1. Yi Wu (CMU) Joint work with ParikshitGopalan (MSR SVC) Ryan O’Donnell (CMU) David Zuckerman (UT Austin) Pseudorandom Generators for Halfspaces TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAA

2. Outline Introduction Pseudorandom Generators Halfspaces Pseudorandom Generators for Halfspaces Our Result Proof Conclusion 2

3. Deterministic Algorithm Program Input Output The algorithm deterministically outputs the correct result. 3

4. Randomized Algorithm Program Input Output Random Bits. The algorithm outputs the correct result with high probability. 4

5. Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Randomized Algorithms 5

6. Is Randomness Necessary? Open Problem: Can we simulate every randomized polynomial time algorithm by a deterministic polynomial time algorithm (the “BPP P” cojecture)? Derandomization of randomized algorithms. Primality testing [AKS] ST-connectivity [Reingold] Quadratic residues [?] 6

7. How to generate randomness? Question: How togenerate randomness for every randomized algorithm? Simpler Question: How to generate “pseudorandomness” for some class of programs? 7

8. Pseudorandom Generator (PRG) Both program Answer Yes/No with almost the same probability Yes /No Yes/ No Input Program Input Program n “pseudorandom” bit PRG Quality of the PRG: number of seed n random bit Seed k<<n random bit 8

9. Why study PRGs? Algorithmic Applications When k = log (n), we can derandomize the algorithm in polynomial time. Streaming Algorithm. Complexity Theoretic Implications Lower Bound of Circuit Class. Learning Theory. 9

10. PRG for Classes of Program Space Bounded Program [Nis92] Constant-depth circuits [Nis91, Baz07, Bra09] Halfspaces[DGJSV09, MZ09] 10

11. Outline Introduction Pseudorandom Generators Halfspaces Pseudorandom Generators for Halfspaces Our Result Proof Conclusion 11

13. Widely used in Machine Learning: Perceptron, Winnow, boosting, Support Vector Machines, Lasso, Liner Regression.12

14. Product Distribution For halfspace h(x), x is sampled from some product distribution; i.e., each xi is independently sampled from distribution Di . For example, each Dican be Uniform distribution on {-1,1} Uniform distribution on [-1,1] Gaussian Distribution 13

15. Index Introduction Pseudorandom Generators Halfspaces Pseudorandom Generators for Halfspaces Main Result Proof Conclusion 14

16. PRG for halfspaces Both program Answer Yes/No with almost the same probability Yes/No Yes/No h(x) = sign(w1x1+…+wnxn-θ) h(x) = sign(w1x1+…+wnxn-θ) Pseudorandom Variable x1, x2 …xn PRG x1, x2 …xnfrom some product distribution k<<n random bit 15

17. Geometric Interpretation, PRG for uniform distribution over [-1,1]2 16

18. Geometric Interpretation, PRG for uniform distribution over [-1,1]2 Total Number of points = poly(dim) Number of points in the halfspace is proportional to area. 17

19. Application to Machine Learning - - - - - - + + + - - + + + - + + - - + + + + How many testing points is it enough to estimate the accuracy of the N dimensional linear classifier? Good PRG implies we only need deterministically check the accuracy on a set of poly(N) points! 18

20. Other Theoretical Applications Discrepancy Set for Convex Polytopes Circuit Lower bound on functions of halfspaces Counting the Solution of Knapsacks 19

21. Outline Introduction Pseudorandom Generator Halfspace Pseudorandom Generators for Halfspaces Our Results Proof Conclusion 20

22. Previous Result [DiGoJaSeVi,MeZu] PRG For Halfspace over uniform distribution on boolean cube ({-1,1}n) with seed length O(log n). 21

23. Our Results:Arbitrary Product Distributions PRG for halfspaces under arbitrary product distribution over Rnwith the same seed length. Only requirement: E[xi4] is a constant. Gaussian Distribution Uniform distribution on the solid cube. Uniform distribution on the hypercube. Biased distribution on the hypercube. Almost any “natural distribution” 22

24. Our Results Functions of k-Halfspaces PRG for the intersections of k-halfspaces with seed length k log (n). PRG for arbitrary functions of k-halfspaces with seed length k2 log (n). 23

25. Outline Introduction Pseudorandom Generator Halfspace Pseudorandom Generators for Halfspaces Our Result Proof Conclusion 24

26. Key Observation: Dichotomy of Halfspaces Under product distributions , every halfspace is close to one of the following: “Dictator” (halfspaces depending on very few variables, e.g. f(x) = sgn(x1)) “majority”(no variables has too much weight, e.g. f(x) = sgn(x1+x2+x3+…+xn). 25

27. Dichotomy of weight distribution Weights decreasing fast (Geometrically) Weights are stable after certain index. 26

28. Weights Decrease fast (Geometrically) Intuition: for sign(2n x1 + 2n-1 x2+ 2n-2 x3 +…xn) If each xi is from {-1,1} , it is just sign(x1). 27

29. Weights are stable Intuition: for sign(100 x1 + x2 + x3+…xn) Then by for every fixing of x1, it is a majority on the rest of the variables. 28

30. Our PRG for Halfspace (Rough) Randomly hashing all the coordinate into groups. Use 4-wise independent distribution within each group. If it is “Dictator-like”: All the important variables are in different groups. If it is “Majority-like” (x1 + x2 +.. xn ) is close to Gaussian. 4-wise independent Distribution (somehow) can handle Gaussian. 29

31. Outline Introduction Pseudorandom Generator Halfspace Pseudorandom Generators for Halfspaces Our Result Proof Conclusion 30

32. Conclusion We construct PRG for halfspaces under arbitrary product distribution and functions of k halfspaces with small seed length. Future Work Building PRG for larger classes of program; e.g., Polynomial Threshold function (SVM with polynomial kernel). 31

33. 32