Introduction: A Brief Tour of Concepts, Techniques, and Applications (5 weeks)

  1. [ pdf ] Discrete Probability Space: Events, The Probabilistic Method, and the Union Bound
  2. [ pdf ] Expectation: definition, linearity of expectation, decomposition into indicator variables, the argument from expectation.
  3. [ pdf ] (Conditional) Probability and Expectation. Independence and Conditional Probability, Random Variables, Expectation and Conditional Expectation, Law of Total Probability, Law of Total Expectation, Randomized Algorithms, Derandomization Using Conditional Expectation.
  4. [ pdf ] The Second Moment: Sampling, (Co)Variance, Moments and Deviation, the Second Moment Method, Basic Concentration/Tail Inequalities (Markov, Chebyshev, Chernoff).
  5. [ pdf ] Tail and Concentration Inequalities: Markov, Chebyshev, Chernoff, Chernoff-Hoefding. The Bernstein moment generating function technique.

Computational Learning Theory (10 weeks)

  1. [ pdf ] Introduction to Learning Theory, The Consistency Model, PAC learning
      [ presentation candidate ] [ pdf ] Pitt, L. and Valiant, L. G. 1988. Computational limitations on learning from examples. J. ACM 35, 4 (Oct. 1988), 965-984.
  2. [ pdf ] Sample complexity and Occam's razor. Sample complexity for finite hypothesis classes.
  3. [ pdf ] VC-dimension and Sauer-Shelah Lemma
  4. [ pdf ] Sample complexity for infinite hypothesis classes. Vapnik-Chervonenkis theorem. The double sampling trick. Lower bound on the sample complexity through VC-dimension
  5. [ pdf ] Dealing with Noises. Inconsistent Hypothesis Model. Empirical error and Generalization error. Empirical loss minimization strategy. Estimation error and approximation error. Uniform convergence theorem.
  6. [ pdf ] Rademacher complexity and the uniform convergence theorem.
  7. [ pdf ] Weak and Strong PAC-learning. Boosting and AdaBoost, training error bound, generalization error bound: naive and margin-based. General margin bound.
  8. A brief introduction to optimization.
  9. Support Vector Machines.
  10. [ No time for this ] Online learning. The mistake-bound model. Learning from expert advices. WMA and RWMA. The Hedging algorithm.