Brief Course Description

This is a year-long seminar on several central topics in the general umbrella of Computational Learning Theory. The topics are chosen partly to fit the (research) interests of the instructors. However, they were also chosen to follow two central themes.

  1. (Fall 2010) We explore two foundational questions: "what is learning?" and "what is learnable?" However, unlike the more experimental approach of statistical machine learning, our focus is on formal theoretical model for the prediction capability and algorithmic efficiency, both in terms of time and space. The complexity emphasis is the hallmark of COLT.
  2. (Spring 2011) We explore many interesting connections between learning (on-line learning, in particular), algorithmic complexity, game theory, and electronic markets.
In the Fall of 2010, we will (mostly) follow the lecture notes and topic outline from Rob Schapire's Theoretical Machine Learning course at Princeton, plus lecture notes and materials from Avrim Blum's Machine Learning Theory Course at CMU.

Instructors

  • Hung Q. Ngo ( hungngo [at] buffalo )
    • Office hours: 9-10am, Mondays and Wednesdays, 238 Bell.
  • Atri Rudra ( atri [at] buffalo )
    • Office hours: by appointment

Prerequisites

Basic knowledge of probability theory. (We assume that you have studied some introductory probability course/book before.)

Work Load

Students are expected to participate in class, and make at least one presentation. Instructors will assign the topic and material to be presented. No A/F grade will be given, only S/U grades.

Some reference materials (you're not required to purchase any book):

  • Michael J. Kearns and Umesh V. Vazirani, "An Introduction to Computational Learning Theory", MIT Press.
  • Vapnik, V. N. "The Nature of Statistical Learning Theory". Springer-Verlag New York, Inc.
  • M. Anthony and N. Biggs. "Computational Learning Theory." Cambridge Univ. Press, 1992.
  • O. Bousquet, S. Boucheron, and G. Lugosi, Introduction to Statistical Learning Theory. [ pdf ]
  • N. Cristianini and J. Shawe-Taylor, Kernel Methods for Pattern Analysis, 2004.
  • N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines (and other kernel-based learning methods), 2000, CUP.
  • M. Anthony and P. Bartlett. Learning in Neural Networks : Theoretical Foundations. Cambridge University Press, 1999.
  • Avrim Blum's COLT Tutorial at FOCS 2003. [ ppt slides ]
  • Robert E. Schapire. "The boosting approach to machine learning: An overview." Nonlinear Estimation and Classification. Springer, 2003. [ pdf ]
  • Valiant's original "Theory of the Learnable" paper. [ pdf ]
  • Robert E. Schapire, Yoav Freund, Peter Bartlett and Wee Sun Lee. Boosting the margin: A new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651-1686, 1998. [ pdf ]
  • Prediction, learning, and games by Nicolò Cesa-Bianchi and Gábor Lugosi. Cambridge University Press, 2006
  • Avrim Blum. "On-line algorithms in machine learning." In Dagstuhl Workshop on On-Line Algorithms, June, 1996. [ ps ]
  • Chris Burges' SVM tutorial. [ pdf ]
  • Shai Shalev-Shwartz and Yoram Singer, Tutorial on Theory and Applications of Online Learning, ICML 2008.
  • Kivinen, Warmuth, Exponentiated Gradient versus Gradient Descent for Linear Predictors 1997. [ pdf ].
  • Stephen Della Pietra, Vincent Della Pietra and John Lafferty. Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(4):380-393, April, 1997. [ ps ]