CSE 720

CSE 720: Inference algorithms in PGMs (Spring 2014)

TTh, 9:30-10:50pm,
338A Davis (map)
Computer Science and Engineering
SUNY at Buffalo

Overview, basics:

  • Some video lectures.
  • Yair Weiss and Judea Pearl, Belief Propagation, CACM. [ pdf ]
  • Zoubin Ghahramani, Graphical Models lecture slides [ pdf ]
  • Kevin Murphy, Exact and approximate inference in GMs [ ppt ]
  • M. Jordan, "Graphical models," Statistical Science: Special Issue on Bayesian Statistics, Feb. 2004. [ pdf ]
  • Martin Wainwright lecture slides [ pdf 1, pdf 2, pdf 3 ]
  • Martin Wainwright's lecture notes. [ pdf ]
  • Martin J. Wainwright and Michael I. Jordan, Graphical Models, Exponential Families, and Variational Inference [ pdf ]
  • Bishop, Christopher M (2006). "Chapter 8: Graphical models". Pattern Recognition and Machine Learning. Springer. pp. 359--418.

Hypergraphs, tree-decompositions, various notions of widths:

  • Hung H. Q. Ngo, C. Re, A. Rudra: Skew Strikes Back: New Developments in the Theory of Join Algorithms. [ pdf ]
  • D. Marx, Tractable hypergraph properties for constraint satisfaction and conjunctive queries, Journal of the ACM, 60(6):42, 2013. (Preliminary version in STOC 2010.) [ pdf ]
  • D. Marx, Can you beat treewidth? Theory of Computing, 6(1):85-112, 2010. (Preliminary version in FOCS 2007.) [ pdf ]
  • M. Grohe, D. Marx, Constraint solving via fractional edge covers. To appear in ACM Transactions on Algorithms. [ pdf ]

Bucket elimination, a.k.a. variable elimination, and the SumProd problem:

  • Mahmoud R. Dechter, "Bucket Elimination: A Unifying Framework for Probabilistic Inference" In "Uncertainty in Artificial Intelligence", UA196, 1996, pp. 211-219. [ pdf, pdf2 ]
  • Rish, I., and Dechter, R., "Resolution versus Search: Two Strategies for SAT" In "Journal of Automated Reasoning", special issue on SAT, Volume 24, Issue 1/2, pp. 225-275, Januray, 2000. [ pdf ]
  • Kalev Kask, Rina Dechter, Javier Larrosa and Avi Dechter. "Unifying Cluster-Tree Decompositions for Reasoning in Graphical Models". In Artificial Intelligence Journal, 2005. [ pdf ]
  • Rina Dechter, Lars Otten, and Radu Marinescu. "On the Practical Significance of Hypertree vs. Tree Width." In ECAI'08. [ pdf ]
  • Dechter, R. and Rish, I., "Mini-Buckets: A General Scheme for Bounded Inference" In "Journal of the ACM", Vol. 50, Issue 2: pages 107-153, March 2003. [ pdf ]

Exact inference and model counting:

  • Hung Tian Sang, Paul Beame, Henry A. Kautz: Performing Bayesian Inference by Weighted Model Counting. AAAI 2005: 475-482. [ pdf ]
  • Hung Mark Chavira and Adnan Darwiche. 2008. On probabilistic inference by weighted model counting. Artif. Intell. 172, 6-7 (April 2008), 772-799. [ pdf ]
  • Hung F. Bacchus, S. Dalmao and T. Pitassi (2009) "Solving #SAT and Bayesian Inference with Backtracking Search", JMLR, Volume 34, pages 391-442. [ pdf ]

Belief propagation, a.k.a. Sum-Product Message Passing:

  • Devansh/Mahmoud Frank R. Kschischang, Brendan J. Frey, Hans-Andrea Loeliger: Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory 47(2): 498-519 (2001). [ pdf ]
  • Devansh/Mahmoud Yedidia, J.S.; Freeman, W.T.; Y. (January 2003). "Understanding Belief Propagation and Its Generalizations". In Lakemeyer, Gerhard; Nebel, Bernhard. Exploring Artificial Intelligence in the New Millennium. Morgan Kaufmann. pp. 239--236. [ pdf ]
  • Yu/Tao Yair Weiss and William T. Freeman, On the optimality of solutions of the max-product belief propagation algorithm in arbitrary graphs, IEEE Trans. Info. Theory (2001). [ pdf ]
  • Yu/Tao Weiss Y. and Freeman W.T., Correctness of belief propagation in Gaussian graphical models of arbitrary topology, Neural Computation 13:2173-2200 (2001). [ pdf ]
  • Need volunteer Martin J. Wainwright, Tommi Jaakkola, Alan S. Willsky: MAP estimation via agreement on trees: message-passing and linear programming. IEEE Transactions on Information Theory 51(11): 3697-3717 (2005). [ pdf ]
  • Need volunteer Vladimir Kolmogorov: Convergent Tree-Reweighted Message Passing for Energy Minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28(10): 1568-1583 (2006). [ pdf ]
  • Need volunteer Chen Yanover, Talya Meltzer, Yair Weiss, Linear Programming Relaxations and Belief Propagation - an Empirical Study, JMLR 2006. [ pdf ]
  • Wei Zheng D. Sontag, T. Meltzer, A. Globerson, T. Jaakkola and Y. Weiss, Tightening LP Relaxations for MAP using message passing. [ pdf ]
  • Need volunteer T. Meltzer, A. Globerson, Y. Weiss, Convergent Message Passing Algorithms: a Unifying View, UAI 2009 [ pdf ]
  • Le Yang Weiss Y., Yanover C. and Meltzer T., Linear Programming and Variants of Belief Propagation [ pdf ]
  • Suchismit Chen Yanover, Yair Weiss, Finding the M Most Probable Configurations Using Loopy Belief Propagation. [ pdf ]
  • E. Maneva, E. Mossel and M. J. Wainwright. A New Look at Survey Propagation and its Generalizations. Journal of the ACM, Volume 54(4), July 2007. pp. 2--41. [ pdf ]

Monte-Carlo Methods:

  • Neeti D. Mackay, Introduction to Monte-Carlo Methods, [ pdf ]
  • Chris Bishop, Pattern Recognition and Machine Learning, Chapter 11.
  • Qi (sec 4) Radford Neal, Probabilistic Inference Using Markov Chain Monte Carlo Methods, [ pdf ]
  • Yaliang David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3 (March 2003), 993-1022. [ pdf | C implementation ]

Books: