{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Latent Variable Models and Variational Bayes" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "\n", "### Preliminaries\n", "\n", "- Goal \n", " - Introduction to latent variable models and variational inference by Free energy minimization \n", "- Materials\n", " - Mandatory\n", " - These lecture notes\n", " - Ariel Caticha (2010), [Entropic Inference](https://arxiv.org/abs/1011.0723)\n", " - tutorial on entropic inference, which is a generalization to Bayes rule and provides a foundation for variational inference.\n", " - Optional \n", " - Bishop (2016), pp. 461-486 (sections 10.1, 10.2 and 10.3) \n", " - references \n", " - Blei et al. (2017), [Variational Inference: A Review for Statisticians](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773) \n", " - Lanczos (1961), [The variational principles of mechanics](https://www.amazon.com/Variational-Principles-Mechanics-Dover-Physics/dp/0486650677)\n", " - Zhang et al. (2017), [Unifying Message Passing Algorithms Under the Framework of Constrained Bethe Free Energy Minimization](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Zhang-2017-Unifying-Message-Passing-Algorithms.pdf)\n", " - Dauwels (2007), [On variational message passing on factor graphs](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Dauwels-2007-on-variational-message-passing-on-factor-graphs)\n", " - Caticha (2010), [Entropic Inference](https://arxiv.org/abs/1011.0723)\n", " - Shore and Johnson (1980), [Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross-Entropy](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/ShoreJohnson-1980-Axiomatic-Derivation-of-the-Principle-of-Maximum-Entropy.pdf)\n", " " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Illustrative Example : Density Modeling for the Old Faithful Data Set\n", "\n", "- You're now asked to build a density model for a data set ([Old Faithful](https://en.wikipedia.org/wiki/Old_Faithful), Bishop pg. 681) that clearly is not distributed as a single Gaussian:\n", "\n", "