{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2017-03-20T15:17:07-06:00\n", "\n", "CPython 3.6.0\n", "IPython 5.1.0\n", "\n", "compiler : GCC 4.4.7 20120313 (Red Hat 4.4.7-1)\n", "system : Linux\n", "release : 4.4.0-66-generic\n", "machine : x86_64\n", "processor : x86_64\n", "CPU cores : 8\n", "interpreter: 64bit\n" ] } ], "source": [ "%load_ext watermark\n", "import warnings\n", "warnings.filterwarnings(\"ignore\") \n", "from IPython.core.display import display, HTML\n", "\n", "import time\n", "\n", "import pandas as pd\n", "import pandas_datareader.data as web\n", "import numpy as np\n", "import scipy.stats as scs\n", "from scipy.stats import multivariate_normal as mvn\n", "import sklearn.mixture as mix\n", "\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "%matplotlib inline\n", "\n", "%watermark" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pandas 0.19.2\n", "numpy 1.12.1\n", "scipy 0.19.0\n", "sklearn 0.18.1\n", "matplotlib 2.0.0\n", "seaborn 0.7.1\n" ] } ], "source": [ "%watermark -p pandas,numpy,scipy,sklearn,matplotlib,seaborn" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### To gain understanding of mixture models we have to start at the beginning with the expectation maximization algorithm and it's application" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### First a little history on E-M\n", "\n", "Reference: 4\n", "\n", "Demptser, Laird & Rubin (1977)\n", " -unified previously unrelated work under \"The EM Algorithm\"\n", " - unified previously unrelated work under \"The EM Algorithm\"\n", " - overlooked E-M works - see gaps between foundational authors\n", " - Newcomb (1887)\n", " - McKendrick (1926) [+39 years]\n", " - Hartley (1958) [+32 years]\n", " - Baum et. al. (1970) [+12 years]\n", " - Dempters et. al. (1977) [+7 years]\n", "\n", "#### *EM Algorithm developed over 90 years*\n", "\n", "## EM provides general framework for solving problems\n", "\n", "Examples include:\n", " - Filling in missing data from a sample set\n", " - Discovering values of latent variables\n", " - Estimating parameters of HMMs\n", " - Estimating parameters of finite mixtures [models]\n", " - Unsupervised learning of clusters\n", " - etc...\n", " \n", "### How does the EM algorithm work? \n", "\n", "EM is an iterative process that begins with a \"naive\" or random initialization and then alternates between the expectation and maximization steps until the algorithm reaches convergence. \n", "\n", "To describe this in words imagine we have a simple data set consisting of class heights with groups separated by gender." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", " | Gender | \n", "Height (in) | \n", "
---|---|---|
0 | \n", "Male | \n", "72 | \n", "
1 | \n", "Male | \n", "72 | \n", "
2 | \n", "Female | \n", "63 | \n", "
3 | \n", "Female | \n", "62 | \n", "
4 | \n", "Female | \n", "62 | \n", "
5 | \n", "Male | \n", "73 | \n", "
6 | \n", "Female | \n", "64 | \n", "
7 | \n", "Female | \n", "63 | \n", "
8 | \n", "Female | \n", "67 | \n", "
9 | \n", "Male | \n", "71 | \n", "
10 | \n", "Male | \n", "72 | \n", "
11 | \n", "Female | \n", "63 | \n", "
12 | \n", "Male | \n", "71 | \n", "
13 | \n", "Female | \n", "67 | \n", "
14 | \n", "Female | \n", "62 | \n", "
15 | \n", "Female | \n", "63 | \n", "
16 | \n", "Male | \n", "66 | \n", "
17 | \n", "Female | \n", "60 | \n", "
18 | \n", "Female | \n", "68 | \n", "
19 | \n", "Female | \n", "65 | \n", "
20 | \n", "Female | \n", "64 | \n", "