![](\"../slides/diagrams/turing-times.gif\")
\n",
"![](\"../slides/diagrams/turing-run.jpg\")
\n",
"| \n", "- Gold medal times for Olympic Marathon since 1896.\n", "\n", "- Marathons before 1924 didn’t have a standardised distance.\n", "\n", "- Present results using pace per km.\n", "\n", "- In 1904 Marathon was badly organised leading to very slow times.\n", "\n", " | \n", "\n",
" Image from\n",
"Wikimedia Commons | \n",
"
![](\"../slides/diagrams/datasets/olympic-marathon.svg\")
\n",
"\n",
"### Regression: Linear Releationship\n",
"\n",
"$$\\dataScalar_i = m \\inputScalar_i + c$$\n",
"\n",
"- $\\dataScalar_i$ : winning pace.\n",
"\n",
"- $\\inputScalar_i$ : year of Olympics.\n",
"\n",
"- $m$ : rate of improvement over time.\n",
"\n",
"- $c$ : winning time at year 0.\n",
"\n",
"## Overdetermined System\n",
"\n",
"###"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.over_determined_system(diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from ipywidgets import IntSlider\n",
"import pods"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('over_determined_system{samp:0>3}.svg',\n",
" directory='../slides/diagrams/ml', \n",
" samp=IntSlider(1,1,8,1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{over\\_determined\\_system}{1}{8}\n",
"![](\"../slides/diagrams/ml/over_determined_system001.svg\")
\n",
"![](\"../slides/diagrams/ml/over_determined_system002.svg\")
\n",
"![](\"../slides/diagrams/ml/over_determined_system003.svg\")
\n",
"![](\"../slides/diagrams/ml/over_determined_system004.svg\")
\n",
"![](\"../slides/diagrams/ml/over_determined_system005.svg\")
\n",
"![](\"../slides/diagrams/ml/over_determined_system006.svg\")
\n",
"![](\"../slides/diagrams/ml/over_determined_system007.svg\")
\n",
"\n",
"### $\\dataScalar = m\\inputScalar + c$\n",
"\n",
". . .\n",
"\n",
"point 1: $\\inputScalar = 1$, $\\dataScalar=3$ $$\n",
"3 = m + c\n",
"$$\n",
"\n",
". . .\n",
"\n",
"point 2: $\\inputScalar = 3$, $\\dataScalar=1$ $$\n",
"1 = 3m + c\n",
"$$\n",
"\n",
". . .\n",
"\n",
"point 3: $\\inputScalar = 2$, $\\dataScalar=2.5$\n",
"\n",
"$$2.5 = 2m + c$$\n",
"\n",
"### \n",
"\n",
"![](\"../slides/diagrams/ml/Pierre-Simon_Laplace.png\")
\n",
"\n",
"### \n",
"\n",
"![](\"../slides/diagrams/laplacesDeterminismFrench.png\")
\n",
"\n",
"### \n",
"\n",
"![](\"../slides/diagrams/laplacesDeterminismEnglish.png\")
\n",
"\n",
"### \n",
"\n",
"![](\"../slides/diagrams/philosophicaless00lapliala.png\")
\n",
"\n",
"### $\\dataScalar = m\\inputScalar + c + \\noiseScalar$\n",
"\n",
". . .\n",
"\n",
"point 1: $\\inputScalar = 1$, $\\dataScalar=3$ $$\n",
"3 = m + c + \\noiseScalar_1\n",
"$$\n",
"\n",
". . .\n",
"\n",
"point 2: $\\inputScalar = 3$, $\\dataScalar=1$ $$\n",
"1 = 3m + c + \\noiseScalar_2\n",
"$$\n",
"\n",
". . .\n",
"\n",
"point 3: $\\inputScalar = 2$, $\\dataScalar=2.5$ $$\n",
"2.5 = 2m + c + \\noiseScalar_3\n",
"$$\n",
"\n",
"### A Probabilistic Process\n",
"\n",
". . .\n",
"\n",
"Set the mean of Gaussian to be a function. $$p\n",
"\\left(\\dataScalar_i|\\inputScalar_i\\right)=\\frac{1}{\\sqrt{2\\pi\\dataStd^2}}\\exp \\left(-\\frac{\\left(\\dataScalar_i-\\mappingFunction\\left(\\inputScalar_i\\right)\\right)^{2}}{2\\dataStd^2}\\right).\n",
"$$\n",
"\n",
". . .\n",
"\n",
"This gives us a 'noisy function'.\n",
"\n",
". . .\n",
"\n",
"This is known as a stochastic process.\n",
"\n",
"### The Gaussian Density\n",
"\n",
"- Perhaps the most common probability density.\n",
"\n",
". . .\n",
"\n",
"$$\\begin{align}\n",
" p(\\dataScalar| \\meanScalar, \\dataStd^2) & = \\frac{1}{\\sqrt{2\\pi\\dataStd^2}}\\exp\\left(-\\frac{(\\dataScalar - \\meanScalar)^2}{2\\dataStd^2}\\right)\\\\& \\buildrel\\triangle\\over = \\gaussianDist{\\dataScalar}{\\meanScalar}{\\dataStd^2}\n",
" \\end{align}$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.gaussian_of_height(diagrams='../../slides/diagrams/ml')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Gaussian Density\n",
"\n",
"![](\"../slides/diagrams/ml/gaussian_of_height.svg\")
\n",
"\n",
"![](\"../slides/diagrams/ml/regression.svg\")
\n",
"\n",
"These points lie exactly on a straight line, that's not very realistic,\n",
"let's corrupt them with a bit of Gaussian 'noise'.\n",
"\n",
"### Noise Corrupted Plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"noise = np.random.normal(scale=0.5, size=4) # standard deviation of the noise is 0.5\n",
"y = m_true*x + c_true + noise\n",
"plt.plot(x, y, 'r.', markersize=10)\n",
"plt.xlim([-3, 3])\n",
"mlai.write_figure(filename=\"../slides/diagrams/ml/regression_noise.svg\", transparent=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"../slides/diagrams/ml/regression_noise.svg\")
\n",
"\n",
"### Contour Plot of Error Function\n",
"\n",
"- Visualise the error function surface, create vectors of values."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# create an array of linearly separated values around m_true\n",
"m_vals = np.linspace(m_true-3, m_true+3, 100) \n",
"# create an array of linearly separated values ae\n",
"c_vals = np.linspace(c_true-3, c_true+3, 100)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- create a grid of values to evaluate the error function in 2D."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m_grid, c_grid = np.meshgrid(m_vals, c_vals)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- compute the error function at each combination of $c$ and $m$."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"E_grid = np.zeros((100, 100))\n",
"for i in range(100):\n",
" for j in range(100):\n",
" E_grid[i, j] = ((y - m_grid[i, j]*x - c_grid[i, j])**2).sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Contour Plot of Error\n",
"\n",
"- We can now make a contour plot."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%load -s regression_contour teaching_plots.py"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f, ax = plt.subplots(figsize=(5,5))\n",
"regression_contour(f, ax, m_vals, c_vals, E_grid)\n",
"mlai.write_figure(filename='../slides/diagrams/ml/regression_contour.svg')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"../slides/diagrams/ml/regression_contour.svg\")
\n",
"\n",
"### Steepest Descent\n",
"\n",
"- Minimize the sum of squares error function.\n",
"- One way of doing that is gradient descent.\n",
"- Initialize with a guess for $m$ and $c$\n",
"- update that guess by subtracting a portion of the gradient from the\n",
" guess.\n",
"- Like walking down a hill in the steepest direction of the hill to\n",
" get to the bottom.\n",
"\n",
"### Algorithm\n",
"\n",
"- We start with a guess for $m$ and $c$."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m_star = 0.0\n",
"c_star = -5.0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Offset Gradient\n",
"\n",
"- Now we need to compute the gradient of the error function, firstly\n",
" with respect to $c$,\n",
"\n",
"$$\\frac{\\text{d}\\errorFunction(m, c)}{\\text{d} c} =\n",
"-2\\sum_{i=1}^\\numData (\\dataScalar_i - m\\inputScalar_i - c)$$\n",
"\n",
"- This is computed in python as follows"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"c_grad = -2*(y-m_star*x - c_star).sum()\n",
"print(\"Gradient with respect to c is \", c_grad)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Deriving the Gradient\n",
"\n",
"To see how the gradient was derived, first note that the $c$ appears in\n",
"every term in the sum. So we are just differentiating\n",
"$(\\dataScalar_i - m\\inputScalar_i - c)^2$ for each term in the sum. The\n",
"gradient of this term with respect to $c$ is simply the gradient of the\n",
"outer quadratic, multiplied by the gradient with respect to $c$ of the\n",
"part inside the quadratic. The gradient of a quadratic is two times the\n",
"argument of the quadratic, and the gradient of the inside linear term is\n",
"just minus one. This is true for all terms in the sum, so we are left\n",
"with the sum in the gradient.\n",
"\n",
"### Slope Gradient\n",
"\n",
"The gradient with respect tom $m$ is similar, but now the gradient of\n",
"the quadratic's argument is $-\\inputScalar_i$ so the gradient with\n",
"respect to $m$ is\n",
"\n",
"$$\\frac{\\text{d}\\errorFunction(m, c)}{\\text{d} m} = -2\\sum_{i=1}^\\numData \\inputScalar_i(\\dataScalar_i - m\\inputScalar_i -\n",
"c)$$\n",
"\n",
"which can be implemented in python (numpy) as"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m_grad = -2*(x*(y-m_star*x - c_star)).sum()\n",
"print(\"Gradient with respect to m is \", m_grad)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Update Equations\n",
"\n",
"- Now we have gradients with respect to $m$ and $c$.\n",
"- Can update our inital guesses for $m$ and $c$ using the gradient.\n",
"- We don't want to just subtract the gradient from $m$ and $c$,\n",
"- We need to take a *small* step in the gradient direction.\n",
"- Otherwise we might overshoot the minimum.\n",
"- We want to follow the gradient to get to the minimum, the gradient\n",
" changes all the time.\n",
"\n",
"### Move in Direction of Gradient"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f, ax = plt.subplots(figsize=plot.big_figsize)\n",
"plot.regression_contour(f, ax, m_vals, c_vals, E_grid)\n",
"ax.plot(m_star, c_star, 'g*', markersize=20)\n",
"ax.arrow(m_star, c_star, -m_grad*0.1, -c_grad*0.1, head_width=0.2)\n",
"mlai.write_figure(filename='../slides/diagrams/ml/regression_contour_step001.svg', transparent=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"../slides/diagrams/ml/regression_contour_step001.svg\")
\n",
"\n",
"### Update Equations\n",
"\n",
"- The step size has already been introduced, it's again known as the\n",
" learning rate and is denoted by $\\learnRate$. $$\n",
" c_\\text{new}\\leftarrow c_{\\text{old}} - \\learnRate \\frac{\\text{d}\\errorFunction(m, c)}{\\text{d}c}\n",
" $$\n",
"\n",
"- gives us an update for our estimate of $c$ (which in the code we've\n",
" been calling `c_star` to represent a common way of writing a\n",
" parameter estimate, $c^*$) and $$\n",
" m_\\text{new} \\leftarrow m_{\\text{old}} - \\learnRate \\frac{\\text{d}\\errorFunction(m, c)}{\\text{d}m}\n",
" $$\n",
"- Giving us an update for $m$.\n",
"\n",
"### Update Code\n",
"\n",
"- These updates can be coded as"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"Original m was\", m_star, \"and original c was\", c_star)\n",
"learn_rate = 0.01\n",
"c_star = c_star - learn_rate*c_grad\n",
"m_star = m_star - learn_rate*m_grad\n",
"print(\"New m is\", m_star, \"and new c is\", c_star)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Iterating Updates\n",
"\n",
"- Fit model by descending gradient.\n",
"\n",
"### Gradient Descent Algorithm"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_plots = plot.regression_contour_fit(x, y, diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('regression_contour_fit{num:0>3}.svg', directory='../slides/diagrams/ml', num=IntSlider(0, 0, num_plots, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{regression\\_contour\\_fit}{1}{28}\n",
"![](\"../slides/diagrams/ml/regression_contour_fit000.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit001.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit002.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit003.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit004.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit005.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit006.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit007.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit008.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit009.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit010.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit011.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit012.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit013.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit014.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit015.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit016.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit017.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit018.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit019.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit020.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit021.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit022.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit023.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit024.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit025.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit026.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit027.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_contour_fit028.svg\")
\n",
"\n",
"### Stochastic Gradient Descent\n",
"\n",
"- If $\\numData$ is small, gradient descent is fine.\n",
"- But sometimes (e.g. on the internet $\\numData$ could be a billion.\n",
"- Stochastic gradient descent is more similar to perceptron.\n",
"- Look at gradient of one data point at a time rather than summing\n",
" across *all* data points)\n",
"- This gives a stochastic estimate of gradient.\n",
"\n",
"### Stochastic Gradient Descent\n",
"\n",
"- The real gradient with respect to $m$ is given by\n",
"\n",
"$$\\frac{\\text{d}\\errorFunction(m, c)}{\\text{d} m} = -2\\sum_{i=1}^\\numData \\inputScalar_i(\\dataScalar_i -\n",
"m\\inputScalar_i - c)$$\n",
"\n",
"but it has $\\numData$ terms in the sum. Substituting in the gradient we\n",
"can see that the full update is of the form\n",
"\n",
"$$m_\\text{new} \\leftarrow\n",
"m_\\text{old} + 2\\learnRate \\left[\\inputScalar_1 (\\dataScalar_1 - m_\\text{old}\\inputScalar_1 - c_\\text{old}) + (\\inputScalar_2 (\\dataScalar_2 - m_\\text{old}\\inputScalar_2 - c_\\text{old}) + \\dots + (\\inputScalar_n (\\dataScalar_n - m_\\text{old}\\inputScalar_n - c_\\text{old})\\right]$$\n",
"\n",
"This could be split up into lots of individual updates\n",
"$$m_1 \\leftarrow m_\\text{old} + 2\\learnRate \\left[\\inputScalar_1 (\\dataScalar_1 - m_\\text{old}\\inputScalar_1 -\n",
"c_\\text{old})\\right]$$\n",
"$$m_2 \\leftarrow m_1 + 2\\learnRate \\left[\\inputScalar_2 (\\dataScalar_2 -\n",
"m_\\text{old}\\inputScalar_2 - c_\\text{old})\\right]$$\n",
"$$m_3 \\leftarrow m_2 + 2\\learnRate\n",
"\\left[\\dots\\right]$$\n",
"$$m_n \\leftarrow m_{n-1} + 2\\learnRate \\left[\\inputScalar_n (\\dataScalar_n -\n",
"m_\\text{old}\\inputScalar_n - c_\\text{old})\\right]$$\n",
"\n",
"which would lead to the same final update.\n",
"\n",
"### Updating $c$ and $m$\n",
"\n",
"- In the sum we don't $m$ and $c$ we use for computing the gradient\n",
" term at each update.\n",
"- In stochastic gradient descent we *do* change them.\n",
"- This means it's not quite the same as steepest desceint.\n",
"- But we can present each data point in a random order, like we did\n",
" for the perceptron.\n",
"- This makes the algorithm suitable for large scale web use (recently\n",
" this domain is know as 'Big Data') and algorithms like this are\n",
" widely used by Google, Microsoft, Amazon, Twitter and Facebook.\n",
"\n",
"### Stochastic Gradient Descent\n",
"\n",
"- Or more accurate, since the data is normally presented in a random\n",
" order we just can write $$\n",
" m_\\text{new} = m_\\text{old} + 2\\learnRate\\left[\\inputScalar_i (\\dataScalar_i - m_\\text{old}\\inputScalar_i - c_\\text{old})\\right]\n",
" $$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# choose a random point for the update \n",
"i = np.random.randint(x.shape[0]-1)\n",
"# update m\n",
"m_star = m_star + 2*learn_rate*(x[i]*(y[i]-m_star*x[i] - c_star))\n",
"# update c\n",
"c_star = c_star + 2*learn_rate*(y[i]-m_star*x[i] - c_star)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### SGD for Linear Regression\n",
"\n",
"Putting it all together in an algorithm, we can do stochastic gradient\n",
"descent for our regression data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_plots = plot.regression_contour_sgd(x, y, diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('regression_sgd_contour_fit{num:0>3}.svg', \n",
" directory='../slides/diagrams/mlai', num=IntSlider(0, 0, num_plots, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{regression\\_sgd\\_contour\\_fit}{0}{58}\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit000.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit001.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit002.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit003.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit004.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit005.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit006.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit007.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit008.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit009.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit010.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit011.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit012.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit013.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit014.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit015.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit016.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit017.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit018.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit019.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit020.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit021.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit022.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit023.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit024.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit025.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit026.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit027.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit028.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit029.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit030.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit031.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit032.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit033.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit034.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit035.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit036.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit037.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit038.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit039.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit040.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit041.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit042.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit043.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit044.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit045.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit046.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit047.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit048.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit049.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit050.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit051.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit052.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit053.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit054.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit055.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit056.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit057.svg\")
\n",
"![](\"../slides/diagrams/ml/regression_sgd_contour_fit058.svg\")
\n",
"\n",
"### Reflection on Linear Regression and Supervised Learning\n",
"\n",
"Think about:\n",
"\n",
"1. What effect does the learning rate have in the optimization? What's\n",
" the effect of making it too small, what's the effect of making it\n",
" too big? Do you get the same result for both stochastic and steepest\n",
" gradient descent?\n",
"\n",
"2. The stochastic gradient descent doesn't help very much for such a\n",
" small data set. It's real advantage comes when there are many,\n",
" you'll see this in the lab.\n",
"\n",
"### Log Likelihood for Multivariate Regression\n",
"\n",
"The likelihood of a single data point is\n",
"\n",
". . .\n",
"\n",
"$$p\\left(\\dataScalar_i|\\inputScalar_i\\right)=\\frac{1}{\\sqrt{2\\pi\\dataStd^2}}\\exp\\left(-\\frac{\\left(\\dataScalar_i-\\mappingVector^{\\top}\\inputVector_i\\right)^{2}}{2\\dataStd^2}\\right).$$\n",
"\n",
". . .\n",
"\n",
"Leading to a log likelihood for the data set of\n",
"\n",
". . .\n",
"\n",
"$$L(\\mappingVector,\\dataStd^2)= -\\frac{\\numData}{2}\\log \\dataStd^2-\\frac{\\numData}{2}\\log 2\\pi -\\frac{\\sum_{i=1}^{\\numData}\\left(\\dataScalar_i-\\mappingVector^{\\top}\\inputVector_i\\right)^{2}}{2\\dataStd^2}.$$\n",
"\n",
"### Error Function\n",
"\n",
"And a corresponding error function of\n",
"$$\\errorFunction(\\mappingVector,\\dataStd^2)=\\frac{\\numData}{2}\\log\\dataStd^2 + \\frac{\\sum_{i=1}^{\\numData}\\left(\\dataScalar_i-\\mappingVector^{\\top}\\inputVector_i\\right)^{2}}{2\\dataStd^2}.$$\n",
"\n",
"### Expand the Brackets\n",
"\n",
"$$\n",
"\\begin{align*}\n",
" \\errorFunction(\\mappingVector,\\dataStd^2) = &\n",
"\\frac{\\numData}{2}\\log \\dataStd^2 + \\frac{1}{2\\dataStd^2}\\sum\n",
"_{i=1}^{\\numData}\\dataScalar_i^{2}-\\frac{1}{\\dataStd^2}\\sum\n",
"_{i=1}^{\\numData}\\dataScalar_i\\mappingVector^{\\top}\\inputVector_i\\\\&+\\frac{1}{2\\dataStd^2}\\sum\n",
"_{i=1}^{\\numData}\\mappingVector^{\\top}\\inputVector_i\\inputVector_i^{\\top}\\mappingVector\n",
"+\\text{const}.\\\\\n",
" = & \\frac{\\numData}{2}\\log \\dataStd^2 + \\frac{1}{2\\dataStd^2}\\sum\n",
"_{i=1}^{\\numData}\\dataScalar_i^{2}-\\frac{1}{\\dataStd^2}\n",
"\\mappingVector^\\top\\sum_{i=1}^{\\numData}\\inputVector_i\\dataScalar_i\\\\&+\\frac{1}{2\\dataStd^2}\n",
"\\mappingVector^{\\top}\\left[\\sum\n",
"_{i=1}^{\\numData}\\inputVector_i\\inputVector_i^{\\top}\\right]\\mappingVector +\\text{const}.\n",
"\\end{align*}\n",
"$$\n",
"\n",
"## Multiple Input Solution with Linear Algebra"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# define the vector w\n",
"w = np.zeros(shape=(2, 1))\n",
"w[0] = m\n",
"w[1] = c"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Design Matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X = np.hstack((np.ones_like(x), x))\n",
"print(X)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f = np.dot(X, w) # np.dot does matrix multiplication in python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"resid = (y-f)\n",
"E = np.dot(resid.T, resid) # matrix multiplication on a single vector is equivalent to a dot product.\n",
"print(\"Error function is:\", E)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Objective Optimisation\n",
"\n",
"### Multivariate Derivatives\n",
"\n",
"- We will need some multivariate calculus.\n",
"- For now some simple multivariate differentiation:\n",
" $$\\frac{\\text{d}{\\mathbf{a}^{\\top}}{\\mappingVector}}{\\text{d}\\mappingVector}=\\mathbf{a}$$\n",
" and\n",
" $$\\frac{\\mappingVector^{\\top}\\mathbf{A}\\mappingVector}{\\text{d}\\mappingVector}=\\left(\\mathbf{A}+\\mathbf{A}^{\\top}\\right)\\mappingVector$$\n",
" or if $\\mathbf{A}$ is symmetric (*i.e.*\n",
" $\\mathbf{A}=\\mathbf{A}^{\\top}$)\n",
" $$\\frac{\\text{d}\\mappingVector^{\\top}\\mathbf{A}\\mappingVector}{\\text{d}\\mappingVector}=2\\mathbf{A}\\mappingVector.$$\n",
"\n",
"### Differentiate the Objective\n",
"\n",
"[Differentiating with respect to the vector $\\mappingVector$ we\n",
"obtain]{align=\"left\"} $$\n",
"\\frac{\\partial L\\left(\\mappingVector,\\dataStd^2 \\right)}{\\partial\n",
"\\mappingVector}=\\frac{1}{\\dataStd^2} \\sum _{i=1}^{\\numData}\\inputVector_i \\dataScalar_i-\\frac{1}{\\dataStd^2}\n",
"\\left[\\sum _{i=1}^{\\numData}\\inputVector_i\\inputVector_i^{\\top}\\right]\\mappingVector\n",
"$$ Leading to $$\n",
"\\mappingVector^{*}=\\left[\\sum\n",
"_{i=1}^{\\numData}\\inputVector_i\\inputVector_i^{\\top}\\right]^{-1}\\sum\n",
"_{i=1}^{\\numData}\\inputVector_i\\dataScalar_i,\n",
"$$\n",
"\n",
"### Differentiate the Objective\n",
"\n",
"Rewrite in matrix notation: $$\n",
"\\sum_{i=1}^{\\numData}\\inputVector_i\\inputVector_i^\\top = \\inputMatrix^\\top \\inputMatrix\n",
"$$ $$\n",
"\\sum_{i=1}^{\\numData}\\inputVector_i\\dataScalar_i = \\inputMatrix^\\top \\dataVector\n",
"$$\n",
"\n",
"## Update Equation for Global Optimum\n",
"\n",
"### Update Equations\n",
"\n",
"- Update for $\\mappingVector^{*}$.\n",
" $$\\mappingVector^{*} = \\left(\\inputMatrix^\\top \\inputMatrix\\right)^{-1} \\inputMatrix^\\top \\dataVector$$\n",
"- The equation for $\\left.\\dataStd^2\\right.^{*}$ may also be found\n",
" $$\\left.\\dataStd^2\\right.^{{*}}=\\frac{\\sum_{i=1}^{\\numData}\\left(\\dataScalar_i-\\left.\\mappingVector^{*}\\right.^{\\top}\\inputVector_i\\right)^{2}}{\\numData}.$$\n",
"\n",
"### Solving the Multivariate System"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.linalg.solve?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"w = np.linalg.solve(np.dot(X.T, X), np.dot(X.T, y))\n",
"print(w)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m = w[1]; c=w[0]\n",
"f_test = m*x_test + c\n",
"print(m)\n",
"print(c)\n",
"plt.plot(x_test, f_test, 'b-')\n",
"plt.plot(x, y, 'rx')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = pods.datasets.movie_body_count()\n",
"movies = data['Y']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(', '.join(movies.columns))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"select_features = ['Year', 'Body_Count', 'Length_Minutes']\n",
"X = movies[select_features]\n",
"X['Eins'] = 1 # add a column for the offset\n",
"y = movies[['IMDB_Rating']]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"w = pd.DataFrame(data=np.linalg.solve(np.dot(X.T, X), np.dot(X.T, y)), # solve linear regression here\n",
" index = X.columns, # columns of X become rows of w\n",
" columns=['regression_coefficient']) # the column of X is the value of regression coefficient"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(y - np.dot(X, w)).hist()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"w"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import scipy as sp\n",
"Q, R = np.linalg.qr(X)\n",
"w = sp.linalg.solve_triangular(R, np.dot(Q.T, y)) \n",
"w = pd.DataFrame(w, index=X.columns)\n",
"w"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reading\n",
"\n",
"- Section 1.3 of @Rogers:book11 for Matrix & Vector Review.\n",
"\n",
"### Basis Functions\n",
"\n",
"### Quadratic Basis\n",
"\n",
"- Basis functions can be global. E.g. quadratic basis: $$\n",
" \\basisVector = [1, \\inputScalar, \\inputScalar^2]\n",
" $$\n",
"\n",
"$$\n",
"\\begin{align*}\n",
"\\basisFunc_1(\\inputScalar) = 1, \\\\\n",
"\\basisFunc_2(\\inputScalar) = x, \\\\\n",
"\\basisFunc_3(\\inputScalar) = \\inputScalar^2.\n",
"\\end{align*}\n",
"$$\n",
"\n",
"$$\n",
"\\basisVector(\\inputScalar) = \\begin{bmatrix} 1\\\\ x \\\\ \\inputScalar^2\\end{bmatrix}.\n",
"$$\n",
"\n",
"### Matrix Valued Function\n",
"\n",
"$$\n",
"\\basisMatrix(\\inputVector) = \n",
"\\begin{bmatrix} 1 & \\inputScalar_1 &\n",
"\\inputScalar_1^2 \\\\\n",
"1 & \\inputScalar_2 & \\inputScalar_2^2\\\\\n",
"\\vdots & \\vdots & \\vdots \\\\\n",
"1 & \\inputScalar_n & \\inputScalar_n^2\n",
"\\end{bmatrix}\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def quadratic(x, **kwargs):\n",
" \"\"\"Take in a vector of input values and return the design matrix associated \n",
" with the basis functions.\"\"\"\n",
" return np.hstack([np.ones((x.shape[0], 1)), x, x**2])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Functions Derived from Quadratic Basis\n",
"\n",
"$$\n",
"\\mappingFunction(\\inputScalar) = {\\color{cyan}\\mappingScalar_0} + {\\color{green}\\mappingScalar_1 \\inputScalar} + {\\color{yellow}\\mappingScalar_2 \\inputScalar^2}\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f, ax = plt.subplots(figsize=plot.big_wide_figsize)\n",
"loc =[[0, 1.4,],\n",
" [0, -0.7],\n",
" [0.75, -0.2]]\n",
"text =['$\\phi(x) = 1$',\n",
" '$\\phi(x) = x$',\n",
" '$\\phi(x) = x^2$']\n",
"\n",
"plot.basis(quadratic, x_min=-1.3, x_max=1.3, \n",
" fig=f, ax=ax, loc=loc, text=text,\n",
" diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{quadratic\\_basis}{0}{2}\n",
"![](\"../slides/diagrams/ml/quadratic_basis000.svg\")
\n",
"![](\"../slides/diagrams/ml/quadratic_basis001.svg\")
\n",
"![](\"../slides/diagrams/ml/quadratic_basis002.svg\")
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('quadratic_basis{num_basis:0>3}.svg', \n",
" directory='../slides/diagrams/ml', \n",
" num_basis=IntSlider(0,0,2,1))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# first let's generate some inputs\n",
"n = 100\n",
"x = np.zeros((n, 1)) # create a data set of zeros\n",
"x[:, 0] = np.linspace(-1, 1, n) # fill it with values between -1 and 1\n",
"\n",
"Phi = quadratic(x)\n",
"\n",
"fig, ax = plt.subplots(figsize=plot.big_wide_figsize)\n",
"ax.set_ylim([-1.2, 1.2]) # set y limits to ensure basis functions show.\n",
"ax.plot(x[:,0], Phi[:, 0], 'r-', label = '$\\phi=1$', linewidth=3)\n",
"ax.plot(x[:,0], Phi[:, 1], 'g-', label = '$\\phi=x$', linewidth=3)\n",
"ax.plot(x[:,0], Phi[:, 2], 'b-', label = '$\\phi=x^2$', linewidth=3)\n",
"ax.legend(loc='lower right')\n",
"_ = ax.set_title('Quadratic Basis Functions')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Quadratic Functions\n",
"\n",
"\\\\startslides{quadratic\\_function}{0}{2}\n",
"![](\"../slides/diagrams/ml/quadratic_function000.svg\")
\n",
"![](\"../slides/diagrams/ml/quadratic_function001.svg\")
\n",
"![](\"../slides/diagrams/ml/quadratic_function002.svg\")
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('quadratic_function{num_function:0>3}.svg', \n",
" directory='../slides/diagrams/ml', \n",
" num_basis=IntSlider(0,0,2,1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Polynomial Fits to Olympic Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from matplotlib import pyplot as plt\n",
"import teaching_plots as plot\n",
"import mlai\n",
"import pods"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"basis = mlai.polynomial\n",
"\n",
"data = pods.datasets.olympic_marathon_men()\n",
"\n",
"x = data['X']\n",
"y = data['Y']\n",
"\n",
"xlim = [1892, 2020]\n",
"\n",
"\n",
"basis=mlai.basis(mlai.polynomial, number=1, data_limits=xlim)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.rmse_fit(x, y, param_name='number', param_range=(1, 27), \n",
" model=mlai.LM, \n",
" basis=basis,\n",
" xlim=xlim, objective_ylim=[0, 0.8],\n",
" diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('olympic_LM_polynomial_num_basis{num_basis:0>3}.svg',\n",
" directory='../slides/diagrams/ml', \n",
" num_basis=IntSlider(1,1,27,1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{olympic\\_LM\\_polynomial\\_num\\_basis}{1}{26}\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis001.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis002.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis003.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis004.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis005.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis006.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis007.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis008.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis009.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis010.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis011.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis012.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis013.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis014.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis015.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis016.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis017.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis018.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis019.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis020.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis021.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis022.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis023.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis024.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis025.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_LM_polynomial_num_basis026.svg\")
\n",
"\n",
"## Underdetermined System"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.under_determined_system(diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Underdetermined System\n",
"\n",
"- What about two unknowns and *one* observation?\n",
" $$\\dataScalar_1 = m\\inputScalar_1 + c$$\n",
"\n",
". . .\n",
"\n",
"Can compute $m$ given $c$.\n",
"$$m = \\frac{\\dataScalar_1 - c}{\\inputScalar}$$\n",
"\n",
"### Underdetermined System"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('under_determined_system{samp:0>3}.svg', \n",
" directory='../slides/diagrams/ml', samp=IntSlider(0, 0, 10, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{under\\_determined\\_system}{1}{3}\n",
"![](\"../slides/diagrams/ml/under_determined_system000.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system001.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system002.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system003.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system004.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system005.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system006.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system007.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system008.svg\")
\n",
"![](\"../slides/diagrams/ml/under_determined_system009.svg\")
\n",
"\n",
"### Alan Turing\n",
"\n",
"| \n",
"\n",
" | \n",
"\n",
"\n",
" | \n",
"
![](\"../slides/diagrams/ml/dem_gaussian001.svg\")
\n",
"![](\"../slides/diagrams/ml/dem_gaussian002.svg\")
\n",
"![](\"../slides/diagrams/ml/dem_gaussian003.svg\")
\n",
"\n",
"### Stages to Derivation of the Posterior\n",
"\n",
"- Multiply likelihood by prior\n",
"- they are “exponentiated quadratics”, the answer is always also an\n",
" exponentiated quadratic because\n",
" $\\exp(a^2)\\exp(b^2) = \\exp(a^2 + b^2)$.\n",
"- Complete the square to get the resulting density in the form of a\n",
" Gaussian.\n",
"- Recognise the mean and (co)variance of the Gaussian. This is the\n",
" estimate of the posterior.\n",
"\n",
"### Main Trick\n",
"\n",
"$$p(c) = \\frac{1}{\\sqrt{2\\pi\\alpha_1}} \\exp\\left(-\\frac{1}{2\\alpha_1}c^2\\right)$$\n",
"$$p(\\dataVector|\\inputVector, c, m, \\dataStd^2) = \\frac{1}{\\left(2\\pi\\dataStd^2\\right)^{\\frac{\\numData}{2}}} \\exp\\left(-\\frac{1}{2\\dataStd^2}\\sum_{i=1}^\\numData(\\dataScalar_i - m\\inputScalar_i - c)^2\\right)$$\n",
"\n",
"### \n",
"\n",
"$$p(c| \\dataVector, \\inputVector, m, \\dataStd^2) = \\frac{p(\\dataVector|\\inputVector, c, m, \\dataStd^2)p(c)}{p(\\dataVector|\\inputVector, m, \\dataStd^2)}$$\n",
"\n",
"$$p(c| \\dataVector, \\inputVector, m, \\dataStd^2) = \\frac{p(\\dataVector|\\inputVector, c, m, \\dataStd^2)p(c)}{\\int p(\\dataVector|\\inputVector, c, m, \\dataStd^2)p(c) \\text{d} c}$$\n",
"\n",
"### \n",
"\n",
"$$p(c| \\dataVector, \\inputVector, m, \\dataStd^2) \\propto p(\\dataVector|\\inputVector, c, m, \\dataStd^2)p(c)$$\n",
"\n",
"$$\\begin{aligned}\n",
" \\log p(c | \\dataVector, \\inputVector, m, \\dataStd^2) =&-\\frac{1}{2\\dataStd^2} \\sum_{i=1}^\\numData(\\dataScalar_i-c - m\\inputScalar_i)^2-\\frac{1}{2\\alpha_1} c^2 + \\text{const}\\\\\n",
" = &-\\frac{1}{2\\dataStd^2}\\sum_{i=1}^\\numData(\\dataScalar_i-m\\inputScalar_i)^2 -\\left(\\frac{\\numData}{2\\dataStd^2} + \\frac{1}{2\\alpha_1}\\right)c^2\\\\\n",
" & + c\\frac{\\sum_{i=1}^\\numData(\\dataScalar_i-m\\inputScalar_i)}{\\dataStd^2},\n",
" \\end{aligned}$$\n",
"\n",
"### \n",
"\n",
"complete the square of the quadratic form to obtain\n",
"$$\\log p(c | \\dataVector, \\inputVector, m, \\dataStd^2) = -\\frac{1}{2\\tau^2}(c - \\mu)^2 +\\text{const},$$\n",
"where $\\tau^2 = \\left(\\numData\\dataStd^{-2} +\\alpha_1^{-1}\\right)^{-1}$\n",
"and\n",
"$\\mu = \\frac{\\tau^2}{\\dataStd^2} \\sum_{i=1}^\\numData(\\dataScalar_i-m\\inputScalar_i)$.\n",
"\n",
"### Two Dimensional Gaussian\n",
"\n",
"- Consider height, $h/m$ and weight, $w/kg$.\n",
"- Could sample height from a distribution: $$\n",
" p(h) \\sim \\gaussianSamp{1.7}{0.0225}\n",
" $$\n",
"- And similarly weight: $$\n",
" p(w) \\sim \\gaussianSamp{75}{36}\n",
" $$\n",
"\n",
"### Height and Weight Models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.height_weight(diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"../slides/diagrams/ml/height_weight_gaussian.svg\")
\n",
"\n",
"Gaussian distributions for height and weight.\n",
"\n",
"### Sampling Two Dimensional Variables"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.independent_height_weight(num_samps=8, \n",
" diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('independent_height_weight{fig:0>3}.png', '../slides/diagrams/ml', fig=IntSlider(0, 0, 8, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{independent\\_height\\_weight}{0}{7}\n",
"![](\"../slides/diagrams/ml/independent_height_weight000.svg\")
\n",
"![](\"../slides/diagrams/ml/independent_height_weight001.svg\")
\n",
"![](\"../slides/diagrams/ml/independent_height_weight002.svg\")
\n",
"![](\"../slides/diagrams/ml/independent_height_weight003.svg\")
\n",
"![](\"../slides/diagrams/ml/independent_height_weight004.svg\")
\n",
"![](\"../slides/diagrams/ml/independent_height_weight005.svg\")
\n",
"![](\"../slides/diagrams/ml/independent_height_weight006.svg\")
\n",
"![](\"../slides/diagrams/ml/independent_height_weight007.svg\")
\n",
"\n",
"### Independence Assumption\n",
"\n",
"- This assumes height and weight are independent.\n",
" $$p(h, w) = p(h)p(w)$$\n",
"\n",
"- In reality they are dependent (body mass index) $= \\frac{w}{h^2}$.\n",
"\n",
"### Sampling Two Dimensional Variables"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.correlated_height_weight(num_samps=8, \n",
" diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('correlated_height_weight{fig:0>3}.png', '../slides/diagrams/ml', fig=IntSlider(0, 0, 8, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{correlated\\_height\\_weight}{0}{7}\n",
"![](\"../slides/diagrams/ml/correlated_height_weight000.svg\")
\n",
"![](\"../slides/diagrams/ml/correlated_height_weight001.svg\")
\n",
"![](\"../slides/diagrams/ml/correlated_height_weight002.svg\")
\n",
"![](\"../slides/diagrams/ml/correlated_height_weight003.svg\")
\n",
"![](\"../slides/diagrams/ml/correlated_height_weight004.svg\")
\n",
"![](\"../slides/diagrams/ml/correlated_height_weight005.svg\")
\n",
"![](\"../slides/diagrams/ml/correlated_height_weight006.svg\")
\n",
"![](\"../slides/diagrams/ml/correlated_height_weight007.svg\")
\n",
"\n",
"### Independent Gaussians\n",
"\n",
"$$\n",
"p(w, h) = p(w)p(h)\n",
"$$\n",
"\n",
"### Independent Gaussians\n",
"\n",
"$$\n",
"p(w, h) = \\frac{1}{\\sqrt{2\\pi \\dataStd_1^2}\\sqrt{2\\pi\\dataStd_2^2}} \\exp\\left(-\\frac{1}{2}\\left(\\frac{(w-\\meanScalar_1)^2}{\\dataStd_1^2} + \\frac{(h-\\meanScalar_2)^2}{\\dataStd_2^2}\\right)\\right)\n",
"$$\n",
"\n",
"### Independent Gaussians\n",
"\n",
"$$\n",
"p(w, h) = \\frac{1}{\\sqrt{2\\pi\\dataStd_1^22\\pi\\dataStd_2^2}} \\exp\\left(-\\frac{1}{2}\\left(\\begin{bmatrix}w \\\\ h\\end{bmatrix} - \\begin{bmatrix}\\meanScalar_1 \\\\ \\meanScalar_2\\end{bmatrix}\\right)^\\top\\begin{bmatrix}\\dataStd_1^2& 0\\\\0&\\dataStd_2^2\\end{bmatrix}^{-1}\\left(\\begin{bmatrix}w \\\\ h\\end{bmatrix} - \\begin{bmatrix}\\meanScalar_1 \\\\ \\meanScalar_2\\end{bmatrix}\\right)\\right)\n",
"$$\n",
"\n",
"### Independent Gaussians\n",
"\n",
"$$\n",
"p(\\dataVector) = \\frac{1}{\\det{2\\pi \\mathbf{D}}^{\\frac{1}{2}}} \\exp\\left(-\\frac{1}{2}(\\dataVector - \\meanVector)^\\top\\mathbf{D}^{-1}(\\dataVector - \\meanVector)\\right)\n",
"$$\n",
"\n",
"### Correlated Gaussian\n",
"\n",
"Form correlated from original by rotating the data space using matrix\n",
"$\\rotationMatrix$.\n",
"\n",
"$$\n",
"p(\\dataVector) = \\frac{1}{\\det{2\\pi\\mathbf{D}}^{\\frac{1}{2}}} \\exp\\left(-\\frac{1}{2}(\\dataVector - \\meanVector)^\\top\\mathbf{D}^{-1}(\\dataVector - \\meanVector)\\right)\n",
"$$\n",
"\n",
"### Correlated Gaussian\n",
"\n",
"Form correlated from original by rotating the data space using matrix\n",
"$\\rotationMatrix$.\n",
"\n",
"$$\n",
"p(\\dataVector) = \\frac{1}{\\det{2\\pi\\mathbf{D}}^{\\frac{1}{2}}} \\exp\\left(-\\frac{1}{2}(\\rotationMatrix^\\top\\dataVector - \\rotationMatrix^\\top\\meanVector)^\\top\\mathbf{D}^{-1}(\\rotationMatrix^\\top\\dataVector - \\rotationMatrix^\\top\\meanVector)\\right)\n",
"$$\n",
"\n",
"### Correlated Gaussian\n",
"\n",
"Form correlated from original by rotating the data space using matrix\n",
"$\\rotationMatrix$.\n",
"\n",
"$$\n",
"p(\\dataVector) = \\frac{1}{\\det{2\\pi\\mathbf{D}}^{\\frac{1}{2}}} \\exp\\left(-\\frac{1}{2}(\\dataVector - \\meanVector)^\\top\\rotationMatrix\\mathbf{D}^{-1}\\rotationMatrix^\\top(\\dataVector - \\meanVector)\\right)\n",
"$$ this gives a covariance matrix: $$\n",
"\\covarianceMatrix^{-1} = \\rotationMatrix \\mathbf{D}^{-1} \\rotationMatrix^\\top\n",
"$$\n",
"\n",
"### Correlated Gaussian\n",
"\n",
"Form correlated from original by rotating the data space using matrix\n",
"$\\rotationMatrix$.\n",
"\n",
"$$\n",
"p(\\dataVector) = \\frac{1}{\\det{2\\pi\\covarianceMatrix}^{\\frac{1}{2}}} \\exp\\left(-\\frac{1}{2}(\\dataVector - \\meanVector)^\\top\\covarianceMatrix^{-1} (\\dataVector - \\meanVector)\\right)\n",
"$$ this gives a covariance matrix: $$\n",
"\\covarianceMatrix = \\rotationMatrix \\mathbf{D} \\rotationMatrix^\\top\n",
"$$\n",
"\n",
"### Sampling the Prior\n",
"\n",
"- Always useful to perform a ‘sanity check’ and sample from the prior\n",
" before observing the data.\n",
"- Since $\\dataVector = \\basisMatrix \\mappingVector + \\noiseVector$\n",
" just need to sample $$\n",
" \\mappingVector \\sim \\gaussianSamp{0}{\\alpha\\eye}\n",
" $$ $$\n",
" \\noiseVector \\sim \\gaussianSamp{\\zerosVector}{\\dataStd^2}\n",
" $$ with $\\alpha=1$ and $\\dataStd^2 = 0.01$.\n",
"\n",
"### Computing the Posterior\n",
"\n",
"$$\n",
"p(\\mappingVector | \\dataVector, \\inputVector, \\dataStd^2) = \\gaussianDist{\\mappingVector}{\\meanVector_\\mappingScalar}{\\covarianceMatrix_\\mappingScalar}\n",
"$$ with $$\n",
"\\covarianceMatrix_\\mappingScalar = \\left(\\dataStd^{-2}\\basisMatrix^\\top \\basisMatrix + \\alpha^{-1}\\eye\\right)^{-1}\n",
"$$ and $$\n",
"\\meanVector_\\mappingScalar = \\covarianceMatrix_\\mappingScalar \\dataStd^{-2}\\basisMatrix^\\top \\dataVector\n",
"$$\n",
"\n",
"### Olympic Data with Bayesian Polynomials"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import mlai\n",
"import pods"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = pods.datasets.olympic_marathon_men()\n",
"x = data['X']\n",
"y = data['Y']\n",
"num_data = x.shape[0]\n",
"\n",
"data_limits = [1892, 2020]\n",
"basis = mlai.basis(mlai.polynomial, number=1, data_limits=data_limits)\n",
"\n",
"max_basis = y.shape[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.rmse_fit(x, y, param_name='number', param_range=(1, max_basis+1),\n",
" model=mlai.BLM, \n",
" basis=basis, \n",
" alpha=1, \n",
" sigma2=0.04, \n",
" data_limits=data_limits,\n",
" xlim=data_limits, \n",
" objective_ylim=[0.5,1.6]\n",
" diagrams='../slides/diagrams/ml')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('olympic_BLM_polynomial_number{num_basis:0>3}.svg', \n",
" directory='../slides/diagrams/ml/', \n",
" num_basis=IntSlider(1, 1, 27, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{olympic\\_BLM\\_polynomial\\_number}{1}{26}\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number001.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number002.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number003.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number004.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number005.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number006.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number007.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number008.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number009.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number010.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number011.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number012.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number013.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number014.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number015.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number016.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number017.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number018.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number019.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number020.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number021.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number022.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number023.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number024.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number025.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_BLM_polynomial_number026.svg\")
\n",
"\n",
"### Hold Out Validation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot.holdout_fit(x, y, param_name='number', param_range=(1, 27),\n",
" diagrams='../slides/diagrams/ml',\n",
" model=mlai.BLM, \n",
" basis=basis, \n",
" alpha=1, \n",
" sigma2=0.04,\n",
" xlim=data_limits, \n",
" objective_ylim=[0.1,0.6], \n",
" permute=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('olympic_val_BLM_polynomial_number{num_basis:0>3}.svg', \n",
" directory='../slides/diagrams/ml', \n",
" num_basis=IntSlider(1, 1, 27, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{olympic\\_val\\_BLM\\_polynomial\\_number}{1}{26}\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number001.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number002.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number003.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number004.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number005.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number006.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number007.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number008.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number009.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number010.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number011.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number012.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number013.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number014.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number015.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number016.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number017.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number018.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number019.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number020.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number021.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number022.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number023.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number024.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number025.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_val_BLM_polynomial_number026.svg\")
\n",
"\n",
"### 5-fold Cross Validation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_parts=5\n",
"plot.cv_fit(x, y, param_name='number', param_range=(1, 27), \n",
" diagrams='../slides/diagrams/ml',\n",
" model=mlai.BLM, \n",
" basis=basis, \n",
" alpha=1, \n",
" sigma2=0.04, \n",
" xlim=data_limits, \n",
" objective_ylim=[0.2,0.6], \n",
" num_parts=num_parts)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('olympic_5cv{part:0>2}_BLM_polynomial_number{num_basis:0>3}.svg', \n",
" directory='../slides/diagrams/ml', \n",
" part=(0, 5), \n",
" num_basis=IntSlider(1, 1, 27, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\\\startslides{olympic\\_5cv05\\_BLM\\_polynomial\\_number}{1}{26}\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number001.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number002.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number003.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number004.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number005.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number006.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number007.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number008.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number009.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number010.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number011.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number012.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number013.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number014.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number015.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number016.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number017.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number018.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number019.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number020.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number021.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number022.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number023.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number024.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number025.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic_5cv05_BLM_polynomial_number026.svg\")
\n",
"\n",
"### Marginal Likelihood\n",
"\n",
"- The marginal likelihood can also be computed, it has the form: $$\n",
" p(\\dataVector|\\inputMatrix, \\dataStd^2, \\alpha) = \\frac{1}{(2\\pi)^\\frac{n}{2}\\left|\\kernelMatrix\\right|^\\frac{1}{2}} \\exp\\left(-\\frac{1}{2} \\dataVector^\\top \\kernelMatrix^{-1} \\dataVector\\right)\n",
" $$ where\n",
" $\\kernelMatrix = \\alpha \\basisMatrix\\basisMatrix^\\top + \\dataStd^2 \\eye$.\n",
"\n",
"- So it is a zero mean $\\numData$-dimensional Gaussian with covariance\n",
" matrix $\\kernelMatrix$.\n",
"\n",
"### References {#references .unnumbered}"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}