{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [], "source": [ "#configure plotting\n", "%matplotlib inline\n", "%config InlineBackend.figure_format = 'svg'\n", "import matplotlib;matplotlib.rcParams['figure.figsize'] = (8,5)\n", "import matplotlib;matplotlib.rcParams['text.usetex'] = True\n", "import matplotlib;matplotlib.rcParams['font.size'] = 16\n", "import matplotlib;matplotlib.rcParams['font.family'] = 'serif'\n", "import GPy\n", "import numpy as np\n", "from matplotlib import pyplot as plt\n", "np.random.seed(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# A brief guide to GP classification with GPy\n", "\n", "### James Hensman, November 2014\n", "#### with edits by Alan Saul\n", "\n", "Gaussian Process classification is perhaps the most popular case of a GP model where the likelihood is not conjugate to the prior. In the classification case, the likelihood is Bernoulli, and we must perform approximate inference over the values of the function. There are many approaches to making such an approximation: here we'll start with the popular EP method. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##The generative model\n", "To illustate GP classification, we'll first consider the generative model of the data. In classification, we consider the labels (0 or 1) to be generated by a conditional distribution given the inputs, $\\mathbf X$. \n", "\n", "In GP classification, the model is that there is a hidden function, drawn from a GP, which takes continuous values. Those values are then squashed through a probit function, into the range $[0, 1]$, and can then be used as probabilities. Bernoulli draws are then made (taking the values 0 or 1) conditioned on the squashed values.\n", "\n", "First we'll set up a kernel for the GP from which to draw some latent function values (a candidate hidden function for the above generative model), given some arbitrary positions $\\mathbf X$ at which we get observations." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " /home/maxz/anaconda/lib/python2.7/site-packages/ipykernel/__main__.py:5: RuntimeWarning:covariance is not positive-semidefinite.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "k = GPy.kern.RBF(1, variance=7., lengthscale=0.2)\n", "X = np.random.rand(200,1)\n", "\n", "#draw the latent function value\n", "f = np.random.multivariate_normal(np.zeros(200), k.K(X))\n", "\n", "plt.plot(X, f, 'bo')\n", "plt.title('latent function values');plt.xlabel('$x$');plt.ylabel('$f(x)$')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we'll squash the latent function values through the probit function. We'll set up a GPy likelihood for this, which contains the squashing probit function as well as assorted mechanisms for doing approximate inference." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "lik = GPy.likelihoods.Bernoulli()\n", "p = lik.gp_link.transf(f) # squash the latent function\n", "plt.plot(X, p, 'ro')\n", "plt.title('latent probabilities');plt.xlabel('$x$');plt.ylabel('$\\sigma(f(x))$')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now binary Bernoulli variables are drawn using the squashed values of our latent function as probabilities:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "Y = lik.samples(f).reshape(-1,1)\n", "plt.plot(X, Y, 'kx', mew=2);plt.ylim(-0.1, 1.1)\n", "plt.title('Bernoulli draws');plt.xlabel('$x$');plt.ylabel('$y \\in \\{0,1\\}$')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Inference\n", "We have now generated some data from our proposed model. We'll now attempt to \n", "\n", "In this case we wish to know, given the observed binary variables, can we recover the latent function, the associated probabilities, and the variance and lengthscale of the GP that we believe produced the data?\n", "\n", "We'll set up a GPy classifier to do this. first we'll use the most general setup for defining a GP object in GPy, allowing us to select the EP inference method and use our existing likelihood and kernel." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Name : gp\n", "Objective : 101.526034536\n", "Number of Parameters : 2\n", "Number of Optimization Parameters : 2\n", "Updates : True\n", "Parameters:\n", " \u001b[1mgp. \u001b[0;0m | value | constraints | priors\n", " \u001b[1mrbf.variance \u001b[0;0m | 7.0 | +ve | \n", " \u001b[1mrbf.lengthscale\u001b[0;0m | 0.2 | +ve | \n" ] } ], "source": [ "m = GPy.core.GP(X=X,\n", " Y=Y, \n", " kernel=k, \n", " inference_method=GPy.inference.latent_function_inference.expectation_propagation.EP(),\n", " likelihood=lik)\n", "print m" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There's a simpler way to build GP classifiers, with some default options (RBF kernel, EP inference) like so:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [], "source": [ "m = GPy.models.GPClassification(X,Y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This model is initialized with incorrect lengthscale and variance, but let's check out how it's fared in inferring the latent fuction values:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "m.plot()\n", "plt.plot(X, p, 'ro')\n", "plt.ylabel('$y, p(y=1)$');plt.xlabel('$x$')\n", "\n", "m.plot_f()\n", "plt.plot(X, f, 'bo')\n", "plt.ylabel('$f(x)$');plt.xlabel('$x$')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the above plots, we can see the posterior over the latent probabilities (top) and latent continuous function values (bottom). We've already run EP to obtain the approximate posterior and marginal likelihood. The above plots suggest that the lengthscale of the kernel is incorrect for this problem, so we'll optimize the parameters of the kernel against the approximate marginal likelihood, $p(y|\\mathcal{M})$, that represents how probable the data is given our model, $\\mathcal{M}$. \n", "\n", "To do this, we interleave runs of EP with optimization of the parameters using gradient descent methods. EP is a method for fitting a Gaussian to the posterior, $p(f|y)$, of the latent (hidden) function, given the data. Whilst the parameters are being optimized, the EP approximation (the parameters of the EP factors) is fixed. " ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Name : gp_classification\n", "Objective : 114.557648448\n", "Number of Parameters : 2\n", "Number of Optimization Parameters : 2\n", "Updates : True\n", "Parameters:\n", " \u001b[1mgp_classification.\u001b[0;0m | value | constraints | priors\n", " \u001b[1mrbf.variance \u001b[0;0m | 1.0 | +ve | \n", " \u001b[1mrbf.lengthscale \u001b[0;0m | 1.0 | +ve | \n", "\n", "iteration: 0 \n", "Name : gp_classification\n", "Objective : 101.130298901\n", "Number of Parameters : 2\n", "Number of Optimization Parameters : 2\n", "Updates : True\n", "Parameters:\n", " \u001b[1mgp_classification.\u001b[0;0m | value | constraints | priors\n", " \u001b[1mrbf.variance \u001b[0;0m | 1.01067796874 | +ve | \n", " \u001b[1mrbf.lengthscale \u001b[0;0m | 0.142282190617 | +ve | \n", "\n", "iteration: 1 \n", "Name : gp_classification\n", "Objective : 99.6116762022\n", "Number of Parameters : 2\n", "Number of Optimization Parameters : 2\n", "Updates : True\n", "Parameters:\n", " \u001b[1mgp_classification.\u001b[0;0m | value | constraints | priors\n", " \u001b[1mrbf.variance \u001b[0;0m | 1.23656640131 | +ve | \n", " \u001b[1mrbf.lengthscale \u001b[0;0m | 0.137464523313 | +ve | \n", "\n", "iteration: 2 \n", "Name : gp_classification\n", "Objective : 99.608364372\n", "Number of Parameters : 2\n", "Number of Optimization Parameters : 2\n", "Updates : True\n", "Parameters:\n", " \u001b[1mgp_classification.\u001b[0;0m | value | constraints | priors\n", " \u001b[1mrbf.variance \u001b[0;0m | 1.24311976022 | +ve | \n", " \u001b[1mrbf.lengthscale \u001b[0;0m | 0.137198614331 | +ve | \n", "\n", "iteration: 3 \n", "Name : gp_classification\n", "Objective : 99.6083643298\n", "Number of Parameters : 2\n", "Number of Optimization Parameters : 2\n", "Updates : True\n", "Parameters:\n", " \u001b[1mgp_classification.\u001b[0;0m | value | constraints | priors\n", " \u001b[1mrbf.variance \u001b[0;0m | 1.24313269983 | +ve | \n", " \u001b[1mrbf.lengthscale \u001b[0;0m | 0.137186966338 | +ve | \n", "\n", "iteration: 4 \n", "Name : gp_classification\n", "Objective : 99.6083668378\n", "Number of Parameters : 2\n", "Number of Optimization Parameters : 2\n", "Updates : True\n", "Parameters:\n", " \u001b[1mgp_classification.\u001b[0;0m | value | constraints | priors\n", " \u001b[1mrbf.variance \u001b[0;0m | 1.24315005803 | +ve | \n", " \u001b[1mrbf.lengthscale \u001b[0;0m | 0.137184773984 | +ve | \n", "\n" ] } ], "source": [ "print m, '\\n'\n", "for i in range(5):\n", " m.optimize('bfgs', max_iters=100) #first runs EP and then optimizes the kernel parameters\n", " print 'iteration:', i,\n", " print m\n", " print \"\"\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now our parameters have been optimised to the values that give the largest marginal likelihood, we can plot the results to see how they differ" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "m.plot()\n", "plt.plot(X, p, 'ro', label='Truth')\n", "plt.ylabel('$y, p(y=1)$');plt.xlabel('$x$')\n", "plt.legend()\n", "m.plot_f()\n", "plt.plot(X, f, 'bo', label='Truth')\n", "plt.ylabel('$f(x)$');plt.xlabel('$x$')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above plots are much more satisfactory: we have managed to infer the correct form of the latent function. We note that it's also possible to run MCMC to obtain Bayesian marginalization of the kernel parameters instead of the (type-II) maximum likelihood approach here. We'll adress this in a forthcoming notebook.\n", "\n", "We've seen the generating process for GP classification, and how to set up models in GPy to do classification. We've used EP to infer the latent function, interleaved with optimization of the kernel parameters. There are other latent-function inference methods in GPy, such as the Laplace approximation and the forthcoming variational approximation. These are illustrated in accompanying notebooks." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plotting the density more fine-grained\n", "\n", "The newest version of GPy allows for densities to be plotted more directly. The outcome of this may surprise you, but it is actually what the density looks like for a Bernoulli likelihood." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "m.plot(plot_density=True)\n", "plt.plot(X, p, 'ro', label='Truth')\n", "plt.ylabel('$y, p(y=1)$');plt.xlabel('$x$')\n", "plt.legend()\n", "m.plot_f(plot_density=True)\n", "plt.plot(X, f, 'bo', label='Truth')\n", "plt.ylabel('$f(x)$');plt.xlabel('$x$')\n", "plt.legend()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.11" } }, "nbformat": 4, "nbformat_minor": 0 }