"
],
"text/plain": [
" true pred pred_var pred_lower pred_upper within_2sigma\n",
"1436 117.054 123.860 39.744 111.252 136.468 True\n",
"572 132.757 127.422 39.623 114.832 140.012 True\n",
"126 112.169 117.339 39.933 104.701 129.977 True\n",
"998 132.897 127.302 39.597 114.716 139.888 True\n",
"80 121.851 124.130 39.607 111.544 136.716 True"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results_1d.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our regression model gives probabilistic predictions and we want to know if the variances of these predictive distributions are correct on average. We will therefore compare the average prediction variance and mean squared error on the test set to see if they are about the same. About 95% of the observations in the test set should be within the $2\\sigma$ prediction interval."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Average prediction variance on test set: 39.7\n",
"Mean squared error on test set: 36.0\n",
"Percentage of test instances within 2 sigma prediction interval: 95.57%\n"
]
}
],
"source": [
"from sklearn.metrics import mean_squared_error\n",
"\n",
"# Model evaluation\n",
"avg_pred_var = results_1d['pred_var'].mean()\n",
"print('Average prediction variance on test set: {}'.format(np.round(avg_pred_var,2)))\n",
"mse = mean_squared_error(results_1d['true'], results_1d['pred'])\n",
"print('Mean squared error on test set: {}'.format(np.round(mse), 2))\n",
"twosigmapercent = 100*results_1d['within_2sigma'].mean()\n",
"print('Percentage of test instances within 2 sigma prediction interval: {}%'.format(np.round(twosigmapercent,2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Multifeature model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We then use all the five features for our GP model."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"# Choose the squared exponential kernel with observation noise. Here variance = \\sigma_f^2, lengthscale = \\ell.\n",
"kernel = GPy.kern.RBF(input_dim=5, variance=1., lengthscale=1.)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"# Create a regression model with normalized targets. Here noise_var = \\sigma_n^2.\n",
"model = GPy.models.GPRegression(X_train_scaled, y_train.values, kernel, normalizer=True, noise_var=1.)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
"Model: GP regression \n",
"Objective: 1193.601528964441 \n",
"Number of Parameters: 3 \n",
"Number of Optimization Parameters: 3 \n",
"Updates: True \n",
"