| \n", "- Gold medal times for Olympic Marathon since 1896.\n", "\n", "- Marathons before 1924 didn’t have a standardised distance.\n", "\n", "- Present results using pace per km.\n", "\n", "- In 1904 Marathon was badly organised leading to very slow times.\n", "\n", " | \n", "\n",
" Image from\n",
"Wikimedia Commons | \n",
"
![](\"../slides/diagrams/datasets/olympic-marathon.svg\")
\n",
"\n",
"Things to notice about the data include the outlier in 1904, in this\n",
"year, the olympics was in St Louis, USA. Organizational problems and\n",
"challenges with dust kicked up by the cars following the race meant that\n",
"participants got lost, and only very few participants completed.\n",
"\n",
"More recent years see more consistently quick marathons.\n",
"\n",
"### Example: Linear Regression"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import mlai\n",
"import numpy as np\n",
"import scipy as sp"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create a weighted linear regression class, inheriting from the `mlai.LM`\n",
"class."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class LML(mlai.LM):\n",
" \"\"\"Linear model with evolving loss\n",
" :param X: input values\n",
" :type X: numpy.ndarray\n",
" :param y: target values\n",
" :type y: numpy.ndarray\n",
" :param basis: basis function \n",
" :param type: function\n",
" :param beta: weight of the loss function\n",
" :param type: float\"\"\"\n",
"\n",
" def __init__(self, X, y, basis=None, beta=1.0, lambd=1.0):\n",
" \"Initialise\"\n",
" if basis is None:\n",
" basis = mlai.basis(mlai.polynomial, number=2)\n",
" mlai.LM.__init__(self, X, y, basis)\n",
" self.s = np.ones((self.num_data, 1))#np.random.rand(self.num_data, 1)>0.5\n",
" self.update_w()\n",
" self.sigma2 = 1/beta\n",
" self.beta = beta\n",
" self.name = 'LML_'+basis.function.__name__\n",
" self.objective_name = 'Weighted Sum of Square Training Error'\n",
" self.lambd = lambd\n",
"\n",
" def update_QR(self):\n",
" \"Perform the QR decomposition on the basis matrix.\"\n",
" self.Q, self.R = np.linalg.qr(self.Phi*np.sqrt(self.s))\n",
"\n",
" def fit(self):\n",
" \"\"\"Minimize the objective function with respect to the parameters\"\"\"\n",
" for i in range(30):\n",
" self.update_w() # In the linear regression clas\n",
" self.update_s()\n",
" \n",
" def update_w(self):\n",
" self.update_QR()\n",
" self.w_star = sp.linalg.solve_triangular(self.R, np.dot(self.Q.T, self.y*np.sqrt(self.s)))\n",
" self.update_losses()\n",
"\n",
" def predict(self, X):\n",
" \"\"\"Return the result of the prediction function.\"\"\"\n",
" return np.dot(self.basis.Phi(X), self.w_star), None\n",
" \n",
" def update_s(self):\n",
" \"\"\"Update the weights\"\"\"\n",
" self.s = 1/(self.lambd + self.beta*self.losses)\n",
" \n",
" def update_losses(self):\n",
" \"\"\"Compute the loss functions for each data point.\"\"\"\n",
" self.update_f()\n",
" self.losses = ((self.y-self.f)**2)\n",
" self.beta = 1/(self.losses*self.s).mean()\n",
" \n",
" def objective(self):\n",
" \"\"\"Compute the objective function.\"\"\"\n",
" self.update_losses()\n",
" return (self.losses*self.s).sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Set up a linear model (polynomial with two basis functions)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_basis=2 \n",
"data_limits=[1890, 2020]\n",
"basis = mlai.basis(mlai.polynomial, num_basis, data_limits=data_limits)\n",
"model = LML(x, y, basis=basis, lambd=1, beta=1)\n",
"model2 = mlai.LM(x, y, basis=basis)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit()\n",
"model2.fit()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x_test = np.linspace(data_limits[0], data_limits[1], 130)[:, None]\n",
"f_test, f_var = model.predict(x_test)\n",
"f2_test, f2_var = model2.predict(x_test)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import teaching_plots as plot\n",
"from matplotlib import rc, rcParams\n",
"rcParams.update({'font.size': 22})\n",
"rc('text', usetex=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fig, ax = plt.subplots(figsize=plot.big_wide_figsize)\n",
"ax.plot(x_test, f2_test, linewidth=3, color='r')\n",
"ax.plot(x, y, 'g.', markersize=10)\n",
"ax.set_xlim(data_limits[0], data_limits[1])\n",
"ax.set_xlabel('year')\n",
"ax.set_ylabel('pace min/km')\n",
"_ = ax.set_ylim(2, 6)\n",
"mlai.write_figure('../slides/diagrams/ml/olympic-loss-linear-regression000.svg', transparent=True)\n",
"ax.plot(x_test, f_test, linewidth=3, color='b')\n",
"ax.plot(x, y, 'g.', markersize=10)\n",
"ax2 = ax.twinx()\n",
"ax2.bar(x.flatten(), model.s.flatten(), width=2, color='b')\n",
"ax2.set_ylim(0, 4)\n",
"ax2.set_yticks([0, 1, 2])\n",
"ax2.set_ylabel('$\\langle s_i \\\\rangle$')\n",
"mlai.write_figure('../slides/diagrams/ml/olympic-loss-linear-regression001.svg', transparent=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"pods.notebook.display_plots('olympic-loss-linear-regression{number:0>3}.svg', \n",
" directory='../slides/diagrams/ml', number=(0, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"../slides/diagrams/ml/olympic-loss-linear-regression001.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic-loss-bayes-linear-regression001.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic-gp-loss-bayes-linear-regression001.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic-gp-loss-samples.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic-gp-loss-bayes-linear-regression-and-samples001.svg\")
\n",
"![](\"../slides/diagrams/ml/olympic-gp-loss-histogram-2020.svg\")
\n",
"