{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building and training a single layer neural network by hand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import packages\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from ipywidgets import interact\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the model\n",
    "\n",
    "As mentioned previously, our model is \n",
    "\n",
    "$ p_{setosa} = f( w_0 + w_1 \\times width + w_2 \\times length ) \\qquad with \\;\\; f(x) = 1/(1 + e^{-x})$\n",
    "\n",
    "Let us define this using Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def probability_setosa( petal_length, petal_width, w0, w1, w2 ):\n",
    "    \"Return the probability that a given specimen belongs to the species setosa\"\n",
    "    # Compute sum of features times weights\n",
    "    x = w0 + w1*petal_width + w2*petal_length\n",
    "    # Apply non-linear function: sigmoid\n",
    "    p = 1./( 1. + np.exp( -x ) )\n",
    "    return( p )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the network: finding the right weights so that the model fits the training data\n",
    "\n",
    "In order to get a sense of what training the network implies, we will try to find the right weights *by hand*. Once, we use Keras, this process will be automated.\n",
    "\n",
    "Let us first load the data from the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('./data/setosa/train.csv')\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then define a function that plots the prediction of the model **for a given set of weights**, along with the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_model( w0, w1, w2 ):\n",
    "    \"Plot the model, along with the training data.\"\n",
    "\n",
    "    # Calculate the probability on a mesh\n",
    "    petal_width_mesh, petal_length_mesh = \\\n",
    "        np.meshgrid( np.linspace(0,3,100), np.linspace(0,8,100) )\n",
    "    p = probability_setosa( petal_width_mesh, petal_length_mesh, w0, w1, w2 )\n",
    "    # Plot the probability on the mesh\n",
    "    plt.clf()\n",
    "    plt.imshow( p.T, extent=[0,3,0,8], origin='lower', \n",
    "               vmin=0, vmax=1, cmap='RdBu', aspect='auto', alpha=0.5 )\n",
    "    # Plot the data points\n",
    "    plt.scatter( df['petal width (cm)'], df['petal length (cm)'], c=df['setosa'], cmap='RdBu')\n",
    "    plt.xlabel('petal width (cm)')\n",
    "    plt.ylabel('petal length (cm)')\n",
    "    cb = plt.colorbar()\n",
    "    cb.set_label('setosa')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use the function `interact` of `ipywidgets` to call this function with adjustable weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interact( plot_model, w0=(-4.,5.), w1=(-2.,2.), w2=(-2., 3.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimal weights: fill these values\n",
    "w0 = \n",
    "w1 = \n",
    "w2 = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing predictions on the test sets\n",
    "\n",
    "Now that we trained the model by finding the optimal weights for the training dataset, let us perform predictions on the test dataset.\n",
    "\n",
    "Let us first load the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test = pd.read_csv('./data/setosa/test.csv')\n",
    "df_test.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now check the accuracy of our model on the first point for instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "probability_setosa( 4.2, 1.5, w0, w1, w2 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More generally, by using pandas syntax, we can perform predictions on the whole dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test['probability_setosa_predicted'] = \\\n",
    "    probability_setosa( df_test['petal length (cm)'], df_test['petal width (cm)'], w0, w1, w2 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next steps\n",
    "\n",
    "While the above procedure yields good results, it is very cumbersome to try to find the weights by hand. Let us use keras [here](Single_layer_keras.ipynb) to automate this process."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}