{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Week 5: Introduction to neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perceptron learning rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This week, we will start working with neural networks. For each of the exercises below you can use the method of your choice but you should display the final boundary of your classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1. \n",
    "As a first exercise, load the binary dataset below and code a few steps of the perceptron learning rule. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import scipy.io as sio\n",
    "data1 = sio.loadmat('perceptron_data_class1.mat')\n",
    "data2 = sio.loadmat('perceptron_data_class2.mat')\n",
    "\n",
    "data1 = data1['perceptron_data_class1']\n",
    "data2 = data2['perceptron_data_class2']\n",
    "\n",
    "# put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2.\n",
    "\n",
    "__2a.__ Load the data below. Using the neural_network module from scikit-learn and its MLPClassifier model, learn a classifier, for the dataset below using \n",
    "\n",
    "- One hidden layer with a linear activation function and \n",
    "    - One neuron\n",
    "    - Two neurons\n",
    "    \n",
    "    \n",
    "    \n",
    "- One hidden layer with a non linear activation function (take Relu for example or a binary step)\n",
    "    - One neuron\n",
    "    - Two neurons\n",
    "\n",
    "How many neurons, hidden layers do you need to learn the distribution of the data? Do you have an idea why?\n",
    "\n",
    "Try increasing the number of neurons and hidden layers. Then try different values of the learning rate. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import scipy.io as sio\n",
    "data1 = sio.loadmat('neural_net_class1.mat')\n",
    "data2 = sio.loadmat('neural_net_class2.mat')\n",
    "\n",
    "data1 = data1['neural_net_class1']\n",
    "data2 = data2['neural_net_class2']\n",
    "\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "# put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__2b.__ Keep the dataset from above. try to change the intialization of the training algorithm. Plot the resulting classifier for a couple of different initializations. What do you see?\n",
    "\n",
    "Do it for a small network first. Then repeat those experiments for larger architectures. I.e. increase the number of neurons and the number of layers. What do you see when you change the initialization?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "# put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 3. \n",
    "\n",
    "__3a.__Load the data below. Try to build the best neural network you can for this dataset. Split the data between a training and a test set and evaluate the models you built. What is the best validation error you can get?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "import scipy.io as sio\n",
    "data1 = sio.loadmat('neural_net_ex2_class1.mat')\n",
    "data2 = sio.loadmat('neural_net_ex2_class2.mat')\n",
    "\n",
    "data1 = data1['neural_net_ex2_class1']\n",
    "data2 = data2['neural_net_ex2_class2']\n",
    "\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "\n",
    "# put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "__3b.__ With the same dataset, add additional features to your model, e.g. $\\sin(x), \\sin(y)$ or other monomials. Can you improve your classifier ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# put your code here\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}