{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WSW9BmePM7zZ"
   },
   "source": [
    "# Exercise 5.3: Neural Networks in Keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "t02FemO-M7za"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# See https://keras.io/\n",
    "# for extennsive documentation\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IrvyHKHTM7ze"
   },
   "source": [
    "Let us visit the problem of wine quality prediction previ- ously encountered in Exercises 3.2 and 4.1 one final time. After linear regression and a self-made network, we can now explore the comfort provided by the Keras library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-H-L5egsM7ze",
    "outputId": "82d3f95f-59b1-43e3-e8d7-4d939ea7eec6"
   },
   "outputs": [],
   "source": [
    "# The code snippet below is responsible for downloading the dataset to\n",
    "# Google. You can directly download the file using the link\n",
    "# if you work with a local anaconda setup\n",
    "!wget https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mDToshKJNWGY",
    "outputId": "8cce4ba8-392b-43a1-d659-26aa95d21113"
   },
   "outputs": [],
   "source": [
    "# load all examples from the file\n",
    "data = np.genfromtxt('winequality-white.csv',delimiter=\";\",skip_header=1)\n",
    "\n",
    "print(\"data:\", data.shape)\n",
    "\n",
    "# Prepare for proper training\n",
    "np.random.shuffle(data) # randomly sort examples\n",
    "\n",
    "# take the first 3000 examples for training\n",
    "X_train = data[:3000,:11] # all features except last column\n",
    "y_train = data[:3000,11]  # quality column\n",
    "\n",
    "# and the remaining examples for testing\n",
    "X_test = data[3000:,:11] # all features except last column\n",
    "y_test = data[3000:,11] # quality column\n",
    "\n",
    "print(\"First example:\")\n",
    "print(\"Features:\", X_train[0])\n",
    "print(\"Quality:\", y_train[0])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VXx8BXB_M7zi"
   },
   "source": [
    "Below is the simple network from exercise 4.1 implemented using Keras. In addition to the network we define the loss function and optimiser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Z0HMdw9eM7zi"
   },
   "outputs": [],
   "source": [
    "# See: https://keras.io/api/models/sequential/ and \n",
    "# https://keras.io/api/layers/core_layers/dense/\n",
    "# We can use the Sequential class to very easiliy\n",
    "# build a simple architecture\n",
    "model = Sequential()\n",
    "# 11 inputs, 20 outputs, relu\n",
    "model.add(Dense(20, input_dim=11, activation='relu')) \n",
    "# 20 inputs (automatically detected by Keras), 1 output, linear activation\n",
    "model.add(Dense(1, activation='linear'))\n",
    "\n",
    "\n",
    "# Set loss function and optimiser algorithm\n",
    "model.compile(loss='mse',  # mean squared error\n",
    "              optimizer='sgd'# stochastic gradient descent\n",
    "             ) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I98jdZcqM7zm"
   },
   "source": [
    "# Training and evaluation below\n",
    "\n",
    "The code below trains the network for 5 epochs using the loss function and optimiser defined above. Each example is individually passed to the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(X_train, y_train, \n",
    "                    validation_data=(X_test, y_test),\n",
    "                    epochs=5, batch_size=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The history object returned by the model training above \n",
    "# contains the values of the loss function (the mean-squared-error)\n",
    "# at different epochs\n",
    "# We discard the first epoch as the loss value is very high,\n",
    "# obscuring the rest of the distribution\n",
    "train_loss = history.history[\"loss\"][1:]\n",
    "test_loss = history.history[\"val_loss\"][1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare and plot loss over time\n",
    "plt.plot(train_loss,label=\"train\")\n",
    "plt.plot(test_loss,label=\"test\")\n",
    "plt.legend()\n",
    "plt.xlabel(\"Epoch-1\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After the training:\n",
    "\n",
    "# Prepare scatter plot\n",
    "y_pred = model.predict(X_test)[:,0]\n",
    "\n",
    "print(\"Correlation coefficient:\", np.corrcoef(y_pred,y_test)[0,1])\n",
    "plt.scatter(y_pred,y_test)\n",
    "plt.xlabel(\"Predicted\")\n",
    "plt.ylabel(\"True\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(y_pred,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KI7zcK_uM7zp"
   },
   "source": [
    "\n",
    "# Problems\n",
    "\n",
    "* Use the notebook as starting point. It already contains the simple network from Exercise 4.1 implemented in Keras.\n",
    "\n",
    "* Currently, SGD is used without momentum. Try training with a momentum term. Replace SGD with the Adam optimizer and train using that. (See: https://keras.io/api/optimizers/)\n",
    "* Add two more hidden layers to the network (you can choose the number of nodes but make sure to apply the ReLu activation function after each) and train again.\n",
    "* Test differet numbers of examples (i.e. change the batch batch size) to be simulataneously used by the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2mUTzx7h2mKJ"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Exercise 6 - Solution",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}