{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 5.2\n",
    "## Interpolation\n",
    "In this task, we implement a simple NN to learn a complicated function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from tensorflow import keras\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "layers = keras.layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generation of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def some_complicated_function(x):\n",
    "    return (\n",
    "        (np.abs(x)) ** 0.5\n",
    "        + 0.1 * x\n",
    "        + 0.01 * x ** 2\n",
    "        + 1\n",
    "        - np.sin(x)\n",
    "        + 0.5 * np.exp(x / 10.0)\n",
    "        ) / (0.5 + np.abs(np.cos(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's simulate the train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_train = 10 ** 4  # number of training samples\n",
    "# Note: \"[:, np.newaxis]\" reshapes array to (N,1) as required by our DNN (we input one feature per sample)\n",
    "xtrain = np.random.uniform(-10, 10, N_train)[:, np.newaxis]\n",
    "ytrain = some_complicated_function(xtrain) + np.random.randn(xtrain.shape[0])  # train data includes some noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "xtrain.shape (10000, 1)\n",
      "ytrain.shape (10000, 10000)\n"
     ]
    }
   ],
   "source": [
    "print(\"xtrain.shape\", xtrain.shape)\n",
    "print(\"ytrain.shape\", ytrain.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simulate test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_test = 10000  # number of testing samples\n",
    "xtest = np.linspace(-10, 10, N_test)\n",
    "ytest = some_complicated_function(xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "xtest.shape (10000,)\n",
      "ytest.shape (10000,)\n"
     ]
    }
   ],
   "source": [
    "print(\"xtest.shape\", xtest.shape)\n",
    "print(\"ytest.shape\", ytest.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Model\n",
    "\n",
    "Define the number of nodes, the number of layers, and choose an activation function.\n",
    "Use `keras.regularizers` to use parameter norm penalties or add a dropout layer via `layers.Dropout(fraction)`.\n",
    "\n",
    "You may use the skeleton below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_nodes = 1\n",
    "nb_layers = 1\n",
    "activation = \"\"\n",
    "\n",
    "model = keras.models.Sequential(name=\"1Dfit\")\n",
    "model.add(layers.Dense(nb_nodes, activation=activation, input_dim=xtrain.shape[1]))  # first layer\n",
    "\n",
    "model.add(layers.Dense(1))  # final layer\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile the model (set an objective and choose an optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choose an optimizer from `keras.optimizers`, e.g., `adam = keras.optimizers.Adam(lr=0.001)`.\n",
    "\n",
    "Further, choose the correct objective (loss) for this <b>regression task</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"\", optimizer=)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the network for a couple of epochs and save the model several times in between."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = \n",
    "save_period =   # after how many epochs the model should be saved?\n",
    "\n",
    "chkpnt_saver = keras.callbacks.ModelCheckpoint(\"weights-{epoch:02d}.hdf5\", save_weights_only=True, save_freq=save_period)\n",
    "\n",
    "results = model.fit(\n",
    "    xtrain,\n",
    "    ytrain,\n",
    "    batch_size=64,\n",
    "    epochs=epochs,\n",
    "    verbose=1,\n",
    "    callbacks=[chkpnt_saver]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the performance of the model during the training. You may use the skeleton below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(nrows=2, figsize=(12, 8))\n",
    "\n",
    "ax1.plot(xtest, ytest, color=\"black\", label=\"data\")\n",
    "saved_epochs = range(save_period, epochs + 1, save_period)\n",
    "\n",
    "colors = [plt.cm.jet((i + 1) / float(len(saved_epochs) + 1)) for i in range(len(saved_epochs))]\n",
    "\n",
    "for i, epoch in enumerate(saved_epochs):\n",
    "    model.load_weights(\"weights-{epoch:02d}.hdf5\".format(epoch=epoch))\n",
    "    ypredict = model.predict(xtest).squeeze()\n",
    "    ax1.plot(xtest.squeeze(), ypredict, color=colors[i], label=epoch)\n",
    "    ax2.plot(epoch, results.history[\"loss\"][epoch - 1], color=colors[i], marker=\"o\")\n",
    "\n",
    "ax1.set(xlabel=\"x\", ylabel=\"some_complicated_function(x)\", xlim=(-10, 13), title=\"\")\n",
    "ax1.grid(True)\n",
    "ax1.legend(loc=\"upper right\", title=\"Epochs\")\n",
    "\n",
    "ax2.plot(results.history[\"loss\"], color=\"black\")\n",
    "ax2.set(xlabel=\"epoch\", ylabel=\"loss\")\n",
    "ax2.grid(True)\n",
    "ax2.semilogy()\n",
    "\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}