{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://raw.githubusercontent.com/autonomio/hyperio/master/logo.png' width=250px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import talos as ta\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1. <a href=#one>Data Preparation</a>\n",
    "##### 2. <a href=#two>Model Preparation</a>\n",
    "##### 3. <a href=#three>Setting the Parameter Space Boundaries</a>\n",
    "##### 4. <a href=#four>Run the Hyperparameter <code>Scan()</code></a>\n",
    "##### 5. <a href=#five>Access the results through the <code>Scan</code> object</a>\n",
    "##### 6. <a href=#six>Analysing the Scan results with <code>Reporting()</code></a>\n",
    "##### 7. <a href=#seven>Evaluating Models with <code>Evaluate()</code></a>\n",
    "##### 8. <a href=#eight>Deploying Models with <code>Deploy()</code></a>\n",
    "##### 9. <a href=#nine>Restoring Models with <code>Restore()</code></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Data Preparation <a name=\"one\"></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this experiment, we're going to use the famous Iris dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = ta.datasets.iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Model Preparation  <a name=\"two\"></a> \n",
    "Talos works with any Keras model, without changing the structure of the model in anyway, or without introducing any new syntax. The below example shows clearly how this works. \n",
    "\n",
    "For this example, we have to import two helper functions from Talos, one for early stopping callout, and the other for using normalized learning rate values. Because we might want to work on trying out several optimizers in a single scan, without normalization, inputting of the values would become cumbersome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from talos.model.normalizers import lr_normalizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the only difference in the model below is how instead of using a label or value to define a given model parameter, we do it using a dictionary label. Also for optimizer we are using a learning rate parameter, which involves the use of two dictionary labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dropout, Dense\n",
    "\n",
    "def iris_model(x_train, y_train, x_val, y_val, params):\n",
    "    \n",
    "    model = Sequential()                            \n",
    "    model.add(Dense(params['first_neuron'],\n",
    "                    input_dim=x_train.shape[1],\n",
    "                    activation='relu'))\n",
    "    \n",
    "    model.add(Dropout(params['dropout']))\n",
    "    model.add(Dense(y_train.shape[1],\n",
    "                    activation=params['last_activation']))\n",
    "\n",
    "    model.compile(optimizer=params['optimizer'](lr=lr_normalizer(params['lr'], params['optimizer'])),\n",
    "                  loss=params['loss'],\n",
    "                  metrics=['acc'])\n",
    "\n",
    "    out = model.fit(x_train, y_train,\n",
    "                    batch_size=params['batch_size'],\n",
    "                    epochs=params['epochs'],\n",
    "                    verbose=0,\n",
    "                    validation_data=[x_val, y_val])\n",
    "    \n",
    "    return out, model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Setting the Parameter Space Boundaries <a name=\"three\"></a> \n",
    "In the last and final step, we're going to create the dictionary, which will then be passed on to Talos together with the model above. Here we have three different ways to input values:\n",
    "\n",
    "- as stepped ranges (min, max, steps)\n",
    "- as multiple values [in a list]\n",
    "- as a single value [in a list]\n",
    "\n",
    "For values we don't want to use, it's ok to set it as None.\n",
    "\n",
    "NOTE: at this point you have to import from Keras the optimizer, activations, and losses you want to scan for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.optimizers import Adam, Nadam\n",
    "from keras.activations import softmax\n",
    "from keras.losses import categorical_crossentropy, logcosh\n",
    "\n",
    "p = {'lr': (0.1, 10, 10),\n",
    "     'first_neuron':[4, 8, 16, 32, 64, 128],\n",
    "     'batch_size': [2, 3, 4],\n",
    "     'epochs': [200],\n",
    "     'dropout': (0, 0.40, 10),\n",
    "     'optimizer': [Adam, Nadam],\n",
    "     'loss': ['categorical_crossentropy'],\n",
    "     'last_activation': ['softmax'],\n",
    "     'weight_regulizer': [None]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Run the Hyperparameter <code>Scan()</code> <a name=\"four\"></a> \n",
    "Now we are ready to run the model based on the parameters and the layer configuration above. The exact same process would apply with any other model, just make sure to pass the model function name in the Scan() command as in the below example. To get started quickly, we're going to invoke the 'grid_downsample' parameter to 1/100 of the entire permutations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/36 [00:00<?, ?it/s]"
     ]
    }
   ],
   "source": [
    "h = ta.Scan(x, y, params=p,\n",
    "            model=iris_model,\n",
    "            dataset_name='iris',\n",
    "            experiment_no='1',\n",
    "            grid_downsample=.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Access the results through the <code>Scan</code> object <a name=\"five\"></a> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# accessing the results data frame\n",
    "h.data.head()\n",
    "\n",
    "# accessing epoch entropy values for each round\n",
    "h.peak_epochs_df\n",
    "\n",
    "# access the summary details\n",
    "h.details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to statistics and meta-data related with the Scan, the used data (x and y) together with the saved model and model weights for each hyperparameter permutation is stored in the Scan object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# accessing the saved models\n",
    "h.saved_models\n",
    "\n",
    "# accessing the saved weights for models\n",
    "h.saved_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Scan object can be further used, and is required, as input for Predict(), Evaluate(), and Deploy(). More about this in the corresponding sections below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Analysing the Scan results with <code>Reporting()</code> <a name=\"six\"></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the Scan process, the results are stored round-by-round in the corresponding experiment log which is a .csv file stored in the present working directory. The Reporting() accepts as its source either a file name, or the Scan object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use Scan object as input\n",
    "r = ta.Reporting(h)\n",
    "\n",
    "# use filename as input\n",
    "r = ta.Reporting('iris_1.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# access the dataframe with the results\n",
    "r.data.head(-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the number of rounds in the Scan\n",
    "r.rounds()\n",
    "\n",
    "# get the highest result ('val_acc' by default)\n",
    "r.high()\n",
    "\n",
    "# get the highest result for any metric\n",
    "r.high('acc')\n",
    "\n",
    "# get the round with the best result\n",
    "r.rounds2high()\n",
    "\n",
    "# get the best paramaters\n",
    "r.best_params()\n",
    "\n",
    "# get correlation for hyperparameters against a metric\n",
    "r.correlate('val_loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the key obsevations, several useful plots are available for analysis of the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a regression plot for two dimensions \n",
    "r.plot_regs()\n",
    "\n",
    "# line plot\n",
    "r.plot_line()\n",
    "\n",
    "# up to two dimensional kernel density estimator\n",
    "r.plot_kde('val_acc')\n",
    "\n",
    "# a simple histogram\n",
    "r.plot_hist(bins=50)\n",
    "\n",
    "# heatmap correlation\n",
    "r.plot_corr()\n",
    "\n",
    "# a four dimensional bar grid\n",
    "r.plot_bars('batch_size', 'val_acc', 'first_neuron', 'lr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Evaluating Models with <code>Evaluate()</code> <a name=\"seven\"></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models can be evaluated with <code>Evaluate()</code> against a k-fold cross-validation. Ideally at least 50% of the data, or more if possible, is kept completely out of the <code>Scan</code> process and only exposed into Evaluate once one or more candidate models have been identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e = ta.Evaluate(h)\n",
    "e.evaluate(x, y, folds=10, average='macro')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once a sufficiently performing model have been found, a deployment package can be easily created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Deploying Models with <code>Deploy()</code> <a name=\"eight\"></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the right model or models have been found, you can create a deployment package with <code>Deploy()</code> which is then easy to transfer to a production or other environment, send via email, or upload to shared remote location. Best model is automatically chosen based on a given metric ('val_acc' by default).\n",
    "\n",
    "The Deploy package is a zip file that consist of: \n",
    "\n",
    "- details of the scan\n",
    "- model weights\n",
    "- model json\n",
    "- results of the experiment\n",
    "- sample of x data\n",
    "- sample of y data\n",
    "\n",
    "The <code>Deploy</code> package can be easily restored with <code>Restore()</code> which is covered in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ta.Deploy(h, 'iris');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Restoring Models with <code>Restore()</code> <a name=\"nine\"></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models can be evaluated with <code>Evaluate()</code> against a k-fold cross-validation. Ideally at least 50% of the data, or more if possible, is kept completely out of the <code>Scan</code> process and only exposed into Evaluate once one or more candidate models have been identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = ta.Restore('iris.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <code>Restore</code> object now consists of the assets from the Scan object originally associated with the experiment, together with the model that had been picked as 'best'. The model can be immediately used for making prediction, or use in any other other way Keras model objects can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions with the model\n",
    "iris.model.predict(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, for book keeping purpose, and for simplicity of sharing models with team members and other stakeholders, various attributes are included in the <code>Restore</code> object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the meta-data for the experiment\n",
    "iris.details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the hyperparameter space boundary\n",
    "iris.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sample of x and y data\n",
    "iris.x\n",
    "iris.y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the results dataframe\n",
    "iris.results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img align=left src='https://img00.deviantart.net/920c/i/2014/137/a/2/pinkie_pie_thats_all_folks_by_dan232323-d7ipnd4.jpg' width=400>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Back to the repository page >> http://github.com/autonomio/talos"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}