{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Python Tour of Data Science: Data Exploitation\n",
    "\n",
    "[Michaël Defferrard](http://deff.ch), *PhD student*, [EPFL](http://epfl.ch) [LTS2](http://lts2.epfl.ch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise: problem definition\n",
    "\n",
    "Theme of the exercise: **understand the impact of your communication on social networks**. A real life situation: the marketing team needs help in identifying which were the most engaging posts they made on social platforms to prepare their next [AdWords](https://www.google.com/adwords/) campaign.\n",
    "\n",
    "This notebook is the second part of the exercise. Given the data we collected from Facebook an Twitter in the last exercise, we will construct an ML model and evaluate how good it is to predict the number of likes of a post / tweet given the content."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Data importation\n",
    "\n",
    "1. Use `pandas` to import the `facebook.sqlite` and `twitter.sqlite` databases.\n",
    "2. Print the 5 first rows of both tables.\n",
    "\n",
    "The `facebook.sqlite` and `twitter.sqlite` SQLite databases can be created by running the [data acquisition and exploration exercise](01_sol_acquisition_exploration.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>likes</th>\n",
       "      <th>comments</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>107201356009441_1349732228423008</td>\n",
       "      <td>Découvrez les photos du tournage de notre band...</td>\n",
       "      <td>2016-10-24 09:56:21</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>107201356009441_1346592535403644</td>\n",
       "      <td>Two architecture student turn downtown Lausann...</td>\n",
       "      <td>2016-10-21 17:00:01</td>\n",
       "      <td>143</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>107201356009441_1346583242071240</td>\n",
       "      <td>Adding a single chemical when making biofuels ...</td>\n",
       "      <td>2016-10-21 13:20:04</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>107201356009441_1345683875494510</td>\n",
       "      <td>Une très belle lumière ce soir sur le campus!</td>\n",
       "      <td>2016-10-20 17:07:48</td>\n",
       "      <td>531</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>107201356009441_1345641658832065</td>\n",
       "      <td>New online programs offered by the EPFL Extens...</td>\n",
       "      <td>2016-10-20 16:26:03</td>\n",
       "      <td>80</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  \\\n",
       "index                                     \n",
       "0      107201356009441_1349732228423008   \n",
       "1      107201356009441_1346592535403644   \n",
       "2      107201356009441_1346583242071240   \n",
       "3      107201356009441_1345683875494510   \n",
       "4      107201356009441_1345641658832065   \n",
       "\n",
       "                                                    text                time  \\\n",
       "index                                                                          \n",
       "0      Découvrez les photos du tournage de notre band... 2016-10-24 09:56:21   \n",
       "1      Two architecture student turn downtown Lausann... 2016-10-21 17:00:01   \n",
       "2      Adding a single chemical when making biofuels ... 2016-10-21 13:20:04   \n",
       "3          Une très belle lumière ce soir sur le campus! 2016-10-20 17:07:48   \n",
       "4      New online programs offered by the EPFL Extens... 2016-10-20 16:26:03   \n",
       "\n",
       "       likes  comments  \n",
       "index                   \n",
       "0         24         0  \n",
       "1        143         1  \n",
       "2         42         0  \n",
       "3        531         6  \n",
       "4         80         2  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>likes</th>\n",
       "      <th>shares</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>789509789755830272</td>\n",
       "      <td>Architecture students design a car-free Lausan...</td>\n",
       "      <td>2016-10-21 16:52:56</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>789500337619083264</td>\n",
       "      <td>A vitamin could help treat Duchenne muscular d...</td>\n",
       "      <td>2016-10-21 16:15:23</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>789492707790028800</td>\n",
       "      <td>RT @epfl_exts: Thanks to @EPFL president Patri...</td>\n",
       "      <td>2016-10-21 15:45:04</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>789471927735840768</td>\n",
       "      <td>A single chemical can transform a waste produc...</td>\n",
       "      <td>2016-10-21 14:22:29</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>789470735093956608</td>\n",
       "      <td>RT @physorg_com: Turning biofuel waste into we...</td>\n",
       "      <td>2016-10-21 14:17:45</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       id                                               text  \\\n",
       "index                                                                          \n",
       "0      789509789755830272  Architecture students design a car-free Lausan...   \n",
       "1      789500337619083264  A vitamin could help treat Duchenne muscular d...   \n",
       "2      789492707790028800  RT @epfl_exts: Thanks to @EPFL president Patri...   \n",
       "3      789471927735840768  A single chemical can transform a waste produc...   \n",
       "4      789470735093956608  RT @physorg_com: Turning biofuel waste into we...   \n",
       "\n",
       "                     time  likes  shares  \n",
       "index                                     \n",
       "0     2016-10-21 16:52:56      8       7  \n",
       "1     2016-10-21 16:15:23      1       2  \n",
       "2     2016-10-21 15:45:04      0       6  \n",
       "3     2016-10-21 14:22:29      5       5  \n",
       "4     2016-10-21 14:17:45      0      12  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from IPython.display import display\n",
    "import os.path\n",
    "\n",
    "folder = os.path.join('..', 'data', 'social_media')\n",
    "\n",
    "fb = pd.read_sql('facebook', 'sqlite:///' + os.path.join(folder, 'facebook.sqlite'), index_col='index')\n",
    "tw = pd.read_sql('twitter', 'sqlite:///' + os.path.join(folder, 'twitter.sqlite'), index_col='index')\n",
    "\n",
    "display(fb[:5])\n",
    "display(tw[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 Vectorization\n",
    "\n",
    "First step: transform the data into a format understandable by the machine. What to do with text ? A common choice is the so-called [*bag-of-word*](https://en.wikipedia.org/wiki/Bag-of-words_model) model, where we represent each word a an integer and simply count the number of appearances of a word into a document.\n",
    "\n",
    "**Example**\n",
    "\n",
    "Let's say we have a vocabulary represented by the following correspondance table.\n",
    "\n",
    "| Integer | Word    |\n",
    "|:-------:|---------|\n",
    "|    0    | unknown |\n",
    "|    1    | dog     |\n",
    "|    2    | school  |\n",
    "|    3    | cat     |\n",
    "|    4    | house   |\n",
    "|    5    | work    |\n",
    "|    6    | animal  |\n",
    "\n",
    "Then we can represent the following document\n",
    "> I have a cat. Cats are my preferred animals.\n",
    "\n",
    "by the vector $x = [6, 0, 0, 2, 0, 0, 1]^T$.\n",
    "\n",
    "**Tasks**\n",
    "\n",
    "1. Construct a vocabulary of the 100 most occuring words in your dataset.\n",
    "2. Build a vector $x \\in \\mathbb{R}^{100}$ for each document (post or tweet).\n",
    "\n",
    "Tip: the natural language modeling libraries [nltk](http://www.nltk.org/) and [gensim](https://radimrehurek.com/gensim/) are useful for advanced operations. You don't need them here.\n",
    "\n",
    "Arise a first *data cleaning* question. We may have some text in french and other in english. What do we do ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<816x200 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 12139 stored elements in Compressed Sparse Row format>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "['lire',\n",
       " 'ly',\n",
       " 'mars',\n",
       " 'martin',\n",
       " 'master',\n",
       " 'may',\n",
       " 'monde',\n",
       " 'more',\n",
       " 'new',\n",
       " 'nous']"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<3211x200 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 25217 stored elements in Compressed Sparse Row format>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "nwords = 200  # 100\n",
    "\n",
    "def compute_bag_of_words(text, nwords):\n",
    "    vectorizer = CountVectorizer(max_features=nwords)\n",
    "    vectors = vectorizer.fit_transform(text)\n",
    "    vocabulary = vectorizer.get_feature_names()\n",
    "    return vectors, vocabulary\n",
    "\n",
    "fb_bow, fb_vocab = compute_bag_of_words(fb.text, nwords)\n",
    "#fb_p = pd.Panel({'orig': fb, 'bow': fb_bow})\n",
    "display(fb_bow)\n",
    "display(fb_vocab[100:110])\n",
    "\n",
    "tw_bow, tw_vocab = compute_bag_of_words(tw.text, nwords)\n",
    "display(tw_bow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exploration question: what are the 5 most used words ? Exploring your data while playing with it is a useful sanity check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hand\n",
      "co\n",
      "http\n",
      "the\n",
      "epfl\n",
      "rt\n",
      "https\n",
      "to\n",
      "of\n",
      "in\n",
      "---\n",
      "off\n",
      "the\n",
      "http\n",
      "de\n",
      "epfl\n",
      "ly\n",
      "bit\n",
      "of\n",
      "la\n",
      "and\n"
     ]
    }
   ],
   "source": [
    "def print_most_frequent(bow, vocab, n=10):\n",
    "    idx = np.argsort(bow.sum(axis=0))\n",
    "    for i in range(10):\n",
    "        j = idx[0, -i]\n",
    "        print(vocab[j])\n",
    "\n",
    "print_most_frequent(tw_bow, tw_vocab)\n",
    "print('---')\n",
    "print_most_frequent(fb_bow, fb_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 Pre-processing\n",
    "\n",
    "1. The independant variables $X$ are the bags of words.\n",
    "2. The target $y$ is the number of likes.\n",
    "3. Split in half for training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3211, 200)\n",
      "(3211,)\n"
     ]
    }
   ],
   "source": [
    "X = tw_bow\n",
    "y = tw['likes'].values\n",
    "\n",
    "n, d = X.shape\n",
    "assert n == y.size\n",
    "\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Split: 1605 testing and 1606 training samples\n"
     ]
    }
   ],
   "source": [
    "# Training and testing sets.\n",
    "test_size = n // 2\n",
    "print('Split: {} testing and {} training samples'.format(test_size, y.size - test_size))\n",
    "perm = np.random.permutation(y.size)\n",
    "X_test  = X[perm[:test_size]]\n",
    "X_train = X[perm[test_size:]]\n",
    "y_test  = y[perm[:test_size]]\n",
    "y_train = y[perm[test_size:]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 Linear regression\n",
    "\n",
    "Using `numpy`, fit and evaluate the [linear model](https://en.wikipedia.org/wiki/Linear_regression) $$\\hat{w}, \\hat{b} = \\operatorname*{arg min}_{w,b} \\| Xw + b - y \\|_2^2.$$\n",
    "\n",
    "Please define a class `LinearRegression` with two methods:\n",
    "1. `fit` learn the parameters $w$ and $b$ of the model given the training examples.\n",
    "2. `predict` gives the estimated number of likes of a post / tweet. That will be used to evaluate the model on the testing set.\n",
    "\n",
    "To evaluate the classifier, create an `accuracy(y_pred, y_true)` function which computes the mean squared error $\\frac1n \\| \\hat{y} - y \\|_2^2$.\n",
    "\n",
    "Hint: you may want to use the function `scipy.sparse.linalg.spsolve()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `solve` and `spsolve` tells you that your matrix is singular, please read this [good comment](http://stats.stackexchange.com/questions/70899/what-correlation-makes-a-matrix-singular-and-what-are-implications-of-singularit). Potential solutions:\n",
    "1. Is there any post / tweet without any word from the vocabulary ? I.e. a row of $X$ made only of zeroes. If yes, remove this row or enlarge the vocabulary.\n",
    "2. Identify and remove redundant features, i.e. words, who are linear combinations of others.\n",
    "3. What else could we do ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mse: 4.3624\n"
     ]
    }
   ],
   "source": [
    "import scipy.sparse\n",
    "\n",
    "class LinearRegression(object):\n",
    "    \n",
    "    def predict(self, X):\n",
    "        \"\"\"Return the predicted class given the features.\"\"\"\n",
    "        return X.dot(self.w) + self.b\n",
    "    \n",
    "    def fit(self, X, y):\n",
    "        \"\"\"Learn the model's parameters given the training data, the closed-form way.\"\"\"\n",
    "        n, d = X.shape\n",
    "        self.b = y.mean()\n",
    "        A = X.T.dot(X)\n",
    "        b = X.T.dot(y - self.b)\n",
    "        #self.w = np.linalg.solve(A, b)\n",
    "        self.w = scipy.sparse.linalg.spsolve(A, b)\n",
    "\n",
    "def evaluate(y_pred, y_true):\n",
    "    return np.linalg.norm(y_pred - y_true, ord=2)**2 / y_true.size\n",
    "\n",
    "model = LinearRegression()\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "\n",
    "mse = evaluate(y_pred, y_test)\n",
    "print('mse: {:.4f}'.format(mse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interpretation: what are the most important words a post / tweet should include ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weight:  3.01, word: see\n",
      "weight:  2.24, word: 10\n",
      "weight:  2.12, word: just\n",
      "weight:  2.01, word: light\n",
      "weight: -1.91, word: upmicblog\n",
      "weight:  1.72, word: who\n",
      "weight: -1.67, word: technologisteu\n",
      "weight: -1.60, word: rt\n",
      "weight:  1.56, word: best\n",
      "weight:  1.46, word: congratulations\n",
      "weight:  1.45, word: startups\n",
      "weight: -1.39, word: 2012\n",
      "weight:  1.30, word: next\n",
      "weight:  1.29, word: rolex\n",
      "weight:  1.23, word: co\n",
      "weight:  1.20, word: years\n",
      "weight:  1.14, word: place\n",
      "weight: -1.08, word: method\n",
      "weight: -1.07, word: like\n",
      "weight:  1.06, word: university\n"
     ]
    }
   ],
   "source": [
    "idx = np.argsort(abs(model.w))\n",
    "\n",
    "for i in range(20):\n",
    "    j = idx[-1-i]\n",
    "    print('weight: {:5.2f}, word: {}'.format(model.w[j], tw_vocab[j]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 Interactivity\n",
    "\n",
    "1. Create a slider for the number of words, i.e. the dimensionality of the samples $x$.\n",
    "2. Print the accuracy for each change on the slider."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nwords = 200\n",
      "mse: 4.3624\n"
     ]
    }
   ],
   "source": [
    "import ipywidgets\n",
    "from IPython.display import clear_output\n",
    "\n",
    "slider = ipywidgets.widgets.IntSlider(\n",
    "    value=1,\n",
    "    min=1,\n",
    "    max=nwords,\n",
    "    step=1,\n",
    "    description='nwords',\n",
    ")\n",
    "\n",
    "def handle(change):\n",
    "    \"\"\"Handler for value change: fit model and print performance.\"\"\"\n",
    "    nwords = change['new']\n",
    "    clear_output()\n",
    "    print('nwords = {}'.format(nwords))\n",
    "    model = LinearRegression()\n",
    "    model.fit(X_train[:, :nwords], y_train)\n",
    "    y_pred = model.predict(X_test[:, :nwords])\n",
    "    mse = evaluate(y_pred, y_test)\n",
    "    print('mse: {:.4f}'.format(mse))\n",
    "\n",
    "slider.observe(handle, names='value')\n",
    "display(slider)\n",
    "\n",
    "slider.value = nwords  # As if someone moved the slider."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 Scikit learn\n",
    "\n",
    "1. Fit and evaluate the linear regression model using `sklearn`.\n",
    "2. Evaluate the model with the mean squared error metric provided by `sklearn`.\n",
    "3. Compare with your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mse: 4.3624\n"
     ]
    }
   ],
   "source": [
    "from sklearn import linear_model, metrics\n",
    "\n",
    "model = linear_model.LinearRegression()\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "\n",
    "mse = metrics.mean_squared_error(y_test, y_pred)\n",
    "assert np.allclose(evaluate(y_pred, y_test), mse)\n",
    "print('mse: {:.4f}'.format(mse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7 Deep Learning\n",
    "\n",
    "Try a simple deep learning model !\n",
    "\n",
    "Another modeling choice would be to use a Recurrent Neural Network (RNN) and feed it the sentence words after words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1606/1606 [==============================] - 0s - loss: 6.7979     \n",
      "Epoch 2/20\n",
      "1606/1606 [==============================] - 0s - loss: 6.0675     \n",
      "Epoch 3/20\n",
      "1606/1606 [==============================] - 0s - loss: 5.6889     \n",
      "Epoch 4/20\n",
      "1606/1606 [==============================] - 0s - loss: 5.4113     \n",
      "Epoch 5/20\n",
      "1606/1606 [==============================] - 0s - loss: 5.2405     \n",
      "Epoch 6/20\n",
      "1606/1606 [==============================] - 0s - loss: 5.0628     \n",
      "Epoch 7/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.9390     \n",
      "Epoch 8/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.8591     \n",
      "Epoch 9/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.8044     \n",
      "Epoch 10/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.6465     \n",
      "Epoch 11/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.5274     \n",
      "Epoch 12/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.4536     \n",
      "Epoch 13/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.3845     \n",
      "Epoch 14/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.2518     \n",
      "Epoch 15/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.1647     \n",
      "Epoch 16/20\n",
      "1606/1606 [==============================] - 0s - loss: 4.1496     \n",
      "Epoch 17/20\n",
      "1606/1606 [==============================] - 0s - loss: 3.9788     \n",
      "Epoch 18/20\n",
      "1606/1606 [==============================] - 0s - loss: 3.9037     \n",
      "Epoch 19/20\n",
      "1606/1606 [==============================] - 0s - loss: 3.8886     \n",
      "Epoch 20/20\n",
      "1606/1606 [==============================] - 0s - loss: 3.7960     \n",
      "mse: 3.1814\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ['KERAS_BACKEND'] = 'theano'  # tensorflow\n",
    "import keras\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Dense(output_dim=50, input_dim=nwords, activation='relu'))\n",
    "model.add(keras.layers.Dense(output_dim=20, activation='relu'))\n",
    "model.add(keras.layers.Dense(output_dim=1, activation='relu'))\n",
    "model.compile(loss='mse', optimizer='sgd')\n",
    "\n",
    "model.fit(X_train.toarray(), y_train, nb_epoch=20, batch_size=100)\n",
    "y_pred = model.predict(X_test.toarray(), batch_size=32)\n",
    "\n",
    "mse = evaluate(y_test, y_pred.squeeze())\n",
    "print('mse: {:.4f}'.format(mse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8 Evaluation\n",
    "\n",
    "Use [matplotlib](http://matplotlib.org) to plot a performance visualization. E.g. the true number of likes and the real number of likes for all posts / tweets.\n",
    "\n",
    "What do you observe ? What are your suggestions to improve the performance ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABMsAAAG3CAYAAACqtdK/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3XuYFPWB7/9P9ZWewWFspwdmGG94iQlRI6h4iYrETVaz\niUZJGYzIJbriZZOVrLfn0SDH/PxJVsHdxJgTjYCaw2Mt0TXrJjGJbFS8RA9wno14Iip4gUFobIaR\nmaEv03X+6Jqhh5lhpmf6Ut39fj0Pj05X9be+Xd3frqpPf7/fMmzbFgAAAAAAAADJU+oKAAAAAAAA\nAG5BWAYAAAAAAAA4CMsAAAAAAAAAB2EZAAAAAAAA4CAsAwAAAAAAAByEZQAAAAAAAICDsAwAAAAA\nAABwEJYBAAAAAAAADsIyAAAAAAAAwFEWYZlpmrNKXQcAg6ONAu5F+wTcjTYKuBftE3C3QrZR30if\naJrm7ZK+IekESV2SXpF0q2VZm7LW+ZOkc7OeZkv6n5ZlXZ/j5mZJWjXSugIoONoo4F60T8DdaKOA\ne9E+AXcrWBsdTc+ycyT9WNI0SRdI8kv6vWmaoax1bEk/lzRe0gRJTZJuGcU2AQAAAAAAgIIZcc8y\ny7Iuyv7bNM25knZKmippbdaiTsuyoiPdDgAAAAAAAFAsIw7LBlCvTE+y2AGPf9s0zdmSPpb0H5Lu\ntiyrK4/bBQAAAAAAAPIiL2GZaZqGpAckrbUs662sRb+U9IGkVkknSfqRpOMlzcyh+MMuvPDCiZLO\nkrQvH/UFkF+TJ08eJ2lKqesBoD/aJ+ButFHAvWifgKuNcbKiwyR9ku/CDdu2R12IaZoPSfqKpLMt\ny9p+kPXOl/RHScdalrVlgOWzlJmgrdeFF144cd68eXxBAQAAAAAAoNfy5cvX//a3v912wMOrLMsa\n1cT/ow7LTNP8iaSvSTrHsqwPh1i3RtJeSV+xLOsPw9zEWZJe3r17t1Kp1KjqCqAw6urq1N7eXupq\nABgA7RNwN9oo4F60T8C9fD6fDj30UEk6W9IreS9/NE92grKLJZ03VFDmOEWZec0G7X02gH2SlEql\nlEwmc68kgIKzbZv2CbgU7RNwN9oo4F60T6AsFGS6rhGHZaZp/lSZIZNfl9RhmuZ4Z9Eey7L2maY5\nSdIVkn6jzPjRkyUtlfSCZVlvjq7aAAAAAAAAQP6NpmfZAmV6if3pgMfnSXpMUkLSBZK+J6lW0keS\n/k3S/zeKbQIAAAAAAAAFM+KwzLIszxDLt0qaPtLyAQAAAAAAgGI7aOAFAAAAAAAAVBPCMgAAAAAA\nAMBBWAYAAAAAAAA4RjPBPwAAAAAAqAD19fXyeOhPA/dIp9Nqa2srybYJywAAAAAAqHIej0exWKzU\n1QB6hcPhkm2b2BgAAAAAAABwEJYBAAAAAAAADsIyAAAAAAAAwEFYBgAAAAAAADgIywAAAAAAAAAH\nYRkAAAAAAECZaWlp0bJly0pdjUFNmzZNc+fOLXU1RoSwDAAAAAAAoALt2LFDS5cu1VtvvVWQ8t95\n5x0tXbpU27Zt67fMMIyCbLMYCMsAAAAAAAAqUE9YtnHjxoKUv2nTJi1dulQfffRRQcovFcIyAAAA\nAACAA9i2rXg8XupqjIpt2zmt39XVlXP55dyDbDCEZQAAAAAAYFjsWFT25rdlx6JlU/4rr7yiCy+8\nUMccc4zOPvtsPfHEE7r//vvV0tLSZ72WlhbdeeedevrppzVjxgxNmjRJL7zwgqRMiLR48WKddtpp\nmjRpks4991z97Gc/6/P8rVu3qqWlRf/2b//Wrw4Hzi/Ws/33339f//iP/6jPfe5z+uxnP6uFCxdq\n3759fZ6bSCS0aNEinXTSSfrMZz6j+fPna/v27UO+7ldffVVf/epXZRiGbrrpJrW0tOjwww/vrd/M\nmTN1wQUX6C9/+YsuvfRSHXvssVqyZMmA9e0xbdo0LVy4UJJkWZYWLFjQW1ZP+a+99lqf57zxxhv6\nu7/7Ox1zzDE666yztHr16iHrXmq+UlcAAAAAAAC4m93VKXv1Ctk7W6VUUvL5ZTQ2y5g5V0aoxrXl\nv/nmm5o9e7bGjx+vm2++WalUSg888IDC4fCAPaLWrl2rZ599VnPmzFE4HO4N1ObMmaPXXntN3/rW\ntzR58mS98MIL+uEPf6gdO3Zo0aJFOderZ9sLFizQEUccodtvv11/+ctftGrVKkUiEd1+++29637/\n+9/Xv//7v+sb3/iGpk6dqpdffllXXXXVkD26jjvuOP3TP/2T7rvvPl155ZWaNm2aJOnUU0/tXScW\ni2n27Nn6+te/rpkzZ6qhoWFY9ZakM844Q/Pnz9fy5cv1ve99T8cee2zvdnts2bJF1157rb71rW/p\nm9/8pp588kktXLhQJ598cp/13IawDAAAAAAAHFRvkBUIZv5JsqPbpdUrZMy+3rXl33ffffJ6vXrm\nmWcUiUQkSV/72td03nnnDbj+5s2b9fzzz/cGP5L03HPP6ZVXXtFtt92mG2+8UVImPFuwYIF+8Ytf\naN68eTriiCNGVL+TTjpJP/rRj3r/3r17t1atWtUblr311lt6+umnNW/ePN1999292/6Hf/gH/fWv\nfz1o2Q0NDZoxY4buu+8+TZ06Vd/4xjf6rRONRrVkyRJdccUVOdf9iCOO0LRp07R8+XKdc845OuOM\nM/qts3nzZj311FM67bTTJGX2/WmnnaYnn3xSd9xxR87bLBaGYQIAAAAAgEHZsej+ICubPyB7Z+uo\nh0wWqvx0Oq21a9fqK1/5Sm9QJklHHnmkzj///AGfc+aZZ/YJyiRpzZo18vl8mj9/fp/H//7v/17p\ndFpr1qwZUf0Mw9CVV17Z57HTTz9du3fvVkdHR++2DcPQvHnz+qx39dVX5zwf2UACgYBM0xx1OYM5\n/vjje4MySQqHw5o0aZI++OCDgm0zHwjLAAAAAADA4NpimaGRA0klM8tdWP6uXbu0b98+HX300f2W\nHXXUUQM+5/DDD+/32LZt2zR+/HjV1PQdDtozjHDbtm0jqp8kTZw4sc/f48aNkyS1tbVJysyD5vF4\n+tX3mGOOGfE2s02YMEE+X+EGHTY3N/d7bNy4cdqzZ0/BtpkPhGUAAAAAAGBw9WHJ5x94mc+fWe7m\n8nMwZsyYfo8N1oPrwDnDBptDLJ1OD7o9j+fgsUw+eo8dTCgUymn97u7unNb3er0DPl7o1zVahGUA\nAAAAAGBQRjgio7FZSib6LkgmZIxvlhGODPzEEpff0NCgMWPGaMuWLf2WDfTYYFpaWrRjxw51dnb2\neXzTpk2S9vcO6+kV1t7e3me9rVu35lTvbIcffrjS6bTef//9Po+/++67w3r+UDcBGMxAvb+SyaR2\n7tyZl/LdjrAMAAAAAAAclDFzroxIk5SIS517pURcRmOTjMvmurZ8j8ejL37xi3ruuef6hDxbtmzR\nn/70p2GX86UvfUmpVErLly/v8/jDDz8sj8ejGTNmSJLGjh2rcDis1157rc96y5cvH3GodP7558u2\nbT366KN9Hn/kkUeGVWZPz7EDA7yhHHXUUfrzn//c57HHH3+8X8+ympoa2badc/lux90wAQAAAADA\nQRmhGhmzr89Mtt8Wk+rDo+5RVozyv//97+vFF1/UxRdfrKuuukqpVEorVqzQCSecoI0bNw6rjC9/\n+cs6++yztWTJEn3wwQeaPHmyXnjhBf3hD3/QNddc0+dOmLNmzdKDDz6om2++WSeddJL+/Oc/a8uW\nLSMedjh58mRdcsklWrlypfbs2aNTTz1Va9eu1QcffDCsMo866iiNGzdOjz/+uGpraxUKhTR16lS1\ntLQc9HmzZs3SbbfdpmuuuUbnnnuu3nrrLb344os67LDD+tXP6/XqwQcf1J49exQIBHTOOecoHC7e\n0NlCoGcZAAAAAAAYFiMckTHpM3kNygpZ/oknnqgnnnhC9fX1uu+++/Tkk0/qlltu0dlnn61gsO/d\nNw3DGLS31ooVK3T11VdrzZo1Wrx4sd59913deeed+sEPftBnvZtuuklXXHGFfvOb3+iee+6Rbdt6\n/PHHRzVccenSpZo/f75eeOEF3XPPPUqn03rssceGVabP59MDDzwgr9er22+/XTfeeKNeffXVIZ/3\n7W9/WzfccINef/113X333dq6datWrVqlmpqaPtuNRCJasmSJdu3apZtvvlk33nhj7/BUafBhmm4f\nvmm4fVI1SVMkrYtGo0omB7k7BoCSCofDisVGeQccAAVB+wTcjTYKuFe1tc9qe73f+c53tGnTJr30\n0kulrgoGcbDPpN/vVyQSkaSpktbne9v0LAMAAAAAABUrHo/3+Xvz5s1as2aNzjrrrBLVCG7HnGUA\nAAAAAKBinXXWWZo5c6aOPPJIffTRR3r88ccVDAZ13XXXlbpqcCnCMgAAAAAAULGmT5+uX//619q5\nc6eCwaCmTp2q2267TUcddVSpqwaXIiwDAAAAAAAV6/777y91FVBmmLMMAAAAAAAAcBCWAQAAAAAA\nAA7CMgAAAAAAAMBBWAYAAAAAAAA4CMsAAAAAAAAAB2EZAAAAAAAA4CAsAwAAAAAAAByEZQAAAAAA\nAAUybdo0LVy4sPfvV199VS0tLXrttdfyto2WlhYtW7Ysb+VVO8IyAAAAAACAAjEMY1iPDWXNmjVa\nunTpsLeBkfOVugIAAAAAAADV4swzz9R7772nQCCQ0/PWrFmjlStX9uml1uO9996Tz0fEky/0LAMA\nAAAAAMhi27bi8XjBys81KJMydTpYeR4PEU++sCcBAAAAAMCwRDuSentXl6IdybIo//7771dLS4ve\nffddXXvttTrhhBP0+c9/Xj/4wQ/6hGEtLS2688479fTTT2vGjBmaNGmSXnjhBUmZkOrhhx/WjBkz\ndMwxx+gLX/iCbr31Vu3Zs6ff9h544AGdeuqpOvbYY2WapjZt2tRvncHmLFu/fr1mz56tyZMn67jj\njtMFF1ygRx99VJJ00003aeXKlb11bWlp0eGHH96n/gfOWfbmm2/qyiuv1AknnKDjjz9el19+udav\nX99nHcuy1NLSojfeeEN33XWXTjrpJB133HG6+uqrFYvFctnVFYU+egAAAAAA4KA6k91auSGqbe0J\npbpt+byGJtYFNOeUiGr8XteW3zOX14IFC3TEEUfo9ttv1/r16/Xoo4+qvb1dDzzwQO+6a9eu1bPP\nPqs5c+YoHA6rpaVFknTLLbdo9erVuvzyy/Wd73xHH374oZYvX66NGzfqmWeekdebqd+PfvQj/eu/\n/qsuuOACnX/++XrzzTd1xRVXKJnsH/wdOMfYiy++qLlz52r8+PG6+uqrFYlE9M477+iPf/yj5s+f\nryuvvFI7duzQSy+9pJ/85CcH7WUmSZs2bdKll16qQw45RDfccIN8Pp+eeOIJffOb39SvfvUrfeEL\nX+iz/p133qn6+notXLhQW7du1cMPP6w77rhDP/3pT3Pf6RWAsAwAAAAAABzUyg1RtbbHNcbnkXyZ\noKe1PaGVG6K67vQJri//qKOO0iOPPCJJmjNnjsaOHavHHntMCxYs0AknnCBJ2rx5s55//nkde+yx\nvc97/fXXtWrVKj344IO6+OKLex8/++yzdcUVV+jZZ5/VxRdfrFgspp/97Gf6m7/5Gy1fvrx3vSVL\nlujHP/7xQeuWTqd16623asKECfr973+vsWPH9ltn6tSpmjRpkl566SVdcsklQ77eJUuWqLu7W888\n80xv6HfZZZfp3HPP1Q9/+EOtXr26z/qHHXaYfvnLX/b+3d3dreXLl2vv3r0D1qfSMQwTAAAAAAAM\nKtqR1Lb2hIK+vhFC0GeotT0x6iGThS7fMAzNmTOnz2Pz5s2Tbdtas2ZN72Nnnnlmn6BMkp599lmN\nGzdO55xzjmKxWO+/z3/+86qtrdXLL78sKdMzLJlMat68eX2ef8011wxZvzfffFMfffSRrr766rwE\nU+l0Wi+++KL+9m//tjcok6TGxkZdcsklev3119XR0dH7uGEY+va3v92njGnTpqm7u1tbt24ddX3K\nET3LAAAAAADAoGJdKaW67d4eX9mSaVuxrpQitX7Xli9JRx99dL+/PR5PnzAoew6wHlu2bNGePXt0\n0kkn9VtmGIY++eQTSdK2bdsG3E44HNa4ceMOWrf3339fhmHo+OOPH96LGcInn3yirq4uTZo0qd+y\n4447TrZtq7W1Vccdd1zv483NzX3W66nzQPOyVQPCMgAAAAAAMKhwyCeft3+QJUl+j6FwaHTRQqHL\nH64xY8b0e8y2bUUikUHnCTvssMN615P6z0U2HEPNP1aM8nrmXctHWZWAsAwAAAAAAAwqUuvXxLqA\nWtsTCmb1/oqnbDXXBUbd66vQ5UuZ+ciyhyRu2bJF6XR6wN5k2Y488kitXbtWp556qoLB4KDr9ZRz\n4HZisdiQvbOOPvpo2batt99+W1/84hcHXW+4QVxDQ4NCoZDee++9fsveeecdGYbRrycZ+mLOMgAA\nAAAAcFBzTomouS6geMrW3kS6N8iac0rE9eXbtq2VK1f2eezRRx+VYRg6//zzD/rcr33ta0qlUlq2\nbFm/Zd3d3Wpvb5cknXPOOfL5fHr00Uf7rPPzn/98yPqdeOKJOuKII/TII4/0ljeQmpoaSdKnn356\n0PI8Ho/OO+88Pffcc73DQyUpGo3qmWee0bRp01RbWztkvaoZPcsAAAAAAMBB1fi9uu70CYp2JBXr\nSikc8uWlx1exyv/www81b948TZ8+XevWrdNTTz2lSy+9tPdOmIM544wzdOWVV+rBBx/Uxo0bdd55\n58nn82nz5s36z//8T91999266KKLFA6Hde211+rBBx/UVVddpRkzZmjjxo36r//6r96hmtmyhzca\nhqF77rlH8+fP15e//GVdfvnlamxs1Lvvvqt33nlHTzzxhKRMqGbbtu644w5Nnz5dHo+nzx06s91y\nyy166aWXdPHFF2vOnDnyer365S9/qUQioTvuuGPQugzn8WpAWAYAAAAAAIYlUuvPa4hVjPINw9BD\nDz2kf/7nf9a9994rr9er+fPn9wmNDMMYdJjjvffeq5NPPllPPPGElixZIp/Pp5aWFs2cOVOnnXZa\n73q33XabxowZo8cff1yvvvqqpkyZolWrVumqq67qV/aBf0+fPl2WZWnZsmX6+c9/rnQ6rSOPPLLP\nXSovuugizZ8/X7/+9a/19NNPy7bt3rDswPKOP/54PfXUU7r33nv14IMPKp1Oa8qUKfrJT36ik08+\n+aB1GerxamCUQVI4RdK6aDSqZHJ0t4sFUBjhcFixWKzU1QAwANon4G60UcC9qq19VurrXbp0qZYt\nW6b//u//1qGHHlrq6iAHB/tM+v1+RSIRSZoqaX2+t82cZQAAAAAAAICDsAwAAAAAAABwEJYBAAAA\nAAAAjhFP8G+a5u2SviHpBEldkl6RdKtlWZuy1glKWirpcklBSc9Jut6yrJ2jqTQAAAAAAMBQFi5c\nqIULF5a6Gigzo+lZdo6kH0uaJukCSX5JvzdNM5S1zgOSvirpMknnSmqW9KtRbBMAAAAAAAAomBH3\nLLMs66Lsv03TnCtppzJ3IlhrmmadpPmSvmVZ1gvOOvMk/V/TNE+3LOv1EdcaAAAAAAAAKIB8zllW\nL8mW1HNfz6nKhHHP96xgWdbbkj6UdGYetwsAAAAAAADkRV7CMtM0DWWGXK61LOst5+EJkhKWZbUf\nsPoOZxkAAAAAAADgKiMehnmAn0r6nKQvDmNdQ5keaP2YpjlL0qzsxyZPnjxu0aJFqqurk20P+DQA\nJeb3+xUOh0tdDQADoH0C7kYbBdyr2tqnx5PPgWfA6Hk8nkHboGEYkqTFixcv27hx454DFq+yLGvV\naLZtjDaAMk3zJ5K+Jukcy7I+zHr8fEl/lHRodu8y0zTfl7TMsqx/GeYmpkhaF41GlUwmR1VXAIUR\nDocVi8WGXhFA0dE+AXejjQLuVW3ts76+nsAMrpJOp9XW1jbgMr/fr0gkImWmAFuf722PqmeZE5Rd\nLOm87KDMsU5SStKXJD3trH+8pCMkvTqa7QIAAAAAgPwZLJQAqtGIwzLTNH+qzJDJr0vqME1zvLNo\nj2VZ+yzLajdN8xeSlpqmuVvSp5L+VdLL3AkTAAAAAAAAbjSanmULlJl77E8HPD5P0mPO/98kqVvS\naklBSb+TdMMotgkAAAAAAAAUzIjDMsuyhhzMbFlWXNI/OP8AAAAAAAAAV2P2PgAAAAAAAMBBWAYA\nAAAAAAA4CMsAAAAAAAAAB2EZAAAAAAAA4CAsAwAAAAAAAByEZQAAAAAAAICDsAwAAAAAAABwEJYB\nAAAAAAAADsIyAAAAAAAAwEFYBgAAAAAAADgIywAAAAAAAAAHYRkAAAAAAADgICwDAAAAAAAAHIRl\nAAAAAAAAgIOwDAAAAAAAAHAQlgEAAAAAAAAOwjIAAAAAAADAQVgGAAAAAAAAOAjLAAAAAAAAAAdh\nGQAAAAAAAOAgLAMAAAAAAAAchGUAAAAAAACAg7AMAAAAAAAAcBCWAQAAAAAAAA7CMgAAAAAAAMBB\nWAYAAAAAAAA4CMsAAAAAAAAAB2EZAAAAAAAA4CAsAwAAAAAAAByEZQAAAAAAAICDsAwAAAAAAABw\nEJYBAAAAAAAADsIyAAAAAAAAwEFYBgAAAAAAADgIywAAAAAAAAAHYRkAAAAAAADgICwDAAAAAAAA\nHIRlAAAAAAAAgIOwDAAAAAAAAHAQlgEAAAAAAAAOwjIAAAAAAADAQVgGAAAAAAAAOAjLAAAAAAAA\nAAdhGQAAAAAAAOAgLAMAAAAAAAAchGUAAAAAAACAg7AMAAAAAAAAcBCWAQAAAAAAAA7CMgAAAAAA\nAMBBWAYAAAAAAAA4CMsAAAAAAAAAB2EZAAAAAAAA4CAsAwAAAAAAABy+kT7RNM1zJN0saaqkJkmX\nWJb166zlyyXNOeBpv7Ms66KRbhMAAAAAAAAopNH0LKuV9H8k3SDJHmSd30oaL2mC82/WKLYHAAAA\nAAAAFNSIe5ZZlvU7Sb+TJNM0jUFWi1uWFR3pNgA3sWNRqS0m1YdlhCOlrg4AAAAAACiAEYdlwzTd\nNM0dknZLWiPpDsuyYgXeJpBXdlen7NUrZO9slVJJyeeX0dgsY+ZcGaGaUlcPAAAAAADkUSEn+P+t\npKskzZB0i6TzJP3mIL3QAFfqDcoCQalmrBQIyo5ul716RamrBgAAAAAA8qxgPcssy7Ky/txomuZf\nJL0nabqk/yrUdoF8smPR/UFZNn9A9s5W2bEoQzIBAAAAAKgghR6G2cuyrC2mae6SdKwGCctM05yl\nA24CMHny5HGLFi1SXV2dbHuw+wgAhZHctV17JXn8/n7L0sm4xqZT8ofDxa+Yy/j9foXZD4Ar0T4B\nd6ONAu5F+wTcyzAygxYXL168bOPGjXsOWLzKsqxVoym/aGGZaZotkg6TtH2wdZwXc+ALmiJpXXt7\nu5LJZAFrCPRne3xKS9JAnz1bavf4ZMSYhi8cDivGfgBcifYJuBttFHAv2ifgXn6/X5FIRIsWLbpJ\n0vp8lz/isMw0zVpleon1zEE2yTTNkyXFnH+LJP1K0sfOekskbZL03GgqDBSTEY7IaGyWHd0u+QP7\nFyQTMsY3MwQTAAAAAIAKM5oJ/k+VtEHSOkm2pPuVSfMWS+qWdJKkZyS9LelhSW9IOteyLLqHoawY\nM+fKiDRJibjUuVdKxGU0Nsm4bG6pqwYAAAAAAPJsxD3LLMt6QQcP2/52pGUDbmKEamTMvl52LCq1\nxaT6MD3KAAAAAACoUEWbswwod0Y4IhGSAQAAAABQ0UYzDBMAAAAAAACoKIRlAAAAAAAAgIOwDAAA\nAAAAAHAQlgEAAAAAAAAOwjIAAAAAAADAQVgGAAAAAAAAOAjLAAAAAAAAAAdhGQAAAAAAAOAgLAMA\nAAAAAAAchGUAAAAAAACAg7AMAAAAAAAAcBCWAQAAAAAAAA7CMgAAAAAAAMBBWAYAAAAAAAA4CMsA\nAAAAAAAAB2EZAAAAAAAA4CAsAwAAAAAAAByEZQAAAAAAAICDsAwAAAAAAABwEJYBAAAAAAAADsIy\nAAAAAAAAwEFYBgAAAAAAADgIywAAAAAAAAAHYRkAAAAAAADgICwDAAAAAAAAHIRlAAAAAAAAgIOw\nDAAAAAAAAHAQlgEAAAAAAAAOwjIAAAAAAADAQVgGAAAAAAAAOAjLAAAAAAAAAAdhGQAAAAAAAOAg\nLAMAAAAAAAAchGUAAAAAAACAg7AMAAAAAAAAcBCWAQAAAAAAAA7CMgAAAAAAAMBBWAYAAAAAAAA4\nCMsAAAAAAAAAB2EZAAAAAAAA4CAsAwAAAAAAABy+UlcAyCc7FpXaYlJ9WEY4UurqAAAAAACAMkNY\nhopgd3XKXr1C9s5WKZWUfH4Zjc0yZs6VEaopdfUAAAAAAECZYBgmKkJvUBYISjVjpUBQdnS77NUr\nSl01AAAAAABQRgjLUPbsWHR/UJbNH5C9szUzNBMAAAAAAGAYCMtQ/tpimaGXA0klM8sBAAAAAACG\ngbAM5a8+LPn8Ay/z+TPLAQAAAAAAhoGwDGXPCEdkNDZLyUTfBcmEjPHN3BUTAAAAAAAMG2EZKoIx\nc66MSJOUiEude6VEXEZjk4zL5pa6agAAAAAAoIz4Sl0BIB+MUI2M2ddnJvNvi0n1YXqUAQAAAACA\nnBGWoaIY4YhESAYAAAAAAEaIYZgAAAAAAACAg7AMAAAAAAAAcIx4GKZpmudIulnSVElNki6xLOvX\nB6zzPyRdLale0suSrrMs692RVxcAAAAAAAAonNH0LKuV9H8k3SDJPnChaZq3SrpR0rWSTpfUIek5\n0zQDo9gmAAAAAAAAUDAj7llmWdbvJP1OkkzTNAZY5XuS7rYs6z+cda6StEPSJZKskW4XAAAAAAAA\nKJSCzFlmmubRkiZIer7nMcuy2iX9WdKZhdgmAAAAAAAAMFqFmuB/gjJDM3cc8PgOZxkAAAAAAADg\nOiMehjlChgaY36yHaZqzJM3Kfmzy5MnjFi1apLq6Otn2oE8FUEJ+v1/hcLjU1QAwANon4G60UcC9\naJ+AexlQxbCBAAAgAElEQVRGZjawxYsXL9u4ceOeAxavsixr1WjKL1RY9rEywdh49e1d1ihpw2BP\ncl7MgS9oiqR17e3tSiaT+a4ngDwIh8OKxWKlrgaAAdA+AXejjQLuRfsE3Mvv9ysSiWjRokU3SVqf\n7/ILMgzTsqwtygRmX+p5zDTNOknTJL1SiG0CAAAAAAAAozXinmWmadZKOlaZHmSSNMk0zZMlxSzL\n+kjSA5LuME3zXUnvS7pb0lZJz4yqxgAAAAAAAECBjKZn2anKDKlcp8w8ZPcr0/VtsSRZlvUjST+W\n9D+VuQtmSNKFlmUlRlNhAAAAAAAAoFCMMpg0f4qkddFolDnLAJdiPgfAvWifgLvRRgH3on0C7tUz\nZ5mkqSqXOcsAAAAAAACAckRYBgAAAAAAADgIywAAAAAAAADHiO+GCQAAKosdi0ptMak+LCMcKXV1\nABQR7R8AgP0IywAAqHJ2V6fs1Stk72yVUknJ55fR2Cxj5lwZoZpSVw9AAdH+AQDoj2GYAABUud4L\n5UBQqhkrBYKyo9tlr15R6qoBKDDaPwAA/RGWAQBQxexYdP+FcjZ/QPbO1szQLAAVifYPAMDACMsA\nAKhmbbHM0KuBpJKZ5QAqE+0fAIABEZYBAFDN6sOSzz/wMp8/sxxAZaL9AwAwIMIyAACqmBGOyGhs\nlpKJvguSCRnjm7krHlDBaP8AAAyMsAwAgCpnzJwrI9IkJeJS514pEZfR2CTjsrmlrhqAAqP9AwDQ\nn6/UFQAAAKVlhGpkzL4+M5l3W0yqD9OjBKgStH8AAPojLAMAAJIyQ7LERTJQlWj/AADsxzBMAAAA\nAAAAwEFYBgAAAAAAADgIywAAAAAAAAAHYRkAAAAAAADgICwDAAAAAAAAHIRlAAAAAAAAgIOwDAAA\nAAAAAHAQlgEAAAAAAAAOwjIAAAAAAADAQVgGAAAAAAAAOAjLAAAAAAAAAAdhGQAAAAAAAOAgLAMA\nAAAAAAAchGUAAAAAAACAg7AMAAAAAAAAcBCWAQAAAAAAAA7CMgAAAAAAAMBBWAYAAAAAAAA4CMsA\nAAAAAAAAB2EZAAAAAAAA4CAsAwAAAAAAAByEZQAAAAAAAICDsAwAAAAAAABw+EpdAQAAAAAAAFSW\naEdSsa6UwiGfIrX+UlcnJ4RlAAAAAAAAyIvOZLdWbohqW3tCqW5bPq+hiXUBzTklohq/t9TVGxaG\nYQIAAAAAACAvVm6IqrU9rjE+Q2ODHo3xGWptT2jlhmipqzZshGUAAAAAAAAYtWhHUtvaEwr6+sZN\nQScwi3YkS1Sz3BCWAQAAAAAAYNRiXSmluu0BlyXTtmJdqSLXaGQIywAAAAAAADBq4ZBPPq8x4DK/\nx1A4VB5T5xOWAQAAACgYOxaVvflt2bHymasGAKpBIb6fI7V+TawLKJ7q27ssnrLVXBcom7tilkek\nBwAAAKCs2F2dslevkL2zVUolJZ9fRmOzjJlzZYRqSl09AKhahf5+nnNKxJnkP6Fk2pbfY6jZuRtm\nuSAsAwAAAJB3vRdigWDmnyQ7ul1avULG7OtLXDsAqF6F/n6u8Xt13ekTFO1IKtaVUjjkK5seZT0Y\nhgkAAAAgr+xYdP+FWDZ/QPbOVoZkAkCJFPP7OVLr12caQmUXlEmEZQAAAADyrS2WGdozkFQysxwA\nUHx8Pw8LYRkAAACA/KoPS75BehL4/JnlAIDi4/t5WAjLAAAAAOSVEY7IaGyWkom+C5IJGeObZYTL\nZ5JnAKgkfD8PD2EZAAAAgLwzZs6VEWmSEnGpc6+UiMtobJJx2dxSVw0Aqhrfz0PjbpjAMJXznTwA\nAACKzQjVyJh9fWay6LaYVB+mxwIAuADfz0MjLAOG0Jns1soNUW1rTyjVbcvnNTSxLqA5p0RU4/eW\nunoAAACuZoQjEhdhAOA6fD8PjmGYwBBWboiqtT2uMT5DY4MejfEZam1PaOUGbnkOAAAAAEClKWjP\nMtM0F0ladMDDf7Us63OF3C6QL9GOpLa1JzTG1zdXDjqBWbQjyZBMAAAAAAAqSDGGYb4p6UuSDOfv\nVBG2CeRFrCulVLct+Yx+y5JpW7GuFGEZAAAoe8zNCgDAfsUIy1KWZTFeDWUpHPLJ5+0flEmS32Mo\nHGLaPwDVi4troPwxNysAAP0V40r/ONM0t0naJ+lVSbdblvVREbYLjFqk1q+JdQG1ticUzOpdFk/Z\naq4LcHEIoCpxcQ1Ujv1zs3p6e9L3zM163ekTSlw7AABKo9AT/L8maa6kr0haIOloSS+apllb4O0C\neTPnlIia6wKKp2ztTaR7g7I5p3DXEADViRufAJWhZ27W4EHmZgUAoBoVtGeZZVnPZf35pmmar0v6\nQJIpaXkhtw3kS43fq+tOn8BwIwAQNz4BKglzswIAMLCiTrhkWdYe0zQ3STp2oOWmac6SNCv7scmT\nJ49btGiR6urqZNt2MaoJDCgcLnUN3Mvv9yvMDgJcKd/tc3viU8njkd/f/wI6nk4p5a9ROHxI3rYH\nVLpSHkMn+eMKjdktv6//8Okao1uTmhoUPiRYgpoB7sA5LuBehpH5oWfx4sXLNm7cuOeAxassy1o1\nmvKLGpaZpjlW0jGSHhtoufNiDnxBUySta29vVzJJV3DAjcLhsGKxWKmrAWAA+W6fvmRSSqeVTCb6\nL0zb8iU7FYtxvAaGq5THUL+kxpBHre3xAedm9Sc7FIt1lKRugBtwjgu4l9/vVyQS0aJFi26StD7f\n5Rc0LDNN858l/YcyQy8nSlosKaX+gRgAACgD3PgEqCxzTok48xAmlEzb8nsM5mYFAFS9Qvcsa5H0\nvyQdJikqaa2kMyzL+qTA2wUAAAXCxTVQOZibFQCA/go9wf+sodcCAADlhItroPJEav20YwAAHEWd\nswwAAFQOLq4BAABQiQjLgAKhxwUAAAAAAOWHsAzIs85kt1ZuiGpbe0Kpbls+r6GJzlw+Nf7+t2YH\nAAAAAADu4Sl1BfIt2pHU27u6FO3gtvUojcyk13GN8RkaG/RojM9Qa3tCKzdES101AAAAAAAwhIrp\nWdbbm+eTvUolUvIFfJp42Fh686Cooh1JbWtPaIzPIzuRkFJJyedXMBBQa3tC0Y5kxQ3JtGNRJXdt\nl+3xyQhzJzwA+WHHolJbTKoP890CAACAoqqYsGzFG9vVuvlDBeOdCtppyfCo9ZOQVqRSuv6sllJX\nD1Ui1pVSKpmS/XGrFN8nOZ9FOzhGyYZmxbpSFROW2V2dslevkL2zVXslpSUZjc0yZs6VEaopdfUA\nlKns75aeHxz4bgEAAEAxVcQwzGhHsjcok8cjeX2Sx6NAokutmz9kSCaKJhzyyffJx5mgLOuzqMQ+\n+XZ9rHCoYvLp/RezgaA8tWOlQFB2dLvs1StKXTUAZSz7u0U1fLcAAACg+CoiLIvtiCqZTGZCiWyG\nR6lkUrEdzBWF4miIt6mpa5finr69x+KGX81dUTXE20pUs/yyY9H9F7PZ/AHZO1szw6cAIEd8twAA\nAMANKiIsOzTeLn+6e8BlvnS3Do23F7lGqFptMV21+3U1pT9VXF51GH7F5VVz+lPN3v1GZv6dStAW\nywyPGkgqWTmvE0Bx8d0CAAAAF6iIMWGR8YepKb1J2z3jFNT+0CwTUrQpMv6wEtYOVaU+rBqvtKBz\ng3Z5QooZIYXtLjWkuyRvZnlFqA9LvkHmXvP5K+d1AiguvlsAAADgAhXRs8wIRzRnzHY1pfb07c2T\n2qOrQh9zFy0UjRGOyGhslpIJNaS7dHx3LBOUJRMyxjdXzGcx+3X2UWGvE0BxZX+37PKEtMkb1i5P\niO8WAAAAFJVh23ap6zCUKZLWRaPOvGSD6Ll7VnRXm2LyK6ykIpF6GZdx9ywU14B3chvfXHGfxezX\n6TOklK2KfJ1AuQuHw4rFymf4Ykf7Xq185jW1Jr1K2ob8hq3mQLfmfP0M1daNLXX1gLwrtzYKVBPa\nJ+Befr9fkUhEkqZKWp/v8ismLOthx6KZOU3qw/wCjZKqls+iHYuqLp1Su8dX0a8TKFfldqL/0Osf\nq7U9rkA61fuDQ8LjV3NdQNedPqHU1QPyrtzaKFBNaJ+AexU6LKuIOcuyGeGIxAU7XKBaPotGOCJ/\nOCyDEwkAoxTtSGpbe0JjfB5JASkQkCQFJbW2JxTtSCpSO8icZgAADCHakVSsK6VwyMfxBMBBVVxY\nBgAAylOsK6VUty35jH7Lkmlbsa4UFzdAGSKgQKl1Jru1ckNU29oTSnXb8nkNTawLaM4pEdX4vaWu\nHgAXIiwDAACuEA755PP2D8okye8xFA5x2gKUEwIKuMXKDVG1tsczPZedH2Ra2xNauSHKEH8AA6qI\nu2GivEQ7knp7V5eiHUPPQQcAqB6RWr8m1gUUT/WdTzWestVcF6BHClBm9gcUhsYGPRrjM3oDCqBY\neob4B319L32DzueRaxIAA+EnWhQNvy4CQGUpxI1M5pwScS6wE0qmbfk9hpqdYwWA8tF3DsL9sgMK\nAnAUQ/YQfzuR2H+3+kCgYof4V9ONxtzyOt1UF+QHYRmKhu7PAFAZ7K5O2atXyN7Zuv+io7FZxsy5\nMkI1oyq7xu/VdadPYI4joMwxByHcIhzyyWekZW/9SIrvk+y0ZHhkB8fI1zixoob4F/L47CZuep1u\nqgvyi2GYKAq6PwNA5eg9KQwEpZqxUiAoO7pd9uoVedtGpNavzzSEuJgGyhRzEMItIrV+Ne94T/FE\nUvJ4JK9P8ngUTyTVvOO9ijrOFOP47AZuep1uqgvyi7AMRdH76+IAen5dBAC4nx2L7j8pzOYPyN7Z\nmhmGAKDqMQch3MKORTU7+rKa7A7F5VWH4VdcXjXbHZodfblijlvVcnx20+t0U12Qf/ykg6Lg10UA\n5YwhgVnaYplhBgeeGEqZx9tiEnN1ABBzEMIl2mKqSXZqQecG7fKEFDNCCttdakh3ScnOyjluVcvx\n2U2v0011Qd6RUKAoen5dbG1PKJg1dwW/LgJwM25MMoD6sOQb5Dvb588sBwAxByFcIuu41ZDuUoO6\n9i+rpONWtRyf3fQ63VQX5B3DMFE0c06JqNnpjr83ke4Nyvh1EYBb7b8xiaGxQY/GOPMsrtxQvd3q\njXBERmOzlEz0XZBMyBjfzB2gAPTDHIQopWo5bvE6i/863VQX5B9hGYqm59fF757ZpHlTGvXdM5t0\n3ekTqrd3BgBX48YkgzNmzpURaZIScalzr5SIy2hsknHZ3FJXDQCAfqrluMXrnFvVdUF+MQwTRRep\n9fPLIgDX670xia//fIs9Nyap1u8yI1QjY/b1mYlr22JSfZhfTx0MNwMA96mW4xavs7rrgvwiLAMA\nYADcmGRoRjjCxLUO5rcDAPerluMWr7P43FQX5AfDMAGgxKIdSb29q6uqh/W5Uc+NSeIpu8/j3JgE\nA2F+OwAAUG64DhkcP4sDQInQE8X95pwScUKQhJJpW36PUdE3JmEI4cj0zG835iDz27E/AQCAW3Ad\nMjTCMgCjEu1IanviU/mSXAxmG868Bft7onh658Xq6Yly3ekTilndoijHuRx6bkxS6SFSJZwwFfLz\nNVTZ2fPb2YmElEpKPr+MQKDq57cbqXL8vkD+8P4DGI5KPz8bKa5D8oOwDMCIZF9cy+OR0umyu7gu\nBLurU/bqFbJ3tu6/YG5sztwpJ1TTu1419UQZ7j5xs0q/MUk5nzAV8vM13LLDIZ98Rlr21o+k+D7J\nTkuGR3ZwjHyNE5nfLgeV8H2BkeP9BzAclfAjXyFwHZJfzFkGYESy5+c5JOhnfh5H7wEqEJRqxkqB\noOzodtmrV/RZr7cnygB6eqJUiuHuE5RGzwlT8CAnTG5WyM/XcMuO1PrVvOM9xRPJzI8HXp/k8Sie\nSKp5x3uccOaA74vqxvsPYDiYJ3RgXIfkF2EZgJyV+8V1odix6P4DVDZ/QPbO1kyXaEe13Gkxe5/s\n8oS0yRvWLk9owH2C0ijnE6Zc2lwhy7ZjUc2Ovqwmu0NxedVh+BWXV812h2ZHX+ZzPkyFfD/hfrz/\nAIaD65CBcR2Sf2WzF3Z3pTS2bGoLVLbs+XkOVNXz87TFMl2eDzxISZnH22K9t5TuudNia3tCwaz9\nWHF3WmyLqbNbeqzmFG33HqKkPPIrrabuT3VV12sam7VPUBplfcKUQ5sraNltMdUkO7Wgc4N2eUKK\nGSGF7S41pLukZOfo6lFNCvl+wv14/+FgHiocDNchg+A6JO9cfAbc1//676i6U6mqH4cMuEFZX1wX\nUn1Y8g1ycPH5M8uzVMWdFuvDeuzQ07Xdc4iCdreC6pYkbfccoscOPV3XH7BPUHxlfcKUY5srWNlZ\n6zaku9SgrvzVo5oU8v2E+/H+Vz3mocqfSg4cuQ4ZBNcheVc2n6Sg19C7sfKYbBiodGV9cV1ARjgi\no7FZdnS75A/sX5BMyBjf3O9uNJVwp8Wh6r4rWK/toQYFE52Ssb+7fNBOanuoQbuC9eKQXHrlesKU\na5srVNmFrEc1YT9Wt+z3f1dw3P4emvE9vP9VopxvNuMW1RA4ch0ysGq8Dik071133VXqOgylSdK1\na9/dqY5ESm1d3TpxfI1qA5XR2IFyNbkxpM2742rr6lYiLXV3p3svrv3eKp4O8fjJ0gfvSe1tzl3x\n7MwB6rK5MvwDH4BqA1411PjL6nutM9mtR9bt1B/e26P12zr0xra9entXlyY3hvq8/1vbE1q/Wwok\nujJdwNOZnmUKjlG8oUknNNaqoYYDcyGFQiF1dXUddB2/16PTJo7VieNrdEIkpOlHj9N5R48rj7Y8\ngjZXkLILWY9qUoX7cThttFp0Hv1Z/eIjn573TNR6b6P+t79Jm+qO0OSvf1WBYGDoAlC2oh1J/eG9\nPf3uzufzGCW9/iu39vnIup29gWPAZ/Tuv8274zpt4thSVy9vsq9D9qVs2ba4DpGq5jqkh9frVW1t\nrST9XNL2fJdv2PbAk/q6yBRJ6/7/3/5FH+3u1N5EWvOmNOozDaFS1wuAMic3KX+NfMlOfo3IYsei\nmbkB6sMV+Wv4Q69/rNb2eJ/JVXt+0cv+9TfakdS/vLpdY3yG7ERi/22sAwHFU7a+e2YTn5sCC4fD\nisVipa5GwRWyzeVSdqW3/WKppv1YLW10OHqOLYF0qvd4kfD4+x1bUHne3tWl5et2amywf9BRquu/\ncjvHzT7nOlClnnPRK2pg1XIM9fv9ikQikjRV0vp8l182wzB7VPU4ZMCFIrV+hcOHKBarzjvPDMYI\nRyp2IuKeuxAd+Otv9l2Iek5Y+nSVDwSkQKZnQKm7ynNyVXkK2eZyKbuS234xsR+rT99jy/7jRVDq\nd2xB5XHTPFTZQxnl8UjpdFkMZazGie8jtf6Ke035wDE0P8oqdSr1xRXcjwtgoPByPRlz03xY1TCX\nB4bGsQJwn2q80Md+bpqHKnvuNL/fr2QyURZzp7kpcMzGMRflqmzCskS3XRaTDaM0uAAGiifXkzE3\nTSDK5MHVjWMF4F5uvdBH8bjhx7Vces+7TbECx+Gez3HMRbkrm6POrJMiGutz/fxqyLPhfhlzAQwU\nz0hPxkrdVb6cT4CRHxwrAPdyU88ilIYbflwr9x6OhQwccw2/OOai3JVNWHZoyKdksrRzIpXzRHnl\nVvdcvoy5AEa5y7V9uqE9u+HX31yV+wlwuSv15zb7WJF9s4lgIFBWx4pS78eRckPP0mIo9Pvjlve/\nUPUox2PLaHDzkIE1xNvU0B6TPGGptrj7ZTQ9HN3wHhUycMwl/Brp9Vk5f4dWy3GukNzQhrKVTVhW\nSnZXp+zVK2TvbN1/J7fGZhkz58oI1ZS6egdVrnXP5cuYC2CUq1zbp5vasxt+/c0VQ3xKwy2f21hX\nSqlkSvbHrc7t1NOS4ZEdHKNkQ7PrjxVu2Y+5qpZhOD3vT3RXm2LyK6ykIg31eXt/3PL+F7oe5Xhs\nGYlc9qNb3vticMN+GUkPRze+R/nuzZ9r+JXr9Vmh92Ehy6+W41whubENSVL/e/Oin943LhCUasZK\ngaDs6HbZq1cc9HnRjqTe3tWlaEfpesSNtO6l1PNlHDzIl3E2LoBRrnJtn25sz5Favz7TECqLi5me\nE+B4qu+Qfob4FJZbPrfhkE++Tz7OBGUej+T1Zf6b2Cffro9df6xwy37M1f4fvwyNDXo0xjmWr9wQ\nLXXV8qpj9eN6aF+Lfnzo2Vpx6Bn68aFn66F9LepY/XheynfL+1+sepTTsWUkctmPbnnvi8Et+2XO\nKRE1O+cLe+Op3vOEwXo4VsN71Bt+DaAn/MqW6/VZofdhIcuvluNcIbm1DRGWDcGORfe/cdn8Adk7\nWzNdBQ/QmezWQ69/rH95dbuWr9upf3l1ux56/WN1JruLVOuMkdTdDXL9MuYCGOUo1/ZZru3Zbfqc\nACfSQ54AY3Tc9LltiLepqWuX4p6+x4S44VdzV1QN8bai1SVXbtqPucj1x69yZceiWrmvSdt94xS0\nuzXWTipod2u7b5xW7msa9fvjlvffLfUod7nsx9Hsczf8aJ+LYu2X4ejp4fjdM5t0/ReP0nfPbNJ1\np08YsJdQtbSLXMOvXK7PCr0PC1l+tRznCsnNbYiwbChtsUxXwIGkkpnlB3BNujyCurvBSHqKcQGM\nspNr+yzT9uw22SfA86Y0HvQEGHngps9tW0xX7X5dTelPFZdXHYZfcXnVnP5Us3e/4e425Kb9mINc\nf/wqV9Edn2i7Z6yC6vujaFDd2u4Zq+iOT0a3Abe8/26pR7nLZT+OYJ+75Uf7nBV4v4xEpNavyRMO\nOfgP71XSLkbSOWHY12eF3ocFLL9ajnMF5eI25O4xB25QH5Z8g3xB+vyZ5VlcNdl8jnV3i5HMFVAt\nc1ygguTaPsu0PbtVqe/MWTXc9LmtD6vGKy3o3KBdnpBiRkhhu0sN6S7JK3e3ITftxxxUyzQJu4N1\nSnp29AvLJCnl8Wp3sE6No9mAW95/t9Sj3OWyH0ewz8v2DoQF3i8F46a6FFiuN+AY9vVZofdhAcuv\nluNcQbm4DdGzbAhGOCKjsVlKJvouSCZkjG/ud5cGN6XLudbdTUbaU6zS57hA5ci1fWavv8sT0iZv\nWLs8obJoz6hebjoOZdelId2l47tjmaCsDNqQm/ZjLqplmoTw+Ij8fn/mphHZ7LR8fr/C40f3/oz0\n/c/3MLxy/Ry6TS77Mdd9Xs5Dwgq5XwrJTXUptJH2zh/q+qzQ57iFfI/cepwrp2HYbm5D3rvuuqtk\nGx+mJknXdnZ2Kp1OD7lyQRw/WfrgPam9zbmDlp154y6bK8PftwHYtvTGtr3yefonzLYtTT96nGoD\nRRzuk0Pd3cTv9ei0iWN14vganRAJafrR43Te0ePk95LvulEoFFJXV1epq1F+cmyfnUd/Vr/4yKfn\nPRO13tuo/+1v0qa6IzT5619VIBgowQtAOSh5+3TTcchNdclVmdZ9cmNIm3fH1dbVrX0pW7at3h+/\nKuWYXhvw6q97De1p7/h/7d17kGRXfdjxb/d0z+7sSCvtsLPaB4FIThCgECIJBILYkJIfhZP4UUQH\nCRPEuqBAgFARCI+ywyI/CkcVEA8bjI1rJduxkrPkD4KJjA15uCyBkFaCpGQQIAHSvmc1+9Ls7EzP\ndOePvjPqnZndnTMzp+e29vupUmmn58ztc8+5v3Pu/d0XfVMNaLavMJtcM8C2y57Pa35qw1n/flEx\nmtD/JxvTfGH3If7msWM8tHeMB/Y+zaOHx7li08Dy27xHt8PSSWnHhLJ7jk/y0N4x+hd4A+GpqRYv\nHB5g47oS91OmdlmOlY7PZ4PB/j42rquv6DFt9n3cjH1Upnku6/if0xL7p6+vj8HBQYA/AvavdLUq\nrdbCV0GVyFXA7pGRERqN1c2MtkZH2vfMXjx01gzn57514Iy3EK7W5c+Lrbu0FENDQ4yOrvz95OfL\nbbVpY8sE/c2p2dcqT1brqzq2qPxyxWeqMs1DZapLql6t+7N9PD/ZmOauh0fY+9TTTE1OUeuvse05\nF3DTlcPnvOoiJUYX0/8zc0Xn1UUrvR/aq9th2aS042LKjow1+NQ39rN2gWTZxFSL91y7pSfib6Xb\nZTlWOj61sG7t4+bsozLMc90Y/3NK7Z96vc7w8DDA1cBDK10fb6JNUBkahkV0Wur93N2w2LpLZTB7\n0HF8kqnpFrW+CtuKGHo2Poh9MfF5+vMQ+6G/fZZtDXT/eYjSEpRpHipTXVL1at2f7c8J7NazU8/V\n/916dm6vbodlk9KOiym7lOf+ltFKt0u3lKkuvaSb+7g5+2i157lSPTt9icoWQybLMvBh89KZLSYu\nevbhtBnNPg9xgbPFM89DdJyRpNWVerA0MtZg/+QJao2VOYhxrlAZT9pLZ+O4tTJsx5Vnsiyj1c4u\nS2Wy2KvFng1nRXI4H9+2c76ccDhf1lPSMzrnRKpVaDZX5Arqbs0Vjlvl5Un73mEftZ2P+7g52I4r\nrystFkJ4F/B+YDPwHeCWGOMD3fhuSeWw2KvFPCuysG7dWlGGHbfz5Tbc82U9Jc3XOSfW63UajckV\nuYI691zhuHV2ZZhDZ3jSvryMo9N18/bhMsXoSnu23IZdJtnfhhlCeAPwOeB9wEeAS4GP7dq160+u\nv/76k4tYxBbg7WP799Ksr/zb3g7tPcCeH+2hOT3F4PoLzlp2ZKzBnuOTtFos6u0frdER2L+n/TaH\ngcEVrUtK2dS6py47tXxKu6S2YarUuueSu81zbltw7jcFjYw1+JvHjrG2VqU1OVm85QRq9RpHx6d5\nySXrZrfLzjfKdpat9PWd842yOdczZ8wt1lLftrOYus++Ped7Izz04yM8sOcEjx6ZPOfbc3LE6Bd2\nH2o/nLQ5Rf/0BLUKHJ2Ex49M8PJtZ277lD5KrXeO8bwb63lo7wH2/WQfp8bHV3WMgzwx0a3l51x2\nL8/nOWOuV9dzMebOiZXGBM3p5oJz4kJ/e652mZkrjhwf59TJCZqtFts2DKzIm9lmxq21tSr9tQq1\naiORNHgAABV+SURBVIWj49PnHLdyKkP/L3UOTdnOe/k4JOf+ds7xuTU6Qv3wQSZOnVrRcWsp83/u\nY5bV3ofK/UbJpbwlskxtvtiy3XgzZ5niueffhhlC+CZwf4zx1uLnCvAk8OkY4+2LWMRVwO4Dn/go\nU40pKv/mLVQG1i27XmPHn+auL32TfY0+Gq0K9UqLrfVpbvrlV87r9NTsf2v8JK0v3knr0L7Zt3lU\nNm09Y91T6pJSNrXuqctOLZ/SLqltmCq17rnkbvOc21anc70p6NHD4+x8YD+DT+0rkl9NqFRhzVrG\nNm5l+8u2cPnGgdnyn71vD/sf/wn9E+OzZSfXDLD1sudz86ue29X1zBlzS7XYs2Ipdf/sfXvY9/gT\nrJk4eVqbb7ns+bxzgTbPFaMjYw0+de8e1hzaO29bmdi0jVtf/dx565yynqn1zjWe517PzrJTlSq1\nVnNVxjjIHxM5l59z2b08n+eMuV5dzxRz58QK0IIzzomp7TLT5iOHjzJKnSEaDG+8eGXG5xK9abFM\n/Z86h6b0Zy8fh+Tc3845Pne2YQ2YghUbt1Ln/9zHLGXah5ppnxxXfqW8JbJMbb7UuuRoxzLGc+63\nYa5MevEMQgh12hX/+sxnMcYW8DXg2qSF1ftpjeyn9cU7V6RuMx29ptLigmqTNZUW+xp93PWlb84v\nO3upfIUL1lRZWzw76a6HRxZc9uyg0L8G1l0A/WvOWvekuiSUTa178rITy6e0S2obpkqtey652zzn\ntpViaKBG7akD7Z2CahX6au3/T56idvjAvHvo3/zYV9g8/hQT1TpjtQEmqnW2jD/Fv33sK11fz5wx\nt1TDg3Uu3zhwzslvsXUfGWs8s5Pf0T/9k+Pse/wJRsYa85adK0ZHx6eYOnRwwW1l6tBBRsenlrye\nS6l3rvE893qeXra1amMc5I+JnMvPuuwens9zxlyvrmeKeXNi7exzYmq7zLT5xto0L6idYmNteuXG\n5+mFT7LPPCahm8rS/0uZQ5O2814+Dsm4b5lzfO5sw+rgyo5bqfN/7mOWMu1DweL3cVPMPA95zVme\nh9ypTG2+1LrkaMdejeflyJosAzYCfcDBOZ8fpP38sjT1flqH9rUv5VyGQ3sPzHZ0p5kOP7T3wOxn\nqcHVGh15ZlBYRN1T6pJSNrXuqctOLZ/SLqltmCq17rnkbvOc21aqjRNH2TJ+mInq6QP2RKXO1vER\nNk4cnf2sNTrCwKEnecep73DLyQe46eT/5ZaTD/D2U99h4NCTXY2hnDGXW0rdRw+O0Gg02jtrnSpV\nphoNRg+e3uY5Y3TDxDFqk+ML1qU2Oc6GiWOnfZyynqn1zjme51zPsoxxkD8mci4/57J7eT7PGXO9\nup6pUuZESGuXnONzmR4eXab+T51DU/qzl49DcsZQzvE557gFafN/7vm8TPtQOaUk+svU5mXan+vV\neF6u1XolwuwV551CCDcCN3Z+dsUVV1y0Y8cO6rUalXqdZmOCC5pT1IeGlvzle374JFOVKmur84Nm\nqllhcrzBULH8/ZMnoNp++OpcE80ppurrGBq6cPazxuH9PA1UFyi/UN1T6pJSNrXuqctOLZ/SLqlt\nmCq17rnkbvOc29Zc9Xr9rL9vHN7Pr594mLs2XMO+6iANqtRp8tzmGDc9/W3WN69ZsP+3MsVWTrQX\n0rdw/Odcz5wxl1tK3Z/35F76W00qCzzLoD7d5Hn11pLjOdWFh/ezbfoE+/svpp/m7OeTVNk2eYSf\nWtNacv+n1jvneJ5zPeeWrcDsTnk3xzjIHxM5l59z2b08n+eMuV5dz1Tz5sRKH/Xq9IJzIqS1S87x\neWgILvvh0+w5Os6a2jO3xUxMTXPp8ACX/4NLlrTcpShT/6fOoSn92cvHITljKOf4PLcNK9Uq/cW/\nlztuQdr8n/uYpUz7UDldVp9gYO0R6rX5t/Otq0xz2ZaNDF3YTuqVqc3Lcsyauy7LiedKpX0C57bb\nbrvjkUceOTbn13fHGO9eUqUKuZNlh4FpYO7suYn5V5tRrMzcFboK2N2YmmqftWnB8WqNylmej3Qu\n/QN1aq0mreb8zq61KvQP1Gefv1RrNKDZpNGYnL+gZota4ySjox1n9Kq19rDXWCADukDdU+qSUja1\n7qnLTi2f0i6pbZgqte655G7znNvWXOd6ZlmrWqO/1eBtTz/I4eoAo5UBhlrjbGyOQ7OxrP7PuZ45\nYy63lLqvHexn8/QJ9lcuYg3Ts+Um6GPr9DHWDvYvOZ5Ttao13nTiYf70oqvZX73wmedtNY/zphPf\n5nj1VUvu/9R659wWc67nvLLVKq1mc8GyueWOiZzLz7nsXp7Pc8Zcr65nqrlz4on6hVzYOLHgnAhp\n7ZJ7H+qGF6/nrocn2Hd8gkazRb1aYev6fm548fqujStQrv5PnUNT+rOXj0NyxlDO8XluG/bX60zO\ntOcyx62Z5S92/s99zFKmfaic6sCmgWrxzLL5b4msN8YYHR0DytXmZTlmzV2X5cTzzDPLduzY8V56\n7ZllMcYGsBu4buaz4gH/1wH3JS+wMUnlkq1UhoaXVa9N2zaztT7NROv0S8knWhW29k+zadszd4jO\nvIJ1Yur0DeNMr2CtDA1T2bQV5nb2GeqeUpeUsql1T112avmUdkltw1Spdc8ld5vn3LZSdfbpxuY4\nL5gebR8UrED/51zPnDGXW0rdK0PD3LR2P1umjjFBH2OVensnf+oYbx44MK/Nc8ZoZWiYweFh3nHs\n/tNvwz12P4ObhpfV/6n1zrkt5lzPsoxxkD8mci4/57J7eT7PGXO9up6p5s6JlzePnHFOhLR2yb0P\nta7ex83XbOY9125h+1WbeM+1W7j5ms0r9gKbxSpT/6fOoSn92cvHITljKOf4nHPcmln+Yuf/3PN5\nmfahcrvpymG2Fv309GRztn9uunLpMbEUvbo/16vxvFx9H/3oR7N+wa5du44Dv7Nr164nd+3aNQH8\nLvBS4K3XX3/9yUUsYgvw9qfv/TqtgUEqr38LlQUu0Uv14ks38fgPnuTodJWJVpUWsLV/mpt+6ZX0\nr+k/rWzyK1hfcAX85DE4frR4y0mrPSicoe4pdUkpm1r31GWnlk9ql8Q2TJVc90xyt3nObavTwMAA\n4+PjZ1/ZjP2fcz1zxlxuKXWvv+DFvOz7f8s/Gf0Bl5/az2tP/ZjXXDRJ/+tvWjjmcsZosex1x0Z4\nzqkjrGs2Vqz/k+udcVvMuZ6dZU9RbW+HqzDGQf6YyLn8nMvu5fk8Z8z17Hqm6miXytQkzenpFWuX\n3PtQAIP9fWxcV2ewv7tJsk5l6v/UOTSlP3v5OCRnDGWdW3LGZ8fyFzP/5z5mKdM+VE71viov33YB\nL7lkHS8cHuC1l17Eay69aMH+KVObl+WYNXddlhrPfX19DA4OAvwRsH9ZlVhApdWafyndSgshvBP4\nAO3bMb8N3BJjfHCRf34VsPvQD7/P1LqVf939ob0HOHL4KBs2XnzOjGjqK1hboyNwdBQuHlpU9jyl\nLillU+ueuuzU8intktqGqVLrnkvuNs+5bcG5b8PslLP/c65nzpjLLaXuqW2eM0Zz9n/u9cxZl9Tt\nfHK8Qf9AfVXHOMgfEzmXn3PZvTyf59zOe3U9U7VGR1jfnGrfkrTC7ZJ7H6osytT/ObfzXj4OyRlD\nOcfnnPE5s/zF9lHuY5Yy7UOVRZnavCzHrLnrkhpDM7dhAleT4TbMriTLlukqYPfISPGmGUmlk5Is\nk9RdxqdUbsaoVF7Gp1ReuZNl3b0nSJIkSZIkSSoxk2WSJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmS\nJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmS\nJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmSVDBZJkmS\nJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEkFk2WS\nJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmSVDBZ\nJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEkF\nk2WSJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmS\nVDBZJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmS\nJEkFk2WSJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmS\nJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEmFWq4FhxB+DDyv46MW8OEY4+25vlOSJEmSJElajmzJMtrJ\nsd8E/hioFJ+dyPh9kiRJkiRJ0rLkTJYBPB1jHMn8HZIkSZIkSdKKyJ0s+1AI4SPAE8BfAHfEGKcz\nf6ckSZIkSZK0JDmTZZ8CHgJGgVcBvwdsBt6f8TslSZIkSZKkJau0Wq1FFw4hfAz44FmKtIAXxRi/\nv8Dfbgf+ELggxthIqOOrgHuPHDnC1NRUwp9J6pb169dz/Pjx1a6GpAUYn1K5GaNSeRmfUnnVajU2\nbNgA8GrgvhVffmL5/wTsPEeZx8/w+f3F9/1D4AcLFQgh3Ajc2PnZ6173um3bt2+faQRJJTU8PLza\nVZB0BsanVG7GqFRexqdUbjt37vzMPffcs3fOx3fHGO9eznKTrixbjhDCrwF3AhtjjMcS/vQ5O3fu\n/Ovt27ffApzKUjlJy3LbbbfdsWPHjveudj0kzWd8SuVmjErlZXxKpbZ2586dn9m+ffvPA0+t9MKz\nPLMshPBK4BXA/wJO0L6V8hPAnyUmygCeuueee/Zu3759xS+rk7QyHnnkkWO0n1EoqWSMT6ncjFGp\nvIxPqdyKXNGKJ8og3wP+J4AbgB3AGuBHwMeBOzJ9nyRJkiRJkrRsWZJlMcaHgWtzLFuSJEmSJEnK\npbraFZAkSZIkSZLKoleSZct6i4Gk7IxRqbyMT6ncjFGpvIxPqdyyxWjX3oYpSZIkSZIklV2vXFkm\nSZIkSZIkZWeyTJIkSZIkSSqYLJMkSZIkSZIKJsskSZIkSZKkgskySZIkSZIkqVBb7QqcSwjhXcD7\ngc3Ad4BbYowPrG6tpPNLCOHDwK8CLwTGgfuAD8YYv99RZg3wCeANwBrgq8A7Y4yHul9j6fxVxOvv\nAp+MMf674jPjU1pFIYStwH8EXgesA34AbI8xPtRR5reAtwIXA/cCN8cYf7gK1ZXOKyGEKnAb8Gu0\njzn3AXfGGH9nTjljVMoshPDTwL8Hrga2AL8SY/zvc8qcNRZDCBuA3wf+FdAE/htwa4xxLKUupb6y\nLITwBuDjwA7gStrJsq+GEDauasWk889PA58BXgH8LFAH/jqEMNBR5pPAvwReD/wMsJX2wCSpS0II\nLwfeRnu+7GR8SqskhDCzMz8B/ALwIuB9wJGOMh8E3g28HbgGGKO9z9vf9QpL558P0Y69d9I+MfwB\n4AMhhHfPFDBGpa4ZBL4NvAtozf3lImPxL2jPtdfR3v/9GeDzqRUp+5Vl7wU+H2P8U4AQwjtor+yv\nA7evZsWk80mM8Rc7fw4hvAU4RDvj/3chhPW04/KGGOP/KcpsB74bQrgmxvitLldZOu+EEC4A/pz2\nmbb/0PG58Smtrg8BT8QY39rx2U/mlLkV+O0Y45cBQghvBg4CvwLErtRSOn9dC3wpxvhXxc9PhBDe\nSPtAfIYxKnVBEYd/BRBCqCxQ5KyxGEJ4Ee0TU1fHGB8uytwCfCWE8P4Y44HF1qW0V5aFEOq0D8S/\nPvNZjLEFfI32gCZp9VxMO9M/Wvx8Ne3ke2e8Pgo8gfEqdcsfAF+OMf7POZ+/DONTWk3/GngwhBBD\nCAdDCA+FEGYTZyGES2nf+tUZo8eB+zFGpW64D7guhPCPAUIILwVeDfyP4mdjVCqBRcbiK4EjM4my\nwtdoH7u+IuX7SpssAzYCfbSzhJ0O0m4gSaugyPB/Evi7GOPfFx9vBiaLwaqT8Sp1QQjhBuCfAR9e\n4NeXYHxKq+ky4GbgUeDngT8EPh1CeFPx+820d+Ld55VWx+8B/xX4XghhEthN+7mf/6X4vTEqlcNi\nYnEz7TugZsUYp2lf5JEUr2W/DXMhFRa4d1VS13wWeDHwzxdR1niVMgshPJd2AvvnYoyNhD81PqXu\nqALfijHO3B79nRDCFbQTaH9+lr8zRqXueAPwRuAG4O9pn3z6VAhhX4zxz87yd8aoVA6LicXkeC3z\nlWWHgWnaZ8Q7bWJ+JlFSF4QQfh/4ReC1McZ9Hb86APQXz0bqZLxK+V0NDAO7QwiNEEIDeA1wa3GG\n/CCwxviUVs1+4LtzPvsu8Lzi3wdo78S7zyutjtuBj8UYd8UYH4kx/mfgDp65WtsYlcphMbF4oPh5\nVgihD9hAYryWNllWnB3fTfsNBsDs7V/X0b6vXFIXFYmyXwb+RYzxiTm/3g1McXq8voD2gcA3ulZJ\n6fz0NeAltM+Ev7T470HaV6zM/LuB8SmtlnuBy+d8djnFQ/5jjD+ivXPfGaPraT9bxX1eKb91zL/i\npElxrGyMSuWwyFj8BnBxCOHKjj+9jnaS7f6U7yv7bZifAO4KIewGvkX77ZjrgDtXs1LS+SaE8Fng\nRuCXgLEQwkw2/1iM8VSM8XgI4U+AT4QQjgAngE8D9/qmPSmvGOMY7dtGZoUQxoCnYozfLX42PqXV\ncwdwbwjhw7TfmvcK2m+tfVtHmU8CvxlC+CHwY+C3gT3Al7pbVem89GXgN0IITwKPAFfRPu78QkcZ\nY1TqghDCIPCPaCe3AC4rXroxGmN8knPEYozxeyGErwJ/HEK4GegHPgPcnfImTCjxlWUAMcYIvA/4\nLeBh4J8CvxBjHFnViknnn3cA64H/Dezr+C90lHkv8JfAFzvKvb6blZQ0a+4ZcuNTWiUxxgeBX6V9\n0un/Ab8B3Nrx8HBijLfT3pn/PO0z3wPA62KMk92vsXTeeTft+fEPaJ98uh34HPCRmQLGqNQ1L6Od\n+9lNe3/248BDwG2w6Fh8I/A92ndf/CXwt8DbUytSabV8JqEkSZIkSZIEJb+yTJIkSZIkSeomk2WS\nJEmSJElSwWSZJEmSJEmSVDBZJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmSVDBZ\nJkmSJEmSJBVMlkmSJEmSJEkFk2WSJEmSJElSwWSZJEmSJEmSVPj/FufkKFHWFNsAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f60ac79a630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline\n",
    "\n",
    "n = 100\n",
    "plt.figure(figsize=(15, 5))\n",
    "plt.plot(y_test[:n], '.', alpha=.7, markersize=10, label='ground truth')\n",
    "plt.plot(y_pred[:n], '.', alpha=.7, markersize=10, label='prediction')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "widgets": {
   "state": {
    "1a4cc8b20a154e6cb11ba51c217c86d7": {
     "views": [
      {
       "cell_index": 17
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}