{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN - Many-to-one\n",
    "> In this post, We will briefly cover the many-to-one type, which is one the common types of Recurrent Neural Network and its implementation in tensorflow. \n",
    "\n",
    "- toc: true \n",
    "- badges: true\n",
    "- comments: true\n",
    "- author: Chanseok Kang\n",
    "- categories: [Python, Deep_Learning, Tensorflow-Keras]\n",
    "- image: images/many-to-one_detail.png"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow: 2.3.1\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "print('Tensorflow: {}'.format(tf.__version__))\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (16, 10)\n",
    "plt.rc('font', size=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Various usage of RNN\n",
    "As we already discussed, RNN is used for sequence data handling. And there are several types of RNN architecture.\n",
    "\n",
    "![various_rnn](image/various_rnn.png) {% fn 1 %}\n",
    "\n",
    "In previous [post](https://goodboychan.github.io/chans_jupyter/python/deep_learning/tensorflow-keras/2020/10/26/02-RNN-Basic.html), we take a look **one-to-one** type, which is the basic RNN structure. And next one is **one-to-many** type. For example, if the model gets the fixed format like image as an input, it generates the sequence data. You can see the implementation on image caption application. Another type is **many-to-many** type. It gets sequence data as an inputs, and also generates the sequence data as an output. Common application of many-to-many type is machine translation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Many-to-one\n",
    "\n",
    "**Many-to-one** type, which is our topic in this post, gets an sequence data as an input and generates some informatic data like labels. So we can use it for classification. Suppose that someone defines the sentiment of each sentence, and train the model with many-to-one type. And when the model gets the unseen sentence, then it will predict the intention of sentence, good or bad.\n",
    "\n",
    "![many-to-one example](image/many-to-one.png)\n",
    "\n",
    "The detailed explanation is like this.\n",
    "\n",
    "Suppose we have a sentence, \"This movis is good\". And we want to classify the sentiment of this sentence. In order to do this, we need to apply tokenization in word level. If this sentensce intends the good sentiment, then word token may contains good words, like \"good\". So we can classify this sentense to good sentiment.\n",
    "\n",
    "So if we want to apply it in RNN model, we need to consider the sentence as a word sequence(many), then clssify its label(one). That is process of many-to-one type model.\n",
    "\n",
    "![many-to-one detail](image/many-to-one_detail.png)\n",
    "\n",
    "But as you notice, computational model cannot accept the word itself as an input. Instead, it needs to convert with numerical vector. **Embedding** Layer can do this.\n",
    "\n",
    "So how can we implement this with tensorflow?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - Word sentiment classification\n",
    "\n",
    "At first, we prepare the dummy data for simple classification. For the simplicity, we defined 1 as a good token, and 0 otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = ['good', 'bad', 'worse', 'so good']\n",
    "y = [1, 0, 0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on this, we can generate the token dictionary, which is the mapping table for each characters. But before we prepare this, we need to consider one exceptional case, the variation of each sentence length. To train the network, the format (or shape) of input data must be fixed. So we need to add the concept of **padding**. Tensorflow has useful API for padding, `pad_sequences`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "char_set = ['<pad>'] + sorted(list(set(''.join(words))))\n",
    "idx2char = {idx:char for idx, char in enumerate(char_set)}\n",
    "char2idx = {char:idx for idx, char in enumerate(char_set)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<pad>', ' ', 'a', 'b', 'd', 'e', 'g', 'o', 'r', 's', 'w']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: '<pad>',\n",
       " 1: ' ',\n",
       " 2: 'a',\n",
       " 3: 'b',\n",
       " 4: 'd',\n",
       " 5: 'e',\n",
       " 6: 'g',\n",
       " 7: 'o',\n",
       " 8: 'r',\n",
       " 9: 's',\n",
       " 10: 'w'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx2char"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<pad>': 0,\n",
       " ' ': 1,\n",
       " 'a': 2,\n",
       " 'b': 3,\n",
       " 'd': 4,\n",
       " 'e': 5,\n",
       " 'g': 6,\n",
       " 'o': 7,\n",
       " 'r': 8,\n",
       " 's': 9,\n",
       " 'w': 10}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char2idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So as we mentioned before, we need to vectorize each tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = list(map(lambda word: [char2idx.get(char) for char in word], words))\n",
    "X_len = list(map(lambda word: len(word), X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[6, 7, 7, 4], [3, 2, 4], [10, 7, 8, 9, 5], [9, 7, 1, 6, 7, 7, 4]]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 3, 5, 7]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_len"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "# Padding the sequence of indices\n",
    "max_sequence=10\n",
    "\n",
    "X = pad_sequences(X, maxlen=max_sequence, padding='post', truncating='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6,  7,  7,  4,  0,  0,  0,  0,  0,  0],\n",
       "       [ 3,  2,  4,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [10,  7,  8,  9,  5,  0,  0,  0,  0,  0],\n",
       "       [ 9,  7,  1,  6,  7,  7,  4,  0,  0,  0]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 0, 0, 1]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we used `pad_sequences` API with several arguments. First, we defined maximum sequence length to 10. So numerical vector has an maximum length of 10. That is, we just consider 10 characters for sequence. And there are some words that has less than 10 characters. In that case, we filled some 0s for padding. Through argument, we can define the direction of padding, `pre` or `post`.\n",
    "\n",
    "And of course, it is efficient to use the dataset with generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<BatchDataset shapes: ((None, 10), (None,)), types: (tf.int32, tf.int32)>\n"
     ]
    }
   ],
   "source": [
    "# Generate data pipeline\n",
    "train_ds = tf.data.Dataset.from_tensor_slices((X, y)).shuffle(buffer_size=4).batch(batch_size=2)\n",
    "print(train_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we can build many-to-one model with simpleRNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim = len(char2idx)\n",
    "output_dim = len(char2idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 10, 11)            121       \n",
      "_________________________________________________________________\n",
      "simple_rnn_1 (SimpleRNN)     (None, 10)                220       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 22        \n",
      "=================================================================\n",
      "Total params: 363\n",
      "Trainable params: 242\n",
      "Non-trainable params: 121\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Embedding, SimpleRNN, Dense\n",
    "\n",
    "model = Sequential([\n",
    "    Embedding(input_dim=input_dim, output_dim=output_dim,\n",
    "              mask_zero=True, input_length=max_sequence,\n",
    "              trainable=False, embeddings_initializer=tf.keras.initializers.random_normal()),\n",
    "    SimpleRNN(units=10),\n",
    "    Dense(2)\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_fn(model, X, y):\n",
    "    return tf.reduce_mean(tf.keras.losses.sparse_categorical_crossentropy(y_true=y, \n",
    "                                                                          y_pred=model(X), \n",
    "                                                                          from_logits=True))\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:   5, tr_loss: 0.248861\n",
      "epoch:  10, tr_loss: 0.018041\n",
      "epoch:  15, tr_loss: 0.003201\n",
      "epoch:  20, tr_loss: 0.001664\n",
      "epoch:  25, tr_loss: 0.001231\n",
      "epoch:  30, tr_loss: 0.001025\n"
     ]
    }
   ],
   "source": [
    "tr_loss_hist = []\n",
    "\n",
    "for e in range(30):\n",
    "    avg_tr_loss = 0\n",
    "    tr_step = 0\n",
    "    \n",
    "    for x_mb, y_mb in train_ds:\n",
    "        with tf.GradientTape() as tape:\n",
    "            tr_loss = loss_fn(model, x_mb, y_mb)\n",
    "            \n",
    "        grads = tape.gradient(tr_loss, sources=model.variables)\n",
    "        optimizer.apply_gradients(grads_and_vars=zip(grads, model.variables))\n",
    "        avg_tr_loss += tr_loss\n",
    "        tr_step += 1\n",
    "    \n",
    "    avg_tr_loss /= tr_step\n",
    "    tr_loss_hist.append(avg_tr_loss)\n",
    "    \n",
    "    if (e + 1) % 5 == 0:\n",
    "        print('epoch: {:3}, tr_loss: {:3f}'.format(e + 1, avg_tr_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we can check the performance of this simple rnn network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X)\n",
    "y_pred = np.argmax(y_pred, axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc: 100.00%\n"
     ]
    }
   ],
   "source": [
    "print('acc: {:.2%}'.format(np.mean(y_pred == y)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6oAAAJECAYAAAAfRdlYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeZifZX0v/vc9M9nIMpOQhSQDJITdZOIShIKKCIgoOZ5yqtiqLVKlLu3pqbbXWX+XwO9sbU/leH51owWt1ZaqtccDigpUsIBbUJIABghrNpIA2ffl+f2RoSfGSfJNMsnznfm+Xtc11zD393lm3vnPt/f9fJ5SVVUAAACgWbTVHQAAAAD2pqgCAADQVBRVAAAAmoqiCgAAQFNRVAEAAGgqiioAAABNpaPuAPszfvz4atq0aXXHAAAA4Ch48MEHX6iqakJfnzVtUZ02bVrmzZtXdwwAAACOglLKs/v7zNFfAAAAmoqiCgAAQFNRVAEAAGgqiioAAABNRVEFAACgqSiqAAAANJWGimop5exSyt2llM2llOWllBtKKe0N3DenlPLdUsqLpZSXSil3lVLOPfLYAAAADFYHLaqllLFJ7kpSJXl7khuSfCzJ9Qe578Te+zqS/GaS9/b+93dLKScfWWwAAAAGq44GrvlgkhFJrqyqan2SO0spY5JcV0r5k961vrwtyeje+9YmSSnlgSQvJHlrks8ccXoAAAAGnUaO/l6e5Dv7FNJbs6e8XniA+4Yk2Zlk415rG3vXyiHmBAAAoEU0UlTPTLJo74Wqqp5Lsrn3s/35+95r/qyUMrGUMjHJjUnWJPnq4cUFAABgsGukqI5NsraP9TW9n/WpqqrlSS5K8q+SrOz9ujLJZVVVrT70qAAAALSCRl9PU/WxVvazvufDUiYn+VqSB7Pn+PDlvf/9zVLKSfu559pSyrxSyrzVq3VZAACAVtRIUV2TpKuP9c70vdP6sj/KnmFNv1ZV1berqvp29uyu7kryh33dUFXVTVVVzamqas6ECRMaiAYAAMBg00hRXZR9nkXtffXMyOzz7Oo+zkzySFVVO15eqKpqe5JHksw49KgAAAC0gkaK6h1JLiuljN5r7aokW5Lce4D7nk0ys5Qy9OWFUsqwJDOTPHPoUQEAAGgFjRTVzybZluTrpZRLSinXJrkuySf2fmVNKWVxKeXmve77yyRTkvxDKeVtpZQrkvzvJJOT3NRf/wAAAAAGl4MW1aqq1iS5OEl7ktuSXJ89r5n5+D6XdvRe8/J9DyZ5S5LRSf46yReTHJfk0qqq5vdHeAAAAAafjkYuqqrq0SRvOsg10/pYuzvJ3YeVDAAAgJbU6OtpAAAA4JhQVAEAAGgqiioAAABNRVE9DMvXbsmbb7w331q4ou4oAAAAg46iehjGjxqWZ17cnIeWrK07CgAAwKCjqB6GoR1tOWvymCxYqqgCAAD0N0X1MPVM7czDy9Zn9+6q7igAAACDiqJ6mHq6O7Nx28489cKmuqMAAAAMKorqYerp7kqSLFzm+C8AAEB/UlQP04wJIzNiSHsWLF1XdxQAAIBBRVE9TB3tbZk5dYyiCgAA0M8U1SMwa2pXHlm+Ljt37a47CgAAwKChqB6Bnu7ObN2xO4tXb6w7CgAAwKChqB6Bnu7OJMmCJY7/AgAA9BdF9QhMO35kRg/ryAKTfwEAAPqNonoE2tpKZk7tzEIDlQAAAPqNonqEek7szM9XbMj2nQYqAQAA9AdF9Qj1TO3K9l2789jzG+qOAgAAMCgoqkfonwcqeU4VAACgXyiqR6h77IiMPW6Iyb8AAAD9RFE9QqWUzOruyoJliioAAEB/UFT7Qc/Uzjy+ckO27thVdxQAAIABT1HtB7O6O7Nrd5VHV6yvOwoAAMCAp6j2g9ndXUmSBUsMVAIAADhSimo/mDRmWCaMHuY5VQAAgH6gqPaDUkp6pnZm4VJFFQAA4Egpqv2kp7sri1dvzMZtO+uOAgAAMKApqv2kp7szVZU84vgvAADAEVFU+8nMqZ1JkoWKKgAAwBFRVPvJhNHDMqVzeOZ7ThUAAOCIKKr9qKe7KwuXekUNAADAkVBU+9Gs7s488+LmrNu8o+4oAAAAA5ai2o96uj2nCgAAcKQU1X7UM7UrSbJgmeO/AAAAh0tR7Uedxw3Jyccfl4UGKgEAABw2RbWfzZramQWKKgAAwGFTVPvZ7O6uLFu7JS9u3FZ3FAAAgAFJUe1ns3oHKi0wUAkAAOCwKKr9bObUzpQSz6kCAAAcJkW1n40a1pEZE0ZlwVKTfwEAAA6HonoU9BioBAAAcNgU1aNgVndnVm3YlpXrt9YdBQAAYMBRVI+Cnt6BSvOXOP4LAABwqBTVo+DsyZ1pbytZaPIvAADAIVNUj4IRQ9tz2sRRnlMFAAA4DIrqUdLT3ZkFS9emqqq6owAAAAwoiupR0tPdlTWbd2Tpmi11RwEAABhQFNWj5OWBSp5TBQAAODSK6lFyxgmjM6S9eE4VAADgECmqR8mwjvacNXlMFiz1ihoAAIBD0VBRLaWcXUq5u5SyuZSyvJRyQyml/SD3XFdKqfbz9e/7J35zmzW1MwuXrcvu3QYqAQAANOqgRbWUMjbJXUmqJG9PckOSjyW5/iC3/mWSX9nn6497P7vjMPMOKD3dndmwdWeefWlz3VEAAAAGjI4GrvlgkhFJrqyqan2SO0spY5JcV0r5k961X1JV1dIkS/deK6X8P0kWVVX10BHmHhBmTe1KkixYujbTx4+sOQ0AAMDA0MjR38uTfGefQnpr9pTXCxv9Q6WUcUkuTfK3h5RwADt90qgM62gzUAkAAOAQNFJUz0yyaO+FqqqeS7K597NG/VqSIdlTcltCR3tbXjFlTBYqqgAAAA1rpKiOTdLX6No1vZ816l1JflpV1eOHcM+A19PdlYeXr8suA5UAAAAa0ujrafpqWWU/6798YSmTs+eY8AGP/ZZSri2lzCulzFu9enWD0ZpbT3dnNm/flSdXb6w7CgAAwIDQSFFdk6Srj/XO9L3T2pd3Zk+x/bsDXVRV1U1VVc2pqmrOhAkTGvzVza2nuzNJPKcKAADQoEaK6qLs8yxqKeXEJCOzz7OrB/CuJPdVVbXk0OINfNPHj8rIoe1ZsLTRTg8AANDaGimqdyS5rJQyeq+1q5JsSXLvwW4upUxLcl5aaNrv3trbSmZO7bSjCgAA0KBGiupnk2xL8vVSyiWllGuTXJfkE3u/sqaUsriUcnMf978ryc4kX+uHvANST3dnHl2xPjt27a47CgAAQNM7aFGtqmpNkouTtCe5Lcn1SW5M8vF9Lu3ovWZf70pyd1VVg2M60mGY1d2V7Tt35/GVG+qOAgAA0PQ6GrmoqqpHk7zpINdM28/6Kw891uAye6+BSq+Y0llzGgAAgObW6OtpOAInjTsuY4Z3eE4VAACgAYrqMVBKSU93VxYuM/kXAADgYBTVY2RWd2cWrdiQrTt21R0FAACgqSmqx8js7s7s3F1l0fMGKgEAAByIonqMzOruSpIsXOr4LwAAwIEoqsfIlM7hOX7k0Mw3UAkAAOCAFNVjZM9Apc4sVFQBAAAOSFE9hmZ1d+WJVRuyefvOuqMAAAA0LUX1GOqZ2pndVfLI8vV1RwEAAGhaiuox1NPdmSRZ4PgvAADAfimqx9DEMcNzwpjhJv8CAAAcgKJ6jM3q7syCZXZUAQAA9kdRPcZmd3fmqdWbsn7rjrqjAAAANCVF9Rib1d2VJHnYrioAAECfFNVjbNbUPQOVvE8VAACgb4rqMTZu5NB0jx1h8i8AAMB+KKo1mN3dlQXLTP4FAADoi6Jag1ndnVny0pas2bS97igAAABNR1GtQU/vc6peUwMAAPDLFNUazOx+eaCS478AAAD7UlRrMGb4kJwyfqSBSgAAAH1QVGsyq7tTUQUAAOiDolqTnu6uPL9+a1at31p3FAAAgKaiqNak5+XnVA1UAgAA+AWKak3OnjwmbSWO/wIAAOxDUa3JyGEdOXXiqCww+RcAAOAXKKo16unuysJl61JVVd1RAAAAmoaiWqOe7s68sHF7VqwzUAkAAOBlimqNZk3dM1DJ8V8AAID/S1Gt0VmTx6SjrRioBAAAsBdFtUbDh7TnjBNGe0UNAADAXhTVmvV0d2bBUgOVAAAAXqao1qynuyvrtuzIcy9trjsKAABAU1BUa/Z/Byo5/gsAAJAoqrU744TRGdrRZvIvAABAL0W1ZkPa23L25DF2VAEAAHopqk2gp7szDy9bl927DVQCAABQVJvArKmd2bR9V556YWPdUQAAAGqnqDaBnu6uJAYqAQAAJIpqUzh14qiMGNKuqAIAAERRbQrtbSUzp47JwmWKKgAAgKLaJGZN7cojy9dl567ddUcBAAColaLaJGaf2JmtO3bniVUGKgEAAK1NUW0Ss6Z2JkkWek4VAABocYpqk5h2/MiMHtaR+UvX1h0FAACgVopqk2hrK5nV3WmgEgAA0PIU1SYyq7szP1+xPtt27qo7CgAAQG0U1SbSM7UrO3ZVeez5DXVHAQAAqI2i2kR6uvcMVFpgoBIAANDCFNUm0j12RMYeN8TkXwAAoKU1VFRLKWeXUu4upWwupSwvpdxQSmlv8N4rSyk/KaVsKaW8WEr5dill5JHFHpxKKZnV3WXyLwAA0NIOWlRLKWOT3JWkSvL2JDck+ViS6xu49/1J/ibJHUkuT/L+JE8k6Tj8yINbz9TOPLFqY7ZsN1AJAABoTY0Uxg8mGZHkyqqq1ie5s5QyJsl1pZQ/6V37JaWU8UluTPJ7VVX9xV4f/cORhh7Mero7s2t3lUdXrM9rTh5bdxwAAIBjrpGjv5cn+c4+hfTW7CmvFx7gvnf2fv+rw8zWknq6u5IkCx3/BQAAWlQjRfXMJIv2Xqiq6rkkm3s/259zkzyW5LdLKUtLKTtKKT8qpZx/2GlbwKQxwzJh9DCTfwEAgJbVSFEdm6Sv7b01vZ/tzwlJzkjyn5L82yRzk2xK8u1SyqRDzNkySimZ3d2ZBcsUVQAAoDU1+nqaqo+1sp/1vX/3qCS/XVXVl6uq+naSf5lkV5Lf7euGUsq1pZR5pZR5q1evbjDa4DNraleeXL0xG7ftrDsKAADAMddIUV2TpKuP9c70vdP6spd6v9/z8kLvc64PJjm7rxuqqrqpqqo5VVXNmTBhQgPRBqee7s5UVfKwXVUAAKAFNVJUF2WfZ1FLKScmGZl9nl3dx8+zZ8e17LNekuw+hIwtZ1Z3Z5JkoedUAQCAFtRIUb0jyWWllNF7rV2VZEuSew9w3+3ZU0ovenmhlNKZ5DVJ5h961NYxftSwTO0a4TlVAACgJTVSVD+bZFuSr5dSLimlXJvkuiSf2PuVNaWUxaWUm1/+uaqqeUm+keTmUspvlVLeluT/JNmR5FP9+G8YlGZN7cwCr6gBAABa0EGLalVVa5JcnKQ9yW1Jrk9yY5KP73NpR+81e3tPkv+d5BNJvpY9JfVNvb+TA5jV3ZlnX9ycdZt31B0FAADgmOpo5KKqqh5N8qaDXDOtj7WNST7U+8UhmN29Z37VwmXr8rrTxtecBgAA4Nhp9PU0HGOzpu4ZqDTf8V8AAKDFKKpNqvO4ITn5+ONM/gUAAFqOotrEerq7stDkXwAAoMUoqk2sZ2pnlq3dkhc2bqs7CgAAwDGjqDaxWd17nlN1/BcAAGglimoTmzm1M6UkCxRVAACghSiqTWzUsI7MmDAqC5eZ/AsAALQORbXJ9UztzPyl61JVVd1RAAAAjglFtcn1dHdm9YZtWbneQCUAAKA1KKpNblZ3V5JkwVLHfwEAgNagqDa5syePSXtbMVAJAABoGYpqkxsxtD2nTRyVBcsUVQAAoDUoqgPA7O6uLFy61kAlAACgJSiqA8Cs7s6s2bwjS9dsqTsKAADAUaeoDgA93Z1JkoeWGKgEAAAMforqAHDW5DEZe9yQ3PnoyrqjAAAAHHWK6gAwpL0tb5k5OXf9fGW2bN9VdxwAAICjSlEdIOb2TM7m7bvyj4tW1R0FAADgqFJUB4hzTzk+40cNy23zl9cdBQAA4KhSVAeI9raSK3om53uPrcqGrTvqjgMAAHDUKKoDyBU9k7Nt5+7c9XNDlQAAgMFLUR1AXn3S2EzpHJ7b5q+oOwoAAMBRo6gOIG1tJVfMnpJ/emJ11m7eXnccAACAo0JRHWCu6JmcHbuqfOeR5+uOAgAAcFQoqgPMrKmdOfn44xz/BQAABi1FdYAppWRuz5Q88OQLeWHjtrrjAAAA9DtFdQC6Yvbk7K6SOxbaVQUAAAYfRXUAOmPS6Jw2cZTjvwAAwKCkqA5ApZTMnT0lP3n2paxYt6XuOAAAAP1KUR2gruiZnKpKvrnArioAADC4KKoD1CkTRuUVU8bkdkUVAAAYZBTVAeyKnil5aMnaLHlpc91RAAAA+o2iOoBd0TM5SXLbguU1JwEAAOg/iuoAduK44/Kqk7pyu+m/AADAIKKoDnBX9EzJoyvWZ/GqjXVHAQAA6BeK6gD3tlmTU0pyu+O/AADAIKGoDnAndA7Pa6eNy23zl6eqqrrjAAAAHDFFdRC4YvaUPLl6UxY9v6HuKAAAAEdMUR0ELp95QtrbSm6b7/gvAAAw8Cmqg8D4UcNy/ozjc/uCFY7/AgAAA56iOkjM7ZmS517anAVL19UdBQAA4IgoqoPEZa84IUPaHf8FAAAGPkV1kOg8bkguPH1CvrlwRXbvdvwXAAAYuBTVQeSKnilZsW5rHnxuTd1RAAAADpuiOohccvakDOtoy+2O/wIAAAOYojqIjBrWkYvPmphvLlyRnbt21x0HAADgsCiqg8wVPVPywsbt+dHTL9UdBQAA4LAoqoPMRWdMzMih7bl9geO/AADAwKSoDjIjhrbnkrMn5Y6Hn8/2nY7/AgAAA4+iOgjN7ZmStZt35P7FL9QdBQAA4JApqoPQ608fnzHDO3Kb478AAMAA1FBRLaWcXUq5u5SyuZSyvJRyQyml/SD3TCulVH183do/0dmfYR3tuewVJ+S7j6zM1h276o4DAABwSA5aVEspY5PclaRK8vYkNyT5WJLrG/wbf5jkV/b6+k+HlZRDMnf2lGzctjP3PLa67igAAACHpKOBaz6YZESSK6uqWp/kzlLKmCTXlVL+pHftQB6rquqHRxqUQ3P+jOMzbuTQ3L5ged4y84S64wAAADSskaO/lyf5zj6F9NbsKa8XHpVUHLGO9rZcPvOE3P3zVdm8fWfdcQAAABrWSFE9M8mivReqqnouyebezw7m86WUXaWUFaWUT5RSRhxGTg7D3NlTsmXHrtz981V1RwEAAGhYI0V1bJK1fayv6f1sf7Yl+VSS305ycZLPJflQ9uzGcgycM21cJo0Zltvmm/4LAAAMHI08o5rsGaS0r7Kf9T03VNWKJL+719I9pZSVST5dSnllVVUP/dIvLOXaJNcmyUknndRgNPanva3krbMm58s/fC7rt+7ImOFD6o4EAABwUI3sqK5J0tXHemf63mk9kK/1fn91Xx9WVXVTVVVzqqqaM2HChEP81fRl7uwp2b5rd+58ZGXdUQAAABrSSFFdlH2eRS2lnJhkZPZ5drUB1T7fOcpedWJXpnaNyG0LHP8FAAAGhkaK6h1JLiuljN5r7aokW5Lce4h/79d6vz94iPdxmEopuWL25Nz3xAtZs2l73XEAAAAOqpGi+tnsGYz09VLKJb3PkV6X5BN7v7KmlLK4lHLzXj9fV0r5s1LKlb333ZDkxiRfr6pqQf/+MziQuT1TsnN3lW8/8nzdUQAAAA7qoEW1qqo12TO1tz3JbUmuz57C+fF9Lu3oveZli7LnPaufT/KtJL+R5E97v3MMvWLKmEwfP9L0XwAAYEBoaOpvVVWPJnnTQa6Zts/Pt8araJpCKSVzeybnz7+3OKs2bM3E0cPrjgQAALBfjRz9ZRCYO3tKdlfJHQsd/wUAAJqbotoiTps0OmdMGu34LwAA0PQU1RYyd/bkzHt2TZav3VJ3FAAAgP1SVFvIFT1TkiTfXLCi5iQAAAD7p6i2kGnjR2bW1M7ctsDxXwAAoHkpqi1m7uzJWbB0XZ59cVPdUQAAAPqkqLaYt/Ue/73d8V8AAKBJKaotZmrXiLzm5LGm/wIAAE1LUW1Bc3smZ9HzG/LEyg11RwEAAPglimoLemvP5LSV5DbHfwEAgCakqLagiaOH59zpx+f2+ctTVVXdcQAAAH6Botqi5s6ekqde2JRHV6yvOwoAAMAvUFRb1FtmnpD2tpLb5jv+CwAANBdFtUWNGzk0rzt1fG5f4PgvAADQXBTVFjZ39pQsXbMlDy1ZW3cUAACAf6aotrA3v2JShra3Of4LAAA0FUW1hY0ZPiQXnjEh31y4PLt3O/4LAAA0B0W1xc2dPSUr12/LT555qe4oAAAASRTVlnfxmRMzfEhbbluwvO4oAAAASRTVljdyWEcuPmtS7lj4fHbu2l13HAAAAEWVZG7PlLy4aXt+8NSLdUcBAABQVEneeMaEjBrWkdvmO/4LAADUT1Elw4e0581nT8q3H34+23c6/gsAANRLUSXJnum/67fuzD89sbruKAAAQItTVEmSXHDq+HSOGOL4LwAAUDtFlSTJ0I62XD7zhNz56Mps3bGr7jgAAEALU1T5Z3NnT8mm7bvyvUWr6o4CAAC0MEWVf3bu9HEZP2pobl+wou4oAABAC1NU+Wcd7W1566zJuXvRymzctrPuOAAAQItSVPkFV/RMydYdu3P3z1fWHQUAAGhRiiq/YM7JY3PCmOG5bb7jvwAAQD0UVX5BW1vJFT2Tc+/jq7Ju84664wAAAC1IUeWXzJ09JTt2VfnmQruqAADAsaeo8kt6ujtz9uQx+cIDT6eqqrrjAAAALUZR5ZeUUnLN66bn8ZUbc//iF+uOAwAAtBhFlT7NnT0540cNzS33P113FAAAoMUoqvRpWEd73n3uyfnHRavy1OqNdccBAABaiKLKfr37vJMytL0tX3jgmbqjAAAALURRZb8mjh6eubOn5KvzlnpVDQAAcMwoqhzQ+y6Yli07duXv5j1XdxQAAKBFKKoc0MypnTl3+rj81QPPZueu3XXHAQAAWoCiykFd87rpWbZ2S7776Mq6owAAAC1AUeWgLjlrUk4cNyK33OdVNQAAwNGnqHJQ7W0lV58/PfOeXZP5S9bWHQcAABjkFFUa8s453Rk1rCOfv9+uKgAAcHQpqjRk9PAhecec7ty+YEVWrt9adxwAAGAQU1Rp2NXnT8uuqsqXfvhs3VEAAIBBTFGlYScfPzKXnDUpX/7Rc9m6Y1fdcQAAgEFKUeWQXHPB9Ly0aXu+8dCyuqMAAACDlKLKITnvlHE5a/KY3HLfM6mqqu44AADAINRQUS2lnF1KubuUsrmUsryUckMppb3RP1JKaSulPFhKqUopVxx+XOpWSsk1F0zLYys35IEnX6w7DgAAMAgdtKiWUsYmuStJleTtSW5I8rEk1x/C33l/kqmHE5DmM3f2lIwfNTS33OdVNQAAQP9rZEf1g0lGJLmyqqo7q6r6bPaU1I+WUsYc7ObeovtfkvzHI0pK0xg+pD3vPvfk3L1oVZ5+YVPdcQAAgEGmkaJ6eZLvVFW1fq+1W7OnvF7YwP3/b5L7k9x96PFoVu8+76QMbW/LF+63qwoAAPSvRorqmUkW7b1QVdVzSTb3frZfpZSeJO9L8oeHG5DmNHH08MydPSVffXBp1m3ZUXccAABgEGmkqI5NsraP9TW9nx3I/5fkU1VVLT7UYDS/910wLZu378pXfrKk7igAAMAg0ujrafp6D0nZz/qeD0t5V5IzkvznRsOUUq4tpcwrpcxbvXp1o7dRk5lTO3Pu9HH5wgPPZOeu3XXHAQAABolGiuqaJF19rHem753WlFKGJPnTJH+cpK2U0pXk5cFLI0spo/u6r6qqm6qqmlNV1ZwJEyY0EI26XfO66Vm2dkvufHRl3VEAAIBBopGiuij7PItaSjkxycjs8+zqXkYm6U7yiewpumuSzO/97NYkPzucsDSfS86alBPHjcgthioBAAD9pJGiekeSy/bZBb0qyZYk9+7nno1JLtrn69d7P/sPSd59WGlpOu1tJVefPz0/eWZNFiztc4MdAADgkDRSVD+bZFuSr5dSLimlXJvkuiSf2PuVNaWUxaWUm5OkqqqdVVXds/dXkh/2Xrqwqqof9eu/glq9c053Rg3ryOfvf6buKAAAwCBw0KJaVdWaJBcnaU9yW5Lrk9yY5OP7XNrRew0tZvTwIXnHnO7cvmB5Vq7fWnccAABggGto6m9VVY9WVfWmqqpGVFU1uaqq/6eqql37XDOtqqqrD/A7nqmqqlRVdfsRZqYJXX3+tOzcXeVLP3y27igAAMAA1+jraeCATj5+ZC45a1K+/KPnsnXHroPfAAAAsB+KKv3mmgum56VN2/ONh5bVHQUAABjAFFX6zXmnjMtZk8fklvueSVVVdccBAAAGKEWVflNKyTUXTMtjKzfkgSdfrDsOAAAwQCmq9Ku5s6dk/KihueW+p+uOAgAADFCKKv1q+JD2vPvck3P3olV5+oVNdccBAAAGIEWVfvfu807K0Pa2fOF+u6oAAMChU1TpdxNHD8/c2VPy1QeXZt2WHXXHAQAABhhFlaPifRdMy+btu/KVnyypOwoAADDAKKocFTOndubc6ePyhQeeyc5du+uOAwAADCCKKkfNNa+bnmVrt+TOR1fWHQUAABhAFFWOmkvOmpQTx43ILYYqAQAAh0BR5ahpbyu5+vzp+ckza7Jg6dq64wAAAAOEospR9c453Rk1rCOfv/+ZuqMAAAADhKLKUTV6+JC8Y053bl+wPCvXb607DgAAMAAoqhx1V58/LTt3V/nSD5+tOwoAADAAKKocdScfPzIXnzkpX/7Rc9m6Y1fdcQAAgCanqHJMXPO6aXlp0/Z846FldUcBAACanKLKMfErpxyfM08YnVvueyZVVdUdBwAAaGKKKsdEKSXXvG56Hlu5IQ88+WLdcQAAgCamqHLM/IvZU3L8yKG55b6n644CAAA0MUWVY2b4kPa8+7yTc/eiVXn6hU11xwEAAJqUosox9Z7zTsqQ9pIv3G9XFQAA6JuiyjE1cfTwzJ09JV99cGnWbVQwwn4AACAASURBVNlRdxwAAKAJKaocc9dcMD2bt+/KV+ctqTsKAADQhBRVjrmZUzvz2unj8vn7n8nOXbvrjgMAADQZRZVaXHPB9CxbuyV3/Xxl3VEAAIAmo6hSi0vPnpTusSNyy33P1B0FAABoMooqtWhvK7n6/Gn58TMvZeHSdXXHAQAAmoiiSm3eec6JGTm0PZ/3qhoAAGAviiq1GTN8SN4x58TctmB5Vq3fWnccAACgSSiq1Orq86dl5+4qX/rhs3VHAQAAmoSiSq2mjR+Zi8+clC/96Lls3bGr7jgAAEATUFSp3TWvm5aXNm3P1x5cWncUAACgCSiq1O5XTjk+rzqpK5/63uJs22lXFQAAWp2iSu1KKfnYpWdkxbqtufXHS+qOAwAA1ExRpSlccOrxee30cfnU9xZ7VhUAAFqcokpT2LOrenpWbdhmAjAAALQ4RZWmce4px+d1p47PZ+55Mpu27aw7DgAAUBNFlabyB5eenhc3bc9f/eCZuqMAAAA1UVRpKq85eWzeeMaE3PT9p7Jh64664wAAADVQVGk6H7309KzdvCOfv/+ZuqMAAAA1UFRpOj3dXbn07En5i396Kus221UFAIBWo6jSlD566enZsHVn/vK+p+qOAgAAHGOKKk3prMlj8rZZk3PLfU/npU3b644DAAAcQ4oqTevfXHJaNu/Ylc99/8m6owAAAMeQokrTOm3S6PyL2VPyxQeezeoN2+qOAwAAHCOKKk3t9y8+Ldt27spn7rGrCgAArUJRpamdMmFUrnx1d770o2fz/LqtdccBAACOAUWVpvf7F5+W3burfPqexXVHAQAAjoGGimop5exSyt2llM2llOWllBtKKe0HuecVpZRv916/rZTyXCnlL0spk/snOq3ixHHH5R1zTsytP16SZWu31B0HAAA4yg5aVEspY5PclaRK8vYkNyT5WJLrD3JrZ5Knk/xhksuSfDzJJUm+VUrpOILMtKDfe9OpSZI//8cnak4CAAAcbY0Uxg8mGZHkyqqq1ie5s5QyJsl1pZQ/6V37JVVVPZDkgb2W7imlLE3y3SQ9SX56ZNFpJVO6RuTXX3tivvyj5/KhC0/NSccfV3ckAADgKGnk6O/lSb6zTyG9NXvK64WH+Pde7P0+9BDvg3z4olPT3lbyybvtqgIAwGDWSFE9M8mivReqqnouyebezw6olNJWShlaSjkjyX9P8pMkPz6MrLS4SWOG5z3nnZx/+NnSPLl6Y91xAACAo6SRojo2ydo+1tf0fnYw30qyLXvK7rgkV1RVtbvhhLCXD71xRoZ1tOeTd9lVBQCAwarR19NUfayV/azv6/eSnJfkvUlGJbmjlDK8rwtLKdeWUuaVUuatXr26wWi0kvGjhuW3zp+W2xYsz+MrN9QdBwAAOAoaKaprknT1sd6Zvndaf0FVVU9UVfWjqqq+lD3Tf1+V5Df2c+1NVVXNqapqzoQJExqIRiv6nTeckpFDO/I/73q87igAAMBR0EhRXZR9nkUtpZyYZGT2eXb1YKqqejbJS0lOOZT7YG9jRw7NNRdMy7cWPp9Hlq+rOw4AANDPGimqdyS5rJQyeq+1q5JsSXLvofyx3oFKx2fP+1XhsP3260/J6OEdufFOz6oCAMBg00hR/Wz2DEP6einlklLKtUmuS/KJvV9ZU0pZXEq5ea+f/0cp5b+XUn61lHJRKeXDSb6T5Mnseb0NHLbOEUPygdefkrt+vjLzlxz0BDoAADCAHLSoVlW1JsnFSdqT3Jbk+iQ3Jvn4Ppd29F7zsnlJXp/k5iTfTPKvk/x9kvOqqtp0xMlpee+7YFq6jhuST9zpWVUAABhMOhq5qKqqR5O86SDXTNvn51tj55SjaPTwIfmdN8zIH397UR589qW85uRxdUcCAAD6QaOvp4Gm9Fvnn5zxo4bmz75rVxUAAAYLRZUB7bihHfnghTPywJMv5gdPvlh3HAAAoB8oqgx47znv5EwaMyw33vl4qqqqOw4AAHCEFFUGvOFD2vORi07Nj595KfctfqHuOAAAwBFSVBkUrjrnxEzpHJ4/+65dVQAAGOgUVQaFYR3t+d03nZaHlqzN9x5bVXccAADgCCiqDBrvmNOdE8eNyCc8qwoAAAOaosqgMaS9Lf/6Tafl4WXr851HVtYdBwAAOEyKKoPKr75qak4ZPzI33vl4du+2qwoAAAORosqg0tHelt+/5LQ8tnJDvvXwirrjAAAAh0FRZdC5omdKTps4Kv/zrieyy64qAAAMOIoqg057W8m/ueT0LF61Mf9n/rK64wAAAIdIUWVQunzmCTnzhNH55F1PZOeu3XXHAQAADoGiyqDU1lby0UtPzzMvbs7Xf2pXFQAABhJFlUHr0rMnpae7M5+8+4ls32lXFQAABgpFlUGrlJI/uPT0LFu7JV+Zt6TuOAAAQIMUVQa1N54+Ia8+qSuf+t7ibN2xq+44AABAAxRVBrVSSj725jOyYt3W3Prj5+qOAwAANEBRZdA7f8bxee30cfnUPU9my3a7qgAA0OwUVQa9Uko+dunpWb1hW770w2frjgMAAByEokpLOPeU4/O6U8fnM/c+mU3bdtYdBwAAOABFlZbx0Tefnpc2bc8XHnim7igAAMABKKq0jFefNDYXnTEhN33/qazfuqPuOAAAwH4oqrSUj156RtZt2ZFb7nu67igAAMB+KKq0lFndnXnz2ZNy8z89nXWb7aoCAEAzUlRpOX9w6enZsG1nbrnfrioAADQjRZWWc9bkMbnkrEn5wgPPZKMJwAAA0HQUVVrShy+akXVbduRvf/Rc3VEAAIB9KKq0pFefNDbnzzg+f/FPT2Xrjl11xwEAAPaiqNKyPnLRqVm1YVv+/qdL644CAADsRVGlZZ0/4/jM7u7M5+59Kjt37a47DgAA0EtRpWWVUvLhi07Ncy9tzjcXrqg7DgAA0EtRpaVdetaknDZxVD79vSeze3dVdxwAACCKKi2ura3kwxfNyGMrN+TuRavqjgMAAERRhcztmZLusSPyqe8tTlXZVQUAgLopqrS8jva2/M6FM/LQkrX5wVMv1h0HAABanqIKSd7xmu6MHzUsn/7ek3VHAQCAlqeoQpLhQ9rzgddPz32LX8j8JWvrjgMAAC1NUYVe7z7v5IwZ3pFP37O47igAANDSFFXoNWpYR64+f1q+88jKPLFyQ91xAACgZSmqsJerL5ieEUPa85l7PKsKAAB1UVRhL+NGDs1vnHtSvjF/eZa8tLnuOAAA0JIUVdjHB15/StpKctP3n6o7CgAAtCRFFfZxQufw/KtXd+fv5i3Jqg1b644DAAAtR1GFPvzOhTOyc9fu3Hzf03VHAQCAlqOoQh+mjx+Zt/VMyZd+8GzWbd5RdxwAAGgpiirsx4ffOCObtu/KF3/wTN1RAACgpSiqsB9nTR6TN505Mbfc/3Q2b99ZdxwAAGgZiiocwEcumpE1m3fkb3+8pO4oAADQMhRVOIDXnDwu504fl7/4/lPZvnN33XEAAKAlNFRUSylnl1LuLqVsLqUsL6XcUEppP8g955RSPl9KWdx732OllI+XUob3T3Q4Nj5y0al5fv3W/MPPltYdBQAAWsJBi2opZWySu5JUSd6e5IYkH0ty/UFuvSrJjCR/nOStST6V5KNJvnwEeeGYe/1p4zNz6ph85p4ns2t3VXccAAAY9DoauOaDSUYkubKqqvVJ7iyljElyXSnlT3rX+vLHVVWt3uvne0opW5N8rpRyclVVzx5ZdDg2Sin5yBtPzYe+/NN8a+GKzJ09pe5IAAAwqDVy9PfyJN/Zp5Demj3l9cL93bRPSX3Zz3q/T2w4ITSBy15xQmZMGJlP3/NkqsquKgAAHE2NFNUzkyzae6GqqueSbO797FCcn2R3kscO8T6oVVtbyYfeeGp+vmJ97nmsr/8PBgAA6C+NFNWxSdb2sb6m97OGlFJOSPIfk/z1AY4LQ9N6+yunZGrXiPz59xbbVQUAgKOo0dfT9PW/yst+1n/5wlKGJvlKko1J/uAA111bSplXSpm3erVdK5rLkPa2XPuGU/Lgs2vy46dfqjsOAAAMWo0U1TVJuvpY70zfO62/oJRSknwxySuSvLWqqjX7u7aqqpuqqppTVdWcCRMmNBANjq2rzjkx40cNzafvebLuKAAAMGg1UlQXZZ9nUUspJyYZmX2eXd2PG7PntTZvr6qqkeuhaQ0f0p5rXjc99z6+Og8vW1d3HAAAGJQaKap3JLmslDJ6r7WrkmxJcu+Bbiyl/Pskv5fkPVVV3XfYKaGJvOe8kzN6eEc+fc/iuqMAAMCg1EhR/WySbUm+Xkq5pJRybZLrknxi76FIpZTFpZSb9/r5N5L81+w59ruslHLeXl/O9TJgjRk+JL/5Kyfnjoefz+JVG+uOAwAAg85Bi2rvM6UXJ2lPcluS67PnOO/H97m0o/eal7259/vVSX6wz9fbjiQ01O19F0zPsI62fO5ez6oCAEB/62jkoqqqHk3ypoNcM22fn6/OnpIKg874UcPyrnNOypd++Gz+zaWnZ2rXiLojAQDAoNHo62mAfVz7hlOSJH/x/adqTgIAAIOLogqHaUrXiPzqq6bmb3/8XF7YuK3uOAAAMGgoqnAEPvjGGdm+a3c+f//TdUcBAIBBQ1GFIzBjwqi8debkfPGBZ7N+64664wAAwKCgqMIR+tAbZ2TDtp356x88W3cUAAAYFBRVOEIzp3bmwtMn5Jb7ns6W7bvqjgMAAAOeogr94CMXnZoXN23PV+YtqTsKAAAMeIoq9IPXTh+Xc6aNzefufTLbd+6uOw4AAAxoiir0kw9fdGqWr9uabzy0rO4oAAAwoCmq0E/eePqEnD15TD5z75PZtbuqOw4AAAxYiir0k1JKPnzRjDy1elO++8jzdccBAIABS1GFfnT5zMmZPn5kPnXP4lSVXVUAADgciir0o/a2kg9dOCMPL1uf7z/xQt1xAABgQFJUoZ/9y1dNzeTO4fn09xbXHQUAAAYkRRX62dCOtnzg9afkR0+/lHnPvFR3HAAAGHAUVTgK3vXaEzNu5NB8+p4n644CAAADjqIKR8FxQztyzQXT8o+LVuXR5evrjgMAAAOKogpHyXt/ZVpGDevIZ+61qwoAAIdCUYWjpHPEkLznvJPzzQXL8/QLm+qOAwAAA4aiCkfRb79ueoa0t+VzdlUBAKBhiiocRRNGD8tV55yYv//p0ixbu6XuOAAAMCAoqnCUXfuGU9LR1pbf/ZufZuuOXXXHAQCApqeowlHWPfa43HjV7PzsubX5919fmKqq6o4EAABNTVGFY+AtMyfnD998ev7hZ8u8WxUAAA6io+4A0Co+ctGpeWLVxvzpdx7LjAmj8paZJ9QdCQAAmpIdVThGSin543/Vk1ee2JU/+LuH8vCydXVHAgCApqSowjE0fEh7bvrN12TscUPygS/Oy6oNW+uOBAAATUdRhWNs4ujh+YvfmpO1m3fk2i8+aBIwAADsQ1GFGrxiSmduvOqVeWjJ2vzbv19gEjAAAOxFUYWavGXmCfmjy87INx5ank99b3HdcQAAoGmY+gs1+vAbZ2Txqo35H999PKdOHJW3zJxcdyQAAKidHVWoUSkl/+3KWXnVSV35g7+bbxIwAABEUYXaDR/SnpveO+f/TgJebxIwAACtTVGFJjBh9LD85W+dk3VbduQDf20SMAAArU1RhSZx9pQxufGqV2bB0rX5o6+ZBAwAQOtSVKGJXPaKPZOAb5u/PH/+jyYBAwDQmkz9hSbzoQtnZPHKjfmzOx/PjImj8tZZJgEDANBa7KhCkyml5L9eOSuvPqkrH/3KQ1m41CRgAABai6IKTWj4kPZ87r1zcvzIYfnAF+dlpUnAAAC0EEUVmtSeScBzsn7rjlz7xXkmAQMA0DIUVWhiZ00ek0++61VZsGxd/vCr800CBgCgJSiq0OQuPXtS/u1bzsztC1bkf91tEjAAAIOfqb8wAPzOG07J4ys35Ma7Hs+pE0flbT0mAQMAMHjZUYUBoJSS/3blrMw5eWw+9tWHsmDp2rojAQDAUaOowgAxrKM9n33va/55EvDz60wCBgBgcFJUYQAZP2pYbr56TjZu3ZkPfHFetmw3CRgAgMFHUYUB5swT9kwCfnj5nknAu3ebBAwAwOCiqMIAdMnZk/Lv3nJmvrlwRT559xN1xwEAgH5l6i8MUNe+4ZQ8sWpjPnn3Ezl14qjMnT2l7kgAANAv7KjCAFVKyX/51Zk5Z9rY/OFX52f+EpOAAQAYHBRVGMCGdbTns+95TSaMNgkYAIDBo6GiWko5u5RydyllcylleSnlhlJK+0HuGVpK+dNSyj+VUraUUkx8gaPg+FHDcvNvnZNN23bm/V/8iUnAAAAMeActqqWUsUnuSlIleXuSG5J8LMn1B7n1uCTvT7I5yQNHFhM4kDNOGJ3/9euvyiPL1+djX33IJGAAAAa0RnZUP5hkRJIrq6q6s6qqz2ZPSf1oKWXM/m6qqmptknFVVV2W5B/6JS2wXxefNSn/4fKz8q2Fz+d/3vV43XEAAOCwNVJUL0/ynaqq1u+1dmv2lNcLD3RjVVW2deAYev/rp+edc7rzv/5xcf7uJ8/VHQcAAA5LI0X1zCSL9l6oquq57DnSe+bRCAUcnlJK/vO/nJU3nD4h/+7rC/P3Dy6tOxIAAByyRorq2CR9vfdiTe9nQBMZ2tGWm977mpw/4/j80dfm5xsPLas7EgAAHJJGX0/T1xHesp/1w1ZKubaUMq+UMm/16tX9+auhpQwf0p6//M1zcs60cfnoV+bnmwtW1B0JAAAa1khRXZOkq4/1zvS903rYqqq6qaqqOVVVzZkwYUJ//mpoOSOGtueWq8/Jq07syu/f+rN855Hn644EAAANaaSoLso+z6KWUk5MMjL7PLsKNJeRwzry+fedk1ndnfndv/lp7v75yrojAQDAQTVSVO9IclkpZfRea1cl2ZLk3qOSCug3o4cPyRfe99qcNXlMPvSln+aex1bVHQkAAA6okaL62STbkny9lHJJKeXaJNcl+cTer6wppSwupdy8942llMtLKb+W5JW9P/9a79fJ/fYvAA6qc8SQfPGa1+bUiaNy7V8/mPueeKHuSAAAsF8HLapVVa1JcnGS9iS3Jbk+yY1JPr7PpR291+ztM0m+muS3e3/+au/XRYcfGTgcXccNzZfff25OGT8y7//iT/KDJ1+sOxIAAPSpVFW/Du7tN3PmzKnmzZtXdwwYdF7YuC2/ftMPs2ztlvzVNa/NOdPG1R0JAIAWVEp5sKqqOX191ujraYBBYvyoYfnyB87NCZ3Dc/UtP86Dz66pOxIAAPwCRRVa0MTRw/O3HzgvE0YPy9W3/Djzl/Trm6YAAOCIKKrQoiaNGZ6/+cB56Ro5JO+9+Ud5eNm6uiMBAP9/e/ceZ2ddH3j88z1nrkkGJlegkYQkXGJE1HBHwHppVayLWi2CuBUQa7v7Qu1uu1vEhbrtWtsXBaWvigSRUhatZd1aeQm8SrdKMCYIQaxAVJKQgApJYEIyk8z1/PaP55yZM2fO3JJhzknm835xOOf5XZ7ne5Lf68nzPb/nIgkwUZVmtF9rb+Wuj55FW0sjl35lA0/9as/4nSRJkqRXmImqNMMdO28Wd115Ji0NeT506wZ+9sLeWockSZKkGc5EVRJL58/max87i4ZccMmaDTy9o7PWIUmSJGkGM1GVBMCyBbO568qzgMQla9azdVdXrUOSJEnSDGWiKmnQ8YvmcNeVZ9FfyJLV7S/uq3VIkiRJmoFMVCUNc+JRbdx5xZns7xvg4jXrea7DZFWSJEnTy0RV0girfu0I7rziTPZ293HxmvX8cvf+WockSZKkGcREVVJVJy8+kr+/4kx2d/VxyZr1vLCnu9YhSZIkaYYwUZU0qtcd287tl5/Bzr09XLxmPTv2mqxKkiTplWeiKmlMpy6dy1cvO4Nf7e7mQ2s2sKuzp9YhSZIk6TBnoippXGcsm8dtHzmdZzv2cemtG+jo6q11SJIkSTqMmahKmpCzV8zn1v94Olt2dXHpVzbw8r6+WockSZKkw5SJqqQJO/eEBdzy4VP5+QudfPi2DezpNlmVJEnS1DNRlTQpv37SIv72Q6t56ld7+N3bHmavyaokSZKmmImqpEl726qjuOni1fz4uZe57Ks/pKunv9YhSZIk6TBioirpgLzj5KP54gffwMbtHVx++w+dWZUkSdKUMVGVdMDedcox3HDR6/nhMy/xjhvX8tDPd9U6JEmSJB0GTFQlHZQLX7+Yu3//HJobc1z6lQ1c/X//nU5PBZYkSdJBMFGVdNBWL5nLd646jyvPW8bXHt7O2294kHVPO7sqSZKkA2OiKmlKtDTm+fS7VnH3x8+mqSHHJbdu4DP/9BNvtCRJkqRJM1GVNKVOXTqP71x1Hlecu4w7N2zjHV94kB9sfrHWYUmSJOkQYqIqacq1NuX5zG+t4hu/dzb5CC5es55rv/UT9vU6uypJkqTxmahKesWcftw87v3E+Vz2xuO4Y/023nHjWjZscXZVkiRJYzNRlfSKam3Kc+27X8PXrzwLgItuWc91//yEs6uSJEkalYmqpGlx5vL53PfJ8/jIOcdx+7pneOcX1vLw1pdqHZYkSZLqkImqpGkzq6mB6/7Da/jalWdRSImLbvkBn/32k+zvHah1aJIkSaojJqqSpt3ZK+Zz3yfO58NnLeW272/lgi+u5dFtzq5KkiQpY6IqqSZmNzfw2QtP5q6Pnklvf4H33/wD/uyeJ+nuc3ZVkiRppjNRlVRT5xy/gPs/dT6XnLGEWx/aygVfWMuj2zpqHZYkSZJqyERVUs3NaW7gz9/7Wu684kx6+gt84OZ1fO47Tzm7KkmSNEOZqEqqG+eesID7PnkeF52+hC8/uIV3fXEtj213dlWSJGmmMVGVVFfaWhr53Pteyx2Xn8H+3gF++0vr+It7Nzm7KkmSNIOYqEqqS+efuJD7PnU+Hzj1WG7+3mbefdNDPP7s7lqHJUmSpGlgoiqpbh3R0sjn338Kt192Onu7+3nfl9bxl/dtoqff2VVJkqTDWaSUah1DVaeddlp65JFHah2GpDrx8v4+/uyeJ/nHR5/jhEVzeM8bFrN6yVxed+yRzGpqqHV4kiRJmqSIeDSldFrVOhNVSYeSf9u0g8/d+xQ/e6ETgHwuWHl0G6uXzOXUpXNZvWQux85rJSJqHKkkSZLGYqIq6bDT0dXLj57dzcbtHWzc3sGPtu+mqzc7JXjBnCbesCRLWlcvaeeUV7XT2pSvccSSJEkqN1ai6vlykg5Jc2c38eaVi3jzykUADBQSP31+72Di+tj23fzLky8A0JALXn3MEaxe0s7q4qzrq+Y66ypJklSvnFGVdNh6qauXx4qJ68Ztu3n8ud3sK866LmxrzhLXJXNZvXQur118JC2NzrpKkiRNF2dUJc1I82Y38dZXH8VbX30UAP0DBX76wl42bt/Nxm1ZAnv/E9msa2M+WHXMEdkpw0uzU4YXtzvrKkmSVAvOqEqa0XZ19vDY9uK1rts6+PFzL7O/L5t1XdTWXJxxbefEo9pYsXAOi9tbyeVMXiVJkg6WM6qSNIoFc5r5jVVH8RurhmZdN5Wudd3Wwcbtu7nviecH27c05li2YA4rFs5mxcI5rFiUfV6+YI43bJIkSZoizqhK0jhe6upl885ONu/ozN53drF5ZyfPvrSPQtkudHF7KysWzWH5gtmDCezxC+ewsK3ZU4glSZIqOKMqSQdh3uwm5s2ex+nHzRtW3t03wLYX941IYr/xzEuDN20CaGtuYPmislnYhXM4ftFslsybTVNDbrq/jiRJUt0zUZWkA9TSmOeko9s46ei2YeUpJZ7f083mHV3F5DV7rXv6Rb658ReD7fK5YOm8WSxfOIcVi4aS2OULZtM+q9FZWEmSNGOZqErSFIsIjjmylWOObOXcExYMq+vs6WfLzk627CxLYnd08eDPdtI7UBhs19yQY2FbM4vamllYfC1qaxlWtqithflzmmjMOysrSZIOLyaqkjSN5jQ3cMqr2jnlVe3DygcKiec6stOIt+zsYsfeHnbu7WHH3m627upiw9aX2L2vb8T6ImDerKbBZHb0pLaZOc0NztJKkqRDwoQS1YhYBdwEnA3sBm4F/jSlNDBOvyOBG4H3ADngHuCqlNKLBxO0JB1u8rlg6fzZLJ0/m7esrN6mp3+AXZ29WQK7p5udnT3s2NPDzs5SUtvDlp1d7NzbM2x2tqSlMTeUxM5pZtERzSyc08yCtmZmNeVpbczT0pindfBzjpZSWfE976N5JEnSNBg3UY2IucADwJPAhcAK4HqyxPOacbr/A3AS8FGgAHwe+CfgvAMPWZJmpuaGPIvbW1nc3jpmu5QSL+/vGzYrmyW3PYPJ7dM7O1m3eRd7uvsnFUNTPjeYwLY25WlpyNPSlKe1VFZMaFuKiW5reVlTnpaGHK1NeZob8jQ15GguvrLP+cHl5oY8zY05mvI5n1srSdIMNJEZ1Y8DrcD7Ukp7gH+JiCOA6yLiL4tlI0TE2cDbgTellB4slv0C2BARb0spPTA1X0GSVC4iaJ/VRPusJk48qm3Mtt19A7zY1cv+3n66+wrs7xugu2+A/b0DdPcX6O4dGCrrG6C7r0B32XJ5u5e6eovLA+zvLQy2GSgc3GPQGvMxLLFtKktmR0t2myqS3cZ8kM/laMgF+VzQkA8aqiznc5GV5bP3hlyOhnwMleeCxnxuzOXSevK5IBd4urUkSQdgIonqO4H7KxLSr5PNjr4J+PYY/V4oJakAKaWHI2Jrsc5EVZJqrKUxP+4M7cHqGxhKWnuKyXBvf4Ge/my5Z6CQvfeXyrPXYJuyz4P1fQV6B4bW0dnTP+o6qp0GPZ3yubJEN7IkOB8xWD7sVVbeUK0+lyMfZO85aMhlM865gFwEUXwfWh76XEqaB+tzk2xfXH+pTTD0mQiCsjbFz5T6ZU0GLdgnDQAAC0BJREFU6xnWtrQtgPLtMlhfuZ6gFEvZZ4rfiaF1xUT6VbShbH2lHxkq+5eU/jyG+g6Ph1L8g/2rtyfKl0dut3y95duuLK8asz+USDpETSRRXQn8v/KClNL2iNhXrBstUV0JbKpS/lSxTpI0AzTmczTmc7S1NNZk+4VCYiAlBgqJ/kKif6BAf2FoeWAg0VcoZMsDWXnlcn+hQP9AKus3VJ+VFYrrzpYLqdg3DdUVitsbfB/WZuSrsk1fX4GBwsDwdsW+hZS9UoKUGFwupOxU8EKprJDK6hnsU95eh6/yhDZbjrLPQ0k1wxLf0ufhCXep/2DzauuuUjZ8eVh0o8Q4cvvl9eXrHfE9yxP7sh5jraNKSOO2q7ad0X4fKI+12jrGi3lkjGOvb2R5xZ/VqH1G2+CIxRHfNSpajKifZPtqqv0AM15c1dpUW1f1NlWjOMDtVVvXyD+HMdtOYh3X/87raGnMV+9Q5yaSqM4lu4FSpY5i3YH0Wz6B7UqSdNByuSBHcIj+Oz3thiW2qUpiW4DEUBKcyMrJ/htKmMl+JKCyrPiZwXUU11dcbyovK9tGGuVz+bZLsSaG6knDt5/K60fpR7Es614eU1l98X+l+lJZKisbzPurxA0j11ltu+XrrSyH4fEOX+/w/pTHXdamWt/SdijvX/ZnNtZ6RsRVbb2VbSvaUNFmMn1TZdDDP46Ia3jfsdtRuY3KWMarr7LN8prR26ZRysdvP2wrFcXjfY/KNtXXUVGWBv834f7D69OY9dXWUbVdlUYTWVdlvKNub5zvkbWp/vdQTfXvNLG/x7FiGKv9oWCij6ep9hVjlPID7hcRHwM+BrBkyZIJhiZJkqZKRGSnF1edC5AkaXpM5CnxHUB7lfIjqT5jOl6/9tH6pZRuSSmdllI6beHChRMITZIkSZJ0uJlIorqJimtKI+JYYDbVr0EdtV/RaNeuSpIkSZI0oUT1XuDtEVH+jIOLgP3A98bpd3REnFsqiIjTyK5PvfcAYpUkSZIkzQATSVRvBnqAb0bE24rXkV4H/HX5I2si4umI+EppOaX0A+B+4I6IeF9EvAf438BDPkNVkiRJkjSacRPVlFIH8FYgT/Yomj8FbgCurWjaUGxT7oNks663AXcAjwLvPbiQJUmSJEmHswnd9Tel9CTwlnHaHFelbDdwWfElSZIkSdK4JnLqryRJkiRJ08ZEVZIkSZJUV0xUJUmSJEl1xURVkiRJklRXTFQlSZIkSXXFRFWSJEmSVFdMVCVJkiRJdcVEVZIkSZJUV0xUJUmSJEl1xURVkiRJklRXTFQlSZIkSXXFRFWSJEmSVFdMVCVJkiRJdcVEVZIkSZJUV0xUJUmSJEl1xURVkiRJklRXTFQlSZIkSXUlUkq1jqGqiNgJbKt1HONYAOyqdRA6ZDheNBmOF02UY0WT4XjRZDheNBkHMl6WppQWVquo20T1UBARj6SUTqt1HDo0OF40GY4XTZRjRZPheNFkOF40GVM9Xjz1V5IkSZJUV0xUJUmSJEl1xUT14NxS6wB0SHG8aDIcL5oox4omw/GiyXC8aDKmdLx4jaokSZIkqa44oypJkiRJqismqpMUEasi4l8jYl9E/DIiPhsR+VrHpfoTER+JiFTl9fFax6bai4jjI+LLEfF4RAxExHertImIuDoino2I/RHxYES8vgbhqsYmOF6eqbK/eb4G4aqGIuIDEfHPEfGLiOiMiEcj4uIq7a6MiJ9HRHexzVtrEa9qayLjJSK+O8rxTEut4lZtRMT7I2JdRLxY3Hf8NCKuiYimsjZTduzSMHWhH/4iYi7wAPAkcCGwArieLOG/poahqb69BdhftrylVoGorrwGuABYDzSN0ua/A58B/gjYBPwh8EBEnJxSMgGZWSYyXgDuAm4qW+59JYNSXfpDYCvwKbLnGV4A3BURC1JKNwFExAeBm4HrgIeAy4B7IuL0lNJPahK1amXc8VL0b8DVFX17pidE1ZH5ZGPhr4DdwBlk+5Gjgf9cbDNlxy5eozoJEfEnwB+TPZh2T7Hsjyn+BZXKJMhmVIGvAm0ppc4ah6M6ExG5lFKh+PluYEFK6dfL6luAF4DrU0qfLZbNBp4BvpxS8sexGWS88VIsfwa4O6X0X6c/QtWLYoKxq6LsLuDslNKy4vJPge+nlC4vLueAx4HHU0qXTnfMqp0JjpfvArtSSu+vQYiqcxHx58B/AuYCzUzhsYun/k7OO4H7KxLSrwOtwJtqE5KkQ1Ep6RjDOcARwDfK+nQB3ybbF2kGmcB4kQCoTDqKHgMWAUTEcuBEhu9bCsA/4r5lxhlvvEgT8CJDZ/pM6bGLierkrCSbwh6UUtoO7CvWSdVsjoj+4nn8v1frYHTIWAkMAD+vKH8K9zca3eUR0RsRL0fE3RGxtNYBqS6cQ3bZEgztPzZVtHkKmBcRC6ctKtWr8vFS8pvF+7Psi4j7I+KUWgSm+hAR+YiYFRHnAlcBX0rZabpTeuziNaqTM5fsfOxKHcU6qdyvyM7RfxjIAxcDN0fErJTSDTWNTIeCuUBnSmmgorwDmBURTSklrz9UuW+RXcP6HPBq4FpgbUS8NqX0ck0jU80Ub5J0IXB5sah0vFJ5PNNRVr9zGkJTHaoyXgC+B/wd8DSwFPg02b7ldSmlZ6Y9SNWDLrLTfAHuILseFab42MVEdfKqXdQbo5RrBksp3Q/cX1Z0b0Q0A9dExBc8lU8TMNr+ZrQ6zWAppU+ULa6NiHXAj8hulHNjbaJSLUXEcWQ32PpWSun2iurKfYj7lhlutPGSUrq2rNnaiHiAbEb+k8WXZp5zgFlkN1P6H8DfAH9QrJuyYxcT1cnpANqrlB9J9ZlWqdLdwO8Ax+HdfzW2DqAtIvIVv0y2A/tSSn01ikuHiJTST4o3zVld61g0/SJiHnAvsB0ov0FSaea0HSifaS8d33g8MwONMV5GSCk9HxHfx33LjJVS2lj8+FBE7AL+LiKuZ4qPXbxGdXI2UXF+dUQcC8xm5LUe0lj8xVrj2UR2yvjxFeUjrpWXxuH+ZoaJiFnAPWQ3OHlX8WYmJaX9R+X1YiuBl1JKnvY7w4wzXsbivkUApaR1GVN87GKiOjn3Am+PiLaysovInpH5vdqEpEPMb5M9p2xbrQNR3VsH7AE+UCooHky8m2xfJI0pIk4GTgIerXUsmj4R0UB2B98TgHemlHaU16eUtgA/Y/i+JVdcdt8yw4w3XkbpcxTwRty3KPPG4vtWpvjYxVN/J+dmsjtbfTMiPg8sJ3uG6l/7DFVVioj/Q3YjpR+T/bp0UfF1ldenqrjjvqC4uBg4IiJKz6j7TkppX0T8BfCZiOhg6KHZOeCmESvUYW288QK8mex0vXuAX5L9en0N2Wl8t09rsKq1vyUbK58gu4vvWWV1j6WUesiOXe4sPnv3+8DvkiUql0xvqKoDY44Xsh+7PkeWzG4DlgB/AhTw2vcZJyLuAx4AniC7u+8bgf8C/ENKaXOxzZQdu0R2J2FNVESsIrtg+Gyy6zhuBa6rcncrzXAR8b/IZlCPJbuI/EngxpTS39c0MNWF4k0rto5SvSyl9ExEBHA18PvAfOARsh86HpuWIFU3xhsvZM+tuwE4hexaoBeB+4CrU0q/nIYQVSeKyedojyVaVrpLa0RcCfw3sn+jngD+KKX0r9MRo+rHeOMF6APWAG8g+3doL/Bd4NMpJS9DmWEi4n8C7yW710o/2f1WvgrcXLr+dCqPXUxUJUmSJEl1xWtUJUmSJEl1xURVkiRJklRXTFQlSZIkSXXFRFWSJEmSVFdMVCVJkiRJdcVEVZIkSZJUV0xUJUmSJEl1xURVkiRJklRXTFQlSZIkSXXl/wPWrLnhhPPE9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.plot(tr_loss_hist)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the performance of this simple network shows almost 100% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this post, we just cover the basic concept of **many-to-one** type RNN model for sentiment classification, and implement it with Tensorflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "{{ 'Reference from stanford CS231n lecture note' | fndetail: 1 }}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}