{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 13 - NLP with Deep Learning\n",
    "\n",
    "> An introduction to Deep Learning and its applications in NLP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lewtun/dslectures/blob/master/notebooks/lesson13_nlp-deep.ipynb)[![slides](https://img.shields.io/static/v1?label=slides&message=lesson13_nlp-deep.pdf&color=blue&logo=Google-drive)](https://drive.google.com/open?id=1g613_b3643zUuPVB1JBBuyxXka_WRRWF)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: Make sure you are connected to a GPU machine when running the Colab notebook by clicking on `Runtime -> Change runtime type` and set hardware type to GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "In this lecture we cover the basics of Deep Learning and its application to NLP. The learning goals are:\n",
    "* The basics of transfer learning\n",
    "* Preprocess data with the fastai data loader\n",
    "* Train a text classifier with the fastai library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "* Practical Deep Learning for Coders - Lesson 4: Deep Learning 2019 by fastai [[video](https://www.youtube.com/watch?v=qqt3aMPB81c)]\n",
    "\n",
    "This notebooks follows fastai's excellent tutotrial in this video and the original notebook can be found [here](https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson3-imdb.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework\n",
    "As homework read the references, work carefully through the notebook and solve the exercises. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "Last time we built a sentiment classifier with `scikit-learn` and achieved around 85% accuracy on the test set. This is already pretty good, but can we do better? We are still wrong 15/100 times. It turns out we can if we use deep learning.\n",
    "\n",
    "<div style=\"text-align: center\">\n",
    "<img src='images/deeper.jpg' width='400'>\n",
    "</div>\n",
    "\n",
    "Deep learning uses an architecture that is modeled after the brain and uses networks of artificial neurons to mimic its behaviour. These models are much bigger than the models we encountered so far and can have millions to billions of parameters. Training these models and adjusting the parameters is also more challenging, and generally requires much more data and compute. A lot of computations are easily parallelizable, which is a strength of modern GPUs. Therefore we will run this notebook on a GPU that enables much faster training than a CPU.\n",
    "\n",
    "<div style=\"text-align: center\">\n",
    "<img src='images/gpu_meme.jpg' width='400'>\n",
    "</div>\n",
    "\n",
    "Since we don't have much training data on the IMDb dataset for deep learning standards we use transfer learning to still achieve high accuracy in predicting the sentiment of the movie reviews."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer learning\n",
    "Training deep learning models requires a lot of data. It is not uncommon to train models on millions of images or gigabytes of text data to achieve good results. Most real-world problems don't have that amount of labeled data ready, and not all companies or individuals who want to train a model can afford to hire people to label data for them.\n",
    "\n",
    "For many years this has been very challenging. Fortunately, it has been solved for image based models a couple of years ago and recently also for NLP. One approach that helps train models with limited labeled data is called **transfer learning**.\n",
    "\n",
    "The idea is that once a model is trained on a large dataset for a specific task (e.g., classifying houses vs. planes), the model has learned certain features of the data that can be reused for another task. Such features could be how to detect edges or textures in images. If these features are useful for another task, then we can train the model on new data without requiring as many labels as if we were training it from scratch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ULMFiT\n",
    "For transfer learning in NLP Jeremy Howard and Sebastian Ruder came up with a similar approach called `ULMFiT` (Universal Language Model Fine-tuning for Text Classification) for texts. The central theme of the approach is language modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Language modeling\n",
    "In language modeling the goal is to predict the next word based on the previous word in a text. An example:\n",
    "\n",
    "`Yesterday I discovered a really nice cinema so I went there to watch a ____ .`\n",
    "\n",
    "The task of the model is to fill the blank. The advantage of language modeling is that it does not require any labels, but to achieve good results, the model needs to learn a lot about language. In this example, the model needs to understand that one watches movies in cinemas. The same goes for sentiment and other topics. With `ULMFiT` one can train a language model and then use it for classifications tasks in three steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Three steps\n",
    "The three steps are visualised in the following figure:\n",
    "<div style=\"text-align: center\">\n",
    "<img src='images/ulmfit_approach.png' width='800'>\n",
    "</div>\n",
    "\n",
    "1. **Language model (wiki)**: A language model is trained on a large dataset. Wikipedia is a common choice for this task as it includes many topics, and the text is of high quality. This step usually takes the most time on the order of days. In this step, the model learns the general structure of language.\n",
    "\n",
    "2. **Language model (domain)**: The language model trained on Wikipedia might be missing some aspects of the domain we are interested in. If we want to do sentiment classification, Wikipedia does not offer much insight since Wikipedia articles are generally of neutral sentiment. Therefore, we continue training the language model on the text we are interested in. This step still takes several hours.\n",
    "\n",
    "3. **Classifier (domain)**: Now that the language model works well on the text we are interested in, it is time to build a classifier. We do this by adapting the output of the network to yield classes instead of words. This step only takes a couple of minutes to an hour to complete.\n",
    "\n",
    "The power of this approach is that you only need little labeled data for the last step and only need to go through the expensive first step once. Even the second step can be reused on the dataset if you, for example, build a sentiment classifier and additionally a topic classifier. This can be done in minutes and allows us to achieve great results with little time and resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other methods\n",
    "Today, there are many approaches in NLP that use transfer learning such as Google's BERT. These models cost on the order of $100'000 to pretrain (1. step) and massive computational facilities. Fortunately, most of these models are shared and can then be fine-tuned on small machines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `fastai` library\n",
    "The `fastai` library wraps around the deep learning framework `PyTorch` and has a lot of functionality built in to achieve great results quickly. The library abstracts a lot of functionality, so it can be difficult to follow initially. To get a better understanding, we highly recommend the [fastai course](https://course.fast.ai/). In this lesson, we will use the library to build a world-class classifier with just a few lines of code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install dslectures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *\n",
    "from dslectures.core import get_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download data\n",
    "The fastai library includes a lot of datasets including, the IMDb dataset we already know. Similar to our `download_dataset()` function we can do this with fastai's `untar_data()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.IMDB)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data structure\n",
    "Looking at the downloaded folder, we can see that there are several files and folders. The relevant ones for our case are `test`, `train`, and `unsup`. The `train` and `test` folders split the data the same way we split it in the previous lecture. The new `unsup` (for unsupervised) folder contains 50k movie reviews that are not classified. We can't use them for training a classifier, but we can use them to fine-tune the language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/storage/data/imdb/README'),\n",
       " PosixPath('/storage/data/imdb/tmp_lm'),\n",
       " PosixPath('/storage/data/imdb/imdb.vocab'),\n",
       " PosixPath('/storage/data/imdb/tmp_clas'),\n",
       " PosixPath('/storage/data/imdb/test'),\n",
       " PosixPath('/storage/data/imdb/train'),\n",
       " PosixPath('/storage/data/imdb/unsup')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the training folder, we find two folders for the negative and positive movie reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(path/'train/').ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In both folders, we find many files, each containing one movie review. This is exactly the same data we used last time. It is just arranged in a different structure. We don't need to load all these files manually - the fastai library does this automatically. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/storage/data/imdb/train/neg/573_1.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/4084_1.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/10475_1.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/1830_1.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/3070_2.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/9556_4.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/6659_1.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/6035_1.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/6056_2.txt'),\n",
       " PosixPath('/storage/data/imdb/train/neg/1579_1.txt')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(path/'train/neg').ls()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tune language model\n",
    "\n",
    "> Note: Fine-tuning the language model takes around 4h. You can skip this step and download the fine-tuned model in the *Load language model* section of the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess data for language modeling\n",
    "In the last lecture, we implemented our own function to preprocess the texts and tokenize them. In principle, we could do the same here, but fastai comes with built-in functions to take care of this. In addtion, we can specify which folders to use and what percentage to split off for validation. The batch size `bs` specifies how many samples the model is optimised for at each step. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = (TextList.from_folder(path)\n",
    "           #Inputs: all the text files in path\n",
    "            .filter_by_folder(include=['train', 'test', 'unsup']) \n",
    "           #We may have other temp folders that contain text files so we only keep what's in train and test\n",
    "            .split_by_rand_pct(0.1)\n",
    "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
    "            .label_for_lm()           \n",
    "           #We want to do a language model so we label accordingly\n",
    "            .databunch(bs=bs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the vectorizer vocabulary, we can have a look at the encoding scheme. The `itos` (stands for id-to-string) object tells us which token is encoded at which position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(data_lm.vocab.itos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the vocabulary contains XXX tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxeos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " 'the']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.vocab.itos[:20]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the first few positions are reserved for special tokens starting with `xx`. The token `xxunk` is used for a word that is not in the dictionary. The `xxbos` and `xxeos` identify the beginning and the end of a string. So if the first entry in the encoding vector is 1 this means that the token is `xxunk`. If the third entry is 1 then the token is `xxbos`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at a processed text. We notice that the token `xxmaj` is used frequently. It signifies that the first letter of the following word is capitalised."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text xxbos xxmaj while the design and locations and photography are strong assets in this film ; it is a turgid and melodramatic affair which demonstrates the limits of cinema to convey truth . \n",
       " \n",
       "  xxmaj the case is the use of the soundtrack music : a mix of xxmaj gustav xxmaj mahler and xxmaj andrew xxmaj lloyd - xxmaj webber that plays constantly and loudly , and would have made xxmaj max xxmaj steiner grimace at its over use as it instructs the audience how difficult ; how ecstatic ; how tortured it is to be an artist . xxmaj and then it really counts the story xxunk the details at the end . \n",
       " \n",
       "  xxmaj this heightened and kitsch exploitation of emotions was once well ridiculed by xxmaj peter xxmaj ackroyd about a xxmaj yukio xxmaj mishima book : xxmaj this is not writing , this is xxmaj barbara xxmaj cartland . xxmaj precisely the same critique can be made of this film : a deceptive , mawkish vanity project ."
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.train_ds[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With fastai we can also sample a batch from the dataset and display the sample in a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>xxmaj max xxmaj steiner grimace at its over use as it instructs the audience how difficult ; how ecstatic ; how tortured it is to be an artist . xxmaj and then it really counts the story xxunk the details at the end . \\n \\n  xxmaj this heightened and kitsch exploitation of emotions was once well ridiculed by xxmaj peter xxmaj ackroyd about a xxmaj yukio xxmaj mishima</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>call it horror . xxmaj there is too much talking , you will get bloodthirsty after watching it xxbos i 'd liked the xxmaj takashi xxmaj miike films i 'd seen so far , but i found this pretty disappointing . i 'd bought it , but i wo n't be keeping it . \\n \\n  i saw it on the xxmaj xxunk xxup dvd , which has just</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>xxmaj gable 's laughable playing of an xxmaj irish patriot in xxmaj parnell . \\n \\n  xxmaj it 's inconceivable that xxmaj bergman would choose both this movie and its director over a lucrative xxmaj hollywood career where she could choose among the finest scripts and directors being offered at that time . xxmaj to begin with , there was no script to work with except a few notes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>worst of the lot ! xxmaj it does not have the feel of \\n \\n  the original or 2nd , xxmaj the enemy 's look stupid and the levels are \\n \\n  even worse . xxmaj it does not look realistic anymore ! i did not enjoy his game unlike the last two , the first had great \\n \\n  background music and the 2nd was not</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>ever made , ever . xxmaj unfortunately my worst fears were confirmed &amp; one has to say that xxmaj curse of the xxmaj wolf is a truly horrible film in every way , both conceptually &amp; technically . xxmaj curse of the xxmaj wolf is the sort of film where the low budget dictates what happens &amp; the script rather than the script dictating the budget . xxmaj you get</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Representation\n",
    "When we vectorized the texts in the last lecture, we represented them as count vectors. The architecture we use in this lecture allows for each word to be processed sequentially and thus conserving the order information. Therefore we encode the text with **one-hot encodings**. However, storing the information as vectors would not be very memory efficient; one entry is 1 and all the other entries are 0. It is more efficient to just store the information on which entry is 1 and then create the vector when we need it.\n",
    "\n",
    "We can look at the data representation of the example we printed above. Each number represents a word in the vocabulary and specifies the entry in the one-hot encoding that is set to one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,    5,  154,    9, 1682,   12, 1854,   12, 1352,   39])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.train_ds[0][0].data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the `itos` we can translate it back to tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in data_lm.train_ds[0][0].data[:10]:\n",
    "    print(data_lm.vocab.itos[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train model\n",
    "We will train a variant of a model called LSTM (long short-term memory) network. This is a neural network with a feedback loop. That means that when fed a sequence of tokens, it feeds back its output for the next prediction. With this the model has a mechanism remembering the past inputs. This is especially useful when dealing with sequential data such as texts, where the sequence of words and characters carries important meaning.\n",
    "\n",
    "<div style=\"text-align: center\">\n",
    "<img src='images/rnn-diagram.png' width='400'>\n",
    "    <p style=\"text-align: center;\"> <b>Reference:</b> https://colah.github.io/posts/2015-08-Understanding-LSTMs/ </p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load pretrained model\n",
    "Training the model on Wikipedia takes a day or two. Fortunately, people have already trained the model and shared it in the fastai library. Therefore, we can just load the pretrained langauge model. When we load it, we also pass the dataset it will be trained on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3, model_dir=\"../data/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Learning rate finder\n",
    "The learning rate is a key parameter when training models in deep learning. It specifies how strongly we update the model parameters. If the learning rate is too small, the training takes forever. If the learning rate is too big, we will never converge to a minimum.\n",
    "\n",
    "With the `lr_find()` function, we can explore how the loss function behaves with regards to the value of the learning rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='0' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      0.00% [0/1 00:00<00:00]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='99' class='' max='8052', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      1.23% [99/8052 00:17<22:46 11.5508]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEGCAYAAABYV4NmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deXzV9Zn3/9eVhCRAFtYQIEAAkUWQRUARd6jivnSzrrVVb6e2te1M2+nMPXN37k5n2vEex3a0Vdu6dKzVcao/17rUgqi4sIMsshOWrASykv36/XFOaMQkRDjfs+X9fDzy4Jzv93POuT6ck1zn8/1s5u6IiIhEWkqsAxARkeSkBCMiIoFQghERkUAowYiISCCUYEREJBBpsQ4gkoYMGeKFhYWxDkNEJGGsXLmywt2HBvHcSZVgCgsLWbFiRazDEBFJGGa2O6jn1iUyEREJhBKMiIgEQglGREQCoQQjIiKBUIIREZFAKMGIiEgglGBERCQQSjAiIgns9Y2lPPjm9liH0SklGBGRBPbHD4t5bNmuWIfRKSUYEZEEVlrdwLDczFiH0SklGBGRBFZS1UB+jhKMiIhEWGl1I8OUYEREJJJqG1uobWwhX5fIREQkkkqqGgB0iUxERCKrtDqUYHSJTEREIupIC0aXyEREJJJKjrRgMmIcSeeUYEREElRpdQPZmWn0S4/PzYmVYEREElQ8z4EBJRgRkYRVWt0Qt/0vEIUEY2apZrbazF7s5NxAM3vWzNaZ2QdmNrXDuUVm9pGZbTOzvw06ThGRRFNS3RC3I8ggOi2Yu4BNXZz7O2CNu58K3AT8DEJJCbgfuBiYAnzJzKZEIVYRkYTQ0tpGeU1j771EZmYFwKXAr7soMgV4A8DdNwOFZjYMmAtsc/cd7t4EPAlcGWSsIiKJ5EBdE21O3C50CcG3YO4Fvge0dXF+LXANgJnNBcYABcBIYE+HcnvDxz7BzG43sxVmtqK8vDxScYuIxLV4n8UPASYYM7sMKHP3ld0U+wkw0MzWAN8AVgMtgHVS1jt7And/yN1nu/vsoUOHnmjYIiIJoX0OTDwnmCAHT88HrjCzS4BMIMfMHnf3G9oLuHs1cAuAmRmwM/zTDxjV4bkKgP0BxioiklCOLBOTG5+TLCHAFoy7/8DdC9y9ELgW+HPH5AJgZgPMLD1891ZgaTjpLAcmmNnY8PlrgeeDilVEJNGUVDWQlmIM6R+/CSbq0z/N7A4Ad38AmAz81sxagY3AV8PnWszs68CrQCrwsLtviHasIiLxqqS6gbzsDFJSOutRiA9RSTDuvgRYEr79QIfj7wITunjMy8DLUQhPRCThxPNWye00k19EJAHF+zIxoAQjIpKQ4nmr5HZKMCIiCSbet0pupwQjIpJgEmGSJSjBiIgknHjfKrmdEoyISIKJ962S2ynBiIgkmERYJgaUYEREEk5pdQM5mWn0TU+NdSjdUoIREUkwJVXxvdFYOyUYEZEEE+9bJbdTghERSTDxvlVyOyUYEZEEkghbJbdTghERSSAVtfG/VXI7JRgRkQRSmiBDlEEJRkQkoSTKHBhQghERSSiJsFVyOyUYEZEEkghbJbdTghERSSCJsFVyOyUYEZEEkghbJbdTghERSSDFVQ0MV4IREZFIcndKqhrIz+kb61B6RAlGRCRB1DS2UN/USn4CjCADJRgRkYTxl43G1IIREZEIak8w6oMREZGIOtKCSYBZ/KAEIyKSMIrDCSYvR30wIiISQSXVDQzJSicjLb63Sm6nBCMikiBKqg4nxEZj7ZRgREQSRCJNsoQoJBgzSzWz1Wb2Yifncs3sBTNba2YbzOyWDud2mdl6M1tjZiuCjlNEJN6VVjeQn0AJJi0Kr3EXsAnI6eTcncBGd7/czIYCH5nZ79y9KXz+fHeviEKMIiJxraG5lYP1zQkzggwCbsGYWQFwKfDrLoo4kG1mBmQBlUBLkDGJiCSiRJtkCcFfIrsX+B7Q1sX5+4DJwH5gPXCXu7eXdeA1M1tpZrcHHKeISFxr38lSfTCAmV0GlLn7ym6KXQSsAUYAM4D7zKz9Utp8d58FXAzcaWbndPE6t5vZCjNbUV5eHsEaiIjEj/YWjEaRhcwHrjCzXcCTwAVm9vhRZW4BnvGQbcBOYBKAu+8P/1sGPAvM7exF3P0hd5/t7rOHDh0aTE1ERGKs+MglMiUY3P0H7l7g7oXAtcCf3f2Go4oVAQsAzGwYMBHYYWb9zSw7fLw/cCHwYVCxiojEu9LqBrIz08jKiMbYrMiIeqRmdgeAuz8A/Ah41MzWAwZ8390rzGwc8Gyo75804Al3fyXasYqIxIviqsMJNYIMopRg3H0JsCR8+4EOx/cTap0cXX4HMD0asYmIJIKSqsSaAwOayS8ikhBKqhNrFj8owYiIxL3m1jbKahoT7hKZEoyISJwrr2nEPbEmWYISjIhI3EvESZagBCMiEvcScZIlKMGIiMS99kmWasGIiEhElVY3kJGWwoB+fWIdyqeiBCMiEueKw3NgwpPPE4YSjIhInCutaki4IcqgBCMiEveKqw8nXP8LKMGIiMQ1d6e0qpFhSjAiIhJJlXVNNLW2MVyXyEREJJKKE3Cr5HZKMCIicawkATcaa6cEIyISxxJ1mRhQghERiWslVQ2kphhDsjJiHcqnpgQjIhLHiqsayMvOIDUlsSZZghKMiEhcK61OvJ0s2ynBiIjEsf2HEnOSJSjBiIjErdY2Z+/Bw4wa1C/WoRwXJRgRkThVUt1AU2sbYwb1j3Uox0UJRkQkThUdqAdgtFowIiISSXsqQwlmzGAlGBERiaDdlXWkpZg6+UVEJLKKKg8zcmBf0lIT8091YkYtItILFB2oS9j+F1CCERGJW0WV9Qk7RBmUYERE4lJ1QzMH65sZowQjIiKRlOhDlEEJRkQkLrUPUR6doEOUIQoJxsxSzWy1mb3YyblcM3vBzNaa2QYzu6XDuUVm9pGZbTOzvw06ThGReLI7nGDUB9O9u4BNXZy7E9jo7tOB84B/N7N0M0sF7gcuBqYAXzKzKVGIVUQkLhRV1jOwXx9yMvvEOpTj1qMEY2b9zSwlfPtkM7vCzI5ZazMrAC4Fft1FEQeyzcyALKASaAHmAtvcfYe7NwFPAlf2JFYRkWRQdKCe0YMTcw2ydj1twSwFMs1sJPAGcAvwaA8edy/wPaCti/P3AZOB/cB64C53bwNGAns6lNsbPvYJZna7ma0wsxXl5eU9CElEJP4VVdYndAc/9DzBmLvXA9cA/+nuVxO6dNX1A8wuA8rcfWU3xS4C1gAjgBnAfWaWA3S2dZt39gTu/pC7z3b32UOHDu1BVURE4ltLaxv7Dh1O6CHK8CkSjJnNA64HXgofSzvGY+YDV5jZLkKXuC4ws8ePKnML8IyHbAN2ApMItVhGdShXQKiVIyKS9PYfaqC1zXtNC+ZbwA+AZ919g5mNAxZ39wB3/4G7F7h7IXAt8Gd3v+GoYkXAAgAzGwZMBHYAy4EJZjbWzNLDj3++h7GKiCS0oiQYogzHboUA4O5vAm8ChDv7K9z9m8fzgmZ2R/g5HwB+BDxqZusJXRb7vrtXhMt9HXgVSAUedvcNx/N6IiKJZndlHZDYkyyhhwnGzJ4A7gBagZVArpnd4+539+Tx7r4EWBK+/UCH4/uBC7t4zMvAyz15fhGRZFJUWU96agrDchJzmf52Pb1ENsXdq4GrCP3RHw3cGFhUIiK9WNGBegoG9SU1pbPxTomjpwmmT3jey1XAc+7eTBejukRE5MQkwxBl6HmCeRDYBfQHlprZGKA6qKBERHord6foQH3CD1GGnnfy/xz4eYdDu83s/GBCEhHpvQ7VN1PT2JLQa5C16+lSMblmdk/7jHkz+3dCrRkREYmgI0OUe0uCAR4GaoAvhH+qgUeCCkpEpLdqX0V5TIKvQwY9vEQGjHf3z3a4/09mtiaIgEREerM9R5bp7xvjSE5cT1swh83srPY7ZjYfOBxMSCIivdfuA3UMzc6gX3pPv//Hr57W4A7gt2aWG75/ELg5mJBERHqvZBmiDD1swbj72vCmYKcCp7r7TOCCQCMTEemFkmWIMnzKHS3dvTo8ox/gOwHEIyLSa9U3tbC/qiEpOvjhxLZMTuw1DERE4syO8tAilycPy4pxJJFxIglGS8WIiETQ1rIaAE7KS44E020nv5nV0HkiMSDxx9CJiMSRbWW1pKVY0lwi6zbBuHt2tAIREenttpbWMmZwP9LTTuTiUvxIjlqIiCSBbeW1TMhLnu/1SjAiInGgqaWN3Qfqk6b/BZRgRETiwq4DdbS2OROSZAQZKMGIiMSFraW1AIwfqgQjIiIRtK2sFjMlGBERibCtZTUUDOxL3/TUWIcSMUowIiJxYFtZco0gAyUYEZGYa21zdlTUMSGJRpCBEoyISMztqaynqaWN8UowIiISSVvLQiPI1IIREZGIal/kUi0YERGJqG1lteTnZJKT2SfWoUSUEoyISIxtK6tNqiVi2inBiIjEkLsrwYiISOTtr2qgvqk1KRNMt/vBRIKZpQIrgH3uftlR574LXN8hlsnAUHevNLNdQA3QCrS4++ygYxURibZtSTqCDKKQYIC7gE1AztEn3P1u4G4AM7sc+La7V3Yocr67V0QhRhGRmNhaGhpBNmFYcs3ih4AvkZlZAXAp8OseFP8S8Psg4xERiTfby2sZ1D+dQf3TYx1KxAXdB3Mv8D2grbtCZtYPWAT8ocNhB14zs5Vmdns3j73dzFaY2Yry8vJIxCwiEjVbS5Ozgx8CTDBmdhlQ5u4re1D8cuCdoy6PzXf3WcDFwJ1mdk5nD3T3h9x9trvPHjp06IkHLiISJe7O1iQdQQbBtmDmA1eEO+ufBC4ws8e7KHstR10ec/f94X/LgGeBucGFKiISfSXVDVQdbuZkJZhPx91/4O4F7l5IKIH82d1vOLqcmeUC5wLPdTjW38yy228DFwIfBhWriEgsrC46BMDM0QNjHEkwojGK7GPM7A4Ad38gfOhq4DV3r+tQbBjwrJlBKMYn3P2VqAYqIhKw1UUHSU9LYfLwTwyyTQpRSTDuvgRYEr79wFHnHgUePerYDmB6NGITEYmV1UWHmDYyl/S05Jzznpy1EhGJc00tbazbV8XMUQNiHUpglGBERGJgU3E1TS1tzBqTnP0voAQjIhITq4sOAjBztFowIiISQav3HCI/J5PhuX1jHUpglGBERGJgVdHBpG69gBKMiEjUldc0sqfysBKMiIhE1po9oQmWs5J0gmU7JRgRkShbXXSQtBRj6sjcWIcSKCUYEZEoW1V0kCkjcsjskxrrUAKlBCMiEkUtrW2s25vcEyzbKcGIiETRltJa6ptak3aBy46UYEREomj1nuSfYNlOCUZEJIpW7T7E4P7pjB7UL9ahBE4JRkQkilbvCU2wDG9HktSUYEREouRQfRM7yut6Rf8LKMGIiETNsu0HAJhTOCjGkUSHEoyISJQs3lxGTmYas3pBBz8owYiIREVbm7NkSznnnDyUtNTe8ae3d9RSRCTGNhZXU17TyPkT82IdStQowYiIRMHizWUAnDtxaIwjiR4lGBGRKPjzR2VML8hlSFZGrEOJGiUYEZGAVdY1sWbPIc7rRZfHQAlGEoy7xzoEkU9t6ZZy3OGCSb0rwaTFOgCRnvrj+mL+4bkNpKbApPwcJg3PZnJ+DmeOH0xeTmaswxPp0uKPyhjcP51pSb7/y9GUYCTu1TW28E8vbOC/V+xl2shcJgzLYnNxDe9uP0BTaxtpKcaFpwzj+tPHcOb4wb1iCQ5JHK1tzptbyrlgUh4pKb3rs6kEI3FtzZ5DfOvJ1eyurOfO88fzrYUn0yc8h6C5tY2tpbU8u3ovT6/cy8vrSxg3pD/nThzKsJxMhmZlkJeTwcT8bPKy1cKR2Fiz5xCH6pt71fDkdkowEpcamlv52RtbeWjpDvJzMnnytjM4fdzgj5Xpk5rClBE5TBkxhb++cCIvry/m9x8U8dTyPdQ3tR4pl5piXDw1n1vmFzJr9MAjLZy2NmfXgTpKqhoYn5dFXnaGWj8ScUs+KiPF4JwJvWd4cjslGIk7H+ys5G//sI4dFXV8YXYBf3/pFHL79un2MZl9UrlmVgHXzCoAoLaxhbLqBspqGnljUylPLt/Di+uKmTYylzPHD2ZjcTVr9xyiuqHlyHMM7NeHifnZTB2RyzWzCpgyIifQekrvsPijMk4bM5Dcft1/hpORJdOonNmzZ/uKFStiHYYcB3dna1ktjy3bxe/eL6JgYF9+cs2pnDVhSESev66xhWdW7+PRd3ay60A9k/KzmT5qADMKBjB8QCbby2r5qLSGzSU1bNhfTVNLG6eNGchN88awaGo+GWnJvXe6BKOsuoG5//IG31s0ka+dd1Ksw+mUma1099lBPLdaMBIzbW3Oit0HeW1DCa9vKmX3gXrM4Cvzx/I3F51Mv/TIfTz7Z6Rx4xljuOH00bS0+ZF+nHZnd7h8UVXfzNMr9/C794u468k1DMlK55sLJnDd3NG9Zg0piYynV+4F4DOTh8U4ktgIvAVjZqnACmCfu1921LnvAteH76YBk4Gh7l5pZouAnwGpwK/d/SfHei21YBJDY0srz63Zz0NLd7CtrJb01BTmjR/MZ6YMY+HkYeTnxkeHfFub8872Cn6xeDvv7jjApPxs/vHyKZw5PjKtKkluDc2tnPXTxUwZkcNvvzI31uF0KdFbMHcBm4BPXNB297uBuwHM7HLg2+HkkgrcD3wG2AssN7Pn3X1jFOKVgByobeTplXt55J2dlFY3Mnl4Dvd8YToXnpJPVkb8NaZTUoyzJwzlrJOG8OqGEv75pU1c96v3uWRaPjfNK+S0MQM/0RISafeHVXupqG3kjnPHxTqUmAn0t9rMCoBLgR8D3zlG8S8Bvw/fngtsc/cd4ed5ErgSUIKJM3WNLZTXNFJW00hZTQNtDsOyM8jPzWRYTiaVdU28uqGEVz4sYfmuStoc5p80mLs/N52zJwxJiFFbZsaiqcM5b2IeDy3dwS+XbOfl9SXkZKZxzslDWTA5j3MmDGVwJ2tMbS6p5plV+6hvamHmqIHMGD2AsYP797r5EL1Na5vzq6U7OLUgl3lHjX7sTYL+2ngv8D0gu7tCZtYPWAR8PXxoJLCnQ5G9wOldPPZ24HaA0aNHn2C40lNV9c3c+tvlLN91sEflJw7L5usXTOCSaflMyk/M0VmZfVL55oIJ3DK/kHe2VfDGpjIWf1TGi+uKMYOpI3I5e8IQzjppCNvKa3l6xV7W76uiT6qRkZbK4+8VAZDbtw/zxg3mi3NHcc6EoaR2SDb1TS38aVMZO8pruXz6CMYPzYpVdeUEvLqhhF0H6vnF9bMS4ktUUAJLMGZ2GVDm7ivN7LxjFL8ceMfdK9sf3kmZTjuL3P0h4CEI9cEcZ7jyKVTVN3Pjw++zubiGuxZMYPSgfuTlZJCXnYkZlFY3UFrdSGl1A31SjYWThzEuif5QZmf2YdHU4SyaOpy2Nmf9viqWbiln6dZyHly6g18s2Q7AlOE5/J/Lp3DljJHk9u3D9vJa1hQdYlXRQV7fWMorG0oYOaAvX5g9ion5Wby0voQ/bSzlcHNoDs+9f9rK2ROGcPO8Qs6flPexRCTxy9158M3tFA7ux0Wn5Mc6nJgKrJPfzP4VuBFoATIJ9cE84+43dFL2WeBpd38ifH8e8EN3vyh8/wcA7v6v3b2mOvmDV3W4mRt/E0ouv7xhFgt66eiYrtQ0NLN8VyV52ZlM7WbdqaaWNl7fWMqTy4t4a2sFEJqHc8m04Vw+fQTjhvTnqeWhkWwl1Q2MGtSXW84cyxfnjKJ/lPurDtQ28tL6YraWhlpVcwoH9upv5ceybHsF1/3qfX589VSuP31MrMM5piA7+aMyDybcgvmbo0eRhc/lAjuBUe5eFz6WBmwBFgD7gOXAde6+obvXUYLpuTV7DvHD5zfwtfPGc2EX37K2ldXQ1OIMzc5gUP90ahtbuPE377OpuJoHbjhNySVCig7Us+/QYWYXfnLQQHNrKBE9/PZOVuw+SG7fPtxwxmhunlcY6AKf9U0tvLahlP9vzT7e2lpBa5uTnppCU2sbk4fncPO8MVw5YyR907ueH1RR28jAfum9ruV108MfsHF/NW9//3wy+8T//KmkSjBmdgeAuz8QPvdlYJG7X3vUYy4h1IeTCjzs7j8+1usowfRMSVUDV9z3NuW1jbjD7eeM47sXTTzyx62suoEfv7yJ59bsP/KYFAv1QTS3tim5xMjK3Qf59Vs7eGVDCX1SUrhixgi+Mn/sx1YcONzUyovr9vPMqn0cOtxMWoqRkmKkpRh9+6SSlZFGVmYaWRlpFAzsy6wxAzllRA4Zaam4O2v2HOK/V+zhhbXF1Da2MCI3kytmjOSqmSMYPagfz63Zz2PLdrG5pIbcvn24euZIrp076ki/mrvz9rYKHlq6g7e2VlA4uB9/dd54rp5ZQHpa8o+427i/mkt+/hbfvWgid54fnxMrj5bwCSZajjfBuDtNrW29YrZ2Q3MrX3jwXbaX1fLU/5rHU8v38F/v7WZO4UDuvXYmr20o4Z7XttDY0sb/OnccU4bnUF7bSHlNIwfqmrhs2nDOPEnzQGJpV0UdD7+zk6dX7OVwcyvzxg3m2rmjWF10iGdW7aW6oYVxQ/ozbmgWbe60tDmtbW0cbmqltrGF2oYWahpaqGkMLZOTnprC1JE51Da2sKW0lr59Urlk2nA+P7uAuYWDPjHizd35YGcl//Xebl7bUEpTaxszRg1gwaQ8XlpfzOaSGoZmZ/C50wp4e2sF6/dVMTw3k9vOHsd1p49OiG/1x+t7/7OWF9YW894PFiTM0jBKMD10PAnmcFMrC+95k2vnjOIbCyYEFFl8cHe++eQaXly3n1/dOJuFU0KtkOfX7ucHf1jH4eZW2hzOnjCE/3vlVMYO6R/jiKU7VfXN/H55EY8t20VxVWhAxaKpw7lu7mjOGDfomP0kZdUNrCo6yKqiQ6zafRAzuHpmAZdPH052Zs/+OFbWNfHs6n08+UERW8tqmZCXxW3njOPKGSOOtIre2lrBfYu38cHOSj4zZRgP3XhaUvbhHKxr4ox/fYPPnlbAv1w9Ldbh9JgSTA8dbwvmyvveBjOeu3N+AFHFj/sXb+PuVz/qtPm+vbyWe17fwqXThnPx1Pyk/AOQrJpb21ix6yAThmXFbL93d6e4qoHhuZldfnYeWrqdf3l5M/90xSncfGZhdAOMgvb6vfKtsxNqKH6QCSb5L4r2wILJw1i75xBlNQ2xDiUwr3xYzN2vfsSVM0bwtfPGf+L8+KFZ3H/dLC6ZNlzJJcH0CS+1E6vkAqHJqCMG9O32s3Pb2eO4YFIeP35pExv2V0UxuuC1tjn/9d5u5o4dlFDJJWhKMMCCyaGNgJZsLo9xJMFYvquSbz65hpmjB/DTz56qBCIxYWb8v89PZ2D/PnzjidXUNbYc+0EJ4s0tZeypPMzN8wpjHUpcUYIhNCFuRG4mf9pUGutQIm5raQ23PraCggF9+c3Nc5K6g1Xi36D+6fzs2pnsOlDHPzz3YazDiZjHlu1mWE4GF56i0ZUdKcEQ+mZ1weQ83tpaQUNz67EfkCBKqhq4+eEPSE9L4bGvzGVQ//RYhyTCGeMG840LJvDMqn38aukO2toSux94Z0Udb24p57q5Y7T46VH0vxG2YPIwDje38u6OA7EOJSKqG5r58iMfUHW4mUe+PIdRg/rFOiSRI765YEKoP+blTVzzy2Ws23so1iEdt8ff201aivGluaNiHUrcUYIJmzduMP3SU3kjCS6TFR2o5/O/fJdtZbU8cONp3S5ZIhILqSnGb26ezT1fmM7eg4e58v53+Ltn11NanVgDbeqbWvjvFXu4eNrwQFdWSFTxtwlHjGT2SeWsk4bw501l+JUe9x3hDc2tVDc0k5f98Q/1su0VfO13q3CHR26Z87GdGkXiiZlxzawCFk4Zxr2vb+Wxd3fxxPtFjBzQl2kjczl1VC5zCwcxa/TAuN3e4NnV+6hpaOGmefG/5lgsKMF0sHDyMF7bWMrG4mpOGRG/3/p3VtRx/a/eY39VA6cW5HLRKfksmprPO9sq+KcXNjJ2SH9+fdNsCjVRUhJATmYf/vHyKVx3+mgWby5j7d5DrN9XxSsbSgAYkZvJZdNHcMX0EZwyIiduvvw1trTyi8XbmT5qALPHDIx1OHFJCaaD8yeFhiu/saksbhPMRyU13PCb92ltc+5aMIE3t5Rz96sfcferHwGwYFIe9147o8czsUXixUl5WZyU95dtHQ7VN/HmlnKeW7Ofh9/eyUNLdzApP5tvLZzARafEfjLwU8v3sO/QYX7y2WkxjyVeaSb/Ua68/x1w57mvnxWhqCJn/d4qbnz4fTLSUvjdradzUl5oH7fiqsO8vrGU1BTj2jmje93qtZL8DtY18ccPS3j4nZ1sK6tl2shcvnvRxJjtitrQ3Mo5/7aYwiH9eer2MxI6wQQ5k18tmKMsnJTHv7++hbLqhph32rW2OQfrm6iobWRraS1/98x6cvv14Ylbz2D04L+MChue25ebNMFLktjA/ulcd/povjhnFM+u3sd/vL6Fmx7+gNljBnL59BEsmJxHwcDojZR8/L3dlNU08p9fmpnQySVoasEcpX257Z9+dhpfnBObLZgPN7Vy8yMfsCK8h327cUP68/itpzNiQN+YxCUSLxpbWnlq+R4eXbaLHeV1QGhb7otOGcYd542nX3pw353rGls4+98Wc8qIHP7rq53u5J5Q1IKJosnDsykY2JcnPtjD508bFZPRK//80kaW76rk1rPGUjCwH0OyMhiSlc60gtxAf3FEEkVGWio3zSvkpnmF7Kyo441NpbyxqYz/XLyNt7dV8PCX5zCgXzATix9dtovKuib++sKJgTx/MtE8mKOYGd9eeDJrwxsvRdvrG0v53ftF3H72OP7+0incfGYhl546nNPHDVZyEenE2CH9ufXscfz+9jP45fWz+HBfNV988L1A5tRUHW7mwTe3s3ByHjNGDYj48ycbJZhOXDNrJHMLB/GTVzZTWdcUtdctq27g+39YxykjcvjOhSdH7XVFksWiqcN59JY57D1Yz+ceWMauin7wdJsAAAr/SURBVLqIPXd1QzN/9+x6qhta+PZn9PvZE0ownTAzfnTVVGobWvjpHzdH5TXb2py/fnot9U0t/OzaGb1id02RIJx50hCeuO0Mahta+NwD7/Lw2zs/0Zppa3PW7jnEvX/awj2vfcTj7+3m9Y2lrNt7iNpOVnl+dUMJC//9Tf64vpjvfObkuJ3GEG90zaULE/Oz+epZY3lw6Q6+MKeA08YMCvT1Hlm2i7e2VvDPV009MvxYRI7P9FEDePqOM/n2U2v4vy9u5EcvbWRu4SAuPCWfnRW1vL6xlNLqRtq7WDsOpkkxmJifw6zRA5g5eiBvbCrljx+WMHl4Dr++eTanFujSWE9pFFk36hpbWHjPmwzol84LX59PdUMLL60v5oU1+6msb2Lh5GFcPDWfUwtyP9VQRXdnS2kty3dVsnxXJSt2HWTfocMsnJzHr26arWGPIhG0rayWF9ft54W1+9leXkffPqmce/JQLjxlGBdMyiMrI40DdU2UVDVQUt3ApuJqVu4+yJqiQ9Q0tpCelsK3Fk7gtrPHJeVqydoyuYcinWAA/ri+mL/63SomD89ha2kNLW3OhLwshuVk8t6OA7S0OSMH9OWCSXlMK8hlyvAcJgzL+tglrpbWNspqGlm2/QBvby3n7W0HqKhtBCAvO4M5hYOYXTiQz88eRVaGGpUiQXB39lQeJi8no0f7IrW2OdvKasnt24f83ORdyFIJpoeCSDDuzp1PrGLd3iouO3UEV84YwaT8bMyMQ/VN/GlTGa98WMyy7QeobwrtJZOWEto+9nBzK7UNLRzusMfMkKx05p80hPknDeGMsYMZNaj7bWZFRIKkBNNDQSSYnmprc3ZX1rNxfzWbiqspqqynf0YqWRlpZGX0YUC/PswpHMSk/Oy4XRlWRHofTbRMACkpxtgh/Rk7pD+Xnjo81uGIiMRc8vVYiYhIXFCCERGRQCjBiIhIIJRgREQkEEowIiISCCUYEREJhBKMiIgEQglGREQCkVQz+c2sHNh91OFcoOoYxzreP9btIUDFCYTZWTw9LfNp63L0/fbbyVSXjrdPpD4nUpeuzulz9pdjem96FuuxygTx3kx092CWcHf3pP4BHjrWsY73j3UbWBHpeHpa5tPWpZs6JE1dIlWfE6mLPmfdf8703iTve3Osn95wieyFHhx74VPejnQ8PS3zaety9P0XuihzvOKhLj2N41hOpC5dndPnLDL03nR/PJbvTbeS6hJZNJjZCg9oYbhoS6a6QHLVJ5nqAslVn2SqCwRbn97Qgom0h2IdQAQlU10gueqTTHWB5KpPMtUFAqyPWjAiIhIItWBERCQQSjAiIhKIXp1gzOxhMyszsw+P47Gnmdl6M9tmZj+3Dvsem9kXzGyjmW0wsyciG3WX8US8Lmb2ZTMrN7M14Z9bIx95lzEF8t6Ez3/OzNzMotJRG9B7c0f4+Boze9vMpkQ+8k7jCaIu3wn/vqwzszfMbEzkI+8ypiDqc46ZrTKzFjP7XOSj/kQcx12HLp7vZjPbGv65ucPxsWb2fvj4U2aWfswnC2r8cyL8AOcAs4APj+OxHwDzAAP+CFwcPj4BWA0MDN/PS+C6fBm4L1nem/C5bGAp8B4wO1HrAuR0KHMF8EoC1+V8oF/49l8BTyXy5wwoBE4Ffgt8Ll7rACwBCo86NgjYEf53YPh2+9+y/wauDd9+APirY71Gr27BuPtSoLLjMTMbb2avmNlKM3vLzCYd/TgzG07oF/xdD/1v/xa4Knz6NuB+dz8Yfo2yYGsRElBdYibA+vwI+DegIcDwPyaIurh7dYei/YGojNYJqC6L3b0+XPQ9oCDYWvxFQPXZ5e7rgLYoVOG469CFi4DX3b0y/DfsdWBRuHV2AfA/4XKP0YO/E706wXThIeAb7n4a8DfALzopMxLY2+H+3vAxgJOBk83sHTN7z8wWBRpt9060LgCfDV+6+B8zGxVcqD1yQvUxs5nAKHd/MehAe+CE3xszu9PMthNKmN8MMNZjicTnrN1XCbUGYimS9YmVntShMyOBPR3ut9drMHDI3VuOOt6ttB6H2wuYWRZwJvB0h8v2GZ0V7eRY+zfINEKXyc4j9E3sLTOb6u6HIhtt9yJUlxeA37t7o5ndQehbywWRjrUnTrQ+ZpYC/Aehy34xFaH3Bne/H7jfzK4D/jdwcyflAxWpuoSf6wZgNnBuJGP8NCJZn1jprg5mdgtwV/jYScDLZtYE7HT3q+m6XsdVXyWYj0shlKVndDxoZqnAyvDd54Ff8vFmfAGwP3x7L/CeuzcDO83sI0IJZ3mQgXfihOvi7gc6HP8V8NPAoj22E61PNjAVWBL+pcsHnjezK9x9RcCxHy0Sn7OOngyXjYWI1MXMFgJ/D5zr7o2BRty9SL83sdBpHQDc/RHgEQAzWwJ82d13dSiyl9CX43YFhPpqKoABZpYWbsX0rL5Bd0DF+w+hDrkPO9xfBnw+fNuA6V08bjlwBn/p4LskfHwR8Fj49hBCzc3BCVqX4R3KXE0ocSbse3NUmSVEqZM/oPdmQocylxPggoVRqMtMYHvHOiXD5wx4lCh08h9vHei6k38noQ7+geHbg8LnnubjnfxfO2ZcsXhD4+UH+D1QDDQTytxfBcYCrwBrgY3AP3bx2NnAh+FfjPv4y6oIBtwTfuz69jckQevyr8CG8OMXA5MS+b05qswSojeKLIj35mfh92ZN+L05JYHr8iegNFyXNcDzifw5A+aEn6sOOABsiMc60EmCCR//CrAt/HNLh+PjCI2c20Yo2WQcKzYtFSMiIoHQKDIREQmEEoyIiARCCUZERAKhBCMiIoFQghERkUAowUhSM7PaKL/esgg9z3lmVmVmq81ss5n9vx485iqL0qrKIj2hBCPyKZhZt6tfuPuZEXy5t9x9JqGJiJeZ2fxjlL8KUIKRuKGlYqTXMbPxwP3AUKAeuM3dN5vZ5YTW9EonNEHuencvNbMfAiMIzZauMLMtwGhCE89GA/e6+8/Dz13r7llmdh7wQ0JLbEwltMzIDe7uZnYJocm4FcAqYJy7X9ZVvO5+2MzW8JdFO28Dbg/HuQ24EZhBaNn+c83sfwOfDT/8E/U8gf86kU9FLRjpjbpaafZt4Ixwq+FJ4HsdHnMacKW7Xxe+P4nQ0uZzgf9jZn06eZ2ZwLcItSrGAfPNLBN4kNDeIWcR+uPfLTMbSGg9u6XhQ8+4+xx3nw5sAr7q7ssIrZH1XXef4e7bu6mnSFSoBSO9yjFWyy0Angrv9ZFOaB2mds+7++EO91/y0KKMjWZWBgzj48u3A3zg7nvDr7uGUAuoFtjh7u3P/XtCrZHOnG1m64CJwE/cvSR8fKqZ/TMwAMgCXv2U9RSJCiUY6W26XGkW+E/gHnd/vsMlrnZ1R5XtuOJvK53/LnVWprNlz7vylrtfZmYnA2+b2bPuvobQIopXuftaM/syH1/9tl139RSJCl0ik17FQztB7jSzzwNYyPTw6VxgX/h2UHurbAbGmVlh+P4Xj/UAd99CaOHR74cPZQPF4cty13coWhM+d6x6ikSFEowku35mtrfDz3cI/VH+qpmtJbQi8ZXhsj8kdEnpLUId8BEXvsz2NeAVM3ub0CrCVT146APAOWY2FvgH4H1C29l27LR/EvhueGjzeLqup0hUaDVlkSgzsyx3rw3vc34/sNXd/yPWcYlEmlowItF3W7jTfwOhy3IPxjgekUCoBSMiIoFQC0ZERAKhBCMiIoFQghERkUAowYiISCCUYEREJBD/P7/7mc9ITbRiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we see that if we choose the learning rate too big, the loss function starts to increase. We want to avoid this at all costs. So we want to find the spot where the loss function decreases the steepest with the largest learning rate. In this case, a good value is `1e-2`. The first parameter determines how many epochs we train. One epoch corresponds to one pass through the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.145812</td>\n",
       "      <td>4.027751</td>\n",
       "      <td>0.294872</td>\n",
       "      <td>24:40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep learning models learn more and more abstractions with each layer. The first layers of an image model might learn about edges and textures in an image, and as you progress through the layers, you can see how the model combines edges to eyes or ears and eventually combines these to faces. Therefore the last few layers are usually the ones that are very task-specific while the others contain general information.\n",
    "\n",
    "For this reason, we usually start by just tuning the last few layers because we don't want to lose that information and then, in the end, fine-tune the whole model. This is what we did above: we just trained the last few layers. Now to get the best possible performance, we want to train the whole model. The `unfreeze()` function enables the training of the whole model. We train the model for 10 more epochs with a slightly lower learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.883628</td>\n",
       "      <td>3.846353</td>\n",
       "      <td>0.312791</td>\n",
       "      <td>25:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.830023</td>\n",
       "      <td>3.806326</td>\n",
       "      <td>0.319647</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.823724</td>\n",
       "      <td>3.776663</td>\n",
       "      <td>0.323472</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.790490</td>\n",
       "      <td>3.747104</td>\n",
       "      <td>0.327012</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.708543</td>\n",
       "      <td>3.720774</td>\n",
       "      <td>0.330030</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.685525</td>\n",
       "      <td>3.700337</td>\n",
       "      <td>0.332860</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.594311</td>\n",
       "      <td>3.683683</td>\n",
       "      <td>0.334876</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.559871</td>\n",
       "      <td>3.672719</td>\n",
       "      <td>0.336452</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.545992</td>\n",
       "      <td>3.668409</td>\n",
       "      <td>0.337152</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.494757</td>\n",
       "      <td>3.668425</td>\n",
       "      <td>0.337193</td>\n",
       "      <td>25:14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save language model\n",
    "Since the step above took about 4h we want to save the progress, so we don't have to repeat the step when we restart the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# uncomment if you want to fine-tune the language mode\n",
    "# data_lm.path = Path('')\n",
    "# data_lm.save('data_lm.pkl')\n",
    "# learn.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_dataset(\"fine_tuned.pth\")\n",
    "get_dataset(\"data_lm.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(Path(\"../data/\"), 'data_lm.pkl', bs=bs)\n",
    "data_lm.path = path\n",
    "\n",
    "learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3, model_dir=\"../data/\")\n",
    "learn.path = Path(\"\")\n",
    "learn.load('fine_tuned');\n",
    "learn.path = path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text generation\n",
    "The objective of a language model is to predict the next word based on a sequence of words. We can use the trained model to generate some movie reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = \"I liked this movie because\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I liked this movie because it had a nice twist , i liked it . i like all of the characters in the movie , the same in each one . Displayed refreshing to see . The story and acting were really good\n",
      "I liked this movie because it was about a person playing far , the same way . It had some good moments of humor , but it really did n't make an epic film . i really did n't understand why the audience was\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "Generate a few movie reviews with different input texts. Post the funniest review on the Teams channel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder\n",
    "As mentioned previously, the last layers of a deep learning model are usually the most task-specific. In the case of language modeling, the last layer predicts the next word in a sequence. We want to do text classification, however, so we don't need that layer. Therefore, we discard the last layer and only save what is called the encoder. In the next step, we add a new layer on top of the encoder for text classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train classifier\n",
    "In this section, we will use the fine-tuned language model and build a text classifier on top of it.  The procedure is very similar to the language model fine-tuning but needs some minor adjustments for text classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess data for classification\n",
    "Preprocessing the data follows similar steps as for language modeling. The main differences are that 1) we don't want a random train/valid split, but the official one and 2) we want to label each element with its sentiment based on the folder name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)\n",
    "             #grab all the text files in path\n",
    "             .split_by_folder(valid='test')\n",
    "             #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)\n",
    "             .label_from_folder(classes=['neg', 'pos'])\n",
    "             #label them all with their folders\n",
    "             .databunch(bs=bs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we display a batch we see that the tokens look the same with the addition of a label column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj by now you 've probably heard a bit about the new xxmaj disney dub of xxmaj miyazaki 's classic film , xxmaj laputa : xxmaj castle xxmaj in xxmaj the xxmaj sky . xxmaj during late summer of 1998 , xxmaj disney released \" xxmaj kiki 's xxmaj delivery xxmaj service \" on video which included a preview of the xxmaj laputa dub saying it was due out</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj some have praised xxunk xxmaj lost xxmaj xxunk as a xxmaj disney adventure for adults . i do n't think so -- at least not for thinking adults . \\n \\n  xxmaj this script suggests a beginning as a live - action movie , that struck someone as the type of crap you can not sell to adults anymore . xxmaj the \" crack staff \" of</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj by 1987 xxmaj hong xxmaj kong had given the world such films as xxmaj sammo xxmaj hung 's ` xxmaj encounters of the xxmaj spooky xxmaj kind ' xxmaj chow xxmaj yun xxmaj fat in xxmaj john xxmaj woo 's iconic ` a xxmaj better xxmaj tomorrow ' , ` xxmaj zu xxmaj warriors ' and the classic ` xxmaj mr xxmaj vampire ' . xxmaj jackie xxmaj</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj to be a xxmaj buster xxmaj keaton fan is to have your heart broken on a regular basis . xxmaj most of us first encounter xxmaj keaton in one of the brilliant feature films from his great period of independent production : ' xxmaj the xxmaj general ' , ' xxmaj the xxmaj navigator ' , ' xxmaj sherlock xxmaj jnr ' . xxmaj we recognise him as</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load model\n",
    "We create a text classifier model and load the pretrained encoder part from the fine-tuned language model into it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5, model_dir=\"../data/\")\n",
    "learn.load_encoder('fine_tuned_enc');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find the learning rate\n",
    "Again, we need to find the best learning rate for training the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='0' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      0.00% [0/1 00:00<00:00]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='94' class='' max='520', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      18.08% [94/520 00:41<03:06 1.9385]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deVyVdfr/8dcFiMoiiCIquOCCSqampLlk2uLStK82tliWY3sz1VTznaXf7E2zNJWTWWPbVLZZ2dRokzNqaiW4g1uoqEgqqKAIynb9/jhH50QHAT039wGu5+NxHp5z35/POe8bhIv7/tz35xZVxRhjjKkuxO0AxhhjgpMVCGOMMX5ZgTDGGOOXFQhjjDF+WYEwxhjjV5jbAQKpffv22r17d7djGGNMo7Fy5coCVY33t65JFYju3buTkZHhdgxjjGk0RGRHTevsEJMxxhi/HC0QIjJBRDaLSLaIPOpnfYyIfCQia0UkS0Ru9VmXIyLrRWSNiNhugTHGNDDHDjGJSCgwA7gIyAXSRWSeqm7waXY3sEFVLxWReGCziLyuqmXe9WNVtcCpjMYYY2rm5B7EUCBbVbd5f+HPAS6v1kaBaBERIAo4AFQ4mMkYY0wdOVkgEoFdPq9zvct8PQv0A/KA9cD9qlrlXafApyKyUkSmOZjTGGOMH04WCPGzrPrMgOOBNUBnYBDwrIi08a4bqaqDgYnA3SIy2u+HiEwTkQwRycjPzw9QdGOMMU4WiFygi8/rJDx7Cr5uBeaqRzawHegLoKp53n/3Ae/jOWT1Hao6S1XTVDUtPt7vqbzGGGNOgZMFIh3oLSLJIhIOTALmVWuzE7gAQEQSgD7ANhGJFJFo7/JIYByQ6WBWU01WXhELsva4HcMY4yLHzmJS1QoRuQdYAIQCs1U1S0Sme9fPBH4FvCwi6/EcknpEVQtEpAfwvmfsmjDgDVWd71RW821fbtvPbS+nU1JWyV+uH8iVZyW5HckY4wJHr6RW1U+AT6otm+nzPA/P3kH1ftuAgU5mM/4tyy5g6ivpJMa2pl1USx55dz2dY1ozrEc7t6MZYxqYXUl9CprqXfgWb8nntpfT6RYXyZxpw3nhpjS6xLVm2msr2ZpfDMCxikpeWZ7DuX/4DzfM+pKsvCKXUxtjnCJN6ZddWlqaOjkXU2WV8sT8TbyyPIcucRGkdmpDv05tGN6zHYO6xDr2uU6pqKwiO7+YdbuKWJtbyDsrc+kZH8Xrtw8jLjIcgJ37S7jyb8uIbBnG1FHJPL94K3lFRxncNZbtBUcoKi1n0tCuPHhRCu2iWrq8RcaY+hKRlaqa5nedFQgoKilnTW4ha3YWsmbXQbbsLeai1ATuu6D3iV+UxccquP/N1SzctI+J/TtSXqls/OYQuwtLAfjB6B48NL4PLUKDd6esskrZkHeIZVsLWJZdQEbOQUrLKwGIbhnG8J7t+MM1A4iNCP9Wv9U7DzJp1pccq6hiUJdYHhyXwqhe7TlUWsFTC7fw6hc7iAwP5bdXncklAzq7sWnGmFNkBeIkyiqq6P+LBZRVViECveKj6BIXwaLN+4hsGcY9Y3txYWoCd7++iq/3FfP4ZWdw0zndTvQvLCnjj59u5h9f7uSsrrE8c8NZJLWNCPSmnbJDR8tZvDmfhRv3smhLPoUl5QD07hDFiJ7tGNQ1lgFJsSS3iyQkxN+lKx6rdh6k+GgF5/Zuj/fkgRO+3nuYR95bx6qdhTw8vg93jen5nTbGmOBkBaIWb6XvJKltBAOSYohu1QLw/NL7/b82sXDTPgCiW4Xxt8mDObe3/2st/rkuj8feW48I/L/Lz+DSAZ0Jc3Fvoqi0nAffXsuizfuoqFLiIsMZ26cDo1PaM7xnOzpEtwro5x0tr+SR99bx4Zo8rhmSxG+vPJPwsODdmzLGeFiBOA3Ltxbw3srd3DmmJ706RJ207Y79R7j3zdWsyy0iMbY1t47sznVnd6GNt+g0lKLScm7++1ds+OYQt41KZlxqAoO6tCX0JHsIgaCqPPXZ1/x14dec0yOOp284K+CFyBgTWFYgGlBllbJw415eXLqdFdsPEBkeyqMT+3LT8O516r81v5jSskpEIESE6FZhJMa2rvMhm0NHy7np7yvYkFfEc5OHcGFqwmlszan5YPVufvzeOlqFhfDYxf24Pq3LSQ9fGWPcYwXCJZm7i3hi/iaWZhfw4s1pXNDv5L+sn1+8ld/9a9N3lie1bc25veM5L6U9I3u1P3EYrLrD3uKQlVfE3yYP4SIXisNxW/OL+cnc9Xy1/QBDu8fx26v606tDtGt5jDH+WYFwUWlZJdc+v5ycghI+uHtEjb8k/7tpH7e9ks741I5cPSSJyipFVckvPsbnXxfwxdb9FB+rIKplGHeO6cnUUcm0ahEKeA7tLNqczxPzN5G9r5gZkwcz/oyODbmZfqkq72Tk8ptPNlJaXslzkwfXWiSNMQ3LCoTLdheWcvmzS4lu1YIP7hpJTMS39wC25hdzxYxldI2L4N3pI2gdHvqd9yivrGLVjoO88Pl2Ptu4l8TY1jw8vg+xES146rOvWbOrkKS2rfnV5f0Z27dDQ21aneQfPsbUV9LZ+M0h1w57GWP8swIRBNJzDvD9F75keM/2vDTl7BMDxkWl5Vw5YxlFpeXMu3cUibGta32v5VsL+M3HG8nKOwRAYmxr7j2/F1cPSQra6zB8B859D399vfcwc9J3oQqPTuxrZz4Z08CsQASJN77ayU/eX09sRAuS20fSvV0kOw+UsHZXIW/ccQ5Dk+Pq/F5VVconmd9wtLyKywZ2bhS/WItKy7l5tmcA/c4xvViWXcDKHQdpESqUVyqjU+KZeeNgIsIdnSLMGOPDCkQQ+WD1br7afoAd+4+QU3CEgyXl/OLSVCYN7ep2tAZx/CyrtbsK6dE+kklDu3D14CT+vWEvP3l/PQOSYnlpytm0jQyv/c2MMafNCkQQU9Vmd9VxSVkFW/cdoX9im29t+4KsPdz75mq6xkUw+5az6doueK5IN6apsgJhGo0vt+3njlcyKC6rYHDXtow/I4FxqR0pKaskPecAK3IOkLm7iAFJsVyXlsTInu3tGgtjToMVCNOo7DpQwvurd7Mga8+JgfjjOse0IrVzDOk5BygqLScxtjXXpiVxy/DudljKmFNgBcI0WrsOlPDfzfuIahnG0OS4ExMhHi2v5N8b9vJ2xi6WZhcQFR7GtNE9uG1UMpEtbZDbmLqyAmGatC17D/PHBZv5dMNe2keFc8/YXkw+p1vQnvJrTDA5WYGwnyDT6KUkRDPr5jTm3jWCXh2iePyjDVz6zFLScw64Hc2YRs0KhGkyBndty5t3nMPMG4dwqLSca2d+wYNvr6Wg+Jjb0YxplKxAmCZFRJjQvyOfPXged47pyby1u7nwz4uZn/mN29GMaXSsQJgmKSI8jEcm9OVf959Ll7YRTP/HKh6bu46Ssgq3oxnTaFiBME1arw7RvHfnCKaf15M56bu45OmlZOQcoCmdnGGMU6xAmCYvPCyERyf25fXbh1FSVsk1M79g/FNLmLl4K3uKjrodz5ig5WiBEJEJIrJZRLJF5FE/62NE5CMRWSsiWSJya137GlNfI3q259MfjeY3V/YnulULfv+vTQz//UJ+/c8NtkdhjB+OXVEkIqHADOAiIBdIF5F5qrrBp9ndwAZVvVRE4oHNIvI6UFmHvsbUW5tWLZg8rBuTh3Ujp+AIf1uUzYtLt9O9fSQ3ntPN7XjGBBUn9yCGAtmquk1Vy4A5wOXV2igQLZ4Z26KAA0BFHfsac1q6t4/kd1cN4Py+HXh8XhYrttt1E8b4crJAJAK7fF7nepf5ehboB+QB64H7VbWqjn0BEJFpIpIhIhn5+fmBym6aidAQ4alJg+gaF8Fdr68kr7DU7UjGBA0nC4S/KTarH+gdD6wBOgODgGdFpE0d+3oWqs5S1TRVTYuPjz+dvKaZatOqBbNuTuNYeRXTXsvgaHml25GMCQpOFohcoIvP6yQ8ewq+bgXmqkc2sB3oW8e+xgRMrw5RPDVpEFl5h/jhW2uorLJBa2OcLBDpQG8RSRaRcGASMK9am53ABQAikgD0AbbVsa8xAXVBvwR++r1U/pW5h59+sN7ObDLNnmNnMalqhYjcAywAQoHZqpolItO962cCvwJeFpH1eA4rPaKqBQD++jqV1Zjjpo5K5sCRY8z471baRoTz4wl93Y5kjGscnThfVT8BPqm2bKbP8zxgXF37GtMQHhrXhwNHyvnbIk+RuGN0D7cjGeMKu7OKMdWICL++oj9FpWX85pONdIptxSUDOrsdy5gGZ1NtGONHaIjwl+sHMbhrLI+8u45t+cVuRzKmwVmBMKYGLcNCefb7gwkPC+Gu11fZ6a+m2bECYcxJdI5tzV+uH8TmvYf5+YeZbscxpkFZgTCmFmP6dOCesb14OyOXdzJ21d7BmCbCCoQxdfDAhSkM79GOn32YybLsArfjGNMgrEAYUwehIcJfbxhEt7hIbp69gpeWbbcL6UyTZwXCmDrqEN2K9+4awQV9O/D/PtrAI++t41iFDVybpssKhDH1ENUyjJk3DuG+8z1jEje++JUVCdNkWYEwpp5CQoQfjevDn64dSHrOQd5Kt4Fr0zRZgTDmFF01OJFhyXE8859sSstsL8I0PVYgjDlFIsKD4/qQf/gYr32Z43YcYwLOCoQxp2FochyjU+J5btFWio9VuB3HmICyAmHMaXrwohQOlpQze+l2t6MYE1BWIIw5TQO7xDIuNYEXlmyjsKTM7TjGBIwVCGMC4EfjUiguq+CFz7e5HcWYgLECYUwA9O3YhksGdOaFz7fzxdb9bscxJiCsQBgTIL+87Ay6xUVwx6sZrMstdDuOMafNCoQxAdI2MpzXpg4jNqIFt8xeQfa+w25HMua0WIEwJoA6xrTiH1OHERoSwo0vriD3YInbkYw5ZVYgjAmw7u0jeW3qUErKKrj++S/J3F3kdiRjTokVCGMc0K9TG9644xxUlaufW877q3PdjmRMvVmBMMYh/RNjmHfvKAZ1ieWHb63llx9toKKyyu1YxtSZowVCRCaIyGYRyRaRR/2sf1hE1ngfmSJSKSJx3nU5IrLeuy7DyZzGOKV9VEv+cfswpozozuxl23n43XVuRzKmzsKcemMRCQVmABcBuUC6iMxT1Q3H26jqk8CT3vaXAj9U1QM+bzNWVe3+jqZRaxEawuOXnUF0qzCe+U823zuzExemJrgdy5haObkHMRTIVtVtqloGzAEuP0n7G4A3HcxjjKvuPb83fTtG89MPMjl0tNztOMbUyskCkQj43kkl17vsO0QkApgAvOezWIFPRWSliEyr6UNEZJqIZIhIRn5+fgBiG+OM8LAQnrh6APsOH+V3n2xyO44xtXKyQIifZTXd5f1SYFm1w0sjVXUwMBG4W0RG++uoqrNUNU1V0+Lj408vsTEOG9glltvP7cGbK3ayfKsdPTXBzckCkQt08XmdBOTV0HYS1Q4vqWqe9999wPt4DlkZ0+j98MIUureL4LG56+1OdCaoOVkg0oHeIpIsIuF4isC86o1EJAY4D/jQZ1mkiEQffw6MAzIdzGpMg2kdHsrvrhrAjv0lTHlpBTv329XWJjg5ViBUtQK4B1gAbATeVtUsEZkuItN9ml4JfKqqR3yWJQBLRWQtsAL4WFXnO5XVmIY2vGc7/njtQDbkHWL8U0t4ZXkOVVU1HYE1xh2i2nT+U6alpWlGhl0yYRqPvMJSHpu7nsVb8hmaHMdzkwfTLqql27FMMyIiK1U1zd86u5LaGBd1jm3Ny7eezR+uGcCqHQd55j/Zbkcy5gQrEMa4TES4Lq0Llw9K5K30XRw4YrctNcHBCoQxQWL6eT0oLa/kleU5bkcxBrACYUzQ6J0QzUWpCbzyRQ5HjlW4HccYKxDGBJM7x/SksKScOem7am9sjMOsQBgTRAZ3bcvQ5Dj+/vk2yipsanDjLisQxgSZO8f0JK/oKPPW1jTxgDENwwqEMUFmTEo8fTtGM3PxVrt4zrjKCoQxQUZEuHNMT7L3FfPuSrtVqXGPFQhjgtClAzoztHscv/54A/sOHXU7jmmmrEAYE4RCQoTfX30mRyuq+PmHWW7HMc2UFQhjglSP+CgeuLA387P28K/137gdxzRDViCMCWJ3nNuD1E5t+Pm8LIpK7DalpmFZgTAmiLUIDeEP1wzgwJEyfvPJBrfjmGbGCoQxQa5/Ygx3nNuDtzNyWbnjoNtxTDNiBcKYRuDe83sRH92S336ykaZ0DxcT3KxAGNMIRLYM40cXpbByx0HmZ+5xO45pJqxAGNNIXDskiZSEKJ6Yv8nmaTINwgqEMY1EWGgIj13cj5z9Jbz+1Q6345hmwAqEMY3ImJR4RvZqx9MLv6ao1E57Nc6yAmFMIyIi/OTifhSWlvP4vCz+uS6Pf2/Yy5It+ew7bFNymMAKczuAMaZ+zugcw/eHduX1r3by/urdJ5anJESx4IHRiIiL6UxTYgXCmEbo11f05weje3KsopJjFVUs3pLPkws2syx7P6N6t3c7nmkiHD3EJCITRGSziGSLyKN+1j8sImu8j0wRqRSRuLr0NaY5ExG6tougd0I0/RNjmDoqmXaR4by8fLvb0UwTUqcCISKRIhLifZ4iIpeJSIta+oQCM4CJQCpwg4ik+rZR1SdVdZCqDgIeAxar6oG69DXG/E+rFqF8f1hXFm7ax479R9yOY5qIuu5BLAFaiUgisBC4FXi5lj5DgWxV3aaqZcAc4PKTtL8BePMU+xrT7E0e1o1QEV79wk6BNYFR1wIhqloCXAU8o6pX4vnL/mQSgV0+r3O9y7775iIRwATgvfr2NcZ4dIxpxcQzO/F2+i6OHKtwO45pAupcIERkODAZ+Ni7rLYBbn+nUtQ0icylwDJVPVDfviIyTUQyRCQjPz+/lkjGNG1TRnTn8LEK5q6yW5Wa01fXAvEAnjGC91U1S0R6AP+tpU8u0MXndRKQV0PbSfzv8FK9+qrqLFVNU9W0+Pj4WiIZ07QN7hrLgKQYXl6eY5P6mdNWpwKhqotV9TJVfcI7WF2gqvfV0i0d6C0iySISjqcIzKveSERigPOAD+vb1xjzbSLClBHd2Zp/hKXZBW7HMY1cXc9iekNE2ohIJLAB2CwiD5+sj6pWAPcAC4CNwNvevY/pIjLdp+mVwKeqeqS2vvXZMGOaq+8N6ER8dEsen5dFYUmZ23FMIyZ12Q0VkTWqOkhEJgNDgEeAlao6wOmA9ZGWlqYZGRluxzDGdV9t289Ns1dwZmIMr98+jFYtQt2OZIKUiKxU1TR/6+o6BtHCe93DFcCHqlpOzQPOxhiXDevRjqeuH8SqnQe5783VVFbZj6upv7oWiOeBHCASWCIi3YBDToUyxpy+i8/sxC8uSeXTDXv5xbxMG7Q29VanuZhU9WngaZ9FO0RkrDORjDGBMmVkMt8cOsrzi7exY38JN53TjfP7diAs1CZyNrWrU4Hwnmn0C2C0d9Fi4JdAkUO5jDEB8sj4vsS0bsEry3OY9tpKOrZpxaShXbjj3B5EtrT5Ok3N6jpI/R6QCbziXXQTMFBVr3IwW73ZILUxNauorGLhpn28/tVOlmzJp2/HaF64OY0ucRFuRzMuOtkgdb3OYqptmdusQBhTN0u25HP3G6toERrCzBuHMDQ5zu1IxiWBOIupVERG+bzhSKA0EOGMMQ1vdEo8H9w9kpjWLZj84pfMWbHTBrHNd9S1QEwHZohIjojkAM8CP3AslTHGcT3jo/jgrpGc06Mdj85dz6XPLuWjtXlUVFa5Hc0EibpOtbFWVQcCA4ABqnoWcL6jyYwxjouJaMFLU87md1edScmxSu59czVj/7SI177IoazCCkVzV6cxCL8dRXaqatcA5zktNgZhzKmrqlL+vXEvMxdvZfXOQrrGRfDguBQuHdCZkBC7z3VTFYgxCL/vexp9jTFBJiREGH9GR+beOYJXbhtKZMsw7p+zhstmLCU950Dtb2CanNMpEDaiZUwTJCKclxLPx/eO4i/XD+TgkXKmzF7BwSM28V9zc9ICISKHReSQn8dhoHMDZTTGuCAkRLjyrCReuvVsSsor+fvS7W5HMg3spAVCVaNVtY2fR7Sq2iWYxjQDKQnRXNy/Ey8vz7Hpw5sZm5DFGFOrey/oRfGxCmYvy3E7imlAViCMMbXq27ENE87oyEvLtlNUWu52HNNArEAYY+rk3gt6cfhoBS/bXkSzYQXCGFMnZ3SO4aLUBP6+dBuHjtpeRHNgBcIYU2f3nd+bQ0creMX2IpoFKxDGmDo7MymGC/sl8NzireQV2nydTZ0VCGNMvfzi0lSqVPnlRxvcjmIcZgXCGFMvXeIiuO+C3szP2sPCjXvdjmMcZAXCGFNvt4/qQe8OUfz8wyxKyircjmMcYgXCGFNv4WEh/ObKM9ldWMrTC7PdjmMc4miBEJEJIrJZRLJF5NEa2owRkTUikiUii32W54jIeu86m8PbmCAzNDmO69KSePHzbWzac8jtOMYBjhUIEQkFZgATgVTgBhFJrdYmFvgbcJmqngFcW+1txqrqoJrmKjfGuOvRif2Iad2CG2Z9yWcbbDyiqXFyD2IokK2q21S1DJgDXF6tzfeBuaq6E0BV9zmYxxgTYHGR4bwzfTidY1tz+6sZPD4vi6PllW7HMgHiZIFIBHb5vM71LvOVArQVkUUislJEbvZZp8Cn3uXTavoQEZkmIhkikpGfnx+w8MaYuukRH8Xcu0Zw28hkXl6ewxUzlpF7sMTtWCYAnCwQ/u44V/0mQ2HAEOB7wHjgZyKS4l03UlUH4zlEdbeIjPb3Iao6S1XTVDUtPj4+QNGNMfXRMiyUn1+ayuwpaeQeLOUn72dyqrczNsHDyQKRC3TxeZ0E5PlpM19Vj6hqAbAEGAigqnnef/cB7+M5ZGWMCWLn903ggQt7s2RLPv/dbEeMGzsnC0Q60FtEkkUkHJgEzKvW5kPgXBEJE5EIYBiwUUQiRSQaQEQigXFApoNZjTEBcvPw7vSIj+TX/9xIWUWV23HMaXCsQKhqBXAPsADYCLytqlkiMl1EpnvbbATmA+uAFcCLqpoJJABLRWStd/nHqjrfqazGmMAJDwvhZ5eksq3gCK9+keN2HHMapCkdJ0xLS9OMDLtkwphgMOWlFazMOch/Hx5D+6iWbscxNRCRlTVdSmBXUhtjHPHT76VSWl7Jnz7d4nYUc4qsQBhjHNGrQxS3jOjOnPSdrMstdDuOOQVWIIwxjrnvgt50iG7JQ++s5ViFXUDX2FiBMMY4JqZ1C35/9QC27C3mqc++djuOqScrEMYYR43t04Hr07rw/OKtrNp50O04ph6sQBhjHPfTS/rRKaY1D729ltIyO9TUWFiBMMY4LrpVC/5wzQC2FRzhyQWb3Y5j6sgKhDGmQYzs1Z6bzunGS8u32zQcjYQVCGNMg/nJxf3o27EN97+5mh37j7gdx9TCCoQxpsG0Dg/l+RuHICL84LWVdj/rIGcFwhjToLq2i+Cvkwaxee9hHpu73qYFD2JWIIwxDW5Mnw48NK4PH67JY/ayHLfjmBpYgTDGuOLO83oyLjWB332y0abiCFJWIIwxrggJEZ68ZiDto1ryw7fW2L2sT5GTh+isQBhjXBMT0YInrx3A1vwj/GG+XR9xKv706RYun7GMqqrAFworEMYYV53bO55bhndj9rLtLM8ucDtOo7Nyx0FUlZAQCfh7W4Ewxrju0Yn96BEfyUPvrKWotNztOI1GVZWyfncRA5NiHXl/KxDGGNe1Dg/lz9cNYu/hYzw+L8vtOI3GtoJiio9VMCApxpH3twJhjAkKg7rEcu/5vXh/9W7mrsp1O06jsGZXEeD52jnBCoQxJmjcM7YXQ5Pj+OkHmWzLL3Y7TtBbl1tIZHgoPeKjHHl/KxDGmKARFhrCXycNIjwshHvfXG13oavF2twi+ifGEOrAADVYgTDGBJlOMa158pqBZOUd4vf/2uR2nKBVVlHFxrxDjh1eAisQxpggdFFqAlNGdOelZTn8e8Net+MEpU17DlFWWcUAh85gAocLhIhMEJHNIpItIo/W0GaMiKwRkSwRWVyfvsaYpuuxi/vSP7ENP3xrDVl5RW7HCTprd3mmJxnYxZkzmMDBAiEiocAMYCKQCtwgIqnV2sQCfwMuU9UzgGvr2tcY07S1DAvlxZvPpk2rMG59KZ3cgyVuRwoqa3OLaBcZTmJsa8c+w8k9iKFAtqpuU9UyYA5webU23wfmqupOAFXdV4++xpgmrmNMK16+bSil5ZVMeSmdohK7iO64tbsKGdglFhFnBqjB2QKRCOzyeZ3rXeYrBWgrIotEZKWI3FyPvgCIyDQRyRCRjPz8/ABFN8YEi5SEaGbdlMbO/SXc8WqGTeoHFB+rIDu/2LEL5I5zskD4K2vVZ5MKA4YA3wPGAz8TkZQ69vUsVJ2lqmmqmhYfH386eY0xQWp4z3b88bqBrMg5wE/sJkOszy1CFQY6eAYTeH5BOyUX6OLzOgnI89OmQFWPAEdEZAkwsI59jTHNyGUDO7Mtv5inPvuaId3bMnlYN7cjueb4/TOcmoPpOCf3INKB3iKSLCLhwCRgXrU2HwLnikiYiEQAw4CNdexrjGlm7ju/N6NT4vl/8zY065sMrc0tpEtca+Iiwx39HMcKhKpWAPcAC/D80n9bVbNEZLqITPe22QjMB9YBK4AXVTWzpr5OZTXGNA4hIcJT1w+ifVQ4d/5jFYUlZW5HcsXaXUWOXv9wnDSlY3lpaWmakZHhdgxjjMPW7Crk2pnLObd3PC/enObIvRCCVUHxMdJ+/Rn/d3E/7hjd47TfT0RWqmqav3V2JbUxptEZ1CWWn12Syn827ePuN1Zx4Ejz2ZM4fmjN6TOYwNlBamOMccxN53SjpKySP326mfScg/z+qjO5MDXB7ViOW7uriBCB/onOFwjbgzDGNEoiwvTzejLvnlG0jwrn9lczePidtU1+XGJdbiG9OkQR2dL5v++tQBhjGrV+ndow755R3D22J++tyuW8Jxfx8rLtlFdWuR0t4FSVdbkNM0ANViCMMU1AeFgID4/vyyf3n0v/xDY8/tEGJjy1hMVbmtbsCrsLS9l/pIyBDVnP6pAAAA7fSURBVDD+AFYgjDFNSN+ObfjH1GG8cHMalVXKLbNXsGjzvto7NhLrcj2z2toehDHGnAIR4aLUBOY/MJqUhCh+/O46DjaRs5zW5hbSIlTo2ym6QT7PCoQxpklq1SKUP183iIMlZfzfB01j/qZ1u4ro16kNLcNCG+TzrEAYY5qs/okx/PCiFD5Zv4cP1ux2O85pqapSMncXNcj1D8dZgTDGNGk/GN2TtG5t+fkHWewuLHU7zinbVnCEw8cqGmz8AaxAGGOauNAQ4c/XDaJKlTteyeDt9F3sKTrqdqx6a6gZXH3ZldTGmCava7sI/njtQH4xL4sfv7cOgJSEKCad3ZVbR3Z39K5sgbIut4iI8FB6dYhqsM+0AmGMaRYmntmJCf07snnvYZZsyefTrL388p8bOHy0gvsv7O12vFqtzS2kf+cYQhtwYkI7xGSMaTZEhL4d2zBtdE/e/sFwrhmSxF8+28Jzi7a6He2kyiur2JB3qEEHqMH2IIwxzVRIiPDE1QMoq6jiifmbCA8LYeqoZLdj+bV5z2GOVVQxwOFbjFZnBcIY02x5BrAHUl5Zxa/+uYGolqFcf3ZXt2N9x/ErqBtqio3j7BCTMaZZCwsN4a+TzmJUr/Y8Pm8DO/eXuJrnaHklLy/b/q1TctflFhIb0YKucRENmsUKhDGm2QsPC+HJawcQGiI8Oneda1ddHz5azi2zV5yYbPBD78V9a3OLODMxpsHPtrICYYwxQKeY1jx2cV+Wb93P2xm7Gvzz9xcf44YXvmTljoP87JJUUhKiuX/OGu5+YxVb9h5u0OsfjrMCYYwxXjec3ZVhyXH8+uON7D3UcBfT7S4s5drnvyB7XzEv3JzG1FHJvDXtHB4al8KCzD1UVmmDn8EEViCMMeYE3zOb/u/9zAY51FReWcWkWV+Qf/gYr00dxti+HQDP2Mg95/dm7l0jmDoqmXN7xzuepTorEMYY46N7+0geHJfCZxv38puPN5Ln8PxNX2zdz64Dpfzh6gGc3T3uO+sHJMXys0tSaR3eMDO4+rLTXI0xpprbRiazfvch/r5sO7OXbWdsnw5MPqcrY1I6EBLgK5nnZ+0hIjz0xJ5DMLE9CGOMqSYsNIRnbjiLJQ+P5a4xvVi3u4jbXs7g5tkrAjo2UVmlfJq1l7F9O9CqRcPvIdTG0QIhIhNEZLOIZIvIo37WjxGRIhFZ43383Gddjois9y7PcDKnMcb40yUugofG92H5o+fzqyv6s3LHQcY/tYT5mXtOtDlaXskXW/czP/MbqqrqN2axcsdBCoqPMeGMjoGOHhCOHWISkVBgBnARkAuki8g8Vd1QrennqnpJDW8zVlULnMpojDF10SI0hJvO6caInu14YM4apv9jJRP7d6SotJyVOw5yrKIKgIn9O/Ln6wbVebxgfuYewsNCgvLwEji7BzEUyFbVbapaBswBLnfw84wxxlE946N4784R3DWmJws37eNgSTmTh3XjxZvT+MnFfZmftYdJs75gXx0OQ6kqC7L2MLp3e6JaBudwsJOpEgHfq01ygWF+2g0XkbVAHvCQqmZ5lyvwqYgo8LyqzvL3ISIyDZgG0LVr8M2hYoxpWsLDQvjxhL48PL5PtSubE0huH8V9b67mihnLmH3r2fTt2KbG91m/u4jdhaX88KIU50OfIif3IPwN9Vc/QLcK6KaqA4FngA981o1U1cHAROBuERnt70NUdZaqpqlqWnx8w58nbIxpnvxNe3FRagLvTB9OpSrXP//lSU+RnZ+5h9AQ4cJ+wXl4CZwtELlAF5/XSXj2Ek5Q1UOqWux9/gnQQkTae1/nef/dB7yP55CVMcYEtf6JMbw1bTgVlVU8MGcNlX4GrlWV+Zl7GN6jHbER4S6krBsnC0Q60FtEkkUkHJgEzPNtICIdxVuGRWSoN89+EYkUkWjv8khgHJDpYFZjjAmY7u0j+dUV/VmRc4Bn/5P9nfVf7ytmW8ERJvQPzrOXjnNsDEJVK0TkHmABEArMVtUsEZnuXT8TuAa4U0QqgFJgkqqqiCQA73trRxjwhqrOdyqrMcYE2lWDk/j86wL+unALI3u1I83nKun5mXsQgXFnJLiYsHbi1rS2TkhLS9OMDLtkwhgTHA4fLed7Ty+lskqZe9cI1uUW8Z9Ne/nnum/okxDNu3eOcDsiIrJSVdP8rQvOc6uMMaYJiG7VgqdvOItrnlvOsN8uBCCqZRjnpcRz7wW9XE5XOysQxhjjoEFdYvnTdQNZl1vE+X07cHb3OMLDGscsR1YgjDHGYZcPSuTyQYlux6i3xlHGjDHGNDgrEMYYY/yyAmGMMcYvKxDGGGP8sgJhjDHGLysQxhhj/LICYYwxxi8rEMYYY/xqUnMxiUg+sKPa4higqJZlvq/9PT/+b3vgVG+B6i9HXdYHMj+c+jbUlv9kbU6Wt/rr2p5b/vq3qe3/UE3bE8j8J8tX23r7GXY2fzdV9X8zHVVt0g9gVm3LfF/7e+7zb0Ygc9RlfSDzn8421Ja/PttQ3/yB+B5Y/pqX1bQ9gcxfl21oiJ8By1+/Ps3hENNHdVj2US3P/b1HIHLUZX1jyX+yNifLW/11XZ6fCstf87KatieQ+evyHo39Z6Cx5/+OJnWIyWkikqE1TIvbWDT2bbD87rL87mro/M1hDyKQZrkdIAAa+zZYfndZfnc1aH7bgzDGGOOX7UEYY4zxywqEMcYYv5ptgRCR2SKyT0QyT6HvEBFZLyLZIvK0iIjPuutEZIOIZInIG4FN/a0MAc8vIlNEJF9E1ngftwc++YkMjnz9veuvEREVEUcH8xz6Hkz3Ll8jIktFJDXwyU9kcCL/j7z//9eJyEIR6Rb45CcyOJF/tIisEpEKEbkm8KlPL3cN73eLiHztfdziszxZRL7yLn9LRMLr/eanek5tY38Ao4HBQOYp9F0BDAcE+Bcw0bu8N7AaaOt93aGR5Z8CPNtYv/7eddHAEuBLIK2xbQPQxqfNZcD8RpZ/LBDhfX4n8FYjy98dGAC8ClwTTLmBRUD3asvigG3ef9t6nx///fM2MMn7fCZwZ32zNts9CFVdAhzwXSYiPUVkvoisFJHPRaRv9X4i0gnPD/EX6vnKvwpc4V19BzBDVQ96P2NfI8vfYBzM/yvgD8BRB+MDzmyDqh7yaRoJOHYWiUP5/6uqJd6mXwJJjSx/jqquA6qCLXcNxgP/VtUD3t87/wYmePeIzgfe9bZ7hVP4OW+2BaIGs4B7VXUI8BDwNz9tEoFcn9e53mUAKUCKiCwTkS9FZIKjab/rdPMDXO09PPCuiHRxLqpfp5VfRM4CuqjqP50OehKn/T0QkbtFZCueQnefg1n9CcT/oeOm4vnrvCEFMn9DqktufxKBXT6vj29LO6BQVSuqLa+XsPp2aKpEJAoYAbzjc0i7pb+mfpYd/ysvDM9hpjF4/nL6XET6q2phYNP6CRWY/B8Bb6rqMRGZjuevjvMDndWf080vIiHAX/AcJnNFgL4HqOoMYIaIfB/4KXCLn/YBF6j83ve6EUgDzgtkxpMJZP6GdLLcInIrcL93WS/gExEpA7ar6pXUvC0B2UYrEP8TgqfiDvJdKCKhwErvy3nAc3x7tzkJyPM+zwW+VNVyYLuIbMZTMNKdDO512vlVdb/P8heAJxxL+12nmz8a6A8s8v6QdQTmichlqprhcPbjAvF/yNccb9uGEpD8InIh8H/Aeap6zNHE3xbor39D8ZsbQFVfAl4CEJFFwBRVzfFpkovnD9LjkvCMVRQAsSIS5t2LOLVtdGIQprE88AxIZfq8Xg5c630uwMAa+qUD5/C/Aa6LvcsnAK94n7fHs+vXrhHl7+TT5ko8xa7RfP2rtVmEw4PUDn0Pevu0uZTTmJzNpfxnAVt9t6Mx5fdZ/zIODVKfam5qHqTejmeAuq33eZx33Tt8e5D6rnrnbIhvYjA+gDeBb4ByPFV4KpAMzAfWAhuAn9fQNw3I9P4gPMv/rkgX4M/evuuPf3MaUf7fAVne/v8F+jam/NXaLML5s5ic+B781fs9WOP9HpzRyPJ/Buz15l8DzGtk+c/2vtcRYD+QFSy58VMgvMtvA7K9j1t9lvfAc7ZWNp5i0bK+WW2qDWOMMX7ZWUzGGGP8sgJhjDHGLysQxhhj/LICYYwxxi8rEMYYY/yyAmGaNBEpbuDPWx6g9xkjIkUislpENonIH+vQ5wpxcPZX0/xYgTCmHkTkpLMPqOqIAH7c56p6Fp6Lzy4RkZG1tL8CsAJhAsam2jDNjoj0BGYA8UAJcIeqbhKRS/HMfRSO5yKpyaq6V0QeBzrjufq1QES2AF3xXIjUFXhKVZ/2vnexqkaJyBjgcTxTHvTHM9XDjaqqInIxngsqC4BVQA9VvaSmvKpaKiJr+N+khHcA07w5s4GbgEF4pgc/T0R+Clzt7f6d7TyNL51pZmwPwjRHNc2cuRQ4x/tX+xzgxz59hgCXq+r3va/74plqeSjwCxFp4edzzgIewPNXfQ9gpIi0Ap7Hc/+BUXh+eZ+UiLTFM6fXEu+iuap6tqoOBDYCU1V1OZ55hh5W1UGquvUk22lMndgehGlWapnxMwl4y3u/gHA889ocN09VS31ef6yeieiOicg+IIFvTyENsEJVc72fuwbPHkgxsE1Vj7/3m3j2Bvw5V0TWAX2A36vqHu/y/iLyayAWiAIW1HM7jakTKxCmualx5kzgGeDPqjrP5xDRcUeqtfWdpbQS/z9L/tr4m4a5Jp+r6iUikgIsFZH3VXUNnonkrlDVtSIyhW/P5nncybbTmDqxQ0ymWVHPHdu2i8i1AOIx0Ls6Btjtfe7UPRg2AT1EpLv39fW1dVDVLXgmUnzEuyga+MZ7WGuyT9PD3nW1bacxdWIFwjR1ESKS6/P4EZ5fqlNFZC2emVMv97Z9HM8hmc/xDCAHnPcw1V3AfBFZimfm06I6dJ0JjBaRZOBnwFd4bi/pO+g8B3jYe2psT2reTmPqxGZzNaaBiUiUqhZ77xs8A/haVf/idi5jqrM9CGMa3h3eQessPIe1nnc5jzF+2R6EMcYYv2wPwhhjjF9WIIwxxvhlBcIYY4xfViCMMcb4ZQXCGGOMX/8febSivL59Dj0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train classifier\n",
    "We start by just training the very last layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.325401</td>\n",
       "      <td>0.261644</td>\n",
       "      <td>0.892640</td>\n",
       "      <td>04:50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we unfreeze the second to last layer as well and train these layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.267379</td>\n",
       "      <td>0.217356</td>\n",
       "      <td>0.913600</td>\n",
       "      <td>05:13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we train the last three layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.233473</td>\n",
       "      <td>0.182271</td>\n",
       "      <td>0.930040</td>\n",
       "      <td>04:54</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we train all layers for two epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.201170</td>\n",
       "      <td>0.175181</td>\n",
       "      <td>0.933120</td>\n",
       "      <td>05:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.182616</td>\n",
       "      <td>0.174323</td>\n",
       "      <td>0.932920</td>\n",
       "      <td>05:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that after the last optimisation step the model can predict the sentiment on the test set with **94%** accuracy. This is roughly 10% better than our Naïve Bayes model from the last lecture. In other words, this model makes **3 times** fewer mistakes than the Naïve Bayes model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Category pos, tensor(1), tensor([3.5403e-06, 1.0000e+00]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"I really loved that movie, it was awesome!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "Experiment with the trained classifier and see if you can fool it. Can you find a pattern that fools it consistently?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}