{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMDb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At Fast.ai we have introduced a new module called fastai.text which replaces the torchtext library that was used in our 2018 dl1 course. The fastai.text module also supersedes the fastai.nlp library but retains many of the key functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *\n",
    "import html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Fastai.text module introduces several custom tokens.\n",
    "\n",
    "We need to download the IMDB large movie reviews from this site: http://ai.stanford.edu/~amaas/data/sentiment/\n",
    "Direct link : [Link](http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz) and untar it into the PATH location. We use pathlib which makes directory traveral a breeze."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_PATH=Path('data/')\n",
    "DATA_PATH.mkdir(exist_ok=True)\n",
    "#! curl -O http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz \n",
    "#! tar -xzfv aclImdb_v1.tar.gz -C {DATA_PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BOS = 'xbos'  # beginning-of-sentence tag\n",
    "FLD = 'xfld'  # data field tag\n",
    "\n",
    "PATH=Path('data/aclImdb/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standardize format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLAS_PATH=Path('data/imdb_clas/')\n",
    "CLAS_PATH.mkdir(exist_ok=True)\n",
    "\n",
    "LM_PATH=Path('data/imdb_lm/')\n",
    "LM_PATH.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The imdb dataset has 3 classes. positive, negative and unsupervised(sentiment is unknown). \n",
    "There are 75k training reviews(12.5k pos, 12.5k neg, 50k unsup)\n",
    "There are 25k validation reviews(12.5k pos, 12.5k neg & no unsup)\n",
    "\n",
    "Refer to the README file in the imdb corpus for further information about the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASSES = ['neg', 'pos', 'unsup']\n",
    "\n",
    "def get_texts(path):\n",
    "    texts,labels = [],[]\n",
    "    for idx,label in enumerate(CLASSES):\n",
    "        for fname in (path/label).glob('*.*'):\n",
    "            texts.append(fname.open('r', encoding='utf-8').read())\n",
    "            labels.append(idx)\n",
    "    return np.array(texts),np.array(labels)\n",
    "\n",
    "trn_texts,trn_labels = get_texts(PATH/'train')\n",
    "val_texts,val_labels = get_texts(PATH/'test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(75000, 25000)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(trn_texts),len(val_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_names = ['labels','text']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a random permutation np array to shuffle the text reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "trn_idx = np.random.permutation(len(trn_texts))\n",
    "val_idx = np.random.permutation(len(val_texts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_texts = trn_texts[trn_idx]\n",
    "val_texts = val_texts[val_idx]\n",
    "\n",
    "trn_labels = trn_labels[trn_idx]\n",
    "val_labels = val_labels[val_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trn = pd.DataFrame({'text':trn_texts, 'labels':trn_labels}, columns=col_names)\n",
    "df_val = pd.DataFrame({'text':val_texts, 'labels':val_labels}, columns=col_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas dataframe is used to store text data in a newly evolving standard format of label followed by text columns. This was influenced by a paper by Yann LeCun (LINK REQUIRED). Fastai adopts this new format for NLP datasets. In the case of IMDB, there is only one text column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trn[df_trn['labels']!=2].to_csv(CLAS_PATH/'train.csv', header=False, index=False)\n",
    "df_val.to_csv(CLAS_PATH/'test.csv', header=False, index=False)\n",
    "\n",
    "(CLAS_PATH/'classes.txt').open('w', encoding='utf-8').writelines(f'{o}\\n' for o in CLASSES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by creating the data for the Language Model(LM). The LM's goal is to learn the structure of the english language. It learns language by trying to predict the next word given a set of previous words(ngrams). Since the LM does not classify reviews, the labels can be ignored.\n",
    "\n",
    "The LM can benefit from all the textual data and there is no need to exclude the unsup/unclassified movie reviews.\n",
    "\n",
    "We first concat all the train(pos/neg/unsup = **75k**) and test(pos/neg=**25k**) reviews into a big chunk of **100k** reviews. And then we use sklearn splitter to divide up the 100k texts into 90% training and 10% validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_texts,val_texts = sklearn.model_selection.train_test_split(\n",
    "    np.concatenate([trn_texts,val_texts]), test_size=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(90000, 10000)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(trn_texts), len(val_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trn = pd.DataFrame({'text':trn_texts, 'labels':[0]*len(trn_texts)}, columns=col_names)\n",
    "df_val = pd.DataFrame({'text':val_texts, 'labels':[0]*len(val_texts)}, columns=col_names)\n",
    "\n",
    "df_trn.to_csv(LM_PATH/'train.csv', header=False, index=False)\n",
    "df_val.to_csv(LM_PATH/'test.csv', header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language model tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we start cleaning up the messy text. There are 2 main activities we need to perform:\n",
    "\n",
    "1. Clean up extra spaces, tab chars, new ln chars and other characters and replace them with standard ones\n",
    "2. Use the awesome [spacy](http://spacy.io) library to tokenize the data. Since spacy does not provide a parallel/multicore version of the tokenizer, the fastai library adds this functionality. This parallel version uses all the cores of your CPUs and runs much faster than the serial version of the spacy tokenizer.\n",
    "\n",
    "Tokenization is the process of splitting the text into separate tokens so that each token can be assigned a unique index. This means we can convert the text into integer indexes our models can use.\n",
    "\n",
    "We use an appropriate chunksize as the tokenization process is memory intensive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chunksize=24000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "re1 = re.compile(r'  +')\n",
    "\n",
    "def fixup(x):\n",
    "    x = x.replace('#39;', \"'\").replace('amp;', '&').replace('#146;', \"'\").replace(\n",
    "        'nbsp;', ' ').replace('#36;', '$').replace('\\\\n', \"\\n\").replace('quot;', \"'\").replace(\n",
    "        '<br />', \"\\n\").replace('\\\\\"', '\"').replace('<unk>','u_n').replace(' @.@ ','.').replace(\n",
    "        ' @-@ ','-').replace('\\\\', ' \\\\ ')\n",
    "    return re1.sub(' ', html.unescape(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_texts(df, n_lbls=1):\n",
    "    labels = df.iloc[:,range(n_lbls)].values.astype(np.int64)\n",
    "    texts = f'\\n{BOS} {FLD} 1 ' + df[n_lbls].astype(str)\n",
    "    for i in range(n_lbls+1, len(df.columns)): texts += f' {FLD} {i-n_lbls} ' + df[i].astype(str)\n",
    "    texts = list(texts.apply(fixup).values)\n",
    "\n",
    "    tok = Tokenizer().proc_all_mp(partition_by_cores(texts))\n",
    "    return tok, list(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_all(df, n_lbls):\n",
    "    tok, labels = [], []\n",
    "    for i, r in enumerate(df):\n",
    "        print(i)\n",
    "        tok_, labels_ = get_texts(r, n_lbls)\n",
    "        tok += tok_;\n",
    "        labels += labels_\n",
    "    return tok, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trn = pd.read_csv(LM_PATH/'train.csv', header=None, chunksize=chunksize)\n",
    "df_val = pd.read_csv(LM_PATH/'test.csv', header=None, chunksize=chunksize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "tok_trn, trn_labels = get_all(df_trn, 1)\n",
    "tok_val, val_labels = get_all(df_val, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(LM_PATH/'tmp').mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save(LM_PATH/'tmp'/'tok_trn.npy', tok_trn)\n",
    "np.save(LM_PATH/'tmp'/'tok_val.npy', tok_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tok_trn = np.load(LM_PATH/'tmp'/'tok_trn.npy')\n",
    "tok_val = np.load(LM_PATH/'tmp'/'tok_val.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 1207984),\n",
       " ('.', 991762),\n",
       " (',', 985975),\n",
       " ('and', 587317),\n",
       " ('a', 583569),\n",
       " ('of', 524362),\n",
       " ('to', 484813),\n",
       " ('is', 393574),\n",
       " ('it', 341627),\n",
       " ('in', 337461),\n",
       " ('i', 308563),\n",
       " ('this', 270705),\n",
       " ('that', 261447),\n",
       " ('\"', 236753),\n",
       " (\"'s\", 221112),\n",
       " ('-', 188249),\n",
       " ('was', 180235),\n",
       " ('\\n\\n', 178679),\n",
       " ('as', 165610),\n",
       " ('with', 159164),\n",
       " ('for', 158981),\n",
       " ('movie', 157676),\n",
       " ('but', 150203),\n",
       " ('film', 144108),\n",
       " ('you', 124114)]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freq = Counter(p for o in tok_trn for p in o)\n",
    "freq.most_common(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *vocab* is the **unique set of all tokens** in our dataset. The vocab provides us a way for us to simply replace each word in our datasets with a unique integer called an index.\n",
    "\n",
    "In a large corpus of data one might find some rare words which are only used a few times in the whole dataset. We discard such rare words and avoid trying to learn meaningful patterns out of them.\n",
    "\n",
    "Here we have set a minimum frequency of occurence to 2 times. It has been observed by NLP practicioners that a maximum vocab of 60k usually yields good results for classification tasks. So we set maz_vocab to 60000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_vocab = 60000\n",
    "min_freq = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "itos = [o for o,c in freq.most_common(max_vocab) if c>min_freq]\n",
    "itos.insert(0, '_pad_')\n",
    "itos.insert(0, '_unk_')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a reverse mapping called stoi which is useful to lookup the index of a given token. stoi also has the same number of elements as itos. We use a high performance container called [collections.defaultdict](https://docs.python.org/2/library/collections.html#collections.defaultdict) to store our stoi mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60002"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stoi = collections.defaultdict(lambda:0, {v:k for k,v in enumerate(itos)})\n",
    "len(itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_lm = np.array([[stoi[o] for o in p] for p in tok_trn])\n",
    "val_lm = np.array([[stoi[o] for o in p] for p in tok_val])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save(LM_PATH/'tmp'/'trn_ids.npy', trn_lm)\n",
    "np.save(LM_PATH/'tmp'/'val_ids.npy', val_lm)\n",
    "pickle.dump(itos, open(LM_PATH/'tmp'/'itos.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_lm = np.load(LM_PATH/'tmp'/'trn_ids.npy')\n",
    "val_lm = np.load(LM_PATH/'tmp'/'val_ids.npy')\n",
    "itos = pickle.load(open(LM_PATH/'tmp'/'itos.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60002, 90000)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vs=len(itos)\n",
    "vs,len(trn_lm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## wikitext103 conversion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now going to build an english language model for the IMDB corpus. We could start from scratch and try to learn the structure of the english language. But we use a technique called transfer learning to make this process easier. In transfer learning (a fairly recent idea for NLP) a pre-trained LM that has been trained on a large generic corpus(_like wikipedia articles_) can be used to transfer it's knowledge to a target LM and the weights can be fine-tuned.\n",
    "\n",
    "Our source LM is the wikitext103 LM created by Stephen Merity @ Salesforce research. [Link to dataset](https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/)\n",
    "The language model for wikitext103 (AWD LSTM) has been pre-trained and the weights can be downloaded here: [Link](http://files.fast.ai/models/wt103/). Our target LM is the IMDB LM. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! wget -nH -r -np -P {PATH} http://files.fast.ai/models/wt103/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pre-trained LM weights have an embedding size of 400, 1150 hidden units and just 3 layers. We need to match these values  with the target IMDB LM so that the weights can be loaded up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "em_sz,nh,nl = 400,1150,3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PRE_PATH = PATH/'models'/'wt103'\n",
    "PRE_LM_PATH = PRE_PATH/'fwd_wt103.h5'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wgts = torch.load(PRE_LM_PATH, map_location=lambda storage, loc: storage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We calculate the mean of the layer0 encoder weights. This can be used to assign weights to unknown tokens when we transfer to target IMDB LM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "enc_wgts = to_np(wgts['0.encoder.weight'])\n",
    "row_m = enc_wgts.mean(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "itos2 = pickle.load((PRE_PATH/'itos_wt103.pkl').open('rb'))\n",
    "stoi2 = collections.defaultdict(lambda:-1, {v:k for k,v in enumerate(itos2)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we try to transfer the knowledge from wikitext to the IMDB LM, we match up the vocab words and their indexes. \n",
    "We use the defaultdict container once again, to assign mean weights to unknown IMDB tokens that do not exist in wikitext103."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_w = np.zeros((vs, em_sz), dtype=np.float32)\n",
    "for i,w in enumerate(itos):\n",
    "    r = stoi2[w]\n",
    "    new_w[i] = enc_wgts[r] if r>=0 else row_m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now overwrite the weights into the wgts odict.\n",
    "The decoder module, which we will explore in detail is also loaded with the same weights due to an idea called weight tying."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wgts['0.encoder.weight'] = T(new_w)\n",
    "wgts['0.encoder_with_dropout.embed.weight'] = T(np.copy(new_w))\n",
    "wgts['1.decoder.weight'] = T(np.copy(new_w))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the weights prepared, we are ready to create and start training our new IMDB language pytorch model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is fairly straightforward to create a new language model using the fastai library. Like every other lesson, our model will have a backbone and a custom head. The backbone in our case is the IMDB LM pre-trained with wikitext and the custom head is a linear classifier. In this section we will focus on the backbone LM and the next section will talk about the classifier custom head.\n",
    "\n",
    "bptt (*also known traditionally in NLP LM as ngrams*) in fastai LMs is approximated to a std. deviation around 70, by perturbing the sequence length on a per-batch basis. This is akin to shuffling our data in computer vision, only that in NLP we cannot shuffle inputs and we have to maintain statefulness. \n",
    "\n",
    "Since we are predicting words using ngrams, we want our next batch to line up with the end-points of the previous mini-batch's items. batch-size is constant and but the fastai library expands and contracts bptt each mini-batch using a clever stochastic implementation of a batch. (original credits attributed to [Smerity](https://twitter.com/jeremyphoward/status/980227258395770882))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wd=1e-7\n",
    "bptt=70\n",
    "bs=52\n",
    "opt_fn = partial(optim.Adam, betas=(0.8, 0.99))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of the LM is to learn to predict a word/token given a preceeding set of words(tokens). We take all the movie reviews in both the 90k training set and 10k validation set and concatenate them to form long strings of tokens. In fastai, we use the `LanguageModelLoader` to create a data loader which makes it easy to create and use bptt sized mini batches. The  `LanguageModelLoader` takes a concatenated string of tokens and returns a loader.\n",
    "\n",
    "We have a special modeldata object class for LMs called `LanguageModelData` to which we can pass the training and validation loaders and get in return the model itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_dl = LanguageModelLoader(np.concatenate(trn_lm), bs, bptt)\n",
    "val_dl = LanguageModelLoader(np.concatenate(val_lm), bs, bptt)\n",
    "md = LanguageModelData(PATH, 1, vs, trn_dl, val_dl, bs=bs, bptt=bptt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We setup the dropouts for the model - these values have been chosen after experimentation. If you need to update them for custom LMs, you can change the weighting factor (0.7 here) based on the amount of data you have. For more data, you can reduce dropout factor and for small datasets, you can reduce overfitting by choosing a higher dropout factor. *No other dropout value requires tuning*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15])*0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first tune the last embedding layer so that the missing tokens initialized with mean weights get tuned properly. So we freeze everything except the last layer.\n",
    "\n",
    "We also keep track of the *accuracy* metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner= md.get_model(opt_fn, em_sz, nh, nl, \n",
    "    dropouti=drops[0], dropout=drops[1], wdrop=drops[2], dropoute=drops[3], dropouth=drops[4])\n",
    "\n",
    "learner.metrics = [accuracy]\n",
    "learner.freeze_to(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.model.load_state_dict(wgts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set learning rates and fit our IMDB LM. We first run one epoch to tune the last layer which contains the embedding weights. This should help the missing tokens in the wikitext103 learn better weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=1e-3\n",
    "lrs = lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e3839cad2e23478a84362ed0a931abf1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                     \n",
      "    0      4.398856   4.175343   0.28551   \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[4.175343, 0.2855095456305303]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.fit(lrs/2, 1, wds=wd, use_clr=(32,2), cycle_len=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we print out accuracy and keep track of how often we end up predicting the target word correctly. While this is a good metric to check, it is not part of our loss function as it can get quite bumpy. We only minimize cross-entropy loss in the LM.\n",
    "\n",
    "The exponent of the cross-entropy loss is called the perplexity of the LM. (low perplexity is better)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.save('lm_last_ft')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.load('lm_last_ft')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1602776e4b864be99452869968c64a93",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                     \n",
      "    0      4.866224   4.736622   0.236569  \n",
      "\n"
     ]
    }
   ],
   "source": [
    "learner.lr_find(start_lr=lrs/10, end_lr=lrs*10, linear=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEOCAYAAAB4nTvgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VGXWwPHfSSGNJLRQA4RepUjAggVRQcHFrqBY31VXd9XVteHuu/ZXd+11V2TVtcHa1gIqqBQpIoQuvXdIaGmk57x/zE0yk0wKMCXlfD+ffLj3uc/c+0yYz5w8XVQVY4wxpiohwS6AMcaY2s+ChTHGmGpZsDDGGFMtCxbGGGOqZcHCGGNMtSxYGGOMqZYFC2OMMdWyYGGMMaZafg8WIhIqIstEZKqXax1F5EcRWSkis0Uk0e3aDSKy0fm5wd/lNMYYUznx9wxuEbkXSAbiVPWictc+Aaaq6r9FZDhwk6peJyLNgBTndQosAQap6mG/FtYYY4xXYf68uVNTGA08BdzrJUtv4B7neBbwhXM8EvheVQ859/keuACYXNmzWrRooUlJSb4puDHGNBBLliw5oKoJ1eXza7AAXgIeAGIrub4CuBx4GbgUiBWR5kA7YKdbvl1OWqWSkpJISUk54QIbY0xDIiLba5LPb30WInIRkKqqS6rIdh9wtogsA84GdgOFgHjJW6G9TERuFZEUEUlJS0vzRbGNMcZ44c8O7qHAGBHZBkwBhovIB+4ZVHWPql6mqgOBPztp6bhqEu3dsiYCe8o/QFUnqmqyqiYnJFRbizLGGHOc/BYsVHWCqiaqahIwFpipquPd84hICxEpKcME4G3neDowQkSaikhTYISTZowxJggCPs9CRB4XkTHO6TBgvYhsAFrh6gjH6dh+Aljs/Dxe0tltjDEm8Pw+dDZQkpOT1Tq4jTHm2IjIElVNri6fzeA2xhhTLQsWxhhTB2XnFbL9YHbAnmfBwhhj6qCb3lnM2c/ODtjzLFgYY0wdtGiba8xPdl5hQJ5nwcIYY+qYb1btLT3OKywOyDMtWBhjTB2Ssu0Qd3y4tPT8pw2BWb3CgoUxxtQh6TkFHuffr90fkOdasDDGmDpEyq2ctyUtMCOiLFgYY0wdIuXWWb14QNuAPNeChTHG1CEFRZ4d2o1CA/M1bsHCGGPqkNxyo59yCooC8lwLFsYYU4fklgsOeRYsjDHGlFd+XsXRfAsWxhhjyilfk8i2YGGMMaa8DfszPc4nL9rB+wtrtI32CbFgYYwxdcjHKbsqpL3/8za/P9eChTHG1BEjXpzjNT01M8/vz7ZgYYwxdYCqsmF/ltdr4jXVtyxYGGNMHVDV6rJSfg0QP7BgYYwxdcC3v+6t9FpEmP+/yi1YGGNMHXDPf1ZUes2aoYwxxlRwUb82AHRuEQNYM5QxxhggNSPX4/yMri2Asr0tdh/J8XsZ/B4sRCRURJaJyFQv1zqIyCzn+koRGeWkJ4lIjogsd37+6e9yGmNMbXX4qOeGR42cPoqD2fkBK0NYAJ5xN7AWiPNy7S/Ax6r6DxHpDXwDJDnXNqvqgACUzxhjajX3VqaBHZoQGuJKOLNbC+ZuPBCQMvi1ZiEiicBoYFIlWZSyIBIP7PFneYwxpi7Kdxs2++6NQ0pXns3OK+SmoUl88rvT/F4Gf9csXgIeAGIruf4oMENE7gRigPPcrnUSkWVABvAXVZ3rz4IaY0xt9ehXq0uP46LCSGwaDUBkeCiP/KZPQMrgt5qFiFwEpKrqkiqyjQPeVdVEYBTwvoiEAHuBDqo6ELgX+EhEKjRjicitIpIiIilpaWl+eBfGGBN8IW7tUCJlG6sWqwauDH6891BgjIhsA6YAw0Xkg3J5/gf4GEBVfwYigRaqmqeqB530JcBmoHv5B6jqRFVNVtXkhIQE/70TY4wJopF9WwMQHuo5RDaAscJ/wUJVJ6hqoqomAWOBmao6vly2HcC5ACLSC1ewSBORBBEJddI7A92ALf4qqzHG1GYl+26veGQEAEVOlAgLDcR0PJdAjIbyICKPAymq+hXwJ+AtEbkHV2f3jaqqInIW8LiIFAJFwO9U9VCgy2qMMbXBt6tcS32Eh7r+vu/eytUNPG5Ih4CVQTSQ9Rg/Sk5O1pSUlGAXwxhjfC7poWkAbH16lM9na4vIElVNri6fzeA2xphabnjPlkBglvWojAULY4ypRdKPFrBmT4ZHWnxUOB2aRQepRC4B77MwxhhTuasn/sy6fZn86fzu7EnP5enLTiInv4jI8OD+bW81C2OMqUXW7csE4PnvNzB50Q6KipXcwiKiwkODWi4LFsYYU4vlFBSRk19EhAULY4wxQOmaT+7yC4vJLSy2moUxxhj4fOkuev7vdxXS5206wIqdRzhyNHDLkXtjwcIYY2qBb3/d5zX9rsnLAFixKz2QxanAgoUxxtQCtX1+tA2dNcaYIPp580Ge/nYtK6upOTSLaRSgEnlnwcIYY4Lowc9WsuPQ0WrzHQrgFqreWDOUMcYEUV5hxRFQreIiKqQNaN8kEMWplAULY4wJojy3LVNLXHdqR5642HMHvHvOr7ClT0BZsDDGmCDyNrciJiKMqwd7Lj8eGWbLfRhjTIOVW1CxZhERFkqjsBAuPzmxNK2RBQtjjDHuIpzAcNPQpNK0fC/NVYFkwcIYY2qZCGeF2b7t4kvTkpOaBas4gA2dNcaYWqdRaNnf8VufHkVhsRIaEryNj8CChTHG1GoiQnhocAMFWDOUMcYETUGRZz9Ei8auWdrFtXDtDwsWxhgTJEfzPYfNRjVyLUNeXPtihQULY4wJlvJzLH4/rCtxkWGc0im4ndneWJ+FMcYESfmaxeldWrDy0ZFBKk3V/F6zEJFQEVkmIlO9XOsgIrOc6ytFZJTbtQkisklE1otI7fztGWOMF0kPTSPpoWlsPZBdZb6ccsEiPCz4HdmVCUQz1N3A2kqu/QX4WFUHAmOBNwBEpLdz3ge4AHhDRIK7p6Axxhyj8ZN+qfJ6TrlmKPchs7WNX0smIonAaGBSJVkUiHOO44E9zvHFwBRVzVPVrcAmYIg/y2qMMb5Q5NY73a5pVJV5525M8zgP9pIeVfF3n8VLwANAbCXXHwVmiMidQAxwnpPeDljolm+Xk2aMMbXavE0HSo8Hdqh8WfEtaVm89MNGj7TaHCz8VjIRuQhIVdUlVWQbB7yrqonAKOB9EQkBvDXcVRhMJiK3ikiKiKSkpaV5eYkxxgTWBwu3l51UMQQ2O6/iarMNtRlqKDBGRLYBU4DhIvJBuTz/A3wMoKo/A5FAC1w1ifZu+RIpa6IqpaoTVTVZVZMTEhJ8/w6MMeYYuW9SVFBUebQIcfv2/fR3p/HP8YMQaYAd3Ko6QVUTVTUJV2f1TFUdXy7bDuBcABHphStYpAFfAWNFJEJEOgHdgEX+KqsxxvhK0+iyvbJzveyCB3A0v5CHP19Vet45oTEX9G3t97KdiIDPsxCRx4EUVf0K+BPwlojcg6vCdqOqKrBaRD4G1gCFwO9V1ftv3RhjapHsvEIAmkaHVxgaW2LS3K2s2JVeeh7dqPYP9gxIsFDV2cBs5/ivbulrcDVXeXvNU8BTASieMcb4THa+K1g0bxxRabDYfTjH4zwyvPYHi9rbm2KMMXVMZm5B6QinmIiwCvMoSvRoXdkA0drLgoUxxvjIt6v2lR5HhYdUGizy3VabfWXcQL+XyxcsWBhjjI+o21jZ6EZhlTZDpecUlB6f3a1ujOS0YGGMMT7iPvQ1Kjy00ppFhluwiI8O93u5fMGChTHG+Eh+YVnzUmR4aLU1i9euqRtNUGDBwhhjfCYz1zUSau3jFxDdqIqaRW4h/ds34aJ+bQNZvBNiwcIYY3wkI7eAsBAhMjyEqEZV1yziIuvWdkIWLIwxxkcycwuIjQxDRFzNUAVFaLn9tAuLilmx80it3Ge7KhYsjDHGRzJzC4mLcnVYl8zKzi0o9sjz7oJtAMzfdDCgZTtRFiyMMcZHNqVmEeqMiIpyZmWX77d4clple8HVbhYsjDHGR1bvyWCLs5VqSbA45f9+8Jp34nWDAlYuX7BgYYwxPlDozMpOah4NQJTTDFXZMuVndGsRmIL5iAULY4zxgaU7jgAwrEdLoPpd76Ib2WgoY4xpcL5asRuADfszAeiSEFN6rfwQ2h6tbCFBY4xpkE7p1ByAh0f1AqBry7KA8OZPm0uPm8c0IjmpaWAL5wMWLIwxxgeOOvtYNHFb66mkBlGybDlAbkEREWG1f/+K8ixYGGOMD8xclwpA44iyvojIcjvgzVi9j+z8IuKj6sbige4sWBhjjA9MX70fcG16VKJ3G8++iVvfXwLAlgNZgSuYj1iwMMYYHwoPLftafeQ3fbzmKamF1CV1a+yWMcbUMtsPZpeuNlue+97ab8/bWnp834gefi+Xr1mwMMaYE3D2s7NLj8/qXvmud49PXVN6PP7Ujv4skl9YM5QxxvjIFYMSq81zx7AuhIZItflqGwsWxpgG6VB2Pue9MIdNqb7rbI4Orzgk9qNbTvE4H5zUzGfPCyS/BwsRCRWRZSIy1cu1F0VkufOzQUSOuF0rcrv2lb/LaYxpWGas3sem1CyecGseOlb70nM9zqMbVQwWcZGew2TT3fbfrksC0WdxN7AWiCt/QVXvKTkWkTsB9w1pc1R1gP+LZ4xpiPKdhf/mbEjjyalr+MtFvY/5HnM2eI5qivISLFrGRXicd3AWGqxr/FqzEJFEYDQwqQbZxwGT/VkeY4wp4T6CaZLbSKWaUlWenb7BI83b4oAtYyNpGesKGMN7tuTkDnVvqQ/wfzPUS8ADQHFVmUSkI9AJmOmWHCkiKSKyUEQu8WMZjTEN0K7DRz3OC4qq/Jqq4MjRAg5k5XmkeWuGAkoDRF3bd9ud34KFiFwEpKrqkhpkHwt8qqruSzN2UNVk4BrgJRHp4uUZtzoBJSUtLc03BTfGNAiTF+30OM/O8z5XojJHvPQ9eGuGck//YvmeY3pGbeLPmsVQYIyIbAOmAMNF5INK8o6lXBOUqu5x/t0CzMazP6Mkz0RVTVbV5ISEysc3G2NMdQ5l5x9T/tdmbqqQVlnNYtvB7OMqU23it2ChqhNUNVFVk3AFg5mqOr58PhHpATQFfnZLayoiEc5xC1yB5/iHLBhjjBvVirvX7UvPRVW9XvNmb3pOhbTISlaTXeZsjHTHsAoNJHVGjYKFiNwtInHi8i8RWSoiI47ngSLyuIiMcUsaB0xRz/+hXkCKiKwAZgHPqKoFC2PMCduXnsst76VUSH9i2lo6TfiG+z9dWaP7lMyXGNmnVWlaSCWT7f5+eT8AurVqfKzFrTVq2ttys6q+LCIjgQTgJuAdYEZNXqyqs3E1JaGqfy137VEv+RcAJ9WwbMYYUyNFxcqpT/9Yev73y/txWpfmnPn3WazdmwHAp0t28dyV/au9V25hEY3CQnjzumS+X7Of+ZsOVJr3yuREzuqeQOv4yBN/E0FS02BREi5HAe+o6goRqXvz1Y0xDdrKXUc8zgclNaVtk6gK+fILi6vdQzuvoJhIJ8/5vVtxfu9WleYVkTodKKDmfRZLRGQGrmAxXURiqWY4rDHG1HZNosK9rtP06ZJdXvMXFyt3T1nGsh2HyS0o8lhVtr6rabD4H+AhYLCqHgXCcTVFGWNMnZFb4Pk3bsmOdRf0ae2R/vB/V3nt6F645SBfLt/DpW8ssGBRidOA9ap6RETGA38B0v1XLGOM8b11+zI8zsOcjYqUioFh6Y7DgGvUU8kcjJIaR3xUOLkFxUSGN5y1WGv6Tv8BHBWR/rhmZG8H3vNbqYwxxsc27s/ksa+9D6os2RLVnYhQVKyc9vRMrpn0CwADOzQBYFiPBPIKi4ioZKhsfVTTYFHoDG29GHhZVV8GYqt5jTHG1Brnv/iTx3nP1mVfYSVBwJ1q2US9FTtdHeOFxa4ayJfL91BYrISFNpxxPjUNFpkiMgG4DpgmIqG4+i2MMabOWf3YSL78w9DS8ym3ngrAH87pWpqWV1BEeo4rWIQ7QWH+poOl1+duPEB4iDVDlXc1kIdrvsU+oB3wrN9KZYwxPpbY1DVE9pnLTiImIsyjCSkiLJRtz4zmvpFle2PnFhZx5Khr/acopyP7h7WezVUZuXVzb4rjUaNg4QSID4F4Z4HAXFW1PgtjTJ0RGR7KqJNaM3ZIhyrzffF7V40jJ7+YNc5EvYzcQj5J2cnok9p45F23L9M/ha2Farrcx1XAIuBK4CrgFxG5wp8FM8YYX8ktKGJTahYLNh+sNm+z6EaAawnzv365ujT9/k9XsutIDj1aNczu2prO4P4zrjkWqQAikgD8AHzqr4IZY4yv7M9wbX9asglRVZo3dgWLp79dV+FaSUd3Q1TTPouQkkDhOHgMrzXGmKDKcuZJ3Ht+j2pyQkxEzTcomnBhz+MuU11T0y/870RkuojcKCI3AtOAb/xXLGOM8Z3sPNe+ao2PIRBUZmjX5qXHdX29p2NR0w7u+4GJQD+gPzBRVR/0Z8GMMcZXsvJco5ZiIk58El2/xLI5GVG23EdFqvqZqt6rqveo6n/9WShjjPGlrGOsWax6tPLtetwDRGXbqNZHVQYLEckUkQwvP5kiklHVa40xprbIdOZDxEbWbC5xbGR4pUuOu68H1Tqu4TRDVRlmVbVhjhEzxtQrWbmuDu7GkTXvs3jwgp6kZubRrWVjjyXL3VeabdG4+tFV9YWNaDLG1HtZeYWIQMwxNBt1bdmYL38/lOSOTT3S3ffZtmYoY4ypRzJzC2kcEYYvNviMjy5ryoqoZje9+qThvFNjTIOVmVtI7HEOmx0zoC2XDGhb2rHt3vTUkHaXtmBhjKn3MnMLaty5XV50ozBeGjuwdCHCVnERvDJuIFcOSvRlEWu9E5+hYowxtVhuQREz1uync4uYE7rPy2MHMnXlHto1iSKxaTRj+rf1UQnrBgsWxph67avlewDYciD7hO7Tu20cvdvG+aJIdZLfm6FEJFRElonIVC/XXhSR5c7PBhE54nbtBhHZ6Pzc4O9yGmPqp5Ld7cyJCUTN4m5gLVAhJKvqPSXHInInMNA5bgY8AiQDCiwRka9U9XAAymuMqUcUCxa+4NeahYgkAqOBSTXIPg6Y7ByPBL5X1UNOgPgeuMA/pTTG1GfPz9gAwHNX9g9ySeo2fzdDvQQ8ABRXlUlEOgKdgJlOUjtgp1uWXU6aMcYck5LFA3/Tv001OU1V/BYsnO1XU1V1SQ2yjwU+VdWikpd7yVOhLikit4pIioikpKWlnUBpjTG1TXGx8sCnKzjvhTmoHntT0oqdR+j+52+JiwynbXykx57b5tj5s2YxFBgjItuAKcBwEfmgkrxjKWuCAldNor3beSKwp/yLVHWiqiaranJCQoJvSm2MqRX+MWczH6fsYlNqFoey84/59Re/Pp/8omJW78k4pjWhjHd+CxaqOkFVE1U1CVcwmKmq48vnE5EeQFPgZ7fk6cAIEWkqIk2BEU6aMaaBeHb6+tLj2etPrOXgeCfkmTIBn8EtIo+LyBi3pHHAFHWrZ6rqIeAJYLHz87iTZoyph3ILivhly8HS81W70j2uxzo1g2enr+Otn7bU6J6dE8om4cVazeKEBeQ3qKqzgdnO8V/LXXu0kte8Dbzt56IZY2qBPo9Mp6hYadckivkPDee/y3Z7XP91TwZn90jg9VmbAfifMzoRElLNukxu3RxWszhxtjaUMSag0nMKWOhWiwAocibO7T6Sw85DR4l2lv5e8NBwAF75cSM7D+WU5j+Qncc787eS9NA0Coq8D7ZMzykoPfbF3tsNnf0GjTEB9ccpy5i1Po0Vj4wgPqriX/xn/n0W4NqFLiG2bIXXP3y0tPT4rL/PIrfAFSSycgtpGtPI4x6qypGcAkRAFRqFNpzVYf3FahbGmIBavy8TgH3puQBsSs30mi8jt4Dw0BCuTnYNjFy3ryxfSaAA2JyW5eW1hRQVa+m2pzaH+8RZsDDG+E1qRi6TF+3wSItw9oXYfeQoAF8sc42K79XGc0WgUKdPYnCnZqVp3raPuOKfroGUe47kcMt7KSQ9NI3+j80AoJGzOZF7k5Q5PtYMZYzxmyH/9yMAZ3RtwTer9nIoO5+tzuqvOw66gkXXlo0B+Pvl/Rj31kKy8lz7ZXdoFg14jmpqGRvB/ow8r886/ZmZFdLeuj6ZJ6etZcKFvXz0jhouCxbGGL/Idr70AWas2c/T367zuL79kCtYLN3hWh+0dXxk6bVze7bkyUv7AnByh6ac0bUF8zYdoHurWK/BorJJe91bxfLezUNO7I0YwJqhjPGbA1l53P7BEtIyvf8lXN+t2l02V+KJqWsqXH9n/jYA3vt5O+CaC1HSyvTomD60iY8qzTush2uFhryCYrY9M7o0PTRECBE49/nZFe7/4AU9T/AdGHcWLIzxg4KiYp6bvp5vf93H67M2Bbs4QTF/04FKr5X0Jahq6d7WkeGh/PO6QYzo3Yo2brUMgIEdmgKweo8rAG17ZjTbnhnNfSN6UKxw+GjFPonTujT3yfswLhYsjPGxomLlDx8tZcpi18LJ7y7Y1iBrF7kFRZVeu2t4VwB2Hsohp6CICRe6agFDu7Zg4vXJhIV6fjX1befq/HYfSgtlK8qWmHHPWaXHNrfCtyxYGHOCtqRleYy2+eiX7UxfvZ+OzaO57ezOACze1vBWqzmQlU9i0yheu2Ygv+nflvtH9gBg0vXJxEe75kWU9FeUdGZXJiIslInXDeKD357ikb54m+d+aN1bxZYeN4m2Wdu+ZKHXmBMwZ0MaN76ziKjwUD665VTaNYniw1920L1VY2bcczYFRcV8uHAHXy7fzaiTarafgqryScouerSOpX/7Jn5+B/5zICuP5o0juKhfWy7q1xZVZXjPlvRqE8fnS3cB8Mf/LAcgvgZf7CP6tK6QNm5Ie75e4bkgdaPQEPKLimka3ahCfnP8rGZh6i1VZfKiHV4nbfnKP2dvJio8lKP5Rfxn8Q6uf3sRW9Ky+f05rmaW8NAQrjmlA9+v2c/R/MJq7uYye30aD3y2kvs+WeG3cp+Ijfsz+dXpvC4sKubuKctK+xLcHcjKJ6Fx2Re2iJTOpejTNt4jr7eZ3DUxqGPT0uOSe0+96wz+dvlJpfM0jG9YsDD11rKdR5jw+SrOfX4Ouw4f9fn996XnsnDrQW49qzPn9WrJ5EU7Wbs3g4cu7MnFA8o2dkzu2JRihd5/nc6Y1+Z5Xcsov7C4tI1/2qq9AOxNzz2uTX/87fwXf+KiV+cBsOPQUb5cvofRr8zz6KPIyS9ic1oWHZvHeL1Hx+aezU5JleSrjvuGRv+57VTA1RR19eAOx3U/UzkLFqbeet8ZkhkicOfkZcf9xauq/GfxjgrNHVNX7kEVLhnQjscu7sttZ3Xmt2d04qrB7T3yDerYtLSzdeWudH7e7LmIHsBv30vhjL/NYu7GND5d4mqiycorZMHmg6gqj329ml+2HPT6Hg5n5/Nxys7SmkvSQ9P408f+r5V8u2qvxxIc7u8rZfsh8guLOau7903JIsND2fjUhaXnMSfQGb3q0RFseupC4mxlWb+yPgtTL70zfyv/Xbab287qTOeEGB78bBVzNqQxrEfLY77XTxsP8OBnqwAY0adV6V+zKdsOk9Q8mqQWrr+KJ4zyPku4eeMIfn1sJEfzC+n7yHRSth/mlM7N+GDhDrYeyOLUzs35aYNrc5/r/rUIgCcv6ctfvviVDxZuJ7+omHfmbyudl3Ba5+Z8+NtTSpfoHvjE9wCs2eOq1QB8tnQXz1/V/5jfa3Xcg9XtHy4tnX0N8MbsTZzT0/X73X3YtUKs+/XywkND+P05XRjS6cSGuNry44FhwcLUO18u381jX69hRO9W3D/SNQ7/xe838s78bccVLL51moUA5qxPY0Sf1izbcZjvVu9jRO9WNb5PdKMwEptG88qPG3nlx42l6R8s9Fw7adyQ9ow/tSOz16fy65501n3tudDez1sOsmDzQc7o1sIj/d0F2yo07/haRo5nv8um1CzCQ4WCImXV7nSSHprGZ7efzidO7SihcYS325S6f6RNnKsrrBnK1CupGbnc/8lKhnRqxivjBhIWGkKjsBAuH9SOuRvTSM3MPab7ZecV8s2qvYw+qQ3NYhrx1Yo9qGrplp/lv7Crc8nAsr6M0Se14b4R3QG4c3hXbhqaBEAPZ/jnSe2asPNQDlsPZHPtKR2IjQhj6p1nEBcZ5rE5UEu3uQePfV02U/pY32tN7M1w1RhKZlQDXDEokUsGtC1dCXbS3C0s2e4a0loy+c7UfVazMPXK9NX7yC8q5slL+hIZXtb5eenAdrw+azMzVu9n/Kkda3y/aSv3kpFbyM1ndCIhNoJ3F2xjzvo0MvMKGXVSa8afUvN7Adx7fnf6to2jU4sYurZsTLHC1YM7kBAbweHsfAqLlMsGJQLQq03ZnIGHR/XiyUv6IiKc2T2Bz5bu4pKBbRnSqRmplUz4W7r9MBf0rdlwXXfr9mWwfMcRxg6p2Em811lW/JYzO7NyVzrhocJjY/ry3Iyy/bK//XUfABcPaHvMzza1lwULU69MXbmXzgkxdCvXVt4loTGt4iL4ZeuhGgeLlbuO8MBnK4mNDOPkDk1o2ySSH9bup1OLGDq1iOHmoTXY2tML9/kCoVI2K7lpTCOeuKRv6bVTOrva8k9qF+/RAVzSkXvdvxaVTvp77sr+dE6I4bI3FnDJgLZMW7WXpTuO0DmhMV0TGte4nEXFygUvzQXgrO4JtIqL9BiCustZ/K9TixiW/u/5Vd7rfy/qXaNnmrrBgoWpN2au288vWw8x4cKeSLmND0SEUzo1Z+GWgxQUFRMeWn3zyKszXWs6XX5yIiJCm/go5j043C9l9yY+Kpyf7j+nwoS1O4Z1Kd0j4s05WwAY2KEJXRIas+jhc4mLCmdTWhYTf9rCxJ+28NSlfbm2hjWguRvTSo9Pf2Ym44a05+nL+pWmzV6fRtv4yAprN7Utdw7YpLh6xhoUTb3x8g8b6dwihpuGdvJ6fWSf1qRm5vHgZyurvVduQRG/bDnIuT1blo4wCoYOzaMrTFhr3yyad28aXHp+4+lJdElw1aRaxkUSGR5Kj1ZlGwn9e8G2Gj8sPxN9AAAYR0lEQVRv1+Ecj/PJi3Z6nG87mE3fdvEVgvENpycx9c4zeOv65NI0mxRXv1iwMHVeyrZD9Prf71ixK51rT+1Yaafq6H5tuOG0jny+dDfbnA14vMkvLOaPU5aTkVvILWd19uj7qC1O69KcW87sRMpfzuPRMX0qXO/RuqwZbsP+LJ6cuobi4urnmRw56toX4tTOZbvTZeS61r06lJ3P5rRsmnsZ4SQi9G0Xz5CkZsQ0Ci1dKNDUHxYsTJ1UMt4/PaeAP3y0jJyCIsJDpdpO1ZKZvcOem82Pa/dXuOekuVv423fr+G71Pu46txundq6dy1xHhIXy59G9aVHJ0NQOzTxnRE+at5WJc7dUe99D2QU0jghjyq2n8eq4gQC8M28bULY/hXsgKS8+Opzlj4zgnvO71+RtmDrE730WIhIKpAC7VfUiL9evAh7Ftaf6ClW9xkkvAlY52Xao6hh/l9XUDarKbe8vYf3+THLyi0jLyuOz208jsWl0pV+eJXq3jeOFq/rz/IwN3PvxCr6/9yxaxkby+w+XsmjbodKlxLskxHDPed0C8Xb8omWc6/fQJDqcK05OZNK8rTw/Yz192sZxZjfvs6oBDh/Np2mMq9nrdGc/iBd/2MDd53Uj06lhlN8ru7ya9AeZuicQHdx3A2uBCp8wEekGTACGquphEXGfMZWjqgMCUD5Tx3y9ci8z1pTVCsYN6cCgjpX/tVveZScn0i+xCSNenMNdk5excEvZ8uFhIcLIvq259czOFdrl65LWca4O56uS2/PwqF7M23SAdfsyue5fizx2mgNYsfMI+zNyOb93Kw5k5ZV2TDdvHMGQTs3YkuZqsvt1dwZg+0Q0VH79XxeRRGA08BRwr5cstwCvq+phAFVN9Wd5TN2XfrSAx79eTf/EeD6/Yyh7juTQKq7iSJzqdG3ZmJPaxXsEiuE9W/Lob/rQwc+zoAOhbZMoZv7p7NIF+lw1LtdM8Lkb0zizWwJ703OIbhTGxa/PB+Ce87qz/eBRTkosWxH2lE7NWLTVtc7T2/O2Ase/Qqyp2/z9J8JLwANAbCXXuwOIyHwgFHhUVb9zrkWKSApQCDyjql/4uaymDnh7/lYOZefz7k1DCA0R2lezaU5VuraMZcWudF67ZiBndk2o0Z4KdUnnhLJO7pfHDmDaqr389cvV/LQhjY7NYjjr2Vn0aVtW4X/xhw0AHkuYlGzq1P0v33JmtxYs2HzwhBb9M3WX3xoXReQiIFVVl1SRLQzoBgwDxgGTRKRkt5cOqpoMXAO8JCJdvDzjVhFJEZGUtLS08pdNLZKTX8Ts9SdecZyzIY0B7ZvQt1189Zmr8eCFPZhwYU8u7Num3gWK8po3juD605Lo2y6OdfsymbLYNU9j9R5X01L3VmWBpbdbAHEfMHAwK58hSTVv7jP1iz97ooYCY0RkGzAFGC4iH5TLswv4UlULVHUrsB5X8EBV9zj/bgFmAwPLP0BVJ6pqsqomJyRU3mlngu/VmRu58Z3FpJzA9qLpOQWs3HWEM7oe23pMlWkZG8ltZ3dpUPMBerWOY+3eDKav3ueR/sJVZd2DF/UrCxCDOjbj+Stdq9duSssiLspqFQ2V34KFqk5Q1URVTQLGAjNVdXy5bF8A5wCISAtczVJbRKSpiES4pQ8F1mDqpNyCIqYsdk3uen/h9uO+z8+bD1KsMNRHwaIh6tkmjgNZrvkS7n0PfdrGER4qhIZIhXkqJaOj8gttq9KGLOBj3ETkcREpGQY7HTgoImuAWcD9qnoQ6AWkiMgKJ/0ZVbVgUUd9sWw3h7Lz6ZcYzzer9pYOTy3xw5r9jHhxDqkZVa+SOn/TAaIbhTKwQ9Mq85nKdXTr47lvZA+GdGrGAxf0QERY8NC5/PxQxeVMmrgFiCYWLBqsgAQLVZ1dMsdCVf+qql85x6qq96pqb1U9SVWnOOkLnPP+zr//CkQ5je+98uNGHv7vKvq2i+OFqwZQWKy8u2CrR55//7yNDfuzmPD5qgqzjFWVX3enU1SszN90gFM6NbNlr09A2yZRpcctYhrx8W2ncccw12zrhNgIWnoZWdbUI1jU774dUzlrgDR+8+Xy3bzw/QYuHtCWJy7pS1xkOKP6tuHfC7bTvVUsPVvH0bxxIxZsPkjnFjH8uC6VS96YjypcfnI7xg7pwAcLt/PktLW0jotkX0Yu1x7D8uKmonZuwcJbYPCmmXuwsGGzDZYFC3NC8gqLeOH7DcRFhvP7c8rWA1q1K52HP1/F4KSmPH9lf8KcWb13nduNaav2cveU5SQ1j2b8qR0pKlZev/ZkUrYf5l9ztxDVKIxHv17DczM2kFNQxID2Tdi43zVH4OxK9nQ2NePeQX1yhyZV5PT+mtZeVpc1DYMFC3PcUjNzufW9JSzfeQSAIZ2aMTipGQs2H+CWf6fQJLoRL48dWBooAHq0juWdGwezek86z83YwNPfrmNIp2b0bB1LrzZxXHdqR1SVBZsPMnXlXuIiw7h9WBcaR4SRmpnn0Yxijp2I8O3dZ9I6LrLGM9Td8/VPrFmAMfWPuG/AXpclJydrSkpKsIvRYGw9kM31b//Cgcx8nrq0L8/P2EBkeAiXnZzIyz9sJKlFNO/dfEqlf4kWFysXvjyX7Yey+fbus+jUIsZrPlM7fLViD/vTc7nlrM7BLorxMRFZ4sxpqzqfBQtzrFSV0a/MY19GLm/fOJgB7Zswe30qt72/hLzCYk7u0IS3bxxc7ciZ3UdyOJyd75MJdsaY41PTYGHNUOaYzVqfypq9GTx3ZX8GtHc1Swzr0ZIVj4zgUHY+reMia7SNZ7smUR4drsaY2suChTkmqsqrMzeR2DSqwt4RkeGh1qdgTD1lA9ZNjR3Mcm1JumzHEW47u4vtW2BMA2I1C1Mjuw4f5cp//kxaZh63ndWZcYPbB7tIxpgAsmBhqpWamcv4Sb+QnVfIf+8Y6rHfgTGmYbB2BOPhUHY+e47klJ4XFBVz+wdL2Z+Rxzs3DbFAYUwDZcGiErPWp/L379aRW1AU7KIE1P2frODi1+eTnVcIwNPfrGPJ9sP87Yp+DOpoC/gZ01BZM5QX36zay52Tl1FUrPy0MY03r0tuEEM8c/KLmLvpAPmFxfxr3lY6J8Tw9vyt3Hh6EmP6t63+BsaYestqFuVMXbmHOycvY0D7JrwybiDbDxxlzKvzWLjlYLCL5ncLtxwkv7CYdk2ieHPOZh78dCUnd2jCw6N6Bbtoxpggs2Dh5qsVe7h7ynJO7tCEf988hDH92/LFH4YSHx3OtZN+4d8LtlFfZrx7M3t9KpHhIbx1fTK5hcVEhofyxrWDbElwY4wFixJfLNvNH6csY1DHprx70xAaO5vSd0lozBe/H8o5PRJ45KvVPPRZxT0X6ovZG9I4rXNzereN463rB/HRLafaKqPGGMCCBQCfL93FvR8v55ROzXn3psHERHh25cRFhjPxumTuGNaF/6Ts5PVZm4JUUv/ZeiCb7QePMqxHSwCG92xFj9axQS6VMaa2aPAd3JtSs7jvkxWc1qU5k64fTFSjUK/5QkKE+0f2YM+RHF74YQN9E+M5x/lirasOZ+czZfFOZq1PZW+6a7hsXX9Pxhj/aPDBomvLxrx+zcmc07MlkeHeA0UJEeHpy/qxfn8Wd09extd3nkHH5nVzae15Gw/w2/cWk1tQTL/EeHq0iuPSAe3o0Dy6+hcbYxocW6L8OOw4eJTfvDaPNvGRfH7H6UQ3qlsxt6ComJEv/YQq/GP8yfRsHRfsIhljgqSmS5Rbn8Vx6NA8mpfHDmD9/kwmfL6qzo2QmrJoB1vSsnl4VC8LFMaYGrFgcZyG9WjJfSN68OXyPbzw/YY6M0IqI7eAF3/YyKmdm3FeL+ufMMbUTN1qP6llbj+7C1vSsnl15iZW7U7nhasG0Cym6t3hgu0fszdzKDufv4zuXeM9mI0xxu81CxEJFZFlIjK1kutXicgaEVktIh+5pd8gIhudnxv8Xc7jERIiPHdlP564pC8LNh1k9CtzWbL9ULCLVandR3L417ytXDawnW1laow5JoFohrobWOvtgoh0AyYAQ1W1D/BHJ70Z8AhwCjAEeEREauUqdiLCdad25PM7Tic8NISr31zICzPWsyUtK9hFq+DZ79YhwH0jewS7KMaYOsavwUJEEoHRwKRKstwCvK6qhwFUNdVJHwl8r6qHnGvfAxf4s6wnqm+7eKbedQYj+7TmlZmbGP78HIY/N5unpq1hxc4jwS4eK3cd4Yvle/jtmZ1s61NjzDHzd83iJeABoLiS692B7iIyX0QWikhJQGgH7HTLt8tJ8yAit4pIioikpKWl+bLcxyUuMpzXrz2ZuQ+cw2Nj+tCuaRTvLtjGxa/P549TlpGakRu0sr05ZwtNosP53dldglYGY0zd5bcObhG5CEhV1SUiMqyK53cDhgGJwFwR6Qt463mtMNxIVScCE8E1z8IHxfaJ9s2iueH0JG44PYnM3AIm/rSFN+ds4Ye1qfzxvG7ccHpSQPevPpiVx4w1+7j+tCRiI8MD9lxjTP3hz2+socAYEdkGTAGGi8gH5fLsAr5U1QJV3QqsxxU8dgHumzwnAnv8WFa/iY0M508jejDjnrMYnNSUJ6et5fYPlgR0bsZ/l+2moEi52vbNNsYcJ78FC1WdoKqJqpoEjAVmqur4ctm+AM4BEJEWuJqltgDTgREi0tTp2B7hpNVZSS1iePvGwTw8qic/rE3lvZ+3B+S5qsp/Fu9kYIcmdG9lCwMaY45PwCflicjjIjLGOZ0OHBSRNcAs4H5VPaiqh4AngMXOz+NOWp0mItxyZmfO6ZHAU9+sZf2+TL8/c9nOI2xMzWKs1SqMMSfA1oYKgrTMPC58+Seax0Tw5R+GVruAYU3NWL2P12dt4sELe3J6lxbk5Bdx2wdLWLLtEIv+fF6FpdeNMcbWhqrFEmIjePaK/qzfn8kz367zyT2/X7OfOz5cyuo9GYyf9At//24dl74xn7kb03jggp4WKIwxJ8S+QYLknJ4tufH0JN5dsI2YiFDuPb8HoSHVL78xe30qRcVKq7hI4qPC2Z+Ry+o9GTw5bQ192sXz5vhB/N83a3lj9mbio8J596YhnN09IQDvyBhTn1mwCKKHR/Uit6CI12dtZuWudF4ZO5CmVawttWjrIW58Z7HXa/0T43nv5iHER4Xz8tgBXDygLT3bxNHOJuAZY3zA+ixqgcmLdvDIl6tpGRfBb/q3JSo8lOhGoYzs05r2zco2Ixo/6RfW7cvgzesGkZaZR3pOAa3iImnXJIpOLWIIC+DcDWNM/VDTPgurWdQC44Z0oFebOO79eDmT5m6hoMgVwN+et5Wpd51Js5hGLNl+mHmbDvDwqJ4M6tgsyCU2xjQ0FixqiQHtmzDzT8MA1052v+5O5+qJC7lr8jL+ffMQXp25kWYxjbj2lI7BLagxpkGydotaKDw0hIEdmvLkJX2Zt+kAd3y4hNnr0/jtmZ1sVJMxJijsm6cWuyq5Pct2HGHyoh00iQ7n+tOSgl0kY0wDZcGilnt0TG8ycwsY1qMlja1WYYwJEvv2qeUiwkJ57ZqTg10MY0wDZ30WxhhjqmXBwhhjTLUsWBhjjKmWBQtjjDHVsmBhjDGmWhYsjDHGVMuChTHGmGpZsDDGGFOterNEuYikAxuP8WXxQLoP81aV53iutQAO1Kh0wXEsv79g3f947lHT15zoZ6K66/a58M+9/fmZqGne2vRd0VFVq98hTVXrxQ8w0Z+vqUneqvIczzUgJdi/V1//zgN9f39+Lk70M2Gfi/r3mfDF56K2fibqUzPU135+TU3yVpXneK/VZv4uty/u78/PxYl+Jqq7bp8L/9zbviuOQ71phqqPRCRFa7CDlWlY7HNhygvEZ6I+1Szqo4nBLoCplexzYcrz+2fCahbGGGOqZTULY4wx1bJgYYwxploWLIwxxlTLgkUdJiIxIrJERC4KdllM8IlILxH5p4h8KiK3B7s8pnYQkUtE5C0R+VJERhzvfSxYBIGIvC0iqSLya7n0C0RkvYhsEpGHanCrB4GP/VNKE0i++Eyo6lpV/R1wFWBDa+sBH30uvlDVW4AbgauPuyw2GirwROQsIAt4T1X7OmmhwAbgfGAXsBgYB4QCT5e7xc1AP1xT/COBA6o6NTClN/7gi8+EqqaKyBjgIeA1Vf0oUOU3/uGrz4XzuueBD1V16fGUJey43oE5Iar6k4gklUseAmxS1S0AIjIFuFhVnwYqNDOJyDlADNAbyBGRb1S12K8FN37ji8+Ec5+vgK9EZBpgwaKO89F3hQDPAN8eb6AACxa1STtgp9v5LuCUyjKr6p8BRORGXDULCxT1zzF9JkRkGHAZEAF849eSmWA6ps8FcCdwHhAvIl1V9Z/H81ALFrWHeEmrto1QVd/1fVFMLXFMnwlVnQ3M9ldhTK1xrJ+LV4BXTvSh1sFde+wC2rudJwJ7glQWUzvYZ8J4E5TPhQWL2mMx0E1EOolII2As8FWQy2SCyz4TxpugfC4sWASBiEwGfgZ6iMguEfkfVS0E/gBMB9YCH6vq6mCW0wSOfSaMN7Xpc2FDZ40xxlTLahbGGGOqZcHCGGNMtSxYGGOMqZYFC2OMMdWyYGGMMaZaFiyMMcZUy4KFCRoRyQrAM8bUcLl3Xz5zmIicfhyvGygik5zjG0XkNd+X7tiJSFL5JbK95EkQke8CVSYTeBYsTJ3nLNnslap+parP+OGZVa2rNgw45mABPAy8elwFCjJVTQP2isjQYJfF+IcFC1MriMj9IrJYRFaKyGNu6V84uwGuFpFb3dKzRORxEfkFOE1EtonIYyKyVERWiUhPJ1/pX+gi8q6IvCIiC0Rki4hc4aSHiMgbzjOmisg3JdfKlXG2iPyfiMwB7haR34jILyKyTER+EJFWznLSvwPuEZHlInKm81f3Z877W+ztC1VEYoF+qrrCy7WOIvKj87v5UUQ6OOldRGShc8/HvdXUxLWb4jQRWSEiv4rI1U76YOf3sEJEFolIrFODmOv8Dpd6qx2JSKiIPOv2f3Wb2+UvgGu9/gebuk9V7cd+gvIDZDn/jgAm4lpNMwSYCpzlXGvm/BsF/Ao0d84VuMrtXtuAO53jO4BJzvGNuDYCAngX+MR5Rm9cewIAXIFrSe8QoDVwGLjCS3lnA2+4nTelbBWE3wLPO8ePAve55fsIOMM57gCs9XLvc4DP3M7dy/01cINzfDPwhXM8FRjnHP+u5PdZ7r6XA2+5nccDjYAtwGAnLQ7XCtTRQKST1g1IcY6TgF+d41uBvzjHEUAK0Mk5bwesCvbnyn7882NLlJvaYITzs8w5b4zry+on4C4RudRJb++kHwSKgM/K3edz598luPZ18OYLde39sUZEWjlpZwCfOOn7RGRWFWX9j9txIvAfEWmD6wt4ayWvOQ/o7dqDBoA4EYlV1Uy3PG2AtEpef5rb+3kf+Ltb+iXO8UfAc15euwp4TkT+BkxV1bkichKwV1UXA6hqBrhqIcBrIjIA1++3u5f7jQD6udW84nH9n2wFUoG2lbwHU8dZsDC1gQBPq+qbHomuzXzOA05T1aMiMhvXNrIAuapaVO4+ec6/RVT+2c5zO5Zy/9ZEttvxq8ALqvqVU9ZHK3lNCK73kFPFfXMoe2/VqfGCbqq6QUQGAaOAp0VkBq7mIm/3uAfYD/R3ypzrJY/gqsFN93ItEtf7MPWQ9VmY2mA6cLOINAYQkXYi0hLXX62HnUDREzjVT8+fB1zu9F20wtVBXRPxwG7n+Aa39Ewg1u18Bq5VQgFw/nIvby3QtZLnLMC1DDW4+gTmOccLcTUz4Xbdg4i0BY6q6ge4ah4nA+uAtiIy2MkT63TYx+OqcRQD1+Ha07m86cDtIhLuvLa7UyMBV02kylFTpu6yYGGCTlVn4GpG+VlEVgGf4vqy/Q4IE5GVwBO4vhz94TNcG8r8CrwJ/AKk1+B1jwKfiMhc4IBb+tfApSUd3MBdQLLTIbwGV/+CB1Vdh2vby9jy15zX3+T8Hq4D7nbS/wjcKyKLcDVjeSvzScAiEVkO/Bl4UlXzgauBV0VkBfA9rlrBG8ANIrIQ1xd/tpf7TQLWAEud4bRvUlaLOweY5uU1ph6wJcqNAUSksapmiUhzYBEwVFX3BbgM9wCZqjqphvmjgRxVVREZi6uz+2K/FrLq8vwEXKyqh4NVBuM/1mdhjMtUEWmCq6P6iUAHCsc/gCuPIf8gXB3SAhzBNVIqKEQkAVf/jQWKespqFsYYY6plfRbGGGOqZcHCGGNMtSxYGGOMqZYFC2OMMdWyYGGMMaZaFiyMMcZU6/8BYcxvhokpKrYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8795d346e02748f78ce46d5c87bff058",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                     \n",
      "    0      4.332359   4.120674   0.289563  \n",
      "    1      4.247177   4.067932   0.294281                     \n",
      "    2      4.175848   4.027153   0.298062                     \n",
      "    3      4.140306   4.001291   0.300798                     \n",
      "    4      4.112395   3.98392    0.302663                     \n",
      "    5      4.078948   3.971053   0.304059                     \n",
      "    6      4.06956    3.958152   0.305356                     \n",
      "    7      4.025542   3.951509   0.306309                     \n",
      "    8      4.019778   3.94065    0.30756                      \n",
      "    9      4.027846   3.931385   0.308232                     \n",
      "    10     3.98106    3.928427   0.309011                     \n",
      "    11     3.97106    3.920667   0.30989                      \n",
      "    12     3.941096   3.917029   0.310515                     \n",
      "    13     3.924818   3.91302    0.311015                     \n",
      "    14     3.923296   3.908476   0.311586                     \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3.9084756, 0.3115861900150776]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.fit(lrs, 1, wds=wd, use_clr=(20,10), cycle_len=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the trained model weights and separately save the encoder part of the LM model as well. This will serve as our backbone in the classification task model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.save('lm1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.save_encoder('lm1_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VFX6B/Dvmw4kIYGEGiD0ptTQexEQWHQVFVdW1p8uirhiRbCwig3RdVlXV8G2dlBQdEFASkAEBULvPRBaCEECIaTO+f0xdyYzmT6Znu/neXi45cydM9zw5sy557xHlFIgIqLQEubvChARkecxuBMRhSAGdyKiEMTgTkQUghjciYhCEIM7EVEIYnAnIgpBDO5ERCGIwZ2IKARF+OuNk5KSVGpqqr/enogoKG3duvWCUirZUTm/BffU1FRkZGT46+2JiIKSiJxwphy7ZYiIQhCDOxFRCGJwJyIKQQzuREQhiMGdiCgEMbgTEYUgBnciohAUdME9N78Iy3af9Xc1iIgCWtAF93s/ycCkL7bh96vF/q4KEVHACrrgfur3AgBAqY4LexMR2RJ0wX1s10YAgOjIoKs6EZHPBF2ErF0jCgAgfq4HEVEgC7rgLozqREQOBV1wN2CPOxGRbUEb3ImIyLagDe6KTXciIpuCLrgLO92JiBwKuuBuxJY7EZFNQRfc2W4nInIs6IK7gWLTnYjIpqAL7oYu92slZf6tCBFRAAu64H4iV59bZvq3u/1cEyKiwBV0wT2/qBQAcDg73881ISIKXEEX3A0PVHUc6E5EZFPwBXctujO2ExHZFnzBXWu7n7tciMXbT/u5NkREgcnp4C4i4SKyXUSW2CkzVkSUiKR5pnr2PbJghy/ehogo6LjScp8CYL+tkyISB+BhAJsqWyl7FmRkefPyREQhwangLiIpAEYB+MBOsRcBzAZQ6IF6ERFRJTjbcp8DYCoAnbWTItIZQCOllM0uG63cRBHJEJGMnJwc12pqQ2FJGVo+8yOW7DrjkesREYUCh8FdREYDOK+U2mrjfBiAfwJ43NG1lFLzlFJpSqm05ORklysLAOFh5tllTuQWoKRMYfbyg25dj4goFDnTcu8DYIyIZAKYD2CwiHxucj4OwHUA1mplegL4wVsPVcf3aGy2//BX2wEAvxcUe+PtiIiCksPgrpSarpRKUUqlAhgHYI1SarzJ+TylVJJSKlUr8xuAMUqpDG9V2tTB7CsAgIJi5pohIjJwe5y7iMwUkTGerIwzdDYmLynOaiIiMopwpbBSai2Atdr2DBtlBla2UnbrYCPVr04BOp1CWBgzvhMRBd0MVXsN9LfTj/iuIkREASzogrutbhkAWL0/23cVISIKYEEX3O0tnrrzVJ4P60FEFLiCLrjzuSkRkWNBF9yZx52IyLGgC+6lZQzuRESOBF1wLy6zmt7G6Pa5v7p0vc3HL+Jw9hXkXCmqTLWIiAKKS+PcA0GJg+C++fhFp6915tI1s18GmbNGuV0vIqJAEnQt90aJ1T12LXbwEFGoCrrg/uSI1nh2VFusnzrIrddvybyI1GlLcS6vEJzLSkShKuiCe3REOO7r1wyNarnXgv/s1xMAgAkfbUaZvRlRRERBLOiCuzN+3H3W5jnRmusHs69g1rIDPqoREZFvhWRwf/CLbcbtHVmX8PLSfcaskWFS3hmztMIvgUvMCU9EISKog3vz5BoOy9z8zga8v/64cV/sdLR3mrnSE9UiIvK7oBsKaWrx5D7Iu1aCvq+lOyx7Jq8QpWU6fLvttA9qRkTkX0Ed3ONiIhEXE2n1XG5+EWrHRhv3+8xa46tqERH5XVB3yxi0qhtrcazrS6uw280skf9adRhz1x2tbLWIiPwmJIL78in9rR7/w9u/uHwtpRT+ueoQXuVIGiIKYiER3MPCBCmJ1TxyrcuFpR65DhGRP4VEcAeA/z3U1yPX6fjCTx65DhGRP4VMcE+sEYXb01L8XQ0iooAQMsEdAGaP7ejR6x3KvuLR6xER+YrTwV1EwkVku4gssXLuARHZLSI7ROQXEWnn2Wr6x7B//uzvKhARucWVlvsUAPttnPtSKXW9UqoTgNkA3qx0zYiIyG1OBXcRSQEwCsAH1s4rpS6b7NYAU6UTEfmVszNU5wCYCiDOVgERmQzgMQBRAAZXvmpEROQuhy13ERkN4LxSaqu9ckqpd5RSzQE8BeBZG9eaKCIZIpKRk5PjVoV9reUzP6LUwdJ+RESBxplumT4AxohIJoD5AAaLyOd2ys8HcLO1E0qpeUqpNKVUWnJyssuV9YZHh7aye76kTKGgpMxHtSEi8gyHwV0pNV0plaKUSgUwDsAapdR40zIi0tJkdxSAwx6tpRcpJx4PVMwSXFDMWaxEFNjcHucuIjNFZIy2+5CI7BWRHdD3u0/wSO3c0K5+vFPlHh6i/32kFLDm8QF4fWwHm2VNV+M7lpOPdjNW4OuMrErVk4jIm1wK7kqptUqp0dr2DKXUD9r2FKVUe6VUJ6XUIKXUXm9U1hnfP9QHt3RuaLfMdw/2NmuNN0uOxW1pjWyWv3i1fIWmQ9n5AICV+7KNx4pLddBxPVYiCiAhNUMVACLDw/DmHZ3sluncONFsuT1HBr2xFs8t3oMTuVeNx0yDeatnl+Hp73a7XlkiIi8JueDurHv7NcW4bo1wX7+mTpX/7LcTeHTBDuNarKsPnAcA4/78LeymIaLAUWWDe2x0BGbd2sHmSk7WbDt5CZNMFt9edygHZVoL3oUvAkREXhfywT0y3Pmo27quzTlaVk34aDNKteAeFV7+T/n1liyMmMO8NETkPyEf3Hs1T8In/9cdTWpXd1i2Tny0wzIVXS3SD4ssKtVPdPp+x2lMXbQLB84xoyQR+U/IBvf0JwYiLiYC79/dFQNaJeOmTvZH0ABA82TLtVgdGfver8bt1GlLMWX+DpevQUTkaSEb3Jsm1cDu54cjOiIcAPDXfk0xqkN97Jhxg83XDG9fz+X3OX7hquNCREQ+JobRHr6WlpamMjIy/PLejqzal437PvVM3XY9PwzxLjy0JSKyR0S2KqXSHJUL2ZZ7ZQxtVxd9WyR55FpZFws8ch0iIlcwuNsQE+mZfxodE0oSkR8wuNvQpHYNj1zn94JilOkU8ouYbIyIfIfB3YZh7ep65Dp3f7QZUxfuwnV/X+GR6xEROYPB3YYezWrjy/t6oHfz2pW+1qJtpzxQIyIi5zG429G7RRIiwz33T3StmIt+EJFvMLg74KkHqwBw6Vqx40JERB7A4O7Aizdd5+8qEBG5jMHdgdqxruebsaXXq2s47p2IfILB3YEwD6fy3Xbyd89ekIjICgZ3B8TDidp1fkr3QERVC4O7j/129KK/q0BEVQCDuxMyZ43Cu3d1Me7/76G+bl+rlAtpE5EPMLg7qU/L8kRi16fUxE+P9jfuPzm8Ncb3bOzUdRZtO4UVe895vH5ERKYY3J0UoT1ZbVNPvxRfjegI47nJg1rgpZuvd/pa93+21bOVIyKqwOngLiLhIrJdRJZYOfeYiOwTkV0islpEmni2mv5XPSoC797VBZ/e2x2A/VE0DROquXTtoW+uw7OLdxv3z18uxIlcLgJCRO5zpeU+BcB+G+e2A0hTSnUAsBDA7MpWLBDdeH191ImLAQAkVIuyWa5Hs1ouXffI+Xx8/ttJ4373V1ZjwOtr3aojERHgZHAXkRQAowB8YO28UipdKWWYnfMbgBTPVC9wVYsKt3kusbrtwG+glEKZTmFH1iVPVouICIDzLfc5AKYCcGbpiXsBLHO7RiFgTMcGAIBpN7axWWblvmzMWXUIN7+zwWaZ7MuFAICzedeQ9tIqHMvJ92xFiShkOQzuIjIawHmllMOngCIyHkAagNdtnJ8oIhkikpGTk+NyZYNFx0YJyJw1ChN6pdosM/Gzrfj3miN2r9PjldVYtvsser26Bhfyi/DlppN2yxMRGUQ4LoI+AMaIyEgAMQDiReRzpdR400IiMhTAMwAGKKWKrF1IKTUPwDxAv0B2pWoeABZM7ImGieUPT58c3hrXN6xp3A93I3fBlkzzSU6Tvthm3L6QXwSllMdnzRJR6BHlwnR4ERkI4Aml1OgKxztD/yB1hFLqsDPXSktLUxkZGS5UNfjodArNnv7Ro9d8cnhrPDiwOT7fdBJju6TY7fsnotAjIluVUmmOyrk9zl1EZorIGG33dQCxAL4RkR0i8oO71w0lYVrLvUnt6h675k97z2HF3mw8t3gPXl9x0GPXJaLQ4ky3jJFSai2Atdr2DJPjQz1aqxCycdpg5OYX4w9v/+KR652/UoQ8bdGPK4UlHrkmEYUezlD1sgYJ1RAZ4bk+8rN5hTibpx9F883WUygudWYAExFVNQzuPpCS6LluGQCYs6r8scaLS/Z59NpEFBoY3H0gNjoCmbNGeeXam48zhTARWWJwD3IdG9V0XIiIqhwGdx/67sHeWDSpt0eveTTHMsFYXkEJDpy77NH3IaLg4tJoGaqczo0TUVhS5tFrbj1huSbrzf/ZgOMXrnqtK4iIAh9b7j4WE+n9SUfHLzBdMFFVx+BORBSCGNxDwIFzl3H60jVkZFofOVNSpjMbVVNapsP3O07DldQTRBRc2OceAkbMWW/cXvhAL+N2mU4hPExw94eb8euxXHx9fy90b1oLc38+htdXHERxqQ63pTXyR5WJyMvYcg8xY9/71bhdUqafvfrrsVwAwJJdZwDo88MDwJMLd/m4dkTkKwzuQaBbaqJbryvVmXe7fPrrCew7c9lsST8iCk0M7n6QOWuU2TDFOXd0slv+zdvtn7eltMwy78zIt9ab7RtG1uw5nYesiwUW5YkoODG4B4CoCPu3oU58tFvX7TRzJZbvOWu3zKA31gIARv/7F/SbnY6NRy649V5EFFgY3APA4DZ1LI5NGtjcuB1eiZWXHvh8m8Myu06VL9L9pw82uf1eRBQ4GNwDQExkuMVs0qdGlC+uHeblZfX2nGaqAqJQw6GQftQwoRoKikstjj87qq3ZvmFFpzABdF4Ymq7A8e5EoYYtdz9aP3UQtj57g8Xx+/o1szj21Ig2WPK3fl6px8X84kpfI3XaUry8lLnliQIFg7sfhYWJsVVuzdiuKaiuLYA9aWBztGsQ75V6/GPlIbvnT+YWIHXaUhzKvgIAUErhg/XHcE5bEcrg/fXHvVI/InIdg3sA+fKvPfDyH68z7r9xW0fsmznCrMzOGcOw5RnvLlmbOm2p2f5jX+8AADz+9U4AwMmLBXhp6X7c/1kGADCNAVEAYnAPIL2bJ+GuHk3slqlZPRLJcdHY9fwwH9UKyNDSCl/WFuQ2TI66XKh/XsDYThR4GNyDVHxMpFevX6YF8EVbTxmPlZbpjxk6ko5fuIqcK0XQmUT3lfuyvVovInIOgztZdamgGOevFOLxb3YajxVrM15zrhQZj3V7eRXOmvS9//XTDN9Vkohscjq4i0i4iGwXkSVWzvUXkW0iUioiYz1bRbJlzeMD8MHdaV65dvrBHHR/ebXZsdho/cjZO+b9Zna83+x0s32lFE7kXsWaA2zFE/mLK+PcpwDYD8DakI2TAP4C4AkP1Imc1Cw5Fs2SY71y7SdMWuwGxy9ctXjYak1JmcLgf6xDmU5xqT8iP3Gq5S4iKQBGAfjA2nmlVKZSahcAy0xVVOWU6nTGPntHuW2IyDuc7ZaZA2AqKhm8RWSiiGSISEZOTk5lLkUBzDTVsDO5bYjI8xwGdxEZDeC8UmprZd9MKTVPKZWmlEpLTk6u7OUoQJ3+/ZrZPsfBE/meMy33PgDGiEgmgPkABovI516tFVXKTZ0a2D3/+A2tvPr+N/7LPGf8qQrB3hkncwtw3ydbcK24zFPVIqpSHAZ3pdR0pVSKUioVwDgAa5RS471eM3JbsyT7D1njq3l3jHxF/WanY//Zy3jgs634ae85AMA9H2/GQpMx9Edz8s0WC+n/ejpW7T+Pnw+z+47IHW6PcxeRmSIyRtvuJiKnANwGYK6I7PVUBcmx6xvWNNvfcybPokzmrFH4c0/7s1+9adq3u7F87zlM/Ezfu5d+MMdsRM6Qf6xDv9npSJ22FOPmla8Du/fMZXbrELnBpeCulFqrlBqtbc9QSv2gbW/RWvc1lFK1lVLtvVFZsq5WjSiz/bjoCPzy1CCLctbSwndpnOCtapnZf7Y8Z/yCLfbXcP3t2EXj9lurD+MbkxY+ETmHM1RDwJu3d8TMm8p/n97aNQUpidUtyllrAP+xc0NvVs2ouLR8oNVTi3a79NpD565g49ELSJ22FEdz8t16/6LSMrMVp4hCHYN7CKgdG427e6WibX39/LKmSTXMzt/XtykAINxk0Y/1Uwdh8eQ+GN+zCQ68OAI1tNTC/uBoYlSZUliYoW+9bz5+0W5ZW2Ys3osxb2/Aqd+5CDhVDVyJKYQsntwbmRcK0CChGgCgca3qOHmxAIPb6tdofXRoK5TqdLgtrRFiIsPRqJa+dR8TGY4GCdVw+Lx7reLKcGbGq06n8O320wCAjzccx82dGqJahV9GS3adQW5+MSb0TrV6je1Z+syW+UWWK18RhSK23ENIdEQ4WteLM+6ve3IgNj09BL2bJwHQpwt+6ebrERNp2Ur//L4e6NTIN/3vrlq844xx+1B2PsZ/aLmI90Nfbsfff7D9HN8wYzbCzuIoRKGEwT2EiQjqxsc4VbZufAwWTeqNN27riAcHNvdyzVyTd63EbH+rll/eFSVauuJD2b7/dkLkDwzuZBQeJhjbNQVxXs4V7wmOhkf+Z+0RdHzhJ+P+SW0M/YNfMB0CVQ0M7mQhMjzwuy6aTv/R7vnZyw8i71oJbn/vV4tz/1p1GIC+/720jLnuKDQxuJOFdvXdX4i7WYWROr6QV1Bi89zmzIv48Bfzhbv/uUq/IPh1f1+BqQt3ebVuRP7C4E4W2jeo6bhQBSmJ+hE6d3Rr5Onq2HXw3BV0nPmT3TIvLtln89y320/jSqHtXw5EwYrBnSzUrB7pcJGN98Z3xQ8P9QGgn/maHBcNoPxvX3j+h70YPudns2Op05Y6nAELmPfZX/+8/V8OBusO5RhH3RAFOgZ3ssk0NUHFTJMjrquHVnX1wy6VAp7/Q3t0apRgnEjlC//dmGn1uDMzYCe5mGd+zYFsTPhoM+b+fNTsuE6n2G9PAYnBnWx6VEsN3Lt5bcwe2wGjO9Q3Ox9uMma8Y6MELJ7cB7Ur5LkJVMu17JS2rDmQjdRpS5Gbr18MPPuy/u+TueUzXLMuFqDf7HS0eGaZ9ypK5CYGd7LJkJ+mT4skREeEo39L8wVWwq1kIqsTH4P/PdQXiyb18kkdveWddH0L/eC5K1bP510rQb/Z6Th9yfVc9US+wOBONjVNqoGN0wZj0gD9pCYF8/7mMK3lPnmQ+aSn61NqomuTWtgx4wZM7N8Mn93b3TcV9iDDRKkVe8/h4tVii/NfbnLcr+9vH/5yHGkvrfR3NchPGNzJrgYJ1YxB3Nps18xZo/Dk8DZWX5tQPQpPj2yLfi1dX1Jxyd/6uvyayjiZW4DLhSWY8f0es+Of/HoCXV4sD5Dzt2Qh+3IhXlt+wOIat767EW2fW+71ujrrxSX7cCHf8hcTVQ1MHEZOG9i6Dh4c2Bz3alkmvanExw8p+7+ebtw2TU9sYDpccsJHm61ew9W0CIu2nkK7BvHGh9BPf7cbDROqYfKgFi5dh8gattzJJVNHtEHtWO8Pdyz145DD+VuyLI698mN5S/2AjX54g31nLts9b/D4NzvN1pv9ctNJvL7ioJO1JLKPwZ0Ckq9b7pWlM/llNPKt9XZKOlZQzLTEVHkM7uQX8yf2NNvv2yIJMZHB++O4Pcv+Kk/5RaVOT7Dy57cWCh3B+7+JgkqT2ubL/nVPrYUeTWuV7zethQMv3mjc79woEQDw4YQ0LJrUG3P/3NU3FXXTre9uNNtPnbbUrAWefbkQAPDeumMOr2UYYHqlsATpB89Xum5cYLxq4gNV8ol1T+oX7DasvBQWJlhwfy/jfsXWarWocIcpEALd+ctFSE3S/xcL0+YEHL9w1aLc6UvXzLp1DFuPLtiBVfvP496+TfHc6HZu10OngCBI9EkexuBOPvXxX7oh0cos1jEd9bNf3787DVdtLIXXum4cDmbbf5gZSDYdz8XS3WdxW1oK7C0A1WfWGrP9w9n56NokEav261vtH/5yHPf2bWpcPtFV+pY7o3tV43RwF5FwABkATiulRlc4Fw3gUwBdAeQCuEMplenBelKIGNSmjtXjLero89Tc0K6uzdeueLQ/9p7Jw9WiMtw+1zJPe6Ax5LhxdQTMre9uxIpH+nusHuyUqZpc6XOfAmC/jXP3AvhdKdUCwD8BvFbZihFZ075BTXQ36at31sNDWnqhNu5JnbYUmVa6Z0ydv1Joth9upen//Y7TyMi8aNwvLCnDteIyi3JbTMpQ1eFUcBeRFACjAHxgo8hNAD7RthcCGCJiJfEIkZ+0rhvnuJAPPfSV/ayUFYP56v2WD1anzN+BsSYrTQ14PR1tZ1jOkP14Q6Z7laSg5mzLfQ6AqQBsDT5uCCALAJRSpQDyANSudO0o5P33nm74/N4eXrn2q7dcb9x2p7XvTXtO25/oNOP7vWb7T3+3G0dz7C/ubchcWdHKfdm4kG/9HIUuh8FdREYDOK+U2mqvmJVjFl19IjJRRDJEJCMnJ8eFalKoGti6Dvq2TPLKte/s3thsf8HEnlj6sG9z1rjryHnLQP7uWn2myjKdwraTtlMdpE5biksF5jllJn6aAZ1O4bXlB4zDMim0OdNy7wNgjIhkApgPYLCIfF6hzCkAjQBARCIA1ARg0dGnlJqnlEpTSqUlJ7ueTIrI4OU/Xmdc2g8A1j4x0Oy8YZFvQ355EaBHs9po36AmNj89BON7mgf+YGAYLvneuqO45T8b7ZbtNNM8G+S2k5fw075svLv2KHq8stpq/hxn7My6xHHzQcJhcFdKTVdKpSilUgGMA7BGKTW+QrEfAEzQtsdqZfgTQF5zV48m+OWpwcb9xrWqWy1n7YewTnwMXrr5evzzjo5eqp13HLtwFW+tPox/rT5sdnz0v9dj1yn7M2QB4KMN5QuFv/KjrbERti3fcw43vbMB32w95fJryffcnqEqIjNFZIy2+yGA2iJyBMBjAKZ5onJEzgoLE7NJTy+MuU5/3M5z/T92TkHv5sHzaGhH1iW8ufKQRat7z+nLGPP2Boev33y8/Mv0DzvPuPz+hj7/qQt3ufxa8j2XJjEppdYCWKttzzA5XgjgNk9WjMgZvzw1CIezy/un69eMwdm8Qgxore/2+/KvPfDtttM2l//74r4euHytFB1nOrdItjXdm9YyC5zBwNoCJBWdv1yId9KP4LnR7RARHmZzcfCi0jKEiSAynNlMAgnvBgW1lMTqNidGAUCrunGYdmMb2BqZKyKoWT2yUnX43YlAGQxe+XG/MR0EAEz7djc++fUE1h+5AMA8p72p1s8uxx/+/Ytb75k6bSke/3qnW68l+xjcKaR88n/d8ZfeqWhQ03LVKE/79sHe6JaaiD90bOD19/KWV5ftRyttge95P+uTmhkelxnSLi/ffQ75RaV4f315n/2zi/Wzb7/YdAKA4xz39izaxj58b2Bwp5DSqm4cnh/T3mZL3ZbOjRMclnludDvUiStfqKRL40R880BvTOiV6mo1A8LVolLMXXcMxRVy53+dkYVLBcVYf1jfYl+QkYXr/r7CrMznv+lTFz/znfmyhBQ4GNyJANzaJcVhmXv7NsWI6+pZHK9ZPRKZs0ZhbFfH1wgk7U0CtungtqcW7bYYSulLu05dwvs/O06NTPYxuBMB+FP3xnjp5uvMju2cMcyspQ4A0RG2/8skVrLv3p/+9tV2l19z2UYfvMGJ3KtInbYUe07nGY+VlOkcjpMf8/YGvOzGUE0yx+BOBP1QyvE9mxj3x3RsgJrVIy3S7PZsph862b5BvMU1Hh/WGnf10E+OcmdVqebJNVx+jacs2XXW5dd0eN7+CKM1B/T5cBZq4+LzCkrQ8plleHfdUYuyaz2wKAmZY3AnMnH81ZGYeVN7/GtcJwDAnd0bAQDqxesf0HZrWgsNasZYtPIBICYyHM+NbodaNaLwn7u6uPzesdGhtbxChDY0crfWcs/R8tvMXn4Q4z/YhLN514xl//LxFlwqKHZqiCY5h8GdyISI4O5eqcYHsjGR4QCAc1o+lviYSGycPgSdGydafX1MZDi2PXcDBrexnZfelsKS4FoUvKI75/0GACgt0yHrYgEWaS32rSd+R0FxKf67sXy0zS9HLqDXq+aLlJSUKXR5sbyvX8e1ZCsltJoKRB7WMUU/iuZvg1t4/b1a1I21WGmqc+MEbD/pOLVAIPj1WC6+3pKFqYssZ7CO/Nd6ZOYW2H19xSUIS3UKUfaWsCK7GNyJ7EhNqoHfpg9B3fhox4UdaFEn1mq2R4M5d3RCbn4RPpzQDekHz6NNvXi0qBNrNrEo0FkL7AAcBnbA8jlFqU6HqAqdCyv3ZaNpUnXjyl37zlzGtZIydG2i/yb16IId+G77aWMqCp1OYdmecxhxXT2rC56EMnbLEDlQr2aMy+PmAaBGVLjZ/pPDW5vtv3l7R6x+fACiIsIwpE0dRIaHYf7EXqgRHYHRHRqgRZ3YStU72PyvQr4ba5kr//ppBoa++bNxf+Rb63Hru+UZMr/bfhoAMHfdUSil8M3WLEz+chs+/+2El2oduNhyJ/KSvTNHYPH203hkwQ4AwLB2dfHqLddj+rf62Z23aGPrD710o9/qGEhMZ8ACQH5RKRKqW88J5Miryw4gTARXi/WLredWwcVK2HIn8iLTSU8iYrGAiDdNHtTcZ+/lDX1fS0fqtKW4fe6veG35AbtrwX6w3nLS0+7TebhapA/uK/Zm20x8FqrYcifyopjIcKQ/MRBJse61QAFg+o1t0CChmssTjZ4c3gbhYWFoWSfWrUlKgWLz8YvYfPyicSUqA9Ng/dLS/WiebN6NdSL3qjG18cHsK/j010zc06epWRmdTkHB+gLkBkWlZYiOCLd5PlCx5U7kZU2TaiAuxv3Zq/cPaG41OVmvZpa56Ed1qG+2/9gNrYI6sZk9zZ/+0Wz/nv9uMdvfeSqm2dgwAAAN0ElEQVTPbP9QtuXD7Ls/2mx2nZX7sjH0zXXGYZhfb8lC62eXY9W+bJy/oh8Oe/zCVXyyMRMHK5EszRcY3In8oFlS5WejfjWxJ6YMaWncv65hPP49rnOlrxsM3Fno7avNJy2O/aKlMzb466cZOHI+H1syL+Js3jXj6J/7Ps1An1lrkH7gPAa9sRZ//2Evhs/52eJ6gYTdMkQ+tn/mCIR5qFn16A2tjMvu/al7E4RVkeF+Taf/6LiQFW2fW479L45wWO4ObUKWqZIy5VaaBn9hcCfysWpR7vXfHntlJBQsuyMMDFkp42IicKWw1OzcoZduREmZDqv2Z0Mp4ObODYNq/LynXCsps3r84LkraF0vzuHrS3XBM4uYwZ0oSBha5U1qV7eahyZKy1iZ/sRA5OYXW5yLigjDTZ0auv3+DwxojvesJP0KNmcuXcOF/CJ0SCnP4b//7GXngntZ5UfcbDqWi46NEoypLbyFfe5EQWbdk4Ow9OF+xv37+zdD2/rlWSqTYqOdClTOMvzSSIqNwn/v6YaZN7X32LX9ofesNRjz9gazEUSPLNiBCR9tdvhaw7h5W3ZkXULqtKXI1nIRFZaU4clvdhofxp7MLcAd837D09pcB29icCcKctNHtsWyKf0cF3RD5qxRuKd3KgB9n/PA1nVwd5CuPFVRxRmx6w7lOHzN2oP2y3y8QT8Ra+NR/YPaZXvO4putp/DKUn1++ve18fipHnig7giDOxHZFRGu7w4qM+lvfjHIW++eUvG5hSHJ24ItWQCAMC1txeIdZ7Bs91l8pqVBSI6rfK4iRxjciaqomzuZj3+v2N3SuFZ1AEDNavox+qZ9xOO6N7boDhrato63qhrQsi4W4P7PMqDTKZy8qE+Q9tsx/WzajzZkGstN+mKbcfvA2cter5fD4C4iMSKyWUR2isheEXnBSpkmIrJaRHaJyFoRCa7FJImqoMmDWpgtI9i1SSK+n9wHTWrrg/pftO6Ye/o0xdMj25h1x0SGh2H6yLb45oFeuEn7JVGzmuuzcHs2qwUAQT3Rqt/sdKzYm43FO04bj8VpD7x3ZllP1/zjnnNer5czLfciAIOVUh0BdAIwQkR6VijzBoBPlVIdAMwE8Kpnq0lEntaybhw2PzPUuF9SptCxUQK+ndQbozvUx+3d9KtQRYaHYWL/5sYHq6ZioyNwR5q+3PUNLZcedMTwrcDabNtg89jXO43bV4pK7Q41NeS88SaHwV3pGebtRmp/Ko4HagdgtbadDuAmj9WQiLzq+8l90KVxAtpoI2xqx0bj7T91cXrZv94tkrD56SEY1r6e48IV3D+gOWKjI8wSrFUFtsbbe5JTfe4iEi4iOwCcB7BSKbWpQpGdAG7Vtv8IIE5Egv9XMVEV0LFRAr59sE+lxl3XiY+xaPEBlg8Ob+uagmHtypcg7NI4EXteGI5aNaKsfjMIVW5kT3CZU/+aSqkypVQnACkAuotIxdWBnwAwQES2AxgA4DQAi+8dIjJRRDJEJCMnx/GwIyIKHrWs5F6vGMRev60j5t2dBgBI1fr2DQ69dCPeG+/6wuKmgmUUT3yM9+ePuvSrUil1CcBaACMqHD+jlLpFKdUZwDPasTwrr5+nlEpTSqUlJye7X2siCjjVosKROWsUOjZKMDlqvYn69f29sHBSb4vjA1vXwRjt4WpaE+uLkNszpqP7M3B9yZ0uLFc5M1omWUQStO1qAIYCOFChTJKIGK41HcBHnq4oEQWHSJPkZRWXFjTo3rQWkmItx3rHRIbjrTs7Y/MzQ/DZvT2cer/Fk/u4V1E/ivBBgjdnWu71AaSLyC4AW6Dvc18iIjNFZIxWZiCAgyJyCEBdAC97pbZEFPDeuK0jAOCpEW1wR7fGOP7qSJevUScuxukEa51MvimU+aIz2wN8kb3TYcePUmoXAIsk0UqpGSbbCwEs9GzViCgYpSbVQOasUcZ9w+LiA1pVviv24SEt8ZaW4tiaatpD4Vs6N0RqUg2Ulunw1pojbr9fz2a1jBOSPMkXmZmZFZKIvG791EEemXI/on09s+A+psLkp2pR4dj9/DBUj4owLp03ZWgrm2mSHfHWF4FwCYxuGSKiSmlUq7pbQy0//b/umH1rB+N+uwbx+HBCmnH/b4NbWLwmLibSbE3U8DBxe5hlh5Sabr3OkaEmw0G9hcGdiAJW/1bJxpmyBkPalgfGlnX1E68WT+6DuX/uavM6G54ajFWP9Xf5/YebjGqZ2L+Zy6+35uv7e6FfS++PFmRwJ6Kg16lRglkgrig5Lhot6ljmuP9zzyZ2r2vae/LI0JZ42Mo3BVf5oEcGAIM7EQWBZVP6YfXjAyp9nW8e6IUGNWOM+0OsZLLMnDUKnRuXj8BpWScW7RvEo3pUBB4b1hrzTL4hvPzHivM5HUu2MgTUGxjciSjgta0fj+bJscb9uX/uihfGuD4btVtqLWycPsS4HxcTaTay5wNt9qxhZE/d+BisfGyA2cpXw9rXw+A2+l8K14pdyxETGS4+WagD4GgZIgpC9rpgXFHxgWlCdX2WyocHt8Sd3RujbnyMtZdhdIf6WHPgPJq6GKibJcU6LuQhDO5EVGUZVkqKi4nAlcJSY394WJjYDOwAcEuXFIzqUB9nLhU69T7hYYIynW8nWLFbhoiqnEeGtgRQPpmoRR3XW9TREeFIrV3duOCILcdeGYn/PdTX5etXFoM7EVU5jwxthcxZo4yzZw0LjjSu5Vo3i4hg/sRedsuEhQlqx+ozZvZu4btM6OyWIaIqb1z3xhjXvbFHrmXo4jFVNz4GPz85CA0SbHf1eBqDOxGRh6QkVkOZThmD+4MDmxvPNa6Qv97b2C1DRFRJL92sH+8+on09/OP2jujaJBHLpvTDYze08lud2HInIqqk29JSkHWxAA8NboG4mEj0npTk7yoxuBMRVVZ0RDimj2zr72qYYbcMEVEIYnAnIgpBDO5ERCGIwZ2IKAQxuBMRhSAGdyKiEMTgTkQUghjciYhCkCjl2xzDxjcWyQFwws2XJwG44MHqBDJ+1tBUlT4rULU+r7c/axOllMMVtv0W3CtDRDKUUmn+rocv8LOGpqr0WYGq9XkD5bOyW4aIKAQxuBMRhaBgDe7z/F0BH+JnDU1V6bMCVevzBsRnDco+dyIisi9YW+5ERGRH0AV3ERkhIgdF5IiITPN3fZwhIo1EJF1E9ovIXhGZoh2vJSIrReSw9neidlxE5C3tM+4SkS4m15qglT8sIhNMjncVkd3aa94Sw8q/fiIi4SKyXUSWaPtNRWSTVu8FIhKlHY/W9o9o51NNrjFdO35QRIabHA+onwERSRCRhSJyQLvHvUL13orIo9rP8B4R+UpEYkLl3orIRyJyXkT2mBzz+n209R6VppQKmj8AwgEcBdAMQBSAnQDa+bteTtS7PoAu2nYcgEMA2gGYDWCadnwagNe07ZEAlgEQAD0BbNKO1wJwTPs7UdtO1M5tBtBLe80yADf6+TM/BuBLAEu0/a8BjNO23wMwSdt+EMB72vY4AAu07Xba/Y0G0FS77+GB+DMA4BMA92nbUQASQvHeAmgI4DiAaib39C+hcm8B9AfQBcAek2Nev4+23qPSn8ef/ync+MfvBWCFyf50ANP9XS83Psf3AG4AcBBAfe1YfQAHte25AO40KX9QO38ngLkmx+dqx+oDOGBy3KycHz5fCoDVAAYDWKL9MF8AEFHxPgJYAaCXth2hlZOK99ZQLtB+BgDEawFPKhwPuXsLfXDP0gJXhHZvh4fSvQWQCvPg7vX7aOs9Kvsn2LplDD9cBqe0Y0FD+2raGcAmAHWVUmcBQPu7jlbM1ue0d/yUleP+MgfAVAA6bb82gEtKqVJt37R+xs+knc/Tyrv6b+AvzQDkAPhY64b6QERqIATvrVLqNIA3AJwEcBb6e7UVoXtvAd/cR1vvUSnBFtyt9TUGzXAfEYkFsAjAI0qpy/aKWjmm3DjucyIyGsB5pdRW08NWiioH5wL+s2oioP8q/65SqjOAq9B/tbYlaD+v1hd8E/RdKQ0A1ABwo5WioXJv7Qn4zxZswf0UgEYm+ykAzvipLi4RkUjoA/sXSqlvtcPZIlJfO18fwHntuK3Pae94ipXj/tAHwBgRyQQwH/qumTkAEkTEsCC7af2Mn0k7XxPARbj+b+AvpwCcUkpt0vYXQh/sQ/HeDgVwXCmVo5QqAfAtgN4I3XsL+OY+2nqPSgm24L4FQEvt6XwU9A9pfvBznRzSnop/CGC/UupNk1M/ADA8TZ8AfV+84fjd2hP5ngDytK9rKwAME5FErRU1DPo+yrMArohIT+297ja5lk8ppaYrpVKUUqnQ3581Sqm7AKQDGKsVq/hZDf8GY7XySjs+Thtx0RRAS+gfSAXUz4BS6hyALBFprR0aAmAfQvDeQt8d01NEqmt1MXzWkLy3Gl/cR1vvUTm+fijjgQceI6EfbXIUwDP+ro+Tde4L/VewXQB2aH9GQt//uBrAYe3vWlp5AfCO9hl3A0gzudb/ATii/bnH5HgagD3aa95GhQd8fvrcA1E+WqYZ9P+BjwD4BkC0djxG2z+inW9m8vpntM9zECYjRALtZwBAJwAZ2v1dDP0oiZC8twBeAHBAq89n0I94CYl7C+Ar6J8llEDf0r7XF/fR1ntU9g9nqBIRhaBg65YhIiInMLgTEYUgBnciohDE4E5EFIIY3ImIQhCDOxFRCGJwJyIKQQzuREQh6P8BzIJeUsSzGEgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.sched.plot_loss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classifier model is basically a linear layer custom head on top of the LM backbone. Setting up the classifier data is similar to the LM data setup except that we cannot use the unsup movie reviews this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trn = pd.read_csv(CLAS_PATH/'train.csv', header=None, chunksize=chunksize)\n",
    "df_val = pd.read_csv(CLAS_PATH/'test.csv', header=None, chunksize=chunksize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "0\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "tok_trn, trn_labels = get_all(df_trn, 1)\n",
    "tok_val, val_labels = get_all(df_val, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(CLAS_PATH/'tmp').mkdir(exist_ok=True)\n",
    "\n",
    "np.save(CLAS_PATH/'tmp'/'tok_trn.npy', tok_trn)\n",
    "np.save(CLAS_PATH/'tmp'/'tok_val.npy', tok_val)\n",
    "\n",
    "np.save(CLAS_PATH/'tmp'/'trn_labels.npy', trn_labels)\n",
    "np.save(CLAS_PATH/'tmp'/'val_labels.npy', val_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tok_trn = np.load(CLAS_PATH/'tmp'/'tok_trn.npy')\n",
    "tok_val = np.load(CLAS_PATH/'tmp'/'tok_val.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60002"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "itos = pickle.load((LM_PATH/'tmp'/'itos.pkl').open('rb'))\n",
    "stoi = collections.defaultdict(lambda:0, {v:k for k,v in enumerate(itos)})\n",
    "len(itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_clas = np.array([[stoi[o] for o in p] for p in tok_trn])\n",
    "val_clas = np.array([[stoi[o] for o in p] for p in tok_val])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save(CLAS_PATH/'tmp'/'trn_ids.npy', trn_clas)\n",
    "np.save(CLAS_PATH/'tmp'/'val_ids.npy', val_clas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create our final model, a classifier which is really a custom linear head over our trained IMDB backbone. The steps to create the classifier model are similar to the ones for the LM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_clas = np.load(CLAS_PATH/'tmp'/'trn_ids.npy')\n",
    "val_clas = np.load(CLAS_PATH/'tmp'/'val_ids.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_labels = np.squeeze(np.load(CLAS_PATH/'tmp'/'trn_labels.npy'))\n",
    "val_labels = np.squeeze(np.load(CLAS_PATH/'tmp'/'val_labels.npy'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bptt,em_sz,nh,nl = 70,400,1150,3\n",
    "vs = len(itos)\n",
    "opt_fn = partial(optim.Adam, betas=(0.8, 0.99))\n",
    "bs = 48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_lbl = trn_labels.min()\n",
    "trn_labels -= min_lbl\n",
    "val_labels -= min_lbl\n",
    "c=int(trn_labels.max())+1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the classifier, unlike LM, we need to read a movie review at a time and learn to predict the it's sentiment as pos/neg. We do not deal with equal bptt size batches, so we have to pad the sequences to the same length in each batch. To create batches of similar sized movie reviews, we use a sortish sampler method invented by [@Smerity](https://twitter.com/Smerity) and [@jekbradbury](https://twitter.com/jekbradbury)\n",
    "\n",
    "The sortishSampler cuts down the overall number of padding tokens the classifier ends up seeing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_ds = TextDataset(trn_clas, trn_labels)\n",
    "val_ds = TextDataset(val_clas, val_labels)\n",
    "trn_samp = SortishSampler(trn_clas, key=lambda x: len(trn_clas[x]), bs=bs//2)\n",
    "val_samp = SortSampler(val_clas, key=lambda x: len(val_clas[x]))\n",
    "trn_dl = DataLoader(trn_ds, bs//2, transpose=True, num_workers=1, pad_idx=1, sampler=trn_samp)\n",
    "val_dl = DataLoader(val_ds, bs, transpose=True, num_workers=1, pad_idx=1, sampler=val_samp)\n",
    "md = ModelData(PATH, trn_dl, val_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# part 1\n",
    "dps = np.array([0.4, 0.5, 0.05, 0.3, 0.1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dps = np.array([0.4,0.5,0.05,0.3,0.4])*0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = get_rnn_classifier(bptt, 20*70, c, vs, emb_sz=em_sz, n_hid=nh, n_layers=nl, pad_token=1,\n",
    "          layers=[em_sz*3, 50, c], drops=[dps[4], 0.1],\n",
    "          dropouti=dps[0], wdrop=dps[1], dropoute=dps[2], dropouth=dps[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_fn = partial(optim.Adam, betas=(0.7, 0.99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = RNN_Learner(md, TextModel(to_gpu(m)), opt_fn=opt_fn)\n",
    "learn.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)\n",
    "learn.clip=.25\n",
    "learn.metrics = [accuracy]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=3e-3\n",
    "lrm = 2.6\n",
    "lrs = np.array([lr/(lrm**4), lr/(lrm**3), lr/(lrm**2), lr/lrm, lr])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lrs=np.array([1e-4,1e-4,1e-4,1e-3,1e-2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wd = 1e-7\n",
    "wd = 0\n",
    "learn.load_encoder('lm1_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.freeze_to(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef6d96007c3b40808b64198c2ac726cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 80%|███████▉  | 625/782 [01:39<00:24,  6.31it/s, loss=1.28] "
     ]
    }
   ],
   "source": [
    "learn.lr_find(lrs/1000)\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c1fd1b1bab94e679afc5c51eb0bbc6e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                      \n",
      "    0      0.365457   0.185553   0.928719  \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.18555279, 0.9287188090884525]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lrs, 1, wds=wd, cycle_len=1, use_clr=(8,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('clas_0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('clas_0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.freeze_to(-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bec8ae53c32945f1822b07897c826990",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                      \n",
      "    0      0.340473   0.17319    0.933125  \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.17319041, 0.9331253991245995]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lrs, 1, wds=wd, cycle_len=1, use_clr=(8,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('clas_1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('clas_1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "affe8780dbe347c1a8910d79ae307155",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                      \n",
      "    0      0.337347   0.186812   0.930782  \n",
      "    1      0.284065   0.318038   0.932062                      \n",
      "    2      0.246721   0.156018   0.941747                      \n",
      "    3      0.252745   0.157223   0.944106                      \n",
      "    4      0.24023    0.159444   0.945393                      \n",
      "    5      0.210046   0.202856   0.942858                      \n",
      "    6      0.212139   0.149009   0.943746                      \n",
      "    7      0.21163    0.186739   0.946553                      \n",
      "    8      0.186233   0.1508     0.945218                      \n",
      "    9      0.176225   0.150472   0.947985                      \n",
      "    10     0.198024   0.146215   0.948345                      \n",
      "    11     0.20324    0.189206   0.948145                      \n",
      "    12     0.165159   0.151402   0.947745                      \n",
      "    13     0.165997   0.146615   0.947905                      \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.14661488, 0.9479046703071374]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lrs, 1, wds=wd, cycle_len=14, use_clr=(32,10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXeYFFXWxt8zeQaGMOQ8gGQljgioCIKKIoZd3TV+pl0ja14FcY3rmte05jUnREQXJQkIIhIHSUMYwpCGOOQ8oft+f3RV963qih1nus/veeah6tatqtvV9HtvnXvuOSSEAMMwDJMcpMS7AQzDMEzsYNFnGIZJIlj0GYZhkggWfYZhmCSCRZ9hGCaJYNFnGIZJIlj0GYZhkggWfYZhmCSCRZ9hGCaJSIt3A/Q0bNhQ5Ofnx7sZDMMwNYolS5bsFUI0sqtX7UQ/Pz8fhYWF8W4GwzBMjYKItjipx+YdhmGYJIJFn2EYJolg0WcYhkkiWPQZhmGSCBZ9hmGYJIJFn2EYJolg0WcYhkkiklb0Z67ZjR0HT8S7GQzDMDElaUX/lk8Kcdmbv8W7GQzDMDElKUW/yuMFAOw5Uh7nljAMw8SWpBT9CkX0GYZhko3kFP2qgOiXHjgex5YwDMPElqQU/Sqv8G/vP1YRx5YwDMPEFkeiT0TDiKiYiDYQ0SiD47cT0UoiWkZEc4moq1KeT0QnlPJlRPROpD9AKIiA5oNA8WsIwzBMjLENrUxEqQDeBHAegFIAi4loohBitVTtSyHEO0r9SwD8G8Aw5dhGIUTPyDY7PISk+ilJ+a7DMEyy4kTy+gLYIIQoEUJUABgL4FK5ghDisLRbC4BANUZuXGoKj/QZhkkenIh+CwDbpP1SpUwDEd1FRBsBvADgbulQWyJaSkS/ENHZYbU2Qnilkb6o1t0TwzBMZHEi+kZD4SCpFEK8KYRoD+BhAI8qxTsBtBZC9AJwP4AviahO0A2IbiWiQiIqLCsrc976EJGF3uNl1WcYJnlwIvqlAFpJ+y0B7LCoPxbAZQAghCgXQuxTtpcA2Aigo/4EIcR7QogCIURBo0a2KR7DRh7ps+gzDJNMOBH9xQA6EFFbIsoAcBWAiXIFIuog7Q4HsF4pb6RMBIOI2gHoAKAkEg0PB81In+07DMMkEbbeO0KIKiIaCWAagFQAHwohVhHRUwAKhRATAYwkoqEAKgEcAHCDcvpAAE8RURUAD4DbhRD7o/FB3CDr/Oa9x9C7df34NYZhGCaG2Io+AAghJgOYrCt7TNq+x+S8bwF8G04Do4Fs3hk1YSUu7t4cR05WokHtzDi2imEYJvokpZe6bNCpqPLinrFL0eefMwzrLizZh/xRk7DzEIdhZhim5pOUou/V2fGnFO0yrfvZgi0AgMWbD0S1TQzDMLEgKUXfbO5WGBw4WekBAHyuiD/DMExNJklFPyDuw7o19W/vPlweVG/Gmj0AgEWb4j7/zDAMEzZJJ/pVHi/Oe2WOf79r88BasU/nb9bUHVe4DQzDMIlE0on+ySptAhV5cdaWfdrY+jsOnoxJmxiGYWJF0om+x6O128uTupNW7tQcO1peFZM2MQzDxIqkEP3vlpYif9QklB0pR6XXfKSv54O5m6LdNIZhmJiSFKL/1SKfbX5j2VFU6vLjcugdhmGSiaQQfTVM6Ky1e7Btv3aRld5nn2EYJpFJWNEv2n4IK0sPAQBSyCf7784pwZ/ena+ppzfvXPnOPM6byzBMwpKwon/xG3Mx4j9zAQBpqebZsfSiv3jzAYxfwq6aDMMkJgkr+jJWKRGNzDts8WEYJlFJCtFPs8h+/ul8Dq/AMEzykBSin25h3jGCB/oMwyQqSSH6aanuPiabdxiGSVSSQvTTLWz6Rgge6zMMk6Akhehbee8YUaWEamhaJysazWEYhokbSSL67j5meZUvhn6Tuiz6DMMkFkkh+m7NO3uP+BZnuTyNYRim2pMUoh/qSF9dycswDJMoJIXor9t9xFX9cl3M/UC5JxLNYRiGiRtJIfq/rt/rqr66SlefM3dBCadMZBimZpPwon+iwoO/nNXW1Tlq9GV92GUvx2FmGKaGk/Civ/PQCWw/eCKo/O4hHTT7nZrk+rc9SqIV/UjfzOzDMAxTU0h40RcAphTtCiq/pm9r3Dgg378//o7+/u3Ozerg8MlK7DikzZH74W/uMmm9NmM9vl+63dU5DMMw0SQhRb9o+yH/tllIhRQC/tC7hX8/Nyvdv92gVgYufPVXlB0p15yz57C7ROmvzFiHe79e5uochmGYaJKQon/xG3P921aZscxcMqu8wtAkVMHmHYZhajgJKfoysuhPu3cgMtNSlPKA6Hdu6rPnr316GABtYpX8Bjn+7ROV7LLJMEzNJuFFf1PZMf92p6a5yKuVAcDXGahh9tV+IU1Zgit76Wzedxz92zUAABw4XhlSG46cDO08hmGYSJPwon/HF79r9tXRvVcI/7ZHUX01w1aVzjVz1IWdDa/95qwNGPDsTNs27NHNDTAMw8SLhBd9PRd3bwbAN3GbnZ4KAGhY2zf6JyKkUPA8gJnt/8VpxUEePiorSg/6t1+Zvk5zbMu+Y64nhRmGYSJBWrwbEGseGtYZdwxqj7rZ6aibnY4Xr+iOczs39h9PTaGgkb5FtkVTjpV7pO0qzbFzXpwNANj83HD3F2YYhgmDpBP91BRCvZwM//6VBa00x1OIglbeWiVWN6NWZqp/W9+JMAzDxIukM+/YkZZCKNpxSFOWKpl31FW6e49a2+mz0gOi72HRZximmsCir+NYhQe/bdjn3+/Rsi5IEv1DJ3yeOHOlIG47Dp6A1yvwybzNOGng1jlv476gMoZhmHiQNKJ/Q/82IZ3Xo1U9jU3+7BdmAYDf3x/wjeR/WLEDj09chVdnrAegXQksx/WR0cf2YRiGiTZJI/odTITXDoLWpn/kpK8DkB16MtNSsOOgzxvnwDFf1i3ZA6jYJJ5/hYdX+DIME1sciT4RDSOiYiLaQESjDI7fTkQriWgZEc0loq7SsdHKecVEdEEkG6+nvMrjt5830+W3TQsx9yERGSZWlwfpVV6B56euBQCcVBKtOLHjc1gHhmFija3oE1EqgDcBXAigK4CrZVFX+FIIcZoQoieAFwD8Wzm3K4CrAHQDMAzAW8r1okKnR6filk8Wo0W9bJx5SkPNsZQQRf+2c9oZ+umflLJoyQJfqYzeb/54sab+d0tLcYuubEWpdsKYYRgm2jgZ6fcFsEEIUSKEqAAwFsClcgUhxGFptxZ8EY2h1BsrhCgXQmwCsEG5XtSYXVyGSo83aGQf6ki/Wd1sGJ3ZsHamf1sr+r5t/Src+75ejplr92B28R5/2Yw1u0NqE8MwTKg4Ef0WALZJ+6VKmQYiuouINsI30r/bzbmRZs+R8iDf+lB87VWMFuRmSy6ZlZJtvsrGTv/Uj6v92x/9tjnkNjEMw4SCE9E3Ussgg7UQ4k0hRHsADwN41M25RHQrERUSUWFZWZmDJtmzdOtBzX44ol83OyOoTP4QqscOAMwqtm5/iRQArk5W0q2NYxgmzjgR/VIA8rLVlgB2WNQfC+AyN+cKId4TQhQIIQoaNWrkoEn2rN55WLOfahI/xwmNcjODyuRVu5OLdoZ03acvOzXkNjEMw4SCE9FfDKADEbUlogz4JmYnyhWISE44OxyAOvSdCOAqIsokorYAOgBYFH6z3RPOSN8IeaTP7vYMw9QUbEVfCFEFYCSAaQDWABgnhFhFRE8R0SVKtZFEtIqIlgG4H8ANyrmrAIwDsBrAVAB3CSHikonEyO3SDb8+NFizb5WRS+bmM9uaHqvycG/BMExscWRUFkJMBjBZV/aYtH2PxbnPAHgm1AZGCrPwyE6pl5OuLXCo11np5v3qA98sxx/7tAyjVQzDMO5ImhW5aSHER/77BZ382xlS2AUhhD/xihFyYva01KR5xAzD1AASVpFe+XMPzf7RcvcpCztKoRvSpU6j7ejJuP4D86mJBSWBAGu1MqK2Fo1hGMY1CSv6ORlay9U+JSaOG2S7vZsVvdmS0BftOGxRk2EYJrYkrOjrrS+heNiE6pWTlRYQ/U17j4Z2EYZhmCiQsKIfKtoBfWiqv2TrAf+21fqALs3qAPDNAehTKjIMw0SDBBZ9YRg+wY5rzwjE3Q814dWXC7f6t8miEa3qZ+NoeRUufmMu/vj2vNBuxjAM44KEEX19QhIhtG6actITK568pJt/W78S9w+93IcNqpOdbnrsp9W78ciElQCAtbuMY+4zDMNEkoQR/f26iVqv0JpWzu/a1NF15Anb0/PzNMdOaVLbdbvkTsSIrs3ruL4mwzBMqCSM6OstMZ2a1tZ40YTgph8EGcaPs6Ztw1qWx0MN+cwwDBMKCSP6euk8pXEu3r6ut38/3BW5AHDwuHu3T4ZhmOpEwoi+EQPaB7JnObXpW/HDcqvgoqHRKi8n4tdkGIYxI2FE324kX13DIWQo7WpZPzvOLWEYJhlImCweEbDe+PnpvoFYtSM4f21WFEIqVCiZtryh+ocyDMO4oHoOf8PkoxtPD+v8jk1ycXmv4OiXcopEldeu6hnWvSqqFNFnzWcYJgYkzkhfmsqN5KhfxsjTplPTXIOa5mx45kIcLa/Cd0u348kfVqNcEX2rqJ0MwzCRInFG+pIeR8JTx4j2jYP99N2GbE5LTUG9nAykK7Z8/0ifh/oMw8SAhBF9ioHo52YGvxg5SW6+6JEhQWUZftH3JRLbd6wC+aMmYfPeY0F1GYZhIkUCmXcCRGIhlhEZBm6fmWn2k7uN62Sh8NGhqC11GulpvhY/8cNqTd3ftx5Avs2CLoZhmFBJoJF+QPajNdL/25AOuG1gO7wjLfoCgPO7NjGsL5c3rJ2JLGkiON3EhVQ198SbhSX7cPEbv6K8Ki4pjRmGiRKJI/rSdrREv05WOkZf1AV92zbwl3mFwLmdGxvWf+e6PqbXMhP98moi+tf8dyGKth/Gln3H490UhmEiSMKIviyi0fLeUZEDuXmEMOxkmtfNssy2lVHNR/oeZWLZwxPMDJNQJIzoZ6SloH6OL4xxtEb6KiQ9Na/XOG7/n09vbXkNs5H+6p3W6RW37juO/FGTsHzbQdt2RoK3Z2/kmEMMk0AkjOgDQJM6WQCiP9KX4/g0qJ3p72SMFm+ZkZZq3Mjvlm63PG9W8R4AwDdLtjm+VzhMXL4DA1+YFZN7MQwTfRLGewcITObKI/3FY4YGJVgJl8y0VCx77DzUzkxDagr5vYXkROrCJtViuono2xGPSMyHT3IqR4ZJFBJqpK/qoSyMjXIz0Vh5A4gk9XIy/EHc1NXAcmdj18/sPlwe2o2Ve3hibPr/dX1ZbG/IMExUSCjRVwkl2UlY91NuN6hTI3+Z3buFVfKUzxdsCcoEpqKeFum3Fzuu/2BRTO/HMEx0SCjR3334ZFzuq47wU1II9w3t6Cu0EeV0i/j+j35fhHvGLjU8pnoOeTlWD8MwIZBQor9PGR2X7D0a0/uqoi9EwJPHztMx0ya+/7b9Wv/4cYu3IX/UJKzb7ftsxbuOoOOYKUH1GIZhrEgo0VdxGwQtXFrl+RKgnNaiXsD8YmPg6dLMOiH6niM+m/+iTftx5GQlXpu5HgDw4W+bAADLSw+hwuPF5JU7XbXV4xUc3I1hkpiEEv1+7fIAGMfIiSbdW9bDtHsH4raB7dC6gS9uTmubNIj1lDUFZniFwKHjlfjTu/Nx15dLcbzC2IPG7QreLo9NxdBXfrGtZ+T2Wl0WjjEMEzoJJfrDuzcHYC+40aBT01ykpBBGdG+Gr/7aD38qaGVZn2wWE1R5BMo9vrg3RdsPmbpNHi13505ZUeVFSZl9JE+jKYPXZq5zdS+GYaofCSX6153RGoseGeI6sUkkISL0b9/AVtTt8AiBT+dtAQDsP1ZhGg7hvTkljq9ZdiREN1GFN2dtDOt8hmHiT0KJPhFFxSc/HggB/GfWBod1BT5bsAWLNu3H2l3mYRyM8v7Git2HT+Jfk9dwLB+GiTMJtSI3GenRqh4e/GYFvv291F+2+bnhhnWdxP5fuvUALn9rXsTap/L38SswZ10ZBndqjP7tG9ifwDBMVEiokX4yQoBG8K04crLSts68jfvCbJEx5ZW++Qk7ryaGYaILi34NJycjePSeP2oSuvxjalD5/eOW214vWit9F27aDyD2q6UZhtHCol/DadPA2FPpRGVwxisnnj680JdhEhsW/WpA3/y8kM+NdKatgyfsTUAMw9RcHIk+EQ0jomIi2kBEowyO309Eq4loBRHNJKI20jEPES1T/iZGsvGJwte39cPap4eFdG6Wixj+TmhYOzOi1wOAQ9yRMEy1wdZ7h4hSAbwJ4DwApQAWE9FEIcRqqdpSAAVCiONEdAeAFwD8WTl2QgjRM8LtTiiIKGTx/m3D3oi2JRqrmR8ev8K/He0ENwzDWOPkF94XwAYhRIkQogLAWACXyhWEELOEEGrkrwUAWka2mYwZoSYuP1npwXX/XYii7VrffTnic91s61ARTpm6aldErsMwTPg4Ef0WAOTcfKVKmRm3AJgi7WcRUSERLSCiy0JoIxMGQgjDmDnPTl6DuRv24uI35mrK5YH4rAcHRbw9HBKaYeKLE9E3eiE3/OUS0XUACgC8KBW3FkIUALgGwKtE1N7gvFuVjqGwrIwzNEWSl34qRsdHp+Ckzpvnk/lbgupWerxIkYb6GWkpSI1Afsb+7QKLsXhFLsPEFyeiXwpAjh7WEsAOfSUiGgpgDIBLhBD+IC9CiB3KvyUAZgPopT9XCPGeEKJACFHQqFEj/eGE5ZOb+2L87f1dnzeiR3PHdb9cuBWAvbvmmp2H0WHMFMxYs8dflkLAbw+fi2/vGKCp61a4d0nJbapY9P3EOvsZwwDORH8xgA5E1JaIMgBcBUDjhUNEvQC8C5/g75HK6xNRprLdEMCZAOQJ4KTmnI6NUGDgrmmXNP3yXs5FXxVoO32ZsXo3AGDOusCbVgoRmtbNQp829f1lS7YcQPtHJmO+wcrdY+VVuOPzJTigS/W4aW8gqufGPYEEN/uOluOzBVsghMDYRVux50h8Mp/Fg017j6Ht6MmYtzGyE/EMY4et6AshqgCMBDANwBoA44QQq4joKSK6RKn2IoDaAL7RuWZ2AVBIRMsBzALwnM7rh5GY+/BgfHtHf7x1bR/LeikuXGDUgbXVqHLZtoN4eXpw2GSj+ywo8Yn9HINE6f+evg5TinbhD2+bx+6ZWhSY1L38rXn4x/dFmF1chlETVuLWT5eYnpdoqJ3rlJU8yc3EFkcB14QQkwFM1pU9Jm0PNTlvHoDTwmlgMtGyfg5a1s/B3PXWo79sF+6dqlnHYyH6l735m2G5kT0/kA4y+HqnNK4NAOig/CvfX+Wqvq3921uVVI/HK3zzDeGGfq5JpFg8R4aJJrwitxpy8ERFUFktKcZO37Z5eHxEV8trdGxSW7P/wtRi1+0wmsMN5AMOPqZ2RtlSW09UaCeQjVI1lilmHfltZO76vQmdqUudMOcpDibWsOhXQ47pRsebnxuOGQ+c498nItx0ZlvLa+hF+bul2123wygRjNoRVHqCBVl9M6jyBG5eoatn9MaxbNtBAAEBXLr1AK77YCH+NXmN6zbXFNTOk/MVM7GGRb8aIq/OHTn4FADuo1NGy2ygipXRKFydgJY7hDOf+1lTR51Ylk05fZTJbLXNahL4j+dtjlCrqx9s3mHiBYt+NUQeYedk+joAVSQa5TqLjRMt10h1NG8k+qkpvv9OVi6dqsjJOQBUs456Wqv6sc9xHGvU79hqroVhogGLfjVEtqXnZipz7UqZU99u2cQSSfyib2De0aMP8QAA7//qy+n73JS1/jK1k1A7hNNa1AUAXNEndtE8pq3ahc8XBC9YixapFnMjDBNNWPSrIRpTjiIO6coo2iwezje398cnN/f170d65etjF/smjtOUdvxvWdD6PD8z1+7B+t1HgkI8AMC2/SeCyp78wefFW6V2JHEIynbbZ0vw6PdFMbuf8hjZvMPEHM6RWw2RR/rqZv1aGXjm8lMxqFNjw3PaN6qNvFoZ/v0qb2Q9X9QRvtMlAnuPBnsg2aF2VOqcwLrdR/zHFpbsw/u/luC96ws0oSJqKv6JXNZ8JsbwSL8a0q5RwN1SFtlrz2iDFvWyDc/Rm31CEV0rAp455p2JPGpNs1lVbIRq3x7znW/EvaI0YB6644vfMWPNHuw/Hvrn2lh2FPmjJuGH5eZvKbFCFf0flu/AwTA+E8O4hUW/GtKpaS7+edmpAICereo5OsfInLPjYLApJVRU0a+0mCuQOx6rQG1Gtn7A3CSVP2oS9iuhHY6etE/5aMb9Xy8DAPztq6UhXyNSyKudX5jmfg0Fw4QKi3415bp+bbDunxeiW/O6tnX7tctDgyhkvJIJiH5gpK8f9b//6yb/ttU4f+SXvxuWO5mHGFe4zbaOGctLjTubeCC/FalB8RgmFrDoV2OcZrEae2t/w5F1KAuyzFC9TWRXUL1b6JItB/zbr8xYb3qtzSaJX7wiEPhNRW+2inZo5lhFvvwkgdcgMNUbFv0EY8KdgTDILxqYDYaf1iyk66o2+u2SycjK80SO1umGv3xaqNnXdyxW5qVIYNeplB44HpSbwIz1u49gz2HjyKGFUgfJMLGERb8Gc9vAdgE/foXereuje0tzk5AcF8cNqg1aNkW4GXV/KrmTumGzFJYZAM7pFHq+hct6+kJSp1nMN1gtavN4Bc56fpbjOYHzXpmDvv+a6a6RDBNlWPRrMKMv6oKVT14QVH5+1yam54xfUmp6zAojofR4hW1yFpWBHUMTazNTUCgMaN8QgLWw24k+AEzXmaBiyaETlXhj5nqO2cOEDIt+AqKGQ1AZd5v77FzB1wwW/c8XbMGpj0/Dln2+0fhNZ+aHfR8901dr483f8OGikK/10LcrbOt4rLyTDLKECiE06wmizdM/rsbL09dh5to99pUZxgAW/QQkVfettqhv7Nvv7prBoj91lU+QN5b5smFlpoVmOrJiXKHxm8nxiir8vDbyI26rRW1GUxjfLd2O81+ZE3Zb1LeIo+VVKD1g/najhqour3I2r8Awelj0ExB9xivVNKOPZfPaVT01++0b1TK9ppHoqx6bhZsPYPySUsNwy9FACIEx3xXh5o8LsWFPZEfZVuYdoyQv6oR18a6jmvLjFe7WE7z8k2/S/c/vzsdZz88yrRdIYuPq8gzjh0U/AdELNJEvJv9LV/bAlHvOBgB8dOPpaK5b3WsVo99I9FX3xrdmb8SD3yzHZ/ONA5aNvbUfAGB499A8h/RMLdqFVTt8Pvd6sTWiyuPF9w7dV3+x8DoyWmX8vRKD6NP5mzXlT//oLiuoet9VOw5b1gsksWHVZ0KDRT8BUfPYGtGlWR1sfm44BnduHCTkbRqYhzQ2Fn3tvlnkzdqKh9EpjWobHnfLp/O3YN1un9jfZbLQS+a9X0twr7Ia147Fm/abHku1CDykf8txm/rRqRuoGtI62usVmMSFRT8B2Vh2zL4SgkWsXnYGNj833FFdwHlQNzeJ3J0w36JTM2LvEeexbaxCRlvp7IgezTX7ThfWqejXH5iN5NV5lEiG2GCSCxb9BCQrXfe1moiVfvSuBjxbMHoIFo0Zojm2S1pkNLSLL9Kn08GmahaJsPY7xsi+LoTAB3M3Ye0urTnFajRvtRhNn7/AbUenf1OwS4JztJwncpnQYNFPQPReNGYLsswiYTatm4XGuVmasvwGgUneS3u2AODcxKB2Lr1b19eUv/DH7o7OD5fZxcF2eq/w2d2HvfqrpnxwZ+PQ1YD159V708ii78T+HiT6NiuPrRaYMYwVLPoJSIbOZzM3yzjxin5Um5eTYVgP0HYQ6cr1t+53tnBKvY9+gVanprmOzg+H0gPHNW8pKmaeRkZSW+nx4qtFWy398fWupfJblJPOUW/eqbQxnbW2mH9hGCtY9BOQdMmeXCfLPE+OKkw5Gal4+9relkKSLnUkVmGTjTAzi3RvWRdvXN3L1bXcssIksqYsxPLqVqN8AR/M3YTRE1bilk8Kg46ZIfenVvGC1PwITkf6LZU1F3VMOnKGsYNFPwFJl0T5sEX8+TQpBeOFNoHY2jYMmHf0i79ChYgwokdzNK+bZV/ZhplrduPQ8cogl0v9W4/KJimmzxeLAvGEjGzpTt09ZWShtxq1G4WsBswXX5Ue8E3gsssmEyos+gmI06xVqalqyj57AdGO9MP/b1M/JzBSdSJfT17SzfTYsm0Hccsnhejx1E+44cNFWLMzMDlrNp8qe9fMXBNYTWs0wh7SxdzOb8ZuyaRUWWUs+me0zfPb5is9Ah//FshH8NK0dUH15Y7Aw6LPhAiLfgIi2/D/du4ppvVSQ8zTauXhItOvXR4AoJ40V/DzA+f4ruHSRHTDgHzTY/qR+MHjlf7tL0wSlIxbHEjGIk/0egxG5ekGbwvqYqyhXQLB7WSTkfzpjN62GtbOQPvGtTXP4YkfAgu6vv29VHO9wycrceBY4HPZTfQyjBks+gnIo8O7+Lcv79XCtJ6qN24HjU4H+g8P64xFY4ZoErarHYDs3fKJw7DLcx8ebFj+sS4hifrmUlHlxc8mgcn+O3eTYbmR/d3o+Tz2v1XYc0Q7QSybaBrUDnzmZyYFxHzP4ZMo+Od07D1agRSy7vyem7LGv939iZ8w4j9z/ftW6wkYxgoW/QSkbnZgpJ9modDkF17nqt+iXrbjkf6+oxVBrp/1c9Jx95AO+PKvZ/jLOjbJDcoL8J9rgid4rT6LjDpC/sBE2J2c64RD0hsFoDWTtZdWH5dJSeqnrdrlT1q/svSQpejL6ScB7SpfHukzoWLu2sHUWEgSZSuddBu867s7B6BVXo4/lLIdRvcmItx/Xsfga991Job++xf/fpu84OBvTt8wJvxeiqZ1s3D4ZKV9ZR12i6Jk9G8Fcochvx14vQKb9x7D+7+WaDqDcHL26heVMYxTeKSf4FiNjt0G7+rVuj4a1s50vNrUYZQGANq3E8B4AtZqHYHM98t24PxX5jh+I5F5fupadH9imm1b1PIZ0iSwLPop0gh+5fZDuOfrZfhi4Vas3O5e6I3ePj41CW7HMHaw6Cd2gQh5AAAgAElEQVQ4VvOldbPTMaB9A7zu0ld+/R77yJaAM68glQa1MtCnTWDFrpHZI82lr6iRWF/Xr7XteYdPVuHAsQp/JE+zj7FQFwPIzPcfAJZvOwggtDj42xwugquprNpxCPeMXRq3IHIby45i4AuzsPeouyB5NRUW/QTnWIW5yKSmEL78az+c3cFdKsMuTeuYHvvLWYHwzPVrORuZA76Rsdz5RCJIm9Gk7MXdmxvUDOaKd+Zh+OtzLevoA7+pbpTlVR4s3Wac+PyYy5g5G/YcxTKlw0hURn65FP9btsOx2TDS/PPH1di6/zg+X5Acb08s+glOywhkzdKTa7HK95ozAiPp0/PzXF1XNsdEYgHYO79s1OyvfOJ8x9Ev5UilZv1Pj1b1NPvqSLXTo1Px2wZth6B+D7IZS5/Exoih//7FcVhoO05UeLCvGoxmJ63YiUkrdvr3450YZpbishtq/uiaBk/kJjih2LXtMFr8Ne62/ujRqm5YKRPl6QcyaXedrDTLVcZW5GalI93lwrIqj9fUvNO9hbHoG6GupJ24fIe/rHVeePFzhBCmz8mIP749D6t3HjYNnx0r1BwIw7v72lFdEsOo31GkWLr1AASCAw3GGx7pJzgpUYjGaDQ5vP3g8bBz5MqCbGbecSNyhvdIc3d+lVeY5g144odVmn03E9eA+wVqetqOnmxY/vJPxcgfNSmoE1q9s3p6/KiPYc76vfFtSIS5/K15+MNb8+LdjCBY9BnXGE3Qnts5sDL12zsGYPzt/V1fVza9mI2aw+3DnPr6q3iF0MwNNMrN9G9v0E1ovzy92DKpuZ4UIsP1COHy3pwSAIEsW6FwtDy0tyk3LN16AO/PKfF38G5TTNqxdtdh5I+ahOJdkc2j7IT3le+gOsKiz7hGPwrf/Nxwja26T5v6KHBpzwe0I18zz5+wR/oO4xKpPDBuuWal7U1n5pvW/d+yHZZJzfWkppDjiWUzZhcHrzhWn2Ooq3aLth/CqY9P09jdo8Hlb83DM5PX2Fd0SEWVF98UbvObiSYr7Z9S5OxzdG1m7qDglkh+rkjDos+4Js+FV44b5M7EbMVpvZzwQgq7dfucUrRL05ZwTVgy4Zp3AODGjxabXtcsZwDgcyk1G82rrqq/rDMOYRFp5O998srQO5r//Lwefx+/AmO+LwIQGCA4nSpo2yh4QWAi4ugXQETDiKiYiDYQ0SiD4/cT0WoiWkFEM4mojXTsBiJar/zdEMnGM/HBbf5Xp8gi2E76Af65oJV/+5Ob+qJVnnuPpJGDfYHn3I70AW0uYLcJz62I1HTLz2t3a/aNRH/gC9o3kNd/Xo9TH5+Gg8eD8weTQSA+IYRhroFIICe5ufML+0T3ZkxQAu99qQTZU/sSpxPE5ZXJEc/I9tdLRKkA3gRwIYCuAK4moq66aksBFAghugMYD+AF5dw8AI8DOANAXwCPE1H1mspOUH75+yB8fNPpUbt+dnrkRrwqsghmSdd//opAWsVWeTm4aUBbuOXBCzoBgGvvHQD49vdAFM8vFkbOlztSCeNv/lib3EUN5ayK2IFjFZosZycrPZi4zOdFtO9YQPQrPV78sq7M3y6vV0AIgd827MWY74twypgpIbfxq0Vb0fWxqUGL1gBg/zHnieut0Hvf+L2CHJ6fLBkonfwC+gLYIIQoEUJUABgL4FK5ghBilhBC/V+1AEBLZfsCANOFEPuFEAcATAcwLDJNZ6xo06AWBnVyHwfeKQ1zfSaeb+8YELFrhmuvN0PNTgU4zzUgI0+I9mvXICJtAgIj8pVPnB+xawKBBXlLtx3AiQoPej09XXO88z+mGp732oz1uOHDRXjwm+UAfIvNpq/ejWv/u9A/eg51tD96wkocr/Dg4An38ZBCRf2m7VaGD1XyJTSuk2lZL1FwIvotAGyT9kuVMjNuAaAOCRydS0S3ElEhERWWlQUnsWaqH+ooqkGU7PsA8NN9A/HZLcFhl436hlZ52Xjpyh6G1/nz6QHzkFFsfDdE8vMeUdYb5GalazqmUBFCYMbqgKknv0EtLNy0z+IMrb17k25F7C/ryrBNN3ouD8MjCAgtC1moqO7Kdou+sjN8y5XiFQYi1jj5BRgNjQyfDhFdB6AAwItuzhVCvCeEKBBCFDRq5C4kABMf3L46h0LHJrmOQ0R4PAJX9GkZVN6iXrYmkUy4ot+7TeSskycrAyEZ5Pj7oTJ55S785dOAqeet2Rvx4rRiw7olSrrIwycrsUexqet/rAePVwbFo3li4iroueXjxTjjXzMctXHHwcgugFI5XhE8KR2w6Vufq74JfLVoW0xcVeONk19AKYBW0n5LADv0lYhoKIAxAC4RQpS7OZepeeQrSdRDmRi1wyrMA2A8kri8t/HL5/aDJzRmo3A8Zh48vyP+YJGURs+ZpzRA56a5psflDuiRi7qY1nOKPqkLAVi1w3pB1h/emoe+/5oJINjbJzczLcjO/Y1BqIKZa/dg92FnE9xRsuCh62PTgsr2K3kL7CZy5ePR6pSqE05EfzGADkTUlogyAFwFYKJcgYh6AXgXPsGX/bymATifiOorE7jnK2VMDefVq3rh3ev7oGX98EIJ6PnoptMx9d6BlnX0tv/bBrbDA+f5JmrnPjwYvz40GJkWHkaDO4X2Njny3A6uXD5fuKIH3rCIYNpcMumc0db9ugYZI2G7yCbZvZ6DuqQwl/RsHvabkZ5xhcGdhl0nb8Qv68psA6Sp2dEOGHgoyZiluUxUbL9RIUQVgJHwifUaAOOEEKuI6CkiukSp9iKA2gC+IaJlRDRROXc/gKfh6zgWA3hKKWNqOHWz03FBt6YRv+7gTo1t7dv60WJmeqrfftuyfg5a5eVYCsnVfYPDKzsJuewWIYTlmgZ5da+bSeycjGDPKSNztNtAbUaxgCjCMnjIYCL3SAixlG74cBEe/b4I+aMmYYqNb/9JG1dM+dlF603k1/Vl+M/P66NzcZc46saFEJOFEB2FEO2FEM8oZY8JIVRxHyqEaCKE6Kn8XSKd+6EQ4hTl76PofAwmmQj6XRqMcvceNR/dndc1EDLi7Wt746u/9sPjI7qF3J5ZDw4yLPd63S8GA4AbLZLAA8Z5j93kLjBDb/qav3GfoQhO+N04GuVD45eH3QbAfeC1H1ZYW4ztTDba+0Ve9b1eges/WISXfloX8WuHAq/IZWocXZTl8qe28P2bZTDytUIeVV94WjP0b98gLDNGHZO3ilZ52SHNeTw+oivuGdLB9PgD53cKKvt1fVlQKGm36IPzlew9hsa5wW6ME3439sAZV1iKOevKMHH5DszfaO01ZEW/Z2ca+vObYZajQO08C7cY5zZQ0d7K/L53f7UU+aMmOW6XikfqVOIdSRTg0MpMDaQgPw8LHxmC+jkZeG/ORtwiJW6JNe0b1TIM4tardT0QUUgTx0SEWpnGHVlORiryamWgQ+PamgxmXy7c6ngy1Yh9R8sNw3A3rZsVVDZ3w17sOXwSjesEH9uy7xj+8T+fh0+oIZx3Hy7HsYoq5GZpQ254vQLlVV5k6zr5X9YZu3k7XfymSXNpcM7Uol1YvHm/Jiy2G+R1HlVeoRkIjF9SinrZ6RgqvX1GGx7pMzWSJnWykJGWgpHndggpHs51/VrjbsmVM1QeG9HNcMGXqiOhrAAGzKOBmg0UZ6wJL07OrsMnDTsoo9g+AND3XzOxeW9wpit51OxmtK7nhEHGt5enF6PLY1Mdu1UKhw7Fcqdh1OTbP1+CD5RJ4VD4TJpwljuYou2H8OA3yzVutrGARZ9JaMy8eP552Wm438BM4pbWeTmGYqm+xrvJZzCsW1P/xK/ZaZ4omQeOV3hcT2IaJR15XPLj3xdGeAWjNJ/fKWYlo3hBRizapPUZ8XoFnp2yBjsPmdv49bkTrFI4Vnm8eG3Gehyz6YTkcNuy6F/9/gLL86IFiz6T0LgRMqeB3BaMHuLfFkIgw2A+wEib7VJXvnN9H/z+j/MAWHjzKNd98tLQJ56NaFArw3WWNbuRdDieMPpwDxvLjmLHoZPKMev75mamYdWOQ5o1Cl6vQOGWA3j3lxL0f/ZnTX25g9Vf+5wXZ5veZ8LS7Xhlxjq8OsN6gvbzBVsD15dEPxSvpUjAos8kNG4Gxj8/MAirn7rA8FgtyY4s27kb1MoMGs13bVYH//5TICTEO9f1xj1DOmCCizhFdi8IA9o3xKJHhlhXcoFXCNfzD3bP9iYT05AT9LkAXpsRcHes8not23qkvCooqX15ldfUX7+F1BlXuTBJHVdG+G6S1VSHUA8s+kxC48aVMT01BVkm8wPP/rG7YXldJb6/PGn50pU90KFJYCXusFOb4b7zOhpOfJpxqYFb5m3ntMOEOwMdh5vr2VHpcZdvF7APwbFy+yH/tttw3JW6EfcJKWTF7OIy1+J5vKJKsw7hhalr/dvllV7/2hAnAeXU8BlqB2Fkwqtl4lHGos8wUcbt6FX9AV/WszmulGL5DOzQ0PE1IvHDrpMVnCxm1LDOOLVF3bCvbYTHK5Bm8axuHJCPJuFEoXT5SPQhIaZLgeT+Ocl9VqrxS0o18ztvzQ64t1Z4vH5vqUe/L8KNHy2yvNbfvloKIDCgMPL48QqgXcPgpCws+gwTJdSYN8+bjNCtWPHE+Xjpyh548coe+NflpwEAcjK03s2dm+aibwgpIcPBaCT+9wvCn4wGfCJrNek8tEsTNK3rPhKoOqHtNnVjpWQyMfJtd5tB7dkpa4P6HTWOf0WV1//9rt11BLOLrSP9zlizW2mXbz+FgMWb96P/szP9nkUeIdC+ce2gc/UTxfGARZ9JSBrW9o1KQ0ntWCcr3b+S9pozWmPzc8ODzBNT7x2IcSbJ39VFY0YMCjHujxl3DXbmdnrzmdZrGYq2H8LrM83DBHiFCJpncLLQyOMV+CEE//ZKaUQ8eeWuoOOhLKbTt7ekzLfOwSf6zt1+1cuo5h0iwkvTirHz0EmsKD3ov5fRBP9n843jBRmFp4gWLPpMQqKaddxMzEUKK9v4xzf1xVOXdsMTI/TJ56LL6Is6Wx5XF1SZ4RN97ef6fetB2/vuP1aBWQbJ2+1YvGk/5m/ch+0HT2DL/mC3yfJK41W4Vuj/K1zxznyMK9yGKq8IepMD7Ds1NWz13qPl/k5INd94dIuwVN6dU4IjJ4MFPpbRPVn0mYREFX2PjXtfPPi//vm40WbkHS760BBW9non9GxVL2ik//rM9bbC+PC3KxwtUKuTlYYRPZr79/8zawOufn8BznzuZ0MvocMhuDsaTeo/NH4FAKC2wQpoO/t7VyUcSOPcLP8CPdXl0ytgumjQ6Kp2oSIiCYs+k5DEY6TfsUmwDTde3HdeR80+EeHJS7phxv3nhHS9ejkZhm8wxbuPWJ43q7gM80vs4/A8enFXPB7ltx8rc/rOQ9pcBEfLq7D7iHVYi3MUU11uVpq/U630eP0rkbPSjeX1t/V7AWijrLY3mPSNFiz6TEJSO9M30o1lsusf/3a2qZ9/NJl+X3D+Aa+A34Sk2vNvGJCPUwwmF50y+sJgE9HdiieLFXJSdhl5niSFyP+d6dFn/1JH2G6xct9dqFu96/EI/Lxmt0ltYO76vf7/W5v2HvMvtBLSfbLSrUf6Z58S8Airk+1uYjocOOAak5A8MaIbWtbPxpAusQtklZGWgow4jKPkNQEqXq/ALQPb4YYB+RFLON+rdXCqyHW7jxrUtGdolyZ45KLOOPflXwAAqSk+kZz14CAMfmm25bnN62Vh9U7rjGBGFG52nsqj0sbLZtO+Y/45jvFSNrE1Ow/73U0zTURfTZ5TLnk0RSI0tlN4pM8kJHVz0vHA+Z3CSo8Yb359aDAm3X1WSOeqMXoiJfiR5t3r+6Bdo8Bbx+ETvpFyK5tQFYC5ye71q3vh5St7GB4DgCd+WG16TO9xVenxWi4t+Mf3RYY2/1dnrMfIL31vP0XS4jQZdXGXvJL37dnhhcV2A4s+w1RTWuXloFvzuhjYsRFGGZhWrIjlyDEU9J3xqh0+gUxLTUG9nHRL11aPVxhmRrvo1KbI0yWYf///Chy155q+bTT7/Z/9GY/ZeDS9ZSPUZgHnrnrPF2hNFv0pRcFuqdGCzTsMU8359Oa+rs+p5pofhLx26+DxSssFUlUeYegCQ0RBbqXnOYxTv2ybvffMnwta4evCbY6uBwRi8wA+058+Ro+bmD2RhEf6DJNA3K1k3LJKDG+HXTTQaOAmo9T8kn04YhDOmBD6xH235vbhLTLSUlzFEDpR6fEvDvzxb8FmumiFybaDRZ9hEohbB7bDg+d3xP/1zw/p/A6Na2PG/efg//q3sa8cIt2aB3vfRMK1lsjYYybbZEJVxslK6c7Ncl11pqUHTvjvbdQXxSt1Ios+wyQQ2empGHluB0cj0teu6onbz2mvKTurQ0Nkpadi+GnN/GWRDh0hx7lXkSNghgoRoaBNsIfRvUO1+YaN8g87yb52Td/WrtuprkszmlDXa/6/fyp2nBUsHFj0GSaBcGLeGKyI+KBOjXHnYK3oP3JRFwBad8OPbjzd0b2fDiOxy59Pb2VbZ1i3prZ1ZHE9S/GD18fp6WCwiM5JAnsiwpvX9Latp/LYxV39iWmMnKj0k+2v/7wBL/9UHFwxwrDoM0wC4cRF86Ob+mLzc8NRNzs9yPShCmSPlgEbt5Nr5jfIwSmNtesFereup9n/RJmQbm6QbF2+hXxvmXeu72PbDhn1bUfvKWSUecupa28TFzkMbj6rrT9yqRAC9aXIoAOenWkYu2hqDLx4WPQZJokxi1bp1r//m9sHaBLEP3h+R0y480xNHTVMhTyB2cygA5D998MhEAVTW66P1Q84F323yWDUkb7HC0wcGZjMVVM/nnlKA019fTiIaMCizzAJgBPTh1smjjzT0OvEiEa5mZow1vq5AkArgP4y/0g4UPbd0u2O7ql/k9CjumvqOzCjSWOnou92sZ8/8J9XoJXBfEBGagr+VNAyqDyasJ8+wyQAb13bO+LB5bq3DBbVR4d3wTu/lGDv0eBgZO0b1Ub9nHQcOF5pKI55tTLQu3U93DM0EAwunMB4ZqGdZz84CIdOVKK7YibSN8VspD/n74ORm5WGXk9P1xy7sk9L3HJ2aFFR1XUDZovlUoj8yVxiBYs+wyQAKSmEjBiEnPjL2e3wl7PbIX/UJMPji8cMxbFyj6F5KC01JcjkI4+EI0W+LmIl6Rwm/9i7ZdBq21QitG5g7JnzjxFdDdNXOsHu8xG5NxmFC4s+wzBh0SovsJgrLTUFdXOCRcws7HTA5BM9n3WhW75byyCaZ7RiNJlFDlUhIsMMW9GEbfoMwwAITrzilCGd7UMd/HSfcRx/O/PHhadq5yqm3RscRtoOr4MOxWriOpzu4NWreuLeoR38pqa/6sxEKQQcr3CfBSwcWPQZhgHgbrSdlZ6Cszs0xOMjuuKhYaEnZ+/f3ue9Iic6l1cD3ztUmwymU1OtW6iq1WueGmZ7D6fo4/XkhmjaAXwunvcO7ejvVI6WawWeQBFZmOYGNu8wTJJzWou6WLn9ECpdpJZc+/SFjuq9dGUP7D5s7oY4ZngXXNevDZrVDZiI5KBpVpaPNg1ysGWfL0GL1eIqef2A3sz03vV9gmz5b13bGx3GTAEAvH1t8GKsnIzUkEfnXy3aqtlPSQFqS29Yz//xtJCu6wYe6TNMkvPlX88AAFxzRuuIX/uKPi1x1+BTTI+np6YEZfOSRV8fNVPmiRHd8M51fdC3bZ5jm7x+XULjOlno3LSOaZ0Dx4OTmI+9tZ+jexkxvHszzT4RaeIFdQkxK5gbeKTPMElOblY6Zj5wjqvVptFE1mUjMV/0yBC8O6cEAzs2QmoKYdipztco3HKW1qYeStAzI1dWlQWjh6DfszNNj+/WLb5KIfKHiwCAzfuOW14/EvBIn2EYtG9U29bTJFakpMjmHUJ6KmkWnzWuk4V/XNw1JI+bP/TWLoSym8awS0azaMwQzX5TgxXGMsd0ZqHKKi9ObREIO5HhIAZQuLDoMwxTrUiVTDrZ6alY/8xFruPu2HFlH5/4G2XgAgJzBDkZ1tE3G9bKdHXfCXcM0OxPXaWNtZOdEf2Ol0WfYZhqhTyCt7Lph8PTl52Kj286HR0NksoDwKwHB2FI58a2pqOUFMLQLs6ycwFAtk0nMrBDQ8vjkYBFn2GYasWVfQJhlqOV1z0rPRWDOjU2Pd6yfg4+uPF05DgYef/3Bm0e3rQQzE5vXN0Lz1x+akwS2VcPIx7DMIxC6wY5qJOVhsMnI5dQ5NHhXfDRb5sjdj09Y2/thyNKexc+MgTHyt25dI7o0TwazTLEkegT0TAArwFIBfBfIcRzuuMDAbwKoDuAq4QQ46VjHgArld2tQohLItFwhmESlxSD6JvhoMYMihRT7jlbM/Hdr11gAViD2ploEJno0FHBVvSJKBXAmwDOA1AKYDERTRRCrJaqbQVwI4AHDS5xQgjRMwJtZRgmSVCNHPHJImtPOP70zetm+ePpxwMnNv2+ADYIIUqEEBUAxgK4VK4ghNgshFgBIDhmKcMwjEuu7utbKGbnPVMT+fq2/v7sYO10EUFjgRPRbwFgm7RfqpQ5JYuIColoARFdZlSBiG5V6hSWlZW5uDTDMInI3y/ohHX/vFCzWjVRaJWXg//rnw8AGHCKu7hAkcCJTd9oOtnNW1drIcQOImoH4GciWimE2Ki5mBDvAXgPAAoKCqrrGx3DMDGCiJCRFn1PlnhxSc/mWLfnCO4cZB6iIlo4GemXApBT1bcEsMPpDYQQO5R/SwDMBtDLRfsYhmESjvTUFIy+sAvqZocewTNUnIj+YgAdiKgtEWUAuArARCcXJ6L6RJSpbDcEcCaA1dZnMQzDMNHCVvSFEFUARgKYBmANgHFCiFVE9BQRXQIARHQ6EZUCuBLAu0Sk5iLrAqCQiJYDmAXgOZ3XD8MwDBNDKJQoc9GkoKBAFBYWxrsZDMMwNQoiWiKEKLCrx2EYGIZhkggWfYZhmCSCRZ9hGCaJYNFnGIZJIlj0GYZhkohq571DRGUAtoR4ekMAeyPYnGjB7YwsNaGdNaGNALcz0sSynW2EEI3sKlU70Q8HIip04rIUb7idkaUmtLMmtBHgdkaa6thONu8wDMMkESz6DMMwSUSiif578W6AQ7idkaUmtLMmtBHgdkaaatfOhLLpMwzDMNYk2kifYRiGsSBhRJ+IhhFRMRFtIKJRMb53KyKaRURriGgVEd2jlOcR0XQiWq/8W18pJyJ6XWnrCiLqLV3rBqX+eiK6IUrtTSWipUT0o7LflogWKvf8WgmhDSLKVPY3KMfzpWuMVsqLieiCKLSxHhGNJ6K1ynPtXx2fJxHdp3znRUT0FRFlVYfnSUQfEtEeIiqSyiL2/IioDxGtVM55nYhcZzwxaeOLyne+goi+I6J60jHDZ2T22zf7HiLRTunYg0QkyBc6Pm7P0hVCiBr/ByAVwEYA7QBkAFgOoGsM798MQG9lOxfAOgBdAbwAYJRSPgrA88r2RQCmwJeVrB+AhUp5HoAS5d/6ynb9KLT3fgBfAvhR2R8H4Cpl+x0AdyjbdwJ4R9m+CsDXynZX5RlnAmirPPvUCLfxEwB/UbYzANSrbs8TvrShmwBkS8/xxurwPAEMBNAbQJFUFrHnB2ARgP7KOVMAXBihNp4PIE3Zfl5qo+EzgsVv3+x7iEQ7lfJW8IWc3wKgYTyfpavPE82Lx+pPeWDTpP3RAEbHsT3/A3AegGIAzZSyZgCKle13AVwt1S9Wjl8N4F2pXFMvQm1rCWAmgHMB/Kj8R9sr/dD8z1L5D91f2U5T6pH++cr1ItTGOvCJKenKq9XzRCB/dJ7yfH4EcEF1eZ4A8qEV1Ig8P+XYWqlcUy+cNuqOXQ7gC2Xb8BnB5Ldv9f86Uu0EMB5ADwCbERD9uD1Lp3+JYt4JN3l7xFBe2XsBWAigiRBiJwAo/zZWqpm1Nxaf41UADwHwKvsNABwUvmQ5+nv626McP6TUj3Y72wEoA/AR+cxQ/yWiWqhmz1MIsR3ASwC2AtgJ3/NZgur3PFUi9fxaKNvRbu/N8I18Q2mj1f/rsCFfAqntQojlukPV9Vn6SRTRDzd5e2QaQVQbwLcA7hVCHLaqalAmLMojAhFdDGCPEGKJg7ZYHYv2806D73X6bSFELwDH4DNHmBGv51kfwKXwmRuaA6gF4EKLe8bredrhtl1Rby8RjQFQBeALtchlW6LWRiLKATAGwGNGh122J+bffaKIfljJ2yMBEaXDJ/hfCCEmKMW7iaiZcrwZgD1KuVl7o/05zgRwCRFtBjAWPhPPqwDqEVGawT397VGO1wWwPwbtLAVQKoRYqOyPh68TqG7PcyiATUKIMiFEJYAJAAag+j1PlUg9v1JlOyrtVSY5LwZwrVBsHiG0cS/Mv4dwaQ9fR79c+S21BPA7ETUNoZ1RfZaGRNN2FKs/+EaGJfB9EepkTrcY3p8AfArgVV35i9BOnL2gbA+HdrJnkVKeB58tu77ytwlAXpTaPAiBidxvoJ3wulPZvgvaicdxynY3aCfVShD5idxfAXRStp9QnmW1ep4AzgCwCkCOcu9PAPytujxPBNv0I/b8ACxW6qqTjxdFqI3DAKwG0EhXz/AZweK3b/Y9RKKdumObEbDpx+1ZOv4s0bx4LP/gmzVfB99M/pgY3/ss+F7JVgBYpvxdBJ9dcSaA9cq/6pdMAN5U2roSQIF0rZsBbFD+bopimwchIPrt4PMg2KD8UDKV8ixlf4NyvJ10/hil/cWIgrcBgJ4ACpVn+r3yQ6l2zxPAkwDWAigC8JkiSnF/ngC+gm+eoRK+0eQtkXx+AAqUz7wRwH+gm3QPo40b4LN9q7+jd+yeEUx++2bfQyTaqTu+GQHRj8uzdPPHK3IZhmGSiESx6TMMwzAOYNFnGIZJIlj0GYZhkggWfYZhmCSCRZ9hGCaJYNFnGIZJIlj0GYZhklr19XgAAAAOSURBVAgWfYZhmCTi/wGVoVgDS6JtlwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_loss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('clas_2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous state of the art result was 94.1% accuracy (5.9% error). With bidir we get 95.4% accuracy (4.6% error)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXeYHMXR/7+1dzrlnJGQTgkFQAJxSgQJhACBbIIBm2SDDZaxjRMv+CfgNbZxAox5nbANBmNjcjLGRiAQIBBJ6CSUA4pIp3hCKKe72/79sTO7Pb09Pd2zs7k/z6NHtxN6ema6a6qrq6uIMQaLxWKxlAexfFfAYrFYLLnDCn2LxWIpI6zQt1gsljLCCn2LxWIpI6zQt1gsljLCCn2LxWIpI6zQt1gsljLCCn2LxWIpI6zQt1gsljKiMt8VEOnSpQurrq7OdzUsFoulqJg3b94OxljXoOMKTuhXV1ejtrY239WwWCyWooKIPtE5zpp3LBaLpYywQt9isVjKCCv0LRaLpYywQt9isVjKCCv0LRaLpYywQt9isVjKCCv0LRaLpYywQr+AeH/Np1hTvy/f1bBYLCVMwS3OKmcu/+sHAID1d07Jc00sFkupYjV9i8ViKSO0hD4RTSailUS0moimSfbfSETLiGgREb1ORH25fVcT0Srn39VRVt5isVgsZgQKfSKqAHAfgHMBDANwORENEw77CEANY2w4gGcB3O2c2wnAjwGMATAawI+JqGN01bdYLBaLCTqa/mgAqxljaxljRwA8CeAC/gDG2JuMsQPOzw8A9Hb+PgfAa4yxnYyxzwC8BmByNFW3WCwWiyk6Qr8XgI3c7zpnmx/XAng55LkWi8ViySI6Qp8k25j0QKKrANQA+LXJuUQ0lYhqiai2vr5eo0qWbPHWx/V4cPbafFfDYrFkCR2hXwfgaO53bwCbxYOIaBKA2wCczxg7bHIuY+wBxlgNY6yma9fAHACWLHL13z7Ez19anu9qWCyWLKEj9OcCGERE/YioCsBlAF7kDyCiEwHcj4TA387tmgHgbCLq6Ezgnu1ss1gsAM64ZxZ+Od1+ZC25I1DoM8YaAdyAhLBeDuBpxthSIrqDiM53Dvs1gDYAniGiBUT0onPuTgA/Q+LDMRfAHc42S4EzZ+2n+a5CWbBux3488LY1p1lyh9aKXMbYdADThW23c39PUpz7NwB/C1tBS3740gMf2JXBFksJYlfkWiwWSxlhhb7FUgAcamjKdxUsZYIV+hZLAfD4nA35roKlTLBC35LkwJHGfFehbGmKS5e+WCyRY4V+AVK/93DwQVngL29ZL5J8weTrHS2WyLFCvwDZsvtgXq57pDGel+taAGZlviVHWKFfgFgBYLFYskVJCf2FG3dh76GGfFcjFPsPp+zp+ZL5jU1eTX/3gQbsKdLnWWzY77wlV5SM0D/U0IQL7nsX3/jnvHxXJRQHOZe9eJ5U/QffWef5PeKOVzH8J6/mpS7lhh3dWXJFyQj9Rsf7YeHGXVkpPx5nWfWwsJ2+vLETuZZcUTJCnzlSc/+RzBa53PzMQsxYujVt+0V/ehcDbp0uOSN67Aeg/LDv3JIrSkboR8Uz8+qkJqKFdbuzel1e08uXy6Ylf+w+aOdOLLnBCv1CgdP0rn+0OOclLOF5Z9WOfFfBUiZYoW8pCvYeasCAW6fjzRXbgw8uQkiWYy5DGGNoaLJrLyxeSkboF71JVOj0ovtkufPxtn1oijP8/o1V+a5KVsiGTf/RDz7BoNtexrY9h6Iv3FK0lIzQz4QJv34T333iI61js7YOgCl/WhxKdcIzG7f1woJEZtINOw9koXRLsWKFPoBPPj2AFxempe6V0tCUHakjeoPmQ7hNGd4z9xe1AABOHtA58jLdwWOpfigt4bBC35AsmF4BpC/Iyoffdve2LXJ+TVOyYfsuBPp0ahV5me6zYlbqWzhKRugXe7tOE/p5uJ/KisKXqMX+nv3Ixsds7vrPAFhTocWLltAnoslEtJKIVhPRNMn+8UQ0n4gaiegSYd9dRLTE+felqCquohijRYrCLB/CrVkRCH2LObqmS0t5ECj0iagCwH0AzgUwDMDlRDRMOGwDgGsAPC6cOwXASAAnABgD4GYiapd5tdWMv/vNrJWdLfOC6FqXD/NOm+bNcn5NS4Jsfm5flawwt5QvOpr+aACrGWNrGWNHADwJ4AL+AMbYesbYIgCiij0MwFuMsUbG2H4ACwFMjqDeSrZm2UXtFy8tk4ZqyITfzvS6Ir64IPfaWSHHf8mFLb8pzrBux/7sX0hGFm+wU+uqrJVtKT50hH4vABu533XONh0WAjiXiFoRURcAZwA4WjyIiKYSUS0R1dbX12sWLRCBvNKZ8GIM+OvsdZFH81y1fZ/n97TnF0dafrGz71D2Uzn+7vVVOOOeWVhTvy/44IiJWuTbyVuLHzpCX9YetVoUY+xVANMBvAfgCQDvA0jrvYyxBxhjNYyxmq5du+oUnRVk/eS/izajetpLyd/ZCnsctpN+sPZT7DscjUAsZDlx87MLASCrmnjt+p0AgK27i38x00dctNlCfq/ZgjGG3QdsPCMZOkK/Dl7tvDcAbdsDY+wXjLETGGNnIfEBKdgllbK+8ZAQY76Q+s+n+w7jsgc+0F5YVsxs25MIQncwwyiqKmKOiSUfwc+itu40cM4MpermquLJuRsx4o5XsXr73nxXpeDQEfpzAQwion5EVAXgMgAv6hRORBVE1Nn5eziA4QDykpVj9fZ9gRqxTNsW+4tKa9pzqCF0rJMw2thhp2Mv37In1DUtXlzh+K3H5uONFdvyW5kM4Rf7VcRKxjNbGzdG0+rteZqjKWACWwNjrBHADQBmAFgO4GnG2FIiuoOIzgcAIhpFRHUALgVwPxEtdU5vBmA2ES0D8ACAq5zyIidoEnLSvW/hyw/NCSgjs+sM/8mr+PojtRqlRIOrmeYr01ZeyKLWGuNU4g/W7szehXIA3yYqY2Wo6icpo76hSaXOQYyx6UjY5vltt3N/z0XC7COedwgJD56C4KMN6qxaMtlJ4tg4oA3NWhluIjqM54zbl6NK6FXuk3+8bMz1s6CIv2a80L9yTJ9Iyy4GytGkpUv5jfsUyARvmnknN1XRowyX2R9pjOOTT7MzZOc1/Vw/0qiFFK8IlLMnWBl1DW2s0OfQaSDZ894xP8fVDqOqUrF0kGsenpuVcvlRnfsotu89hH8v2JSV6/FE/ezLyuQnIeqRUylRMkI/ijauU0a2+lKYYl1zRDF17+v+MRf/XZTZwrNshdnwmncS/1/zt7n43pMLsOvAkaxc0+WO/y4NPsiAchn9NcWZNPfEK87iyfJ4CmaUjNCPAql5x8ykH/7aITpp1BO5ueggM5dvxw2PZ+Zimi17LW/ecZ+pu7q7MaqJEx8ONUT7IYsXX/ipUFz0p3cx8LaX812NoqIshL6uQJVO5ArDxJ37sqPxhREpO/YlfNd3ldkilKwJ/RLqDdn+SBUKi+p2K/eXyYDHiBJq5v6oXjz/QdBpH2+vChkmIgu8u9qbTHvr7kOY98lnocsrlg6SLXvt9MWpeErFbhNvKjKh/8aKbVi4Ue1dZ4mGkhH6qiau2sf3DemIoEDng15ZsgVz1nl9ySf+ZhYu/vN7GZfduqoi4zKySS7czt2mUKzZpxqLzL7ztb/X4oL73o283EIOIpgvSkboq1CZd/h9MuXoQ0Gw/nrGysjq5a1I+iZVcvTrH52Pl5d4I30ecEIUxENqeW4HefS6MaHOzxVpayeygPssitXfO1tpPS3FT8kIfaVgV5znkY957Cefk+Sn/el/loUqK1N77glHdwh97kcbPlN+rKIgF3I4LalNkWmMTUWm6WeLYhuh5YKSEfoqlDZ9MOnfuaZ/1zZp22YuDxf/Jex9+D2n4348A5/tD57AXlS3Cxf96T3838yPta73hT+9i+N/MsOkigCAtTmIee8+ih1ZmrjPNlFM5B5qaLKRKkuQshD6KhiT/63Luh37ccvzizOeOJMJ6i0hQ/yG1W6OOBq6aD7Zd7gRtRoTxFc+mIhttGKLPLKhOBqbv2EX9mrEyd+062DgMVGTNnKMSB845c438NicT6IpTEEUE7mXPfABRtyRl/iIkWEV/XRKRuirsgOpNF+P0De85qGGJtzw+Hw88eEGLNtcfJEun59fh7VcwpA/z1qjdd7ry7dJQxwHCfCwH6On5m70/B5d3SlcQQaIdY3KGWbTroO47V9Lkr+zl34z8wovKAFvmlwvUvvi/e+jetpLWQsVEgUlI/RVk3vbnVjsMnjXPNMG0hRnyQU9TRE1rig8U3TdDW98eiHO/d1so7LX1u/Dtf+oTSY1cTnUkPoIzPpY7tYaVfeLxRIT7GuzmOFKfIbZcuHM1vyEtennltXb92HBxl1Jx49CVgJLRuirUCXF4LuyqTZHlBLSGZt3nNN/duFxgccGTZSayKfDhiEN3NsUG/Xry7cn/26KM6zYmt7owwpOWU6DL97/Pib+5q1Q5emQrulnR+h7VgFH6FtfrN47UQvLXK1XmHTvW7iQczl1r3rgSCPq9/ornfmgLIR+h1bNfPfxnfnAEfNQ/xRxKITeHVsFHnPzs4uU+7MhoNxRULOKxP2Kk6n/+sgblEw2ARi2WrkKhaG6xql3vYlV2zLLwiQbSfJCX2feRJfGIhX6SzapV9iaEnZeLFPcV33Rfe9h1C9m5qUOfpSF0K9Q2Ez4fmiaJo8xuaafibYiq+r/PL0QZ//fW1hbvw+7DhxJE7Bp9Qp99eAy/VfDeq8qM7dF5h2VA3km+0C9t+bTyMvkH2eU9udiNe9E7UGXtXU1Abj3sdJRFAopAJ5WEpViR/m8uX23/sss7nicpWz6n3Kufet27Mewo9oZlbVqu799+rn5dQCAib95Cz3atQiuVxaHtLsOhndhjCwEdA6kfjY6aVCJKuXElGKKveMJhVI81U4iy8P78uKtaNkstbJ97Y79GCBxy84HZaHpq9rRocbUBOSSTSkNXafT3/vax8kh+bcfn5/cHsYjw/WcqftM7Z7oRn1UoRciOlzvOv+PekvlZc/gYx/zyO4DDaie9hIe/UDuynhA8BTKhWCQXSLTD4H0fG5TlJ48vNBv07ywdbtMPOj8aJXDUCKT7n07bdtLi7fg2n+kUqcW0sdMS+gT0WQiWklEq4lommT/eCKaT0SNRHSJsO9uIlpKRMuJ6PeUizX0AqrO+qc3V0u36yhKD7+7Xro9kxtsVpH5d1jHk8hUEQwukhS/EvCTvTwbdh4AADw2Z4N0/wNvr/XWJagqESBrMx9l6MIYVO8ouwZv0y/04HGeRfERVbVVVaF96ArnHQRKGCKqAHAfgHORyHd7ORGJeW83ALgGwOPCuScDOAXAcADHARgFYELGtY4QPzt+JrP+mfTdCGS+lnlHJtTatgjfUTKxTLiLwnTvPRf2UdkVMk3eElTtigiF/mwuGmyBy3wh0m00lS3WmEm5QKeXjwawmjG2FgCI6EkAFwBIBoZhjK139om9ggFoAaAKCeWvGYBwsQUyQNXo/bSrfGlHsQhaq873SnbMCUd3wNr6cItKxGrLbsOvWne9vAKAvtDLxZvJhklc1qZ4IRfFu3fh54gKXdP3RrqNpsxCk/mF9Ap0dKteAPglkXXOtkAYY+8DeBPAFuffDMbYcvE4IppKRLVEVFtfH328eqXQ99meSUfJRGBE0fF1zDuiRvXe6h2YvWqHIuSBukytx+Vz0IfrEwtadCcyg/IjPD13I9bt2I+an78WemWkbDSR7Y7r9+pvemYhLvpT+LDDbr0ZY6hdv7OgPEkAb1vUifEUlrtfWYHqaS9lrXwVhfTEdYS+rClq3QMRDQQwFEBvJD4UE4lofFphjD3AGKthjNV07dpVp2gj/jp7rf9On44WZwidF/WuV1bg4XfXYbvGpKtIFB4ceuYd7+8lmzPLQPTqsuABXFCtdmm6zPLl7D3kPeflJVvxw+cW4Yx7ZmHHviN4utYbwkEXWV1fWbpV6qmhXaZsHpfb5vfBf3ZeHT7aEH4+wRWqz83fhEv+8j7+s2hL6LKyQTYmcmWP8k+aYUayQSF9Z3WEfh2Ao7nfvQHoZra+CMAHjLF9jLF9AF4GMNasipmzKMSCj6Y4wy9eShuUaPHJpwfw0/8sw/WPzjM+N1dhGMRDgkYYuWiz9YpwGR64yot29g/Wen3pQ4+6fM6bdO/boUNH5yuKq/sM3IT0GwxHP7kcGURlltmm25bKEB2hPxfAICLqR0RVAC4D8KJm+RsATCCiSiJqhsQkbjhJmgFhGhJjzMjXWSasPwsRlrZFM3NXs25tm3t+60xCf7rfrFOYmrtkcjGoiL2H9VZE88WIH6tnaus8v3WDyKVfw7+yv3t9VbgyA+4/E5PiZs4sV/fZAWm5s1YmTKemUUvd9vTfRZtRPe0l7NN8T7rwt11EywuMKKR8DIFCnzHWCOAGADOQENhPM8aWEtEdRHQ+ABDRKCKqA3ApgPuJaKlz+rMA1gBYDGAhgIWMsf9k4T6C7sH4nKY4M/IAkLlamngAXXdqPwDA8b3a61/UQbyKzmVv//dSz+9ATd/wEQZNWmYCnww7W14aqgWtK7eGM/FIff9DleRl9qp6nHznG3hlScJss12I9ZL+KswemjtH9Mc3Eu7NG3ceUB1uDN8uii3Noy6FdFtaPnqMsekApgvbbuf+nouE2Uc8rwnANzKsY1bxCysQRdRME6HvxgcKY9MXP2o6GiMfFRMINiuZaqFSoZ8DZSeqj0A2NLMg5SOspr/YMV9+tHEXJh/X02fuILXR1C04LbVAxI+GL08WsluvDKa9zsHk2KgoJA+qsl2R+8j767FGEZqXMeD5+eoYNzwyU5DJi3YPjUUxkatxXdM2b6zpy8w7muc2NMWxnzMhTD62h++xQSuYw5KNPirV9BnDsJ7tMrqme55/XCTv6E/HQ4w34bjKiysoo/4g8qUdaAgr9LNzbFQUUliMshD6ohBkjOH2fy/FhX9817cBmy7Oki37dl/07oMNWK+Z4i/MAh2xEetEWFQJCBlxxrA0wMNHPF5nm4xrHv4Qx/44lUaxR3v/eEP3vOoNqJXJJ7MrNzeSjT7qd/uxmHvNzC6aajqyUVZqm84z+smLKfOfO+p1zzOp5qGGJvx7wSZ1DmtuX99OwVFmpWUYHBtV7gsTCikAXskL/camOBoaRaGf+H/v4UZf90bTDnisJMCaW/aF972L0++ZpTzfvVoYP32xpne9siLwnPTFVOrrbtx5EFN+/452naTPT/ORvrva64GjqhrvvcMYw6EMVs22zXaMGj+hn9Sgw+GaRFRC2fQjNtdZOwGk2rH7HnS7xpy1n+LKB+fge08uwPuKCKV83cIKZJP+mg9TS1VF7mIBBVHyQn/sr95Ihjd14V/58b07SM8z/TDL2tGnzkKTdRpafsq8E3ytSUO7eX67Me5dZq/aoX09XUyja7odq6EpjmnPLcKW3Qez4r/QwLkJPTpnQ2ZJM7jHuHyLf3jssFeQT25zZpOQwuiPTvyoz5x1JXwpvTq09L22ik8+TU3WNolCX/MJfOmBDzDPCUi4R5VKkyvu7lfChUI2Evp5ULpbNS8coV9oUYkiZ8e+dNdEvnOt9glpbKpxRJUYXce8c7QwBG7Xwj9JTFSYrpR0O9bsVfV4cu5GbNtzCAO7qUPLih8zHfjn/vz8OsWRwfCjLJVbY2jbu+91E/9v33MYhxqafN12a9fvxN5DjThjiPw57Tuc0Pif5dxWB3Vvg027DnrCa5hWP2XecT9OhgUEEMUcgUmd8mHeKaRV0CWv6cvgH/8TH8ojO5pqRplO1LiXE7135m9Iz6YkVq1Lm+ZpxwRhakUyTZLtPj9Xi21iwJAe6hwDM32icKrYfzg18ZfJqlVAf2Fc2A7sF9rB/dh887H5uOKvH/ief8lf3sdX/z43sHz+g+WWvW3vIe44s3q7H/CUpp9YJbwtxIpzGWJ9wqwDEMtQecHlw7xTQPO4pSv0z/vdbCyuk0886rxzma1/dHUn3+Mz9S92ryba1t1Eyyoy1Vy27TkUKMiCrnDhCUd5frsdyx25xOMsqcGq0lfK6Ny6ynefaLrz40hjHFt2R+Pp8/qK7aFi+vDP8C0ueTwvn+ZnEm5B4gHW3Um68wKXbc1Us/7LW4kFbu76iC27DuKmZxbi6r99GLquPGJt3lsdbJ4U4QV5PM6UI+9sJhnyvabV9LPPsi17cNsL8kxYOo0+zoCegtfIhMGJuEAyTx0+AUuUE4KqCTCXMJonf8rn//BOYIL0IPNV80rvM3EPr3QEUFOchW747Vv5C31dbv3XYoz71RtKP3CT6n3x/veN68CXv7guJdyj9hnnp3gGdG2dtt80Ufff31vv+f23d9cBkJtO/dHz3gHCPQ++bS1TzMcA0SZLF9e7+FFAzjulJfTPPc7rz70oA02/Kc7ShogsadtU0xDmDftUitcIxXq4ZNqGt+89jF+97PX4Gdy9rUcDDdKOxA+p2wljEQj9KIzIM5ZuBQB8stNfQze5imk+5UT5qSvw2nimSzOO7pSYrB3co61TXqpAV4DybXnG0syim/MmtSiQPfeH312H383UD3fBlxHUzqJU9MVYT34UVRiGYiLIZmwCn//WJbkIJkAT6dm+pfH1El4cxqcByM7QccYPxuOZ68clfweZkMTdj7yfSH3oCpsmxrgQv+HqJBtBud4pQex1vEemPWeWB9mPUPfAncNP2IvtzNT8cMbgxMRuOycJDt8+3b+iTNASpEmbInuWP/3PMvzfzI/1y+D0rIUB809R9pdvPTY/+CDkZ0GYHyUl9HW/plo2fcaSafxS5ScI6j4dDW3Wbp10u6VYfR0hsW7HfvS/5SWjITkvPIKGxOJedy7CVTAZY6G1HfesXh3TBbxpB25QRMjMtocFX3qMKHk9UejLJu9VJP3znf8rOa3+wJHEx46/xJVj+gAAVmzdE6mpQ4Xq0UahBfPtICiPdJT3rBs2xdr0s4Tuu9RpZLKGkfJIUZ+7OEQoZwaT2CFivYLPOeOeWYgz4LZ/6Wu6fG127FO7bPq1afee4izdC0Q1QSsru7Ii/fmYdmDV8SYlhVL0uZP4Vy2+dlOZlPLzT9jrW3JzTq7X1Z6DKY+Y6s6tsXr7Xkz+7ey0Fc06dG9n7i2mIj22T/oDeHXpVlRPewm7fSLX8kJ1+uKt0mPc5xylANYX+pFdMmNKSujramp6mn76NneyMkg4N/iEQVAvRdfX9EX3SbPViNqHBt4n3wH9PqRuCQs27krW061uUBhpcQQji2RqHvJZqXLqE6IT88+IFxbiczYdcbin7z7YgMm/fRv/4jx1zh6WmOdq0Sz17JoYS37E5603G1UA8vcgIoZ3Vt1RmtCXHOMmQvLz1uLP4d8xP+qOJT3JFJUxRHcF/dbd3tHHD59diEfeXx9dRQwoKaGvKwB0jpKVdVSHhDdPWPPolt3+w04Tm744kjAJdTtzeWIST+daQUrM1H/Wpn4EaPqyQ4Lq0JT8SCT+/+VFxwMA2rdMdWTT9RGquQmTksJoi/wpFTFKuVgKz8G0ZFfw/O71VclV4C7uMx7GhQmJM5ZcxR3G1TjInLj7QANOvetN7fJUI2/33bsjlZVb5fMJ/Pvo0iY1guQ/AO5zjnJxlq7QX7vDuwj06dq6tPDmuaKkhL7Ou2SMaWlSYsN+6OqapAALOyUmxjkXMQ2C5uIucXeF4fUTBuDgkSY88PaaNM3WvfVjurcNLDeoPnO4NQR+T5QXaD98dhGAhEa673Bj4NBYrHuPdi3w5bF9PeeZTnqqgtGZaNhhxAYvmGKKiVxTmaR6jG7ZnVpzweTiKSeFMPbtoA/t/iNmi6tU9+teytXwf+QjKPkyendMrVjnq+q25yjNO7pzZMN7ycO95IOSEvqXj+4TeEyc6XVYXhuYNLQ7zhzaPZmhqn9X/3AClYoeqIq0F0U7dIVh66oK/Hbmx/jl9BV4caE8PLS4BkGGyYiGMYae7VvgB5OO8Wz304T+9ObqQI8St3O6HZcoIeD4dIUjjjbrTFFN4mW6NqIiRlyQvczKVpnhUkHSUmXyc9kL63Z7RoqL6nbhqw+rF10FCX3ToIFiafzt676vRolGz58fjzMccW4834uz8h2SoaSEvhiTRkaccx1UIZt0G9u/Mx69dkyaYOPhG7zoYqg2JxsY9SXs2Hc4efpvXvsYHzua0T4fn2qdZ2Ak9AE0r4zhmpOrtY5vYiwwd0Cyw3I+/weONHmCd9X09V8lLUOl5WV7IpcnRqnOLwpJU/OD6j3JInjGmddp4Ef/XpL8+/w/vos3V6avDeGp6dsxUaZPNWWvVem9k7aTE5DO3+P6d1bWqYFbXHg+tzrcfZYruGxndbsOYswvZ4ZaVR0Wvt31u2W64sjsU1JCX4c4Y7jh8WDfWj8N49RBXdKiWvLwUTJFAbPvcCOONMblsUUyk/mo+flMjz3X7bh+WoVWohWDGsVZQuNs70yctdTI9SvT9PkVs+7AKGX7JjwzzxtUzVRAqrRUk6LCKGseRQKUrLvolWQ6GlG9J5mmH2fMc8asACEv0q9L+ipfHtNEQOneO96/py/egubN1KKKb8/8RLOr1btzWQDw3Lw6bNtzGE/O3WhUz0woII9NPaFPRJOJaCURrSaiaZL944loPhE1EtEl3PYziGgB9+8QEV0Y5Q2Ywphe6GFemIiCU9WoDzWkNA5RIP151hpc9dAcHMclCEleA2aadctmFbhs1NGBx/kNZf3aIO9GaWre4Q8/GLA8fdnmPVJPjON+kno27vNLavpSDdLce8cvoBc/odinUyu8GyIGjAq+/CbGkh+1SiGetqnNWfWeeJfZ5LUzNG+4H06/64bJCcHDZ7Tbc7AB33psfuCHib8lWXz+e1/7mNvvOghkVE0jispPn4gqANwH4FwAwwBcTkTDhMM2ALgGwOP8RsbYm4yxExhjJwCYCOAAgFcjqLcvd18yXLlf28NHcZyuIiOa8Bub4r4B1BJCM1HwrecNCSy7bYtK3Hmx+l4Bf5OS3+29+J1Tk3+bdF4GGA1V/D68vEBKmXf862MqwHbuP4LjfjwDqyQfHNHmviniVIx8WJDJYo3TAAAgAElEQVSmOFNo+vpl7j7YgD/PWuO7P+mxEvd+cDKRy0HPXFa02kPH+/ueV1MC+iuaQd08OYB5jzGW3peTK8NzGBrh2n/U4rE5n+Tseip0NP3RAFYzxtYyxo4AeBLABfwBjLH1jLFFAFTN9RIALzPG9P0LQxDk+60rJPiOJzZK3UVUoqZfyQ07l21Odz1zi506foBW+Toc8ZEgfh8//oPG/11VGdBUJOapRJo8jUr68M7qesxdvxNrHc1P9tjd17nhU/9mxUeYdOFtvDLiJgsnNPh4215854mPPOW7bVGc/A9STHYdSJnxbufs8TKSvumekWt4TzEgPQBbpqiEb9B7cvF46Ti3durALmn7AK4/51j5/ul/luX2gj7oCP1eAHjjV52zzZTLADwh20FEU4molohq6+vN7IsiQVq4rluyp5OkXUNT6Aut7epx1cm/Xbe2hqY43lixDU/N3YgDigiQIrqamnRlcZx5FvDw8PfWmYvTH7QKU7ai+HtPLvD96Ojw25mrcOlf3k/a8WXP3X1Pk3/3tm85339qQdq2p2vT7bm8rBXt3pkiRrZsirOk6a1SWOwUZLK68emFyb9lygOPew9xYQQVcWDPQFSRPaOwfIgfNYCL+yT0AffYZ+bVYaXmRyUqVm/P7fVk6Ah9+WjNACLqCeB4AOnGbACMsQcYYzWMsZquXbuaFJ1GkEDWXYyicusKGxWRH8ZvdhJd3PnyCnzt77XqdHISdDU12X08N7/ON2MY//g6cfb9poBk636KcSbuaZ8I2rvKvGPywZSV7XL+iKNw0Ym9sPtAAz4yTByjQtTePeadNE1fXRbvG743oN3IbPpiTKmokVVfpeUuChG2RMSrpHmfq/js3Wexc/8RnPNbf2Uhao40xjHp3txdzw8doV8HgJ8x7A1gs+F1vgjgX4wx83i0hgQJfT9vDzEOzHJOA2gumDbCTlQxiTby0DvrQpWlWwXZ3e5UpD70+5gE+WYzJq9TUMweFe66CBfZxzbIXCeW4SLNV8sYqipjqIwR9hxqxONz5FnVwiBWM85SoabFRWpBr5ave9BIyi2atye/tmyb70ffD9lzjmpu8ruc2SssoscP4K/p59yuU2DoCP25AAYRUT8iqkLCTPOi4XUuh49pJ2qCtHA/ISF65Hy4LhUn++cXHhe6Pv9ewGUsEswHuUB2HbVft3x7cJRNJv1gfP+p8B1avKRsLiVo5HaUT+hlcQS0fc8hbN59CNMXb8nY+0R6vTRNH5z3jvd6XXw+VC78qmLVBxxIPbM19V6f9M2GWcSOBCTZ4cnV4iPRDdXljRWJtJvuyFpU9PxiY4nMWrkds1dlZm4uRAKFPmOsEcANSJhmlgN4mjG2lIjuIKLzAYCIRhFRHYBLAdxPRMm10kRUjcRI4a3oq59OoKbvI7xEn3FelnRuI2qc+kLhe0+m7MmiW5ksQYouujVw23tXR5B8d+JApWnI79504unLTuVdWE3RWeIeNGXgJ4CO7dXe8/vjbQnN98CRJsSysHpFrEdTPM557wgumwEfWBOPJb+PuGyNxAFF+ATZZGuu5wVE6jjvKv6RuBPNrius+DzF4Gd+XPPwXHz5IfOUkH+4/ETtRYr5QCuvH2NsOoDpwrbbub/nImH2kZ27HuEmfkMR1GF9hb7QO1QaZFihIObxvFaR5DqIzZoN171i59ZVqN97GJ1aVylNNb5CP8imj+jT/umgCm0B+NvHxXzH/OvPxn2I356Epi+36Qd9YE2CzPm9TzFT2uxV9aEEXKEw++N6nCCE5Kj0Me9k21Xz8yOOQpsWlZF7OUVFya3IDavpi4JclaA67OiVF/rVAasa59x6ZriLCBxqaMJNzyxMpvd7/qNN+PlLy32PJ58WIRM0fTunwl6Ii7NyRZAA9DOjiR2fF/QqE6EoWHQRq9nEGB5z5gzEPKtB7csoMqbmSwkS+CYfmnxYzH/zWnqWrVhyIjc711Sv5cnzMEhBdBm8C4Sghz3xN3IrU5vmzQBEuxhHhG8jMVIPj1WB20x4bl6dJzyDX97gVL18NH1Jz7l4ZGpw52feiZp2LSo9nk5BphC/3aIQ4+uuakNRpbSMxxkeeDuxqEp0Zwya7wkadfFEJXxUCeVzTVOcOdFy1cf5ee9EharYiLpvVig7TV/GpKHdI88GJINvfIGrGiPqrKYNXmysU8f3ByDXLp+auxFDf/QKpi/eklGOX0AvQioA9BCigwZ7Fcn31ylW26raUHjPLe9v3oQjxsAP0kxN4g1FJXxMigkrY78yrq/WcVN+PxsDb3s5bXujMMHj572TzXlmN7y57jUefncdqqe9hOeEmFLZpASFfvq2zw3vifu/fJLvOSP7dsBNZw/OYq0SiBO5qgnVVlXBActMr6mDKNRuPW8oxvTrJC1n066DONjQhF+9vNwTRiIMpwxUR1F0SbeNB08wyxDdMe+ZkUobqIxlI9n25Yfm4J8fJFwil2zajU+FCegtuw+mp0SMs+QzFVedBn2oVTkB0uubPZXTr5phvXfED7of7vMSn6m4ZsHfph8NsnLcD82/F+h5tbvrF34TIm1lWEpP6Euk/qBubZWx2wmE4wRvDhVR2PSDOkaLZhVakSpNrqmD7DGl+ZGnCTDzgHFREaTp62rFtZ+k0gaamndmr9qBH72QCIfwuT+8g8//4Z3kviWbdmPcr97A72au8taLq7fodhlksjKbyNU+1JjDPm6cSzaHW2yVaZx7ca7K9YoS+0BUj0TWh91thxrNzGG5nAcpPaEv6ZU3TByICkU45KCIkFHBN5LmzSoCF9Z00kwcrr6m2fGy5ycKfdfk47Jp10GjHL8ydDVS0Qz02rJtnlg0ImEEiUpQ6pjdeM8qdxHUsi3ecAkPcovy0sMEqMt370mcAJYRlZlQViW/aKWDugVnZZNh+qrEtv2Z0A7yoem7YZ3btTCbLs1lEM4SFPrp24IiJvKhXHUI6/I1Z+3OZKJmnSEw71F05pBumHXT6cbXjGISSxT6sg8DAzJS9YNOHdgtka3sa6f2S9s3c/l23/PC5ENV2/SNizMm6J01xON46+N6DPnRK4FlRTX6ysViwkyVHDGOjuu0INY9qsVjsmIe//oYAMBFJ0o92AuCkhP6fprNnkP+ESCmL95idI2wbeb5jzZJY6H4wWu/V4zpo23z5DHtrDKvIdE0JnvCyzbvzlDTV2OyIpQnTOx4lXZsngow+Pri/I3rXutHU5xpx/qPynvntWXbgg9yeOT99dLtrrCNxxm2SFYEZ+qxJprJXBOL2ATE1clhEd/thGO6YqAzyunRzqyv5jLMc8kJfb9k2yrzgSsXVWkQdRjbPzh1n1sLrcxVghuhLGy0qwH7YboiVibwxHkSmSDZse9IRlplkKlL9bx+/pJ/MK/ozTtmZale82AnOb0bAtjFTSDvRxQrck3xi8oq46+z10m3u9X+81trMO5Xb2Ddjv3S/bqIglK8V/dnU5wlV6RHidJl01CyWvNOBvg1cp3O+r1Jg7D4J2eHvrYqYbpYD710hX4/UjSryP4rFDV932ecwTWem68WKqr3t+uAv2bsJ0jGH+MfzTVKl00/JQRICS3TMuOa8ydd2zYP/SFu07wSK342OXXNCFY4vb/mU8S5UYobadbFVNsVn1uHVl7zEO+y2cFxpdQlkxAp/LV1sRO5GeDXgXTfgU4H7N+1NTq0Sm9EOpdwo07qfNm9q0TlpWc6JNYRCmkN2PcZh6/LFaPVqR/DakJ+Nv0+neSB2IDwE7mqmC7qlIb++zLhqaljQ0/kxsibkMjEY8iPqx6ag4ffW++73/QS4ohHnBNwd//hjVVYZRhV9OuP1JpVRkDlLSjDavoZ4Pesdb1DdARXq6pKLLg9fURg8p51huh8cZmMYDIl3byjrkt7Q60KCDZThcVPQ1W5w6oEpepxb/zMP069rF11d+y+A7q2wZfH9sUt5wanyXS5/+21WseF/RCLz0AmNMMU/cmn+5O5DNJGaIaSTzQJiiMFV7ufsVR/PiJZtsYcktq8Y/pwrE0/NGLsexfd4VYmQtRkIYyOeWfsgNSCJb/Om6lLns7Z/1noXWjiF4vdvaUwg4+KACNo1Jq+n90ZCO+9I7uUu0122skDErb8b54+AD+78DiM7a+3QE0XouiWZsmEYNhR5ibHrPPasq2e7Zlq+u6z7u/Etbph4sBQ9dPF/ci4c3l8bUw1/VxSckLfTwiGFfruZJv+9fWOC4qDDgA/+fyxgeVmOlEX5qPhtyjHtdGGKTOqWEMu702biL6dW2lPek4+tkfy77DmHWmiEVVCcGef2zb5osN4HYkQwmv6Ov2loYnh3gxWkop1M/U0E5+RGx+oa9vmGN2vk7Fd/cCRRszf8FnwgQ6pD3qw80MQ2QoKJ6P0hL7Pdt3oiJWCxnn64OD0ja5nAJH+ZOaTc9NztIrwych9NX3N6/mhc744f+HXnt2GHqZOph00iKM6tMSkod21JyA7tk7do9hhec8PlRBtkHgguSGLZKeJMo4XHk9ptI8gmlj4XLitmyfMX0Fmt9+/sdqoXL46/128RUiEYlRUmtBf63gDMYT74H33iQX4wp/ew2caCpl7HSD1seIVl6jbc5SUntD3edHDjmqndX5FjNCFS5ryw8nBdtZzju2euLbWFRKYanL+E9TZn8j93PCent9+DTqlserX6YVvn4KZN04I1PTDLA6qiFEki7P++pWa5N/NFCu7ZUHp3KurquFejr9skK++DrsOhHej/d8pwwDouSGbcuaQbgASJqN5XPgLcdHU0J7qPuv3btft2I/dBxuM733xpkQ4db+RrIhbX7cve4S+8USutemHxu9Rq15CT2HRk/vuurSpMvpim3g4iBEBAeCOC46VHJlAbBRVjqum+YSROWJVgz5AJu29d8eWGNitTeBz3hKQNOa0QV3StsWIoBt6Xgx7zdOmeQUe/uoo/PPa0cp7O9KoMuX442r4fNlRrICNs/BKgbu4KOqAbUTkUQr4uQLGvApFlc/8nIuf4lS/9zBWbN0baUz7TbsOonraSx7zj3t1t99XcgqBqZ++Ne9kgK/tWyFUxIUbKeGl12ie+DAxFH9szgbtc2TxflRB33YJmp+rVWYq888c0j3wmCc+9Eak9LvHHU5ceJM6uYe2baHv8eM3z3Jinw4ec1xFTD8MA29ukwmLMwZ3w2mD1Ka+6x+dl7bNLckvETtPd8NVnEE0xVnGgi/bZgr+CcQZ82jLW3ap81vsCRgNRSn0312VWFvAR2d1X5+rNB57VKr/Fr2mT0STiWglEa0mommS/eOJaD4RNRLRJcK+PkT0KhEtJ6JlTs7crBFGM0mzrTpF6Lb3MJNubk5Wz3UVx4v73Et+Z+Ig42vz3PPFETjeIMIo4D/putcJwGXS2dxjg7Q6nubN0o9ljlb7xytG4qMfnQUg0fHChWHw/m7fMuX/LbavTQGCyS1L6dmTHFmmlI8oFkON6N0h7V4evXaM1rmpPpBdoc+vyo0zb9vavledI/naf3h96acc39MjPKP6Xn28ba9yUWVNdSc8/62T8c0JA5LbTD+Wew414sHZa5XBA6MisKcRUQWA+wCcC2AYgMuJaJhw2AYA1wB4XFLEIwB+zRgbCmA0AP/oWBEQpo1OEFZnug09kwbfprl5UjKdmC+uTdltU6cM7IKvndLPvIIObZpXYlD3aH3kTZ5alAHBYpS4n47OIh1d05eoZYnv3TMSFIpsCLD/qu7PvarskDhLRdHsKFkIqEPLqgrsFyJh6gojdwI3aplP5C3zf52Q1EDmJq1eHVt6IpxGFWF0+ZY9qbL4KnJ/j+zT0dPewlz75y8tx/97Th2CIwp01KvRAFYzxtYyxo4AeBLABfwBjLH1jLFFADw9wPk4VDLGXnOO28cY81/BEgGm7aZ5ZQw/OMsbcycTLcc9I8ywWHWKayP8y1WJZDB9O6dy7Gbatk1HR4wx5f397MLj9K8dUceMS5K46A6xxdGAGBNGxXtrPlXub1DZ+ZOafno9py/egiE/egULNu7yjABUfGFkr7RtYppDnYVz3z1zEFpVJZSWqK07mYbgVhGPs6SJEQhfdwaW9PUHEu/HLcuTE8OR+lHez2eKkCJRoSP0ewHg/cfqnG06HANgFxE9T0QfEdGvnZGDByKaSkS1RFRbX59ZzAsTbSFGwN2XDPdNEhI0GTPj++PxyvdPk+4L43euEr6uYHDtvlHaAC+WCAsVcab2Yjl9cDftssLIfNkpshy9upr+bC5i5TUnV3sm5IKufeu/FivLvuuVFb77VEJj5bZEmOC563ZqhRC46+Lj8fkRR6Vt78sJL8BcGamMOLaTqOnzxOMMl9aow3GoYPD2f1Olze1/jCE5WgSA7z7xEZZuTuRD4PUD0TwXBbmw7eu8UWkf0yy/EsBpAG4CMApAfyTMQN7CGHuAMVbDGKvp2jXYL16FidBf+6spuOCEdIGna94Z3KMthvRoh35CxwKAk/p21K6HizI+i1A3JtkXlpMHpnu+qGAAmim+iGEmcjPFtenzqARcC25egA/89ZPzj016RskwHZmI+W95dITGw+/6rxzm+dKoPqjbmT6IDgqbdMWY9NzE2/ekTCSqZ+GiElTfmOBNuCN7T8l9SKTnrO7cCn+7pkZ6jIo4Yx6hHNY8K5MhDzlJb2TWnSg1/ShiHAWhI/TrAPCf394A9BJAJs79yDENNQJ4AcBIsyqaIT4z3aExj6lNnw/94J5y63lDja+rwhVgrqzl2+VVY/uiRbMYvjy2b6i4N6YwxtC/a/qHziVIMPITx+JiuLDEJQuRVOad4b1Si/XSJ/KzO3npauRJoaG4nhugTwtJOapcCHdccCzelkST5GPn6/QBlZ51zcnVwrHqBWNVlTHMuvkMTBzSHVdKPkhB9eA/QGFfI2P+yqMs5WmU7SWKCfwgdHrcXACDiKgfEVUBuAzAi5rlzwXQkYhc9X0iAP/g5xHAv5R3p03EGzdNMC7DfYW671L20mUeJkEoc7M6tapIavqp+6zu0horfnYufnbhcaHtmG/edDrev2WidJ9oqmLMzG4vci2X/SrM3IdsZMKQ/vxU5h3++YmaarbDpnRq1SzxTDVGpUF5BoIQ2yb/s6oihjpJRjle29R5PSq3WJnJ0s+MKQpa3eTiLoyxNBdQE7Y6I5w4Y77KIt9WUh9to8soWVgXLr+wCYGSydHQbwAwA8ByAE8zxpYS0R1EdD4AENEoIqoDcCmA+4loqXNuExKmndeJaDES8vSv2bmVBPykXI92LdDOwP/bxXQil+8YA5yY+mGEWZAGlDgmZXeUl6F/Xb6O/bq0Rs/28nDD04Toj3HG0rI9mcAL4zBzHzedPRhv33xGWp3SbPqKonmFSlSuopy8/I4k6FcsRog7AipKgTGid7rrbZh74fuQjo1d5RYr3h8DfO0hYjF+OXj9iDNvsheTXBOi66jfiJlf7KcKplfIaPkVMsamA5gubLud+3suEmYf2bmvARieQR2N4CcYVS/j7ov9q+QKe79okiK8AH70ujFYvGk3mld6hWLn1lVK+y4QYNNPfojSr+k5LqiyHGFd5FR22SAmDunmEURhVhRXxAh9OrfybIuz9A+eMoGJJ+aL9zmoUgOa1rZXh/QPaQURGKLzZHE/nEdJrpUuvFJX9Hv7fDgJPiYRkGh/onA+3BjHQ++sw3Wn9Utr9+L9ydpO/66tsbZ+Py44IX0imqdT6yploMLtew+hdn1qxWxrTbfpeJxhx76U10+cMV8zyytLt2L3gQa0591osz00jJiSW5Hr5qgE1O9C1SBMBRo/BO/SpjnOGNwtTcPS0fxV3jvu6UHl5KL9MYRf6XnNydW+tnZZKAWdfQDw8da9ac+cr2ONMLGu0vRV3jKmty07viJGCfszWCT24Aev9p/0FL1vxMvJQpE3NqUeiPiuZBrwn2etwa9nrMSDknDVouhkYGmtvLpzaxzXq11ylOzHFwNGHTOWbvN8wHVH2wl3X66OjCkVordWJeZBcpnXNkpKTujronphpv1Q1kDEBpf5ykbdyWX96+gq+uJxOjFdbj1PHqhu/DFdfQXduAHh48kfbGhSeu+IoTZUKzf5cmTmGRNkH/KZyxMjif2HmyLR9JNhQ7Tqk6KqIobvTUpf0e216XtLleWa/ctbawAgbSGYjJ3704PAMckaCxk6kXL5D7j2Og3m/fjGmToWTvI+i9S8U9JCX6VFqaMeGr5GSVliZ9EpUhaPJ1We83/MtenLbyAnmj4LXsMwdfyAtG2uGcJPAfv6af1xVHt5/BnZ7b518+me32mjK+5hvLNqh2cfA7Bx5wH88NmFGN7bK0z4Yv7n7MG++1RUT3sJm3cdlCoXa+r3O/+bpfDzw2T+iG/bZx/bHV1aq73bRPPbn686CSt/Pll6rF/ydN6DZ9uew+kmH+jNPegcw4fz0H0u4hqPprha03d3ZWMi9+QMFB9dSlroi0wamgoupnqppmbmMU74WX4+IW3Bl1mRabRzhtVuOX61z4XWwZieeefpb4xL/v2lmqPxxNSxAPw/qs0qYrhstNxNTyY8+VXJADBzuTfCBy+wxLPjjOGmZxbi6do6zP/EmzhDd1S2fe+hNJdEnlkr65XKhWxBWRjcMkyVlYoYGV+/S5vmaXZ7F1kkVMaAmuqUae2TT/fjBcErJ645o61zf+66ginDe2oL/ffW7PD0m3iAecfdp0qiEpagFd5RUFZC/74rT9Q6zrQjXHJSYg6bj5IpCo4TNRZrqS7r2jtVAbz4/ab7TLi0preWYOSjYd51yXCMqk58HEdX+8do9ytVN0Qyj8p5Y8mmPfjYWfXa4BSeXFCnfIapnaN/8bonebhIYzyuNBPIQkeEocLAvMNTVRFTrj6WoSNHjxaSzvNtRbbugLF0O3/Ya7u0qKzQPv6jDbs8dWRM3d7cUXZyRbXmdZ74+lh83zGnZeL9linmUcGKGD8NRWTJpj2hyufffYyA847vgctG9UG/Lq2xbc8hvLRoi/p8jcbjCgk/TUQlRC46oRee54bgj3xtdPAFBdbfOQVAQssNws8E1F4RQMzvGRwTIihc0IfJjXNy9yuJlH/fOn2Ac57/OemmCX+p/svpy3GowSs9+nZulUwMXhGjSIZm7ohGzHAmgzcLVlbE0KdTK8XR6eho23zT7NCqmd5cg07b1zjGuzgr/YQpx/fES4u9/bChyevuG6Tpu3t0XTaH9GiLFVv3YtyAztiyO7Eu4pxje2DHvsOYLZgdc0FZafo8OpOYQaniVGUREf505UkYf0xXHN2plZHPsIqghq/W9L07xx+jF/JCJthkAvWeS0cEHhOErKM+OXUsbp1ivsLZdK2E6foMAMqAJKLAB4Dnvnky7vzC8QASSoh4pbm3TdK/tgOf/+G3XzpBeaxY3ZP6mmXGUj1Stz0xlgj+tv7OKWjRrCLwQ6HrAqzzwXGzXskObdmsQpqzorEp7lGW4ky+7oOvL6Bv03/umyfj3WnpCx+zvfLbj/IV+hruViN66+XV1VlOryOAdD5Ebil+ZgPVhyrKNiZ20mOPapc0c/kdw/Ot0wfg2evHpW2XnTK2f2ftUZrf9U1u3eRYU6c9QspdWCZYZN4xQfBN68IT1cHzMo3npRpJ8iEd+ONUTb93x5ZpLpP+1w5GXAtz8zmpifjXbhwvbV9iLmE3ho9ffKm0FdwBNWvdvNJnvYbytKxRdkK/kxM9T6fxm07oqg7X0WR0FkulPizyY++7cqSvBwx/P307mw3rRUR3ONlHSOXh88PJQ1Ajse1HOSn26AefGB3vXttkwZhprJREusDE301xM5u+a1oTMRnRqIKj6RQTRnFwvZVkjO3fWXtC20QzbunMtQzpkZpX6t2xlfQem+Jeoe/66fvNd7ivPNOImNlOUON73bxcNY9M4IagQUSqGUf0pIMmctu1aIYJg+VmG17AvPjtU7WvKTVfxcRj9ExAQYin3P45MV+PPqdwMXpMEqQrRxVC/UwTrxNSz0VlQjDB5DlnupxI167OH7dyq/8cmWqR2iBh1GqihE0dn4jueaqwqE/2rMSPb5wltvmFCBG9d0z0lDOHdseI3u3x3TMHWfNOrjAb5usd7a5q7MTF4BbRilaocS1Xg1GHQ5Zfi//wqCZTdRDvRyb7MklC4/I1LjibKSdyi3kOHPFfAyFy18UJm/sPJh2Ttk9sE6bKHlFKeB1uiHtWv4Yl6Dnzq5mVLqQa1wrqE4camtAQ95prVCOn5+bXJUJoSPZ1FPqTyajI9aoSP+Bj+qX7wccZ8/SNeJyBseDYPe4kbH1AWkee9i2b4d83nIp+XVpHnqBGl7Ly3gGAWY7dUSc7kq7MOr5Xe/z8wuPw+eH+sUN0XrCsQz41dWwyoQaQsA++8T8TpHFWXPzrHV0rSwvZKynaved2LfSbmel34vGvj8EVf50j3Wcc18c5/Myh3fHM9eMwsk+wm61pDl4CJTW8D9fvNKufD0GjSF4ZUc1lnTW0O15dtg1frJGG0QKgfj+jqjtiyI9eSTsuMCY/Sx85AultzOR1+h1b3SVh1mxdVYG2LZph655DaZp+Q1PCvONnNnM1fTeBzgNvr8W3zzBfuZ3tpPN+lJ3QdwM2zd/wme8xY/t3wgdrd2oPv4gIV43tG3hMEAMlsUfG9O+MMf292kn/gBglfoRPH5eOeDsybZOI8MuLjjcKr2Bq01dFURXrNKxnOyzboueOO8pnLYF4m0s3G4bCpegX0AWFG+D3qjT9P14xErNX1WPiEP/MZ7qjN/49BmnMH67fiXH909vIoO5t8P5abrGSwYPz62/8dlfoNsaZ52N4z6sr0aJZTFso7z4YLsWhtennGNWkqRtLO5veLjIyNbn48WXngxT2fmTJv/1STIpcMaaPNLOYH6Z1VD1XUdb89zunYs0vz/O/ttmlAQAbdqbHo1eRMO9E29mDFAp+P2PACJ8YNrDAjEoAABshSURBVFWVMZw5tLuyPHePrAw+jMjm3annUiUJ6iYiG62I4TFMnpuOvHajevbt1NrzMVywcRc+WLtTusIYSHdaCAoG6Ifsdrq3M/feMqWMhb7/PpMAVrq4ZfXu6G+WyRbJrFshhU0HyVyFWFYuMnbJUGljovCKxSjjIbV4dpPhUmECcKhRf35Bh8DIq9zfccbw/DdPxupfnBvqWu4j/YLENZRf1MgvOjqxT7Drs2yEd/HIXph544RkngCj+TiNtu6miuzZvkVGrqw3hDDtAImPSz4oX6GvkPpuH4pSIQu18Cck6en/nP9DlneFJB6OKGd+d5leiIuoUc3NGCfG1log5P1tbNMnwlYfDTIsQd8xWZ6BsAnP3bK+Mq4v/nLVSVrnnKgxNyJ79ETkWXciez/NKgjv3zIxbR7Cf2FVeujlJuY177QNiMN/WFh0F2ZtBQBp1rJcUL5CX+mvnJlmLIMPn/DhbWdGVq4M0X2SX7EZBpkmyZd198XDQzd8EVONq4fPmgRAP7SuCV8/zZvoe88hs+xOmdboprPTPYpk7fScY1PBBV9YkAq90TKimC9EhKpKXft+NPh93Hq2b4lOQrRQv77r5mTu17V1Sug73jou15+eHiGWZ/8R7zuP0vUyynUqfpTdRK6LSra48WlM07Wp4P3ru7X1F1RR8OTcjZ7fbuPO1iDjUoW3hymmdTyqg/+zNC1L5/BB3duiR7sWyXyqphBltipWJmBEL6VVvzjX88HjRyOyRCXL75CHSg5TF/lxoYrXKicVEkFUdORltKyqwMPXjMJwLrWkOFoLqm+mi7LyjZamT0STiWglEa0mommS/eOJaD4RNRLRJcK+JiJa4PzTTaiedXSi3Jn43xYizStjeHLq2KyblrKxyGRA19bKlJYuKm3eE4YhwiqaRqbkyYYmJ5oqm1XEpO6qp/qs7WhZVRFqBKA7ksr0nvkwJ7oTnfy7f+y6Mbj7klRbOmNIN3Ru09xX029oVAv1NPOpVo30yEU2rkBNn4gqANwH4CwAdQDmEtGLjLFl3GEbAFyDRBJ0kYOMMXUUqBwy88bxmHTv21r2yCg/6ClNP3daQr8urTG2f+dkTJRMGudTU8eG1m7DMOGYbvjiqOCk3KpJTN4jRJYWUGRt/T6tIHRhkrm7EGXWsWUfbt0mZTr/EKYuMjL94PLRLHu0a4FteyTKmOLWTvH52Lkfxrhg0/+/mR8r6/PgO+vw4DvrlMcAwI1nHYMjEs83l+aVsWSAOJemECHETdHR9EcDWM0YW8sYOwLgSQAX8AcwxtYzxhYByEGVM2Ngt7ZYf+cUtFX4d7uETRwuw9WG8zEwTM5ROI18gmZ0TZ4x/TvjghPUwbyioHObhKeQ7hyB7gKsB78yKvn3//pE7Hx5yVatsvyEiA6ZNinZB0c3FESmQt91ZXbRDS2SqSa8eFNiLcTyLXt937d4ZzofJHekImr6pvjJie+eOQg3nTNYug9IhFcWuWqsPIlQlOjY9HsB4I3EdQDGGFyjBRHVAmgEcCdj7AWDc/NKlEI/ltT0IysyEPdaSW8k+AftKhQuGNELBMLnhvfUOp43MYhCiX/WfGIPWcRDIBFCQIfrTuuPx+Zs0DpWhCEzASMTerornhvCZKLhry1cWtu8oxOCROOZHN2ppe8HRBxB67jmehdnhSfsuTIz4UUBUVKjQKe1yJ6eyX32YYxtJqL+AN4gosWMsTWeCxBNBTAVAPr0yf6XTpcoBXSrqsSjdlMr5pIYL/ULnFiMAsMDe47nBEozoRPx749fIOSn8PbprLeILBPzTovKCuUaET9+5ASek127cxu9UVGmmr4oSHVHWVHNp2zYeUDb1VSnau79JGLtRGty00EWujkXSqGO0K8DwBtXewPY7HNsGoyxzc7/a4loFoATAawRjnkAwAMAUFNTUzBT41Fq+u1bNsPMGyckF2f98YoTMWtlPS4e2RvNm8WwPYv28mSjLJgnGx18vxE1J15/4zum33vVXThnHNPH4dKTeiMWI0w4pivuemWF0bluLl4xXrwJmQp9UbiphN15x6dMFzpPS6evdWnTHN8/cxCueDAVa0nMYpW8poYgrowRYpRIvCK7epc2zbFjX7Azh6hs6CLT9HPRRXU+m3MBDCKifkRUBeAyAFpeOETUkYiaO393AXAKgGXqs0qXgd3aJKP/fW74Ubjn0hEYN6AzRvbpiMnH6ZkzTHA7kqvRlKDM95gYxBgvvCAQV6XK0P3GB2n6fuaj7u0S7qVDe7aV7lfhXvHPs1Ybn+t+zExTI4ps2uVdTKR6DG2bN+OOy8y809YxX43s0xFtDIL3BUFEqIzFEouzJNe/6ET/AIo8gQHlfJDFJMrG2hKRwNoyxhoB3ABgBoDlAJ5mjC0lojuI6HwAIKJRRFQH4FIA9xPRUuf0oQBqiWghgDeRsOkXjdAvcnfcJP9ZmBiYzfvEP8hcscKbHPwyHQHAEc4twu+96lp2g4SY72rQZCLt8B07jCnh6nHVAKCMzBoGld2cf5Z+VZ5yfErRUT17954rYxR5nyRyXV4l+SACPu7uR1ScS9JFNkLok2FyIx20PpuMsekApgvbbuf+nouE2Uc87z0Ax2dYx7wRpXknH7i1X7E1EZp5FReiuVTgBWgzYYWo3+vL9L0Gafr+IQAyumzy2qarRy4+qTdeWboV153WL6Nrj6r2hlNQfYC8oyz5cfzpqmdzwQlH4ZH3P0HLqoo00eza4sM+2ooYJVw2JQUEad3DerZDswoKbe4LGwojU8o2DIMOxSry3QBX35noDQRVFSLPbDFRKWj6vNbEd2pf27bmCw/q5NlYBOcWGSZgXKfWVXjumyejZ/twmr7rPnvzOUM825VCn//hcxh/vkro//jzx2Lhj89Gi2YV6NLGG/zPPS3shzxGhKa4/NUHvccmxjJ615Mdl80ffz58drgwWKFfgrR2PIU6tkp0kG5OpxU7TKa0aV6Zt+iaMsThMl83Xib4yQdVHHmeIMHrJwhUmdV0yYd26I5sxFXsKj99/hn7PS5+u8q8UxGj5Lvs3bEVHvxKTdoxTwmhR3SJEXw1/aDvazyeWbrLEUd3wPo7p+Crp2Q2AjOlbGPv6FCs1h3RWecXFx2Prz9SG9r26Me8H02KtLxMETV9Hl6o+CV18Yszn34dc/POvV8cEcnitnxkW/L7iKnMH/zzlsXTX3D7WbjjP6npvbnr9eebjpZMSJukw+SJJc075p09U02f55GvjQ5MNhMVVtNXUKQyP2nndoe8LZpl5zU3r6xQJxHPMaqYOLxFRyY0TJB19FvOHaLc/4WRvbUF9pdq0sNPuO9Uti9XiPelnJDmnresjXRoVYVdITNOyeYC3IQoplSQI/Ql+56urVOeG2VYi/HHdDXKMJcJVuirKFJV/+KRCY1ycHeva2Augjnlkz0KIdKpVbBpxXWnDUKm6X9jQiocb6bKOB9DXmRlHibjk/kYxBW5Su+dYMIGNOSv2tHJNhd2BMSQiI8v6+pBsabiEWr6ucQK/RLkghN6Yf2dU5IuermI0V0IqARNUCrKUwbqa1nZnshVzZPkwo9bxC9Kq9K8o6EwhfV64fnXt05xLhju/J37j+CZeXVShSjoUTdlaNPPF1bolxFFOnDRJh/2bhmZhpq++KTe+NmFx/mUnVHRoUgl4dGvi05TyyBCNQBgULc2qDbIv6wkRN9oiltNv2B47pvj8MuLMl8eUCoysgjbZShUi7NywbRzh+Drp/XLeFxVEaNkMnuRKZqB6KLEvZ+0gGsq847Qeebelj7pH/Yj7bZn3pMp074qOz9ISWq0mn7hcFLfTsmkx2FwY68X4ftUYjV9f0xNYPxqUpfrJwzAbVOGaYccDkLWhjMNpRCG1MhFP/aO2NRkobLDaskDurbBNyb0x/1cTozvnjkoVFkuOn3jSuF9JMw7xSclSlLoZ8pfHT/gKGyOhUBp3EUwMrv8JSf1xhmD0/MH/CTDBTGqELhR2d3dIGs8+Zif8bsdtZ++hk0/5HMiItxy7lDP4rt+GZp5dJwcbjrbGxu/sYkVZd+yQl+CG+Ap4kRDeafUvXdG9umYtu2eS0fg4a+OTts+cUh3z++rfMwpfqjcQ6PS/mSl5EOx5NbNerabaPryYwqnPYYZBS/bsidjT618YIW+BLcxF3sC5CRF2DCzjail1lSnfzBUnDbIP/vYDycPjnQh3ICuci32Fxcdhz9fOTKy6/iRzPomdAfliEaj67g5JgoBv+o+/NVUxjXZ7VrzTomQymeb33pETandTyakJQQx7Lyq+YOTB3RB7f9Gt1rZT7BceEIvnCuZW4ga9+riyFet6Qc3trAhiaPkvON7YFC3Nr4K3hmDU6E5CISbhfSHVtMvEVy7aSENPzOhXPz0TSZyRS01zBM6bVAiV+7dFw+X7s+GQOCr3bp5bjTl5MhXNO9oxt5x+fYZAzy/C0FJJsWK3PSDgZOFVbMm4SMKBSv0JbiNMcOUogVHaXzC0mnprKTtpZn5CkifpA8zqeh6pPhN+Ge6bkD1vob0ME/EEha/ka+JyyYAnDnUO49SCEI/RokY/fM1ck0QASf26Zj28So2CseoVkC4AqDY4+m7FELnyiZLf3oOduw/jG5tW2ifk2aPDvGM3Obhd2pFjNDQlHkbIs/f+XuZYnfgP5TXndoPD76zLnWs5JMlhq/I1r2YxCdyo2z+4Y3gjGTMUQJVYTKKAavpSyhVm36pqvqxGBkJfPccz+8Q8udwYyKyoyyKJKCO+qmDK1RbciGN8+K946ME8UJ/xNEd8Mr3T0v+PqZ7+kgk7Xlk6V5+ZOCOGyNCk2ZHb3SG/sW4CpfHavoSUqGJS0NKFncTzQ6ZTuQCSGrxfiFxMzXvDOjaGv9z1jG4+KRUUrrGJm/e41zgdyW+CjEij+b+/UnHpB0vurlmS3ialBoj8jXjukmI3rr5dLy0eAs6Ox5ZxS70tVQRIppMRCuJaDURTZPsH09E84mokYgukexvR0SbiOiPUVQ626TMO3muSMSUykcsCsRuG6YfJ807PucGxd0PgojwnTMHeXLbuvMWuUy88cWaxEenZ3vvaIr/8FTEvB8B2UdJ3Mb/+sfX0tdShMXkXVbE/EMku5FT+3ZujW+dnspCV+xCP1DTJ6IKAPcBOAtAHYC5RPSikOB8A4BrANzkU8zPALyVWVVzh9s2S8VPvxh9ibON2HHDfeCZtCwX3VDNJnRqXYX1d06JvFwVV59cjS+Pq04X2tx9ExF6OB8Fv9XOfh/B337pBEw4xn/dgykmcwUVMX/zThsf76gC8DTNCJ3qjwawmjG2ljF2BMCTAC7gD2CMrWeMLQKQNlAiopMAdAfwagT1zQnucL1dAaUCtERLy6oK3HPpCIzu1wlAOK08aCJ3066DAJDm280jS/1XaBCRVorIti2aYf2dU3CNzyhE/Di6wjbqcCcmOk7CvGP2xS92JUrHpt8LAJ+Asg7AGJ3CiSgG4DcAvgzgTOPa5Ym+nVvjx58fhvNysPAlF4zs0wGXntQbNwiJ0sudS07qjfNHHIUd+w6H0spdUREkA1TJQiYN6+67r5jQkduicHdH0lFPT5gs+lJp+r7nlIHQl92h7lP6FoDpjLGNqq8jEU0FMBUA+vQJHx0zSnKdrDibVFbE8OtLR+S7GgVJVWXMYzM34dtnDMDcdTtxUl91CAfXy6eU0dHWRWGZGilFK0RNRg4xIuO0h3mO4J0xOtWvA8A7vvYGsFmz/HEAbiCi9QDuAfAVIrpTPIgx9gBjrIYxVtO1a3S2PYslm5zUtxMW//QcdAhIxXiiJBBcqeCuStZBlMXxLGn6JlTEzM07JT+RC2AugEFE1A/AJgCXAbhCp3DG2JXu30R0DYAaxlia94/FUoq88O1T8IuXlkmTds+8cTw27VLnYC0GTOzb6eYdt4xo6vLQ1TVYsmmP0Tmiead9y2bYHZCwveSFPmOskYhuADADQAWAvzHGlhLRHQBqGWMvEtEoAP8C0BHA54nop4yxY7Nac4ulwDnh6A545vqTpfsGdmuLgd1yF0ohWyTluIayLArL1CnRCNEzh3ZPC/UQBAE41OAuugKaaeRw5Ce1x0fodZQrtBZnMcamA5gubLud+3suEmYfVRl/B/B34xpaLJaCxSRkiX+Qu/y5Rtc5HlYAcNGJvfH2qvrAc/jbaCrCAF1FPiVhsVjyiav06pjFSYzCUADhTl5bui35t7jAzA9+xLK5CE10VuhbLJYM0Nf0RfNOIax8P9KU0tRjRFr2+v2HG5N/r9uxPyv1yiZW6FssltCYaOviAji/OP35Yt2O/TixT4fA4zZzJqFixAp9i8USmpQcDxbc4gI4MjAN5YI563bic8PTPa1EeC+kQUUYZtkKfYvFEhpCeBNNIeai1rHp8xPShVNzfWxoZYvFEhp3daqu3L7vipHo0iaxmO2mswdj+95DmDikW8BZuWPCMcF14TX9Qvpg6WKFvsViCQ0ZTOQCwJThqXhWfTq3wpNTx2WlXroM69kOy7akFnTxCWv84GMpFZ/It+Ydi8WSASm7fDGKP+DbZ6QHITyuVzv8RhGralR1p+TfxXjbVtO3WCyh+X+Th2D3wQZMMlwJWyjIbPj//c5p6Rs5jj2qXfJva96xWCxlxdGdWuGf12pFWi9IeLf87u2a568iOcSadywWS9nCB4xr20IvaVKrqopkOO3i0/Ot0LdYLGUMb905daBemGgiwt2XDAdQnHMZVuhbLJayhdf0TUImp9YYRF6lrGOFvsViscA0t27ifyv0LRaLpYgIa54p5kQqVuhbLJayJazLZSrQXPGp+lboWyyWsiWszE5FCC0+rNC3WCxlS4/2LZJ/myRoL+aVyFpCn4gmE9FKIlpNRGmJzYloPBHNJ6JGIrqE296XiOYR0QIiWkpE10dZeYvFYsmEE/t0xKPXjsG4/p3x9dP6a59HEeX1zQeBK3KJqALAfQDOAlAHYC4RvcgYW8YdtgHANQBuEk7fAuBkxthhImoDYIlz7uZIam+xWCwZcuqgLjh1kJ6PvkgRKvpaYRhGA1jNGFsLAET0JIALACSFPmNsvbPPkyWYMXaE+9kc1pxksVhKgKTLZn6rEQodIdwLwEbud52zTQsiOpqIFjll3GW1fIvFUvSUuPeOzHilfaeMsY2MseEABgK4mojSwvER0VQiqiWi2vr6et2iLRaLJS+4Nv0ilPlaQr8OwNHc794AjLV1R8NfCiAtbilj7AHGWA1jrKZr166mRVssFktOqXDsO2Le32JAR+jPBTCIiPoRURWAywC8qFM4EfUmopbO3x0BnAJgZdjKWiwWSyHQqXUVbj5nMB67rvjCSgcKfcZYI4AbAMwAsBzA04yxpUR0BxGdDwBENIqI6gBcCuB+IlrqnD4UwBwiWgjgLQD3MMYWZ+NGLBaLJZd8+4yBqO7SOt/VMIYKbSKipqaG1dbW5rsaFovFUlQQ0TzGWE3QcdaF0mKxWMoIK/QtFouljLBC32KxWMoIK/QtFouljLBC32KxWMoIK/QtFouljLBC32KxWMqIgvPTJ6J6AJ+EPL0LgB0RVqeQKZd7LZf7BMrnXsvlPoHc3mtfxlhgHJuCE/qZQES1OosTSoFyuddyuU+gfO61XO4TKMx7teYdi8ViKSOs0LdYLJYyotSE/gP5rkAOKZd7LZf7BMrnXsvlPoECvNeSsulbLBaLRU2pafoWi8ViUVAyQp+IJhPRSiJaTUTT8l0fU5xcwm8S0XIiWkpE33O2dyKi14holfN/R2c7EdHvnftdREQjubKudo5fRURX5+ueVBBRBRF9RET/dX73I6I5Tp2fchL2gIiaO79XO/uruTJucbavJKJz8nMnaoioAxE9S0QrnHc7rhTfKRH9wGm3S4joCSJqUSrvlIj+RkTbiWgJty2yd0hEJxHRYuec3xORLEVtdDDGiv4fgAoAawD0B1AFYCGAYfmul+E99AQw0vm7LYCPAQwDcDeAac72aUgklweA8wC8jEQO47EA5jjbOwFY6/zf0fm7Y77vT3K/NwJ4HMB/nd9PA7jM+fsvAL7p/P0tAH9x/r4MwFPO38Oc99wcQD/n/Vfk+74k9/kPANc5f1cB6FBq7xRALwDrALTk3uU1pfJOAYwHMBLAEm5bZO8QwIcAxjnnvAzg3KzeT74faEQvZRyAGdzvWwDcku96ZXhP/wZwFhLpJXs623oCWOn8fT+Ay7njVzr7LwdwP7fdc1wh/EMiz/LrACYC+K/T2HcAqBTfJxIZ28Y5f1c6x5H4jvnjCuUfgHaOMCRhe0m9U0fob3QEWqXzTs8ppXcKoFoQ+pG8Q2ffCm6757hs/CsV847b6FzqnG1FiTPcPRHAHADdGWNbAMD5v5tzmN89F8Oz+C2AHwKIO787A9jFEqk5AW+dk/fj7N/tHF8M99kfQD2Ahx1T1oNE1Bol9k4ZY5sA3ANgA4AtSLyjeSjNd+oS1Tvs5fwtbs8apSL0ZTawonRLIqI2AJ4D8H3G2B7VoZJtTLG9ICCizwHYzhibx2+WHMoC9hX0fTpUImEW+DNj7EQA+5EwBfhRlPfq2LMvQMIkcxSA1gDOlRxaCu80CNN7y/k9l4rQrwNwNPe7N4DNeapLaIioGRIC/zHG2PPO5m1E1NPZ3xPAdme73z0X+rM4BcD5RLQewJNImHh+C6ADEVU6x/B1Tt6Ps789gJ0o/PsEEnWsY4zNcX4/i8RHoNTe6SQA6xhj9YyxBgDPAzgZpflOXaJ6h3XO3+L2rFEqQn8ugEGOt0AVEpNDL+a5TkY4M/YPAVjOGLuX2/UiAHem/2okbP3u9q843gJjAex2hpkzAJxNRB0dDexsZ1tBwBi7hTHWmzFWjcR7eoMxdiWANwFc4hwm3qd7/5c4xzNn+2WOJ0g/AIOQmBArGBhjWwFsJKLBzqYzASxDib1TJMw6Y4moldOO3fssuXfKEck7dPbtJaKxzrP7CldWdsj3BEmEEy3nIeHxsgbAbfmuT4j6n4rEsG4RgAXOv/OQsHW+DmCV838n53gCcJ9zv4sB1HBlfQ3AauffV/N9b4p7Ph0p753+SHTw1QCeAdDc2d7C+b3a2d+fO/825/5XIsseDxnc4wkAap33+gISnhsl904B/BTACgBLAPwTCQ+ckninAJ5AYq6iAQnN/Noo3yGAGue5rQHwRwgT/1H/sytyLRaLpYwoFfOOxWKxWDSwQt9isVjKCCv0LRaLpYywQt9isVjKCCv0LRaLpYywQt9isVjKCCv0LRaLpYywQt9isVjKiP8Paog5CjNthV8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_loss()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}