{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ULMFiT Language Model in Malay Language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Project\n",
    "\n",
    "### Background\n",
    "\n",
    "_This work is part of my project while studying [fast.ai's 2018 edition of Cutting Edge Deep Learning for Coders, Part 2](http://course.fast.ai/part2.html) course._\n",
    "\n",
    "I took this opportunity to implement [Universal Language Model Fine-tuning for Text Classification (ULMFiT) paper](http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html) in different languages together with the fast.ai community. fast.ai will soon launch a [model zoo with pre-trained language models for many languages](http://forums.fast.ai/t/language-model-zoo-gorilla/14623). You can learn more about ULMFiT in lesson 4 and lesson 10. What I learned from lesson 10 ([my notes](https://cedrickchee.gitbook.io/knowledge/courses/fast.ai/deep-learning-part-2-cutting-edge-deep-learning-for-coders/2018-edition/lesson-10-transfer-learning-nlp)) is:\n",
    "- how pre-training a full language model from scratch can greatly surpass previous approaches based on simple word vectors\n",
    "- transfer learning for NLP by using this language model to show a new state of the art result in text classification, in some sense like [NLP's ImageNet moment has arrived](http://ruder.io/nlp-imagenet/)\n",
    "\n",
    "---\n",
    "\n",
    "### Project Goal\n",
    "\n",
    "The goal of this project is to train Malay word embeddings using the fast.ai version of [AWD-LSTM Language Model](https://arxiv.org/abs/1708.02182) by Salesforce Research—basically LSTM with dropouts—with data from [Wikipedia](https://dumps.wikimedia.org/mswiki/20180901/mswiki-20180901-pages-articles.xml.bz2) (last updated Sept 2, 2018). The AWD-LSTM language model achieved the state of the art performance on the English language.\n",
    "\n",
    "Using 90/10 train-validation split, I achieved perplexity of **29.30245 with 60,002 embeddings at 400 dimensions**, compared to state-of-the-art as of June 12, 2018 at **40.68 for English WikiText-2 by [Yang et al (2017)](https://arxiv.org/abs/1711.03953)** and **29.2 for English WikiText-103 by [Rae et al (2018)](https://arxiv.org/abs/1803.10049)**. To the best of my knowledge, there is no comparable research in Malay language at the point of writing (Sept 21, 2018).\n",
    "\n",
    "My workflow is as follows:\n",
    "- Perform 90/10 train-validation split\n",
    "- Minimal text cleaning and tokenization using our own tokenizer\n",
    "- Train language model\n",
    "- Evaluate model based on perplexity and eyeballing\n",
    "- Get embeddings of dataset from train set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Pre-trained model**\n",
    "\n",
    "You can download the files from [Google Drive](https://drive.google.com/drive/folders/1p5fsrD97iRD-Vz6C_ae5fo4c5wY0KrJd?usp=sharing):\n",
    "- Weights for the pre-trained model (lm_malay_final.h5.tar.gz)\n",
    "    - Uncompress and put the weights (.h5 file) into `{project_root}/data/models/`.\n",
    "- Index-to-word mapping (itos.pkl.tar.gz)\n",
    "    - Uncompress and put the pickled objects (.pkl files) into `{project_root}/data/model/malay/tmp/`.\n",
    "- Pre-processed training dataset of Malay Wikipedia:\n",
    "    - tokenized training text data (tok_trn.npy.tar.gz)\n",
    "    - tokenized validation text data (tok_val.npy.tar.gz)\n",
    "    - indexed representation of train set (trn_ids.npy.tar.gz)\n",
    "    - indexed representation of validation set (val_ids.npy.tar.gz)\n",
    "    - Uncompress and put the numpy array binary (.npy files) into `{project_root}/data/model/malay/tmp/`.\n",
    "\n",
    "Note:\n",
    "\n",
    "The weights (model state dict) and the optimizer state for the model were saved at the end of the training.\n",
    "\n",
    "_Note: the model was last trained on 2018-09-22 and the weights last updated on 2018-09-22._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import pathlib\n",
    "import html\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Standardize data format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "BOS = \"xbos\"  # beginning-of-sentence tag\n",
    "FLD = \"xfld\"  # data field tag\n",
    "\n",
    "DATA_PATH = \"data\"\n",
    "EXTR_PATH = pathlib.Path(f\"{DATA_PATH}/wiki_extr/ms\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "LM_PATH = Path(f\"{DATA_PATH}/model/malay/\")\n",
    "LM_PATH.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "232\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['data/wiki_extr/ms/AB/wiki_65',\n",
       " 'data/wiki_extr/ms/AB/wiki_11',\n",
       " 'data/wiki_extr/ms/AB/wiki_62',\n",
       " 'data/wiki_extr/ms/AB/wiki_84',\n",
       " 'data/wiki_extr/ms/AB/wiki_23']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LANG_FILENAMES = [str(f) for f in EXTR_PATH.rglob(\"*/*\")]\n",
    "print(len(LANG_FILENAMES))\n",
    "LANG_FILENAMES[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT = []\n",
    "for i in LANG_FILENAMES:\n",
    "    for line in open(i):\n",
    "        LANG_TEXT.append(json.loads(line))\n",
    "\n",
    "LANG_TEXT = pd.DataFrame(LANG_TEXT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>666100</td>\n",
       "      <td>Senarai penyakit anjing\\n\\nSenarai penyakit an...</td>\n",
       "      <td>Senarai penyakit anjing</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>666132</td>\n",
       "      <td>Son\\n\\nSon, adalah satu siri televisyen Turki ...</td>\n",
       "      <td>Son</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>666144</td>\n",
       "      <td>Kanashiki Amefuri / Adam to Eve no Dilemma\\n\\n...</td>\n",
       "      <td>Kanashiki Amefuri / Adam to Eve no Dilemma</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>666153</td>\n",
       "      <td>Bulakan, Cibeber\\n\\nBulakan merupakan sebuah K...</td>\n",
       "      <td>Bulakan, Cibeber</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>666155</td>\n",
       "      <td>Cibeber, Cibeber, Cilegon\\n\\nCibeber merupakan...</td>\n",
       "      <td>Cibeber, Cibeber, Cilegon</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666155</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id                                               text  \\\n",
       "0  666100  Senarai penyakit anjing\\n\\nSenarai penyakit an...   \n",
       "1  666132  Son\\n\\nSon, adalah satu siri televisyen Turki ...   \n",
       "2  666144  Kanashiki Amefuri / Adam to Eve no Dilemma\\n\\n...   \n",
       "3  666153  Bulakan, Cibeber\\n\\nBulakan merupakan sebuah K...   \n",
       "4  666155  Cibeber, Cibeber, Cilegon\\n\\nCibeber merupakan...   \n",
       "\n",
       "                                        title  \\\n",
       "0                     Senarai penyakit anjing   \n",
       "1                                         Son   \n",
       "2  Kanashiki Amefuri / Adam to Eve no Dilemma   \n",
       "3                            Bulakan, Cibeber   \n",
       "4                   Cibeber, Cibeber, Cilegon   \n",
       "\n",
       "                                          url  \n",
       "0  https://ms.wikipedia.org/wiki?curid=666100  \n",
       "1  https://ms.wikipedia.org/wiki?curid=666132  \n",
       "2  https://ms.wikipedia.org/wiki?curid=666144  \n",
       "3  https://ms.wikipedia.org/wiki?curid=666153  \n",
       "4  https://ms.wikipedia.org/wiki?curid=666155  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LANG_TEXT.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting rid of the title name in the text field\n",
    "def split_title_from_text(text):\n",
    "    words = text.split(\"\\n\\n\")\n",
    "    if len(words) >= 2:\n",
    "        return ''.join(words[1:])\n",
    "    else:\n",
    "        return ''.join(words)\n",
    "    \n",
    "LANG_TEXT[\"text\"] = LANG_TEXT[\"text\"].apply(lambda x: split_title_from_text(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>666100</td>\n",
       "      <td>Senarai penyakit anjing ialah pilihan penyakit...</td>\n",
       "      <td>Senarai penyakit anjing</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>666132</td>\n",
       "      <td>Son, adalah satu siri televisyen Turki yang di...</td>\n",
       "      <td>Son</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>666144</td>\n",
       "      <td>Kanashiki Amefuri / Adam to Eve no Dilemma (悲し...</td>\n",
       "      <td>Kanashiki Amefuri / Adam to Eve no Dilemma</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>666153</td>\n",
       "      <td>Bulakan merupakan sebuah Kelurahan yang terlet...</td>\n",
       "      <td>Bulakan, Cibeber</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>666155</td>\n",
       "      <td>Cibeber merupakan sebuah desa yang terletak da...</td>\n",
       "      <td>Cibeber, Cibeber, Cilegon</td>\n",
       "      <td>https://ms.wikipedia.org/wiki?curid=666155</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id                                               text  \\\n",
       "0  666100  Senarai penyakit anjing ialah pilihan penyakit...   \n",
       "1  666132  Son, adalah satu siri televisyen Turki yang di...   \n",
       "2  666144  Kanashiki Amefuri / Adam to Eve no Dilemma (悲し...   \n",
       "3  666153  Bulakan merupakan sebuah Kelurahan yang terlet...   \n",
       "4  666155  Cibeber merupakan sebuah desa yang terletak da...   \n",
       "\n",
       "                                        title  \\\n",
       "0                     Senarai penyakit anjing   \n",
       "1                                         Son   \n",
       "2  Kanashiki Amefuri / Adam to Eve no Dilemma   \n",
       "3                            Bulakan, Cibeber   \n",
       "4                   Cibeber, Cibeber, Cilegon   \n",
       "\n",
       "                                          url  \n",
       "0  https://ms.wikipedia.org/wiki?curid=666100  \n",
       "1  https://ms.wikipedia.org/wiki?curid=666132  \n",
       "2  https://ms.wikipedia.org/wiki?curid=666144  \n",
       "3  https://ms.wikipedia.org/wiki?curid=666153  \n",
       "4  https://ms.wikipedia.org/wiki?curid=666155  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LANG_TEXT.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT.to_csv(f\"{LM_PATH}/wiki_malay_corpus.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT = pd.read_csv(f\"{LM_PATH}/wiki_malay_corpus.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT = LANG_TEXT.assign(length = 0)\n",
    "LANG_TEXT.columns = [\"id\", \"text\", \"title\", \"url\", \"length\"]\n",
    "LANG_TEXT = LANG_TEXT.assign(labels = 0).pipe(lambda x: x[[\"labels\", \"text\", \"length\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Senarai penyakit anjing ialah pilihan penyakit...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Son, adalah satu siri televisyen Turki yang di...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>Kanashiki Amefuri / Adam to Eve no Dilemma (悲し...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>Bulakan merupakan sebuah Kelurahan yang terlet...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Cibeber merupakan sebuah desa yang terletak da...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   labels                                               text  length\n",
       "0       0  Senarai penyakit anjing ialah pilihan penyakit...       0\n",
       "1       0  Son, adalah satu siri televisyen Turki yang di...       0\n",
       "2       0  Kanashiki Amefuri / Adam to Eve no Dilemma (悲し...       0\n",
       "3       0  Bulakan merupakan sebuah Kelurahan yang terlet...       0\n",
       "4       0  Cibeber merupakan sebuah desa yang terletak da...       0"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LANG_TEXT.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT[\"length\"] = LANG_TEXT[\"text\"].str.len()\n",
    "LANG_TEXT = LANG_TEXT.sort_values(by=[\"length\"], ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>298556</th>\n",
       "      <td>0</td>\n",
       "      <td>Dalam agama Islam, Asmaul Husna () merupakan s...</td>\n",
       "      <td>200069.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>287674</th>\n",
       "      <td>0</td>\n",
       "      <td>Tasbih Cinta merupakan sebuah Drama lipur lara...</td>\n",
       "      <td>154934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>289705</th>\n",
       "      <td>0</td>\n",
       "      <td>Ketuanan Melayu merupakan kontrak sosial yang ...</td>\n",
       "      <td>101706.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315332</th>\n",
       "      <td>0</td>\n",
       "      <td>Dato' Sri Siti Nurhaliza binti Tarudin (Jawi: ...</td>\n",
       "      <td>98675.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95126</th>\n",
       "      <td>0</td>\n",
       "      <td>Kebebasan beragama di Malaysia adalah tertaklu...</td>\n",
       "      <td>94949.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        labels                                               text    length\n",
       "298556       0  Dalam agama Islam, Asmaul Husna () merupakan s...  200069.0\n",
       "287674       0  Tasbih Cinta merupakan sebuah Drama lipur lara...  154934.0\n",
       "289705       0  Ketuanan Melayu merupakan kontrak sosial yang ...  101706.0\n",
       "315332       0  Dato' Sri Siti Nurhaliza binti Tarudin (Jawi: ...   98675.0\n",
       "95126        0  Kebebasan beragama di Malaysia adalah tertaklu...   94949.0"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LANG_TEXT.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT.to_csv(f\"{LM_PATH}/wiki_malay_corpus_sorted_by_len.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT = pd.read_csv(f\"{LM_PATH}/wiki_malay_corpus_sorted_by_len.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "319374"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(LANG_TEXT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Dalam agama Islam, Asmaul Husna () merupakan s...</td>\n",
       "      <td>200069.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Tasbih Cinta merupakan sebuah Drama lipur lara...</td>\n",
       "      <td>154934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>Ketuanan Melayu merupakan kontrak sosial yang ...</td>\n",
       "      <td>101706.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>Dato' Sri Siti Nurhaliza binti Tarudin (Jawi: ...</td>\n",
       "      <td>98675.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Kebebasan beragama di Malaysia adalah tertaklu...</td>\n",
       "      <td>94949.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   labels                                               text    length\n",
       "0       0  Dalam agama Islam, Asmaul Husna () merupakan s...  200069.0\n",
       "1       0  Tasbih Cinta merupakan sebuah Drama lipur lara...  154934.0\n",
       "2       0  Ketuanan Melayu merupakan kontrak sosial yang ...  101706.0\n",
       "3       0  Dato' Sri Siti Nurhaliza binti Tarudin (Jawi: ...   98675.0\n",
       "4       0  Kebebasan beragama di Malaysia adalah tertaklu...   94949.0"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LANG_TEXT.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT = LANG_TEXT[LANG_TEXT['length'] > 10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "LANG_TEXT = LANG_TEXT.iloc[0:1000000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "316081"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(LANG_TEXT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some statistics of Malay Wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    Dalam agama Islam, Asmaul Husna () merupakan s...\n",
      "1    Tasbih Cinta merupakan sebuah Drama lipur lara...\n",
      "2    Ketuanan Melayu merupakan kontrak sosial yang ...\n",
      "3    Dato' Sri Siti Nurhaliza binti Tarudin (Jawi: ...\n",
      "4    Kebebasan beragama di Malaysia adalah tertaklu...\n",
      "5    England sudah dihuni manusia sejak lebih darip...\n",
      "6    Perang Bosnia merupakan sebuah konflik bersenj...\n",
      "7    Masakan Zaman Pertengahan termasuk makanan, ta...\n",
      "8    Perang Vietnam, ada kalinya disebut juga \"Pera...\n",
      "9    Operasi Market-Garden adalah satu operasi sera...\n",
      "Name: text, dtype: object\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(316081, 3)"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(LANG_TEXT[\"text\"][:10])\n",
    "LANG_TEXT.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of words in all the documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26509826"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LANG_TEXT[\"text\"].apply(lambda x: len(x.split(\" \"))).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of unique tokens across documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1742401"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(\"\".join(LANG_TEXT[\"text\"].values).split(\" \")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We perform the following minimal text processing:\n",
    "\n",
    "- Remove html tags\n",
    "- The token `xbos` is used to note start of a text since we will be chaining them together for the language model training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean plain text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "re1 = re.compile(r'  +')\n",
    "\n",
    "def fixup(x):\n",
    "    x = x.replace('#39;', \"'\").replace('amp;', '&').replace('#146;', \"'\").replace(\n",
    "        'nbsp;', ' ').replace('#36;', '$').replace('\\\\n', \"\\n\").replace('quot;', \"'\").replace(\n",
    "        '<br />', \"\\n\").replace('\\\\\"', '\"').replace('<unk>','u_n').replace(' @.@ ','.').replace(\n",
    "        ' @-@ ','-').replace('\\\\', ' \\\\ ')\n",
    "    return re1.sub(' ', html.unescape(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_texts(df, n_lbls=1):\n",
    "    labels = df.iloc[:, range(n_lbls)].values.astype(np.int64)\n",
    "    texts = f\"\\n{BOS} {FLD} 1 \" + df[n_lbls].astype(str)\n",
    "    for i in range(n_lbls + 1, len(df.columns)):\n",
    "        texts += f\" {FLD} {i-n_lbls} \" + df[i].astype(str)\n",
    "    texts = texts.apply(fixup).values.astype(str)\n",
    "\n",
    "    tok = Tokenizer().proc_all_mp(partition_by_cores(texts)) # splits the list into sublists for processing by each core\n",
    "    # Lower and upper case is inside the tokenizer\n",
    "    return tok, list(labels)\n",
    "\n",
    "def get_all(df, n_lbls):\n",
    "    tok, labels = [], []\n",
    "    for i, r in enumerate(df):\n",
    "        print(i)\n",
    "        #pdb.set_trace()\n",
    "        tok_, labels_ = get_texts(r, n_lbls)\n",
    "        tok += tok_;\n",
    "        labels += labels_\n",
    "    return tok, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the data into train and validation sets\n",
    "# Splitting 10% for validation.\n",
    "trn_texts, val_texts = sklearn.model_selection.train_test_split(LANG_TEXT, test_size=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(284472, 31609)"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(trn_texts), len(val_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "trn_idx = np.random.permutation(len(trn_texts)) # generate a random ordering\n",
    "val_idx = np.random.permutation(len(val_texts))\n",
    "\n",
    "df_trn = trn_texts.iloc[trn_idx, :] # sort things randomly\n",
    "df_val = val_texts.iloc[val_idx, :] # sort things randomly\n",
    "\n",
    "df_trn.columns = [\"labels\", \"text\", \"length\"]\n",
    "df_val.columns = [\"labels\", \"text\", \"length\"]\n",
    "\n",
    "df_trn.to_csv(LM_PATH / \"train.csv\", header=False, index=False)\n",
    "df_val.to_csv(LM_PATH / \"test.csv\", header=False, index=False) # saving the data in our new format to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "chunksize = 10000\n",
    "df_trn = pd.read_csv(LM_PATH / \"train.csv\", header=None, chunksize=chunksize)\n",
    "df_val = pd.read_csv(LM_PATH / \"test.csv\", header=None, chunksize=chunksize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We write a simple custom function to perform tokenization. Out of 24,911,449 tokens from all of training set, we chose 60,000 for the vocabulary size (plus one for unknown and another for padding) of tokens which are not rare words (appeared more than twice and not typos) in the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n",
      "11\n",
      "12\n",
      "13\n",
      "14\n",
      "15\n",
      "16\n",
      "17\n",
      "18\n",
      "19\n",
      "20\n",
      "21\n",
      "22\n",
      "0\n",
      "1\n",
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "# Finally, tokenize text data\n",
    "tok_trn, trn_labels = get_all(df_trn, 1)\n",
    "tok_val, val_labels = get_all(df_val, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tmp directory to store the upcoming numpy arrays\n",
    "(LM_PATH / \"tmp\").mkdir(exist_ok=True)\n",
    "\n",
    "# Save the train and validation tokens in the tmp directories\n",
    "np.save(LM_PATH / \"tmp\" / \"tok_trn.npy\", tok_trn)\n",
    "np.save(LM_PATH / \"tmp\" / \"tok_val.npy\", tok_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "tok_trn = np.load(LM_PATH / \"tmp\" / \"tok_trn.npy\")\n",
    "tok_val = np.load(LM_PATH / \"tmp\" / \"tok_val.npy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([list(['\\n', 'xbos', 'xfld', '1', 'gräfenhainichen', 'merupakan', 'sebuah', 'bandar', 'terletak', 'di', 'daerah', 'wittenberg', ',', 'saxony', '-', 'anhalt', ',', 'jerman', '.', '\\n ', 'xfld', '1', '94.0']),\n",
       "       list(['\\n', 'xbos', 'xfld', '1', '19', 'merupakan', 'tahun', 'biasa', 'yang', 'bermula', 'pada', 'hari', 'ahad', 'dalam', 'kalendar', 'gregory', '.', '\\n', '<', 'br', 'clear', '=', 'all', '>', '\\n ', 'xfld', '1', '92.0'])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sanity check: get the first 2 train tokens\n",
    "tok_trn[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>,</th>\n",
       "      <td>1102901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>.</th>\n",
       "      <td>999701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yang</th>\n",
       "      <td>548167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di</th>\n",
       "      <td>532424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dan</th>\n",
       "      <td>509869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>465667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>xfld</th>\n",
       "      <td>448944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>\\n</th>\n",
       "      <td>403986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>\"</th>\n",
       "      <td>378261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-</th>\n",
       "      <td>355521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pada</th>\n",
       "      <td>232686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>xbos</th>\n",
       "      <td>224472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dalam</th>\n",
       "      <td>210035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(</th>\n",
       "      <td>202054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>)</th>\n",
       "      <td>195057</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merupakan</th>\n",
       "      <td>189399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t_up</th>\n",
       "      <td>188865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sebuah</th>\n",
       "      <td>184610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dengan</th>\n",
       "      <td>174237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>untuk</th>\n",
       "      <td>151121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ini</th>\n",
       "      <td>145098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>terletak</th>\n",
       "      <td>138483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>\\n</th>\n",
       "      <td>134853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tahun</th>\n",
       "      <td>115035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dari</th>\n",
       "      <td>110815</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 0\n",
       ",          1102901\n",
       ".           999701\n",
       "yang        548167\n",
       "di          532424\n",
       "dan         509869\n",
       "1           465667\n",
       "xfld        448944\n",
       "\\n          403986\n",
       "\"           378261\n",
       "-           355521\n",
       "pada        232686\n",
       "xbos        224472\n",
       "dalam       210035\n",
       "(           202054\n",
       ")           195057\n",
       "merupakan   189399\n",
       "t_up        188865\n",
       "sebuah      184610\n",
       "dengan      174237\n",
       "untuk       151121\n",
       "ini         145098\n",
       "terletak    138483\n",
       "\\n          134853\n",
       "tahun       115035\n",
       "dari        110815"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the Counter object from all the splitted files.\n",
    "# Identify the most common tokens\n",
    "freq = Counter(p for o in tok_trn for p in o) \n",
    "freqs = pd.DataFrame.from_dict(freq, orient=\"index\")\n",
    "freqs.sort_values(0, ascending=False).head(25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24911449"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sanity check\n",
    "len([p for o in tok_trn for p in o])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnt = []\n",
    "for i in range(49):\n",
    "    row_cnt = freqs[freqs[0]>=i+1].shape[0]\n",
    "    cnt.append(row_cnt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.lines.Line2D at 0x7f24d52484e0>"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD8CAYAAACLrvgBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X2QHNV57/HvMzM7szsjidXLIrBekAAJjEkQsBHCOAaDLSRsLO6NA7jiIHMpyxVIYipQCVBJERvj2HXjgKkYxVQgCMcxqLAJgggrQkZ2fG0wwsgIkIFFBiRZb+htJa32/bl/9JnVaDU7MytpXqT5faq6uvvp031OJ2sene7TZ8zdERERKUWs2g0QEZFjh5KGiIiUTElDRERKpqQhIiIlU9IQEZGSKWmIiEjJiiYNMzvDzFbnLO1mdrOZjTGz5Wb2VliPDuXNzO4zszYze8XMzsu51vxQ/i0zm58TP9/M1oRz7jMzC/G8dYiISHUUTRru/oa7z3D3GcD5QAfwBHAbsMLdpwErwj7AXGBaWBYACyFKAMCdwAXATODOnCSwEPhCznlzQnyoOkREpAqG+3jqMuBtd38XmAcsCvFFwFVhex7wiEeeB5rN7GTgcmC5u+9w953AcmBOODbK3Z/36EvDRwZdK18dIiJSBYlhlr8W+H7YHu/um8L2ZmB82J4ArM85Z0OIFYpvyBMvVMeQxo0b51OmTCnlXo7cG29E6zPOqEx9IiJl8tJLL73v7i3FypWcNMwsCXwauH3wMXd3MyvrfCSF6jCzBUSPwpg8eTKrVq0qZ1MOuOSSaL1yZWXqExEpEzN7t5Ryw3k8NRf4lbtvCftbwqMlwnpriG8EJuWcNzHECsUn5okXquMg7v6Au7e6e2tLS9FEKSIih2k4SeOzHHg0BbAEyI6Amg88mRO/LoyimgXsDo+YlgGzzWx0eAE+G1gWjrWb2awwauq6QdfKV0dt+Nu/jRYRkTpR0uMpM8sAnwC+mBP+OrDYzG4A3gWuDvGlwBVAG9FIq+sB3H2Hmd0FvBjKfcXdd4TtG4GHgSbgmbAUqqM2fPzj1W6BiEhF2fE2NXpra6tX7J3G6tXResaMytQnIlImZvaSu7cWKzfc0VOS6+abo7VehItIndA0IiIiUjIlDRERKZmSRvDEyxv43gslDVMWEalbShrB07/exH+88F61myEiUtP0IjxIpxJ0dPcN76Svfa08jRERqVFKGkEmGWdfV+/wTvrwh8vTGBGRGqXHU0E6eRg9jZ//PFpEROqEehpBJhVnX3cv7k74Daji7rgjWus7DRGpE+ppBOlkAnfo7OmvdlNERGqWkkaQScUB2Nc9zPcaIiJ1REkjSCejJ3UdXcN8ryEiUkeUNIIR6mmIiBSlF+HBQE9jOEnj3nvL1BoRkdqkpBEMvNMYzuMpTYkuInVGj6eCbE9jWB/4PftstIiI1An1NIJMNmkM5wO/r341WusX/ESkTqinEaTD46lhvdMQEakzShrBQE9DQ25FRIakpBE0NsQwU09DRKQQJY3AzMgkE+ppiIgUoBfhOdLJ+PB6Gt/5TvkaIyJSg0rqaZhZs5k9bma/MbO1ZnahmY0xs+Vm9lZYjw5lzczuM7M2M3vFzM7Luc78UP4tM5ufEz/fzNaEc+6zMM3sUHWUSyaVGN7oqTPOiBYRkTpR6uOpbwE/cvczgXOAtcBtwAp3nwasCPsAc4FpYVkALIQoAQB3AhcAM4E7c5LAQuALOefNCfGh6iiLdDJOx3C+03jqqWgREakTRZOGmZ0AfBR4EMDdu919FzAPWBSKLQKuCtvzgEc88jzQbGYnA5cDy919h7vvBJYDc8KxUe7+vLs78Miga+WroywyycTw5p765jejRUSkTpTS05gKbAP+zcxeNrN/NbMMMN7dN4Uym4HxYXsCsD7n/A0hVii+IU+cAnUcxMwWmNkqM1u1bdu2Em4pv3QqPvxf7xMRqSOlJI0EcB6w0N3PBfYx6DFR6CH40W9eaXW4+wPu3ururS0tLYddRzR6SkNuRUSGUkrS2ABscPcXwv7jRElkS3i0RFhvDcc3ApNyzp8YYoXiE/PEKVBHWUSjp9TTEBEZStGk4e6bgfVmlh0mdBnwOrAEyI6Amg88GbaXANeFUVSzgN3hEdMyYLaZjQ4vwGcDy8KxdjObFUZNXTfoWvnqKItMSj0NEZFCSv1O4y+A75lZElgHXE+UcBab2Q3Au8DVoexS4AqgDegIZXH3HWZ2F/BiKPcVd98Rtm8EHgaagGfCAvD1Ieooi2xPw90Jo34L++53y9kcEZGaU1LScPfVQGueQ5flKevATUNc5yHgoTzxVcDZeeLb89VRLplUgt5+p7uvn1QiXvyESZOKlxEROY5oGpEc6WSY6bbUqUQeeyxaRETqhKYRyZFJZX9To5fRmWTxExYujNbXXFPGVomI1A71NHJkBn4nXCOoRETyUdLIkR74nXCNoBIRyUdJI4d6GiIihSlp5Mi+CFdPQ0QkP70Iz5F9EV5yT+Pxx8vYGhGR2qOkkSOT7WmUOtPtuHFlbI2ISO3R46kc6eyQ21IfTz38cLSIiNQJJY0cTQ3ZdxolPp5S0hCROqOkkSMeM5oahvk74SIidURJY5BMKj683wkXEakjShqDpJOJ4f1OuIhIHVHSGCSdVE9DRGQoGnI7SCaVKP2dxtKl5W2MiEiNUdIYJJ2Ms6ezxKSRTpe3MSIiNUaPpwbJJIfR07j//mgREakTShqDpFPx0r/TWLw4WkRE6oSSxiDD6mmIiNQZJY1B0vpOQ0RkSEoag2SSCbp7++np6692U0REak5JScPM3jGzNWa22sxWhdgYM1tuZm+F9egQNzO7z8zazOwVMzsv5zrzQ/m3zGx+Tvz8cP22cK4VqqOchj09uohIHRlOT+Nj7j7D3VvD/m3ACnefBqwI+wBzgWlhWQAshCgBAHcCFwAzgTtzksBC4As5580pUkfZZKdHL+m9xsqV0SIiUieO5PHUPGBR2F4EXJUTf8QjzwPNZnYycDmw3N13uPtOYDkwJxwb5e7Pu7sDjwy6Vr46yubA9OjqaYiIDFZq0nDgv83sJTNbEGLj3X1T2N4MjA/bE4D1OeduCLFC8Q154oXqKJth9TT+8R+jRUSkTpT6RfhH3H2jmZ0ILDez3+QedHc3Mz/6zSutjpDIFgBMnjz5iOpJJ4fR03j66Wh9661HVKeIyLGipJ6Gu28M663AE0TvJLaER0uE9dZQfCMwKef0iSFWKD4xT5wCdQxu3wPu3ururS0tLaXc0pAyqWH0NERE6kzRpGFmGTMbmd0GZgOvAkuA7Aio+cCTYXsJcF0YRTUL2B0eMS0DZpvZ6PACfDawLBxrN7NZYdTUdYOula+OshnoaWj0lIjIIUp5PDUeeCKMgk0A/+HuPzKzF4HFZnYD8C5wdSi/FLgCaAM6gOsB3H2Hmd0FvBjKfcXdd4TtG4GHgSbgmbAAfH2IOspmoKeh39QQETlE0aTh7uuAc/LEtwOX5Yk7cNMQ13oIeChPfBVwdql1lNOwehpNTWVujYhIbdHU6IOkk8PoaTzzTPEyIiLHEU0jMkhDPEYyEdM7DRGRPJQ08sgk4+wrpadx113RIiJSJ5Q08kgnE+wrZcjtihXRIiJSJ5Q08sik4nRoGhERkUMoaeRRck9DRKTOKGnkkUnFNTW6iEgeGnKbRzqZYPvejuIFx44tf2NERGqIkkYemWSJPY0f/KD8jRERqSF6PJVHOpXQhIUiInkoaeQRfadRQk/j9tujRUSkTujxVB6ZVIL9PX309TvxmA1d8Be/qFyjRERqgHoaeWTCpIX7ezSCSkQkl5JGHmlNjy4ikpeSRh4Z/RCTiEheeqeRR3Z69KKTFk6cWPi4iMhxRkkjj0wq+j9L0W81/v3fK9AaEZHaocdTeQz0NPSthojIQZQ08hjoaRT7VuPmm6NFRKRO6PFUHiX3NFavrkBrRERqh3oaeWRHT2nIrYjIwZQ08sh+p6EhtyIiBys5aZhZ3MxeNrOnw/5UM3vBzNrM7DEzS4Z4Kuy3heNTcq5xe4i/YWaX58TnhFibmd2WE89bR7kl4zESMdOkhSIigwynp/ElYG3O/jeAe9z9dGAncEOI3wDsDPF7QjnM7CzgWuBDwBzg/pCI4sC3gbnAWcBnQ9lCdZSVmZEuZdLC6dOjRUSkTpSUNMxsIvBJ4F/DvgGXAo+HIouAq8L2vLBPOH5ZKD8PeNTdu9z9t0AbMDMsbe6+zt27gUeBeUXqKLtMKdOjP/BAtIiI1IlSexr3An8N9If9scAud8/+V3UDMCFsTwDWA4Tju0P5gfigc4aKF6rjIGa2wMxWmdmqbdu2lXhLhZXU0xARqTNFk4aZfQrY6u4vVaA9h8XdH3D3VndvbWlpOSrXzKQSxYfcLlgQLSIidaKU7zQuAj5tZlcAjcAo4FtAs5klQk9gIrAxlN8ITAI2mFkCOAHYnhPPyj0nX3x7gTrKLp2MF/+47803K9MYEZEaUbSn4e63u/tEd59C9CL7x+7+J8BzwGdCsfnAk2F7SdgnHP+xu3uIXxtGV00FpgG/BF4EpoWRUslQx5JwzlB1lF0mWUJPQ0SkzhzJdxp/A/yVmbURvX94MMQfBMaG+F8BtwG4+2vAYuB14EfATe7eF3oRfw4sIxqdtTiULVRH2UW/E653GiIiuYY1jYi7rwRWhu11RCOfBpfpBP54iPPvBu7OE18KLM0Tz1tHJUS/E66ehohILs09NYRMKT2NGTMq0xgRkRqhpDGETDLOvu5e3J3ok5E87r23so0SEakyzT01hHQqgTt09vQXLywiUieUNIaQKWV69M99LlpEROqEHk8NIZ3M+SGmEUMU2rChcg0SEakB6mkMIZPST76KiAympDGEgZ6GkoaIyAAljSEM9DQ0aaGIyAC90xhCST2NCy+sUGtERGqDksYQsr8TXrCn8Q//UKHWiIjUBj2eGkL2d8L1TkNE5AAljSEM9DQKTSXyR38ULSIidUKPp4bQ2BDDDDoKTVq4fXvlGiQiUgPU0xiCmYXf1NDoKRGRLCWNAtLJuN5piIjkUNIoIJNK6DsNEZEceqdRQNGexmWXVa4xIiI1QEmjgEyySE/j7/6uco0REakBejxVQDoV14SFIiI5lDQKiHoaBZLG3LnRIiJSJ/R4qoBMKl74d8L3769cY0REakDRnoaZNZrZL83s12b2mpl9OcSnmtkLZtZmZo+ZWTLEU2G/LRyfknOt20P8DTO7PCc+J8TazOy2nHjeOiolXaynISJSZ0p5PNUFXOru5wAzgDlmNgv4BnCPu58O7ARuCOVvAHaG+D2hHGZ2FnAt8CFgDnC/mcXNLA58G5gLnAV8NpSlQB0Vke1puHslqxURqVlFk4ZH9obdhrA4cCnweIgvAq4K2/PCPuH4ZWZmIf6ou3e5+2+BNmBmWNrcfZ27dwOPAvPCOUPVURHpZILefqe7r7+S1YqI1KyS3mmE3sBLwOlEvYK3gV3unn12swGYELYnAOsB3L3XzHYDY0P8+ZzL5p6zflD8gnDOUHVURCYZZrrt6iOViB9a4FOfqmRzRESqrqSk4e59wAwzawaeAM4sa6uGycwWAAsAJk+efNSum05lZ7rtZXQmz+uUW289anWJiBwLhjXk1t13Ac8BFwLNZpZNOhOBjWF7IzAJIBw/AdieGx90zlDx7QXqGNyuB9y91d1bW1pahnNLBWUGfr1PU4mIiEBpo6daQg8DM2sCPgGsJUoenwnF5gNPhu0lYZ9w/McevUleAlwbRldNBaYBvwReBKaFkVJJopflS8I5Q9VREemB3wkfYgTVJZdEi4hInSjl8dTJwKLwXiMGLHb3p83sdeBRM/sq8DLwYCj/IPBdM2sDdhAlAdz9NTNbDLwO9AI3hcdemNmfA8uAOPCQu78WrvU3Q9RREeppiIgcrGjScPdXgHPzxNcRjXwaHO8E/niIa90N3J0nvhRYWmodlZJOFulpiIjUGU0jUkAmpZ6GiEguJY0CskNuNWmhiEhEc08VkB1y2zHU9OhXX13B1oiIVJ+SRgFNDUV6GjfeWMHWiIhUnx5PFRCPGU0NBWa67eiIFhGROqGeRhGZVHzo0VNXXBGtV66sWHtERKpJPY0i0smERk+JiARKGkWkkwV6GiIidUZJo4hMSj0NEZEsJY0i0sm4vtMQEQn0IryITDLB1vau/Ac///mKtkVEpNqUNIrIpBLsHeqdhpKGiNQZPZ4qIvqd8CGSxvvvR4uISJ1QT6OIdDLBvqFehH8m/NSHvtMQkTqhnkYRmWSc7t5+evr6q90UEZGqU9IoIq3p0UVEBihpFJHRDzGJiAxQ0ihi8pg0AGs3tVe5JSIi1acX4UWcP2U0TQ1xfvLmNi774PiDD/7Zn1WnUSIiVaKkUUQqEefDp43lJ29uO/TgNddUvkEiIlWkx1Ml+Oj0Ft7d3sE77+87+MD69dEiIlInlDRKcPH0FoBDext/+qfRIiJSJ4omDTObZGbPmdnrZvaamX0pxMeY2XIzeyusR4e4mdl9ZtZmZq+Y2Xk515ofyr9lZvNz4ueb2Zpwzn1mZoXqqLQp4zKcMjbNT/M9ohIRqSOl9DR6gVvc/SxgFnCTmZ0F3AascPdpwIqwDzAXmBaWBcBCiBIAcCdwATATuDMnCSwEvpBz3pwQH6qOirt4egs/f3s7Xb36XkNE6lfRpOHum9z9V2F7D7AWmADMAxaFYouAq8L2POARjzwPNJvZycDlwHJ33+HuO4HlwJxwbJS7P+/uDjwy6Fr56qi4i6e3sL+nj1Xv7KxWE0REqm5Y7zTMbApwLvACMN7dN4VDm4HseNQJQO7b4Q0hVii+IU+cAnUMbtcCM1tlZqu2bSvPI6RZp44lGY/lH0UlIlInSk4aZjYC+AFws7sf9KVb6CH4UW7bQQrV4e4PuHuru7e2tLSUpf5MKsEfTB3NT97ISRq33BItIiJ1oqSkYWYNRAnje+7+wxDeEh4tEdZbQ3wjMCnn9IkhVig+MU+8UB1VcfH0Ft7YsodNu/dHgSuvjBYRkTpRyugpAx4E1rr7P+UcWgJkR0DNB57MiV8XRlHNAnaHR0zLgNlmNjq8AJ8NLAvH2s1sVqjrukHXyldHVVw8/USAA6Oo3ngjWkRE6kQpX4RfBPwpsMbMVofYHcDXgcVmdgPwLnB1OLYUuAJoAzqA6wHcfYeZ3QW8GMp9xd13hO0bgYeBJuCZsFCgjqqYPn4EJ41q5CdvbuOaP5gMX/xidEC/pyEidaJo0nD3nwE2xOHL8pR34KYhrvUQ8FCe+Crg7Dzx7fnqqBYz4+LpLSx9dRO9ff2ag0VE6o6+CB+mi89oYU9nL6vX76p2U0REKk5JY5guOn0c8Zhp6K2I1CUljWE6oamBcyc1K2mISF3SY/nD8NHpLdzz7Ju03/LXjGpKVrs5IiIVo57GYbh4egvu8Nykc+DjH692c0REKkZJ4zD83oQTGJNJ8tayn8Hq1cVPEBE5TihpHIZYzPjDaeP42L98Db/55mo3R0SkYpQ0DtPF01vo6euno6u32k0REakYJY3D9IfTookRt+3trnJLREQqR0njMLWMTNEyMsWW9k7atu6tdnNERCpCSeMITB6TJmbw90teI5o9RUTk+KbvNI5Awze+ztZXfsfP2t5n2WubmXP2ydVukohIWamncSQ+/GEu/8L/5syTRnLX02vZ363fDxeR45uSxpH4+c9JvPA8X/70h9i4az8LV7ZVu0UiImWlx1NH4o47ALhg5UrmzfgA//LTdXzm/ElMHpuucsNERMpDPY2j5I4rPkhDzPjK069XuykiImWjpHGUjB/VyF9eNo1n127hud9U9afMRUTKRknjKLr+oqmc2pLhy0+9RlevXoqLyPFHSeMoSiZi/P2VH+Kd7R088JN11W6OiMhRpxfhR+Leew8JfXR6C5/8/ZP55vI3GdGY4PqLplahYSIi5aGkcSRmzMgb/uYfn0NvXz9ffup13t/bxa2zz8DMKtw4EZGjr+jjKTN7yMy2mtmrObExZrbczN4K69EhbmZ2n5m1mdkrZnZezjnzQ/m3zGx+Tvx8M1sTzrnPwn9dh6qjpjz7bLQM0tgQ5/4/OZ/PzpzEt597m9t/uIbevv4qNFBE5Ogq5Z3Gw8CcQbHbgBXuPg1YEfYB5gLTwrIAWAhRAgDuBC4AZgJ35iSBhcAXcs6bU6SO2vHVr0ZLHvGY8bX/9Xv85aWn8+iL67nxe7+is0cvx0Xk2FY0abj7T4Edg8LzgEVhexFwVU78EY88DzSb2cnA5cByd9/h7juB5cCccGyUuz/v0Yx/jwy6Vr46jhlmxl/NPoMvf/pDLF+7hese+iW79/dUu1kiIoftcEdPjXf3TWF7MzA+bE8A1ueU2xBiheIb8sQL1XHMmf/hKdx37bm8/N5OrvnOL1izYXe1myQicliOeMht6CGUdV7wYnWY2QIzW2Vmq7Zt21bOphy2K8/5AP/2+Zm8v7eLT3/7Z9yy+Ndsae+sdrNERIblcJPGlvBoibDOfgK9EZiUU25iiBWKT8wTL1THIdz9AXdvdffWlpaWw7yl8vvItHE8d+slfPGjp/HUr3/HJf93Jd969i3Njisix4zDTRpLgOwIqPnAkznx68IoqlnA7vCIaRkw28xGhxfgs4Fl4Vi7mc0Ko6auG3StfHXUju98J1qGYWRjA7fNPZMVt1zMpWeeyD3Pvsml31zJEy9voL9fP+QkIrXNiv3inJl9H7gEGAdsIRoF9Z/AYmAy8C5wtbvvCP/h/2eiEVAdwPXuvipc5/8Ad4TL3u3u/xbirUQjtJqAZ4C/cHc3s7H56ih2Q62trb5q1apS77/qfvnbHdz19Ous2bibU1syXH/RVP7ovAmkk/qERkQqx8xecvfWouWOt58prWjSeOqpaH3llUd0mf5+5+k1m3jwf9bx6w27GdWY4LMXTOa6C6cwobnpKDRURKQwJY1KuOSSaL1y5VG5nLvzq/d28dD/+y0/enUzAHPOPonPXXAKM6eOIR7TV+UiUh6lJg09A6khZsb5p4zm/FNGs3HXfh75xTt8/4X3+K9XNjFuRIrLPzSeuWefzKxTx5CIa65JEak8JY0aNaG5idvnfpAvXTaNFWu38qNXN/PDX23key+8x+h0A584azxzzj6JWaeO1fsPEakY/demxqWTCa485wNcec4H2N/dx0/e3MaPXt3E0jWbWbxqAw1xY8akZi48dSwXnjaOcyc309gQr3azReQ4paRxDGlKxplz9knMOfskunr7eGHdDn7+9nZ+sW47//xcG/f9uI1UIsb5p4ymdcoYzp3UzDmTmhmTSVa76SJynNCL8COxPsyMMmlS4XIV0N7Zw4u/DUnk7e38ZnM72c8+ThmbZsakZs6ZGCWRM08aSSalfy+IyAEaPVXn9nX1smbjblav38Xq93axev0uNodpS8xg6tgMH/zAKM46eRRnhfWJI1P63Q+ROqXRU5Xw2GPR+pprqtuOPDKpBLNOHcusU8cOxDbt3s+rG9t5/XftvL5pN69s2MV/vbJp4PjIxgSntYzgtJYRnH7iCE5ryXDaiSOYPCZNg0ZriQjqaRyZo/ydRjW0d/bwm017WLupnbe37aVt617e3raXLe1dA2XiMeMDzY1MGZvhlLFppozNMHlMmslj00xobmJkY0MV70BEjgb1NKQkoxobmDl1DDOnjjko3t7Zw7pt+2jbupd3t+/jne0dvLd9H0tW/472zt5B10gwYXSUQCY0NzJhdBMnndDEySc0ctKoRsaPaiSZUE9F5HigpCF5jWpsYMakZmZMaj7k2K6Obt7d3sF7OzrYuGs/G3fuZ+Ou/azf0cHz67azt6v3kHPGjUgyPiSQlhEpWkYevIwbkWLciCQjUgm9VxGpYUoaMmzN6STN6STn5EkoALv397ClvZNNuzvZsjtab27vZPPu/Wxp7+TVjbvZvq+bvjyz+qYSMcaNSDF2RDJaZ5KMGZFkTDrJmMzBS3M6yahGJRmRSlLSkKPuhKYGTmhqYPr4kUOW6et3dnZ0s21PF+/v7RpYb9/bzbaw3tLeyeu/a2fHvm66+/rzXiceM05oaqC5qYHmdEOU0JoaGBWWEwYto5oSjGqMjmWScSUckWFS0jgSjz9e7RYcs+IxC4+kUkXLujv7uvvYsbebHR3d7NgXJZXd+3vY1dHDrv3d7OzoYXdHD1v3dPLmlj3s3t/Dns5DH5Pliln0+yajmhKMTDUwojHBqMYEI1IJRjQmGNnYwIhUgpGNCTLJEAvHMqmoXCaVIN0QJ6bJJKVOKGkciXHjqt2CumBm0X/IUwkmj02XfF5fv7Ons4fd+w8sezp7ad/fQ3tnD+37e8O6h71dvezp7OV3uzrZ29Ub9nvo6SttdGEmGR9IJOlUnEwyJJRktJ2NNSXjNDXESSfjOdsJmpIxGhui/aZknMZEtE4lYuoNSU1R0jgSDz8crT//+Wq2QoYQj9nA+5fD1dnTx76BJBKt92bXXb10dPeytysqky23v7uPvV29bNvTxb7uXjq6+tjXHR0/nB9nbGyI0dQQp/GgJTcWozERpzEkm1RDjFQiRioRJ5nIbsfCdjgej5FqiJGMHyg/cDyU10zKko+SxpFQ0jjuZf8jPbaEx2jFuDvdff3s7+6jIyzRdi+dvVG8qzeKdfb00dHTR2dPP109fezvCfFQrrMnWto7e+jsOXBuZ08/Xb19JfeQCokZJBMxkvEYyUScZNyi/ewSP5BosrFU/NDjDQPlou2GgbgdVCaK28B2QzzaTmS3YzEaEjESseg8PRKsDiUNkQoxs/Av+TjNpT9lOyz9/VGC6gpJpKu3Pyx9dIft7jyxrp6+nPP66e47UK4nbHfnxLt7+9nV0T1QtqsnlMs53ns43asSxGM2kFiS8RiJuJGIRQkmEY+SS0OIN8RixGMWbcdjA+fGYzEaQjwRz25H5yZyjsfjRiIW7UdrC2Xy78didlA8Wh9oQ974QfVE8ZhRc48nlTREjkOxmNEYi4dp8qv7xX42gWWTTk+fH5RYegaO+cB2dNzpHdiPjvX2HbhWb851stfu7Xd6+5ze/uic3r4o1tPXT29/P/vnk7tcAAAEq0lEQVR7nL6w39fvB47lOae33/MOC6+0Q5JPPEbMDuznLg/N/4Nhvfc7rPaU9eoiUvcOTmDHFncfSC7ZdW9OwsmN9fQ5/Z6NR4moL3t+3+Dr9B+IZ8/JSVZRHUTX6Q/X6QvH+qNj/aFsts7+fifVUP73UEoaIiJDMMs+uqp2S2qHksaRWLq02i0QEamomh9TZ2ZzzOwNM2szs9uq3Z6DpNPRIiJSJ2o6aZhZHPg2MBc4C/ismZ1V3VbluP/+aBERqRM1nTSAmUCbu69z927gUWBeldt0wOLF0SIiUidqPWlMANbn7G8IsYOY2QIzW2Vmq7Zt21axxomI1JtaTxolcfcH3L3V3VtbWlqq3RwRkeNWrSeNjcCknP2JISYiIlVQ60njRWCamU01syRwLbCkym0SEalb5l79z+QLMbMrgHuBOPCQu99dpPw24N3DrG4c8P5hnnusq+d7h/q+/3q+d6jv+8+991Pcvejz/ZpPGpVkZqvcvbXa7aiGer53qO/7r+d7h/q+/8O591p/PCUiIjVESUNEREqmpHGwB6rdgCqq53uH+r7/er53qO/7H/a9652GiIiUTD0NEREpmZJGUNOz6R5lZvaQmW01s1dzYmPMbLmZvRXWo6vZxnIxs0lm9pyZvW5mr5nZl0K8Xu6/0cx+aWa/Dvf/5RCfamYvhL//x8J3UcclM4ub2ctm9nTYr6d7f8fM1pjZajNbFWLD+ttX0uAYmE336HsYmDModhuwwt2nASvC/vGoF7jF3c8CZgE3hf9f18v9dwGXuvs5wAxgjpnNAr4B3OPupwM7gRuq2MZy+xKwNme/nu4d4GPuPiNnqO2w/vaVNCK1PZvuUebuPwV2DArPAxaF7UXAVRVtVIW4+yZ3/1XY3kP0H48J1M/9u7vvDbsNYXHgUuDxED9u79/MJgKfBP417Bt1cu8FDOtvX0kjUtJsuse58e6+KWxvBsZXszGVYGZTgHOBF6ij+w+PZ1YDW4HlwNvALnfvDUWO57//e4G/BvrD/ljq594h+gfCf5vZS2a2IMSG9bevn3uVQ7i7m9lxPazOzEYAPwBudvf26B+ckeP9/t29D5hhZs3AE8CZVW5SRZjZp4Ct7v6SmV1S7fZUyUfcfaOZnQgsN7Pf5B4s5W9fPY2IZtOFLWZ2MkBYb61ye8rGzBqIEsb33P2HIVw395/l7ruA54ALgWYzy/4j8nj9+78I+LSZvUP0CPpS4FvUx70D4O4bw3or0T8YZjLMv30ljYhm043ud37Yng88WcW2lE14hv0gsNbd/ynnUL3cf0voYWBmTcAniN7rPAd8JhQ7Lu/f3W9394nuPoXof+M/dvc/oQ7uHcDMMmY2MrsNzAZeZZh/+/q4LxjubLrHMjP7PnAJ0QyXW4A7gf8EFgOTiWYJvtrdB78sP+aZ2UeA/wHWcOC59h1E7zXq4f5/n+hlZ5zoH42L3f0rZnYq0b++xwAvA59z967qtbS8wuOpW939U/Vy7+E+nwi7CeA/3P1uMxvLMP72lTRERKRkejwlIiIlU9IQEZGSKWmIiEjJlDRERKRkShoiIlIyJQ0RESmZkoaIiJRMSUNEREr2/wEpvU3CBmUuLAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cnt)\n",
    "plt.axvline(x=2, color=\"red\", linestyle=\"--\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Numericalize the text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We trained the language model based on 224,472 texts and validated on 31,609 texts from Malay Wikipedia using various tricks detailed in the [ULMFiT paper](https://arxiv.org/abs/1801.06146). The validation perplexity is 29.30245."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Truncating our vocab to ignore the rare words\n",
    "max_vocab = 60000\n",
    "min_freq = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "itos = [o for o, c in freq.most_common(max_vocab) if c > min_freq] # getting rid of the rare words\n",
    "itos.insert(0, \"_pad_\")\n",
    "itos.insert(0, \"_unk_\") # itos is the list of all the strings in the vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60002"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a index-key dictionary for our vocabulary\n",
    "stoi = collections.defaultdict(lambda: 0, {v:k for k, v in enumerate(itos)})\n",
    "len(itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a index representation for our train and validation dataset\n",
    "trn_lm = np.array([[stoi[o] for o in p] for p in tok_trn])\n",
    "val_lm = np.array([[stoi[o] for o in p] for p in tok_val])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving our indexed representation of our dataset to disk.\n",
    "# We also save the index-word mapping to retrieve the complete text representation from these numpy arrays\n",
    "np.save(LM_PATH / \"tmp\" / \"trn_ids.npy\", trn_lm)\n",
    "np.save(LM_PATH / \"tmp\" / \"val_ids.npy\", val_lm)\n",
    "pickle.dump(itos, open(LM_PATH / \"tmp\" / \"itos.pkl\", \"wb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the indexed representation of our dataset from disk.\n",
    "# We also load the index-word mapping to to help us convert the indexes to word datasets, if need be.\n",
    "trn_lm = np.load(LM_PATH / \"tmp\" / \"trn_ids.npy\")\n",
    "val_lm = np.load(LM_PATH / \"tmp\" / \"val_ids.npy\")\n",
    "itos = pickle.load(open(LM_PATH / \"tmp\" / \"itos.pkl\", \"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60002, 224472, 31609)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check vocabulary size\n",
    "vocab_size = len(itos)\n",
    "trn_set_size = len(trn_lm)\n",
    "val_set_size = len(val_lm)\n",
    "vocab_size, trn_set_size, val_set_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([list([9, 13, 8, 7, 0, 17, 19, 58, 23, 5, 43, 33863, 2, 3841, 11, 7682, 2, 146, 3, 24, 8, 7, 1675]),\n",
       "       list([9, 13, 8, 7, 700, 17, 25, 298, 4, 225, 12, 119, 2533, 14, 978, 1386, 3, 9, 687, 1835, 2184, 669, 1215, 648, 24, 8, 7, 1587])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sanity check\n",
    "trn_lm[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "em_sz = 400  # size of each embedding vector\n",
    "nh = 1150    # number of hidden activations per layer\n",
    "nl = 3       # number of layers\n",
    "\n",
    "wd = 1e-7\n",
    "bptt = 70\n",
    "bs = 64\n",
    "# opt_fn = partial(optim.Adam, betas=(0.8, 0.99))\n",
    "opt_fn = partial(optim.SGD, momentum=0.9)\n",
    "weight_factor = 0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you're overfitting, increase this. Underfitting? decrease this.\n",
    "drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15]) * weight_factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 0 ns, sys: 0 ns, total: 0 ns\n",
      "Wall time: 6.44 µs\n"
     ]
    }
   ],
   "source": [
    "%time\n",
    "trn_dl = LanguageModelLoader(np.concatenate(trn_lm), bs, bptt)\n",
    "val_dl = LanguageModelLoader(np.concatenate(val_lm), bs, bptt)\n",
    "md = LanguageModelData(DATA_PATH, pad_idx=1, n_tok=vocab_size, trn_dl=trn_dl, val_dl=val_dl, bs=bs, bptt=bptt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner= md.get_model(opt_fn, em_sz, nh, nl, \n",
    "    dropouti=drops[0], dropout=drops[1], wdrop=drops[2], dropoute=drops[3], dropouth=drops[4])\n",
    "\n",
    "learner.metrics = [accuracy]\n",
    "learner.clip = 0.2\n",
    "learner.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method Learner.summary of SequentialRNN(\n",
       "  (0): RNN_Encoder(\n",
       "    (encoder): Embedding(60002, 400, padding_idx=1)\n",
       "    (encoder_with_dropout): EmbeddingDropout(\n",
       "      (embed): Embedding(60002, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDrop(\n",
       "        (module): LSTM(400, 1150)\n",
       "      )\n",
       "      (1): WeightDrop(\n",
       "        (module): LSTM(1150, 1150)\n",
       "      )\n",
       "      (2): WeightDrop(\n",
       "        (module): LSTM(1150, 400)\n",
       "      )\n",
       "    )\n",
       "    (dropouti): LockedDropout(\n",
       "    )\n",
       "    (dropouths): ModuleList(\n",
       "      (0): LockedDropout(\n",
       "      )\n",
       "      (1): LockedDropout(\n",
       "      )\n",
       "      (2): LockedDropout(\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=60002, bias=False)\n",
       "    (dropout): LockedDropout(\n",
       "    )\n",
       "  )\n",
       ")>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# AWD-LSTM network\n",
    "learner.summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hyper-parameter search**\n",
    "\n",
    "Properly setting the hyper-parameters of a neural network can be challenging, thankfully, there are some technique that can help.\n",
    "\n",
    "We will speed up training using Leslie Smith's work on 1cycle policy that he described the [super-convergence] phenomenon in this [paper](https://arxiv.org/abs/1708.07120). Here's an [application of super-convergence to win the DAWNBench challenge](http://www.fast.ai/2018/04/30/dawnbench-fastai/).\n",
    "\n",
    "Based on my own experiments with this method previously, the AWS-LSTM model converged faster, instead of 15 epochs, now it takes just 10 epochs.\n",
    "\n",
    "We will be using an [implementation of this method](http://forums.fast.ai/t/the-1cycle-policy-an-experiment-that-investigate-super-convergence-phenomenon-described-in-leslie-smiths-research/14737):\n",
    "- A simple guide on how to use 1cycle policy [Cyclical Learning Rate (CLR)](http://forums.fast.ai/t/using-use-clr-beta-and-new-plotting-tools/14702)\n",
    "- Some [tips on super-convergence(ish) on WikiText-2](http://forums.fast.ai/t/super-convergence-ish-on-wikitext-2/17091), a similar LM task like ours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec21c7f8b3654333a6ed135a80eedc15",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  9%|▉         | 500/5559 [09:32<1:32:51,  1.10s/it, loss=7.94]"
     ]
    }
   ],
   "source": [
    "# Find learning rate\n",
    "learner.lr_find2(num_it=500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAK9CAYAAADVK0CmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XmcXFWd9/HPr6q6u3pP79k3CAlhJy0EWURZBETiLrihA2YQFdQZZ5h5Zh63mXl0Rh33QQYRZlBGBAREBoisKgh0IEAWAiH73kkn6U7vy+/5o24nlUovRaerb3X39/16lVX31u26Pxvob51z7j3H3B0REZGBRMIuQEREsp/CQkREBqWwEBGRQSksRERkUAoLEREZlMJCREQGpbAQEZFBKSxERGRQCgsRERlULOwChktlZaXPnDkz7DJEREaVpUuX7nL3qsGOGzNhMXPmTOrq6sIuQ0RkVDGzDekcp24oEREZlMJCREQGpbAQEZFBKSxERGRQGQsLM7vFzHaa2fKkfR80sxVm1mNmtQP87EVmttrM1pjZDZmqUURE0pPJlsWtwEUp+5YD7wOe6u+HzCwK/Bi4GJgPXGFm8zNUo4iIpCFjYeHuTwENKftWufvqQX70NGCNu6919w7gf4BFGSqThuYO/t+Dq1i+ZV+mTiEiMupl430WU4BNSdubgdP7OtDMFgOLAaZPnz6kk8Wixq1Pr6e9q4fjp5QO6TNERMa6UT3A7e43uXutu9dWVQ16A2KfSuI5vH1uNb97ZRs9PVqPXESkL9kYFluAaUnbU4N9GXP+/Brqm9p5fef+TJ5GRGTUysaweB6YY2azzCwXuBy4P5MnrJ1RBsALG/dk8jQiIqNWJi+dvQN4BphrZpvN7Coze6+ZbQbOAH5nZg8Hx042swcB3L0L+BzwMLAKuNPdV2SqToDp5QXkxSKsrVfLQkSkLxkb4Hb3K/p56zd9HLsVuCRp+0HgwQyVdphIxJhWXsDGhpaROqWIyKiSjd1QoZhWls+mhtawyxARyUoKi0B1cZzdze1hlyEikpUUFoGKolx27+/Q5bMiIn1QWAQqivLo6nEa2zrDLkVEJOsoLAKVRbkA7NrfEXIlIiLZR2ERqCzKA2D3fo1biIikUlgEygoSLYuGZrUsRERSKSwCxfHELSdNbV0hVyIikn0UFoGSeA6ABrhFRPqgsAgUqWUhItIvhUUgGjGK8mIKCxGRPigskhTHY+qGEhHpg8IiSUk8hyaFhYjIYRQWSUry1Q0lItIXhUWSkniOuqFERPqgsEhSHI/R2KqWhYhIKoVFkpJ8tSxERPqisEhSEs+hsbUTd01TLiKSLJNrcN9iZjvNbHnSvnIzW2JmrwfPZf38bLeZLQse92eqxlQl+TF6HJo7ukfqlCIio0ImWxa3Ahel7LsBeNTd5wCPBtt9aXX3k4PHZRms8RAHpvxoVVeUiEiyjIWFuz8FNKTsXgTcFry+DXhPps4/FCX5mh9KRKQvIz1mUePu24LX24Gafo6Lm1mdmf3ZzEYsUA62LHRFlIhIslhYJ3Z3N7P+RpJnuPsWM5sNPGZmr7j7G6kHmdliYDHA9OnTj7imkvzeyQTVshARSTbSLYsdZjYJIHje2ddB7r4leF4LPAGc0s9xN7l7rbvXVlVVHXFxmqZcRKRvIx0W9wNXBq+vBO5LPcDMyswsL3hdCZwJrByJ4g6MWagbSkTkEJm8dPYO4BlgrpltNrOrgG8CF5jZ68D5wTZmVmtmNwc/eixQZ2YvAY8D33T3EQmL3tXydDWUiMihMjZm4e5X9PPWeX0cWwdcHbx+GjghU3UNJCcaIT8nqm4oEZEUuoM7RUl+jH1qWYiIHEJhkaKsIJc9LQoLEZFkCosU5YW57GnuCLsMEZGsorBIUVaQS0OLwkJEJJnCIkVZYQ571Q0lInIIhUWKsoJc9rZ00N2jacpFRHopLFKUFeTS47rXQkQkmcIiRVlh4i7uPRq3EBE5QGGRoqwgF1BYiIgkU1ikKC8MwqJZ3VAiIr0UFil6Wxa6fFZE5CCFRYqyAy0LhYWISC+FRYrC3Cg5UdOUHyIiSRQWKcwscRd3c3vYpYiIZA2FRR+qS/Kob1JYiIj0Ulj0oaY4zo5GhYWISC+FRR+qS+LsbGoLuwwRkayhsOhDTUkeu/Z30NndE3YpIiJZIZNrcN9iZjvNbHnSvnIzW2JmrwfPZf387JXBMa+b2ZWZqrE/NSVxAI1biIgEMtmyuBW4KGXfDcCj7j4HeDTYPoSZlQNfAU4HTgO+0l+oZEpNSR4AOxUWIiJABsPC3Z8CGlJ2LwJuC17fBrynjx99J7DE3RvcfQ+whMNDJ6OqixMtix2NGrcQEYGRH7OocfdtwevtQE0fx0wBNiVtbw72jZjebqidCgsRESDEAW53d+CIVhgys8VmVmdmdfX19cNUGVQU5hKNmC6fFREJjHRY7DCzSQDB884+jtkCTEvanhrsO4y73+Tute5eW1VVNWxFRiJGdXEe29WyEBEBRj4s7gd6r266Erivj2MeBi40s7JgYPvCYN+ImlgaZ/s+hYWICKQRFmb2r2ZWYmY5ZvaomdWb2cfS+Lk7gGeAuWa22cyuAr4JXGBmrwPnB9uYWa2Z3Qzg7g3AN4Dng8fXg30janJpPlv3to70aUVEslIsjWMudPe/MbP3AuuB9wFPAbcP9EPufkU/b53Xx7F1wNVJ27cAt6RRW8ZMnhDn96t24O6YWZiliIiELp1uqN5AeRfwa3ffl8F6ssbkCfm0d/VoqnIREdILiwfM7FVgAfComVUBY74zf/KEfAB1RYmIkEZYuPsNwFuBWnfvBJpJ3Fw3pk0JwmKLwkJEJK0B7g8Cne7ebWb/QGKsYnLGKwuZWhYiIgel0w31j+7eZGZnkbiC6WfAf2S2rPCVFeQQz4koLERESC8suoPndwE3ufvvgNzMlZQdzIzJE/LZunfMD8+IiAwqnbDYYmY/BT4MPGhmeWn+3Kg3ZUI+m/a0hF2GiEjo0vmj/yESd1C/0933AuXAlzNaVZaYWVHI+l3NJKaxEhEZv9K5GqoFeAN4p5l9Dqh290cyXlkWmFlZSGNbFw3NHWGXIiISqnSuhroe+AVQHTxuN7PPZ7qwbDC7shCAdbuaQ65ERCRc6Uz3cRVwurs3A5jZt0jM+fTDTBaWDWYlhUXtzPKQqxERCU86YxbGwSuiCF6Pi8mSppblE4uYWhYiMu6l07L4OfCsmf0m2H4PiXstxrxYNML08gKFhYiMe4OGhbt/18yeAM4Kdn3K3V/MaFVZZFZlocJCRMa9fsPCzJI76dcHjwPvhbHGRBhmVhbypzd20dPjRCLjovdNROQwA7UslpJYI7v3L2TvzQYWvJ6dwbqyxqzKQto6e9je2HZgvigRkfGm37Bw91kjWUi2Sr58VmEhIuPVuJi240jMqkqExVqNW4jIOKawGERNcZx4ToT1CgsRGcdCCQszu97MlpvZCjP7Qh/vn2tm+8xsWfD4v2HUCRCJGDMrdEWUiIxvg146m3JVVK+mYNW8N83Mjgc+DZwGdAAPmdkD7r4m5dA/uPulQznHcDuqqohXtoyLpcdFRPqUTsviBaAeeA14PXi93sxeMLMFQzjnscCz7t7i7l3Ak8D7hvA5I+aYmmI27WmhpaMr7FJEREKRTlgsAS5x90p3rwAuBh4ArgV+MoRzLgfONrMKMysALgGm9XHcGWb2kpn9r5kd19cHmdliM6szs7r6+vohlJKeY2qKcIc1O/dn7BwiItksnbBY6O4P924E05Of4e5/BvLe7AndfRXwLeAR4CFgGYfOPQWJ1swMdz+JxISF9/bzWTe5e62711ZVVb3ZUtJ2zMRiAF7bobAQkfEpnbDYZmZ/a2YzgsffADvMLAr0DOWk7v4zd1/g7ucAe0h0cSW/3+ju+4PXDwI5ZlY5lHMNhxnlBeTGIry2oymsEkREQpVOWHwEmEri2/29wPRgX5TEKnpvmplVB8/TSYxX/DLl/YlmZsHr04I6dw/lXMMhFo1wdFURq7crLERkfEpnIsFdQH+LHaVewZSuu82sAugEPuvue83smuB8NwIfAD5jZl1AK3C5h7y26TE1RTy3blxMhyUicph0Lp09BvhrYGby8e7+jqGe1N3P7mPfjUmvfwT8aKifnwnHTCzm3mVbaWzrpCSeE3Y5IiIjKp31LH4N3AjczOED0ePG3JrEIPfrO5pYMEOr5onI+JJOWHS5+39kvJIsd+ykEgCWb2lUWIjIuJPOAPdvzexaM5tkZuW9j4xXlmUmlcapLs5j2aa9YZciIjLi0mlZXBk8fzlp37hZz6KXmXHytAkKCxEZl9K5GkrrWgROmjaBR1buYG9LBxMKcsMuR0RkxAy0rOo73P0xM+tz3iZ3vydzZWWnU6ZNAGDZpr2cO7c65GpEREbOQC2LtwGPAe/u4z0Hxl1YnDC1FDOFhYiMPwMtq/qV4PlTI1dOdiuO5zCnukjjFiIy7qRzU14e8H4Ovynv65krK3udPG0CS1buwN0JZiQRERnz0rl09j5gEdAFNCc9xqWTp5Wxp6WTDbtbwi5FRGTEpHPp7FR3vyjjlYwSJycNcs+sLAy5GhGRkZFOy+JpMzsh45WMEsfUFJGfE9W4hYiMK+m0LM4CPmlm64B2wAB39xMzWlmWikUjnDi1lLoNmoFWRMaPdMLi4oxXMcqcPaeSbz/yGvVN7VQVv+nFAkVERp1+u6HMrCR42dTPY9zqvcfiydcyt+63iEg2GWjMonf1uqVAXfC8NGl73DpucglVxXk8sXpn2KWIiIyIgW7KuzR41txQKcyMc4+p4uEV2+nq7iEWTec6ARGR0Sutv3JmVmZmp5nZOb2PTBeW7c6dW01jW5euihKRcWHQsDCzq4GngIeBrwXPXz2Sk5rZ9Wa23MxWmNkX+njfzOwHZrbGzF42s1OP5HyZcNacSqIR43F1RYnIOJBOy+J64C3ABnd/O3AKMOSv02Z2PPBp4DTgJOBSMzs65bCLgTnBYzGQdSv1lebnsGBGGY+9qkFuERn70gmLNndvg8Q8Ue7+KjD3CM55LPCsu7e4exfwJJA6Dfoi4L884c/ABDObdATnzIh3zKtm1bZGtu1rDbsUEZGMSicsNpvZBOBeYImZ3QdsOIJzLgfONrMKMysALgGmpRwzBdiUXEOwL6ucNy9xCe2jq9QVJSJjWzor5b03ePlVM3scKAUeGuoJ3X2VmX0LeITEhITLgO6hfJaZLSbRTcX06dOHWtKQHV1dxKzKQh5avp2PLZwx4ucXERkpA7YszCxqZq/2brv7k+5+v7t3HMlJ3f1n7r7A3c8B9gCvpRyyhUNbG1ODfamfc5O717p7bVVV1ZGUNCRmxsXHT+SZtbvZ03xEvxIRkaw2YFi4ezew2syG9Wu7mVUHz9NJjFf8MuWQ+4FPBFdFLQT2ufu24axhuFxywiS6e5z/Xb497FJERDImnbmhyoAVZvYcSetYuPtlR3Deu82sAugEPuvue83smuBzbwQeJDGWsQZoAbJ2tb7jJpdwdHURv3lxMx85feS7wkRERkI6YfGPw31Sdz+7j303Jr124LPDfd5MMDPee8oU/u3h1WxqaGFaeUHYJYmIDLt0roa6JBirOPAg8a1fAotOngzAvS8eNqwiIjImpBMWF/SxT9OWJ5laVkDtjDKNW4jImDXQFOWfMbNXgLnBlBu9j3XAyyNX4ujwzuMmsnJbI5satDa3iIw9g01R/m4SVya9O+mxwN0/NgK1jSoXHlcDwMMr1LoQkbGn37Bw933uvt7dr3D3DUkPrSfahxkVhZwwpZRfPb+JxPi8iMjYoYUYhtEn3zqT13fu549rdoVdiojIsFJYDKNLT5pEZVEet/xxXdiliIgMK4XFMMqLRfn4whk8vrqeN+r3h12OiMiwUVgMs48unE5uNMKtf1ofdikiIsNGYTHMKovyWHTyZO5aupl9LZ1hlyMiMiwUFhnwqTNn0drZza/qNoZdiojIsFBYZMD8ySUsnF3OLX9cT2vHkJbqEBHJKgqLDPnC+cewvbGNW/6kK6NEZPRTWGTIwtkVXDC/hp88vob6pvawyxEROSIKiwy64eJ5tHf18O+/T10IUERkdFFYZNBRVUV8/IwZ3PHcRpZt2ht2OSIiQ6awyLAvXXAMVUV5/MO9r9DdozmjRGR0UlhkWHE8h3+8dD7LtzTy38+sD7scEZEhCSUszOyLZrbCzJab2R1mFk95/5NmVm9my4LH1WHUOVwuPXESZ8+p5NuPvMaOxrawyxERedNGPCzMbApwHVDr7scDUeDyPg79lbufHDxuHtEih5mZ8fVFx9PR3cM3HlgZdjkiIm9aWN1QMSDfzGJAAbA1pDpGzKzKQq499ygeeHkbDy3fFnY5IiJvyoiHhbtvAb4NbAS2Afvc/ZE+Dn1/sIzrXWY2bUSLzJBrzz2ak6aW8uW7XtbyqyIyqoTRDVUGLAJmAZOBQjNLXab1t8BMdz8RWALc1s9nLTazOjOrq6+vz2TZwyI3FuFHHzkVgOv+50W6untCrkhEJD1hdEOdD6xz93p37wTuAd6afIC773b33tuebwYW9PVB7n6Tu9e6e21VVVVGix4u08oL+Of3nsCLG/dy45NvhF2OiEhawgiLjcBCMyswMwPOA1YlH2Bmk5I2L0t9f7S77KTJXHbSZL73+9d5YeOesMsRERlUGGMWzwJ3AS8ArwQ13GRmXzezy4LDrgsurX2JxJVTnxzpOjPtG4uOZ9KEONf891JdTisiWc/cx8ZdxbW1tV5XVxd2GW/Kq9sbed9PnmZOTTG/WryQeE407JJEZJwxs6XuXjvYcbqDO0TzJpbw3Q+dzEub9vL397zCWAluERl7FBYhu+j4iXzpgmO458Ut/OvDq8MuR0SkT7GwCxD4/DuOZntjG//xxBtUFOZy9dmzwy5JROQQCossYGZ8Y9Hx7Gnu4J9+t4rywlzed+rUsMsSETlA3VBZIhoxvnf5ybz1qAr++tcv8buXNSWIiGQPhUUWyYtF+c9P1HLq9DKu/58XNYeUiGQNhUWWKcyL8fNPvYUTppbymV+8wE1PvaGrpEQkdAqLLFQcz+GXVy/k4uMn8i8Pvsrf3v0yHV2aR0pEwqOwyFL5uVF+dMWpXPeOo7mzbjMfu/lZ6pvaB/9BEZEMUFhksUjE+NKFc/n+5Sfz8pa9XPrDP2guKREJhcJiFFh08hTu+cyZ5MWifPinz/Djx9doenMRGVEKi1Fi/uQSfvu5s7hw/kT+7eHVXPDvT/Hoqh1hlyUi44TCYhQpLcjhxx89lZs/UUssYlx1Wx1//euX2NmkWWtFJLMUFqPQ+fNreOC6s7j23KO498UtvP3fnuCHj75Oa0d32KWJyBilsBil8mJR/uaieSz50ts4e04V31nyGu/4zhP8um4TnRrPEJFhprAY5WZVFnLjxxdw51+eQXVxHl++62Xe/u0nuP3PG2jrVEtDRIaHFj8aQ9ydR1ft5EePr2HZpr1UF+ex+JzZfOT06RTkas5IETlcuosfKSzGIHfn6Td28+PH1/D0G7spK8jh6rNn8xdnziI/V6vxichBWR0WZvZF4GrASazD/Sl3b0t6Pw/4L2ABsBv4sLuvH+gzFRZ9e2HjHn702Boee3Unk0rjfPKtM3nfqVOpKs4LuzQRyQJZGxZmNgX4IzDf3VvN7E7gQXe/NemYa4ET3f0aM7sceK+7f3igz1VYDOzZtbv5ziOv8dz6BqIR4x3zqvlQ7TTOnVtFTlRDVyLjVbphEVZHdgzIN7NOoADYmvL+IuCrweu7gB+ZmflY6TMLwemzK7jzmjN4o34/d9Zt4u6lW1iycgeVRXm8/9QpfLB2GkdXF4VdpohkqbC6oa4H/hloBR5x94+mvL8cuMjdNwfbbwCnu/uu/j5TLYs3p7O7hydX13Nn3SYee3UnXT3OqdMncMkJk6guiVOan8OE/BwmlsapLs7DzMIuWUQyIGtbFmZWRqLlMAvYC/zazD7m7rcP4bMWA4sBpk+fPqx1jnU50Qjnz6/h/Pk11De185sXN3Nn3Wb+6XerDju2ND+HY2qKOKammLkTizmmJvEoL8wNoXIRCUMYYxYfJNFquCrY/gSw0N2vTTrmYeCr7v6MmcWA7UDVQN1QalkcOXdn1/4O9rV2sK+1k70tnWze08prO5p4bUcTq7c30djWdeD4yqI85k4sYk51cogUURzPCfH/hYi8GVnbsgA2AgvNrIBEN9R5QOpf+fuBK4FngA8Aj2m8IvPMjKrivH6vlHJ3djS2HxIer+1o4s66TbQkTTUyuTTOMROLmT+phAUzyjhleplaISKj3IiHhbs/a2Z3AS8AXcCLwE1m9nWgzt3vB34G/LeZrQEagMtHuk45nJkxsTTOxNI45xxTdWB/T4+zZW9rIjx2NvHa9iZe3d7EH1/fRVdPIuPLC3OZWVHAMTXFHDuphGMnlTBvUjElaoWIjAq6KU8yprWjm5c27+XlzXtZt6uZtfXNvLajiT0tnQeOmVaez7ETS5g/OREg8yeVMLUsXwPqIiMkm7uhZJzIz42ycHYFC2dXHNjX25W1cts+Vm1rYuW2RlZtbWTJqh30fm8pjseSAqSY+ZNKmVNTRDyn/7vP3V0Bk+V6epym9i7cHXfocafHwUlsH9x3cNtJHHNwn+Mktru6nfauHjq6emjv6j7wuqOrh56kL8GpX4dzoxFyohFyY4lHTtTIiyX2Rcww45DniAEkni3YNoL3I4Zx8HgL3otY8r7EdiwSIRKBqBnRiA3p39fuHmfbvlY27m5h/e4WNjQ0s3F3C1XFeXx90fFH8E9ncAoLGVHJXVnvmFdzYH9LRxevbm9i1bZGVm1rZOXWxkPGQmIR4+jqIiaWxtnT0klDcztNbV20dyb+UEQjRk1JnMml+dSUxikvyGFCQS5lBTlMmpDPtLICppXna/B9hOxv72L19kZWbjv4z3T19qZDxrbGu4hBNJIIjvycKAW5MQpyoxTkxSjIiVKYFyU/N0Z+ToT6pnY2NLSwuaGVjqRZpXOjEaaW5zOhIPNjguqGkqzV0+NsaGhh5dZGVm7bx4qtjeza305ZQS4VhbmU5OcQz4mSG43Q2dPD9n1tbNvXxo7GNvY0dxxy5VavWMSIRY2cSIRY1IhFIwf25edEKY7nUJQXoygeozgvduB1UV6MvJwo8ViEeE6UorwYE4JAmpCfQ0l+DtHI6GrZ7G/vYlNDC7v2t7Nrfzu793dQv7+dfS2ddHT30NXtdHb3BI/DX/e+39HH69akGY9L4jHmJXUx9n5b7/1Wbinf4nu/tR/4Fn/gW/qh3+KjkUSLIC8WJTcWCV4nWguRlG/tvZvuHPj/0NHVQ0d3Nx1dTkd3D51BiyQxzHZoi6a3pXJwO9jXewyHtn56kp5793e7092T8gj2tXZ009LRTUtHV8pz4nVFYR4zKgqYXlHAzIpCZpQXMKOykIkl8SP+907dUDLqRSLGrMpCZlUW8q4TJ73pn+/q7mFPSydb97ayaU8LmxpaaWzrpLvn4B+7rp7e58R/sE3tnext6WDTnhb2t3Wxv70r7W/DJfHYgdZMUTxGfk7wTTE3Sn5uNPj2mPi22Ls/HuwryI0eOD4/eBTkRIkN41QsLR1d1K3fw9Nv7OaZtbtZvmUf3T2HflnMjUaYUJATdM8kumhyohFi0Qi5USMWiZCfGyEnktifEzv4OhYcmxM1SvNzmDexhGMnlzC5NK4uwjFAYSFjViwaOXAp8EnTJgz5c7q6e2jp7Kats5v2zh7aOrtpau9iX2sn+1o62dPSwd6WzuDelA72tnbS1NZFQ3MrrcE3xNbOblo7ug9cHZau3GgkJWgOBk5+ToSC3BhlBbnUlORRXZJHTXGcyuI8Gpo72NjQwqaGFjY2tLB+VzMrtzXS2e3EIsbJ0yZw7blHceykEiqL8qgsyqWiKI+SeEx/2KVPCguRQcSiEUqikWG5zLejqyfR5dDZdaDrobUzeO7oSnrdfaAboq3zYLdE7/59rZ3s2NdNc0cXDc0d/bZ+ohFj8oQ408oKuOqs2ZxxVAW1M8oozNN/+vLm6N8YkRHUewVOKcM70L6/vYudjW3saEyMP0woyGF6eQGTJ+RrVmEZFgoLkTGgKC9GUVURs6s0c7Bkhr5yiIjIoBQWIiIyKIWFiIgMSmEhIiKDUliIiMigFBYiIjKoMTM3lJnVAxuO4CMqgX7X+Bb9fgag383A9PsZWNi/nxnuXjXYQWMmLI6UmdWlM5nWeKXfT//0uxmYfj8DGy2/H3VDiYjIoBQWIiIyKIXFQTeFXUCW0++nf/rdDEy/n4GNit+PxixERGRQalmIiMigFBYiIjKocR8WZnaRma02szVmdkPY9WQTM5tmZo+b2UozW2Fm14ddUzYys6iZvWhmD4RdS7YxswlmdpeZvWpmq8zsjLBryiZm9sXgv63lZnaHmcXDrqk/4zoszCwK/Bi4GJgPXGFm88OtKqt0AX/l7vOBhcBn9fvp0/XAqrCLyFLfBx5y93nASej3dICZTQGuA2rd/XggClweblX9G9dhAZwGrHH3te7eAfwPsCjkmrKGu29z9xeC100k/kOfEm5V2cXMpgLvAm4Ou5ZsY2alwDnAzwDcvcPd94ZbVdaJAflmFgMKgK0h19Ov8R4WU4BNSdub0R/DPpnZTOAU4NlwK8k63wP+BugJu5AsNAuoB34edNPdbGaFYReVLdx9C/BtYCOwDdjn7o+EW1X/xntYSBrMrAi4G/iCuzeGXU+2MLNLgZ3uvjTsWrJUDDgV+A93PwVoBjQuGDCzMhI9GbOAyUChmX0s3Kr6N97DYgswLWl7arBPAmaWQyIofuHu94RdT5Y5E7jMzNaT6MJ8h5ndHm5JWWUzsNnde1ujd5EID0k4H1jn7vXu3gncA7w15Jr6Nd7D4nlgjpnNMrNcEoNL94dcU9YwMyPR37zK3b8bdj3Zxt3/zt2nuvtMEv/uPObuWfvNcKS5+3Zgk5nNDXadB6wMsaRssxFYaGYFwX9r55HFFwDEwi4gTO7eZWafAx4mcSXCLe6+IuSyssmZwMeBV8xsWbDv7939wRBrktHl88Avgi9ja4FjDVoGAAAgAElEQVRPhVxP1nD3Z83sLuAFElcevkgWT/2h6T5ERGRQ470bSkRE0qCwEBGRQSksRERkUAoLEREZlMJCREQGpbCQccPM9o/AOS4b6dmLzexcM8vam7lkbBjX91mIDIWZRd29u6/33P1+MnBjp5nF3L2rn7fPBfYDTw/3eUV6qWUh45KZfdnMnjezl83sa0n77zWzpcEaA4uT9u83s++Y2UvAGWa23sy+ZmYvmNkrZjYvOO6TZvaj4PWtZvYDM3vazNaa2QeC/REz+0mwxsMSM3uw972UGp8ws++ZWR1wvZm928yeDSbl+72Z1QQTPF4DfNHMlpnZ2WZWZWZ3B///njezMzP5u5TxQS0LGXfM7EJgDokp6g2438zOcfengL9w9wYzyweeN7O73X03UAg86+5/FXwGwC53P9XMrgX+Gri6j9NNAs4C5pFocdwFvA+YSWINlWoSUzzc0k+5ue5eG5yzDFjo7m5mVwN/4+5/ZWY3Avvd/dvBcb8E/t3d/2hm00nMUHDskH9hIigsZHy6MHi8GGwXkQiPp4DrzOy9wf5pwf7dQDeJCRWT9U6suJREAPTlXnfvAVaaWU2w7yzg18H+7Wb2+AC1/irp9VTgV2Y2CcgF1vXzM+cD84NAAygxsyJ3z/iYjYxdCgsZjwz4f+7+00N2mp1L4g/tGe7eYmZPAL3LXLb1MU7RHjx30/9/S+1Jr62fYwbSnPT6h8B33f3+oNav9vMzERItkLYhnE+kTxqzkPHoYeAvgnU6MLMpZlYNlAJ7gqCYR2Ip2Uz4E/D+YOyihsQAdTpKOTiF/pVJ+5uA4qTtR0hM4AeAmZ089FJFEhQWMu4Eq5H9EnjGzF4hMY5QDDwExMxsFfBN4M8ZKuFuEms9rARuJzHr6L40fu6rwK/NbCmwK2n/b4H39g5wE6zrHAzeryQxAC5yRDTrrEgIescQzKwCeA44M1j/QSQracxCJBwPmNkEEgPV31BQSLZTy0JERAalMQsRERmUwkJERAalsBARkUEpLEREZFAKCxERGZTCQkREBjVm7rOorKz0mTNnhl2GiMiosnTp0l3uXjXYcWMmLGbOnEldXV3YZYiIjCpmtiGd49QNJSIig1JYiIjIoBQWIiIyqIyFhZndYmY7zWx50r4PBmsb95hZ7QA/e5GZrTazNWZ2Q6ZqFBGR9GSyZXErcFHKvuUklp98qr8fMrMo8GPgYhJrFF9hZvMzVKOIiKQhY2Hh7k8BDSn7Vrn76kF+9DRgjbuvdfcO4H+ARRkqE4AVW/fR2d2TyVOIiIxq2ThmMQXYlLS9OdiXEftaO3nXD/7I5375QqZOISIy6mVjWKTNzBabWZ2Z1dXX1w/pMwpyo+TGIqzc1jjM1YmIjB3ZGBZbgGlJ21M5uEj9Idz9JnevdffaqqpBb0DsU040wvXnzWFTQyv1Te1D+gwRkbEuG8PieWCOmc0ys1zgcuD+TJ5wwYwyIDF2ISIih8vkpbN3AM8Ac81ss5ldZWbvNbPNwBnA78zs4eDYyWb2IIC7dwGfAx4GVgF3uvuKTNUJcOykEgB1RYmI9CNjc0O5+xX9vPWbPo7dClyStP0g8GCGSjtMaX4OlUW5bGpoGalTioiMKtnYDRWKiaVxtu1rC7sMEZGspLAITCzJZ7vCQkSkTwqLwMTSPLY3KixERPqisAiUF+axr7WT7h4PuxQRkayjsAiUFeTgnrijW0REDqWwCJQV5AKwp6Uj5EpERLKPwiIwoSAHgL0KCxGRwygsAgdaFs3qhhIRSaWwCKgbSkSkfwqLQEl+4mb2praukCsREck+CotAUZ7CQkSkPwqLQCwaoTA3SmObxixERFIpLJKU5OfQqPssREQOo7BIUhLPUctCRKQPCoskJfkxjVmIiPRBYZGkWC0LEZE+KSySlMRjNLaqZSEikkphkaQkXy0LEZG+ZHIN7lvMbKeZLU/aV25mS8zs9eC5rJ+f7TazZcHj/kzVmKoknkNTWxfumqZcRCRZJlsWtwIXpey7AXjU3ecAjwbbfWl195ODx2UZrPEQJfkxunuclo7ukTqliMiokLGwcPengIaU3YuA24LXtwHvydT5h6I4nph5Vl1RIiKHGukxixp33xa83g7U9HNc3MzqzOzPZjZigVLSGxYa5BYROUQsrBO7u5tZf4MDM9x9i5nNBh4zs1fc/Y3Ug8xsMbAYYPr06UdcU+9kgmpZiIgcaqRbFjvMbBJA8Lyzr4PcfUvwvBZ4Ajiln+Nucvdad6+tqqo64uIOtiwUFiIiyUY6LO4HrgxeXwncl3qAmZWZWV7wuhI4E1g5EsUVxzXzrIhIXzJ56ewdwDPAXDPbbGZXAd8ELjCz14Hzg23MrNbMbg5+9FigzsxeAh4HvunuIxIWJfka4BYR6UvGxizc/Yp+3jqvj2PrgKuD108DJ2SqroH0tizUDSUicijdwZ0kLxYlnhOhUd1QIiKHUFikSNzFrZaFiEgyhUWKYk0mKCJyGIVFCk0mKCJyOIVFipK4llYVEUmlsEiRaFmoG0pEJJnCIkVxPKYBbhGRFAqLFIluKK1pISKSTGGRoiQ/Rkd3D+1dPWGXIiKSNRQWKTSZoIjI4RQWKQ7OD6VBbhGRXgqLFAfmh9Igt4jIAQqLFOqGEhE5nMIiRemB1fLUDSUi0kthkaJ3zGJfS0fIlYiIZA+FRYqyglwAdjcrLEREeiksUuREI5Tm59CgsBAROUBh0YeKolx271dYiIj0yuQa3LeY2U4zW560r9zMlpjZ68FzWT8/e2VwzOtmdmWmauxPRWEuu5vbR/q0IiJZK5Mti1uBi1L23QA86u5zgEeD7UOYWTnwFeB04DTgK/2FSqZUFOapZSEikmTQsDCzQjOLBK+PMbPLzCxnsJ9z96eAhpTdi4Dbgte3Ae/p40ffCSxx9wZ33wMs4fDQyajyolyNWYiIJEmnZfEUEDezKcAjwMdJtBqGosbdtwWvtwM1fRwzBdiUtL052DdiKgtzaWjpoLtHM8+KiEB6YWHu3gK8D/iJu38QOO5IT+yJOcCP6K+xmS02szozq6uvrz/Skg4oL8zFHfbqXgsRESDNsDCzM4CPAr8L9kWHeL4dZjYp+NBJwM4+jtkCTEvanhrsO4y73+Tute5eW1VVNcSSDldRlAfoXgsRkV7phMUXgL8DfuPuK8xsNvD4EM93P9B7ddOVwH19HPMwcKGZlQUD2xcG+0ZMRVFwY54GuUVEAIgNdoC7Pwk8CRAMdO9y9+sG+zkzuwM4F6g0s80krnD6JnCnmV0FbAA+FBxbC1zj7le7e4OZfQN4Pvior7t76kB5RlUU9rYsdPmsiAikERZm9kvgGqCbxB/wEjP7vrv/20A/5+5X9PPWeX0cWwdcnbR9C3DLYLVlSm/LQldEiYgkpNMNNd/dG0lc5vq/wCwSV0SNWWUFuZjBria1LEREIL2wyAnuq3gPcL+7d3KEVzFlu2jEqCrKY0ejwkJEBNILi58C64FC4CkzmwE0ZrKobDCpNM62xrawyxARyQqDhoW7/8Ddp7j7JZ6wAXj7CNQWqomlcbbvaw27DBGRrJDOdB+lZvbd3pvfzOw7JFoZY9qk0ny27VPLQkQE0uuGugVoInGZ64dIdEH9PJNFZYOJpXGa2rrY367lVUVEBr10FjjK3d+ftP01M1uWqYKyxaTSOADb97VxdHVRyNWIiIQrnZZFq5md1bthZmcCY74zf2LJwbAQERnv0mlZfAa4zcxKASMx7fgnM1lUNphUmg/AVg1yi4ikNd3HMuAkMysJtsf8ZbOQGLMwgy17FBYiIv2GhZl9qZ/9ALj7dzNUU1bIjUWYXJrPxoaWsEsREQndQC2L4hGrIktNLy9gw+7msMsQEQldv2Hh7l8byUKy0YyKAn6/akfYZYiIhC6dq6HGrWnlBeza30Gz7rUQkXFOYTGAmRWJG9XX7VJXlIiMbwqLARxVnQiLtQoLERnn0ln8KA94PzAz+Xh3/3rmysoOMysKiRi8sXN/2KWIiIQqnZvy7gP2AUuBcbXAQzwnyrTyAtbUKyxEZHxLJyymuvtFw3lSM7se+DSJO8L/092/l/L+uSRCal2w656wWjJHVRWpZSEi4146YfG0mZ3g7q8MxwnN7HgSQXEa0AE8ZGYPuPualEP/4O6XDsc5j8RRVYX8ac0uunucaMTCLkdEJBTpDHCfBSw1s9Vm9rKZvWJmLx/BOY8FnnX3FnfvAp4E3ncEn5dRR1UV0d7Vw9a9mvZDRMavdFoWFw/zOZcD/2xmFSRmr70EqOvjuDPM7CVgK/DX7r5imOtIS+/05Gt27mdaeUEYJYiIhC6dZVU3ABOAdwePCcG+IXH3VcC3gEeAh4BlQHfKYS8AM9z9JOCHwL19fZaZLe5dwa++vn6oJQ3oqKpEWLyhQW4RGcfSWVb1euAXQHXwuN3MPn8kJ3X3n7n7Anc/B9gDvJbyfqO77w9ePwjkmFllH59zk7vXunttVVXVkZTUr7LCXMoLcxUWIjKupdMNdRVwurs3A5jZt4BnSHzjHxIzq3b3nWY2ncR4xcKU9ycCO9zdzew0EqG2e6jnO1JHVRXyxk7dmCci41c6YWEc2k3UHew7EncHYxadwGfdfa+ZXQPg7jcCHwA+Y2ZdJMY1Lnd3P8JzDtnR1UU8vEITCorI+JVOWPwceNbMfhNsvwf42ZGc1N3P7mPfjUmvfwT86EjOMZyOqiqioXkTDc0dlBfmhl2OiMiIS2elvO+a2RMkLqEF+JS7v5jRqrJM7yD32vr9lBeWh1yNiMjIG2ilvBJ3bzSzcmB98Oh9r9zdGzJfXnZIviKqdqbCQkTGn4FaFr8ELiUxJ1TyeIEF27MzWFdWmVKWT35OlFXbmsIuRUQkFAOtlHdp8Dxr5MrJTtGIccLUUl7ctDfsUkREQpHOfRaPprNvrFswo4wVW/bR1pl6/6CIyNjXb1iYWTwYr6g0szIzKw8eM4EpI1VgtlgwvYyuHuflzfvCLkVEZMQNNGbxl8AXgMkkxi16761oJIsuax0pp84oA2Dphj2cNkuD3CIyvgw0ZvF94Ptm9nl3H/Ld2mNFeWEusysLWbphT9iliIiMuHTus/hhsAbFfCCetP+/MllYNjp1RhmPv7oTd8dMa1uIyPiRzgD3V0jMA/VD4O3AvwKXZbiurLRgRhm7mzvYsLsl7FJEREZUOosffQA4D9ju7p8CTgJKM1pVllqQNG4hIjKepBMWre7eA3SZWQmwE5iW2bKy09FVRRTHYyzdqLAQkfElnYkE68xsAvCfJK6K2k9iivJxJxIxTplexgtqWYjIOJPOSnnXuvveYFbYC4Arg+6ocWnB9DJW72iisa0z7FJEREbMQBMJnjrQe+7+QmZKym4LZpThDss27uWcYzKzOp+ISLYZqBvqO8FzHKgFXiJxY96JQB1wRmZLy04nTSslYolBboWFiIwX/XZDufvb3f3twDbg1GCt6wXAKcCWkSow2xTHc5g7sYTn14+bGdpFRNK6Gmquu7/Su+Huy4FjM1dS9jtnTiXPr2/QuIWIjBvphMXLZnazmZ0bPP4TePlITmpm15vZcjNbYWZf6ON9M7MfmNkaM3t5oPGTMFwwv4bObufJ1fVhlyIiMiLSCYtPASuA64PHymDfkARTh3waOI3EDX6XmtnRKYddDMwJHouB/xjq+TLhlOllVBTmsmTljrBLEREZEenMDdUG/HvwGA7HAs+6ewuAmT0JvI/ENCK9FgH/5e4O/NnMJpjZJHffNkw1HJFoxHjHvGoeWrGdzu4ecqLpZK6IyOg10HoWdwbPrwRdQYc8juCcy4GzzazCzAqASzj8jvApwKak7c30sYaGmS02szozq6uvH9kuoQvm19DU1sWzazXQLSJj30Ati+uD50uH84TuvsrMvgU8AjQDy4AhLT/n7jcBNwHU1tb6IIcPq7PnVBHPibBk5XbOmlM5kqcWERlxA106uy143tDX40hO6u4/c/cF7n4OsAd4LeWQLRza2phKll2um58b5ayjq1iycgeJ3jIRkbFroG6oJjNr7OPRZGaNR3JSM6sOnqeTGK/4Zcoh9wOfCK6KWgjsy5bximQXzq9h6742Vmw9ol+HiEjWG2ilvOIMnvduM6sAOoHPuvteM7smOO+NwIMkxjLWAC0cwdVXmfSOY6sxgyUrd3D8lHE5a7uIjBPpzDoLHGgNJK+Ut3GoJ3X3s/vYd2PSawc+O9TPHymVRXksmF7GkpU7+OIFx4RdjohIxqSzUt5lZvY6sA54ElgP/G+G6xo1Lphfw8ptjWzeo9XzRGTsSucGgW8AC4HX3H0WiVXz/pzRqkaRi4+fBMB9y7aGXImISOakExad7r4biJhZxN0fJzELrQDTKwo4bVY5dy3drKuiRGTMSics9ppZEfAU8Asz+z6J+yMk8MEFU1m3q5kXtNyqiIxR6YTFIhJXJH0ReAh4A3h3JosabS45YRIFuVF+9fymwQ8WERmF0gmLvwQmuXuXu9/m7j8IuqUkUJgXY9HJk/ntS9vY16ppy0Vk7EknLIqBR8zsD2b2OTOryXRRo9FHTptBa2c3v3lhc9iliIgMu0HDwt2/5u7HkbjvYRLwpJn9PuOVjTInTC3lpKml/OLZjRroFpEx583Mrb0T2A7sBqozU87o9tGFM3h9536eW6eZaEVkbEnnprxrzewJ4FGgAvi0u5+Y6cJGo3efOJmSeIxbn14fdikiIsMqnek+pgFfcPdlmS5mtMvPjfKJM2by4yfW8PqOJubUZHJ6LRGRkZPOmMXfKSjS9xdnzSIei/KTJ94IuxQRkWGj9UCHWXlhLh9bOJ37lm1h3S7duygiY4PCIgM+fc5s8mJRvrskdU0nEZHRSWGRAdXFca46axa/fWkrL2/eG3Y5IiJHTGGRIYvfNpuKwly+9tuVuu9CREY9hUWGlMRz+NuL5rF0wx5+82JWLR8uIvKmKSwy6AMLpnLStAn8y4Ov0timOaNEZPQKJSzM7ItmtsLMlpvZHWYWT3n/k2ZWb2bLgsfVYdR5pCIR4xuLjmN3czvf//3rYZcjIjJkIx4WZjYFuA6odffjgShweR+H/srdTw4eN49okcPoxKkTuPwt07n16fWs3t4UdjkiIkMSVjdUDMg3sxhQAIzpNUm//M65FMdj3HDPy3R09YRdjojImzbiYeHuW4BvAxuBbcA+d3+kj0Pfb2Yvm9ldZjatr88ys8VmVmdmdfX19Rms+siUF+byT+85nhc37uVfHlwVdjkiIm9aGN1QZSRW35sFTAYKzexjKYf9FpgZTFi4BLitr89y95vcvdbda6uqqjJZ9hG79MTJ/MWZs7j16fXct0xXR4nI6BJGN9T5wDp3r3f3TuAe4K3JB7j7bndvDzZvBhaMcI0Z8XeXzOMtM8u44e5XNH4hIqNKGGGxEVhoZgVmZsB5wCF9M2Y2KWnzstT3R6ucaIQff+RUCvNifOb2pTTpcloRGSXCGLN4FrgLeAF4JajhJjP7upldFhx2XXBp7Uskrpz65EjXmSnVJXF+/JFT2NDQwpd//bLu7haRUcHGyh+r2tpar6urC7uMtP3nU2v55wdX8XcXz+Mv33ZU2OWIyDhlZkvdvXaw43QHd0iuPnsWl5wwkW899CpPv7Er7HJERAaksAiJmfGvHziJWZWFXHfHi2zf1xZ2SSIi/VJYhKgoL8ZPP76Alo5urrl9Ka0d3WGXJCLSJ4VFyI6uLua7HzqJlzbv5fN3vEBXt+7wFpHso7DIAhcdP4mvXXYcv1+1k//zm+W6QkpEsk4s7AIk4RNnzGRnYzs/enwNNSV5fOnCuWGXJCJygMIii/zVhcews6mNHzy2htKCXK46a1bYJYmIAAqLrGJm/Mt7T2BfayffeGAlOxrbuOGieUQiFnZpIjLOacwiy8SCKUE+vnAGNz21ls/+8gXaOnWVlIiES2GRhWLRCF9fdBz/8K5jeWjFdj7802fY1NASdlkiMo4pLLKUmXH12bO58WMLWFvfzLt+8AceWr497LJEZJxSWGS5dx43kQeuO4sZFYVcc/tSPvXz51izU9Obi8jIUliMAjMqCrnrM2fw95fMo279Ht75vT/wf+9bTkNzR9ilicg4obAYJfJiURafcxRPfPlcrjhtGrf/eQNv+7fHuempNzRNiIhknKYoH6Ve29HEvzy4iidW11NZlMs1bzuKj54+g/zcaNilicgoku4U5QqLUe65dQ18/9HX+NOa3QoNEXnTFBbjTGpofOT0GXz4LdOYMiE/7NJEJIspLMap59Y18JMn1vDka/UYcO7caq4+axZnHFVBYslzEZGDsjoszOyLwNWAk1iH+1Pu3pb0fh7wX8ACYDfwYXdfP9BnKiwOtamhhTvrNnHHc5vYtb+dYyeV8L5TpvDukyYzsTQednkikiWyNizMbArwR2C+u7ea2Z3Ag+5+a9Ix1wInuvs1ZnY58F53//BAn6uw6FtbZzf3vLCFX9Vt4qVNezGDM2ZX8J5TpnDR8RMpieeEXaKIhCjbw+LPwElAI3Av8AN3fyTpmIeBr7r7M2YWA7YDVT5AsQqLwa3b1cy9L27h3mVb2LC7hdxYhPPmVbPo5MmcO7eaeI4GxUXGm6wNCwAzux74Z6AVeMTdP5ry/nLgInffHGy/AZzu7rtSjlsMLAaYPn36gg0bNoxE+aOeu7Ns017uW7aVB17eyq79HRTkRjmqqohZlYXMqixk3sRiameWU1WcF3a5IpJBWRsWZlYG3A18GNgL/Bq4y91vTzomrbBIppbF0HR19/DM2t08umonb9TvZ92uZrbsbaX3X4vZlYW8ZWY5p81KPKaW5WugXGQMSTcswljP4nxgnbvXA5jZPcBbgduTjtkCTAM2B91QpSQGumWYxaIRzp5Txdlzqg7sa+vsZtW2Rp5b18Dz6xv43+Xb+FXdJgAmlcYPCY+jq4q03obIOBBGWGwEFppZAYluqPOA1CbB/cCVwDPAB4DHBhqvkOEVz4lyyvQyTplexl++7Sh6epzXdjbx3LoGnlvXwJ/X7ub+l7YCUByPMae6iLkTi1k4u4Izj66kskhdVyJjTVhjFl8j0Q3VBbxI4jLa/wPUufv9ZhYH/hs4BWgALnf3tQN9prqhRo67s2F3C8+tb+ClTXtZs3M/r25vYl9rJwDVxXlMKy9gWlk+08oLmFFRyKnTJzCrslBdWCJZJmvHLDJFYRGu7h5nxdZ9/GnNbtbW72fznlY27Wlh2742unsS/45VFedx0tQJzJ1YxDE1xcydWMzsyiJyY5rPUiQs2TxmIWNQNGKcOHUCJ06dcMj+zu4eNuxu5vn1e3h27W5WbmvkidU76QoCJBYxZlUWcszEYubWFHN0dRFHVxcxs6JQISKSoqfHqd/fzobdLWxsSDw2NbRQUZjLP1w6P6PnVlhIRuVEIxxdXczR1cVccdp0ANq7ulm3q5nV25t4bUcTq7fv55XN+/jdy9sO/FwsYhw3uYRTZ5Qxt6aYzu4eWju7ae3ooa2rm9aObiJmVBXnUVOSR3VxnOqSPGqK45Tkx9TdlUXcndbObva3ddHY1sX+9i72t3Wxv72TpmC7ub2L9q6exKOzm7bOHtq7ug/sa+vsfd1Ne2diX3ePE4lA1IyIGZGIETGImJETjZAbi5ATTbzOi0XIiR585MYi5Abv5cQi5KYcnxscn5t0fCxiRCNGLJI418HtCLGokReLEM+JHvIci775Lzxtnd1samg5JBCSg6G9q+fAsWYwuTSf02eXD+c/sj4pLGTE5cWizJtYwryJJYfsb+noYm19M2/UJ8ZAXtiwhzue20hbZ88hx+VGI8RzInT3OM19rOWRF4tQUxJPhEhJnNzgP1gL/scSr8iJGpVFeVSX5FFdnEdVcZzq4sR2Xkw3KL4Zze1dvLx5Hy9t3suyjXtZt6uZ/e1dNLV1sr+9i540ervNIB6LkpeT+OOeF4sSz0k8J/4AR5iQnxO8HyViRo87Pe509zjuie7Qbne6unvo7HY6untoauuiobuHzt59XT10BNsdXQf3Z0IskgiRgrwYxXkxiuIxivKCRzyxL54bpb6pnY1BOOxsaj/kMwpzo0wrL+CoqkLePreK6eUFTP//7d15mFx1ne/x97equnpfks5C9gQIhIgg0GIioMg2OC7ICFeccVSUYRxFGdfBufc66DijzjBuIzpyFZ3BBTQ4PFF5ABHUYTEkISwmAQkhe0Jn6b27umv53j/O6U7R6VpIUl2V7s/refqpc06dqvr2gdSnf7/fOb/TWs/8qXXMaakdtxa4wkIqRl08xqlzmjl1TjOXhduS6Qx7uhLUVEWpjUepGfXXWt9givaeQdq7E7wYPrb3DPJid4I9XQk27uommQnCxh2yh+gGUxkO9A2O+UXWUldFa32c1obq8DFOa3010xriTK2vZnpjNbOaa5jZVDMhusvSGSeRTNM/lCaRTDMQLg+E6/1DwbagdZdiYCho6e3vHeSpHV08194zchwXtNZx0sxGmmuraKiO0Vhz8Mvx4PpLn6urjhKPRsrWInT3kXBJhgEyNBImQcCk3UlnMqTSQTilwmBKp51k+qUtoOzHRDLDQDI10orqTaTY1tcfLA+m6B9M09oQZ/7UOl5/0nAY1AWPU+uYWh+viJaywkIqWlU0wrypdTmfr6+Osag6xqJp9Yf1/umMs79vkPbuQfb2DNLek6C9e5D2nkEO9A2xr3eQ59p7WfXCEB39Q4w+H8QMpjVUM72hmpa6KlrqqmiuraK5Nh6s1w5vi9McLrfUVVFbFS35F0AynWHDrm7Wbu3gyR2dHOgbYmD4Sz/ry79/KM1QKlP4DUeJxyI01cR4xexmLj31OF41v4XT57YwtT5egt+mtMyMeMyC4NeZ32NSWMikFo1YMN7RWHgm3lQ6Q0d/ciRc9nQl2NU1wO7OBPv7BukaSPLci710DiTp6k8ylM79BUxxy4sAACAASURBVByPRmgOg2U4UGrjsZG+8+pYhOqqSNB1UR2joSb4S7y5toqp9XGm1seZUldFLBohmc6ws2NgpF97y76+kS6h4f7tWc01HNdcQ21VlJa6qqClVhWlLh6lJn5wubYqSk1VlLp4jNp45OByuH9tfOwWnkx8CguRIsWiEaY3Bl1QS47Lv+/woG7XQJLO/uCna2AoWB5Ijmwf3rarM8FAMvgLP3sgN1/gQHBRZN+oMYF4LMLSWU38xWsWcNaCKZy5oIVZzboJlhwZhYVICZgZdfEYdfHYEX1RD6bS9A0On0mUpDuR5EDfEB19Q+wPH5tqq5gX9m8vaK1jZmONpmCRo05hIVLBgjOBosfkOIBMLOp0FBGRghQWIiJSkMJCREQKUliIiEhBCgsRESlIYSEiIgUpLEREpKAJc/MjM9sLbD2Ct5gG7DtK5UxEOj656djkp+OTX7mPzwJ3n15opwkTFkfKzNYUc7eoyUrHJzcdm/x0fPI7Vo6PuqFERKQghYWIiBSksDjolnIXUOF0fHLTsclPxye/Y+L4aMxCREQKUstCREQKmvRhYWaXmtmzZrbJzG4odz2VxMzmmdmDZrbBzNab2fXlrqkSmVnUzNaZ2S/KXUulMbMWM1thZs+Y2UYzW17umiqJmX00/Lf1BzP7sZkVvmVjmUzqsDCzKHAz8EZgKfBOM1ta3qoqSgr4uLsvBZYBH9LxGdP1wMZyF1Ghvgbc4+5LgNPRcRphZnOAjwBt7n4qEAWuKm9VuU3qsADOBja5+2Z3HwJuBy4rc00Vw913u/vj4XIPwT/0OeWtqrKY2VzgTcB3yl1LpTGzZuB1wHcB3H3I3TvLW1XFiQG1ZhYD6oBdZa4np8keFnOA7VnrO9CX4ZjMbCFwBrCqvJVUnK8CnwLy3yx7cloE7AW+F3bTfcfM6stdVKVw953ATcA2YDfQ5e73lbeq3CZ7WEgRzKwBuBP4W3fvLnc9lcLM3gy0u/vactdSoWLAmcC33P0MoA/QuGDIzKYQ9GQsAmYD9Wb2rvJWldtkD4udwLys9bnhNgmZWRVBUPzQ3X9W7noqzDnAW81sC0EX5gVm9oPyllRRdgA73H24NbqCIDwkcBHwgrvvdfck8DPgtWWuKafJHhargcVmtsjM4gSDSyvLXFPFMDMj6G/e6O5fLnc9lcbdP+3uc919IcH/Ow+4e8X+ZTje3H0PsN3MTg43XQhsKGNJlWYbsMzM6sJ/axdSwScAxMpdQDm5e8rMrgPuJTgT4VZ3X1/msirJOcBfAk+b2RPhtr9397vLWJMcWz4M/DD8Y2wzcHWZ66kY7r7KzFYAjxOcebiOCr6aW1dwi4hIQZO9G0pERIqgsBARkYIUFiIiUpDCQkREClJYiIhIQQoLmTTMrHccPuOt4z17sZmdb2YVezGXTAyT+joLkcNhZlF3T4/1nLuvpAQXdppZzN1TOZ4+H+gFHjnanysyTC0LmZTM7JNmttrMnjKzz2Ztv8vM1ob3GLg2a3uvmf2bmT0JLDezLWb2WTN73MyeNrMl4X7vNbNvhMvfN7Ovm9kjZrbZzK4It0fM7JvhPR5+ZWZ3Dz83qsbfmNlXzWwNcL2ZvcXMVoWT8t1vZjPDCR4/AHzUzJ4ws/PMbLqZ3Rn+fqvN7JxSHkuZHNSykEnHzC4BFhNMUW/ASjN7nbv/Dnifux8ws1pgtZnd6e77gXpglbt/PHwPgH3ufqaZfRD4BHDNGB83CzgXWELQ4lgB/BmwkOAeKjMIpni4NUe5cXdvCz9zCrDM3d3MrgE+5e4fN7P/AHrd/aZwvx8BX3H3h8xsPsEMBacc9gETQWEhk9Ml4c+6cL2BIDx+B3zEzC4Pt88Lt+8H0gQTKmYbnlhxLUEAjOUud88AG8xsZrjtXOCn4fY9ZvZgnlrvyFqeC9xhZrOAOPBCjtdcBCwNAw2gycwa3L3kYzYycSksZDIy4Avu/u2XbDQ7n+CLdrm795vZb4Dh21wmxhinGAwf0+T+tzSYtWw59smnL2v534Evu/vKsNYbc7wmQtACSRzG54mMSWMWMhndC7wvvE8HZjbHzGYAzUBHGBRLCG4lWwoPA28Pxy5mEgxQF6OZg1Povydrew/QmLV+H8EEfgCY2asOv1SRgMJCJp3wbmQ/Ah41s6cJxhEagXuAmJltBL4I/L5EJdxJcK+HDcAPCGYd7SridTcCPzWztcC+rO0/By4fHuAmvK9zOHi/gWAAXOSIaNZZkTIYHkMws1bgMeCc8P4PIhVJYxYi5fELM2shGKj+RwWFVDq1LEREpCCNWYiISEEKCxERKUhhISIiBSksRESkIIWFiIgUpLAQEZGCJsx1FtOmTfOFCxeWuwwRkWPK2rVr97n79EL7TZiwWLhwIWvWrCl3GSIixxQz21rMfuqGEhGRghQWIiJSkMJCREQKUliIiEhBCgsRESlIYVFhUukMmglYRCqNwmKcZDLO0zu66EkkX7L9QN8Qv3m2nQN9Q/Qkkrz53x/i7d96hF2dA6QzCg0RqQwT5jqLSpNMZ9i4u5vO/iSLZzbwsTue5NHN+6mtitJQE6MuHuWU45p4bMsBDvQNUVMVYXZLLVv39xMxeO0XHyBiMLW+mouXzuSf3nYqkYiV+9cSkUlKYXEUPbm9k3vX7+GR5/ezq3OA9p7BkeciBh++4ES6BpL0DabpG0zxzJ5uzpzfwlWvns8vntrFrs4EH73oJBZNq2fNlgPs6x1i875efvzYNhqqo1z3hsU011Xx8KZ9fP6XG3nF7Ca+9PbTiOYIkWQ6w/pd3Zw2p1lBIyJHZMLcKa+trc3LcQX3YCqNYdyzfg8fu+MJUhnnzPktALzj1fOY3VLL0zu7WH58K2fMn/Ky39/d+eSKp1ixdgdVUWP+1Dqe39tHa32c/X1DXLx0JqfMauLnT+6ibzBFXTxKxuG45hq27OujvWeQvzn/BP7u0iUved+hVIbVWw7wxPZO/vSVs1g0rf6oHA8RObaY2Vp3byu4n8Li8Px644vc9cQu7l2/h6FUBoDFMxq45d1tR/2L1915akcXv3x6N09s7+SNpx7HO8+ez7d/u5n/+O3zDCTTLDt+Kgtb6+kfSgOwq3OAlroqYpEI96zfw/UXLiYWMXoGU2w/0M//PLeP3sEUANMaqrn92tdw4oxGIBhkj5ipNSIyCSgsSuQPO7t45Pl9/PPdzwDQWB3j/CUzOHvRVP787Pk5u4RKJZNxehIpmuuqxnx+MJXmqlt+z7ptnQDUVEVora/mdSdN44IlMzmuqYarv7+aRDJNY02M7oEkfUNpFrbW8f/e3cbimY3j+euIyDhTWJRAe0+Cs//p1wAc11TDbe8/m/mtdVTHoiX93COVzji7OgeY0VQ9Zq3P7+3l5gc3ETWjqbaK+uoYP35sG4lkmk9duoT5U+tIpTPs7Bxg0bR6zlt86ASVPYkkA0Np0u4c11SDmVolIscChcVRtqm9h0/89Cme2N7JW06fzT9ffiqNNWP/NT8R7Owc4P3fX80ze3oOee6Ks+Zy/YWLae9JsG5bJz9/ajdPbu8cef68xdP42lVnMLU+nvP9U+mg687MiBgKF5EyqYiwMLNLga8BUeA77v7FUc9XA/8FnAXsB97h7luynp8PbABudPeb8n1WKcMikUxzymfuwR3+4jXz+afLX1mSz6k0qXSGHR0D7OsdJBIxZjXX8KNV27j5wU1kXwLyyjnNXLx0JlPr4+zvHeLmBzfR2hDng284kaFUhu6BJN2JJN0DKXZ1DvDCvj72dCcO+bxYxHjzabP49J+ewsymmnH8TUUmr7KHhZlFgT8CFwM7gNXAO919Q9Y+HwROc/cPmNlVwOXu/o6s51cADqwqZ1h8asWT/GTNDuZOqeVnf/NaZkzyL7KndnSyblsn86fWsXhmA3On1L3k+ad3dPHBH61l+4GBkW2N1TEaa2Ic11zDwmn1zJ1SRyxiZNxxD/4j7+8d5KdrgrO+rr9oMe9atoC6uM7uFimlSgiL5QQtgj8J1z8N4O5fyNrn3nCfR80sBuwBpru7m9nbgHOAPqC3XGGxu2uA5V94gHcvX8Bn3/oKdZcUaTCVpr17kKaaKhpqYkUP/G/Z18dnf76eB5/dS2NNjCvOmsuVZ83jhBn1I+Mtw2Mw1VURZjRO7uAWOVLFhkUp/2ybA2zPWt8BvCbXPu6eMrMuoNXMEsDfEbRKPlHCGgv69cZ2AP5y2QIFxctQHYsyb2pd4R1HWTitnlvf+2rWbu3gvx7dyg9+v5XvPbwFs+CkgtqqKNs7+kmmnXg0wkcuPJG/fv0JVEU1c41IKVVqG/9G4Cvu3pvvC9rMrgWuBZg/f35JCrl/44vMn1rHiTMaSvL+cigzo23hVNoWTqW95xQe3rSPbfsH2Hagn4FkiktecRwLWut4aNM+brrvj9z99B4+eenJuDud/Un6BlNUV0WprYpSU3WwNeLupN3D5WBb2oPtGYeTZjZw5vwp+qNAZAylDIudwLys9bnhtrH22RF2QzUTDHS/BrjCzP4FaAEyZpZw929kv9jdbwFugaAb6mj/Ajs6+vntH/fyvnMW6QukTGY01nD5GXPHfO6dZ8/nLaft5v/ctZ6rv7f6qHzeomn1XHHWXC4/Yw6zW2qPynuKTASlDIvVwGIzW0QQClcBfz5qn5XAe4BHgSuABzwYRDlveAczu5FgzOIbjLPfPLsXd7iybewvKym/S0+dxfLjp/HEjk6aamK01MWpr44ylMqQSKYZGMpgBmYQjVhwZbpZuMzIsgOPbNrHirU7+Nd7n+Wm+57lgpNn8K5lC3jdSdNzjrm4Ox39SeriB1sxIhNRycIiHIO4DriX4NTZW919vZl9Dljj7iuB7wK3mdkm4ABBoFSM+ze+yKJp9Zysq5grWnNdFa8/6dALBV+uK9vmcWXbPLbt7+cna7Zz++rt/Pr7q5k3tZZLlh5HOuP0DqboH0pxoG+I3V0JdnclGEplaKmr4ppzF/Hu1y6kaQJffyOTly7KG+Xe9XsYGErztjPm0Pb5+zn/5OncdOXpR6FCOdYMpTLct2EPtz26lce3dVBbFaW+OkZ9dYyW2ipmtdQyq7mGmU01PLJpH79+pp2mmhjvO3cRV792Uc4pWEpV63PtPWzY1c2mvb0sP76V1580Xd2nUlAlnA11TPrr29YCsOz4Vvb1DnLKrKYyVyTlEo9FePNps3nzabML7vv+cxfxh51dfP3Xz/HV+5/j1ode4APnn8DVr11Ebfzod0/t6Urw6OZ9/P75Azy1s4tN7T0k08Effmbw7d9u5oz5LXz0opM4b/E0hYYcMYVFDvdt2APAWQte/rTiMjmdOqeZW97dxvpdXfzbfX/kX+55lu8/vIXrL1rM/2qbR08iuIJ9T1eC3V0D7OpKsLszeHyxO8G0hmqWHT+VZce3ctaCKdTFY7g7+/uGeGFfHy/s6+OJ7Z38/vn9bN7XB0BzbRWvmtfC+SdPZ+msJpbObmJOSy13Pr6Dmx/YxLtvfYy2BVP424tO4pwTW0saGu7OYCpD32CKroEke3sGae8ZZG/PIHt7B2nvHiQagdPntXDGvCmcNLOB2GGe8jzcI1Lq36ejP8mL3Qn29gwyZ0otx0+rL2vwDl+/FBzXBO09g7zYnWBKXZxrzju+pJ+tbqhRFt7wSwAuOmUGz77Yw/986oIjfk+ZnB574QD/cs8zrNnaQTRih9wmtypqzGyqYXZzLTOba9jR0c9TO7pIZ5yqqLGwtZ49XQl6wqnkARqqY5y9aCrLj29l+QmtnDKrKefg+2AqzU/WBKGxpzvBwtY6LjxlJhcumUHbwqnEYy/vizqTcbYe6OeZ3d1s3N3Nht09bNnfR28iRd9Qiv6hdM5bAVdFjekN1SRSGQ70DQFQF49y2txmXjVvCicf10DEDHdeclV/OpNhX+9QGDwJ2rsHeTF8rIpGOGF6PSfMaOCE6Q2cGD4uaK2jKhrB3UmmnVQmQzLlJDOZYPqZRJKu/iTdiSDURn76h9jbO8iL3YPs6QoCYiicw2zYlLoqzlowhbMWTKVt4RReOaf5qJzY0D+UGgmB9p4EL3YHj3tHbesaSB7y2mjEWH58Kz+4ZvRlbMUp+xXc4+1ohIW7s+jTdwPBVN5vOW02/6rxCjkC7s6Dz7az6oUDzGysYXZLDcc11zK7uYZpDdWH3DOkdzDFmi0H+P3mAzz3Yg9zptSyaFo9C6fVc/y0eua01L7sv8YTyTT/vW7nyF0ch1IZGqtjvO6k6cydWktPIkX3QJKeRIqeRHLknigv/T0Ir3MJnotYcJrxiTMaaK6toi4eo6E6Rl11lPp4MLXLjMYaZjRVM72hmubaKiIRw93ZfmCAdds7WLetk3XbOli/q5tUgfvNB+9XPfKeMxqrGUpl2LS3l+fbXzrX2PBZboXeM9dnzGyqyfoJ1lvr42zZ38eaLR2s3dox0rKrihqnzmnmhOkNI2fgJcLHwWSaRDJDIpUmlQ6u78l48JPODK9DKpMhkcwcUk88GmF6Y/XI7zujsWakvulZ26bWx4/o1ggKi8NwoG+IM//xVyPrn3/bqbxr2YIjLU2kYvQPpXh4034eeOZFHnimnY7+JE01MRprqkYea+NRxvrqmd1Sy9JZTSyZ1chJMxuP2qnCiWSanZ3BPGIRM4xg3MUwIhFora8uOO7TO5ji+fZent/bywv7+si4E4tEiMcixCJGVTRCVSxCPGo01lTRXFtFU/jYXPvypqSBYB6zx7d1smbrAdZu6WBn5wA1VVGqYxGqq6LUxCLUVEWpqQoeY5EI0cjB07ezH6MRY0pdPPjybwoCYGZTELDj0eWlAe7DsLNj4CXrp89tKVMlIqVRF49x8dKZXLx0ZrlLGVFTFeWE6Uc2Q0JDdYzT57Vw+rzx+Tfb2lBdccex1DShTpadnf0jy/FohJOO0xQfIiKgsHiJ9p7BkeUlsxor/g54IiLjRWGRpaPv4JkGr5zTXMZKREQqi8YssnT0D9FYE+N//+kpLDu+tdzliIhUDIVFlo7+IabUxbnq7NJMdy4icqxSN1SWjv4kU8ZxPh8RkWOFwiJLZ/8QLXXxcpchIlJxFBZZOvqHmFqvsBARGU1hkaWzL0mLuqFERA6hsAgNpTL0DKaYom4oEZFDKCxCnQPBTJga4BYROZTCItTZH1yQpwFuEZFDKSxCHeEc+xrgFhE5lMIi1NEfhIUGuEVEDqWwCHWE3VAa4BYROZTCIjTcslBYiIgcSmER6uxPUh2LFLwjl4jIZKSwCHX0DalVISKSg8Ii1JNI0VijSXhFRMaisAgNJNPUqQtKRGRMCovQQDJNTZXCQkRkLAqLUCKZ1uC2iEgOCotQ/5C6oUREclFYhAaG1A0lIpKLwiKUSKapVViIiIxJYRHS2VAiIrmVNCzM7FIze9bMNpnZDWM8X21md4TPrzKzheH2s83sifDnSTO7vJR1ujsDalmIiORUsrAwsyhwM/BGYCnwTjNbOmq39wMd7n4i8BXgS+H2PwBt7v4q4FLg22ZWsivmBlMZ3KFGLQsRkTGVsmVxNrDJ3Te7+xBwO3DZqH0uA/4zXF4BXGhm5u797p4Kt9cAXsI6GRhKA6hlISKSQynDYg6wPWt9R7htzH3CcOgCWgHM7DVmth54GvhAVngcdQPJICx0NpSIyNgqdoDb3Ve5+yuAVwOfNrOa0fuY2bVmtsbM1uzdu/ewPyuZzgAQj1bs4RARKatSfjvuBOZlrc8Nt425Tzgm0Qzsz97B3TcCvcCpoz/A3W9x9zZ3b5s+ffphF5pMB71csagd9nuIiExkpQyL1cBiM1tkZnHgKmDlqH1WAu8Jl68AHnB3D18TAzCzBcASYEupClXLQkQkv5KdYeTuKTO7DrgXiAK3uvt6M/scsMbdVwLfBW4zs03AAYJAATgXuMHMkkAG+KC77ytVramRloXCQkRkLCW9gYO73w3cPWrbZ7KWE8CVY7zuNuC2UtaWbShsWVSpG0pEZEz6UxpIjYSFDoeIyFj07cjBAW6FhYjI2PTtCCQzQctCZ0OJiIxNYQEkUzobSkQkH307AqmMrrMQEclHYcHB6yw0ZiEiMjZ9O5I1wB3R4RARGYu+Hck6dTambigRkbEoLDjYDRVTy0JEZEz6diRrgDuiloWIyFiKCgsz+5mZvcnMJmS4hFlBRGEhIjKmYr/8vwn8OfCcmX3RzE4uYU3jLhOmhbJCRGRsRYWFu9/v7n8BnEkwVfj9ZvaImV1tZlWlLHA8ZDwIi6jSQkRkTEV3K5lZK/Be4BpgHfA1gvD4VUkqG0dpH25ZKCxERMZS1BTlZvbfwMkE04a/xd13h0/dYWZrSlXcePHhMQuFhYjImIq9n8XX3f3BsZ5w97ajWE9ZpDVmISKSV7HdUEvNrGV4xcymmNkHS1TTuNOYhYhIfsWGxV+5e+fwirt3AH9VmpLG3/DZUKZuKBGRMRUbFlHL+iY1sygQL01J4y/jalWIiORT7JjFPQSD2d8O1/863DYhpN01XiEikkexYfF3BAHxN+H6r4DvlKSiMsi460woEZE8igoLd88A3wp/JpxMRmEhIpJPsddZLAa+ACwFaoa3u/vxJaprXGnMQkQkv2IHuL9H0KpIAW8A/gv4QamKGm/pjKOGhYhIbsWGRa27/xowd9/q7jcCbypdWeMr466WhYhIHsUOcA+G05M/Z2bXATuBhtKVNb40wC0ikl+xLYvrgTrgI8BZwLuA95SqqPGWzmheKBGRfAq2LMIL8N7h7p8AeoGrS17VOHNdZyEiklfBloW7p4Fzx6GWsklnNGYhIpJPsWMW68xsJfBToG94o7v/rCRVjbOMqxtKRCSfYsOiBtgPXJC1zYEJEhZOZELeXVxE5Ogo9gruCTdOkU1nQ4mI5FfsFdzfI2hJvIS7v6/A6y4luP1qFPiOu39x1PPVBBf4nUXQcnmHu28xs4uBLxLMbDsEfNLdHyim1sORzjhRhYWISE7FdkP9Imu5Brgc2JXvBeFZVDcDFwM7gNVmttLdN2Tt9n6gw91PNLOrgC8B7wD2Edy+dZeZnQrcC8wpstaXzR1dwS0ikkex3VB3Zq+b2Y+Bhwq87Gxgk7tvDl9zO3AZkB0WlwE3hssrgG+Ymbn7uqx91gO1Zlbt7oPF1Pty6WwoEZH8DndYdzEwo8A+c4DtWes7OLR1MLKPu6eALqB11D5vBx4vVVCAxixERAopdsyih5eOWewhuMdFSZnZKwi6pi7J8fy1wLUA8+fPP+zPUViIiORXbDdU42G8905gXtb63HDbWPvsMLMY0Eww0I2ZzQX+G3i3uz+fo65bgFsA2traDhmAL1bG0amzIiJ5FPUVaWaXm1lz1nqLmb2twMtWA4vNbJGZxYGrgJWj9lnJwTmmrgAecHc3sxbgl8AN7v5wMTUeCVfLQkQkr2L/nv4Hd+8aXnH3TuAf8r0gHIO4juBMpo3AT9x9vZl9zszeGu72XaDVzDYBHwNuCLdfB5wIfMbMngh/Co2RHLbDbpKIiEwSxZ46O1aoFHytu98N3D1q22eylhPAlWO87vPA54us7Yi5g9oVIiK5FduyWGNmXzazE8KfLwNrS1nYeHLQhRYiInkUGxYfJriS+g7gdiABfKhURY03d1fLQkQkj2LPhurj4HjChKSGhYhIbsWeDfWr8Ayl4fUpZnZv6coSEZFKUmw31LTwDCgA3L2DwldwHzM0wC0ikl+xYZExs5FLpM1sIRPojFPHMfVDiYjkVOyps/8beMjMfkvwR/h5hNNsTARqWYiI5FfsAPc9ZtZGEBDrgLuAgVIWNt7UsBARya3YiQSvAa4nmN/pCWAZ8Cgvvc3qMcsnTIeaiEhpFDtmcT3wamCru78BOAPozP+SY4fjmDqiRERyKjYsEuHUHIQ3IXoGOLl0ZY0vdzRoISKSR7ED3DvC6yzuAn5lZh3A1tKVNf6UFSIiuRU7wH15uHijmT1IcN+Je0pW1ThTw0JEJL9iWxYj3P23pSikrBxMNz8SEclJX5FogFtEpBCFBeFFecoKEZGcFBYhhYWISG4KCybQJFciIiWisGD45kdqWoiI5KKwIDx1VlkhIpKTwgLNDSUiUojCQkREClJYMNwNpX4oEZFcFBYA7hreFhHJQ2GBBrhFRApRWISUFSIiuSks0NlQIiKFKCwIJxJUP5SISE4KC8KJBMtdhIhIBVNYoFlnRUQKUViMUFqIiOSisECzzoqIFFLSsDCzS83sWTPbZGY3jPF8tZndET6/yswWhttbzexBM+s1s2+UskYIZ51Vw0JEJKeShYWZRYGbgTcCS4F3mtnSUbu9H+hw9xOBrwBfCrcngP8LfKJU9Y2mrBARya2ULYuzgU3uvtndh4DbgctG7XMZ8J/h8grgQjMzd+9z94cIQmNcqGUhIpJbKcNiDrA9a31HuG3Mfdw9BXQBrcV+gJlda2ZrzGzN3r17D7tQXZQnIpLfMT3A7e63uHubu7dNnz798N8H3SlPRCSfUobFTmBe1vrccNuY+5hZDGgG9pewpjHpOgsRkfxKGRargcVmtsjM4sBVwMpR+6wE3hMuXwE84D7+nUKadVZEJL9Yqd7Y3VNmdh1wLxAFbnX39Wb2OWCNu68EvgvcZmabgAMEgQKAmW0BmoC4mb0NuMTdN5SqXnVDiYjkVrKwAHD3u4G7R237TNZyArgyx2sXlrK2UZ81Xh8lInJMOqYHuI8WB11oISKSh8ICQLPOiojkpbBgeIBbcSEikovCIqSoEBHJTWGBBrhFRApRWKDrLEREClFYoNuqiogUorAIaYBbRCQ3hQXBRIIiIpKbwgJ1Q4mIFKKwILyfhdJCRCQnhUVIEwmKiOSmsAhpfFtEJDeFBbooT0SkEIUF4UV55S5CRKSCKSzQbVVFRApRWBBcZ6EBbhGR3BQWIbUsRERyU1gQXmchIiI5KSzQrLMiIoUoLBhuWSgtRERyTxRAXAAAB1ZJREFUUViE1LIQEclNYQGgWWdFRPJSWKBZZ0VEClFYhNQNJSKSm8ICdUKJiBSisAjpCm4RkdwUFmjWWRGRQhQWIY1ZiIjkprBAYxYiIoUoLEJqWIiI5FbSsDCzS83sWTPbZGY3jPF8tZndET6/yswWZj336XD7s2b2J6WsU0MWIiL5lSwszCwK3Ay8EVgKvNPMlo7a7f1Ah7ufCHwF+FL42qXAVcArgEuBb4bvVzKmQQsRkZxK2bI4G9jk7pvdfQi4Hbhs1D6XAf8ZLq8ALrTgW/sy4HZ3H3T3F4BN4fuVhM6GEhHJr5RhMQfYnrW+I9w25j7ungK6gNYiX3tUdPUn6U6kSvHWIiITxjE9wG1m15rZGjNbs3fv3sN8EzhxRgPLT2g9usWJiEwgsRK+905gXtb63HDbWPvsMLMY0AzsL/K1uPstwC0AbW1th9WX1Fxbxf0fe/3hvFREZNIoZctiNbDYzBaZWZxgwHrlqH1WAu8Jl68AHvBgAGElcFV4ttQiYDHwWAlrFRGRPErWsnD3lJldB9wLRIFb3X29mX0OWOPuK4HvAreZ2SbgAEGgEO73E2ADkAI+5O7pUtUqIiL52UQ5E6itrc3XrFlT7jJERI4pZrbW3dsK7XdMD3CLiMj4UFiIiEhBCgsRESlIYSEiIgVNmAFuM9sLbD2Ct5gG7DtK5UxEOj656djkp+OTX7mPzwJ3n15opwkTFkfKzNYUc0bAZKXjk5uOTX46PvkdK8dH3VAiIlKQwkJERApSWBx0S7kLqHA6Prnp2OSn45PfMXF8NGYhIiIFqWUhIiIFTfqwKHSf8MnMzOaZ2YNmtsHM1pvZ9eWuqRKZWdTM1pnZL8pdS6UxsxYzW2Fmz5jZRjNbXu6aKomZfTT8t/UHM/uxmdWUu6ZcJnVYFHmf8MksBXzc3ZcCy4AP6fiM6XpgY7mLqFBfA+5x9yXA6eg4jTCzOcBHgDZ3P5Vgdu6ryltVbpM6LCjuPuGTlrvvdvfHw+Uegn/oJbm97bHKzOYCbwK+U+5aKo2ZNQOvI7gVAe4+5O6d5a2q4sSA2vDmb3XArjLXk9NkD4txu9f3sc7MFgJnAKvKW0nF+SrwKSBT7kIq0CJgL/C9sJvuO2ZWX+6iKoW77wRuArYBu4Eud7+vvFXlNtnDQopgZg3AncDfunt3ueupFGb2ZqDd3deWu5YKFQPOBL7l7mcAfYDGBUNmNoWgJ2MRMBuoN7N3lbeq3CZ7WBR1r+/JzMyqCILih+7+s3LXU2HOAd5qZlsIujAvMLMflLekirID2OHuw63RFQThIYGLgBfcfa+7J4GfAa8tc005TfawKOY+4ZOWmRlBf/NGd/9yueupNO7+aXef6+4LCf7fecDdK/Yvw/Hm7nuA7WZ2crjpQoJbJUtgG7DMzOrCf2sXUsEnAJTsHtzHglz3CS9zWZXkHOAvgafN7Ilw29+7+91lrEmOLR8Gfhj+MbYZuLrM9VQMd19lZiuAxwnOPFxHBV/NrSu4RUSkoMneDSUiIkVQWIiISEEKCxERKUhhISIiBSksRESkIIWFTBpm1jsOn/HW8Z692MzON7OKvZhLJoZJfZ2FyOEws6i7p8d6zt1XUoILO80s5u6pHE+fD/QCjxztzxUZppaFTEpm9kkzW21mT5nZZ7O232Vma8N7DFybtb3XzP7NzJ4ElpvZFjP7rJk9bmZPm9mScL/3mtk3wuXvm9nXzewRM9tsZleE2yNm9s3wHg+/MrO7h58bVeNvzOyrZrYGuN7M3mJmq8JJ+e43s5nhBI8fAD5qZk+Y2XlmNt3M7gx/v9Vmdk4pj6VMDmpZyKRjZpcAiwmmqDdgpZm9zt1/B7zP3Q+YWS2w2szudPf9QD2wyt0/Hr4HwD53P9PMPgh8ArhmjI+bBZwLLCFocawA/gxYSHAPlRkEUzzcmqPcuLu3hZ85BVjm7m5m1wCfcvePm9l/AL3uflO434+Ar7j7Q2Y2n2CGglMO+4CJoLCQyemS8GdduN5AEB6/Az5iZpeH2+eF2/cDaYIJFbMNT6y4liAAxnKXu2eADWY2M9x2LvDTcPseM3swT613ZC3PBe4ws1lAHHghx2suApaGgQbQZGYN7l7yMRuZuBQWMhkZ8AV3//ZLNpqdT/BFu9zd+83sN8DwbS4TY4xTDIaPaXL/WxrMWrYc++TTl7X878CX3X1lWOuNOV4TIWiBJA7j80TGpDELmYzuBd4X3qcDM5tjZjOAZqAjDIolBLeSLYWHgbeHYxczCQaoi9HMwSn035O1vQdozFq/j2ACPwDM7FWHX6pIQGEhk054N7IfAY+a2dME4wiNwD1AzMw2Al8Efl+iEu4kuNfDBuAHBLOOdhXxuhuBn5rZWmBf1vafA5cPD3AT3tc5HLzfQDAALnJENOusSBkMjyGYWSvwGHBOeP8HkYqkMQuR8viFmbUQDFT/o4JCKp1aFiIiUpDGLEREpCCFhYiIFKSwEBGRghQWIiJSkMJCREQKUliIiEhB/x9UcpkbX7r8kwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x864 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set learning rate\n",
    "lr = 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f505918b0b34cb18bb9bbefb96e4d3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                       \n",
      "    0      4.114716   3.936571   0.367859  \n",
      "    1      3.83864    3.711561   0.382893                       \n",
      "    2      3.669321   3.603781   0.391633                       \n",
      "    3      3.63252    3.560706   0.394518                       \n",
      "    4      3.478959   3.513905   0.399009                       \n",
      "    5      3.518267   3.480469   0.401523                       \n",
      "    6      3.409158   3.465206   0.402808                       \n",
      "    7      3.426483   3.437133   0.405097                       \n",
      "    8      3.296175   3.409095   0.409595                       \n",
      "    9      3.185208   3.377671   0.413643                       \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([3.37767]), 0.41364255045710613]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.fit(lr, 1, wds=wd, cycle_len=10, use_clr=(10,33,0.95,0.85), best_save_name='best_lm_malay_1cycle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize loss history and learning rate history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VOW9x/HPL5OdLSxh3xdBUBGMC6AIAhbBqqWb2lptbWmtrbbe1rq02lqt1Nv2WqtX5bbVti51r61YRCmIioIB2VRWCTskLAlhyzbP/WNOhiRkMklmspzx+3698sqZMzPn/A6O33nyPM85x5xziIiI/yW1dAEiIhIfCnQRkQShQBcRSRAKdBGRBKFAFxFJEAp0EZEEETXQzezPZpZvZmuqrPuimX1oZkEzy2naEkVEpD7q00J/HJhaY90aYAawKN4FiYhI4yRHe4FzbpGZ9a+x7mMAM2uaqkREpMGiBno8denSxfXv3785dyki4nvLli3b65zLjva6Jg90M5sJzATo27cvubm5Tb1LEZGEYmZb6vO6Jp/l4pyb7ZzLcc7lZGdH/YIREZFG0rRFEZEEUZ9pi08D7wJDzWy7mV1rZp8zs+3AGGCOmb3W1IWKiEjd6jPL5YoIT70U51pERCQG6nIREUkQCnQRkQShQBcRSRC+CPRt+4/w5vqCli5DRKRVa9YzRRtr4m8WUh505M2a3tKliIi0Wr5ooZcHdSNrEZFofBHoIiISnQJdRCRB+CLQZ4zu1dIliIi0er4I9I6ZqbRJDbR0GSIirZovAt0ADYuKiNTNH4GuGyOJiETli0AHcGqii4jUyReBbmY4dbqIiNTJH4GOWugiItH4ItBFRCQ6fwS6aZaLiEg09bkF3Z/NLN/M1lRZ18nMXjezDd7vjk1apCnRRUSiqU8L/XFgao11twDznXNDgPne4yZjQFCd6CIidYoa6M65RcD+GqsvBf7iLf8FuCzOdVWjBrqISHSN7UPv5pzb5S3vBrrFqZ5aJZnh1EIXEalTzIOiLpS0EdPWzGaaWa6Z5RYUNO6uQ6Eul0YWKCLyKdHYQN9jZj0AvN/5kV7onJvtnMtxzuVkZ2c3amfmnfuvVrqISGSNDfR/Ald7y1cDL8ennNpVXstFeS4iEll9pi0+DbwLDDWz7WZ2LTALmGJmG4DJ3uMmY3gt9KbciYiIz0W9SbRz7ooIT02Kcy0RJYVb6A7QpRdFRGrjizNFK7tcNDAqIhKZTwK9sstFiS4iEolPAj30W4OiIiKR+SPQKwdFFegiIhH5I9ArW+jqchERicgXgZ6kLhcRkah8EeiVXS664qKISGT+CPRwl4uIiETik0DXoKiISDT+CHTvty7OJSISmS8CXYOiIiLR+SLQK7tcNCgqIhKZTwI99FtxLiISmU8CXYOiIiLR+CPQvd8aFBURicwfga4uFxGRqHwR6EnqchERiSqmQDezG81sjZl9aGY/iFdRJ+zH+61ZLiIikTU60M3sFOBbwFnASOBiMxscr8Kq7yv0W3EuIhJZLC30k4Elzrkjzrly4E1gRnzKqu74LBdFuohIJLEE+hrgPDPrbGaZwDSgT80XmdlMM8s1s9yCgoJG7ej4LJdG1yoikvAaHejOuY+BXwPzgLnACqCiltfNds7lOOdysrOzG7UvzUMXEYkupkFR59yfnHNnOOfGAweA9fEpq7rKa7loUFREJLLkWN5sZl2dc/lm1pdQ//k58Smr5n5CvxXnIiKRxRTowAtm1hkoA653zhXGoaYTJGlQVEQkqpgC3Tl3XrwKqY+g8lxEJCJfnClaOSiqThcRkch8Eei6wYWISHS+CHSj8gYXLVyIiEgr5o9AD89yUaKLiETii0BXl4uISHS+CHTQPUVFRKLxRaCbWugiIlH5ItCTwtMWRUQkEl8Eum5wISISnS8CPcmrUnkuIhKZLwLdNCgqIhKVLwIdXW1RRCQqXwS6rrYoIhKdLwI9YDr1X0QkGn8EuneqaHmFEl1EJBJfBHpyIBToFWqii4hE5ItAD7fQg8EWrkREpPWKKdDN7Idm9qGZrTGzp80sPV6FVZWcpBa6iEg0jQ50M+sF3ADkOOdOAQLA5fEqrKrjLXQFuohIJLF2uSQDGWaWDGQCO2MvqZadeKeKqoUuIhJZowPdObcD+A2wFdgFFDnn5tV8nZnNNLNcM8stKCho1L7UQhcRiS6WLpeOwKXAAKAn0MbMvlrzdc652c65HOdcTnZ2dqP2dbwPXYOiIiKRxNLlMhnY7JwrcM6VAS8CY+NTVnWahy4iEl0sgb4VOMfMMs3MgEnAx/EpqzrNQxcRiS6WPvQlwPPAcmC1t63ZcaqrGvWhi4hElxzLm51zdwJ3xqmWiDTLRUQkOp+dKapAFxGJxBeBrlkuIiLR+SLQK1voZZrlIiISkS8CPSUQKlPTFkVEIvNFoAeSjJSAUVJe0dKliIi0Wr4IdIDUQBIl5epDFxGJxDeBnpYSUAtdRKQO/gn05CRKytRCFxGJxF+Bri4XEZGIfBToAUoV6CIiEfkn0FOS1IcuIlIH3wS6ZrmIiNTNN4EeaqEr0EVEIvFPoCdr2qKISF18FOiatigiUhffBHpGSoAjpWqhi4hEEstNooea2YoqPwfN7AfxLK6qNmnJHC4tb6rNi4j4XqPvWOScWwecDmBmAWAH8FKc6jpBm7RkDpco0EVEIolXl8skYJNzbkuctneCtmkByiqcBkZFRCKIV6BfDjwdp23Vqk1a6I+JIyUKdBGR2sQc6GaWClwCPBfh+ZlmlmtmuQUFBY3eT2WgH1K3i4hIreLRQr8IWO6c21Pbk8652c65HOdcTnZ2dqN30tYLdA2MiojULh6BfgVN3N0CkJkaANDAqIhIBDEFupm1AaYAL8annMjahrtc1IcuIlKbRk9bBHDOHQY6x6mWOlX2oauFLiJSO9+cKdpWg6IiInXyTaCrhS4iUjcfBXpoUFTXcxERqZ1vAj0tOUBKwNTlIiISgW8CHXQ9FxGRuvgr0FOT1UIXEYnAX4GeFlALXUQkAp8FejKHdWKRiEitfBXo7dJTKD5W1tJliIi0Sr4K9C5tUtl7qLSlyxARaZV8FegZqQHd4EJEJAJfBXpacoCSsmBLlyEi0ir5K9BTkigpV6CLiNTGX4GenERpRZBg0LV0KSIirY7PAj10PZfSCrXSRURq8lmgh8pVP7qIyIn8FegpXqBrpouIyAlivQVdlpk9b2ZrzexjMxsTr8Jqk+51uWhgVETkRDHdgg74PTDXOfcFM0sFMuNQU0TpKbomuohIJI0OdDPrAIwHrgFwzpUCTXoaZ7v0ULn7DpcA7ZpyVyIivhNLl8sAoAB4zMw+MLM/mlmbONVVqwNHQt8XD/5nY1PuRkTEl2IJ9GRgNPCwc24UcBi4peaLzGymmeWaWW5BQUEMu4MxAzsDsHjTvpi2IyKSiGIJ9O3AdufcEu/x84QCvhrn3GznXI5zLic7OzuG3UHX9ukAfH5075i2IyKSiBod6M653cA2MxvqrZoEfBSXqurQs0M6Zk29FxER/4l1lsv3gSe9GS6fAF+PvaS6tU3XfUVFRGoTU6A751YAOXGqpV7apOm+oiIitfHVmaIAbdOSOXhMgS4iUpPvAr19hm5DJyJSG98FeoeMFA4eVaCLiNTku0BPMth7qFTXRBcRqcF3gf7Ee1sBePjNTS1ciYhI6+K7QP/5Z4cDcGqvDi1ciYhI6+K7QB8zqAsASzbr9H8Rkap8F+htvSsuPrRAXS4iIlX5LtB7dkhv6RJERFol3wW6mZHdLq2lyxARaXV8F+gAKUmhq3OV6lZ0IiJhvgz0b5w7AEDXdBERqcKXgZ6VmQrAB1sPtHAlIiKthy8DfXfRUQCu/UtuC1ciItJ6+DLQ+3Vu0luXioj4ki8D/bMje4aXj5VVtGAlIiKthy8DvaqdhUdbugQRkVYhpkA3szwzW21mK8ysWTu0Z44fCMCDCzY2525FRFqteLTQJzrnTnfONeut6C4Y1hWAF5fvaM7dioi0Wr7tcmmbFuv9rUVEEkusge6AeWa2zMxm1vYCM5tpZrlmlltQUBDj7o7LSA3EbVsiIokg1kA/1zk3GrgIuN7Mxtd8gXNutnMuxzmXk52dHePujhvYRVMXRUSqiinQnXM7vN/5wEvAWfEoqj7MLLycm7e/uXYrItJqNTrQzayNmbWrXAYuBNbEq7CGuO+1dS2xWxGRViWWFno34G0zWwksBeY45+bGp6z6eeG6sQAs3awWuohIo6eKOOc+AUbGsZYGy257/Lrou4qO0qNDRgtWIyLSsnw7bRGgU9vU8PKYe/9DeYWujy4in16+DvSac9H/uXJnC1UiItLyfB3oNb2v2S4i8imWUIG+o/AYt764mqOlugKjiHz6+D7QN/1qWnh50foCnl66lZPvmEvRkbIWrEpEpPn5PtADScabP55wwvpHFm1q/mJERFqQ7wMdar+DkbpdROTTJiECvTaPL86jpFyhLiKfHgkb6ABDfzqXvL2HW7oMEZFmkTCBPrJ3h1rXT/jNQoqOaoBURBJfwtwl4q/Xns3uomOYwYX/s6jac3sOHqNDRkoLVSYi0jwSpoXeISOFod3bcVK3drxw3Zhqz9UMeBGRRJQwgV7VGf068YPJQ6qtu2/uWiqCDoA31xewdvfBas8XHytjw57iZqtRRCTeEqbLpaa+nTKrPf7fhZv4YGshN04ewtV/XgpA3qzp4edP/fm8E9aJiPhJQrbQAUb2yQJgxqhe4XXvfrKPy2e/F368/cARNuwp5rUPd4fXvbBsO/1vmcPhkvLmK1ZEJA7MOddsO8vJyXG5ubnNtr9K/W+Z0+D3zP3BeQzr3r4JqhERaRgzW+acy4n2uphb6GYWMLMPzOyVWLfVVCrvbNQQU+9/i8IjpQy+7VV2Fh5l36GSJqhMRCR+4tHlciPwcRy202TO6NeRp755doPfd/pdr1MedIyd9R/OuPsNns3dRjDYfH/RiIg0REyBbma9genAH+NTTtMZO7hLePnKs/s2ahs3P7+Kgbe9ylNLtvLUkq00Z3eViEg0sbbQ7wduBnx177eLTunO2l9Orbbuq+fUP+Rve2k1t720mvfzDsS7NBGRRmt0oJvZxUC+c25ZlNfNNLNcM8stKCho7O7iYmB26KqMg7Lbkp4SYOb4gQBsvncad192Kg9eOYrkJGP93RfVa3s1LymwaH0BSzfrrkki0jIaPcvFzO4FrgLKgXSgPfCic+6rkd7TUrNcKjnnqAg6kgPRv8euf3I5c1bvivq6718wmHbpyfzq1bXhdQ2dy75+TzGDs9uSlGQNep+IfDrUd5ZLXKYtmtkE4EfOuYvrel1LB3pDPbxwE+cO7sKgrm342T8+5IXl27luwiAeXlj3zTP6dMrgnstO5Yx+HWmTVve5W2t3H2Tq/W/xowtP4v43NnDnJSO46px+8TwMEfE5BXoTWrOjiIv/8HbU140d1JmnvnVO+PGb6wu4/431PPftMSQHkli8cS9X/nHJCe/LmzWdZ3O3kRpI4rIqJ0aJyKdTfQM9Lqf+O+cWAgvjsS0/OKVX7ZfqrWnxpn2ccudrPPb1Mzmjb8fwJQeWbN7Ps7nbeHnFzlrf99g7m/nFvz4C4IH5GwgkGQ9eOZo2aQG2HzhK306Z9MzKiM/BiEjC+FScKdoUnn1/Gze/sKpF9j2yTxYvXz8u/Lgi6Ph418F6f9FUWr+nmN1Fxxg3uAuHS8tpn65LDIu0Rs3aQv80mnRy1xPWJScZ5c1w4tHKbYUA7Cw8ihmMufc/ALx6w3kM71n/yxVUXlb4K2f35cklW1l/90WkJifs5X1EEp4CvZE6t03j+e+MIT0lwMV/eJsOGSmsvPNCoHHXjmmom55dwYvLd1RbN+2Bt+jcJpXhPdvzt2urnxk778PdDO/Znt4dM1m8cS9rdx+/VPCTS7YC8MneQ5zUtR2rdxRxSq8OvPfJPsYO6oyZZt+I+IECPQY5/TtReKQUgB9Wuf56ZmqAY2UVrP3lRfzp7c38eu7aSJsA4P3bJ3PmPW9w05STOK13B6557P2o+64Z5pX2HS7lrQ17mffhbi4c0R2AYNAx82+h0wWW/2xKrQOxELp+TeUsnl5ZGewoPMpZ/TuxNG8/XdqmsfdQyQlTMtfvKaZb+/QG3RFq76ESbnlhNbM+fyoPL9zEtecOoGdWBmt3HyQrI5XuHdLrvS0ROU596HEQDDrMiNiSHXXXPA4cCZ2ElDdrOu9s3MtX/riE6ycO4sefGQbAgcOldMhIISnJwi38f33vXNJSkrjwfxbxi0tG8LUx/Rhw66sNqu2tmyfy+YcXk18cn4uLLf/ZFLIyUjCDBevy+cbjuYzo2Z45N5xX723c9MwKXvzg+BfS+JOy+es3zqL/LXNIDSSx6OaJpKckkZWZGpeaRfyuWact1leiBno0P//nhzy+OA84ftJRaXkwYn91WUWQsoogmakn/gH1wPwN/O719U1WazSzrzoj3Nqv6fIz+/CTqcPo2OZ4EM9ds5s5q3fxhytGkZu3ny888m6t733nlgsYN+s/4ccdMlKYMrwbw7q3Y+v+I1w9tj+DstuGny8pr2DDnkMNHgj+8XMr2XbgCKf1zmL2ok90QxPxBQV6K1JSXsFtL67h2nMHNGjQsjblFUEG3/7vOFV23FfP6csT722Ny7aqDq5W/rVxzdj+4S+1xujePp33bpsEwLGyCmb+bRmL1hfw3q2T6t1FU1oe5KSfVv+32/SraQS8M3TzDx7jX6t28Y1x/eM+blB0pIy/v7+Vb543kGfe38aM0b1ITwmwdPN+Bma3oUvbtLjuTxKLAj2B/XbeOh5csJHK/3Sb752GmfHZP7zN6h1FnNStLev3HIq6nQlDs8lICfDvNbvJmzW92mDud84fxCNvhs6Ivedzp3D7S2saXGe79GSKj8Xvzk+ThnVl/tr8Wp/r0jaN3J9Orrau/y1z+K8pJ/H9SUO44LcL+aTgcK3vff2H43l5xU4eXLARgLTkJErKj19vrmor3jnH715fz6Wn92Rw13b1rv26J5bx7zW7mTqiO3M/3M0PJg/hxklDGHDrq/TtlMkrN5xLRkqAlHpclqKhSsorCJjV65IX0jop0D8Fht8xlyOlFeHAKTpSxl/ezeN7Ewezed9hXl21i9963TP/97Ucpgzvxth757Oz6Bgr77iQNmkBzIyyiiDpKQEOHivjtJ/P4xeXjODqsf0pqwhSUh6krXf5guaYvROrjfdcRHIgiY35xUz+3aK4bDNv1nTmf7yHQdltue7J5Xy86yDt05N54bqxtM9IoVv76n8h5B88xrGyIH07Z7Ixv5h75nzMgnUnXpjurZsnct59C8KPLxzejcnDu5GWnMSNf1/BP783jtN6Z4Wf37b/CCu2FfLZkT0bVH//W+Zw3pAu7C46Rkl5kEU3T2zgv0B8Oec0c6qBFOifAgcOl3K0rCLiWaPOufAg6vKfTaFTm1SOllZQHgzSrhEnEZ3689fi1uIe2bsDK7cXAZDTryO5W0KXIq7518XFp/XglVXRL5JWVc8O6ewsOhaXOgG+nNOHZ3K3RXz+ka+OZuopPcKPB932KhVBx70zTuXWF1dHfF9maoAjpRV17nv93Rdx4Egp6ckBRt4VupH5GzeNp3fHTNJTAnW+d+nm/SzbcuCEWVZ5s6bz0c6DLN60lwlDs1m4roCrxvTjWGmQwqOllJYHGdKt/n99RPI/r6/n9/M3sPT2Sfx18RZumnISxSXljPzF8UaD1I8CXYDjreoN91wU85/zq7YXcsmD74QfzxjVi999+fRq+/n86N68sHx7tfdddU4//vbelvDjR686gwuGdeWyh97h6Znn0D49Jfz+vFnTKT5Wxsb8Qwzp1o62acmc/98L2LLvSEy11yYlYJRVxOfz/7lRvbj23AEcLinny1VuRB6rsYM6s3jTvlqf++o5fbn7slPDj/P2HmbCbxbyxk3nE0gyJv5mYaP3W7WbqfLaRQ9/ZTQXndqjjndVV/Mvuue+M4asjBSmeCe0vXPLBXRrl9YkXUHbDxyhS9s0dhUdo01agK7t/D0VVoEuAOwqOkq79JRwt0msDpWU887GvXz7b8v436+MZpr3P/g/V+5keI/2DO4amolS+T9zt/ZpLLltMgXFJXzp0XeZc8O5tc7eKSmvwDlqbXUeOFzKqF++Hpf6q6o5bjD55K688XGoj/792yczZ9VOfu5dU6e1+nJOH76Y05uc/p247KF3WOGdRRyrjfdcxI+eW8mInh2459Xjd5g8vU8Wl53ek6+N6V/tcs9PvLeFXh0zmDi0Kz9+biXPLdte22Z56ltnc+X/HT8P4ks5vfnHBzs5rXcH8vYd5h/Xj6N3x8zw8yu2FbKz8Gj4c1YfwaBj4G2vMvnkbrzx8R5SAsaGe6Y15PBbHQW6NKn1e4o5qY4/yyuD8jMjuvHoVVE/h1FVdmNUyps1nYqg4zfz1vHwwk08ds2Z/Pj5VeytcjPvi07pzn9/cSSn3PkaELpZ+Mk92lERdBwtraBr+/TwuMEfrhjF9FN7kJRklFcEw63GeI4bjO6bxfKthcwY3SviiWEA3z5/II+++UmDtt2tfRp7DjbfjcxvnDSEH045Kfy48t9pYHabiIPP9fHt8wcyY1RvNuQX80nB4fAU3ceuOZPsdmn0ysqoNi0WQrOe0lMCOOd4aulW9h8qDY8dVcqbNZ25a3azMb+YAV3asn5PMd8+fyCl5UEOHCkjLTkpLhe823PwGCu3FXL+0GzmrtnNJSN7xmW8QIEuLWrltkJufXE1T888p0FnkUYz+LZXKQ+6cJdAMOjYe7gk/Cd10ZGycF9z7k8n06VtGu9u2kdpRZDzT8pu8P5qBvpZ/TvxpTP7cM+cj8Ini9XH18b04ydTh/Gntzfz3QmDyC8uYaw37z5v1nR2Fh5l5bZCxg7qQofMFF8MQFdq7i+Tqt1Bc1bt4vqnlvPUN8/m+WXbq52wFst2C4pL+PrjS3noytH069ym3tsYN+s/7Cg8Gh6b+Os3zmJ8Iz53NSnQJSEVHyvDQZ1XhoznuEHlbJkLh3dj3kd7wgOgpeVBRt01jzs/O4JJJ3dl3Z7icFdCn04ZzL1xPOv2FDPjfxez7u6ppCWf2JW0/3ApBie0OAE+2nmQaQ+8VW3dsO7t6N0xkzc+3tOgY3ji2rP57pPLGNkni8e/fhaDbjt+tvE/rh/HH+ZvoGdWBnddOoLVO4qqjZO0Vpee3pP7v3x6g8+crsvKOy7kwvvf5GcXD+d7T30QXj9xaDaj+3bk+omDq3UzfbiziKzMVHplZfD3pVt5fHFetWskAZzcoz2v3nBuzK10Bbp8am3ML6Y86BjWPbaTuKrad6iE2Ys+4UefGRrxS6Lyi+SSkT154IpRMe+zZiu9sgV5zWNLWbiugI/u+gy3vri62nX177o0dMeryqCr7UzY0vIgz+Ru48qz+oZPqqprv7FIMgg6WHr7JM66Z354fdUzjs8b0oWvnN2P7zxR5+2JW9wVZ/Xh3hmnhR/X99/pW+cNoPhYOdeM69/oz6QCXaSZ3fWvj/jzO5t56+aJ9OmUGf0NUVTO1x5w6xxy+nXkue+MBeBIaTn5B0vo3yXUFVD1zleVAV5eESSQZI1qGQ65/dVqs39evn4cw3q0Y/x9C+rsWvn6uP489k4eAL/94khG9c2id8dM9h8upXuH9PA02m+dN4Dbpw8HIL/4GJ0yU0kOJMX1i+S8IV14a8PeuG0vHv50dQ6TTu7WqPc2eaCbWTqwCEgjdNXG551zd9b1HgW6SMOVVwRJMot4E/GdhUer9cfHqvhYGS8s2877eQeYs3oXS26bRLf26QSDjoff3MQVZ/WlU5vUauc5/P7y07n09F6s3l7Emp1FXHFW3wbvt+hoGSN/Ma/auhV3TOFHz60Mzz4alN2GTfUYdF16+yR+NedjLji5G1NHdA9f8uELZ/Tm5qlDGX/fAr47YTA3TBrCjsKj1a4j1FT++LUcJg9vvYFuQBvn3CEzSwHeBm50zkWchKtAF2kaRUfK6JAZ3ztOFR0t44OtB5gw9MSbuVSqbFU/M/Mczh7YOeZ9Pr10K0kGP3khdEJW5RfUwws3sWzLAf54dQ5rdx/k2sdz2VF4FAjNZ++YmcLwO17jrktH8LUx/U/Y7q6io/z13S38+MKhJ3wxlpRXMPSnc2OuvaYnv3k2X6lyqepYvmyb/I5FLvRNUHlKX4r303z9NyISFu8wh9AVL+sKc4CXvjuWVduL4hLmQLhlv2JbIeOHHJ8dct2EQeHlYd3b84UzevP7+RsA6OVNN6wrMHt0yOAnU4fV+lxacoDrJw7ioQWbwutW3nkh7dOTeWLJVn72jzX0ysqgR4f08BnNlWaM6hWeWTP55G584YxenNm/E/PX5jNucBfm/9f5TPrtmw35J4hJTH3oZhYAlgGDgYeccz+p6/VqoYtIPCzbsp/PP/wu100YFDGoG8I5R0l5kC8+8i6rdxTxya+mhVvyC9blc97gLiQHksInLQHhKYlvbShg7prd3PO5U2vd9tLN++nePp2+nRs/rtKsg6JmlgW8BHzfObemxnMzgZkAffv2PWPLli21bEFEpGGCQRdxXKGxio6UsXX/EU7tHfk6+5XdTC9fP46RfbIivi6e6hvocbmIgnOuEFgATK3ludnOuRznXE52duwT7EVEgLiHOYS6ruoKc4DffWkkV4/p12xh3hCN7kM3s2ygzDlXaGYZwBTg13GrTESkFZoxujczRvdu6TJqFcsVm3oAf/H60ZOAZ51zr8SnLBERaahYZrmsAmI/HU5EROJC96QSEUkQCnQRkQShQBcRSRAKdBGRBKFAFxFJEAp0EZEE0azXQzezAqCx5/53AVrXBY7jR8fmP4l6XKBja436OeeinmrfrIEeCzPLrc+1DPxIx+Y/iXpcoGPzM3W5iIgkCAW6iEiC8FOgz27pApqQjs1/EvW4QMfmW77pQxcRkbr5qYUuIiJ18EWgm9lUM1tnZhvN7JaWrqc2ZvZnM8s3szVV1nUys9fNbIP3u6O33szsAe94VpnZ6Crvudp7/QYzu7rK+jPMbLX3nge8m3Q317H1MbNN6K2vAAAD0ElEQVQFZvaRmX1oZjcmyvGZWbqZLTWzld6x/cJbP8DMlnj1PGNmqd76NO/xRu/5/lW2dau3fp2ZfabK+hb7/JpZwMw+MLNXEuy48rzPywozy/XW+f7zGDPnXKv+AQLAJmAgkAqsBIa3dF211DkeGA2sqbLuPuAWb/kW4Nfe8jTg34AB5wBLvPWdgE+83x295Y7ec0u915r33oua8dh6AKO95XbAemB4Ihyft7+23nIKsMSr41ngcm/9I8B13vJ3gUe85cuBZ7zl4d5nMw0Y4H1mAy39+QVuAp4CXvEeJ8px5QFdaqzz/ecx5n+Xli6gHv/hxgCvVXl8K3BrS9cVodb+VA/0dUAPb7kHsM5bfhS4oubrgCuAR6usf9Rb1wNYW2V9tde1wHG+TOgOVQl1fEAmsBw4m9DJJ8k1P4PAa8AYbznZe53V/FxWvq4lP79Ab2A+cAHwilen74/L218eJwZ6Qn0eG/Pjhy6XXsC2Ko+3e+v8oJtzbpe3vBvo5i1HOqa61m+vZX2z8/4UH0WoJZsQx+d1S6wA8oHXCbU8C51z5bXUEz4G7/kioDMNP+bmcD9wMxD0HncmMY4LwAHzzGyZhW5EDwnyeYxFLLegkwZwzjkz8/WUIjNrC7wA/MA5d7Bqt6Kfj885VwGcbmZZwEvAsBYuKWZmdjGQ75xbZmYTWrqeJnCuc26HmXUFXjeztVWf9PPnMRZ+aKHvAPpUedzbW+cHe8ysB4D3O99bH+mY6lrfu5b1zcbMUgiF+ZPOuRe91QlzfADOuUJgAaHuhCwzq2zwVK0nfAze8x2AfTT8mJvaOOASM8sD/k6o2+X3+P+4AHDO7fB+5xP6Ej6LBPs8NkpL9/nUo68smdBgxQCOD76MaOm6ItTan+p96P9N9UGa+7zl6VQfpFnqre8EbCY0QNPRW+7kPVdzkGZaMx6XAX8F7q+x3vfHB2QDWd5yBvAWcDHwHNUHD7/rLV9P9cHDZ73lEVQfPPyE0MBhi39+gQkcHxT1/XEBbYB2VZYXA1MT4fMY879NSxdQz/+A0wjNrNgE3N7S9USo8WlgF1BGqM/tWkJ9kPOBDcAbVT4sBjzkHc9qIKfKdr4BbPR+vl5lfQ6wxnvPg3gnhTXTsZ1LqM9yFbDC+5mWCMcHnAZ84B3bGuAOb/1A73/qjV4Ipnnr073HG73nB1bZ1u1e/euoMiuipT+/VA903x+XdwwrvZ8PK/edCJ/HWH90pqiISILwQx+6iIjUgwJdRCRBKNBFRBKEAl1EJEEo0EVEEoQCXUQkQSjQRUQShAJdRCRB/D+7Un4Sk4aVOgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.sched.plot_loss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsoAAAEKCAYAAAALlnNpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4lGX28PHvySSTTCihhV4FBAGlSglkbbtKUVCKAnZQVCDJ7roWtthdf++uogkgCjawUERQgWBd1IReEnrvRZBQQslAkpn7/SMzGDHAEDJ5Zibnc125mHkyM8/Jbnxy5sy5zy3GGJRSSimllFK/FWZ1AEoppZRSSgUiTZSVUkoppZQqgibKSimllFJKFUETZaWUUkoppYqgibJSSimllFJF0ERZKaWUUkqpImiirJRSSimlVBE0UVZKKaWUUqoImigrpZRSSilVhHCrAyisWrVqpmHDhlaHoZRSl2zFihVZxphYq+MoTXrNVkoFK1+v2QGVKDds2JDly5dbHYZSSl0yEdlldQylTa/ZSqlg5es1W1svlFJKKaWUKoImykoppZRSShVBE2WllFJKKaWKoImyUkoppZRSRdBEWSmllFJKqSL4NVEWkb+IyDoRWSsiU0Qkyp/nU0oppZRSqqT4LVEWkTpAItDBGNMKsAED/XU+pZRSSimlSpK/Wy/CAYeIhAPRwH4/n+93ftj0Cxt+Pl7ap1VKKXWOk2fymbZsN1t/OWl1KEqpELD7cA4vz12P2238dg6/JcrGmH3Aq8Bu4Gcg2xjzzbmPE5FhIrJcRJYfOnSoxON44P1l9EhO4730HRjjv/8hlVJKXdjJ0/k89dkaluw4bHUoSqkgN2f1fnqlpDF12R62Z53y23n82XpRGegDNAJqA+VE5J5zH2eMmWCM6WCM6RAbW7K7v+a73GdvvzBnPQ9PXsHRU7kleg6llFK+cdhtADhzXRZHopQKVs5cF6NmrmbkJxk0qVGe1MR4mlQv77fz+bP14o/ADmPMIWNMHjATiPPj+X4nJ6/gYvz3ns155tYW/Lj5F3qmpLF0x5HSDEMppRQQ7UmUczRRVkoVw+aDJ+gzLp0pS/fw6HWNmf5IF+pVifbrOf2ZKO8GOotItIgIcBOwwY/n+x1v1SLaHs6Qbo2Y+VhXIsPDGDhhEWO+34LLjz0tSimlfivCFkaETTRRVkpdEmMMU5fupvfYdI6cymXykI483aM5ETb/Tzn2Z4/yEmAGsBJY4znXBH+dryi/JsoFVYyr68YwJzGe21rX5rVvN3Pvu0v45fjp0gxJKaXKNEeEjdN5migrpXxz4nQeiVMzeXrmGto3qExqUjx/uLJkW3UvxK+puDHmWWNMc2NMK2PMvcaYM/4837lyzkmUAcpHhvPGXW34T/9ryNh9jB7Jafyw6ZfSDEsppcqsaHs4Obn5VoehlAoCq/ceo1dKOqlrfuaJW5oxeUgnqlco3S05QnpnPmdewcXYYQ//zXER4c4O9Zid0JXYCpE88P4yXpm3gbxCi/+UUkqVvGi7TVsvlFIXZIzhnbTt9Bu/kHyXm2nDOjPihibYwqTUYwnpRNl7MXZE2Ir8fpPqFfh8RFfu7lSft3/czoC3FrHnSE5phqiUUmVKVIRNp14opc7ryKlchk5azktzN3BDs+qkJsXToWEVy+IpE4ly4daLc0VF2Hj5jqsZN7gd2w6dpGdKGqlrfi6tEJVSqkzRirJS6nwWbz9Mj+SfSN+SxfO9W/L2ve2pFG23NKaQTpS9VQvHBRJlr17X1CI1MZ7GseUZ/vFK/jFrjS44UUqpEuaw286O7lRKKQCX2/DGd5sZPHEx0fZwZg6P4/64hhQMTbNWSCfKvlSUC6tXJZpPH+3CI9ddwcdLdnP7uAVs/eWEP0NUSqkyJdpuw6mL+ZRSHgeyTzN44mLe+G4Lt7epw+yEbrSqE2N1WGeFdKLs9FQtoiPCL/LIX0XYwhjV4yo+ePBaDp04w21jFjB92R7d/loppUpAtD387LVZKVW2zd9YsBHc6r3ZvDqgNaPvakP5SN9zttIQ2olyrnfqhW8V5cKu9zSQt61fiSc/W82fp2Vy8oxWQZRS6nI47LqYT6myLjffzctz1/PgB8uoXiGSOYnd6N++rtVhFSmw0vYSlpPrIjxMsIcX7/1AjYpRfDi0E2/O38rr321m1Z5jjBnUjqvrBs5HAkopFUyiI3Qxn1Jl2e7DOSRMWcmqvdnc16UBf+95FVHnmU4WCEK6opyT6zrvaDhf2cKEhJuaMnVYF87ku+k7fgHvpe/QVgyllCoGh92GM8+l11ClyqDZq/bTKyWNHVmneOuedrzQp1VAJ8kQ4omyM9dVrLaLonRsVIXUxHiuu7I6L8xZz8OTl3P0VG6JvLZSSpUVDrsNY+B0nm7wpFRZ4cx1MWrmahKmZNC0RnnmJsbTvVUtq8PySUgnyjl5Lp8nXviicjk7E+9rz7O3teCnzVn0TElj6Y4jJfb6Sil1OUSku4hsEpGtIvJ0Ed9vICLfi8hqEflBROqe8/2KIrJXRMb6K8ZoT/VIt7FWqmzYfPAEfcalM2XpHh67vjHTHulCvSrRVofls5BOlJ25+b/bvvpyiQgPdm3EzOFxRIaHMXDCIlK+34LLrR8jKqWsIyI2YBzQA2gBDBKRFuc87FVgsjHmGuAF4JVzvv8i8JM/44z2XJO1T1mp0GaMYcrS3fQem86RU7lMHtKRp7o3J8IWXKlncEV7iZwlXFEurFWdGOYkxtO7dW1Gf7uZe95ZwsHjp/1yLqWU8kFHYKsxZrsxJheYCvQ55zEtgP95bs8v/H0RaQ/UAL7xZ5Dedjjd0Emp0HX8dB4JUzIYNXMNHRpUITUpnj9cGWt1WMUS0olyTq7/EmWA8pHhvH5XG/7T/xoy9xyjZ3IaP2z6xW/nU0qpC6gD7Cl0f6/nWGGrgL6e23cAFUSkqoiEAa8Bf/N3kN5rslaUlQpNq/Yc49aUdOatPcATtzRj8pCOVK8QZXVYxRbSibKzBKZeXIyIcGeHesxO6EpshUgeeH8Zr6RuIDdfF6oopQLO34DrRCQDuA7YB7iA4UCqMWbvxV5ARIaJyHIRWX7o0KFLDsARoYmyUqHI7Ta8k7ad/m8tJN/lZtqwzoy4oQlhYdZvQ305Qn6OcklNvbiYJtUr8PmIrrw0dz1v/7SdxTuOMHZQ26BqWFdKBbV9QL1C9+t6jp1ljNmPp6IsIuWBfsaYYyLSBYgXkeFAecAuIieNMb9bEGiMmQBMAOjQocMlL87wXpOdebqYT6lQceRULo9Pz2T+pkPc3KIG/+l/DZWi7VaHVSJCuqLs79aLc0VF2Hjp9qt58+52bD90kp7Jacxd/XOpnV8pVaYtA5qKSCMRsQMDgS8LP0BEqnnaLABGAe8BGGPuNsbUN8Y0pKDqPLmoJLkk6GI+pULL4u2H6ZH8Ewu2Hub53i15+972IZMkQ4gnys7cfBwRpV8073l1LVIT42lcvTwjPlnJ32et0YUrSim/MsbkAyOBr4ENwHRjzDoReUFEensedj2wSUQ2U7Bw7+XSjlN7lJUKDS634fVvNzN44mKi7eHMHB7H/XENEQnuVotz+S2LFJFmwLRCh64AnjHGvOGvcxZmjCnxOcqXol6VaD59tAuvfbOZt37cxoqdRxk7uC1Na1SwJB6lVOgzxqQCqecce6bQ7RnAjIu8xgfAB34IDyjUeqGJslJB60D2aZKmZrBkxxH6tq3DC7e3onxkaHbz+q2ibIzZZIxpY4xpA7QHcoBZ/jrfuc7kuzGGUutRLkqELYynezRn0pCOZJ08w21j05m+bI9u3aqUKrOiz/Yoa6KsVDD638aD9Ej+iTX7snltQGtG39UmZJNkKL3Wi5uAbcaYXaV0vrPVCqsqyoVdd2Us85LiaVe/Mk9+tpo/T8vkxOk8q8NSSqlSFxWurRdKBaPcfDcvzVnPkA+WUzPGweyEbvRrX/fiTwxypfUWYCAwpZTOBRRsXw2BkSgDVK8YxYdDOzH+h62M/nYzmXuOMXZQO66uG2N1aEopVWrCwoSoiDCcuoW1UkFj1+FTJEzJYPXebO7r0oC/97yKKD+P3w0Ufq8oe1Zf9wY+Pc/3L2sm5/l4L8KB9H+kLUwYeWNTpj3Shdx8N33HL+Dd9B3aiqGUKlOi7eFaUVYqSMxetZ9eKenszDrFW/e044U+rQIqt/K30mi96AGsNMYcLOqbxpgJxpgOxpgOsbElt71hztnWi8Drm7m2YRXmJcVz3ZXVeXHOeh6atJyjp3KtDksppUqFI8Kmi/mUCnDOXBejZq4mYUoGV9YoT2pSPN1b1bI6rFJXGonyIEq57QIKJ8qB+a6nUrSdife157nbWpC2JYseyWks2X7Y6rCUUsrvou02rSgrFcA2HzxBn3HpTF22h+HXN2baI12oW7lsbqDm10RZRMoBfwJm+vM8RfFWK6ycenExIsIDXRsxc3gcURFhDJq4mOTvtuByayuGUip0RdttZ9eRKKUChzGGKUt303tsOkdO5TJ5SEee7N6cCFtIb7txQX79yY0xp4wxVY0x2f48T1GcAbaY70Ja1YlhTmI8vVvX5vXvNnPPO0s4ePy01WEppZRfOOw2TmtFWamAcvx0HglTMhg1cw0dGlQhNSme+KYl1xIbrEL2LcLZ1gsLduYrjvKR4bx+Vxv+2/8aMvcco0dyGvM3/WJ1WEopVeKi7eHk5OnUC6UCxao9x7g1JZ15aw/wxC3NmDykI9UrRFkdVkAI2UTZO/UikFsvziUiDOhQj9kJ3aheIZIH31/Gv1M3kJvvtjo0pZQqMY4I7VFWKhC43YaJP22n3/iFuNyG6Y90ZsQNTQgLC61tqC9HyCbKOUHQo3w+TaqX5/MRXbmnc30m/LSdAW8vYvfhHKvDUkqpEuGw69QLpax2+OQZhk5axsupG7ixeXXmJnajfYMqVocVcEI/UQ7SWX9RETZeuv1qxt/dju2HTtIrJY25q3+2OiyllLpsOvVCKWst2naYnilpLNh6mBf6tOTte9tTKdpudVgBKWQTZWeei8jwMGxB/vFBj6trkZoYT5Ma5RnxyUpGzVzDaV0trpQKYlpRVsoaLrfh9W83c/c7iylnD2fWiDju69IQkeDOlfwpOFa6FUNObn5QTLzwRb0q0Ux/pAuvfbOZt37cxspdRxk7uC1Na1SwOjSllLpk0RHh5Lrc5LvchJfhsVNKlaYD2adJnJrB0h1H6NuuDi/2aUW5yJBNA0tMyF6hnLnugNyVr7gibGE83aM5k4Z0JOvkGW4bm860Zbt1+2ulVNDxFjGc+umYUqXifxsP0iP5J9buy+a1Aa0ZfWcbTZJ9FLqJcl5+UC7ku5jrroxlXlI87RtU5qnP1pA0NZMTp/OsDksppXzmvTZr+4VS/pWb7+bFOesZ8sFyasY4mJ3QjX7t61odVlAJ2bcTObmukGm9OFf1ilFMHtKJ8T9s5fXvtrBq7zHGDGrLNXUrWR2aUkpdlHeRtS7oU8p/dh0+RcKUDFbvzeb+Lg0Y1fMqooJ0wIGVQrainJPrCulfCFuYMPLGpkwd1pm8fDf9xi/k3fQd2oqhlAp43iKGJspK+ceXq/bTKyWdnVmneOue9jzfp1VI50T+FLKJsjOEK8qFXduwYJvJ65tV58U563lo0nKOnMq1OiyllDqvs60XujufUiXKmevi6c9Wkzglg2Y1K5CaFE/3VjWtDiuohWyiHEpTLy6mUrSdCfe257nbWpC2JYueyWks2X7Y6rCUUqpI3oXWWlFWquRsOnCC3mPTmbZ8D8Ovb8zUYZ2pWzna6rCCXsgmys5cF46IkG3B/h0R4YGujZg5PA6H3cagiYtJ/m4LLre2YiilAou2XihVcowxTFm6m95j0zmak8fkIR15sntzInT0YokI2f8VnXllo/XiXK3qxDA7oRt92tTh9e8KhoofPH7a6rCUUuosb+uFbp6k1OU5fjqPkVMyGDVzDR0bVSE1qRvxTWOtDiukhGyiHMpTLy6mfGQ4o+9szasDWrNqTzY9ktOYv+kXq8NSSilAK8pKlYTMPcfolZLGV2sP8GT3Zkx6sCPVK0RZHVbICclE2eU2nMl3h+QcZV+JCP3b12V2QjeqV4jkwfeX8fLc9eTmu60OTSlVxul4OKWKz+02TPxpO/3HL8TthumPdGb49U0IC9NtqP0hJJt4vbs9OXQUCk2ql+fzEV15ee4GJqbtYOnOo4wZ2Jb6VbXBXylljV83HNGpF0pdisMnz/C3T1cxf9MhbmlZg//0a01MdITVYYW0kKwo53guvmW19eJcURE2Xry9FePvbsf2QyfplZLGnNX7rQ5LKVVG2W1h2MJEK8pKXYJF2w7TMyWNBdsO82Kflrx1T3tNkkuBXxNlEakkIjNEZKOIbBCRLv48n5d3W1SHPSQL5sXW4+papCbG06RGeUZ+UtD8r1vIKqVKm4gQHWHTRFkpH+S73Iz+djOD31lMOXs4s4bHcW+Xhohoq0Vp8HcmmQx8ZYzpLyJ2oFQ+7/defLWi/Hv1qkQz/ZEuvPbNZt76cRsrdh1h3OB2NK1RwerQlFJliMNu0zfqSl3Ez9lOkqZmsnTHEfq1q8sLfVpSLlKLgKXJbxVlEYkB/gC8C2CMyTXGHPPX+Qo726OsiXKRImxhPN2jOZOHdOTIqVxuG5vOtGW7dftrpVSpibbbzl6rlVK/9/2Gg/RMTmPtvmxG39ma1+5srUmyBfzZetEIOAS8LyIZIvKOiJTz4/nO8lYponUx3wX94cpYUpPiad+gMk99tobEqZmcOJ1ndVhKqTLAYQ/X1gulipCb7+aF2esZOmk5tWIczEnoRt92da0Oq8zyZ6IcDrQDxhtj2gKngKfPfZCIDBOR5SKy/NChQyVy4l9bL/Sd18VUrxDF5CGdeOKWZqSu+Zlbx6Szem+pFP6VUmWYIyIMZ55OvVCqsJ1Zp+g3fiHvLdjB/V0aMHN4HFfElrc6rDLNn4nyXmCvMWaJ5/4MChLn3zDGTDDGdDDGdIiNLZndZLxTLxz2kBzqUeJsYcKIG5owbVhn8vLd9Bu/kHfStmsrhlJBRkS6i8gmEdkqIkUVJhqIyPcislpEfhCRup7jbURkkYis83zvLn/HGq0VZaV+44vMfdw6Jp3dR3J4+972PN+nFVH6ybjl/JZJGmMOAHtEpJnn0E3Aen+drzCdelE8HRpWITUpnhuaVeeluRt4aNJyjpzKtTospZQPRMQGjAN6AC2AQSLS4pyHvQpMNsZcA7wAvOI5ngPcZ4xpCXQH3hCRSv6MVxfzKVUgJzefp2asJmlqJs1qViA1KZ5bWta0Oizl4e+SawLwsYisBtoA//bz+YBCrRf6TuySVYq2F7yT7d2StC1Z9ExOY8n2w1aHpZS6uI7AVmPMdmNMLjAV6HPOY1oA//Pcnu/9vjFmszFmi+f2fuAXoGQ+4juPaLuOh1Nq04ET9B67gOkr9jDihsZMG9aZOpUcVoelCvFromyMyfS0VVxjjLndGHPUn+fz0qkXl0dEuD+uITOHx+Gw2xg0cTHJ323B5dZWDKUCWB1gT6H7ez3HClsF9PXcvgOoICJVCz9ARDoCdmCbn+IENFFWZZsxhk+W7Kb32HSO5eTx4ZBOPHFLc8Jt2jIaaELy/xFnroswgcjwkPzxSk2rOjHMTujG7W3q8Pp3m7n7ncUcPH7a6rCUUsX3N+A6EckArgP2AWezVRGpBXwIPGiMcRf1AiW1ANsREc5pHQ+nyqDjp/MY+UkGf5+1ho6NqjAvKZ5uTatZHZY6j5DMJHNyXUTbw3XXmhJQPjKc0Xe14dUBrVm1J5seyWnM3/iL1WEppX5vH1Cv0P26nmNnGWP2G2P6eiYR/cNz7BiAiFQE5gL/MMYsPt9JSmoBdkFFOV8XDasyJXPPMXqlpPHVugM81b05kx7sSGyFSKvDUhcQkomyMy9f2y5KWP/2dZmT2I3qFSJ58INlvDx3Pbn5RRaclFLWWAY0FZFGnp1QBwJfFn6AiFQTEe91fxTwnue4HZhFwUK/GaURrMNuw23gjF5HVBngdhsm/LSN/uMX4nbD9Ee68Nj1jQkL04JeoAvJRDkn14VDF/KVuMax5fl8RFfu69KAiWk7GPDWQnYfzrE6LKUUYIzJB0YCXwMbgOnGmHUi8oKI9PY87Hpgk4hsBmoAL3uO30nBTqoPiEim56uNP+P1XqN18oUKdYdPnmHIpGX8O3Ujf7yqBqmJBRt9qeAQkvPTClovNFH2h6gIGy/0aUVc46o8OWM1vVLS+Hffq7mtdW2rQ1OqzDPGpAKp5xx7ptDtGRTMtD/3eR8BH/k9wEK81+icPBeaMqhQtXBbFn+emskxZx4v9mnJPZ0baFtokAnJirIz16WtF37WvVUt5ibG07RGeRKmZDBq5mqtDCmlfOa9RjtzdXc+FXryXW5Gf7uZu99ZQvmocD4f3pV7uzTUJDkIhWhFOV8ryqWgXpVopj3ShdHfbmb8D9tYsesoYwe348oaFawOTSkV4KI9G0LpiDgVan7OdpI0JZOlO4/Qv31dnu/dknKRIZlulQmhWVHOc+OI0F/K0hBhC+Op7s2ZPKQjR07l0ntsOlOX7taV7EqpC4q2a4+yCj3frT9Ij+Q01u7P5vW7WvPqgNaaJAc5nxNlEYn2ZyAlyakV5VL3hytjSU2Kp0ODKjw9cw0JUzI4cTrP6rCUUgHKUahHWalgdybfxQuz1/PQ5OXUjnEwJ6Ebd7Sta3VYqgRcNFEWkTgRWQ9s9NxvLSJv+j2yy6CL+axRvUIUk4d05IlbmjFv7QF6paSzeu8xq8NSSgUgnXqhQsXOrFP0H7+I9xbs4IG4hswaEccVseWtDkuVEF8qyq8DtwCHAYwxqygYIxSwnLkuonQ8nCXCwoQRNzRh2rDO5Lvc9Bu/kHfStmsrhlLqN85OvdBEWQWxLzL3ceuYdHYfyWHCve15rndLIsM1/wglPrVeGGP2nHMoYK9sxhhy8rSibLUODauQmhTPDc2q89LcDQydtJwjp3KtDkspFSB06oUKZjm5+Tw5YxVJUzNpXrMCqUnx3NyyptVhKT/wJVHeIyJxgBGRCBH5GwXD7ANSrsuNy200UQ4AlaLtvH1ve57v3ZL0LVn0SP6JxdsPWx2WUioA6NQLFaw2HjhO77EL+HTFXkbe0ISpwzpTp5LD6rCUn/iSKD8KjADqAPuANsBwfwZ1Obz9bg67rjINBCLC/XENmTk8jmh7OIMnLuaN7zbjcmsrhlLnIyIdRGSWiKwUkdUiskZEVlsdV0ny9ihroqyChTGGj5fsos/YBRzLyePDIZ342y3NCLeF5AAx5eFLNtnMGHN34QMi0hVY4J+QLo/Ts4JaK8qBpVWdGGYndOOZz9fyxndbWLTtMMkD21IzJsrq0JQKRB8DTwBrALfFsfiFLUyIDA/jtE69UEEg25nH32euYe6an4lvWo3Rd7YhtkKk1WGpUuDL26AxPh4LCN7qhCbKgad8ZDij72rDawNas2ZfNj1T0vjfxoNWh6VUIDpkjPnSGLPDGLPL+2V1UCUt2m7TirIKeBm7j9IrJY2v1h3gqe7NmfRgR02Sy5DzVpRFpAsQB8SKyF8LfasiELBZ6NnWC516EbD6ta9Lm/qVGPlJBkM+WM5D3RrxZPfm2MP14yulPJ4VkXeA74Ez3oPGmJnWhVTyHBGaKKvA5XYbJqZt579fb6JGxSimP9KF9g0qWx2WKmUXar2wA+U9jym8J/FxoL8/g7ocOWd7lDVRDmSNY8sza3gc/07dwDvpO1i68whjBrWlQdVyVoemVCB4EGgORPBr64UBQitRtttw5unUCxV4sk6e4fHpq/hx8yG6t6zJ/+t3DTHREVaHpSxw3kTZGPMj8KOIfBBMH/nleEYNaetF4IuKsPFCn1bENa7KkzNW0yslnVf6Xs1trWtbHZpSVrvWGNPM6iD8LdoerhVlFXAWbs3iz9MyOebM48XbW3FPp/qIiNVhKYv4spgvR0T+C7QEzq68MsbceLEnishO4AQFc5fzjTEdihmnz35tvdCpF8Gie6tatKoTQ+KUDBKmZLBwWxbP3NpSPxVQZdlCEWlhjFlvdSD+5NAeZRVA8l1uUr7fwpj5W2lUrRwfPNiRFrUrWh2Wspgv2eTHwDTgVgpGxd0PHLqEc9xgjMkqRmzFoov5glPdytFMe6QLr3+7mfE/bmPFrqOMHdyOK2tUuPiTlQo9nYFMEdlBQY+yAMYYc421YZWsaLtNNyJSAeHnbCdJUzJZuvMI/dvX5YU+Lc/O+lZlmy+/BVWNMe+KSFKhdoxl/g6suHQ8XPCKsIXxZPfmdGlclb9My6T32HSeva0lA6+tpx97qbKmu9UBlIZou429R7WirKz13fqD/G3GKvLy3bx+V2vuaFvX6pBUAPFlzECe59+fRaSXiLQFqvj4+gb4RkRWiMiwYkV4iZy6mC/oxTeNJTUpng4NqjBq5hoSpmRw/HTexZ+oVOgw5/kKKY6I8LPXbKVK25l8F8/PXsdDk5dTp5KD2QndNElWv+NLRfklEYkBHqdgfnJF4C8+vn43Y8w+EakOfCsiG40xPxV+gCeBHgZQv3593yM/jxwdDxcSqleIYvKQjoz/cRujv93M6r3ZjBnUltb1KlkdmlKlYS4FibFQsDakEbCJgrUiIcNhDzu7AFup0rQj6xQJU1aydt9xHohryKiezYkM17xB/d4FK8oiYgOaGmOyjTFrjTE3GGPaG2O+9OXFjTH7PP/+AswCOhbxmAnGmA7GmA6xsbHF+BF+KycvH7stTLeUDAFhYcKIG5ow/ZHOuNyG/m8t5J207bh1+2sV4owxVxtjrvH825SCa+ciq+MqaTr1Qlnhi8x93JqSxp4jTibc257nerfUJFmd1wWzSWOMCxhUnBcWkXIiUsF7G7gZWFuc17oUzlyXtl2EmPYNqjA3sRs3NKvOS3M3MHTSMl0ApMoUY8xKoJPVcZQ0R4SNM/luXPrmV5WCnNx8npyxiqSpmVxVqyLzkuK5uWVNq8NSAc6X1osFIjKWgskXp7wHPRfuC6kBzPIswgoHPjHGfFXcQH2Vk+vShXwhqFK0nbfvbc+Hi3fx0pwN9EhkB1HnAAAgAElEQVT+ieSBbel8RVWrQ1OqxJ2zG2oY0A7Yb1E4fuO9VjvzXJSP1AkDyn82HjjOyE8y2HboJCNvaMKf/9hUP3lWPvHlytTG8+8LhY4Z4IJzlI0x24HWxYyr2LSiHLpEhPu6NKR9g8okfJLB4ImLSbixKYk3NcUWplMxVEgpPBcxn4Ke5c8sisVvzibKuZooK/8wxvDxkt28OGc9FR0RfDS0E12bVLM6LBVELnplMsbcUBqBlBRnnlaUQ13L2jHMTujGv75YS/L3W1i8/TDJA9tSMybq4k9WKjisN8Z8WviAiAwAPj3P44OSwzOnVidfKH/IduYxauZqUtccIL5pNUbf2YbYCpFWh6WCTMh97pCTm0+07soX8spFhjP6zja8NqA1a/Zl0yP5J/638aDVYSlVUkb5eCyoeYsaOXk6+UKVrIzdR+mVksY36w7ydI/mTHqwoybJqlhCLqN05rqIibZbHYYqJf3a16VN/UqM/CSDIR8s56FujXiye3Ps4SH3HlCVASLSA+gJ1BGRlELfqkhBC0ZI8Y7x1MkXqqS43YaJadv579ebqFExiumPdqFd/cpWh6WCWMglyjm5LmrFaOtFWdI4tjyzhsfxSuoG3knfwdKdRxgzqC0NqpazOjSlLtV+YDnQG1hR6PgJfJ9fHzQchXqUlbpcWSfP8Pj0Vfy4+RA9WtXk//pdQ4wjwuqwVJC7aKIsIn2LOJwNrPHMRw4oOvWibIqKsPF8n1Z0aVyNJ2esoldKOq/0vZrbWte2OjSlfGaMWQWsEpFPjDEhvx3l2dYLTZTVZVq4NYukaZlkO/N48fZW3NOpPp6pW0pdFl8qykOBLsB8z/3rKah0NBKRF4wxH/optmJx5unUi7Kse6uatKpTkaSpmSRMyWDB1iyeva2l/k6oYNNRRJ4DGlBwnRbAGGOusDSqEvZrohxyXSWqlOS73CR/v4Wx87dyRbVyTB7SkatqVbQ6LBVCfEmUw4GrjDEHAUSkBjCZguH3PwGBlShrRbnMq1s5mqnDOvP6t5sZ/+M2Vuw6ytjB7WhWs8LFn6xUYHiXglaLFUDIllu9Uy9O54Xsj6j8aP8xJ0lTM1i28yj929flhT4tibaHXEepspgvK57qeZNkj188x44AAfXRoNttPBVl/Q+lrIuwhfFk9+ZMHtKRozl59B6bzpSluzFGdwBTQSHbGDPPGPOLMeaw9+tiTxKR7iKySUS2isjTRXy/gYh8LyKrReQHEalb6Hv3i8gWz9f9Jf0DFSVaF/OpYvp2/UF6pqSxfv9xXr+rNa8OaK1JsvILXxLlH0Rkjuciej/whedYOeCYf8O7NKfzCy62WlFWXvFNY0lN6kbHRlUYNXMNCVMyOH46oN7fKVWU+SLyXxHpIiLtvF8XeoKI2IBxQA+gBTBIRFqc87BXgcnGmGso2ETqFc9zqwDPUvBJYUfgWRHx+6gAh/Yoq0t0Jt/Fc1+u4+HJy6lTycGcxHjuaFv34k9Uqph8efs1AugHdPXcnwx8ZgpKcwG1GYn3YusdOaQUQPUKUUx6sCNv/bSN177ZzOq92YwZ1JbW9SpZHZpS59PJ82+HQscutiNqR2CrZ1dURGQq0AdYX+gxLQDv9tjzgc89t28BvvV8UoiIfAt0B6Zcxs9wUZHhYYjo1Avlmx1Zp0iYspK1+47zQFxDRvVsTmS4/r1X/uXLznwGmOH5Cmjei60u3FLnCgsThl/fhE6NqpA4JZN+4xfyVPfmDO3WiDDd/loFmGLuiFoH2FPo/l5+Tbi9VgF9gWTgDqCCiFQ9z3PrFCOGSyIiREfYtKKsLurzjH38Y9Yawm1hTLyvA39qUcPqkFQZcdHWCxHp6+lZyxaR4yJyQkSOl0Zwl8p7sdXWC3U+7RtUITUxnpuuqs7LqRsYOmkZh0+esTospX5DRGqIyLsiMs9zv4WIDC2Bl/4bcJ2IZADXAfu4xMWCIjJMRJaLyPJDhw5ddkAOezhO3ZlPnUdObj5PfLqKP0/LpEXtisxLitckWZUqX3qU/wP0NsbEGGMqGmMqGGMCcvaKd8SQJsrqQmKiI3jrnva82KclC7YdpmdKGou2XXSdlFKl6QPga8A7CHwz8OeLPGcfUK/Q/bqeY2cZY/YbY/oaY9oC//AcO+bLcwu9xgRjTAdjTIfY2FjffpoLiLZrRVkVbcPPx7ltTDozVu4l4cYmTHm4M7UrOawOS5UxviTKB40xG/weSQlw5nl7lHXlq7owEeHeLg2ZNTyOcvZwBr+zmNHfbsbl1qkYKiBUM8ZMB9wAxph8Ll75XQY0FZFGImIHBgJfFn6AiFQTEe91fxTwnuf218DNIlLZs4jvZs8xv4u227RHWf2GMYaPFu+iz7gFHD+dz0dDO/H4zc0It/mSsihVsnzJKJeLyDQKFn2c/YzaGDPTb1EVk1NbL9Qlalk7htkJ3Xjmi3WkfL+FxdsPkzywDbVitGqhLHXK0ztsAESkMwU7op6XMSZfREZSkODagPeMMetE5AVguTHmSwo2jHpFRAwFc/BHeJ57RERepCDZBnjBu7DP3xx229kih1LZzjye/mw189Ye4A9XxjL6ztZUKx9pdViqDPMlUa4I5FBQYfAyQMAlytqjrIqjXGQ4r93Zmq5NqvLPz9fSMzmNVwe05qartA9OWeavFFSDG4vIAiAW6H+xJxljUoHUc449U+j2eRdmG2Pe49cKc6nR1gvltXL3URI+yeDg8dOM6tGch+Ov0MXWynK+TL14sDQCKQneinKUjodTxdC3XV1a16vEyE8yGDppOUO7NeKp7s2xh+vHfap0GWNWish1QDMKtq/eZIwJyQHgjggbR06F5I+mfOR2GyakbefVrzdRMyaK6Y92oV19v4/xVson502UReRJY8x/RGQMno//CjPGJPo1smLQxXzqcjWOLc+s4XG8krqBd9N3sHTHEcYObkuDquWsDk2VIZ7NQ3oCDSm4Tt8sIhhjRlsamB847OE4c3XqRVmVdfIMf52+ip82H6JHq5r8X79riHFEWB2WUmddqKLsXcC3/HJO4LngLwf2GWNuvZzXupicPG/rhS7mU8UXFWHj+T6t6NK4Gk/OWEWvlHT+3fdqereuffEnK1UyZgOngTV4FvSFKp2jXHYt2JrFn6dlku3M46XbW3F3p/qIaKuFCiznzSiNMbM9/066zHMkUZB0+32knDPXhQhERehH5erydW9Vk1Z1KpI0NZPEKRks2JLFc71b6oY2qjTU9WwzHfIcOvWizMl3uXnjuy2M+2ErV1Qrx+QhHbmqVkBOnVXKpw1HrhSRCSLyjYj8z/vly4uLSF2gF/DO5QbqC2euC0eETd+RqhJTt3I0U4d1ZsQNjZm+Yg+9x6az6cAJq8NSoW+eiNx88YcFv2idelGm7D/mZOCExYydv5X+7eoyO6GbJskqoPnSo/Ap8BYFye6lXs3eAJ4EKlzi84olJ8+l/cmqxEXYwnjiluZ0vqIqf5m2it5j03n2tpYM6lhP35Qpf1kMzPLMPM6jYEGfCdTNni5HtN1GvtuQm+/WhbMh7pt1B3hixuqCivJdbbi9rd93SVfqsvmSKOcbY8Zf6guLyK3AL8aYFSJy/QUeNwwYBlC/fv1LPc1vOHNd+rG48pv4prHMS4rnr9Mz+fusNSzYlsUrfa+mYpQuPFElbjTQBVhjjAnpXXAcnjUlzlyXJsoh6ky+i1dSN/LBwp20qlORMYPa0aiaLpBWwcGXq9JsERkuIrVEpIr3y4fndQV6i8hOYCpwo4h8dO6DSnI71JzcfBw6Gk75UWyFSCY92JEnuzfjq7UH6JWSRuaeY1aHpULPHmBtqCfJwNlrdk6eTr4IRTuyTtH3zYV8sHAnD3ZtyGePxWmSrIKKLxXl+z3/PlHomAGuuNCTjDGjKNgiFU9F+W/GmHuKEaPPcnJdZ6sTSvlLWJgw/PomdGpUlcQpGfQfv5CnujdnaLdGOhxflZTtwA8iMo/f7ogacuPhvO1yOvki9MzK2Ms/Z60lIjyMifd14E8tdBMnFXwumFV6+uPuMcYsKKV4Losz10W0VpRVKWnfoDKpifE8+dkqXk7dwIJtWbw2oDVVdbtVdfl2eL7snq+Q5W2X08kXoSMnN59nvljHjBV7ubZhZZIHtqV2JYfVYSlVLBdMlI0xbhEZC7S9nJMYY34Afric1/BFTq6LWjHaL6pKT0x0BG/d056PFu/ixbkb6JGcxhsD2xDXuJrVoakgZox5HkBEynvun7Q2Iv/RinJo2fDzcUZ8spIdWadIuLEJSTc1JdymvecqePny2/u9iPSTIFjefzpPF/Op0ici3NulIbOGx1E+Kpy731nC6G83k+8K6X0ilB+JSCsRyQDWAetEZIWItLQ6Ln/wJso6Ii64GWP4cPEu+oxbwInT+Xw8tBOP39xMk2QV9Hz5DX6EghFxZ0TkuIicEJHjfo6rWHJydTycsk7L2jHMHtmNvm3rkvL9Fga/s4Sfs51Wh6WC0wTgr8aYBsaYBsDjwESLY/ILR4R36oUu5gtW2c48hn+8kn99vpbOV1RlXlI8cU30UzUVGi6aKBtjKhhjwowxdmNMRc/9gJzlmZObr9tXK0uViwzntTtbM/rO1qzdl03P5DS+33DQ6rBU8ClnjJnvveNpXwvJUQHaehHcVu4+Ss/kNL5df5BRPZrzwQPXUk3XaagQ4lNWKSKVgaZAlPeYMeYnfwVVXM48F1G6mE8FgL7t6tKmXiVGfpLB0EnLGdK1EU/1aEZkuP5+Kp9sF5F/AR967t9DwSSMkOPQRDkoud2Gt3/azqvfbKJWTBSfPtqFtvUrWx2WUiXuoomyiDwEJAF1gUygM7AIuNG/oV2aPJebPJfR1gsVMK6ILc/M4XH837yNvLdgB8t2HmHMoLY01Bmi6uKGAM8Dn3nupwEPWheO/+jUi+Bz6MQZ/jo9k7QtWfS8uiav9L2GGIcupFehyZce5STgWmCXMeYGCiZgBNwOC95qhCbKKpBERdh4rndL3r63PbuP5HDrmHS+yNxndVgq8DUG6lFwjbYDNwEB9yleSfCO9NSKcnBYsDWLnilpLN1xhJfvaMW4we00SVYhzZfWi9PGmNMigohEGmM2ikgzv0d2ibzVCJ16oQLRLS1r0qpODElTMkiamsnCrYd5tncL7alX5/Mx8DdgLRDS41PCbWHYbWG6M1+Ay3e5eeO7LYz7YSuNY8vz4dCONK8ZkMuVlCpRvvyV3isilYDPgW9F5Ciwy79hXTrvaCGtKKtAVaeSg6nDOvP6d5t584dtrNh9lHGD29GsZgWrQ1OB55AxZrbVQZQWh93Gaa0oB6x9x5wkTclg+a6j3NmhLs/1bqlv8lWZcdHfdGPMHZ6bz4nIfCAG+MqvURVDjme0kHfUkFKBKNwWxhO3NKfLFdX487RMeo9N59nbWjKoYz2CYFS5Kj3Pisg7wPf8dgvrmdaF5D/Rdpu2XgSor9cd4MkZq8l3uUke2IY+bepYHZJSpcrXqRfdgKbGmPdFJBaoQ8H2qgHDqT3KKoh0a1qNeUnx/HV6Jn+ftYYFW7N4pd/VVIzSXj8FFCzcaw5E8GvrhQFCMlF22G3k6IYjAeVMvotXUjfywcKdtKpTkbGD2ulCZFUm+TL14lmgA9AMeJ+CC/dHQFf/hnZpcrRHWQWZ2AqRTHqw49kRS6v3HWPMoHa0qVfJ6tCU9a41xgTcWhB/cUTYdOpFANl+6CQJUzJYt/+4jrZUZZ4vUy/uAHoDpwCMMfuBgGuqPJso6xxlFUTCwoTHrm/M9Ee64HZD//ELmfDTNtxuY3VoyloLRaSF1UGUloLWC13MFwhmZezl1jHp7Dvm5J37OvDMbS00SVZlmi+Jcq4xxlDwsR8iEpCfvTg9K6a19UIFo/YNKpOaGM8fr6rBv1M3MmTSMg6fPHPxJ6pQ1RnIFJFNIrJaRNaIyGqrg/IXhz1cK8oWO3Umn8enr+Iv01bRqnYM85Li+WOLGlaHpZTlfOlRni4ibwOVRORhCgbhT/RvWJfu1znKuphPBaeY6AjG39OOj5bs5sU56+mRnMYbA9sQ17ia1aGp0tfd6gBKU3SEjZ81UbbM+v3HGTllJTuyTpF4YxMSb2pKuM2XOppSoc+XqRevisifgOMU9Ck/Y4z51u+RXSKdo6xCgYhwb+cGtK9fmZFTVnL3O0tIuLEpiTc20T9cZYgxJuBGcPpTtN12dsSnKj3GGD5avIsX526gkiOCjx/qpG/MlTqHT+VXT2IccMlxYTr1QoWSFrUrMiehG898sY6U77eweNthkge1oVaMw+rQlCpxDrsu5itt2Tl5PPXZar5ad4DrrozltTtbU618pNVhKRVwzluiEpETInK8iK8TInK8NIP0RU6eiwibEKFVNxUiou3hvDqgNa/f1Zq1+7PpkZzGd+sPWh2WUiVO5yiXrhW7jtIzJY3vNhzk7z2b8/4D12qSrNR5nDerNMZUMMZULOKrgjEm4PatdOa6iNKJFyoE3dG2LnMSulGnkoOHJi/nhdnrOZOvSYUKHY6IgtYLnfbiX2634c0ftnLn24sQgU8f7cKwPzQmLEw3O1LqfEKm/JqTm69tFypkXRFbnpnD43ggriHvLdhB//GL2Jl1yuqwlCoRDs8i7NP6BtBvDp04w/3vL+U/X22ie8uazE2Mp239ylaHpVTA81uiLCJRIrJURFaJyDoRed5f54KCqRc68UKFsshwG8/1bsmEe9uz+0gOt45J54vMfVaHpdRl8xY5tP3CP9K3ZNEzJY2lO47w7zuuZuzgtsQ4dBdQpXzhz4ryGeBGY0xroA3QXUQ6++tkzlyXbjaiyoSbW9YkNSme5jUrkDQ1kydnrNLNGhQAItLdM3t5q4g8XcT364vIfBHJ8Mxn7uk5HiEikzzzmjeIyKjSjNs7rUgX9JWsfJeb/369kXvfW0KMI4IvRnZlcKf6iGirhVK+8luibAqc9NyN8Hz5rQHNmefS1gtVZtSp5GDqsM6MvKEJn67YS++xC9h4IODW2KpSJCI2YBzQA2gBDCpid79/AtONMW2BgcCbnuMDgEhjzNVAe+AREWlYGnHDrxVlHRFXcvYdczJwwmLGzd/Gne3r8eXIrjSvGXDLi5QKeH7tURYRm4hkAr8A3xpjlvjrXDm5Lp2hrMqUcFsYf7ulGR8O6cSxnDz6jF3Ax0t2UbCRpiqDOgJbjTHbjTG5wFSgzzmPMYA3W4oB9hc6Xk5EwgEHkEvB7PxSoa0XJevrdQfomZzGxgMnSB7Yhv/X/xptTVSqmPyaKBtjXMaYNkBdoKOItDr3MSIyTESWi8jyQ4cOFftczlytKKuyqVvTasxLiqdjoyr8Y9ZaRn6SQbYzz+qwVOmrA+wpdH+v51hhzwH3iMheIBVI8ByfAZwCfgZ2A68aY474NdpCHBEFSZy2EF2e03kunvtyHY98uIL6VaKZk9CNPm3O/RVQSl2KUpl6YYw5BsyniG1ZjTETjDEdjDEdYmNji32OnLx87VFWZVZshUgmPdiRp7o356t1B+iVkkbG7qNWh6UCzyDgA2NMXaAn8KGIhFFQjXYBtYFGwOMickVRL1BSxY3CtEf58m0/dJK+by7kg4U7GdqtETMe60LDauWsDkupoOfPqRexIlLJc9sB/AnY6K/zOXNdZ0cMKVUWhYUJj13fmOmPdMEYGPDWIt7+cZvOpi079gH1Ct2v6zlW2FBgOoAxZhEQBVQDBgNfGWPyjDG/AAuADkWdpKSKG4Vp68XlmblyL7eOSWd/tpN37+/Av25tQWS4Fo6UKgn+rCjXAuaLyGpgGQU9ynP8dbIcbb1QCoD2DSqTmhjPH6+qwSvzNvLgB8vIOnnG6rCU/y0DmopIIxGxU7BY78tzHrMbuAlARK6iIFE+5Dl+o+d4OaAzfixsnMv7aaBWlC/NqTP5PD59FX+dvopWtWOYlxTPTVfVsDospUKK30qwxpjVQFt/vf4559KpF0oVEhMdwfh72vHRkt28OGc9PZPTeOOuNsQ1qWZ1aMpPjDH5IjIS+BqwAe8ZY9aJyAvAcmPMl8DjwEQR+QsFC/geMMYYERkHvC8i6wAB3vdcw0vFrxVl7VH21fr9xxk5ZSU7sk6ReFNTEm9sQrgtZPYQUypghESvwpl8N8agUy+UKkREuLdzA9rXr8zIKSu5+90lJNzQhMSbmuof1BBljEmlYJFe4WPPFLq9HuhaxPNOUjAizhLeiQzOPLdVIQQNYwwfLd7Fi3M3UMkRwccPdSKusb4BVspfQuKvpbevLVoX8yn1Oy1qV2ROQjf6tatLyv+2MnjiEn7OdlodllJnRUWEIQJOrShfUHZOHo99tJJ/fbGOuMZVmZcUr0myUn4WIolywcVV50QqVbRoezivDmjN63e1Zt3+bHokp/Hd+oNWh6UUUPDphyPCpov5LmDFrqP0TEnjuw0H+UfPq3jv/mupWj7S6rCUCnkhkSh7F4BEaeuFUhd0R9u6zE7oRp1KDh6avJznZ6/jTL4mJ8p6jggbOboz3++43YY3f9jKnW8vIiwMZjwWx8N/uIKwMN2GWqnSEBIlWG29UMp3V8SWZ+bwOF5J3cj7C3aybOcRxgxqRyOduaos5LDbdOrFOQ6dOMNfp2eStiWLXlfX4pV+V1MxKsLqsJQqU0Kionw2UdaKslI+iQy38Vzvlky4tz17jji5NSWNLzLPHbmrVOmJttt06kUh6Vuy6JGcxtIdR/j3HVczdnBbTZKVskBIJMrOvIKLq069UOrS3NyyJvOS4rmqVkWSpmby5IxVmqwoSzjs4dqjDOS73Pznq43c+94SKkVH8MXIrgzuVB8RbbVQygoh0XrhzC0YKaSL+ZS6dLUrOZg6rDPJ329h7PytrNx9jLGD29K8ZkWrQ1NlSHSEjdNlvEd53zEniVMyWLHrKHd1qMezvVvo3zWlLBYSFeVfp15oRVmp4gi3hfH4zc34aGgnsp159Bm7gI8W78IY3f5alY6C1ouymyh/ve4APd74iU0HTpA8sA3/r/81miQrFQBCIlF2eqoQ2nqh1OXp2qQaqYnxdLqiKv/8fC0jPllJtjPP6rBUGVBWF/OdznPx7BdreeTDFTSoWo45Cd3o06aO1WEppTxCIlH2ViEcOvVCqcsWWyGSDx64lqd7NOebdQfplZJGxu6jVoelQlxZnKO8/dBJ+r65kEmLdjG0WyM+eyyOhjp9RqmAoomyUup3wsKER69rzPRHuwAw4K1FvP3jNtxubcVQ/lHWpl7MXLmXW8ek83O2k3fv78C/bm2BPTwk/iQrFVJC4r9KZ24+URFhOoBdqRLWrn5l5ibG86cWNXhl3kYe/GAZWSfPWB2WCkEOe/jZNrpQdupMPn+dnslfp6+iVZ0YUpPiuemqGlaHpZQ6j5BIlHNyXbroQSk/iXFE8Obd7Xjp9lYs2n6YHslpLNyaZXVYKsRE223kuQx5LrfVofjNuv3Z3DYmnVkZ+0i6qSmfPNSJWjEOq8NSSl1ASCTKzjyXtl0o5Uciwj2dG/DFiK5UjArn7neX8No3m8gP4aRGlS7v1KJQrCobY5i8aCd3vLmQk2fy+eShzvzlT1cSbguJP8FKhbSQ+K/UmevS0XBKlYKralVkdkI3+rery5j/bWXQxMXsP+a0OiwVArxTi0Jt8kV2Th6PfrSCZ75YR1zjqsxLiqdL46pWh6WU8lFIJMo5migrVWqi7eH8d0Br3rirDev3H6dnShrfrj9odVgqyHmv4aE0+WLFriP0TEnj+w2/8I+eV/He/ddStXyk1WEppS5BSCTKzlwXUdp6oVSpur1tHeYkxlO3soOHJy/n+dnrOJMfOkmOKl3e9rlQmHzhdhvGzd/KnW8vJiwMZjwWx8N/uEIXnCsVhPyWKItIPRGZLyLrRWSdiCT561w5eflaUVbKAo2qleOzx+J4IK4h7y/YSb/xC9mRdcrqsFQQcngWZAd768WhE2e4//2l/PfrTXRvVZO5ifG0qVfJ6rCUUsXkz4pyPvC4MaYF0BkYISIt/HEinXqhlHUiw20817slE+/rwN6jTm5NSeOLzH1Wh6WCTCi0XqRtOUSP5DSW7jjCK32vZuygtlSMirA6LKXUZfBbomyM+dkYs9Jz+wSwAfDLvpzOXJduX62Uxf7UogapifG0qF2RpKmZPPHpqpD4GF2Vjl9bL4IvUc5zufnPVxu5772lVI6O4MuR3RjUsT4i2mqhVLArlTKsiDQE2gJL/PH6zjxdzKdUIKhdycGUhzuT/P0Wxs7fysrdRxk7uB1X1apodWgqwHmv4aeDbDzc3qM5JE3NZMWuowy8th7P3tZSCzdKhRC/L+YTkfLAZ8CfjTHHi/j+MBFZLiLLDx06VKxz5GhFWamAEW4L4/Gbm/Hx0E4cP51Pn3EL+GjxLozR7a/V+Xnb54KpovzV2gP0TE5j04ETpAxqy//1u0b/FikVYvyaKItIBAVJ8sfGmJlFPcYYM8EY08EY0yE2NvaSz+FyG3Lz3URHaI+yUoEkrkk15iXF0/mKqvzz87WM+GQl2c48q8NSAcphD56pF6fzXDzzxVoe/WgFDauVY25iN3q3rm11WEopP/Dn1AsB3gU2GGNG++s83ouqwx4Sk+6UCinVykfywQPXMqpHc75Zd5BeKWlk7D5qdVgqAHl7lAN96sW2Qye5482FTF60i6HdGjHj0TgaVC1ndVhKKT/xZ3bZFbgXuFFEMj1fPUv6JN6LqkOnXigVkMLChEeua8z0R7sAMOCtRbz14zbcbm3FUL+yh4cRHibkBHCP8mcr9nLbmHQOZDt59/4O/OvWFtjDtUijVCjzW3ZpjEkH/L7k19vPFq0bjigV0NrVr8zcxHhGzVzN/83byMJthxl9Z2uq6U5lysNhtwVkRfnUmXz+9flaZmbso2OjKiQPbEOtGIfVYSmlSkHQvxU+myjrAgqlAl6MI4Jxg9vx0u2tWLz9MD2S0yBOecUAABItSURBVFiwNcvqsFSAiLbbAq5Hed3+bG4bk87nmftIuqkpUx7urEmyUmVI0CfKzjxv64UmykoFAxHhns4N+GJEVypGhXPPu0t49etN5LvcVoemLBZtD8eZFxi/B8YYJi3cyR3jFnIqN5+PH+rMX/50JTbdhlqpMiX4E+WzFWXtUVYqmFxVqyKzE7rRv11dxs7fyqCJi9l/zGl1WMpCjggbzgCoKB/LyeWRD1fw7Jfr6NqkKqmJ8XRpXNXqsJRSFgj6RNn7MZ22XigVfKLt4fx3QGveuKsN6/cfp0dyGt+uP2h1WMoiBa0X1vYor9h1hF4p6czf9Av/7HUV795/LVW1j16pMivoE2Vv60WULuZTKmjd3rYOcxLjqVfFwcOTl/Pcl+s4kx94i7oCnYh0F5FNIrJVRJ4u4vv1RWS+iGSIyOrCk4hE5BoRWSQi60RkjYhElW70BS10ViXKbrdh3Pyt3Pn2YmxhwoxH43go/grCtNVCqTIt6BNlXcynVGhoVK0cnz0Wx4NdG/LBwp30fXMhO7JOWR1W0BARGzAO6AG0AAaJSItzHvZPYLoxpi0wEHjT89xw4CPgUWNMS+B6oNR3hylovSj9RPmXE6e5772l/PfrTf+/vTsPkqJM8zj+ffqCbsDmaAZQQMCLQR2OaTkbh3AcR1oZldhV0FlkVRAdBK8wPFbX3YiJQZ3YEFFH2JEhFpHxBlYEdCd0bDnltkFQQFAQoT0Amebobt79o97GaiyO7q6qrKz+fSIqKuutrMznoV4f3856M5PBF7TlzXFFdO/QPOlxiEjq0UBZRFJGo6xM/n3I+fz3iEJ27DnAlU+VMGvVjqDDCovewCbn3Bbn3GHgr8BVx6zjgNP8cj7wpV++DFjrnFsD4Jz7xjmX9BFrXk4m5RXJnaP8/idlFE8sYfm2b5kw9EImDe/JaY2zkxqDiKSu0A+UDxy9M58GyiLp4lfd2vDWuIF0O/007nxpNfe+siblLhuWgs4Avoh6vd23RXsU+K2ZbQfeAu7w7ecCzswWmNlKM7sv0cHGkpuTlbQjyhVVR3hs/gZGTF1Gi7wc5owtYljvjkRuKisiEhH+gXJFFZkZRk5m6FMRkSinN89l5qi+3HHJ2by2MnJHtI937gs6rLAbDkxzzrUHioHpZpZB5OZTRcAN/vkaM/tlrA2Y2WgzW25my8vKyuIaXF6Sbjiy/btyrpu8mD+9t5nhvTswZ2wR57ZplvD9ikj4hH50WX64irzsTB0FEElDWZkZ3HPZecy4uQ/7DlZy1TMLmb5kG87p9tcx7AA6RL1u79ui3Qy8DOCcWww0BgqIHH1+3zn3tXOunMjR5l6xduKcm+KcK3TOFbZu3TquCUSmXlQl9PudX7qT4oklfLJrP5OG9+QPQ3+mXyRF5LhCP1A+cLhKRU4kzfU/u4B54wfSt0srHp5Vyu0zVrL3QNLPNUt1HwLnmFlnM8shcrLenGPW+Rz4JYCZ/ZTIQLkMWABcaGZ5/sS+XwDrkxa5l5uTiXNwqDL+Nx05WFHFw7NKGfPCSjoVNGHuuCKGdD897vsRkfQS+oFyuQbKIg1CQdNGTBt5EQ8M7so763dRPLGElZ9/F3RYKcM5VwmMJTLo/ZjI1S3Wmdl/mtlv/Gr3AKPMbA0wExjpIr4D/ovIYHs1sNI5NzfZOeT6y3zG+xJxm8v2c82zi5i+ZBu3FHXm1TH9ObNVk7juQ0TSU+hvZ1d+uOpocRWR9JaRYdz6i7Po3bkld8xcxbXPLebeX5/HaF3vFgDn3FtEpk1Etz0StbweGHCcz75A5BJxgam+elH54UpaNsmJyzZfXbGdR2aX0igrg6kjC7mka5u4bFdEGobQH1E+UFGpS8OJNDA9O7Zg7riBXHZ+GybM28DIaR/y9f5DQYcl9ZSbEzl2E48T+vYfquRuf8WUC8/IZ974izVIFpFaC/1AufxwFXk5oT8wLiK1lJ+bzTPX9+L311zA0i3fMHhiCQs3fR10WFIPeXGaelG6Yy9DJn3ArNU7uPPSc3hxVF/a5if9RoMikgZCP1DWyXwiDZeZcUOfM5k9dgD5udn89vml/HHBRiqr4n8ymCRe9a+DByrqNlB2zjFt4WcMfXYR5YcreXFUX+689FwyNS1HROoo/APliipNvRBp4Lq2PY05Ywfwzz9vz9PvbmLYlCXs2HMg6LCklqoPetRl6sWe8sPcOn0Fj/7veorOKWDe+Ivp26VVvEMUkQYm9ANlncwnIgB5OVk8/k/dmTisBx/v3EfxxBLeXvdV0GFJLVRPo6vt1IvlW7+leGIJ727czb9d8VOev7EwbicDikjDFvqBsqZeiEi0q3qcwdxxA+nQMpfR01fw6Jx1HKpMzm2RpX5+uDzcqd2uvOqI45l3N3HdlCVkZWbw6pj+3DKwi25AJSJxk7CBsplNNbPdZlaaqH045yg/rKteiEhNnQqa8Npt/blpQGemLdrK0GcXsaVsf9BhyUnk1mKO8u7vDzJi6lKeWLCRwRe05c1xRXTv0DzRIYpIA5PII8rTgMsTuH0OVR7hiENXvRCRH2mUlckjQ7rx5xGF7NhzgCGTPuCNVduDDktO4IfrKJ94oPz+J2UUTyxhxbbvmDD0QiYN78lpjbOTEaKINDAJGyg7594Hvk3U9iFyS1JAc5RF5Lgu7daGeeMHcv7p+dz10hrueXkN/zh0aj/tS3Kd7M58FVVHmDBvAyOmLqNlkxzmjC1iWO+OmmohIgkT+BxlMxttZsvNbHlZWVmtPltdTDX1QkROpF1+Li+O6sO4S87m9VXbGfL0B6z/cl/QYckxMjKMxtkZRw+CRPvi23KunbyY5/6+meG9OzL7d0Wc26ZZAFGKSEMS+EDZOTfFOVfonCts3bp1rT5bPVDWyXwicjJZmRncfdl5zLilD/sPVnL1swuZvngrzrmgQ5MoeTlZPzqZb37pTq54qoRNu/YzaXhP/jD0QtV9EUmKwAfK9VF9rU1NvRCRU9X/rALeGj+Qfl1a8fDsddw+YyV7yyuCDku83OzMowdBDlZU8fCsUsa8sJLOBU2YO24gQ7qfHnCEItKQhHqgXH3UQSfziUhtFDRtxF9GXsSDxV15Z/0uip+KnBgmwcvNyeTA4So27d7P1c8sZPqSbYwa2JlXxvSnY6u8oMMTkQYmkZeHmwksBs4zs+1mdnO891FeoakXIlI3GRnG6IvP4pUx/TCDaycv5k/vbebIEU3FCFJeTibzSr9iyKQP2P39If4y8iIeuqIbOVmhPq4jIiGVsEOxzrnhidp2tQM6mU9E6qlnxxbMHTeQB1//iMfmb2DR5q/5842FNMpSXQnCvgORaTDdO+Tz5HU9aZvfOOCIRKQhC/WchZ80a8TgC9rqVqUiUi/5udk8fX1PBiwr4JNd32uQHKBHhnTj82/K+Zd+ncjM0GXfRCRYoR4oF3ZqSWGnlkGHISJpwMy4vk/HoMNo8C7p2iboEEREjtKkLxERERGRGDRQFhERERGJQQNlEREREZEYNFAWEREREYlBA2URERERkRg0UBYRERERiUEDZRERERGRGDRQFhERERGJwZxzQcdwlJmVAdtq+bEC4OsEhJNsyiO1KI/UEZYcznTOtQ46iGSqY82G8HynJ5IOOYDySDXKI3lOqWan1EC5LsxsuXOuMOg46kt5pBblkTrSIQepKR2+03TIAZRHqlEeqUdTL0REREREYtBAWUREREQkhnQYKE8JOoA4UR6pRXmkjnTIQWpKh+80HXIA5ZFqlEeKCf0cZRERERGRREiHI8oiIiIiInEX6oGymV1uZhvNbJOZ3R90PABmNtXMdptZaVRbSzN7x8w+9c8tfLuZ2VM+/rVm1ivqMzf69T81sxuj2n9uZh/5zzxlZpaAHDqY2btmtt7M1pnZ+JDm0djMlpnZGp/Hf/j2zma21O/7JTPL8e2N/OtN/v1OUdt6wLdvNLNfR7UnpQ+aWaaZrTKzN0Ocw1b/na82s+W+LVR9SuonWX2tNkw1O5XySJua7feluv3DdsJbt51zoXwAmcBmoAuQA6wBuqVAXBcDvYDSqLbHgfv98v3AY365GJgHGNAXWOrbWwJb/HMLv9zCv7fMr2v+s4MTkEM7oJdfbgZ8AnQLYR4GNPXL2cBSv8+XgWG+/TngNr98O/CcXx4GvOSXu/n+1Qjo7PtdZjL7IHA38CLwpn8dxhy2AgXHtIWqT+lRr+9fNVs1+2R5pE3N9nGobqdAv6r3v0HQAdTjy+sHLIh6/QDwQNBx+Vg6UbPobgTa+eV2wEa/PBkYfux6wHBgclT7ZN/WDtgQ1V5jvQTmMxv4VZjzAPKAlUAfIhdBzzq2HwELgH5+OcuvZ8f2rer1ktUHgfbA34BLgDd9TKHKwW97Kz8uuKHtU3rU+vtXzVbNrk0Ooa3Zftuq2ynYr+ryCPPUizOAL6Jeb/dtqaiNc26nX/4KaOOXj5fDidq3x2hPGP8TUE8if9mHLg//09dqYDfwDpG/wvc45ypj7PtovP79vUCrk+SRjD74JHAfcMS/bkX4cgBwwNtmtsLMRvu20PUpqTPVbNXsk0qTmg2q26cSbyjqdlbQATQ0zjlnZi7oOE6FmTUFXgPudM7ti546FJY8nHNVQA8zaw68AXQNOKRaMbMrgd3OuRVmNijoeOqpyDm3w8x+ArxjZhui3wxLn5KGJUz9UjU7Nahup5cwH1HeAXSIet3et6WiXWbWDsA/7/btx8vhRO3tY7THnZllEym4M5xzr/vm0OVRzTm3B3iXyE9Wzc2s+o/E6H0fjde/nw98Q+3zi6cBwG/MbCvwVyI/400MWQ4AOOd2+OfdRP4H2JsQ9ympNdVs1exTFuKaDarbpxpvOOp20HM/6vogcjR8C5EJ7tWT2c8POi4fWydqznd7gpoT3x/3y1dQc+L7Mt/eEviMyKT3Fn65pX/v2InvxQmI34D/AZ48pj1sebQGmvvlXKAEuBJ4hZonVNzul39HzRMqXvbL51PzhIotRE6mSGofBAbxw0khocoBaAI0i1peBFwetj6lR736gGq2avbJ8kirmu1jGYTqdqjrduAB1PNLLCZydu9m4KGg4/ExzQR2AhVE5tvcTGSu0d+AT4H/i+ogBjzj4/8IKIzazk3AJv/416j2QqDUf+Zp/E1j4pxDEZF5SWuB1f5RHMI8fgas8nmUAo/49i7+P85NvnA18u2N/etN/v0uUdt6yMe6kaizcpPZB6lZcEOVg493jX+sq95P2PqUHvXuB6rZqtknyiOtarbf3yBUtwPtV/V96M58IiIiIiIxhHmOsoiIiIhIwmigLCIiIiISgwbKIiIiIiIxaKAsIiIiIhKDBsoiIiIiIjFooCwpxcwW+edOZnZ9nLf9YKx9iYhI3ahmS7rT5eEkJfnbft7rnLuyFp/Jcs5VnuD9/c65pvGIT0REfqCaLelKR5QlpZjZfr84ARhoZqvN7C4zyzSzJ8zsQzNba2a3+vUHmVmJmc0B1vu2WWa2wszWmdlo3zYByPXbmxG9L4t4wsxKzewjM7suatvvmdmrZrbBzGaYmVVvz8zW+1j+mMx/IxGRVKGaLeku6+SriATifqKOTvjiudc5d5GZNQIWmtnbft1ewAXOuc/865ucc9+aWS7woZm95py738zGOud6xNjXUKAH0B0o8J9537/Xk8htRL8EFgIDzOxj4Bqgq3POmVnzuGcvIhIuqtmSlnREWcLiMmCEma0GlhK5heY5/r1lUQUXYJyZrQGWAB2i1jueImCmc67KObcL+DtwUdS2tzvnjhC5LWwnYC9wEHjezIYC5fXOTkQkvahmS1rQQFnCwoA7nHM9/KOzc6766MQ/jq4UmSd3KdDPOdcdWAU0rsd+D0UtVwHVc+p6A68CVwLz67F9EZF0pJotaUEDZUlV3wPNol4vAG4zs2wAMzvXzJrE+Fw+8J1zrtzMugJ9o96rqP78MUqA6/ycutbAxcCy4wVmZk2BfOfcW8BdRH7+ExFpyFSzJS1pjrKkqrVAlf85bhowkchPaCv9yRllwNUxPjcfGOPnpG0k8lNetSnAWjNb6Zy7Iar9DaAfsAZwwH3Oua980Y6lGTDbzBoTOWpyd91SFBFJG6rZkpZ0eTgRERERkRg09UJEREREJAYNlEVEREREYtBAWUREREQkBg2URURERERi0EBZRERERCQGDZRFRERERGLQQFlEREREJAYNlEVEREREYvh/9NqoPuffymQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.sched.plot_lr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.save('lm_malay_final')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.save_encoder('lm_malay_enc_final')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how our model did. Losses after 10 epochs:\n",
    "- Training: 3.185208\n",
    "- Validation: **3.377671**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.30244618263257"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(3.377671)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Analysis**:\n",
    "\n",
    "The **perplexity** of this language model after 10 epochs is **29.30245** (accuracy of 41.3643%).\n",
    "\n",
    "It took me ~1 hour 24 minutes (5016.09s) to train 1 epoch on one Tesla K80 GPU, roughly 1.1 iteration/s. The full training took me ~14 hours."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate text using the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We perform eyeballing test by having the model \"fill in the blanks\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load model from saved weights\n",
    "learner.load('lm_malay_final')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = learner.model # initialized model\n",
    "m.eval() # ensure you switch your model to inference mode\n",
    "m[0].bs = 1 # set batch size to 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60002"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the index-word mapping from the picked file on disk to help us convert the indexes to words.\n",
    "itos = pickle.load(open(LM_PATH / \"tmp\" / \"itos.pkl\", \"rb\"))\n",
    "\n",
    "# String to index lambda function\n",
    "stoi = collections.defaultdict(lambda: 0, {v:k for k, v in enumerate(itos)})\n",
    "len(itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility functions\n",
    "\n",
    "def gen_text(ss, topk):\n",
    "    s = ss.strip().split(\" \")\n",
    "    t = LongTensor([stoi[i] for i in s]).view(-1, 1).cuda()\n",
    "    t = Variable(t, volatile=False)\n",
    "    m.reset()\n",
    "    pred, *_ = m(t)\n",
    "    pred_i = torch.topk(pred[-1], topk)[1]\n",
    "\n",
    "    return [itos[o] for o in to_np(pred_i)]\n",
    "\n",
    "def generate_sentences(ss, nb_words):\n",
    "    result = []\n",
    "    s = ss.strip().split(\" \")\n",
    "    t = LongTensor([stoi[i] for i in s]).view(-1, 1).cuda()\n",
    "    t = Variable(t, volatile=False)\n",
    "    m.reset()\n",
    "    pred, *_ = m(t)\n",
    "    \n",
    "    for i in range(nb_words):\n",
    "        pred_i = pred[-1].topk(2)[1]\n",
    "        pred_i = pred_i[1] if pred_i.data[0] < 2 else pred_i[0]\n",
    "        word = itos[pred_i.data[0]]\n",
    "        if word != \"xbos\":\n",
    "            result.append(word)\n",
    "        else:\n",
    "            break\n",
    "        pred, *_ = m(pred_i[0].unsqueeze(0))\n",
    "    \n",
    "    result = re.sub('\\s+([.,])', r'\\1', \"{} {}\".format(ss, \" \".join(result).rstrip()))\n",
    "\n",
    "    return(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "strings = [\n",
    "    \"Menara Petronas\",\n",
    "    \"Dr Mahathir merupakan Perdana Menteri\",\n",
    "    \"Tunku ialah Bapa Kemerdekaan\",\n",
    "    \"Syarikat penerbangan\",\n",
    "    \"Durian ialah buah\",\n",
    "    \"P Ramlee ialah seorang\",\n",
    "    \"Pemenang badminton Piala Thomas\",\n",
    "    \"Lee Chong Wei dan badminton\",\n",
    "    \"Jurulatih Rashid Sidek\",\n",
    "    \"Pokok getah\",\n",
    "    \"Industri kelapa sawit di Malaysia\",\n",
    "    \"Penyelidikan minyak sawit\",\n",
    "    \"Negara terbesar di Asia Tenggara ialah\",\n",
    "    \"Proton Saga adalah\",\n",
    "    \"Penyanyi terkenal\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Menara Petronas... \n",
      "menara petronas. \n",
      " pada tahun 2005, sebuah syarikat yang dikenali sebagai t_up petronas, yang dimiliki oleh t_up petronas, telah membeli sebuah syarikat yang dimiliki oleh t_up petronas, t_up petronas. pada tahun 2008, t_up petronas membeli saham mereka daripada t_up petronas, dan t_up petronas, dan t_up petronas, dan t_up petronas, dan t_up petronas, dan t_up petronas, dan t_up petronas, yang telah membeli saham mereka pada tahun 2008. t_up\n",
      "\n",
      "Dr Mahathir merupakan Perdana Menteri... \n",
      "dr mahathir merupakan perdana menteri malaysia yang pertama. beliau juga merupakan seorang ahli parlimen bagi kawasan t_up dun t_up n53. beliau juga merupakan ahli parlimen bagi kawasan t_up dun t_up n53. beliau juga merupakan ahli parlimen bagi kawasan t_up dun t_up n53. beliau juga merupakan ahli parlimen bagi kawasan t_up dun t_up n53. beliau juga merupakan ahli parlimen bagi kawasan t_up dun t_up n53. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, negeri sembilan. beliau juga\n",
      "\n",
      "Tunku ialah Bapa Kemerdekaan... \n",
      "tunku ialah bapa kemerdekaan malaysia. beliau juga merupakan seorang ahli politik yang pernah berkhidmat sebagai perdana menteri malaysia ke-6. beliau juga merupakan seorang ahli parlimen bagi kawasan bukit besi, malaysia. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, pulau pinang. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, pulau pinang. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, pulau pinang. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, pulau\n",
      "\n",
      "Syarikat penerbangan... \n",
      "syarikat penerbangan ini. \n",
      " pada tahun 2005, sebuah syarikat penerbangan antarabangsa, t_up bae systems, telah ditubuhkan. pada tahun 2005, syarikat penerbangan ini telah melancarkan operasi untuk melancarkan pesawat pejuang generasi baru, \" t_up atr 72 \". pada tahun 2009, syarikat penerbangan ini telah melancarkan operasi untuk melancarkan pesawat pejuang generasi baru, \" t_up atr 72 \". pada tahun 2014, syarikat penerbangan ini telah melancarkan operasi untuk melancarkan pesawat pejuang generasi\n",
      "\n",
      "Durian ialah buah... \n",
      "durian ialah buah - buahan yang ditanam di dalam hutan. \n",
      "  xfld 1 1137.0\n",
      "\n",
      "P Ramlee ialah seorang... \n",
      "p ramlee ialah seorang ahli perniagaan yang berjaya. beliau juga merupakan seorang ahli perniagaan yang berjaya. beliau juga merupakan seorang ahli perniagaan yang berjaya. beliau juga merupakan seorang ahli perniagaan dan ahli perniagaan. beliau juga merupakan seorang ahli perniagaan dan ahli perniagaan. beliau juga merupakan seorang ahli perniagaan dan ahli perniagaan. beliau juga merupakan seorang ahli perniagaan dan ahli perniagaan. beliau juga merupakan seorang ahli perniagaan dan ahli perniagaan. beliau juga merupakan seorang ahli perniagaan dan\n",
      "\n",
      "Pemenang badminton Piala Thomas... \n",
      "pemenang badminton piala thomas, t_up pbb. \n",
      " pada tahun 2005, persatuan bola sepak malaysia ( t_up fam ) mengumumkan bahawa t_up fam akan menubuhkan persatuan bola sepak malaysia ( t_up fam ). pada tahun 2009, t_up fam mengumumkan bahawa t_up fam akan menyertai t_up aff pada tahun 2013. t_up fam juga telah mencadangkan bahawa t_up fam akan menyertai t_up aff pada tahun 2013. t_up fam juga telah bersetuju untuk menyertai t_up aff pada tahun 2014. t_up\n",
      "\n",
      "Lee Chong Wei dan badminton... \n",
      "lee chong wei dan badminton. \n",
      " pada tahun 2005, sebuah lagi acara sukan diadakan di stadium nasional bukit jalil, kuala lumpur. pada tahun 2009, persatuan bola sepak malaysia ( t_up fam ) telah mengadakan perlawanan persahabatan menentang t_up fam, t_up fam, t_up fam, t_up fam, t_up fam, t_up fam, t_up fam, t_up afc dan t_up fifa. pada tahun 2009, t_up fam mengumumkan bahawa mereka akan bertanding dalam piala malaysia pada tahun\n",
      "\n",
      "Jurulatih Rashid Sidek... \n",
      "jurulatih rashid sidek, seorang ahli politik yang telah berkhidmat sebagai perdana menteri malaysia. beliau juga merupakan seorang ahli parlimen bagi kawasan t_up dun t_up n53. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, negeri sembilan. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, negeri sembilan. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, negeri sembilan. beliau juga merupakan ahli parlimen bagi kawasan bukit gelugor, negeri sembilan. beliau juga merupakan\n",
      "\n",
      "Pokok getah... \n",
      "pokok getah. \n",
      " pada tahun 2011, sebuah lagi projek baru yang dikenali sebagai \" the new york times \" telah dilancarkan. pada tahun 2011, sebuah lagi projek baru yang dikenali sebagai \" the new york times \" telah dilancarkan. pada tahun 2013, sebuah siri \" the last airbender \" telah diterbitkan semula sebagai siri \" the last airbender \". pada tahun 2013, \" the new york times \" melaporkan bahawa \" the new york\n",
      "\n",
      "Industri kelapa sawit di Malaysia... \n",
      "industri kelapa sawit di malaysia. \n",
      " pada tahun 2005, sebuah syarikat yang dikenali sebagai t_up petronas, telah ditubuhkan untuk membangunkan dan memajukan t_up petronas sebagai sebuah syarikat yang bertanggungjawab untuk pembangunan dan pembangunan teknologi maklumat. t_up petronas telah ditubuhkan pada tahun 1990, dan pada tahun 1992, t_up petronas telah menjadi syarikat yang pertama untuk membangunkan dan menghasilkan produk - produk yang berkualiti. t_up petronas telah membeli dan membeli sebuah syarikat yang dikenali sebagai t_up petronas t_up petronas,\n",
      "\n",
      "Penyelidikan minyak sawit... \n",
      "penyelidikan minyak sawit, dan juga beberapa jenis produk yang berkaitan dengan industri. \n",
      "  xfld 1 1137.0\n",
      "\n",
      "Negara terbesar di Asia Tenggara ialah... \n",
      "negara terbesar di asia tenggara ialah t_up unesco. \n",
      " sejarah. \n",
      " pada tahun 2011, sebuah lagi muzium sejarah di indonesia, iaitu muzium negara indonesia, telah dibuka di jakarta, indonesia. muzium ini telah dirasmikan oleh t_up yab perdana menteri malaysia, tun dr. mahathir bin mohamad pada 1 jun 2007. muzium ini mempamerkan koleksi seni bina yang unik dan menarik. muzium ini mempamerkan koleksi seni bina yang unik dan menarik. muzium ini mempamerkan koleksi seni bina\n",
      "\n",
      "Proton Saga adalah... \n",
      "proton saga adalah sebuah kereta kebal utama yang digunakan oleh tentera udara diraja malaysia. xfld 1 1137.0\n",
      "\n",
      "Penyanyi terkenal... \n",
      "penyanyi terkenal, penyanyi, penyanyi, penulis lagu, komposer, komposer, komposer, komposer, dan artis. \n",
      " lagu. \n",
      " lagu ini digubah oleh komposer terkenal, komposer terkenal, komposer terkenal, ahmad nawab. lagu ini digubah oleh ahmad nawab, dan liriknya ditulis oleh ahmad nawab. lagu ini digubah oleh ahmad nawab, dan liriknya ditulis oleh ahmad nawab. lagu ini digubah oleh ahmad nawab, dan dinyanyikan oleh p. ramlee.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for s in strings:\n",
    "    print(f\"{s}... \\n{generate_sentences(s.lower(), 80)}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We extract the embedding layer of the encoder to be used in the same manner as word2vec. We can also create sentence vector by summing or averaging the vectors. For more details about word2vec use cases, see word2vec_examples.ipynb."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_weights = list(learner.model.named_parameters())[0][1]\n",
    "emb_np = to_np(emb_weights.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>390</th>\n",
       "      <th>391</th>\n",
       "      <th>392</th>\n",
       "      <th>393</th>\n",
       "      <th>394</th>\n",
       "      <th>395</th>\n",
       "      <th>396</th>\n",
       "      <th>397</th>\n",
       "      <th>398</th>\n",
       "      <th>399</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>semua</th>\n",
       "      <td>0.013038</td>\n",
       "      <td>0.515365</td>\n",
       "      <td>0.050658</td>\n",
       "      <td>0.403895</td>\n",
       "      <td>-0.076790</td>\n",
       "      <td>-0.274187</td>\n",
       "      <td>-0.294301</td>\n",
       "      <td>-0.482214</td>\n",
       "      <td>-0.942632</td>\n",
       "      <td>-0.382407</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000483</td>\n",
       "      <td>0.049201</td>\n",
       "      <td>-0.237344</td>\n",
       "      <td>-0.195282</td>\n",
       "      <td>0.318259</td>\n",
       "      <td>-0.085800</td>\n",
       "      <td>-0.082405</td>\n",
       "      <td>-0.008426</td>\n",
       "      <td>-0.548818</td>\n",
       "      <td>-0.450443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kanak</th>\n",
       "      <td>-0.261232</td>\n",
       "      <td>0.187345</td>\n",
       "      <td>0.054948</td>\n",
       "      <td>-0.388966</td>\n",
       "      <td>0.347622</td>\n",
       "      <td>0.906619</td>\n",
       "      <td>-0.085901</td>\n",
       "      <td>-0.067504</td>\n",
       "      <td>-0.227947</td>\n",
       "      <td>0.385188</td>\n",
       "      <td>...</td>\n",
       "      <td>0.133744</td>\n",
       "      <td>0.541352</td>\n",
       "      <td>-0.151185</td>\n",
       "      <td>0.144295</td>\n",
       "      <td>0.467201</td>\n",
       "      <td>0.060089</td>\n",
       "      <td>0.736018</td>\n",
       "      <td>0.356920</td>\n",
       "      <td>0.567058</td>\n",
       "      <td>-0.463647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>album</th>\n",
       "      <td>0.493681</td>\n",
       "      <td>-0.060911</td>\n",
       "      <td>-0.029997</td>\n",
       "      <td>0.046737</td>\n",
       "      <td>-0.580396</td>\n",
       "      <td>0.530397</td>\n",
       "      <td>-0.035759</td>\n",
       "      <td>0.317195</td>\n",
       "      <td>-0.029852</td>\n",
       "      <td>0.295080</td>\n",
       "      <td>...</td>\n",
       "      <td>0.263408</td>\n",
       "      <td>-0.229109</td>\n",
       "      <td>0.182056</td>\n",
       "      <td>-0.561614</td>\n",
       "      <td>-0.144355</td>\n",
       "      <td>-0.297672</td>\n",
       "      <td>-0.030824</td>\n",
       "      <td>0.462025</td>\n",
       "      <td>0.147016</td>\n",
       "      <td>0.053593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>menteri</th>\n",
       "      <td>-0.222996</td>\n",
       "      <td>-0.038149</td>\n",
       "      <td>0.061630</td>\n",
       "      <td>-0.323144</td>\n",
       "      <td>-0.102095</td>\n",
       "      <td>0.199310</td>\n",
       "      <td>-0.400978</td>\n",
       "      <td>-0.040864</td>\n",
       "      <td>-0.019013</td>\n",
       "      <td>-0.120582</td>\n",
       "      <td>...</td>\n",
       "      <td>0.412610</td>\n",
       "      <td>0.142238</td>\n",
       "      <td>-0.350635</td>\n",
       "      <td>0.059112</td>\n",
       "      <td>-0.300906</td>\n",
       "      <td>0.396439</td>\n",
       "      <td>0.035477</td>\n",
       "      <td>0.269761</td>\n",
       "      <td>-0.146999</td>\n",
       "      <td>-0.197764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>turut</th>\n",
       "      <td>-0.314148</td>\n",
       "      <td>-0.329103</td>\n",
       "      <td>0.084600</td>\n",
       "      <td>0.258733</td>\n",
       "      <td>-0.126275</td>\n",
       "      <td>-0.426062</td>\n",
       "      <td>-0.482648</td>\n",
       "      <td>-0.129092</td>\n",
       "      <td>0.201955</td>\n",
       "      <td>0.143144</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.096362</td>\n",
       "      <td>-0.009566</td>\n",
       "      <td>-0.645949</td>\n",
       "      <td>0.033635</td>\n",
       "      <td>-0.120077</td>\n",
       "      <td>0.654878</td>\n",
       "      <td>-0.348052</td>\n",
       "      <td>0.138207</td>\n",
       "      <td>-0.021835</td>\n",
       "      <td>-0.045728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kira</th>\n",
       "      <td>-0.989307</td>\n",
       "      <td>-0.055768</td>\n",
       "      <td>0.138539</td>\n",
       "      <td>-0.709911</td>\n",
       "      <td>-0.174012</td>\n",
       "      <td>-0.861263</td>\n",
       "      <td>-0.491223</td>\n",
       "      <td>0.044874</td>\n",
       "      <td>-0.403155</td>\n",
       "      <td>0.109385</td>\n",
       "      <td>...</td>\n",
       "      <td>0.157908</td>\n",
       "      <td>-0.044533</td>\n",
       "      <td>0.267241</td>\n",
       "      <td>-0.251010</td>\n",
       "      <td>-0.141131</td>\n",
       "      <td>-0.171218</td>\n",
       "      <td>0.155060</td>\n",
       "      <td>-0.295976</td>\n",
       "      <td>0.203567</td>\n",
       "      <td>-0.208005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>murid</th>\n",
       "      <td>0.248733</td>\n",
       "      <td>-0.059398</td>\n",
       "      <td>-0.023560</td>\n",
       "      <td>-0.273422</td>\n",
       "      <td>-0.298870</td>\n",
       "      <td>0.082709</td>\n",
       "      <td>-0.414381</td>\n",
       "      <td>0.048601</td>\n",
       "      <td>-0.398422</td>\n",
       "      <td>0.156399</td>\n",
       "      <td>...</td>\n",
       "      <td>0.035820</td>\n",
       "      <td>-0.110192</td>\n",
       "      <td>0.296525</td>\n",
       "      <td>0.256502</td>\n",
       "      <td>-0.831308</td>\n",
       "      <td>-0.100444</td>\n",
       "      <td>-0.141868</td>\n",
       "      <td>0.223650</td>\n",
       "      <td>-0.933487</td>\n",
       "      <td>0.428067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kuala</th>\n",
       "      <td>0.151282</td>\n",
       "      <td>-0.164024</td>\n",
       "      <td>-0.214619</td>\n",
       "      <td>-0.244270</td>\n",
       "      <td>-0.234031</td>\n",
       "      <td>-0.298997</td>\n",
       "      <td>0.021982</td>\n",
       "      <td>0.546422</td>\n",
       "      <td>0.604748</td>\n",
       "      <td>-0.260872</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.326803</td>\n",
       "      <td>0.072062</td>\n",
       "      <td>-0.017195</td>\n",
       "      <td>0.266519</td>\n",
       "      <td>0.331591</td>\n",
       "      <td>-0.068582</td>\n",
       "      <td>0.175815</td>\n",
       "      <td>0.104983</td>\n",
       "      <td>0.110367</td>\n",
       "      <td>0.049074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>semula</th>\n",
       "      <td>0.125103</td>\n",
       "      <td>-0.197420</td>\n",
       "      <td>0.751799</td>\n",
       "      <td>-0.990424</td>\n",
       "      <td>-0.583707</td>\n",
       "      <td>0.293708</td>\n",
       "      <td>-0.224855</td>\n",
       "      <td>-0.778633</td>\n",
       "      <td>-0.698253</td>\n",
       "      <td>-0.438961</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.231747</td>\n",
       "      <td>0.037555</td>\n",
       "      <td>0.303921</td>\n",
       "      <td>-0.279731</td>\n",
       "      <td>-0.914102</td>\n",
       "      <td>0.339698</td>\n",
       "      <td>0.337318</td>\n",
       "      <td>0.274428</td>\n",
       "      <td>-0.103200</td>\n",
       "      <td>0.126072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perempuan</th>\n",
       "      <td>0.344372</td>\n",
       "      <td>0.385741</td>\n",
       "      <td>-0.095603</td>\n",
       "      <td>0.161351</td>\n",
       "      <td>0.259355</td>\n",
       "      <td>0.608875</td>\n",
       "      <td>-0.228196</td>\n",
       "      <td>-0.217302</td>\n",
       "      <td>-0.557234</td>\n",
       "      <td>0.143539</td>\n",
       "      <td>...</td>\n",
       "      <td>0.383469</td>\n",
       "      <td>-0.132593</td>\n",
       "      <td>0.077143</td>\n",
       "      <td>0.672117</td>\n",
       "      <td>-0.278821</td>\n",
       "      <td>0.018481</td>\n",
       "      <td>-0.199514</td>\n",
       "      <td>0.894665</td>\n",
       "      <td>0.157668</td>\n",
       "      <td>0.372751</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 400 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                0         1         2         3         4         5    \\\n",
       "semua      0.013038  0.515365  0.050658  0.403895 -0.076790 -0.274187   \n",
       "kanak     -0.261232  0.187345  0.054948 -0.388966  0.347622  0.906619   \n",
       "album      0.493681 -0.060911 -0.029997  0.046737 -0.580396  0.530397   \n",
       "menteri   -0.222996 -0.038149  0.061630 -0.323144 -0.102095  0.199310   \n",
       "turut     -0.314148 -0.329103  0.084600  0.258733 -0.126275 -0.426062   \n",
       "kira      -0.989307 -0.055768  0.138539 -0.709911 -0.174012 -0.861263   \n",
       "murid      0.248733 -0.059398 -0.023560 -0.273422 -0.298870  0.082709   \n",
       "kuala      0.151282 -0.164024 -0.214619 -0.244270 -0.234031 -0.298997   \n",
       "semula     0.125103 -0.197420  0.751799 -0.990424 -0.583707  0.293708   \n",
       "perempuan  0.344372  0.385741 -0.095603  0.161351  0.259355  0.608875   \n",
       "\n",
       "                6         7         8         9      ...          390  \\\n",
       "semua     -0.294301 -0.482214 -0.942632 -0.382407    ...     0.000483   \n",
       "kanak     -0.085901 -0.067504 -0.227947  0.385188    ...     0.133744   \n",
       "album     -0.035759  0.317195 -0.029852  0.295080    ...     0.263408   \n",
       "menteri   -0.400978 -0.040864 -0.019013 -0.120582    ...     0.412610   \n",
       "turut     -0.482648 -0.129092  0.201955  0.143144    ...    -0.096362   \n",
       "kira      -0.491223  0.044874 -0.403155  0.109385    ...     0.157908   \n",
       "murid     -0.414381  0.048601 -0.398422  0.156399    ...     0.035820   \n",
       "kuala      0.021982  0.546422  0.604748 -0.260872    ...    -0.326803   \n",
       "semula    -0.224855 -0.778633 -0.698253 -0.438961    ...    -0.231747   \n",
       "perempuan -0.228196 -0.217302 -0.557234  0.143539    ...     0.383469   \n",
       "\n",
       "                391       392       393       394       395       396  \\\n",
       "semua      0.049201 -0.237344 -0.195282  0.318259 -0.085800 -0.082405   \n",
       "kanak      0.541352 -0.151185  0.144295  0.467201  0.060089  0.736018   \n",
       "album     -0.229109  0.182056 -0.561614 -0.144355 -0.297672 -0.030824   \n",
       "menteri    0.142238 -0.350635  0.059112 -0.300906  0.396439  0.035477   \n",
       "turut     -0.009566 -0.645949  0.033635 -0.120077  0.654878 -0.348052   \n",
       "kira      -0.044533  0.267241 -0.251010 -0.141131 -0.171218  0.155060   \n",
       "murid     -0.110192  0.296525  0.256502 -0.831308 -0.100444 -0.141868   \n",
       "kuala      0.072062 -0.017195  0.266519  0.331591 -0.068582  0.175815   \n",
       "semula     0.037555  0.303921 -0.279731 -0.914102  0.339698  0.337318   \n",
       "perempuan -0.132593  0.077143  0.672117 -0.278821  0.018481 -0.199514   \n",
       "\n",
       "                397       398       399  \n",
       "semua     -0.008426 -0.548818 -0.450443  \n",
       "kanak      0.356920  0.567058 -0.463647  \n",
       "album      0.462025  0.147016  0.053593  \n",
       "menteri    0.269761 -0.146999 -0.197764  \n",
       "turut      0.138207 -0.021835 -0.045728  \n",
       "kira      -0.295976  0.203567 -0.208005  \n",
       "murid      0.223650 -0.933487  0.428067  \n",
       "kuala      0.104983  0.110367  0.049074  \n",
       "semula     0.274428 -0.103200  0.126072  \n",
       "perempuan  0.894665  0.157668  0.372751  \n",
       "\n",
       "[10 rows x 400 columns]"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malay2vec = pd.DataFrame(emb_np)\n",
    "new_itos = itos\n",
    "# replace space with token\n",
    "new_itos[2] = '_space_'\n",
    "# replace space for named entities with _\n",
    "new_itos = [re.sub(' ', '_', i) for i in new_itos]\n",
    "malay2vec.index = new_itos\n",
    "malay2vec.iloc[200:210]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60002, 400)"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malay2vec.to_csv(f\"{LM_PATH}/malay2vec_embeddings.csv\", sep=\" \", header=False, line_terminator=\"\\n\")\n",
    "malay2vec.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from gensim.models import KeyedVectors\n",
    "\n",
    "# model = KeyedVectors.load_word2vec_format(f\"{LM_PATH}/malay2vec.csv\", binary=False,\n",
    "#                                           unicode_errors = \"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model.save_word2vec_format(f'{LM_PATH}/malay2vec.bin', None, True)"
   ]
  }
 ],
 "metadata": {
  "gist": {
   "data": {
    "description": "mywork/Telugu_Language_Model.ipynb",
    "public": false
   },
   "id": ""
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}