{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Neural Machine Translation" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "%reload_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Translation files" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from fastai.text import *" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Download dataset**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "French/English parallel texts from http://www.statmt.org/wmt15/translation-task.html . It was created by Chris Callison-Burch, who crawled millions of web pages and then used *a set of simple heuristics to transform French URLs onto English URLs (i.e. replacing \"fr\" with \"en\" and about 40 other hand-written rules), and assume that these documents are translations of each other*." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/ubuntu/data\n" ] } ], "source": [ "%cd data\n", "%mkdir translate" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "~20 minutes to download at 1.5 MB/s" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[#48abf9 32KiB/2.4GiB(0%) CN:5 DL:\u001b[32m73KiB\u001b[0m ETA:\u001b[33m9h34m4s\u001b[0m]\u001b[0m \n", "07/02 05:11:30 [\u001b[1;31mERROR\u001b[0m] CUID#10 - Download aborted. URI=http://www.statmt.org/wmt10/training-giga-fren.tar\n", "Exception: [AbstractCommand.cc:350] errorCode=29 URI=http://www.statmt.org/wmt10/training-giga-fren.tar\n", " -> [HttpSkipResponseCommand.cc:224] errorCode=29 The response status is not successful. status=503\n", "\n", "07/02 05:11:30 [\u001b[1;31mERROR\u001b[0m] CUID#8 - Download aborted. URI=http://www.statmt.org/wmt10/training-giga-fren.tar\n", "Exception: [AbstractCommand.cc:350] errorCode=29 URI=http://www.statmt.org/wmt10/training-giga-fren.tar\n", " -> [HttpSkipResponseCommand.cc:224] errorCode=29 The response status is not successful. status=503\n", "\n", "07/02 05:11:30 [\u001b[1;31mERROR\u001b[0m] CUID#7 - Download aborted. URI=http://www.statmt.org/wmt10/training-giga-fren.tar\n", "Exception: [AbstractCommand.cc:350] errorCode=29 URI=http://www.statmt.org/wmt10/training-giga-fren.tar\n", " -> [HttpSkipResponseCommand.cc:224] errorCode=29 The response status is not successful. status=503\n", " *** Download Progress Summary as of Mon Jul 2 05:12:30 2018 *** \n", "===============================================================================\n", "[#48abf9 95MiB/2.4GiB(3%) CN:2 DL:1.6MiB ETA:24m43s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:13:30 2018 *** \n", "===============================================================================\n", "[#48abf9 206MiB/2.4GiB(8%) CN:2 DL:1.9MiB ETA:19m40s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:14:31 2018 *** \n", "===============================================================================\n", "[#48abf9 322MiB/2.4GiB(13%) CN:2 DL:1.9MiB ETA:18m41s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:15:31 2018 *** \n", "===============================================================================\n", "[#48abf9 439MiB/2.4GiB(17%) CN:2 DL:1.9MiB ETA:17m40s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:16:32 2018 *** \n", "===============================================================================\n", "[#48abf9 576MiB/2.4GiB(23%) CN:2 DL:2.3MiB ETA:13m14s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:17:32 2018 *** \n", "===============================================================================\n", "[#48abf9 721MiB/2.4GiB(29%) CN:2 DL:2.3MiB ETA:12m12s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:18:33 2018 *** \n", "===============================================================================\n", "[#48abf9 866MiB/2.4GiB(34%) CN:2 DL:2.3MiB ETA:11m13s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:19:34 2018 *** \n", "===============================================================================\n", "[#48abf9 0.9GiB/2.4GiB(40%) CN:2 DL:2.3MiB ETA:10m12s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:20:34 2018 *** \n", "===============================================================================\n", "[#48abf9 1.1GiB/2.4GiB(46%) CN:2 DL:2.3MiB ETA:9m11s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:21:35 2018 *** \n", "===============================================================================\n", "[#48abf9 1.2GiB/2.4GiB(50%) CN:2 DL:1.2MiB ETA:15m59s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:22:35 2018 *** \n", "===============================================================================\n", "[#48abf9 1.2GiB/2.4GiB(53%) CN:2 DL:1.0MiB ETA:17m51s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:23:36 2018 *** \n", "===============================================================================\n", "[#48abf9 1.3GiB/2.4GiB(55%) CN:2 DL:917KiB ETA:20m17s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:24:37 2018 *** \n", "===============================================================================\n", "[#48abf9 1.4GiB/2.4GiB(58%) CN:2 DL:901KiB ETA:19m38s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:25:37 2018 *** \n", "===============================================================================\n", "[#48abf9 1.4GiB/2.4GiB(60%) CN:2 DL:901KiB ETA:18m38s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:26:38 2018 *** \n", "===============================================================================\n", "[#48abf9 1.5GiB/2.4GiB(62%) CN:2 DL:1.3MiB ETA:11m23s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:27:39 2018 *** \n", "===============================================================================\n", "[#48abf9 1.5GiB/2.4GiB(66%) CN:2 DL:1.3MiB ETA:10m7s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:28:40 2018 *** \n", "===============================================================================\n", "[#48abf9 1.6GiB/2.4GiB(70%) CN:2 DL:1.9MiB ETA:6m24s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:29:40 2018 *** \n", "===============================================================================\n", "[#48abf9 1.8GiB/2.4GiB(74%) CN:2 DL:1.9MiB ETA:5m24s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:30:41 2018 *** \n", "===============================================================================\n", "[#48abf9 1.9GiB/2.4GiB(79%) CN:2 DL:1.9MiB ETA:4m23s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:31:41 2018 *** \n", "===============================================================================\n", "[#48abf9 2.0GiB/2.4GiB(84%) CN:2 DL:1.8MiB ETA:3m29s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:32:42 2018 *** \n", "===============================================================================\n", "[#48abf9 2.1GiB/2.4GiB(88%) CN:2 DL:1.6MiB ETA:3m3s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " *** Download Progress Summary as of Mon Jul 2 05:33:42 2018 *** \n", "===============================================================================\n", "[#48abf9 2.2GiB/2.4GiB(92%) CN:2 DL:1.6MiB ETA:2m3s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:34:43 2018 *** \n", "===============================================================================\n", "[#48abf9 2.3GiB/2.4GiB(95%) CN:2 DL:1.5MiB ETA:1m5s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 05:35:44 2018 *** \n", "===============================================================================\n", "[#48abf9 2.4GiB/2.4GiB(99%) CN:2 DL:1.7MiB ETA:1s]\n", "FILE: /home/ubuntu/data/training-giga-fren.tar\n", "-------------------------------------------------------------------------------\n", "\n", "[#48abf9 2.4GiB/2.4GiB(99%) CN:2 DL:\u001b[32m1.7MiB\u001b[0m]\u001b[0m \n", "07/02 05:35:45 [\u001b[1;32mNOTICE\u001b[0m] Download complete: /home/ubuntu/data/training-giga-fren.tar\n", "\n", "Download Results:\n", "gid |stat|avg speed |path/URI\n", "======+====+===========+=======================================================\n", "48abf9|\u001b[1;32mOK\u001b[0m | 1.7MiB/s|/home/ubuntu/data/training-giga-fren.tar\n", "\n", "Status Legend:\n", "(OK):download completed.\n" ] } ], "source": [ "!aria2c --file-allocation=none -c -x 5 -s 5 http://www.statmt.org/wmt10/training-giga-fren.tar" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "!tar -xf training-giga-fren.tar" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "%mv giga-fren.release2.fixed.en.gz giga-fren.release2.fixed.fr.gz training-giga-fren.tar translate/" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/ubuntu/data/translate\n" ] } ], "source": [ "%cd translate/" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tar: This does not look like a tar archive\n", "tar: Skipping to next header\n", "tar: Exiting with failure status due to previous errors\n" ] } ], "source": [ "# Strange error\n", "!tar -xzf giga-fren.release2.fixed.en.gz" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# Resolve the previous issue\n", "!gunzip giga-fren.release2.fixed.en.gz" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "!gunzip giga-fren.release2.fixed.fr.gz" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/ubuntu\n" ] } ], "source": [ "%cd ../.." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Setup the directories and files**" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "PATH = Path('data/translate')\n", "TMP_PATH = PATH / 'tmp'\n", "TMP_PATH.mkdir(exist_ok=True)\n", "fname = 'giga-fren.release2.fixed'\n", "en_fname = PATH / f'{fname}.en'\n", "fr_fname = PATH / f'{fname}.fr'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tokenizing and Pre-processing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Training a neural model takes a long time\n", "\n", "- Google's model has 8 layers\n", "- we are going to build a simpler one\n", "- Instead of a general model we will translate French questions" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "# Question regex search filters\n", "re_eq = re.compile('^(Wh[^?.!]+\\?)')\n", "re_fq = re.compile('^([^?.!]+\\?)')\n", "\n", "# grabbing lines from the English and French source texts\n", "lines = ( (re_eq.search(eq), re_fq.search(fq))\n", " for eq, fq in zip(open(en_fname, encoding='utf-8'), open(fr_fname, encoding='utf-8')))\n", "\n", "# isolate the questions\n", "qs = [(e.group(), f.group()) for e, f in lines if e and f]" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "# save the questions for later\n", "pickle.dump(qs, (PATH / 'fr-en-qs.pkl').open('wb'))" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "# load in pickled questions\n", "qs = pickle.load((PATH / 'fr-en-qs.pkl').open('rb'))" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "52331\n", "[('What is light ?', 'Qu’est-ce que la lumière?'), ('Who are we?', 'Où sommes-nous?'), ('Where did we come from?', \"D'où venons-nous?\"), ('What would we do without it?', 'Que ferions-nous sans elle ?'), ('What is the absolute location (latitude and longitude) of Badger, Newfoundland and Labrador?', 'Quelle sont les coordonnées (latitude et longitude) de Badger, à Terre-Neuve-etLabrador?')]\n" ] } ], "source": [ "# ======================================== START DEBUG ========================================\n", "print(len(qs))\n", "print(qs[:5])\n", "# ======================================== END DEBUG ========================================" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[('x', 3), ('y', 4), ('z', 5)]\n", "('x', 'y', 'z')\n", "(3, 4, 5)\n" ] } ], "source": [ "# ======================================== START DEBUG ========================================\n", "# Python zip method: https://www.programiz.com/python-programming/methods/built-in/zip\n", "# What is iterable, iterator: https://stackoverflow.com/questions/9884132/what-exactly-are-iterator-iterable-and-iteration\n", "coord = ['x', 'y', 'z']\n", "value = [3, 4, 5, 0, 9]\n", "result = zip(coord, value)\n", "result_list = list(result)\n", "print(result_list)\n", "# unzip result_list\n", "c, v = zip(*result_list)\n", "print(c)\n", "print(v)\n", "# ======================================== END DEBUG ========================================" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tokenize all the questions." ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [], "source": [ "en_qs, fr_qs = zip(*qs)" ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [], "source": [ "en_tok = Tokenizer.proc_all_mp(partition_by_cores(en_qs))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "_Note: tokenizing for French is much different compared to english_" ] }, { "cell_type": "code", "execution_count": 68, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz\n", " Downloading https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz (39.8MB)\n", "\u001b[K 100% |████████████████████████████████| 39.8MB 71.0MB/s ta 0:00:01\n", "\u001b[?25hInstalling collected packages: fr-core-news-sm\n", " Running setup.py install for fr-core-news-sm ... \u001b[?25ldone\n", "\u001b[?25hSuccessfully installed fr-core-news-sm-2.0.0\n", "\u001b[33mYou are using pip version 9.0.3, however version 10.0.1 is available.\n", "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n", "\n", "\u001b[93m Linking successful\u001b[0m\n", " /home/ubuntu/anaconda3/envs/fastai/lib/python3.6/site-packages/fr_core_news_sm\n", " -->\n", " /home/ubuntu/anaconda3/envs/fastai/lib/python3.6/site-packages/spacy/data/fr\n", "\n", " You can now load the model via spacy.load('fr')\n", "\n" ] } ], "source": [ "# Download spaCy 'fr' model.Otherwise, you'll encounter errorr \"OSError: [E050] Can't find model 'fr'...\"\n", "!python -m spacy download fr" ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "scrolled": true }, "outputs": [], "source": [ "fr_tok = Tokenizer.proc_all_mp(partition_by_cores(fr_qs), 'fr')" ] }, { "cell_type": "code", "execution_count": 73, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "([['what', 'is', 'light', '?'],\n", " ['who', 'are', 'we', '?'],\n", " ['where', 'did', 'we', 'come', 'from', '?']],\n", " [['qu’', 'est', '-ce', 'que', 'la', 'lumière', '?'],\n", " ['où', 'sommes', '-', 'nous', '?'],\n", " [\"d'\", 'où', 'venons', '-', 'nous', '?']])" ] }, "execution_count": 73, "metadata": {}, "output_type": "execute_result" } ], "source": [ "en_tok[:3], fr_tok[:3]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check stats for the sentences length" ] }, { "cell_type": "code", "execution_count": 74, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(23.0, 28.0)" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 90th percentile of English and French sentences length.\n", "np.percentile([len(o) for o in en_tok], 90), np.percentile([len(o) for o in fr_tok], 90)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are keeping tokens that are less than 30 chars. The filter is applied on the English words, and the same tokens are kept for French." ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [], "source": [ "keep = np.array([len(o) < 30 for o in en_tok])" ] }, { "cell_type": "code", "execution_count": 85, "metadata": {}, "outputs": [], "source": [ "en_tok = np.array(en_tok)[keep]\n", "fr_tok = np.array(fr_tok)[keep]" ] }, { "cell_type": "code", "execution_count": 90, "metadata": {}, "outputs": [], "source": [ "# save our work\n", "pickle.dump(en_tok, (PATH / 'en_tok.pkl').open('wb'))\n", "pickle.dump(fr_tok, (PATH / 'fr_tok.pkl').open('wb'))" ] }, { "cell_type": "code", "execution_count": 101, "metadata": {}, "outputs": [], "source": [ "def toks2ids(tok, pre):\n", " \"\"\"\n", " Numericalize words to integers.\n", " \n", " Arguments:\n", " tok: token\n", " pre: prefix\n", " \"\"\"\n", " freq = Counter(p for o in tok for p in o)\n", " itos = [o for o, c in freq.most_common(40000)] # 40k most common words\n", " itos.insert(0, '_bos_')\n", " itos.insert(1, '_pad_')\n", " itos.insert(2, '_eos_')\n", " itos.insert(3, '_unk')\n", " stoi = collections.defaultdict(lambda: 3, { v: k for k, v in enumerate(itos) }) #reverse\n", " ids = np.array([ ([stoi[o] for o in p] + [2]) for p in tok ])\n", " np.save(TMP_PATH / f'{pre}_ids.npy', ids)\n", " pickle.dump(itos, open(TMP_PATH / f'{pre}_itos.pkl', 'wb'))\n", " return ids, itos, stoi" ] }, { "cell_type": "code", "execution_count": 102, "metadata": {}, "outputs": [], "source": [ "en_ids, en_itos, en_stoi = toks2ids(en_tok, 'en')\n", "fr_ids, fr_itos, fr_stoi = toks2ids(fr_tok, 'fr')" ] }, { "cell_type": "code", "execution_count": 110, "metadata": {}, "outputs": [], "source": [ "def load_ids(pre):\n", " ids = np.load(TMP_PATH / f'{pre}_ids.npy')\n", " itos = pickle.load(open(TMP_PATH / f'{pre}_itos.pkl', 'rb'))\n", " stoi = collections.defaultdict(lambda: 3, { v: k for k, v in enumerate(itos) })\n", " return ids, itos, stoi" ] }, { "cell_type": "code", "execution_count": 111, "metadata": {}, "outputs": [], "source": [ "en_ids, en_itos, en_stoi = load_ids('en')\n", "fr_ids, fr_itos, fr_stoi = load_ids('fr')" ] }, { "cell_type": "code", "execution_count": 116, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(['qu’', 'est', '-ce', 'que', 'la', 'lumière', '?', '_eos_'], 17573, 24793)" ] }, "execution_count": 116, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Sanity check\n", "[fr_itos[o] for o in fr_ids[0]], len(en_itos), len(fr_itos)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Word vectors" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Facebook's fasttext word vectors available from https://fasttext.cc/docs/en/english-vectors.html" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Download word vectors:\n", "\n", "We are using the pre-trained word vectors for English language, trained on Wikipedia using fastText. These vectors in dimension 300 were obtained using the skip-gram model: https://fasttext.cc/docs/en/pretrained-vectors.html" ] }, { "cell_type": "code", "execution_count": 118, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " *** Download Progress Summary as of Mon Jul 2 14:55:13 2018 *** \n", "===============================================================================\n", "[#fe81f7 2.9GiB/9.6GiB(30%) CN:5 DL:55MiB ETA:2m5s]\n", "FILE: data/translate/wiki.en.zip\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 14:56:14 2018 *** \n", "===============================================================================\n", "[#fe81f7 5.8GiB/9.6GiB(60%) CN:5 DL:52MiB ETA:1m14s]\n", "FILE: data/translate/wiki.en.zip\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 14:57:14 2018 *** \n", "===============================================================================\n", "[#fe81f7 8.6GiB/9.6GiB(89%) CN:5 DL:52MiB ETA:19s]\n", "FILE: data/translate/wiki.en.zip\n", "-------------------------------------------------------------------------------\n", "\n", "[#fe81f7 9.6GiB/9.6GiB(99%) CN:2 DL:\u001b[32m39MiB\u001b[0m]\u001b[0m \n", "07/02 14:57:39 [\u001b[1;32mNOTICE\u001b[0m] Download complete: data/translate/wiki.en.zip\n", "\n", "Download Results:\n", "gid |stat|avg speed |path/URI\n", "======+====+===========+=======================================================\n", "fe81f7|\u001b[1;32mOK\u001b[0m | 47MiB/s|data/translate/wiki.en.zip\n", "\n", "Status Legend:\n", "(OK):download completed.\n" ] } ], "source": [ "!aria2c --file-allocation=none -c -x 5 -s 5 -d data/translate https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.en.zip" ] }, { "cell_type": "code", "execution_count": 119, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " *** Download Progress Summary as of Mon Jul 2 15:01:29 2018 *** \n", "===============================================================================\n", "[#2e6d55 2.5GiB/5.5GiB(45%) CN:5 DL:34MiB ETA:1m29s]\n", "FILE: data/translate/wiki.fr.zip\n", "-------------------------------------------------------------------------------\n", "\n", " *** Download Progress Summary as of Mon Jul 2 15:02:29 2018 *** \n", "===============================================================================\n", "[#2e6d55 5.3GiB/5.5GiB(95%) CN:5 DL:42MiB ETA:5s]\n", "FILE: data/translate/wiki.fr.zip\n", "-------------------------------------------------------------------------------\n", "\n", "[#2e6d55 5.5GiB/5.5GiB(99%) CN:1 DL:\u001b[32m11MiB\u001b[0m]\u001b[0m \n", "07/02 15:02:47 [\u001b[1;32mNOTICE\u001b[0m] Download complete: data/translate/wiki.fr.zip\n", "\n", "Download Results:\n", "gid |stat|avg speed |path/URI\n", "======+====+===========+=======================================================\n", "2e6d55|\u001b[1;32mOK\u001b[0m | 41MiB/s|data/translate/wiki.fr.zip\n", "\n", "Status Legend:\n", "(OK):download completed.\n" ] } ], "source": [ "!aria2c --file-allocation=none -c -x 5 -s 5 -d data/translate https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.fr.zip" ] }, { "cell_type": "code", "execution_count": 123, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Archive: data/translate/wiki.en.zip\n", " inflating: data/translate/wiki.en.vec \n", " inflating: data/translate/wiki.en.bin \n" ] } ], "source": [ "!unzip data/translate/wiki.en.zip -d data/translate/" ] }, { "cell_type": "code", "execution_count": 124, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Archive: data/translate/wiki.fr.zip\n", " inflating: data/translate/wiki.fr.vec \n", " inflating: data/translate/wiki.fr.bin \n" ] } ], "source": [ "!unzip data/translate/wiki.fr.zip -d data/translate/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To use the fastText library, you'll need to download [fasttext word vectors](https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md) for your language (download the 'bin plus text' ones)." ] }, { "cell_type": "code", "execution_count": 125, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting git+https://github.com/facebookresearch/fastText.git\n", " Cloning https://github.com/facebookresearch/fastText.git to /tmp/pip-9tychtgg-build\n", "Collecting pybind11>=2.2 (from fasttext==0.8.22)\n", " Downloading https://files.pythonhosted.org/packages/12/90/0f92a575dc60c8fba6d0c91d6b45abdb1058da9ebed40400cbcfad2ac0a7/pybind11-2.2.3-py2.py3-none-any.whl (144kB)\n", "\u001b[K 100% |████████████████████████████████| 153kB 1.8MB/s ta 0:00:01\n", "\u001b[?25hRequirement already satisfied: setuptools>=0.7.0 in ./anaconda3/envs/fastai/lib/python3.6/site-packages (from fasttext==0.8.22)\n", "Requirement already satisfied: numpy in ./anaconda3/envs/fastai/lib/python3.6/site-packages (from fasttext==0.8.22)\n", "Installing collected packages: pybind11, fasttext\n", " Running setup.py install for fasttext ... \u001b[?25ldone\n", "\u001b[?25hSuccessfully installed fasttext-0.8.22 pybind11-2.2.3\n", "\u001b[33mYou are using pip version 9.0.3, however version 10.0.1 is available.\n", "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n" ] } ], "source": [ "!pip install git+https://github.com/facebookresearch/fastText.git" ] }, { "cell_type": "code", "execution_count": 126, "metadata": {}, "outputs": [], "source": [ "import fastText as ft" ] }, { "cell_type": "code", "execution_count": 127, "metadata": {}, "outputs": [], "source": [ "en_vecs = ft.load_model(str((PATH / 'wiki.en.bin')))" ] }, { "cell_type": "code", "execution_count": 128, "metadata": {}, "outputs": [], "source": [ "fr_vecs = ft.load_model(str((PATH / 'wiki.fr.bin')))" ] }, { "cell_type": "code", "execution_count": 130, "metadata": {}, "outputs": [], "source": [ "def get_vecs(lang, ft_vecs):\n", " \"\"\"\n", " Convert fastText word vectors into a standard Python dictionary to make it a bit easier to work with.\n", " This is just going through each word with a dictionary comprehension and save it as a pickle dictionary.\n", " \n", " get_word_vector:\n", " [method] get the vector representation of word.\n", " get_words:\n", " [method] get the entire list of words of the dictionary optionally\n", " including the frequency of the individual words. This\n", " does not include any subwords. \n", " \"\"\"\n", " vecd = { w: ft_vecs.get_word_vector(w) for w in ft_vecs.get_words() }\n", " pickle.dump(vecd, open(PATH / f'wiki.{lang}.pkl', 'wb'))\n", " return vecd" ] }, { "cell_type": "code", "execution_count": 131, "metadata": {}, "outputs": [], "source": [ "en_vecd = get_vecs('en', en_vecs)\n", "fr_vecd = get_vecs('fr', fr_vecs)" ] }, { "cell_type": "code", "execution_count": 132, "metadata": {}, "outputs": [], "source": [ "en_vecd = pickle.load(open(PATH / 'wiki.en.pkl', 'rb'))\n", "fr_vecd = pickle.load(open(PATH / 'wiki.fr.pkl', 'rb'))" ] }, { "cell_type": "code", "execution_count": 133, "metadata": {}, "outputs": [], "source": [ "# DEBUG\n", "ft_vecs = en_vecs" ] }, { "cell_type": "code", "execution_count": 136, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2519370" ] }, "execution_count": 136, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# DEBUG\n", "ft_words = ft_vecs.get_words(include_freq=True)\n", "ft_word_dict = { k: v for k, v in zip(*ft_words) }\n", "ft_words = sorted(ft_word_dict.keys(), key=lambda x: ft_word_dict[x])\n", "len(ft_words)" ] }, { "cell_type": "code", "execution_count": 149, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(300, 300)" ] }, "execution_count": 149, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dim_en_vec = len(en_vecd[','])\n", "dim_fr_vec = len(fr_vecd[','])\n", "dim_en_vec, dim_fr_vec" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Find out what the mean and standard deviation of our vectors are. So the mean is about zero and standard deviation is about 0.3." ] }, { "cell_type": "code", "execution_count": 179, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(0.0075652334, 0.29283327)" ] }, "execution_count": 179, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# en_vecd type is dict\n", "en_vecs = np.stack(list(en_vecd.values())) # convert dict_values to list and then stack it\n", "en_vecs.mean(), en_vecs.std()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exclude the extreme cases**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Often corpuses have a pretty long tailed distribution of sequence length and it's the longest sequences that tend to overwhelm how long things take, how much memory is used, etc. So in this case, we are going to grab 99th to 97th percentile of the English and French and truncate them to that amount. Originally Jeremy was using 90 percentiles (hence the variable name):" ] }, { "cell_type": "code", "execution_count": 157, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(29, 38)" ] }, "execution_count": 157, "metadata": {}, "output_type": "execute_result" } ], "source": [ "enlen_90 = int(np.percentile([len(o) for o in en_ids], 99))\n", "frlen_90 = int(np.percentile([len(o) for o in fr_ids], 99))\n", "enlen_90, frlen_90" ] }, { "cell_type": "code", "execution_count": 158, "metadata": {}, "outputs": [], "source": [ "en_ids_tr = np.array([o[:enlen_90] for o in en_ids])\n", "fr_ids_tr = np.array([o[:frlen_90] for o in fr_ids])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Create our Dataset, DataLoaders**" ] }, { "cell_type": "code", "execution_count": 159, "metadata": {}, "outputs": [], "source": [ "class Seq2SeqDataset(Dataset):\n", " def __init__(self, x, y):\n", " self.x, self.y = x, y\n", " \n", " def __getitem__(self, idx):\n", " return A(self.x[idx], self.y[idx]) # A for Arrays\n", " \n", " def __len__(self):\n", " return len(self.x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Split the training and testing set**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is an easy way to get training and validation sets. Grab a bunch of random numbers — one for each row of your data, and see if they are bigger than 0.1 or not. That gets you a list of booleans. Index into your array with that list of booleans to grab a training set, index into that array with the opposite of that list of booleans to get your validation set." ] }, { "cell_type": "code", "execution_count": 160, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(45219, 5041)" ] }, "execution_count": 160, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.random.seed(42)\n", "trn_keep = np.random.rand(len(en_ids_tr)) > 0.1\n", "en_trn, fr_trn = en_ids_tr[trn_keep], fr_ids_tr[trn_keep] # training set\n", "en_val, fr_val = en_ids_tr[~trn_keep], fr_ids_tr[~trn_keep] # validation set\n", "len(en_trn), len(en_val)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Create training and validation sets**" ] }, { "cell_type": "code", "execution_count": 162, "metadata": {}, "outputs": [], "source": [ "trn_ds = Seq2SeqDataset(fr_trn, en_trn)\n", "val_ds = Seq2SeqDataset(fr_val, en_val)" ] }, { "cell_type": "code", "execution_count": 163, "metadata": {}, "outputs": [], "source": [ "# Set batch size\n", "bs = 125" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Most of our preprocessing is complete, so making `numworkers = 1` will save you some time.\n", "- Padding will pad the shorter phrases to be the same length.\n", "- Classifier → padding in the beginning.\n", "- Decoder → padding at the end.\n", "- Sampler - so we keep the similar sentences together (sorted by length)." ] }, { "cell_type": "code", "execution_count": 165, "metadata": {}, "outputs": [], "source": [ "# arranges sentences so that similar lengths are close to each other\n", "trn_samp = SortishSampler(en_trn, key=lambda x: len(en_trn[x]), bs=bs)\n", "val_samp = SortSampler(en_val, key=lambda x: len(en_val[x]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Create DataLoaders**" ] }, { "cell_type": "code", "execution_count": 167, "metadata": {}, "outputs": [], "source": [ "trn_dl = DataLoader(trn_ds, bs, transpose=True, transpose_y=True, num_workers=1,\n", " pad_idx=1, pre_pad=False, sampler=trn_samp)\n", "val_dl = DataLoader(val_ds, int(bs * 1.6), transpose=True, transpose_y=True, num_workers=1,\n", " pad_idx=1, pre_pad=False, sampler=val_samp)\n", "md = ModelData(PATH, trn_dl, val_dl)" ] }, { "cell_type": "code", "execution_count": 169, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[(38, 29), (21, 7), (21, 8), (38, 13), (38, 21)]" ] }, "execution_count": 169, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Test - inspect\n", "\n", "it = iter(trn_dl) # trn_dl is iterable. turns iterable into iterator.\n", "# Return the next item from the iterator.\n", "its = [next(it) for i in range(5)]\n", "[(len(x), len(y)) for x, y in its]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Architecture" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Initial model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Architecture diagram](https://s15.postimg.cc/x710hbkdn/1_1f_KDa_Dsww_Vu3w2_Zt_Cg-_Uow.png)\n", "\n", "- The architecture is going to take our sequence of tokens.\n", "- It is going to spit them into an encoder (a.k.a. backbone).\n", "- That is going to spit out the final hidden state which for each sentence, it’s just a single vector.\n", "- Then, it will need to be passed to a decoder that will walk through the words one by one." ] }, { "cell_type": "code", "execution_count": 170, "metadata": {}, "outputs": [], "source": [ "def create_emb(vecs, itos, em_sz):\n", " \"\"\"\n", " Creates embedding:\n", " 1. rows = number of vocab\n", " 2. cols = embedding size dimension\n", " \n", " Will randomly initialize the embedding\n", " \"\"\"\n", " emb = nn.Embedding(len(itos), em_sz, padding_idx=1)\n", " wgts = emb.weight.data\n", " miss = []\n", " \n", " # goes through the embedding and replace\n", " # the initialized weights with existing word vectors\n", " # multiply x3 to compensate for the stdev 0.3\n", " for i, w in enumerate(itos):\n", " try:\n", " wgts[i] = torch.from_numpy(vecs[w] * 3)\n", " except:\n", " miss.append(w)\n", " print(len(miss), miss[5:10])\n", " return emb" ] }, { "cell_type": "code", "execution_count": 171, "metadata": {}, "outputs": [], "source": [ "nh, nl = 256, 2" ] }, { "cell_type": "code", "execution_count": 172, "metadata": {}, "outputs": [], "source": [ "class Seq2SeqRNN(nn.Module):\n", " def __init__(self, vecs_enc, itos_enc, em_sz_enc, vecs_dec, itos_dec, em_sz_dec, nh, out_sl, nl=2):\n", " super().__init__()\n", " \n", " # encoder (enc)\n", " self.nl, self.nh, self.out_sl = nl, nh, out_sl\n", " \n", " # for each word, pull up the 300M vector and create an embedding\n", " self.emb_enc = create_emb(vecs_enc, itos_enc, em_sz_enc)\n", " self.emb_enc_drop = nn.Dropout(0.15)\n", " \n", " # GRU - similiar to LSTM\n", " self.gru_enc = nn.GRU(em_sz_enc, nh, num_layers=nl, dropout=0.25)\n", " self.out_enc = nn.Linear(nh, em_sz_dec, bias=False)\n", " \n", " # decoder (dec)\n", " self.emb_dec = create_emb(vecs_dec, itos_dec, em_sz_dec)\n", " self.gru_dec = nn.GRU(em_sz_dec, em_sz_dec, num_layers=nl, dropout=0.1)\n", " self.out_drop = nn.Dropout(0.35)\n", " self.out = nn.Linear(em_sz_dec, len(itos_dec))\n", " self.out.weight.data = self.emb_dec.weight.data\n", " \n", " def forward(self, inp):\n", " sl, bs = inp.size()\n", "\n", " # ==================================================\n", " # Encoder version\n", " # ==================================================\n", " \n", " # initialize the hidden layer\n", " h = self.initHidden(bs)\n", " \n", " # run the input through our embeddings + apply dropout\n", " emb = self.emb_enc_drop(self.emb_enc(inp))\n", " \n", " # run it through the RNN layer\n", " enc_out, h = self.gru_enc(emb, h)\n", " \n", " # run the hidden state through our linear layer\n", " # note: we are only using the last hidden state to 'decode' into another phrase\n", " h = self.out_enc(h)\n", " \n", " # ==================================================\n", " # Decoder version\n", " # ==================================================\n", " \n", " # starting with a 0 (or beginning of string _BOS_)\n", " dec_inp = V(torch.zeros(bs).long())\n", " res = []\n", " \n", " # will loop as long as the longest english sentence\n", " for i in range(self.out_sl):\n", " \n", " # embedding - we are only looking at a section at time\n", " # which is why the .unsqueeze is required\n", " emb = self.emb_dec(dec_inp).unsqueeze(0)\n", " \n", " # rnn - typically works with whole phrases, but we passing\n", " # only 1 unit at a time in a loop\n", " outp, h = self.gru_dec(emb, h)\n", " \n", " # dropout\n", " outp = self.out(self.out_drop(outp[0]))\n", " \n", " res.append(outp)\n", " \n", " # highest probability word\n", " dec_inp = V(outp.data.max(1)[1])\n", " \n", " # if its padding ,we are at the end of the sentence\n", " if (dec_inp == 1).all():\n", " break\n", "\n", " # stack the output into a single tensor\n", " return torch.stack(res)\n", "\n", " def initHidden(self, bs):\n", " return V(torch.zeros(self.nl, bs, self.nh))\n", " " ] }, { "cell_type": "code", "execution_count": 173, "metadata": {}, "outputs": [], "source": [ "def seq2seq_loss(input, target):\n", " \"\"\"\n", " Loss function - modified version of cross entropy\n", " \"\"\"\n", " sl, bs = target.size()\n", " sl_in, bs_in, nc = input.size()\n", " \n", " # sequence length could be shorter than the original\n", " # need to add padding to even out the size\n", " if sl > sl_in:\n", " input = F.pad(input, (0, 0, 0, 0, 0, sl - sl_in))\n", " input = input[:sl]\n", " return F.cross_entropy(input.view(-1, nc), target.view(-1))#, ignore_index=1)" ] }, { "cell_type": "code", "execution_count": 174, "metadata": {}, "outputs": [], "source": [ "opt_fn = partial(optim.Adam, betas=(0.8, 0.99))" ] }, { "cell_type": "code", "execution_count": 175, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n", "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n" ] } ], "source": [ "rnn = Seq2SeqRNN(fr_vecd, fr_itos, dim_fr_vec, en_vecd, en_itos, dim_en_vec, nh, enlen_90)\n", "learn = RNN_Learner(md, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n", "learn.crit = seq2seq_loss" ] }, { "cell_type": "code", "execution_count": 176, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6e45ba3d891a43c8bc1cb451c9afa881", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " 65%|██████▍ | 235/362 [01:11<00:38, 3.28it/s, loss=29.5]" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEOCAYAAACaQSCZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VFX6+PHPkw5ptACphNB7MSAoIoiCYsEGyFqwomtZt7m6+v1tcb+2dXW/u6urorKWtQtYAEFkUUBpoffekhAIPbSQhOf3xwwa2QlMkrlzJ8nzfr3mlXvP3Dv3mcMwz5xz7r1HVBVjjDHmbMLcDsAYY0zNYAnDGGOMXyxhGGOM8YslDGOMMX6xhGGMMcYvljCMMcb4xbGEISLpIjJTRNaIyCoRedBb/gcRyRORpd7H0Ar2v1RE1onIRhF5xKk4jTHG+Eecug5DRJKBZFVdLCLxwCLgamAEcFhV/3KGfcOB9cAlQC6wEBilqqsdCdYYY8xZOdbCUNWdqrrYu1wErAFS/dy9N7BRVTer6gngfWCYM5EaY4zxR1DGMEQkE+gBzPcW3S8iy0VknIg09LFLKrCj3HouFSQbERkjIjnex5gAhm2MMaYcx7qkvj+ASBzwDfCEqk4QkWbAHkCBP+Hptrr9tH2GA0NU9U7v+s1Ab1V94EzHatKkiWZmZjrwLowxpnZatGjRHlVN8mfbCCcDEZFIYDzwjqpOAFDVXeWefxWY5GPXXCC93HoakH+242VmZpKTk1OtmI0xpi4RkW3+buvkWVICvA6sUdXny5Unl9vsGmClj90XAm1EpKWIRAE3AJ85Fasxxpizc7KFcT5wM7BCRJZ6yx4FRolIdzxdUluBuwFEJAV4TVWHqmqpiNwPTAPCgXGqusrBWI0xxpyFYwlDVecA4uOpKRVsnw8MLbc+paJtjTHGBJ9d6W2MMcYvljCMMcb4xRKGMcYYv1jCMMaYGmxtwSEWbdsXlGNZwjDGmBrs2anruOutRRwvKXP8WJYwjDGmhlpXUMSMtbsZ3TeTmMhwx49nCcMYY2qoV2Ztol5kOLf0bRGU41nCMMaYGuhwcSmfL8tnRHYaDWOjgnJMSxjGGFMDzdu0l5IyZUjn5kE7piUMY4ypgeZs3ENMZBg9M3zNEOEMSxjGGFMDfbtxD70yGwVlsPsUSxjGGFPD7Dp0nA27D9OvdZOgHtcShjHG1CC7Dx3nsYmeWSH6tQluwnB0AiVjjDGBo6rc+q+FbCw8zMOXtqdTSmJQj28Jwxhjaoj5W/axeuchnrmuCyN7ZQT9+NYlZYwxNcTbc7eRWC+Sq7qlunJ8SxjGGFMD5B04xrRVBQw/J416UcE7M6o8SxjGGFMDPDllDeFhwq3nZ7oWgyUMY4wJcd9t2sPk5Tu5d0Br0hrWdy0OSxjGGBPixs7aTPOEGO6+MMvVOCxhGGNMCNt7uJjZG/ZwdY/UoF7V7YtjCUNE0kVkpoisEZFVIvKgt/xZEVkrIstFZKKINKhg/60iskJElopIjlNxGmNMKJuyYidlJ5Vh3VPcDsXRFkYp8CtV7QD0Ae4TkY7AdKCzqnYF1gO/PcNrDFTV7qqa7WCcxhgTsj5Zmk/bZnG0bx7vdijOJQxV3amqi73LRcAaIFVVv1TVUu9m84A0p2Iwxpia7PNl+Szatp8R2emIiNvhBGcMQ0QygR7A/NOeuh34ooLdFPhSRBaJyJgzvPYYEckRkZzCwsJAhGuMMa4rOHicxyauoHt6A0afl+l2OEAQEoaIxAHjgZ+r6qFy5Y/h6bZ6p4Jdz1fVnsBleLqz+vvaSFXHqmq2qmYnJSUFOHpjjHHHuwu2U1RcyvMjuhEZHhrnJzkahYhE4kkW76jqhHLlo4ErgBtVVX3tq6r53r+7gYlAbydjNcaYUDJtZQG9MhuRlRTndijfc/IsKQFeB9ao6vPlyi8FHgauUtWjFewbKyLxp5aBwcBKp2I1xphQsrnwMOt2FXFpp+BNv+oPJ1sY5wM3Axd5T41dKiJDgReAeGC6t+xlABFJEZEp3n2bAXNEZBmwAJisqlMdjNUYY0LG1FUFAFwaxPm6/eHY7c1VdQ7ga1h/io+yU11QQ73Lm4FuTsVmjDGh6NiJMh6ftIrxi/LokdGAlAb13A7pR2w+DGOMCQEnTyo//2AJX67exU3ntuDega3cDum/WMIwxpgQ8NI3m5i2ahe/u6Ijt/dr6XY4PoXGuVrGGFOH7T9ygpe+3sTgjs24zcXbl5+NJQxjjHHZP7/eyJETpTw0pF1IXNFdEeuSMsYYlxQWFfPrj5bxzfpCruuZRptm7t8v6kwsYRhjjEuen76OuZv28shl7bk1RG7/cSaWMIwxxgW5+4/y8aJcRvXO4J4LQ++MKF9sDMMYY1zw4syNADUmWYAlDGOMCbqcrft4f+EObu6TGXIX552JJQxjjAmi4tIyfjN+OSmJ9fjV4LZuh1MpNoZhjDFB9Ma3W9lceIQ3butFbHTN+gq2FoYxxgTJnsPFvPCfjVzUvikD2jV1O5xKs4RhjDFB8tLXmzhaUsajQzu4HUqVWMIwxpggOHS8hPcXbOfKrsm0bho6kyJVhiUMY4wJgg8W7ODIiTLu6JfldihVZgnDGGMctmjbPl7+ZhPntmxEl7REt8OpMksYxhjjoO827WHEK/OIjY7g8WGd3Q6nWmrWOV3GGFODqCrPfLGW5gkxTPpZPxJiIt0OqVqshWGMMQ6ZsWY3y3IP8rNBrWt8sgBLGMYY45h/zNxIi8b1ubZnmtuhBIRjCUNE0kVkpoisEZFVIvKgt7yRiEwXkQ3evw0r2H+0d5sNIjLaqTiNMcYJS7bvZ9mOA9zRryWR4bXjt7mT76IU+JWqdgD6APeJSEfgEWCGqrYBZnjXf0REGgG/B84FegO/ryixGGNMKHrju63ER0fUmtYFOJgwVHWnqi72LhcBa4BUYBjwpnezN4Grfew+BJiuqvtUdT8wHbjUqViNMSaQ1u8qYvLynQzPTieuht0v6kyC0k4SkUygBzAfaKaqO8GTVABfN1RJBXaUW8/1lvl67TEikiMiOYWFhYEM2xhjKu3gsRLGvJVDw9go7rmw5l6k54vjCUNE4oDxwM9V9ZC/u/koU18bqupYVc1W1eykpKSqhmmMMQHx1+nryd1/jH/e2JOmCTFuhxNQjiYMEYnEkyzeUdUJ3uJdIpLsfT4Z2O1j11wgvdx6GpDvZKzGGFNdR4pLGb8olyu6JtMrs5Hb4QSck2dJCfA6sEZVny/31GfAqbOeRgOf+th9GjBYRBp6B7sHe8uMMSZkTVySR1FxKTf3zXQ7FEc42cI4H7gZuEhElnofQ4GngUtEZANwiXcdEckWkdcAVHUf8CdgoffxuLfMGGNCkqry9txtdEpJoGdGA7fDcYRjw/eqOgffYxEAg3xsnwPcWW59HDDOmeiMMSawFmzZx7pdRTxzXRc8HSy1T+24msQYY1z21rxtJNaL5KpuPk/orBUsYRhjTDXtPnScaSsLGH5OGvWiwt0OxzGWMIwxppo+zNlB6Unlpj4t3A7FUZYwjDGmGk6eVD7MyaVvVmMym8S6HY6jLGEYY0w1zNu8l+37jnJD7/Szb1zDWcIwxphqeGf+dhLrRTKkU3O3Q3GcJQxjjKmi1fmHmLxiJzeem0FMZO0d7D7FEoYxxlTRc1+uIyEmgrv7t3I7lKCwhGGMMVWw8+AxZqzdzR39skisX/OnX/WHJQxjjKmCjbsPA9C7Ze27yWBFLGEYY0wVbNlzBIBWSbX7VNryLGEYY0wVbC48QmxUOEnx0W6HEjSWMIwxpgo27zlCy6TYWnujQV8sYRhjTBVs2XOYlk3i3A4jqCxhGGNMJR0vKSN3/zGyavmtQE5nCcMYYypp+76jqEJWHRrwBksYxhhTaZsLPWdItbQWhjHGmDNZkXcAoNbfnfZ0ljCMMaYSdh48xr++3crFHZqREFM3rvA+xRKGMcZUwpNT1lJ2Uvn9lR3dDiXoIpx6YREZB1wB7FbVzt6yD4B23k0aAAdUtbuPfbcCRUAZUKqq2U7FaYwx/jpcXMoXK3Yy+rxM0hvVdzucoHMsYQBvAC8Ab50qUNWRp5ZF5Dng4Bn2H6iqexyLzhhjKmn+5r2UnlQGtW/qdiiucCxhqOosEcn09Zx4Lo0cAVzk1PGNMSbQ5mzcQ3REGD1bNHQ7FFe4NYZxAbBLVTdU8LwCX4rIIhEZE8S4jDGmQt9u3EPvlo3qxGRJvjjZJXUmo4D3zvD8+aqaLyJNgekislZVZ/na0JtQxgBkZGQEPlJjjAF2HzrO+l2Hua5nmtuhuCboLQwRiQCuBT6oaBtVzff+3Q1MBHqfYduxqpqtqtlJSUmBDtcYYwD461frARhYR8cvwJ0uqYuBtaqa6+tJEYkVkfhTy8BgYGUQ4zPGmB/5ZEke7y3Ywb0DWtG2Wbzb4bjGsYQhIu8Bc4F2IpIrInd4n7qB07qjRCRFRKZ4V5sBc0RkGbAAmKyqU52K0xhjzmRT4WEenbiCXpkN+eUlbd0Ox1VOniU1qoLyW32U5QNDvcubgW5OxWWMMf4qKTvJfe8sJiYynL+P6kFEeN2+1rluv3s8H4gvVxWwMu9Ml4QYY+qiKSt2sragiCeu7kxyYj23w3FdnU8YJ1X51YfLePO7rW6HYowJIarKuDlbaNkkliGdmrsdTkio8wkjOiKcQR2aMn3NLkrLTrodjjEmRCzefoBluQe57fxMwsLqzjSsZ1LnEwbApZ2bc+BoCQu27HM7FGNMiBg3ZwsJMRF1+rqL01nCAC5s25SYyDC+WFngdijGmBCQu/8oX6zcyajeGcRGu3V9c+ixhAHUiwpnQNumTF1VQIl1SxlT5709dxsiwi3nZbodSkixhOE1PDuNwqJiplorw5g66eiJUnbsO8rUlQWM+3YLl3VuTmoDOzOqPGtreQ1s15TMxvX517dbuLJbitvhGGOC6ORJ5dZxC1mw1TOO2T29AU9c08XlqEKPJQyvsDBh9HmZ/PHz1SzZvp8eGXXz9sXG1EXvzN/Ggq37uP38liQnxnBD73Ti69j0q/7wq0tKRB4UkQTxeF1EFovIYKeDC7bh2ek0rB/J89PXux2KMSZIDhw9wTNT13FBmyb8vys6cFf/LEsWFfB3DON2VT2E50aAScBtwNOOReWSuOgI7hvYmtkb9jBng032Z0xd8MHCHRwuLuWxyzvgmdvNVMTfhHGqFocC/1LVZeXKapWb+7YgtUE9npu+zu1QjDEOKy07yVtzt9EnqxHtmye4HU7I8zdhLBKRL/EkjGne24/XyvNPoyPCGdM/iyXbD7Bom13IZ0xtNmPtbvIOHOPW81q6HUqN4G/CuAN4BOilqkeBSDzdUrXS8Ow0EutF8uqsLW6HYoxx0NxNe6kXGc7FHerupEiV4W/C6AusU9UDInIT8D9Arb29a/2oCG48N4NpqwuYv3mv2+EYYxyyrqCIts3j6/xty/3lby29BBwVkW7Ab4BtwFuORRUC7r6wFS2bxHL3vxexdc8Rt8MxxgSYqrK24BDt6/AMepXlb8IoVVUFhgF/U9W/AbW6lhPrRfKvW3uhCr8ZvxzP2zfG1BaFRcXsP1pC++Ra/VUWUP4mjCIR+S1wMzBZRMLxjGPUai0ax/Lwpe1ZsGUfny3LdzscY0wArS0oAqBdc0sY/vI3YYwEivFcj1EApALPOhZVCBnZK53OqQn87+Q17C467nY4xpgAWedNGHY6rf/8ShjeJPEOkCgiVwDHVbVWj2GcEh4mPHt9N4qOl/DAu0tskiVjaok1BYdoGh9No9got0OpMfy9NcgIYAEwHBgBzBeR650MLJR0SE7gyWu6MH/LPh6buNLGM4ypBdYVFFl3VCX52yX1GJ5rMEar6i1Ab+D/nWkHERknIrtFZGW5sj+ISJ6ILPU+hlaw76Uisk5ENorII/6+GSdd2zONn13Umg9ydvDMVLsK3Jia7HhJGesKiuiUkuh2KDWKv3erDVPV3eXW93L2ZPMG8AL/ffrtX1X1LxXt5B1QfxG4BMgFForIZ6q62s9YHfOLS9qy98gJXv5mE41iIxnTv5XbIRljqmD1zkOUnlS6pzdwO5Qaxd+EMVVEpgHveddHAlPOtIOqzhKRzCrE1BvYqKqbAUTkfTyn87qeMESEx4d15sCxEp6cspaG9aMYnp3udljGmEpauv0AgCWMSvJ30PshYCzQFegGjFXVh6t4zPtFZLm3y8rXpBOpwI5y67neMp9EZIyI5IhITmFhYRVD8l94mPD8iG70a92ERyasYMaaXY4f0xgTWMtyD9A8IYbmiTFuh1Kj+H09vKqOV9VfquovVHViFY/3EtAK6A7sBJ7zsY2vu+BWOMqsqmNVNVtVs5OSkqoYVuVER4Tzys3n0DE5gZ+9t+T70/OMMTXDsh0H6JZu4xeVdcaEISJFInLIx6NIRA5V9mCquktVy1T1JPAqnu6n0+UC5ft50oCQu2ouNjqCV2/JJjY6gpten8/MdbvPvpMxxnUHjp5g696jdLPuqEo7Y8JQ1XhVTfDxiFfVSl/tIiLJ5VavAVb62Gwh0EZEWopIFHAD8FlljxUMzRNjeOuO3jSsH8lt/1rIwx8v59DxErfDMsacwbcbPTcU7ZFu0zBXlmO3aBSR94C5QDsRyRWRO4A/i8gKEVkODAR+4d02RUSmAKhqKXA/MA1YA3yoqqucirO62jdP4PMH+nHvgFZ8tGgHQ/46i1nrnR9LMcZUzfsLt5OSGEPvlo3cDqXGkdp0EVp2drbm5OS4dvylOw7wqw+XsnnPEZ6+tgsje2W4Fosx5r/t2HeU/s/O5MFBbfj5xW3dDickiMgiVc32Z1u7CXwAdU9vwOSfXUD/Nkk8PH4FT01Zw8Gj1kVlTKj4KGcHAoyw0+GrxBJGgMVEes6gGn5OGmNnb6b/szMZO2sTx0vK3A7NmDpvbUERbZrGk9Kgntuh1EiWMBwQExnOs8O7MfmBC+iR0YAnp6zl4ue/YdLyfLsPlTEuOnishMT6tX5mBsdYwnBQx5QE3ritN+/ceS5x0RHc/+4SrnvpO6atKqDspCUOY4Lt4LESEmIsYVSVJYwgOL91Eyb/7AKevrYLuw4Vc/fbi7joua/517dbOFxc6nZ4xtQZRcdLSaxnCaOqLGEESXiYcEPvDL55aAAv/qQnjWOj+OPnq+n71AyemrKGXYdsciZjnHbwWIkljGrw9+aDJkAiwsO4vGsyl3dNZvH2/bw+Zwuvzt7M2/O2cf9FrbmjX0uiI8LdDtOYWqe07CSHi0tJqGdfe1VlLQwX9cxoyIs/6cnMXw+gX+sm/HnqOgb/dZbd0NAYBxw67un+tRZG1VnCCAEtGscy9pZs3rq9N+Fhwh1v5vCTV+cxffUuGxw3JkAOHfNcE2UJo+osYYSQ/m2TmPpgf/7n8g5sLjzCXW/l0P/PM3nj2y2cKLW5xI2pjoOWMKrNOvNCTFREGHdekMXo8zL5avUu/vXtVv7w+Wpem7OFa3qkcmW3FNo2s3mIjamsUwkjwRJGlVnCCFGR4WFc1iWZSzs35+t1hbw2ZzMvztzIP/6zkXbN4rmyWzJXdE0hs0ms26EaUyNYC6P6LGGEOBFhYPumDGzflMKiYr5YuZPPl+Xzly/X85cv19M1LZEru6Zweddku92BMWdwauoBSxhVZwmjBkmKj+aWvpnc0jeT/APHmLx8J58vz+eJKWt4YsoaemU25MpuKVzWOZmk+Gi3wzUmpHzfJWVXeleZJYwaKqVBPe7qn8Vd/bPYuucIk5bn8/mynfzu01X88fPVDGzXlJ+cm86FbZsSHuZr1ltj6paDx0qICg8jJtLO9akqSxi1QGaTWO6/qA33X9SGdQVFTFySx8eLdvDVml2kJMZwY58WjOqdQaPYKLdDNcY1h46VkFAvEhH7AVVVlmprmXbN43nksvZ898ggXrqxJy2TYnl22jr6PDWD33y8jFX5B90O0RhXHDpWSqJd5V0tVnu1VFSE5yyry7oks66giDfnbmXi4jw+zMmld2YjRp+XyZBOzYgIt98Mpm446G1hmKqzhFEHtGsez5PXdOHhIe35aNEO3py7lfveXUzzhBhu6J3OqN4ZNEuIcTtMYxx18FgJjeOsW7Y6LGHUIYn1I7nzgixuO78l/1m7m7fnbeP/vtrAP/6zkcEdm3FTnxac16qx9fGaWunQ8RKykuy6pepwLGGIyDjgCmC3qnb2lj0LXAmcADYBt6nqAR/7bgWKgDKg1N8Jyo1/wsOESzo245KOzdi65wjvLtjORzk7+GJlAVlNYhnVO4NreqbSJM5OzTW1h02eVH1OdmC/AVx6Wtl0oLOqdgXWA789w/4DVbW7JQtnZTaJ5dGhHZj720E8P6IbDWOjeGLKGvo8OYOf/nsRM9ftthsgmhrv5EnlkM2FUW2OtTBUdZaIZJ5W9mW51XnA9U4d31ROTGQ41/ZM49qeaWzYVcQHC3cwYUkeX6wsIDkxhuHnpDE8O530RvXdDtWYSjtaUsZJhfgY64WvDjdPkbkd+KKC5xT4UkQWiciYM72IiIwRkRwRySksLAx4kHVRm2bx/M8VHZn320H888aetG0Wzz9mbuSCP8/kptfm88mSPA4cPeF2mMb4rbikDPD8MDJV50q6FZHHgFLgnQo2OV9V80WkKTBdRNaq6ixfG6rqWGAsQHZ2tvWdBFBURBhDuyQztEsyeQeO8XFOLh/m7ODnHywlPEw4J6MhV/dIZXh2GpF2eq4JYSfKPNMDREXY57Q6gl57IjIaz2D4jarq8wteVfO9f3cDE4HewYvQ+JLaoB4PXtyG2b8ZyIR7z+PeAa04dLyERyeuYNBz3/DJkjxO2liHCVHFJZ6EEW0Jo1qCWnsicinwMHCVqh6tYJtYEYk/tQwMBlYGL0pzJmFhQs+MhvxqcDu+ePACxt2aTWx0BD//YClD/z6br1bvooLfAca4xloYgeFY7YnIe8BcoJ2I5IrIHcALQDyebqalIvKyd9sUEZni3bUZMEdElgELgMmqOtWpOE3ViQgXtW/G5Af68bcbunO8pIw738rhupe+s+llTUj5oYVhYxjV4eRZUqN8FL9ewbb5wFDv8magm1NxmcALCxOGdU9laJdkPl6Uyz9mbOCut3Jo0bg+N53bgnOzGtG+eYL9ujOuOVHmGfS2z2D12DlmJmAiw8MY1TuD689JY9qqAsbN2cITU9YAEBUexrlZjfjlJW3pkdHQ5UhNXWNjGIFhCcMEXGR4GFd0TeGKrins2HeU5bkHWZZ7gAmLc7nmn9+R3aIhI3ulc3nXZOpH2UfQOK+4zBJGIFjtGUelN6rP5V2TeXRoB75+aCCPDm3PvqMneOjj5fR5cgYfLtxhg+TGcadaGNYlVT1WeyZo4qIjGNO/FTN+eSEf3dOX9skJ/Gb8ci7722w+WLid496Lq4wJtOJSz2fLBr2rxxKGCToRoVdmI96/qw9/Ge45v+Hh8Ss47+n/MHbWJkscJuBOlFqXVCBYB7JxTViYcP05aVzXM5V5m/fx0jebeHLKWl6cuYkruiZzbc80emY0sNutm2ortoQREJYwjOtEhL6tGtO3VWPmbtrL+wu3M35xLu/M305m4/pc1zONkb3TaRpvkzyZqjnVwrAxjOqxhGFCyqnEUXS8hKkrC5iwOI/npq/n7//ZwGWdk7m5bwuyWzS0VoeplB9aGDaGUR2WMExIio+JZHh2OsOz09lceJi3523j45xcPluWT5umcYzqncF1PdNIrG/zG5izsxZGYFjCMCEvKymO31/ZiYeGtGPSsp28s2A7j09azTNT1zKoQ1P6tmpC36xGtEqKs5aH8am4tIyIMCE8zD4f1WEJw9QY9aMiGNErnRG90lmVf5D3F+xg+updTFlRAECrpFhuPLcFgzo0pUVjm7vZ/OBE6Ukb8A4ASximRuqUksifrk7k8WGd2L7vKHM27uHDhTt4fNJqHp+0mo7JCYzslc6I7HTqRVm/dV1XXHrSuqMCwBKGqdFEhBaNY2nR2NO62Fx4mJnrCvl0aR6//2wVf5uxgVvPy2R030wb76jDikvLbMA7ACxhmFolKymOrKQ47ujXkoVb9/HS15t4fvp6XvlmEz85N4M7+mXRPNFOz61rTlgLIyAsYZhaq1dmI3rd2og1Ow/xyjebGPftVt78bhvX9kxlTP8sspLi3A7RBEmxjWEEhNWgqfU6JCfwfzf0YOavBjCyVzoTluQx6PlvuO+dxew+dNzt8EwQnCg9SXSkfd1Vl9WgqTMyGtfnT1d35tuHL+KnF7ZixtpdDP37HKauLLD5yGu54tKTRIXb1111WQ2aOicpPprfXNqeT+/rR4P6kdzz70UM/ftsPl+Wb9PK1lKe02pt0Lu6LGGYOqtd83imPngBfx3ZjdKTygPvLWHYi3NYuHWf26GZACsuLbNB7wCwGjR1WkR4GNf0SOPLn/fnbzd0Z+/hEwx/eS73v7uYjbuL3A7PBIgNegeGnSVlDJ5brQ/rnsolHZvxyjebeWXWJiav2MmQjs25d2AruqY1cDtEUw12Wm1gOFqDIjJORHaLyMpyZY1EZLqIbPD+bVjBvqO922wQkdFOxmnMKfWjIvjFJW359uGLuH9ga77btIerXviWm1+fz9xNe2062Rqq2MYwAsLplPsGcOlpZY8AM1S1DTDDu/4jItII+D1wLtAb+H1FicUYJzSOi+ZXg9vx7SMX8chl7Vmzs4hRr87j2pe+46vVuyxx1DDFdlptQDhag6o6Czh9BHEY8KZ3+U3gah+7DgGmq+o+Vd0PTOe/E48xjouPieSeC1sx5+GB/GlYJwqLirnzrRyufvFb5mzY43Z4xk/FpWV2Wm0AuFGDzVR1J4D3b1Mf26QCO8qt53rL/ouIjBGRHBHJKSwsDHiwxgDERIZzc99MZv56AM9c14XComJuen0+N742jxW5B90Oz5yFXbgXGKFag75uWu+zD0BVx6pqtqpmJyUlORyWqesiw8MY2SuDmQ8N4HdXdGTNziKufGEOD76/hHUFRdZVFYJU1dMlZS2ManPjLKldIpKsqjtFJBnY7WObXGBAufU04OsgxGaMX6Ijwrm9X0uuz07jlW828dpXXUjWAAAQyUlEQVTsLXy6NJ8mcdH0yWrEpZ2bc0nHZjbQGgJKyjxJPDrS/i2qy42E8RkwGnja+/dTH9tMA54sN9A9GPhtcMIzxn8JMZE8NKQ9o/tmMnPdbuZu2su3m/YyaflOGsVGcU2PVLqkJtI+OZ52zeJtRkAXFJeWAdgYRgA4mjBE5D08LYUmIpKL58ynp4EPReQOYDsw3LttNnCPqt6pqvtE5E/AQu9LPa6qdvmtCVlNE2IY2SuDkb0yKDupzN5QyPsLdvDmd1sp9d5uJCk+mn6tm9ApJYGWTWK/f1gScdap+bxtDKP6HE0YqjqqgqcG+dg2B7iz3Po4YJxDoRnjmPAwYUC7pgxo15RjJ8rIO3CUxdsPMHvDHmatL2Tikrzvt+2UksDovplc1T2FGOsycUTxqYRhF+5Vm13pbYyD6kWF07ppPK2bxjMiOx1V5cDRErbsPcLKvIO8M287vxm/nP+dvJqL2jdlRHY6fVs1tlZHAJ1KGHald/VZwjAmiESEhrFRNIyNomdGQ27u04L5W/bxUU4uM9bu4pOl+XROTeDu/q0Y2iWZ8DBLHNX1fZeUnYBQbZYwjHGRiNAnqzF9shpzvKSMiUvyeHXWZh54bwkvfb2JEdlpNIyNokH9KGKjwmmWEEN6o/puh12j2KB34FjCMCZExESGM6p3BiOz05m0YifPfLGWP3y++r+2S21Qj1ZN42jbNI7szIZc0CaJ2Gj7r1wRG/QOHPuUGRNiwsKEq7qlcHmXZPYfPcGBoyUcPHaCoyfK2Fx4hAVb97Ft7xHemreX1+ZsISoijP5tmjCkU3Mu7tCMhrFRbr+FkPL9GIa1MKrNEoYxISo8TGgSF02TuOjvyy5ok8To8zIBzy/nxdv3M21VAdNWFvDVGs81sHHRETSMjaRRfc9YSaeUBPpkNeacFg2pH1X3/sv/0MKwMYzqqnufHmNqiaiIsO/HP353RUdW5B1k9oY97DlczIGjJew7coJdh4qZvWEzL87cRHiY0Dopjk4pCbRrHk98TCSTV+RTcPA4nVMT6ZKaSNe0BnROTahVieXUGIadVlt9tedTYUwdJiJ0TWvgc6Knw8WlLNq2n5yt+1iZd5A5G/cwwXstSEpiDB1TEpi/eR+fLs0HICYyjKu6pXBzn0y6pCUG9X04wU6rDRxLGMbUcnHREVzYNokL2/5wc86Dx0rYe7iYjEb1ifD27e8+dJwVeQf5as0uPlmSz4c5uXRKSeCqbilc0S2F1Ab13HoL1WIX7gWOJQxj6qDEepEk1ov8UVnThBgGJcQwqEMzfju0AxMX5zFhSR5PfbGWp75YS3aLhlzVPYVh3VJJrB9ZwSuHHmthBI4lDGPMf0mIiWT0eZmMPi+TbXuPMGn5Tj5bms/vPl3FE5PXcHmXZG7onUGvzIYhf1V6ccmpMQwb9K4uSxjGmDNq0TiW+wa25r6BrVmZd5D3Fmzn06X5TFiSR6ukWG7olcF156TRKERP59116DgxkWEkxNjXXXVZG80Y47fOqYk8cU0XFjw2iD9f35XEepE8MWUNfZ+awUMfLWNlXujNPph/4DgpDeqFfEuoJrCUa4yptPpREYzITmdEdjprCw7x9txtTFicx0eLcumQnMDwc9IY2Ss9JK5AzztwjJTEmjlgH2qshWGMqZb2zRN44pouzHt0EH+8qhNREWE8Pmk1fZ+awbPT1rK76Lir8eUfOEZKgxhXY6gt3E//xphaIbHeDwPlS7bvZ+yszfzz6028OmsL1/ZM5a7+WbRKigtqTMWlZewuKialhp4SHGosYRhjAq5HRkNeuukctuw5wmuzN/PxolzeX7iDSzo2Y1TvdLIzG5EQ4/ypubsOFgNYwggQSxjGGMe0bBLLE9d04ReXtOWt77by1rxtTF+9C/BcZT64U3Pu6NfSsVu25x04BlBjLzoMNZYwjDGOaxIXzS8Ht+Pega1ZtG0/S7bvZ1X+If49bxtvfLeVrCaxDOncnKu7p9KueXzAjrvzoCdhWAsjMCxhGGOCJiYynPNbN+H81k0AyN1/lC9WFDBrQyFjZ23mpa830b55PFd1T+GqbimkNaxeyyPf28JITrRB70AIesIQkXbAB+WKsoDfqer/ldtmAPApsMVbNEFVHw9akMaYoEhrWJ+7+mdxV/8s9hwuZsqKnXy6NJ8/T13Hn6euo2l8NEXHSxnWPYXbzm9Jm6ZxhFVi2tq8A8dpEhdFjN3aPCCCnjBUdR3QHUBEwoE8YKKPTWer6hXBjM0Y454mcdHc0jeTW/pmsmPfUT5bls/mwiMAjF/sGTRvEhfF6L6ZXNMz1a/Wh+eUWuuOChS3u6QGAZtUdZvLcRhjQkh6o/rcN7D19+u/HtKW2Rv28MWKnTw3fT3PTV9PaoN6nJvViAvbJjGgXdP/upliYVExK/IO0jercbDDr7XcThg3AO9V8FxfEVkG5AO/VtVVvjYSkTHAGICMjAxHgjTGuCs5sd73V5Zv3H2YORsKmb9lHzPX7mbC4jyiI8K4pGMz+rdJYkin5kRGCA+8t5gjxaU/SjymekRV3TmwSBSeZNBJVXed9lwCcFJVD4vIUOBvqtrmbK+ZnZ2tOTk5zgRsjAk5ZSeVZbkHmLg4jykrdrL3yAlio8KpFxXBnsPFPD+iG9f2THM7zJAmIotUNduvbV1MGMOA+1R1sB/bbgWyVXXPmbazhGFM3aWqrMw7xOtzNnPwWAn3DWxNdmYjt8MKeZVJGG52SY2igu4oEWkO7FJVFZHeeO55tTeYwRljahYRoUtaIv93Qw+3Q6m1XEkYIlIfuAS4u1zZPQCq+jJwPfBTESkFjgE3qFtNIWOMMYBLCUNVjwKNTyt7udzyC8ALwY7LGGNMxez25sYYY/xiCcMYY4xfLGEYY4zxiyUMY4wxfrGEYYwxxi+WMIwxxvjFtSu9nSAihcAR4IxXhAdAInAwCPuebduKnq9M+ellp683weqzoudCtT59HdeJ/dyoT19lteUzGuj6rKj89LI2qproR3yey+lr0wPICcIxxgZj37NtW9HzlSk/vczHep2vT3/rLlTqszp1Gur16VadBuMzGuj6rET9+f3erEuqaj4P0r5n27ai5ytTfnpZdd5bVYV6fVb0XKjWZ3WOG+r16c9xnRCMz2ig67Oi8ip/RmtVlxSAiOSonzfSMmdn9RlYVp+BZ3UaPLWxhTHW7QBqGavPwLL6DDyr0yCpdS0MY4wxzqiNLQxjjDEOsIRhjDHGL5YwjDHG+MUShjHGGL/UqYQhIgNEZLaIvCwiA9yOpzYQkVgRWSQiV7gdS00nIh28n82PReSnbsdT04nI1SLyqoh8KiKD3Y6nNqgxCUNExonIbhFZeVr5pSKyTkQ2isgjZ3kZBQ4DMUCuU7HWBAGqT4CHgQ+dibLmCER9quoaVb0HGAHU6esKAlSfn6jqXcCtwEgHw60zasxptSLSH8+X/Vuq2tlbFg6sxzM/eC6wEBgFhANPnfYStwN7VPWkiDQDnlfVG4MVf6gJUH12xXMfnxg8dTspONGHnkDUp6ruFpGrgEeAF1T13WDFH2oCVZ/e/Z4D3lHVxUEKv9ZyZU7vqlDVWSKSeVpxb2Cjqm4GEJH3gWGq+hRwpi6S/UC0E3HWFIGoTxEZCMQCHYFjIjJFVU86GniICtTnU1U/Az4TkclAnU0YAfp8CvA08IUli8CoMQmjAqnAjnLrucC5FW0sItcCQ4AGwAvOhlYjVao+VfUxABG5FW/rzdHoap7Kfj4HANfi+TEzxdHIaqZK1SfwAHAxkCgirVX1ZSeDqwtqesIQH2UV9rGp6gRggnPh1HiVqs/vN1B9I/Ch1AqV/Xx+DXztVDC1QGXr8+/A350Lp+6pMYPeFcgF0sutpwH5LsVSG1h9BpbVZ2BZfbqspieMhUAbEWkpIlHADcBnLsdUk1l9BpbVZ2BZfbqsxiQMEXkPmAu0E5FcEblDVUuB+4FpwBrgQ1Vd5WacNYXVZ2BZfQaW1WdoqjGn1RpjjHFXjWlhGGOMcZclDGOMMX6xhGGMMcYvljCMMcb4xRKGMcYYv1jCMMYY4xdLGMY1InI4CMe4ys/btAfymANE5Lwq7NdDRF7zLt8qIiFxvzMRyTz9NuM+tkkSkanBism4wxKGqfG8t732SVU/U9WnHTjmme7DNgCodMIAHgX+UaWAXKaqhcBOETnf7ViMcyxhmJAgIg+JyEIRWS4ifyxX/ol3Rr9VIjKmXPlhEXlcROYDfUVkq4j8UUQWi8gKEWnv3e77X+oi8oaI/F1EvhORzSJyvbc8TET+6T3GJBGZcuq502L8WkSeFJFvgAdF5EoRmS8iS0TkKxFp5r0l9z3AL0RkqYhc4P31Pd77/hb6+lIVkXigq6ou8/FcCxGZ4a2bGSKS4S1vJSLzvK/5uK8Wm3hmRJwsIstEZKWIjPSW9/LWwzIRWSAi8d6WxGxvHS721UoSkXARebbcv9Xd5Z7+BKizc8zUCapqD3u48gAOe/8OBsbiuRtpGDAJ6O99rpH3bz1gJdDYu67AiHKvtRV4wLt8L/Cad/lWPJMRAbwBfOQ9Rkc8cysAXI/nduJhQHM886Vc7yPer4F/lltvyA93S7gTeM67/Afg1+W2exfo513OANb4eO2BwPhy6+Xj/hwY7V2+HfjEuzwJGOVdvudUfZ72utcBr5ZbTwSigM1AL29ZAp47V9cHYrxlbYAc73ImsNK7PAb4H+9yNJADtPSupwIr3P5c2cO5R02/vbmpHQZ7H0u863F4vrBmAT8TkWu85ene8r1AGTD+tNc5dev6RXjmlfDlE/XM27FaPDMvAvQDPvKWF4jIzDPE+kG55TTgAxFJxvMlvKWCfS4GOnrm8wEgQUTiVbWo3DbJQGEF+/ct937eBv5crvxq7/K7wF987LsC+IuIPANMUtXZItIF2KmqCwFU9RB4WiPACyLSHU/9tvXxeoOBruVaYIl4/k22ALuBlAreg6kFLGGYUCDAU6r6yo8KPRMKXQz0VdWjIvI1nulgAY6ratlpr1Ps/VtGxZ/t4nLLctpffxwpt/wPPFP9fuaN9Q8V7BOG5z0cO8PrHuOH93Y2ft8ATlXXi8g5wFDgKRH5Ek/Xka/X+AWwC+jmjfm4j20ET0tumo/nYvC8D1NL2RiGCQXTgNtFJA5ARFJFpCmeX6/7vcmiPdDHoePPAa7zjmU0wzNo7Y9EIM+7PLpceREQX279Szx3WQXA+wv+dGuA1hUc5zs8t/IGzxjBHO/yPDxdTpR7/kdEJAU4qqr/xtMC6QmsBVJEpJd3m3jvIH4inpbHSeBmPHNln24a8FMRifTu29bbMgFPi+SMZ1OZms0ShnGdqn6Jp0tlroisAD7G84U7FYgQkeXAn/B8QTphPJ7JeVYCrwDzgYN+7PcH4CMRmQ3sKVf+OXDNqUFv4GdAtneQeDWe8YYfUdW1eKYSjT/9Oe/+t3nr4WbgQW/5z4FfisgCPF1avmLuAiwQkaXAY8D/quoJYCTwDxFZBkzH0zr4JzBaRObh+fI/4uP1XgNWA4u9p9q+wg+tuYHAZB/7mFrCbm9uDCAicap6WEQaAwuA81W1IMgx/AIoUtXX/Ny+PnBMVVVEbsAzAD7M0SDPHM8sYJiq7ncrBuMsG8MwxmOSiDTAM3j9p2AnC6+XgOGV2P4cPIPUAhzAcwaVK0QkCc94jiWLWsxaGMYYY/xiYxjGGGP8YgnDGGOMXyxhGGOM8YslDGOMMX6xhGGMMcYv/x/D1MMwB8CmGwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Find the learning rate\n", "learn.lr_find()\n", "learn.sched.plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Fit the model (15-20 mins to train)**" ] }, { "cell_type": "code", "execution_count": 177, "metadata": {}, "outputs": [], "source": [ "lr = 3e-3" ] }, { "cell_type": "code", "execution_count": 178, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "fa19dcfa8db64e78baf878c7ba7fdada", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=12), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss \n", " 0 5.209042 5.980303 \n", " 1 4.513244 4.566818 \n", " 2 4.056711 4.515142 \n", " 3 3.775803 4.026515 \n", " 4 3.595237 3.857968 \n", " 5 3.519258 3.773164 \n", " 6 3.160189 3.705156 \n", " 7 3.108818 3.66531 \n", " 8 3.142783 3.613333 \n", " 9 3.192778 3.680305 \n", " 10 2.844773 3.637095 \n", " 11 2.857365 3.5963 \n" ] }, { "data": { "text/plain": [ "[array([3.5963])]" ] }, "execution_count": 178, "metadata": {}, "output_type": "execute_result" } ], "source": [ "learn.fit(lr, 1, cycle_len=12, use_clr=(20, 10))" ] }, { "cell_type": "code", "execution_count": 180, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VNX9//HXJ3vCvgSUzYgiqMhmREBUBBcELW51r9S9WmurX7UorqhotVXbX63+rHvrvn1xV1SUWhEMAiqb7IssCWsCIfv5/jE3Y5aZJGSSTO7k/Xw8eGTmzp25nznKm5Nzzz3XnHOIiIj/xUW7ABERaRgKdBGRGKFAFxGJEQp0EZEYoUAXEYkRCnQRkRihQBcRiREKdBGRGKFAFxGJEQlNebDOnTu7jIyMpjykiIjvzZ07d4tzLr22/Zo00DMyMsjKymrKQ4qI+J6ZranLfhpyERGJEQp0EZEYoUAXEYkRCnQRkRihQBcRiREKdBGRGKFAFxGJEb4I9E8Xb+Yfny+PdhkiIs2aLwL9ix9z+OfMldEuQ0SkWfNFoCfGx1FcqptZi4jUxBeBnhBvFJWWRbsMEZFmzReBnhQfR4kCXUSkRr4I9IS4OMoclJZp2EVEJBxfBHpiggFQrF66iEhYvgj0pPhAmQp0EZHwfBHoCXHlPXQNuYiIhOOLQE9MUA9dRKQ2/gh0DbmIiNTKJ4GuIRcRkdr4JNADZWouuohIeL4I9IS4QJm6WlREJDxfBHpSgoZcRERq44tA15CLiEjtfBHoGnIREamdLwJdQy4iIrXzRaBryEVEpHa1BrqZPW1m2Wb2Q4VtD5rZEjP7zszeMrP2jVlk+ZCLLiwSEQmvLj30Z4GxVbZNB/o75wYAPwI3N3BdlWjIRUSkdrUGunNuJrCtyraPnXMl3tOvgR6NUFuQLv0XEaldQ4yhXwJ80ACfE1aCAl1EpFYRBbqZTQZKgBdq2OcKM8sys6ycnJx6Had8LZciDbmIiIRV70A3s4nAKcAFzrmwSeuce8I5l+mcy0xPT6/XsZIT4gEoKlEPXUQknIT6vMnMxgJ/BI51zuU3bEnVJXvroRcUlzb2oUREfKsu0xZfAmYBfc1svZldCvwdaANMN7P5ZvZ4YxZZHuiF6qGLiIRVaw/dOXdeiM1PNUItYZkZSQlxFJaohy4iEo4vrhQFSEmIo7BYPXQRkXB8E+jJifEachERqYF/Aj0hjkKdFBURCctfga4euohIWD4K9HhNWxQRqYFvAj0lUT10EZGa+CjQ1UMXEamJbwI9OSGOAs1DFxEJyzeBnpIYr3noIiI18E2gq4cuIlIz3wR6alICe4oU6CIi4fgm0Du2SmTb7iJqWKlXRKRF802gpyUlUOagSHctEhEJyTeBnpoYuMmFhl1ERELzT6AneYGuuegiIiH5JtDTvEDPVw9dRCQk3wR6ioZcRERq5JtAT9OQi4hIjXwT6DopKiJSM/8EusbQRURq5JtAT0sK3M9aKy6KiITmm0AvH3JRD11EJDT/BHpwyKUkypWIiDRP/gl0r4euIRcRkdB8E+hJCXEkxJmGXEREwvBNoEOgl6556CIiofkr0JPiNQ9dRCQM/wW6eugiIiH5K9AT4zWGLiISRq2BbmZPm1m2mf1QYVtHM5tuZsu8nx0at8yA1KR4zXIREQmjLj30Z4GxVbZNAj51zvUBPvWeN7q0JPXQRUTCqTXQnXMzgW1VNk8AnvMePwec1sB1hZSaqJOiIiLh1HcMvatzbiOA97NLw5UUXmpSgk6KioiE0egnRc3sCjPLMrOsnJyciD4rNTFOPXQRkTDqG+ibzWxfAO9ndrgdnXNPOOcynXOZ6enp9TxcQFpSgtZyEREJo76B/jYw0Xs8EZjWMOXULEVXioqIhFWXaYsvAbOAvma23swuBe4HTjCzZcAJ3vNGl5YUT3Gpo7i0rCkOJyLiKwm17eCcOy/MS2MauJZatUoOlJtfWEq7NF9dEyUi0uh8lYptvEDPLSiOciUiIs2PvwI9JRDouwp1YlREpCpfBXprBbqISFj+CnRvyGVXgQJdRKQqXwV6+ZCLxtBFRKrzWaAnAhpyEREJxVeBriEXEZHwfBXoaUnxmEGeAl1EpBpfBbqZ0To5gTyNoYuIVOOrQAdom5JInsbQRUSq8V2gt0lJ0Bi6iEgIvgv0wJCLAl1EpCrfBXqblATyCjWGLiJSlQ8DPVE9dBGREHwX6G1TNeQiIhKK7wI90EMvxjkX7VJERJoV3wV6+9REiksdu3WzaBGRSnwX6J1aJwOwJa8wypWIiDQvvgv0zq2TANi6W4EuIlKRDwM90EPPySuKciUiIs2LbwNdPXQRkcp8F+idvCGXLeqhi4hU4rtAT4yPo31aIlt2qYcuIlKR7wIdoFOrJA25iIhU4ctA79w6WUMuIiJV+DLQ09sksym3INpliIg0K74M9D5d2rB2Wz4FxbpaVESknC8DvUeHVAA27VQvXUSknC8DvVv7QKCv2ZYf5UpERJqPiALdzK4zs4Vm9oOZvWRmKQ1VWE0O6dYWgBlLspvicCIivlDvQDez7sC1QKZzrj8QD5zbUIXVpF1qIgCfKdBFRIIiHXJJAFLNLAFIAzZEXlLdHH9wF/K1hK6ISFC9A9059xPwZ2AtsBHY6Zz7uOp+ZnaFmWWZWVZOTk79K63isO7t2bKrUDNdREQ8kQy5dAAmAPsD3YBWZnZh1f2cc0845zKdc5np6en1r7SKjM5pAKzViVERESCyIZfjgVXOuRznXDHwJjCiYcqqXUanVgCs3rK7qQ4pItKsRRLoa4FhZpZmZgaMARY3TFm1Kw/0lQp0EREgsjH02cDrwLfA995nPdFAddWqXVoiPTum8t36HU11SBGRZi0hkjc75+4A7migWvZav33asjx7V7QOLyLSrPjyStFyvdNbsXpLPiWlZdEuRUQk6nwd6Aekt6aotIz12/dEuxQRkajzfaADrMjRsIuIiK8D/aCugUBfuCE3ypWIiESfrwO9TUoiXdok6+IiERF8HugAvTqmsU6BLiIS2bTF5mDNtnxy8grZVVhC62Tffx0RkXrzfQ99TL8uACz8aWeUKxERiS7fB/ofjj8IgCWb8qJciYhIdPk+0Lu2TaZTqyS+Vw9dRFo43we6mTGgRzut6SIiLZ7vAx1gQI/2LMvexa7CkmiXIiISNTER6JkZHXAOXp6zNtqliIhETUwE+sgDOwNwz3uLcc5FuRoRkeiIiUA3M4ZmdATg27Xbo1yNiEh0xESgA/zl7IEAzFurk6Mi0jLFTKB3a58KaNhFRFqumAn0+DgLPv5sSXYUKxERiY6YCXSAp3+dCcDzs9ZEuRIRkaYXU4E+ul9XhvRqz27NRxeRFiimAh2g7z5tyFqzXaEuIi1OzAX6Ed70xTe/XR/lSkREmlbMBfrpg7sDcNu0hVGuRESkacVcoJv9PNtlzdbdUaxERKRpxVygA7x19QgAjn3w8+gWIiLShGIy0Af36hB8rIuMRKSliMlAB5gwqBsAz/x3dXQLERFpIjEb6Dee1BeAKe8uori0LMrViIg0vogC3czam9nrZrbEzBab2fCGKixSPTqk0SopHoCRf/osytWIiDS+SHvofwU+dM71AwYCiyMvqeH8cNdJAGzOLeTtBRuiXI2ISOOqd6CbWVvgGOApAOdckXOuWa1da2a8csUwAK59aZ5OkIpITIukh94byAGeMbN5ZvakmbVqoLoazJG9O3Fot7YAzF2jm1+ISOyKJNATgCHAY865wcBuYFLVnczsCjPLMrOsnJycCA5Xf38/fwgAZz0+i7Iy9dJFJDZFEujrgfXOudne89cJBHwlzrknnHOZzrnM9PT0CA5Xfxmd0oKPpy34KSo1iIg0tnoHunNuE7DOzPp6m8YAixqkqgZmZsy77QQAnpi5KsrViIg0joQI3/874AUzSwJWAhdHXlLj6NAqCYDFG3MpKC4lJTE+yhWJiDSsiKYtOufme8MpA5xzpznnmvVZx2uOOxCAfrd9yNJNeVGuRkSkYcXslaKh/GbUAcHHJz0ys8Z9i0vLyJj0HpPf+r6xyxIRaRAtKtBbJ1ceYdq+uyjsvmf//1kAvDB7rZbhFRFfaFGBDnDzyf2Cj//wyvyQ+2zfXcS8tT9fI6VleEXED1pcoF957AGsum8cADOX5YS89+hXK7YCcN8ZhwW3FRSXNk2BIiL11OICHQLTGN+4agTOwWtZ66q9/uBHS0iIM846vAf/unQoAEfc+0lTlykisldaZKADHL5fB1onJ3DnO4tYty0/uP3pL1exems+JWWOxPg4hvfuBEBeQQkXPjk73MeJiERdiw10gBMP7QrA0Q/MYOPOPeTkFTLl3cC1Ufd7wy0J8XE8fM5AAL5cvoVXQ/ToRUSagxYd6H/55cDg4+H3fRYcVhnYsz1nZ/YMvnb64B5MHL4fADe9/h2T3/qe3ILipi1WRKQWLTrQzYz3rz262va3rhpBXJxV2nbXhP7Bxy/MXsuAOz9u9PpERPZGiw50gEO6tWXmjccFn6+cOq5amJd746oRlZ5nTHqPJZtytc66iDQLka7lEhN6dkxl/GH7curAbmHDHAInUlffP55XvlnLH98IXEE69pH/AHDUgZ144bJhTVKviEgoLb6HDoGhl0cvGMLY/vvUaf9zjujFM78+otK2/y7fyj8+X94Y5YmI1Ik15XBBZmamy8rKarLjNbb12/NZsjGPmctyeH7WmuD2OZPH0KVNShQrE5FYYmZznXOZte2nHnoEenRI4/hDujJlQn+uHX1gcPvQez+tcZ0YEZHGoEBvINef2JcpEw4NPh9893QenaEhGBFpOgr0BnTR8AxevuLnE6MPfrSUdxZsiGJFItKSKNAb2LDenfjnRZlk7tcBgN+9NI8d+Rp+EZHGp0BvBCcc0pXXrxpB25TArNBBU6ZHuSIRaQkU6I3oW+/G1IAuPhKRRqdAb0QJ8XFceUxvAFZu+fmuR+W3t8uY9J6GY0SkwSjQG9mpA7sB8K8K89RvefPn+5QOmjKdW/9X9y0VkcjpwqImcMS9n5CTV1infVffP76RqxERv9GFRc3IMO8mGRWN6pvOd3eeyBlDulfafvOb3zVVWSISYxToTeDyo/ev9PzBswbw7MVDaZuSyENnD+LxCw/nwmG9AHhpzjqWZ+dFo0wR8TkNuTQh5xzOEXZFx9fnrueG1xYwqm86z148tImrE5HmSkMuzZCZ1bg875ne8MvnS3MoK3MUFJfyzH9X6e5IIlInWg+9GTEzjj+4C58szmbQlI/JLSgB4K53FvHjPSeTlKB/f0UkPCVEM3PuEYGx9PIwL3fnOwuDc9dX5OyKRmki0swp0JuZ0f26cHSfzsHnN43tC8CLs9cGt435yxc1fsbGnXv4eOGmximwHopLy5ixJJuMSe9xz7uL2LBjT7RLEolJEQe6mcWb2Twze7chCmrp4uKM5y8ZylmH9+C+Mw7j6lEHhtzv0me/Ye3WfNZty2fGkuxKrw2/7zOu+NdcdhWWhHxvU1i3LZ+Ln5nDpp0F9Jn8ARc/+w0AT365ihH3f8afP1oatdpEYlXEs1zM7HogE2jrnDulpn1b+iyXSBSXlpEYH8eMJdnBcKyqW7sUbhrbjz+8Mj+4bdV94zALfyK2Mazbls/RD8yodb9hvTvy4FkD6dkxrQmqEvGvJpnlYmY9gPHAk5F8jtQuMT7wn2pU3/Sw+2zYWVApzAGG3de0d08qLCnlV0/NrrZ91s2jWXL3WN64anhw29crt3H0AzMoKC5tsvpEYlmkQy6PADcBZeF2MLMrzCzLzLJycnIiPJyYGW9ePYIHzhzANccdyK+G7UeHtMRq+z18zkAANucWMvju6WRMeo/Jb31PcenP/6nWbs0nt6CYxRtzmfj0HGat2BpRbc45+t76Iau35gMw55YxjOqbztCMjuzbLpWUxHgO368jq+8fT3KFGTv9bvswouOKSEC9h1zM7BRgnHPuajMbBdygIZfo2V1YwpZdhbw4ey1XH3cg7VITuey5LD5ZvHmvPmfOLWPo0rZuN7henp3HRU/NYcPOAg7s0pq2KQl8u3YHABmd0vj8xuPCvvf79Tu5/tX5LMsOzNhZcPuJtAvxD5OI1H3IJZJAvw/4FVACpABtgTedcxeGe48Cvemt25bPXe8s5JPF2bXvDNxx6iH07dqG85+cza3jD+ayo3uH3Tdj0nsht3972wl0bJVUp+M98OES/vH5Ci4avh9TJvSv03tEWppGD/QqBxuFeujNXkFxKcf9+XM27iyo9toLlx3JBU9WH/tees9YkhPiKStznPn4V8xbu4MeHVI5c0gP/vrpspDH2ZsVI4tLy+gz+YO9fp9IS1LXQNeVoi1ISmI8s24eA8CO/CJaJSeQGB+Hcy7sTJi+t37I6vvHc8NrC5jnDaes374nGObTrzuGPl3bUFBcinOQmhS/VzWVn+wFuOG1BYw4oBM9O6ZxREZHAEpKy5ixNIfjD+5CSZmrtL+IVKbFuSRo665Cnpi5ki5tU7h4RAa9b3m/1vc0RK967pptnPnYrJCfO/TeT8j21pJPS4pn2m+Pok/XNhEfU8RP1EOXvdapdTI3jzs4+Pyly4dx3j+/Dj4/fXB3Hj5nUIMf9/D9Olbb9sH3Gzk8o0MwzAHyi0o54eGZjVZHbkExA+78GIAbT+rLb48LfVGXSHOlHrqEVVbmgr30KRMO5cIj96txtchIbd9dxPWvzmfG0tqnt/713EFMGNS91v3qouL3rOida0ZyWI92DXIMkUg06UnRulKg+09uQTFlZY72aXWbtdIQqs6eeeScQbRLS+TiZypfITv7ljF0reMUy705XkXRuNK2JVqZs4vWKQl0aRP5f89YpPXQpUG0TUls0jAHmH/7CQAM6tmeBbefyIRB3TiubxeW3XsyH/7haJK8E6NHTv2Ub1Zvq/b+hz5eylX/nktZWfXOygMfLuF/Xl1AqbfefNUw/+T6Y3nnmpHB56/NXR98vGxzHjN/DP3bw6MzlnPT6wsoKC6ltMxRUhr2WrtKFqzbwby12+u0byTmrtnO0k3RuRPWrsIS1m3LZ3dhCZ8s2szijbmUlTm+WrGFRRtyARj9ly8Yeu+ndW43CU09dPGdguLS4NWlnVsnkXXrCcHXFm7Yyfi/fQnAuUf05P4zBwRf+2rFFs7/Z2Bq5rjD9mHJpjxW5uwG4BcDu/G38wYH9127NZ9jHgysR/O38wbTvX0qZz72VfD18pO2N762gPg44+Vv1lWr86XLhzH8gOr3k4XA8NKQe6ZT8a/fb449gFMH7suh3eo2zDPzxxxWbdnNxBEZYfcpLXOs2bqb0VVW6GyK3zzKyhyfLcnmsucDf+dTEuMoKK45sP9xwRDGHbZvo9blRxpykZhX3rv+xcBu/DKzB1mrt1ebGz9hUDemzd9Q62eFuoHIIbd/SH5R6HVmLjlqf0b26cQlz9b8//ODZw3gl5k9g88HTfmYHfk134Hqr+cOIievkPOG9qJVcuh5CxVnBn1y/bEc2KV1tX027Sxg2H2fhnx/ny6tmXbNUaQlNd68iJqGssI5ZN+2dGufSkanNG4edzDxjXjOxk805CIx797TA1eWvr1gA796ak6lMB/oncysGuZtUxI4p0LA/s8JB7H6/vEh7wa1aMpYUhMrz6u/7viDAHj6v6uqhfnjFx7O0P07csGRvYLbbpv2A+Wdpj1FpbWGOcDvX57PPe8t5tA7Pgq7z5X/+jb4+PiHqq+Pn19UEjbMAZZl7+KQ2z/iiuez2NZAi7flF5UEb5e4KcTFaxD4B3bGDaPokJbIeUN7sWjKSSTFx3HPaf05eN+2LNqYyyeLN/Pkl6u4fdoPDVJXXf20Yw+bcwuqDdVtzi1o0gXuIqEeuvja+L/9h4XeOGy58uGQUD3E8umIa7buZvJbP/D8JUNrnLnzz5kruff9xQB8f+eJtElJrPa5K6aOY3t+EZ1bJwe3Pf7FCh74cAnl2fDAWQNYsG4HL3g3KklKiOO9340MzqkvvzDr4NsrL1R2/pG9mHr6YdXqOvvxWSzelEtehTtbLZpyUrDH/e+v13Dr/wYC8dUrh3Not7a8MHsNB3ZpTc8OaZzw8MxKn3fd8Qdx9XEH7NWFW9t2FzFv7XYufS6LL/94HCP/FBiimnfbCQy+ezoAZwzuztQzDmPhhlz2FJVy1IGdwg71XPXvuXzww883ZunYKoltu4uY9tujGNizfZ3rqo/NuQUcOfXnfwAvHbk/+7ZLYb9OrbjcGzKK5qwnDblIi7FlVyGlZY4/f7SUW8YdTIcw68hs3VVIu9REEvYitJxzzFu3g14d04KB/ePmPE70ArE85EPJzi1g6NTqveRl954cNji37y6ipMxRWuaCPezyf6C27iokziwYludk9mTCoG6cX2HJhmG9O3J0n3Qe9G4gcnL/fXjswsOrHefNb9dz/asLwn7vOZPHVJpxsnNPMSmJcSQnBH5jefI/K7nnvcVh319u+b0n17m9y0P13d+N5IbXFrCkwkncxy4YwsmNOLb+xMwVTH1/Sa37RWvWkwJdpBn4accejrr/s+Dz0f268PSvj6jTe3//8jymzd9A6+SEkHefuvioDO449VDufHshz361OuRnrJg6Luw4dPlMn1BDO/u0TeHrWwLLRBSWlNL31sBvDlNPP4zHvljOum2130bwPzcdV++blzz15SrufndRpW2NudZP+W9d7/5uJKf8vy/D7hfJd4qExtBFmoHu7VMrBdH4vehlXjR8P4CQYT68dycmDs8A4M5fHEpifOjQrumkYnyc0So5gdX3j+eLG0fRJiWB7u1TAdiUW0BJaRlFJWXBMAe45a3vq4X5+RXOGTx09kCOPSid7+48MaLgu3Tk/pw+uHul8x0vzF5T78+rSfmYeZxB/+7tmDN5DJPHHcwlR+3PG1cN5/MbRgVnQL2aVX02U3OiHrpIE1i2OY8p7y7iyYmZwWGLuqg6Xt+9fSrnDe3JNaP7VNt3V2EJ/b3e9ouXHUnv9Nbs027vL9R57qvV3PH2QiDwD9B7328Mud/KqeOC5x9KSsuIj7NGGY64+91FPPXlquDzcLN6Vm/Zzag/f77XyzYsz87j+Idm8sCZAzj7iJ4h96n4W8rrvxlOZkb15Soak4ZcRGJAflEJj85Yzu/HHBRyJk5VW3YVkpwQF3Zcvy72FJVWOzn71aTRbM8v4rWs9Zx/ZC+S4uPI6Nyq3sfYG8459r+58tIML15+JCMO6Bx8Pn/dDk579L/B5z06pPLlH0cz9pGZwbH4K4/tzakDugWHVN64ajiHdmvHHdMW8krWOj6+7hgOqmHhtxMf/oIfNwduyNLUSz0r0EWk3t79bgPXvDgv+Lw5rFVfdV79qQO7MTSjA7dNW1jvz2yVFM9u71qDms43QOWZMPec1p8Lh+1X7+PuLQW6iETs0RnLaZea2KThVZOde4oZeNfHYV9//9qjefI/K3lz3k/Bbf32aVNpxkwoY/p14ak6nKx+Y+56/ue1wOyg2ma8zF65lXOe+Hm10khmyCjQRSQm5RUUc9idlUN95IGd+fdlRwKVh2jeu3ZkcCmFafN/4o9vfMdXk8bQsVUSD0//MXgx2rDeHXn5iuG1HruszNH3tg8oLg3k5pBe7fl27Q5OGbAvfz9/CFPfX8wTM1dy7hE9qy0H8cg5gzhtcP1WCFWgi0jMyi0oZmd+MUc/MIObT+7HlcceUOn13YUllDpH2xrOJVRcNvnNq0cwpFeHOh07O6+AofdWv77g9d8M56zHK9+opVfHNF6/ajj/mrWG644/qN7LTyvQRURq4ZxjT3Fpvda0uf6V+bw57yfi44zSECt7Apw3tCf3nTEg5Gt7Q/PQRURqYWb1XqDsoXMGseq+cayYOq7S9levHM5TEwPZe/nRvSOucW/oFnQiIvVUfpKzfFXPM4f0YOj+gTnq0ZgZpEAXEYnQn84cQPvURH7vrcYZLQp0EZEIpSTGc9eE/tEuQ2PoIiKxQoEuIhIjFOgiIjFCgS4iEiMU6CIiMUKBLiISIxToIiIxQoEuIhIjmnRxLjPLAepzY8DOwJYGLidWqG1CU7uEpnYJrzm3zX7OufTadmrSQK8vM8uqy0pjLZHaJjS1S2hql/BioW005CIiEiMU6CIiMcIvgf5EtAtoxtQ2oaldQlO7hOf7tvHFGLqIiNTOLz10ERGpRbMPdDMba2ZLzWy5mU2Kdj2NzcyeNrNsM/uhwraOZjbdzJZ5Pzt4283M/ua1zXdmNqTCeyZ6+y8zs4nR+C4Nycx6mtkMM1tsZgvN7PfedrWNWYqZzTGzBV7b3OVt39/MZnvf8xUzS/K2J3vPl3uvZ1T4rJu97UvN7KTofKOGZWbxZjbPzN71nsduuzjnmu0fIB5YAfQGkoAFwCHRrquRv/MxwBDghwrbHgAmeY8nAX/yHo8DPgAMGAbM9rZ3BFZ6Pzt4jztE+7tF2C77AkO8x22AH4FD1DYO7zu29h4nArO97/wqcK63/XHgKu/x1cDj3uNzgVe8x4d4f8eSgf29v3vx0f5+DdA+1wMvAu96z2O2XZp7D30osNw5t9I5VwS8DEyIck2Nyjk3E9hWZfME4Dnv8XPAaRW2P+8Cvgbam9m+wEnAdOfcNufcdmA6MLbxq288zrmNzrlvvcd5wGKgO2obvO+4y3ua6P1xwGjgdW971bYpb7PXgTEWuDnmBOBl51yhc24VsJzA30HfMrMewHjgSe+5EcPt0twDvTuwrsLz9d62lqarc24jBIIN6OJtD9c+Md1u3q/Cgwn0RNU2BIcV5gPZBP6RWgHscM6VeLtU/J7BNvBe3wl0Ijbb5hHgJqDMe96JGG6X5h7oFmKbpuX8LFz7xGy7mVlr4A3gD8653Jp2DbEtZtvGOVfqnBsE9CDQezw41G7ezxbRNmZ2CpDtnJtbcXOIXWOmXZp7oK8HelZ43gPYEKVaommzN1yA9zPb2x6ufWKy3cwskUCYv+Cce9PbrLapwDm3A/icwBh6ezMrvxF8xe8ZbAPv9XYEhvlirW2OAn5hZqsJDNeOJtBjj9l2ae6B/g3QxzsrnUTgRMXbUa4pGt4GymdjTASmVdh+kTfgLN+2AAABH0lEQVSjYxiw0xt2+Ag40cw6eLM+TvS2+ZY3lvkUsNg591CFl9Q2Zulm1t57nAocT+AcwwzgLG+3qm1T3mZnAZ+5wNm/t4Fzvdke+wN9gDlN8y0annPuZudcD+dcBoHs+Mw5dwGx3C7RPitb2x8CsxV+JDAmODna9TTB930J2AgUE+gZXEpgHO9TYJn3s6O3rwGPem3zPZBZ4XMuIXDyZjlwcbS/VwO0y0gCv+Z+B8z3/oxT2ziAAcA8r21+AG73tvcmEDzLgdeAZG97ivd8ufd67wqfNdlrs6XAydH+bg3YRqP4eZZLzLaLrhQVEYkRzX3IRURE6kiBLiISIxToIiIxQoEuIhIjFOgiIjFCgS4iEiMU6CIiMUKBLiISI/4PCEuEjE16Y78AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.sched.plot_loss()" ] }, { "cell_type": "code", "execution_count": 181, "metadata": {}, "outputs": [], "source": [ "learn.save('initial')" ] }, { "cell_type": "code", "execution_count": 182, "metadata": {}, "outputs": [], "source": [ "learn.load('initial')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Test" ] }, { "cell_type": "code", "execution_count": 188, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "quelles composantes des différents aspects de la performance devraient être mesurées , quelles données pertinentes recueillir et comment ? _eos_\n", "which components within various performance areas should be measured , whatkinds of data are appropriate to collect , and how should this be done ? _eos_\n", "what aspects of the and and be be be be be be be be be ? ? _eos_\n", "\n", "le premier ministre doit - il nommer un ministre d’ état à la santé mentale , à la maladie mentale et à la toxicomanie ? _eos_\n", "what role can the federal government play to ensure that individuals with mental illness and addiction have access to the drug therapy they need ? _eos_\n", "what minister the minister minister minister minister minister , , , , health health and health ? ? ? ? _eos_\n", "\n", "quelles sont les conséquences de la hausse des formes d’ emploi non conformes aux normes chez les travailleurs hautement qualifiés et chez ceux qui occupent des emplois plus marginaux ? _eos_\n", "what is the impact of growing forms of non - standard employment for highly skilled workers and for those employed in more marginal occupations ? _eos_\n", "what are the consequences of workers workers workers workers workers workers and and workers and workers workers workers workers workers workers ? ? ? _eos_ _eos_\n", "\n", "que se produit - il si le gestionnaire n’ est pas en mesure de donner à l’ employé nommé pour une période déterminée un préavis de cessation d’ emploi d’ un mois ou s’ il néglige de le\n", "what happens if the manager is unable to or neglects to give a term employee the one - month notice of non - renewal ? _eos_\n", "what happens the the employee employee employee employee employee the the the the the or or the the the ? ? _eos_\n", "\n", "quelles personnes , communautés ou entités sont considérées comme potentiels i ) bénéficiaires de la protection et ii ) titulaires de droits ? _eos_\n", "which persons , communities or entities are identified as potential ( i ) beneficiaries of protection and / or ( ii ) rights holders ? _eos_\n", "who , , , , , or or or or or or or or protection ? ? ? ? _eos_\n", "\n", "quelles conditions particulières doivent être remplies pendant l’ examen préliminaire international en ce qui concerne les listages des séquences de nucléotides ou d’ acides aminés ou les tableaux y relatifs ? _eos_\n", "what special requirements apply during the international preliminary examination to nucleotide and / or amino acid sequence listings and / or tables related thereto ? _eos_\n", "what specific must be be be be sequence sequence or or or or sequence or or sequence or sequence or sequence in in ? ? ? ? _eos_ _eos_\n", "\n", "pourquoi cette soudaine réticence à promouvoir l’ égalité des genres et à protéger les femmes de ce que , dans la plupart des cas , on peut qualifier de violations grossières des droits humains ? _eos_\n", "why this sudden reluctance to effectively promote gender equality and protect women from what are – in many cases – egregious human rights violations ? _eos_\n", "why is the so for such of of of of of and rights and rights rights of rights rights ? ? ? ? _eos_ _eos_\n", "\n", "pouvez - vous dire comment votre bagage culturel vous a aidée à aborder votre nouvelle vie au canada ( à vous adapter au mode de vie canadien ) ? _eos_\n", "what are some things from your cultural background that have helped you navigate canadian life ( helped you adjust to life in canada ) ? _eos_\n", "what are you new to to to to to to to to life life life life ? ? ? ? _eos_ _eos_\n", "\n", "selon vous , quels seront , dans les dix prochaines années , les cinq enjeux les plus urgents en matière d' environnement et d' avenir viable pour vous et votre région ? _eos_\n", "which do you think will be the five most pressing environmental and sustainability issues for you and your region in the next ten years ? _eos_\n", "what do you see the next priorities priorities next the next the and and in in in in in in ? ? ? ? ? _eos_ _eos_\n", "\n", "dans quelle mesure l’ expert est-il motivé et capable de partager ses connaissances , et dans quelle mesure son successeur est-il motivé et capable de recevoir ce savoir ? _eos_\n", "what is the expert ’s level of motivation and capability for sharing knowledge , and the successor ’s motivation and capability of acquiring it ? _eos_\n", "what is the nature and and and and and and and and and and and and and to to to ? ? ? ? _eos_ _eos_ _eos_\n", "\n" ] } ], "source": [ "x, y = next(iter(val_dl))\n", "probs = learn.model(V(x))\n", "preds = to_np(probs.max(2)[1])\n", "\n", "for i in range(180, 190):\n", " print(' '.join([ fr_itos[o] for o in x[:, i] if o != 1 ]))\n", " print(' '.join([ en_itos[o] for o in y[:, i] if o != 1 ]))\n", " print(' '.join([ en_itos[o] for o in preds[:, i] if o != 1 ]))\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bi-direction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Take all your sequences and reverse them and make a \"backwards model\" then average the predictions. Note that with deeper models, not all levels may be bi-directional." ] }, { "cell_type": "code", "execution_count": 191, "metadata": {}, "outputs": [], "source": [ "class Seq2SeqRNN_Bidir(nn.Module):\n", " def __init__(self, vecs_enc, itos_enc, em_sz_enc, vecs_dec, itos_dec, em_sz_dec, nh, out_sl, nl=2):\n", " super().__init__()\n", " self.nl, self.nh, self.out_sl = nl, nh, out_sl\n", " self.emb_enc = create_emb(vecs_enc, itos_enc, em_sz_enc)\n", " self.emb_enc_drop = nn.Dropout(0.15)\n", " self.gru_enc = nn.GRU(em_sz_enc, nh, num_layers=nl, dropout=0.25, bidirectional=True) # for bidir, bidirectional=True\n", " self.out_enc = nn.Linear(nh * 2, em_sz_dec, bias=False) # for bidir, nh * 2\n", " self.drop_enc = nn.Dropout(0.05) # additional for bidir\n", " \n", " self.emb_dec = create_emb(vecs_dec, itos_dec, em_sz_dec)\n", " self.gru_dec = nn.GRU(em_sz_dec, em_sz_dec, num_layers=nl, dropout=0.1)\n", " self.out_drop = nn.Dropout(0.35)\n", " self.out = nn.Linear(em_sz_dec, len(itos_dec))\n", " self.out.weight.data = self.emb_dec.weight.data\n", "\n", " def forward(self, inp):\n", " sl, bs = inp.size()\n", "\n", " # ==================================================\n", " # Encoder version\n", " # ==================================================\n", "\n", " # initialize the hidden layer\n", " h = self.initHidden(bs)\n", "\n", " # run the input through our embeddings + apply dropout\n", " emb = self.emb_enc_drop(self.emb_enc(inp))\n", "\n", " # run it through the RNN layer\n", " enc_out, h = self.gru_enc(emb, h)\n", "\n", " # Additional for bidir\n", " h = h.view(2, 2, bs, -1).permute(0, 2, 1, 3).contiguous().view(2, bs, -1)\n", " \n", " # run the hidden state through our linear layer\n", " h = self.out_enc(self.drop_enc(h)) # new for bidir; dropout hidden state.\n", "\n", " # ==================================================\n", " # Decoder version\n", " # ==================================================\n", "\n", " # starting with a 0 (or beginning of string _BOS_)\n", " dec_inp = V(torch.zeros(bs).long())\n", " res = []\n", "\n", " # will loop as long as the longest english sentence\n", " for i in range(self.out_sl):\n", "\n", " # embedding - we are only looking at a section at time\n", " # which is why the .unsqueeze is required\n", " emb = self.emb_dec(dec_inp).unsqueeze(0)\n", "\n", " # rnn - typically works with whole phrases, but we passing\n", " # only 1 unit at a time in a loop\n", " outp, h = self.gru_dec(emb, h)\n", "\n", " # dropout\n", " outp = self.out(self.out_drop(outp[0]))\n", "\n", " res.append(outp)\n", "\n", " # highest probability word\n", " dec_inp = V(outp.data.max(1)[1])\n", "\n", " # if its padding ,we are at the end of the sentence\n", " if (dec_inp == 1).all():\n", " break\n", "\n", " # stack the output into a single tensor\n", " return torch.stack(res)\n", "\n", " def initHidden(self, bs):\n", " return V(torch.zeros(self.nl * 2, bs, self.nh)) # for bidir, sel.nl * 2" ] }, { "cell_type": "code", "execution_count": 192, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n", "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n" ] } ], "source": [ "rnn = Seq2SeqRNN_Bidir(fr_vecd, fr_itos, dim_fr_vec, en_vecd, en_itos, dim_en_vec, nh, enlen_90)\n", "learn = RNN_Learner(md, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n", "learn.crit = seq2seq_loss" ] }, { "cell_type": "code", "execution_count": 193, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c58d98c9ed43427687de852a42598a7f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=12), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss \n", " 0 4.766771 4.495123 \n", " 1 3.918195 4.018911 \n", " 2 3.682928 3.852527 \n", " 3 3.654867 3.653316 \n", " 4 3.540806 3.581977 \n", " 5 3.38937 3.518663 \n", " 6 3.337964 3.461221 \n", " 7 2.868424 3.439734 \n", " 8 2.783658 3.426322 \n", " 9 2.743709 3.375462 \n", " 10 2.662714 3.39351 \n", " 11 2.551906 3.373751 \n" ] }, { "data": { "text/plain": [ "[array([3.37375])]" ] }, "execution_count": 193, "metadata": {}, "output_type": "execute_result" } ], "source": [ "learn.fit(lr, 1, cycle_len=12, use_clr=(20, 10))" ] }, { "cell_type": "code", "execution_count": 196, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8W9X9//HXkbzikb2nMyAEsggmC0ghCSE4lBRK+TIKhEJTSvnSFtr+wh4tgdKWFlqgX6AEaCmblpSEQCYUyN57TzLs2Fl24qnz+0OyYtmSl2TLV34/H488LF1d6X508sjbJ+eee66x1iIiIs7ninYBIiISGQp0EZEYoUAXEYkRCnQRkRihQBcRiREKdBGRGKFAFxGJEQp0EZEYoUAXEYkRcQ15sLZt29r09PSGPKSIiOMtX778sLW2XXX7NWigp6ens2zZsoY8pIiI4xljdtdkPw25iIjECAW6iEiMUKCLiMQIBbqISIxQoIuIxAgFuohIjFCgi4jECEcE+tyNh3hhwbZolyEi0qg5ItA/35LNy1/siHYZIiKNmiMC3e0ylHh0M2sRkao4ItDjXIaSUgW6iEhVnBHobhel6qGLiFTJGYHuMpR4PNEuQ0SkUas20I0xrxpjsowx68pt+50xZpMxZo0x5l/GmJb1WaTbZfBY8KiXLiISUk166K8B4ytsmw30t9YOBLYA90W4rgDxbm+ZOjEqIhJatYFurf0CyK2w7TNrbYnv6SKgaz3U5ud2GQCNo4uIVCESY+g/AD6JwOeEFOcL9GKNo4uIhBRWoBtjHgBKgDer2GeyMWaZMWZZdnZ2nY5TFuilmrooIhJSnQPdGHMLcAVwo7U2ZNJaa1+y1mZYazPatav2lnhBuTWGLiJSrTrdU9QYMx74f8C3rLUnI1tSZWU9dE1dFBEJrSbTFt8CFgJ9jTH7jDG3AX8B0oDZxphVxpi/1meR/kDXkIuISEjV9tCttdcH2fy3eqglpDi3ZrmIiFTHEVeKul1lY+gachERCcURgR7vH0NXD11EJBRHBLpbY+giItVyRKCXjaGrhy4iEpozAt03hl6qMXQRkZAcEui+S/815CIiEpIjAl2Lc4mIVM8RgR6nS/9FRKrljED3z3LRGLqISCiOCHS35qGLiFTLEYFedscijaGLiITmiEAvm4derCEXEZGQHBHoCb4eemGJAl1EJBRHBHpinLfMIgW6iEhIjgj0BAW6iEi1nBXoGkMXEQnJGYHuVg9dRKQ6jgj0OLcLl1Ggi4hUxRGBDt5hFw25iIiE5pxAd7vUQxcRqYJzAj3OrXnoIiJVcEygJ8aphy4iUhXHBHqc2+iORSIiVXBMoLtdhmItziUiEpJjAj3e5dJ66CIiVXBMoLtdRsvniohUwTGBHu82ukm0iEgV4qJdQE0dOVlM7smiaJchItJoOSbQ9+SejHYJIiKNmmOGXEREpGqOCfTvD+9O65SEaJchItJoVRvoxphXjTFZxph15ba1NsbMNsZs9f1sVb9lQrN4N6eKSuv7MCIijlWTHvprwPgK26YAc621ZwBzfc/rVbOEOE4Vl+LR1EURkaCqDXRr7RdAboXNE4HXfY9fB74T4boqaRbvBnSjaBGRUOo6ht7BWnsAwPezfeRKCq5ZvLfUU8UadhERCabeT4oaYyYbY5YZY5ZlZ2fX+XOSE7wzLE8WlUSqNBGRmFLXQD9kjOkE4PuZFWpHa+1L1toMa21Gu3bt6ng4SErwDrkUqIcuIhJUXQN9OnCL7/EtwEeRKSe0ZN8Y+knNdBERCaom0xbfAhYCfY0x+4wxtwFPAZcaY7YCl/qe16tmvh66pi6KiARX7aX/1trrQ7w0JsK1VCnJ10PXSVERkeAcc6VosnroIiJVckygN1MPXUSkSo4J9LIeen6hpi2KiATjmEBv3iwegOMFCnQRkWAcE+iJcS7i3YYTCnQRkaAcE+jGGNKS4jlRUBztUkREGiXHBDpAamKcxtBFREJwVKAnJ7jJ17RFEZGgHBXoKYlxWpxLRCQERwV6coKb/EL10EVEgnFUoKckqIcuIhKKowI9OVE9dBGRUBwV6Oqhi4iE5qhAT07ULBcRkVAcFegpCXEUlXgoLtWNokVEKnJUoJct0KW7FomIVOaoQE9J1I2iRURCcVSgn15CVz10EZGKHBXoKQnqoYuIhOKsQPcNueRpCV0RkUocFehpSd5A100uREQqc1SgN0/y3rVIa6KLiFTmqEAv66HrrkUiIpU5KtBTFegiIiE5KtDj3S6axbs15CIiEoSjAh28wy7qoYuIVObMQC9UD11EpCLHBXrzZvHqoYuIBOG4QE9Litc8dBGRIBwY6HGcOKUhFxGRihwX6N1bJ7PjcD4HjxVEuxQRkUYlrEA3xvzcGLPeGLPOGPOWMSYpUoWFMrZfewAmPPff+j6UiIij1DnQjTFdgLuBDGttf8ANXBepwkI5s0MaADn5RfV9KBERRwl3yCUOaGaMiQOSgf3hl1S1NN96Lm6Xqe9DiYg4Sp0D3Vr7DfB7YA9wADhmrf0sUoVV5YZh3WnRLL4hDiUi4hjhDLm0AiYCPYHOQIox5vtB9ptsjFlmjFmWnZ1d90rL6dQ8idz8IgqKdeciEZEy4Qy5jAV2WmuzrbXFwIfAyIo7WWtfstZmWGsz2rVrF8bhTuvUshkAe3NPRuTzRERiQTiBvgcYboxJNsYYYAywMTJlVW1wtxYArNl3rCEOJyLiCOGMoS8G3gdWAGt9n/VShOqqUtdWyQAcPK656CIiZeLCebO19hHgkQjVUmNJ8W5aNIvnkAJdRMTPcVeKlunYPElXi4qIlOPYQG/fPFE9dBGRchwb6B2bJ2kMXUSkHOcGeosksk8UUlLqiXYpIiKNgmMDvUPzJDwWDudpTRcREXBwoHdp5b246JujurhIRAQcHOjdfIH+4YpvolyJiEjj4NhA79k2FYB9R05FuRIRkcbBsYHudhnO69GKohKdFBURAQcHOninLh44ph66iAg4PNDP6JDKrpyTHM4rjHYpIiJR5+hAH9jVu+ri7A2HolyJiEj0OTrQR53RjsQ4F5sPnoh2KSIiUefoQI9zuxjZuw1vL90T7VJERKLO0YEO3tkuBcUetmXlRbsUEZGocnygXz+0OwA7shXoItK0OT7QB3TxnhjVUroi0tQ5PtDbpCYC8NaSvVGuREQkuhwf6G6XAWDDgeNRrkREJLocH+gAF/RpA8Cmgwp1EWm6YiLQp4zvB8BHq/ZHuRIRkeiJiUDv36U5AC8u2K7FukSkyYqJQDfG0KttCgBLduZGuRoRkeiIiUAHePeOEQBsPqRlAESkaYqZQG+bmkiblAS2aF0XEWmiYibQAc7skMYm9dBFpImKqUDv3jqZ1XuPcqqoNNqliIg0uJgK9A4tkgDo9/CsKFciItLwYirQ7x7dx//YWhvFSkREGl5MBXqc28VTVw8AYOfh/ChXIyLSsGIq0AHO7d4KgA9W7ItyJSIiDSusQDfGtDTGvG+M2WSM2WiMGRGpwurqjPapADw/f3uUKxERaVjh9tCfBWZZa88CBgEbwy8pPC6X4TuDOwOQpTXSRaQJqXOgG2OaA6OAvwFYa4ustUcjVVg4br+oFwBDp86NciUiIg0nnB56LyAbmGaMWWmMecUYkxKhusJydqfm/scfaixdRJqIcAI9DhgCvGitPRfIB6ZU3MkYM9kYs8wYsyw7OzuMw9Wcy2X45KcXAXDPu6t5/etdDXJcEZFoCifQ9wH7rLWLfc/fxxvwAay1L1lrM6y1Ge3atQvjcLXTr1wv/ZHp6xvsuCIi0VLnQLfWHgT2GmP6+jaNATZEpKoIWfPoOP/j+ZuyoliJiEj9C3eWy/8Cbxpj1gCDganhlxQ5zZPimXvvtwCYv1mBLiKxLaxAt9au8g2nDLTWfsdaeyRShUVK73apJLhdvLFwd7RLERGpVzF3pWgwnVt6F+06XlAc5UpEROpPkwj0R688B4AVuxvdfyBERCKmSQT6oK4tAZg0bWmUKxERqT9NItBbpST4HxcU6+YXIhKbmkSgAzxz7SAA3lm6N8qViIjUjyYT6H07pgHei4zUSxeRWNR0Ar1Dmv/xWQ/NUqiLSMxpMoEe53YF3KLuD59tjmI1IiKR12QCHeCecX355WXelQpe/u9OJk1bEuWKREQip0kFOsBPLjndS1+wOZucvMIoViMiEjlNLtABdj6ZyX2XnwXAu8u0XrqIxIYmGejGGG4ZmQ7Ab2dtim4xIiIR0iQDHSAp3u1//NGqb6JYiYhIZDTZQAd4744RALy4YHuUKxERCV+TDvTz01vTNjWBFs3iK72WdbyAU0Waqy4izhEX7QKibWTvtkxfvZ89OSc5nF/I1S98XWmfG4d154mrBkShOhGRmmvSPXSAC/u0BWDMMwuChjnAm4v38OXWww1ZlohIrTX5QP9eRlcAikttwPZ4twl4/v2/LebQ8YIGq0tEpLaa/JCLMYbx53Rk1vqDgPfG0s2TAsfU06fMAGDY1LnsempCg9coIlITTb6HDqd76ZNH9aoU5gBPf3eg/7GmOIpIY2WstdXvFSEZGRl22bJlDXa82igq8ZAQF/r3297ck1z09HzAe6WpMSbkviIikWSMWW6tzahuP/XQfaoKc4BurZP9j5fszK3vckREak2BXgt/ueFcAKZ9tSu6hYiIBKFAr4UJAzoBMGv9QfbmnoxyNSIigRTotWCMYWy/DgB85/mvolyNiEggBXotvXKL97xETn4Rmw+eiHI1IiKnKdDr4OWbvaF+2Z++iHIlIiKnKdDrYGy/9v7HX2zJDvvzdufk8+j09RSXesL+LBFpuhTodWCM4bnrvTNeHp2+PqzPOlVUykMfree1r3fR/5FP/dtLPZbfztrEn+du5eAxLTkgItVr8pf+19WVgzpz91sr2XE4n1KPxe2q3YVGn2/JZntWHo9/vMG/rbDEwyMfrWN8/05c//Ii//Y/zN4CwLYnLifOrd/BIhKc0iEMzZO8vw973z+TA8dO1fh9CzZnccurSwLCvMzrC3cHhHl5f1+0u26FikiTEHagG2PcxpiVxpiPI1GQk8y4+yL/4xFPzmPdN8eqfU+pxzJp2tJK228a3oNxZ3cI2Lbq4Ut58cYhZPRoBcBj/9lAQy7VICLOEokhl58CG4HmEfgsR+nWOpmXb87gh29416e54s9f+l+75ryu/HTMGXRqkRQwTDJ7wyH/40kj07l33JkkxrmJcxlmrjvAZ77Xtz5xOfFuF5cP6MTlAzr5V3ycszGLSysEv4gIhNlDN8Z0BSYAr0SmHOe59OwOLLl/TKXt7y/fx0VPz6fPA58EbL/n3VUArH5kHI9eeQ5pSfEkxLlwuQxXDOzs3y++wlj5pz8bBcBd/1wRds2nikqZte5g0N5+qceSX1gS9jFEpOGF20P/E/ArIC3UDsaYycBkgO7du4d5uMapffMk3vrh8JBj30fyi5i94RCtUhI4WVRKvNsEvY8peE98BtO3Yxpndkhly6E8jhcUB13mt6IH/rWWNxfvAeBX4/ty58V98Hgs/R6eBcCHd46kX8fmLNqZw7NztrJq71H/e68f6v27unfcmbRNTaz2WCISfXVePtcYcwWQaa290xhzMfALa+0VVb2nMS+fG0n5hSXMXHuArBOF/O7TzZVe//P15/LtQZ2DvLNq8zdlcetrSxl3dgdeurnqlTR3Hc7n4t8vqPUxgn6WbuohElUNsXzuBcCVxphdwNvAaGPMP8L4vJiRkhjH9zK68ZNL+gR9/fL+Hev0uUO6e0+OfrbhEOlTZlQ5/BKpMIfTd2wK16frD5I+ZQZfbTtMbn5RRD5TRE6rc6Bba++z1na11qYD1wHzrLXfj1hlMeK33x1QaVtd55K3SI5nZO82/ucfrzlA+pQZTPlgTcB+JeWuOF3ywBi2VhjG+eDHI3j+hiGVPn/71Ex2PTWBDY9fVum19CkzSJ8yg6teqPuiZM/N3QrAja8sZsivZ5PxmzmatSMSQbqwqJ5ddk5Hpq/ez2NX9mflniOMPqt99W+qwjPXDmb4k3MDtr29dC/XD+3On+Zs4elrBvHned7gnDCwE+3TkgDvrJlSjyUp3u1/36gzxzFz7QGuzegWcAem5IQ4dj01gSU7c7n2/xYGHGvlnqNYa2t9x6Zff7yB9fuPB2w7nFdIz/tmsn1qJjuy83jqk03cO64vZ3duchOmRCJCt6BzsOfnbws6Rl9m6QNjaZcW3gnNEwXFLNmZy22vB/69bX3icnLyiujYwvsL46a/Lea/Ww8DsPbRcaQlxTNz7QGemLGRKwZ24v++2OF/b/8uzSko9rAtKy/oMdc/dhlulwn45SPSlNV0DF2B7nAlpZ5KUyPLRPpkZk5eIef9Zk61+z3y7bOZNDKdnvfNrPTayzdn+OfRbz10gkv/GHrFytWPjAs5G6g2Jv7lS1bvO8a0W8/nkr7h/Q9JJBp0T9EmIs7tYtqt53PZOR2YNDIdgDPap/KvO0dG/FhtUhP5+21Dq93vsf9sCBrmGx8fH3BRVJ/2qVV+ztAn5rDpoHeYxlrL7A2HarQi5fbsPPbkeO8odSS/iNX7vFfw3jptKc/O2cqxk8Us352r1S0l5qiHHkPK/i5rO75d22OUhfXV53bB7TK8t3wfAP/91SVc9PT8gP1/PfEcHvpoPU9fM5BrM7pV+dmFJaX0fXBW0Nd6tUthR3Y+1w/tzpNXVz7RXObQ8QKGTfWeY3j/jhE8N29blUscu12G7VMzq6xLJNo05CL15uCxAvIKS/w9bGsth44X0rFFEst35/LdF0+fSN311AR2Hs4nvU1yjX/R7M09yXNzt/p/UVQ0/a4LGNi1JR6P5ZnZW7jmvK6kt03BWsttry9j3qasSu/Z8pvLOfPB4ENTADufzKy3X4QN8YtWYpsCXaLqVFEpCXGuWi8rXF5BcSmPf7yBf/qudi3vtVvP5x+L9jBno3ftmxUPXcqQX88O+Vm7nppATl4hM9cd5OIz23HHP5YHzLqZftcFTPlgLbdf1JNNB08weVQvFu3ICViOoSq5+UX88r3VPPM/gyuN+9/zzio+XPkN6x67jNTE0xPL5m06REGxh0zfzcdFQlGgS8zYlpXH2Gc+58I+bfly2+Fq9//2oM5cOagzM9bsZ+bag8y4+0LO6BC4OkV+YQn3vruaWesPVvlZs38+imYJblolJ5CSGHyWb8ULr7ZPzfT/Iit/xe7VQ7pwbUY3hvdqg8dj6XW/d+hqyuVncce3elf7vcqcKirl+fnb+OGoXhE5aSyNnwJdYlJRiSdg6GRQt5asLrcGzT9vH8bQnq1rfPFWqcfS+/7KJ3CDuaBPG3q0SWFoemvG9+/IqaJSbnhlMRsPHK+074d3jmT5riM8MXNj0M/5altOwLbF94+hfVoiJR5baWE2AI/HMn31fvbknuQZ3w1PBnRpwX/+98Ia1S7OpkCXmPXesr388n3v1bG7nprgX+Om7HltTftqJwC7c07y2te7ABhzVnvmBhmLD6VVcjyDurVkwea63WM2LSmO5AQ3h44XAvD4xHO4aXgPPBb+PG8r6/cfD1h6uYzW2WkaFOgSs6y1/PydVZzXoxU3jUgHvCdq492GNhFcGfKFBdt4elboC7fKW/rAWPIKS/jeXxdyOK8w4LVXJ2XQv0sLhj4ReIXvzSN6cH9mP856KPjMHvCukhmqhtYpCSx/cKxOtjYBCnSRCHrtq510bZXMqDPbsWxXLje8shiA+zPP4rYLewWc/D2SX8QHK/aRnBDH9zK6+odQyg8XffHLS+jeJhmA0b9fwI7D+TWqo+w8ws0jevDGwt28+6MRDO3Zutbfx1rLD15bys0j03WxlQMo0EXq0ZH8IrZm5dUpTIPdVLzsxOryB8eyfv9xWqckBNwBCyBzQEdeuPE8ANbvP8aE57yv12bY5URBMQMe/Sxgm4ZtGr+aBroW5xKpg1YpCXUKcyDoVM7pd11AXmEJbVITGXVmOwAeyOzHEzM3smNqJst2H2FI95b+/c/p3ML/eMHmLC6uope9au9RVu89yi0j0wNuYlImJ68wokNVEj3qoYs41D3vruLDFd8A8PAVZ3PFwE60b+5dLK1sumTZlbrBZA7oyMy1B/mfjG7cflHPSlM7pfHQWi4iMW54z9Nr4z/+8QaGTp3Lum+OYa3lF++tBggZ5jufzOTxif0BeGfZXi794xe8sGBbnepYvfcov521qV7Xtr/vwzX+Nfk9nsDjnCgoZuH2HK2tj4ZcRBzru+d1pcRjuf9fa/3bKo67h2KMoW1qIuent2LpriMAvDB/O3eM6o3LZbDW8uW2w4zs3Tbk1b5Pz9rECwu2+593SEtk0gU9w/hGwc1ad5C3luz1P/9k3UHGnt2eC56ax+G8wDtf7ZiaiSuMq5OdToEu4lBul+GGYd0p8XhwGcOD/14Xct9Nvx5PYpyLT9cfYky/0+Pt024dyuebs7n77ZXkFZbQ6/6ZNE+Ko3ubZNZ9c5zbL+xJnNvFnI2HmHPPtwDvWjtZJwoCwhzgeEFJxL9jQXEpd/xjOQAtmsVz7FQxP6ni1ouTXlvKGz+ofkXQcFlrWbg9hxG92zSqaaMaQxeJEV9tO8yNvumU4J0bf7ygmC4tm1V7s5C7/rmCj9ccqPOxu7RsxjdHT7HzyUxOFJbgNibkUgnlFZV4ePm/O/j+sB60SK68jMHcjYf8N1fZ+WRm0GWZL+7bjj7tUnnlS+8FYisfupSfvrOKn489g1V7j3LjsB4kxNV+dPlUUSn9Hp7Fj0b14r7MfoD3vMWpolIu6NOWB/+9jmevG0xeYQk3DO1er8GuaYsiTdDVL3xF+7Qk/nrTebV6X05eIX9ftJs/zdlaq/e9efswurVKZtTv5ld6rWw6ZEmphxlrD5DeJoVB3VpSXOphb+5J2qQmMuixzwI+a2TvNhSWeEiKd7M396R/Oeavp4ymc8tmfLB8H/f6zg+0TU3ggx+PpEebFCD0zcxvHtHDf76gJvYfPUWnFknMWneQH7/p/d/Aziczmbsxi9vfCJ1fGx6/jOSE+hn0UKCLSK0dOl7Aoh05PDt3K1OvGsB1Ly3iiav6s3bfMd5eujdg32evG8zEwV0AeP3rXTwyPfAE7I3DunPX6D6MeHJewPY2KQnk5AeOfZcZ26+DfwXN8sovb1xQXEpBcSktkxMC9inrUVeUkuBm/ePjQ35nay2Ld+aSV1BCUamHO98MPaRTnfqa069AF5F64fFY5m/O4pK+7QNOQPZ98BMKSyJ/F6ipVw3ghmHda7Tv9S8tYuGOnErbQ91+cNa6A9zxj9oF+OBuLVm19yivTsqgfVoSLZrF+/8nsfk340mMczNs6hwOHS+M2Dr7CnQRaVBZJwqYueYAN49I9y8NXObFG4fQt2Mao//wecD2JQ+MoW1KIkWlHv+aNn3ap/pvID68V2venjyiVnXk5BWSEOfihpcXM3FwZ34zw7vi5cL7RvOHz7ZwXo9WXD+0O7n5RVWuoQ/eJRp++MYyNh86AUCvtinM+8XFlfZ7+KN1vLFwd9DPKL+ccl0p0EUkqg7nFTL69wt45Zbz/VfVFhSX8vtPN/Ojb/WmXVrg1akFxaX8cc4WJl/Ui6R4N3e+uYL7Ms/irI7Nw6oj1Nh6ecN7tWbRjlzA28v++8LdHC8o4Z5Lz6S41MMfPtvCVed2oU/71KDhfODYqUpDS+VNGNiJ528YUufvoEAXEQH+vfIbfvbOqpCvl91JqqC4lOwThXRrnVyn41hrWb77CDn5RfTtkMaunHwmTVvqf/2tHw5nRO82VXxCaLpSVEQEmDj49G0E/3LDuUybdL7/+cqHLvXfFjAp3l3nMAfvxVoZ6a257JyOpLdNYWDXlgGvN29W/5f9qIcuIlJPXlywnQv7tOWMDqnVXgtQFa22KCISZT++uOb3io0EDbmIiMQIBbqISIxQoIuIxAgFuohIjFCgi4jEiDoHujGmmzFmvjFmozFmvTHmp5EsTEREaiecaYslwL3W2hXGmDRguTFmtrV2Q4RqExGRWqhzD91ae8Bau8L3+ASwEegSqcJERKR2InJhkTEmHTgXWBzktcnAZN/TPGPM5jocoi1wuK71xTi1TXBql+DULqE15rbpUZOdwr703xiTCnwOPGGt/TCsDwt9jGU1uey1KVLbBKd2CU7tElostE1Ys1yMMfHAB8Cb9RXmIiJSM+HMcjHA34CN1tpnIleSiIjURTg99AuAm4DRxphVvj+ZEaqropfq6XNjgdomOLVLcGqX0BzfNg26fK6IiNQfXSkqIhIjGn2gG2PGG2M2G2O2GWOmRLue+maMedUYk2WMWVduW2tjzGxjzFbfz1a+7cYY85yvbdYYY4aUe88tvv23GmNuicZ3iaRQVyarbcAYk2SMWWKMWe1rm8d823saYxb7vuc7xpgE3/ZE3/NtvtfTy33Wfb7tm40xl0XnG0WWMcZtjFlpjPnY9zx228Va22j/AG5gO9ALSABWA2dHu656/s6jgCHAunLbngam+B5PAX7re5wJfAIYYDiw2Le9NbDD97OV73GraH+3MNulEzDE9zgN2AKcrbax+L5jqu9xPN7rQYYD7wLX+bb/Ffix7/GdwF99j68D3vE9Ptv3bywR6On7t+eO9veLQPvcA/wT+Nj3PGbbpbH30IcC26y1O6y1RcDbwMQo11SvrLVfALkVNk8EXvc9fh34Trntb1ivRUBLY0wn4DJgtrU211p7BJgNjK//6uuPDX1lstrGK8/3NN73xwKjgfd92yu2TVmbvQ+M8c1amwi8ba0ttNbuBLbh/TfoWMaYrsAE4BXfc0MMt0tjD/QuwN5yz/fRNJcX6GCtPQDeYAPa+7aHap+YbrcKVyarbfAPK6wCsvD+ktoOHLXWlvh2Kf89/W3ge/0Y0IbYbJs/Ab8CPL7nbYjhdmnsgW6CbNO0nNNCtU/MtpvvyuQPgJ9Za49XtWuQbTHbNtbaUmvtYKAr3t5jv2C7+X42ibYxxlwBZFlrl5ffHGTXmGmXxh7o+4Bu5Z53BfZHqZZoOuQbLsD3M8u3PVT7xGS7hbgyWW1TjrX2KLAA7xh6S2NM2XpN5b+nvw18r7fAO8wXa23ioPKUAAABSklEQVRzAXClMWYX3uHa0Xh77DHbLo090JcCZ/jOSifgPVExPco1RcN0oGw2xi3AR+W23+yb0TEcOOYbdvgUGGeMaeWb9THOt82xqrgyWW1jTDtjTEvf42bAWLznGOYD1/h2q9g2ZW12DTDPes/+TQeu88326AmcASxpmG8Redba+6y1Xa216XizY5619kZiuV2ifVa2uj94ZytswTsm+EC062mA7/sWcAAoxtszuA3vON5cYKvvZ2vfvgZ43tc2a4GMcp/zA7wnb7YBt0b7e0WgXS7E+9/cNcAq359MtY0FGAis9LXNOuBh3/ZeeINnG/AekOjbnuR7vs33eq9yn/WAr802A5dH+7tFsI0u5vQsl5htF10pKiISIxr7kIuIiNSQAl1EJEYo0EVEYoQCXUQkRijQRURihAJdRCRGKNBFRGKEAl1EJEb8f6jvc8+z15obAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.sched.plot_loss()" ] }, { "cell_type": "code", "execution_count": 194, "metadata": {}, "outputs": [], "source": [ "learn.save('bidir')" ] }, { "cell_type": "code", "execution_count": 195, "metadata": {}, "outputs": [], "source": [ "learn.load('bidir')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Test**" ] }, { "cell_type": "code", "execution_count": 197, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "quelles composantes des différents aspects de la performance devraient être mesurées , quelles données pertinentes recueillir et comment ? _eos_\n", "which components within various performance areas should be measured , whatkinds of data are appropriate to collect , and how should this be done ? _eos_\n", "which aspects of should should should be be and and how how how be be be ? ? _eos_ _eos_\n", "\n", "le premier ministre doit - il nommer un ministre d’ état à la santé mentale , à la maladie mentale et à la toxicomanie ? _eos_\n", "what role can the federal government play to ensure that individuals with mental illness and addiction have access to the drug therapy they need ? _eos_\n", "who is the minister minister minister to minister mental mental mental mental mental health health ? ? ? _eos_\n", "\n", "quelles sont les conséquences de la hausse des formes d’ emploi non conformes aux normes chez les travailleurs hautement qualifiés et chez ceux qui occupent des emplois plus marginaux ? _eos_\n", "what is the impact of growing forms of non - standard employment for highly skilled workers and for those employed in more marginal occupations ? _eos_\n", "what are the implications of of of of of workers workers workers workers workers workers workers in less workers ? ? ? _eos_ _eos_\n", "\n", "que se produit - il si le gestionnaire n’ est pas en mesure de donner à l’ employé nommé pour une période déterminée un préavis de cessation d’ emploi d’ un mois ou s’ il néglige de le\n", "what happens if the manager is unable to or neglects to give a term employee the one - month notice of non - renewal ? _eos_\n", "what happens if the employee of the the the the of of of or or or or of of of\n", "\n", "quelles personnes , communautés ou entités sont considérées comme potentiels i ) bénéficiaires de la protection et ii ) titulaires de droits ? _eos_\n", "which persons , communities or entities are identified as potential ( i ) beneficiaries of protection and / or ( ii ) rights holders ? _eos_\n", "which communities are are or as or or or or or , , of ? ? ? ? ?\n", "\n", "quelles conditions particulières doivent être remplies pendant l’ examen préliminaire international en ce qui concerne les listages des séquences de nucléotides ou d’ acides aminés ou les tableaux y relatifs ? _eos_\n", "what special requirements apply during the international preliminary examination to nucleotide and / or amino acid sequence listings and / or tables related thereto ? _eos_\n", "what special requirements requirements be be for for for / sequence / or or sequence or or sequence sequence or or sequence sequence ? ? _eos_ _eos_\n", "\n", "pourquoi cette soudaine réticence à promouvoir l’ égalité des genres et à protéger les femmes de ce que , dans la plupart des cas , on peut qualifier de violations grossières des droits humains ? _eos_\n", "why this sudden reluctance to effectively promote gender equality and protect women from what are – in many cases – egregious human rights violations ? _eos_\n", "why is such such such women women of women , , , , rights rights ? ? ? ? ? _eos_ _eos_\n", "\n", "pouvez - vous dire comment votre bagage culturel vous a aidée à aborder votre nouvelle vie au canada ( à vous adapter au mode de vie canadien ) ? _eos_\n", "what are some things from your cultural background that have helped you navigate canadian life ( helped you adjust to life in canada ) ? _eos_\n", "what is your you to you you to to to to to life life life life in life canada ? ? ? ? _eos_\n", "\n", "selon vous , quels seront , dans les dix prochaines années , les cinq enjeux les plus urgents en matière d' environnement et d' avenir viable pour vous et votre région ? _eos_\n", "which do you think will be the five most pressing environmental and sustainability issues for you and your region in the next ten years ? _eos_\n", "what do you see the the the the the the , , future and and and and and future future future future future ? ? ? ? _eos_\n", "\n", "dans quelle mesure l’ expert est-il motivé et capable de partager ses connaissances , et dans quelle mesure son successeur est-il motivé et capable de recevoir ce savoir ? _eos_\n", "what is the expert ’s level of motivation and capability for sharing knowledge , and the successor ’s motivation and capability of acquiring it ? _eos_\n", "what is is expertise of the of and and and and and and and and and and and and and and and and ? ? ?\n", "\n" ] } ], "source": [ "x, y = next(iter(val_dl))\n", "probs = learn.model(V(x))\n", "preds = to_np(probs.max(2)[1])\n", "\n", "for i in range(180, 190):\n", " print(' '.join([ fr_itos[o] for o in x[:, i] if o != 1 ]))\n", " print(' '.join([ en_itos[o] for o in y[:, i] if o != 1 ]))\n", " print(' '.join([ en_itos[o] for o in preds[:, i] if o != 1 ]))\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Teacher forcing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When the model starts learning, it starts out not knowing anything about the different languages. It will eventually get better, but in the beginning it doesn't have a lot to work with.\n", "\n", "**idea:** what if we force feed the correct answer in the beginnging?" ] }, { "cell_type": "code", "execution_count": 198, "metadata": {}, "outputs": [], "source": [ "class Seq2SeqStepper(Stepper):\n", " def step(self, xs, y, epoch):\n", " self.m.pr_force = (10 - epoch) * 0.1 if epoch < 10 else 0\n", " xtra = []\n", " output = self.m(*xs, y)\n", " if isinstance(output, tuple):\n", " output, *xtra = output\n", " self.opt.zero_grad()\n", " loss = raw_loss = self.crit(output, y)\n", " if self.reg_fn:\n", " loss = self.reg_fn(output, xtra, raw_loss)\n", " loss.backward()\n", " if self.clip: # gradient clipping\n", " nn.utils.clip_grad_norm(trainable_params_(self.m), self.clip)\n", " self.opt.step()\n", " \n", " return raw_loss.data[0]" ] }, { "cell_type": "code", "execution_count": 199, "metadata": {}, "outputs": [], "source": [ "class Seq2SeqRNN_TeacherForcing(nn.Module):\n", " def __init__(self, vecs_enc, itos_enc, em_sz_enc, vecs_dec, itos_dec, em_sz_dec, nh, out_sl, nl=2):\n", " super().__init__()\n", " self.nl, self.nh, self.out_sl = nl, nh, out_sl\n", " self.emb_enc = create_emb(vecs_enc, itos_enc, em_sz_enc)\n", " self.emb_enc_drop = nn.Dropout(0.15)\n", " self.gru_enc = nn.GRU(em_sz_enc, nh, num_layers=nl, dropout=0.25)\n", " self.out_enc = nn.Linear(nh, em_sz_dec, bias=False)\n", " self.emb_dec = create_emb(vecs_dec, itos_dec, em_sz_dec)\n", " self.gru_dec = nn.GRU(em_sz_dec, em_sz_dec, num_layers=nl, dropout=0.1)\n", " self.out_drop = nn.Dropout(0.35)\n", " self.out = nn.Linear(em_sz_dec, len(itos_dec))\n", " self.out.weight.data = self.emb_dec.weight.data\n", " self.pr_force = 1. # new for teacher forcing\n", "\n", " def forward(self, inp, y=None): # argument y is new for teacher forcing\n", " sl, bs = inp.size()\n", " h = self.initHidden(bs)\n", " emb = self.emb_enc_drop(self.emb_enc(inp))\n", " enc_out, h = self.gru_enc(emb, h)\n", " h = self.out_enc(h)\n", " \n", " dec_inp = V(torch.zeros(bs).long())\n", " res = []\n", " \n", " for i in range(self.out_sl):\n", " emb = self.emb_dec(dec_inp).unsqueeze(0)\n", " outp, h = self.gru_dec(emb, h)\n", " outp = self.out(self.out_drop(outp[0]))\n", " res.append(outp)\n", " dec_inp = V(outp.data.max(1)[1])\n", " \n", " if (dec_inp == 1).all():\n", " break\n", " if (y is not None) and (random.random() < self.pr_force): # new for teacher forcing\n", " if i >= len(y):\n", " break\n", " dec_inp = y[i]\n", " return torch.stack(res)\n", "\n", " def initHidden(self, bs):\n", " return V(torch.zeros(self.nl, bs, self.nh))" ] }, { "cell_type": "code", "execution_count": 200, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n", "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n" ] } ], "source": [ "rnn = Seq2SeqRNN_TeacherForcing(fr_vecd, fr_itos, dim_fr_vec, en_vecd, en_itos, dim_en_vec, nh, enlen_90)\n", "learn = RNN_Learner(md, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n", "learn.crit = seq2seq_loss" ] }, { "cell_type": "code", "execution_count": 201, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "beee982a4da04610b5522175ee71409b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=12), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss \n", " 0 3.972275 11.894288 \n", " 1 3.75144 8.904335 \n", " 2 3.147096 5.737202 \n", " 3 3.205919 4.434411 \n", " 4 2.89941 4.337346 \n", " 5 2.837049 4.195613 \n", " 6 2.9374 3.801485 \n", " 7 2.919509 3.679037 \n", " 8 2.974855 3.600216 \n", " 9 2.98231 3.551779 \n", " 10 2.871864 3.418646 \n", " 11 2.674465 3.432893 \n" ] }, { "data": { "text/plain": [ "[array([3.43289])]" ] }, "execution_count": 201, "metadata": {}, "output_type": "execute_result" } ], "source": [ "learn.fit(lr, 1, cycle_len=12, use_clr=(20, 10), stepper=Seq2SeqStepper)" ] }, { "cell_type": "code", "execution_count": 202, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VNXh//H3yU4IS4CwLwFl30Qim4oiCAiura22dalV0VpbtV+1uNVd0C7aurW0ys99ResCIiigAgKGHdn3VRIMgUAI2c7vj5lMZpKZzAQymTvJ5/U8PNy5c+fOmQv55My5ZzHWWkREJHrERLoAIiJSPQpuEZEoo+AWEYkyCm4RkSij4BYRiTIKbhGRKKPgFhGJMgpuEZEoo+AWEYkyceE4aYsWLWx6eno4Ti0iUictXbr0gLU2LZRjwxLc6enpZGZmhuPUIiJ1kjFmR6jHqqlERCTKKLhFRKKMgltEJMoouEVEooyCW0Qkyii4RUSijIJbRCTKOCq4//nlJr7amB3pYoiIOJqjgvvFeVuYv0nBLSJSFUcFd1ysobhUixeLiFTFWcEdYyhRcIuIVMlRwR0bE6Mat4hIEI4K7rgYQ0mJgltEpCqOCu7YGLVxi4gE46jgjos1lJSWRroYIiKO5qjgVo1bRCQ4RwW3epWIiATnqOBWrxIRkeAcFdyqcYuIBBdScBtjbjPGrDHGfG+MuT1chVEbt4hIcEGD2xjTB7gRGAT0By40xnQNR2FcNW71KhERqUooNe6ewCJrbb61thj4CrgsHIWJjTEUawCOiEiVQgnuNcBwY0xzY0wyMA7oUPEgY8wEY0ymMSYzO/vEZvhz9eNWcIuIVCVocFtr1wFPArOBmcBKoNjPcVOstRnW2oy0tLQTKox6lYiIBBfSzUlr7UvW2tOttcOBHGBTOAqjXiUiIsHFhXKQMaaltTbLGNMR+AkwNByFUa8SEZHgQgpuYJoxpjlQBPzOWnswLIVRrxIRkaBCCm5r7dnhLgioxi0iEgqNnBQRiTKOCu7YmBj14xYRCcJRwa0at4hIcI4K7lit8i4iEpSjglu9SkREgnNUcKtXiYhIcI4KbrVxi4gE56jgVq8SEZHgHBXc8bGGIrVxi4hUyVHBHRcTg7WouUREpArOCu5YA0Cxat0iIgE5K7hj3MGtdm4RkYCcFdyxruIouEVEAnNUcMe7m0p0g1JEJDBHBXdcjGrcIiLBOCu4y2rcJapxi4gE4qjgjvf0KlGNW0QkEEcFd6y7qUQTTYmIBOao4I6PKWsqUY1bRCSQkILbGHOHMeZ7Y8waY8xbxpikcBRG3QFFRIILGtzGmHbAH4AMa20fIBa4MhyFKbs5mV9YHI7Ti4jUCaE2lcQBDYwxcUAysDcchVm4+QAAz83dHI7Ti4jUCUGD21q7B/grsBPYBxyy1s4KR2F+ltEBgHO6pYXj9CIidUIoTSWpwCVAZ6At0NAYc5Wf4yYYYzKNMZnZ2dknVJhWjcPSdC4iUqeE0lQyCthmrc221hYBHwDDKh5krZ1irc2w1makpZ1YjTklMQ6AwwVq4xYRCSSU4N4JDDHGJBtjDDASWBeOwsTGGFIS48grKArH6UVE6oRQ2rgXA+8Dy4DV7tdMCVeBUhLjOKIat4hIQHGhHGStfRB4MMxlAaBRUhx5Cm4RkYAcNXIS3MF9XE0lIiKBODC449VUIiJSBccFd4qaSkREquS44G6cFKfugCIiVXBccDdKiueI2rhFRAJyXHCnJMZRUFSqVXBERAJwXHA3SnL1UFQ7t4iIfw4M7ngA9SwREQnAgcFdNl+J2rlFRPxxXnAnqqlERKQqzgvusqaS4wpuERF/HBjcZTVuNZWIiPjjuOBOUa8SEZEqOS64y2rcaioREfHPccGdGBdLQlyMepWIiATguOAGaJwUz+FjqnGLiPjjyOBumhxPbn5hpIshIuJIjgzu1OR4Diq4RUT8cmRwN01OIDdfbdwiIv44MrhV4xYRCSxocBtjuhtjVnj9OWyMuT2chUpNTuBgfhHW2nC+jYhIVAq6yru1dgNwGoAxJhbYA3wYzkI1T0mgsLiUo4UlpCSGtBC9iEi9Ud2mkpHAFmvtjnAUpkxao0QAsvOOh/NtRESiUnWD+0rgrXAUxFtaShIAWYcLwv1WIiJRJ+TgNsYkABcD7wV4foIxJtMYk5mdnX1ShfLUuI+oxi0iUlF1atwXAMustfv9PWmtnWKtzbDWZqSlpZ1UoVqqqUREJKDqBPcvqIVmEoAmDeKJjzVkKbhFRCoJKbiNMcnA+cAH4S2OS0yMoajE8urC7bXxdiIiUSWkvnbW2nygeZjLUsnRwpLafksREcdz5MhJgPF920S6CCIijuTY4J6+eh8Aq3bnRrgkIiLO4tjgvmtMdwCKSzXsXUTEm2OD+4z0ZgCs3KUat4iIN8cGd7OG8QA8/MnaCJdERMRZHBvcp7ZsFOkiiIg4kmODG+DXw9JppNkBRUR8ODq4WzdJIu94MUeOa+FgEZEyjg7usjlLNEugiEg5Rwd3s4YJABzU+pMiIh5REdw/anpXEREPRwd38xRXU8lHK/ZGuCQiIs7h6OBu17QB4Br+XqoRlCIigMOD29uynQcjXQQREUdwfHC/M2EIAHkF6hIoIgJRENxtmriaS7T+pIiIi+ODu0UjV8+SXTn5ES6JiIgzOD64kxNcQ96fnbM5wiUREXEGxwe3N2vVs0REJCqC+/7xPQE4dEwjKEVEQl3lvakx5n1jzHpjzDpjzNBwF8xb2Q3K5VpUQUQk5Br3P4CZ1toeQH9gXfiKVFm/9k0AWLZDfblFRIIGtzGmMTAceAnAWltora3Vqm+HZsmc3rEpz87ZTEFRSW2+tYiI44RS4+4CZANTjTHLjTH/NcY0DHO5KrlsQDsAXl6wrbbfWkTEUUIJ7jjgdOBFa+0A4CgwseJBxpgJxphMY0xmdnZ2DRcTfjW4EwB7Dh6r8XOLiESTUIJ7N7DbWrvY/fh9XEHuw1o7xVqbYa3NSEtLq8kyAhATY+jfvgk7NRBHROq5oMFtrf0B2GWM6e7eNRKIyNLrLVIS+WbTAfXnFpF6LdReJb8H3jDGrAJOA54IX5ECy8kvBGDNnsOReHsREUcIKbittSvczSD9rLWXWmsj0i/v7jE9ANh3SO3cIlJ/RcXIyTKdW7g6s2TlaaZAEam/oiq4m6ckYIyCW0Tqt6gK7vjYGFKTEzigublFpB6LquAGyDlayJuLd0a6GCIiERN1wV1ma/aRSBdBRCQioi64bx/VFYDz/vZVhEsiIhIZURfcfzivq2c7192vW0SkPom64I6JMfx6WDoASzXNq4jUQ1EX3AA3n3MKAHe8syLCJRERqX1RGdytmyQB0DQ5IcIlERGpfVEZ3ABXntGBA0eOU1KqCadEpH6J2uA+vVMq+YUl6hYoIvVO1AZ333audSjvnrYqwiUREaldURvc3Vs1AmD5Tq38LiL1S9QGd0yM8WwXFpdGsCQiIrUraoMb4P/O7wbApqy8CJdERKT2RHVwn93NtbblC3O3RLgkIiK1J6qDu1ebxgBMX70vwiUREak9UR3cCXExdHGviqMFhEWkvojq4Aa44owOABwtLIlwSUREakdcKAcZY7YDeUAJUGytzQhnoaojtaFr2PvBo4WkJIb0cUREolp1atwjrLWnOSm0AZq7g/vsp+by7ZYfI1waEZHwi/qmkr7tm3i2f/GfRRwvVpOJiNRtoQa3BWYZY5YaYyaEs0DV1bJREv+48jTP4zV7DkWwNCIi4RdqcJ9prT0duAD4nTFmeMUDjDETjDGZxpjM7OzsGi1kMJec1o7v7hsFwMLNai4RkbotpOC21u51/50FfAgM8nPMFGtthrU2Iy0trWZLGYK0RokA/G32Ro4eL6719xcRqS1Bg9sY09AY06hsGxgNrAl3wU7ENUM7AfDRir0RLomISPiEUuNuBcw3xqwElgDTrbUzw1usE3PriFMBuPfD1azdezjCpRERCY+gwW2t3Wqt7e/+09ta+3htFOxEtGyc5Nl+ft7mgMftP1xA+sTpPDVzfW0US0SkRkV9d8CKNj1+AQDTVwWev2T0018D8MI8TU4lItGnzgV3fGwMAzo2rfKYQ8eKPNtr9hyioEh9v0UketS54AYY3as1EFqf7gufnc9Nry0Nd5FERGpMnQzusq6BFz47n9IKq8BbazEGbhrexbPvq43ZWi1eRKJGnQzunm0aeba73DvD57njxaVYC02S43n00j6e/VotXkSiRZ0M7t5tm/DR7870+1y+e/rX5PhYrh7SiQ9uGQbA0h0Ha618IiIno04GN0D/Dk25a0x3ABZtLR8GXzaqMjnBNQXsae1dNzInfrCaj1bsqeVSiohUX50NbijvPXLllEWefcfcPUgaJMQCvqvF3/b2Cq2kIyKOV6eDu6zGPaRLM88+T1OJO7gBvr5rhGe78z0z1D1QRBytTgd3fGwMF/dvy66cY559+YWuppIGXsHdsXkyi+4Z6Xnc44GZbD9wtPYKKiJSDXU6uAFObZnCntxjnrbt3HxX80njpHif41o3SeLnGe09j8/967xaK6OISHXUi+AG2OauQeccLQSgpbuvt7enLu/P1F+fUXuFExE5AfUmuDdnufppl9W8GwZYWHhEj5ae7fSJ00mfOF3LoYmIo9T54G6f2gCATVl5ABwuKCI2xvjcnKxoytUDfR53v9+Rs9iKSD1V54M7OSGO2BjDl+uyAMg5WkRqcjzGmICvOb9Xq9oqnohItdX54AY4Ja2hJ6gPHi2kaXJClccbY9g+eTxf/LF8ac3hT80NaxlFREJVL4J7QIdU9uYeo6TUknuskNTk+OAvAk5t2YhfDe4IwM6cfNInTufAkePhLKqISFD1Irj7d2jKoWNF7M09Rm5+UdAat7dbzzvV5/Ed76yo6eKJiFRLvQju9BbJgKvWvCsnn7ZNkoK8olybJg3YPnk83Vq5eqd8s+kAxSWlYSmniEgo6kVwl/XZ/mLdfo4WlvisTRmqWXec49n+++yNNVY2EZHqCjm4jTGxxpjlxphPw1mgcGif6qpxT12wHYCGVXQFrMpr1w8CXGtV3vzaUp6fu5mrX1pcabEGEZFwqk6N+zZgXbgKEk5J8b5BnRxg8E0wZ3dN82zP/P4H/vL5Br7ZdIDL/7VQ4S0itSak4DbGtAfGA/8Nb3HC599eg2oaJpxYcAPcP75npX3LdubS5d4ZFKntW0RqQag17meAu4GoTaY+7Zp4tpMTT6ypBOCGs7vw5g2D/T732ZofTvi8IiKhChrcxpgLgSxrbZVLoRtjJhhjMo0xmdnZ2TVWwJrSrmkDz3bKCTaVlBl2ague+mk/Lurflrl3nsst554CwLZsTQUrIuFngq34YoyZBFwNFANJQGPgA2vtVYFek5GRYTMzM2uynDVi2c6DvPvdLh66uHeldu+TdebkOTRKiuOKMzpQXGK50WsVeRGRYIwxS621GSEdW52luowx5wJ3WmsvrOo4pwZ3OKVPnO7zePvk8REqiYhEo+oEd73ox10byobGi4iEW7WC21o7L1htu7567NI+JMaVX86yubz9KS21pE+czsw1+2qreCJSh6jGXUOMMWx47AKeueI0n/0VA3xXTj4Pfvw9ADe/voxFW3+s1XKKSPQ7ue4VUkl6i4Z+9y/feZBrXlpCnnsFnjJXTlmk9nARqRbVuGvYaR2a+t1/2QsLK4V2mSlfbwlnkUSkjlGNOwxWPzSazVlHOF5cytIdB/nL5xsqHbPk3pHk5Bcy9plveGLGei7q35Y2TRr4OZuIiC8Fdxg0SopnQMdUAIZ0ae4J7m6tUnxmGfSepXDopDlqMhGRkCi4a8HU685g5a5cbh/Vrcrjsg4X8P6y3Tw10xX0fdo15tPfn10bRRSRKKI27lowonvLgKG96fELGNy5GeCarKostAHW7DnM07M3snr3oVopp4hEh2qNnAxVfRw5eTIKikro8cDMKo+5IqMDHy7fw4bHxla5Qr2IRCeNnIwyFedNmXvnuWx6/AKffe9k7qKwpJTZa/dX+/wz1/zg6U9eUFRyUmUVkchTcDtEU6+V5zu3aEh8rP9/mgmvLWXNntCbTr7fe4ibXy+f2PGDZXtOvJAi4ggKboeYdfvwSvu+f3gMt444tdL+C5+dH/R8eQVFFJeU8ubinT777/1w9YkXUkQcQcHtEC0bJ/HEZX2ZdUd5gDdMjOPOMd25sF8bAD73CveyexMrd+Uy/Km55BcWkz5xOmc9OYeSUkvfh2Zx6n2feY6f9JO+nu0Vu3KrLEt23nF+8sICHv7k+xr5bCJSs3RzMsr87F8L+W77QT6+9Uz6tW/qdyKr1OR4DuYXeR4P7tyMd24aynVTlzB3g2uRC2NgxQOjiYmB5IQ4YmNcNzy/257Dz/71ree1yx44n2YNE8L8qURENyfrsPvH9wLg4ucW8OOR436P8Q5tgA7NXKvc33zOKZ591kL/R2bR96FZ9HjgM7LzjtP/4Vk+oQ2wcX9eyGUrKCrxfBN45JO1fLXReSsh1XelpZb3l+7mWKFuUkczBXeUSW9ePonV+0t3+zz384z2fl9z6JgryAd3ac62SeMqPV9UYjnj8S88xwGM69sagOU7Xc0qD338Pf/9ZmvAcr2ycDs9HphJ53tmsDX7CC8v2Ma1Ly8J8VNJbely7wzufG8lL8zbHLb32HfoGPsPF9T4eY8Xl/Dzf3/L/E0Havzc0UYjJ6NME6/eJ5NnrifGwPcPj6WotJTGSfEs3XGQLdlHWfngaAqKSnjo4++5d1z5yvTGGLZPHh9wrvAy/7xyADNWf8aTM9eTkhjL/1u4HXD94hjVq5XPsV+u2++Zqhbgr7PKBxGVlFpPM4w4x7NzNvPsHFd4v3njYIad0qJar5+xeh8piXEM75bms/94cQlDJ80BYMsT4zz/9vmFxcTHxgTsLRWMdxPeH95ezrIHzj+h89QVqnFHoanXnQG4mjtKLTRIiKVxkivQ37pxCK9dP4gmDeJp1TiJF68a6Gkq8fbVXefSv0NT3r95qM/+z247m8X3jiTO6wfsgY/KQ/mGVzP547srWLbzIPsPF3D3+yu5/hXf+xkzVpevdj91wbaT/8BhdvBoIXtzj0W6GGG3Oct/s9f/vbsy5HNYa3lryU5ueWMZ17y8hINHC32eH+YObYDTH53N+h8Oc7y4hF5//pwxz3x9QuWe8GqmTxNeToX3rI90czJKedeYa2Jyqu/3HqJn68bEeNWOy26EhuqlazMqhTjAtknjHD3as+xaOr2cJ8paS+d7ZgR8vmlyPCv+PLrKcxQUlbB4Ww6xxnDVS4s9+/94fjf+MLKr55hgI4DLbqpXZcMPeRw6VsTATql0vW8GpX4i6tqhnbhzTHdSEuPqzL+Zbk7WA91bNQKotOLOierdtolPaAP8/eehn/uDW4b5fN0e0LH8h/PdzF0nX8AwKCm1PPPFRs/jf35Zc+2+4agQ+bNqd26lWi+47n/0eOAzJryaWanv/td3jfB5nFvhZrY/t7+9gmtfXuIT2gBxseX/Z7wHegVy8XMLyNyeU2n/scISco4WsuGHPMY88zU///e3/G3WBp/QvrBfGx69tA8Ar3y7g74PzaLzPTN4eb7zv9XVNLVxR6mhpzRnw/48urkDPBw6NEvmkUt6M6BDKr3aNubNJTsZ3LkZo5+u/JW3e6tGNEgoH7o/7eZhZO44yM///S1/mraaK84IfTHlQ8eK2PBDHoPck2+Fy61vLuOzNeXNOk9/sZHbRnU96fNm5RUw6PEv+cvl/diZk8+zczaz+fELfJqfakJRSSkXP7eg0nTBAHe+52r+mOVnioSOzZM939KGPzWXnTn5zF2fxYgeLSsdW1VtvUmDeN5YtJMxvVsTF2OY5+5q+uilfbDW8ueP/I8DuNzd7NGpeTI7fsxn9h3DOd/P/6kX5pUvMPLN3SPo0CyZwuJSHvjfGp/jHvl0Lef1aBlw9am6KGhwG2OSgK+BRPfx71trHwx3waRq947ryfm9WtGrbeOwvs81Q9M921cP6QTAOxOGsGhrDuP7tSHnaCE92jSiYaLrv5J3s4138BaVlIZ0Y8q7CWj2HcPpGqZfTP/9ZqtPaJcpLC4lIe7kAnZXTj4Ad72/yrNv5e5DDOyUelLnrWjHj6732bj/CJc+v4AXrzqdNk0aVLkI9U3ndPF5fNmAdvzjy008OXM9T3+xkWm/Hebz7xQotMtucB86VsTIv30FQKOkOPIKirnyjA7Ex8b4/N8BOFxQRL+HZlUqv7/Q9nbfuJ6e+zQJcTF89LszueT5BT7HrNydG3JwW2uZtmwPvds25qcvLmTWHcNpn1r5PpCThfI/9DhwnrW2P3AaMNYYMyS8xZJgEuJiOPPU6vUEqCmDuzTntlFdObVlCoM6N/PcGPXndncNdv7m4F24Kn7l/8mLC0+uoFV4bPo6z/aHtwzjkUt6A3Dp8wtOuo/zntzKXeG2ZB05qXP6s+3AUc/2il25DJ00h1/+ZxE3v77M7/H3jevJ3WN6+Owr+/dZ/0Meq3Yf4u3vypu1vLuHAmx4bCzz/zSC7+4b5ff8eQXFXNy/bcBf0I2T4rl7bPfgHwzo376JZ/vG4b6/bPp3aMrKB0cz/Q9nceUZHQC47e0VIZ0XXE1id763kgv+8Q35hSWc9eTckF/rFEGD27qU/a+Ld/+pnQY8iXq/GORqIrlu6ndBjx3w6Gyfx3kFxRwvrvmBIke91v5cdM9IBnRMZXDn5gCs3XeYj1dWfyKubQeOkj5xOne8s4I/vLW80vN3TyuvfW/Ndv045eYXkl9YeR1Say1vLt5JXkERBUUlLN/p/wbxtgOVfxks3PKjZ/vfVw9kw2NjAXj6iv7cOLxLpa6Zxhgedf/SAnjgf2tYvNV1Du9eKF/fNYLEuFjapyaT1igRgI2PXUDjJN8v7dl5/geFlZlwdhemXD2wymPuGNWNj249i+2Txwe88d6kQTy92zZh8k/7efZN/mw9T8xY5/d4b8/N3VRp34EAg9mcKqTvhMaYWGPMCiALmG2tXRzsNSIArbyWZxv7zNcBR3t6e+36QVzcvy0A3e+fyb5DNdtVb9oy18ClG87qTOsmrvJ1a5XieX722qxqnW9XTj4j/joPgA+XBw79/3y9lS/X7ee8v33Fq99u57RHZtPrz5/z549cbbZHjhezevchhk6aw70frqbvQ7OYNGMdl72w0Kd2Da6AfHbOZto1beA33B67tA+je7UiMS6W7ZPHc9kA/4OzAK4emk6P1uVNUldMWcSp987wNPXcP74nHZtXbkpIiIth1UNj2D55vKeL6otXnR7wfQDiYmMY3bs1n98+nJeuzSAp3hVBY3u3Zvvk8WybNK7a9xn6uxfo/tdXW5jy9dYqu6CWlFqKSirXOzMe+6Ja7xlpIQW3tbbEWnsa0B4YZIzpU/EYY8wEY0ymMSYzO1tDnaVcWTPE+h/yGOjnB2RzVp6nbbtnm8ac3TWNLdnltcmhk+Z4Jsa678PVJ92LoOym2Zg+rT37jDGeUaVfrCu/oWet5eqXFntqof5c9oL/Jp35fxrB1ifGefrKr913mA3uKQS8b9y9+u0OTn90Nn0e/JyLnpvPD16jDl/5dgeA5xcDQHFJKWc8/gV5BcXscfc/X/PwGK7I6OA55peDOlarm9zM24cz7JTm5e9Ratma7fpl8ZszOwd9/YjuLdk+eTxNk0Ob16Z760aM7NmK0b1c/wbtUl0LZZ9I175pFcYiPPzJWgY8MotVu13/Zx75ZC3XTXWN4j37yTk+xy69v7zZJze/kE3785i+KvA9Aqeo1l0Ya20uMA8Y6+e5KdbaDGttRlpaWqXXSv1V8SaVt5JSy6i/l9+c2vGjKyzeuGGwz3GXPr+Ad77byRuLd/LIp2vpdt9nAduiDx1zNTFsyT5C+sTp5BX4ttW2auz6qj+wo+/NQmMMvdq4bvaW/SJ5af42vtl0gCumLAr4Gcq+Zj91efnX9k9uPYv2qcnExBgy0pvRJa0hO3Py2XEg3+85QhlUcqywhKy8Ap9ZH8uWvUtJjOPJy/t5mhcqdu0MxQu/8l9bPpFzher6s1y/FEb66dESqrjYGD659SyffQfzi7j4OdcNzJcXbGPuhmzSJ05n7yHXL8XXrx/Mp78/i+YpiZ4uhlMXbOf8p7/md28uY/fBfA4XBO8mGSlBB+AYY9KAImttrjGmATALeNJa+2mg12gAjlT08cq9PPPFRk8tDlwDXir2Wnj9+sGc1dV103XG6n3c8ob/G21lrhrSkccuLZ+ytrik1CfYAFqkJLL43pHM33yAB/63hp05+fTv0JSPfndmpfPtyT3GmZPnVNoP8JfL+/Ezr1ptmZF/m0dxqeW9m4cy6PEvgcqDogJNMfDrYeme6QS8vfqbQVxTYa6XqdedwY2vZFLs1bl542MXnHQvmIpW7sr19NrIvH8ULVISa/T8FYXa4yiYeRuymLch2+/19DaiexpTrxvkeXwov4j+j8yqdFzzhgksrcWh9TU9AKcNMNcYswr4Dlcbd8DQFvHn4v5t+a3X7IRQuavZ9snjPaENMK5vG9Y+MsZnLvGKXl9UvlBEdt7xSqENrhrx/f9bzbUvL2Gnu6veTRV6KpRp17QB152Z7ve5u95f5VN7X7fvMH0e/Jwt2Ufp0qIhLRslMffOc1nu54f9ztG+i0U/+4sBdGnRkLvHdmfab4fy1E/7seLP53tqzMO7pTH/T66BMv+5xvWzvHbvYZ/QXvXQ6BoPbcCni2m4QxuokdAGOLd7S+4e251HL+1DVS0u51SYX8V7/h9vPzp4aH3QftzW2lXAgFooi9RxZe2Y/nz5f+f43Z+cEOfp/3zmqc1JToijS4uG9GjTiIc+XktBUQmHjhUxc80+/jQt8Oo+by3xHb05rm+bgMc+eFFvurZs5He1oL4PzWL5A+eT2jCBC/7xjWd/99ausOscoC/xred15ZxuLbnouflM++0wBnZK5SL3DdiBnZoxsFPlwUbtU5N9au5/+dw1eVfPNo357LazA5b/ZMXHxjDpJ31p2iBwN0+nSk6I4+ohnbhqcMeAfdD9jQ0Y2CmVpTtcvXfuuaAHkz6jfiHBAAAHGUlEQVRbD8CSbTk+4xEKi0vZkn2Enm3CO34iGM1VIrWm4gCMMoM6N+Pdm4b6eUW5bQeO0qlZsk9763uZu3wGuXjLvH8U3275keHd0uj/cPl7vnHD4JD7vxcWl1JqLQmxMcxet5+bXisf0r3ywdH88j+L+H7vYQC++OM5nNoyJdCpTpp3U8vzvzyd8f0C/+IRl8LiUr7d+iPndEujtNSyJ/cYB/ML/c6Vsisnn7OfcvXn3j55PO9+t4u7p60irVEiCyeeR3xsDEeOF9Pnwc8BuHdcDyYMP6XSeU5GdZpKNORdao33QJ0tT4yj1Fqy8o7TxqvLYCD+arK92zaptG/ST/p6+o6X1WivOzOdqQu2A9DHz2sC8W6GGNO7NZ1bNPR0y/vt60s9oQ3QvopvEzXh7QlDuNJ9gzScvyDqkoS4GE+zSEyMoUOzZL8zZYJreodWjRM9PXMuH9ieu6etIjvvOBc9O5+Ztw/3hDbAEzPW88SM9Z7H53ZPo3/7ptxxfrdK5w4H1bilVv191gZaN2nALweHPndJVa7672LPqEzv+Z+9ec9adzIzKVprmbpgO498utZn/9w7zw3YRFKTFm4+QFGprdRGK+Hh/S2n4nKAgcz/04gTHj5fnRq3gluinrWWUkuVCzaUzR8SqMZVnffybjut2ENB6o7RT3/Fxv2VR6d+8cdzGPX3rwK+7kQrB2oqkXrFGENskK7GJxvY3u/16e/PYs76LC4f2J6mAXokSPT7+NazmL12P7/3msKgLJTfuGEwHZsl07JxIoeOFfHW4l087Z4i+OjxYs+ka+Gi4Bappj7tmtCnXeht5RKdkuJjuah/W348cpxjRaX09uom6X2Du2WjWG4b1ZV2qQ3IyisIe2iDgltEpEq/DmHIP7huaNYWrYAjIhJlFNwiIlFGwS0iEmUU3CIiUUbBLSISZRTcIiJRRsEtIhJlFNwiIlEmLHOVGGOygR0n8NIWwIEaLk5doOsSmK6Nf7ou/jn5unSy1oY0g1hYgvtEGWMyQ51kpT7RdQlM18Y/XRf/6sp1UVOJiEiUUXCLiEQZpwX3lEgXwKF0XQLTtfFP18W/OnFdHNXGLSIiwTmtxi0iIkE4JriNMWONMRuMMZuNMRMjXZ5wM8a8bIzJMsas8drXzBgz2xizyf13qnu/Mcb8031tVhljTvd6zbXu4zcZY66NxGepScaYDsaYucaYdcaY740xt7n31+trY4xJMsYsMcasdF+Xh937OxtjFrs/4zvGmAT3/kT3483u59O9znWPe/8GY8yYyHyimmWMiTXGLDfGfOp+XLevi7U24n+AWGAL0AVIAFYCvSJdrjB/5uHA6cAar31PARPd2xOBJ93b44DPAAMMARa79zcDtrr/TnVvp0b6s53kdWkDnO7ebgRsBHrV92vj/nwp7u14YLH7874LXOne/y/gt+7tW4B/ubevBN5xb/dy/3wlAp3dP3exkf58NXB9/gi8CXzqflynr4tTatyDgM3W2q3W2kLgbeCSCJcprKy1XwM5FXZfArzi3n4FuNRr/6vWZRHQ1BjTBhgDzLbW5lhrDwKzgbHhL334WGv3WWuXubfzgHVAO+r5tXF/vrKVa+PdfyxwHvC+e3/F61J2vd4HRhpjjHv/29ba49babcBmXD9/UcsY0x4YD/zX/dhQx6+LU4K7HbDL6/Fu9776ppW1dh+4Agxo6d4f6PrU6evm/ho7AFftst5fG3dzwAogC9cvoi1ArrW22H2I92f0fH7384eA5tTB6wI8A9wNlLofN6eOXxenBLe/NbrV3aVcoOtTZ6+bMSYFmAbcbq09XNWhfvbVyWtjrS2x1p4GtMdVG+zp7zD33/XiuhhjLgSyrLVLvXf7ObROXRenBPduoIPX4/bA3giVJZL2u7/m4/47y70/0PWpk9fNGBOPK7TfsNZ+4N6ta+Nmrc0F5uFq425qjClb9Nv7M3o+v/v5Jria5uradTkTuNgYsx1XE+t5uGrgdfq6OCW4vwO6uu8EJ+C6afBxhMsUCR8DZb0frgU+8tp/jbsHxRDgkLu54HNgtDEm1d3LYrR7X9Rytze+BKyz1v7d66l6fW2MMWnGmKbu7QbAKFzt/3OBy92HVbwuZdfrcmCOdd2F+xi40t27ojPQFVhSO5+i5llr77HWtrfWpuPKjTnW2l9R169LpO+Olv3B1TtgI652u/siXZ5a+LxvAfuAIly/7a/H1db2JbDJ/Xcz97EGeN59bVYDGV7n+Q2uGymbgesi/blq4Lqchesr6ipghfvPuPp+bYB+wHL3dVkD/Nm9vwuugNkMvAckuvcnuR9vdj/fxetc97mv1wbggkh/thq8RudS3qukTl8XjZwUEYkyTmkqERGRECm4RUSijIJbRCTKKLhFRKKMgltEJMoouEVEooyCW0Qkyii4RUSizP8HBQj0FZtYefQAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.sched.plot_loss()" ] }, { "cell_type": "code", "execution_count": 203, "metadata": {}, "outputs": [], "source": [ "learn.save('forcing')\n", "learn.load('forcing')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Test**" ] }, { "cell_type": "code", "execution_count": 204, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "quelles composantes des différents aspects de la performance devraient être mesurées , quelles données pertinentes recueillir et comment ? _eos_\n", "which components within various performance areas should be measured , whatkinds of data are appropriate to collect , and how should this be done ? _eos_\n", "what elements of the should be be be be and and and and ? ? ? ?\n", "\n", "le premier ministre doit - il nommer un ministre d’ état à la santé mentale , à la maladie mentale et à la toxicomanie ? _eos_\n", "what role can the federal government play to ensure that individuals with mental illness and addiction have access to the drug therapy they need ? _eos_\n", "what is the minister of the the of the and and and and and and mental health ? ? ? _eos_\n", "\n", "quelles sont les conséquences de la hausse des formes d’ emploi non conformes aux normes chez les travailleurs hautement qualifiés et chez ceux qui occupent des emplois plus marginaux ? _eos_\n", "what is the impact of growing forms of non - standard employment for highly skilled workers and for those employed in more marginal occupations ? _eos_\n", "what are the implications of of of of of of workers in in in and and and workers and workers and workers ? ? _eos_ _eos_\n", "\n", "que se produit - il si le gestionnaire n’ est pas en mesure de donner à l’ employé nommé pour une période déterminée un préavis de cessation d’ emploi d’ un mois ou s’ il néglige de le\n", "what happens if the manager is unable to or neglects to give a term employee the one - month notice of non - renewal ? _eos_\n", "what if if not is not a a or or or or or or or ? ? ? ? ? ? ?\n", "\n", "quelles personnes , communautés ou entités sont considérées comme potentiels i ) bénéficiaires de la protection et ii ) titulaires de droits ? _eos_\n", "which persons , communities or entities are identified as potential ( i ) beneficiaries of protection and / or ( ii ) rights holders ? _eos_\n", "who communities or persons , as as as as ( ( ( , protection and ? ? ? _eos_\n", "\n", "quelles conditions particulières doivent être remplies pendant l’ examen préliminaire international en ce qui concerne les listages des séquences de nucléotides ou d’ acides aminés ou les tableaux y relatifs ? _eos_\n", "what special requirements apply during the international preliminary examination to nucleotide and / or amino acid sequence listings and / or tables related thereto ? _eos_\n", "what special conditions to to to to the the / / / / sequence sequence sequence of of / / / / ? ? ? ? ? _eos_\n", "\n", "pourquoi cette soudaine réticence à promouvoir l’ égalité des genres et à protéger les femmes de ce que , dans la plupart des cas , on peut qualifier de violations grossières des droits humains ? _eos_\n", "why this sudden reluctance to effectively promote gender equality and protect women from what are – in many cases – egregious human rights violations ? _eos_\n", "why encourage such such such such such such as as human human human ? ? ? _eos_ _eos_\n", "\n", "pouvez - vous dire comment votre bagage culturel vous a aidée à aborder votre nouvelle vie au canada ( à vous adapter au mode de vie canadien ) ? _eos_\n", "what are some things from your cultural background that have helped you navigate canadian life ( helped you adjust to life in canada ) ? _eos_\n", "what are the you you you to to to to to to to to your your in in in in canada ? ? ? _eos_\n", "\n", "selon vous , quels seront , dans les dix prochaines années , les cinq enjeux les plus urgents en matière d' environnement et d' avenir viable pour vous et votre région ? _eos_\n", "which do you think will be the five most pressing environmental and sustainability issues for you and your region in the next ten years ? _eos_\n", "what do you see as the most most most important future and and and and future future future ? ? ? ? _eos_\n", "\n", "dans quelle mesure l’ expert est-il motivé et capable de partager ses connaissances , et dans quelle mesure son successeur est-il motivé et capable de recevoir ce savoir ? _eos_\n", "what is the expert ’s level of motivation and capability for sharing knowledge , and the successor ’s motivation and capability of acquiring it ? _eos_\n", "what is the expert of and and and and and and and and and and and and and ? ? ? ? ?\n", "\n" ] } ], "source": [ "x, y = next(iter(val_dl))\n", "probs = learn.model(V(x))\n", "preds = to_np(probs.max(2)[1])\n", "\n", "for i in range(180, 190):\n", " print(' '.join([ fr_itos[o] for o in x[:, i] if o != 1 ]))\n", " print(' '.join([ en_itos[o] for o in y[:, i] if o != 1 ]))\n", " print(' '.join([ en_itos[o] for o in preds[:, i] if o != 1 ]))\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Attentional model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our RNN model exports the hidden state at every time step, along with the hidden state at the last time step. Initially we are only using the LAST hidden state to 'decode' into another phrase.\n", "\n", "Can we use the rest of those hidden states?\n", "\n", "**goal:** use some percentage of all hidden states and add another trainable parameter to find good answers in the model.\n", "\n", "**idea:** expecting the entire sentence to be summarized into a vector is a lot. Instead of having a hidden state at the end of the phrase, we can have a hidden state after every single word. So how do we use the hidden information after every word." ] }, { "cell_type": "code", "execution_count": 205, "metadata": {}, "outputs": [], "source": [ "def rand_t(*sz):\n", " return torch.randn(sz) / math.sqrt(sz[0])\n", "\n", "def rand_p(*sz):\n", " return nn.Parameter(rand_t(*sz))" ] }, { "cell_type": "code", "execution_count": 220, "metadata": {}, "outputs": [], "source": [ "class Seq2SeqAttnRNN(nn.Module):\n", " def __init__(self, vecs_enc, itos_enc, em_sz_enc, vecs_dec, itos_dec, em_sz_dec, nh, out_sl, nl=2):\n", " super().__init__()\n", " self.emb_enc = create_emb(vecs_enc, itos_enc, em_sz_enc)\n", " self.nl,self.nh,self.out_sl = nl,nh,out_sl\n", " self.gru_enc = nn.GRU(em_sz_enc, nh, num_layers=nl, dropout=0.25)\n", " self.out_enc = nn.Linear(nh, em_sz_dec, bias=False)\n", " self.emb_dec = create_emb(vecs_dec, itos_dec, em_sz_dec)\n", " self.gru_dec = nn.GRU(em_sz_dec, em_sz_dec, num_layers=nl, dropout=0.1)\n", " self.emb_enc_drop = nn.Dropout(0.15)\n", " self.out_drop = nn.Dropout(0.35)\n", " self.out = nn.Linear(em_sz_dec, len(itos_dec))\n", " self.out.weight.data = self.emb_dec.weight.data\n", "\n", " # these 4 lines are addition for 'attention' \n", " self.W1 = rand_p(nh, em_sz_dec) # random matrix wrapped up in PyTorch Parameter\n", " self.l2 = nn.Linear(em_sz_dec, em_sz_dec) # this is the mini NN that will calculate the weights\n", " self.l3 = nn.Linear(em_sz_dec + nh, em_sz_dec)\n", " self.V = rand_p(em_sz_dec)\n", "\n", " def forward(self, inp, y=None, ret_attn=False):\n", " sl, bs = inp.size()\n", " h = self.initHidden(bs)\n", " emb = self.emb_enc_drop(self.emb_enc(inp))\n", " enc_out, h = self.gru_enc(emb, h)\n", " h = self.out_enc(h)\n", "\n", " dec_inp = V(torch.zeros(bs).long())\n", " res, attns = [], [] # attns is addition for 'attention'\n", " w1e = enc_out @ self.W1 # this line is addition for 'attention'. matrix product.\n", "\n", " for i in range(self.out_sl):\n", " # these 5 lines are addition for 'attention'.\n", " \n", " # create a little neural network.\n", " # use softmax to generate the probabilities.\n", " w2h = self.l2(h[-1]) # take last layers hidden state put into linear layer\n", " u = F.tanh(w1e + w2h) # nonlinear activation\n", " a = F.softmax(u @ self.V, 0) # matrix product\n", " attns.append(a)\n", " # take a weighted average. Use the weights from mini NN.\n", " # note we are using all the encoder states\n", " Xa = (a.unsqueeze(2) * enc_out).sum(0)\n", " \n", " emb = self.emb_dec(dec_inp)\n", " # adding the hidden states to the encoder weights\n", " wgt_enc = self.l3(torch.cat([emb, Xa], 1)) # this line is addition for 'attention'\n", " \n", " outp, h = self.gru_dec(wgt_enc.unsqueeze(0), h) # this line has changed for 'attention'\n", " outp = self.out(self.out_drop(outp[0]))\n", " res.append(outp)\n", " dec_inp = V(outp.data.max(1)[1])\n", " if (dec_inp==1).all():\n", " break\n", " if (y is not None) and (random.random() < self.pr_force): # why is teacher forcing logic still here? bug?\n", " if i >= len(y):\n", " break\n", " dec_inp = y[i]\n", "\n", " res = torch.stack(res)\n", " if ret_attn:\n", " res = res, torch.stack(attns) # bug?\n", " return res\n", "\n", " def initHidden(self, bs):\n", " return V(torch.zeros(self.nl, bs, self.nh))" ] }, { "cell_type": "code", "execution_count": 224, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n", "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n" ] } ], "source": [ "rnn = Seq2SeqAttnRNN(fr_vecd, fr_itos, dim_fr_vec, en_vecd, en_itos, dim_en_vec, nh, enlen_90)\n", "learn = RNN_Learner(md, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n", "learn.crit = seq2seq_loss" ] }, { "cell_type": "code", "execution_count": 208, "metadata": {}, "outputs": [], "source": [ "lr = 2e-3" ] }, { "cell_type": "code", "execution_count": 209, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "17cb27cf39134f16ae7c87780a8764a2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=15), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss \n", " 0 3.780541 14.757052 \n", " 1 3.221531 5.661915 \n", " 2 2.901307 4.924356 \n", " 3 2.875144 4.647381 \n", " 4 2.704298 3.912943 \n", " 5 2.69899 4.401953 \n", " 6 2.78165 3.864044 \n", " 7 2.765688 3.614325 \n", " 8 2.873574 3.417437 \n", " 9 2.826172 3.370511 \n", " 10 2.845763 3.293398 \n", " 11 2.66649 3.300835 \n", " 12 2.697862 3.258844 \n", " 13 2.659374 3.267969 \n", " 14 2.585613 3.240595 \n" ] }, { "data": { "text/plain": [ "[array([3.24059])]" ] }, "execution_count": 209, "metadata": {}, "output_type": "execute_result" } ], "source": [ "learn.fit(lr, 1, cycle_len=15, use_clr=(20, 10), stepper=Seq2SeqStepper)" ] }, { "cell_type": "code", "execution_count": 210, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VNX9x/H3yUx2lkAIOxg2wZUtLogILiCCS7W11f7aamur1dZqa6tQxa0u1LZq3UpptbWtVVtrrQqiIIvihmHf97AvgUACgZDt/P6YO5OZZCaZhAyz5PN6njyZuXNn5ns1fObMueeca6y1iIhI/EiKdgEiItI4Cm4RkTij4BYRiTMKbhGROKPgFhGJMwpuEZE4o+AWEYkzCm4RkTij4BYRiTPuSLxohw4dbG5ubiReWkQkIS1cuHCftTYnnH0jEty5ubnk5+dH4qVFRBKSMWZLuPuqq0REJM4ouEVE4oyCW0Qkzii4RUTijIJbRCTOKLhFROKMgltEJM7EVHA/8+F65q0rjHYZIiIxLaaCe8q8jcxfr+AWEalPTAV3siuJiipdvFhEpD5hBbcx5g5jzApjzEpjzJ2RKibZlUR5VXWkXl5EJCE0GNzGmNOBHwBnAwOBy40x/SJRTIrLUF6p4BYRqU84Le5TgM+ttUestZXAPODqSBST7E6iQi1uEZF6hRPcK4ALjDHZxpgMYBzQIxLFpLgU3CIiDWlwWVdr7WpjzK+BmcBhYClQWXs/Y8zNwM0APXv2bFIxya4kyit1clJEpD5hnZy01r5orR1irb0AKALWB9lnqrU2z1qbl5MT1lrgdSS7kzhWWdWk54qItBRhXUjBGNPRWrvXGNMTuAYYFoliUt1JOjkpItKAcK+A8x9jTDZQAfzIWnsgEsWkupM4fKxOL4yIiPgJK7ittSMiXQh4gnv/YbW4RUTqE1MzJ1PdLvVxi4g0IMaCWzMnRUQaElPBneJO4liFgltEpD4xFdyp7iSOaVSJiEi9Yiu4k10aDigi0oCYCu4Ul2cCjrWaPSkiEkpMBXeqO4lqC5XVCm4RkVBiK7iTPeWou0REJLSYCu4Ul6ccnaAUEQktpoI7NdkFoEk4IiL1iKng9ra41VUiIhJaTAW3t49bXSUiIqHFVnC7PV0lanGLiIQWU8Gd4va2uNXHLSISSkwFd6o3uLVeiYhISLEZ3FohUEQkpJgK7hS1uEVEGhRTwe09Oak+bhGR0GIsuDWOW0SkITEZ3BrHLSISWljBbYz5qTFmpTFmhTHmVWNMWiSKqekqUXCLiITSYHAbY7oBPwHyrLWnAy7gukgUo9UBRUQaFm5XiRtIN8a4gQxgZySKqVkdUCcnRURCaTC4rbU7gN8CW4FdQLG19oOIFJNkSHYZdZWIiNQjnK6SdsBVQC+gK5BpjPlWkP1uNsbkG2PyCwsLm1xQiitJXSUiIvUIp6vkEmCztbbQWlsBvAmcV3sna+1Ua22etTYvJyenyQWlJrvUVSIiUo9wgnsrcK4xJsMYY4CLgdWRKijVnaSZkyIi9Qinj/sL4A1gEbDcec7USBWU4k6iXGuViIiE5A5nJ2vtA8ADEa4FgDS3i6Pl6ioREQklpmZOAmSkujhaoeAWEQkl5oI7M8VN6bHKaJchIhKzYi64M1JcHFFXiYhISDEX3JmpbkrL1eIWEQkl5oI7I8XFkWNqcYuIhBJzwa0Wt4hI/WIuuDNSXJRVVFNVbaNdiohITIq54M5M8QwtP6JWt4hIUDEX3BmpnospHCpTcIuIBBNzwd2hVSoARaXlUa5ERCQ2xVxw13SVaGSJiEgwMRfc6SmerhL1cYuIBBdzwZ2Z6g1utbhFRIKJueDOSFZXiYhIfWIuuDN9o0oqolyJiEhsirngbpOeDEDxUQW3iEgwMRfcya4kWqW6FdwiIiHEXHADtE1PVnCLiIQQk8HdJj2ZEgW3iEhQMRncbdPVVSIiEkqDwW2M6W+MWeL3U2KMuTOSRaW6XazaWRLJtxARiVsNXuXdWrsWGARgjHEBO4D/RrKoeesKve+NMSaSbyUiEnca21VyMbDRWrslEsV43XR+LwAO6aLBIiJ1NDa4rwNejUQh/s7o1haAvSXHIv1WIiJxJ+zgNsakAFcC/w7x+M3GmHxjTH5hYeFxFaWlXUVEQmtMi/syYJG1dk+wB621U621edbavJycnOMqql2mZ/ZkUala3CIitTUmuK/nBHSTQE2Lu/CwWtwiIrWFFdzGmAxgNPBmZMvxyM5MIclAYUnZiXg7EZG40uBwQABr7REgO8K1+LhdSWS3SmWPTk6KiNQRkzMnATq1SWXvIbW4RURqi9ng7to2nU37SqNdhohIzInZ4O7VIZNdxWpxi4jUFrPB3TYjmfLKasoqdAkzERF/sRvczpVwDh7RKoEiIv5iPri1vKuISKCYDe52GSkA7DusIYEiIv5iNrg7tfHMntyv9UpERALEbHC3TvN0lRwu09KuIiL+Yja4W6V6JnUeKlMft4iIv5gN7owUFwCPv7cmypWIiMSWmA1u7yXL2mUkR7kSEZHYErPBDTD61E7ktE6NdhkiIjElpoO7e7t0dhw4irU22qWIiMSMmA7ublnplJZXaRKOiIifmA7uLm3TAbTYlIiIn5gObu8knL2HNHtSRMQrpoO7jbNeicZyi4jUiOngbp3mnYSj2ZMiIl4xHdze2ZPPzd4Q5UpERGJHTAd3ZoonuHccPBrlSkREYkdYwW2MyTLGvGGMWWOMWW2MGRbpwgCSkozvdukxdZeIiED4Le7fAzOstQOAgcDqyJUU6NGrTwdgze5DJ+otRURiWoPBbYxpA1wAvAhgrS231h6MdGFe3dtlADBnzd4T9ZYiIjEtnBZ3b6AQ+IsxZrEx5s/GmMzaOxljbjbG5Btj8gsLC5utwLyT2gHw3BydoBQRgfCC2w0MAf5grR0MlAITau9krZ1qrc2z1ubl5OQ0W4GZzsgSERHxCCe4twPbrbVfOPffwBPkJ8zZvdoDcECXMRMRaTi4rbW7gW3GmP7OpouBVRGtqpZbLugNQMH+0hP5tiIiMSncUSW3A68YY5YBg4DHIldSXd7FpnYe1GJTIiJhdSBba5cAeRGuJaRu7bzBrYk4IiIxPXPSq216Mq1T3ZpBKSJCnAQ3eFrd2w8ouEVE4ie4s9LV4hYRIZ6Cu106Ow4ciXYZIiJRFzfB3alNGiVllewt0cgSEWnZ4ia4yyqqAHheU99FpIWLm+C+/aJ+ABQe1vUnRaRli5vgTnEn0S0rnUJdOFhEWri4CW6Ac3q315BAEWnx4iq4O7RKZVdxma+/W0SkJYqr4PZeg3LD3sNRrkREJHriKriH980GYL+WdxWRFiyugrt9ZgoARaU6QSkiLVdcBXd2ZioARaUVUa5ERCR64iq4W6e5cSUZtbhFpEWLq+BOSjJUVVuen7Mx2qWIiERNXAW3v4qq6miXICISFXEX3K3TPEMC+937noYFikiLFHfB/eFdI323L3lynibjiEiLE3fB3bF1WsD9VbtKolSJiEh0hBXcxpgCY8xyY8wSY0x+pItqjLW7D0W7BBGRE6oxLe4LrbWDrLVRu9q714JfXsxjV58BwMQ3l2OtjXJFIiInTtx1lQB0bJPGN8/p6bu/aV9pFKsRETmxwg1uC3xgjFlojLk5kgU1xm++diYAi7YciHIlIiInTrjBPdxaOwS4DPiRMeaC2jsYY242xuQbY/ILCwubtchQvjK4G8bAtiJdRFhEWo6wgttau9P5vRf4L3B2kH2mWmvzrLV5OTk5zVtlCMmuJDq2TmVnsS4gLCItR4PBbYzJNMa09t4GxgArIl1YuLq0TWe3gltEWhB3GPt0Av5rjPHu/09r7YyIVtUIXbPSWKMhgSLSgjQY3NbaTcDAE1BLk3Rpm877K/dgrcX5cBERSWhxORywtqpqq4sIi0iLEffBfVZuewAK9msst4i0DHEf3Gd0bwvAtiK1uEWkZYj74O7UOhVjYHeJRpaISMsQ98HtdiWRnZlC4SFdzkxEWoa4D26AfYfLeXXB1miXISJyQiREcIuItCQJEdzdstIBrVkiIi1DQgT3lYO6ArBk28EoVyIiEnkJEdzXDO4GeNaeFRFJdAkR3O0zUwAoOqyRJSKS+BIiuLMyUjAG9peWR7sUEZGIS4jgdiUZrIWP1u+LdikiIhGXEMHttbSek5PWWqqr1QsuIvEvYYL7ioFdfX3dwZz7+Ifc9PKXJ7AiEZHISJjg7tUhk6LSciqqqus8Zq1lT8kx5qwtpPRYZRSqExFpPgkT3FnpyQBsKqy7vOs/Pt/iu71lvybpiEh8S5jgPrVrG6DuKoFjnprHpP+t9N1/Y+H2E1qXiEhzS5jgPik7A4AttS6osG7P4YD7R8rVVSIi8S1hgrtT6zQA7vdrXR8qqwjY55QubbRut4jEvbCD2xjjMsYsNsa8G8mCmiopqeZCwVudfuyrnvvEt+3xa86gf6dWrNMV4UUkzjWmxX0HsDpShTSHW0b2BmDN7hIANu3zdJv8fMzJXH92T/p3bsPO4jJKarXERUTiSVjBbYzpDowH/hzZco7PTcN7AfDQO6sCtt82qi8AvXMyASjYpwsLi0j8CrfF/TRwN1B3kHQMaedMwNlx0HPh4J7tM/jKoK6+bpTcbCe4NSRQROJYg8FtjLkc2GutXdjAfjcbY/KNMfmFhYXNVmBjJLuS+M6wkwCoqKrm8LFKWqW5fY/3bO+MPFGLW0TiWDgt7uHAlcaYAuA14CJjzD9q72StnWqtzbPW5uXk5DRzmeFb45x8/GJTEQeOlNOhVarvsfQUFx1bp7JVV8oRkTjWYHBbaydaa7tba3OB64DZ1tpvRbyyJvrFpf0BWLmzGGups35Jz/YZbFFwi0gccze8S3wZ2rMdAI+/twaANLcr4PHN+0rZX1pOeWU1Ke6EGcYuIi1Io5LLWjvXWnt5pIppDv7juQHSUgKD+/x+HQDYp6vliEicSsgm5+VndvHdrr0G93l9sgGYvnzXCa1JRKS5JGRwX3dWT9/tTm3SAh4b1b8jAI9Mi+m5RCIiISVcHzd4ukOWPjCGVTtLOLd3+4DH/INc/dwiEo8SNrXapiczrE82xpg6j53dyxPmr3259USXJSJy3BI2uOvzp+/kAZ6VBI+WV0W5GhGRxmmRwd3WuVoOwN8+K4haHSIiTdEigxvgt9cOBGrGe4uIxIsWG9wX9o/etHwRkePRYoM7u1UqP76wL64kw8Q3l/HKF1safpKISAxIyOGA4TopO4OqasurC7YB2/i/c06KdkkiIg1qsS1ugIqqwFmVP319SZQqEREJX4sO7q8N7R5w/7+Ld0SpEjhaXkXuhGl89Q+fsrHwMPkFRZRXxvR1K0QkSlp0V0mKO4l/3TKMe/6zjM3OxRWqq22dhapOhJ+/sRSAhVsOcPHv5gEwpGcWw/pkk5Hi5kcX9j3hNYlIbGrRLW7wzKKc8/NRXDGwKwDnTZ4dlTq27K97VZ5FWw/y/JyN/Ob9tazYUcwTM9ZgrQ3y7ODKKjyt+NwJ07jq+U/Y6VzSTUTiW4sPbq9Jl58CwO6SspD7zFixm76/nM6/8rc163uXV1azubD+y6ld/ux8Xpi7kb9/Hv7ol+/+5Uvf7aXbDnLe5Nl8vml/k+usrbraMvrJeeROmBb0g0dEIkPB7ejYumbxqdwJ06io8vQv3/3GUnInTOOKZ+fzw38spLLacvcbyxjxxGzfRYmP19LtByktr+Lxa84A4EcX9gm57/3/Wxn2634WJKSvm/p54wsM4Z1lO1m/9zAAI38zt9leV0Tqp+D2c/Xgbr7b/e59j8nvreFf+dsBWL6jOGDfbUVHGT65ecL7Tx9tAmB4nw4UTB7PLy4dwLxfjGLyNWew+fFxAfvmtE4N9hJ1VFWH36XSFMVHK7jjtcBROLXXPm+qsgrP+jE/e30JuROm8eQHa5vldePJ8u3F/G9J9E6WS2xr0Scna/vttQMDRpZMmbexwecMnzyb6T8Zwald2zT5fT9YtQeATm1rQvmk7ExOys4E4PlvDqFdRjK/m7mOhVsOUFZRRVqyK+hrefX55XQARvTrwMfr95HiTqKq2lJVbamoqibZ1bTP7DMeeJ9DxyoDtg3vm80nG/bz+eb9nNenQ5Ne981F2yktr2LSWyvqPPbM7A3cfnG/Jtccj654bj4AS7cVk+JO4v2Vu/njt4dycqfWUa5MYkHL+ZcQBleSYfXDY+tsf+nGvHqf98oXW9h58Cgfry9s0vtmpLg4tUsbUt3Bw3j8mV04r28H3zeCtc6V7ENZ4fft4Nze2XxlUFf+fcswxp7eGYArn/ukSXXuPHi0TmgDfH9EbwDeauJwyqLScn72r6VBQ9try/7YucBzeWU1j09fzfdfzvd9O2iqyqpqlmw7CMCB0nLfyWSvlz7ZzJR5G9m8r5QxT33EobKK43o/SQwK7lrSU1zcNqqmj/mtHw3nogGdeO3mc0l1J7Fo0mgW3ncJGx69jM7ORRle+WIr502ezbdfXMAtf89v1IqD+w8f40h5FRef0rHBfYee5LkQ8rYD9YfYnDV7AchMcXHryD48fd1gBvbIIsNppa/eVRJ2ff4+3RjYZ96pTSoFk8cz6mTPui/ebqXGqq9L4Johng+rptbcHGau2sOHqz3fisoqqjj5vvf440ebmLV6D2c9OqtJr+ntyup773t85flPeP3LrQz+1cwGn/f1PzbfOQqJXwruIG6/qB/d26Xz95vOZlCPLMDTcl37yGW0z0whu1UqblcSn028qM5z31+5h/v/t5Lio+G1jCa+uRyAdhkpDe7bo30GAD/+5+J69/vdzHUALJw0OmBM+m+uHUjvnEwG98wKq7bavCNH7hk7AIB3bx8BEHCxisa0CK215E6YxkPvrArY/pOL+zH356OYeNkAHrzyNADmr9/XpJqPV0VVNT/4Wz43vZwPwIBJMwIe79AqvHMO/rYVHaHPL6fz2PSay+fd85/ldfZ77Ooz6mxr6ANsW9ERNhYebnRNEl8aDG5jTJoxZoExZqkxZqUx5qETUVg0pae4mH/PRYzoV/8KgsYY3v7x8KCPDXzoA8DTfeBtrQWTleFZG/yb5/QMuY9Xq9SaUxKhrlK/x284Y7B+8EE9sthdHHrIYyjbDxzh2dkbAE+tBZPHB5wo/da5nvqXbisO+vxg5m8IDOO/fPcsAK4d2p3cDpncMrIPbdI8/31eb8IQzOIjx9+t8ON/LvLdvj7IiBzvxK3GeOmTzQBMdU5K1zb120MZe1pnvjq0GxsevYy7Rp/M+3deENZrj3hiDhf/bh6lQbq0JHGE0+I+BlxkrR0IDALGGmPOjWxZ8ePM7llM/fZQAOb9YhRv3nae77HcCdO48/Ul3PRyfkDryquq2vq6Fxo62eh1pTNR6KX5m4M+/pwTrjcMC75gVq/sTHYVl3GkvHH/sH8/a73vtv+FKLyuHdoDgG+9+EXYr3nbP2pC8ZXvn8OF/TtSMHm875tFbbe/Wv83DS9vS37gwx9QeCj4BxzAKZNmBPQn1zbyN3N4f2XNh653eGVO61Rm/rQmSBs7gucvnxQE3T64ZxYFk8cz5rTOTPn2UFLdLtyuJG6/uB/9O7fmF5f2BzwfJsHe03+E00PvhD9sVOJPg8FtPbzfvZKdn8iONYszY07rTMHk8ZyUncmQnu2C7jP1o0385ZPNLN56wLdtewN91cH87uueC0C8MDf4iJcFm4sAuOeyAUEfz+3gGanyxIy1vrHq4fBeVHntI3VP3oLnBKvX4TBbeyNO9oxA+XzixQzvG3o0ivcaoe8s3Rnym4a/e/6zzHc7VB/08u3FHHVOLAZrmVdV25AnRJ+5bjD9OrVmsjPuPpwZqbe/upifvLq4zgfFaV3bsGjSaO4e2583bz0vxLM9vLNm3122i4/WBZ4Izy8oYniUZv02h/V7DrGruHln9lprE3a9n7D6uI0xLmPMEmAvMNNaW6dZZYy52RiTb4zJLyxs2uiKRBFsZArAQ++s4uoXPuXpWetYtPUAG5zJK//8wTlhv3ayK4lkl6dPefHWAzzy7qqAAPb+8WekBB/pOaCzZzjZXz8t4Ipn54f1nlv3H+GVLzwXVg418qVvx1a+2+GcSJy+fBfTl++md4dMOrdNq3ff12+u+YKX90jDJwNrnyQNtkyAd7gdeP5b1Lb3UE130ts/Hs5N5/fy3R/WJxuAXs6HYEN9ytXVlneW7uTtpTt9226+oDcFk8cz7ScjaJ+Zwm2j+ga9sLW/b5+b67v93b9+6RuB8tTMdXxtymcB+zb1RHG0jH7qI4Y93nwfPGUVVfSaOJ2T73sv7IZEPAkruK21VdbaQUB34GxjzOlB9plqrc2z1ubl5LTsq8ukp7j4+O4LufG83KAh/vSs9Vzzwqe+E165znjtcHkvu3b1C5/y5/mbee3LbZRXVlNWUUVJWaWvOyWYfn7jgNc0MKzQ935hTIAxxvgC9qN1hRyrrKq3JXrbK55ukk1h9BEbY7h7bP9695m3rpDrp37Oy0FC2Du7M5SnZq2rs22yc0m7l27M48zuWb6RRlO+NcS3T89sT5fOjX5LCwRTO9j75GTyy3Gn1PucYNpm1O2iAvj9h+sD7ndvlw54PrDCnRR1+FglY5/+iC8LihpdV1Os2FHMPz7fwt8/3xLwLaQxa/HU5zO/EVDLth9slteMJY0aVWKtPQjMBYI3KcWnR/sMHrzyNNJTXGx6bFxAi622xo5MGNg9cFTIpLdWMO6Zj30jHlql1T+v6sErTm3U+1U5/5gevuq0evc7K9fTpfHs7A30v28G502eHbB8gJf/Cb2nvjEwrBpuHdmHW53wXLfnELNW7WGR0+10rLKKG15awGeb9vPA256+3ezMFN/4+ztfW8JVz82vt2to7tq9Aff/t8TTOj6nl6d1nd3KM/Rx7OldfPv4/3+rPZqmyG9M9uinPgp4bNbPRoZ1zMEUTB4f0C3lr3u7dDY/Po5hvZ1vBBOn0/uX05kybyPbiurvllu05QBrdh8Kei4mXC/M3cDP/7203n3+OG8juROmcfmz87nvrRV1xu7PWr03xDNrbCs6EtDNWFZRVeck8a/erRmpdMvfF4ZTflwJZ1RJjjEmy7mdDlwC6Aq7jZCUZJh0+akUTB7PhMsG1BmO5+0/DtdJ2XVP3m3wa1VWVdXfarlxeC/GnuaZjFN7wkcwVVWWvh1b8Z1hufXuF2o53H73vhdw/8LfzvXdvnpwd8JhjCHHCcoxT33E9/+WzzUvfEruhGm8sbBut0D+fZdwthO6q3aVsHR7Mcu2H+TPH2/yHW/vDpm+E8v+reZ1e2q+iWSmhv4QTHYlcd94T8v5jAc/4K3FO3wt3GnLdtbZ/4X/G8KMO0c02CXSkCPlNZN+Xv7e2b7bl5zSCWMM1wwJ/G86+b01jHhiTp3zAxsLD/P0rHUcKqvge3/1HP/irU1rnT749kqemLGWNxZuD+iaWLWzhG/+6XO+PuUzcidMa/Di3D/4W369j5eUVTDiiTmc/+s5vm0//uciLvztXJ6YUfPa3kYEwKGySoqPVnC0/PgmS8WScKa8dwFeNsa48AT9v6y170a2rMT1w5F9+OFIT8vx9S+3suNg44fmGWMomDwe8Iytrr3A052j+zX4GrW/vhceOhZyHZQZK3fTLsTX9HB9vL6wweGVDfnK4G48/O6qOtvv/W9gq23mTy/AGBMwfBLgq38I7Ad+5vrBnOa3VMEdry3m4StP5w/OiV//iVihjD+zC49M87RS73x9CXeGuIpSx9apjDujS9DHGut31w7krn8vZehJ7Rh5cg7tM1MoKi3H7Xxwevvga8t7ZBYjT87hgStOJTPV7Vv3/elZgV0tP/z7QqY4H2jhqP3Bv2Dzfi4a0AmAcc98HPJ5Z3RrS5t0N59s2M8vxw3gseme4P3+y1/y5DcG+YaC+nveGTUFnuUXlj04xtdKf2HuRu4eO4DVu0p4PX8bWRnJDOjcms83FfmG53p5//3EqwaD21q7DBh8Amppcb5xVsNjtxtyUnYm2Zkp7C8tB8L/g3zn9vMDJpPMXrOHw8eq+GDlbh69+nS6ZWWQnuLy9TnW1/L0t+6Ryzh4pJwtRUfYXVzmG8L37RcXUDB5PEVOnQC/v25QWK/p1S4jmTO6tWX5jmIG98yq0zpc9uAYWqe6A1q094wdwK9nBG/lnd6tbcD9/y3Z6esiAbhrTP396gDZmfV3cz359YEM7JFFn5xW9e7XGN4ROOOdD4Ip3xrK1//4GcP71YzMuXVUH98HkH8ozltXyEVOYIcyY+VucidMY9Nj4yg8fIxObUKfPH5/5e46277313w2Pz6Ou99YFuQZ8PMxJ9OhVSrXnR3493+4rJJnZm9g1uq9nPngB3X+lq+d8ilfFtSMyjp0rJJeE6cH7DPprRW+pY+rqy23jurL55sW1KnhWGVVyBPt8UCLTCWATydeRP/7ZvDGD4eF/Zy0ZBfLHhzDsm3FfOvFLwJm7l3ypKdPdtNj43xfr2++oHdYr5viTqJjmzQ6Ov/Yz+7VnnMe+xCAS5/6CLczIubhq07jqkHdQr5OMMYY3rn9fN/94qMVvpbUub3bB22h3TrK0zd+01+/5ENnKYCxp3UO6K//8t5Lgg4bdIVxJaT6urkGdm9bp9uiOXRum8bC+y4hy5lte3av9iy5f7TvPng+sO6+tL/vQ6yy2vLEjPpPMv/lxrN4bs4GFm7xhOM7y3Zyx2tLuOn8Xky6PPh5kfwQJzOnfrSJf9fqwrrxvFwG9mgbsnusTa35AUWl5bTPrDkm/9AOxX+9+pk/G0l2ZvAZyW8t3tEsDadoUXAngFS3q0lf/dqkJTO8b/Cv1QA/eW0xe0s8we0d+tZYndqkceN5ufz10wLW+vUd553Uvp5nhcd/ItCfvlP/QmBPfmMQT81cxz1jB5Be6+ReTutU/vydPL7fQP9qKB/89AI6tEpl58GjXP7sfLIyklly/5gmvVa4smud0M4KsmSC/zePW0f24UBpOX/6OHDi1tyfj+KLzftZtr2YCwd0ZPHWA77g9i7b++L8zbw4fzNrfjU2YKLYtqIjvtfb+Ng4XEmGKfM2Mvm9Ncx/sm1WAAAJAklEQVRcVTNx6V+3DGNwz6wGV3f0dtW5kwyV1ZanZ63j4avqDGDz1HRDnm9UFsCw3tl11p/3flPo1SGzzsnLp2et59qhPSirrAo5dDaWmeYafuMvLy/P5uc37R+BnHhz1uzlu3+tf0jb8gfH0DpIizYcB0rL6yygtPC+S+qET1PsOHiUNmnuJtfmZa31XOGoYyvmb9jHlQO7Nkt9sWbeukJueMnTdfDM9YPrDB3dWHjY1/cdTMHk8bw4fzO/encVp3drw4odJb7tXhf9bi4lRyvYd7ic8Wd24flvDgn1cgGstSzYXMSZ3bM45X5PN94PR/bhyoFd6duxFWc+9D5lFdXcNfpkbr/Ycx5n6baDZKa66dgmlTMf9Hz7GtGvA9cM6eZr2a/eVcKzs9cz8uScoGvCrH1kbEx0mxhjFlpr62+BePdVcIuXtZb+982ge7v0OuOrj/dkzu7iMjYWHublTwuYt66QVQ+PDasrQppXWUUVAybN4O6x/bltVPALUM9duzfk2PSubdPYWWutm+zMFBZOGu27f9srC5m+3NP3/c6Pz+eM7oHnEsIRaqTTxMsGcMvI4CeN56zZy6Z9pfUOvQ312rFwsrIxwR1/3xEkYowxrHv0MgDufG0xbzkn6kL1bzZG57ZpdG6bVu/Udom8tGQXmx8fV++QxFH9O7L58XH8Z9EOxp7emVapbpZuO+i54HSQBco+vCtwXHrXtum+293apdfePSwf330hI56YU2f79gOhJ3VdOKAjF4bx2u/dMYLLfh96tEs8UHBLUE9fN5inr9NgokQUzjhyYwxfG1pzEnFgj+BLAX9418g6/euVfrM124c4OdiQHu0z+Hpe9zpT9++8pOGhrg05pUsbLj+zC+8u2+XbdqisIqC7rfRYJac98D5t0twsmjQadxhXX1q5s5jDZZWclds+5JyG5qL1uEUkLB/f7WnP3jLSs85KweTxQYc5/uLS/ow5tRML7r34uN7v2jzPipPeyWJQ96RsUz33zSF8fPeFPP0Nz5BU7wS2DXsPMWXeRk574H0ASsoqGRZi8a6SMs+kni8Liig8dIyX5hdw2yuLOM75VWFRi1tEwtKjfUZYfcGZqW6mNjDKJxxn5bb3vd/Zj87igSvqX3KhsXq0z/Atj/v49DXcNeZkvhFkzfV2Gcn85ZPN9OqQyaj+HfnVu6t40VlWuXeHTN/5oN45mQzqkXXcM2PDoeAWkZi34N5LIvK63rXfFxQUBQ3tZJehvLK6zlWavPxP4m8qLOWqgY2bm9BUCm4RabHqG9l07dDupCW7Aib1NOTg0fKGd2oG6uMWkRZt/j3Bx6Lcf8WplIa4UtT9l5/qWz7Xn/8J3UjSOG4RafGstfxn0Q5apbq49LTOvn7qkrIKbnhpAX/6Th7LdxTz+cb93HFJP99sy9W7SigqLadfp1ZUVVu6tG3a8EfQBBwRkbjTmOBWV4mISJxRcIuIxBkFt4hInFFwi4jEGQW3iEicUXCLiMQZBbeISJxRcIuIxJmITMAxxhQC4U/wr9EB2NfM5cSSRD8+SPxj1PHFt1g+vpOstTnh7BiR4G4qY0x+uDOH4lGiHx8k/jHq+OJbohyfukpEROKMgltEJM7EWnBPjXYBEZboxweJf4w6vviWEMcXU33cIiLSsFhrcYuISANiJriNMWONMWuNMRuMMROiXU+4jDEvGWP2GmNW+G1rb4yZaYxZ7/xu52w3xphnnGNcZowZ4vecG5z91xtjbojGsQRjjOlhjJljjFltjFlpjLnD2Z4Qx2iMSTPGLDDGLHWO7yFney9jzBdOra8bY1Kc7anO/Q3O47l+rzXR2b7WGHNpdI4oOGOMyxiz2BjzrnM/0Y6vwBiz3BizxBiT72xLiL/RoKy1Uf8BXMBGoDeQAiwFTo12XWHWfgEwBFjht+0JYIJzewLwa+f2OOA9wADnAl8429sDm5zf7Zzb7aJ9bE5tXYAhzu3WwDrg1EQ5RqfOVs7tZOALp+5/Adc526cAtzq3bwOmOLevA153bp/q/N2mAr2cv2dXtI/P7zh/BvwTeNe5n2jHVwB0qLUtIf5Ggx5vtAtw/oMNA973uz8RmBjtuhpRf26t4F4LdHFudwHWOrf/CFxfez/geuCPftsD9oulH+B/wOhEPEYgA1gEnINnkobb2e77+wTeB4Y5t93Ofqb236z/ftH+AboDHwIXAe869SbM8Tn1BAvuhPsb9f7ESldJN2Cb3/3tzrZ41clauwvA+d3R2R7qOOPi+J2vzYPxtEoT5hidboQlwF5gJp7W5EFrrfdKsf61+o7DebwYyCaGjw94GrgbqHbuZ5NYxwdggQ+MMQuNMTc72xLmb7Q2d7QLcJgg2xJxuEuo44z54zfGtAL+A9xprS3xXkw12K5BtsX0MVprq4BBxpgs4L/AKcF2c37H1fEZYy4H9lprFxpjRnk3B9k1Lo/Pz3Br7U5jTEdgpjFmTT37xusx+sRKi3s70MPvfndgZ5RqaQ57jDFdAJzfe53toY4zpo/fGJOMJ7Rfsda+6WxOqGMEsNYeBObi6ffMMsZ4Gzb+tfqOw3m8LVBE7B7fcOBKY0wB8Bqe7pKnSZzjA8Bau9P5vRfPh+/ZJODfqFesBPeXQD/nTHcKnpMib0e5puPxNuA9I30Dnn5h7/bvOGe1zwWKna9w7wNjjDHtnDPfY5xtUWc8TesXgdXW2if9HkqIYzTG5DgtbYwx6cAlwGpgDvA1Z7fax+c97q8Bs62nQ/Rt4DpnVEYvoB+w4MQcRWjW2onW2u7W2lw8/65mW2v/jwQ5PgBjTKYxprX3Np6/rRUkyN9oUNHuZPc7ETAOz4iFjcC90a6nEXW/CuwCKvB8Yt+Ep0/wQ2C987u9s68BnneOcTmQ5/c63wM2OD/fjfZx+dV1Pp6vi8uAJc7PuEQ5RuBMYLFzfCuA+53tvfEE0wbg30Cqsz3Nub/Beby332vd6xz3WuCyaB9bkGMdRc2okoQ5PudYljo/K735kSh/o8F+NHNSRCTOxEpXiYiIhEnBLSISZxTcIiJxRsEtIhJnFNwiInFGwS0iEmcU3CIicUbBLSISZ/4f61oy/gMisMwAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.sched.plot_loss()" ] }, { "cell_type": "code", "execution_count": 211, "metadata": {}, "outputs": [], "source": [ "learn.save('attn')" ] }, { "cell_type": "code", "execution_count": 226, "metadata": {}, "outputs": [], "source": [ "learn.load('attn')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Test**" ] }, { "cell_type": "code", "execution_count": 228, "metadata": {}, "outputs": [], "source": [ "x, y = next(iter(val_dl))\n", "probs, attns = learn.model(V(x), ret_attn=True)\n", "preds = to_np(probs.max(2)[1])" ] }, { "cell_type": "code", "execution_count": 236, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "quelles composantes des différents aspects de la performance devraient être mesurées , quelles données pertinentes recueillir et comment ? _eos_\n", "which components within various performance areas should be measured , whatkinds of data are appropriate to collect , and how should this be done ? _eos_\n", "what components of the performance should be be be data be and and how ? ? _eos_ ?\n", "\n", "le premier ministre doit - il nommer un ministre d’ état à la santé mentale , à la maladie mentale et à la toxicomanie ? _eos_\n", "what role can the federal government play to ensure that individuals with mental illness and addiction have access to the drug therapy they need ? _eos_\n", "what is the minister minister ’s minister minister to to minister to health health ? and mental mental health _eos_ _eos_ mental _eos_\n", "\n", "quelles sont les conséquences de la hausse des formes d’ emploi non conformes aux normes chez les travailleurs hautement qualifiés et chez ceux qui occupent des emplois plus marginaux ? _eos_\n", "what is the impact of growing forms of non - standard employment for highly skilled workers and for those employed in more marginal occupations ? _eos_\n", "what are the implications of of - statistics - workers - workers workers and and skilled workers workers workers older workers _eos_ ? workers ? _eos_ _eos_\n", "\n", "que se produit - il si le gestionnaire n’ est pas en mesure de donner à l’ employé nommé pour une période déterminée un préavis de cessation d’ emploi d’ un mois ou s’ il néglige de le\n", "what happens if the manager is unable to or neglects to give a term employee the one - month notice of non - renewal ? _eos_\n", "what if the manager is not to to employee employee employee a employee the employee for retirement time hours employee after a employee of ? after _eos_\n", "\n", "quelles personnes , communautés ou entités sont considérées comme potentiels i ) bénéficiaires de la protection et ii ) titulaires de droits ? _eos_\n", "which persons , communities or entities are identified as potential ( i ) beneficiaries of protection and / or ( ii ) rights holders ? _eos_\n", "who , or or or or considered as as recipients of of of protection protection protection _eos_ ? _eos_ _eos_\n", "\n", "quelles conditions particulières doivent être remplies pendant l’ examen préliminaire international en ce qui concerne les listages des séquences de nucléotides ou d’ acides aminés ou les tableaux y relatifs ? _eos_\n", "what special requirements apply during the international preliminary examination to nucleotide and / or amino acid sequence listings and / or tables related thereto ? _eos_\n", "what specific conditions conditions be be during the international examination examination in the for nucleotide or amino amino / or or ? _eos_ ? ? _eos_ tables _eos_ ?\n", "\n", "pourquoi cette soudaine réticence à promouvoir l’ égalité des genres et à protéger les femmes de ce que , dans la plupart des cas , on peut qualifier de violations grossières des droits humains ? _eos_\n", "why this sudden reluctance to effectively promote gender equality and protect women from what are – in many cases – egregious human rights violations ? _eos_\n", "why this this to to to to to to women to and and and women to , of _eos_ of many people ? ? of _eos_ ? human human\n", "\n", "pouvez - vous dire comment votre bagage culturel vous a aidée à aborder votre nouvelle vie au canada ( à vous adapter au mode de vie canadien ) ? _eos_\n", "what are some things from your cultural background that have helped you navigate canadian life ( helped you adjust to life in canada ) ? _eos_\n", "what is your your of your you to you to to in canada canada canada life canada canada canada _eos_ _eos_ _eos_ _eos_ _eos_\n", "\n", "selon vous , quels seront , dans les dix prochaines années , les cinq enjeux les plus urgents en matière d' environnement et d' avenir viable pour vous et votre région ? _eos_\n", "which do you think will be the five most pressing environmental and sustainability issues for you and your region in the next ten years ? _eos_\n", "what do you think in the next five five next , , next and and and and and and you and in ? _eos_ ? ? _eos_ ?\n", "\n", "dans quelle mesure l’ expert est-il motivé et capable de partager ses connaissances , et dans quelle mesure son successeur est-il motivé et capable de recevoir ce savoir ? _eos_\n", "what is the expert ’s level of motivation and capability for sharing knowledge , and the successor ’s motivation and capability of acquiring it ? _eos_\n", "what is the the of the the and and and and and and and to and to and and ? ? ? _eos_ _eos_\n", "\n" ] } ], "source": [ "for i in range(180, 190):\n", " print(' '.join([fr_itos[o] for o in x[:, i] if o != 1]))\n", " print(' '.join([en_itos[o] for o in y[:, i] if o != 1]))\n", " print(' '.join([en_itos[o] for o in preds[:, i] if o != 1]))\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Visualization" ] }, { "cell_type": "code", "execution_count": 237, "metadata": {}, "outputs": [], "source": [ "attn = to_np(attns[..., 180])" ] }, { "cell_type": "code", "execution_count": 248, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(29, 38)\n", "(38,)\n", "[0.00093 0.38696 0.50663 0.05482 0.00831 0.0026 0.00418 0.00047 0.00101 0.00141]\n" ] } ], "source": [ "# DEBUG\n", "print(attn.shape)\n", "\n", "# graph 1\n", "print(attn[0].shape)\n", "print(attn[0][:10])\n", "# END DEBUG" ] }, { "cell_type": "code", "execution_count": 241, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3QAAAJCCAYAAACIxIW/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XmYnNV15/HvqbWrd0nd2iUkgRAIMMIWsh1sFm+AMwNOjB3IZifOECdm7MniBGdmSELiiZeJ7cmEJGYcx9lsgokzVhzGeGOzHbAEFosEQkIIqbW2lm51d1XXeuePqre71Oqlqmuv+n2eR093Vb1d7y2kvrznPfeeY845REREREREpPH4aj0AERERERERmR8FdCIiIiIiIg1KAZ2IiIiIiEiDUkAnIiIiIiLSoBTQiYiIiIiINCgFdCIiIiIiIg1KAZ2IiIiIiEiDUkAnIiIiIiLSoBTQiYiIiIiINKhArQcwVV9fn1uzZk2thyEiZfbUU0+dcM7113ocpdD8JNJ8NDeJSD0qZm6qu4BuzZo1bN++vdbDEJEyM7NXaz2GUml+Emk+mptEpB4VMzdpyaWIiIiIiEiDUkAnIiIiIiLSoBTQiYiIiIiINCgFdCIiIiIiIg2qpIDOzG4ws91mttfM7pzm9feb2aCZ7cj9+ZVSzleqAyejjCfTtRyCiMg5RsaTHBmO1XoYIiIF23t8lEzG1XoYIkIJAZ2Z+YF7gBuBjcBtZrZxmkP/yTm3KffnC/M9X6liiTTXf+4x/vHJA7UagojItD73nT3cdu8TtR6GiEhBjg6P8/bPPso3dx6t9VBEhNIydFuAvc65fc65BHAfcHN5hlV+Lx0bIZZMMzgSr/VQRETOcnwkzuGhcZzT3W4RqX8nRuM4l83SiUjtlRLQrQAO5j0eyD031bvN7Fkze8DMVk33RmZ2u5ltN7Ptg4ODJQxpZi8ePQNANJGqyPuLiMxXNJ4ikc4wnszUeigiInOK5bavHB7SUnGRelBKQGfTPDf19vK/Amucc68BvgP87XRv5Jy71zm32Tm3ub+/oIboRXvx6AgAY3HtoROR+hJNZOeloViixiMREZmbN2cdUkAnUhdKCegGgPyM20rgcP4BzrmTzjlvjeP/AV5XwvlKsjsX0ClDJyL1xpuXhmPJGo9ERGRu0Xh2zlKGTqQ+lBLQbQPWm9laMwsBtwJb8w8ws2V5D28CXijhfCXxArqxhDJ0IlJfvHlpOKqATkTqX36GTnt/RWovMN8fdM6lzOwO4CHAD3zRObfTzO4GtjvntgIfNrObgBRwCnh/GcZctMGROCfHskuZvLtKIiL1Ijax5FIBnYjUv2huD914MsPpaJKFHaEaj0iktc07oANwzj0IPDjlubvyvv8Y8LFSzlEOXnauKxxQhk5E6s6YllyKSAOJ5W1fOTwUU0AnUmMlNRZvFF6Fy02re7WHTkTqTlRLLkWkgUTzbo4PnNY+OpFaa4mAbvfREfo6Q6xa2K4qlyJSV5LpDIlUtl2BMnQi0gjyAzoVRhGpvdYI6I6NcNHSbjpCfmXoRFqEmd1gZrvNbK+Z3TnLcbeYmTOzzdUcnyf/wkhtC0SkEUQTKRa0B2kL+hTQidSBpg/o0hnH7qMjbFjaRXsoQDSRJpNRRSaRZmZmfuAe4EZgI3CbmW2c5rgu4MPAk9Ud4aRYXkA3HNMNJxGpf9FEmvZQgOW9EQ4PK6ATqbWmD+hePTlGPJVhw9IuOsJ+AGJJLbsUaXJbgL3OuX3OuQRwH3DzNMf9EfApYLyag8s3lrdqYCiqDJ2I1L9YIk17yM+K3giHtIdOpOaaPqDzKlxelMvQwdkXUCLSlFYAB/MeD+Sem2BmVwCrnHPfqObAporm7es9oz10ItIAxvIDuqGa3Q8TkZymD+hePDqCGaxfPJmhi6owikizs2mem1hrbWY+4LPAb835Rma3m9l2M9s+ODhYxiFmeft6+zpD6kMnIg0hlkgRCflZ3hvhxGicca18Eqmppg/odh8dYc2iDiIhvzJ0Iq1jAFiV93glcDjvcRdwKfCIme0H3gBsna4winPuXufcZufc5v7+/rIP1CuKsqwnoiqXItIQook0Hbk9dABHhpWlE6ml5g/ojo2wYUkXAB25gC6m5uIizW4bsN7M1ppZCLgV2Oq96Jwbds71OefWOOfWAE8ANznntld7oJMBXRtnYkkVbRKRuhdLpInkllyCWheI1FpTB3SxRJr9J8fYsDQb0LXnllyOKaATaWrOuRRwB/AQ8AJwv3Nup5ndbWY31XZ0Z/NWDCzvjZBxMBLXCgIRqW9jidTEHjqAQwroRGoqUOsBVNKe4yM4BxcvOztDF9UFk0jTc849CDw45bm7Zjj22mqMaTrefLS8tw3IFkbpiQRrNRwRkTl5bQuW9IQxU4ZOpNaaOkP3Yq7C5Yal3QC0h5ShE5H6Ek1O7qEDGIpqH52I1DevbUE44Ke/M6zWBSI11twB3ZER2oI+Vi9sB6AjnMvQqSiKiNSJaDyNz2BxVxhAhVFEpK4lUhlSGTdxk3zFAjUXF6m1pg7odh87w4VLuvD7shXMJzJ0alsgInViLJGiIxSgtz0EwFBMzcVFpH55heUiuW0sy3sjHFYvOpGaau6A7uhkhUuAcMCHz5ShE5H6EUukaQ/7J/bNKUMnIvXMK+Q0kaHrjXBoKIZzqtArUitNG9CdGI1zYjQxUeESwMzoCAWUoRORujGWKy7gBXTaQyfSHMzsBjPbbWZ7zezOaV6/2syeNrOUmd0y5bW0me3I/dk69WdryWu1kh/QJVIZToxqdYFIrTRtlcvduYIoF+UKonjaw35l6ESkbsRy5b/bgj5CAR9nlKETaXhm5gfuAd4ODADbzGyrc25X3mEHgPcDvz3NW8Scc5sqPtB5iE0EdJNLLiFb6bI/txdYRKqraTN0kxUuu856viMUUJVLEakbY/E0HaEAZkZPJKgMnUhz2ALsdc7tc84lgPuAm/MPcM7td849C2RqMcD5ik5Zcum1XFHrApHaadqAbvfRM/R1hs65W9Qe9qsPnYjUjWgiRSR3YdQbCWoPnUhzWAEczHs8kHuuUG1mtt3MnjCzd013gJndnjtm++DgYCljLUp0oijK5JJLUHNxkVpq4oBu5JzsHGSXCIxpyaWI1IloIk1HOHth1KOATqRZ2DTPFVM1ZLVzbjPws8DnzOz8c97MuXudc5udc5v7+/vnO86iTd1D1xMJ0hHyK6ATqaGSArq5NvzmHXeLmTkz21zK+QqVzjh2Hxthw5Luc17rCPknJiMRkVqL5oqiAPS2BxlSQCfSDAaAVXmPVwKHC/1h59zh3Nd9wCPAFeUcXCm8JZcduXnLzHKtCxTQidTKvAO6vA2/NwIbgdvMbOM0x3UBHwaenO+5inXgVJTxZIaLpsvQhQOMacmliNSJsVxRFIDuSFBFUUSawzZgvZmtNbMQcCtQULVKM1tgZuHc933AVcCu2X+qemLJs5dcgnrRidRaKRm6OTf85vwR8Cmgar/pu4+eAc4tiALK0IlIfTkrQxcJMRRV6W+RRuecSwF3AA8BLwD3O+d2mtndZnYTgJldaWYDwHuAz5vZztyPXwxsN7NngIeBT0ypjllTU5dcAqxYENGSS5EaKqVtwXQbfl+ff4CZXQGscs59w8ymK8vrHXc7cDvA6tWrSxhS1otHRzCDC5fMsIdOGToRqQOpdIZEKnPWXpSxRJpkOkPQ37RbnEVagnPuQeDBKc/dlff9NrJLMaf+3A+Byyo+wHnyCsu1BfICut4Ip8YSxBLpszJ3IlIdpVwxzLrh18x8wGeB35rrjcq9sXf30RHOW9g+7aTSEc5m6JwrZm+yiEj5RZNn3+nubc82F1dhFBGpV9lVBX58vsnLwInWBcPK0onUQikB3VwbfruAS4FHzGw/8AZgazUKo8xU4RKyGbpUxpFIN1TbFxFpQtF4NqDrCGcXS/REFNCJSH2LJtNnLbcEWN6Ta11wWgGdSC2UEtDNuuHXOTfsnOtzzq1xzq0BngBucs5tL2nEcxhPptl/cowNS8+tcAnZPXQweSElIlIrY1Ma9HoBnZqLi0i9mm5Z5YoF2YBOlS5FamPeAV0hG35rYTiWJONgaXfbtK+35+6Ee0udRERqJTZRXCCXocstuVSlSxGpV2PxFO3Bs0swLOluw2cK6ERqpZSiKHNu+J3y/LWlnKtQI+PZO96dbdN/NK9vSlSFUUSkxrwCTR1TM3QxVboUkfoUS56boQv6fSzpbuOQWheI1ETTlVEbzV0gdYWnD+jaw9lJaEytC0Skxrzy397FUa+3h05LLkWkTkUTaTrC5xadW9Eb4dBQtAYjEpHmC+iUoRORBuEFdF5RlO6Joiian0SkPkUTaSLBc6+x1FxcpHaaL6CLZ+9sd86UoQspQyci9cErihIJZueloN9HZzigJZciUreiidQ5VS4hG9AdGY6RyagtlEi1NV1AN7GHbo6ALprQHXARqa3YlAwdZPfRqW2BiNQrrw/dVCt620imHYOj8RqMSqS1NV1AN7GHbqYll7kLpzG1LRCRGpvatgByAZ320IlInYol0hOVefN5rQsOVbjS5ch4kkRKvYRF8jVfQJfL0HUoQycidS4aT+MzCAcmp2Jl6ESkXjnnZl1yCZVvXfBTf/FDPvnNFyt6DpFG03wBXTxFW9BH0D/9R/PuKilDJ9LczOwGM9ttZnvN7M5pXv+gmT1nZjvM7PtmtrHaY4wm0nSEApjZxHO97UGGFNCJSB2KpzJkHOe0LYDqBHTjyTR7j4/yg70nKnYOkUbUdAHdSDxFZzg44+t+n9EW9ClDJ9LEzMwP3APcCGwEbpsmYPuyc+4y59wm4FPAZ6o8zOyd7inlv5WhE5F65VXmnS5D190WpKstwKHTlQvovGDxpWMjE308RaQJA7rR8dSM++c8HaHAxN4VEWlKW4C9zrl9zrkEcB9wc/4BzrkzeQ87gKqXZhubZi+Kt4fOOVWKE5H64t0M75hmDx14vegq17pgIBcsZhw8d2i4YucRaTTNF9DFUzNWuPS0h/1EteRSpJmtAA7mPR7IPXcWM/uQmb1MNkP34SqNbUJsmr0oPe1BEukM40lt+heR+uJV5p1uySV4vegql6E7eHqycfkzB4cqdh6RRtN8Ad343AGdMnQiTc+mee6clJdz7h7n3PnA7wL/bdo3MrvdzLab2fbBwcGyDnIsfm75755cc3H1ohORejPbkkuA5b1tFa1yOXA6RtBvrOiNsEMBnciEpgvoRuIpOudYctke8k9MSiLSlAaAVXmPVwKHZzn+PuBd073gnLvXObfZObe5v7+/jEOEaPLcJZe9kRCA9tGJSN3xbobPlKFb0dvOcCw50UKq3AZOx1jeG+G15y1QQCeSp+kCutF4kq65MnThgDbTijS3bcB6M1trZiHgVmBr/gFmtj7v4U8Ce6o4PgCi8RQd0xRFAdSLTkTqTmwiQzf9ddby3jYAjlQoS3fwVJSVCyJsWtXLkeFxjp2p3H49kUbSfAHduDJ0Iq3OOZcC7gAeAl4A7nfO7TSzu83sptxhd5jZTjPbAfwm8L5qjzOaSBMJTsnQtXtLLhXQiUh98a6dOmbM0FW2ufjA6RirFrSzaVUvgLJ0IjmzRz4NxjlXUFEU7aETaX7OuQeBB6c8d1fe9x+p+qCmiCZmydApoBOROlNIURSAwxWodDmeTHNiNM7KBREuWd5NwGfsODjE9ZcsLfu5RBpNU2Xo4qkMybSbO0OnKpciUgembVvQriWXIlKfvLYFMy257O8KYwbHR8of0HktC1YuaKct6OfiZd2qdCmS01QBnbcJd849dMrQiUiNpdIZEqnMOdXiOkMBfKYMnYjUn7E5qlwG/T4Wtoc4PhIv+7m9lgWrFmazgJtW9fLswDDpjHp2ijRVQOcVOpl7D12A8WRGk4CI1Ew0Of2Fkc9ndEeCalsgInUnlkjjMwgHZr587O8Kc/xM+QO6/AwdwOWrehmNp9g3OFr2c4k0mqYK6EbGcwFdODjrcd4FVCypZZciUhvesu+OaVYU9EaCDMe0ikBE6ks0t0zcbLpWn1n9XWEGRysR0EUJBXz0d4YBJgqj/FjLLkVKC+jM7AYz221me83szmle/6CZPWdmO8zs+2a2sZTzzcVbcjlXUZT2XBGCqFoXiEiNTO5FOXfpUk8kyFBUGToRqS+xZGrGgiie/q4wgxVoJzBwKsbK3gg+XzaYXNfXQVdbQJUuRSghoDMzP3APcCOwEbhtmoDty865y5xzm4BPAZ+Z90gLMJrL0HXNseSyI7eZd0ytC0SkRqKz9HPqaQ9xRnvoRKTOjMXTM+6f8yzuamNwNI5z5d3WMnA6yooFkYnHPp9x+cpeFUYRobQM3RZgr3Nun3MuAdwH3Jx/gHPuTN7DDqCim9YKztDlJiM1FxeRWvHmnxkzdAroRKTORKepzDvV4q4wybRjqMyVegdOxyb2z3k2rerlxaMjE+0URFpVKQHdCuBg3uOB3HNnMbMPmdnLZDN0Hy7hfHMaKbAoirdnRc3FRaRWZiqKAt4eOgV0IlJfYsnUnBm6/q7sHrdyVrqMJlKcHEuwMi9DB9mALp1xPH94uGznEmlEpQR00+2IPScD55y7xzl3PvC7wH+b9o3Mbjez7Wa2fXBwcN4DGh0vMkOn1gUiUiOzFUXpiQQ5E0uSUSVeEakj2QzdXEsuvYCufPvovAqXqxaenaG7PFcYRcsupdWVEtANAKvyHq8EDs9y/H3Au6Z7wTl3r3Nus3Nuc39//7wHNBpPEvTbrOV0IS9Dp+biIlIj3g2lSHCaDF17kIybXHUgIlIPovH0tHNWvsXdbQAMljFDN5DrQTc1Q9ffFWZFb0SVLqXllRLQbQPWm9laMwsBtwJb8w8ws/V5D38S2FPC+eY0Op6iIzx7OV1Qhk5Eas/b8zFdhq47km29osIoIlJPojVacjnZgy5yzmubVvey44ACOmlt8w7onHMp4A7gIeAF4H7n3E4zu9vMbsoddoeZ7TSzHcBvAu8recSzGImn5lxuCZNVLtW2QERqZWyWtgW9uYCu3EUFRKR6CmjtdLWZPW1mKTO7Zcpr7zOzPbk/Fb12KkYskaZ9juusznCA9pC/rM3FD56KEs7rQZdv08peDg3FypoRFGk0c0c/s3DOPQg8OOW5u/K+/0gp71+s0fHCAjqvD53aFohIrcQSaXzGtEvEe3IBnQqjiDSmvNZObye7RWWbmW11zu3KO+wA8H7gt6f87ELg94HNZGsTPJX72dPVGPtsook07XMsuYTsPrpyNhfPVriMTLsCa9PqyX10b9u4pGznFGkkJTUWrzej8dScPegAQn4fAZ9NNPYVEam2sXiajtD0S8R72nMZupiai4s0qEJaO+13zj0LZKb87PXAt51zp3JB3LeBG6ox6NlkMo5Ycu6iKJBddnm8jM3Fp2tZ4Ll0eQ9+n6nBuLS0pgvoCsnQmRntIT9jKooiIjUSTaSIzHBh1BsJAcrQiTSwglo7VeBnK2Y8lcY5iMzRhw4mm4uXy8DpKKsWnrt/DiAS8rNhSRfPDCigk9bVXAHdeIrOtmBBx3aEA8rQiUjNRBPpaQuiwOSSS+2hE2lYBbV2KuVny9XyqVDRiUJOhWXoBsu0h240nuJ0NDljhg5yhVEODqnVi7SspgroCi2KAtlCBNpDJyK1Ek3MXC2uLegjFPCpyqVI4yq2tVPRP1uulk+F8irzztW2ALIB3Ug8NfEzpZipZUG+Tat6GRlPse/EWMnnE2lETRXQjY4XtocOchk6VbkUkRoZi8+8F8XM6IkElaETaVxztnaaxUPAO8xsgZktAN6Re66mvAxde0FLLrPVKMtReXLglNeyYOYM3RVqMC4trmkCulQ6QyyZVoZORBpCNJme9cKoNxLUHjqRBlVIayczu9LMBoD3AJ83s525nz0F/BHZoHAbcHfuuZqardXKVF5z8eMjpRdGOZjL0K2aJUO3rr+TznBAhVGkZZXUtqCeeAVOCg3oOkIBjpVhohERmY9oPMWK3rYZX+9RQCfS0Apo7bSN7HLK6X72i8AXKzrAIk0suSykymVn+ZqLD5yOEQn6WdgRmvEYv89Y19/Bq6eiJZ9PpBE1TYZuJJ698OkscMllJOQnqiqXIlIj0USaSHCWDF17kCEFdCJSJyaKohSy5LK7jEsuT0dn7EGXb0l3W1lbJYg0kqYJ6EZz++G6isjQjanKpYjUSDSRmrVaXHckqKIoIlI3vMrghWToFraH8PusPEsuT8VYtXDm/XOeJd1hjimgkxbVPAHdeHaiKTRD1x72T9xtEhGptrFEetYLo95IiKGoGouLSH2YLIoyd0Dn8xl9nSGOl6F1gZehm8uSrjZOR5OMJ3VtJ62naQK6kVyGrpg9dNFEGufUs0REqiuVzpBIZWZdutQTCTKWSJNMZ6o4MhGR6RUT0EF5mosPx5KcGU8VFtD1ZPckl2OZp0ijaZqAzsvQFdq2oD3sJ51xxFO6WBJpRmZ2g5ntNrO9ZnbnNK//ppntMrNnzey7ZnZetcYWTc59YdQTyc5lKowiIvUgNlHlsrDrrP6ucMkZukOnsy0LVs3SssCzJFdZU8supRU1T0A3kaELFnS8d2dcyy5Fmo+Z+YF7gBuBjcBtZrZxymE/BjY7514DPAB8qlrj8woyzdq2oD1b0U0BnYjUg2giTcBnhAKFXTou7gqXXOXy4ERT8cL20AEcVUAnLah5Arpi99Dl7oyPqbm4SDPaAux1zu1zziWA+4Cb8w9wzj3snPNqXD/BDOXDK8ErLjBbUZSeSPbmlJqLi0g9iM6x73eqxV1hTo3FSWfmv7Vl4LTXVHzuJZdLJzJ0WnIpradpArqReAozaA8WNtl0hJWhE2liK4CDeY8Hcs/N5APA/6voiPJM7kWZZQ9dezagU6VLEakH0USq4P1zkF1ymXFwsoR9dAOno3SE/PS2z736qicSJBTwqXWBtKSmCehGx1N0hgL4fLP3KfFMZOjUukCkGU03EUx7m9jMfh7YDHx6htdvN7PtZrZ9cHCwLIMrpLjARIYupkqXIlJ70US6oB50nv6ubMaslGWXXsuCuXrQAZgZS7rDWnIpLal5Arp4suDllpCXoVNzcZFmNACsynu8Ejg89SAzexvwX4GbnHPTXnU45+51zm12zm3u7+8vy+DGJooLzNa2IBvQDWvJpYjUgVixSy7L0Fy80JYFnqXdbSqKIi2piQK6VMEtC0AZOpEmtw1Yb2ZrzSwE3ApszT/AzK4APk82mDtezcF5N5I6Zpmzur2ALqY5SkRqL5pIF7fksjMb0M23ubhzjkOnYwUVRPEs7m7THjppSU0T0I2Mp4rL0E1UudTFkkizcc6lgDuAh4AXgPudczvN7G4zuyl32KeBTuCrZrbDzLbO8HZl5807kVn2/Ab9PjrDAS25FJG6EE2kiBS15LK0DN2ZWIqReGE96DxLurIZOvUYllZT+G9mnSs6Qxf2qlxqyaVIM3LOPQg8OOW5u/K+f1vVB5Xj7aGbLUMH2X10alsgIvUgmkizrKfwDF1b0E9PJDjvPXTFtCzwLO0JE02kGY2n6GorrI2VSDMoKUNXT417R8dTBTcVB2XoRKR2CtlDB7mATnvoRKQORBPpiZvhhSqlufjAREBXRIZOzcWlRc07oKu3xr3FZui8pU7K0IlItcUSaXwG4Tka9CpDJyL1IpYsbg8dZHvRDc6zbYHXg25VMXvoutSLTlpTKRm6umrcOzqeojNceHrd5zPaQ35l6ESk6sbi2fLfc5Xi7okEGVJAJyJ1INuHrridOou7wvMuinLwVJSutsBET85CLO1Rhk5aUykBXdka95ba5ymTcYwmiiuKAtnlTmNqLC4iVRZLpgoq/93brgydiNReOuMYT2ZmLeQ0HW/J5XyKlAwUWeESsgEkoF500nJKCejK1ri31D5P0WQa56CriCWXAO2hANG4MnQiUl1j8fScBVEA+jrDnBpLkExnqjAqEZHpxZJeIadil1y2EU9lGJnHtVY2oCt8/xxkC011hQPz3rcn0qhKCejK1ri3VKPj2YlCGToRaQTZpUtzXxit6esgnXEcPBWd81gRkUqZaLVS7JLLXHPxYgOswZE4r5wcY82i4jJ0AEt61FxcWk8pAV3dNO4djWeXJBVTFAWyd3K0h05Eqq3QBr1r+7IXM/tPjlV6SCIiM4rlbn63F7vkcp7Nxf/8e3tIZxy3bVld1M8BLOkO18WSy3TGMapVYFIl8w7o6qlx70gpGTpVuRSRKhtLpAsqLrC2rxOAfYMK6ESkdrxrpaKrXHYX31z8wMkoX/7RAX7mylWs6+8s6nyQbS5eD0suf+eBZ3n7Zx4lk1GTc6m8khqL10vjXu8OSLF76DpCWmctItUXjadYnqvGNpsF7UG62wLK0IlITcWSud6ZRV5n9Xdm57liAro//fZu/D7jI29dX9S5PEt62jg+Mk4m4/D5Zq8kXCnf3nWMf356AIA9x0fZsLSrJuOQ1lFSY/F6Me89dGH/RINfEZFqiRaYoTMz1vZ18MoJBXQiUjvRxPwydN2RAKGAr+CAbufhYb6+4zC/fNXaiSbhxVrSFSaZdpyKJub186Uaiib4vX95jhW92YIu2/afqsk4pLU0RUDnVU8qeg9dKDAxSYmIVEs0kSq4Wtzavg72n1BRFBGpHe9aqdi2BWaW60VXWED3qW/upicS5FevOb/oMXq8QLBWhVHu/sYuTo0l+PwvvI7FXWEFdFIVTRHQeRm6riIai0MuQ6cNqyJSZWOJdEF96CBb6fLQUIzxpG4+iUhteAXkis3QQa4XXQFFUf795ZM8+tIgH7rufHoixV3P5VuSW85eiy0133vxGF97+hC/fu35XLqihyvXLGT7/tNVH4e0nuYI6HJBWbH9UTpCAeKpDCn1eBKRKkmlMyRSGToKLP+9tq8DgFdPKksnIrUxueSy+NILi7vCcy65dM7xiW++yLKeNn7xjWvmM8QJXoau2pUuh2NJPva159iwpIs73nIBAFeuWcChoRiHhmJVHYu0nqYJ6CJBPwF/cR/Hu9MU1Z1vEakSb74p9E63F9BpH52I1MpE24Iib5xDtrn4XEv14/gYAAAgAElEQVQuH9p5lGcODvEbb7uQtiKXdZ57vmxlzWovufz4v+3ixGiCT7/nNYQD2c+wec1CALZr2aVUWFMEdCPjqaILokC2Dx1AVK0LRKRKYkXe6V6jgE5Eaiw6zz50kF1yORRNEk9Nf62VSmf41EO7uWBxJz/92hUljRMg6PfR1xniWBWXXD6y+zj3bx/gV69ex2tW9k48f/GybjrDAe2jk4prioBuNJ4qumUBTN4hV6VLEamWsSKXiHe3BenrDLFfAZ2I1Eg0kSbk9xW9EgomM2YnRqevOvnAUwPsGxzjo9dvmNf7T3/Otqpl6M6MZ5darl/cyUfednarBb/PeO15C7SPTiquOQK68eT8MnQhZehEpLrmUy1OrQtEpJaiiVTBhZym8pqLH58mwIqn0nzuO3u4YnUv79i4pKQx5lvaU72A7i8feZljZ8b59Hsun1hqme/K8xaw+9gIw9FkVcYjrak5Arp4quiWBTC5FlwZOhGpFi+g6yhizlqzqINX1FxcRGokmkjTMc+AzmsuPt0+ugeeGuDomXF+6+0bMCtfE/Al3eGqLbn84d4TXLlmIZtW9U77+uY1C3EOnjqgZZdSOU0R0I2Mzy+gm8jQKaATkSoZm0f57zV9HQyOxBkZ1x1ekUZhZjeY2W4z22tmd07zetjM/in3+pNmtib3/Bozi5nZjtyfv6r22KeKFdFqZSovQze10mUqneGvHn2ZTat6ueqCRSWP8axzdrVxcixOssJVzMeTaXYdOcOm1dMHcwCbVvUS9BvbtOxSKqgpArrR+PyKokzsodOSSxGpEm+JdzHlv9epdYFIQzEzP3APcCOwEbjNzDZOOewDwGnn3AXAZ4FP5r32snNuU+7PB6sy6FlEE6l5tSwAWNQRwuzcDN3WZw5z8FSMD113QVmzc5BdcuncuUFkub1w5AzJtOOKGbJzAJGQn0tX9LDtFWXopHKaJqCbV1GUsDJ0IlJd82nQq0qXIg1nC7DXObfPOZcA7gNunnLMzcDf5r5/AHirlTuyKZOxEjJ0Ab+PRR0hBvOai2cyjr945GUuWtrFWy9aXK5hTliSywpWuhfdjoNDAGxatWDW47asWcizA8OMq02WVEjDB3TOOUbn27ZAGToRqbL57qEDBXQiDWQFcDDv8UDuuWmPcc6lgGHAW3u41sx+bGaPmtmbKz3YucRK2EMH0N/Vdla27KGdR9l7fJRfv+4CfL7yx7CLu3L79ioc0D1zcIgl3WGW9rTNetzmNQtJpDM8d2i4ouOR1tXwAV08lSGVcXSGg0X/bLv20IlIlU30cyri4igS8rOsp02tC0Qax3RRiivwmCPAaufcFcBvAl82s+5zTmB2u5ltN7Ptg4ODJQ94NqUsuYRs6wJvyaVzjnse2cvavg5+8rJl5RriWbwAq9KFUXYcHJqxGEq+zedlM3g/0rJLqZCGD+hGxrPBWGeBPZ3yhQI+gn5jLKEMnYhURzSRwmcQDhQ3/a7t62CfAjqRRjEArMp7vBI4PNMxZhYAeoBTzrm4c+4kgHPuKeBl4MKpJ3DO3euc2+yc29zf31+BjzCplKIokG0u7mXoHn1pkOcPneHXrjkffwWycwAL20MEfFbRJZdD0QT7T0a5vICAbkFHiPWLO9muBuNSIQ0f0I3mmvTOZ8klZLN00bgydCLNpoAKc1eb2dNmljKzW6o1rrF4mo5QoOgiAGv6Otiv1gUijWIbsN7M1ppZCLgV2DrlmK3A+3Lf3wJ8zznnzKw/V1QFM1sHrAf2VWnc04om00WtKphqcS6gy2Qc9zy8l+U9bbzriqkrUMvH5zMWd4Ur2otucv/c3AEdZJddbn/1NJnM1EStSOkaPqAb8wK6eSy5hOw+OmXoRJpLgRXmDgDvB75czbHFkvNr0Luur4OhaJLTY4kKjEpEyim3J+4O4CHgBeB+59xOM7vbzG7KHfbXwCIz20t2aaV34+lq4Fkze4ZssZQPOudqmtqJxkvP0KUyjm/tOsq2/ae5/ep1hIpcpVCsJT1tHK/gkssdB4cwg8tW9BR0/JVrFjAynmL3sZGKjUla1/wXRNeJySWX88zQhQPaQyfSfCYqzAGYmVdhbpd3gHNuf+61yjYqmmIsni6qIIpnojDKyTEWdITKPSwRKTPn3IPAg1Oeuyvv+3HgPdP83D8D/1zxARYolc6QSGcmevfOh1ek5I//7QX6OkPcumV1uYY3oyVdbewdHK3Y+z9zcIj1izvpaissoXDlmoUAbN9/iouXnbMlUqQkDZ+h85Zcds1zyWVHyD9RpEBEmkYhFeZqIppIEQkWf6d7bX82oFNhFBGppmiy+EJOU3nNxQdOx/jAm9bRNo85sFhLuiu35NI5V3BBFM/KBRGWdrepwbhURBMEdEmghAxdKDDR6FdEmkYhFeYKe6MyV5KLJtJ0zKOI06oF7fhMrQtEpLpiuZveJS257MwGdN1tAX7+DZXPzkF2yeXIeKoiq7AOnopxOposqCCKx8y4cu1Ctu0/hXPaRyflVVJAVw9FB0bHSyuK0hH2M6YllyLNppAKcwUpdyW5sUR6XuW/QwEfKxe0K6ATkaryahWUkqFb2tNGR8jPr7x5XcFLFEu1pKtyrQt+fDCbZSsmQwfZfXRHhsc5NBQr+5iktc07oKuXogMj8RL30IUCWnIp0nwKqTBXE7FEat4XRmv7OhTQiUhVTfbOnP8euragn8d/9y3857dcUK5hzWlJtxfQlX/Z5Y6DQ7QFfWxY0lXUz20+z9tHp2WXUl6lZOgmig445xKAV3RggnNuv3PuWaBiRQdGx1ME/VZ0TydPR9g/cfdJRJpDIRXmzOxKMxsgW5Tg82a2sxpjG4vPL0MH2YBu/4kxLdcRkaqJlWEPHcDCjlDR7VpKsbQnu8yzEgHdMweHuGxFDwF/cdeeG5Z20dUW4EfqRydlVkqVy+mKDrx+Pm9kZrcDtwOsXl3c2urReIrOcPE9nTzK0Ik0pwIqzG0juxSzqqKJ1Lz20EE2oBtLpBkcjU9UjRMRqaTJDF3lC5mU0+IKZegSqQzPHz7DL77hvKJ/1u8zXnfeAjUYl7IrJUNXtqIDpexRGR1PzXv/HHh96FK64y0iVRFNzL+f05q+XOuCQS27FJHqiOZWMUWCjdXpqiscIBL0l30P3e6jIyRSGTatLm7/nGfL2oW8dGyU4yOVa3ouraeUgK5sRQdKMRJPzbupOGT70DkH48mqtqISkRaUSmeIp+bfz2ldLqDbf1IBnYhUh5ehm+/KgloxM5b2tJU9Q7djngVRPFevzyYuHn/pRNnGJFJKQFcXRQdGx1N0zbMgCmQzdIAqXYpIxZXaz2l5b4SQ38c+FUYRkSrx5q1S2hbUyuKu8vei+/HBIfo6Q6zojczr5y9Z3k1/V5hHXyq9DY6IZ94BXb0UHRiNl7bkMpK7U65edCJSabESq8X5fcbqRe1qLi4iVRNLeG0LGmvJJWQrXZZ7yeUzuYbi863dYGZcvb6fx/YMks5ou4+UR0l96JxzDzrnLnTOne+c+3juubucc1tz329zzq10znU45xY55y4px6DzeUVR5ksZOhGpFq+ibilLl9YsUusCEakeb8llJNh4GTpvyWW56iQMx5K8PDjG5Svnt9zSc82GfoaiSZ4dGCrLuERKCujqwUiJRVHac8FgVAGdiFRYOS6M1vV38OrJKBnd2RWRKogm0oQDPvy+6rUcKJfFXWHiqQzDsWRZ3u+5gWGAeRdE8bz5gj58hpZdStk0fEA3Gk+WZw+dllyKSIVNFheY/5y1ZlEH8VSGIxXorSQiMlW21UrjLbeEbIYOKNuyS68gymtKzNAt6Ahx+apeHtmtgE7Ko6EDumQ6w3gyU9KSS29NuDJ0IlJpYxN7UUpYctnXDlS2dcFzA8O8+y9/yON7dLEh0uqiiXRDLreE7B46KF8vuh0Hh1jX30FPZP7V1T3XXNjPMwNDnB5LlGFk0uoaOqDz9qOU1IcurAydiFRHqUVRANb1dQLwSoVaF6TSGT76wDM89eppfvGLP+Jz33lJG/dFWlgskW64puKeJV3ZgO5oGQI65xw7Dg7Pu13BVNduWIxz8PhetS+Q0jV0QDcyngvolKETkQbg3YQq5eJoSXeYSNBfsQzdl364nxePjvCZ917OT21awee+s4f3/82PODla3kpxItIYxho4oFvcHQbgeBkCukNDMU6MxssW0F22oocF7UEe2X28LO8nra2hA7rR3MVRVzkydAll6ESksqKJ0vrQQbbk9Zq+joo0Fz88FOMz336Jt1y0mJ+6YgV/+t7L+ZOfvownXznFT/7Z99m+/1TZzyki9S2WSDVkywKAtqCf3vZgWTJ0zxzMFUQpU0Dn9xlvXt/PYy+dUJErKVlTBHSd4fmvZW4L+DGbvHMuIlIp5SiKArC2r70irQv+8F93knGOP7zpEswMM+O2Lav52q/9BOGgj1vvfYIvPL6vbCXARaT+RRs4QwewrCfC13cc5ve//jzPHBya9/z11KunCQV8XLS0u2xju3ZDPydG4+w6cqZs7ymtqbEDuvHS99D5fMa6vg4efWlQFykiUlHRRAqfQThQ2tR7weIuDpyKsvvoSJlGBt/ZdYyHdh7jw29dz6qF7We9dumKHv71P7+Jt168mD/+txf4xrNHynZeEalvsUSaSAMHdH/y05dxzYX9fGXbQW6+5we87TOPcs/Dezk8FCv4PXYdPsM/Pvkq117YT6jE+Tvfm9f3A2pfIKVr6IBuJF76HjqAX37TWp4dGObJV7ScSEQqJ3unO4BZaf2cfuEN57GgPchH7vsx48nSl4tHEyl+f+tOLlzSyX9687ppj+luC/KXP/c6Llraxf/81m4SqUzJ5xWR+jeWSDV0hm7Tql7+/Gdfy7b/+jb+5KcvY2FHiE8/tJurPvk9PvrVZ0ilZ5/LRsaTfOjLT9MTCfI/fvqyso6tvyvMpSu6eVTtC6REDR3QeRm6UvbQAbz7tStZ2BHi/zy2rxzDEhGZVrRMF0b9XWE+fcvlvHh0hE8/tLvk9/uz7+7l0FCMP37XZQT9M/9vweczfveGi3j1ZJR/2nag5POKSP3zbkQ1up5IkNu2rOarH/wJHv3otfzST6zlq08N8NEHnp2xkq9zjju/9hwHTkX537ddQV9nuOzjuvbCxTx14HTZmp9La2rsgC6e/cdfaoauLejnF994Ht998Th7j5dvCZOISL6B07GSb0B5rrtoMe9743n89fdfKalf3O6jI3zh8X28d/NKtqxdOOfx127o5/VrF/K/vru3qnuPE6mMlsWL1EAjty2YyXmLOrjrP27kt99xIf/y40N87GvPTluY5B+eeJV/e/YIv/WOC3n9ukUVGcs1G/pJZxw/VPsCKUFjB3TjKcxKqxjn+YU3nEc44OMLj79ShpGJiJzt+UPDPL7nBDddvqJs7/mxd17M+sWd/Nb9z3BqHs1pMxnHf/2X5+hqC3DnjRcX9DNmxu/eeBEnRuP89ferM1++enKMLf/jO7zrL37ID3TRI1I1iVSGVMY1XUDnueMt6/nwWy7g/u0D3LX1+bNuGj03MMwffeMFrtvQzwevPr9iY7hiVS9dbQHto5OSNHRANxJP0RkufT8KwKLOMLe8biVfe/oQx0dKL28rIpLvz767h+62AO+/ak3Z3rMt6Odzt25iKJrkzn9+tqgM1suDo/ynv9vO9ldP87F3XszCjlDBP/va1Qu4/pIl3PvYvor3p0ukMnz4Kz8mk3EMnhnn577wJLfd+wRPvXq6oucVkWx2DiDSBEsuZ/Ibb7+QX71mHf/wxAHu/sYunHMMx5L8+pefoq8zxGfeuwmfr/TrzJkE/D7edEEfj+xWcT6Zv4YO6EbHU3SVuNwy3wfetJZkJsPf//urZXtPEZHnDw3zrV3H+MCb1tETmX+blelcsryH37lhA9/adYx/2nZwzuNPjSX4/a8/z/WffYwnXznFnTdexHtet7Lo8370+g1EEynuefjl+Qy7YJ/59ks8MzDMJ9/9Gh7+6LX8/n/cyJ7jI7z7L3/IL39pGzsPD1f0/CKtbCyRXVbdrBk6yK46uPOGi/ilq9bwNz/Yzye++SK/88AzHBka589/7rUsKOJm13xdu6Gfo2fGeenYaMXPJc2psQO6eKqklgVTrevv5O0XL+Hvn3iVaEJ96USkPCqRncv3y1et5U0X9PGH/7qLfYPTXxCMJ9N8/tGXuebTD/MPTx7g1i2reOSj1/LBa86f1yqHCxZ38Z7XreIfnniVg6eipX6EaX1/zwn+6tGX+dnXr+bGy5YRDvj5pavW8tjvXMdHr9/A9v3Zhuf/86HdurMtUgFe78xmDuggG9Td9R828vNvWM3nH93HQzuPceeNF/Ha1Quqcv6rL8y2L3hk9/GqnE+aT0Pn0EdzSy7L6far1/GtXcd44KkBfvGNa8r63iLSerzs3G+87cKyZ+c8Pp/xp++9nOs/9xjv+5sfsWFJN5ANcLw458WjIxwaivGWixbzsRsvYv2SrpLP+1/evp7/u+MQn/32S3zmZzaV/H75To7G+Y37d7B+cSf//Sc3nvVaeyjAh667gJ9/w3l8/N928ecP7yWRzvCxGy8qyxJ8EcleY33i/70IwPLeSI1HU3lmxt03XUpHKEA0keYDb1pbtXMv64mwYUkXj740yK9eU7n9etK8GjqgGxlP0V3mC6TNaxby2tW9fOHxV/i515+Hv4LrpkWk+VU6O+dZ0t3Gn916BX/6rd0TDXO92MYM1vS188l3v4Y3re8r2zmX9UR4/1VruPexffynq9dx8bLusryvc47f/uozDMeS/N0vb5mxqXFPJMgn3/0aIkE/9z62j1Ta8d//w8UK6kRKNHA6yq/87Xb2HB/lD2+6hM3nVSdTVWs+n/GxdxZWIKrcrr2on88/uo+rPvE9Ll7WzcZlXWxc3s3GZT2sXBCp6D4+aXwNHdCNxlOsqMBdo9uvXscH/+FpvrXzKDdetqzs7y8iraEa2bl8V1/YP7F0p1p+/ZoL+MqTB/jUN1/kk7e8hiND4xwZjnE49/XUWJKfOH8R77xs2YyB2VR/84P9PLx7kLtvvmTOINHM+IObLsHnM774g1dIZzL8wU2XKKgTmaft+0/xq3//FIl0hi/90pW8eX1155RW9cGrz2dBe4hdh8+w68gZvvfiMbxOCgs7Qvz0FSu47fWrOb+/s7YDlbrU2AHdePmXXAK8feNSzlvUzucf28cNly5t2QuDRCrD8ZFxjp0ZJ5l2XLlmoTKWIkWoVnaulnrag/zatRfwyW++yJaPf/es10IBH53hAP/89AB/sHUnN1+xnFuvXM2lK3pmfL/nDw3zif/3Im+7eAm/8IbzChqDt/8l4DP+z+OvkHaOu2+6VHe0RYr01e0H+b1/eY6VC9r5wvs2K3ioogUdIT6Yt9wylkjz0rERdh05w/f3nOBLP9zPF77/Cm9ct4ifff1qrr9kKaFAQ5fCkDIqKRoysxuA/wX4gS845z4x5fUw8HfA64CTwM845/aXcs585S6K4vH7jF9501r++9d38r0Xj/OWixY3dVB3cjTOjoND/PjAEC8ePcOR4WwQd2L07L5W6/o7+LVrzuddV6wg6NckIvWt1vNTtbNztfRLuYC1PeRnWU8by3sjLOtpm2iF8MS+U9y//SBf3T7APzxxgI3LurnldSvpbAswFE1wOppkKJpkKJrg6QOnWdAR5FO3vKaoedfM+L13Xozf5+OvHn2ZdMbx8XddpqBOJM/e4yP845MHiAT92T+h3J+gn2cHhvnSD/dz1QWLuOdnX0tve+WrO8rMIiE/l6/q5fJVvdy2ZTXHR8b56vYBvvKjA/znr/yYvs4QN1y6FJ8Zo+MpRuOTfzLOcdUFfVx/yVI2rezVPNgCbL6VwczMD7wEvB0YALYBtznnduUd8+vAa5xzHzSzW4Gfcs79zGzvu3nzZrd9+/Y5z5/JONb93oN85K3r+Y23XzivzzCbWCLNNZ9+mOMjcZZ2t/Gm9X28eX0fV13QR19n+Jzjx5NphmNJOsKBimQNy2U0nuLFI2d47tDwRBB3IFehzu8z1i/uZHlvhCXdYZZ0t7G0u40l3W2cGU/y+Uf3sevIGVb0Rrj96nX8zJWraAs2d+UrKR8ze8o5t7lK56rp/ARw+99t54l9J3n8d9/S9AFdoYZjSbbuOMR92w6y8/CZiecDPqO3PcSC9iB9nWE+esOGeVeXc87xp996iT9/eC8XLO7kug3ZZahXrlmo+UqmVam5qZSbSmb2MeADQBr4sHPuodnOVejc9OhLg9zxj08TTaZJZ869/vuFN5zHXf9xo27a1rFMxvHYnkG+/OQBHtszSFvQT0coQFdbYOIaNJ5Ks33/aVIZx5LuMO/YuJTrL1nK69ctrNjfbTKdYXQ8RTKdIe0cqbQj4xypjCOTcfh8RsBn+H1GwOfLfTVCAR/hgI+A/s2do5i5qZSA7o3AHzjnrs89/hiAc+5P8o55KHfMv5tZADgK9LtZTlropDQynuSyP/gW/+0nL+ZX3rxuXp9hLsdHxvnuC8f5/p4T/ODlEwxFkwBsXNbNos4Qp8YSDEWTnI4mJkr7AqxcEOGipV1ctLSbDUu7uGhpFws7QiTTjkQqQyKdIZHKkEx7fxypTIZU2pFMZ0hlHD7LVnLrCPuJBLNf20OBs9Pref8V085NvGc8lZk4z6mx+MR67F2Hz7D/5GR58aXdbVyxujf3ZwGXLu+ZdY+Lc45HXhrknu/tZfurp+nrDPFzrz+P8xa1090WpDsSpDsSoLstSEc4gBm4DDgczmWH65zD77OJX2yfTf6CN3MWVKoe0NV0fnr+0DD/4X9/n99424V85G3rS/w0zenVk2P4zOhtD9IZDpT19985x1efGuDrOw6x7ZXTJNIZ2oI+Xr92EVdf2M+K3raz5slk7qthBP1GMOAj6PcR8me/mpGbw7JzWSb3TyTg89Eezl5MtYf8tIf8dIQDtAX8BPzZeS3o91V0qXomk71oyuSNK+Mm59zMHP+P95nhs+xXM+9x9rlWmZMrMTeVclPJzDYCXwG2AMuB7wAXOufSU8/jKeZmkyeZzhBLphlPpIkm0vh9xqqF7UV+UqlXw9Ek39t9jIeeP8ajLw0SS6bpCgdYvaidZT1tLO1pY1lPhKXdbSzuDpNx2a028VQ69zVDPJlmLJFmLC/7NxZPMRZPMxJPMTKeZGQ8+3U8mSlpvH6fEc4Fd+GAn/awn65wgK627P8jutoCdLZl59pwwE844MsFg9nv24J+2oI+IkE/4bwMdCjgI5C75gz4fQT9kwFlNeY55xzpzOQcnXGOSNBf0HmLmZtKSSWtAPK72A4Ar5/pGOdcysyGgUXAiRLOC2QzTUBFs2GLu9q4bctqbtuymnTG8fyhYb6/9wQ/2HuC0XiKJd1tbFjaxYLcneXe9hBD0QQvHh3hxaMjPLx7cNo7YLVw3qJ2Ni7r5t2vXcnG5d1csryHpT1tRb2HmXHdhsVct2ExP3rlFPc8vJf/9d09ZR2n98vlMzCyFxiW+z47BrDcWM76VbBpv50w49/CDC94T3vX9pOP84+ZfDDfFlgz/T7btJ9i+uNnOreb4cPNfHzx/u+vX8XG5eWpbFhmNZ2fWmHvXKnOW9RRsfc2M967eRXv3byKaCLFk/tO8ehLgzy2Z5A/+sauud+g7ONh4gaW5c9tTF5MzPS/9oxzE4FZxgF5FwXV+t+LNy8bnD3+KZ8FpszNM32oIuegmebh6eZg75mf2byKP3rXpbN9rErbAux1zu0DMLP7gJuB/H+ANwN/kPv+AeDPLfsP4mbgPudcHHjFzPbm3u/fyznAYO6GRXebVhA0o572ID91xUp+6oqVxBJpHt8zyON7TnBoKMahoXGeevU0p3OJirkEfDaR/esMZ5MMPZEgKxdE6G6bDLo6w4GJAMrnM/xmBPzZuS+TC2xSmcmvqVxiI57MBZCpNPFUhvFcIDkynmJ0PMmxM+OMxlOMjKeIJlJln/u8m1n5c9xMx+Xf+DKA3A0/b152Lvs4nfd4qn3/450zXv/NVynRUCHXzQVdW5vZ7cDtAKtXry7o5L2REH/9vs1sWFp6L6VC+H02sZb5Q9ddUNDPjCfTvDw4yu6jI4yMpwh5d30DPkJ+y/2j9xHwZ+/iBnJ3cwN+I5OBWDJ7FySayH1Npokn02dF9d53PoNQIHsnIv/9u9uCbFjaRVeZJ+wtaxeyZe0WhqLZLOWZ8SRnYqnc1+REwO39zz3/Loj3S+39QmdyX70MnvcLkHHZx/nB1WSmb3IshQZW8w2ebOrjaf77n/ugAPO8qJnOTHd6ZhxSkf8tZrKos273WNR0fvrFN67h+kuWaqllHWgPBbjuosVcd9FiAA4PxRiOJXPzpG/iazC3+iGZW+mQyK2eSKQyOByGdzcXyAUxqbQjmkgRzd3BjibSjCVSjCczpHKrLVJpRzqTIZmb67x5zpvLYPabMvkZNPIuOrxVDT4Dv9nEHpnJoHHy+9lMXojkvubuJDu8AHIyoMyffydWXuRdsOTPzd7r5bhplT3+7NenDYQn5mxj06qZC+9USSk3lVYAT0z52RWVG6o0u0jIzzsuWco7Lll61vOxRJqjZ8Y5fmY8lyHzEw5m50Tva0c4QDjgq6uMfSo3R+cHguPJbCAYS6YZz/2JJbPZRm8u9lbBpXOP8+c5bw6c+Rrs7Hlv4njHWdm+yYAvOzebZVdrZOfs7HOVUEpANwCsynu8Ejg8wzEDuSVNPcCpqW/knLsXuBeyywYKOXkk5OetFy+Zx7Crpy3o55LlPVyyvOb/Y6mY3vaQNk5LParp/FTOXm9SXst7I7M3ST53i7TIfJRyU6liN5tE8kVCftb2dbC2r3IrJioh4M/uudPl56RSdiBuA9ab2VozCwG3AlunHLMVeF/u+1uA7822P0VEpEw0P4lILRVzU4kpN5UK+ZNsKqkAACAASURBVFmcc/c65zY75zb396tXnEgrm3dA55xLAXcADwEvAPc753aa2d1mdlPusL8GFuXWf/8mcGepAxYRmYvmJxGpsVJuKm0FbjWzsJmtBdYDP6rSuEWkAZVUUcQ59yDw4JTn7sr7fhx4TynnEBGZD81PIlIruT1x3k0lP/BF76YSsN05t5XsTaW/z91UOkU26CN33P1kC6ikgA/NVuFSRKR+G6aJiIiINKhSbio55z4OfLyiAxSRpqEufiIiIiIiIg1q3o3FK8XMBoFXi/iRPsrQN6qBtNrnhdb7zM36ec9zzjX0zv0i56dm/XucTat95lb7vNCcn1lzU/Nrtc/cap8XmvMzFzw31V1AVywz215oF/Vm0GqfF1rvM7fa521Wrfj32GqfudU+L7TmZ242rfh32GqfudU+L7TmZ86nJZciIiIiIiINSgGdiIiIiIhIg2qGgO7eWg+gylrt80LrfeZW+7zNqhX/HlvtM7fa54XW/MzNphX/DlvtM7fa54XW/MwTGn4PnYiIiIiISKtqhgydiIiIiIhIS1JAJyIiIiIi0qAaNqAzsxvMbLeZ7TWzO2s9nkowsy+a2XEzez7vuYVm9m0z25P7uqCWYywnM1tlZg+b2QtmttPMPpJ7vpk/c5uZ/cjMnsl95j/MPb/WzJ7MfeZ/MrNQrccqhWv2+Ulzk+YmzU2NqdnnJtD81Ozzk+am6TVkQGdmfuAe4EZgI3CbmW2s7agq4kvADVOeuxP4rnNuPfDd3ONmkQJ+yzl3MfAG4EO5v9dm/sxx4C3OucuBTcANZvYG4JPAZ3Of+TTwgRqOUYrQIvPTl9DcpLlJc1NDaZG5CTQ/Nfv8pLlpGg0Z0AFbgL3OuX3OuQRwH3BzjcdUds65x4BTU56+Gfjb3Pd/C7yrqoOqIOfcEefc07nvR4AXgBU092d2zrnR3MNg7o8D3gI8kHu+qT5zC2j6+Ulzk+am3PNN9ZlbQNPPTaD5iSafnzQ3Ta9RA7oVwMG8xwO551rBEufcEcj+EgOLazyeijCzNcAVwJM0+Wc2M7+Z7QCOA98GXgaGnHOp3CGt9O+7GbTq/NTUv6cezU2amxpYq85N0OS/q55WmZ80N52rUQM6m+Y59V9oEmbWCfwz8F+cc2dqPZ5Kc86lnXObgJVk76BePN1h1R2VlEDzU5PS3KS5qcFpbmpirTQ/aW46V6MGdAPAqrzHK4HDNRpLtR0zs2UAua/HazyesjKzINkJ6R+dc1/LPd3Un9njnBsCHiG7Br7XzAK5l1rp33czaNX5qal/TzU3aW5qAq06N0GT/6626vykuWlSowZ024D1uYo2IeBWYGuNx1QtW4H35b5/H/D1Go6lrMzMgL8GXnDOfSbvpWb+zP1m1pv7PgK8jez694eBW3KHNdVnbgGtOj818++p5ibNTc2gVecmaO7f1ZaanzQ3Tc+ca8yMpJm9E/gc4Ae+6Jz7eI2HVHZm9hXgWqAPOAb8PvB/gfuB1cAB4D3OuambfxuSmb0JeBx4Dsjknv49smvBm/Uzv4bs5l0/2Rss9zvn7jazdWQ3rC8Efgz8vHMuXruRSjGafX7S3ARobtLc1ICafW4CzU+5p5t2ftLcNL2GDehERERERERaXaMuuRQREREREWl5CuhEREREREQalAI6ERERERGRBqWATkREREREpEEpoBMREREREWlQCuhEREREREQalAI6ERERERGRBqWATkREREREpEEpoBMREREREWlQCuhEREREREQalAI6ERERERGRBqWATkREREREpEEpoBMREREREWlQCuhEREREREQalAI6ERERERGRBhWo9QCm6uvrc2vWrKn1MESkzJ566qkTzrn+Wo+jFJqfRJqP5iYRqUfFzE11F9CtWbOG7du313oYIlJmZvZqrcdQKs1PIs1Hc5OI1KNi5iYtuRQREREREWlQCuhERERE5sHMbjCz3Wa218zunOb1q83saTNLmdktU15Lm9mO3J+tec+vNbMnzWyPmf2TmYWq8VlEpHEpoBMREREpkpn5gXuAG4GNwG1mtnHKYQeA9wNfnuYtYs65Tbk/N+U9/0ngs8659cBp4ANlH7yINBUFdCIiIiLF2wLsdc7tc84lgPuAm/MPcM7td849C2QKeUMzM+AtwAO5p/4WeFf5hiwizUgB3Sx+fOA0x86M13oYIiJVlUpn+M6uYzjnaj0UkXq2AjiY93gg91yh2sxsu5k9YWZe0LYIGHLOpeb5nk1vx8EhjgzHaj0MkbqigG4Gzjne98Uf8RcP7631UEREquo7LxznV/5uO7uOnKn1UETqmU3zXDF3QVY75zYDPwt8zszOL+Y9zez2XEC4fXBwsIjTNi7n/j97dx4f110e+v/znX1GGmm0r973PYvjJHYIZCELS8KlpISyJJTelP229LaF322BUtpS2l4KtxQSIFAoIQVKmxQc0kD2xY5N7HhJbNmWbUm21hmtM5r9+/vjzMhjeSTNqm2e9+ull0czZ84c5RUdnec8z/d5NPd+92W++qsTc30oQswrEtBNoX80xEgwii8QmetDEUKIWdU1GACM86AQYkpdwJKU71uB85m+WWt9PvFvO/A0cDkwAHiUUsmxUlPuU2v9gNZ6u9Z6e13dgh6jl7GekSBDgQjdw1I9JUQqCeimcHrAD8BoUAI6IURpSZaaDwbCc3wkQsxr+4A1ia6UNuBu4NEZ3gOAUqpKKWVPPK4FdgGvaaPO+Skg2RHzHuCRgh/5AtXWOwZAn9xsEuIiEtBN4YzXCOhGxiWgE0KUluTd70G/nP+EmEpindvHgceB14Efa62PKqW+oJS6A0ApdZVSqgu4C7hfKXU08fYNwH6l1KsYAdyXtNavJV77U+BTSqmTGGvqvjN7P9X8dqJ3FJDqASEms8y8SWk6PWCUHI0GozNsKYQQi4tk6ITIjNZ6N7B70nOfTXm8D6NscvL7XgS2TLHPdowOmmKSE4kMnc8fIhbXmE3plhwKUXokQzeF0wPGSUMCOiFEqelJBHQ+vwR0Qoj5o63PyNDFNXjHJEsnRJIEdFM4M5Ghk5IjIUTp0FrTO2xcKA1JUyghxDyhteZk7xiNFQ5A1tEJkUoCujTicc0Zrx+lwB+OEYvLLCYhRGnw+cOEY/GJx0IIMR90DwcZDUXZuboGkHV0QqSSgC6NnpEgoWicVXXlAIxJ2aUQokQkyy0tJiVr6IQQ80ZboiHKdatrAQnohEglAV0ayZEFW1sqARiRskshRInoSXS4XFVXLgGdEGLeSDZE2bkqEdDJGjohJkhAl0YyoNvSagR00hhFCFEqkhm6DU1uBv0RjLFYQggxt9p6R6ktt9NY6aDCYaFvRIaLC5EkAV0aZwb8OKwm1tS7AcnQCSFKR+9wEJOCtY1uwrE4/nBsrg9JCCFo6xtjTb2xFKbObZcMnRApJKBL44zXz/KaMiqdVkAydEKI0tE9HKTObaeu3A7AoDRGEULMMaPD5ShrG4yArt7toG9EAjohkiSgS6N9wAjo3A5j7rqMLhBClIqekSCNFQ6qXDZAhosLIebeuaFx/OEYaxqMyinJ0AlxMQnoJonG4nT6AiyvTQ3oJEMnhCgNvSNBGiocVJUZAZ2MLhBCzLUTfUZDlLWJgK7ebZcul0KkkIBukvNDQSIxzcraMtyOZMmlZOiEEKWhezhIU6WD6jLJ0Akh5ocTiZEFyZLLOredQDjGWEhuuAsBEtBd4rTX6HC5vLYMm8WE3WKSDJ0QoiQEwlFGg1EaKh1UuYwbWoN+uaElhJhbbb1j1LnteBKl4HVuY42vZOmEMEhAN8npfiOtv7zWBYDbYWVEAjohRAlIzqBrrHBQ4bBiUpKhE0LMvRMpDVHAaIoCyOgCIRIyCuiUUrcppY4rpU4qpT6d5vXrlVKvKKWiSql3TXotppQ6mPh6tFAHXixnvAHKbOaJDm8VDouUXAohSsJEQFfpwGRSVLlssoZOCDGn4nHNib6xiVFSkJKhk8YoQgBgmWkDpZQZ+DrwZqAL2KeUelRr/VrKZh3AvcD/TrOLca31ZQU41llxesDPiroylFIAuB0WKbkUQpSE5FDxxgrj7rfHZWUoIDe0hBBz59zQOIFwbKIhCkjJpRCTZZKh2wGc1Fq3a63DwMPAnakbaK3PaK0PAfEiHOOsOp0YWZBklFzKBY0QYvGbCOgqjYCuukwydEJMJ9cKJqXUZUqpl5RSR5VSh5RS70557XtKqdMp1U0L5qZ4MZzou7ghCoDHacVqVvRJQCcEkFlA1wJ0pnzflXguUw6l1H6l1B6l1DuyOrpZFo7G6RoMsKI2NaCTDJ0QojT0DAepcFhw2YzijSqXTdbQCTGFlAqm24GNwHuUUhsnbZasYHpo0vMB4ANa603AbcA/KqU8Ka//sdb6ssTXwaL8AAtEW6/R22BNSobOZFLUlsvoAiGSMgnoVJrndBafsVRrvR34HYwT1qpLPkCp+xJB3/7+/v4sdl1YnYMB4pqLAroKh1XW0AkxT5XS+t7Z0DMcnMjOAbKGTojp5VzBpLVu01qfSDw+D/QBdbNz2AtLW+8oDRV2Kp3Wi56vd9slQydEQiYBXRewJOX7VuB8ph+QOFGhtW4HngYuT7PNA1rr7Vrr7XV1c3c+OzNwYWRBkmTohJif8rw7Don1vYmvO4p6sAtEcqh4UlWZjaFABK2zuYcnRMnIt4IJAKXUDsAGnEp5+q8SpZhfUUrZ8zvMhe1E79hF6+eS6mS4uBATMgno9gFrlFIrlFI24G4go7vZSqmq5IlIKVUL7AJem/5dc+d0IqBbMWkNXSAcIxpb8MsDhVhsSmp972xIDhVPqi6zEo7F8Ydjc3hUQsxb+VYwoZRqAn4AfFBrnTxPfQZYD1wFVAN/OsV750V1UzHF45qTkzpcJklAJ8QFMwZ0Wuso8HHgceB14Mda66NKqS8ope4AUEpdpZTqAu4C7ldKHU28fQOwXyn1KvAU8KVJ3THnldMDfiqdVqrKbBPPuR3GWpKxkGTphJhnZmV9bylcNAFEY3EGxkITHS7BKLkEGJSySyHSyauCSSlVAfwC+DOt9Z7k81rrbm0IAd/FuHl1iflS3VRM54bGGY/EWJPSECWpzu3A6w/JDXchyGBsAYDWejewe9Jzn015vA/jRDb5fS8CW/I8xllzxuu/aP0cXAjoRoNRPC5burcJIeZGIdb3nldKrQSeVEod1lqfmryR1voB4AGA7du3L9raw/6xEHENDZVpArpAmCXVrrk6NCHmq4kKJuAcRgXT72TyxkTF038A39da/2TSa01a625lzE96B3CksIe9cLT1XtrhMqnObUdr8PnD1KfciBKiFGU0WLxUnBkIpAnojEW4MrpAiHmn6Ot7S0l3Yqh4asllslpBGqMIcak8K5h+G7geuDfNeIIfKqUOA4eBWuCLs/hjzSvJDper05Rc1idm0UljFCEyzNCVgmAkxrmh8Ytm0AFUJDJ0I+NScinEPJPP3fEqIKC1DqWs7/1y0Y50AehNBHSpTVGqyy5k6IQQl8qjgulfgX+dYp83FvgwF6wTvaM0Vjgu6XAJMlxciFSSoUs46w0AsLz24rKiZIZORhcIMb+U0vre2TAxVPyiNXTG+W/QL+c/IcTsa+sbTbt+DqCuXAI6IZIkQ5eQ7HC5svbiE0fqGjohxPxSKut7Z0PPcBCb2TSRlQNjDqdJSYZOCDH7kh0u33v1srSv102UXAZn87CEmJckQ5dwxpucQXdxhq7CKRk6IcTi1zMSpKHSjtGHwWAyKRkuLoSYE52DAYKReNqGKAAOq5kKh0UydEIgAd2E0/1+asttEyWWSZKhE0KUgp7h4EXllkkel1UydEKIWZdsiLImzVDxpPoKhzRFEQIJ6Cac9vovaYgCYDWbcFhNjMocOiHEItYzEqSx0nnJ89VlNllDJ4SYdcmRBWvq02fowFhHJxk6ISSgm3Bm4NIZdEluh1VKLoUQi5bWOpGhs1/yWpXLJhk6IcSsO9E7SnOl45LKqVR1bjv9YxLQCSEBHeAPRekbDbF8yoDOwkgRSi77RoP0DMtiXiHE3BoejxCKxi8aWZAka+iEEHOhrXds2nJLMGbR9Y2E0FrP0lEJMT9JQMeFDpfTZ+gKH9D96U8P8ckfHSj4foUQIhsXhopfWnJZVWZjKBCRCyYhxKw64/Wzqm7qckswMnTjkRj+cGyWjkqI+UkCOlI6XKZZQwfGcPGR8cKXXLb1jtGeCCaFEGKuTMygq7y05LK6zEo4FpcLJiHErAmEowTCsYnRBFOZGF0wItVOorRJQIexfg4uHVmQ5HZYCr6GLhyN0z08zsBYiFBULpSEEHOnN5Ghm6rkEmBQyi6FELMkWeZdkzIXM516t3HOksYootRJQAecHgjQWOHAZUs/Z91tL3zJ5fmhceKJCiZZRyeEmEvdw0GUunBxlGoioJPGKEKIWZIM6KpnCOiSGTppjCJKnQR0wOmBsSmzcwAVTkvBA7qzvsDE43ND4wXdtxBCZKN3JEhNmR2b5dI/CVWJCyppjCKEmC3exPmmasYMXbLkUgI6UdokoAPOegNTNkQBoynKeCRGJBYv2Gd2pAR054ckQyeEmDvGDLr0a1WSd8glQyeEmC2+scxKLiudVqxmJRk6UfJKPqCLxTVefzhtqVGS22GUYo4VMEvX4fVjMxv/+c9Lhk4IMYeMGXSXdrgEqHIZM6B8MlxcCDFLJkouy6cP6EwmRW25XTJ0ouSVfEA3FjKCtGTQlk5yqGUhyy47fAGW1bioLbfRPSwBnRBi7kyXoatwWDEpGJIMnRBilnj9Yaxmhds+9bVZUr0MFxeCmX9TFrlk98qKRNCWTjLYGylgp8sO3zhLq104bWbOScmlEGKOBCMxhgIRGtN0uATjDrgMFxdCzCafP0R1mQ2l1Izb1rntch0lSl7JZ+iSWbfpM3SFDei01nR4/SytcdFc6ZSSSyHEnOmdmEGXvuQSwOOyyho6IaaglLpNKXVcKXVSKfXpNK9fr5R6RSkVVUq9a9Jr9yilTiS+7kl5/kql1OHEPr+mMolsFhGfP0x12fQz6JLq3A76RyWgE6VNArqJgG7qDF1FgUsuff4w/nCMpdUumjwOzg+No7UuyL6FECIb3YmxKVNl6MBojDIoa+iEuIRSygx8Hbgd2Ai8Rym1cdJmHcC9wEOT3lsNfA64GtgBfE4pVZV4+RvAfcCaxNdtRfoR5iWvPzxjQ5SkOrcdrz9MtICN64RYaCSgS2TdMsnQFSqgS44sWFrtosXjJBCOMTJe2LEIQgiRiQsZuqnvhle5bJKhEyK9HcBJrXW71joMPAzcmbqB1vqM1voQMDniuBV4Qmvt01oPAk8AtymlmoAKrfVL2rjb+33gHUX/SeaRQX94xhl0SXVuO1pfGHUgRCkq+YBuJKOALpmhK8wd6s5EQLesxkWzxyhzkll0Qoi50DM8c8mlrKETYkotQGfK912J5/J5b0vi8bT7VErdp5Tar5Ta39/fn9VBz3feLAK65Cy6/lFpjCJKV8kHdJmUXBY8Q+c1ArrWqgsBnayjE0LMhe7hIOV2C+XTdJOrKrMxFIhIabgQl0q3ti3TX5Sp3pvRPrXWD2itt2utt9fV1WX4kfNfOBpnNBjNKkMH0Cfr6EQJk4Aug6YoVrMJp9VcsAxdhy9AY4UDh9VMs8dYt3JeRhcIIeZA70iQhorpmw9Ul1kJx+L4w7FZOiohFowuYEnK963A+Tzf25V4nMs+F7xkebdk6ITIXMkHdCPBCDazCYfVPO12boelYBm6Dl+ApdUuAGrL7FjNivPSclcIMQd6RoI0TVNuCUbJJRjrWoQQF9kHrFFKrVBK2YC7gUczfO/jwC1KqapEM5RbgMe11t3AqFLqmkR3yw8AjxTj4Ocj75hxnsm0KUptuQR0QpR8QDcajE6bnUsqaEDnDbAkEdCZTIomGV0ghJgjPcNBGqbpcAkpAZ00RhHiIlrrKPBxjODsdeDHWuujSqkvKKXuAFBKXaWU6gLuAu5XSh1NvNcH/CVGULgP+ELiOYCPAN8GTgKngMdm8ceaU8n1uplm6BxWMxUOC30S0IkSJoPFg1EqnFOvn0tyO6wFmUMXjMToGQmyrMY18VxzYnSBEELMplhc0zcamrbDJRhr6ABpjCJEGlrr3cDuSc99NuXxPi4uoUzd7kHgwTTP7wc2F/ZIFwav3wjMasozC+gA6isckqETJU0ydMFIxhm6kQJk6LoGL4wsSGr2SIZOCDH7vGMhYnE9bYdLuHCnXDJ0Qohiu5Chy2ywOEBduV0ydKKkSUCXYcllhcNakKYoHckZdKkZukonvaMhGYophJhVmQwVB6hyGVUMPhkuLoQoMp8/jEmBJ4PqqaT6Crtk6ERJk4AuGMFtz6TksjBr6Dq86TN0ydInIYSYLT0jmQV0FQ4rJgVDkqETQhSZ1x+mymXDZEo3vSG9unIjoJPRKqJUSUCXVVOU/O9On/UFcNnMF3VvmhhdIGWXQohZ1DVonHNaqqYvuTSZlAwXF0LMisEshoon1bntjEdijIUK07xOiIVGArpgdNqh4kluh5VgJE4kz7LIzsTIAqMTsaElMVz8nAR0QohZ1OH147ZbJkoqp+NxWWUNnRCi6Lz+8EQjpkzVV8joAlHaSjqgi8U1Y6FM19AZ2+RbdnnWG7io3BKgKRHQJdezCCHEdIYDEQ50DOa9nw6fMUIl9QbTVKrLbAzKGjohRJH5/OGMZ9Al1ZUblU6ydEWUqowCOqXUbUqp40qpk0qpT6d5/Xql1CtKqahS6l2TXrtHKXUi8XVPoQ68EJKp+cxKLo072PmUXWqt6fAFLhpZAFBut1DhsEjJpRAiI9998TS/ff9LBMJ53mBKcz6aSpXLJhk6IUTR+XIsuQTJ0InSNWNAp5QyA18Hbgc2Au9RSm2ctFkHcC/w0KT3VgOfA64GdgCfU0pV5X/YhZEMzioyKrnMP0PXPxoiFI1fkqEDGV0ghMhc70iISExzoncs533E4pou3/hFHXenI2vohBDFFotrBgPZZ+jqJaATJS6TDN0O4KTWul1rHQYeBu5M3UBrfUZrfQiYvMDsVuAJrbVPaz0IPAHcVoDjLohkcJZNhm5kPPcM3dmJkQVll7zW4nFybkhKLoXIxmKtHpiJLzF493jPaM776B0JEo7FWVZ96fkonaoyG0OBiHSRE0IUzVAgjNZknaHzuKxYzUpKLkXJyiSgawE6U77vSjyXiXzeW3QXArrMM3T5DBdPN7IgqdnjpHtYMnRCZGoxVw/MJJkpO5ZHQHd2mvNROtVlVsKxOP5wLOfPFEKI6UwMFS/PfKg4gFKKereDo+eH5aaTKEmZBHTpVstn+tuS0XuVUvcppfYrpfb39/dnuOv8JUsuMx0snvqeXJz1BTCpC10tUzV5HAwFIvil5a4QmVq01QMzSV70HO8dyXkfHT4/QFZr6MBoKS6EEMXgTZxfsi25BHjfNct47sQA/+/Jk4U+LCHmvUwCui5gScr3rcD5DPef0Xu11g9orbdrrbfX1dVluOv8ZVdymf8auk5fgKZKJzbLpf/ZWyY6XUqWTogMzUr1wFzdcJrORECXZ4bOYlI0VU4/VDwpGdDJOjohRLFMZOhyCOg+/MaVvPOKFv7vE208+mqml6lCLA6ZBHT7gDVKqRVKKRtwN/Bohvt/HLhFKVWVKGe6JfHcvHAhQzdzyWV5AQK6Dt+lIwuSmidm0ck6OiEyVPTqAZi7G05TicU1Q+MRKhwWBsbCDIzltmakwxegpcqJxZzZ9JrkXCjpdCmEKBZfHhk6pRR/884t7Fhezf/+yav85uzUo11O9Y9x9wMv8Rf/dTTnYxViPpnxL7nWOgp8HCMQex34sdb6qFLqC0qpOwCUUlcppbqAu4D7lVJHE+/1AX+JERTuA76QeG5eGMkiQ2c1m3BazfmVXHqnbhGeDOi6pdOlEJkqevXAfJRsGnD1yhog9yzddDeY0qmWgE4IUWTJgM7jyj6gA7BbzHzz/VfSVOngvu/vpzPRjC4pHtd894XTvPVrz7Gn3ccjB8/nvebu16/38pUn2vLahxD5yujWrNZ6t9Z6rdZ6ldb6rxLPfVZr/Wji8T6tdavWukxrXaO13pTy3ge11qsTX98tzo+Rm5FgBJvZhMNqzmj7Cqcl5wxdIBxlYCzEkikuoBrcdkwKGV0gROYWbfXAdJIXPDtXGQFdro1RprvBlE6Vy5r4fBkuLoQoDp8/jNthSbs0JVPVZTYevPcqIrE4H/qXfYwkbsSfGxrnfd/Zy1/812tcu7KGT960Bp8/TNdgftdd33vxDF/99Ql6hqXCSsyd3H9jFoHRYDSj7FyS22FlNJTbxUyHb/qOchaziYYKh5RcCpGhxVw9MJ1kQLe2wU1NmY3jPdk3RhkORBgej2Q8sgCMxlAmZWQIhRCGDEan2JVS/5Z4fa9Sanni+fcqpQ6mfMWVUpclXns6sc/ka/Wz+1PNHa8/+xl06ayqK+eb77uS9n4/H3/oAD/e38ltX3mWVzuH+Jt3buHBe6/ilo0NABzsHMr5c7TWHD43DMDjR3vyPm4hcpV5NLMIZR/Q5Z6hS44smO6OuAwXFyI7WuvdwO5Jz3025fE+jHLKdO99EHiwqAdYBMmArsplY12jO6eSy+QNpqkqBtIxmZQMFxciRcrolDdjlHHvU0o9qrV+LWWzDwGDWuvVSqm7gb8F3q21/iHww8R+tgCPaK0PprzvvVrr/bPyg8wjPn8op4Yo6excXcsX37GZT//sMM+29bNjRTX/cNe2ifPeukY3douJg51DvH1bc06f0TU4zlDAuNG/+3A39+xcXpBjFyJbJR7QRTJqiJLkdlgZzvHu9EwZOjACukNdud8pEkIsfr7EOaim3AjoHn65k3hcYzKl6/OS3tksRxYkeVxWWUMnxAUTo1MAlFLJ0SmpAd2dwOcTj38K4/SrxgAAIABJREFU/JNSSumLF269B/hR8Q93/vOOhWmtyu68NJ27dywlHDOm1rzv6mUXnSetZhObWyp5NY8M3aEuIzt34/p6njreR/9oiDp3djP0hCiEki+5rHDOUobOF6DCYZl2oW+zx0H3cJB4XIZiCiHS840lmwZYWd/oZjwSm7hhlKlsh4onVZfZGJQ1dEIkZTL+ZGKbRJn4MFAzaZt3c2lA991EueWfK6Uyv1uzwPkKVHKZ6gPXLucD1y5Pe9PrsiUeDp8bJhKbPKo0M4fPDWM1K/7w5rVoLWWXYu6UeEAXwW3PPENX4bBMdMbMVocvwNIZ7oY3VzoJR+MTgzWFEGIyrz+M227BbjGzrrECyL4xSqcvQG25jTJ7dkUaVS6bZOiEuCCT8SfTbqOUuhoIaK2PpLz+Xq31FuANia/3p/3weTgjMx9aawYDYarLCxvQTWfbEg+haDznbsGHzw2xrtHN5pYKVtaV8diR7gIfoRCZKfGALoemKDmOLejwztwiPDm6QNbRCSGmknrBs7ahHKWyH11wNoPzUTqyhk6Ii2Qy/mRiG6WUBagEUhsw3c2k7JzW+lzi31HgIYzSzkvMtxmZ+RoJRonEdMEzdNO5rNUDwKs5LHfRWnO4a5gtLR6UUrxlcxN72n14c5wNKkQ+JKDLZg2d3UIoGicczS41H4trugbHWTpDR7lmjwOQgE4IMTWfP0xVonTbZbOwtNpFW292AV2HL8Cymsw7XCZVlRkZunznNgmxSGQyOuVR4J7E43cBTybXzymlTBgdeB9ObqyUsiilahOPrcDbgCOUgMHEzaJCNUXJxJJqJ9VlNg52ZB/QdfgCjASjbG2tBOD2LY3E4ponXust9GEKMaOSDehicc1YKPsul0DWWbqekSDhWHzGO+ItyQydzDIRQkxh8hqTdQ1ujmUxuiAcjXN+eDynDF11mZVITOMPx7J+rxCLTSajU4DvADVKqZPAp4DU0QbXA13JpioJduBxpdQh4CBwDvhWkX+UeSG53KRqFgM6pRTbWitzytAlxxVsaTECuo1NFSyrcbH7iKyjE7OvZLtcjoWMtXDZBHQVTiObNxqMUlOeeRejTEYWAFQ6rTitZsnQCSGm5POH2dBUMfH9+kY3vz7WRzASw2E1z/j+rsEAWmffEAWYyAwO+sOUZ7n+TojFKIPRKUGMLFy69z4NXDPpOT9wZcEPdAFIlnPPZsklwGVLqni6rT/rzueHu4axmU2sbXADRnB4++Ymvv1cO0OB8LRN8IQotJLN0CWzbBVZji0w3ptdY5TODEYWgHEyaPY4JKATQqSltb40Q9dYQSyuOdk3ltE+zvoyu8GUTjKgk8ZNQohC8/mNtWezWXIJsG1JJVpfyLhl6vC5YdY3ubFZLlxKv2VLI1EpuxRzoIQDuuwzdMltR7IsuTzr82MxKZoqHTNuK8PFhRBTCYRjhKLxi0qS1jUad4czbYySrBiYqetuOsn3tPdnFjwKIUSmvBMZutmd47Yt0RjlYBbz6LTWHD43PFFumbSlpZIWj5PHpOxSzDIJ6LLK0OW2hq7DN05LlROLeeb/3C0ep6yhE0Kk5UvTNGB5jQubxcTxDBujdPgCOK1m6rIoG09aVVeO02qeGKYrhBCF4hsL47SacdpmLh0vpKoyG8trXFkNGD/rDTAajF4S0CmleMuWRp470Z/1zX8h8lHCAZ3xi5bVGrpE8JftLLpTfWMZd5Rr9jjpHw0RikrTASHExdKtMbGYTayuK894Fl1yZEEus4rNJsXmloqsS5OEEGImPn941sstk7Yt8fBqZ+bntUPJhiitlZe8dvuWJiIxza9fl7JLMXtKOKDLveQymzV0wUiM472jbG259Jc+nWRZZo9k6YQQk/im6AK3vtHN8Qw7XXb4/DmVWyZtafFw9Pww0Vh241uEEGI6Xn+YmlkcKp7qsiUeekaCGV97HTk3jM1yoSHKRftq9dBU6WD3YSm7FLOnhAO6ZIYu85LLZFe3bEouj54fJhbXbFviyWj75OiCc7KOTggxyVRd4NY1uukdCTEUmL5ZidbamEGXQ4fLpK2tlQQjcU7KOjohRAHNdYYOMl9Hd6hriA1NFVjTLKUxmRS3bW7kmbb+iY7qQhRbyQZ0Izlk6CxmEy6bOasM3YHEsMptSzLL0DUnArruIcnQCSEuNlWGLtkYZaayy/7REMFIPL8MXaLESNbRCSEKaS4Duo1NFVjNKqOALh7XHDk3Mm3l1Vu2NBGOxnnyWF8hD1OIKZVwQBfBZjZlNLcpldthySpD92rXMC0eJ/XumTtcAjQmSi6l06UQYjKvP4zVrHBPmgG3vtGYSzdTp8uzGY5Qmc6KmjLK7RYOS0AnhCggnz9M9RzNbnNYzWxoqsioMcoZr5+x0KUNUVJdubSKeredxw53F/IwhZhSyQZ0o8FoVtm5JLfDmlWG7tXOoYyzc2CcVGrLbZwfloBOCHGxwcQd7MkNTRoq7FQ6rTNm6M56kzPoMmvSlI7JpNjUXDHRFEAIIfI1Ho4xHolRPUdr6MAYX3D4nLFMZjqHp2mIkmQyKW7f3MiTx/omKiuEKCYJ6LJU4bBkHNB5x0J0+AITM04y1exxck5KLoUQk3j94Ynh3qmUUqzLoDFKhy+ASV1Yq5urra2VvN49QkQaowghCsCbGCo+eX3wbLpsiYexUJRTM6wPPtw1jN1iYk19+bTb/c7VywhF4/zo5Y5CHqYQaZVwQBfJqiFKktthzXi2SHKNyWUZNkRJaq1y0uH1Z31sQojFbTAwdRe49Y1u2nrH0Hrqu8sdXj9NlU5slvxO/VtaPYSjcdoynH0nhBDTuTBjc3aHiqfKtDHKoXPDbGyumHG28LpGN29YU8v3XzpDOCo3v0RxlXBAl2vJZeYZugOdQ5gUbM5wZEHSpuZKzngDDI/LUEohxAW+KTJ0YFw8jIWidA1OXa591hdgWR4NUZKSzQBkHZ0QohC8EwHd3GXoVtaW4XZYpg3o4nHN0XPD066fS/W7u1bQOxLisSOylk4UVwkHdJE81tBlFmi92jnE2gY3ZfbsPidZoikXS0KIVN6x0JQlSesTnS6na4zSWaCAblmNC7fDIuvohBAF4RtLP5JlNplMim2tnmkbo7QP+PGHYxkHdG9cW8fKujK+8/zpaasnhMhXCQd0USpyKLmscFgmRh5MR2vNq11DWZdbwoWFtq92ZTYPRQix+EVicUaC0UtGFiQlB9wen6IMciwUZWAszJI8OlwmKaXY2lopN52EEAUxUXI5h01RwBgxdaxnlPFwLO3rRzJoiJLKZFJ8cNcKDnUN85uzgwU7TiEmK+mALrc1dBbC0TihaPpf9qSz3gBDgUjGA8VTVTqtrKgty6h9rhCiNAwGpr+D7XZYafE4p+x02ZHscFmde4fLVFtaPBzrGZnxXCiEEDOZaiTLbLtsSRWxuObo+fQ3qw51DeOwmlhdN31DlFS/dUULlU4rD75wulCHKcQlSjKgi8U1Y6HcxxYAM66jS2bXcsnQAWxrrZTBvUKICYN+o9R7uqYB66fpdNnhMxotFaLkEoxOl5GYnnH2nRBCzMTnD6UdyTLbtiUyb1OtoztybphNzZUzNkRJ5bJZeM+OpfzySA+diVmgQhRaSQZ0YyEjGMu1KQrMHNAd6BjCaTXP2NZ2KltbPfSMBOkdkfEFQogLbb2ryqauLFjX6Ka935/2oqEj8VwhSi6BiTUkcuNJlDql1G1KqeNKqZNKqU+ned2ulPq3xOt7lVLLE88vV0qNK6UOJr6+mfKeK5VShxPv+Zqa60inyHz+yJQNn2ZTfYWD5koHz58cuKQzZSyuOXI+84Yoqe7ZuQylFN9/6UxhDlSISUoyoEs2NcllDd2FDN30jVFe7RpiS2t2d3FSJYeRS9mlEAIuZOhqpsnQ3bV9CWV2C/c8+PIlw2zPegN4XFYqndmf99JprXJS5bLKOjpR0pRSZuDrwO3ARuA9SqmNkzb7EDCotV4NfAX425TXTmmtL0t8fTjl+W8A9wFrEl+3FetnmA98/tCUI1lm222bm3j6eD/Xf/kpHnj21MT13umBMQJZNERJ1VTp5C1bmnj45c6JpIIQhVSiAV3uGbqKxHtGxqf+hQxH4xw9P5JzuSXAxqZKzCYld7+FEIBxwQPTZ+hW1Jbx7Xu20zU0zof+Zd9FC/s7fAGWFSg7B0ZjlC2tHul0KUrdDuCk1rpdax0GHgbunLTNncC/JB7/FLhpuoybUqoJqNBav6SN1ojfB95R+EOfP3z+8JzOoEv152/bwPc+eBUrasv4693H2PmlJ/nbXx7jyWN9QOYNUSb73V3LGQ1F+en+zkIerhBAyQd0xcnQHesZIRyNT4wfyIXTZmZdg1s6XQohgAtzmmYqS7pqeTVfu/syDnYO8YkfHSAaM8qGOnwBltYUpiFK0taWStp6RwlGitsYJR7X/K+HD/Dcif6ifo4QOWgBUq/QuxLPpd1Gax0FhoGaxGsrlFIHlFLPKKXekLJ91wz7RCl1n1Jqv1Jqf3//wv7d8PrDczqyIJVSijetq+dH913DIx/bxfVr6rj/mVP89e5jOK1mVmXRECXV5UuruGKph+++eIZ4XEYYiMIq0YDOCMaKtYYuWSZ52dLcAzowyi4PdQ3L7BIhBIP+MJVOK9YMyrhv29zE59++iV+93sufP3KUaCzOucFxllY7C3pMW1oricU1r3Wnb8RSKMd7R3nk4Hn+7D+PEInFZ36DELMnXaZt8h/tqbbpBpZqrS8HPgU8pJSqyHCfaK0f0Fpv11pvr6ury/Kw549wNM5oMDqnQ8Wnsm2Jh6+/9wqe/KM38YFrl/GRN63CbMp9OePvXreCs94Av05k+4QolBIN6PIpuTQydCPTZOgOdA5RW26nudKR2wEmbG31MDwe4axXuiIJUeq8/nBWFzz37FzOR960ih+93MH/+Y8jROO6YCMLkpJrSYq9jm5vuxcw1gH+WMqVxPzSBSxJ+b4VOD/VNkopC1AJ+LTWIa21F0Br/RvgFLA2sX3rDPtcNJIjWeZjQJe0vLaML9y5mU/etCav/dy2qZHmSgffeb69QEcmhKFEA7pkhi77ksvyDDN0ly2pzLv9brJkU8ouhRCDgewCOoA/uXUd77y8hX9LBEFLCzSyIKmp0kFtua3oa31fPuOjxeNk+7IqvvqrE1MO/RViDuwD1iilViilbMDdwKOTtnkUuCfx+F3Ak1prrZSqSzRVQSm1EqP5SbvWuhsYVUpdk1hr9wHgkdn4YeaCd2z6GZuLicVs4q7tS9jT7puxuZ4Q2SjJgG4kjwyd2aQos5mnDOhGghFO9fvzaoiStLahHIfVxKud0nRAiFLnHQtn3dZbKcWXfmsrb1hTCxhNUwpJKcWWlkoOnyveTSetNS+f9nH1imr+5Lb19I2G+JeXzhTt84TIRmJN3MeBx4HXgR9rrY8qpb6glLojsdl3gBql1EmM0srkaIPrgUNKqVcxmqV8WGvtS7z2EeDbwEmMzN1js/IDzYFkR975nKErpE3NFQCc6Bub4yMRi0lGAV0xZqzMpdFgFJvZhMNqzun9lU4rJ/pG065tO5QIvrYVIKCzmE1saq7kkGTohEhrsZ2bpuPLsWmAzWLigfdv598/spOGivzKwNPZ0urhZN8Y/jStuP2hKA8+fzqvNt2n+scYGAtz9cpqdqyo5oZ1dXzj6VMMj8vdbTE/aK13a63Xaq1Xaa3/KvHcZ7XWjyYeB7XWd2mtV2utd2it2xPP/7vWepPWepvW+gqt9X+l7HO/1npzYp8f14t4MX1yxuZ8GVtQbGsb3ACc7JWAThTOjAFdEWeszJnRYCSn7FzSe69ZxnMnBvj6UycveS1ZHrk1jw6Xqba2VnLk/PBEpzohhGExnpumorU2Si5zvOBx2sxcuayqwEdl2NpSSVxzSWOUQX+Y3/n2Xr7w89d4+OWOnPe/97SRsNixwmgK+Me3rmd4PMIDz57K/aCFEPOGL8MOvovFkmoXdouJtt7RuT4UsYhkkqEr+IyVuTYSjOYV0H30Tav4H5e38Pf/3cbPD128TvlAxxAr68oKNrz3siUegpE4bXInR4jJFt25aSqjoSiRmKZ6Hl7wJGcypTZG6R4e5677X+L17hGqy2w8f3Ig5/3vbfdR77azPLH+b2NzBXdsa+bB58/QNxrM7+CFEHNu0B9GKfDMw/NbMZhNitX15bRJyaUooEwCumLMWLnIbM9SMTJ0uQdcxrqULWxfVsUf/fhVDnQMAsZd9IOdQ1xWoOwcXMj0zUbZ5Wgwws8PnScm81HEwlD0c9N8MTiP15g0VDhoqLBzODFg/FT/GO/6xkv0DAf5/u/u4G1bm9jb7iMczb7KQGvN3tNerl5Zc1GTqU+9eS2RWJx/evLSKgkhxMLi9Rvrg/MZB7DQrG1wc0IydKKAMgnoijFj5eINZ3mWymieGToAu8XM/e+/koYKB//z+/vpGgzQPRxkYCyU9/y5VMtrXFQ4LLxa5C5yAA88287HHzrAB7+3j6FEG2Eh5rGin5tgfgzv9c7jgA5gS4uHQ11DHO4a5q5vvkQoGuPh+67hmpU17Fpdy3gkNnHjKxsdvgC9IyF2rKi+6PnltWW8+6olPLS3gw4Z6yLEgubLciTLYrCmoZzu4eC0I7CEyEYmAV0xZqzMqXzX0CXVlNt58N6rCEXjfOh7+yfKirYVMEOnlGLbEs/EsPJiev7kAHVuOy+dGuCOf3qBYz3FHRYsRJ5m5dw0H4b3zucMHRhrfdsH/LznW3twWs385MM72ZyYUXfNyhpMCl7Ioexyb7uxfu6aSQEdwCdvWoPFrPjKr9ryO3ghxJzKdsbmYrC23miMckKW04gCySSgK/iMlcIceu6MDF1h1ritri/nG++9kpP9Y/zZfx7BZjaxoSntjf6cbW2t5HjvKMFI8WYvjQYjHOoa5t3bl/DwfdcSjMR45z+/yO7D3UX7TCHytOjOTVOZ9xm61kq0NubS/ftHdl40HqHSaWVrqyendXR7TnupKbOxur78ktcaKhx8cNcK/vPgObn5JMQClmsH34Us2elSyi5FocwY0BVxxsqcKUTJZarr1tTyl3duJhyNs7G5ApulsOP9trZ6iMU1R88X76Jlb7uPWFyzc3UNVy6r4r8+cR3rG9189Iev8OVfHpN1dWLeWYznpqnM9zlN16+p48vv2spPPnwtjZWXjka4bnUtr3YNZ11etLfdx44V1Retn0v14etX4bZb+Jvdx3I6biHE7NJaMx6O0T8a4qzXz9Hzw/SPhubtua1YWqucOK1maXgnCiajqEZrvRvYPem5z6Y8DgJ3pXnfvwP/nucxFlQsrhkLFS5Dl/Q7Vy8lFo/TUuUs6H7hQgnnq51DRWs9/sKpAewWE1csNfbfUOHgR/ddw+cffY1/fvoUr3eP8MAHtmM1l+QsejFPLaZz03QG/WHsFhMuW26zM4vNbFL89vYlU76+a3Ut//TUSfa2+3jzxoaM9tk1GODc0Di/94YVU25T6bLyiRvX8Fe7X+fZtn6uXzs3JbFCiOmNBiO851t7eO38COnuDzd7Cn/tNJ+ZEp0uT/RJhk4URuHSVAtEcsBtRQEzdEnvv3Z5wfcJ0FhpdJErZqfLl0552b686qJh63aLmb955xbWNZTz+f8yZkkV62cUQkwtucZkAU5cAOCKZR6cVjMvnBzIOKB7OTF/7uoVNdNu94Gdy/jBnrP89e7X2bW6tqQ65QmxUHz/pbMcOTfC7123gppyO2V2M2U2C2V2C26HpWg3q+ezNQ3lOa0tFiKdkgvoRhMlPxUFztAV29ZWD4eK1OlyYCzEsZ5R/vjWdWlfv2fncnYf6eGrvz7JO69opcxecv/bCDGnBhd40wC7xcyOFdU8dyLzLqF7231UOq2sb3TPuO9P376ej/7wFX6yv5O7dyzN93CFEAU0ForyrefauXF9PX/2to1zfTjzxtoGNz975RzD45GCzS4Wpavk6udGg0aGrpBr6GbDtkQXueHxwre4ffGUFzDKotJRSvHp29czMBbiwedPF/zzhRDTWwxd4K5bXcupfj/dw+MZbf/yGR9XLa/GlEHG7fbNjWxfVsU/PNE2UYUhhJgf/nXPWYYCET5x4+q5PpR5ZW2D0exJGqOIQijhgG5h3Q3ZtsRYR3e4CFm6F08O4HZY2JJoM57OFUuruHVTA/c/2453LFTwYxBCTG0xzGlK3jB64aR3xm37RoKcHvBzdZpxBekopfg/b91A/2iI+585lddxCiEKJxCO8q1n27l+bR2XLy29ssrprEmMLpDGKKIQSjCgMzJcCy1Dt7Ul0RilCOvoXjg1wDUra2Zce/LHt64nEI7y9afkgkmI2bTQSy4B1je6qSmzZbRmZE9y/dzKzAI6gMuXVvH2bc1867n2jLOAQoji+uGeDrz+MP/rpjVzfSjzTosn2elSMnQifyUY0C3MkstKl5XlNS72nfEVdB5dpy9Ap2+cXaumbzwAxsy9396+hH/dc5ZOX6BgxyCEmFooGmM0FKXatbADOpNJsXN1Lc+fHEDr6cegvHzaS7ndwsYsZ3r+ya3riGv4u8eP53OoQogCGA/HuP/Zdq5bXVuSTU9mYjIp1jRIp0tRGCUY0CUzdAur5BLgimVVPH28n02fe5yb/uFpPvbDV/jqr07wyyM9OZdBJu+WT7V+brI/uHktSsFXnmjL6fOEENkZChjnrOryhR3QAVy3uob+0RAn+qYvMdrb7mP78iosWY5JWVLt4nd3reBnr5wrSnm6ECJzD73cwcBYiE9Kdm5Ka+rdUnIpCqLkArqRBZqhA/jc2zfxT79zOR990ypW1pVz5Pww//jrNj78r7/hLV97LqfM3QunvNS77ayuL89o+8ZKBx/ctYL/OHiO17uLN+hcCGHwjiWGii/wDB3AdWuMOXHPn5i67NI7ZgR8OzJcPzfZR29YRXWZjS/+4rUZM4FC5EspdZtS6rhS6qRS6tNpXrcrpf4t8fpepdTyxPNvVkr9Ril1OPHvjSnveTqxz4OJr/rZ+4kKIxiJ8c1nTnHtypqcf5dLwdqGcvpHQwwFwnN9KGKBK7mAbjQYxWY2XTRvbaGodFp529Zm/uiWdXzrA9t55o9v4Ohf3MpX776M3pEQjxw8l9X+tNa8dGqAnatqsppv9ZE3rsJtt/DlXx7L9kcQQmTJ508EdAt8DR0Ya0ZW1JZNu44u0/lzU6lwWPnDm9ew97SPJ17rzWkfQmRCKWUGvg7cDmwE3qOUmtyX/0PAoNZ6NfAV4G8Tzw8Ab9dabwHuAX4w6X3v1VpflvjqK9oPUST/tq+T/lHJzs1kbYPRGGWmqgUhZlKCAV1kQWbnpuKyWbhjWzPrG91894UzWd2RbusdY2AszM4Myy2TKl1WPnbDap463s+e9pk71gkhcucLLJ6ADmDX6hr2tHuJxOJpX9972ofTamZr69Rdd2fynh1Laa1y8tDLHTnvQ4gM7ABOaq3btdZh4GHgzknb3An8S+LxT4GblFJKa31Aa30+8fxRwKGUss/KURdZKBrjG0+fYseKaq7NYH1+KVuTGF0gjVFEvkowoIsuqoAOjJbdH9y1nGM9o+xN3N3ORPIu+c4cTrj37FxOU6WDLz12TMqahCgiX2J97GIJ6K5bXYs/HOPVzvQde/ee9nHlsiqsWa6fS2Uxm7htUyMvnvTKXDpRTC1AZ8r3XYnn0m6jtY4Cw8DkP7q/BRzQWqcuhv9uotzyz9UUJTRKqfuUUvuVUvv7+/vz+TkK6sf7u+gZCUpnywy0eJyU2cyckHV0Ik8lGNBFFmRDlJnceVkLHpeVf3nxTMbvefHUAMtqXLRWubL+PIfVzCdvWsPBziH2nRnM+v1CiMz4AhGUAs8iWEMHcO3KWpSC5yatoxsYC/HZR47wevcI12QxrmAqN29sIByL81zb/LnQFYtOukBr8h3OabdRSm3CKMP8/ZTX35soxXxD4uv96T5ca/2A1nq71np7XV1dVgdeLJFYnG88dZIrl1XldLO41CilWN3glgydyFvJBXQjizBDB0aAdfdVS3n8aA/nhmaewRSNxdnb7mPnquzKLVPdsa0Zu8XELw6dn3ljIUROfP4QHqd1xjmRC0Wly8rWlsqJCgF/KMpXf3WCN375KX64t4P3Xr2UD+5akffnbF9WhcdllXV0opi6gCUp37cCk/8gTmyjlLIAlYAv8X0r8B/AB7TWEwNetdbnEv+OAg9hlHYuCPvPDHJ+OMjvXbciq7X5pWxtfbl0uhR5K7mAbrGtoUv1vmuWAvCDl87OuO2hc8OMhqLsWp37HbQyu4Ub1tXz2JEeYnEpuxSiGAb9kUVTbpm0a3UtBzqH+M7zp3nj3z3NV37VxvVr6/jvP7yev/ofWyiz53+OtphN3LiunieP9xGdYr2eEHnaB6xRSq1QStmAu4FHJ23zKEbTE4B3AU9qrbVSygP8AviM1vqF5MZKKYtSqjbx2Aq8DThS5J+jYJ490Y/FpLhuTe43i0vN2gY3A2MhBv3S6VLkrgQDuuiiLLkEaK1yccvGRh7e1zHjCIMXE3fHr12ZX0nEW7c20TcaYv+ZzNfuCSEy5/WHFl1Ad92aWmJxzV/+/DVW1pbxs4/u5Bvvu5JVdZmNT8nUmzc2MBSIsP+slIWLwkusifs48DjwOvBjrfVRpdQXlFJ3JDb7DlCjlDoJfApIjjb4OLAa+PNJ4wnswONKqUPAQeAc8K3Z+6ny88zxfq5cVrVor7OKQRqjiEJYnKmqaSzGpiip7t21nF8e7eGRg+d491VLp9zuxVNeNjRVUFOeX1OtG9fXG2WXh7u5Os/gUAhxKZ8/zIrasrk+jIK6ank1v//GlVy1rJqbNtQXrTTrDWvrsJlN/Oq1Xq6R85MoAq31bmD3pOc+m/I4CNyV5n1fBL44xW6vLOQxzpa+0SCvdY/wJ7etm+tDWVCSowva+sbkOkohPnLtAAAgAElEQVTkrKQydLG4Ziy0eDN0AFevqJ5xhEEwEmP/2UF2FWDBcpndwo3rpexSiGLx+SNUly2KbuYTrGYTn7l9AzdvbCjqOptyu4Wdq2t44vVe6cYrRJE912ZU/ly/Zn40aFkomioduO0WTkiGTuShpAK6ZPvqikWcoVNKce9OY4TBy1OMMNh/ZpBwNM6uLOfPTeWtW5voHw2xT8ouhSioeFwzGAhTXbZ4b0IV280bGjjrDXBSBvcKUVTPtPVTW25nY1PFXB/KgmJ0uiyXkkuRl5IK6EaDEYBFXXIJxgiDSqeV700aYTAejvHPT5/kIz/8DeV2C1etyL81OBhllw6riV8c6i7I/oQQhtFglFhcL7oM3Wy6eUMDAP8t3S6FKJpYXPPciX6uX1uLaZF05J1Na+vdMotO5KXEArpkhm5x3+122szcvWPJxAiDaCzOwy938Ka/f4ov//I4O5ZX8+8f2Ul5ATrJAbhsUnYpRDF4/cmh4ov7nFVMjZUOtrZW8qvXJaATolgOnxtmMBDhjWul3DIXaxrK8frDeMdCM28sRBolGdAt5jV0Se+/ZhkAn3vkCLf+47N8+meHafY4+bf7ruE7917FukZ3QT/vrVuaGRgLTVnmKYTIni/RxloydPl584YGDnYO0TcanOtDEWJReratH6XgugIt5Sg1E41RJEsnclRiAV1plFyCMcLgzRsb+NXrfQDc//4r+dlHdhatg9IN6+uMssvDMmRciEKZCOhci2tswWy7eWMDWsOTifOhEKKwnmnrZ2tLZd6ds0tVMqA70Sfr6ERuSiygS2boFn9AB/CXd27mm++7ksf/4Hpu3dRY1G5yLpuFm9Y38EspuxSiYCYCunIJ6PKxvtFNa5WTJ2QdnRAFNxyIcKBjkOul3DJnDRV23A6LNEYROSuNyCbhQoZu8ZdcAtRXOLhtc+Osfd5btjTxi8Pd7D3tZecqKbsQIl++gGToCkEpxc0bGvjRyx0EwlFctpL60ydEUb1waoC4RtbP5UEpxdoG94IsudRac7JvjGfa+tnT7qPCaWFlbRkr68pZWVfG8poyHFbzXB/moldSf9VGSixDN9tuWF+H02rmF4e6JaATogCOnh/BbbfgtMkfw3zdsrGB7714hudODHDrptm70SXEYvfM8X7cDguXLfHM9aEsaGsbyieqnMzzvFPoSDDCiycHeKatn2eO93N+2FifvLzGRTAS52evnJvYVilornSyc1UNt29pZNfqWuwW+ZtWaCUV2YwGo9jMJrlTUCQum4UbN9Tz+NEe/uKOTVjMJVXRK0RBvXzaxy8OdfOxG1bN9aEsCletqKbCYeFXr/VKQCdEgWiteaatnzesqZW/+Xm6ZmUNP3q5k3u/+zJfu/tyqsrmX2XG0fPDfPeFMzx68DzhWBy33cKu1bV84qY6rl9bR4vHCYA/FOX0gJ/2AT+n+/209Y7yy6M9/OQ3XbjtFm7aUM9tm5t407o6uSYvkBIL6CKSnSuyt25p4heHunn5tI+d0u1KiJxEY3E++8gRWjxOPnbD6rk+nEXBajZxw/p6njzWtyDugE8lGIkRjMSodFqLui5aiEyc6BujZyTI9Wuk3DJfd17WQiAc43OPHOVt/+957n//lWxuqZzrwyIW1/z69V4efOE0e9p9OK1m3n3VEt6+rZnLl3qwpgnky+wWNrdUXnT84WicF04N8Njhbp54rZf/PHgel83MrtW13LCunjetq6M5ERCK7JVUdDMajEpAV2Q3rKvHaTXz88PdEtAJkaPvv3SWYz2jfPN9V8p6rwK6eUMDjxw8z4GOQbYvr57rw5nS8HiEV84O8lr3CD3DQbqHx+keDtIzHMSbaJRTZjPTWuViSbUz8a+LFbUutrV6pNOgmDXPHO8HkIYoBfKeHUvZ0FTBR//1N7zzGy/yxXds5re3L5mTYxkJRvjp/i6+9+IZOnwBWjxO/r+3rOfd25dS6cq+F4XNYuKGdfXcsK6eaCzO3tM+HjvSzVPH+icaVq1tKOeGdfW8cV0d25dVY7NI1jdTJXWlYGToSqMhylxx2szctKGex4/08Ke3rs/pl16IUtY3EuQrT7TxxrV13LqpYa4PZ1F547o6rGbF3/7yGH9w81quXVmDaY4zdVprzg2Ns//MIPvP+th/ZpDjvaPoRLNgj8tKY4WDZo+TbUs8NFU4cNrMnBsap9M3TtdggJdOefGHYxP7XFbj4oqlVVyx1MPlS6tY3+iWcjhRFM+09bO2oVwyKwV02RIP//WJ6/jEjw7wJz89xMHOIT739o2ztu7syLlhfrj3LP954DzjkRjbl1Xx6dvXc8vGhoKdRyxmE7tW17Jrde1EU5Wnj/fzdFsfD75wmvufbcdpNbN9eRXXrqrh2pU1bGmplPPYNEosoJMM3Wx45xUt/PxQN9u+8N+sqivjiqVVXL60isuXeljb4M641Mk7FmLfGR/LaspY1+Ce8wsvITLVMxzEaTNT6cz+hsbfPHaMUDTO5+/YJCV1BVbhsPKZ2zfwtSdP8N5v72VptYu7dyzhXVe2Uu92zNpxBCMx9rR7efp4P08d7+OsNwCA227h8mVVvHVLE9uXV7O1tZIy+8x/s7TWDAYinOgd5UDnEK+cHeS5EwP8xwGjMYHTamZ9k5vNzZVsaq5gU3MlaxvLpTFBCTo3NM6f/vQQX3zHZpbXluW1r0A4ysunfXzg2mUFOjqRVFNu5/u/u4O/++/j3P9MO4e7hvnYDau5eUP9jEFN70iQl0/7cDss1JbbqSm3UV1mm/b3PRiJsftwNz/Yc5YDHUM4rCbu3NbC+65ZxpbW4pZ9KqVY0+BmTYOb/3n9SsZCUV465eWFkwPsaffy5V8eB6DcbmHHimo2NVewpNrFkkSFQlOlc8GW0BeS0np+zQzbvn273r9/f0bbPn9igGtWVmccsd/ylWdYUVvG/e/fns8higzsP+Nj72kfBzoGeaVjaGKeVpnNzBXLqtixvJodK6rZtsRz0YLYTl+Ax4/28N+v9bL/jI94yl3qHcuruXplDdesrGZDY0VGAV48rukbDdHhC9A/GsLnD+H1h/H5w8a/Y2HGIzEisXjiSxOOGo9dNjMel3Ei9LisVLtsVJXZqHfbaaly0upx0eRxpK0fF5dSSv1Ga72gf/kyPT+979t7OdE3ypfeuZUb1tdnvP+97V7e/cAePnHjav7olnX5HKqYRjAS4/GjPTy0t4O9p31YTIo3b2zgrVub2LG8mvqKqYO7eFzzes8Ie9p9DAXCeFw2PE4rVWXWicd2q5lwNH7hKxYjFI1zqt/PU8f6ePHUAMFIHIfVxK5VtbxhTS07VtSwrjHzG14z0VrTNTjOKx2DHOwc4uj5EV4/P8JoyOj2bDEpVteXs67RzdoGN2vqy1nb4GZJtavkLo5K6dy0p93L7//gN8Tjmi+/ayu3b2nK+TOfOtbHB7+3jx98aAdvkDV0RfPY4W6+8PPX6B4O0lBh591XLeXuq5ZclBUd9Id57EgPj756jr2nfaS7tHc7LFS5bCgFca2Jx43zRExrxoJR/OEYK+vKeN/Vy/itK1rnTYXVwFiIve0+Xjw1wEvtXs4M+Ekdd2w1K5o9Tlo8RnDX7HHQWOmgudJJk8dBTZkx428hNl/J5ty0YAO6tt5RbvnKs6yoLeMPbl7D27c2z3iBf+3f/Jpdq2v5+7u2FepwRQa01nT4AhzoGOI3ZwfZd8bHsR5jeKbNbGLbkko2NFWw78wgr3ePAMYg4Fs2NfKGNbV0eAPsPe1lT7uPDp9xJ7vCYaHZ48TjslLlMgKuSqeNCqeFgdEwHT4/Z70BOnwBQtH4JcfkcVmpLrNR7bLhsluwmRUWkwmrxYTVrLCaTAQiMYYCYQYDYQb9EQYDYQIpZU0AJgUNFQ5aPE5aqoyTSYvHkTipGCcWaV5gKMZFk1LqNuCrgBn4ttb6S5NetwPfB64EvMC7tdZnEq99BvgQEAM+qbV+fKbPy/T8dLhrmP/9k1c53jvKb13RymfftnHGP46RWJy3fe15xkJRfvWpN8qogllyqn+Mh1/u4Ke/6WIwYMwqXVbj4splVVy1vJqrllcRjmr2tHvZ0+5l72kfw+PGdkqR9sJpOkurXdy43mgAcM3Kmlm9yIjHjXPx0fMjHD0/zGvdI5zoHePc0PjENg6riZW15dS57VSX2ahy2agus1KVOF9WOq1UOK1UOq1Uuqy47ZYFf37L99xUjPPQTPucLJub4V2DAT720AFe7Rzi3p3L+cxb1ueUrf38o0d5eF8HBz97y4K8WF5IorE4Tx3v54d7z/JMWz8KuHF9A9evreXp4/0829ZPNK5ZWVvG27c1c9OGeiKxOANjYbxjYbxjxs3socRsU5NSKKUwm4zHdouJWzc1cu2qmnn/+xyJxekeCtLhC9A5aFzndfoCnB8y1hr3jgQvCviSbGYT5Q4L7sSXy2bBbjElvszYEo+t5uSXwmo2YUn8azObJraxpbzHYk7899LJf4wH0ZgmGI0TDMcYjyS+wjGC0RifuX1DRj9rwQO62bxoyvSkpLXmidd6+b9PtHGsZ5T1jW4+9ea1vHljw5T/M27+3OPctb2Vz71904z7F8U1FAiz74wR3O097eP17hEua/Vwy6YG3ryxgWU16UtBzg+Ns/e0l31nBukfDTEcMAKtofEIQ4EwkZjGaTWztNrF0hoXy6pdLKtxsbSmjIaKCxcouWbV/n/27jxOrrJK+Pjv1Nr73kk66YR0SCAJBAJkQTYVEIMLcQHZXkXHGXSUGbeZEeedUYeRGX2d0Vl0HBkFcWMRXDIaxAVEZUnSgRASEqCzd9be1+qu7Xn/uPd2Kp1eqqurupZ7vp9Pf7r61q3q56bTT9/zLOcMRWIc7x3icFeI1q4Qrd0h+/EgR3pCHOsZIhI79Xeq2O+loaqIhko70KssoqGqmLqyIGVBq2MpC/oosz8HfZ6c71BTke6ATkS8wKvAm4BWYAtwszHm5YRzPgKcZ4z5sIjcBLzTGHOjiCwHHgDWAHOB3wBnGWNio79PoqncNA1HY3ztiRb+63d7qCsL8M/vWsGVS8ffE/etP+zlC7/YxTffe5Gm1c+CSCzOy0d62bK/ky37rb1sTgISx4KaEi5eVMPFi2q5eFEtcyqK6BuKWoM+Cf3QcCR+yh/8gM+6GZhTWcTC2pKc+/3uH47y2vE+Xj3ex6vH+9nb1k/nQJhOezCr357VG4tHoKLYT7HfO3JjFPSfvEnyegSfR/CM+uwV6/HIZw/4PB489k2m1+M5+dn+9zIYjLHvm4whbiASjxOJGsKxWMKMaJxLF9dx69rklgJOp2/KRD9kv2zC9xxtKn0TWBkHv/jYbu59eh/nN1bytVsuZH5NSdKvN8Zw5b8+xRm1JXznA2uSfp2avkOdgzyw+SAPNx+ivT9MQ2URbz9/LtedP5dz5lbkXP8y06KxOCf6hjlqJ5TqHAjTNxSlbyhK/3Bk5PFgOEo4Gmd45CM28nU0ZkZWbo0VHE5Hkd/DS59/c1L3oWkN6Gb6pmmqnVI8bvjf7Uf4t9+8xr72Ac5vrOTjbzqL6pIA+9r72ddm18FoH2DnkV4+dtUSPvGmsyZ/Y5V3jDGEIjGK/d6sdWjxuKG9f5gjPUMc6Q7ZH0Mc67U+H+0JcaJveMJRfZ9HKA1awV1p0EtJwHpcHPCO3Bj6vUIgYSRJxBpl88jJkTdPwjERGfMcr4DHc/J8Y6x/R2Nfi8FKWRyLG6Jxq4OLxgyRuPX59isWMXuC5WmJMhDQvQ74vDHmzfbXnwEwxvxzwjmP2+c8KyI+4BhQD9yZeG7ieRN9z6n2T2DN1v31Iy+y+1gf77pwHp+65myCPg9x+87UAN2DEd79jWdYtbCa+96/2vV/kHOBMYZ97QM07+/C5xXWLqodqbHkNsPRGN2DETr6w/SEIvSEIvTan3tCEXqHIgxFrGWlwxHrxmjI/hyLW0u6ojFD3Fj9iNOnxO3nYnFrCVjUvnlynreeG7uzFAHBSq4Q9FqrK5wR9IDPw9vOa+DjVyf3t36aAV3a+yH7ZRO+52ip9E0Av9xxlL9+ZDsCfOnd53HhGdUIgIBg/d2IxQ177Vpirx7v47Xj/bxyvI+eUITPvX05H7i0acrfV01fOBrnYOcAi+rKNMdABsXiJ4M7J+BL/ByJx3H+9Z2/3QJ4PUKR30txwEux3zsy6DWVn9VU+qZkMoSsAVqMMXvtN38QWA8kjhSt52Qn9AjwNbGuaj3woDFmGNgnIi32+0140zQVHo+wfuU83rqigR8/f5h//+1rfOC+LSefF+x0zqX8yaVN3LCqMV3fWuUYEcl6inePR5hVUcSsiiJWzq8a85yIPXrU2R+mbzjCwHCM/uEI/UNR+oaj1lr24Sj9wzEGhqMMhKP0D0dp7x+2OpSYNSI98jgWtwMxe128PXKdsWsU6ybK7xGuv6gx6YAuA+YBhxK+bgXWjneOMSYqIj1ArX38uVGvnZeJRq5orGTDHZfxn0+8xn/9bg8/fv7wmOcFvB4+/3ZNhJIrRIRF9WUsqi/LdlOyLujzMrvCm5XfdZPQnwl2IJdbvyOZ6ocme8+0WHduA8sbKvnID7fy5z94ftLzy4t8nDW7nLesaGBZQzk3XJSdlPrKKgOweFZ5tptR8Lwewevx5vyy4mTufvPipsnn9fCe1fNZf8FcfrXzOEGfh0X1pcyvKdFMXiqn+L0ea89dhkf7nRk2J8hzgr6YMdZNUvzkczH7ebFHZZ1ZPY/9tc9rffg9UxtdyrCxGjI6lB3vnGRea72ByO3A7QALFiyYSvtGBHwePnXN2bz1vAY27+u0R8BP/vuKwIp5ldPOOqdUoXFWEuSwTPRDY63FOq1/SkffBLCgtoRH//wSHnvpGAPh6CnLWo31fVhQU8LZs8uZXRHMtYBaKUVyAV3Gb5rS1SmBNZL49vPnTus9lCoETuDlHfPXsCC0AonDw43AkXHOabWXOlUCnUm+FgBjzD3APWAta5pOg5fOqWDpnIrpvIVSKrdkqh+atH9KZ98U9Hl5xwUZGW9XSs2AZDJDTKWzIpWbJmPMPcaYVcaYVfX1mvpWKZWULcASEWkSkQBwE7Bh1DkbgNvsx9cDTxhr4/AG4CYRCYpIE7AE2DxD7VZKFY5M9EPJvKdSSo1IZoZupGMBDmN1LLeMOsfprJ4lobMSkQ3AD0XkK1hJUfSmSSmVFvby7juAx7Ey8N5rjNkpIncBzcaYDcC3ge/Z+3c7sfov7PMextoLHAU+OlmGS6WUGi1T/dBY7znT16aUyh+TBnR606SUylXGmI3AxlHHPpvweAi4YZzX3g3cndEGKqUKXib6obHeUymlxpNUSkC9aVJKKaWUUkqp3JNadWWllFJKKaWUUlk3aWHxmSYibcCBKbykDmjPUHNykduuF9x3zYV6vWcYY/I669EU+6dC/TlOxG3X7LbrhcK8Zu2bCp/brtlt1wuFec1J9005F9BNlYg0J1tFvRC47XrBfdfstustVG78Obrtmt12veDOay40bvwZuu2a3Xa94M5rTqRLLpVSSimllFIqT2lAp5RSSimllFJ5qhACunuy3YAZ5rbrBfdds9uut1C58efotmt22/WCO6+50LjxZ+i2a3bb9YI7r3lE3u+hU0oppZRSSim3KoQZOqWUUkoppZRypbwN6ERknYi8IiItInJnttuTCSJyr4icEJEdCcdqROTXIvKa/bk6m21MJxGZLyJPisguEdkpIh+zjxfyNReJyGYRedG+5n+wjzeJyCb7mh8SkUC226qSV+j9k/ZN2jdp35SfCr1vAu2fCr1/0r5pbHkZ0ImIF/g6cC2wHLhZRJZnt1UZ8R1g3ahjdwK/NcYsAX5rf10oosCnjDHLgIuBj9o/10K+5mHgSmPM+cBKYJ2IXAx8Cfiqfc1dwAez2EY1BS7pn76D9k3aN2nflFdc0jeB9k+F3j9p3zSGvAzogDVAizFmrzEmDDwIrM9ym9LOGPN7oHPU4fXA/fbj+4F3zGijMsgYc9QY87z9uA/YBcyjsK/ZGGP67S/99ocBrgQesY8X1DW7QMH3T9o3ad9kHy+oa3aBgu+bQPsnCrx/0r5pbPka0M0DDiV83Wofc4PZxpijYP0SA7Oy3J6MEJGFwAXAJgr8mkXEKyLbgBPAr4E9QLcxJmqf4qb/34XArf1TQf+eOrRv0r4pj7m1b4IC/111uKV/0r7pdPka0MkYxzRdZ4EQkTLgUeDjxpjebLcn04wxMWPMSqARawR12VinzWyr1DRo/1SgtG/SvinPad9UwNzUP2nfdLp8DehagfkJXzcCR7LUlpl2XEQaAOzPJ7LcnrQSET9Wh/QDY8yP7cMFfc0OY0w38DusNfBVIuKzn3LT/+9C4Nb+qaB/T7Vv0r6pALi1b4IC/111a/+kfdNJ+RrQbQGW2BltAsBNwIYst2mmbABusx/fBvwsi21JKxER4NvALmPMVxKeKuRrrheRKvtxMXA11vr3J4Hr7dMK6ppdwK39UyH/nmrfpH1TIXBr3wSF/bvqqv5J+6ax5W1hcRF5C/BvgBe41xhzd5ablHYi8gDwBqAOOA58Dvgp8DCwADgI3GCMGb35Ny+JyGXAH4CXgLh9+G+x1oIX6jWfh7V514s1wPKwMeYuEVmEtWG9BngB+D/GmOHstVRNRaH3T9o3Ado3ad+Uhwq9bwLtn+zDBds/ad80trwN6JRSSimllFLK7fJ1yaVSSimllFJKuZ4GdEoppZRSSimVpzSgU0oppZRSSqk8pQGdUkoppZRSSuUpDeiUUkoppZRSKk9pQKeUUkoppZRSeUoDOqWUUkoppZTKUxrQKaWUUkoppVSe0oBOKaWUUkoppfKUBnRKKaWUUkoplac0oFNKKaWUUkqpPKUBnVJKKaWUUkrlKQ3olFJKKaWUUipPaUCnlFJKKaWUUnlKAzqllFJKKaWUylMa0CmllFJKKaVUnvJluwGj1dXVmYULF2a7GUqpNNu6dWu7MaY+2+2YDu2flCo82jcppXLRVPqmnAvoFi5cSHNzc7aboZRKMxE5kO02TJf2T0oVHu2blFK5aCp9ky65VEoppZRSSqk8pQGdUkoppZRSSuUpDeiUUkoppZRSKk9pQKeUUkoppZRSeUoDujyy43APv9xxNNvNUEqpU0Rjcf7n93sZDEez3RSllMppBzsG+ckLrdluhiowGtDlkf9+ag9/+eA2+of1pkkplTteONTN3Rt38b8vHsl2U5RSKqfd98w+Pvnwi0Rj8Ww3RRUQDejyyIm+YcLROE/sPpHtpiil1IjuwQgAm/Z1ZrklSimV2w50DGIMOjiv0koDujzS3jcMoMsulVI5pSdkBXSbNaBTSqkJHegYAKBvSAM6lT5JBXQisk5EXhGRFhG5c4znrxCR50UkKiLXJxxfKSLPishOEdkuIjems/Fu02YHdE/ubiMUjmW5NUpln/ZNucEJ6Fq7QhzuDmW5NUoplZviccOhLquP7B2KZLk1qpBMGtCJiBf4OnAtsBy4WUSWjzrtIPB+4Iejjg8C7zPGnAOsA/5NRKqm22g3GorE6BuOcsmZtYQiMZ56tS3bTVIqq7Rvyh1OQAeweV9HFlui1MzKxKCSiHxHRPaJyDb7Y+VMXY/KrGO9Q4Sj1t45naFT6ZTMDN0aoMUYs9cYEwYeBNYnnmCM2W+M2Q7ERx1/1Rjzmv34CHACqE9Ly13GmZ1723lzqS7x67JLpbRvyhm9oQhlQR8VRT427dVll8odMjyo9NfGmJX2x7aMXICacQc6Bkcea0Cn0imZgG4ecCjh61b72JSIyBogAOwZ47nbRaRZRJrb2nTmaSxt/VZA11BZxJuWz+a3u04wHNVll8rVMt43qeT0hCJUlfhZ01Sj++iUm+igkpqSg50DI497Q7rkUqVPMgGdjHHMTOWbiEgD8D3gA8aY0/K0GmPuMcasMsasqq/X/mwszgxdfXmQa89toG84yjMturRJuVrG+yb7HB1wmkRPKEJlsZ+1TbXsbR/gRO9Qtpuk1EzI5KDS3fZSzK+KSHCc12nflGdOnaHTgE6lTzIBXSswP+HrRiDpYkMiUgH8Avg7Y8xzU2ueciQGdJcsrqU86GPjS7rsUrnajPRNOuA0OSegW9NUA8Dm/TpLp1whU4NKnwGWAquBGuDTY71W+6b8c6BzkHlVxYAuuVTplUxAtwVYIiJNIhIAbgI2JPPm9vk/Ab5rjPlR6s1UbX3DiEBNaYCgz8tVy2bx613HiWhhSuVe2jflCCegO2duBaUBr+6jU26RkUElY8xRYxkG7sNa2qkKwMGOQRbVl1Lk99CndehUGk0a0BljosAdwOPALuBhY8xOEblLRK4DEJHVItIK3AB8U0R22i9/D3AF8H7N1jQ9bf3D1JQE8HutH9m6cxvoHozojZNyLe2bcocT0Pm8Hi5aqPvolGtkZFDJnrVDRAR4B7Ajra1WWXOgY4AzaksoL/LrkkuVVr5kTjLGbAQ2jjr22YTHW7BGpka/7vvA96fZRoU1Q1dffnIZ/evPqqfY7+WxHUe5bEldFlumVPZo35QbnIAOYG1TDV9+/BU6B8LUlAay3DKlMscYExURZ1DJC9zrDCoBzcaYDSKyGitwqwbeLiL/YGe2dAaVakXk/fZbvt/OaPkDEanHWtK5DfjwzF6ZyoSewQi9Q1HOqCmlvMhHry65VGmUVECnsm90QFcc8HLl0lk8vvM4d60/F69nrKX8SimVWUORGOFonIqEgA5gy/5O3nzOnGw2TamMy8SgkjHmyjQ3U+WAA3aGywX2DJ1muVTplMweOpUD2vqGqS87NdHVunPn0N4/zNYDXVlqlVLK7Zyi4s4M3YrGSoI+jy4HV0qpBE6GyzNqS6go8mlSFJVWGtDlAWMMbf2nztABvHHpLAI+D49pkXGlVJaMDuiCPi8XLqhm834tq6KUUo6DnVZAt6CmhPMoBd0AACAASURBVArdQ6fSTAO6PNA7FCUcjZ8W0JUFfVyxpJ5f7jhGPD6lTMlKKZUWowM6gDVNNbx8pJdevWFRSinASohSXx6kJOCjXGfoVJppQJcHEmvQjXbtuXM42jPEi63dM90spZSiZ/D0gG7tohriBrbu1+XgSikF1pLLM2pKADSgU2mnAV0eGAnoyk4P6K5eNhufR/jljmMz3SyllBqZoatICOgumF+N3ys8t0+XXSqlFFhLLhfUOgGdn1AkprWEVdpoQJcH2vrHn6GrLPFzyeI6Ht+pAZ1SauY5yyoTZ+iKA17Ob6zSenRKKYWVDfhY7xBn1JQC1gwdoLN0Km00oMsDEy25BHjdolr2dwyOLH1SSqmZMjJDV3RqFZw1TTW81NrDYFhvWJRS7tbaNYgxVoZLgIoiawBME6OodNGALg+09Q3j98opI+CJljaUA/DK8b6ZbJZSStETilAW9OHznvrnZE1TDdG44fkDur9XKeVuTsmCk0sudYZOpZcGdHmgvX+YurIgImMXD186xwrodh/rnclmKaUUPaHImINNqxbW4BHYpPvolFIuN1KDrubkHjpAMwGrtNGALg+09Z1egy7RnIoiKov97DqqM3RKqZnVG4qckhDFURb0ce68SjbpPjqllMsd7BykLOijpjQAnJyh6w3pDJ1KDw3o8kBb3/CYGS4dIsLSOeU6Q6eUmnHWDJ1vzOfWNtWw7VA3Q5HYDLdKKaVyx4GOARbUlIystNI9dCrdNKDLA239E8/QASxrqODVY31aYFwpNaPGW3IJsKaplnA0zvMHtR6dUsq9DnQOjiREAd1Dp9JPA7ocF4sbOpII6JbOKWcgHKO1KzRDLVNKqYkDutedWUt50MeDmw/NcKuUUio3xOKG1s7QSEIU0IBOpV9SAZ2IrBORV0SkRUTuHOP5K0TkeRGJisj1o567TUResz9uS1fD3aJzIEzcjF+ywHG2nRhlly67VErNoIkCurKgj5vWzOcXLx3lcLcONiml3OdY7xDhWHykBh2Az+uhJODVJZcqbSYN6ETEC3wduBZYDtwsIstHnXYQeD/ww1GvrQE+B6wF1gCfE5Hq6TfbPUZq0E2whw7grNnliMBuTYyiXEQHm7JrOBpjKBIfN6AD+MClTQhw3x/3zVzDlFLKtuNwDy0n+rP2/Q90DACcsuQSrFk6zXKp0iWZGbo1QIsxZq8xJgw8CKxPPMEYs98Ysx2Ij3rtm4FfG2M6jTFdwK+BdWlot2u09U9cVNxRGvRxRk2JJkZRrqGDTdnnFBWfKKCbW1XM285r4IHNB0fOV0qpmfLpR7fzD/+7M2vf/6BTg65mdEDn1yWXKm2SCejmAYkbIFrtY8lI6rUicruINItIc1tbW5Jv7Q4jM3STBHQAS+dU8MoxnaFTrqGDTVnWawdoY5UtSPSnly9iIBzjwc0HZ6JZSik1oqM/zJ5sztB1DuL3CnOrik85Xl7k04BOpU0yAd1Y1ayTTaWY1GuNMfcYY1YZY1bV19cn+dbu4AR0dZMsuQRrH92+jgFCYU0Rrlwh44NNoANOE0lmhg7g3HmVXHJmLfc9vZ9wdHRsrZRSmdMTinCkZyhr90YHOwZprC7B6zn1lriiyK976FTaJBPQtQLzE75uBI4k+f7Tea3CCuhKA15Kg2PXeUq0rKEcY+DV4zpLp1wh44NNoANOE0k2oAP4sysWcax3iF+8pH8CVGHJxF5eEblIRF6y3/M/xClgpqZkOBojZNfB3Nc+kJU2HOgcOG25JegMnUqvZAK6LcASEWkSkQBwE7Ahyfd/HLhGRKrt/SnX2MdUkpKpQedYOqcCQPfRKbfQwaYsm0pA94az6lkyq4x7fr8PY7RepioMGdzL+w3gdmCJ/aFLwlOQuG83GwGdMYYDHYOnJUQBaw+dJkVR6TJpQGeMiQJ3YAViu4CHjTE7ReQuEbkOQERWi0grcAPwTRHZab+2E/hHrKBwC3CXfUwlqa1vKOmAbkFNCcV+L7t1H51yBx1syrKeweQDOhHhzy5fxK6jvTyzpyPTTeMPr7VxqHMw499HuV7a9/KKSANQYYx51lijH98F3pHxKylAvacEdDO/j657MELfUHTMGbqKIh+9OkOn0iSpOnTGmI3GmLOMMWcaY+62j33WGLPBfrzFGNNojCk1xtQaY85JeO29xpjF9sd9mbmMwtXWl/wMnccjnD2nXEsXKFfQwabs6wlZNyOTJUVxrL9gLnVlQe75/d5MNoueUIQ/+c4Wvvn7PRn9PkqRmb288+zHqbynStA9eDKg25uFGboD9qDSGbWlpz1XXuQjHI0zHNW8B2r6Jt+YpbKqrW+YyxbXJX3+0jnlPL7zGMYYdMm9KnTGmI3AxlHHPpvweAvWcsqxXnsvcG9GG1jgekIRSgNe/N6kxgYJ+ry8/5Iz+Jdfvcorx/o4e055Rtr1213HicQMnQPhjLy/UgkysZc3qfcUkduxlmWyYMGCJL+luzhLLsuCvmkvuYzFDXFjku7vYPwadHByIKxvKEqwzDuttimV/P9KNeOGIjF6h6JJZbh0LJ1TTtdghBN2dkyllMqUnlAkqeWWiW5dewbFfi/f+kPmZuke23EMgK4B3Z+iMi4Te3lbOXUgasz31IRNk3MCuvPnV04roIvFDX/ynS3c+M1np/S68WrQgTVDB2hiFJUWGtDlsA57dDnZJZcASxusxCi7jmpiFKVUZvWEIkkvt3RUlwa4YVUjP912mBO9Q2lvU/9wlKdetcpLdGshc5V5ad/La4w5CvSJyMV2dsv3AT/LROMLnbPk8oL51XQPRuhKcdb+337zKk+92saLrT0MRZJfInmgc5DZFUGK/KfPwJUHnRk67afU9GlAl8OmUlTcsdRewqQFxpVSmdabwgwdwAcvayIaN3x/U/oLjT+x+wThaJyFtSV0D+qSS5VZGdzL++fAt4AWYA/w2AxeVsE4OUNXBaS2j+7JV07wn0+0sKiulFjcTOn+6mDHIGfUnL5/Dk7O0PWGdIZOTZ8GdDkslYCuqiRAQ2WRZrpUSmVcKksuwUoQ8Pqz6nloy0GisfQWGv/ljqPUlwe5cunsUxIiKJUpmUgcZ4xpNsaca7/nHUZrfaSkJxShvMjH4lllwNRLF7R2DfKJh7axrKGC/37vRQDsPJL8CqgDnQMsGGP/HFhlC0Bn6FR6aECXw1IJ6ADOnlOuSy6VUhmXakAHcMuaBRzvHea3u0+krT2D4ShP7m5j3TlzqC0LEIrEprQ8SilVWJw+qrG6GJ9HplS6YDga46M/eJ5YzPCNWy9kcX0Z5UEfLx/tSer1Q5EYx3uHOWOM/XOge+hUemlAl8OcgK62dGoB3dI5Fexp6yccTe/It1JKJZpOQHfl0lnMqSjiB2lcdvnUK22EIjGuPXfOSLt6dB+dUq7VE4pQVeLH7/WwoKaEvW3Jz9B94ee7eLG1hy/fcD4L60rxeIRlcyt4OckZuoN2yYLxZuic/cdaXFylgwZ0Oaytf4jqEj8B39R+TMsayonEDHuzUERTKeUO4WicUCSWckDn83q4ac18/vBa20gmuOl6bMcxakoDrGmqobokAECX7qNTyrW6B8MjfVRTXWnSSy5/tu0w33vuAH92eRPrzp0zcnx5QwW7jvYRi0++AvZAx/g16MAqpQA6Q6fSQwO6HDaVouKJls6xMl1qYhSlVKY4M1+VJakFdAA3rp6PAA9smf4s3VAkxm93Heea5bPxeT1U2e3SfXRKuVdPKEJVsTW44wR08UmCsdeO93Hnoy+xemE1f7Nu6SnPLZ9bQSgSY3/H5IHhSA26cZZcej1CWdCnM3QqLTSgy2GpBnSL6kvxe4VdR6cW0LX3D/Odp/dxoi/9qcSVUoXFuQlJdYYOoKGymKuWzeZHzYemvUT8j6+1MxCOce2KBgAN6JRSp5RWaaovZTga5+gk5VLu3riL4oCX/7z5wtOKiJ8z1xowT2bZZcuJfiqL/SN90VjKi3w6Q6fSQgO6HNbWP0z9FIqKO/xeD2fWl7H72NQSo3zrD/v4/P++zGVffJLP/Hg7e9p0yaZSamzODF1FUeoBHcCtaxfQ3h/mVy8fm9b7bNxxlMpiP5ecWQtYGX8BLV2glEsZY07Z59tUZy193DfBPrpILM6mvZ287bwG5lQWnfb8klnl+L2SVKbLzfs6Wb2wGquU4NisgE4HndT0aUCXo4wxKc/QASxrqGD3FGfonm5pZ3lDBTesauTR5w9z9Vee4s++28zWA52Tv1gp5SojAd00ZugArlhST2N1MT94LvVll+FonF+/fJyrl80eGVGvdmboNCmKUq4UisSIxMzIDNmiOqd0wfiD1dtbewhFYqxtqh3z+YDPw+JZ5bw8SSbx471D7G0fGPd9HBVFfp2hU2mhAV2O6h+OMhSJpxzQLZ1TzrHeoaRHp7sHw+w40sObz5nD3e9cwTN3XslfvHExm/d18u5vPMuN33xWs8UppUb0hqa/5BLA4xFuXrOAZ/d20HIitVUBz+xpp28oyltWnExeUOz3EvB6NCmKUi7lLLd2+qjZFUFKAt4Ji4tv2tcBwNpFNeOec04SmS6f22u9z8WLJg7odMmlShcN6LJkOBrjIz/Yyl/96MUxn0+1Bp1jaYO1zjvZAuPP7e3AGLh0sdX51JUF+eQ1Z/PsZ67k79+2nOYDXXz2ZztSaotSqvD0pCmgA3jPqvn4PMIDm1ObpXvspWOUBX1ctqRu5JiIUFXip0f30CnlSk4fVWX3USIyaabLTXs7WTyrjLoJtrssb6igvX+YExPsxXtubyflQR/L7T134ykv8mtSFJUWSQV0IrJORF4RkRYRuXOM54Mi8pD9/CYRWWgf94vI/SLykojsEpHPpLf5+Skai/MXP3yBjS8d49HnWznSHTrtnJGAruz0NdzJWDqnHIDdSRYY/2NLO6UBL+fPrzrleEnAxwcva+Ivr1zCz7Yd4acvHE6pPUqpwtIzmL6Arr48yJvPncOjz7dOuRB4NBbnVy8f46plswj6vKc8V1Xi1xk6pVxq9AwdTFy6IBqL07y/k4snmJ0DRoK0nRPcX23a28Gaphq8nvH3z4HO0Kn0mTSgExEv8HXgWmA5cLOILB912geBLmPMYuCrwJfs4zcAQWPMCuAi4ENOsOdW8bjhbx7Zzq9ePs6HrliEMfCTMYKktv7pzdDNKg9SXeJPeobumZYO1i6qPS2jk+OjbzyTi86o5u9/uoNDnempGaXUdOlgU/b0hCLWssYp1skcz61rFtA9GGHjS0dPOd7RP8w9v9/D2//zj3z4e1v56QuHT1n+vWlfJ12DEa5NqBXlqCoOaJZLpVxqrH2+i+pKOdQ5OGZW3R1HehkIj79/zrF8kkyXJ+z9c5MttwRrhq5vKIIxk9e1U2oiyfwlXgO0GGP2GmPCwIPA+lHnrAfutx8/AlwlVlofA5SKiA8oBsLA1FIvFhBjDJ/bsJMfv3CYv7rmLD7zlmVcvKiGR7a2nvbLPN0llyLC0jkV7EoioDvSHWJv+8BIdrix+Lwe/u3GlRjgkw9vS6qoplKZpINN2ZWYPS4dXndmLYvqSvnBpoMYY3hmTzt3/PB5Lv7n3/JPG3fj8QgvHOri4w9tY9UXfs377t3MDzcd5EfNhyj2e3n9WbNOe8+qEr8GdEq5lLPPN7FsQFN9KXEDB8cYmHb2vU20fw6sRCbza4rHDeie22clkksuoPMRiRmGp1m2RalkArp5wKGEr1vtY2OeY4yJAj1ALVZwNwAcBQ4C/2KMcW3KxP/3+Ct877kDfOj1i/joGxcDcP1F89nXPsDzB7tOObetbxivR0bWfqdiWUMFrxzrJRSeeAnT0y3tAFy6uG7C8+bXlPCP7ziHLfu7+MbvWlJul1JpooNNWZTugE5EuGXtArYe6OL1X/4dt/zPJn7/ahv/5+Iz+PUnruBnH72UZ++8ih9/5BL+5NImDnQM8Lc/eYmfbjvClUtnURzwnvaeVSV+ukO65FIpN3J+909dcmllutw7RlmmTXs7WFRfyqzyybe6LG+oGDfT5XN7O5LaPwcnZw91H52armQCurEWAI+enhnvnDVADJgLNAGfEpFFp30DkdtFpFlEmtva2pJoUv75+pMtfON3e7h17QLuXLd0pC7JtefOoSTg5UfNraec394/TF1ZAM8k668ncs05sxmKxE9bwjTaM3s6qCsLcPbs8knf8x0r53Hd+XP56m9eY9uh7pTbplQazMhgkxv6p1SkO6ADePeFjcwqDzK7IshX3nM+m//v1Xzu7eewxO6bPB7hwgXVfOYty/jdX72Bxz52OXdeu5RPXnPWmO9XXaJLLpVyq55QBK9HKAv6Ro411dq16Ebto4vG4mzZ3zXpckvHOXMr2d8xQP/w6fvfntvbweok9s8BVBRZbesN6T46NT3JBHStwPyErxuBI+OdY494VwKdwC3AL40xEWPMCeBpYNXob2CMuccYs8oYs6q+vn7qV5HjHtx8kC8//grvWDmXf1x/7ilFJkuDPt6yooGfbz96ykzadGrQOdY21bCwtoSHthwa9xxjDE+3tPO6M+uSCh5FhH98x7nMqSji4w++wMAYnZlSMyTjg01Q+P1TqnpCkWnXoButujTA5v97NT/68CW868JGivynz7o5RIRlDRV8+PVncmZ92ZjnVJb4GY7GJ12loJQqPN2D1qBT4j1XZYmf2tLAaQHdy0d76R+OTpoQxbG8oQJjTk88d6J3iL1tA0m/T7kd0GlxcTVdyQR0W4AlItIkIgHgJmDDqHM2ALfZj68HnjDWprCDwJViKQUuBnanp+n54/ubDnDuvAq+fMP5YwZN11/USP9wlMd3Hhs51tY/TP0EaXOTISK8Z/V8Nu/vHHN5AUDLiX5O9A1z6QT750arLPbzlfecz4HOQe7635en1UalpiHjg01qfL0ZmKFLt+qSAIAuu1TKhcZbRdBUV3paLbpNe5Pf9wYJiVFGBXRT2T8HVlIUQDNdqmmbNKCzlyndATwO7AIeNsbsFJG7ROQ6+7RvA7Ui0gJ8EnCyzX0dKAN2YAWG9xljtqf5GnJeZ3+Ys2dXjJtBcs3CGubXFPPI1pPLLtMxQwdw/YWNeD3Cw6OWdDqS3T832tpFtdx+xSIeaj7E/glquiiVQTrYlEWZWHKZbs4e5K4BHf1Wym0mCuhGz9A9t7eDprpSZlckVyqqobKI6hL/aYlRRvbPNUy+fw4SZ+g0oFPTk1S+aWPMRmPMWcaYM40xd9vHPmuM2WA/HjLG3GCMWWyMWWOM2Wsf77ePn2OMWW6M+XLmLiU3GWPoGAhTUzr+jY/HI7z7wkae3tPO4e4Q8bihvT+cloBuVkURbzy7nkefbyUSOz2L0h9bOlhQU8L8mpIpv/frz7KWnx3pOb2OnlKZpoNN2ROJxRkIx3I/oNMZOqVca7yAblF9GW19wyPLHGNxw+b9naxtSm6ZJFgroJbPrWDnGAHd6qYafOMM4I9WMTJDp4NOanrSU0BIjSsUiTEcjVNTOnFw9u4LG62adM+30jUYJhY3015y6bhx9QLa+oZ5cveJU45HY3E27e3g0sXJL7dMVGe3r6Nfb5ZUduhgU3Y46cAri32TnJldTrpyTYyilPv0hCKnlCxwNNVZiVH2t1ulC3Yd7aVvKJr0MknH8oYKXjneNzJYfqJvavvn4OQMnWa5VNOlAV2GOcHORDN0YJUEcGrSnRipQZfc1P9k3nh2PfXlQR5uPjU5ykuHe+gbjk55uaXjZEA3PO02KqXyh1Owt3KMm6VcMrKHTgM6lSEisk5EXhGRFhG5c4zngyLykP38JqfepYjcKiLbEj7iIrLSfu539ns6z51eZFFNykmKMtqieiug29tu5RZItv7caOfMrSQcjbO3zVq+OdV9eAClAR8iuuRSTZ8GdBnWNegEdJPPtl1/0Xz2dwzyyx1WcpR0LLkEqyj4uy9s5MlX2jjROzRy3Nk/97opjko5qor9eATadYZOKVcZCehyfsmlvYduUPsolX4i4sVavn0tsBy4WUSWjzrtg0CXMWYx8FXgSwDGmB8YY1YaY1YC7wX2G2O2JbzuVud5O3GTmoJ43NA7NHZAt6CmBJGTpQue29vJGbUlNFQWT+l7OIlRdh7psd+ng7Ip7J8Da8tNWdCnAZ2aNg3oMqxjILkZOjhZk+7+Z/cD6QvoAG5cPZ9Y3PDI8yeTozzd0sGyhgpqU1za6fEINaVBOgZ0hk4pN8mXgK7I7yXo84y0V6k0WwO0GGP2GmPCwIPA+lHnrAfutx8/AlwliXn0LTcDD2S0pS7TNxzFmLH7qCK/l3lVxexrHyAeN2yZ4v45x6K6UoI+z0hilOf2drB6YXXS++ccFUV+XXKppk0DugzrGkh+hs6pSecsD0pnQNdUV8qaphoe3nIIYwyhcIytB7q4LMX9c466soDO0CnlMvkS0IG17NLph5VKs3lA4l6GVvvYmOfYiZx6gNF/eG/k9IDuPnu55d+PEQCqSfQMTtxHOZkudx/roycUSbqgeCKf18PSOeW8fLSXE31D7GkbmPI+PLD20ekMnZouDegyrNMJ6Oy9HJO5/qJGAIr9XkoD4xfVTcWNq6wlnZv2ddJ8oJNwLM4lKe6fc9SVBXUPnVIu4yRFSXdh8UyoKvHTrTN0KjPGCrTMVM4RkbXAoDFmR8LztxpjVgCX2x/vHfObi9wuIs0i0tzW1ja1lhc4Z9Cpapx7r0V1pextG0h5/5zDyXSZyv45R0WRf6RPVSpVGtBlWOdAGK9HqEgyG9yahTUsqCmhrjxAugfl3rKigfKgj4e3HOLplg58HmHNwtQ6MUetztAp5Tr5NENXVeIfGa1XKs1agfkJXzcCR8Y7R0R8QCXQmfD8TYyanTPGHLY/9wE/xFraeRpjzD3GmFXGmFX19fXTuIzC45QqmWiGrn84ys+3H6GxupjG6qmXbgIr02VPKMJPXjhMWdDHOXOT3z/n0Bk6lQ65nXO6AHQOhKkuST4483iEu995Ll0ZuAEpDni5buVcHtnaSmN1MRcuqKY0OL3/ArWlOkOnlNv0hCIU+T0EfeldRZAJVcUB9rT1Z7sZqjBtAZaISBNwGCs4u2XUORuA24BngeuBJ4wxBkBEPMANwBXOyXbQV2WMaRcRP/A24DeZvpBCM9mgU1N9GQDPH+weWRmViuVzKwF4YvcJ3nh2/ZT3z4EV0L16Qged1PToDF2GdQ6EqS1Nbrml4/Il9Vx3/tyMtOfG1fMZjsbZ0zbAJdPcPwfWDN1AOEYoHEtD65RS+WC8gr25qLpUl1yqzLD3xN0BPA7sAh42xuwUkbtE5Dr7tG8DtSLSAnwSSCxtcAXQ6tTHtAWBx0VkO7ANK1D8nwxfSsE5ueRy7H5qkV2LDkgpIYpj6ZxynPH6VJZbApQX+XWGTk2bztBlWOdAmOokMlzOlBXzKlk6p5zdx/pSrj+XyCl+3jEwTGMgtSULSqn8kk8BXWVxgO7BMMaYtC9jV8oYsxHYOOrYZxMeD2HNwo312t8BF486NgBclPaGukz3JElR5lYVE/B5CEfjKQdiYCWza7L3461NOaCzllxqH6WmQ2foMqxzMExtEhkuZ4qI8OdvOJMV8yo5v7Fq2u9XW2bNPuo+OqXcI58CuuoSP5GYYVBXESjlGr2hCEGfhyL/2MvCvR5hYW0J86qKmV8zvcHoc+dWUh70cW4K++fASi4VixtCEe2jVOp0hi7Dcm2GDmD9ynmsXzk6s3JqnBp2uo9OKffoCUWZV1WU7WYkJbG4+HT3DCul8kP34OSDTndcuSQt3+vT1y7l/ZcuTGn/HFgzdAC9oSglAe2jVGr0f04GRWNxekKRpGrQ5Stnf2CHztAp5Rq9oQjLGsqz3YykOGnLuwcjNFZnuTFKqRnRE4qMu3/Oka5cBfOqiplXVZzy68uLrHb2DUWYU5kfA2Uq9+iSywzqDkUwBmom6VTyWZ09Q9c+oDN0SrlFbx4tuayy29mtpQuUco18WhY+MkOniVHUNCQV0InIOhF5RURaROTOMZ4PishD9vObRGRhwnPnicizIrJTRF4SEdcMP3Q5RcXLCneGrjhgFUBv79MZOqXcIBY39A1H8+ZmaWSGLqR9lFJu0Z1HAV2FHdD1Demgk0rdpAGdiHiBrwPXAsuBm0Vk+ajTPgh0GWMWA18FvmS/1gd8H/iwMeYc4A2Aa/7HdjgBXcnUyhbkm9qyIB06Q6eyQAebZl6vnQ68oig/bpaqR/bQueZPj1KuZ60iyI97r4qRJZc6Q6dSl8wM3RqgxRiz1xgTBh4E1o86Zz1wv/34EeAqsXKvXgNsN8a8CGCM6TDGuCaNz8gM3RTr0OWb2rKA7qFTM04Hm7JjsoK9uabSDuh6BrWPUsotugfDedNHOXvoenWGTk1DMgHdPOBQwtet9rExz7ELbfYAtcBZgBGRx0XkeRH5m+k3OX90uCSgqysL0q5ZLtXM08GmLMi3gC7o81IS8OoeOqVcIhKLMxCOTZoUJVeUjyy51Bk6lbpkArqxqhyaJM/xAZcBt9qf3ykiV532DURuF5FmEWlua2tLokn5wZmhy7WyBelWVxbQOnQqG2ZksKlQ+6dUjQR0eXKzBFZiFF1yqZQ79ObZoFNJwIvXI7qHTk1LMgFdKzA/4etG4Mh459hLmSqBTvv4U8aYdmPMILARuHD0NzDG3GOMWWWMWVVfXz/1q8hRHQNhyoI+gr6xC1sWitrSIJ0Dw8Tjo+N8pTIq44NNULj9U6rybYYOrMQoPZoURSlX6M6zPkpEKAv6dIZOTUsyAd0WYImINIlIALgJ2DDqnA3Abfbj64EnjDEGeBw4T0RK7EDv9cDL6Wl67usaDBf8ckuw9tDFzclOVKkZkvHBJnW6/AzodIZOKbfIx1UE5UUa0KnpmTSgs5cp3YEVnO0CHjbG7BSRu0TkOvu0bwO1ItICfBK4035tF/AVrKBwG/C8G+L9TQAAIABJREFUMeYX6b+M3NQ5EKbaBQGdU4uuQ/fRqZmlg01ZkI8BXXVJgG5NiqKUK/QM5l8fVVHkH1kqqlQqfMmcZIzZiDWCnXjsswmPh4Abxnnt97GyyblO50CY2RWFnwm9tswKWtv6h1kyuzzLrVFuYYyJiogz2OQF7nUGm4BmY8wGrMGm79mDTZ1YQR/GmC4RcQabDLDRTYNN09EbihDweSjy589S8soSvyZFUcolnEGnqjwK6HSGTk1XUgGdSk3nQJilcyqy3YyMOzlDpyPgambpYNPM68mjgr2O6hI/3aEIxhisJKdKqUKVj6sIyov8tHYNZrsZKo8ls4dOpcAYQ+dAeGT2qpDV2stKdcmlUvklHI2z9UAX1irU5ORjQFdVHCAWN/QN6wi4UoXOmY2vyKN+qkJn6NQ0aUCXIYPhGMPRONUlhR/QVZcE8MjJuntKqfzw2I6jvPsbz/C3P9lBNBZP6jX5GNCdLC6uyy6VKnQ9oQhlQR9+b/7c4lpLLrV/UqnLn//teabTDm5qXZAUxeMRakq1uLhS+eZgh7XE54HNB/mz7zYzkMQMVj4GdM7Amu6jU6rwdYfCeddHVRT76RuOavknlTIN6DKkc6SoeOEHdKDFxZXKR0d7h6gpDfBP71zBU6+2cdM9z3Gib2jC1+RjQFdlz9B1aaZLpQpebx72UeVFPoyBgbAuu1Sp0YAuQzrtGwc31KEDKzGK7qFTKr8c6xliTkURt6xdwLduW0XLiX7e9V/P0HKif9zX5GNAV20HdForU6nCl499VHmR1V7dR6dSpQFdhnT2uyugqy0L6B46pfLM0Z4hGiqt0ipXLp3NQx+6mKFIjHd/4xm27O887fxY3NA3FM2rZAMAlcXOkkvto1T6icg6EXlFRFpE5M4xng+KyEP285tEZKF9fKGIhERkm/3x3wmvuUhEXrJf8x+i6VmT1j2YjwGdlXReAzqVKg3oMqTLZTN0taVB2vt0hk6pfHK8d4g5lSdrZZ7XWMVPPnIptWUBbvmf57jz0e20nOgbed7ZtJ9vN0vOkkvdQ6fSTUS8wNeBa4HlwM0isnzUaR8Euowxi4GvAl9KeG6PMWal/fHhhOPfAG4Hltgf6zJ1DYWmJxQZ+Z3PFydn6LSPUqnRgC5DOgbC+DxCRZE7Sv3VlgUYCMcIhWPZbopSKglDkRidA+GRGTrH/JoSHv3wJbxn1Xx+8sJhrv7K7/nAfZt5pqU9L+s7Afi9HsqCPt1DpzJhDdBijNlrjAkDDwLrR52zHrjffvwIcNVEM24i0gBUGGOeNVZNke8C70h/0wtTd14uubTuFXs1oFMp0oAuQ7oGwlSXBlxTxLbeKS4+oLN0SuWD471W8pM5lcWnPVddGuDud67gmTuv5BNXn8VLh3u45VubuPVbm4D8C+jAmqXTsgUqA+YBhxK+brWPjXmOMSYK9AC19nNNIvKCiDwlIpcnnN86yXsiIreLSLOINLe1tU3/SgrAUCRGOBofKVWSLyp0D52aJg3oMqRjIEyNC2rQOZwC6h2a6VKpvHC0xw7oKorGPae2LMjHrl7CHz99JV981wqK/F4AGqtPDwJzXVWJX2foVCaMNWo7Ovf8eOccBRYYYy4APgn8UEQqknxPjDH3GGNWGWNW1dfXT7HZhSlfVxFUjMzQaUCnUuOO9YBZ0DUQds3+ObBu/ACtRadUnjjmBHSV4wd0jiK/l5vWLOA9q+ZzvG+IhjFm9XJddUlAs1yqTGgF5id83QgcGeecVhHxAZVAp72cchjAGLNVRPYAZ9nnN07ynmoMzj7ZfAvodA+dmi6docuQTrcFdKU6Q6dUPjk6hYDO4fFIXgZzYN3gaVIUlQFbgCUi0iQiAeAmYMOoczYAt9mPrweeMMYYEam3k6ogIouwkp/sNcYcBfpE5GJ7r937gJ/NxMXkO2eGrqo4v+6/ivwefB7RJZcqZTpDlyGdg+4K6OqcGTrdQ6dUXjjWE6K8yEdZ0B1/BqpK/Fq2QKWdMSYqIncAjwNe4F5jzE4RuQtoNsZsAL4NfE9EWoBOrKAP4ArgLhGJAjHgw8YYp17InwPfAYqBx+wPNQnndzzfZuhEhPIin87QqZS54y/5DIvG4nQPRqh2UUBXHPBSGvDqDJ1SeSKxBp0bVJcE6AlFiMcNHo87klWpmWGM2QhsHHXsswmPh4Abxnjdo8Cj47xnM3Buelta+EZm6PIsKQpARbGf3pDO0KnUJLXkMtWimQnPLxCRfhH5q/Q0O7c5+zRqXRTQgbWPTvfQqZmkfVPqjvUOjZnhslBVFvuJG80ip1QhcwK6ijyboQN0hk5Ny6QBXRqKZmIfc81ygc4Ba5bKTTN0YGW61Bk6NVO0b5qeoz1DNEyQ4bLQVNtZh7tD2kcpVah6QhE8AuV5uJS8POjXASeVsmRm6KZVNFNE3gHsBXamp8m5zwno3DZDV6czdGpmad+UokgsTnv/8JQSouQ7ZwlWlyZGUapg9YQiVBT783JZtTVDpwGdSk0yAV3KRTNFpBT4NPAPE32DQiuOOTJD56I6dAB1ZQHadYZOzZyM901QeP0TwIm+YYzBVXvoqpwZOk2MolTB6h6M5F1CFEd5kV+XXKqUJRPQTado5j8AXzXG9E/0DQqtOObIDF2ZuwK62tIgnQPDxOOn1T9VKhMy3jdB4fVPYGW4hKmVLMh3zgydli5QqnD1hCJU5WlAV1Hs08LiKmXJLDJOuWgmsBa4XkT+H1AFxEVkyBjztWm3PIc5AV0+ZlmajtqyAHFjJYVxU8kGlTXaN6XIqUGXrzXlUlGtM3RKFbxue8llPiov8tM/HCUWN3jzcMmoyq5kArqRopnAYaz6KbeMOscpmvksCUUzgcudE0Tk80C/G26YOgfClAd9BH3ebDdlRjm16Dr6hzWgUzNB+6YUHXOKirsoKUpFkfXnTvfQKVW4ekMRFtSUZLsZKXGWwB/oGGBRfVmWW6PyzaRLLu19J07RzF3Aw07RTBG5zj7t21j7UlqATwKnpQ93k86BsOsyXMLJJaZtmhhFzQDtm1J3tGeIYr+XiuL8ywSXKp/XQ0WRbyStuVKq8PSEIlTmab+26oxqAJoPdGW5JSofJfW/PtWimaPO/3wK7ctLXYNhV85QnZyh0yVNamZo35SaY3ZRcTvhp2tUlQR0yaVSBcoYYwd0+bnk8sz6MqpK/DTv7+Q9q+ZP/gKlEiRVWFxNTUe/OwM6p0xDh87QKZXTjvaEXJUQxVFV4tcll0oVKGf/WVVxft5/eTzCqjNq2LJfZ+jU1GlAlwFunaGrLgngEegY0BFwpXLZsZ4hlwZ0Abp1yaVSBcnJYJuvM3QAqxdWs699gLY+HRhXU6MBXZoZY+gYCLuuqDhYo0s1pVpcXKlcFosbjvcNu6oGnaOq2K9LLpUqUM7+2Mo8zjC+amENAFsPdGa5JSrfaECXZoPhGOFo3JVJUUCLiyuV69r7h4nFDXNcVLLAUV3i1zp0ShWo3lD+z9CtmFdJ0OfRZZdqyjSgSzOnBp0bl1yClelS99AplbuckgUNLipZ4KgsCdA7FCEWH11/XimV77oLIKAL+DysnF/Flv06Q6emRgO6NBsJ6ErcGdDVlQV1D51SOcwpKu7GPXTVJX6MOTmSr5QqHM6Sy6o8XnIJsHphDTuP9DIwHM12U1Qe0YAuzUYCujJ3BnS1pUHadTOvUjnrWE8IwJ176OwbvS7dR6dUwSmEpCgAqxZWE4sbth3qznZTxmSM4WDHIMboSodcogFdmrl9hq62LMBAOEYoHMt2U5RSYzjaO0TA63HlsvAqu1/WTJdKFZ6eUISA10Ox35vtpkzLRWdU4xHYvC+3ll2GwjEe2nKQt/3nH7niy0/y6Ue3a1CXQ5IqLK6S5/YZujr7ujsGhmkMlGS5NUqp0Y71DDG7Mui6ouJgZbkENNOlUnkgEovj80jSfVVPKEJFsT/v+7byIj9L51TQnCOZLve29fP95w7yyNZD9A5FOWt2GdedP5eHm1spL/Lzd29dlvf/5oVAA7o06xwM4/cK5UF3/tPWlQUBq7h6Y7UGdErlmqM9QzRUuC/DJVi1MgHNdKnSRkTWAf8OeIFvGWO+OOr5IPBd4CKgA7jRGLNfRN4EfBEIAGHgr40xT9iv+R3QAITst7nGGHNiBi4nZxhjuOpfn2L1whr+5YbzkgoYekJhKosL495r9cJqfrS1lUgsjt+bncV0fUMR/vKBF3jylTZ8HmHduXN43+sWsnphNWAl//v2H/dRUeTnY1cvyUob1UmF8T8/h3T2h6kuCbh2tKLWDui0Fp1SuelYzxAr51dluxlZ4eyh04BOpYOIeIGvA28CWoEtIrLBGPNywmkfBLqMMYtF5CbgS8CNQDvwdmPMERE5F3gcmJfwuluNMc0zciE5aF/7AAc7BznYOcjC2hL+4qrJA4aeUGRkWXW+W91Uw/3PHmDX0V7Oa8xOf/0fv32N373axsevXsItaxcwq/zUfdeffdty+oaifPU3r1Je5ONPLmvKSjuVRffQpVnnYNiVe1McTkH1Dq1Fp1TOMcZwrGfIlQlRwFrKJKJLLlXarAFajDF7jTFh4EFg/ahz1gP3248fAa4SETHGvGCMOWIf3wkU2bN5CkYSgqxpquFff/0qP99+ZMLzjTF09IfzPiGKY9UZVoHxbO2j29PWz31P7+c9F83n41efdVowB+DxCF969wrWnTOHu37+Mj9qPpSFliqHBnRp1jng7oDOWXLZPqAzdErlms6BMOFY3JUlCwC8HqGy2K9JUVS6zAMS72JbOXWW7ZRzjDFRoAeoHXXOu4EXjDGJfzjvE5FtIvL34sIlP9sOdVMa8HL/B9aw6oxqPvXwi+NmfRwMR/nUwy+y+1gf58ytmOGWZsacyiLm1xTTnKUC41/4+csU+7389bqzJzzP5/Xw7zev5PIldXz60e38csfRGWqhGk0DujTrGghT7eKArjjgpTTg1Rk6pXKQU4POrTN0AGfUlmbtJkkVnLECrdFp/yY8R0TOwVqG+aGE5281xqwALrc/3jvmNxe5XUSaRaS5ra1tSg3PddsOdbOisZLigJdvvvciZlUE+dP7mzncHTrlvNeO97H+a0/zk22H+cTVZ/Hxq8/KUovTb/XCGpoPdM54Jsknd5/gyVfa+NjVS0YG6ScS9Fk/owsWVPMXD7yQc9k53SKpgE5E1onIKyLSIiJ3jvF8UEQesp/fJCIL7eNvEpGtIvKS/fnK9DY/93QMhEeWHbpVbVlQ99CpGaF909QcGykq7s6kKADvumAeLx/tZcfhnmw3ReW/VmB+wteNwOi1gSPniIgPqAQ67a8bgZ8A7zPG7HFeYIw5bH/uA36ItbTzNMaYe4wxq4wxq+rr69NyQblgKBJj19FeLlhgJd+oLQty722rGY7E+OB3ttBvF9z+yQutXPe1p+kaDPP9D67lY1cvwespnMnM1QtraO8Ps79jcMa+Zzga5x9//jKL6kt53+sWJv26koCPe9+/mrlVxXz60e0MRbR01UybNKBL2PR7LbAcuFlElo86bWTTL/BVrNEmOLnpdwVwG/C9dDU8F0VjcXpCkZFMam5VWxbQGTqVcdo3Td2xXp2hW79yLgGvh0e2tma7KSr/bQGWiEiTiASAm4ANo87ZgNXHAFwPPGGMMSJSBfwC+Iwx5mnnZBHxiUid/dgPvA3YkeHryCk7j/QSiZlTkjctmV3O1269kFeP9/GxB17gMz9+iU889CIrGiv5xV9ezqWL67LY4sxwsklumcEZr+88s4+97QP8/duWE/BNbRFfZbGfu9+xgn3tA/zXky0ZaqEaTzI/Ld30m6QuO3NarUtr0DlqS3WGTs0I7Zum6FjPEF6PJLWMplBVlQS45pzZ/HTbYYajOoqsUmfvibsDK0PlLuBhY8xOEblLRK6zT/s2UCsiLcAnAWclwR3AYuDv7b1y20RkFhAEHheR7cA24DDwPzN3Vdn3wkFrSfQFo7Lxvv6sej5/3Tn8dvcJHth8kI+84Ux++KdrmV1RmANUZ9aXUV3iZ8v+mQno2vqG+Y/ftnDl0lm88exZKb3HZUvqeOcF8/jGU3toOdGX5haqiSRTtmCsTb9rxzvHGBMVEWfTb3vCOWNt+gWsdeDA7QALFixIuvG5psvOnOb2Gbr68gAvto69eVmpNMp43wSF0z+BtYdudnmwoJYlpeI9q+bz8+1H+c3LJ3jreQ3Zbo7KY8aYjcDGUcc+m/B4CLhhjNd9AfjCOG97UTrbmG+2HepmbmURs8YI1N73uoX4vR4aq4u5fEnhLDMdi4iwamENzQdmZs/vlx/fzXA0xt+9ddm03ufv3rqMJ185wd/+eAcP3n4xHpf/vZkpyczQZWrT78kTC2QduLPM0PV76EqDdA6EicdndiOvcp2M901QOP0TwLHekGszXCa6dHEdcyuLeFjTbCuVc7Yd6mblgvFrr928ZkHBB3OO1Qur2dc+wIm+oYx+n+2t3fxoaysfuLSJRfVl03qv2rIgf3vtMjbv7+RHW7WPnSnJBHQZ2fRbiEZm6Nwe0JUFiMWNpgZXmaZ90xQd7RnSgA6rfMH1FzXy+9faODIqa55SKnva+4dp7Qqdsn/OzVYttOrRbc1gZt5Y3PD5DTupLQ3yF1cuTst73rCqkTVNNfzTxt26BWeGJBPQpX3Tb6HqGNAZOrBGZ8Baj61UBmnfNAVOUfE5Fe7NcJno+ovmYwz8+PncS45ypDvEV3/9Kh/6XjNffGw3j25tZXtrNwN2dj+lCtW2g9Z2jZXzq7Pcktxw7txKivwetmQgoDPG8Kudx7j233/P8we7+fS6sykvSk9hdhHhn965gsFwlC/8/OW0vKea2KR76Ox9J86mXy/8//buO77q6m7g+OfcnT0hZBISQBJAwh6ioJVCVcTWAa1V3Ba181Fra7V2WmtbtdX20SruRdUK1kepW0Q2YagEAoGQQEjInnef5497E9kkkuSu7/v1yuv+7s0d5+Tmfu/5/s5iceekX2C91noZvkm/z/on/dbja1jB4ZN+7/Lf9nWtdU1vVyQYNPgTusQIn0NXmB6PUvDKhgruPP/IRQeF6B0Sm3qm2e6m3emJ6BUuD5WTEs3UvBSWrK/kpplDAz7Pw+PVfLi9hhfW7OWD7TVoYHByNO+X1ODyfDmSODMxigm5Sdx1QWFEL24jwtOmikaMBsXozIRAFyUoWEwGirITWV/euwujfLqrlvuXb6d4byN5qTH8/fJxfGPUoF59jaEDY1k0cyh/fa+Ui8dnRcww2UDpzqIofTXpN+zUtzmJs5l6vNRruBk6MJZLx2fx1Kd7+O6UwQxOiQl0kUSYktjUfV/uQScJXafLJmbx45c3s3ZPPVPyUgJShvo2J8+uKufldXvZ32RnQJyVRTPzWTAxh+zkaFweL+V17eysaaG0upUdNa28/dkB1u2u5x/fHc8YGZomwkhxRQMjBsURZTEGuihBY2JuMn//cBd1rY6uEVBf1dbKJv64vIQVpbWkJ9i47+LRXDwuC5Oxb9qtN83M5z+b9/OL1z9j+Y/OwmaW97WvRHbm0cvq25wkR/hwy07/8/XTMBsN3Pt/JYEuihACqGryzRWTHrovzRmZTpzVFLDFUXZUtzD3b5/wwLs7yB8Yyz8uH8end5zDbbNHkJ0cDYDZaGDowFjmjErn+18bxt++PZZXF01DKcWlj65iyTpZdECEB69Xs6WiSebPHWFeUQYAf3u/53u71bY6eGPzfn722lZm3P8Bcx/+hM/2NflWorx1JvMn5vRZMgdgMxv57TdHUV7Xzu2vbMHt8fbZa0W6bvXQie6RhO5LafE2vjcjn7+8s4M1ZXVMDtDZbyGEj/TQHS3KYmRuUQavbazkVxeO7LX5I92xcmct33tuAzazkaU3n9GjnrZRmQm88f3p/ODFYm5/dQubKhv55dxCrCY5+y1C166DrbQ43JLQHWHowDgWTMzmudXlXDl18ElXoWzqcPHw+6WsKK2l5IBvL7g4q4nJeclcNS2XS8Zn9Wusm5afyk/njOC+t0vweDUPLijC3IdJZKSSv+gpanW4+c+W/Xz/xWLW7aknJUbmNHS6/sw80hNs/PbNbbKFgRABdqDZjlIwME4SukNdOj4Lu8vLf7ZU9dtrLllXwcLFa8lIiOL1HiZznZJjLDx19URunJHHC2v2suCx1VQ39+3S5kL0peIK34IoY0+wZUGk+vGs4djMRu5968SjnrTW3PavzSxeuYfUWCu3zT6N128+g+K7Z/H4wolcfcaQfk3mOi2amc8vzi/gza1V3PT8RhxuT7+XIdxJQvcVNLY7eXndXq55ah3jfvMOt7xQzKc7a7moKJNbZw8PdPGCRpTFyO1zTmPrvib+Xbwv0MURIqIdaLKTGmuN+Dm+RyrKTmTYwNh+GXbp9WruX17C7a9uYWp+Cv9aNJXMxK++6qjJaOBn3yjgke+MY/uBFs7/6wre2tp/iakQvWlTRSNxNhN5qae2D1o4So31za1954tqVpfVHfd+L66t4L9fVHPHnBE8d91kbj57KEXZiX06rLK7rjszj1/PG8k7X1Rz47MbsLskqetNgX+HQ0xFfTuzHviYn766lR3VLVwxZTBLbpzK2jvP5b5LTmfEoPhAFzGozBuTyZisBO5fvp12pyy5LUSgVDXZZf7cMSiluGxCNsV7Gymtbumz12lzuPnhy5t45INdLJiYzeKrJhLfS2fKzz89nddvPoO0eBuLnt/Iouc29PlGxEL0tk17GxmTlRjwFWeD1bXTh5CeYON3xxn1tLOmlV//53POHJbKtdOHBKCEJ3fl1Fzu/dZoPtpxkOueXi/twl4kc+h6oL7NycLFa3G6vby6aCrjcpJQSgLPiRgMil9cUMil/7uKxz4u40fnBncPpsPtYVdNGzuqW0iNtTIhN0lWZRJh4UCTncEp0YEuRlC6aGwm971dwpOf7uH33xzdrcfsrGlhdVk9TrcXp8fru3R7cXm8tDjc1LY4qGtzUtvqoK7VSat/D7nb55zGohn5vf7dMTwtjtdvPoN/rijjwXdL+XRXHb84v4BLxmfJ95QIeu1ONyUHmrn57N7Z2Doc2cxGbpt9Gj9Zspllm/dz0djMrt853B5+8GIx0RYTf750TFAnxd+elIPZaOD2VzZz1ZPreGLhhIAMAw03ktB1k93l4bqn11HZ2MEL101m/ODkQBcpZEzMTea80YN49KMyFkzMCZpFGTxezZqyOjZVNlJS1ULJgWbKDrbhPuTMl9VkYGJuMmcMTeXMYakUpscHdaAU4niqmjqYkidx61gGxFn5zuQcnllVTl5qDNedmXfC+6/bU8/CxWtpdx4+ZEgpsBgNxFhNpMZaSI21cnpWYtfx2JxEpuWn9lk9zEYDN80cyuyRg7jj1S3c9soWlm3ez++/Obpr1UwhgtHWyia8GlkQ5SQuKsrkyZV7+OPbJcwZNajrhPOflm/ni6pm/nnlBAbGB0cb60QuGZ+FxWTgxy9vYub9H3LDWXlcMXUw0RZJS74q+ct1g8er+cGLxRRXNPKPy8cxIVcaRT11x5wC3v2ihj/9dzt/unRMQMtSWt3Cqxv38XrxPg74FxHITIxixKA4ZhWmcdqgeIanxVLVaGdFaS0rd9Zy39sl3Pc2JEWbOfu0gcwtymD60FRZqUmEhDaHm2a7m7QgOZkSjO6+oJCDLQ5+++Y2Yq0mFkzKOeb91u6u56on1zIowcY/r5xASowFi8mA2WjAZFBB0RuWPyCWl2+YyvNryvnDWyWc+5ePuHBMBt+dMlj2rRNBaZN/QRRJ6E7MYFD8/LwCvv3P1TzxyW5uPnsoK0oP8s8Vu7liymBmFaYFuojdduGYDLKTovjLOzu4960SHvu4TBK7UyB/sZPQWvOrNz7nv19Uc8/cQuaMSg90kUJSTko0V5+Ry2MrypgxfABzx2T06+vXtzl5Y/N+Xt1YyZbKJowGxczhA7jrgkKmD0slIero7v4Rg+I5e8RAAGqa7azcVcuKHbW8s62a14r3kRRt5huj07lwTAaTcpOl504Erc4TFzKH7vhMRgMPLiii/ZkN/OzfW4mxmo6KU4cmcy9dPyWoz4QbDIorpuZyTkEaD7+/k6Wb9vGvDZWMzkzg8sk5XFiUIY0mETQ2VTSSnRx1yhtnR4Kp+SmcW5DGPz7cxdcL0/jJks0MGxjLnecXBLpoPTY2J4lnr53MhvJ6Hny39LDEbv7EbBKjZSuw7lJaB9dy8hMmTNDr168PdDG6/O9Hu/jDWyXccFYePz8v9D4swaTF7uLqJ9exvryBW84eyk9mDT9hElTV1MFzq8sZFG9jQm4yw9PiMPYgadJas6mikWdWlfPmliqcHi+F6fF8a1wm84oyGRD31b44HG4PH++oZdnm/bz7RTUdLg9p8VZmjxzEzNMGMCUvRRpKx6CU2qC1nhDocpyKYItP3bVyZy2XP76GF6+fwtR82RPyRDqcHhYuXsvGvQ08duV4zhnhO+O9pqyOq59aR3qCjReDPJk7lma7i6XF+3hu9V62V7cQZzNxUVEm54wYyKQhycRYIzdmSWwKvKn3vseE3GT+9u2xgS5KSNh1sJXZD3yMxWTA7dEsveUMCtJDf1G+zsRuRWktACMGxTF5SDKThqQwaUjyV263haqexCZJ6E5g6aZ9/PClTcwdk8FD84ukB6YXONwe7n79c15eX8GswjQemF9E7BENCafby+KVu/nre6WHzVGJs5kYPziJibnJjB+cxJDUGFJjrUcleXaXh2Wb9/PsqnK27msixmLk4vFZLJiYQ2FG7wa8dqebd7fV8Mbm/awoPYjd5cViNDBpSDIzhg/grOEDGJ4WGxTDsAJNGk197+MdB3l90z7aHR7sbg8dTg92t5faFgf7Gjv44NaZDEmNCXQxg16L3cXlj69h+4EWnrp6EgbFl8ncDVNCei8/rTUbyht4bnU5//fGUIXyAAAQe0lEQVTZAZxuLyaDoig7kWn5KUwbmsrYnMSI2qQ8kmJTh9PDE5+UceOM/KCZMnCgyc6Ue9/jrgsKg3Z1xmD0y6Wf8fSqcu6+oJBrwuzvtrWyiY921LBmdz0byhu62oL5A2IozEggOymK7ORospOiyUmOJj3RFjT/z71JErpesHTTPn6yZDMTc5N4+ppJEfXl1te01jz96R5+8+Y28gfE8PiVE8nxr763cmctdy/9jF0H25hVmMbdFxQCvkUI1pc3sH5PPTuqW7uey6B8CxqkxdtIi7cRZzPxfkkNje0uhg6MZeHUwXxzXNZRSWNfsLs8rN/TwEc7avhox8GucqbGWhibk8S4nCTG5SRyelYiUZbI+3+KpEaT3eXBbDT0qEf5VOxr7OA3b3zB258fIDnGQmqshSizEavZSJT/Jyspip+fVyAnprqpoc3JZY+uYn9jBxrCIpk7kt3lYUN5Ayt31rJyVx1bKxvxarCYDAxJiSE3NZohqbEM8V/mpkYzINYadieoIik2vbW1ikXPb+T80ek8tKAoKPYne/uzKr733EZeXTSN8YOTAl2ckGF3eVi1q44ZwweEdVx3ebx8tq+JtbvrWbu7np0HW9nX0HHYAnYGBekJUWQnR5GdFE1WUrTvODmaQfE2UmOtIdnukoTuFC1ZV8FPX9vCpNxknrhqYr8kA5Hok9Jabn5hI0rBb+aN4u3PD/DmlipykqO558LCrqFOR2psd1Jc0ci+hg5qmu0caLZT3eygutlObauDibnJXDF1MFPzUgLa8Khq6uDjHQdZs7ue4r2N7K5tA8BoUBSkxzEqI4Hs5GiykqK6gk84NpY6RVKj6dZ/baaqqYOHFowltQ/nhDjcHh5fsZu/vV8KwPfPGcZ1Zw6RE1C9pLrZzvxHV2EyGnjh+slhlcwdS1OHizVldawvb6DsYBt76toor2vD5fmynRBlNnY1mrKT/T9JUaTGWUmOtpAUYyHeZgqpOBZJsQng8RVl/PbNbVw4JoMH5hf124mn47n3rW0s/mQ3W++ZLdsEiW5xe7wcaLZTUd9BRX07FQ3t/kvf9ZoWx1GPibEYSYm1khJrISXGSmqsheQYCymxvuOUGN/vkqItxNpMRJuNAU+UIyKh83o19y0vYf6EbPIGxPba6z/96R5+ucy3MeNjV0wIyYw+lOypbeP6Z9ZTWtOK1eRbcvvGGXlhGdTr25wU721g494GNpY3UlrTQm2r87D72MwGMhOjyEiMIiPBd5meaCMjIYqB8VYSoswkRJmxmgwh1WCCyGo0LVlXwV1LPyMp2sIjl4/tk21OPtpxkHuWfc7u2jbmjBzEXXMLyUyM6vXXiXR2lweDUlhMge/JCASPV7O/sYOy2jb21LZRUd/OXv9PRX07bUds3QBgMiiSYiwkR1uIjzIRZzMTZzMR77+Ms5mJsRqxHdKDHG0xYrMYibOaiI8yE28zYzP3T5yLpNjUqXN9gG+NzeT+S8cENKmb/+gq7C4PS2+ZHrAyiPBid3nY1+hP7pod1Lb59gOta/XtD3rQv09ofZsTzzE2aQffNjQxFhOxVhOxNhMJUeauETCdyV9KrJWkaDMxVt/9YqwmYi0mYqxGTEYDWms8Xo3be/hlckz3FnvpSWzqVteTUmoO8BBgBB7XWv/hiN9bgWeA8UAdMF9rvcf/u58B1wIe4Ada6+XdqsVJ7DzYyrOrynl8xW4um5DNj84dRtopTlLvDHCzCtN4+Dtj5Sx3P8hNjeG1m6bx0toKZo8c1DX0Mhwlx1j4WkEaXyv4suex3emm0n9G6dDLqqYOSg60cPAYZ5nAt9dVfNSXDSSb2UiU5cvGkc1ixGYyYjEZfD9G5b80YPU3nqL8j/Edm7CZfUMEDUqhFBiU6vqSd7q9ONwe7C4vDv+x0+1lQm5yQHuwgzE2AVw2MZuRmfHc9PxG5j+6mp+dV8A1Z+SecuPU49W8u62aJ1fuZnVZPUNSY3j6mknMGD6gl0oujhSOJ5d6wmhQXT1xR/6faa1paHdR2dDuaxy1Omlo9zWSGtqd1LU6abG7qWmxs7PGTYvdRYvdfdhQqRMxGxXxNjPx/pNYnTHMbDR0bRXh1borHjncXhwu30bv549O59bZp/XFn+QofRGHTvacp+p7M/Jxe7z86b87MBgUf7z49D7pjWh1uFlTVscnO2tZU1ZPU4cLp8eL0+3F5b90ezULpw7u9dcWkctmNpI/IJb8k3T4eL2aZruLWn+y54tdLtocblocblrtblodLlodbhrbXVTUt1O8t5H6NgcnC2NGgzpuslj2+/N6/fN20paYUsoIPALMAiqBdUqpZVrrLw6527VAg9Z6qFJqAXAfMF8pVQgsAEYCGcC7SqnhWuujT+n10PC0OD667Wwefr+UF9bu5d/FlVxzxhBunJF/zCXoT0RrzYPvlvLQe6XMHZPBXy4bE5aTK4NVnM3M9WedeCPfcBVtMTE8LY7haXHH/L3D7aG6ycH+pg4Otjhotrto7nDTbHfR1OGiucPXQLK7PDS0Odnv8tDh8tDh9GJ3+Ro5To+3z8r/5g+mMzIjoc+e/0SCNTZ1GpmRwLJbpnPbvzbzm/98wYbyeu67+HTibD2LTwBN7S6WrK/g6VV7qGzoICPBxp3nFXDltMFy4kkEjFKK5BhLt882g+/71u7y0u50+2OVL2a1O33HLQ43zR0umv3Jn+/YF+NchyQC7R2ergVdLCYDVpOBWKvJf2wkp582Uu+LOOR/zMme85Tdcs4wXB7NQ++VYjYqfnfR6FNuZDa0Odle3cLqsjo+Ka1lU0Ujbq/GajIwITeJgvR4LCZ1WGJuNRm5ZEJWL9VKiO4zGBSJ0RYSoy0MHdj90X5er6apw0Vdm4OGdl/C1+b/aXV4aHO4cbg9mAy+/UmNRoXRf5LcZFD0xdjI7pxanwTs1FqXASilXgLmAYcGlnnAPf7jV4CHle9U9DzgJa21A9itlNrpf75VvVH4AXFWfjVvFNdOz+PP72zn7x/u4oW1e1k0I58x2YkoX3n9vQ0ACrvLc9gfvs3poaSqmdc37eeyCVnc+63TAz6eXIhOVpORnJToU+q51Fp3nRHtPIvd2ZBqd3YmgG7sLi9erfFqfJde37FGYzUZsfobTZ1fwFazIdArJgZtbOqUEGXm0SvG89jHZfxx+XZKqlbyiwsKiLGYumKTL06BV3PYe9T5nhXvbeC1jfvocHmYlJvMnecVMKswLSgWMxCip5RSvtEE4TOdoS/iEN14zl7xo3OH4fZ6eeSDXQDMHZPhG6WBr7Hb2Xby+oeOeb0aj//Y6fayt76dXQdb2VXTxs6DrdS3+aYRKAWnZyZww1l5TB+ayrjBSRHf2y3Ch8E/rDypByez+lp3ErpMoOKQ65XA5OPdR2vtVko1ASn+21cf8djMr1za48hJieahBWO5/sw8/rh8O/e+VdKjxxsNimunD+FOWQFOhCGllD8hC7sv06CPTeD7+984I5+i7ERuebGYa57q2aJPFqOBC4syuGpaLqMyA9MbKoQ4rr6KQyd7zl6hlOLWr5+G26t59KMyXlxbcfIHHSE5xkL+gBhmj0zrGuY2NidRNoUWoh91J6E7VoZzZG/h8e7TnceilLoBuAEgJyenG0U6tlGZCTxzzSS2VTXT0OZEA9rfw6D9vQ5RZuPhkxetpn6beC2E6FV9Hpug9+LT5LwU3v3xDD7f33RUbOosUNc8oc7eUKORpBjzVxqmKYToF30Rh47V/d5nbSelFHfMGcHc0zNosbvRWqPxtZk6205Gg2/ImMGguuZam42KrKToHg25FUL0je4kdJVA9iHXs4D9x7lPpVLKBCQA9d18LFrrx4DHwLdSU3cLfzwF6b27ebQQIij1eWyC3o1PCdFmpg1NPZWnEEIEl76KQ/3adlJKyQgAIUJYdyZhrAOGKaWGKKUs+CbwLjviPsuAhf7jS4D3tW8/hGXAAqWUVSk1BBgGrO2dogshIpzEJiFEoPVFHOrOcwohRJeT9tD5x3vfAizHt3zuYq3150qpXwPrtdbLgCeAZ/0TeuvxBR/891uCbyKvG7i5N1eRE0JELolNQohA66s4dKzn7O+6CSFCR8huLC6ECC2RuHmvECL4SWwSQgSjnsQmWfdaCCGEEEIIIUKUJHRCCCGEEEIIEaKCbsilUuogUN6Dh6QCtX1UnGAUafWFyKtzuNZ3sNZ6QKALcSp6GJ/C9X08kUirc6TVF8KzzhKbwl+k1TnS6gvhWedux6agS+h6Sim1PtTHvvdEpNUXIq/OkVbfcBWJ72Ok1TnS6guRWedwE4nvYaTVOdLqC5FZ50PJkEshhBBCCCGECFGS0AkhhBBCCCFEiAqHhO6xQBegn0VafSHy6hxp9Q1Xkfg+RlqdI62+EJl1DjeR+B5GWp0jrb4QmXXuEvJz6IQQQgghhBAiUoVDD50QQgghhBBCRKSQTeiUUnOUUtuVUjuVUncEujx9QSm1WClVo5T67JDbkpVS7yilSv2XSYEsY29SSmUrpT5QSm1TSn2ulPqh//ZwrrNNKbVWKbXZX+df+W8fopRa46/zy0opS6DLKrov3OOTxCaJTRKbQlO4xyaQ+BTu8Uli07GFZEKnlDICjwDfAAqBbyulCgNbqj7xFDDniNvuAN7TWg8D3vNfDxdu4H+01gXAFOBm//saznV2AOdorccARcAcpdQU4D7gAX+dG4BrA1hG0QMREp+eQmKTxCaJTSElQmITSHwK9/gksekYQjKhAyYBO7XWZVprJ/ASMC/AZep1WuuPgfojbp4HPO0/fhq4qF8L1Ye01lVa643+4xZgG5BJeNdZa61b/VfN/h8NnAO84r89rOocAcI+Pklsktjkvz2s6hwBwj42gcQnwjw+SWw6tlBN6DKBikOuV/pviwRpWusq8H2IgYEBLk+fUErlAmOBNYR5nZVSRqXUJqAGeAfYBTRqrd3+u0TS/3c4iNT4FNaf004SmyQ2hbBIjU0Q5p/VTpESnyQ2HS1UEzp1jNtkuc4woZSKBV4FfqS1bg50efqa1tqjtS4CsvCdQS041t36t1TiFEh8ClMSmyQ2hTiJTWEskuKTxKajhWpCVwlkH3I9C9gfoLL0t2qlVDqA/7ImwOXpVUopM76A9LzW+jX/zWFd505a60bgQ3xj4BOVUib/ryLp/zscRGp8CuvPqcQmiU1hIFJjE4T5ZzVS45PEpi+FakK3DhjmX9HGAiwAlgW4TP1lGbDQf7wQWBrAsvQqpZQCngC2aa3/csivwrnOA5RSif7jKOBcfOPfPwAu8d8trOocASI1PoXz51Rik8SmcBCpsQnC+7MaUfFJYtOxhezG4kqp84AHASOwWGv9uwAXqdcppV4EZgKpQDXwS+B1YAmQA+wFLtVaHzn5NyQppaYDK4CtgNd/88/xjQUP1zqfjm/yrhHfCZYlWutfK6Xy8E1YTwaKge9qrR2BK6noiXCPTxKbAIlNEptCULjHJpD45L85bOOTxKZjC9mETgghhBBCCCEiXagOuRRCCCGEEEKIiCcJnRBCCCGEEEKEKEnohBBCCCGEECJESUInhBBCCCGEECFKEjohhBBCCCGECFGS0AkhhBBCCCFEiJKETgghhBBCCCFClCR0QgghhBBCCBGi/h/TEFoeyQrrRAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig, axes = plt.subplots(3, 3, figsize=(15, 10))\n", "for i, ax in enumerate(axes.flat):\n", " ax.plot(attn[i])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## All (seq2seq + bi-directional + attention)" ] }, { "cell_type": "code", "execution_count": 249, "metadata": {}, "outputs": [], "source": [ "class Seq2SeqRNN_All(nn.Module):\n", " def __init__(self, vecs_enc, itos_enc, em_sz_enc, vecs_dec, itos_dec, em_sz_dec, nh, out_sl, nl=2):\n", " super().__init__()\n", " self.emb_enc = create_emb(vecs_enc, itos_enc, em_sz_enc)\n", " self.nl, self.nh, self.out_sl = nl, nh, out_sl\n", " self.gru_enc = nn.GRU(em_sz_enc, nh, num_layers=nl, dropout=0.25, bidirectional=True)\n", " self.out_enc = nn.Linear(nh * 2, em_sz_dec, bias=False)\n", " self.drop_enc = nn.Dropout(0.25)\n", " self.emb_dec = create_emb(vecs_dec, itos_dec, em_sz_dec)\n", " self.gru_dec = nn.GRU(em_sz_dec, em_sz_dec, num_layers=nl, dropout=0.1)\n", " self.emb_enc_drop = nn.Dropout(0.15)\n", " self.out_drop = nn.Dropout(0.35)\n", " self.out = nn.Linear(em_sz_dec, len(itos_dec))\n", " self.out.weight.data = self.emb_dec.weight.data\n", "\n", " self.W1 = rand_p(nh * 2, em_sz_dec)\n", " self.l2 = nn.Linear(em_sz_dec, em_sz_dec)\n", " self.l3 = nn.Linear(em_sz_dec + nh * 2, em_sz_dec)\n", " self.V = rand_p(em_sz_dec)\n", "\n", " def forward(self, inp, y=None):\n", " sl, bs = inp.size()\n", " h = self.initHidden(bs)\n", " emb = self.emb_enc_drop(self.emb_enc(inp))\n", " enc_out, h = self.gru_enc(emb, h)\n", " h = h.view(2, 2, bs, -1).permute(0, 2, 1, 3).contiguous().view(2, bs, -1)\n", " h = self.out_enc(self.drop_enc(h))\n", "\n", " dec_inp = V(torch.zeros(bs).long())\n", " res, attns = [], []\n", " w1e = enc_out @ self.W1\n", " for i in range(self.out_sl):\n", " w2h = self.l2(h[-1])\n", " u = F.tanh(w1e + w2h)\n", " a = F.softmax(u @ self.V, 0)\n", " attns.append(a)\n", " Xa = (a.unsqueeze(2) * enc_out).sum(0)\n", " emb = self.emb_dec(dec_inp)\n", " wgt_enc = self.l3(torch.cat([emb, Xa], 1))\n", " \n", " outp, h = self.gru_dec(wgt_enc.unsqueeze(0), h)\n", " outp = self.out(self.out_drop(outp[0]))\n", " res.append(outp)\n", " dec_inp = V(outp.data.max(1)[1])\n", " if (dec_inp == 1).all():\n", " break\n", " if (y is not None) and (random.random() < self.pr_force):\n", " if i >= len(y):\n", " break\n", " dec_inp = y[i]\n", " return torch.stack(res)\n", "\n", " def initHidden(self, bs):\n", " return V(torch.zeros(self.nl * 2, bs, self.nh))" ] }, { "cell_type": "code", "execution_count": 250, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n", "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n" ] } ], "source": [ "rnn = Seq2SeqRNN_All(fr_vecd, fr_itos, dim_fr_vec, en_vecd, en_itos, dim_en_vec, nh, enlen_90)\n", "learn = RNN_Learner(md, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n", "learn.crit = seq2seq_loss" ] }, { "cell_type": "code", "execution_count": 251, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d5b77ca806c5432a92631c29ed5642b4", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=15), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss \n", " 0 3.848361 13.931393 \n", " 1 3.392174 6.029147 \n", " 2 3.092935 4.806828 \n", " 3 2.786907 4.489991 \n", " 4 2.630582 4.586274 \n", " 5 2.865972 4.124985 \n", " 6 2.795535 3.689954 \n", " 7 2.744906 3.453802 \n", " 8 2.713428 3.494858 \n", " 9 2.882232 3.303313 \n", " 10 2.613028 3.27942 \n", " 11 2.597443 3.218975 \n", " 13 2.469049 3.22439 \n", " 14 2.278426 3.235787 \n" ] }, { "data": { "text/plain": [ "[array([3.23579])]" ] }, "execution_count": 251, "metadata": {}, "output_type": "execute_result" } ], "source": [ "learn.fit(lr, 1, cycle_len=15, use_clr=(20, 10), stepper=Seq2SeqStepper)" ] }, { "cell_type": "code", "execution_count": 252, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4lFXax/HvnU5ooQSkKKEpIAhipKigIirt1V1XXexbFLe4a10F21rW7rquu76rrOXdVeyK64o0wYINDSi9I0iTBJAaIO28f8xkMsnMJJOQyRR+n+vKlafNzH1wvHOe85xizjlERCR+JEU7ABERqR0lbhGROKPELSISZ5S4RUTijBK3iEicUeIWEYkzStwiInFGiVtEJM4ocYuIxJmUSLxp69atXU5OTiTeWkQkIc2bN2+bcy47nGsjkrhzcnLIy8uLxFuLiCQkM1sf7rVqKhERiTNK3CIicUaJW0Qkzihxi4jEmbASt5lda2aLzWyJmV0X6aBERCS0GhO3mfUGrgIGAH2BMWbWPdKBiYhIcOHUuHsCXzjnCp1zJcBHwI8jG5aIiIQSTuJeDAw1s1ZmlgmMAo6sepGZjTOzPDPLKygoqFMwT8xaxUcr6/ZaEZHDRY2J2zm3DHgImAlMAxYAJUGum+icy3XO5WZnhzX4J8BTH61hjhK3iEi1wno46Zx71jnX3zk3FNgBrIpEMGkpSRSVlkXirUVEEkZYQ97NrI1zLt/MjgLOAwZHIpj0lCQOFitxi4hUJ9y5St40s1ZAMfBb59wPkQgmPSVZNW4RkRqElbidc0MiHQh4mkoOlpQ2xEeJiMStmBo5qaYSEZGaxVziVlOJiEj1Yipxp6nGLSJSo5hK3OkpyRxUjVtEpFoxlbg9NW49nBQRqU5MJe70lCSKSlTjFhGpTowl7mQOKnGLiFQrphK3px+3EreISHViKnF7mkrUxi0iUp2YS9yqcYuIVC8mE7dzLtqhiIjErNhK3KnJABSXKnGLiIQSU4k7LdkTjiaaEhEJLaYSd3pqeeJWO7eISCixlbhTPOFoEI6ISGgxlbjTUlTjFhGpSUwl7vQUz8NJ1bhFREKLqcSth5MiIjWLqcSth5MiIjWLrcStphIRkRqFlbjN7HozW2Jmi83sZTPLiEQwFQ8n1VQiIhJKjYnbzDoAvwdynXO9gWRgbCSCUXdAEZGahdtUkgI0MrMUIBPYHIlg0tUdUESkRjUmbufcJuBR4DtgC7DLOTej6nVmNs7M8swsr6CgoE7B+JpKtGCwiEhI4TSVtADOBToD7YHGZnZp1euccxOdc7nOudzs7Ow6BVP+cFILBouIhBZOU8lw4FvnXIFzrhh4CzgpEsFU1Lj1cFJEJJRwEvd3wCAzyzQzA84AlkUiGLVxi4jULJw27rnAG8B8YJH3NRMjEYx6lYiI1CwlnIucc38E/hjhWDAz0pK1fJmISHViauQklC8YrMQtIhJKzCXutJQkjZwUEalGzCVurfQuIlK92EvcqclqKhERqUbMJW7Pw0k1lYiIhBJziTs9NYkDGvIuIhJSzCXulo3T2LGvKNphiIjErJhL3I3TUygsKol2GCIiMSv2EndaMoVFauMWEQkl5hJ3ZloK+w6qxi0iEkoMJu5k9mt2QBGRkGIuce/aX0xxqeOAkreISFAxl7hLyxwA67bvi3IkIiKxKeYS96g+7QDYc0Dt3CIiwcRc4s7KTAVgZ2FxlCMREYlNMZe4W2SmAbCzUINwRESCibnE3dxb4961XzVuEZFgYi5xN01PITnJ1FQiIhJCzCVuM6N5o1R+UFOJiEhQMZe4AbIapbJTTSUiIkHVmLjN7Bgz+8bvZ7eZXRfJoLIyU9mlphIRkaBqXOXdObcC6AdgZsnAJmByJIPKykwjf8+BSH6EiEjcqm1TyRnAGufc+kgEUy6rUaoeToqIhFDbxD0WeDkSgfhrnqnELSISStiJ28zSgHOA10OcH2dmeWaWV1BQcEhBtchMY+/BEopLtYSZiEhVtalxjwTmO+e2BjvpnJvonMt1zuVmZ2cfUlBZGoQjIhJSbRL3RTRAMwlA80aar0REJJSwEreZZQJnAm9FNhyPLO98Jdv2HmyIjxMRiSthJW7nXKFzrpVzblekAwJoluHppbh4U4N8nIhIXInJkZM9jmgGwJ+mLItyJCIisScmE3ejtGQAspumRzkSEZHYE5OJG+CMHm1o3USJW0SkqphN3B1bNGLjD4XRDkNEJObEcOLOZM+BEvXlFhGpImYTd5tmnmaSAk02JSJSScwm7ibpni6Bew+WRjkSEZHYErOJu7E3cRceLIlyJCIisSVmE3dFjVuJW0TEX8wm7kxvX+59RUrcIiL+YjZxN83wTDS1YIOGvYuI+IvZxN26iWeiqZlLg84iKyJy2IrZxG1mNElPoW0zjZ4UEfEXs4kbYHDXVuxTd0ARkUpiOnHntMpk7ba9lJW5aIciIhIzYjpxt22WQXGp07B3ERE/MZ249xzwdAVcsHFnlCMREYkdMZ24z+zVFoD9RWrnFhEpF9OJu3w+7jUFe6MciYhI7IjpxJ2Z7hk9+eiMlVGOREQkdoS7ynuWmb1hZsvNbJmZDY50YACN01Ia4mNEROJKuJnxr8A059z5ZpYGZEYwJp/kJKNbmyZs33uwIT5ORCQu1Ji4zawZMBT4GYBzrggoimxYFVbne9q39xeV+hYRFhE5nIXTVNIFKACeN7OvzewZM2sc4bh8jmrpqdyv37GvoT5SRCSmhZO4U4D+wD+cc8cD+4DxVS8ys3FmlmdmeQUFBfUW4DXDugEw4vE59faeIiLxLJzEvRHY6Jyb691/A08ir8Q5N9E5l+ucy83Ozq63AEf3aQfASV1b1dt7iojEsxoTt3Pue2CDmR3jPXQGsDSiUflpnJ7CcR2bk5oc0z0XRUQaTLi9Sn4HTPL2KFkL/DxyIQXKbpLONxs07F1EBMJM3M65b4DcCMcS0qzl+QAU7DlIdlPNzy0ih7e4aH/4Ub/2AHy3ozDKkYiIRF9cJO6rT+0KwPe7DkQ5EhGR6IuLxF0+2dT2fRpBKSISF4m7RaZnxfcV3++JciQiItEXF4k7JTmJFpmp7D1YEu1QRESiLi4SN0Dn1o0p2KOmEhGRuEncR7bMZE3BXpzTwsEicniLm8Tdt2MWW3cfZOVWrYYjIoe3uEnc3do0AeCNeRuiHImISHTFTeI+MaclAM0bpUY5EhGR6IqbxF2+iILWnxSRw13cJG4REfGIy8StniUicjiLq8R99aldANhXVBrlSEREoieuEvfRbZoCaNV3ETmsxVXiLu9RsumH/VGOREQkeuIqcZd527bf+npTlCMREYmeuErcZ/ZqC8CcVfW3iryISLyJq8RtZgBs3a02bhE5fMVV4ga4aMBR0Q5BRCSqwkrcZrbOzBaZ2TdmlhfpoKozdfEWAF77SnOWiMjhqTY17tOdc/2cc1Fb7R3gisE5AHy5bkc0wxARiZq4ayq5/syjaZGZSnpK3IUuIlIvws1+DphhZvPMbFwkAwpHm6YZbN6pvtwicnhKCfO6k51zm82sDTDTzJY75z72v8Cb0McBHHVUZB8grti6hxVbtXCwiByewqpxO+c2e3/nA5OBAUGumeicy3XO5WZnZ9dvlFV0ad0YgMdmaopXETn81Ji4zayxmTUt3wbOAhZHOrDq3H9eHwCemLUqmmGIiERFOE0lbYHJ3sEvKcBLzrlpEY2qBoO6tPJt7ywsIiszLYrRiIg0rBpr3M65tc65vt6fY51z9zVEYDW5bFAnAFbla/FgETm8xG2fuvIRlEs3745yJCIiDStuE3dO60wA/vjOkihHIiLSsOI2cWemVTTPHyjWijgicviI28QN0CGrEQCXPTs3ypGIiDScuE7cvz29GwBfrfuB9dv3RTkaEZGGEdeJ++KBFSM0/zZ7dRQjERFpOHGduAF+fHwHAN6YtzHKkYiINIy4T9x/+Wm/aIcgItKg4j5xA1x7RncAvli7PcqRiIhEXkIk7jHHtQNg7MQv+HabHlKKSGJLiMTdrU0T3/bpj34YvUBERBpAQiRuM+OJi46PdhgiIg0iIRI3wDl923NWr7aARlKKSGJLmMQNkOSZepYed0R11lkRkYhKqMQ9ss8Rvu2S0rIoRiIiEjkJlbjP7deBtGRPkSbN/S7K0YiIREZCJW6AV64eBGi6VxFJXAmXuP27BoqIJKKES9zNMlJJTrJohyEiEjHhLBYcd7q3acLy7/dwoLiUFd/v4eY3FrJi6x7f+SV3n03j9IQsuogcBsKucZtZspl9bWbvRjKg+vDzk3MAz9wl5z75aaWkDfDPOWujEJWISP2oTVPJtcCySAVSn8pr0z97/qug5x9/f1VDhiMiUq/CStxm1hEYDTwT2XDqx/FHtQg4dsXgTnxw02m+/cKikgaMSESk/oRb434cuBmIi1Et7ZtnVNr/fMIw7j63N51bN6Z5o1QAet05nR/2FUUjPBGRQ1Jj4jazMUC+c25eDdeNM7M8M8srKCiotwDrwqyiV8mfftSbds0b+fYnXTnQt/3yV5UH6ezaX0zO+Cmszq/cJi4iEkvCqXGfDJxjZuuAV4BhZvZi1YuccxOdc7nOudzs7Ox6DrPuMlKTK+379/N+eNoKpi/5HgDnHOf/4zMAhj/2ccMFKCJSSzUmbufcBOdcR+dcDjAWmO2cuzTikR2i8pkCU6r06c5ITWbRXWf59q9+YR7n/+MzbnhtAavy9/qOl5W5hglURKSWEm4ATrn2WZ7mkYI9BwPONc1I5eGfHOfbz1v/A5O/3lTpmrz1P+CckreIxJ5aJW7n3IfOuTGRCqY+nXWsp8bdo13ToOcvyO0Y9Pi7vzsFgAuf/pzOE97j5AdnRyZAEZE6Stga90ldW7PgzrMY0j14e7uZsfSeswOOH9u+WaX9TTv3k7duR0RiFBGpi4RN3ADNM1OrPZ+ZlsK6B0dzzendAGiakVKpR0q585/6PCLxiYjUhSbsAG46+xh+ckJH2md5+n/n3T6cf368lqc/rhgaf6C4NKCHyoD73qdxekqlgT0iIpGW0DXu2ujcujHpKZ7E3LpJOhNG9WTdg6PpcYSnjfy2yYsDXpO/5yDfbtvHa3kbGjRWETm8KXHX4P9+PgCAN+dvrHR80879vu2b31jIDG9/cBGRSFPirsERfsPn/ft2V+1tMu6FageWiojUGyXuMJx3fAcAlm7ZHXDu5G6tfNs7NPeJiDQAJe4wXDa4EwBbdx8AYMGGnb5zk64c5Nvuf+9M9hwobtjgROSwo8QdhvJJqtZtLwTg3Cc/BeAPZx8DwIzrh/qu7XPXDAA27CjknQWbGzJMETlMqDtgGNo0TQfg3neXcrCk1Hd8zHHtAM9Saf7ueHsxH67MZ8OO/Qzq3JI2zSpPMysicihU4w5Dkt9EVQ9PW+Hb7tSqMeAZhbnqvpG0apwGwAtfrGfDDk+vk1/+K4+c8VPIGT+F5d8HtpGLiNSWEneYRntr1+VW3zey0n5qchKzgwzEWbRpl297xONzWLJ5F7dOXqTZB0WkzpS4w/TI+RWzCd577rGkJAf+0zVvlMr8O8707acFuWb0E5/w0tzv+MdHayITqIgkPCXuMGWmeR4H/KR/Ry4bnBPyupaN0/jVqV0BuNzbGwXg7nOOrXTdI9NX0JDKm2sOFJfWfLGIxDQl7lr49oFRPHrBcTVed/PZx7D0nrO5fUwvxhzXjhNzWnDZoE4B181atjUSYVZSWFTCsXdO8+3/dVbkVrjP33OA299exMqtWvpNJJKUuGvBzILOHlhVUpL5auh/v7g/r//qJJKSzFcTLzdn1baIxOmv153T2VdUUcv+x4drKI1Q+/qtby3mxS++46y/aOk3kUhS4m5Alww8yredlpzEK36LFReXlnH9q99wykP1t3CDf9dFf11vfa/ePsPf+353EGqSEYkcJe4G1D6rEcN7tuXVcYPISE0iNTmJ306aT874KXS/bSqTv97Exh/211uPk2Nur2giefPXgyud+6fflLW1NXbi51zx3Jd8urrijqGktKzSNd/4jS4VkfqlxN2AkpOMZ67IZWCXVnRskcmeAyVMWbQl4LqX/Wri9eE/vz2ZEzq15EO/7or3vbesTmtqlpU5vli7g49WFnDJM3P5cEU+AGP+9gkAw3t6lowbO/ELdu3X8H+RSFDijpKRvY8Iee62yYu5dfIivt22r87vv357xWv7HpkFQE7rxgzv2cZ3/IlZq2v1no/NXEmXKs0sP3v+K9YU7GX5954Hkqf3qFgqru/dMyLWnh7Ke4u2sGFHYYN+pkhDqzFxm1mGmX1pZgvMbImZ3d0QgSU6/wE9g7q0BCo3Z7w09ztOf/TDOr//A+8tD3r87xf3923/5f2VtXrPJ0L0SDnjzx/5tkf3acdVQzr79l+p57uH6mzYUchvJs1nyMMfMOqvc3hr/kZyxk9h2uLAu5p48cmqbfx99qqQzWf7i0oZ+vAHlf5QS+ILp8Z9EBjmnOsL9ANGmNmgGl4jNejYItO3/fzPBjDrxlM5oVNLerarvFhx+YyEtTXNu7DD134DggAyUpNZ9+Bo3/5iv5GdtTGsR5uAYx/edBpZmWmk+g08um3yYnLGT+H6V7+p0+eEa8nmXQx5+APf/tItu7nhtQUA/OrF+SEf1Mayf322jkufncujM1ayeHPw/04975zGdzsKGfnXOTz5wWrNTnmYqDFxO4+93t1U74/Gax+itJSKf/pGacl0zfZMVDX12iGVug36r7QTrtX5Ff2os0IsmFw+MdaYv31CcZUHi8HkV/kD8s/Lc/l0/DDf/iPnH0dOa8/cLdcM68Z5/TtUun7y15vCC76Oxj79RbXnD+VhbKSF6oHzx3eW+LYXb/LMc7N19wHf9YVFJb7zhUWlPDJ9hW92SklsYbVxm1mymX0D5AMznXNzIxvW4eGlqwby0E/6BBwfP7IH06/zTBW7uQ6Je/hjnn7Uw3q0CdnvfNJVA33b3W+bWuN7zl6e79u+bVRPkpOMDlmNeP+GoTz/sxO5IPdI3/nMtBQeu7AfQ7q3rvQeL82NXLPJnoMl1Z7/+we1a89vKK/nbaDHHdN44Yv13P3fJb5nAn+ZWbkZ69bJiygpLWPg/bO44rkvWbp5N73unB70PT9fsz3icUt0hZW4nXOlzrl+QEdggJn1rnqNmY0zszwzyysoKKjvOBPSSV1b89MTjwp6rnzF+Wte+rpWD/j8r714QPD3BmjTNIO+HZv79mtqStjvreV9On4YVw3t4jverU1TTg/SbALwwi8HVtq/dfIiAKYu2kLO+Cms+L7+R1heMvAoZl4/lLX3j+KvY/vx+QTPXcEZ3t4useSFz9fxhzcWAp6pgJ//dB3Ltuxm5dY9QUe4btnlueuZ++0ORj0xJ+T7PjqjYadTkIZXq14lzrmdwIfAiCDnJjrncp1zudnZ2QGvldppmlHRxLEqP/wEVz517K2jejC8V/XJ6s8X9vVtr9tWfU+MJZt307pJOu2b125u8XUPjq7Upv7F2u38etJ8wJOs6sPagr2+7T/9qDfd2zYlKck4t18H3yIYUxZuYX9RaUB/82i44rkvyRk/hTv+syTg3OdrtlcaefrXsf3o5m3WmhTijmXdg6O5MLcjd4zpBcC89T/wwYr8oNfW1oHi0jp1G5XICqdXSbaZZXm3GwHDgeBdFiQiRjw+J+wBLaOf8PSn9n/4GUq3Nk15+7cnA/D4+yu56t95Qf8nLSkt4415G2ndJC2sIf/VGTuxoi26R7umh/Re5b7+zvNvc9GAo4LGd0KnFoDnQd7wxz4KON9QnHMcLCnlo5Wh70jve2+ZbzszLZlz+3XggfM8zWlPBZlRslFqMgAPn9+XX55S0Zvn2pe/rnOcSzfvJmf8FDpPmEKPO6bR9261m8eacGrc7YAPzGwh8BWeNu53IxuWADx1aUXXvR89+amvaWF/UWnQB1r+SffsY0P3E/d3ZAtPjXTq4u+ZuXQr3wfpxfLKVxsAfH216+Lcfu0Djv378/V1evha1TLvIs4TRvUIev65n53o2163vZB1h9A//lC8OX9TpdGs5TJSkyrdlZRbeo/nxrbHEcH/wJ3Xv0OlaYQB3wPjztlNgr0kLOc/9RkA5V+n3QcCnx8MeXg2OeOncM9/l2pu+SgIp1fJQufc8c6545xzvZ1z9zREYAIjerdjQOeWvv3H319JcWkZPe+cRo87pnGLt310bcFenHMU+k0mlZwUXs24pXfVnnKDHwicK+V2b5PG4z/tV+sylHvk/IpmmRF+f1TePsTeJne9s4RnPvkWgGYZwXvQNG9U+fjvX6m5NhqJ5oGbXl/g237y4v6sum8klw/uxMzrTw249pNbTvdtN61Srg9uOo3HLuzL/T/uQ6O05ErnOmQ14qIBR7Jgw05W5+9hbw0PbYMJ9kf/tTzPH+/Fm3bR9db3fCs8Pffpt/x3odZWbWgaORnjJl050Je8py7+vlIPkFfzNpAzfgrD/vwRnSe8x0PTPC1YwXqqhGJmvkWPg/FPYOf0Daw1hystJYlLBh7F8Udl8dRlJ/jmKn9k+ooaa2xlZS7gDqM8rv/7bF1Yn3/vucfSuoln7dCFG3f55idfnb834Nof/++ndJ7wHuPfXMiOfUVhvX+59dv3Bb0bOrpKz53Rx7UjNTmJe87tzZEtPc1aF5zQEYD5d5wZ0NTVJdvT1fL9G4bSuXVjzuvfkYzUyknb91ltPTX04Y99zJX/+qpW8YPnjq5c+bqqN7+xkA07Chnzt08CHpavLdDgn4ZmkahZ5Obmury8vHp/38NZzvgpYV/71KX9GdG7Xc0Xeh0oLuWB95bxr8/X+45dNaQzt43uxYINO32r2ge7nT8U5WX68KbTfH3A/e0+UMyvXpjHF2u3U+aga3Zj1oRIEreO6sG4oV2Dngv2mf7WPTiaXfuL2bGviM6tGwdcM+26IfQ4ovLAqLIyx9pt+2icnsyqrXu5/LkvfeeObNmIOTcP4+OVBVz+3Jec0KkF89b/EPCZVZWWOb7dtpdubQ6t7X/Tzv2c/GDFnZP/ZxWXllFYVBpwF+Lvgqc+w8x47erB7DtYwrF/DN7t8I4xvbj33aX0bNeMqdcOOaSYBcxsnnMuN5xrVeOOE/6DXaDyUmpVJdXyAWJGajJ3n9ubF/267/1zzreUlTmme0dgPntFWN+nWrnrfzy9IP48M3Do/cqtezjurhl8tsaTtIGQSfvY9s24vJpVifxVbRMGePKD1fS9ewanP/ohP3/+y4DzIx6fw+veu5tdhZ6Rif9ZsInhj33E4AdmV0raABt27GfDjkLfcf+kvfzeEXz7wKigsSUn2SEnbfA0l0y6suK/pf8dwOAHZtP37hkBdzmlZY4bXv2G9dv3sXnnAV/vocbpKXx52xlBP+eKwZ3ISE1i2ZbdzFq2lbx1OwKu2bCjEOccq7buicu28Kc+WsNr3mc8sUQ17jjy6lff8dC0FUy6cqBvaPzq/L28OX8jawv2Mn3JVo5t34x3f3dKnXt/DHl4tq/90qziAdWa+0eF3W4erl2FxfS9p6LHwnu/H0LPdk2Z8NYi3wPRcNT2TuCm1xfQvnkGvdo351cvzqvVa8/r34Frz+jOqY98WKvXAcy7fTitvM01DeHed5fyrLf9P+/24eT+6X3fuccu7Mt5/TvinKPMwcerCvj58xXNKr85rSs3j6h42Ot/F/Lb07sy9sSjOLJlJg9OXV6pt8uCO8+ieWYqew4U+0ZxtmmaTv6eg0D93LUVFpVwzt8/5e5zjuXkbq1DXrfvYAm3v72YW0b04Igwu7G+8MV67nh7MTeeeTTFpWU8MdszcGv1fSODrjNbn2pT406JaCRSr3564lEBA3a6tWnCLSOC96aoizk3D+ObDTv50ZOf4v83vb6TNkDzKsPxQw0q6dWuGb85vSvXvPQ1w3q0YfbyfLKbpjPl96ewe3/tH749ekHfGq95+rITOK5jc4Y89AElfjXFt+Zv4q35gQ9Ubx/dk1F92lFa5irNmTKy9xFkZabSKDUl4EFwpHX2a37yT9oAN7y2gD4dmnNmiNWKqia6d393CrdNXsTL4wb5VncC6FKliWvUE3P4dPww32AhwJe0wTN1QptmtRsL4O8vM1eyeed+Vufv5ZJn5nLX//SiuNRx33vLaJyWTNtmGfzt4uPp1a6Zr4ln5tKtLL77bMDzIL95o9SQf0DLxxZUvQtcXbA3oLksmpS4JUC/I7P4/bBuvtpGJC295+yQQ7cBzju+A495e7OMOc7zcDR/zwHSU5Jp3iiVQ21ZuP/HfXwjOstdPbSLr2fFjOuHMuzPoft+L793BBt/2O8bJAOeNvvTvDM7Pj62H+kpwR8iRtrQ7tUPhAuVtCHwD3XvDs35zzWnBFyX3axyAizv3rmzMPhkVwPun1XnWvczc9YGjCi9679Lfdv7ikpZu22fbyxDub0HS9hfVEpqsvn+W1aN4eQHZ1fbNXXZlt0xlbjVVCIhld8eXze8O9cNPzqin/XrF+cxdfH3vv2Fd51F0/SUQx7wE44tu/ZjeD5n94FiX6+Mqt6ct5Eb/br0lTcLxLJPVm3j0mc9UwulJSex8r6R3PXOkhp748y+8VS6hNEXvKS0jN9Mms+lgzoFtPWHsuq+kaQmJ3GguJTSMseUhVs4r3+HGpsizvn7JyzcWLfZLI9p25T/vbS/bwri1k3Sybt9OODpodR5QvDl/B67sC83v7GQq0/twh/Orr8722Bq01SixC0h7T1YQmZqMkkRaCapyjnH63kb+WT1Nv58Yd9KU8PGkq/W7aBbdhOaN0ptkH+X+lBYVMIb8zZyycBOvpp0sN41t4/uyZVDugQcD9e4f+cxY+nWSse+mHAGWZmppKck+ZLjrBtPpWt2k4AYcju14JEL+tI+K8N3l+Kc45HpK/jfDwNHjfrr1a4ZxaVlrKrSvbPHEU1DDhz7xyX9fdMv+HvgvD70bNeMuWu3c/WpXTnzsY9IT03i3d9FtueMEreIVOuz1du4+Jm5zL7xVBqnp/D4+yu5Y0yvSu3XdVE1GZfXrgGmLd7Cr14MTJRVDezcklev9iwq8upX33HLm5Wbslo3SeeGM49jIO+AAAAHo0lEQVRm38ES7p+6jAtO6MjD3gFeRSVlXPrMXC4b3InhPdtiBj3uCBytWp3l946o1Ef+9rcX8eIX3/HVbcPJbhq5h8tK3CISFeU9hV4ZN4i+HbMqjez072lSky9vO4M2TTMC/hCMOa5dpVWcwlFa5ujqt+Te3FvPYOD9swKuO6NHG87s1ZaxVWbVnLpoC7+eNJ+XrxrE4K6tavXZtaF+3CISFc0zU1n34GgGdWkVMBy/6tD96gy4b1bAtAMjex9RaUbLcPk/aP3kltNp2yyD1OTAZq6bzj4mIGmD58EswLz1gf3Uo0W9SkSkwdw5phc79hXRtU1jMlKSGdnHM8LXOUdJmeOFz9dzz7ueniKv+vXl/3zCMN8UvXWx8k8jKSwqISvT0yXzoz+czkkPzubXp3XllhE92LW/OORo0nberpGPzljJNcO61zmG+qTELSIN5hd+U8/6MzNSk41fnNKZ4tIyHpi6nPFvedq2//WLAYeUtMEzV05aSkU/+vZZjSp1CaxuCgD/3i4Few5y4n3v06dDc9655uQG6fUUjJpKRCSmdK4yqGdo99CjIxtK+Zzob8zbCMCiTbu42Ts7ZzQocYtITDmzV1t+N6wbf/pRbz4dPyxqtVp/5Ytrl8/ACfD6vI1s2FHI7gPF7DtYwuXPfcniTXXrZ15baioRkZhiZtx4VuiphqPBf2Ssv/LpDSb/5iQ+XlnAxysL6n0WzWBU4xYRqUH5Q03wTMBV1YVPf+7bbog1OlXjFhEJg39N+vozj660qElxqSdZ//sXAxokFtW4RURqKTU5KWjNe+jR2Q3SJq/ELSJSBzeP6MG3D4xiSPfW5LTK5PMJw2p+UT1RU4mISB2ZGS/4rRzVUGqscZvZkWb2gZktM7MlZnZtQwQmIiLBhVPjLgFudM7NN7OmwDwzm+mcW1rTC0VEpP7VWON2zm1xzs33bu8BlgEdIh2YiIgEV6uHk2aWAxwPzA1ybpyZ5ZlZXkFBQf1EJyIiAcJO3GbWBHgTuM45t7vqeefcROdcrnMuNzu7+rXuRESk7sJK3GaWiidpT3LOvRXZkEREpDrh9Cox4FlgmXPusciHJCIi1Qmnxn0ycBkwzMy+8f6MinBcIiISQkTWnDSzAmB9HV7aGthWz+HEkkQvHyR+GVW++BbL5evknAvrAWFEEnddmVleuItlxqNELx8kfhlVvviWKOXTXCUiInFGiVtEJM7EWuKeGO0AIizRyweJX0aVL74lRPliqo1bRERqFms1bhERqUHMJG4zG2FmK8xstZmNj3Y84TKz58ws38wW+x1raWYzzWyV93cL73Ezsye8ZVxoZv39XnOF9/pVZnZFNMoSTKhpfROljGaWYWZfmtkCb/nu9h7vbGZzvbG+amZp3uPp3v3V3vM5fu81wXt8hZmdHZ0SBWdmyWb2tZm9691PtPKtM7NF3nEmed5jCfEdDco5F/UfIBlYA3QB0oAFQK9oxxVm7EOB/sBiv2MPA+O92+OBh7zbo4CpgAGDgLne4y2Btd7fLbzbLaJdNm9s7YD+3u2mwEqgV6KU0RtnE+92Kp4J1AYBrwFjvcefAn7t3f4N8JR3eyzwqne7l/d7mw509n6fk6NdPr9y3gC8BLzr3U+08q0DWlc5lhDf0aDljXYA3n+wwcB0v/0JwIRox1WL+HOqJO4VQDvvdjtghXf7aeCiqtcBFwFP+x2vdF0s/QD/Ac5MxDICmcB8YCCeQRop3uO+7ycwHRjs3U7xXmdVv7P+10X7B+gIzAKGAe96402Y8nnjCZa4E+47Wv4TK00lHYANfvsbie85v9s657aAZz5zoI33eKhyxkX5q0zrmzBl9DYjfAPkAzPx1CZ3OudKvJf4x+orh/f8LqAVMVw+4HHgZqDMu9+KxCofgANmmNk8MxvnPZYw39GqYmXNyWDLIidid5dQ5Yz58led1tdCr2Qdd2V0zpUC/cwsC5gM9Ax2mfd3XJXPzMYA+c65eWZ2WvnhIJfGZfn8nOyc22xmbYCZZra8mmvjtYw+sVLj3ggc6bffEdgcpVjqw1Yzawfg/Z3vPR6qnDFdfgs+rW9ClRHAObcT+BBPu2eWmZVXbPxj9ZXDe745sIPYLd/JwDlmtg54BU9zyeMkTvkAcM5t9v7Ox/PHdwAJ+B0tFyuJ+yugu/dJdxqehyLvRDmmQ/EOUP5E+go87cLlxy/3PtUeBOzy3sJNB84ysxbeJ99neY9FnVnIaX0Tooxmlu2taWNmjYDheJbn+wA433tZ1fKVl/t8YLbzNIi+A4z19sroDHQHvmyYUoTmnJvgnOvonMvB8//VbOfcJSRI+QDMrLF51sPFzBrj+W4tJkG+o0FFu5Hd70HAKDw9FtYAt0U7nlrE/TKwBSjG8xf7l3jaBGcBq7y/W3qvNeBJbxkXAbl+7/MLYLX35+fRLpdfXKfguV1cCHzj/RmVKGUEjgO+9pZvMXCn93gXPIlpNfA6kO49nuHdX+0938XvvW7zlnsFMDLaZQtS1tOo6FWSMOXzlmWB92dJef5IlO9osB+NnBQRiTOx0lQiIiJhUuIWEYkzStwiInFGiVtEJM4ocYuIxBklbhGROKPELSISZ5S4RUTizP8DDrKs+oKk8qcAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.sched.plot_loss()" ] }, { "cell_type": "code", "execution_count": 253, "metadata": {}, "outputs": [], "source": [ "learn.save('all')" ] }, { "cell_type": "code", "execution_count": 254, "metadata": {}, "outputs": [], "source": [ "learn.load('all')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Test**" ] }, { "cell_type": "code", "execution_count": 255, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "quelles composantes des différents aspects de la performance devraient être mesurées , quelles données pertinentes recueillir et comment ? _eos_\n", "which components within various performance areas should be measured , whatkinds of data are appropriate to collect , and how should this be done ? _eos_\n", "what components of the different aspects of should be be measured , and and how how ? _eos_\n", "\n", "le premier ministre doit - il nommer un ministre d’ état à la santé mentale , à la maladie mentale et à la toxicomanie ? _eos_\n", "what role can the federal government play to ensure that individuals with mental illness and addiction have access to the drug therapy they need ? _eos_\n", "who is the minister minister to minister minister to mental health mental and mental ? ? ? _eos_\n", "\n", "quelles sont les conséquences de la hausse des formes d’ emploi non conformes aux normes chez les travailleurs hautement qualifiés et chez ceux qui occupent des emplois plus marginaux ? _eos_\n", "what is the impact of growing forms of non - standard employment for highly skilled workers and for those employed in more marginal occupations ? _eos_\n", "what are the implications of increasing employment forms of workers workers workers workers workers workers workers workers workers workers workers workers more more ? _eos_ _eos_ _eos_\n", "\n", "que se produit - il si le gestionnaire n’ est pas en mesure de donner à l’ employé nommé pour une période déterminée un préavis de cessation d’ emploi d’ un mois ou s’ il néglige de le\n", "what happens if the manager is unable to or neglects to give a term employee the one - month notice of non - renewal ? _eos_\n", "what happens the manager does not to to the employee employee employee employee a a employee a employee or employee or or or or or or or or ?\n", "\n", "quelles personnes , communautés ou entités sont considérées comme potentiels i ) bénéficiaires de la protection et ii ) titulaires de droits ? _eos_\n", "which persons , communities or entities are identified as potential ( i ) beneficiaries of protection and / or ( ii ) rights holders ? _eos_\n", "who , communities communities or entities as as potential as beneficiaries of ( protection and protection protection protection ? ? _eos_ _eos_ _eos_\n", "\n", "quelles conditions particulières doivent être remplies pendant l’ examen préliminaire international en ce qui concerne les listages des séquences de nucléotides ou d’ acides aminés ou les tableaux y relatifs ? _eos_\n", "what special requirements apply during the international preliminary examination to nucleotide and / or amino acid sequence listings and / or tables related thereto ? _eos_\n", "what special conditions must be required during the international preliminary preliminary in for nucleotide or sequence amino or sequence or or or or tables ? ? _eos_ _eos_ _eos_\n", "\n", "pourquoi cette soudaine réticence à promouvoir l’ égalité des genres et à protéger les femmes de ce que , dans la plupart des cas , on peut qualifier de violations grossières des droits humains ? _eos_\n", "why this sudden reluctance to effectively promote gender equality and protect women from what are – in many cases – egregious human rights violations ? _eos_\n", "why this sudden effect of to to women women women and and of of of , , of of of human human human human ? _eos_ _eos_ _eos_ _eos_\n", "\n", "pouvez - vous dire comment votre bagage culturel vous a aidée à aborder votre nouvelle vie au canada ( à vous adapter au mode de vie canadien ) ? _eos_\n", "what are some things from your cultural background that have helped you navigate canadian life ( helped you adjust to life in canada ) ? _eos_\n", "what can you you your your cultural your your you to to to canada canada canada life life life life canada ? _eos_\n", "\n", "selon vous , quels seront , dans les dix prochaines années , les cinq enjeux les plus urgents en matière d' environnement et d' avenir viable pour vous et votre région ? _eos_\n", "which do you think will be the five most pressing environmental and sustainability issues for you and your region in the next ten years ? _eos_\n", "what do you see be be the the next five five five , and and and and and and and your your ? ? ? _eos_ _eos_ _eos_\n", "\n", "dans quelle mesure l’ expert est-il motivé et capable de partager ses connaissances , et dans quelle mesure son successeur est-il motivé et capable de recevoir ce savoir ? _eos_\n", "what is the expert ’s level of motivation and capability for sharing knowledge , and the successor ’s motivation and capability of acquiring it ? _eos_\n", "what is the expert ’s and and knowledge knowledge knowledge knowledge and and and and and and and and and and and ? ? ? _eos_ _eos_\n", "\n" ] } ], "source": [ "x,y = next(iter(val_dl))\n", "probs = learn.model(V(x))\n", "preds = to_np(probs.max(2)[1])\n", "\n", "for i in range(180, 190):\n", " print(' '.join([fr_itos[o] for o in x[:, i] if o != 1]))\n", " print(' '.join([en_itos[o] for o in y[:, i] if o != 1]))\n", " print(' '.join([en_itos[o] for o in preds[:, i] if o != 1]))\n", " print()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 2 }