{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# FAST version of the notebook:\n", "# *Faster than training from scratch - Fine-tuning the English GPT-2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)*\n", "\n", "> Tutorial on how to use fastai v2 over Hugging Face's Transformers and Tokenizers libraries to fine-tune an English pre-trained transformer-based language model (GPT-2) to any language other than English" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n", "- Date: July 2020 (14/07/2020)\n", "- Post in medium: [Faster than training from scratch - Fine-tuning the English GPT-2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787)\n", "- Notebook with all explanation: [finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb)\n", "- Hugging face model page of [GPorTuguese-2](https://huggingface.co/pierreguillou/gpt2-small-portuguese): a language model for Portuguese text generation (and more NLP tasks...)\n", "- Other posts in medium of the GPT-2 series: \n", " - [NLP & fastai | GPT-2](https://medium.com/@pierre_guillou/nlp-fastai-gpt-2-16ee145a4a28)\n", " - [Byte-level BPE, an universal tokenizer but...](https://medium.com/@pierre_guillou/byte-level-bpe-an-universal-tokenizer-but-aff932332ffe)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial, instead of training from scratch, we will see how to fine-tune in just over a day, on one GPU and with a little more than 1GB of training data an English pre-trained [transformer](https://arxiv.org/abs/1706.03762)-based language model to any another language. \n", "\n", "As a practical case, we fine-tune to Portuguese the [English pre-trained GPT-2](https://github.com/openai/gpt-2) by wrapping the [Transformers](https://github.com/huggingface/transformers) and [Tokenizers](https://github.com/huggingface/tokenizers) libraries of Hugging Face into [fastai v2](https://github.com/fastai/fastai2). We thus create a new language model: [GPorTuguese-2](https://huggingface.co/pierreguillou/gpt2-small-portuguese), a language model for Portuguese text generation (and more NLP tasks...)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![The 3 main steps of fine-tuning the English GPT-2 to Portuguese with Hugging Face and fastai v2 (image edited from fast.ai NLP)](images/GPT2_tf_ft_approach.png \"The 3 main steps of fine-tuning the English GPT-2 to Portuguese with Hugging Face and fastai v2 (image edited from fast.ai NLP)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Acknowledgment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This tutorial was made possible thanks to the computing power of the [AI Lab](https://www.linkedin.com/company/ailab-unb/) (University of Brasilia) to which I am attached as an Associate Researcher in NLP and the participation of its directors in the definition of the NLP strategy, Professors [Fabricio Ataides Braz](https://www.linkedin.com/in/fabricio-braz-b356457/) and [Nilton Correia da Silva](https://www.linkedin.com/in/nilton-silva-6097853/). Thank you so much!\n", "\n", "And special thanks to Sylvain Gugger for his [tutorial on Transformers and fastai v2](https://dev.fast.ai/tutorial.transformers) which is the basis of this tutorial." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Post, notebooks, Web App and model download" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The main code of the tutorial is published in this post ([Faster than training from scratch - Fine-tuning the English GPT-2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787)), organized by paragraph. \n", "\n", "The complete code is in this notebook. However, as this notebook is very detailed, you can use this fast notebook finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2_FAST.ipynb (nbviewer version) if you just want to execute the code without explanation.\n", "\n", "In addition, our **GPorTuguese-2 (Portuguese GPT-2 small) , a language model for Portuguese text generation (and more NLP tasks...)**, is testable online in the [Hugging face model hub](https://huggingface.co/models) with all usage information at this address: https://huggingface.co/pierreguillou/gpt2-small-portuguese" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![GPorTuguese-2 (Portuguese GPT-2 small) , a language model for Portuguese text generation (and more NLP tasks...)](images/hfmh.png \"GPorTuguese-2 (Portuguese GPT-2 small) , a language model for Portuguese text generation (and more NLP tasks...)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Main coding steps to fine-tune a Hugging Face language model with fastai v2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The 6 main steps detailed below can be summarized in 3 main ones:\n", "\n", "1. **Initialization & download** (download of Portuguese Wikipedia and GPT-2 English pre-trained model and tokenizer)\n", "2. **GPT-2 tokenizer with a Portuguese vocab** (train a GPT-2 tokenizer with a vocab in Portuguese, wrap it into a fastai v2 tokenizer and update the embeddings matrix of the GPT-2 English pre-trained model according to the new Portuguese vocab: keep the embeddings vectors of the common tokens between English and Portuguese vocabs)\n", "3. **Fine-tune on Portuguese Wikipedia the GPT-2 model with fastai v2 training functionalities**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Initialization" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "nbpresent": { "id": "151cd18f-76e3-440f-a8c7-ffa5c6b5da01" } }, "outputs": [], "source": [ "from fastai2.text.all import *\n", "from nlputils_fastai2 import * \n", "\n", "%reload_ext autoreload\n", "%autoreload 2\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "nbpresent": { "id": "6ceb4db2-e4cf-4fe0-a393-91df4a7ed3e7" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cuda device: 0\n" ] } ], "source": [ "gpu = 0\n", "torch.cuda.set_device(gpu)\n", "print(f'cuda device: {torch.cuda.current_device()}')\n", "# print(f'cuda device name: {torch.cuda.get_device_name(gpu)}')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'archive_path': '/storage/archive/',\n", " 'data_path': '/storage/data/',\n", " 'model_path': '/storage/models/',\n", " 'storage_path': '/storage/data/',\n", " 'version': 2}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get config of paths\n", "config = Config()\n", "config.d" ] }, { "cell_type": "markdown", "metadata": { "nbpresent": { "id": "cf070ab7-babb-4cf0-a315-401f65461dc8" } }, "source": [ "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents. (For other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "nbpresent": { "id": "70da588b-8af1-4f97-97c2-c9f2d4d46e1a" } }, "outputs": [], "source": [ "lang = 'pt'" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "nbpresent": { "id": "701ab344-0430-4f43-bbe2-337a12cae6be" } }, "outputs": [], "source": [ "# setup new path_data and create the corresponding folder\n", "name = f'{lang}wiki'\n", "data_path = config['data_path']\n", "path_data = data_path/name\n", "path_data.mkdir(exist_ok=True, parents=True)" ] }, { "cell_type": "markdown", "metadata": { "nbpresent": { "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2" } }, "source": [ "### 2. Download Wikipedia in Portuguese" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note**: all the following methods come from the file nlputils_fastai2.py." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(Path('/mnt/home/pierre/course-v4/nbs'),\n", " Path('/mnt/home/pierre/.fastai/data/ptwiki'))" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Path.cwd(), path_data" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/mnt/home/pierre/.fastai/data/ptwiki/ptwiki already exists; not downloading\n" ] } ], "source": [ "get_wiki(path_data,lang)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If `get_wiki(path_data,lang)` breaks, fix the download manually no terminal:\n", "- mkdir -p /mnt/home/pierre/.fastai/data/ptwiki\n", "- cd /mnt/home/pierre/.fastai/data/ptwiki\n", "- wget -c https://dumps.wikimedia.org/ptwiki/latest/ptwiki-latest-pages-articles.xml.bz2\n", "- bzip2 -dk ptwiki-latest-pages-articles.xml.bz2\n", "\n", "And re-run `get_wiki(path_data,lang)` once the download is successful." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "Astronomia\r\n", "\r\n", "Astronomia é uma ciência natural que estuda corpos celestes (como estrelas, planetas, cometas, nebulosas, aglomerados de estrelas, galáxias) e fenômenos que se originam fora da atmosfera da Terra (como a radiação cósmica de fundo em micro-ondas). Preocupada com a evolução, a física, a química e o movimento de objetos celestes, bem como a formação e o desenvolvimento do universo.\r\n" ] } ], "source": [ "!head -n4 {path_data}/{name}" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/mnt/home/pierre/.fastai/data/ptwiki/docs already exists; not splitting\n" ] } ], "source": [ "dest = split_wiki(path_data,lang)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/mnt/home/pierre/.fastai/data/ptwiki/docs/Fotografia.txt\n", "/mnt/home/pierre/.fastai/data/ptwiki/docs/Espadanedo (Macedo de Cavaleiros).txt\n", "/mnt/home/pierre/.fastai/data/ptwiki/docs/Jacques-Germain Soufflot.txt\n", "/mnt/home/pierre/.fastai/data/ptwiki/docs/Faculdade de Medicina da Universidade de São Paulo.txt\n", "/mnt/home/pierre/.fastai/data/ptwiki/docs/Escola do Teatro Bolshoi no Brasil.txt\n" ] } ], "source": [ "dest = path_data/'docs'\n", "for file in dest.ls()[:5]:\n", " print(file)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "203205 files - 193686269 tokens\n", "CPU times: user 1min, sys: 20.8 s, total: 1min 21s\n", "Wall time: 11min 16s\n" ] } ], "source": [ "%%time\n", "# Size of downloaded data in the docs folder\n", "num_files, num_tokens = get_num_tokens(dest)\n", "print(f'{num_files} files - {num_tokens} tokens')" ] }, { "cell_type": "markdown", "metadata": { "nbpresent": { "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2" } }, "source": [ "#### Create text and csv files of wikipedia in Portuguese" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dest = path_data/'docs'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Text file" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "get_one_clean_file(dest,lang)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### csv file" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "get_one_clean_csv_file(dest,lang)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Download a GPT-2 English pre-trained model and train a GPT-2 tokenizer with a vocab in Portuguese" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are following 3 steps in order to **get a GPT-2 tokenizer with the vocab in Portuguese**:\n", "- 3.1) **Get the pre-trained GPT-2 Tokenizer & Model (pre-training with an English corpus) from the Transformers library (Hugging Face)**: it will give us the tokenizer structure we need and the pre-trained model weights (it's better to start training our GPT-2 model in Portuguese from weights already trained even in another language than from random values)\n", "- 3.2) **Train a Byte-level BPE (BBPE) Tokenizer on the Portuguese Wikipedia corpus by using the Tokenizers library (Hugging Face)**: this will give us the vocabulary files in Portuguese of our GPT-2 tokenizer.\n", "- 3.3) **Import the tokenizer Portuguese config files into the pre-trained GPT-2 Tokenizer**: it will give us a GPT-2 tokenizer structure with the vocab in Portuguese." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 3.1 Get the pre-trained GPT2 Tokenizer & Model (pre-training with an English corpus)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "transformers==3.0.0\r\n" ] } ], "source": [ "# ! pip install transformers\n", "!pip freeze | grep transformers" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from transformers import GPT2TokenizerFast, GPT2LMHeadModel" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 13 s, sys: 922 ms, total: 13.9 s\n", "Wall time: 7.48 s\n" ] } ], "source": [ "%%time\n", "pretrained_weights = 'gpt2'\n", "tokenizer_en = GPT2TokenizerFast.from_pretrained(pretrained_weights)\n", "model_en = GPT2LMHeadModel.from_pretrained(pretrained_weights)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# To correct the warning about token_pad (GPT2TokenizerFast), run the following code\n", "# source: https://github.com/huggingface/transformers/issues/2648#issuecomment-616177044\n", "tokenizer_en.pad_token = tokenizer_en.eos_token" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 3.2 Train a Byte Level BPE (BBPE) Tokenizer on the Portuguese Wikipedia" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tokenizers==0.8.0\r\n" ] } ], "source": [ "# !pip install tokenizers\n", "!pip freeze | grep tokenizers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Training" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "50257" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get GPT2 tokenizer_en vocab size\n", "ByteLevelBPE_tokenizer_pt_vocab_size = tokenizer_en.vocab_size\n", "ByteLevelBPE_tokenizer_pt_vocab_size" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 5h 47min 23s, sys: 43min 12s, total: 6h 30min 35s\n", "Wall time: 10min 52s\n" ] }, { "data": { "text/plain": [ "['/mnt/home/pierre/.fastai/data/ptwiki/ByteLevelBPE_tokenizer_pt/vocab.json',\n", " '/mnt/home/pierre/.fastai/data/ptwiki/ByteLevelBPE_tokenizer_pt/merges.txt']" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "# ByteLevelBPETokenizer Represents a Byte-level BPE as introduced by OpenAI with their GPT-2 model\n", "from tokenizers import ByteLevelBPETokenizer\n", "\n", "ByteLevelBPE_tokenizer_pt = ByteLevelBPETokenizer()\n", "\n", "# Get list of paths to corpus files\n", "paths = [str(path_data/'all_texts_ptwiki.txt')]\n", "\n", "# Customize training with <|endoftext|> special GPT2 token\n", "ByteLevelBPE_tokenizer_pt.train(files=paths, \n", " vocab_size=ByteLevelBPE_tokenizer_pt_vocab_size, \n", " min_frequency=2, \n", " special_tokens=[\"<|endoftext|>\"])\n", "\n", "# Get sequence length max of 1024\n", "ByteLevelBPE_tokenizer_pt.enable_truncation(max_length=1024)\n", "\n", "# save tokenizer\n", "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n", "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n", "if not (path_to_ByteLevelBPE_tokenizer_pt_rep).exists():\n", " path_to_ByteLevelBPE_tokenizer_pt_rep.mkdir(exist_ok=True, parents=True)\n", "ByteLevelBPE_tokenizer_pt.save_model(str(path_to_ByteLevelBPE_tokenizer_pt_rep))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now have both a vocab.json, which is a list of the most frequent tokens ranked by frequency, and a merges.txt list of merges." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Check our tokenizer pre-trained in Portuguese" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# Load the tokenizer ByteLevelBPE_tokenizer_pt\n", "from tokenizers import ByteLevelBPETokenizer\n", "\n", "# Get the path to ByteLevelBPE_tokenizer_pt config files\n", "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n", "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n", "\n", "ByteLevelBPE_tokenizer_pt = ByteLevelBPETokenizer(\n", " vocab_file=f'{path_to_ByteLevelBPE_tokenizer_pt_rep}/vocab.json',\n", " merges_file=f'{path_to_ByteLevelBPE_tokenizer_pt_rep}/merges.txt'\n", ")\n", "\n", "# Get sequence length max of 1024\n", "ByteLevelBPE_tokenizer_pt.enable_truncation(max_length=1024)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Create a fastai tokenizer and update the embeddings matrix of the GPT-2 English pre-trained model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's see how we can use fastai v2 to fine-tune this model on Wikipedia in Portuguese, using all the fastai v2 training utilities.\n", "\n", "We will follow these 2 following steps:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 4.1) **GPT2TokenizerFast (imported GPT-2 tokenizer) --> fastai Tokenizer**: to process the data to train a model, we need to build a fastai tokenizer from the GPT-2 tokenizer with vocab in Portuguese.\n", "- 4.2) **Change vocab embeddings (wte matrix) in the GPT-2 pre-trained model to adapt to the Portuguese vocab**: as the vocab embedding matrix (wte) of the pre-trained GPT-2 model corresponds to the English vocabulary, we'll keep the embeddings vectors of the common tokens between the English and Portuguese vocab." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "from fastai2.text.all import *" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 4.1 GPT2TokenizerFast (imported GPT-2 tokenizer) --> fastai Tokenizer" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "class TransformersTokenizer(Transform):\n", " def __init__(self, tokenizer): self.tokenizer = tokenizer\n", " def encodes(self, x): \n", " toks = self.tokenizer.tokenize(x)\n", " return tensor(self.tokenizer.convert_tokens_to_ids(toks))\n", " def decodes(self, x): return TitledStr(self.tokenizer.decode(x.cpu().numpy()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Tokenizers" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 16.3 s, sys: 831 ms, total: 17.1 s\n", "Wall time: 6.94 s\n" ] } ], "source": [ "%%time\n", "# Load the GPT2 tokenizer in English\n", "from transformers import GPT2TokenizerFast, GPT2LMHeadModel\n", "pretrained_weights = 'gpt2'\n", "tokenizer_en = GPT2TokenizerFast.from_pretrained(pretrained_weights)\n", "model_en = GPT2LMHeadModel.from_pretrained(pretrained_weights)\n", "\n", "# To correct the warning about token_pad (GPT2TokenizerFast), run the following code\n", "# source: https://github.com/huggingface/transformers/issues/2648#issuecomment-616177044\n", "tokenizer_en.pad_token = tokenizer_en.eos_token" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "# Get the path to ByteLevelBPE_tokenizer_pt config files\n", "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n", "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n", "\n", "# import the pre-trained GPT2TokenizerFast tokenizer with the tokenizer_pt config files\n", "tokenizer_pt = GPT2TokenizerFast.from_pretrained(\n", " str(path_to_ByteLevelBPE_tokenizer_pt_rep), \n", " pad_token='<|endoftext|>')\n", "\n", "# Get sequence length max of 1024\n", "tokenizer_pt.model_max_length = 1024" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 4.2 Change vocab embeddings (wte matrix) in the GPT-2 pre-trained model to adapt to the Portuguese vocab" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Check vocabs size" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(50257, 50257, 0)" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer_fastai_en = TransformersTokenizer(tokenizer_en)\n", "old_vocab_size = tokenizer_fastai_en.tokenizer.vocab_size\n", "\n", "tokenizer_fastai_pt = TransformersTokenizer(tokenizer_pt)\n", "new_vocab_size = tokenizer_fastai_pt.tokenizer.vocab_size\n", "\n", "old_vocab_size,new_vocab_size,old_vocab_size-new_vocab_size" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Changing vocabs and the vocab embeddings matrix (ie, setup new embeddings matrix)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "# Get weights of the old wte\n", "old_wgts = model_en.transformer.get_input_embeddings().weight.clone().detach()\n", "\n", "# Get the mean embedding vetor of the old wte\n", "wgts_m = old_wgts.mean(0)\n", "\n", "# Initialize vocab size and weights of the new wte\n", "new_vocab_size = tokenizer_fastai_pt.tokenizer.vocab_size\n", "new_wgts = old_wgts.new_zeros(new_vocab_size,old_wgts.size(1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Save**" ] }, { "cell_type": "code", "execution_count": 74, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Portuguese wte matrix setup done!\n", "\n", "We kept 12948 embeddings vetors from the English one.\n", "We did not kept 37309 embeddings vetors from the English one (we used the old wte mean vetor).\n", "\n", "15 first tokens IN common between the 2 vocabs:\n", "[('ĠQuit', 40195), ('Smith', 32470), ('Ġomit', 39040), ('oc', 574), ('ym', 18252), ('Ġactual', 9443), ('ck', 911), ('ĠPremier', 16558), ('Ġeste', 987), ('ĠInd', 3438), ('Ġbol', 4203), ('phen', 35836), ('ĠParticip', 36689), ('ĠZeus', 19316), ('Ġnan', 39770)]\n", "\n", "15 first tokens NOT in common between the 2 vocabs:\n", "[('PSDB', 23151), ('Ġenvio', 19270), ('Ġocupação', 5938), ('Ġdocumentada', 30011), ('Ġduros', 36706), ('visto', 44422), ('ĠSiro', 43061), ('Ġdestacavam', 47397), ('Ġarqui', 49060), ('ĠArte', 5977), ('ĠValor', 29721), ('Ġalinhados', 38446), ('Ġnúmeros', 4626), ('Ġpênis', 31686), ('cisa', 29710)]\n" ] } ], "source": [ "# Get the new wte keeping the embeddings vetors of tokens in common in the 2 vocabs\n", "# A token present in the new vocab but not in the old one gets the mean embedding vetor of the old wte\n", "old_vocab = tokenizer_fastai_en.tokenizer.get_vocab()\n", "new_vocab = tokenizer_fastai_pt.tokenizer.get_vocab()\n", "same_tokens_list = list()\n", "different_tokens_list = list()\n", " \n", "for w,idx_new in new_vocab.items(): \n", " idx_old = old_vocab.get(w, -1)\n", " if idx_old>=0:\n", " new_wgts[idx_new] = old_wgts[idx_old]\n", " same_tokens_list.append((w,idx_new))\n", " else:\n", " new_wgts[idx_new] = wgts_m\n", " different_tokens_list.append((w,idx_new))\n", "\n", "# setup in model the new wte\n", "new_wte = nn.Embedding(new_vocab_size,old_wgts.size(1))\n", "#new_wte.weight.data.normal_(mean=0.0, std=model.config.initializer_range)\n", "new_wte.weight.data = new_wgts\n", "model_en.transformer.set_input_embeddings(new_wte)\n", "print(f'Portuguese wte matrix setup done!\\n\\nWe kept {len(same_tokens_list)} embeddings vectors from the English one.\\nWe did not kept {len(different_tokens_list)} embeddings vectors from the English one (instead, we used the old wte mean vector).\\n')\n", "\n", "# Check identical tokens between the 2 vocabs \n", "num = 15\n", "print(f'{num} first tokens IN common between the 2 vocabs:\\n{same_tokens_list[:num]}\\n')\n", "print(f'{num} first tokens NOT in common between the 2 vocabs:\\n{different_tokens_list[:num]}')\n", "\n", "# save new_wgts\n", "torch.save(new_wgts, path_data/'new_wte_wgts.pt')\n", "# save same_tokens_list and different_tokens_list\n", "torch.save(same_tokens_list, path_data/'same_tokens_list.pt')\n", "torch.save(different_tokens_list, path_data/'different_tokens_list.pt')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Load**" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Portuguese wte matrix setup done!\n", "\n", "We kept 12948 embeddings vetors from the English one.\n", "We did not kept 37309 embeddings vetors from the English one (we used the old wte mean vetor).\n", "\n", "15 first tokens IN common between the 2 vocabs:\n", "[('ĠQuit', 40195), ('Smith', 32470), ('Ġomit', 39040), ('oc', 574), ('ym', 18252), ('Ġactual', 9443), ('ck', 911), ('ĠPremier', 16558), ('Ġeste', 987), ('ĠInd', 3438), ('Ġbol', 4203), ('phen', 35836), ('ĠParticip', 36689), ('ĠZeus', 19316), ('Ġnan', 39770)]\n", "\n", "15 first tokens NOT in common between the 2 vocabs:\n", "[('PSDB', 23151), ('Ġenvio', 19270), ('Ġocupação', 5938), ('Ġdocumentada', 30011), ('Ġduros', 36706), ('visto', 44422), ('ĠSiro', 43061), ('Ġdestacavam', 47397), ('Ġarqui', 49060), ('ĠArte', 5977), ('ĠValor', 29721), ('Ġalinhados', 38446), ('Ġnúmeros', 4626), ('Ġpênis', 31686), ('cisa', 29710)]\n" ] } ], "source": [ "# load new_wgts\n", "new_wgts = torch.load(path_data/'new_wte_wgts.pt')\n", "# load same_tokens_list and different_tokens_list\n", "same_tokens_list = torch.load(path_data/'same_tokens_list.pt')\n", "different_tokens_list = torch.load(path_data/'different_tokens_list.pt')\n", " \n", "# setup in model the new wte\n", "new_wte = nn.Embedding(new_vocab_size,old_wgts.size(1))\n", "new_wte.weight.data = new_wgts\n", "model_en.transformer.set_input_embeddings(new_wte)\n", "print(f'Portuguese wte matrix setup done!\\n\\nWe kept {len(same_tokens_list)} embeddings vectors from the English one.\\nWe did not kept {len(different_tokens_list)} embeddings vectors from the English one (instead, we used the old wte mean vector).\\n')\n", "\n", "# Check identical tokens between the 2 vocabs \n", "num = 15\n", "print(f'{num} first tokens IN common between the 2 vocabs:\\n{same_tokens_list[:num]}\\n')\n", "print(f'{num} first tokens NOT in common between the 2 vocabs:\\n{different_tokens_list[:num]}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Changing lm_head weights with the new embeddings matrix" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Linear(in_features=768, out_features=50257, bias=False)" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_en.lm_head.weight = model_en.transformer.wte.weight\n", "model_en.lm_head" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5. Create fastai v2 Datasets and Dataloaders" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 5.1 fastai v2 Datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*(text from Sylvain Gugger Transformers Tutorial)* You can then group your data with this `Transform` using a `TfmdLists`. It has an s in its name because it contains the training and validation datasets.\n", "\n", "We indicate the indices of the training dataset and the validation dataset with `splits` (here, 80% of the indices randomly chosen, then all the remaining indices)." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "203205" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lang = 'pt'\n", "fname = f'all_texts_{lang}wiki.csv'\n", "df = pd.read_csv(path_data/fname)\n", "len(df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Sample (this allows us to quickly test our code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- train: 80%\n", "- val = 20%" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "df_sample = df[:1000]\n", "\n", "num = int(0.8*len(df_sample))\n", "\n", "idxs = np.random.randint(0, len(df_sample), len(df_sample))\n", "idxs_train = idxs[:num]\n", "idxs_val = idxs[num:]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We gather all texts in one numpy array (since it will be easier to use this way with fastai):" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1.46 ms, sys: 218 µs, total: 1.68 ms\n", "Wall time: 1.61 ms\n" ] } ], "source": [ "%%time\n", "all_texts = np.concatenate([df_sample.iloc[idxs_train].text.values, df_sample.iloc[idxs_val].text.values])" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 322 ms, sys: 43.1 ms, total: 365 ms\n", "Wall time: 33.3 ms\n" ] } ], "source": [ "%%time\n", "splits = [list(idxs_train), list(idxs_val)]\n", "tls = TfmdLists(all_texts, TransformersTokenizer(tokenizer_pt), splits=splits, dl_type=LMDataLoader)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We specify `dl_type=LMDataLoader` for when we will convert this `TfmdLists` to `DataLoaders`: we will use an `LMDataLoader` since we have a language modeling problem, not the usual fastai `TfmdDL`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### All data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- train: 80%\n", "- val = 20%" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "# num = int(0.8*len(df))\n", "\n", "# idxs = np.random.randint(0, len(df), len(df))\n", "# idxs_train = idxs[:num]\n", "# idxs_val = idxs[num:]\n", "\n", "# save idxs train and valid\n", "# torch.save(idxs_train, path_data/'idxs_train.pt')\n", "# torch.save(idxs_val, path_data/'idxs_val.pt')" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "# load idxs train and valid\n", "idxs_train = torch.load(path_data/'idxs_train.pt')\n", "idxs_val = torch.load(path_data/'idxs_val.pt')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We gather all texts in one numpy array (since it will be easier to use this way with fastai):" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 42.6 ms, sys: 3.79 ms, total: 46.4 ms\n", "Wall time: 44.6 ms\n" ] } ], "source": [ "%%time\n", "all_texts = np.concatenate([df.iloc[idxs_train].text.values, df.iloc[idxs_val].text.values])" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 340 ms, sys: 31.9 ms, total: 372 ms\n", "Wall time: 151 ms\n" ] } ], "source": [ "%%time\n", "splits = [list(idxs_train), list(idxs_val)]\n", "tls = TfmdLists(all_texts, TransformersTokenizer(tokenizer_pt), splits=splits, dl_type=LMDataLoader)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We specify `dl_type=LMDataLoader` for when we will convert this `TfmdLists` to `DataLoaders`: we will use an `LMDataLoader` since we have a language modeling problem, not the usual fastai `TfmdDL`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 5.2 fastai v2 Dataloaders" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*(text from Sylvain Gugger Transformers Tutorial)* The fastai v2 library expects the data to be assembled in a `DataLoaders` object (something that has a training and validation dataloader). We can get one by using the `dataloaders` method. We just have to specify a batch size and a sequence length. \n", "\n", "Since the GPT-2 model was trained with sequences of size 1024, we use this sequence length (it's a stateless model, so it will change the perplexity if we use less)." ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 6h 43min 14s, sys: 59min 46s, total: 7h 43min\n", "Wall time: 31min 27s\n" ] } ], "source": [ "%%time\n", "bs,sl = 8,1024\n", "dls = tls.dataloaders(bs=bs, seq_len=sl)" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 4 µs, sys: 1e+03 ns, total: 5 µs\n", "Wall time: 11 µs\n" ] } ], "source": [ "%%time\n", "\n", "# IMPOSSIBLE TO SAVE Dataloaders at the date of 07/01/2020\n", "# source: https://forums.fast.ai/t/how-to-save-dataloaders/73828/6\n", "\n", "# save\n", "# fname = 'dls_ptwiki_tokenizerGPT2.pkl'\n", "# torch.save(dls, path_data/fname)\n", "\n", "# load\n", "# dls = torch.load(path_data/fname)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6. Fine-tuning the model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*(text from Sylvain Gugger Transformers Tutorial)* The Hugging Face model will return a tuple in outputs, with the actual predictions and some additional activations (should we want to use them is some regularization scheme). To work inside the fastai training loop, we will need to drop those using a `Callback`: we use those to alter the behavior of the training loop.\n", "\n", "Here we need to write the event `after_pred` and replace `self.learn.pred` (which contains the predictions that will be passed to the loss function) by just its first element. In callbacks, there is a shortcut that lets you access any of the underlying `Learner` attribute so we can write `self.pred[0]` instead of `self.learn.pred[0]`. That shorcut only works for read access, not write, so we have to write `self.learn.pred` on the right side (otherwise we would set a `pred` attribute in the `Callback`)." ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "class DropOutput(Callback):\n", " def after_pred(self): self.learn.pred = self.pred[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 6.1 Splitter (get layers groups)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model has 2 main layers groups: `transformer` and `lm_head`. As we can read in [The illustrated GPT2](http://jalammar.github.io/illustrated-gpt2/#model-output), the `lm_head` is a copy of the embeddings matrix `wte`. Therefore, we need to split only the `transformer` layers group to get all layers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We decided to follow the fine-tuning method showed in the notebook [10_nlp.ipynb](https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb) by creating 4 layers groups: 3 layers groups of 4 decoder blocks and one embeddings groups with the wte and wpe matrices." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Split a GPT2 model in 4 groups for differential learning rates" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [], "source": [ "def splitter(model):\n", " \"Split a GPT2 `model` in 3 groups for differential learning rates.\"\n", " \n", " # First layers group : decoder blocks from 0 to 3\n", " modules = []\n", " for i in range(4): modules.append(model.transformer.h[i])\n", " groups = [nn.Sequential(*modules)]\n", "\n", " # Second layers group : decoder blocks from 4 to 7\n", " modules = []\n", " for i in range(4,8,1): modules.append(model.transformer.h[i])\n", " groups = L(groups + [nn.Sequential(*modules)])\n", "\n", " # Third layers group : decoder blocks from 8 to 11\n", " modules = []\n", " for i in range(8,12,1): modules.append(model.transformer.h[i])\n", " groups = L(groups + [nn.Sequential(*modules)])\n", " \n", " # Fourth layers group : embeddings matrices wte and wpe + LayerNorm at the model output\n", " groups = L(groups + [nn.Sequential(model.transformer.wte,model.transformer.wpe,model.transformer.ln_f)])\n", " \n", " return groups.map(params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 6.2 Learner" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*(text from Sylvain Gugger Transformers Tutorial)* Now, we are ready to create our `Learner`, which is a fastai object grouping data, model and loss function and handles model training or inference. Since we are in a language model setting, we pass accuracy and perplexity as metrics, and we need to use the callback we just defined. Lastly, we use mixed precision to save every bit of memory we can (and if you have a modern GPU, it will also make training faster)." ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [], "source": [ "# Learner: basic class for handling the training loop\n", "# source: https://dev.fast.ai/learner#Learner\n", "learn = Learner(dls, model_en, loss_func=CrossEntropyLossFlat(),\n", " splitter = splitter,\n", " cbs=[DropOutput], \n", " metrics=[accuracy, Perplexity()]).to_fp16()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can check how good the model is without any fine-tuning step." ] }, { "cell_type": "code", "execution_count": 97, "metadata": {}, "outputs": [ { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 4h 45min 53s, sys: 1h 1min 36s, total: 5h 47min 30s\n", "Wall time: 53min 2s\n" ] }, { "data": { "text/plain": [ "(#3) [9.949938774108887,0.09898579120635986,20950.939453125]" ] }, "execution_count": 97, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "# loss, accuracy, Perplexity() of validation dataset\n", "learn.validate()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Loss = 9.95\n", "- accuracy = 0.099\n", "- perplexity = 20950.94" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have a `Learner`, we will use during training all the **fine-tuning techniques** seen for classification model training (see the notebook [10_nlp.ipynb](https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb) about \"NLP Deep Dive: RNNs\") to take advantage of the **Transfer Learning** of the GPT-2 pre-trained embeddings and model from Hugging Face Transformers:\n", "- **learning rate finder** (method that helps finding the best learning rate to train the model)\n", "- **Mixed precision training** (some of the operations will be done in FP16, others in FP32 in order to speed up the training)\n", "- **gradual unfreezing** (the model has 4 layers groups created by our method `splitter` : the embedding one and the 3 groups of 4 decoder blocks each)\n", "- **1cycle policy** with the method [fit_one_cycle()](https://dev.fast.ai/callback.schedule#Learner.fit_one_cycle) (The 1cycle policy was introduced by Leslie N. Smith et al. in Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates. It schedules the learning rate with a cosine annealing from `lr_max/div` to `lr_max` then `lr_max/div_final` (pass an array to `lr_max` if you want to use differential learning rates) and the momentum with cosine annealing according to the values in `moms`. The first phase takes `pct_start` of the training. You can optionally pass additional `cbs` and `reset_opt`.)\n", "- **differential learning rates** (each layers group with a learning rate different: the biggest one for the embeddings group, and the smallest one for the first 4 decoder blocks)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 6.2.1 Freeze all layers but the last layers group (do not freeze `wte`, `wpe` embeddings matrices and last `LayerNorm`)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 4min 21s, sys: 58.9 s, total: 5min 20s\n", "Wall time: 1min 5s\n" ] }, { "data": { "text/plain": [ "SuggestedLRs(lr_min=0.005754399299621582, lr_steep=2.2908675418875646e-06)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhV5bn38e+9M5KQgYQEQpgFFWSUiFLn4lwrak+ttLZap9raqp3eY9u357ztOVU7D9rW0jq1Kq2tUrEialXAAdQgICigBETDlEBCQubpfv/YG4y4AwGzp+T3ua59Za+119rrzjbuH89az3oec3dERET2F4h1ASIiEp8UECIiEpYCQkREwlJAiIhIWAoIEREJSwEhIiJhJce6gJ40cOBAHzlyZKzLEBFJGMuXL9/p7gXhXutVATFy5EhKS0tjXYaISMIws81dvaZTTCIiEpYCQkREwlJAiIhIWAoIEREJSwEhIiJhKSBERCQsBYSISAJ7Y2sNS96qjMh7KyBERBLY/cs2882/r4rIe0csIMzsbjOrMLM1ndb91MzWmdnrZjbPzHK72PccM1tvZhvM7OZI1SgikuhqG9vITo/MPc+RbEHcC5yz37qngQnuPgl4C/jO/juZWRLwW+BcYDww28zGR7BOEZGEVdPYSk6/lIi8d8QCwt2XAFX7rXvK3dtCi8uAoWF2nQ5scPeN7t4C/BWYFak6RUQSWW1TK9mJFhDdcCXwRJj1xcB7nZbLQ+tERGQ/tY2tZKf3ooAws+8BbcAD4V4Os84P8F7XmlmpmZVWVkbmSr6ISLxKyFNMXTGzy4Hzgc+5e7gv/nJgWKflocDWrt7P3ee4e4m7lxQUhB2xVkSkV3J3apvayO6XeBepP8TMzgH+E7jA3Ru62OxVYKyZjTKzVOBSYH60ahQRSRT1Le20d3jitSDMbC6wFDjKzMrN7CrgDiALeNrMVprZnaFth5jZAoDQReyvAk8Ca4GH3P2NSNUpIpKoahtbASJ2DSJiEwa5++wwq+/qYtutwHmdlhcACyJUmohIr1DbFAqIRGtBiIhIZNU0BAMi4U4xiYhIZNU2BW8r61XdXEVE5KOraVQLQkREwth3kbo3dHMVEZGes7cFkaVTTCIi0lltUytZackkBcINQPHRKSBERBJUbWNbxLq4ggJCRCRh1TRGbiRXUECIiCSs2qbWiE0WBAoIEZGEVRvBkVxBASEikrBqdYpJRETCieRcEKCAEBFJSG3tHdS3tEdsmA1QQIiIJKQ9e8dhitBd1KCAEBFJSJEehwkUECIiCWnfXBA6xSQiIp3ta0FkKCBERKST2sbIzgUBCggRkYRUE+GhvkEBISKSkPZeg9BFahER+YDaxlaSA0a/lKSIHUMBISKSgPbeRW0WmbkgQAEhIpKQapsiOxcERDAgzOxuM6swszWd1n3azN4wsw4zKznAvu+Y2WozW2lmpZGqUUQkUUV6LgiIbAviXuCc/datAS4GlnRj/9PdfYq7dxkkIiJ9VW1jZOeCgAgGhLsvAar2W7fW3ddH6pgiIn1FbVNityA+CgeeMrPlZnbtgTY0s2vNrNTMSisrK6NUnohIbEV6siCI34A40d2PBc4FrjezU7ra0N3nuHuJu5cUFBREr0IRkRhxd2ob2yJ6FzXEaUC4+9bQzwpgHjA9thWJiMSPptYOWto7+l4LwswyzSxr73PgLIIXt0VEhE4juUZwmA2IbDfXucBS4CgzKzezq8zsIjMrB2YAj5vZk6Fth5jZgtCug4AXzGwV8ArwuLsvjFSdIiKJZt84TBE+xRSx+HH32V28NC/MtluB80LPNwKTI1WXiEiiq43CZEEQh6eYRETkwN4/xaSAEBGRTqIx3SgoIEREEs77kwUl6EVqERGJjPcnC1ILQkREOqltbCUjNYmUpMh+hSsgREQSTE1ja8S7uIICQkQk4dQ2RX4cJlBAiIgknNrGtojfRQ0KCBGRhFMThZFcQQEhIpJwapt0DUJERMKIxnSjoIAQEUkoHR1OXXObAkJERD5oT3Mb7pG/ixoUECIiCSVaI7mCAkJEJKFEa5gNUECIiCSU2ihNFgQKCBGRhFLdEAyIvMzUiB9LASEikkCq6psBGJCpFoSIiHRSVR9sQQzIUAtCREQ6qW5oISs9OeJDfYMCQkQkoVTVt5AfhesPoIAQEUko1Q0tDEj0gDCzu82swszWdFr3aTN7w8w6zKzkAPueY2brzWyDmd0cqRpFRBJNVX0LeVG4/gCRbUHcC5yz37o1wMXAkq52MrMk4LfAucB4YLaZjY9QjSIiCaW6vhe0INx9CVC137q17r7+ILtOBza4+0Z3bwH+CsyKUJkiIgmlqqElKvdAQHxegygG3uu0XB5aF5aZXWtmpWZWWllZGfHiRERipbGlnabWjqh0cYX4DAgLs8672tjd57h7ibuXFBQURLAsEZHY2hW6SS4vCjfJQXwGRDkwrNPyUGBrjGoREYkb1VG8SQ7iMyBeBcaa2SgzSwUuBebHuCYRkZiramgBojMOE0S2m+tcYClwlJmVm9lVZnaRmZUDM4DHzezJ0LZDzGwBgLu3AV8FngTWAg+5+xuRqlNEJFFU10c3ICI2JZG7z+7ipXlhtt0KnNdpeQGwIEKliYgkpKooB0Q8nmISEZEwqhtaCFh05oIABYSISMKoqm9hQEYqgUC4zp49TwEhIpIgojkOEyggREQSRjTHYQIFhIhIwqiqb4nKTHJ7KSBERBJEVX1r1HowgQJCRCQhuHvwGoROMYmISGe1TW20d7haECIi8kHRvosaFBAiIglh7zhM6uYqIiIfsK8FoWsQIiLSWbTHYQIFhIhIQqjWKSYREQlnV30LqUkBMlOTonZMBYSISAKoDt1FbRadgfpAASEikhCq6lujepMcKCBERBJCdUNLVC9QgwJCRCQhVNfHaUCY2RFmlhZ6fpqZ3WBmuZEtTURE9qqK4xbEw0C7mY0B7gJGAQ9GrCoREdmnrb2Dmsb4vQbR4e5twEXAr9z960BR5MoSEZG9ahpbcY/uTXLQ/YBoNbPZwOXAv0LrojdrhYhIHxaLm+Sg+wHxRWAG8CN332Rmo4D7D7SDmd1tZhVmtqbTujwze9rM3g79HNDFvu1mtjL0mN/dX0ZEpDfaVRf9cZigmwHh7m+6+w3uPjf0pZ7l7rcdZLd7gXP2W3cz8Iy7jwWeCS2H0+juU0KPC7pTo4hIb/V+CyK6J26624tpkZllm1kesAq4x8x+caB93H0JULXf6lnAfaHn9wEXHmK9IiJ9TlV9KxC/1yBy3L0WuBi4x92nAWccxvEGufs2gNDPwi62SzezUjNbZmYKERHp0/a1IKJ8iim5u9uZWRFwCfC9CNaz13B332pmo4FnzWy1u5eF29DMrgWuBRg+fHgUShMRia6q+hYyU5NIT4neQH3Q/RbED4EngTJ3fzX0xf32YRxvRyhoCP2sCLeRu28N/dwILAKmdvWG7j7H3UvcvaSgoOAwShIRiW/Bgfqi23qA7l+k/ru7T3L3L4eWN7r7pw7jePMJdpUl9PPR/TcwswGd7toeCJwIvHkYxxIR6RVicRc1dP8i9VAzmxfqtrrDzB42s6EH2WcusBQ4yszKzewq4DbgTDN7GzgztIyZlZjZn0K7jgNKzWwV8Bxwm7srIESkz6qub4n69Qfo/jWIewgOrfHp0PJloXVndrWDu8/u4qWZYbYtBa4OPX8JmNjNukREer2qhhZGF/SP+nG7ew2iwN3vcfe20ONeQCf8RUSioKquhdyM6A9e0d2A2Glml5lZUuhxGbArkoWJiAg0tbZT39JOfrxegwCuJNjFdTuwDfgPgsNviIhIBO29ByIvMy3qx+5uL6Z33f0Cdy9w90J3v5DgTXMiIhJBVfV7AyJ+TzGF840eq0JERMJ6PyDitAXRBeuxKkREJKxEbUF4j1UhIiJhVcewBXHA+yDMbA/hg8CAfhGpSERE9qmqb8EMcvpFvwVxwIBw96xoFSIiIh9W1dBCbr8UkgLRP6v/UU4xiYhIhFXVx2YcJlBAiIjENQWEiIiEpYAQEZGwqupbFRAiIvJB7k51jOaCAAWEiEjcqm1so73DYzIXBCggRETiVtW+gfoUECIi0klVfTOggBARkf1U1bcCCggREdmPWhAiIhKWWhAiIhJWVX0z6SkBMlIPOGxexCggRETiVFV9K3kx6uIKBxnN9aMys7uB84EKd58QWpcH/A0YCbwDXOLu1WH2vRz4v6HF/3X3+yJZa09pbe/glU1VLFyznWfXVWAGw/MyGJGfwYj8TCYW5zBxaA7Z6dEfuldEEkt1QwsDYnR6CSIcEMC9wB3Anzutuxl4xt1vM7ObQ8v/2XmnUIj8N1BCcD6K5WY2P1yQxIPW9g5eKtvFv1Zt5em1O9jd0Eq/lCROOXIg6SlJvFvVwFNv7GBXaOIPgCMKMpk2YgCnHlnISWMGkpPx4cDo6AjeRVmxp5ncjBSKcjQFh0hfsiuG4zBBhAPC3ZeY2cj9Vs8CTgs9vw9YxH4BAZwNPO3uVQBm9jRwDjC3p2tsbmvn1gXrOGv8ID42ZuAh7VvT2MqPF65jwept7G5oJSstmTPHD+LsCYM5ZWwB/VKTPrh9Qyuvb9nNqvd2s/K93Sxcs52HSstJChhThuUyICOF2sY2apta2d3Qys66Zto6gvM1JQeMq08ezQ0zx8TsfKSIRFd1fQsj8zNidvxYfNMMcvdtAO6+zcwKw2xTDLzXabk8tK7HtbU7z79dyeOrt7HghpMpyOr+tH7/b/4bzF+1lfMnFXH+pCGccuRA0pKTutw+JyOFk8cWcPLYgtCxO1hVvptF6yt5YcNOttU0kZWezPC8DCYUp1CYlUZhVhoFWeksWl/BnYvLmL9yC//1yWM4+5hBmH14AhH3YKCEe01EEkssR3KF2AREd4T7dgs7B7aZXQtcCzB8+PBDPlBmWjJ3fPZYLvzti3zjoZXc98XpBLoxc9PTb+5g3oot3DhzLF8/88hDPi5AclKAaSPymDYij2+eddQBt/3EpCIuOW4Y3//nGq67fzlZackMyklncHY6uRkpVNW3sL22ie01TSQHjONH5zNjdD4fG5PPmIL+JCepP4JIImlua6euua33XqTuwg4zKwq1HoqAijDblPP+aSiAoQRPRX2Iu88B5gCUlJSEDZGDGVeUzX9/8hi+O281v19cxvWnjzng9rsbWvjuvNUcPTjroNv2pONG5vHY107ikdfKeXNrLTtqm9le20R5dQN5makcPTiLU48soKG5naUbd/H0mzsACBgUZKUxODudwaFQGZSTTlFOOlOHDWDkwMyo/Q4i0j3Ve++B6N+3AmI+cDlwW+jno2G2eRK4xcwGhJbPAr4TyaJmTx/G0o27+PlT6zluZB7TR+V1ue0P//UmVfUt3HPFcaQmR/df5ilJAT5zXPdaSlt2N7KsbBebqxrYXtPItpomNlbW81LZLvY0tQGQnhLgwWtO4NjhAw7ybiISTVWhTi29tgVhZnMJtgQGmlk5wZ5JtwEPmdlVwLvAp0PblgDXufvV7l5lZv8DvBp6qx/uvWAdwVq55aIJrC7fzQ1zV7DwppPJDfMf5tl1O3jktS3c8PExTCjOiWRJH1lxbj8+NW1o2Nfqm9t4t6qB6+5fzpX3vso/rpvBmMKsKFcoIl2pDo3kGsturhH956+7z3b3IndPcfeh7n6Xu+9y95nuPjb0syq0bam7X91p37vdfUzocU8k69wrKz2F22cfS2VdMz9euO5Dr+9uaOE7jwRPLX3142OjUVLEZKYlM64omz9fOZ3kQIAv3PUK22uaYl2WiITs7Raf31sDIhFNHJrDlSeOZO4r71H6zvuNFnfne/9cw666Fn726clRP7UUKSPyM7n3i8dR29TG5Xe/QlllHW3tHfte317TxJwlZXziN88z49Zn+M4jr/Psuh00tbbHsGqR3q+6PvYtiHjtxRRTN51xJI+/vo3vzVvDv244iZSkAI+u3Mrjr2/j22cfFfenlg7VhOIc5nx+Gpff8wozf76YlCRjeF4G/dNTeL18N+4weWgOU4blMn/lVua+8h4ZqUmMHZRFUXY6RbnpDBuQwfmTiijMTj+kY9c3t9HY2s7A/t3vXizSF+yqb8EMcvvFbtQFBUQYmWnJ/GDWBK75cyl/en4TF0wZwvcfXUPJiAFcd+oRsS4vIj42ZiALbzqF5Zur2bSznk2V9VTWNXPjzLHMmlLMqFBPp+a2dpZtrOLZtTvYuLOeDZV1PP92JfUt7dyyYC3nTSziihNHMnVYbth7MTo6nI0761i0vpJF6yt5ZVMVLe0djC/K5vSjCzjtqEImD809YAutprGV8uoGtlQ3khQwcjNSGZCRwsCsNA1hIr1GdX0LOf1SYtpF3fbeWNUblJSUeGlpaY+937V/LmXJ25UcNTibDTv28MSNpzA8hnc1xit3551dDfxl6Wb+Xvoee5rbGDqgH4Oz0ynISiO/fypV9S1srKxn0856mtuCp7DGFvbn9KMLyc1IYfH6Sko3V9Pe4SQHjCMK+nN0URYj8jOpqm9m2+4mtuxuZMvuxn09sMKZNDSHjx9dyBnjBnHMkGwA2jqc1vYOtlQ3UhaqYevuRnL6pTCwf+q+YNmbZ+7Q1Brsg17X3EZLWwdDB2QwpjCTEfmZpOieEomC6x94jbXba3n2m6dF9DhmttzdS8K+poDo2tbdjZzxi8U0tLTzk09N4pLjhvXYe/dW9c1tPPJaOS9vqmJnXTM761rYVddMbkYqowdmMrogkzGF/TlxzECGDvhg2NY0tvLShp2s3lLD+u17WLd9D1t2N5KbkcKQnH4Mye3HkNDprKED+lE8oB8dHuztsbuhhfKqRp5bX8GK94KnxZICRntH+L/v7PRk6prb6OLlLiUHjHFF2cyaMoQLpgyhMOvQTqmJdNfsOctobe/gH1/+WESPo4D4CBau2cab2/bw9TPGaviKGGht7zjkf7HvrGvmuXUVbNxZT0pSgJSAkZwUoCgnndEFmYwcmEl2esq+wRAr65qp269Vkp6SRP+0ZPqnJ5McMN6taqCsso4NFXW88PZOVpXXkBQwTh47kJlHF3LsiAEcPTibpG7chS/SHWf/cgkj8jOY84Ww3909RgEh0sM2VNQxb0U5/1yxlS27GwHITE1i0tBchudlUJSbzpCcfhw5OIvJQ3P0jws5ZMf96N/MPLqQ2z41KaLHOVBA6CK1yGEYU9ifb599NN866yjKqxtZvrma5Zureb18N8+sq2BnXfO+bYcO6McnJw/hgslDOHpwlsJCDsrdqa6P7VwQoIAQ+UjMjGF5GQzLy+DCqe8PONzc1s6OmmZefaeK+au2MmfJRn6/qIzRAzM585hBnDV+MFOH5XZrYEjpe2qb2mjr8JjeJAcKCJGISEtOYnh+BsPzM/jUtKHsqmvmiTXbefKN7dz1/Cb+sHgjBVlpnDV+EOdMGMwJo/PVO0r22XeTXAzHYQIFhEhU5PdP47ITRnDZCSOoaWxl0foKnnxjO/NWbOGBl98lOz2ZmeMGMXNcIaccWaD7Ofq4vcNsxHIkV1BAiERdTr8UZk0pZtaUYppa23n+7Z08sWYbz62rYN6KLSQHjONG5nHBlCHMmjJEMwj2QdVxMJIrKCBEYio9JYkzxw/izPGDaO9wVrxbzbPrKnjqzR1855HV3PL4Wj41bSiXnTBco+32IfuG+tY1CBGB4I19JSPzKBmZx7fPPorX3q3mL0s38+DL73LvS+9w+lEFfOnUIzh+VJ56QvVyVQ0KCBHpgpntm472++c37wuJS+csY8qwXK47dTRnjR+sXlC9VHV9C6nJATJSu57jPhrUbUIkzuX3T+NrM8fy4s0f538vnEB1QwvX3f8aZ/9qCY+u3PKB4dmld9hW00RhVlrMW4oKCJEEkZ6SxGUnjODZb57Gry+dghnc+NeVnPGLxTyxelusy5MeVFZZxxEF/WNdhgJCJNEkBYxZU4pZeOMp3HnZNNJTkvjyA6/xlQeWf+AObklMHR3Oxsp6BYSIHL5AwDhnwmAe+9pJfPvso/j3mxWc+YvFPLpyC71pjLW+ZlttE42t7RxRmBnrUhQQIokuJSnA9aeP4fEbTmJEfiY3/nUlV977Ku9VNcS6NDkMZRV1AGpBiEjPGTsoi4e//DG+f/54Xt5UxVm/XMKcJWW6iJ1gyioVECISAUkB46qTRvH0N07lxDH53LJgHRf97iW1JhJIWWUd2enJDIzxMBuggBDplYpz+/HHL5Twu88dyzu76rngjhd44e2dsS5LuqGsop4jCvvHvIsrxCggzOxGM1tjZm+Y2U1hXj/NzGrMbGXo8V+xqFMkkZkZ500sYv5XT6IgK40v3P0ydy4u0wXsOBcvXVwhBgFhZhOAa4DpwGTgfDMbG2bT5919Sujxw6gWKdKLjBqYybyvnMi5E4u47Yl1fOWB19jT1BrrsiSM2qZWKvY0992AAMYBy9y9wd3bgMXARTGoQ6TPyExL5o7ZU/nueUfz1Js7mPXbF3lrx55YlyX72VhZD8ARBbHv4gqxCYg1wClmlm9mGcB5wLAw280ws1Vm9oSZHdPVm5nZtWZWamallZWVkapZJOGZGdeecgQPXH08tY2tXPjbF5m/amusy5JO9nVxLeyjLQh3Xwv8GHgaWAisAtr22+w1YIS7TwZuB/55gPeb4+4l7l5SUFAQoapFeo8TRufz+A0nM64omxvmruA3z7yt6xJxoqyyjuSAMTwvI9alADG6SO3ud7n7se5+ClAFvL3f67XuXhd6vgBIMbOBMShVpFcalJ3O3GtO4OJji/nF02/xg8fepKNDIRFrZZV1jMjPiJvpZ2My3LeZFbp7hZkNBy4GZuz3+mBgh7u7mU0nGGS7YlCqSK+VmhzgZ/8xmbyMVP70wiaqG1r42acnx82XU19UFidjMO0Vq/kgHjazfKAVuN7dq83sOgB3vxP4D+DLZtYGNAKXutrAIj0uEDC+94lx5PVP5ScL11PT2MrvPzeNfjGeh6Avam3vYPOues4cPyjWpewTk4Bw95PDrLuz0/M7gDuiWpRIH2VmfOW0MeRlpPLdeav5/F0vc9cVx5HTLyXWpfUp71U10NrucdWCUFtSRAC4dPpw7vjssawq382lc5ZRuUdDh0dTWZx1cQUFhIh0ct7EIv50+XG8s7OeT9/5EuXVGsMpWvYO0jdaLQgRiVenHlnA/VdPp6q+hUvuXMo7O+tjXVKfUFZRR0FWWlyd2lNAiMiHTBuRx4PXnEBjazuX/GEpGyp013WkBcdgip/TS6CAEJEuTCjO4W9fmkGHw2f+sIy122pjXVKv5e5x18UVFBAicgBHDsrioS+dQEpSgNl/XMbyzVWxLqlX2lXfQk1jqwJCRBLL6IL+PPSlGeT2S2H2H1/mMY3f1OPWbQuewouXMZj2UkCIyEENz8/gka+cyKTiHL42dwW/fW6Dxm/qQQvf2EZ6SoCSEQNiXcoHKCBEpFvyMlO5/+rjuWDyEH765Hq+9ffXqW/ef5xNOVRt7R08sXo7M8cNIjMtVoNbhKeAEJFuS09J4teXTuHGmWN5ZEU55/76eV59R9clPoqlG3exq76FT04aEutSPkQBISKHxMz4+plH8rdrZ+A4l/xhKbcuWEtTa3usS0tIj63aSv+0ZE47Kv6mK1BAiMhhmT4qjyduPIVLjxvOH5Zs5IxfLGbeinING34ImtvaWbhmO2cdM4j0lPgbIFEBISKHrX9aMrdePJEHrj6enH4pfP1vq/jE7S/w3PoKXcTuhuff2kltUxufnBx/p5dAASEiPeDEMQN57Ksn8etLp1DX3MoX73mVWb99kcdf30a7WhRdeuz1reRmpHDSmPicD00BISI9IhAwZk0p5plvnMYtF02ktrGV6x98jZk/X8TcV96ltb0j1iXGlcaWdv795g7OnTA4bidpis+qRCRhpSYH+Ozxw3nmm6fx+88dS06/FL7zyGrO/MVi5q/aqmsUIc+tr6C+pT0uey/tpYAQkYhIChjnTizin9efyN1XlJCeksQNc1dw/u0v8NQb2/t8UDy2aisD+6dx/Oj8WJfSJQWEiESUmfHxowex4IaT+dVnplDX3Ma1f1nOGb9czF9febdPdo+tbWrl2XUVfGLiYJICFutyuqSAEJGoCASMC6cW8+w3T+X22VPJSE3i5kdWc/JPnuP3i8rY09Qa6xKjZuHq7TS3dXDh1OJYl3JA8XVft4j0eslJAT45eQjnTyripbJd3Lm4jB8vXMfvFm3gCzNGcMXHRlGQlRbrMiNq3ootjBqYyZRhubEu5YAUECISE2bGiWMGcuKYgawur+H3izfwu0Vl/HHJJs6fVMTnZ4xgyrBczOL3FMzh2Lq7kWWbdnHTzCPj/ndTQIhIzE0cmsPvPjeNjZV13PfSOzz82hYeWbGFicU5fH7GCC6YPCQu7zQ+HI+u3Io7XDg1fnsv7RWTaxBmdqOZrTGzN8zspjCvm5n9xsw2mNnrZnZsLOoUkegaXdCfH8yawLLvzuR/Zh1DU2s7/+cfr3PCrc9wy4K1bN6V2PNjuzvzVpQzbcQARuTH1/Si4US9BWFmE4BrgOlAC7DQzB5397c7bXYuMDb0OB74feiniPQB/dOS+fyMkVx2wgiWbaziL8ve4a4XNjFnyUYmFudwxrhBzBxXyDFDsuP+NE1nb26r5a0ddfzPhRNiXUq3xOIU0zhgmbs3AJjZYuAi4CedtpkF/NmDg7ksM7NcMyty923RL1dEYsXMmHFEPjOOyGd7TROPrCjnmbUV/OqZt/jlv9+iOLcf50wYzHkTi5g6LJdAHHcZBfjnii2kJBnnTyyKdSndEouAWAP8yMzygUbgPKB0v22Kgfc6LZeH1n0oIMzsWuBagOHDh0eiXhGJA4Nz0vnKaWP4ymlj2FnXzLPrKnjqje38Zelm7nphE4Oz0zlnwmA+MamIacMHxF1YtHc4j67cymlHFTIgMzXW5XRL1APC3dea2Y+Bp4E6YBWw/7RU4f7Lhr3t0t3nAHMASkpK+vatmSJ9xMD+aVxSMoxLSoYFbzpbW8Hjq7fx4Cvvcu9L71CYlcY5EwZzytgCpo/OIzs9Jcq4ybsAAAlvSURBVNYl81LZTir2NHNRnN/70FlMejG5+13AXQBmdgvBFkJn5cCwTstDAc2ULiIfkp2ewoVTi7lwajF1zW08s3YHT6zezkOl7/HnpZsJGEwcmsvHjsjn5DEDmTZyAGnJ0e0R9eKGndz88Gpy+qXw8aMLo3rsj8JiMWa7mRW6e4WZDQeeAma4e3Wn1z8BfJXg6afjgd+4+/SDvW9JSYmXlu5/tkpE+qKm1nZWvLubpWU7ebFsF6ve201bh5OeEmD6qHxmjM7nuJEDmDg0J2KBUd/cxq1PrOX+Ze8yemAmP79kMlOHD4jIsQ6XmS1395Jwr8XqPoiHQ9cgWoHr3b3azK4DcPc7gQUEw2ED0AB8MUZ1ikiCSk9J2neB+xtAXXMbL2/cxfNv7+TFDTv58cJ1QHD02UnFOUwelsukoTlMLM5hZH7mIV/DqG1qZUt1I5t31bNxZz2bKut5ccNOttU2cfVJo/jW2Ucl3L0cMWlBRIpaECLSXbvqmlm+uZrSzdWUvlPFG1traW4LzlmRmhygMCuNQdnpFGalkZocoK3DaW932jqcto4OWts7aG13ahtb2bK7kT1NH7yUWpiVxthB/bnpjCM5bmReLH7FbonHFoSISEzl90/jrGMGc9YxgwFoa+/g7Yo6Vm+poayijoo9zeyobeLtijra2jtIChjJgQCBgJGaZCQnBUhJMopz+zF9VB7Fuf0oHtCPEXmZjCrIpH9a4n+9Jv5vICLSA5KTAowrymZcUXasS4kbGu5bRETCUkCIiEhYCggREQlLASEiImEpIEREJCwFhIiIhKWAEBGRsBQQIiISVq8aasPMKoHdQE2n1Tmdlvc+D7duILDzMA7b+b26+/rB1u3/ek/XfbCau1NjuHUHex6Lz7qr9YlY94GWVffB6zrY64dTd2/4Lhnr7jlhj+juveoBzOlqee/zLtaV9sTxuvP6wdZ19Tv0VN0Hq7kn6j7Q5x7Nz7o31d2dv23VHd26e9N3SbhHbzzF9NgBlh87wLqeOl53Xj/Yuq5+h56quzv7ftS6D/a5H47D+ay7Wp+IdXfnb7vzc9V9aK8fTt29+rukV51i+ijMrNS7GNEwniVi3YlYM6juaFPdsdcbWxCHa06sCzhMiVh3ItYMqjvaVHeMqQUhIiJhqQUhIiJhKSBERCQsBYSIiISlgOgGMzvZzO40sz+Z2Uuxrqc7zCxgZj8ys9vN7PJY19NdZnaamT0f+rxPi3U9h8LMMs1suZmdH+tausvMxoU+63+Y2ZdjXU93mdmFZvZHM3vUzM6KdT3dZWajzewuM/tHrGvpjl4fEGZ2t5lVmNma/dafY2brzWyDmd18oPdw9+fd/TrgX8B9kaw3VNtHrhmYBRQDrUB5pGrtrIfqdqAOSCex6gb4T+ChyFT5YT30t7029Ld9CRCVrpk9VPc/3f0a4ArgMxEst3N9PVH3Rne/KrKV9qDDueMvkR7AKcCxwJpO65KAMmA0kAqsAsYDEwmGQOdHYaf9HgKyE6Fm4GbgS6F9/5EonzUQCO03CHgggeo+A7iU4BfW+YlSd2ifC4CXgM8mUt2h/X4OHJuAdUfl/8mP+kiml3P3JWY2cr/V04EN7r4RwMz+Csxy91uBsKcHzGw4UOPutREsF+iZms2sHGgJLbZHrtr39dRnHVINpEWizv310Od9OpBJ8Muh0cwWuHtHvNcdep/5wHwzexx4MHIV7zteT3zeBtwGPOHur0W24qAe/vtOCL0+ILpQDLzXabkcOP4g+1wF3BOxig7uUGt+BLjdzE4GlkSysIM4pLrN7GLgbCAXuCOypR3QIdXt7t8DMLMrgJ2RDocDONTP+zTgYoJhvCCilR3Yof59f41gqy3HzMa4+52RLO4ADvXzzgd+BEw1s++EgiRu9dWAsDDrDnjHoLv/d4Rq6a5DqtndGwiGWqwdat2PEAy3WDvkvxEAd7+350s5JIf6eS8CFkWqmENwqHX/BvhN5MrptkOtexdwXeTK6Vm9/iJ1F8qBYZ2WhwJbY1RLdyVizaC6o011R1ei1t0tfTUgXgXGmtkoM0sleHFxfoxrOphErBlUd7Sp7uhK1Lq7J9ZXySP9AOYC23i/u+dVofXnAW8R7IHwvVjXmeg1q27Vrbrjs+6P8tBgfSIiElZfPcUkIiIHoYAQEZGwFBAiIhKWAkJERMJSQIiISFgKCBERCUsBIb2amdVF+Xh/MrPxPfRe7Wa20szWmNljZpZ7kO1zzewrPXFsEUD3QUjvZmZ17t6/B98v2d3beur9DnKsfbWb2X3AW+7+owNsPxL4l7tPiEZ90vupBSF9jpkVmNnDZvZq6HFiaP10M3vJzFaEfh4VWn+Fmf3dzB4DnrLgrHeLLDgL2zozeyA0/DSh9SWh53UWnNVvlZktM7NBofVHhJZfNbMfdrOVs5TgyKGYWX8ze8bMXjOz1WY2K7TNbcARoVbHT0Pbfjt0nNfN7Ac9+DFKH6CAkL7o18Av3f044FPAn0Lr1wGnuPtU4L+AWzrtMwO43N0/HlqeCtxEcP6H0cCJYY6TCSxz98kEh1y/ptPxfx06/kEHdjOzJGAm74/x0wRc5O7HAqcDPw8F1M1AmbtPcfdvW3AqzrEE5yyYAkwzs1MOdjyRvfrqcN/St50BjA/9ox8g28yygBzgPjMbS3DI5pRO+zzt7lWdll9x93IAM1sJjARe2O84LQRnEgNYDpwZej4DuDD0/EHgZ13U2a/Tey8Hng6tN+CW0Jd9B8GWxaAw+58VeqwILfcnGBixnB9EEogCQvqiADDD3Rs7rzSz24Hn3P2i0Pn8RZ1ert/vPZo7PW8n/P9Lrf7+Rb6utjmQRnefYmY5BIPmeoJzIHwOKACmuXurmb1DcA7v/Rlwq7v/4RCPKwLoFJP0TU8BX927YGZTQk9zgC2h51dE8PjLCJ7aguDw0Afk7jXADcC3zCyFYJ0VoXA4HRgR2nQPkNVp1yeBK81s74XuYjMr7KHfQfoABYT0dhlmVt7p8Q2CX7YloQu3b/L+DF8/AW41sxcJTkYfKTcB3zCzV4AioOZgO7j7CmAVwUB5gGD9pQRbE+tC2+wCXgx1i/2puz9F8BTWUjNbDfyDDwaIyAGpm6tIlJlZBsHTR25mlwKz3X3WwfYTiTZdgxCJvmnAHaGeR7uBK2Ncj0hYakGIiEhYugYhIiJhKSBERCQsBYSIiISlgBARkbAUECIiEpYCQkREwvr/rKryNsD2i3AAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "%%time\n", "learn.freeze()\n", "learn.lr_find()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The learning rate finder curve suggests picking a lr min of 6e-3. Let's use 2e-3." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracyperplexitytime
03.8033443.6407770.32517738.1214415:48:31
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "IOPub message rate exceeded.\n", "The notebook server will temporarily stop sending output\n", "to the client in order to avoid crashing it.\n", "To change this limit, set the config variable\n", "`--NotebookApp.iopub_msg_rate_limit`.\n", "\n", "Current values:\n", "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", "NotebookApp.rate_limit_window=3.0 (secs)\n", "\n" ] } ], "source": [ "learn.fit_one_cycle(1, 2e-3)" ] }, { "cell_type": "code", "execution_count": 101, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3xU9Z3/8dcnyeQGAUIIdzCgCAJCwIgoSrEoCmqxShV/unWxu3TVrsX+/FVa13qpWtatdtvdquu17dZbi/qgraCiglQFFBQhIMhFLuEaAgmX3JPv749MhtwJmUlOzuT9fDx45MyZM2c+35nhnW++c873mHMOERHxnxivCxARkZZRgIuI+JQCXETEpxTgIiI+pQAXEfGpuLZ8sh49eriMjIy2fEoREd9bvXr1Qedcet31bRrgGRkZrFq1qi2fUkTE98xsR0PrNYQiIuJTJw1wM3vezA6YWXaNdd3NbLGZbQ7+TG3dMkVEpK7m9MB/B1xeZ91c4D3n3BDgveBtERFpQycdA3fOLTOzjDqrpwOTgsu/B5YCd0ewLhERAMrKysjJyaG4uNjrUlpdYmIi/fv3JxAINGv7ln6J2cs5txfAObfXzHo2tqGZzQZmAwwcOLCFTyciHVVOTg4pKSlkZGRgZl6X02qcc+Tl5ZGTk8OgQYOa9ZhW/xLTOfe0cy7LOZeVnl7vKBgRkSYVFxeTlpYW1eENYGakpaWd0l8aLQ3w/WbWJ/ikfYADLdyPiMhJRXt4VzvVdrY0wP8C3BxcvhlY0ML9NMvW3GN8uPlgaz6FiIjvNOcwwpeB5cBQM8sxs+8B84BLzWwzcGnwdqu55PEPuOm5la35FCIiDcrPz+eJJ5445cdNmzaN/Pz8VqjohOYchXJDI3dNjnAtTdTQVs8kIlJbdYDfdttttdZXVFQQGxvb6OMWLlzY2qW17an0IiJ+M3fuXLZu3UpmZiaBQIDOnTvTp08f1qxZw4YNG7j66qvZtWsXxcXF/PCHP2T27NnAialDjh07xtSpU7nwwgv5+OOP6devHwsWLCApKSns2nwV4JWVjpiYjvFlhojU98Bf17Nhz5GI7nN43y7cd9WIRu+fN28e2dnZrFmzhqVLl3LFFVeQnZ0dOtTv+eefp3v37hQVFXHuuedy7bXXkpaWVmsfmzdv5uWXX+aZZ57huuuu47XXXuOmm24Ku3ZfzIUy7ezeAJRVVnpciYh0dOPGjat1nPZvfvMbRo8ezfjx49m1axebN2+u95hBgwaRmZkJwDnnnMP27dsjUosveuCj+3dj4bp9lFc4EnxRsYi0hqZ6ym2lU6dOoeWlS5fy7rvvsnz5cpKTk5k0aVKDx3EnJCSElmNjYykqKopILb7ogcfFVpVZXqFvM0WkbaWkpHD06NEG7ysoKCA1NZXk5GQ2btzIihUr2rQ2X/RnA7FV494aQhGRtpaWlsaECRMYOXIkSUlJ9OrVK3Tf5ZdfzlNPPcWoUaMYOnQo48ePb9PafBHgcTHqgYuId1566aUG1yckJLBo0aIG76se5+7RowfZ2aHZuLnrrrsiVpcvhlBCPfAK9cBFRKr5JMCDPfBK9cBFRKr5IsDj1AMXEanHHwEeHANXgIuInOCLAK8eA9eXmCIiJ/gkwKvHwNUDFxGp5osAPzEGrh64iLRvnTt3BmDPnj3MmDGjwW0mTZrEqlWrwn4uXwR4QGdiiojP9O3bl/nz57fqc/jkRB6diSki3rj77rs57bTTQvOB33///ZgZy5Yt4/Dhw5SVlfHQQw8xffr0Wo/bvn07V155JdnZ2RQVFTFr1iw2bNjAWWedFbG5UHwR4NU98LJyBbhIh7ZoLuxbF9l99j4bpjZ+UbGZM2cyZ86cUID/6U9/4q233uLOO++kS5cuHDx4kPHjx/Otb32r0WtaPvnkkyQnJ7N27VrWrl3L2LFjI1K6rwJcJ/KISFsbM2YMBw4cYM+ePeTm5pKamkqfPn248847WbZsGTExMezevZv9+/fTu3fvBvexbNky7rjjDgBGjRrFqFGjIlKbLwJcJ/KICNBkT7k1zZgxg/nz57Nv3z5mzpzJiy++SG5uLqtXryYQCJCRkdHgNLI1neoV55vDH19iajIrEfHQzJkzeeWVV5g/fz4zZsygoKCAnj17EggEWLJkCTt27Gjy8RMnTuTFF18EIDs7m7Vr10akLl/0wANx6oGLiHdGjBjB0aNH6devH3369OHGG2/kqquuIisri8zMTIYNG9bk42+99VZmzZrFqFGjyMzMZNy4cRGpyx8BHhwDL1WAi4hH1q078eVpjx49WL58eYPbHTt2DKi6qHH1NLJJSUm88sorEa8prCEUM/uhmWWb2XozmxOpouqKjwsGuI5CEREJaXGAm9lI4J+BccBo4EozGxKpwmqKVw9cRKSecHrgZwErnHOFzrly4APg25Epq7ZQgKsHLtIhOdcxDmA41XaGE+DZwEQzSzOzZGAaMCCM/TUqJsaIizEFuEgHlJiYSF5eXtSHuHOOvLw8EhMTm/2YFn+J6Zz70sz+HVgMHAO+AMrrbmdms4HZAAMHDmzp0xEfF6MAF+mA+vfvT05ODrm5uV6X0uoSExPp379/s7cP6ygU59xzwHMAZvYIkNPANk8DTwNkZWW1+FdofFyMxsBFOqBAIMCgQYO8LqNdCivAzaync+6AmQ0ErgHOj0xZ9QViY3QcuIhIDeEeB/6amaUBZcDtzrnDEaipQfGxMZRoCEVEJCTcIZSLIlXIycTHxeiCDiIiNfhiLhSo6oGXlld4XYaISLvhnwBXD1xEpBbfBHggVseBi4jU5JsA12GEIiK1+SbAA7E6kUdEpCbfBHhCnI4DFxGpyTcBrlPpRURq802A60xMEZHafBPg8RoDFxGpxTcBnhiIpVgBLiIS4qMAj6G4TGdiiohU81GAx1JUVhH1k7qLiDSXrwLcOV0XU0Skmq8CHKC4TAEuIgK+CvCqUks0Di4iAvgpwOPUAxcRqck/AR4cQilSD1xEBPBVgFeVqkMJRUSq+CjAq4dQFOAiIuDHANfZmCIigK8CvKrUolL1wEVEwFcBXtUDL9GFjUVEAB8GuMbARUSqhBXgZnanma03s2wze9nMEiNVWF2JcdVHoWgMXEQEwghwM+sH3AFkOedGArHAzEgVVldSvI4DFxGpKdwhlDggyczigGRgT/glNezEmZgKcBERCCPAnXO7gV8CO4G9QIFz7p2625nZbDNbZWarcnNzW15ojBEfG6MhFBGRoHCGUFKB6cAgoC/Qycxuqrudc+5p51yWcy4rPT295ZUCCbqog4hISDhDKJcAXzvncp1zZcDrwAWRKathiYFYBbiISFA4Ab4TGG9myWZmwGTgy8iU1TBdVk1E5IRwxsBXAvOBz4B1wX09HaG6GpQUiNUYuIhIUFw4D3bO3QfcF6FaTqrqyvTqgYuIgI/OxISqQwk1hCIiUsVfAR4fq8msRESCfBXgXZMCHCku97oMEZF2wVcB3i0pQH5hqddliIi0C/4K8OQABUVlVFY6r0sREfGcrwK8a1KASgdHNYwiIuKvAO+WHA9AfpGGUURE/BXgSQEA8gvLPK5ERMR7vgrw1E7BAC9SgIuI+CrAq4dQPtjU8mlpRUSiha8C/LTuyQC88PHXHlciIuK9sOZCaWtxsTF0io8lIXiBYxGRjsxXPXCA73/jdA4dL6VA4+Ai0sH5LsDHDOwGwKJ1ez2uRETEW74L8MwBVQG+Zle+x5WIiHjLdwGekhhgUI9O7M4v8roUERFP+S7AAcYP7s6KbXmaG1xEOjRfBvgFp/egrMKxavthr0sREfGMLwN8aO8UAOa8+rnHlYiIeMeXAX5mr6oAL6vQtLIi0nH5MsABArFGQVEZy7fmeV2KiIgnfBvgD199NgA3PLMC59QTF5GOx7cBft25A0LLf1y508NKRES80eIAN7OhZramxr8jZjYnksWdzCf3TAYg53BhWz6tiEi70OLJrJxzm4BMADOLBXYDb0SormbpmZLImb06s3n/sbZ8WhGRdiFSQyiTga3OuR0R2l+zBWJjeH/jAQ4d12XWRKRjiVSAzwRebugOM5ttZqvMbFVubuQvxBAfV9WEWb/7NOL7FhFpz8IOcDOLB74F/Lmh+51zTzvnspxzWenp6eE+XT2v33oBAF9ocisR6WAi0QOfCnzmnNsfgX2dMjNjWPDMzJueXelFCSIinohEgN9AI8MnbeVHl54JwIdbDlJSrgmuRKRjCCvAzSwZuBR4PTLltMyUEb05Pb0TABv2HPGyFBGRNhNWgDvnCp1zac65gkgV1FJP3HgOAN9+4mOPKxERaRu+PROzriE9O4eWH3tnk4eViIi0jagJ8JgY47xB3QFYuinyhyuKiLQ3URPgAK9+/3wA1u0uYNchnV4vItEtqgIcoFtyAICLHl3icSUiIq0r6gL8ldnjQ8srt2mucBGJXlEX4MN6d+H126rOzrz+6RXkF2qOFBGJTlEX4ABjB6aGljMfXOxhJSIirScqAxzgq4emel2CiEiritoAj4+L4Zox/QDYcuCox9WIiERe1AY4wNDgJFeXPL7M40pERCIvqgN89sTBoeWyikoPKxERibyoDnAzI8aqlifquHARiTJRHeAAn987BYC9BcVUVDqPqxERiZyoD/CuwTMzAZ5cusXDSkREIivqAxxg488vB+BIcbnHlYiIRE6HCPDEQCwj+nZh0z4dTigi0aNDBDiAc/DBV7nszi/yuhQRkYjoMAF+zdiqk3omzHvf40pERCKjwwT49y4cFFr+Yle+h5WIiERGhwlwM+Ox74wGYPpvP/K4GhGR8HWYAAe49pz+XpcgIhIxHSrAAZICsQBkzH3T40pERMLT4QL8F9ecHVr++uBxDysREQlPWAFuZt3MbL6ZbTSzL83s/EgV1lqmZ/YNLV/8y6XeFSIiEqZwe+C/Bt5yzg0DRgNfhl9S6zKzWhd7KCqt8LAaEZGWa3GAm1kXYCLwHIBzrtQ554vj8+LjYnjixrEA3P+X9R5XIyLSMuH0wAcDucALZva5mT1rZp3qbmRms81slZmtys3NDePpImvC6T0AeHXVLpZsOuBxNSIipy6cAI8DxgJPOufGAMeBuXU3cs497ZzLcs5lpaenh/F0kVVzlsJZL3zqYSUiIi0TToDnADnOuZXB2/OpCnTf+NGlZ4aWndNc4SLiLy0OcOfcPmCXmQ0NrpoMbIhIVW3kjslDuP+q4QD8z7JtHlcjInJqwj0K5V+BF81sLZAJPBJ+SW1raO8uAMxbtNHjSkRETk1YAe6cWxMc3x7lnLvaOXc4UoW1lfMGdQ8tD/23RR5WIiJyajrcmZh1xcQYj147CoCS8kq25h7zuCIRkebp8AEOcN25AzirT9VQyuTHPiDvWInHFYmInJwCPOivP5gQWj7noXc9rEREpHkU4EFxsTG1JroSEWnvFOA13DBuIFcHJ7v6wUufeVyNiEjTFOB13BK89Nrf1u5lwZrdHlcjItI4BXgdo/p3Cy1vP1joYSUiIk1TgDdg+7wrAPjVu19xpLjM42pERBqmAD+JUfe/43UJIiINUoA34o3bLggtr9p+yMNKREQapgBvxJiBqdwz7SwAZjy13ONqRETqU4A34Z8nDg4t5xeWeliJiEh9CvCT+OP3zgMg88HFLFy31+NqREROUICfxLgasxXe9uJn7Mkv8rAaEZETFOAnER8XQ9ekE5df+8Z/LPGwGhGRExTgzfDFfVP45KeTASircBwvKeeLXfkeVyUiHV2c1wX4Rc8uiWQO6MaaXfmMuO9tAAZ0T+LvP/6mx5WJSEelHvgpuGvK0Fq3dx3SeLiIeEcBfgpGD+hab11xWYUHlYiIKMBPSUpigO3zrmDDg5cxsl/VFXzufm0tlZXO48pEpCNSgLdAcnwcC26/EIAFa/Yw+KcLPa5IRDoiBXgLxcZYrduaL0VE2poCPAxbH5kWWp7x1HIef2eTh9WISEcTVoCb2XYzW2dma8xsVaSK8ovYGAvNHQ7wm/e3kDH3TQ8rEpGOJBI98Iudc5nOuawI7MuX5tW5GHL27gKPKhGRjkRDKBEwPbMfsyZkEBccF7/x2ZU4pyNTRKR1hRvgDnjHzFab2eyGNjCz2Wa2ysxW5ebmhvl07VNSfCz3XTWCzQ9PBaCgqIxBP1nIq5/u9LgyEYlm4Qb4BOfcWGAqcLuZTay7gXPuaedclnMuKz09Pcyna9/MjH/5xumh23e/to5vP/GRhxWJSDQLK8Cdc3uCPw8AbwDjIlGUn82dOozf33LiZfh8Zz7lFZUeViQi0arFAW5mncwspXoZmAJkR6owP/vGmelsq3GI4Rn3LNI0tCISceH0wHsBH5rZF8AnwJvOubciU5b/xcQYr916fuj2jrxCMua+SUFRmYdViUg0aXGAO+e2OedGB/+NcM49HMnCosE5p3Wvt270A+94UImIRCMdRtjKNv78ciYP68nUkb1D63Syj4hEgrXl8cpZWVlu1aoOd8JmyIEjxYx75L166x+9dhTXnTvAg4pExA/MbHVDJ0uqB96GenZJ5MeXD623/sevreV4SbkHFYmInynA29htk87gwekj6q2vvkybiEhzKcA98N3zM/jkp5O5Lqs/T900NrR+4bq9HlYlIn6jMfB24M21e7n9pc9Ct38+fQSPL/6Kz382xcOqRKS90Bh4O3bFqD61bt+7YD2HC8vIeuhd3sre51FVItLeKcDbiS3BibBqOnishH/542qccxSW6ktOEalNQyjtTM7hQlZtP8ycV9c0us3L/zye8YO7Y2aNbiMi0aOxIRQFeDtVXFbBd55azrpmXByi5lWBRCT6aAzcZxIDsfz1Xy/kkW+fzfmD05rcNmPumyzesL+NKhOR9kI9cB/ZsOcI6SkJnPvwu41uo964SPTREEoU2rTvKJf957J661+YdS4XD+0JwLGScvILS+mfmtzW5YlIhCjAo9iBI8W8+ukuHlv8VaPbTM/sy4I1e/jP6zO5eky/NqxORMKlMfAo1rNLIrdffEaT2yxYsweAOa+u4Z431gFwvKScrbnHdAFmEZ9SDzyK7D9SzI68Qq77n+VMOCONj7bkNetxP7r0TO6YPKSVqxORltIQSgd0+HgpR4rLSE9JIC4mhq8PHm9wzLymWyYM4mdXDQegotJx6eMfcMfkIRp2EfGQAlxCWnJBiaV3TSKjR6dWqEZETkYBLiEFRWXgoGty4JTD/OtfTKt1BmhxWQWf7TjMBWf0iHSZIhKkAJeTOlZSzsg685IP7J7MzkOFzXr8/VcNp0tSgE+3HyY1OcD4wWkUl1UwZUTvkz9YRBqlAJdmOVZSTnIglpiYE71s5xyDfrKwxfvMfuAyOifERaI8kQ5JhxFKs3ROiKsV3gBmxvZ5V/DAt+pfSQhg1oSMJvc58r63yZj7Jm9l7+W3S7bw1f6jnPfIu2TMfZNZL3xSa9t1OQWUlFeE1QaRjiLsHriZxQKrgN3OuSub2lY9cP9buG4vI/p24bS0+l9oVo+nr3/gMuLjYvhgUy7/9IdTf7+vHdufx64bDcAbn+fQp2sS408yH0xzOedwjnq/pETas1YbQjGzHwFZQBcFeMdWUVn1WYqtEY7lFZXc/MInzT4mvTH3TDuL4rIKHlv8FedmpPJfN4yld9fERrcvKCrj0bc2MueSM0lPSQitnzDvffqnJvHK7PEcOFpCz5QETcsr7V6rBLiZ9Qd+DzwM/EgBLk355OtDvLY6h5H9unDvgvW8PWciv/v4a17+ZFdom/GDu7Ni26Fm7W/qyN4kBmJ54/PdoXVvzbmINTvzmfv6utC6V2eP55FFG/nm0J786t360w1UTwBWUek4eKyETglxOOfonBCHmVFaXsmOvOMM6ZXS0qaLhKW1Anw+8AsgBbiroQA3s9nAbICBAwees2PHjhY/n0Sv8opK4mJjKK+o5Ix7FgEwoHsSP58+kn984dNWfe4B3ZOYd80obnx2ZZPb3TbpdC44vQc3PXdiu8V3TiQ9JYHMBxdzxdl9+O2NY5vYg0jLRDzAzexKYJpz7jYzm0QjAV6TeuDSHEWlFSzfdpBvDusFwL6CYmY89TH/eX0mYwem8uv3NvPr9zafdD+3TBjE8x993drl1nN6eie25h7noatHUlBUxiVn9WJo76Z771tzj7HzUGFoFsnG5B0rYW9BMSP7dY1kydLOtUaA/wL4B6AcSAS6AK87525q7DEKcImEykrHexsPcOnwXpRVVBKIjeEHL31Gv9QkXlqxk0dnjGLq2VUXis49WsL7G/fTvVMC727YzyPXnF1rjL7uiUzrH7iMT7cfYlvucR7824aI1ZyaHOBwYVmtdXdMHsLGvUd4p8bFOL64bwpdkwLsLSjieEkFK7blMT2zLymJgVr1/u/3xrFiWx7fHNaTrknxnNGzc619HzhSTGqneAKxzTvQ7HhJOTFmJMXHhtNMaSWtehy4euDiV8dKytm07wg/nr+We68czqQ6PeCDx0p4/8sDjD0tlUse/wCoCvnk+Fim/vrvbNx3FIAP776YJ5Zu5aWVO1ulzhf+8VzO6NmZix5d0ug2i354EXe/tpYRfbuEvleIj4thXEZ37r1yOL/7+GvO6JnCe1/uZ8zAbvy/y4Zx4Ggx4x5+L7SPX1xzNj95fR2/v2UcWw8cY9aEDD7aksdpackM6F41p/yXe48QF2P0S01i+M/e5vvfGMzdlw0jJsYoKCrjk68PcenwXrVq25F3nIS42Ca/eG7MvoJiuiTFkRzfcc8lUICLhKmgsIyjJWWhi2McKS5jX0ExPTon0L1TPAD/8fZGTk/vTFrnBOJijLP7d2XU/e94WbYnnrxxLD94+XP+8YIMrs7sx1X//SEAC++4iECscfBYKTc8syK0/egB3fhiVz4PXT2Sm8afxr+/tZEnl27lu+efxh+WV31vdqpXmzpaXEYgNobEQP2/KlZuy+P6p1fw9pyJ9EtNqnWiWfWFUmac059ffmc0t7/4GReckcY9b2QD8KvrR7N8ax7Tzu6Dc3DxsBO/9D/acpAbn13Juz/6Rr2/isKhMzFFPFQ91HPoeClJgViWbDrAbS9+BsDff3wxA7onszYnn+c+/Jqdhwp54FsjGNSjEzvyCrnyvz6sta/P772U837xHqXllXx276WM/fnisOs7f3Aay7eFd6hnW7ln2ll8kZPP39bu5Q+3jGPimelA1TDQsq9yuTX4ulYb2iuFTfuPNrnPP37vPLolB/jdx9uZvzrnlGtKjo+lsLT2CWjXjOnH49dnUlRaQXllZWgYrCUU4CLtTFFpBYFYI64Z49Sl5ZWc+W+LuOSsnjx787kN7uvBv63noiHpjBnYjT5dk6gMHhaZnpLA2+v3kV9YRlJ8LHExMWzYW8Bvl2wFYPW/XUJa5wQqKx2VzvH9/11NflEZB44WU1JWyR//6Tym/KrhaYinjuzN0eJyPtxyMLwXowNYeMdFDO/bpUWPVYCLSC2l5ZXszi9i0ClME7x6xyEGpCZz4GgJyzbnctukqitBFZaWM/xnVROhffXQVD7acpBJQ9PZkVfI7z7ezpxLhrBgzR7KKip56M0vAfjbv15Ij84JdEmKo7C0gh6dExh539scKyln5U8n06tLImUVlcSasXrnYb7z1PJm13l91gDuuGQIE+a93+R2P58+gnsXrK+3PiUhjqMl5QC8c+fE0C+wLQ9PZWvucXp3SWTjviP8n2dXhk5gq/bTacN4ZOHGevvc8vDUZv2ybogCXERa1eodhzmzV+eTDhWUV1RS4RwJcad2xMszy7ZRUFTGfy/Zwi+/M5qhvVK4689fhIZHnrxxLJ0T4+iSGGD0gG4AvPrpTpLj47hqdF9e/XQnYwamUlhawcdbD4Z++ew6VMjDb37J+5sOcO8VZ/EP52cAVdMu1DxLt+7tmuvv+vNaMgd0DT0W4IanV7B8Wx4Zacn83ylDuWp031Nqb00KcBERn9JshCIiUUYBLiLiUwpwERGfUoCLiPiUAlxExKcU4CIiPqUAFxHxKQW4iIhPKcBFRHxKAS4i4lMKcBERn1KAi4j4VJtOZmVmuUBLL0vfA+hIkw6rvdFN7Y1ukW7vac659Lor2zTAw2FmqxqajStaqb3RTe2Nbm3VXg2hiIj4lAJcRMSn/BTgT3tdQBtTe6Ob2hvd2qS9vhkDFxGR2vzUAxcRkRoU4CIiPtXuA9zMLjezTWa2xczmel1POMxsu5mtM7M1ZrYquK67mS02s83Bn6nB9WZmvwm2e62Zja2xn5uD2282s5u9ak9dZva8mR0ws+wa6yLWPjM7J/j6bQk+tv4lwttQI+2938x2B9/jNWY2rcZ9PwnWvsnMLquxvsHPuJkNMrOVwdfhVTOLb7vW1WdmA8xsiZl9aWbrzeyHwfVR+R430d728x4759rtPyAW2AoMBuKBL4DhXtcVRnu2Az3qrHsUmBtcngv8e3B5GrAIMGA8sDK4vjuwLfgzNbic6nXbgrVNBMYC2a3RPuAT4PzgYxYBU9the+8H7mpg2+HBz28CMCj4uY5t6jMO/AmYGVx+CrjV4/b2AcYGl1OAr4Ltisr3uIn2tpv3uL33wMcBW5xz25xzpcArwHSPa4q06cDvg8u/B66usf4PrsoKoJuZ9QEuAxY75w455w4Di4HL27rohjjnlgGH6qyOSPuC93Vxzi13VZ/2P9TYlycaaW9jpgOvOOdKnHNfA1uo+nw3+BkP9jy/CcwPPr7ma+cJ59xe59xnweWjwJdAP6L0PW6ivY1p8/e4vQd4P2BXjds5NP0CtncOeMfMVpvZ7OC6Xs65vVD1gQF6Btc31na/vSaRal+/4HLd9e3RD4JDBs9XDydw6u1NA/Kdc+V11rcLZpYBjAFW0gHe4zrthXbyHrf3AG9o/MvPxz1OcM6NBaYCt5vZxCa2bazt0fKanGr7/NLuJ4HTgUxgL/BYcH3UtNfMOgOvAXOcc0ea2rSBdb5rcwPtbTfvcXsP8BxgQI3b/YE9HtUSNufcnuDPA8AbVP1ptT/4pyPBnweCmzfWdr+9JpFqX05wue76dsU5t985V+GcqwSeoeo9hlNv70Gqhhzi6qz3lJkFqAqzF51zrwdXR+173BnjfbUAAAFbSURBVFB729N73N4D/FNgSPCb2nhgJvAXj2tqETPrZGYp1cvAFCCbqvZUfwt/M7AguPwX4LvBb/LHAwXBP0/fBqaYWWrwT7cpwXXtVUTaF7zvqJmND44dfrfGvtqN6iAL+jZV7zFUtXemmSWY2SBgCFVf2DX4GQ+OAS8BZgQfX/O180TwdX8O+NI593iNu6LyPW6sve3qPfbqG97m/qPqm+yvqPoW9x6v6wmjHYOp+vb5C2B9dVuoGgd7D9gc/Nk9uN6A3wbbvQ7IqrGvW6j6gmQLMMvrttWo62Wq/qQso6rX8b1Itg/ICv5n2Qr8N8EzidtZe/832J61wf/QfWpsf0+w9k3UOLqisc948DPzSfB1+DOQ4HF7L6TqT/y1wJrgv2nR+h430d528x7rVHoREZ9q70MoIiLSCAW4iIhPKcBFRHxKAS4i4lMKcBERn1KAi4j4lAJcRMSn/j/bhjUBDwV19gAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "#hide\n", "learn.recorder.plot_loss()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In just **one epoch**, our model passed \n", "- from an accuracy of 9.90% to **32.52%**\n", "- from a perplexity of 20950.94 to **38.12**\n", "\n", "Not too bad! \n", "\n", "Now, We can pass `-2` to `freeze_to` to freeze all except the last two parameter groups." ] }, { "cell_type": "code", "execution_count": 105, "metadata": {}, "outputs": [], "source": [ "learn.save(path_data/'GPT2_pt_1epoch_lr2e-3')\n", "learn = learn.load(path_data/'GPT2_pt_1epoch_lr2e-3')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 6.2.2 Freeze all layers but the last 2 layers groups" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracyperplexitytime
03.4539133.3018860.36287927.1638165:38:18
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.freeze_to(-2)\n", "learn.fit_one_cycle(1, slice(1e-3/(2.6**4),1e-3))" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "scrolled": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deXgUVdbG35NOZyWQkAQIBAj7FgKEiOw7GkDBBRUV95FxG3c/wV0cNcO4z6iMzui4ITq4C4ggmyJb0BD2PUhYQyCBhKyd+/3RVd3V1VXV1Z3O0p3ze548VFfdqr7VCW/dPvfc95AQAgzDMEzgE9LQHWAYhmH8Aws6wzBMkMCCzjAMEySwoDMMwwQJLOgMwzBBQmhDvXFCQoJISUlpqLdnGIYJSDZv3nxKCJGodazBBD0lJQXZ2dkN9fYMwzABCREd0jvGIReGYZgggQWdYRgmSGBBZxiGCRIaLIbOMAzjLVVVVcjPz0d5eXlDd6XOiYiIQHJyMqxWq+lzWNAZhgkY8vPzERMTg5SUFBBRQ3enzhBCoLCwEPn5+ejUqZPp8zjkwjBMwFBeXo74+PigFnMAICLEx8d7/U2EBZ1hmIAi2MVcxpf7DDhBP1dehW9yjjR0NxiGYRodASfos77YivsW5GDH0bMN3RWGYZoYRUVFeOutt7w+b9KkSSgqKqqDHrkScIJ+rLgMAHC+srqBe8IwTFNDT9BtNpvheYsXL0ZsbGxddctBwGW5NJX4GcMwjY9Zs2Zh//796N+/P6xWK5o1a4akpCTk5ORgx44duOyyy3D48GGUl5fjvvvuw8yZMwE4rU5KSkowceJEDB8+HL/++ivatWuHb775BpGRkX7pn0dBJ6IIAGsAhEvtFwohnla16QDgAwCxACwAZgkhFvulh+r+SP9y4TyGado8+912v4dee7dtjqcv7aN7PCsrC9u2bUNOTg5WrVqFyZMnY9u2bY7Uwvfeew8tW7ZEWVkZLrjgAlx55ZWIj493ucbevXvx6aef4t1338XVV1+NL774AjNmzPBL/82M0CsAjBVClBCRFcAvRLRECLFe0eYJAJ8LId4mot4AFgNI8UsPVcgDdC6FyjBMQzNo0CCXPPE33ngDX331FQDg8OHD2Lt3r5ugd+rUCf379wcADBw4EHl5eX7rj0dBF/Yq0iXSS6v0o5ZTAaC5tN0CwFF/ddCgX3X9FgzDNGKMRtL1RXR0tGN71apVWL58OdatW4eoqCiMHj1aM488PDzcsW2xWFBWVua3/piaFCUiCxHlADgJYJkQYoOqyTMAZhBRPuyj87/oXGcmEWUTUXZBQYFPHSYp6MJyzjBMfRMTE4Nz585pHisuLkZcXByioqKwa9curF+/XrNdXWJK0IUQNiFEfwDJAAYRUaqqybUA/iuESAYwCcBHROR2bSHEO0KIDCFERmKipj+7ZzjkwjBMAxEfH49hw4YhNTUVjzzyiMuxzMxMVFdXIy0tDU8++SQGDx5c7/3zKstFCFFERKsAZALYpjh0m7QPQoh10kRqAuwjer8iT4q+tnwPhnQZ4u/LMwzDGDJ//nzN/eHh4ViyZInmMTlOnpCQgG3bnNL58MMP+7VvHkfoRJRIRLHSdiSA8QB2qZr9AWCc1KYXgAgAvsVUPPbH/u+Gg6fr4vIMwzABi5mQSxKAlUSUC2AT7DH074loDhFNkdo8BOB2ItoC4FMAN4t6mrWsqREoreBFRgzDMGayXHIBDNDY/5RieweAYf7tmjYE14VFl/zjF+w4dhY75lyMqLCAWyfFMAzjNwJu6X/OYVc/hB3H7AsLzpbxKJ1hmKZNwAl6WZXTM+GFxTsd2+Ghvt3KtiPF+Gi9bhFthmGYgCHgYhTRYRaUVtpF/Z01Bxz7bT6G7C/5xy8AgBsGd6x95xiGYRqQgBuht4vTNrGp4cR0hmEaGc2aNQMAHD16FNOmTdNsM3r0aGRnZ/vl/QJO0PVgPWcYprHStm1bLFy4sM7fJ+AEXZ3lIlNZXVOr6x4oKPHciGGYJs2jjz7q4of+zDPP4Nlnn8W4ceOQnp6Ovn374ptvvnE7Ly8vD6mp9gX2ZWVlmD59OtLS0nDNNdf41csl4GLoenbory7bg1eu6e/zdQ8UlKJzYjOfz2cYpp5ZMgs4vtW/12zTF5iYpXt4+vTpuP/++3HXXXcBAD7//HP88MMPeOCBB9C8eXOcOnUKgwcPxpQpU3RrN7z99tuIiopCbm4ucnNzkZ6e7rfuB9wIPUTnQ/ry99rVGQ3zMUuGYZimw4ABA3Dy5EkcPXoUW7ZsQVxcHJKSkvDYY48hLS0N48ePx5EjR3DixAnda6xZs8bhf56Wloa0tDS/9S/gRuhGCCF8rmgUExFUHwXDBD8GI+m6ZNq0aVi4cCGOHz+O6dOn45NPPkFBQQE2b94Mq9WKlJQUTdtcJXVVeS3ghqVG2SzT5q3z+nox4XYhLy6r8rlPDMM0HaZPn44FCxZg4cKFmDZtGoqLi9GqVStYrVasXLkShw4Zr2sZOXIkPvnkEwDAtm3bkJub67e+BZygj+vVSvfY5kNnXF7nnSpFeZVx8dZzkg/Mze9vqn3nGIYJevr06YNz586hXbt2SEpKwvXXX4/s7GxkZGTgk08+Qc+ePQ3Pv/POO1FSUoK0tDTMnTsXgwYN8lvfAi7O0DI63HMjADuPncXE138GAORlTa7LLjEM08TYutU5GZuQkIB167SjAyUl9uy5lJQUh21uZGQkFixYUCf9CrgRuln2nNCuKlJeZUNNjbmk9S2Hi3CqpMKf3WIYhqkzAk7QPbny3vHRZhwpcs3r3H38nOPcnk/+gM6PLTb1XlPfXIspkjUAwzBMYyfgBN0TP2w/jmFZK1wmT698+1f8uv8U7p7/m+nrVFTbY+9Hi41nqxmGqV+aSoF4X+4z4GLoZnngsy2ObSEErnvXta510flKxEaF6Z5f25WnDMP4n4iICBQWFiI+Pr7OUv8aA0IIFBYWIiIiwqvzglbQlcjujEq2HinGiG7ahapragTeWrW/rrvFMIyXJCcnIz8/HwUFdVLhslERERGB5ORkr84JOEGXv4X8cP8IZL72s8/XsYToP93nLt2NeatZ0BmmsWG1WtGpU6eG7kajJWBj6MlxUbU63yJ9XdMSdhZzhmECkYATdAH/TIgs2XYcZZU22KQUxgtS4vxyXYZhmIYi4ARdprbTIf/9NQ+7FbnqVTb9B0VJRTVqagSOFfvP5pJhGMbfBJyg+zNjyaKYJa+y6We1nCmtxD9W7MOQF1dgr86CJYZhmIYm4ARdRpmxdMeoLj5do7DUuQq0Whqha+V+Wi0heHX5HgDAhFfX+PReDMMwdU3ACfrU/u2w8I4hCA+1OPY9OKE77h7jvagrDbmqauwj9Aof8s//9EE2lu/Q9z9mGIapDwJO0Nu0iEBGSkuX7BSrhWAJqd2tyCGXc+XVbscGv/iT4bnLd57Anz70T5FXhmEYXwm4PHQtiAihBnnlZqi2CUx8/WfsPHbWq/OayjJkhmEaPwE3QtfDaKGQGapswmsxB4C/L91tuu3K3SexTArNVFTbcOIs+8QwDOM/AnqE/vzlqSiRQiTVOmmH7WIj3dwXtTDKcjHCG4uAW6SY/cEXJ6HHEz8AAN65YSAu6tPGp/dmGIZREtCCfv2FHR3bZTqViSpNCnV9lqBT5ryv3lPgtaBvP1qMxJhwtIrxzriHYZjgxmPIhYgiiGgjEW0hou1E9KxGm1eJKEf62UNERXXTXX3OlWsLsq8jb3+w8eBpfJ971G1/dY2zT3qFOIyY/MYvGPfy6lr1jWGY4MNMDL0CwFghRD8A/QFkEtFgZQMhxANCiP5CiP4A/gHgS/931ZiwUN+nA67JaO+2b96Mgfjl0TGa7U+XVpq67tX/Wod75v+OgnOuVY9siopJeYXnTffTViMc52pl4+ghhMCKXSd4ApdhghyPKijslEgvrdKPkTJcC+BTP/TNK6wW+61c3Kc1bh6a4tgfqpHOGGZx3ZccF+nWJjLMomsAlv7cMny68Q/TfbvxvY0ur5XxfrXYGzFy7kqkPr3UZd/UN9di+N9WALA/aLTK63226TBu/W82/pedb/q9GIYJPEwNa4nIQkQ5AE4CWCaE2KDTriOATgBW6ByfSUTZRJTtbz9jWcgGdozDQxd1d+yff/uFbm3DVaP5tftPubWxWoyzZmZ/udX0SF2dPVOtEt2zOuEimfwz55EyaxGOFJW5zBVUVNuw5XAR8s+U4cTZcqQ/twz/XLnP7fyj0qTwMa6+xDBBjSlBF0LYpHBKMoBBRJSq03Q6gIVCCM0ZSiHEO0KIDCFERmKidnEJX7FJ4YQQIpdKJt1bx7i1VRc6WX/gtFub/QWlHt9TL26vxcTXnd7tNpWgj5q70vDcL387orlfzpQBgOOSWL+ybI/pPjEME1x4FXgWQhQBWAUgU6fJdDRAuAVwmnYRkaETY1yUFSEmctbN2OmqhdkI5ShdOSkKAGfOV+GkQU56hNXzr8moGpfcy0qbdiYQwzDBgZksl0QiipW2IwGMB7BLo10PAHEA1vm7k2aQi0JbSF/cvrprKJbeP9KU9W7HltEAgIcV4Rv393R9fcZkCGb439xH5INecLcXWLL1GFJmLcKeEyVux9QYPVx+/8OedPRNjnvGDcMwwYOZEXoSgJVElAtgE+wx9O+JaA4RTVG0uxbAAtFAqRSuI3RtyR7QIQ6tmke4FZf986jObm3lrJkJvfVzxNUj7Yznl6OmRuDxr7Zix1HvV52qeelH+yrUhZs9T2Yu3nrMsV1R7ToS79e+BQAgLblFrfvEMEzjxePCIiFELoABGvufUr1+xn/d8h0iZ1Wj6DCLZht1xKW1YoHO9Ava48Ur+jpE32jis1LlzGirEThaXIZPNvyBTzaYz4JRsmzHCazddwrtYiNNxfFl3v35oGN7z/ES9FWIt+xMGWHV/jwYhgkOAnqlqJKkWLsox0eHI9JqwZ2ju2BKv7YAgPG9Wrv4pqhH6DcNTcGc73c42iqPG8Wvz1e6x6S1winecLsfXBuPFJW5CHqU9GBrEWmt9bUZhmm8BI2gzxzRGZ3io5GZ2gZEhEczezqO/fumDJe26oCMJYTwwPjueHX5HiTGhLsc6xhvj6XHRIS6LeYprTC/uMcM/opWhaseQiHSA0orR51hmOAhaNwWQy0hmNg3yW30rUWIoo0cfvnL2K747p7h6Nc+1qWtLLKhIYQv7hzicuzdnw/Usteu6PnONI/w7rl7i6JwB+C8R5ZzhglugmaE7g2yns+bkY60ZLuAh4SQS5hCJjrc/hHdNDQFrZu7mmFp5a/XhmFZmuuxcNaLZf5ayGmaW/KL3Y7ZagRCyD0MxTBM4BE0I3RvkEfofdq2QNtY92X/SiKsFuRlTcb947sjOS4Kc6b2qbN+nSoxl/boLbJUbzns6plWWV2DLo8txp8/2lwn78swTP3SJAVd9m6xWry//RuHpGimOepxdUay1+9RX8jpjT9yPVSGCQqapKC/PWMg/nndALRp4Zuf+CzFhCsATEzVz1VvHtHwmSV6i46Os7cLwwQVTVLQW0aH4ZK0tj6fT0SOVED5enoM75bg8/v4C51iTpj65lrH9qFC8znvDMM0TpqkoPsDi2ISsW87/RWYI7v514TMF2w12tkzyjz6M+frr2ITwzB1Awu6j5xT5KB3adVMt53SCKyVKsd9VPe6EfskVShJmQ2ZnXcauflFbjnvi6TKSuVVNvz5o2wcPGUfsd+/4HekzFpUJ/1kGMa/sKD7AXXBDD3aqQppfHDrIBx8cZJf+zKlX1s3v3flCH3avHWY8s+1uHdBjksb2Tpg48HTWLr9BJ76ZhsA4Oscp9AzDNO4YUH3A2GhIXjzunTHa72Y+u0jOuNvV/YFYC9xB3if/92+ZSQmpyUZ9kXtMaMuqAEA323Rdl6UM38qVNfwpuQdwzANQ5NcWORvwkJDMDktCXfPt7+WwxlqG4FxvVohPNSCay7o4PN7VVTVOPLoe7aJwa7jrkWmw0JD3FacerPkPyzUfu0qW42j0hFgtzlQ3w/DMI0LHqH7ATnksnbWWAD2NMHlD47E8gdHubSzatQ39ZYbBneEXB1v5sjOWP7gSJfj4aEhqKhyCnpxWRXeWOFelk4P+RvD738UYahi5WqJj74158qrcPcnv+HkOU6RZJi6hgXdD8je6XLsuryqBl1bxbi5G+pVSrp/fDeP7/HcZanY9Vwm7hnbFRbpwWCrES5FsP88sjPCQkNQoRih553yLh3x9eV7NfefOa+9ivVceRV+2qm/MOnr349g0dZjePh/uV71g2EY72FB9wPyCF0Wdj2TLT0yDRYmyVw3qAMirBYQEeQ5WFuNgFUxATp7Ui+EW+wx9LEvr4IQwrA0nRohBFbv0S7erecLf927G3DbB9k4XlwOIQSqVPcuP8TW6FyXYRj/wYLuB2RRNZvtoqZnm+ZY+fBowzbKwb1jhC4ErKpRv/xQOVBQio/XH/Kq7JxRGbtCHZ+ZrUfshl97T57DhFfXoNvjS1yOt4yyTxBn9vH80GIYpnbwpKgfkEMtEVYLnrqkN8b0bOX1NaLDjasJKbNhlCN0dRhHrk4EAE9+s92rPmhlw8johVxkrJYQ7Dtpr31abatBqNTJGkdpQK+6wjCMD7Cg+8hfL0tFta0G1w/u6GLydevwTj5d72yZ+UnHUEUMPaGZa+ZJWKjvX7pqDApsFHoogK3Mfd+YdxpDu9gtD+RygCzoDFP3cMjFR2YM7oibh3Uy5djYzoNFL2DPL1ejJ85y2qJWiGTjQe882nslNccVA9oB0PdjB4DPNh12s99Vopyczc4749iuURTvZhimbmFBrweW3D8C62aPNWyjDJXIqBcIycj2v/HN3BcweZsrPqF3a6RKXjRGfi62GuFi5qVGmZGpfNDIOfAhLOgMU+dwyKUeaB5h9clG9+qMZHyene+2/6ahKWgbG4GLpYnGnx5y5rsP6RKP//6aZ/o9moVbYNFJp/QGpf+XHLo5V16Fw6fPA3Cv48owjP/hEXojYt6MgVjx0Ci8dFU/AMA9Y7Tz0y0hhMxUZ/3ULonN0CXRbhDmbaZNlc19YtWIN1dqL1KqUii6PELv+8yPeHnZHgDAseIyh7gzDFM3sKA3IjJT26BzYjNMG5iMvKzJbguTzOBtZCMuKgyhXgj635fudmyfOOtc/XnFW78anrcp7wxGzF3pXecYhvEKFvRGjC9OAXqTjx1aRmnun5jaBqWqZf2tm4cjqUWE5kStzNnyKlz4wk+ax5pHWrFVoyC1Wf4oPI+UWYsw6wvn6tKi85U44yHThmGaOizojRhfYtt6Z1wmZbIouX98N8RFhyFHlb1SI4B1s8fh8Okyt3MA4EhRGT5Ym6fbh6wlu3DpP3/RPX7pP35ByqxFuqZh8rkLNh127Os/ZxkGPLcMALBk6zGkzFqEYoNJ3CNFZThQUKJ7nGGCERb0RowvmSF6D4G7x3Rx2ze+V2sA7l7n6gIZaoZlrXDExn1BXl2qZ5FQXGZcPWnemgMAgP2n9AV7WNYKjH15tY89ZJjAhAW9EePTCF3jlIV3DNFMiwyVbBuX7zzpsv/uMV29fl9f+M8vB306R86HN1gHxTBNEhb0RozFhxG61qg+Olw7O1WeDFU/NyZII/exPlgYeEJp3rX9qHucfcUufedGAHju+x2KV6zoDKOEBb0R4006oUzPNjFu+2KjjLNlhnVN0HxftXOiP1CGUxZvPe52/Nb/Zpu+Fo/QGcYVj4JORBFEtJGIthDRdiJ6Vqfd1US0Q2oz3/9dZcwQGxWGJfeNcNmX1EI7W6Wy2q6ISj/256b2cWyry9D5A3W8vjYPDT09L6t0voc31ZoYJtAxs1K0AsBYIUQJEVkB/EJES4QQ6+UGRNQNwGwAw4QQZ4jI/9/VmzAPjO/uVftIq7Zz4/s3X4DOidEY9fdVAIDOidEAgE4JzRxtLuwc79hWCqO/UF+z2iag012P6In1E19vc2zbhEAIr1NlmggeBV3YC2TK6QRW6Uf9P+l2AG8KIc5I55wE4xfysiZ7fY6eqZfa1jdCUlJ5clTNXy9LxQuLd2KDl4ZfRry4ZJfL6+qaGgC+KbpNJ+by5e9Ou4TaPDAYJtAwFUMnIgsR5QA4CWCZEGKDqkl3AN2JaC0RrSeiTJ3rzCSibCLKLijgCjZ1RbgHC92VD4/GN3cPc7xWrhRVamS/9rH47M9D/Nq3Fbtcn/U3vrcRBecqdNsfK9bOhQdc/WOUTO3X1rH9zLfbsb+gBGfLjVMhGSYYMCXoQgibEKI/gGQAg4goVdUkFEA3AKMBXAvg30QUq3Gdd4QQGUKIjMTExNr1nNElQjEkndq/rdvxTgnR6Nfe+evxtdKSmu//Mtzrc37/owgXPL9c9/ju4+d0jylH6KdKKlAkFeFQrpb9LPswxr28GhNf+9nrvjFMoOGV26IQooiIVgHIBLBNcSgfwHohRBWAg0S0G3aB3+SvjjLmiQ4PxcqHR8NqIVNe7KEKQe/eupluu0cu7uHi5aLG0zcDT+w6ftZtn9HErE0xRM/4q/2hkJc1WTNifqRIf6TPMMGCmSyXRHm0TUSRAMYD2KVq9jWAMVKbBNhDMAf821XGGzolRCM5LsrrwhJa7bu1aoap/du6jfb7tG3u8ro21ZIA4Mmvt7ntq6yu0Z38PFBQqn2hWs6Bfrguz6O1AMM0Rsz8D0wCsJKIcmEfcS8TQnxPRHOIaIrUZimAQiLaAWAlgEeEEIV102Wmvln24Ci8Pn2A2/5F97qmR9ZG0FfvKdBcGZscF4nzVdrZNn9dtNPNWAwAyAdF/7+FWzBv9X4AwFNSLdZjZ3lUzwQWZrJccgG4/W8WQjyl2BYAHpR+mABkztQ+6JygH24BtEfvvz05AemSaVaEhr2AWW56b6NmtaXLPdjyfvX7EcwY3FHVT+/eu/h8laOQyB2jnJ43epOuDNNY4ZWiDADgxiEpGN4twbBNhMYIvGW0swxeZFjt8gONsl302HfS1aBr25Fi3fH5w//bgmqNhUzXvLPOsa0sn3fmPNv1MoEFCzpjmvhmxvVKIwwSvr3Np79MIztHi//+mgehyHbJP1OmO0JfuDkfe0+6OzTuUmTSvLbc6SLZysv6rL5yvLhc09eGYbyFBZ3xCi1hHtDBLUO11hw4pTPhqYHShveZb7cberzYPFgBrNnjXB9h9IDyJ0OzfsLkN/T94xnGLFwkmqk1X9451PC4VWclqhH9kmORa7LqUaUitfH42XJUG4j2/oISpLZroXtcOU9QUw/uX6dLK8F2M4y/4BE6U2uIyDA9cs3/jfHqeu1bRuLGIR09N5Sosrkq4le/H9Ft2zzC6Ty5ek8B1h9wTcZSVm/yNJr3B8oQz/lKZ8bOibPlGPvyKi6szXgFCzrjNXOm9sG8GQNNtc3oGOdwe5w7Lc3UOT//31i016mBqoU3wtsswvml9Kb3NmL6O+t128rXTZm1yJHSCABbDhdhUe4xAEBJRTVy84s0zzeD0r++37M/OrYXbs7HgYJSzN/4h8/XZpoeLOiM19w4JAWZqW0M2wyXPNaVxl9pyc5Qx7wZA/HtPcPczpPxZtVpjRCI0SniocabKEp1jXBkxWQt2YXTpZU4X1mNqW+uxd3zfwMA3PnxZkz551o3W2BzfREOuwLA9ZvGTzvthT58KXLCNF1Y0Jk6QU5hbKvwYleORsf0TERasvtk6nUXdgCgnfMu8+VdQ/HIxT0cr3VXjGogC/TsL7d6bGurES7WA+nPLcMz3253vC4+X4XfDp0B4F4ftaLahpRZi/D68r261//3zwfxdc5RzWO//WEf9ftQ44RpwrCgM35lxmC7IKd3iMPcaWmYc5nTx00pTnKN0/7tXUV9sMKPXY/0DnG4Mj3Z8frad9frriZVI0+YfmoilGGrES4TroA9FCLTb86PKJX83dX2BKdK7CPvV5frF9NetUfbZVrpDMnzpYw3sKAzfuWi3vZQzKBOcbg6oz2aKUIhWqPue8e5FqQ2OyJVZ85oxdGbR7iHYaq9WP5ZXSNcSuYB+oW71Zk1Z0qNFyVVVtcg5w/t2HthifNcb714mKYNCzrjV0Z2T8Su5zIxsGNLt2NaBaw3Hjzj0/voCauS3GcudttXbTM/5q0RAqNfWuWyL0FncdV177pOrp4qMV71+tdFOxyjezXKW+MYOuMNLOiM39FbkKMlTsqFPGbolWR3eIw2OQmqxihHXc3Js+6irC6oLbPnhOsK1KcVsXYtth3RzrHPzS/yyVyMYQAWdKYe0Rps3jIsxatryH7tVh+LcnhTlFrOZFHSwSCdUmlBoFyUtPmQewm/33TCLSXl1dimsAEQHEVnvIAFnak3QqRYgjJcok5/VKYVTk5LcmzPHNkZr1zdD3OmOidZR3gwE9OitouFjM5Xph0ePu203jUqCqKGiBAX5TQ88/XBxTRN+K+FqTfkkItyoK72UFfKZeuYCADAwI5xeGxSL1yRnowWkc6Vnmbi6Gq+zjnqMb5txNLtx3WPHdTxn1GuPvUEkeuEr1Haox5llTZs9GNhbyZwYEFn6g1Zf5VxbGuI/p+gNdR+wvherTWPe2MPILNmT4GjXJ0v7DKocXrxa2s0LYB7tGnutu/aQe01r/Hz3gKXz6fSVoNVu7XTG5XMW70fK6V2s77MxdX/Wsdl95ogLOhMvaHllx4SQsjLmowVD43CiG4JGNXNWTx8TI9WAIChXbRz01s3j/Bb35Ja+OdayhxyGa3vEbGKsIqSN1fud8tpf2RhrsvrW97fiI/XHwJg94N/bfkeZC3ZhVvet5fwlf1pSsrt3jDr9he6LIhighcWdKbeiJGMsbQEunNiM3x024VoEeUMqQzuHI+8rMno17729rwd4429YZ66pHet3wPQTs1UuzaWV9nw9qr9LvtevqqfY1udiaOO26/cXYAnpPqr1767Hq+pwjInpOwcOSR143sb8N9f83BO42HjDUrzMKZxwoLO1CvZT4zHezdf4JdreZPe98mfLjQ8rpx8fGB8d5/7JDTMYo6qQh9Xvm1cVk8t4OrFTTJVthpDDxk5Fi8L+3mdvHczfLQuD72fWsruj40cFq14EYwAABwhSURBVHSmXkloFu63whHxzbTDFnrva0SohTA5LQlEwH3ju6F3knvc2wxqK1/AaQMgs/3oWcNrLJOMuWRsNQLvrjng1m6ahweDGm9y8NU8KRXO/mGb/qQw0/CwoDMBS+vmEZh/+4WYqOH8KOe3f3zbhbiwU0uP7o1hlhC8eV06Dr5or8hk9dB+zSPaHu8Xv7bGLQYOAF/+lm8ohspIzfwN7j4zzy/eidOllUiZtcixb0t+sZvZi2suvLzP/q9WPVVvqeLK2Y0arljEBDRDuyRg1W77atMIawjKq+yCI5t8De+W4LH4NeAu4FYPKZFtY/UnUbVE78HPtwAAdsxxtyMAYKpq0cLNh932qU97/SdnPF0Wd7lNbUboMp6+6TANC4/QmYBHHhHfPdpp9GUmuv713U4/9lCVgBst6Ll3XDfNyU8ZI7+Y5xftdNuXlzXZVGz6hcW73PaVVLhOVConSNX67Y8KTC11snOYxgELOhPwXNDJbgQ2qFNLjOtpT3U041IYqYjlqwU81KAOakp8lGPVqxZLDEIrp0srNYtqT9fJS68NjvCL9I83tgd6+Oqhw9QPLOhMwHNxnzb4/ckJuLBzvCO8YGaErkwn3HnMdaIyTCXwyhi8pxWqD/9vi+6xs+VVjhWwSpIUhUD8hSOGLn0qthqB99ceRMqsRaio9i3jRWstAdN4YEFngoK4aHsoYNpAe+GLXm09Z6koQxDtYl0FVTlC3/PXibgqw1lQwxfLAZm1+wqRV+i0COjROsbna3ni4tfWAHBm3vy6vxBvSDH2c+Wec8rPlldhU95plCnSHeujcDbjOyzoTFAxqW8S8rImuwm0Fkpx6qVKU1SGYMJCQxCqsChQ579fO6iD5vUv7OTuCQ847QPmTkvD0gdGeuynv1i775Tjns08kjJfXYOr5q1DYanTzkArz55pPLCgM00WZR67OstFPUmqnAQ9Xerq19JGx4IgRqNikky/9rG4OsP/cXMjEpqF46w0MtdbrKTkaHE5ADgyhwD3EXppRTV+NDAsY+oXFnSmyZIc57QDUAv4WVVIQhmCsagMxXLztd0UbTUCgztrj9LV7+crmX3cc/CVaHnLAK61UbVQ5tIr/dzVEZfHvtqKmR9txp4T+qZlTP3hUdCJKIKINhLRFiLaTkTParS5mYgKiChH+vlT3XSXYeoGdZbLil1258JJfe2CqRyhq4tOFOrUD+3XPhYLZg7RPFabOLySHzyMjt9QpDEOUXjoGKVdAq45649+sdWxrfalOVBgnw+oja0A4z/MjNArAIwVQvQD0B9AJhEN1mj3mRCiv/Tzb7/2kmH8QOvm+oti9AQ2ItQiHXfuU68E1dPmP43orP9+GoL6y6PO1adyZSYAiKpFZkmlTTtcovahV6M3+akWdFn4/fGNo9pWg9IKNgCrDR4FXdiRCyZapR+eGWECjuYRVs+NVIQ6DK6c/1XUWqeeUHWcayByWnnuyhBQVJgz/q4s6uEtSv2tttWguRTXf2XZHsxbvR97T5zTnOhUrnaVc/sBd6GX7QSM8vbN0vXxJejz9NKgmXgtraj2S+6/N5iKoRORhYhyAJwEsEwIsUGj2ZVElEtEC4lIc7aHiGYSUTYRZRcUeFccmGFqy3SdbBQjlmy1hzSUI2q1qClL5SmRBV1rubynkIvVQo4smeE6hanN8Hm20y7gX2sOuIzEspbswoRX1+DfPx90O8+mWO2q9G6XR+iy6MojdE8WL2rHSSM2BEm1pT5PL8Wg530vpuILpgRdCGETQvQHkAxgEBGlqpp8ByBFCJEGYDmAD3Su844QIkMIkZGYmKjVhGHqjFuHpeDgi5Pw4ITuuCAlztQ556QQgHIE2q99C5c2WjYBNw9NQai0/6eHRmGsYpQLAEfOGAtclU04snBsJkasH9+mbQ9cUe1U2vwzZaisdlfebI0i1soRelmVMwxSUwMcKixFp9mL8X3uUccI1Cg//X/ZhzE0a4VmsexDhaVYtsPVXbKwRHtOIhA5c752HvTe4lWWixCiCMAqAJmq/YVCCDmX610AA/3SO4bxI0QEIsK947rhf3cMNWzbOTHa5bVyEnFgR9fMFa2xdn9FUY4WkVb849oBLsf3nixRn+JCzuEiPH9ZX/xlbFfNEXq0Iq6eEh9lyoAMcBV4I5Q2wMoJT5sQyM0vBmD/9iL71lQbDNHXH7AL+f4C95qr415ejds/zHYJs5RX2VBaUe3VqJ6xYybLJZGIYqXtSADjAexStVF+55wCwN2BiGEaIekaviqAe7xdndmiRMs3pktiM5fX0eGhePv6dK/6Fhcdhocu6uEQUCWdFA+cvEJ3Y6+Mjua+gegxcu5Kx/Y+xcNHCOEYjVtCyCHk+06WIGXWIs36p1/8Zk+R1LIwlkM2Hyssg8uqbLj6X+swNGtFre6hIVE+oI4V19+DycwIPQnASiLKBbAJ9hj690Q0h4imSG3ulVIatwC4F8DNddNdhvEv828fjM1PjHfbn9rOPtE5a2JPAK4xZTXqBUSPTeqJvskt3NoZGXoZ0UojO+eKAckaLZ1kHzpj+vpa83bKEEq+Ijxkq7ELLgBsPVLsGMlvlt5vzvc7HG3vW/A7nv1uu+Jc/c/wSamkHgAkxoR7LAICALuPn8NDn2/BFx5y6tXcPf83XP7WWgD2bwPX/GsdTuuknvqK8l4PaHwzqSs8WqcJIXIBDNDY/5RiezaA2f7tGsPUPRFWi2YFJXmAJYc2jLzE1WmFesWrPeV+A8DAjnHYfOgMtj/r9E2/c1QXtI6JwGUD2mH8K6tx8FQpRvVIRM/sGIeNQG1QpyIaYRMCs7+056UfPFXq+HxkQVeK1zc5R13OfXPlPlyR7nwQTXr9Z833iDHp6Ch71XzxWz6uHGj8gFOyKPeYS582HDyN9OeWIS9rsulreGKL4ltVfSbtsBcmw2jg+D8oibDW6LJ183BHQWZ/8cWd7rF9InIIlvzwiQqz6ArFrucy0fPJH0y/Z5kXi4LUKYWl0rme5gQAeyz+hv9swJR+bfHyj3tw/Gy5Zjtl7nzR+UpYLSEO215bjUBldY2b62NpRbVP1r51IbbVthqXurGxUb6nnXoLL/1nGA1GSBORae3soRN5hD6+lzNb5au7huH16f3dRt56I3F/5Fe/c8NAPD6pF5JaRDpG1k9f2tulTYTVgpuHpjhee8pjL62sRt6pUkfB6XM6dgFA7dwWjxWX4+e9p/DIwlxdMQdcJ277z1mG0S+tcrx+8PMc9HrK/WHV5+mlPvXp043u5f5qi3pOg0foDNPATOybhK3PXIQYaXJUFuN0xWRj29hITO3fDoUlrqN0vciKP/5ft28ZhdtHdna5npwe+fikXsg/4z5B6il0v/dECUa/tAptW0Tg19njcMJAbM3quZkKTHp8vP6Qy+uCc87PVx3GMcNXv+cjhAhT+7dzO6Zn21Abrv/3epfXRhlA/oZH6AyjQ4zJlaXxzcLx3T3DMb5XawD6I3St4tG14ZiU1ic/UG4f2RnPTlUvEfGcCy1Pch4tLkfKrEUY/8oa3bZm72GEIkvGWyamai/U8qUfAPDAZ1tw34Icj+1SZi3CW6v2eWy3Ke+07vunzFrkFoarTw95FnSGMYGcmqj39blvcguH34vegNjf/6/l+LWyjqiMeqLW25RJPWqEcDy46gozNgL3feZZoH1h7g+7DY9/sTkfV81bhwmvrjZ9TRZ0hmlkyGELozi4fEgv5OJNNkltuWdsV5fXZtIAzWATwqM/ye5aZt6YGX1/t8Vc6MWTmF7ar62p68j8vNduWaK1SEqPDQdP44dtxzw39AMs6AxjgssG2OOvE/vqhwNuHpYCwDXOrqSuBF1rcZTS3AsALunnOYxhhse/2oYiD8Ux5HRCXzFjdWCW99e6+9TInDhbjh6Sq2U/jXUDWih9bczyyrI9uOPj37w+zxdY0BnGBN1bxyAva7LbClAlQ7skIC9rMlppFIEGXAVdmS1TWwZ1ivfYRun8OKVfW6x+ZDQm9PYcOln+4Ci3fVsOaxf08BfKCknesP5AoctkbGV1DX77Q3+B1YUv/ORYGGW0zkBJpwT7Ct1R3X3zovK1OLdZWNAZpp6QR81XpLfDW9f7z+5IWbhCiycm94Iysj97Uk90jI/Guzdm6NY9lemi8rSpD55TrDb1hunvrHeZjE19eikWb3UWALl63jq30ntyRo+WaZkWT39rX/m64WAhTpdW4owiS8ZTWuonGw6hxxM/OCyH6wJOW2SYeuKi3q3xwuV9cUV6O48FJrzB5iEtbkS3RN3URU/90PKpMSIqzNJoqhdVqoRzY95pRwxcZsEmu73wXpVfjfq+D58+j69+P+J4ffmAdkh/bhkAOFaYehrlP/vdDke/QjUcOv0BCzrD1BNEhOsu9N6T3RMXpBiPsnu0iUHeKecknnIgqWX9WxvMhi4aCtLNQXJiqxEItRB+3luA85U2/GX+724PhzCNz63awO8HcH4LqMusFw65MEyAEx3meVzWMT4K3VrZ4//Kghty7VR/YTZ0ocXDF3X3Wz/W7NEuoPPyMuO0RMCeXvrTzhO44T8b8eePNruJOQD8ur/Qsf3Kj/Zrml1A9Pelu3ULi9cWFnSGCXDMuDgSEZY9OAp5WZP9Gu4B7MWw/cE9Y7t5bKNlwavFje9t1NxvxvmwrMqG2z7INmyjDNG8scK+GMnTCF3mw3WHsPOYf9JI1bCgM0yA4il8897NGbh7TJe674gPaYbNvDTS2l9QgmpbDfolGz885CyU2vDCIu/LORw8VYoBUkzdDP4OdcmwoDNMgPLC5X0NLV/H9myNRy7u6dO1J/RujUX3DjfV9qI+bdAi0opLdGqraiGX1zPLuJdX46Uf97iUxtPi4KlSrNQosuENnynqsJpljMJATOYig7TQupoUZUFnmCaMVrRm2sBkvHRVP/Rpa19s8909w/HTQ+756DLtYiOx5emL0NmL0bEvi6w25Z02Fda45f1NXl+7LnjnxgzdY2Em7A18gQWdYZowWgkXL13Vz8Vyt29yC8MFVRbpqeDNCk9fFoNuPnRG03bg2kH+zRzyMlPTJ44W6Tta1gYWdIZpwnRtpS/UZglxFAExf46vq/vVpeLysiZjU95p3y6mQ31Y7mhlzvgDFnSGacJo5VProTcSlkfm6pGtUfKNrx7hJ89VYIrKUKtXUnOfrlWXtNaoA6uEJ0UZhvE76hTGbgYj9hev6OvyurckpOol75FSmbwl943UvE5iTLjbaD5DMjSLDnOv76omVPWkuEUyRdMjPtp7Q63a8MujY/DrrHGGbeoqqsOCzjBNmH9c61r/3Wz2yb1ju2KHlEutDnncM7Yrds7JREsdIV376FiXEfqIbglYKNVS/eDWQR7f+7tcV+tcT98y1CZk3nwr8YWW0WGOeQU9OrSMqpP3ZkFnmCZMe5WwhIaYk4SBCruBYxoTfJFhFpeRtDwCj4kIRVhoiG62SoYHGwMADodEGU/hC/XCqw9v8/zQMMvyB92/hZgJp4zq4ZtboydY0BmGcXBVRrKpds3CnaERWV7VY1KLIjVPdoTM7NMGgLEp2N+npQEABnaMw+J7R3jsi6cKR+rBsreLmvQIs4Sga6sY9/6YWLlrqaNUGhZ0hmEcGKUnKokOD8WIbgkAgMGd7aNqeXJUznpRipYc5pDbfDZzsON8NfIkZ5sWEbophB/eOghf3mUP01g9fKtIiXfNjw8PDTFckGUWvUwVMw6VZuwafIEFnWGaON/d41wRanbg2Cw8FG9MH4BpA5MxY3BH+z7JJCwuyp7DrowjW6URuTx/2q11DJ6+tA8AYHhXV2FPbdcCr1zdD1lX9NVNIRzZPRHpHeKkaxt3+uoL2uPj2y50vPbX4HhcT/8VKfEXbJ/LME2cvoryayfPVaCPiXOiwkIRFx2Gl67q59g3c1RnxESE4qqM9gBcQw/yttI6tmurZvh11lgktXCv8HRFuj30s+5AvtuxJy/p7fLaU9w/NIQwXPFtQOk2KTNvRjoqqmtw3wLPxafXPDIGryzbjdmTenlsW9+woDMMg9eu6Y/7P8vBkM6ey9kB2q6H4aEW3Dysk+O1coQeLcWt1atJ28ZGGr6PVkWlfQqnQwCw6sTQ505LQ2KzcEelqPWzx+GP0+fd6oLOntgTmalJbsUv9GjfMhKvTR/guSGAn/9vDGqEwH0LcpBTx6X7ABZ0hmFgL4ItF8I2Q4TVc764MpbcPMIehqnxsrhD99buk47lVa4VkfSMrlrFhGN0D2dYpE2LCLTR+DaQInnQmC08YRQj3/f8RJfXchZRm+badWb9DcfQGYbxGk951jJ5WZORlzUZzSPtY0e93HQ9tN5GbewVZbUgPjoM16vshD2FYiKs9uMDpZTKAe3jvOqbFnqfy8xRnWt9bTOwoDMMU+cM75qAv13ZF49P9i7uHKIxGlZXRQoJIWx+coJjclbGUzqjvKJVbqVMs3z3xgxc7sU3Fhm90bs8gVvXeBR0Ioogoo1EtIWIthPRswZtpxGRICJ930iGYQKWaQOTcddo74tmEBGuuaCDI55t/jz3fe104u7q0fH+ghLNdjKO/HnpTZSTuF0So03lkwPACgNrYS3mzUj3qr03mPl0KwCMFUKUEJEVwC9EtEQIsV7ZiIhiANwLYEMd9JNhmEaAMqulPlCPeG8d1gmzJmoX7VDLb6sY47j136f1w8s/7kbzCLsMKr8NWELIdHpjZ5O5+zKZqeYLgXiLR0EXducd+VFnlX60Zg+eAzAXwMN+6x3DMIyCpy7trXtMLcC92xq7ME7o3drF50WZLRNChDiNeP+lKqfHxoap7z9EZAGwGUBXAG8KITaojg8A0F4I8T0R6Qo6Ec0EMBMAOnTwryk9wzBNG/Vo3mzIRO/8O0Z2wb9WHwAAPDShO3q0icFFknWBL7x/ywVI1MiB9yemJkWFEDYhRH8AyQAGEVGqfIyIQgC8CuAhE9d5RwiRIYTISEysG3MahmGCC7PL9NvFRjrCJ0DtLGrDrSGIjXJWbfrLuG61EnMAGNOjFVLbtfDcsBZ4leUihCgCsApApmJ3DIBUAKuIKA/AYADf8sQowzD+4uI+rXUXEMlEWC3IfeZi/Gm4fXGTVsjELCFEpjxZZDL7tNGdrK1PPIZciCgRQJUQooiIIgGMB/A3+bgQohhAgqL9KgAPCyGy/d9dhmGaIv+6wfz48IlLeuOJS/Rj7WbwdnQ/74aBtXo/f2Emhp4E4AMpjh4C4HMpVj4HQLYQ4ts67SHDMEwDoecI2Vgxk+WSC8DNuEAI8ZRO+9G17xbDMEz9M7xrAn7Zd8rhDvmRwqUxEGAvF4ZhGIm3ZqRjx9GzDu+ZQIOX/jMMw0g0j7BisEnHycYICzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAkeBZ2IIohoIxFtIaLtRPSsRps7iGgrEeUQ0S9E1LtuusswDMPoYWaEXgFgrBCiH4D+ADKJaLCqzXwhRF8hRH8AcwG84ud+MgzDMB4I9dRACCEAlEgvrdKPULU5q3gZrT7OMAzD1D0eBR0AiMgCYDOArgDeFEJs0GhzN4AHAYQBGOvPTjIMwzCeMTUpKoSwSeGUZACDiChVo82bQoguAB4F8ITWdYhoJhFlE1F2QUFBbfrNMAzDqPAqy0UIUQRgFYBMg2YLAFymc/47QogMIURGYmKiN2/NMAzDeMBMlksiEcVK25EAxgPYpWrTTfFyMoC9/uwkwzAM4xkzMfQkAB9IcfQQAJ8LIb4nojkAsoUQ3wK4h4jGA6gCcAbATXXWY4ZhGEYTM1kuuQAGaOx/SrF9n5/7xTAMw3gJrxRlGIYJEljQGYZhggQWdIZhmCCBBZ1hGCZIIPvK/gZ4Y6ICAId8PD0BwCk/dqexw/cb3PD9Bjf+vt+OQgjNhTwNJui1gYiyhRAZDd2P+oLvN7jh+w1u6vN+OeTCMAwTJLCgMwzDBAmBKujvNHQH6hm+3+CG7ze4qbf7DcgYOsMwDONOoI7QGYZhGBUs6AzDMEFCwAk6EWUS0W4i2kdEsxq6P75CRHmKwtrZ0r6WRLSMiPZK/8ZJ+4mI3pDuOZeI0hXXuUlqv5eIGpXLJRG9R0QniWibYp/f7pGIBkqf4T7pXKrfO3RF536fIaIj0u85h4gmKY7Nlvq+m4guVuzX/Bsnok5EtEH6HD4jorD6uztXiKg9Ea0kop1S8fj7pP1B+fs1uN/G9fsVQgTMDwALgP0AOsNe6m4LgN4N3S8f7yUPQIJq31wAs6TtWQD+Jm1PArAEAAEYDGCDtL8lgAPSv3HSdlxD35vifkYCSAewrS7uEcBGAEOkc5YAmNgI7/cZAA9rtO0t/f2GA+gk/V1bjP7GAXwOYLq0PQ/AnQ14r0kA0qXtGAB7pHsKyt+vwf02qt9voI3QBwHYJ4Q4IISohL060tQG7pM/mQrgA2n7AzgrP00F8KGwsx5ALBElAbgYwDIhxGkhxBkAy2BcTapeEUKsAXBatdsv9ygday6EWCfs/wM+hE6lrPpC5371mApggRCiQghxEMA+2P++Nf/GpdHpWAALpfOVn129I4Q4JoT4Tdo+B2AngHYI0t+vwf3q0SC/30AT9HYADite58P4Q23MCAA/EtFmIpop7WsthDgG2P+AALSS9uvddyB+Hv66x3bStnp/Y+QeKczwnhyCgPf3Gw+gSAhRrdrf4BBRCuw1EzagCfx+VfcLNKLfb6AJulYMLVDzLocJIdIBTARwNxGNNGird9/B9Hl4e4+Bcu9vA+gCoD+AYwBelvYHxf0SUTMAXwC4Xwhx1qipxr5guN9G9fsNNEHPB9Be8ToZwNEG6kutEEIclf49CeAr2L+KnZC+akL696TUXO++A/Hz8Nc95kvb6v2NCiHECSGETQhRA+Bd2H/PgPf3ewr2MEWoan+DQURW2MXtEyHEl9LuoP39at1vY/v9BpqgbwLQTZoNDgMwHcC3DdwnryGiaCKKkbcBXARgG+z3Is/y3wTgG2n7WwA3SpkCgwEUS19nlwK4iIjipK96F0n7GjN+uUfp2DkiGizFH29UXKvRIIubxOWw/54B+/1OJ6JwIuoEoBvsk4Caf+NSHHklgGnS+crPrt6RPvP/ANgphHhFcSgof79699vofr8NNWvs6w/ss+V7YJ8pfryh++PjPXSGfXZ7C4Dt8n3AHkf7CcBe6d+W0n4C8KZ0z1sBZCiudSvsEy77ANzS0Pemus9PYf8aWgX7yOQ2f94jgAzpP9B+AP+EtPK5kd3vR9L95Er/yZMU7R+X+r4bigwOvb9x6e9mo/Q5/A9AeAPe63DYQwK5AHKkn0nB+vs1uN9G9fvlpf8MwzBBQqCFXBiGYRgdWNAZhmGCBBZ0hmGYIIEFnWEYJkhgQWcYhgkSWNAZhmGCBBZ0hmGYIOH/AS00mrP+a81eAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "#hide\n", "learn.recorder.plot_loss()" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [], "source": [ "learn.save(path_data/'GPT2_pt_2epoch_lr1e-3')\n", "learn = learn.load(path_data/'GPT2_pt_2epoch_lr1e-3')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 6.2.3 Freeze all layers but the last 3 layers groups" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracyperplexitytime
03.3333893.2073900.37457924.7144876:20:51
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.freeze_to(-3)\n", "learn.fit_one_cycle(1, slice(5e-4/(2.6**4),5e-4))" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deXwV1dnHf89NQha2BAgYCBg22VfDIiIiIKBYtYpKra2vb6t1qdpqrdQVta3WtrZ1X/pqW9ci1ooiWlGQyh4EwiLIFiSyhS0kQEKW8/5xZ+49d+6ZmTN3yU1uni+ffLh35szMmXvn/ubMc56FhBBgGIZhkhdfojvAMAzDxBcWeoZhmCSHhZ5hGCbJYaFnGIZJcljoGYZhkpzURHfASocOHURBQUGiu8EwDNOkWL169UEhRK5qXaMT+oKCAhQVFSW6GwzDME0KItplt45NNwzDMEkOCz3DMEySw0LPMAyT5DQ6Gz3DMIxXampqUFpaiqqqqkR3Je5kZGQgPz8faWlp2tuw0DMM0+QpLS1F69atUVBQACJKdHfihhAChw4dQmlpKbp37669HZtuGIZp8lRVVaF9+/ZJLfIAQERo37695ycXFnqGYZKCZBd5k0jOk4W+CXPgWBX+s3FforvBMEwjh4W+CTPjxeW44dXVqK2rT3RXGKZZc/ToUTz77LOet7vwwgtx9OjROPQoFBb6JsyOg8cT3QWGYWAv9HV1dY7bffjhh8jOzo5XtwKw100SwDXCGCaxzJw5E9u3b8fQoUORlpaGVq1aIS8vD2vXrsWmTZtw6aWXYvfu3aiqqsLtt9+OG264AUAw5UtlZSUuuOACjB07FkuXLkWXLl3w3nvvITMzMyb9Y6FPAuq5HCTDBHjo/Y3YtOdYTPfZv3MbPPidAbbrH3vsMWzYsAFr167FokWLMG3aNGzYsCHgAvnyyy+jXbt2OHnyJEaMGIHLL78c7du3D9nH1q1b8eabb+Kll17ClVdeiXfeeQfXXHNNTPrPQp8EsM4zTONi5MiRIX7uTz75JN59910AwO7du7F169Ywoe/evTuGDh0KADjzzDNRUlISs/6w0DdRyk/WBF4v/roMkweclsDeMEzjwWnk3VC0bNky8HrRokVYsGABli1bhqysLIwfP17pB5+enh54nZKSgpMnT8asPzwZ20Q5Jgn9i4t3JLAnDMO0bt0aFRUVynXl5eXIyclBVlYWNm/ejOXLlzdw73hE32Tx+YJBE2yjZ5jE0r59e5x99tkYOHAgMjMz0alTp8C6qVOn4vnnn8fgwYPRp08fjB49usH7x0LfREmVhL6OdZ5hEs4bb7yhXJ6eno758+cr15l2+A4dOmDDhg2B5b/4xS9i2jc23TRR5Cjo+npWeoZh7GGhb6pI2s6mG4ZhnGChb6LI0l7HI3qGYRxgoW+iyKN4HtAzDOMEC30TRRZ3oZEEYUdZJY4cPxXHHjEM01hhoW+iyCN6HdPNhD9+jp+++WU8u8QwTCOFhb6JEjKi1zTdLNl2KD6dYRjGE61atQIA7NmzB9OnT1e2GT9+PIqKimJyPBb6JIC9bhimadK5c2fMmTMn7sdhoU8w+8qrUDBzHq58fplrWyEENu/zZ+ULmYyNW+8YhtHh7rvvDslHP2vWLDz00EOYOHEihg8fjkGDBuG9994L266kpAQDBw4EAJw8eRIzZszA4MGDcdVVV8U01w1HxiaYNd8cAQCsLDns2vbNlbtxz7vr8eqPRqJrTlZgObtXMozE/JnAvvWx3edpg4ALHrNdPWPGDPzsZz/DzTffDACYPXs2PvroI/z85z9HmzZtcPDgQYwePRoXX3yxbc3X5557DllZWSguLkZxcTGGDx8es+6z0CcYL3V+N+0tBwDsPHgcXbKDBQlKj8Tuzs8wjHeGDRuGAwcOYM+ePSgrK0NOTg7y8vLw85//HIsXL4bP58O3336L/fv347TT1JlmFy9ejNtuuw0AMHjwYAwePDhm/XMVeiLKALAYQLrRfo4Q4kGbttMBvA1ghBCiiIgKAHwFYIvRZLkQ4sYY9DupuXtOMd5b9y02P3JByHJC8K7AY3iGscFh5B1Ppk+fjjlz5mDfvn2YMWMGXn/9dZSVlWH16tVIS0tDQUGBMj2xjN1oP1p0bPTVACYIIYYAGApgKhGFpV8jotYAbgOwwrJquxBiqPHHIq/BP4t2o6omvOD3ln3+NKhC+O31DMM0HmbMmIG33noLc+bMwfTp01FeXo6OHTsiLS0NCxcuxK5duxy3HzduHF5//XUAwIYNG1BcXByzvrkKvfBTabxNM/5UKvMIgMcBON+ymIgx7fhCCI6GZZhGxoABA1BRUYEuXbogLy8P3//+91FUVITCwkK8/vrr6Nu3r+P2N910EyorKzF48GA8/vjjGDlyZMz6pmWjJ6IUAKsB9ALwjBBihWX9MABdhRAfEJE1v2Z3IloD4BiA+4QQ/41Bv5sd3x4N2uEFgFjPv9bW1WPt7qMoLGhn22bP0ZNIT/Whfat02zYM05xZvz44CdyhQwcsW6b2pqus9I+dCwoKAumJMzMz8dZbb8WlX1rulUKIOiHEUAD5AEYS0UBzHRH5APwJwJ2KTfcC6CaEGAbgDgBvEFEbayMiuoGIioioqKysLJLzSHoOVwbTFwgRnvbgteXOj4Vu/HnBVkx/flnAC0jFmMc+w5m/XhDVcRiGaXg8+dELIY4CWARgqrS4NYCBABYRUQmA0QDmElGhEKJaCHHI2HY1gO0AzlDs90UhRKEQojA3NzeiE2lO1CtMN/f9e4O6sSabDfv/gYrqqPbDMEzjw1XoiSiXiLKN15kAJgHYbK4XQpQLIToIIQqEEAUAlgO42PC6yTXMPiCiHgB6A+ACpxI1EZSHOnmqzlM0LE/cMs2B5nKdR3KeOiP6PAALiagYwCoAnxi2+IeJ6GKXbccBKCaidQDmALhRCOEeGdSMOFlT53mbmrp6T5OxOvb8OHl1MUyDkJGRgUOHDiW92AshcOjQIWRkZHjaznUyVghRDGCYYvkDNu3HS6/fAfCOpx41M3rmtrJdV18vAkXAZSGurvUq9F5G/6HvN+4px7Qnv8CCO8bpH5BhGpj8/HyUlpaiOczxZWRkID8/39M2HBkbIVU1dUjxEdJSok0XZC/CdULAh/Chdk2d0MpBX1VTh50Hj6NHbkvXtl/vr1Aun7t2DwDgk00HXPfBMIkiLS0N3bt3T3Q3Gi2c1CxC+t7/Ea7QSETmhjyC/nD9XsxZXRp4L+ew2bT3WOB1TV29ljnml3OKccFf/ovDloIj+49V4VBlNW56bTWOVdUAAHYdOqHcx5ET/m117menauvxwufbcao2PNiLYZjEwSP6KFi7+2jU+5D1+ubXQwuDmEK/o6wSv5wTjJLLSk/RskWu3uV3layoqg1ZPuq3nwZef7xxH3Y8Os12H7OL/DcenUnjvy3diUfnb4aPCNeP6+HanmGYhoFH9Amm3mFobq45ZBmRv/D5Dpw85T6J6zO+3Zo6+xF2h7DgJ3V/fv/xlpD3hyqr8cQnX4csq6z292np9oMAgIqqGhTMnIePNuxz7SvDMPGDhV7B0u0HcdFTDRPA6zRONkftCzbtD1v3wmJ3L9UUYwa31mE0XlZZHXIjkO87W23s9gBw5q8X4MlPt+KZhdsCy8zZhIVbyjB//V7sPHgcAPD0wq2BNgcqqvD518k/YcYwjQkWegVXv7QCG749hq8ku3i8cLLA1At/6gOVqO8t109N7HwzAXrfOz/wvlZS+vP/tNh139aRvsm89XsD5yZn3bzqheW49uWVrvtlGCZ2sNA7YDV5VNXU4R/LShzNLV5xtLUL4OSpWuWqr/f7c2X07mjvnmnu2Yt75R3/XKvd1orsAupP0xC+3BzlJ7u/M8M0JljoHfh8S6iJ4anPtuKB9zbivXXfxuwYTnJXLwT+vtQ5h815fTuqt60XAU+a49Xqm4WK2ihuYvLIHRS8wewrr8LQh/+DHWWVgdWxror11KdbsV3aP8MwQVjoHdhkMd2Y3ivHTuoLpxtOA9uKqlq8teob5boWhr+jXUDr7KLdgdfWSdN4IY/c9x49GRi1H6ioxtETNfij1I9Y6nz5Sf++v/fi8tjtlGGSCBZ6B1qkhn48KUaUajSjXitOZpVxv19o69bYOdsIgbZRetlTZ803QTdQHW8dFYPz23pqX1xajtdXhN6k5Dq3XsxJrhi7iiSdBMM0B1joPZBqCH1dfewCgiKVuzpDKMlG6e1s4P0e+Cii41UrKl4BQEZa8BJatCUYPVtbL/CvL0NNXGkpwb56MSe5YuzWGi/AMIyfpBT6iqoaHKioinrSdOOeUNONLw4j+kgnJc17jTUZ2d7yk1hfWh7zClR2I/BqIwq2urYOX37jHEAm18PUKWheeuSEVp79yghuGut2Hw2LGGaYZCUpI2Mn/PFzlBl51X/73UG4elS3iPaz7UDo5J45evYqoqaYqwr/RirItYbSy3sUQmDc435zzyVDO0e2Y/grSXXOzgxZltkiRdnW7P+P/17kul+f1NmObdyrVF33yipsPVCJaYPykNOyhW07+UlCl0ueWQIAKHnMPiqYYZKFpBzRl0nFM15fYT8i/MXb61Awc572fiNN5XvPuxvQ/VcfKtfpJCdTYXqtyH2qF8FUBe8ZycgAYFAXb/b19d+Why1rm5kW8j7FF/ph/HfrQdf9Egjpqb7AazfMkfoJF9v7XxYEA7K2l1XGxXWzvl6wSyjTZElKoZdxsrLICcRMHvlgk217U5q8/uDfXOmflFSZkiLVDlPofZLSr9hxSNl2ZHf7OrBO+5axxhS0iDJrp84NTve+KlfFmvjHz8PmBqLlVG09etzzIR52uDZ0OHL8FJZtV39HDBNPkl7ovYry/32x03ZdUckRY5+R9cXMFCkTqbm/ViH0V/91hbqtQ64bFSqhX74jtF6M1SNJF/Oz0zlvs4nTd2iaYGRUTyQy5SfDvwcnKozv7ZUlJZ62s/LDl1fiey8td8w9xDDxIOmE3ioKsXTjW1niF7vPIrAJA+obRKTmAFOMrSYUFTUe7yY6n5ksltsO2OfEkSEpiMo871Ulh7F0m7PZx6k76xQZRN2CsbwWUn920XZP7e3YYtTljXWwGMO4kXRCb/0N6fymyk/UhNj13Vjj4l1y+XNL8QdFDhiVgEb6kw+O6N3b1nmsS+v13jNXmg9w4uv9FYHzNXPWX/H8Mtsnkb3lVQBi7HMPYM03Rzy1P+bxCcAW47uK9fkwjBtJJ/SRjOhHPboAI36zIGZ9WL3rCJ6WsjqaqEZyXkf0PznXn+e9TYZ/ctTqHaOixqPfv9cJ4ic/Cz9XFR8U7w2c72XPLdXev9cRsNukuVedlQ//3WfDTUW6mN3iET3T0CSN0J88VYfuv5qH94tDR5c7yo67bltlEwwUazbvCzdxeBWdTq39EbEHK/1PIH1Oa+26jVOaYhUxjAcL37fRlaMn9EfJXnXR7SGn5FDwmtCJtZBvxm5Pc06cMmzzLPRMQ5M0Qr+n/CSEAP7wccPkdbFjpMOTwQ8V6Xmj/cnr2OhrPSq322RhQ7gZygIshMD76/bgFksFLjtU8Qoy26Wb/ymNidFYna35sX21V29Og2FiRdIIfZYR0FNYkBPT/W47UIHSI+p6qioOeLD1A2rTUm5r+2Aiq67vOeoeYapTBlAm1cV1srjU2aslFsyXqlLVCYFb31yDeev3hrQ5YZPCOVr2lp8MeNoAsb+xZWeluTdimBiSNEKfmeYX+jM6uZsyaurq8Y1NMWwrk55YjLG/W+jYpr5eeHoc/3p/BRZu9nvuqD1x7Lf1WZS+sMDdR96rqUDOSaPi443xLw14XBJx+YHk3TWlgbiE6c+pi7OrBvRCCLy0eEdY2gPVZ3PWo5/hoqe+CB4/Rjpvlm3MSFNHGUdKXb3w7ELLNC+SRujNSMsjGvlLfv3BJoz7vbN4e+Hm179Ez3vCI1/tEndN/tNiXPe3VSiYOU9pFnAaQVo1LEUjXFfXb3tYt2wAwO1vORcfiZW7oRPyZyA/9fz8n+vwq3+tBxCeRtrKPe+ux//+bRUAv0vlbz78Cr94e11IGzNXj5Vdh04ESinWxWhEb35VsUyKBwCXPbsEvaQqYQxjJWmE3uSvDgFPJktiHJ34kc0Id8CDH7tuqxJ1R08hi7D7NIRedzI2Jys0n4zbyD5atigmp03kUbRXd0RzAv6NFd/gs80HUFZRjfvf2wgAISYZAHjy061h25uc/6fFOHCsKuamm1gPvtc1gCmNadokj9B70KRIc7K7EStBcDIVWE9TJ/+O7ojeOrHbJiMN0wbnhbUb2KWN1v7cuOFV+0RoIkTove3XWny8SsqVQyB0khKq7T9W5bivIydqsOeocxtd2L2SSRTJI/QOWNPYfqsxgamiR4eWjus/2+w9YlZ1b3AawVqFXUvoNYVFHsHvOnQc9UKgXVZ41kjT1hwtuyzzJMeqagKRro9++FVgebQBRuc8HjTTCQj0PS14o3L1MIJAO4fMmV4gDphiEoSr0BNRBhGtJKJ1RLSRiB5yaDudiAQRFUrLfkVE24hoCxFNiVXHw49tv8606UaLW+DRPe96P44qOMnRcgPvphtdm3CKL3g5zJq7EUdO1CgjbxdZaunGgnnFe3He7xfhkmeWoPxEDSqq5cnY2AkjgUI+8R1lx0NG/FZ8RBHdwO2ODfCInml4dEb01QAmCCGGABgKYCoRjbY2IqLWAG4DsEJa1h/ADAADAEwF8CwRxdblwDyWw7rSIydw19vrsNwmu6Muuw+HPwlcLkV47j/mzbUS8D6itwpvLG30adLOFxpi7uaTHg3j++QGXq/dfSRQ/vBvS0tC2sVSFwVC0w3vOHgcP3l1tW37WJ19RVUN9hlmolhN7jKMLq5CL/yYFTjSjD/VlfoIgMcByAbNSwC8JYSoFkLsBLANwMjouqzGSZAOHKvG26tLMePF5Xjhc3uPkUhs7Kt3ecubMnvV7pD3UZtuNI6pa6NXebFEovNPXDlEq13/vKAJRf7+2rUKNZUocwRFKJarSo6EfeayTd/69BCr+5xcmCWWTygykZokmeRHy0ZPRClEtBbAAQCfCCFWWNYPA9BVCPGBZdMuAGRlKzWWxRynAFH5B/Do/M2YOuA0Zbt4/P6s+cc/+Wq/6zbOk7GhJ6ojRNs10kAAwFZLRS3V8XS4bHi+Vjv5NE9Jbo5h+YoUH4hbOUenVMRON1Lrfr/cFXnKA5k1UpbNWJpuZLPTqp2HHVoyzRktoRdC1AkhhgLIBzCSiAaa64jIB+BPAO5UbKpSibCrnIhuIKIiIioqK4vM/uskSEO6Zoe8T09Tn7bTDzDSgBQ5uGhSv46otBSw9upHbz1NpyeZWBQc0cmOqcJakUqFfJqyucaaB0f1tbjlnHfypqkXAqe1yVCus34Gv3yn2PE4usgf4xdGWuZYeH89aLiNArF7+mCSD09eN0KIowAWwW9vN2kNYCCARURUAmA0gLnGhGwpgK5S23wAYTlthRAvCiEKhRCFubm51tVaOF3kumLlNNL78wJ7f2snZAGrqxdYZpknUB1TJWzn9O4AwJvNuFu7LA+t1UQqHjrFPewyLZywCKDqM7rsWefslzvKwp9OgvuzzxF00qVsIRBqCntt+S4ccHHRBEI/x5f+uwM7yirR74GP8HbRblRU1UQ8ypf3qzNfwzRPdLxucoko23idCWASgM3meiFEuRCigxCiQAhRAGA5gIuFEEUA5gKYQUTpRNQdQG8A4Zm94oz10d/uR+Vky96yP/pEVCpzg8os4TwZq/9jtkYJr7xnova28vGybAqDO9HRIV+PydtF4aUcAeB5yzxKJJOXv5zjMBJ3EPp/LCtx3feXxrzMt0dP4r5/b8D1/yjCv74sxUcb9tpuI2dIraqpx9fG9TRv/V4MmvUfPPz+RrtNHRl+ejC3E+s8Y4fOiD4PwEIiKgawCn4b/QdE9DARXey0oRBiI4DZADYB+AjALUKIuEQrOV3kVoG1C3tXpRE2iUX5N5WAf6GoruSU/0YWqB+edbrj8bq1Dx3Rd5TMFVecqWdHBwGzf3KWXlsA3x3mn4IZnJ/t0lI/Adx1r6zSPr7JsSr7hGf1QtgKvdW0pt7e/79Z0OXQ8VO4Y/Y63Pjal/jDx1uwaU/opLZT9K1pvnl3TWR1binkNSs9o0bH66ZYCDFMCDFYCDFQCPGwsfwBIcRcRfvxxmjefP8bIURPIUQfIUTcEnI4jXStI3g7G/h/v7afH4iF0KueJD4oth8Fypg3CTmp2aju7R23mdxfPekM6CV/A/yfq04qZBPzHK8/p7v2Ng1N0a4j2HlQPUGtk7bYvNTM/+XL6emF2zD9+VCz0hOf2KfONjeN1I1VPnak8ylM8pM0kbFO17h1RG9nusmTqjVdPKRzyLpTNk8BXojEDpuTlYYFd5yLHrn+qFw5UtVNG1qm25tcrKN9O7YfqPQk9OYZjurhfBNqrOjc0K2DCmupQescgwpToKPVZvnaZu98xo7kEXoH1etrqcK00CayU/bEyLHkDF9V4s1fXkUkQr/mgcno1bEV7pvWH6/9aFRInhk3/c1qkWq7bkxPtRDfN61fyPv/bNrvaV7g7ql9tNs2RrxU4zKfrioUWUqf/HSrY62Am4wiKtHGTsmR1dZ9Ldt+CFe9sIxTGDNJJPQO63RHpPLoaEyvDlH2KJwvoyhDl5GWgrG9O4TYYVU3t8uGB8MUMh0mUVN8hA6twnO49FaYdLyM6PNzovf0Mfnx2IY3/xzSSHNt4vSpPPHJ1xjz2Geu+zDdRCOdSJXF3Tpp/fN/rsWKnYc9F8Nhko/kEXqnyVjNUZo58nnsskGYYhNUFS86t1X7dYfh4k73xJVDAyN9pzTDPiJlcjKVpuvkvHdD3sWkfp20tnGrdBUPdPLa+Cw2+mhQ2fkB4ECFXsZMeTO7iFv2xmGSSOjtr2adCTYgOKK3VnFqCH43fbBWu1C/aXUb8/fewkEofUTaJhmfzW7euH4UXrluBJ6/5kwAzjcr+WYx8wJn807vjq0AAKmNdHbRyxOOLnLcwWeb92Pkbz4NS7esRLpDtEwPNdWpEuYxzZOkEXondG3jq0r8IeSJCDzpo+kFI/fMrZ9OgpTiI6WAq2zGdvsZ07MDzuvTMRAFm28J0PrTVUPwg9F+F1D55nn4uHMwlWk+SY1z4ROZkoPHtZPemR9RLDIZqCZu1+72m3O+1MijJPehpY2pjt0umWYh9LppZv+xbBcA+4jNuKL5W5TF3e1+lGo3FIf/aUD3huZmurFb/d1h+biysGvgeCaV1c5Cb9Z1TdP4IqyT5rpcWRgaRzD+D4sw48XlWtua7rm6idW85u83s4jqDFAypfqz1uacJJMxaRZC75VEjOh1R11eQt5TfIT7L+pvsx+16UalDdqmLFWgl7FQPlardD1x1jGRRFoEJTMtBWsfOD+ibU0B1RVS3Y9v24EK7D58AinGk4xb/QMAIUVR7CKqudAJw0KvIB42WDd07y2hXjfObVN9hKkD7SeVbzy3Z8j7aYPCywYCGiN6h3WmxshC392lUpfucQGgQiOSVQURIVtRPUuHes9Cb38eIZPUTywOqYalc/OXRXzJ9oMhOX7MNUdO6HsSMckJC72CRFQAUv2k/zJjaHg7DyN6n48cJzSt2S07Z2dghcJOrTuiV03+mYnVvj+qW2CZyq1Thc4Nd59GQjEVXp/a/vfsoKtnwHSjOdnp1EfVzSwQTKXRRflSfeHzHZjwx88D700T2LQnv3CsosUkPyz0CpYo8s8kgkuGOqfuN4VgZIFfsC838sC3krwvnIReFpk/XjEEd07uExZFfNeUPlE94eS0bIGSx6Zhxsig0OuG+0cyGWvm2nHD6yndNrEX3rh+FICguOqMB8pPOM9HqG6i5o1E9/ZqhzxgeWPFN1p7M6mvF3jwvQ2B5GtM06ZZC/07N41RLr/QxoQRT4gIf7jCvTKTPBI1X5t22on9OgIA5t02NvA04CTSbY2JzJysNFx+Zj4y0lIwsW/HkDYdW6e7mlDMQB1HE4XjHtR8+lX4JPqHt51j277ksWlhkb12eHWh9fmCcxrmSF7nya+6znkkPa53dIF5ug+fdon87Pj26En8fdmuiBLKMY0P+xj5JOeMTq1sPTYSYqNHaP54OdVBSDuF6SbF4qVxevuWOL293w5uFbTvjeyGxZJ/9roHJyM91cXf3mU40CbD/zn2Oc3eRdTpXjH9zHzMWR2esljlXuiWMlk3yMqr6SaFghbz4GSsu8q6zacuUNzMTPdSVcUvu7Yye46exCebQiuZVdd6M93Eyh+h/GQNlm0/iKkDG37wxARptiN6H5Gt++EC40ey5ddTlevjAVFo4rQ7J6uDikL96I3/jRcq7wpzNG7mh3/0skFYMnNCYH3bzDRkSC56VrOKz6e2Iw/q0jbwemCXtnj1RyNxr8No2mli0S7dcu9OrcKWuQm0UzRw6H60mgXwxx0YI3rhF3lVjhsr768Lq7PjyitLSgAgTKxV3P/vDWHLrntlFR6cG5rf3ksOHyD4OUfrsfOzt9bgxte+xO7DJ6LaDxMdzVboU3wUcGOzYuYyT09Nwd1T+zZIfwiEWmn4ZydooX70xojeWKQyJbRMT8W9F/bDPzVzyocXH1enKZbd+gDgnN65SE+1H207VwBTr7x1Qm9P+wGcYwd0jmlHii94q6oXAs8s3OZa5QoAfvPhV56OEwtURXJ0XDVNdh48HqiMFq3Q7z7iT+ymU7mLiR/NWujtJirlUaGuK6DMSz8sdG3z2Z3nhi6gUPHcYFMTVTbFkGVEb2czvn5cD+3zGJKfjfP7d8JdU/xPFEO6ZoeM8p/9/nAAwE/O7aG1Pzvkalc+ImUNV1VSNjezmm7aBM8jeqLA5yAA/OE/9jnmGyN1Hkb0lz+3FC8u3gEA2H+sOiovNPNzZlf+xNJsbfTFpeW2opGtmfN92uA8zFMUDtER1VxLqT2i0KpMOvVezVFpD+N4kQYPybRI9QVuVLec1yuwfO0D58PnI7TJSEPJY9M879f6OcrVrlJ8hC45mWFuiC0Ucwc6LqV6/fE+GWtuksgApEufWYKJfQn8sLEAACAASURBVDsiIy3FU6bNiqpa1NTVa0UbW/3ub3xttdbgRUWsTEBMdDRboQfsA3L+Z0xB4LWTsIzq3g43j++JvUer8ON/BIpqaU1kuQlWnkY2S1PTbhrfC4Pys3HuGZEVVtch0uAiEydh9ZE/WGu1ZfI1TWGGidU8eSTRz+Y2VRqFReLF2t1HsXa393TX/yzajZJDx/H6j0dh5c7Djmm4rZqsM1fgBgt9Ymm2ppseuS1tR3+yZ4eTZ8WgLm0xoHNbTOofmnZXR0KsTxPWbXRGprLXTTxFPhY4nY3PR7ju7AIUz5qM6VItW5UfvZdRrBOR3DD2lfvtzfe8uz7i45omsUSwYudhPLNwO67+6wosbaBYkYBLqoPO19TV41iVc7wBEx3NSugvHHQaZl7gn1zdUXbc9scuu+ipRiKvXDcC3xvZFcO65UTcl7BJT8uCLQ6Fyu320Zhxm4wl8puF5GYqe7tKMFpnpIY8hUXbHzvMdAtHXIKgnEj0DXnHQb/L5n4p331VTR1ufHU1vjkUe88Yu3z7QDB//m1vrsHgWf+J+bGZIEkp9Ha29/ycrBBxN23AVnu4PBmb2zrUhLLwF+NxXp+OePQyvfzxKm6b2DvM3dDa4wPH3KsCJSL5WqQ4uVfKJjT5lFTfY8+OofMfPTq0xPpZUzDr4gGOx7+qsGuIG+fLhgujHT0U8yyxiK84WOmt2tNvY+y1Y40FAIAvth7ERxv3Ydb7G5XbRIM1yMxk58Hj6HHPh/igeA/mb9gX8+MyoSSl0MucI0Ue/mJyn8CFl9s6HempKVhwx7l4+uphIdvIboJnnh46ao/EC+eZq4dj1neCWSRnjOgabrqxaMh3hrgHmDQpoXca0UtXoRw4pTq/MBdOzY/gscsH4eFLBgbeH3YxAT12efiNPBaf9zm9vY3oTe+XWPHvtX6/flnog6NuPTt6VU0dXlu+y7ailYx5mVs9dzbu8XuVfbg+3JmBiT1JJfR3nH8GgNAL9oZxQTfAFqm+gIlkYGd/5Gmvjq0ci2hHy/n9O2Ha4Dz8z9nd0amN3yuGSGWjD33v5B1hRvQ20gJMShxt9JKAynqgI6xPXBme+E15fI8irWrutgtrjnsVKT7CnBv1YhoaCq/3rz998jXu+/cGPPXZNo19B11SQ5YbV8SH66Mfzd/z7noUzJwX9X6SmaQS+lsn+N0Br5MyDZ5hqdwUiCYNyRkT237IoiLvOpCVUCF71h+bk9CbNuImNKB3VHo7k4hbet0nvzcMQ7tmhy0f07M9HrlkAO69UC/vjQpVj5xuPOf07oDHpw8JCyRrrMinYl6PukFN5tPQnxa4xxKY36119B/La9drwrbmSFIJPRFhx28vDElsFebNYlxhIRe69ObPV+mNEO34zpDOKGifFSjrJouDeanr3Fh07MFeR6mNFbvTkOuoKrezWf7G9aPxg7MKMC6KiU9Vn8ySiU4cc+kz4I+XmNxfr0C6zPTn3CNxdVnzzVE8atr/jXNdvuOw1rZeLjtz/sVqumlKT6PJQFIJPWAGtoQMV0LXG+/JZkTvdBE/9b1h9iulfRERbjGeLuT9maPPDJfEXEBiEqvFEyfzr108gzVgali30NG7m+BEWhJyTM/2ypvoOQ6ZJs3z65HrPofTItWHFyMIQCrSqCGry6vLd+GFxTtQUVWjbZs3CRm8uGxrNq0La6fyqHLvR1298Dyh7ZW6eoHaOm/ZPhs7SSf0VsLMJMaVJ+uoKvWvivF99EeI5nHl3f1lxlC8/9OxgWyPim4F0DEBRFpdKRE4/YbtPvNCaSK8eNZkvHXD6JD1bhWYIn3iObtXh5A9TxnQSXt/unMGQHh0dCKoF0BVjZ6ozV23B2+uDDWTWOdj31ldiq1Srp2g6QYomDkPv/toMwD1iF7nfvPw+xtR+OsFqIij333Pez5Er3vna002NxVcZyGJKAPAYgDpRvs5QogHLW1uBHALgDoAlQBuEEJsIqICAF8B2GI0XS6EuDFmvXfgril90LVdFlpYhnUqG738+3XyxvAyyjb3KYtRVotUDMpvq24fQcZ2p/TCjQ2nyEhVcFhGmi8knkHn5mjFKY++tbpWyHaWp8IXfuA++v7CCEDac/Ska1uTebeOxcjffqrV1mm02yU7E996OK5Mfb1AWYXeCPm2N9eELTty4lRI6o07314HAIE0GQGhN/r/3KLtyG2VHlIcx2THwUrkZLVARVUtCmy82z40XDFPnKpDa8U1EUsqqmu1zHVNAR13k2oAE4QQlUSUBuALIpovhFgutXlDCPE8ABDRxQCeAGDm+N0uhIjO8B0Bcp6Wf908BgM7+wXWvEnvKQ8GjMii//qKXbjWJvjGi3udOUHkFK7eL68Nvtp7DEBkk1O6CbwaA+YPvbMitYPqBuoh2aItTt+XU4CVjyIrlAJ4q4rV1qYegooRv1lguy5SkQeA2noRltLYC0u2HXSshGZ+B7Lp5uEPNinb7j58EpOeWAwAtvmUAtW3GuDS92rSasy4DgmFH7MCQprxJyxtjklvW1rXJ5rh3XIC9t4t+/xdXScJsHzRfL3fvtiDl4vrGyP/tlO9UNkUEcl125QmY4N1UMP7rLpf1WoovdvZO2UsdrpHEoLVpLpkZ7r2A/BPwgPO34l1lZeBw8HK+BT41vmcnXDLYWPndRPJvgBnzzX3bQVqbGzvH2/ch4KZ81BcGtSFoydqMOjBj7Fse3gd5aaG1rM/EaUQ0VoABwB8IoRYoWhzCxFtB/A4gNukVd2JaA0RfU5EyjpwRHQDERURUVFZWZmqScxQXSC6Pzi3knoAsNS4KHp39BfNUKXfNZHNSl5Ee/ZPzsLwbtlaGS4bC53aZKCgfRYeuTQYwaoyo5k46UJPjQlPu/3qQCSZ3jR3oVPwxM4DLJ64Bfj90ZJuuVKjmIrMt0eCTxOqdMZ2AVMqvtEoTmLuJZLR9stLStD73vk4pJjMnb1qNwDg4qeXBJatKz2KiupaPPXZVs/HMpm7bg8KZs7DHf9cG/E+YoGW0Ash6gzzSz6AkUQ0UNHmGSFETwB3A7jPWLwXQDchxDAAdwB4g4jCauQJIV4UQhQKIQpzcxs+F4ju703HRn/c+KHcY7h4tnGw8amcgwZ2aYOze7V3PMbI7u3wr5vPVqbxbay0SPVh0V3nYULfoFuhqkiIznfRM7eVVlvn74uMPjjvRFeMz+/nP6++DuUUrfvStbzZjUJ1GGIzJ2RiLeG4zmNmzFekVBJ/VvjVe0lTvKPsuGsbcz/hXjzu/OtL/7nuLQ9/ylYly1v8tX/eRecGtGz7IazeFeqeuv9YVWBe419rvvXc31jiSSmEEEcBLELQ/q7iLQCXGu2rhRCHjNerAWwHcEZEPY0R1pwbQOgP0GkErjPqvny4PzrSHK3rjjzMXX9w6zl4/cejnRsnCe/cNAY/Gts9ZFJ5xoiuAIABndU1c4HgqM7t+3Baba6zuxl4GdHPv/0cXGAUlM9ra2/qaZ0ROiWm+xT3xdbIM016fWpQXa75OfbnJAu4HCk75tFPsXT7QamesfuxdW58R41gQdUTgu5vTdVMNZf2jnFjKD3iPgfyvZeW4/LnloUsO2lJZ116JHHlFF2FnohyiSjbeJ0JYBKAzZY2cs23aQC2StumGK97AOgNILbJOzxiztTLCa7kH8NtE8PL10WCzkhGHk02JXt7rBiU3xb3X9Q/5NzPPaMjAGfbeGBCzmX/OqY2uwyZpolPRygz09zjIi4e0hnv3DTGtZ2K6/62KqLtAP1CLCaqkbKT0Nll8txTXoXH5m/2NKL/x/Jdjutl+7kyG6bLIRr6J2btzlOfuqeMiBc6I/o8AAuJqBjAKvht9B8Q0cOGhw0A/JSINhp2/DsAXGssHwegmIjWAZgD4EYhhF74XZzpJI3c5e//0mGdY7L/KiOc3GlyNzXSiJ4kxhRxJ4E1f+Su1aY0ftkqIRQQgYlcuWB7NMe5bWJv9MgNL3YeC9o7xFx4dczy6jvuFO9RL4RrmUsZq3gfq6pBwcx5+Ot//WNDOVJatT+3Y5g370QVQUlk8RUdr5tiIcQwIcRgIcRAIcTDxvIHhBBzjde3CyEGCCGGCiHOE0JsNJa/YywfIoQYLoR4P76n447qupd/qKoEZ/NuG4uHXNLgmlx/jj+JGhdDjgxzROlkXzfLEFpNIVZ0RrO5ivKLQgAbvvV7Z+m4LuqMFAvax2/iXH5SsH5uKT7CXz1E4f7XxkxkFx9wxZn2idzq6oM3Gq9Vqk7V1gdy1P96nj9Vw/Hq4G9K9eThJqSBLJ2eehI75m/Yl7Bsnc12SCnb88jlUxjQua2tb72VbsYPuimlEG5MmKMy+eNraUkZ8cBF/fHnq4Y6Bj0BeqPZ1348KqSqFeA3AVR6iLzUmaSPx/Ww4aEp+PL+80OCi76873wU3Tcp8J6IPAX6zS7arVw+9c+LlctPORjfv9p7LGA+m+dR4FQVp258bXXgtfnkIf+O3YTetO979dgx5410kPdtfcKorK7Fza9/6enYsaLZCb3q96Zjy43FcRh36i0j+uJZk7Hy3kkhbTJbpODSYV1c5zWcBM6sM9A5OxM/GH16yDoBgYFdnL1VZHS+63hcD63SU8NMJ22z0kIiVb1e211tXHaP2aTb6N3R3ssIiHzuya3f5ohe1lI3043pPaNq1T/PfvK/ZXoq/rZkJ5bvcPenX7It2EbHXGWydPvBgFdQPGh2Qq8iFqOtt288K6QeqJd9ZnuIkEx2zAGi+UNvk5GGlopweR2cvoMQMbTcEHaWHQ88mXVo5Z5zqKbW+Qf92GWDEjbZ7iN4isYzI7V1ERBYt/solm5Xm3wijd6udpkbMUVUDviqqxd4/KPNrmkorC6lANDFwbPIR8Cs9zdhxovLleurJDOtnAvIi9Bf/dIK3DF7nXZ7rzRboVdV2ImGEQXtQtIu6F7f62dNxtKZE6LvQJJgBh5FKu4yujdba9qCf6/15vPsdpgZI7vZrhtZ0A73X9Tfdn20ePW68crnW8pwyTNLcPVLYTGUjsd3+8xUPvky5u9XDuwtLi3Hs4u241ZFTh4ZVf56J1F2u47u+/cG5XIvQh9vmq3Qy8QjJbDuPltnpMW1wlVTY9qgPNw2sTd+ObWPe2MXdL9W66izU5sM7RD7J64cojR39Oqo52Ez+8az8P1R9jcCXS4anIeOimyYVpGa0Ldj1MeS+Y/LJKvdd+BmJn9rlXquwEQ1ojdfWz2lTtXWBwIZ7ah1EmWXS2HLvmC2zoIOwWsh2vQSsaTZKox8/cdjoownYyMjNcUXKAkZLbo3W+t3JVcocxOky4aHe51s/+2FAPzpbnWIxUDj6auHK5d/vHEfxkp59OOZ3tdH4b7s8aqrELDRS1pqNy98zV9XYGWJs1d3nYMob5NcpD/ZtB/nOxSN6SfZ+t+JwOb+dtFuXFGoP/mrS7Mb0Qd9aYPL4nEtJlvhkKaIyi7+0c/OCctrH16/NzpXvBSfN0+XeA4KrMFOq0piV7zEimpQbI0OjdmxbGz0KtxEHgBq6+y/6U83Hwi8vv4fRWHr5Zun3IXXlnsvcXjXnGLP2+jQ7Eb05u8v1EbPI/rmQt/Twr0rrN9Vq4zUwLKGSAUd70Mk8kosi1M1KFPUZXHXDUgaUZAT8v7r/RVYsTP8ZpCZlqIVD1NyKJjaoLEWK2l2I3ozyViWRjm/aGCdb3p0bJ2ORy4diMuH5yMnKw23TuiFN64fFffjxtsjJxrpOTeKurtA/G6UpummRhLWkkP+pGhuQW7DTw8V+pUKkQecc/zYYd5sTpyynxNIRJ77Zjeiv3ZMAYSAdgBUpOiUAmQaB6bOpvooxKf+zsnRTwg3BnTSONgRfSrs6IVeVcLT1MpPvwpOBpuphp2qxAHAC5/vwK8u6Bd4X3JQnTUzkgyZ5hOGqhqXyeHjp9BeEZEdT5rdiD4txYfrx/WIe4rfMzq1xhvXj8LmR5wSfTKNiVYuKRUaA/de2M+9kYWRBc4RxE4kMj+Lydb9leh7//yQZaagyvnwZROKG7KJ5a9f7FS2KTumNjtt+LYcR2xuJubntdqhkPvGPcFYhRcXb8esKCp86dL4r+wmzJieHdwbMXHlqe8Nc0x5DPgzZd55/hm4dJh9SbzGQtd23s0JOVE8XXoxOY/q3k5h647+RqEyxZij7b55zpG5dlTV1rm6NVfYuGRe9NQX6JHbEp/dOT5snfl52WX1BIBdUn7733642bZdLGl2I3qmefGdIZ1ds0YSEW6d2Ns2/F+mT6fIhEWXKQM64YGL+uODW8finN6qgUJ0ppBObeJnMlA5IMTrgcAckefn2H9ne8tP4pq/BgO5cqQI9GjMWYB9kZT135a7bpuIWs88omcYD4zp1R5b9le4N4yQF34QzDQ5rGt2WDZJIuCV/xkRd2cCEy+apAoQipfhJxAw5eAW+czCbfhiW/Dzk0fZjgFSUfDGim/w2+8OcmwTz5utHTyiZxgP3Dm5j6eUw9HMBamkKIUI5/XtiFE9nMtNXm0TbevV7TdXEW1rx7pS99FsrDhQ4befb7LJzVN+ssbRj/2/W6OvTb1Q8q/3Qm4r+yp28YKFnmE80Co9FYvuOk+r7Yp7JmLlPRMjPpbK7KEos6vErGFr9ViRZf7WCb3gxs3j/W3G9nKfb4rWHOIFM7/MIx9sCiybZpRzBIAhD/3Hcfuf/zP6BGKRVv6KxJsnWth0Y/B/1xaiu5TXm2Gc+O8vz0N6mrPqdnKoP6yDqr6xbg4e0/vDqbWqTqqVFqk+FJ6eE2IC8UJDeu3EK7fMiIIcTxHFBTPnOa5PhB89j+gNJvbrFLdSb0zy0bVdFjq2ju8juEoPdC0v5rbWYCz5vZsZx0ySJmegnDKgE9Y9MFmvEwC+OXQCGS43xFhR42Cvj4ZTLvv1XK6RhZ5hmicv/OBMPHLpwJBlKjnQtbGb25qtTdG++4K+gTZu+XjMsphys6wWqWiblYadj16o1Y8dB4+jqsZ+pD3IQ4EXN+LlzFLrUEULAEa7zJdYqRfAF1sPhtTAjTdsumGYRsCUAaeFLVPa6DWF3oxoPaunX4QyDS+dgVJMgZ0w5rXNwFs3jA7sw6d4CohV2obJ/TtpuSRauXVCL8x8Jz4JwKy4FUGprRfo3qEldtpE2Fo5cvwUbnh1NUb3iDyQzSs8omeYRkoPac7ILMii68XT57TWWDJzAn40tnvI8tAbhVqs95ZX4fT2LQNivnR7sDxerNPyqPb3/DXqlMsyx6vrwnLWx8t0s+1ApeP6unrhyTupyrhxyHns4w0LPcM0Uq4oDOa6f+QSv1lHLmzhRpfszIBYB232wfUpEfz6Y20dUT0ZOAVBmXy8cV/YMq8VndaXluNX/1rvaRsVtfUCLTx8mOZkbENWoGLTDcM0UogIPzm3B3JbpWPGyG6OJQm19wn9yViTKQM64eONzpWkIu6Pogs6/VKlRfDqdfOdp7/w1F7F20W7sW73UQzvlu3YLj3VFzAB1bPQMwwjI2dZjAaVq6au0NtFn7bNTIt6QlHlLqrqqw6JqNH66vJdAJxTI/94bHe8X7wH+40kaebcbk0D9pdNNwzTDLhocGcAQFsp34udzvfMDY0nkSssydI096dne+7HL6f2Qbo0z2Dtw5WF4aUZZaafab9ex0avE/gVCSkON837LuofckP7ZJPf7NSQAWYs9AzTDLhrch+se3Ay2mYGhd6MerVy7zT7p4jPJNE/vX14gKGbrXpkQbvQ6m6W9RP7dQoRRWtitw4Oedx1RvTxSk++p7wKhZaCJnbEywzmBAs9wzQDfD4KEXkAaN8qNH1xtjHab5FinzDNWtTj7F6hPuSnXHzOe3dsHZICwGo+cnvf3WEyWidRWZds72menZB7l+ew7xqXz8XEqfB4NLgKPRFlENFKIlpHRBuJ6CFFmxuJaD0RrSWiL4iov7TuV0S0jYi2ENGUWJ8AwzCRIWvokpkT0MmI9PUSeJSe6i2LZkoKhYy8fT6rsMPx/ZWFXW33XVZR5Xr8UQ6+625mIze+OWxf+OSQS9Urk5d+WOjeKAJ0RvTVACYIIYYAGApgKhGNtrR5QwgxSAgxFMDjAJ4AAEPwZwAYAGAqgGeJqGHyqzIM44g8WpZHuk7BUFbTzAENcZVxsmX7jx06GbthzzHLevvtD1a6i6lTrqBfX+qcXtiNdRq5gxKFq9ALP2bEQJrxJyxt5G+jpbT+EgBvCSGqhRA7AWwDMDLqXjMMEzVWyTMF1mlEf93YgpD3VuGe0Lej4zHltAvPXO0eGFVWoS7nFylO9xmzIIiurV1mSFdn98pEo2WjJ6IUIloL4ACAT4QQKxRtbiGi7fCP6G8zFncBIIevlRrLrNveQERFRFRUVhZ9nmiGYdxJ8ZEyY6vTqNlaHcnq6dLnNOcKXPL2qSnhxyHjX7w4r4/9jcjnI3xw61i8ct0Iz/u1uzl+Z0jnsGWxzO+ji5bQCyHqDLNMPoCRRDRQ0eYZIURPAHcDuM9YrDr9sBkTIcSLQohCIURhbm54xXeGYWKPz0f46Gfn4KuH/QXsTYF1GtEXWgqNWycZb5/Y2/WYJilE4Sl7pWN7KfCii53XjZnCYGCXtmidkaZso2KzkcZAZZL6/fTB+NOVQ8KWn9a2kRceEUIcBbAIfnu7HW8BuNR4XQpAnj3JB7DHyzEZhokPPiKkp6YEEp6ZphunEb11RGwV+ow0/Sm4FOWIHmiZ7t9Hvzznou6RYHcTsz7Z6NZ1NaNdVcFnWS1SkapwN7XLhhkvH39Az+sml4iyjdeZACYB2GxpI9/GpwHYaryeC2AGEaUTUXcAvQGsjEXHGYaJDjsp8+J1M97BFOKGnZie3r4lXvmfEfj9FeGj4WixvYlZHizevTk8GOyMTvb1KlS7NW9YMi1SfLZuoJEWd9FBJwVCHoC/G94yPgCzhRAfENHDAIqEEHMB/JSIJgGoAXAEwLUAIITYSESzAWwCUAvgFiFEXTxOhGGY2OAlBfEZnZxt8k44eeCc5zKpG2usxUA6tG4R1sYpZYQ1t//vLh+Ec88IN0OfqqsPK/jeELgKvRCiGMAwxfIHpNe3O2z/GwC/ibSDDMPEB6tumVpnN6L/9aVhU3OYNigP97zrzwA5UypqooOq8Ems8tx7ZXtZaCrivLaZ+ODWsbjoqWDiM6e+WW8CV42IPgFdLOHIWIZpptiNUK3LxxjFS7q2C58cbZuVFijKnZ/jLepU7XWTGI6cCE/ONrBLW1wlBWjdNeUMAMBfZgwNa5ug+5M2LPQM00yxmzi1ita0wX4h76FwxQSA+y7qh0uGdsakfv7w/bum9Am8dkJ1o2lsgimbdCb07YRtv7kAlwwN8xDXzgTqhMoVM1aw0DMME4JVtK4e2Q2bH5mqHNEDfjPHX2YMC9w4bjmvF87r6+4mneoLl594+tBHQp3Fdq/yogH85q4nFK6UXuiXF/l8hxss9AzDhGAVeiLy5DYJAPWSZ8nvpw/GF3efh3duGgMgmEZBZaOPNBe9HY9cMiBs2VUO+XKs2GXEtE60EhGyWth/RukaWTNj8VRgu++47ZlhmCaFKWmx0BtTH09vn4UrCrsiPycLZxqpBUwxVwm9E6rMk6k+wqzv9Fe09tOzY7hL5O+mD9Y+pp3QT+wX6hXkI2ehXnDHufibS8Stx4/DEyz0DMOEEAu9MSdapw48LWydmTahrl7AGhhrfS+z6K7xAICLjDkDANj22wvxg7MKbLcZ2jVbK/AqI00thXbdUT31OAl913ZZrjEH8TRbsdAzDAMA4ekIouCKM7vip+f1ckyJUF0bHlJj9Wc3adkiBWmGyaeyujZkndNIOKtFKubffo5Gj22w+UjChB7en1CsxHMimoWeYZgQYiE4LVJ9+MWUPshqYR+qo0qoZtX5v/+vP9nt098PZrrs2Dq0ylQ8fe/t5gxUc7LW3Pq65BgFX+J5HlwcnGGaGWvuP9+1ElRDkJbiC8t2aRXWc8/IxZf3n492LYORqk7lBGNNvc3HZBVlosht7FMH5uHNld+wjZ5hmNiR07IFOrVp+AyKVtJSfBh3Ri4+v2t8II+9SlhlkQeCE70qLhmq9kW/b1q/QGCXF64Zfbpyucoe71ZUxY60FDNrKI/oGYZJMkyBO719y8BoVmeWwGku4eyeHfDe2vAEuT8+p0ckXcTY3uqMkirTTaSmF3NyOlLTjw48omcYBoCeyMYSWRjN13aTsTKObSStvNRmdK9iokMk7+OXD8acG88KWaYafevo9LVnhT8hnDzln1xu6eCHHy0s9AzDJJyfntcL7Vu2wMgC++LdJnZpfoFQAf7zjLBcjLY4RbVeOaJrWMEVldDvLXevn3vH5D5hy6YM8LugDuvmvYShLmy6YRjGQnzTELx941lYufNwyLIhXbOx+v7ztbavrbMX+kh7np7qbTStHNFrDOmtEbJ5bTNwwaA87Hz0Qva6YRimAWgg282IgnYYoTFyt8Na1UpGkT4nLlht9ELoVaWyavmyX000lsf35spCzzBMCI0tg6QVa0FyGa+eKyO7t8P90+xTKNihEmadY0fqmRMtbKNnGAYAcPUof7GM3NYN56ceCbec1xNAMJgK8NvYP/7ZuAj21QuD8tt63k4l2DqRsXbZL+MNj+gZhgHgd0GM1A2xIemR2wolj00LWXbZ8HwAwNf7KzztSyerpAqViShBGq5FI+4awzCMN3QtI7N/chZyW6djSH52RMdRu1fqHfy+af0iOmY0sNAzDJM06IrtyO7tsOreSciM0HfdepzxfToqC6moSMRTE5tuGIZJGhpqqtO0xxMBa++fjDaZqVi2/VADHd07LPQMwyQN8XZTNDlQ4Q+OEsJfIN1/8AY5dESw6YZhmKTB1Pl4phMAgLKK6rBlHVsnPlGcHSz0DMMkDfHMACmj8uXvpShb2Fhg0w3DMElDQ1lP2mSmKZe/cf0odG4bXts20bDQMwyTNDRUuczG7QAACOBJREFUCgQ7xvRUpzVONGy6YRgmaTAnY+OdtufiwfopkBsDrkJPRBlEtJKI1hHRRiJ6SNHmDiLaRETFRPQpEZ0urasjorXG39xYnwDDMIxJQ5luAp42TQQd0001gAlCiEoiSgPwBRHNF0Isl9qsAVAohDhBRDcBeBzAVca6k0KIobHtNsMwTDgNNRnb1HAd0Qs/lcbbNONPWNosFEKcMN4uB5Af014yDMNowDqvRstGT0QpRLQWwAEAnwghVjg0/xGA+dL7DCIqIqLlRHSpzf5vMNoUlZWVaXeeYRhGhkf0arS8boQQdQCGElE2gHeJaKAQYoO1HRFdA6AQwLnS4m5CiD1E1APAZ0S0Xgix3bL/FwG8CACFhYUNXbqSYZgkoSFl/pmrh6NTm8ad0tnEk3ulEOIoES0CMBVAiNAT0SQA9wI4VwhRLW2zx/h/h7HtMAAhQs8wDBMLGioFAgBMG5zXYMeKFlehJ6JcADWGyGcCmATgd5Y2wwC8AGCqEOKAtDwHwAkhRDURdQBwNvwTtQzDMDGnqVhu5tx4ljKNQrzQGdHnAfg7EaXAb9OfLYT4gIgeBlAkhJgL4PcAWgF427ijfiOEuBhAPwAvEFG9se1jQohN8TgRhmGYpmKjL4yiZm4kuAq9EKIYfnOLdfkD0utJNtsuBTAomg4yDMPoYur8iVN1ie1II4MjYxmGSRpq6uoT3YVGCQs9wzBJAzXmpPAJhIWeYZik4ciJU4nuQqOEhZ5hmKRh16ET7o2aISz0DMMkDSLueSubJiz0DMMkDYJ1XgkLPcMwSUOKjydjVbDQMwyTNHTJbnxl/BoDLPQMwyQNFww8DQDwyCUDEtyTxgXXjGUYJmlITfGh5LFpie5Go4NH9AzDMEkOCz3DMEySw0LPMAyT5LDQMwzDJDks9AzDMEkOCz3DMEySw0LPMAyT5LDQMwzDJDks9AzDMEkOCz3DMEySw0LPMAyT5LDQMwzDJDks9AzDMEkOCz3DMEySw0LPMAyT5LDQMwzDJDks9AzDMEmOq9ATUQYRrSSidUS0kYgeUrS5g4g2EVExEX1KRKdL664loq3G37WxPgGGYRjGGZ0RfTWACUKIIQCGAphKRKMtbdYAKBRCDAYwB8DjAEBE7QA8CGAUgJEAHiSinFh1nmEYhnHHVeiFn0rjbZrxJyxtFgohThhvlwPIN15PAfCJEOKwEOIIgE8ATI1JzxmGYRgttGz0RJRCRGsBHIBfuFc4NP8RgPnG6y4AdkvrSo1l1v3fQERFRFRUVlam13OGYRhGCy2hF0LUCSGGwj9SH0lEA1XtiOgaAIUAfm8uUu1Osf8XhRCFQojC3NxcvZ4zDMMwWnjyuhFCHAWwCArzCxFNAnAvgIuFENXG4lIAXaVm+QD2RNRThmEYJiJ0vG5yiSjbeJ0JYBKAzZY2wwC8AL/IH5BWfQxgMhHlGJOwk41lDMMwTAORqtEmD8DfiSgF/hvDbCHEB0T0MIAiIcRc+E01rQC8TUQA8I0Q4mIhxGEiegTAKmNfDwshDsf+NBiGYRg7SIgwk3lCIaIyALui2EUHAAdj1J2mAJ9vcsPnm9zE8nxPF0IoJzkbndBHCxEVCSEKE92PhoLPN7nh801uGup8OQUCwzBMksNCzzAMk+Qko9C/mOgONDB8vskNn29y0yDnm3Q2eoZhGCaUZBzRMwzDMBIs9AzDMElO0gg9EU0loi1EtI2IZia6P9FARCVEtJ6I1hJRkbGsHRF9YuT1/8RM90x+njTOu5iIhkv7aZS1AIjoZSI6QEQbpGUxOz8iOtP4/LYZ26pyLjUYNuc7i4i+Nb7jtUR0obTuV0bftxDRFGm58honou5EtML4HP5JRC0a7uzCIaKuRLSQiL4yaljcbixPyu/Y4Xwbz3cshGjyfwBSAGwH0ANACwDrAPRPdL+iOJ8SAB0syx4HMNN4PRPA74zXF8KfLZQAjAawwljeDsAO4/8c43VOos/N6Ns4AMMBbIjH+QFYCeAsY5v5AC5ohOc7C8AvFG37G9dvOoDuxnWd4nSNA5gNYIbx+nkANyX4fPMADDdetwbwtXFeSfkdO5xvo/mOk2VEPxLANiHEDiHEKQBvAbgkwX2KNZcA+Lvx+u8ALpWW/0P4WQ4gm4jy0IhrAQghFgOwpsKIyfkZ69oIIZYJ/6/iH9K+EoLN+dpxCYC3hBDVQoidALbBf30rr3FjJDsB/oI/QOhnlxCEEHuFEF8arysAfAV/evKk/I4dzteOBv+Ok0XotfLeNyEEgP8Q0WoiusFY1kkIsRfwX1gAOhrL7c69qX0msTq/LsZr6/LGyE8NU8XLFKy85vV82wM4KoSotSxvFBBRAYBhAFagGXzHlvMFGsl3nCxCr5X3vglxthBiOIALANxCROMc2tqde7J8Jl7Pr6mc93MAesJfnnMvgD8ay5PmfImoFYB3APxMCHHMqaliWZM7Z8X5NprvOFmEPqny3gsh9hj/HwDwLvyPdPuNR1YY/5vpoO3Oval9JrE6v1IES1nKyxsVQoj9wl/Qpx7AS/B/x4D38z0Iv6kj1bI8oRBRGvyi97oQ4l/G4qT9jlXn25i+42QR+lUAehsz0y0AzAAwN8F9iggiaklErc3X8Ofw3wD/+ZheB9cCeM94PRfADw3PhdEAyo3H4qZWCyAm52esqyCi0YZt84fSvhoNpuAZfBf+7xjwn+8MIkonou4AesM/8ai8xg0b9UIA043t5c8uIRif+/8B+EoI8YS0Kim/Y7vzbVTfcaJmqmP9B//M/dfwz1rfm+j+RHEePeCfbV8HYKN5LvDb6T4FsNX4v52xnAA8Y5z3egCF0r7+F/6Jnm0Arkv0uUn9ehP+R9ka+EcxP4rl+cFfznKDsc3TMCLAG9n5vmqcT7Hxw8+T2t9r9H0LJG8Su2vcuGZWGp/D2wDSE3y+Y+E3LRQDWGv8XZis37HD+Taa75hTIDAMwyQ5yWK6YRiGYWxgoWcYhklyWOgZhmGSHBZ6hmGYJIeFnmEYJslhoWcYhklyWOgZhmGSnP8H3H77vNJCoZcAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "#hide\n", "learn.recorder.plot_loss()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "learn.save(path_data/'GPT2_pt_3epoch_lr5e-4')\n", "learn = learn.load(path_data/'GPT2_pt_3epoch_lr5e-4')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 6.2.4 Unfreeze all layers" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracyperplexitytime
03.2884333.1867210.37738024.2089066:06:29
13.2325693.1678640.37988523.7566876:16:22
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.unfreeze()\n", "learn.fit_one_cycle(2, slice(1e-4/(2.6**4),1e-4))" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deZgU1dW43zPDMMO+DosMMKAoyjbggPuGqCiRkOinJJqoiSEaTUxMjPiLS9SYGP0+Y0yM0URNjAsqxmBAXFAQFRDZd2Tf931fZu7vj66aqa6u6q7qrupl+r7P089037p161ZP9T33nnPuOaKUQqPRaDT5R0GmO6DRaDSazKAFgEaj0eQpWgBoNBpNnqIFgEaj0eQpWgBoNBpNnlIv0x3wQ+vWrVV5eXmmu6HRaDQ5xcyZM7crpUrt5TklAMrLy5kxY0amu6HRaDQ5hYiscSrXKiCNRqPJU7QA0Gg0mjwloQAQkRIRmS4ic0VkoYg86FDnFhGZLyJzROQzETnNKC8XkUNG+RwR+avlnNONc5aLyFMiIsHemkaj0Wji4cUGcAQYqJTaLyJFwGciMl4pNc1S51Wl1F8BRGQo8AQw2Di2QilV4dDuM8AIYBrwrlF/fJL3odFoNI4cO3aM9evXc/jw4Ux3JXRKSkooKyujqKjIU/2EAkBFggXtNz4WGS9lq7PX8rGR/bgdEWkPNFVKTTU+vwQMQwsAjUYTMOvXr6dJkyaUl5dTlxUNSil27NjB+vXr6dKli6dzPNkARKRQROYAW4EPlVJfONS5TURWAI8BP7Ec6iIis0XkExE5zyjrAKy31FlvlDlde4SIzBCRGdu2bfPSXY1Go6nh8OHDtGrVqk4P/gAiQqtWrXytdDwJAKVUlaHGKQMGiEhPhzpPK6VOBO4G7jWKNwGdlFJ9gTuBV0WkKeD0n3BcNSilnlNKVSqlKktLY9xYNRqNJiF1ffA38XufvryAlFK7gUnU6vedGEVEnYNS6ohSaofxfiawAjiZyIy/zHJOGbDRT19S4fCxKt6auR4dCluj0eQzXryASkWkufG+ATAIWGKr083ycQiwzHJuofG+K9ANWKmU2gTsE5EzDe+f7wJjArgfT/z23cX8/M25fLZ8e7ouqdFo8pTdu3fzl7/8xfd5V1xxBbt37w6hR7V4WQG0ByaKyDzgSyI2gLEi8pDh8QNwu+EiOoeIqucGo/x8YJ6IzAVGA7copXYax24F/g4sJ7IySJsBePOeiI7swJHj6bqkRqPJU9wEQFVVVdzz3n33XZo3bx5WtwBvXkDzgL4O5fdb3t/hcu5bwFsux2YAMbaEdFCr+MkPvaBGo8kcI0eOZMWKFVRUVFBUVETjxo1p3749c+bMYdGiRQwbNox169Zx+PBh7rjjDkaMGAHUhr7Zv38/l19+Oeeeey5TpkyhQ4cOjBkzhgYNGqTct5yKBRQUpuo/T+xCGo3G4MH/LmTRxr2JK/rgtBOa8sCVPVyPP/rooyxYsIA5c+YwadIkhgwZwoIFC2pcNV944QVatmzJoUOH6N+/P1dddRWtWrWKamPZsmW89tpr/O1vf+Oaa67hrbfe4vrrr0+573kpAMw1QIGWABqNJs0MGDAgyk//qaee4u233wZg3bp1LFu2LEYAdOnShYqKyH7a008/ndWrVwfSl7wUANXmCiCz3dBoNGkm3kw9XTRq1Kjm/aRJk5gwYQJTp06lYcOGXHjhhY5+/MXFxTXvCwsLOXToUCB9yctgcKb7p14AaDSasGnSpAn79u1zPLZnzx5atGhBw4YNWbJkCdOmTXOsFxZ5uQIwjcBaAGg0mrBp1aoV55xzDj179qRBgwa0bdu25tjgwYP561//Su/evTnllFM488wz09q3/BQANUZgLQE0Gk34vPrqq47lxcXFjB/v7AFv6vlbt27NggULasp/8YtfBNav/FQBGX/18K/RaPKZ/BQANTYALQI0Gk3+kqcCIPJXD/8ajSafyU8BgPYC0mg0mvwUADUrAC0BNBpN/pLfAkCP/xqNJo/JTwGgVUA5w1sz13PKveM5VlWd6a5oNGmhcePGAGzcuJGrr77asc6FF17IjBkzUr5WXgqAVdsPADB23qYM90STiN+MW8SR49XsO6xDd2vyixNOOIHRo0eHeo28FABb9h4B4IOFm0O/1vGqar7z/BfMXLMzcWVNDGbAvmqdvU2To9x9991R+QB+/etf8+CDD3LxxRfTr18/evXqxZgxsfmwVq9eTc+ekYj5hw4dYvjw4fTu3Ztrr702sFhAebkT2KReQfjyb/2uQ3y6bDtrdhxk8i8vCv16dQ1TTacFgCYQxo+EzfODbbNdL7j8UdfDw4cP56c//Sk/+tGPAHjjjTd47733+NnPfkbTpk3Zvn07Z555JkOHDnXdm/TMM8/QsGFD5s2bx7x58+jXr18gXc9rAVBYoI0A2U7ND0KP/5ocpW/fvmzdupWNGzeybds2WrRoQfv27fnZz37G5MmTKSgoYMOGDWzZsoV27do5tjF58mR+8pOfANC7d2969+4dSN8SCgARKQEmA8VG/dFKqQdsdW4BbgOqgP3ACKXUIhG5BHgUqA8cBe5SSn1snDOJSLpJcy1zqVJqaxA35ZUNu4NZRnlBG5yTw/zaqrUA0ARBnJl6mFx99dWMHj2azZs3M3z4cF555RW2bdvGzJkzKSoqory83DEMtJUwIhd40YEcAQYqpfoAFcBgEbGHrHtVKdVLKVUBPAY8YZRvB65USvUikif4X7bzrlNKVRivtA7+AB2ap55SLRE67lBqmDYApZcAmhxm+PDhjBo1itGjR3P11VezZ88e2rRpQ1FRERMnTmTNmjVxzz///PN55ZVXAFiwYAHz5s0LpF9ecgIrIrN6gCLjpWx1rDnWGpnHlVKzLeULgRIRKVZKHUml00HRtEFR6NfQcYdSo9YGkNl+aDSp0KNHD/bt20eHDh1o37491113HVdeeSWVlZVUVFTQvXv3uOffeuut3HTTTfTu3ZuKigoGDBgQSL882QBEpBCYCZwEPK2U+sKhzm3AnUTUPQMdmrkKmG0b/F8UkSoiieN/o1SspU9ERgAjADp16uSlu55JhwlAj1upUeMFpCWAJseZP7/W+Ny6dWumTp3qWG///sh8u7y8vCYMdIMGDRg1alTgffLkBqOUqjLUO2XAABHp6VDnaaXUicDdwL3WYyLSA/g98ENL8XWGaug84/Udl2s/p5SqVEpVlpaWeumuZ9IxKdeB51Kjxgasx3+NJnB8+UEqpXYDk4DBcaqNAoaZH0SkDHgb+K5SaoWlrQ3G333Aq0AwaxofLNiwl5FvBaNLc0dLgFSodQIKTgIcr6rm0NGqwNrTaHKVhAJAREpFpLnxvgEwCFhiq9PN8nEIsMwobw6MA+5RSn1uqV9PRFob74uArwELyACjvlyXluvo8T85zIB9VQGqgG5/dTan3v9eYO1psh8H7XKdxO99elkBtAcmisg84EvgQ6XUWBF5SESGGnVuF5GFIjKHiB3gBrOciN3gPhGZY7zaEHEpfd9ocw6wAfibr55nEf+etZ7Pl293PJYnz11orN15EIB/TlkdWJvvpWEHuCZ7KCkpYceOHXVeCCil2LFjByUlJZ7P8eIFNA/o61B+v+X9HS7n/gb4jUvTp3vsY9Zz5xtzAVj96JCYY7UJ6KPXAMu27OO6v3/B2J+cS5sm3v9h+Uo692xo6hZlZWWsX7+ebdu2ZboroVNSUkJZWZnn+nm9Ezid2FVAz3+2iq37jvDR4q18a0Cw3k11E61E0yRHUVERXbp0yXQ3spK8DAbnxIINe9h3+Fjg7dbxVWfaCNtja9eBo2zdF38npkZT19ACgIiP+df+9Bnf+8eXoV1D7wNLjbD3bPR9+EMGPPJRuBfRaLIMLQCo1dPPXLMrhLYjrbvFs8/UCmH7/iOUjxzHGzPS4wWVKjp9p0YTPFoAEK6LmNn0pj3R6oVMrwjMpDivp8kNNlUy/X1pNHURbQSmdgUQhhjINhvA4WNVUYNproyrWgBoNMGjVwCEO0gv2LgnvMaToPt973H27z7OOsGk0WjSjxYA1OrpwxgUfzk67FAT/tlx4GjN+1Rn1os27uXX7ywMfZNNGDaAfg9/yO/fW5K4okZTR9ECIAFHj1fzxAdLOXjU2Yj7u/GLa96vM3at+iFTce6DGrCveOpT/jFlNTstQiUbGTNnA3+bvDKqbOeBozwzaYXLGRGOHNcxgzR1Fy0AgC173NMTvDFjHU99vJynPlruePzZT2oHlfcW+AkxkB1K7aBm1mFHaz6xtFFK598xag6PvLs4cUUL7y3YxCn3vsfiTXsdj+86cJSZa3am1C+NJpNoAQAcrap2PXb4WGQGePS4ex2TXMpaFXRPw1IBDezeBoBeZc1DaT8eExZHktTN3+Bsxxn+3DSuesY5prtGkwtoAQDEGw43G+6b/5y6OmErU1fs8H/lDMmMl6cZKegCWogcD2kJYHbPTcDsOnDU58orGFZtP8DSLfvSfl2NJki0ACD+IGz6yXsJR7x5r59Ml2aqSB+nBMjYeZsCbS/IcM1OuLX+w3/N5JaXZ7JtX/R3f+WfPuPO1+eEcvGlm/dx0f9OCqZtjSaDaAGQgAMuxl8njsdRJWWa5yavoOKhD2LKg5I/oa0AEmQEM8NFH7N99/M37OHfszekdm2X8g27o4392fx/12jioQUAsP+I+yDvJ5n78WrFrS/PZOD/TfJ8jpsR9ujxalcPlFXbD/C7dxf70rv/9t0l7D4YfLA7k6rqsAfB+Pca798UtH1Ch6XQpIu9h4/x/GerQrOxaQEAXPvsNNdjfoKQKaUYv2AzK7cdSLlP/R+ZQI/733c8dtOL03l28krW7Ih1O12xbT9ff/pz9nqMbOpXBfXZsu18uiw2rnpYKwBzHu72/DsZ3rfsjQ67kexvx+tpuWP61+Qavx6zkIfHLuIzl4RTqaIFALVeQPXrxX4dflYAvX14qiQalPYcOuY6qK52GPhN/vDhV8xdt5tJS70lv/A7m73++S/4zvPTY8qPV4WsAkpUz3Ifr3yxtub9ym37qU519mT/imyfu/1qfBpyS2vykd2HIhO5I8fCWWF7yQlcIiLTRWSukfbxQYc6t4jIfCPl42cicprl2D0islxElorIZZbywUbZchEZGdwtpc7WfYf5vw+WUl2tfK0ATmnXJLxOOeA0sG01jKHpTn8XuhFYWd+rmtj9iW7zqmem8NWW/YH2xemRSFduaU1+EpaziJdgcEeAgUqp/UYC989EZLxSyqo3eVUp9VcAI0/wE8BgQxAMB3oAJwATRORk45yngUuA9cCXIvKOUmpRMLeVGr94cx6Tv9rGed1KKfDxzadj0H32k/g7V6evimxM2r7f287coB6ssFRAHy7aAkSreioe+pA9h44x4c4LLCk3nc/fdfAYT074KtA+vTs/WA8qjSZTJFwBqAjmFKrIeClbHetWyUaW418HRimljiilVgHLgQHGa7lSaqVS6igwyqibWYxem5u/qpWKEQD/mraGJz50HlCScQaxNr986z7KR45jYZwAcr8bXxu7pklJkWu9MLKbxSOdK4A9xrJ4zY5aW0s8OfaBIUQA3k8mIbzt1j5eUvdzy2qyi7Dmlp5sACJSKCJzgK3Ah0qpLxzq3CYiK4DHgJ8YxR0A69p4vVHmVu507REiMkNEZoSd1NnLTt77/rOApz5axrIt+2KMjX9IMNOctTZ+wpn3F0YGqv/O9TbDjNdfczzec/BYWlYmxy1eQNXViic+WMrWvcGlWHS6g6pq5fuH8cN/zfRc102ohJ2dTKNJF54EgFKqSilVAZQBA0Skp0Odp5VSJwJ3A/caxU4/FRWn3OnazymlKpVSlaWlpV66mzROg4mbi+glf5jMGb/1l0IwkaGw1uDpcVSLU00pxertB+jz0Af8y9z1G+eabizdvI8nJ3yVUI1iXQHMXreLpz5ezp1vzI3fuA+chFiUDSSFQfnX7yz0Vd+PWlCjCYKwHjlfXkBKqd3AJGBwnGqjgGHG+/VAR8uxMmBjnPKMEvY82TpwLNzoHGAM4GMjBg3AgSPHmeLiApaovxt3HwLg/jH+Bjgrlz05mScnLOPJCcvi1rPaAExVmKlKS4WKju6eVZFrpv5f+8eU1b7qF+olgKaO4MULqFREmhvvGwCDgCW2Ot0sH4cA5mjxDjBcRIpFpAvQDZgOfAl0E5EuIlKfiKH4nVRvJlVCj2lvEQBmgDHrUGK6Mi7bWuu1ctfouXz771+wwRjMrcTrrlLQqDjYhG/WHa/27+qLlbVRMb26bnqhZaP6xvVij1lXHencnKUXAJq6gpcRoj3wTxEpJCIw3lBKjRWRh4AZSql3gNtFZBBwDNgF3ACglFooIm8Ai4DjwG1KqSoAEbkdeB8oBF5QSiU/TQ2IIIf/tTsO0qlVw6iywiR2XSzZHAk4dsghJEUiVdF/5yZeVPmReaNnrq95/8Lnq/n+uV1qPq/YViu0rOPjz16fw6fLtjHj3ku8Xwh4/rNVrLUYeZ3u9dDRqowE09MqIE26CHtSmlAAKKXmAX0dyu+3vL8jzvmPAI84lL8LvOu5pylwvKqa+8Ys5EcXnkjHlg1d6wX5XVc5NJbawBF7btwVAIq/f7YqhevFYrWHzF23O+qYPRYPRB7et414PIs37aVTy4aeVyUPj414BJvhoJ3udcnmfQndQP1yvKqaW16exR0Xd6tp+5dvzWNoxQmUFBUGeq2wGfzkZE5p14Q/Do/5+Wo0QJ7sBJ6+eievTV/LXaODM0omoqpaxQyKTruKVdRxf9eIJ6+8CjM/u2StdbftO0L5yHE1n607gZ3u4/I/fsotL3vzwFm9Pda905dwTkGSr9x+gAmLt3DnG9GRRJ1UcNnOks37GDMn46Y1TQr4iUSQDHkhAPxw+FhVTQ6AVBj0xCf0fCA6lo+T7TDhAJzA08dOswaRvQF94hhPrUxb6T2jlVXPbt+rYHUDnWj4yc9aG71K+GKVt2td6BBqWRFJ6/hzm2eRk4AIaiHn9tNbv8tZGCS1x6AOsmXvYcpHjmPaSv/5MXKByV9tY4FLkqBcQwsAG7e/OrsmxHCqHLFlEXNSAXm91qrtsQHmnGRHzw5NAWhgqCuCxNp9u3usdQXg+sNPYWRWSnHHqDm8NavWDmEaiCNNh6srHWMJLV3fxZjzxAfxXWWPV1Xz8rQ1dT58tCno47kf5zLffWE6X/vTZ2m5Vtg2AC0AbExYvCVxpSRxGsSPHU/wDzYG3R+8NMPTNcxZesoB0BJg3/i7fX9tQpYCFzfJZAbpeB5FJ7VpHJUvYNy8Tfzv+0tDMQw/9XFtTmi3+0i0Wn9p6hru/c8CXpoa7sC468BRbnoxNmBfpjhw5DhD//wZSza7uz5r4pMV+wDyDftA4jXEshs7D8TG57EOJn4HrjctXjk3/3MGY+dtTJhAJSysEUoL4zyty7fuZ+KSra7HXXG4H6VqVyJKwW2vzuLPE5cHIvySsa8kMvKbkR3NUBZhcd+YBUz0GA02HUxdsYN56/fw2HtLM90VjQ0tAHzQ+9exGbX88m+LCgPsuuvYkSVeboGZa2p16hMWb+H2V2fXGI3CXgHEw22j1LEqxaAnPuGmf3zpu02n72b3oaMcNsLkWu837DvvWtrYsbwgwa+pxl4RbHdi8GprCYqPl2ypCdoHtWoL+/2mO0KtJjF5JQD8Pn9hLLvs4RFKmxTXvP9jgt22dhJtBPPejrfKXoWKmwooOdwTwvzq7QU1762Hgxhn4t1BPZf7K64XvN0lGeKtwMLge/+Y4aiiNCcjpsNANq1KNBHySgD4JR0TFuts2W40tuPHO8nPCuB1j7Hsl2za56leYQrjj10YebXJVFuMEm6pNH31I86xZENBpGtczrZIFX6e24NHj/Pa9LVpWS38d+5GHvxv8vtPdx44SvnIcZ42XGYrWgDEIUzPkg7NGwBwkos6wYlLnvgk6vOXq2OX+rWhrL33xc2t0U49jyO7nwGyqlqxy2Ibcfvdj/z3fM9tvvj5as914+E2YF/VzzFwrfdgFGF7dgQoAZ6euJzykeOS8lwye+EnVPhD/13EPf+ez5QV4buQ/vi12Sk9K2Zujh+/NjugHqUfLQDiEFYaNoB2zUp8n7PP5np5zHC9tP7ACmqMwMkPMks27+VPH8WqowoTKblr+uB9AHp47CL6Pvwh+48cZ8f+I2zZl9wejLANq1acUodC4hl+uuIVBRms7k8fR56Doz4EgP3R8zMZMb3JDrhE4c0mrKFRwiLsdVCw0cKynB0HjjqGLHDjrVnh/YODXOL+ZFTtDGTrXiMlZJJtbdt3hMFPfgrArReeGHWsyOMKwI8AGDsvkvvg4NHjDHjEX3htK8u3Bpv2EZKwGXkc4MP+UQdpAzC/g2SElwQwGclmdjh49XnhjxOWsXnvIX73zd6ezwlr8pBXK4DlW/fH7CSNx9EEOvlkqapWNbtkX5yyikVxQkN7Ydaa2kQzK429Bn5sANbx4sevzbKURz90XmeWHVo08HztWo+R1B7woPXrYfzc0mYDCHAFkEysJfvMOBmPNBFh4cY9LN/qze6US/xhwle8Nt1fDumw1NF5JQAA3vFhsAkrz+35j02sef/58h1c8dSnSbc1c81ONjkY2fx03fr73HvouKU8uhGvg/RJbbzbNYIO5pYqz36yEkhylp4l9+C03yRpkvgiPrPlr0j2WRzy1GcMemKy/w7UIcJ+pPJCACQ7u/SjLvJDkIHFrnpmqmO532X34+8voXzkuKh7nppkLBc/iWDMwergkdQ8d4IK0Rym2s8kbI2IPUxHGOoxPyRzu27/zSkrtkftOq/rhJ6kKuT2c5ryVo0y3YWk8TvImN4QVlfUeev9BbxSSvGz1+cw2xYEzgvnPz4xcaU4ZMMKQinF1n2HXd1Q09VFuw1g3nr//w+TVFQPZi/8qIAS1fz2375g+HPTku5TrqJtABng8p7tMt2FpEl2J7B17HjLpstNNMhOWbGDt2dv8KVmC4p0ZgRz48vVuxjwyEfc9sqsuPXCDlx3QvNoD7NUhGMqqxXThhS0EThdK5pPvtoWSGTgbEYLgDiEZAJIC770ri4D0kpb8LpE48gHGQyHHMag6jZwJhrPJix2jnVkfp+vfxmumqlt02gBkIp6zH6rv313se+NT9UBaVLT7U10wwvTGfb052m9ZrrxkhO4RESmi8hcEVkoIg861LlTRBaJyDwR+UhEOhvlF4nIHMvrsIgMM479Q0RWWY5VBH97qZHJeDqp4mvZbam6ZkfyobBf/mJt0uemSjr/VclODMyZa9g6bPuAn0pSEfug+9zklZ43PiWjAqo51zF3RuRvOnc6b96b3ArgeFU1Px01m6Wbo72YXpq6OvVOBYiXfQBHgIFKqf0iUgR8JiLjlVJWRdxsoFIpdVBEbgUeA65VSk0EKgBEpCWwHLBGVLtLKTU6kDsJgVwWAH4mxIpgBtACgdSDMCRHOndjuq02+nRsHpMq00rYA9eNL05nkkO8nSCum8rz4c8LyL2y+XtMR07m+T7tX3aWbtnHf+ZsZMnmfbz30/Nryu8fk/HU51EkXAGoCKbSrch4KVudiUopc+o4DShzaOpqYLylXtpIVj2Qy+P/C5+v8lz3mUkrOOTDc8cNPzPN8fM3sS/F8NqZwm0AamVJUOPE3BQHlUQ4Df6Qmn3Ez0/gjRnRvu0z10b2p3hV3Sil+HRZxIXUeQWQPgEwdl5qdizzlpdszu59DJ5sACJSKCJzgK3Ah0qpL+JU/z4w3qF8OPCarewRQ230BxEpdjgHERkhIjNEZMa2bclFE/z3rA2JKzngJ4ZJthHGg/dVAuObn5nmra/MiomMmk0cOlrlOgEodgsFkaDNvp28pekMmiCMwF4mUb8cPS/qsxljyutKesLirXH33tQ0Y7uf5yav4N7/eI8V5YVs+eWHPQn1JACUUlVKqQoiM/sBItLTqZ6IXA9UAo/bytsDvQBrktx7gO5Af6AlcLfLtZ9TSlUqpSpLS0u9dDcGux7OK1VpXAKMM0IiZDOTv4ovgP3OzNYFlHozDOLt1XBNCJNAAoaRptMLYaieqj1MjsyB3+s8KpG+3fze7ffz23eX8PK0YO1Pfid/9r0vqaqPq6sV7y3YXCuIsiEjmFJqNzAJGGw/JiKDgF8BQ5VSdivXNcDbSqmaNb9SapOhXjoCvAgM8Nl3zyT7z/DykAfFqC8zZ0DV+OOwi59/v04t4p43J459IEySNQLHU91c9/d4SgDz/MjfZH5/Rx1SpaZTBeS3z3YBkKr24B9TVnPLyzNrJ10hDUVevIBKRaS58b4BMAhYYqvTF3iWyODv5AP3LWzqH2NVgESezmHAAofzAuGqfk4micTcmsCfO0isSdVzlYNH/dkRgrA7hMm/ZzurDq2JaOKx/8hxykeO4/nPIvYY6/dzrKqax95bwqSlW7n6mSmhxZ2C5AfMbftq53H28dDPLnHPg6ml3qY9sSswuwA4+d7x/HRUOMb/VAfwVM/fGGC0gHh48QJqD/xTRAqJCIw3lFJjReQhYIZS6h0iKp/GwJvGbGOtUmoogIiUAx2BT2ztviIipUQWN3OAW1K/HWe6lmb/jt5ctjckSyoup+nA7/9kmS1w2UwjSN8j4xZFDaYQSQ36l0kr+MukSEz51TsOcHLbJin01p1Mb5ELYh/Aym37+dPHy4Fam8bR49X8Z044mw7d/vfTPabbTDWOWLqGg4QCQCk1D+jrUH6/5f2gOOevBmIyaCilBnruZYos25LZWChesMf61+Qeh2wroBtemA5Efsx/NZKHmBy1rfjCHKQ9pnGIy84DR2lUnFz0eLcVwMSlW7npxS+Zes9A2jdrENda/cN/zWSZ4YQQ5mrJxN7nx99fwrqdh/hqi7M90X6LMxySNTlxvKqa+8Ys4LaLTqKsRUPX62eFDSBXeeTdxZnuQkIWb0otJLQm8/haMdgjrYYoAbzaAKav2lmjroqcWPu2aUlR0td3EwCjpkfsXjV7JzyqihKlTg0C+//y6Ykr4oY4sd9j84bxXYJNpq/ayWvT1/GLN6M94tK16zkvBIBGkw5S8/wITwLEs2SDjBYAACAASURBVAGMmr6WVUaIimuencrDYxfVHlTWt8nfm5tcNPcnOH1tMRPgNOmxRs9cz83//BK/gYDt6Vk7tmzoUtOGcV/TVu6MCtqXLhVQXguA9y079DSaVEnFjpOsq+bomesT7oKO1/bIf89n6J8/czw2e13iAWn9Lnc7jml7s85mlVI1+YVrMoY5nJspi9gv3pxr7EfwJwHsOv8ORkC+kqL4Q6x1k95THy2veR92wECTvBYAbrldNZpkSMWRK1FO45+/MZcPF22JKf/Fm3MTBmdLtBN43+Hj0TN/g/9YvKDc+hfP88vUaVvHxgfeWchJv4rsEzUFQDaGXPGbD9xcZR09Xs3bs9fX3PPhY9WUjxzH/32w1OU85/b0CiANZNo7QlO3sG6Uc8sJYPLl6l1Rn7/xlylx6781az0/eGlGUv1yG2SsM/Mo3b/Bym210WCTGaR7ntA05jovTV1T8742XLTvplMmUdIit70ebjRrELGR/PGjr/jZ63N5f0F0ZNx/GPk27Bx10TVpG4BGk8Occu97cY+nM2eCmxE40RjTsLh253Kj+sl5AEE8G4DRDw9tBJ3v4cH/1gZlcxps/eYBaGnEgdqyN+Luu/Ogt7ScVs+xCYu3oJSiulqlTSjmtQAI0rC0+tEhwTWmyUoSzeqzFftsd+u+yOCWaFZ/cpvafQnJ6KS9qniSme2Wjxzn+xwrKyyrm2uejaRVtX5Pfu055i2I7XMNLmON3UD/109W0vX/vRujcgtLW5HXAkCj8UMiPX228syk2j0IM9fsZMAjHzFmzoaEQ3rzhrWun17H6KueiVZl/WvaGhZudHZxtg9+CzYE5wo9b/3u+LtpLfdjV8cBrPEZp8ouIO1Cz20An78hOkLs60ZImC1J5iHwS14LgHT4E2vqDumIQeOFg0eP882/fM6SzbUDZrzAeserI4bI8pHjWGQMxtNX7Uw4qPcqa+a7b+buZ4gIjfv+4xw2Y9OeQzErhNdt4aRTYeifP+fsRz+OKVdKMXON8yYt6/fhd7OZ/bv0ajOZZAuwaAYUnGXLqx2WRiivBUA+hl/QJE+2CIAvVu5k1trdPDq+NiRXPCOyU79FEqt1Glt2/gb9Sznrdx+7q0sIL+/vS1PXcNUzU5lu89tftHFvinsd7CuA6ON7Dx+PiUp8vKo6JodEup+x5C07Gk2ekc5UhPFwiooZL83kjDWxKg6lEqt1rMZjNz19vDYOJ3CljOcF9Nr0tba6cZvyjJtgueKpT/n1lacl3a59wHf6vi57cnLUZ9Md1soqWx7u2gaT7lpc8noFkIXux5osJl6OgCB44oOlNaqcUdPXUj5ynGPWtGqbwTEZZq7Zxb0u6hmT6A1c/q9x8Gj8+FZvG/sMDhw9Hlfn/fTE5a7HgiSVJEqmNqFGrZWkdtlNKxHWxrC8FgCaus24n5wbaHtDnnLeMRsUT328vMbX/2+frgTiuyMmNSs2TlqyeR+jZ653rbb38DFufPFLr805UuIxAc6fPl7OgTjBEB9/33kTVTZhn/EHnUwqrMmqFgCaOkvQvuOQuvthIkzjY416JE7dsPTkANv3uauUrMTLY+HVvfPwsaq07XyNJ7DcNmV5oXZVJsbnYG/oqY+WBdqeSV4LALdl1S8uPdlXO528Bn7SpJUssdkmhbXrbgPp6iTyKRzxmITHbow8Xq0cM+TF+447tGjg6VoRtUfm9bHJ5g6HyF6C8pHjOGbofoL2MJy7fk/iSkmgjcAOnNG1la/6LRomHypXEx65KADMYdCMfT9n3W66tWkcWPu/GectNLpdAFz0v5PoXx6b8jKIPMfHqqrTtgJIlUShtTfsitiJciHHN+T5CsANv7mAc+TZzTvCUAGlQqP6/gfLX46eF6P/TUecGKdxzr5h6qWpq3l28krXNrx281iVSqgyScVAa+XAkdR2cyd6orLFVdgrCVcAIlICTAaKjfqjlVIP2OrcCdwMHAe2Ad9TSq0xjlUB842q1lSRXYBRQEtgFvAdpZS3ABoh43d/wGNX9w6pJ5pUMAN0ZQvJJmffnYEdyLcnCDENcP+YhXGP+/kVhSXTTJuNGarlrVnuhm834m2ys1OYLb7CHvGyAjgCDFRK9QEqgMEicqatzmygUinVGxgNPGY5dkgpVWG8hlrKfw/8QSnVDdgFfD/puwgYvz/U7u2ahtQTTSq0a1aSsM6PLjwxDT2J4OWpWrX9QIxHjD1BerKCxA9z1+1OXCkBfgb1216ZFcD1FIeOViWM9OmXdZacB4nCgeSaAPCSE1gBprtBkfFStjoTLR+nAdfHa1MiT/BA4NtG0T+BXwPPeOl0ULg9oDn2P9SkQDaq777/z2j3y1zdr+Inns1Ktw1QPqhWcOr979VE5gyDjXvi7wXJNQHgyQYgIoUiMgfYCnyolPoiTvXvA9YtbiUiMkNEponIMKOsFbBbKWVOddbjkDjeuPYI4/wZ27Ztc6oSOOmYYaWTn1/iz6spnygqzD4z2LSV0WEKclUA/GPK6rRez7SN7DwQsCbZmhoz0e7pYK8cOp6efqVUlVKqAigDBohIT6d6InI9UAk8binupJSqJDLbf1JETsT5e3L8apVSzymlKpVSlaWlpV666xm3f2YdG//58cXdOK29VlM5Udbcm6tiJrEbSMM2Au/2GMs+2/jtu0tcj42Zk7yL54s+BFmujR2+pj9Kqd3AJGCw/ZiIDAJ+BQxVSh2xnLPR+LvSOLcvsB1oLiKmCqoMSF+GDPOiLn7KOfY/9ERB9k10M4bpunjdGZ0oTpCzNRuwCoB4u3eD4skJ4Ww6CpsXPo/NamZyx6g5SbdrD+IWj72H44e/yDYSPv0iUioizY33DYBBwBJbnb7As0QG/62W8hYiUmy8bw2cAywy7AoTgauNqjcAY1K/HX+0cNEV5poU94LpEtmzQ26sBMLM1/zcd09n9aNDeOQbvaiXA5LR6pQ2c82uOqeirEtMX+UcajoIwlj5eXn62wMTRWQe8CURG8BYEXlIREyvnseBxsCbIjJHRN4xyk8FZojIXCID/qNKKTP79N3AnSKynIhN4PmA7kljoXXjiJB7eFhP+nZqzs3nds1wj7wRpt3ivG61qsQsNAE4kKNGgDpCWIHY/BJGQiIvXkDziKht7OX3W94Pcjl3CtDL5dhKYIDnnqaVujPDalIS8YWv6Nict390DuPn58YOxTDj3FgpTOMKYF+cgGfxSPcu2bq6wAg7jlPYhGH6yYn5jyZ57D/mYzmy537i0vR4fNUrzP7Rzu/O9FTJtd2sYdKiYRFVcQLepZMweqEFQBoI0y85EfYf89i5abe1J0XQm3ncqJcDftvpltlTVuxI7wWznI1xQnKnk50HvEVo9YMWAI54/8X9/ipHDVcNnVs1ZMxt56TaoaSxj285sgBIWQD84Lwunup1ad0opeukA7vx71/T1oR6vcWbgkvOnuvsOngsa4I9rt8VfEIiLQAc8KprW/3oEK7t3ylunUIROmYwXLR9BWBf3f/2G/EFWKZ4YGiPtFyndePitFwnFaxCe/fBo0z+Kj3qMU2EXQfTH4vJifJWwU9WtABwwEsMGa8M7N4GgAHlLX2dF5Rmwu4yaP3UunEx3z4jvgDLFIUijL7lLCbceUFS59clV8njlvyCQcTo0eQmYdhm8l4A3HROOQ1tYXob1Q8uTUJhkkbGVP7Zl5zW1tJO9DFrs51aRjbCVXRsnrBNU5ClCxGoLG/JSUnGwk9HyOR0Yb2VbNFHa9JPGO6oeS8AHriyB+d1ax1Vlg1DRyoC4G/frax5b89M5BQj/4azOyds05r4w2r3eOvWs5LpYkJSnetkY4yfZEkljIGm7hBGoLm68yvxSceWtWEgHvlGr6iZcTbMHpN1T3/hxsqoz3Z/eqd2v9G3jHZN46u9LjildvNUcb1aYXB6Z3+qLa+kutpt0TDa86rYZWdxLmiK/jMnNzy3NOGi9wEEyOS7Lqp537pxMfcOOa3mc6Pi5FRAr/3gTF79wRm+zpl6z0DH8kKXkenFG/u7tvX/rujORackp6r5Rj/HYKw1nGVJk5mOQTPVbF43nlMe9XlAl+AFlZlkRKPJVfJWANiNhFZ/8PpJqg/OOrEVZ3RxySfsMp61b+YckM5JBXRCsxLXdpoU12PE+Scmbfy88exy12NPXNOHji0bcrFhB8gFA6tXFVC2pY3UaNKJTgpvUG7xB09lpeU2nPgdZrqWNmLu+j3RbYi42gauOzOxHj/SD+fz27qogD6560I6G+5nT1/Xj+37jzBrbfieKDkgYzSatKJVQCFyhkVFkIoNIJmBq5XDTuFOLj6/Ts0vf+Ry7h58ireLpTCwlhQVUtaioecm7MZ1PyurdK0ytKDR5AortwcfH0sLAIOSokKKDJfNqjQbgZ2uNqC8RUyZiLNqqF5hQWgDZrIqki6tG/Hnb/WLKpv8y4t469az4553qpG4xmvY6qW/iUlNodHUSXQsoJD53rmR8AFuBliTLq0b8YtLncMVBzUQN20Qu/08IgCiyxLtZLVnArMnHE+EUwhaL7d49omtaNawiL9eXysE2jUroU9Zs7jnXd6zHUseHkz3drX9LsmBpC0aTS6if1kWRg7uzqrfXUG9BKqKp4b35faB3dLUq1ouObVdjArnf/+nd9xzGpdEm3mG9+/o65otGsUKot3G1vh4wscUEv06t7CVJ5YeJUXRG/Pm3H9pwnOSRWuANPmMFgAWRCRwVYqpQqnyGYXN3o8vfzWIXw05NaZeA9tgmYjGxf4CWzl9H5v2RIJSbd8fiU446FR319M2TaKNy27f7ojzu9KofiHDKuK7o8b0z6HFM7uGszchHk1L6qY/xeAe7TLdBY3BKW2bBN6ml5SQJSIyXUTmishCEXnQoc6dIrJIROaJyEci0tkorxCRqcZ580TkWss5/xCRVUYGsTkiUhHsrYVHMluyZ6zZ5d6eg83BPqyVNimmsEBiBrxEuwPtR4sCiH9v7pNoYgx6z32nMqaOmxmlwKW//++KU1n40GA6tfIfOO+9n54X9dn6HV1TWRb33KAEfrcQfpzZQBi7TzXJcUJzZ5fxVPCyAjgCDFRK9QEqgMEicqatzmygUinVGxgNPGaUHwS+q5TqQSSR/JNmfmGDu5RSFcYr+azNdYgfDzwJgDZNvEWp9PsDLfKZa9ep9dmGG+g+IwG226AeFPHGaBHo3LJRTJlJtzbpGZjTHSspXYT9v9VkloSjgYpg+h8VGS9lqzNRKXXQ+DgNKDPKv1JKLTPebwS2AqXUcZ64pg+VnWO9eNywfpl3XNyNV24+gzO6Om8osw+GfpOaW10xk/Uq8JJEJUiPhUSeSPbvJJ7A6GxbYQQ1vN16wYkBtZQe7h7c3VO9HEiYllF+dUWsWtbKdzzuz8kUnkYPESkUkTlEBvAPlVJfxKn+fWC8QxsDgPrACkvxI4Zq6A8ikrbA7B/+7PxQ2/9mvzIu8jAjNBO2W6lXWMA5J7V2qB3B/ntMpMGwD8TWHbL2U0deHjsoOLV/aY9ItNFOcfIcBOlJm2iVE5PzwGFY79amCd87pwv/vMlfGuoHPeYlyLWZ8mU92iauRHbfVzYkarHGFHPi/itPi3s803gSAEqpKqVUBZGZ/QAR6elUT0SuByqBx23l7YF/ATcppczwlPcA3YH+QEvgbpc2R4jIDBGZsW1bMIkwskVfe9M53rJWxcNvRiurDcDLGO00mJa1iAz8/ePkOLCH2LbiZ3UE8QWA4G0FUCCRH2N5a3d1kRM3nF2eFQNN0Hi1fWRzfuBsCEmSqA9B9dCvs4dXfOkPlFK7gUlE9PlRiMgg4FfAUKXUEUt5U2AccK9SapqlrU2GeukI8CLgODVTSj2nlKpUSlWWltYt7dH/JDBQeiHRgxFrBE7d8auycwseu7o3D329dnZcarNZ/NxlnwTEN4gnwmkwtt9jBx/GMi+DSCqD4LCKE5I+N0y83lEWLwCygkTPRjYIqXh48QIqNQ23ItIAGAQssdXpCzxLZPDfaimvD7wNvKSUetN2TnvjrwDDgAWp3Uqw/CegPL7x/v/mw5NIXZLICOrn3Pr13FVATuw97LQRTLimsmNU1FR72IeGASbVsTKsb6ybqLlCuHfIqfzlun48cKW/dJJ/ua5fjACD2uV9KmqQn13iLgjDYtCpidU7Xselvp38rdbSSXU2hG1P9PtLTzeSxst0sD0wUUTmAV8SsQGMFZGHRGSoUedxoDHwpuHS+Y5Rfg1wPnCjg7vnKyIyH5gPtAZ+E9RNBYGXLFmpYt/w5MZvhtVq3GJSPCZcgkYfL4pjBHb6PaWanD1o7H00926sfnQIN5/XlSt6tadBHPWTE1f0as8DDrpac0d4op3h8ciECuXvN8S65drxGuLD78bBdHJymjy84pF4BZCmjiRJwmmaUmoe0Neh/H7L+0Eu574MvOxyzDkQfh2hb6eIAOnvENMH4JWbz6BxgrwDw/t3ZNSX66J+rH4fqBivIZ9uHZ7dTI2BuXXj+rz/0/hG9p4dmrJgw15f/Qibr/U+gYHd23Da/e/XlG3cHUm/WBd94b0+R9mswvjJxd24/vl4/ijhk3gFnr3fH+idwJ745/cG8P+u8OY2Z3L2ia2Ze/+lDOweWY7bvS7sxkgn+hirELvrohcuMjJ4dS2Nvo5bZiw3/A5+Iy8/lVYJ4hMFuXIP8ufVsH69SM4Fg6NVEX+Fg0f9xU+yks1eNLlOQUHtZsRM4aQ6zCW0APBA7w7N+Ga/WoOt1wGsmcVgGW/J7RZ+enj/jrz30/Oi3EK9DidnGvsI7Ebi4nqFPPrNSE7fItseAnOyYk0P6cXn3y9ZoLp1xWnGtutgrB3ECWvgu5r2gAFxvKUyRdAT069nwNhd2rg44wm8e5wQP7hhUISREB60APCESPTAG4Re10sLIhIVFdMP11R25IwuLbn5vK6Ox2654ESeHB4dfcOMV/T1vrU/Zr9LWC+1k32UO7dqyA1nl0fN0oMmFXXP4J7tefY7p0eVFYjQMc5+iaBp29TbjDRo1US5S/6KMOnWtkkoodu9GNFT4XfGBCwbqJsRrFzo07E5R49XJ65oI2JotH4OsFP4GxDNa/fq0Iw3fniWa70WjerzusvxggJx3PRlfjcl9fz7HPu5h2QS7nz88wto1biYZg2KmHLPxZSPHAcE/79IdcFz1onRO7jTrQK+ttKb0TbobmXKTnLwaPBOCt85qzMTFm8JvF2Ti5MIGxJW6tK8EgDPXn867ZKYPdpXAMnQwpb1q0XD2s9ndW3FB4u8PXAdjU1Yw/p28O3tkog2xuwx3g7fRIQ14HUtbRxOwzasqzs3A348mpYUcUrbJizdsg9IvxvgD86PXfE54eX/5Mf2pE0duYlWAXnAb/gFJ+772ql8s1/Eh71Px+ZRg/d9X/O+XbxN0xKWPDyY751T7r8TCfhW/048f0NlTT/B+wB2/skRO4UXlVWgRuAE/wy/utPolV5yo9odg2pzRaTbC6RJibddy4lmlH3KmvHJXRd5vu6mPYejPjuFOQmCRJ5zfslIGG+Hrz5RBrxGAd+3SV4JgGR/iwV2FVAS87qG9etxvREYyn62X0+RkqLCUAaWggLh4lPbJtX2N/qWMe/Xl3LaCR4EQAAGLa9hrU1h4/WWrCuAZP3/Ky0rh2z1AhSBZ79zOj9326jms+P2fBfWZ6hRgCvVoL/Oy3u2D7jFxDiNH7dfFD/B1Ju3uKt7UyG/BECy5wkp+eKbuM18U0lCn0009Tj7DOJ237n9XO667JTUG7Jh1WUnq9e2JsHJ1lg6AlzWox0/vth54Jm7brev9uzflfXTgQD19LHRcIP/flPZ+OcFp+YTXdJvzC+v5JUASFYC2H/E3dokq4+OjHz2f3a1f7t02gjazgDBeO6d2r4pt110kuf6yaxqzFO+fUYn3+fWtAHcFIK6LmUCHuPsAiAIwXe6Q9DAIFe+b95yFiu27Y8pz4TM3n/Yfa9J+xC93vJKAHgNveCI8VA0Lq6XMGewx6ZqyIaYJm60bxZ8FiJzxRO0PjcIrAOX+b5+Cv/vApHQ9LfxsMdmshO0V4l9wG8eQATVNTsOxpTZB2fzl9Osgf/r9S9vWROY8KeD0pfj2+mbd0sZO/bH5zL2x+eG1pe8EgBeVRR2RIKZFZjJWxrb+pHNAiAMzLu99cJIEpVk3OLCwkkFlJKLYwAeZMnw5LUVfGuAu0to8O6ztQ0O798xypEA4JYkEuY0qB87PNm7bU4mfvuN1Hzr0zkZcVrFtGzkbDTv2aFZwp31qZBXAiBZgtLj9i5rxj2Xd+eJa/pElZsCwB62oc5iSIAw9LepYu2S+X9PRQBk6hZbNS7md9/s7Xo86G5Z73PUl+tqHB6cjnvFKdS5ffC867LIfha/i7RvGlFlr+wT2fRon4OFGeLB6auo7zNES1BoAeABcXnvux0RfnjBibS2SXRz9Re28SlbuOeKU2nRsKgmyXU23bbV5mEOWn7jJ1mxbyJ0wmv+5yAJ2ovM2txdl50Sm6UtKAFg+3zrhSey+tEhvu/niWsju+BN/bp9R/Hndw9k6W9i0p6ERiKVXVhoAeAB68MVhrLG1P9lq8dI0FxyWltm339pajaZkPjj8NrAt+b/o7MR5uD8k2sTEnldrXmZ+b6XIHqqV8pa+EiEE8gVa7E+u2UtGsRmaUviik4rL7efSND3U79eAcVJ7Ij3grMXkDA6JFfPeGgB4IGwl/FmjP62IVr7Nd5oawmEZw5A5r+/lUVP28SjztjLwNeyUf1A1GF+vEUCn2vY2rNPZiqT2FU9vL+T91WkXXvK0WRXNKYNIbWVvc/6WZQmJvvcMLIQ68MVxr/upDaNefzq3qEHocoUv7+ql+NDn479Dxd1b8Pvxi/hil7+N/yYG/TMoKlRxnrjmfjzt/uybd8R+6lR1eL94D+560IAPvr5BVzw+CTffTT54/AKzj3JuxrB2qeupY1Yue1A0teG6AFfRGIEgBmd1g+dHEJRmHLS/o2m+rtM6+Lbdq0w3TwToQVAlvA/HoN45SLXOs7k0sPJbZuw+tEhSZ1rDjbmYGb11DN/w1/rnXwY5GYNimrUS51bNaK0SXFcYWLnwaE9eOCdhQB8vSI2VWZcAh7w7AEEg2i+e7vYjF+uKqCAN2eGxc3ndokpm3rPxenthAUvOYFLRGS6iMwVkYUi8qBDnTtFZJGIzBORj0Sks+XYDSKyzHjdYCk/XUTmi8hyEXlKsj11jibvMB/IWgHgf7QQwXGzkRP3fe00X+6I55zkf1ZtEuSvbfWjQ6IyzYlD+8lcr3nDWNdIc+ViHy5S3W2cilrGz2PRuXWjrHJ68GIDOAIMVEr1ASqAwSJypq3ObKBSKdUbGA08BiAiLYEHgDOAAcADImIqA58BRgDdjFf6TO4ajQdM1zzzB5uMykoQ9h72llBmaJ8TWPDgZb5aTxarimaEQ84Iv9jDpdsH6OJ6hYywRCp1i+X01q1nx71OOtxq/Qr6k3xEBijI0L4QNxIKABXBnMIUGS9lqzNRKWVu25sGmOmzLiOSRH6nUmoX8CERAdIeaKqUmqoiv6qXgGGp344mN8mmn0QtpgAwB0sHE0BCwhyw3DYPecHareEDOvHiTf1T71ACbjy7vOb9sSrnQdYa/qHaYXesKVjsX2uy37NTL7yO/98aEFFtjhpxpmM2OJMzutRmhLPbRl4fYZ9LpxdPXkAiUigic4CtRAb0eJmYvw+MN953ANZZjq03yjoY7+3lTtceISIzRGTGtm3bvHQ3FMx/W3EWui5qwsHMcmUOLtaZodfxRkQ4MaRcBqnsXo1100yNKOHo0prf9VNBgXDpad4cI1L1rLF+H1772bdTJGd368bFnNutNEFt4zpEr47OSMI4HiSeBIBSqkopVUFkZj9ARHo61ROR64FK4HGzyKm5OOVO135OKVWplKosLfX2JYdBk5Iifjn4FF7/YWYldl0iWwNgDDPy29bOsGONwPUKvHlQC+4bvTKpC47dqBVcZ4K8L/sOWbe2kw0x7jTb96oCSuY2C8RdVJ3aviknNCtxD9EdAr72ASildgOTcNDXi8gg4FfAUKWU6cqwHrC6t5QBG43yMofyrOZHF54U2mxOkz2YA705SPbrHJnpffes2vAGTT0GHxPBdaS4e3BsWk6vlBQVpF2AeE3yMn3VTsdyrzaUoX1qPavs+yO83vN1KURw9SpLLu3RrvYUr3ojcb+HRsX1mHLPxZx5YvpWBV68gEpFpLnxvgEwCFhiq9MXeJbI4L/Vcuh94FIRaWEYfy8F3ldKbQL2iciZhvfPd4ExgdyRJhB+cnG3mJAV+cLN53WhQVEh5xh+9W2alLD60SGc162UXh2a1dTxgog4qicWPHhZjQ45Gfxk63IiGRXQb7/Ri9aNE29am79hT/IdA576Vt8a191C20rLVb1kG3+/YxHW8XBaOXhdTVhDhHhdEZ7lQeWTzogAXnrdHpgoIvOAL4nYAMaKyEMiMtSo8zjQGHhTROaIyDsASqmdwMPGeV8CDxllALcCfweWAyuotRtosoA7LzmZGfcOSsu1ssktDqB3WXMWPzzYMSCYuTu4yEf0Mev9mb7tqe78te5YTga3WD1N4qRIPLltE2bce0nC2EXmdzTE2Hz3A4/C0olkvqdZ913iKTUpWDPGCQONqLTJ7A1oUL8wYdau135wJh1bNkxor0gmtHWyJLQiKaXmAX0dyu+3vHcdKZRSLwAvOJTPABxtCRpNtuNHaFkH2399/wwWbNgTSByklMIXxHz23tqVFSfw7CcrXY+bg7ZppDbVpskMrIWF3lRA1qaT9Y7q2aEZHy/ZGpXRLR72vvQvb+lc0UC5JISyY8q8sLKARV0r9CtoNC7kYhqEZLps/b23blyfizKQ/+CBK0+L+pxKUqMrEuTRTSl/gr0tl9EyjEfnjou78c7t59CrrJmn+n49j7zWNw3y6cgTxCwmBwAAFA5JREFUogWAJuNkmQbIE376HL1JyvuZl57WNsq/3Z61yk9bieJMVXRqTpfWjfhlHMO0ORwluuxBY1dusp45VuLlGraSalwpMa7Vu6y593McOtMigExojYxAd93axIbCCBotADSakEnWR/2571bS1eJ15sfuYKcgway8cXE9Jv7iQip8DIBuzDTSLJqkYuPxaxC9so+/2Ex7DkV2abuF6+jY0j3EtlPPJv3iooTB3RLmh2hawqs/OIMnh1fErxgAWgBoMka/TqZ7ZXlmO+KHZGaaKQyAy7fWDkzmwFFSVOC72SC0MrWhk5NrrEUSuvl6LiEjUpnxj7y8dpUzYdEWAF75Yq1j3dM7+Qtj3axhESe3jZ6523vq5fs7+8TWaUlTqQWAJmO0aRpxrzw3Q9mQUsGP+iUoL6eaQGjJJFfx2Akv1bzej32MNge0Nk2Ka9QcifC6Ya3Gm8fh2AmWGfnqR4dE5SdOpGffsPuQ6zG3vnSzxQZ6cGgPzuraqmbncDZ5vWkBoNH4IFUjsF+sLpfmwGH/66kPlsrmQOSXeDYArzPyt249i7E/OZcij2k27SsXM4jc5S75HZz69j2HEMwmiXodzxPH69d/avumvDbizKzMgKcFgEaTBP6MwMmLgG37Y/MDJLNRyGpMtc6Ak+G09k257aLk2ji9c0vaNCnhgpO9hXWxG4Hr1ytgxr2D+N03eyU892u9I0Ii3vdv7nto29R5b4N9I5qVZHMTZNECQAsAjcYPjhPd5RPgk8dg7uuwdhrs3YhQXXM4lR+80/XM9vwIFus4muomMhHhrsu6x5SZfNsIw1CzYnD4Brzqt+2CQhBaNy72ZBA3N/LFW50M6BLZmfuzQc7xd+Jdxu37T/R/yabUJzojmEaTBFG/4VWfwudPRh1fWlLEuurW8K/nqd+0Iz8sPMJ6VQob2kHzcmjY0rcyuGYQT2L8sA46yRqEu3rcmGTPDpYKTUr8uVVax/pr+3fkxc9Xc1mPdvxm3GLH+qYNoKGLQPJqO/FD9gz/WgBoNKlzyYNw4UjYvQ52r4FdqynYuYZOu9bA3rXU2ziLe4oM18i/PRX5W78xNO8MLTpD806W951pxCEOEOt+WGsE9k9hgTDo1DZMWLw1JqG6V7zOXN2SvSR3zeTrd2/XNGE6UHN14CYUE7nP5jpaAMRh2j0X1/gJazQQZ3NTUQMoPTnyIvqHdeRYFZX3/ZuOso3xN3SGXWsMQRERFqz8BI7VJmVfWAI7VWN49iSeLqrPOtWG9aqUsh076SqH2UcSCe4F/u9/Kpi4dCsnxdlg5DTgvnP7Ocyy+fYDnH9yKddUlsWU+1HPJCImZEXA47G5YnCzqyRjb0lkEM8iDZAWAPFo16yEdgk2dWjyE7+umPtpyGLVGbo7zEiVgoM7Yfdq2LWGR197n46yjesaKrrLYgYVzKJYjsPcF7ncHDf/tx206MwfiuqxTpXCrJ2RlUSLztC0DAqjf9oFIjRrWI9hfX0mjycSHM9ph+xL3xvgWN/Nd9/KbRedRMcWDXl28gq+2uKeM9k+WAYdHaG6Zm+DM05eSG7ZzLyibQAaTY6S1D6wRL93EWjUKvLqcDp/rYqM8td9ZwgXjxyHUE0bdnPfuY34aMp0Ti7ewa0n1Yfda+hfsJihTIF3/mNprxCadYDmnfl9vULWq9bUW7AXWnWJqJoatwUX75b6Kew2NjGDwZW1iKixWjeJ3QBWVFjAVaeX8fGSrXEFQNhYo4E6YVcBNWtQxGU92rluHLO2mQtoAaDRJIEvH/wUzX6KArbQki3NT+Pt6mJaFtTn1mGXAHDuyHHU4zjLf9k7olbavTZKxXRR4Ve0kd0wZnRtg4XFht2hU43dwfx7UvPOuCft84YZaO72i06i5wnNuOgU9+B3j13dm2v7d+S7L0x3qRHMbPmZ6/qxZPO+mPLa5D/O53VoHmuL+c2wnvxmmHsg4xwa/7UA0Gj8EMoKIEWOUw9adom8bAwYOY5ijrL4ztMo2LM2YnMw7Q+718DGWXCoVr8vwPziBqxXreG1l6OEQ43AKI4fpMxcAdQrLGBQgpy+jYrrcb7HPQGpcHmv9o6bx2rCW7j8k9oZLrN9ypoxd/2euHVr20ylp+lFCwCNJmSCHv/9tneE+hS0OQXanOJc4fCeqJXD6HGfUCbbONXBQA1Ag5a2lUMnOu2sT1c5wgbVOuVkN+mkOoEX0AWnlHJlnxP43jnlfOMvU9LYs/SgBYBG44PGxs5RPzHv/Rr9pt4zkMPHqmPKa2ervppLTEkzaNcr8gIeHFMOwOofDTEM1DsM4bA6WsW0eT4sfReqjnIlcKVhoD44uQ0s6eqoYqJphxgDdTxMb6Ehvdozbv6m4JPCG3/d2i2uV8ifvtWXY1XVNCmpx31fO825YgB9yQQJ/xMiUgJMBoqN+qOVUg/Y6pwPPAn0BoYrpUYb5RcBf7BU7W4c/4+I/AO4ADATiN6olJqT2u1oNOHy52/15a1ZG2pSO3rB73jdvplzCOJa1UIaZ9gi0Kh15FV2euzx6mrYt4l/T5zCp9Nn0lG28Y2y43Qp2A5rp8KC0aAswsxioI4IhXJo0Zl+siGyUa66OspA3axBESt/ewXzNuxh3PxNibsbwC07UVRYwPxfX+aprteMYtmAF1F8BBiolNovIkXAZyIyXik1zVJnLXAj8AvriUqpiUAFgIi0JJL/9wNLlbtMYaHR5AJtmpZw64X+4uCkMmNv36yETXsOB9Ze4BQUQLMOnHj6Jdw5rSEAnfv2ocvpxv6AqmOwZ3203WGXYaxe9iHsj4Rj/rfp3vrbn0OzjlErh4LmnehS0oHm7OPayh6h3Eaqhnor/1NZxu/fW0LzAJLDhI2XnMAKMP20ioyXstVZDSAisevWWq4GxiulDibVU40mR0nF79u6qaomp6yP88fcdg5z1u1O+vpe6dOxdp9A1D6AwiJXAzUAxw7B7rXc+Ic36SjbePjsJrWCYv0MOBzpezNgTgnwcROYbTNKN+9M070lNOSw8zXSTH0j0qk9L0A24kkZJyKFwEzgJOBppdQXSVxrOPCErewREbkf+AgYqZSKCX0oIiOAEQCdOnVK4rIaTe7ipE+2ypNzT2rN8AEdXc/v07F51OCcDurFiaAZQ1EDKD2FSdV9AXj4UttGucN7alcM1lXEzpWwciIci8wnLwYWlcD+Zc3gua7RdgdT1dSsDIrCV880LSniX98fQO8O7t/7Fb3acYVLSOt04kkAKKWqgAoRaQ68LSI9lVILvF5ERNoDvYD3LcX3AJuB+sBzwN3AQw7Xfs44TmVlZe5YVzSagDnJlmgE4OWbz8hAT+ITZCwgSppB+96Rlx2Lgfr4jlVMnPYl55UehP3rogzUUTRpHyUczt1fxZGCEkr2t4eqlr4M1PE4r1t819a/XOdgT8kAvu5WKbVbRCYBgwHPAgC4BnhbKVUTWEcpZVp0jojIi9jsBxqNppa3bj2rxjgcpL46DLyEgggEi4G6XtnpXNLn6ujjhoE6auVgejGtmQLz3+RmVc3N9YGxj8C79SJeSjX7HmyriMZts8wAkzpevIBKgWPG4N8AGAT83ud1vkVkxm9tt71SapNEFKTD8CdQNJq8olWjYlo3LuaEZiU8MDQcQ6hJWYsG9EkhOXwyAdQAhvQOWCViGKhp1gE6nx17/PhR7nrhXTauXsJ95zSme8nOWkHx1ftwYGt0/XoltTuoY1RMnaFBi5wTEF5WAO2Bfxp2gALgDaXUWBF5CJihlHpHRPoDbwMtgCtF5EGlVA8AESkHOgKf2Np9xRAuAswBbgnihjSauohIxLg45Z6LQ7/WZ3cPTOl8P3skTBKFbQ6FevXZWu8EPq8uYtOJ/ene3Ray4uhB2LPO4r20unY1YTFQ11Dc1EU4GGXFsSq8TOPFC2ge0Neh/H7L+y+B2Liw1HgIxYQgVEql9pRpNHlEtqt9rPgyAmeY7UbKzaNVDg6M9RtC6SmRlxOHdscap3evjTFQ19CwlXNojebl0Lwj1PMWIjtI9E5gjSaLyaW4Mq0b12f7/qM0Ks6+5OduLNy4F4BpK3dwWY92/k5u0DzycjNQH9huWTlYBMWmubB4LFRbc41IxEDtkCAoEuK7AxQE/71qAaDR5AC5oFoubVLC9v1Hk1IBZYoBXVoyfdVOBtrVP6kiAo1LI6+yytjj1VWGgXqtbYPcGlj9Oex9g6jtVgX14IeToW2w9h8tADSaHCAXVgJV1RE1Si4JADNwXdpVbAWFkX0JzcpcDdTsXR8tHJq57/dIFi0ANJo0UVLkXzeeCzN/kyojuH4YidTDojYhTGb7EUO9+tCya+QV5mVCbV2j0QDw/A2VOREaIBVq8uvm0AqgJrxG7nQ5ULQA0GjSwMWnxk+MkohcCDFcVRNbP3dG05oVQA55WQVJ7vhraTR5TC4MUKYKKJcSwiTKB1DXyQsBEESia41GE5+2RvrE4no59HurWQHkJ3mhApp536Ca2YlGk0us23kIgGPV8SKtZwfPfud0Pl++nTZNcychSi6o1sIkLwRAk5LsT8yg0cTjiEOKyGyjdeNivl4Rs+k/q8lFw3WQ5NBaTaPJPzo0j0QAbVKSF3O1tHPJaRHjvPk95xv6qdJoNHnLiPO7MnxAJ5o1yE8tgV4BaDSavEVE8nbwBy0ANJqspmH93Amspsk9tApIo8liXrixP/+ZvYGyFvmpo9aEixYAGk0W07FlQ358cbdMd0NTR9EqII1Go8lTEgoAESkRkekiMldEForIgw51zheRWSJyXESuth2rEpE5xusdS3kXEflCRJaJyOsiUj+YW9JoNBqNF7ysAI4AA5VSfYAKYLCInGmrsxa4EXjV4fxDSqkK4zXUUv574A9KqW7ALuD7vnuv0Wg0mqRJKABUhP3GxyLjpWx1Vhu5gz1tVxQRAQYCo42ifwLDvHZao9FoNKnjyQYgIoUiMgfYCnyolPrCxzVKRGSGiEwTEXOQbwXsVkodNz6vxyFxvHHtEcb5M7Zt2+bjshqNRqOJhycBoJSqUkpVAGXAABHp6eManZRSlcC3gSdF5EScg+85RmVSSj2nlKpUSlWWlpb6uKxGo9Fo4uHLC0gptRuYBAz2cc5G4+9K49y+wHaguYiYbqhlwEY/fdFoNBpNanjxAioVkebG+wbAIGCJl8ZFpIWIFBvvWwPnAIuUUgqYCJgeQzcAY/x3X6PRaDTJIkrFj4ctIr2JGGkLiQiMN5RSD4nIQ8AMpdQ7ItIfeBtoARwGNiuleojI2cCzRIzDBcCTSqnnjXa7AqOAlsBs4Hql1JEEfdkGrEnyXlsTWXnkC/p+6zb6fus2Qd9vZ6VUjA49oQCoK4jIDMMWkRfo+63b6Put26TrfvVOYI1Go8lTtADQaDSaPCWfBMBzme5AmtH3W7fR91u3Scv95o0NQKPRaDTR5NMKQKPRaDQWtADQaDSaPCUvBICIDBaRpSKyXERGZro/XhGRF0Rkq4gssJS1FJEPjTDaH4pIC6NcROQp4x7niUg/yzk3GPWXicgNlvLTRWS+cc5TRpC+jCEiHUVkoogsNkKP32GU18l7dgu17hYqXUSKjc/LjePllrbuMcqXishllvKse/aN2GKzRWSs8bnO3q+IrDaetzkiMsMoy57nWSlVp19ENrCtALoC9YG5wGmZ7pfHvp8P9AMWWMoeA0Ya70cCvzfeXwGMJxJn6UzgC6O8JbDS+NvCeN/CODYdOMs4ZzxweYbvtz3Qz3jfBPgKOK2u3rPRh8bG+yLgC+M+3gCGG+V/BW413v8I+KvxfjjwuvH+NOO5Lga6GM97YbY++8CdRELHjzU+19n7BVYDrW1lWfM858MKYACwXCm1Uil1lMju469nuE+eUEpNBnbair9OZGc2RIfR/jrwkoowjUispfbAZUQiuO5USu0CPiSS06E90FQpNVVFnqSXyHBIbqXUJqXULOP9PmAxkSixdfKejX47hVp3C5Vu/R5GAxcbM76vA6OUUkeUUquA5USe+6x79kWkDBgC/N34HC80fM7frwtZ8zzngwDoAKyzfHYNPZ0jtFVKbYLIgAm0Mcrd7jNe+XqH8qzAWO73JTIrrrP3LLZQ60RmsG6h0mvuyzi+h0hodb/fQyZ5EvgltblD4oWGrwv3q4APRGSmiIwwyrLmec6HpPCeQ0/nOG736bc844hIY+At4KdKqb1x1Jo5f89KqSqgQiIBF98GTnWqZvz1e19OE7yM3a+IfA3YqpSaKSIXmsUOVevE/Rqco5TaKCJtgA9FJF4gzbQ/z/mwAlgPdLR8zvXQ01uMpR/G361Gudt9xisvcyjPKCJSRGTwf0Up9W+juE7fM0SFWj8T91DpNfdlHG9GREXo93vIFOcAQ0VkNRH1zEAiK4K6er+o2nD4W4kI+AFk0/OcSQNJOl5EVjkriRiLTMNQj0z3y0f/y4k2Aj9OtAHpMeP9EKINSNNVrQFpFRHjUQvjfUvj2JdGXdOAdEWG71WI6DGftJXXyXsGSoHmxvsGwKfA14A3iTaK/sh4fxvRRtE3jPc9iDaKriRiEM3aZx+4kFojcJ28X6AR0MTyfgqRXCpZ8zxn/EFI0z/iCiIeJSuAX2W6Pz76/RqwCThGRNp/n4gO9CNgmfHXfBAEeNq4x/lApaWd7xExlC0HbrKUVwILjHP+jLEzPIP3ey6RJew8YI7xuqKu3jPQm0go9HlGn+43yrsS8e5YbgyOxUZ5ifF5uXG8q6WtXxn3tBSLJ0i2PvtEC4A6eb/Gfc01XgvN/mTT86xDQWg0Gk2ekg82AI1Go9E4oAWARqPR5ClaAGg0Gk2eogWARqPR5ClaAGg0Gk2eogWARqPR5ClaAGg0Gk2e8v8BI8+3cOSiIw0AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "#hide\n", "learn.recorder.plot_loss()" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "learn.save(path_data/'GPT2_pt_5epoch_lr1e-4_v2')\n", "learn = learn.load(path_data/'GPT2_pt_5epoch_lr1e-4_v2')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model sharing and uploading in the Hugging Face model hub" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can share our Portuguese GPT-2 on the Hugging Face [model hub](https://huggingface.co/models) following the tutorial [Model sharing and uploading](https://huggingface.co/transformers/model_sharing.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Your model now has a page on huggingface.co/models 🔥\n", "\n", "Anyone can load it from code:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tokenizer = AutoTokenizer.from_pretrained(\"namespace/awesome-name-you-picked\")\n", "model = AutoModel.from_pretrained(\"namespace/awesome-name-you-picked\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In our case, the code is:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from transformers import AutoTokenizer, AutoModelWithLMHead\n", "\n", "tokenizer = AutoTokenizer.from_pretrained(\"pierreguillou/gpt2-small-portuguese\")\n", "model = AutoModelWithLMHead.from_pretrained(\"pierreguillou/gpt2-small-portuguese\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check our [Hugging face model page](https://huggingface.co/pierreguillou/gpt2-small-portuguese) to get more information." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Text Generation by our Portuguese GPT-2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have a GPT-2 in Portuguese, we can use it for different tasks in NLP (Text Generation, Reading Comprehension, Translation, Summary) as showed in the post \"**GPT-2 use cases: beyond Text Generation**\".\n", "\n", "For now, let's use it to generate new texts, which allows us to check that it works properly and also have a little fun." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Text Generation techniques " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**At each stage of text generation, GPT-2 provides a vector of 50.257 probabilities** (each corresponds to a possible token of the vocabulary whose size is 50.257). To decide how to choose the output token from these probabilities, **there are at least 5 methods: Greedy, Beam Search, Sampling with temperature, Top-k sampling and Top-p (nucleus) sampling**.\n", "\n", "In this tutorial, we will test only 2 of these text generation methods: **Top-k sampling** and **Top-p (nucleus) sampling**.\n", "\n", "Note: to get more information on text generation techniques for transformer-based language model, read the article \"[How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate)\" from [Patrick von Platen](https://huggingface.co/patrickvonplaten) (Hugging Face, 03/18/2020)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (Use case 1) Top-k sampling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our use case 1 follows the same method used by OpenAI in page 20 of the paper [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by choosing **Top-k sampling** text generation technique with a **value of 40**.\n", "\n", "This text generation method is implemented in the `model.generate()` function of a Transformers model thanks to the following arguments:\n", "- `top_k` (int): the number of highest probability vocabulary tokens to keep for top-k-filtering. Between 1 and infinity. Default to 50." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (Use case 2) Top-p (nucleus) sampling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our use case 2 follows the [top-p (nucleus) sampling](https://huggingface.co/blog/how-to-generate#top-p-nucleus-sampling) method with Top-p sampling (top_p = 0.95), top-k sampling (top_k = 50), temperature (temperature = 0.7) and repetition penalty (repetition_penalty = 1.2).\n", "\n", "This text generation method is implemented in the `model.generate()` function of a Transformers model thanks to the following arguments:\n", "- `top_p` (float): the cumulative probability of parameter highest probability vocabulary tokens to keep for nucleus sampling. Must be between 0 and 1. Default to 1.\n", "- `top_k` (int): the number of highest probability vocabulary tokens to keep for top-k-filtering. Between 1 and infinity. Default to 50.\n", "- `temperature` (float): the value used to module the next token probabilities. Must be strictly positive. Default to 1.0.\n", "- `repetition_penalty` (float): the parameter for repetition penalty. Between 1.0 and infinity. 1.0 means no penalty. Default to 1.0." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Text n°1 | Famous OpenAI generated text about unicorns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At the time of publication of GPT-2 in the article \"[Better Language Models and Their Implications](https://openai.com/blog/better-language-models/)\" (02/14/2019), the media retained from its different possibilities in NLP that of text generation because of the now famous text generated on unicorns from this small paragraph:\n", "\n", ">In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From this text, GPT-2 wrote the following [sample](https://openai.com/blog/better-language-models/#samples) out of 10 tries:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. \n", "\n", "The scientist named the population, after their distinctive horn, Ovid's Unicorn. These four-horned, silver-white unicorns were previously unknown to science. \n", "\n", "Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. \n", "\n", "Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. \n", "\n", "Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez. \n", "\n", "Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. \n", "\n", "While examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” \n", "\n", "Dr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America. \n", "\n", "While their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.” \n", "\n", "However, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.\n" ] } ], "source": [ "openai_generated_text_en = \"In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. \\\n", "\\n\\nThe scientist named the population, after their distinctive horn, Ovid's Unicorn. These four-horned, silver-white unicorns were previously unknown to science. \\\n", "\\n\\nNow, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. \\\n", "\\n\\nDr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. \\\n", "\\n\\nPérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez. \\\n", "\\n\\nPérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. \\\n", "\\n\\nWhile examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” \\\n", "\\n\\nDr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America. \\\n", "\\n\\nWhile their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.” \\\n", "\\n\\nHowever, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.\"\n", "\n", "print(openai_generated_text_en)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Reload Portuguese GPT-2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### English pre-trained GPT-2" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 14.4 s, sys: 1.8 s, total: 16.2 s\n", "Wall time: 7.72 s\n" ] } ], "source": [ "%%time\n", "from transformers import GPT2TokenizerFast, GPT2LMHeadModel\n", "\n", "pretrained_weights = 'gpt2'\n", "tokenizer_en = GPT2TokenizerFast.from_pretrained(pretrained_weights)\n", "model_en = GPT2LMHeadModel.from_pretrained(pretrained_weights)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Portuguese tokenizer" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from transformers import GPT2TokenizerFast\n", "\n", "# Get the path to ByteLevelBPE_tokenizer_pt config files\n", "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n", "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n", "\n", "# import the pre-trained GPT2TokenizerFast tokenizer with the tokenizer_pt config files\n", "tokenizer_pt = GPT2TokenizerFast.from_pretrained(\n", " str(path_to_ByteLevelBPE_tokenizer_pt_rep), \n", " pad_token='<|endoftext|>')\n", "\n", "# Get sequence length max of 1024\n", "tokenizer_pt.model_max_length = 1024" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Datasets" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "class TransformersTokenizer(Transform):\n", " def __init__(self, tokenizer): self.tokenizer = tokenizer\n", " def encodes(self, x): \n", " toks = self.tokenizer.tokenize(x)\n", " return tensor(self.tokenizer.convert_tokens_to_ids(toks))\n", " def decodes(self, x): return TitledStr(self.tokenizer.decode(x.cpu().numpy()))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# get data\n", "lang = 'pt'\n", "fname = f'all_texts_{lang}wiki.csv'\n", "df = pd.read_csv(path_data/fname)\n", "\n", "# load idxs train and valid\n", "idxs_train = torch.load(path_data/'idxs_train.pt')\n", "idxs_val = torch.load(path_data/'idxs_val.pt')\n", "\n", "all_texts = np.concatenate([df.iloc[idxs_train].text.values, df.iloc[idxs_val].text.values])\n", "splits = [list(idxs_train), list(idxs_val)]\n", "tls = TfmdLists(all_texts, TransformersTokenizer(tokenizer_pt), splits=splits, dl_type=LMDataLoader)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Dataloaders" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 27min 56s, sys: 3min 20s, total: 31min 16s\n", "Wall time: 25min 8s\n" ] } ], "source": [ "%%time\n", "bs,sl = 8,1024\n", "dls = tls.dataloaders(bs=bs, seq_len=sl)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Split a GPT2 model in 4 groups for differential learning rates" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "class DropOutput(Callback):\n", " def after_pred(self): self.learn.pred = self.pred[0]" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def splitter(model):\n", " \"Split a GPT2 `model` in 3 groups for differential learning rates.\"\n", " \n", " # First layers group : decoder blocks from 0 to 3\n", " modules = []\n", " for i in range(4): modules.append(model.transformer.h[i])\n", " groups = [nn.Sequential(*modules)]\n", "\n", " # Second layers group : decoder blocks from 4 to 7\n", " modules = []\n", " for i in range(4,8,1): modules.append(model.transformer.h[i])\n", " groups = L(groups + [nn.Sequential(*modules)])\n", "\n", " # Third layers group : decoder blocks from 8 to 11\n", " modules = []\n", " for i in range(8,12,1): modules.append(model.transformer.h[i])\n", " groups = L(groups + [nn.Sequential(*modules)])\n", " \n", " # Fourth layers group : embeddings matrices wte and wpe + LayerNorm at the model output\n", " groups = L(groups + [nn.Sequential(model.transformer.wte,model.transformer.wpe,model.transformer.ln_f)])\n", " \n", " return groups.map(params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Learner" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "# Learner: basic class for handling the training loop\n", "# source: https://dev.fast.ai/learner#Learner\n", "learn = Learner(dls, model_en, loss_func=CrossEntropyLossFlat(),\n", " splitter = splitter,\n", " cbs=[DropOutput], \n", " metrics=[accuracy, Perplexity()]).to_fp16()\n", "learn = learn.load(path_data/'GPT2_pt_5epoch_lr1e-4_v2')" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# model\n", "model_pt = learn.model" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# put model into eval mode and on GPU\n", "model_pt.eval();\n", "model_pt.to('cuda');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Get translated famous unicorn text in Portuguese" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "from transformers import MarianMTModel, MarianTokenizer" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['>>fr<<', '>>es<<', '>>it<<', '>>pt<<', '>>pt_br<<', '>>ro<<', '>>ca<<', '>>gl<<', '>>pt_BR<<', '>>la<<', '>>wa<<', '>>fur<<', '>>oc<<', '>>fr_CA<<', '>>sc<<', '>>es_ES<<', '>>es_MX<<', '>>es_AR<<', '>>es_PR<<', '>>es_UY<<', '>>es_CL<<', '>>es_CO<<', '>>es_CR<<', '>>es_GT<<', '>>es_HN<<', '>>es_NI<<', '>>es_PA<<', '>>es_PE<<', '>>es_VE<<', '>>es_DO<<', '>>es_EC<<', '>>es_SV<<', '>>an<<', '>>pt_PT<<', '>>frp<<', '>>lad<<', '>>vec<<', '>>fr_FR<<', '>>co<<', '>>it_IT<<', '>>lld<<', '>>lij<<', '>>lmo<<', '>>nap<<', '>>rm<<', '>>scn<<', '>>mwl<<']\n" ] } ], "source": [ "# MarianMT Translator\n", "model_name = 'Helsinki-NLP/opus-mt-en-ROMANCE'\n", "tokenizer_en_pt = MarianTokenizer.from_pretrained(model_name)\n", "print(tokenizer_en_pt.supported_language_codes)\n", "model_en_pt = MarianMTModel.from_pretrained(model_name)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['>>pt_BR<< In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.']\n" ] } ], "source": [ "src_text = [\n", " '>>pt_BR<< In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.',\n", "]\n", "print(src_text)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito.'" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "translated = model_en_pt.generate(**tokenizer_en_pt.prepare_translation_batch(src_text))\n", "tgt_text = [tokenizer_en_pt.decode(t, skip_special_tokens=True) for t in translated]\n", "prompt = tgt_text[0]\n", "prompt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Get generated text" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "504\n" ] } ], "source": [ "# Get the number of tokens of the OpenAI English generated text\n", "openai_generated_text_en = \"In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. \\\n", "\\n\\nThe scientist named the population, after their distinctive horn, Ovid's Unicorn. These four-horned, silver-white unicorns were previously unknown to science. \\\n", "\\n\\nNow, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. \\\n", "\\n\\nDr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. \\\n", "\\n\\nPérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez. \\\n", "\\n\\nPérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. \\\n", "\\n\\nWhile examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” \\\n", "\\n\\nDr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America. \\\n", "\\n\\nWhile their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.” \\\n", "\\n\\nHowever, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.\"\n", "\n", "openai_generated_text_en_ids = tokenizer_en.encode(openai_generated_text_en, return_tensors='pt').to('cuda')\n", "max_length = openai_generated_text_en_ids.shape[1]\n", "\n", "print(max_length)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "# encode\n", "input_ids = tokenizer_pt.encode(prompt, return_tensors='pt').to('cuda')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Use case 1 (Top-k sampling)" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ">> Generated text 1\n", "\n", "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito. Um deles, um escocês chamado Thomas (um anatomista irlandês) que havia trabalhado com o britânico em seu trabalho como assistente do diretor experimental da universidade, John Holley, foi chamado de por eles para um encontro sobre os dois conceitos, que não tinham encontrado muito trabalho até o momento. Eles foram palestrantes, mas ficaram impressionados que Thomas (como todos os outros cientistas da época) não sabia falar Inglês. Quando Thomas fez isso com os unicórnios por engano, Holley a escreveu e enviou um telegrama à BBC perguntando: \"\"Tão, o que é tal dizer se você quer saber de nenhum outro, eu não sei como ele fala inglês?\"\" Eles disseram a ele: \"Não, mas é um idiota.\" Então, uma semana depois, no mesmo dia, Holley o enviou, uma carta com um questionário para os membros do elenco de \"\"Doctor Who\"\". Thomas, como qualquer um dos cientistas mencionados, não se lembra muito bem dos eventos que aconteceram em Londres.\n", "\n", "Depois do segundo episódio, o editor-produtor Michael Gambon deu o que ele achou que valia, então disse: \"Vamos fazer um comentário, para o elenco e equipe deDoctor Who.\" Como resultado, eles deram permissão para que eles assistissem outro episódio. Isto foi o suficiente para que \"The Magician\" aparecesse para o cânone da BBC.\n", "\n", "Na segunda temporada, \"The Magician\" teve um episódio chamado \"The End of the World\", e o ator John Huston, que interpreta John Watson, era o mais conhecido ator inglês do programa. \"The End of the World\" foi ao ar em 13 de outubro de 2000. A BBC anunciou que \"The Magician\" seria exibida ao vivo em janeiro de 2003 da HBO em 15 de janeiro de 2003. Ele também foi transmitido na televisão em 16 de janeiro de 2004 da BBC One, em uma transmissão que também marcou o cancelamento de um episódio.\n", "\n", "A série é uma produção da BBC-NBC Television Limited. Ela foi produzida por Paul Rudd, John Swarns e Jonathan Seagal e foi criada por David Tennant, Graham Hill, Colin Smith, Tom Baker, Jack Davies e Paul Thompson. O primeiro episódio foi exibido em 17 de outubro de\n", "\n", "---\n", ">> Generated text 2\n", "\n", "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito. \"Não é mais estranho que a nossa forma tivesse o inglês com dois de suas asas como se o macho fosse inglês — o que é interessante. Mas a sua natureza inata seria estranha para o inglês\", acredita eles.\n", "\n", "Em 2015, cientistas realizaram uma nova análise sobre as formas dos unicórnios. De acordo com especialistas na área, os membros superiores do grupo foram provavelmente derivados de outra espécie de escorpião — uma espécie com características semelhantes. Uma nova equipe de cientistas calculou que uma fêmea unicornada da Eurásia seria originalmente uma humana. \"Isto significa que o ancestral do unicórnio, um híbrido de um esquilo e um escorpião macho não nasceu.\n", "\n", "Um estudo recente estimou que cerca de 12% do corpo humano é composto por membros de qualquer um dos grupos mais diverso de animais extintos, incluindo o ser humano e o unicórnio-do-sul. \"A análise dos dados mostra que a maioria dos membros do gênero é composta por um exito e um exito macho que compartilham uma única espécie de corpo. Em alguns casos, estes membros compartilham um mesmo conjunto (a linhagem) de partes em dois espécimes.\" O estudo indica que os membros de \"P. rubi\" são semelhantes em aparência e morfologia aos membros humanos modernos, como as fêmeas modernas e machos robustos. \"Como é evidente com os membros de \"P. rubi\", os ancestrais e o ancestral eram similares na forma e na composição das semelhanças em um organismo.\"\n", "\n", "O DNA do \"P. rubi,\" chamado por sua forma em inglês de \"sonoroplasto\", revela que o \"sonoroplasto\" inclui três genes de alto nível e quatro genes relativamente reduzidos (e ausentes) e um gene de baixo nível (e ausentes) com uma concentração de cloroplasto em cada núcleo. A \"sonoroplasto\" se assemelha à \"P. rubi\" em características morfológicas e comportamentais, embora as diferenças na morfologia sejam menores. \"A espécie \"P. rubi\" apresenta cinco pares de cromossomos separados (com 6 pares se aproximando e 8 pares se afastando) e um \"sonoroplasto de base\" (com 12 pares se aproximando e 15 pares se afastando), sugerindo que o membro tenha uma composição semelhante ao ancestral \"P. rubi\n", "\n", "---\n", ">> Generated text 3\n", "\n", "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito.\n", "O jogo de corrida da FIAT foi introduzido no Salão Internacional de FIAT (em 1989) em um evento que teve como principal objetivo fornecer uma oportunidade a todos que competirem em corridas de carros nacionais, como as categorias de base e pilotos. Esse evento trouxe o título de \"National Le Mans Series\" (1973), além de um terceiro título que o vencedor da competição ganharia. O primeiro campeonato foi disputado entre 1983 e 1985, e foi vencido pelo piloto estadunidense Dennis Young.\n", "\n", "O Campeonato de Le Mans foi criado por uma equipa formada em 1983 na rua General Motors, em Long Beach, Califórnia, e patrocinado pela \"Frampton Racing\", criada pelo empresário de corridas do Reino Unido Derek J. Cox, e é atualmente patrocinado pela equipe da equipe de Fórmula 1 da Ferrari, Lotus e McLaren, \"Lusa Motorsports\" e pela equipe de corrida britânica Team Lotus. \"Lusa\" foi fundada como uma \"casa\", mas a companhia não conseguiu competir em corridas por cinco anos devido à falta de patrocinador, e os pilotos que competiram em Lusa foram proibidos de pilotar a equipe McLaren em 1997. A equipe mudou seu nome para Lusa Motorsports em 1999. As corridas de Fórmula 1 só foram retomadas duas vezes por falta de patrocinadores, no começo de 2000, e novamente em 2003 por \"franqueza\".\n", "\n", "Em 2003, a Williams-Renault obteve o seu primeiro título da temporada, uma corrida de Le Mans de estilo \"Grand Prix\", em que os carros eram todos monopostos, mas com carros \"primo\", mais do que os carros de Fórmula 1. A temporada de 2003 teve muitas modificações em termos de desempenho, como novos pneus e mudanças na configuração dos carros, e foi a sexta temporada do campeonato do campeonato com mais pontos no campeonato na temporada de 2004, com cinco vitórias no campeonato, três na temporada de 2008 e cinco na de 2009.\n", "\n", "Em 2005, a Toyota assinou um contrato exclusivo para a produção de veículos para a Temporada de Le Mans e o primeiro ano em que a equipe Toyota foi a primeira equipe a ser a primeira a dar uma vitória ao correr \"Le Mans Series\", uma prova de Fórmula 1, na qual a equipe Toyota produziu dois carros da temporada. Apesar de ter vencido a prova, foi a\n", "\n", "---\n", "CPU times: user 7.77 s, sys: 372 ms, total: 8.15 s\n", "Wall time: 8.16 s\n" ] } ], "source": [ "%%time\n", "#set top_k = 40 and num_return_sequences = 3\n", "sample_outputs = model_pt.generate(input_ids, pad_token_id=50256,\n", " do_sample=True, \n", " max_length=max_length, \n", " min_length=max_length,\n", " top_k=40,\n", " num_return_sequences=3)\n", "\n", "for i, sample_output in enumerate(sample_outputs):\n", " print(\">> Generated text {}\\n\\n{}\".format(i+1, tokenizer_pt.decode(sample_output.tolist())))\n", " print('\\n---')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Use case 2 (Top-p nucleus sampling)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ">> Generated text 1\n", "\n", "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito. Eles não sabiam onde exatamente eram falantes nativos do idioma, e acreditaram que eles simplesmente migraram das terras altas da região de Mendoza ao norte como consequência do declínio populacional que ocorreu na Cordilheira das Cobras.\n", "\n", "Em 2004, o Departamento de Antropologia da Universidade do Colorado anunciou que havia encontrado uma fêmea no vale do rio Orinoco na Bolívia, mas essa fêmea foi morta durante a investigação. No entanto, no início de 2006, as autoridades locais anunciaram que havia identificado uma fêmea encontrada em uma área próxima à Cordilheira dos Andes, no Vale do Cauca. A equipe de pesquisadores relatou que esta fêmea era chamada de \"El Maria\" ou \"El Maria\".\n", "\n", "O estudo revelou que o grupo de unicórnios habitava um ecossistema bastante diverso, com espécies endêmicas incluindo espécies como as tiláceas gigantescas (que são encontradas principalmente nos países subdesenvolvidos) e as quelupus (\"Erica azoricae\").\n", "\n", "Um dos principais objetivos do estudo da espécie é determinar se os europeus teriam colonizado a região entre a década de 1940 e 1960 e se estes últimos grupos étnicos sobreviveram até hoje. Os cientistas acreditam que as populações desses grupos poderiam ter sido muito maiores antes disso; por exemplo, a teoria sugere que a população europeia provavelmente teria introduzido os humanos primitivos na América Central depois que os espanhóis invadiram a região, embora isso seja controverso.\n", "\n", "O gênero \"El Maria\" tem um ancestral comum, os \"Looney-do-the-Bone\", um pequeno grupo de \"Looney-da-Daíndia\" encontrados apenas no leste dos Estados Unidos, Canadá e México. O gênero possui parentesco próximo ao gênero \"Lontrapyrus\", também conhecido como lontras negras. Acredita-se que esses indivíduos tenham migrado para o leste dos Andes, atravessando regiões montanhosas do sul de América Central e América Central.\n", "\n", "Os membros desta família são geralmente confundidos com os lontras brancos.\n", "\n", "As fêmeas têm cerca de seis centímetros de comprimento, pesando de 9 quilogramas e medindo 11 cm de largura. A cabeça é branca, com manchas escuras pretas escuras sobre seus flancos. As patas posteriores podem ser amarelas, enquanto sua cauda pode estar preta ou branca, dependendo da cor utilizada na identificação. As costas apresentam quatro dedos dorsais bem desenvolvidas\n", "\n", "---\n", ">> Generated text 2\n", "\n", "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito.\n", "\n", "O fóssil é encontrado no Vale de La Guaira, na região de San Juan, com cerca de 1 metro (1,8 m) de comprimento e cerca de 4 metros (2 pés) de largura, em Wyoming. A cabeça possui cinco dedos no pé direito e duas garras direita. No topo do crânio há quatro placas pretas-escuros. O cérebro tem um formato triangular, a base apresenta apenas uma pequena porção da metade anterior e três partes do pescoço são pretos-claro; a face tem um tom vermelho escuro-escuro. A mandíbula também contém oito pares de maxilas bem desenvolvidas.\n", "\n", "A região de San Juan foi descoberta por volta de 14 mil anos atrás. Foi escavado durante a Guerra dos Sete Anos e tem aproximadamente 6 metros quadrados de espessura. O local era habitado pelo povo de Tacuareña, que vivia entre o final do e o início do e tinha uma cultura diferente das outras regiões próximas ao território de El Berriol. Os indígenas não eram caçadores conhecidos pelos nativos e como \"carijós\".\n", "\n", "O fóssil foi descoberto em 2002 e depois enviado para uma equipe de arqueólogos liderada pela Universidade de Wyoming. A equipe encontrou os restos humanos e o chamou de \"caçadores\" ou \"coletores\", pois eles tinham sido mortos pelas expedições anteriores ao assentamento.\n", "\n", "O arqueólogo Brian C. Fanning, da Universidade de Wyoming, fez várias entrevistas com os esqueletos humanos e descobriram os crânios de dois ou três homens que foram enterrados em um túmulo localizado abaixo do túmulo de um antigo homem chamado Huehue, que estava enterrado por volta de 10500 anos atrás. O homem que estava enterrado é o membro de um grupo étnico conhecido como \"Huehue\".\n", "\n", "A equipe de pesquisadores da equipe de Fanning incluiu a presença humana no local onde o homem enterrado, incluindo um cemitério indígena que está próximo à entrada do cemitério. Eles encontraram evidências ósseas humanas enterradas perto dos ossos do homem e outros artefatos encontrados dentro de um depósito de ossos de um homem chamado \"Creator\".\n", "\n", "Os pesquisadores acharam vários ossos humanos semelhantes aos ossos de Huehue, mas sem as marcas físicas. Eles notaram semelhanças anatômicas entre esses membros de um grupo étnico chamado \"Caçadores\", possivelmente porque ambos viviam no Vale de\n", "\n", "---\n", ">> Generated text 3\n", "\n", "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito.\n", "\n", "Os arqueólogos descobriram um conjunto de moedas cunhadas por indígenas e outros achados arqueológicos encontrados no local durante as escavações foram feitos ao longo da década de 1960 em um esforço de resgate arqueológica e reconhecimento pela população indígena.\n", "\n", "O achado arqueológico é o único relato sobre a cultura do gênero \"Saccharus\", que se refere às culturas terrestres primitivas não relacionadas com o \"Homo sapiens\".\n", "\n", "A pesquisa envolveu cerca de 60 anos de escavação, incluindo levantamento topográfico efetuado entre 1972 e 1977. Em 1988, o arqueólogo americano John Deere, que pesquisou as ruínas do complexo arqueológico do Monte dos Andes, observou que a cultura deste gênero estava fortemente relacionada com outras civilizações mesoamericanas, como o olmeca e a olmeca.\n", "\n", "O arqueólogo franco-americano John Deere observou que a cultura desses gêneros era bem próxima à civilização olmeca e \"a cultura dos astecas\" foi muito similar àquelas encontradas na América Central em seus sítios prévios.\n", "\n", "Em 1991, o arqueólogo americano John Deere publicou uma extensa monografia descrevendo a cultura dos Andes. Ele escreveu:\n", "\n", "Muitos estudiosos têm alegado que a cultura dos Andes pode ter sido transmitida através de migrações humanas do continente americano. Contudo, estudos recentes, baseados nos dados históricos obtidos em diferentes lugares dos Andes sugerem que tais migrações ocorreram antes ou imediatamente após as primeiras invasões americanas de europeus.\n", "\n", "Pesquisas iniciais demonstraram que esta migração ocorreu provavelmente entre o final do século XVIII e meados do século XIX, quando sociedades modernas dominavam o solo chileno, Peru, Bolívia e Colômbia. As tribos inca e xamãs também estavam envolvidas nesta atividade.\n", "\n", "A história das Américas começou em torno de um grupo de caçadores sulamericanos pertencentes aos Andes liderados pelo índio A. Iruña, que estabeleceu sua capital no atual assentamento chamado \"San Juan del Sur\". Acreditava-se que ele teria fundado um assentamento que seria conhecido por sua fama de ser capaz de caçar com seus guerreiros e construir suas casas com o passar do tempo.\n", "\n", "Durante o século XVIII, as terras baixas eram propícias para o crescimento populacional e, consequentemente, o desenvolvimento econômico das novas cidades situadas em torno delas.\n", "\n", "Os primeiros exploradores espanhóis chegaram ao Novo Mundo em 1692 e estabeleceram assentamentos permanentes nesses territórios até a década de 1830. As tentativas espanholas\n", "\n", "---\n", "CPU times: user 35.3 s, sys: 1.85 s, total: 37.2 s\n", "Wall time: 37.2 s\n" ] } ], "source": [ "%%time\n", "#set top_p = 0.95, top_k = 50, temperature = 0.7, repetition_penalty = 1.2 and num_return_sequences = 3\n", "sample_outputs = model_pt.generate(input_ids, pad_token_id=50256,\n", " do_sample=True,\n", " max_length=max_length, \n", " min_length=max_length,\n", " repetition_penalty=1.2,\n", " temperature=0.7,\n", " top_k=50, \n", " top_p=0.95, \n", " num_return_sequences=3)\n", "\n", "for i, sample_output in enumerate(sample_outputs):\n", " print(\">> Generated text {}\\n\\n{}\".format(i+1, tokenizer_pt.decode(sample_output.tolist())))\n", " print('\\n---')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Text n°2 | Recent text on the coronavirus disease (Covid-19)" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n" ] } ], "source": [ "# source (uol, 07/13/2020): https://www.uol.com.br/vivabem/noticias/redacao/2020/07/13/russia-vacina-conclui-testes-distribuicao-em-agosto.htm\n", "prompt = \"A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\"\n", "print(prompt)" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [], "source": [ "# encode\n", "input_ids = tokenizer_pt.encode(prompt, return_tensors='pt').to('cuda')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Use case 1 (Top-k sampling)" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ">> Generated text 1\n", "\n", "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n", "\n", "Entre os primeiros casos confirmados de COVID-19 na Rússia estava um homem russo com idade entre 50 e 49 anos, segundo o jornal \" Moscow\" e o primeiro caso em 28 de fevereiro, a primeira no Hospital Pulkai.\n", "\n", "No entanto, os primeiros casos foram mais sérios devido sua \"maturidade sem gravidade\" e a falta de uma pessoa estar disponível para testes de suas condições de vida. Entre os pacientes que foram considerados estão funcionários de um hospital ou enfermeiros na cidade de Moscou (ver abaixo) ou médicos. Depois de serem testados negativos após um teste positivo, o paciente se recupera completamente.\n", "\n", "Os russos também anunciaram que serão realizados testes de coronavírus de outros países, como a França, que inicialmente acreditava que os vírus da gripe tinham sido transmitido pela Europa (a expectativa é de 20 a 50 casos por dia). A situação foi resolvida em 24 de fevereiro, quando o Ministério da Saúde confirmou sua conclusão de que a COVID-19 é transmitida de via aérea.\n", "\n", "Até ao dia do seu primeiro caso, a Rússia tinha o menor número de funcionários e médicos com doença grave antes de o vírus ter se tornado um vírus no país. Os números de funcionários com doença grave não chegaram a ser confirmados, segundo o Ministério da Saúde, devido a sua falta de apoio.\n", "\n", "O vírus que está em curso no país é transmitido pela primeira vez nos Estados Unidos, onde foi isolado em 14 de fevereiro. Em Portugal o Ministério dos Negócios Estrangeiros declarou oficialmente em 11 de fevereiro que a COVID-19 está presente no país, mas não anunciou nenhum impacto na prática. No entanto, o Ministério do Trabalho declarou no dia seguinte que o coronavírus existe \"em todo os países que não têm regulamentação\" e que nenhum \"aedes\", que era identificado em 11 de fevereiro, tinha entrado na corrente sanguínea.\n", "\n", "Em 9 de março, autoridades russas confirmaram que o paciente não está hospitalizado e que está em quarentena no Hospital Pulkai.\n", "\n", "Mais dois casos de COVID-19 foram anunciados em 9 de março, mas foram considerados casos de \"propaganda\" e não de risco.\n", "\n", "Em 9 de maio, autoridades russas confirmaram os confirmados em todo país. Em 10 de maio,\n", "\n", "---\n", ">> Generated text 2\n", "\n", "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto. As populações de Wuhan, Wuhan, Qinghai, Shenzhen na costa oeste do China e no norte de Xangai, na província de Hubei, e os subúrbios da cidade de Tvershend em Hong Kong já estão de acordo com o número de casos confirmados. Uma nova equipe de infecologistas para ajudar a determinar os possíveis efeitos da pandemia já está em Xangai. Em 14 de março, a Rússia também tem a oportunidade de testar novas vacinas virais e testar novas versões para garantir a propagação de maisvírus na China através da transmissão de novos coronavírus.\n", "\n", "As autoridades russas e autoridades de saúde chinesas têm se preparado para assumir essa atitude de emergência com base nos dados coletados pela máscaras, máscaras de voo e máscaras faciais. Um relatório de uma empresa francesa informou que mais de 30 pessoas morreram na área de transmissão ao longo do mês de março, em relação aos dois primeiros casos. Em 21 de março, mais de 2,3 milhões de pessoas foram afetadas em áreas urbanas. Segundo o Conselho de Saúde, cerca de 10% dos habitantes são da China continental e quase metade do Distrito de Pequim, incluindo as regiões Norte e Noroeste. A doença foi descrita pela primeira vez em uma carta enviada às autoridades sanitárias no dia 15 de março. Três dias depois, um estudo genético para o vírus causou grande preocupação aos especialistas em transmissão, que acreditavam que a doença se espalhava através de duas regiões do país. A China continental relatou que apenas 8 pessoas foram diagnosticadas no país.\n", "\n", "Os vírus estão propagando-se por todo o continente europeu, do Ásia e de diversas partes do globo. De acordo com análises recentes do Centro de Controle e Prevenção da Doenças, em 14 de março, havia um total de 785 casos confirmados confirmados em todo o mundo. O surto da doença chegou aos Estados Unidos, com o surto de gripe na cidade de Wuhan, China, sendo o quarto maior já registrado em um período desde a epidemia de surtos no ano de 1994. Além disso, os casos das novas epidemias trazidas pela epidemia das SARS de 2013 no Haiti em 2020 tornaram-se os maiores problemas para a saúde da China. Em 14 de março, cerca de 4,85 milhão de pessoas estavam infectadas e cerca de 8%\n", "\n", "---\n", ">> Generated text 3\n", "\n", "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n", "O Papa Francisco pediu ao povo russo que confirme a data da chegada da vacina. O Papa também pediu aos cidadãos russos que realizem uma conferência de imprensa oficial para o anúncio. Em 7 de dezembro de 2016, quando se iniciou o trabalho de tradução para mais idiomas, mais de 300 pessoas no mundo tiveram acesso aos benefícios da vacina.\n", "\n", "A primeira vacina para o coronavírus foi testada no Japão em 9 de março de 2020.\n", "\n", "Em 6 de fevereiro, um estudo realizado em Taiwan revelou que a epidemia do SARS é mais provável devido ao aumento do hábito. Por causa do aumento nas visitas dos cidadãos a hospitais para verificar que a vacina é eficaz, a saúde do país começou a declinar. O ministro da Saúde japonês, Toshihiko Hayashi disse que a crise da saúde era uma consequência da melhora das medidas de prevenção e contra o coronavírus em relação ao início do período de alta infecção.\n", "\n", "A Rússia começou a distribuir a vacina no início de outubro, após a confirmação por funcionários da saúde pública de uma nova coronavírus. Foi lançada no mesmo dia em todas as regiões ocidentais do país. O total de pessoas hospitalizadas pelo vírus foi de aproximadamente 431 em Taiwan. Além disso, entre 15 e 17 de outubro, uma pessoa foi internada no Hospital Universitário de Pequim, nos arredores de Pequim, devido a um surto viral de SARS. Os sintomas de pneumonia, febre, prurido e pneumonia permaneceram altas durante toda a primeira quinzena de outubro. Os outros dois dias depois, no Hospital Universitário de Moscou, a doença causou mais mortes confirmadas do que não provocado anteriormente, com cerca de 211. O número de passageiros do Hospital Universitário de Moscou caiu 297 pessoas.\n", "\n", "A agência russa \"DPS\" declarou como uma das principais preocupações sobre a doença a saúde pública na República Popular da China. Em janeiro de 2020, o governo chinês lançou três programas de vacinação oral nos distritos urbanos em resposta à melhora global e a pandemia no Vale do Silício. O programa nacional de vacinação, chamado de \"Omgao\", consiste num conjunto de 36 cidades com um total de 1.429 municípios (em regiões mais remotas do país). A aplicação do programa foi feita em 27\n", "\n", "---\n", "CPU times: user 7.56 s, sys: 242 ms, total: 7.81 s\n", "Wall time: 7.8 s\n" ] } ], "source": [ "%%time\n", "#set top_k = 40 and num_return_sequences = 3\n", "sample_outputs = model_pt.generate(input_ids, pad_token_id=50256,\n", " do_sample=True, \n", " max_length=max_length, \n", " min_length=max_length,\n", " top_k=40,\n", " num_return_sequences=3)\n", "\n", "for i, sample_output in enumerate(sample_outputs):\n", " print(\">> Generated text {}\\n\\n{}\".format(i+1, tokenizer_pt.decode(sample_output.tolist())))\n", " print('\\n---')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Use case 2 (Top-p nucleus sampling)" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ">> Generated text 1\n", "\n", "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n", "\n", "A Rússia tem um mercado interno bruto (PIB) estimado entre US$ 1,2 bilhões e US$ 7 bilhões. O PIB \"per capita\" em 2007 foi de US $ 4,348. A maior parte do crescimento econômico anual ocorreu na agricultura. Em 2003, a taxa média anual era de 22%.\n", "\n", "Em 2004, cerca de 3 milhões de pessoas foram infectadas pelo coronavírus. Cerca de 17 mil casos foram identificados entre 2001 e 2002, incluindo aproximadamente 2,8% das crianças e adolescentes com idade inferior a seis anos; 5,1 milhão deles eram endêmicas ou sem cuidados médicos; 8,711 mortes ocorreram nos três primeiros meses de vida após o contato com o vírus; 13,6 mil haviam retornado à Rússia; e 14,4% estavam doentes devido aos sintomas causados pela doença durante seus últimos dias no exterior, principalmente idosos e recém-nascidos infectados.\n", "\n", "Cerca de 6 milhões de russos estão vivendo em países fora da União Europeia, sendo eles os Estados Unidos, China, Índia, Itália, Japão, Reino Unido, Austrália, Nova Zelândia, Países Baixos, Rússia, Turquia, Espanha, Suécia e África do Sul. Em novembro de 2008, estimava-se que 20 milhões de russos estariam potencialmente infetados naquele ano.\n", "\n", "A Organização Mundial da Saúde divulgou dados detalhados sobre as medidas antivirais prescritas por cada país segundo os padrões estabelecidos.\n", "\n", "O número total de infeção por coronavírus foi inicialmente estimado em 12.000 em 2005. Em janeiro de 2006, estimava-se que 924.300 pessoas tinham sido contaminadas. No mesmo mês, o Ministério da Saúde confirmou que havia aumentado o número de casos confirmados desde o início da pandemia. O número de mortes atribuídas ao coronavírus caiu de 50 para 54 em janeiro de 2006, enquanto novos surtos começaram em março de 2006. Os números de casos suspeitos aumentaram ainda mais após o início da epidemia em janeiro. A Organização Mundial da Saúde divulgou estatísticas oficiais em fevereiro de 2006 mostrando que todos os indivíduos que tenham sido contestados como tendo casos não apresentaram qualquer sintoma claro ou grave risco.\n", "\n", "Desde o início da pandemia, um grande número de pessoas têm sido afetadas por COVID-19. Mais tarde,\n", "\n", "---\n", ">> Generated text 2\n", "\n", "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n", "\n", "Em junho, a Organização Mundial da Saúde divulgou um relatório sobre o surto e previu que os casos confirmados pela Rússia naquele ano ultrapassariam 10 milhões de pessoas. Em julho, foi relatado que o vírus havia matado 50% das crianças entre 15 e 21 anos. Em agosto, o Ministério da Saúde confirmou que as mortes foram estimadas no total devido à infecção causada pelo vírus \"S\". Em setembro, uma nova pesquisa revelou que 90% dos infectados haviam sido mortos antes mesmo da chegada da vacina.\n", "\n", "O governo soviético aprovou em dezembro de 2008, como forma de comemorar o centenário do nascimento da primeira vacina contra o coronavírus (CVD), dois novos centros comerciais russos estão sendo construídos na cidade de Moscou. No entanto, apenas três deles são permanentes: um na Rússia central; outro na fronteira leste com a Ucrânia; e outro nos subúrbios de Vyacheslavsky-Kaliningrado. Em 29 de janeiro de 2009, o governo anunciou planos futuros para construir estações comerciais em todas as regiões afetadas pelos surtos.\n", "\n", "Até maio de 2013, todos os postos de saúde oficiais estabelecidos no país estavam fechados desde então. Em 1º de março de 2014, a Rússia fechou seus aeroportos, deixando suas fronteiras fechadas para os voos domésticos e internacionais.\n", "\n", "Desde abril de 2015, cerca de 318 mil imigrantes estrangeiros vêm anualmente da Rússia, principalmente alemães étnicos russos residentes na Alemanha Oriental e eslavos orientais radicados lá. A entrada desses migrantes trouxe também grandes quantidades de problemas econômicos ao país. Os últimos emigrantes vindos ilegalmente vieram sobretudo após a Segunda Guerra Mundial.\n", "\n", "No final de 2019, muitos milhares de trabalhadores europeus emigraram para o leste da Europa para escapar da guerra civil russa. A maioria destes emigrantes veio da Rússia depois da invasão alemã de 1941.\n", "\n", "De acordo com a Organização Mundial da Saúde, a taxa de mortalidade infantil na Rússia diminuiu substancialmente desde o início do século XX, mas o número caiu em grande medida entre 1990 e 2001, quando o número aumentou novamente, alcançando a cifra de 7,5% entre 1995 e 2000. No entanto, esse número permaneceu estável até 2007, quando passou de 4,1%, e subiu novamente para 5,9%. As taxas gerais de mortalidade diminuíram consideravelmente durante este\n", "\n", "---\n", ">> Generated text 3\n", "\n", "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n", "\n", "A Organização Mundial de Saúde (OMS) estima que, no final de dezembro de 2015, havia pelo menos 50 milhões de pessoas infectadas com o vírus na Rússia e Ucrânia. Estimativas semelhantes foram feitas pela OMS sobre os casos registrados desde a década passada. Em março de 2018, o Ministério da Saúde confirmou a descoberta de um novo coronavírus e recomendou aos cidadãos russos manter a disposição para evitar viagens ao exterior após o início das aulas médicas ou exames médicos.\n", "\n", "Em novembro de 2016, o Ministro da Saúde russo confirmou que todos os indivíduos infectados poderiam ser testados em março; assim como seus familiares e amigos, eles podem fazer teste clínico em uma unidade cirúrgica do hospital da cidade em que são colocados, caso necessário. Também foi anunciado no mesmo dia que o Ministério do Trabalho revelou que as autoridades russas estão trabalhando em conjunto visando reduzir o número de mortes causadas pelas epidemias. A agência informou que a Agência Nacional de Vigilância Sanitária Russa (Anvisa) começou a monitorar a pandemia através de máscaras faciais nos hospitais.\n", "\n", "Em abril de 2019, o Ministério da Saúde divulgou que 582 mil pessoas haviam sido infetadas com o vírus no país entre janeiro de 2019 e maio de 2020. Cerca de 370 mil desses pacientes estariam diretamente relacionados à doença.\n", "\n", "A Rússia também tem planos promissores para produzir vacinas que sejam eficazes contra o coronavírus, incluindo a vacina anti-SIDA e antivirais (ver Lista Vermelha da OMS).\n", "\n", "Em 1º de julho de 2017, a Organização Mundial de Saúde lançou uma nota oficial alertando que \"um grande aumento pode vir da necessidade de medidas preventivas necessárias\" antes do início das aulas médicas em escolas públicas nas cidades ucranianas.\n", "\n", "Em outubro de 2017, a Secretaria Municipal de Saúde ucraniana publicou uma nota oficial informando que os profissionais responsáveis ​​pela coordenação da vacinação deveriam estar preparados, bem como suas famílias e amigos durante a realização de exames adicionais para determinar sua saúde mental.\n", "\n", "Em 30 de junho de 2019, o Ministério da Saúde lançou um comunicado afirmando que três grupos escolares teriam dificuldades de administrar adequadamente o vacina contra o coronavírus na Rússia:\n", "\n", "A primeira ministra russa, Valentina Tereshinakova, disse em entrevista coletiva que não haverá\n", "\n", "---\n", "CPU times: user 37.9 s, sys: 1.78 s, total: 39.7 s\n", "Wall time: 39.6 s\n" ] } ], "source": [ "%%time\n", "#set top_p = 0.95, top_k = 50, temperature = 0.7, repetition_penalty = 1.2 and num_return_sequences = 3\n", "sample_outputs = model_pt.generate(input_ids, pad_token_id=50256,\n", " do_sample=True,\n", " max_length=max_length, \n", " min_length=max_length,\n", " repetition_penalty=1.2,\n", " temperature=0.7,\n", " top_k=50, \n", " top_p=0.95, \n", " num_return_sequences=3)\n", "\n", "for i, sample_output in enumerate(sample_outputs):\n", " print(\">> Generated text {}\\n\\n{}\".format(i+1, tokenizer_pt.decode(sample_output.tolist())))\n", " print('\\n---')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are the first, fortunately surprised by the efficiency of fine-tuning in Portuguese an English pre-trained transformer-based language model like GPT-2 small.\n", "\n", "In about 1 day using 1 GPU and a little over 1 GB of Portuguese texts, we managed to obtain a **GPorTuguese-2 capable of generating contextual Portuguese texts of a level comparable to that of the GPT-2 used by OpenAI in 2019**.\n", "\n", "Happy.\n", "\n", "The next step would be to apply our fine-tuning method to most recent NLP models like GPT-3, BART, T5 or Reformer. Let’s do it?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## END" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 }