{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Byte-level BPE, an universal tokenizer but...\n",
    "\n",
    "> Study about the \"universality\" of a Byte-level Byte-Pair-Encoding tokenizer (BBPE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n",
    "- Date: July 2020\n",
    "- Post in medium: [Byte-level BPE, an universal tokenizer but…](https://medium.com/@pierre_guillou/byte-level-bpe-an-universal-tokenizer-but-aff932332ffe) (07/03/2020)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this study, we will see that, while it is true that a BBPE tokenizer (Byte-level Byte-Pair-Encoding) trained on a huge monolingual corpus can tokenize any word of any language (there is no unknown token), **it requires on average almost 70% of additional tokens** when it is applied to a text in a language different from that used for its training.\n",
    "\n",
    "This information is key when it comes to choosing a tokenizer to train a natural language model like a Transformer model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a tokenizer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just read the great [tutorial \"Tokenizer summary\"](https://huggingface.co/transformers/master/tokenizer_summary.html) from Sylvain Gugger (Hugging Face)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the Byte-level BPE (BBPE) tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the [tutorial \"Tokenizer summary\"](https://huggingface.co/transformers/master/tokenizer_summary.html), read the paragraphs [Byte-Pair Encoding](https://huggingface.co/transformers/master/tokenizer_summary.html#byte-pair-encoding) and [Byte-level BPE](https://huggingface.co/transformers/master/tokenizer_summary.html#byte-level-bpe) to get the best overview of a Byte-level BPE (Byte-level Byte-Pair-Encoding) and read the Abstract and Conclusion paragraphs of the original paper: [Neural Machine Translation with Byte-Level Subwords](https://arxiv.org/pdf/1909.03341.pdf]) (Facebook AI, 12/05/2019).\n",
    "\n",
    "> **[Abstract]** Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. **In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is.** We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. **Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE.** In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.\n",
    "\n",
    "> **[Conclusion]** We proposed BBPE which builds a byte-level subword vocabulary for machine translation. It results in a much more compact vocabulary than character-based ones do without the loss of performance. In multilingual settings, the former often outperforms the latter. **BBPE does not have any out-of-vocabulary tokens, allowing us to transfer a model using BBPE between languages with non-overlapping vocabularies.** This transfer learning paradigm is actually very generic and can be applied to any languages and datasets for performance gain or training acceleration. With the same vocabulary size, BBPE segments sentences into shorter sequences than character-based methods do, leading to faster training and inference. Our future work includes: eliminating source-target sentence length imbalance; evaluating BBPE in one-to-many and many-to-many translation settings; exploring better segmentation algorithms for byte-level subwords."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the tokenizers and NLP libraries used in this study"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenizers and NLP libraries used to perform this study were:\n",
    "- English pre-trained GPT2 tokenizer ([GPT2TokenizerFast](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizerfast)) from the [Transformers](https://github.com/huggingface/transformers) library (Hugging Face, version 3.0.0): it is a Fast GPT-2 BBPE tokenizer (backed by Hugging Face's tokenizers library)\n",
    "- Portuguese trained ByteLevelBPETokenizer tokenizer from the [Tokenizers](https://github.com/huggingface/tokenizers) library (Hugging Face, version 0.8.0)\n",
    "- Deep Learning library [fastai v2](https://github.com/fastai/fastai2) (fastai2, version 0.0.17) and the Wikipedia downloading functions of the file [nlputils_fastai2.py](https://github.com/piegu/fastai-projects/blob/master/nlputils_fastai2.py)\n",
    "\n",
    "**Note about the Deep Learning libraries**: the [Tokenizers](https://github.com/huggingface/tokenizers) and [Transformers](https://huggingface.co/transformers/) libraries from [Hugging Face](https://huggingface.co/) are today the most up-to-date NLP libraries (Natural Language Processing) used all over the world when fastai v2 is a great tool for training Deep Learning models, especially with powerful fastai tools like Learning rate finder, Mixed precision training, Distributed training, Gradual unfreezing, Differential learning rates and 1cycle policy.\n",
    "\n",
    "**Note about the choice of the GPT2 tokenizer**: we could have chosen another pre-trained BBPE tokenizer for this study. The key point is to use BBPE tokenizers trained on huge corpus because they can thus tokenize any word of any language without using the unknown token."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbpresent": {
     "id": "151cd18f-76e3-440f-a8c7-ffa5c6b5da01"
    }
   },
   "outputs": [],
   "source": [
    "from fastai2.text.all import *\n",
    "from nlputils_fastai2 import * \n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'archive_path': '/mnt/home/pierre/.fastai/archive',\n",
       " 'data_path': '/mnt/home/pierre/.fastai/data',\n",
       " 'model_path': '/mnt/home/pierre/.fastai/models',\n",
       " 'storage_path': '/mnt/home/pierre/.fastai/data',\n",
       " 'version': 2}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get config of fastai2 paths\n",
    "config = Config()\n",
    "config.d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "cf070ab7-babb-4cf0-a315-401f65461dc8"
    }
   },
   "source": [
    "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents (for other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "nbpresent": {
     "id": "70da588b-8af1-4f97-97c2-c9f2d4d46e1a"
    }
   },
   "outputs": [],
   "source": [
    "lang = 'pt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "nbpresent": {
     "id": "701ab344-0430-4f43-bbe2-337a12cae6be"
    }
   },
   "outputs": [],
   "source": [
    "# setup new path_data and create the corresponding folder\n",
    "name = f'{lang}_wiki'\n",
    "data_path = config['data_path']\n",
    "path_data = data_path/name\n",
    "path_data.mkdir(exist_ok=True, parents=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2"
    }
   },
   "source": [
    "## Download Wikipedia in Portuguese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By fastai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: all the following methods come from the file [nlputils_fastai2.py](https://github.com/piegu/fastai-projects/blob/master/nlputils_fastai2.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Path('/mnt/home/pierre/course-v4/nbs'),\n",
       " Path('/mnt/home/pierre/.fastai/data/pt_wiki'))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Path.cwd(), path_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# Download Wikipedia in Portuguese (zip of 1.62Go)\n",
    "# duration: 40m 30s\n",
    "get_wiki(path_data,lang)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `get_wiki(path_data,lang)` breaks, fix the download manually in a terminal:\n",
    "- mkdir -p /mnt/home/pierre/.fastai/data/pt_wiki\n",
    "- cd /mnt/home/pierre/.fastai/data/pt_wiki\n",
    "- wget -c https://dumps.wikimedia.org/ptwiki/latest/ptwiki-latest-pages-articles.xml.bz2\n",
    "- bzip2 -dk ptwiki-latest-pages-articles.xml.bz2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "--2020-07-02 18:35:23--  https://dumps.wikimedia.org/ptwiki/latest/ptwiki-latest-pages-articles.xml.bz2\n",
    "Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 208.80.154.7, 2620:0:861:1:208:80:154:7\n",
    "Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|208.80.154.7|:443... connected.\n",
    "HTTP request sent, awaiting response... 200 OK\n",
    "Length: 1744626313 (1.6G) [application/octet-stream]\n",
    "Saving to: ‘ptwiki-latest-pages-articles.xml.bz2’\n",
    "```\n",
    "\n",
    "```\n",
    "ptwiki-latest-pages-articles.xml.bz2\n",
    "100%[==================================================================================>]   \n",
    "1.62G   791KB/s    in 40m 30s\n",
    "2020-07-02 19:15:54 (701 KB/s) - ‘ptwiki-latest-pages-articles.xml.bz2’ saved [1744626313/1744626313]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And re-run `get_wiki(path_data,lang)` once the download is successful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "extracting...\n",
      "CPU times: user 22.7 ms, sys: 18.3 ms, total: 41 ms\n",
      "Wall time: 10min 3s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Download Wikipedia in Portuguese\n",
    "get_wiki(path_data,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<doc id=\"220\" url=\"https://pt.wikipedia.org/wiki?curid=220\" title=\"Astronomia\">\r\n",
      "Astronomia\r\n",
      "\r\n",
      "Astronomia é uma ciência natural que estuda corpos celestes (como estrelas, planetas, cometas, nebulosas, aglomerados de estrelas, galáxias) e fenômenos que se originam fora da atmosfera da Terra (como a radiação cósmica de fundo em micro-ondas). Preocupada com a evolução, a física, a química e o movimento de objetos celestes, bem como a formação e o desenvolvimento do universo.\r\n"
     ]
    }
   ],
   "source": [
    "# Check the content downloaded\n",
    "name = f'{lang}wiki'\n",
    "!head -n4 {path_data}/{name}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "100000\n",
      "200000\n",
      "300000\n",
      "400000\n",
      "500000\n",
      "600000\n",
      "700000\n",
      "800000\n",
      "900000\n",
      "1000000\n",
      "1100000\n",
      "1200000\n",
      "1300000\n",
      "1400000\n",
      "1500000\n",
      "1600000\n",
      "1700000\n",
      "1800000\n",
      "1900000\n",
      "2000000\n",
      "2100000\n",
      "2200000\n",
      "2300000\n",
      "2400000\n",
      "2500000\n",
      "2600000\n",
      "2700000\n",
      "2800000\n",
      "2900000\n",
      "3000000\n",
      "3100000\n",
      "3200000\n",
      "3300000\n",
      "3400000\n",
      "3500000\n",
      "3600000\n",
      "3700000\n",
      "3800000\n",
      "3900000\n",
      "4000000\n",
      "4100000\n",
      "4200000\n",
      "4300000\n",
      "4400000\n",
      "4500000\n",
      "4600000\n",
      "4700000\n",
      "4800000\n",
      "4900000\n",
      "5000000\n",
      "5100000\n",
      "5200000\n",
      "5300000\n",
      "5400000\n",
      "5500000\n",
      "5600000\n",
      "5700000\n",
      "5800000\n",
      "5900000\n",
      "6000000\n",
      "6100000\n",
      "6200000\n",
      "6300000\n",
      "6400000\n",
      "6500000\n",
      "6600000\n",
      "6700000\n",
      "6800000\n",
      "6900000\n",
      "7000000\n",
      "7100000\n",
      "7200000\n",
      "7300000\n",
      "7400000\n",
      "CPU times: user 36.3 s, sys: 13.8 s, total: 50.1 s\n",
      "Wall time: 5min 26s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Split global download file to one article by text file\n",
    "dest = split_wiki(path_data,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mnt/home/pierre/.fastai/data/pt_wiki/docs/Fotografia.txt\n",
      "/mnt/home/pierre/.fastai/data/pt_wiki/docs/Espadanedo (Macedo de Cavaleiros).txt\n",
      "/mnt/home/pierre/.fastai/data/pt_wiki/docs/Jacques-Germain Soufflot.txt\n",
      "/mnt/home/pierre/.fastai/data/pt_wiki/docs/Faculdade de Medicina da Universidade de São Paulo.txt\n",
      "/mnt/home/pierre/.fastai/data/pt_wiki/docs/Escola do Teatro Bolshoi no Brasil.txt\n"
     ]
    }
   ],
   "source": [
    "# Check the splitting\n",
    "dest = path_data/'docs'\n",
    "for file in dest.ls()[:5]:\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204315 files - 194843190 tokens\n",
      "CPU times: user 38.5 s, sys: 15.7 s, total: 54.1 s\n",
      "Wall time: 3min 45s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# number of files and total tokens\n",
    "num_files, num_tokens = get_num_tokens(dest)\n",
    "print(f'{num_files} files - {num_tokens} tokens')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.6G\t/mnt/home/pierre/.fastai/data/pt_wiki/docs\r\n"
     ]
    }
   ],
   "source": [
    "# Size of downloaded data \n",
    "!du -hs {dest}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By Hugging Face"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[nlp](https://github.com/huggingface/nlp) is a lightweight and extensible library from Hugging Face to easily share and access datasets and evaluation metrics for Natural Language Processing (NLP).\n",
    "- Colab tutorial: https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb\n",
    "- Online dataset explorer: https://huggingface.co/nlp/viewer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[ WARNING ]** We did try to use it in order to download Wikipedia in Portuguese but without success. However, to help people solving this issue, we decided to leave our code in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# source: https://huggingface.co/nlp/viewer/?dataset=wikipedia&config=20200501.pt\n",
    "# !pip install nlp\n",
    "\n",
    "# Issues\n",
    "# source: https://github.com/huggingface/nlp/issues/227\n",
    "# !pip instal apache_beam\n",
    "# !pip install dill==0.3.1.1\n",
    "# !pip install apache-beam[interactive]\n",
    "# !pip install mwparserfromhell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nlp==0.3.0\r\n"
     ]
    }
   ],
   "source": [
    "!pip freeze | grep nlp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c6d3ba8825184c71aa82c44d5e318d85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=13008.0, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f2a6cf0107f44488a393822998a4ec1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=28653.0, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Downloading and preparing dataset wikipedia/20200501.pt (download: Unknown size, generated: Unknown size, total: Unknown size) to /mnt/home/pierre/.cache/huggingface/datasets/wikipedia/20200501.pt/1.0.0...\n"
     ]
    },
    {
     "ename": "MissingBeamOptions",
     "evalue": "Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/\nIf you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). \nExample of usage: \n\t`load_dataset('wikipedia', '20200501.pt', beam_runner='DirectRunner')`",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mMissingBeamOptions\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-48-05e2dc27b99b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mnlp\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mload_dataset\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mdataset\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mload_dataset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'wikipedia'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'20200501.pt'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.conda/envs/fastai2/lib/python3.7/site-packages/nlp/load.py\u001b[0m in \u001b[0;36mload_dataset\u001b[0;34m(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)\u001b[0m\n\u001b[1;32m    522\u001b[0m         \u001b[0mdownload_mode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdownload_mode\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    523\u001b[0m         \u001b[0mignore_verifications\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mignore_verifications\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 524\u001b[0;31m         \u001b[0msave_infos\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msave_infos\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    525\u001b[0m     )\n\u001b[1;32m    526\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/fastai2/lib/python3.7/site-packages/nlp/builder.py\u001b[0m in \u001b[0;36mdownload_and_prepare\u001b[0;34m(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)\u001b[0m\n\u001b[1;32m    430\u001b[0m                 \u001b[0mverify_infos\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0msave_infos\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mignore_verifications\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    431\u001b[0m                 self._download_and_prepare(\n\u001b[0;32m--> 432\u001b[0;31m                     \u001b[0mdl_manager\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdl_manager\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverify_infos\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mverify_infos\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mdownload_and_prepare_kwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    433\u001b[0m                 )\n\u001b[1;32m    434\u001b[0m                 \u001b[0;31m# Sync info\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/fastai2/lib/python3.7/site-packages/nlp/builder.py\u001b[0m in \u001b[0;36m_download_and_prepare\u001b[0;34m(self, dl_manager, verify_infos)\u001b[0m\n\u001b[1;32m    821\u001b[0m                 \u001b[0;34m\"Dataset is small enough, you can use the local beam runner called \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    822\u001b[0m                 \u001b[0;34m\"`DirectRunner` (you may run out of memory). \\nExample of usage: \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 823\u001b[0;31m                 \u001b[0;34m\"\\n\\t`{}`\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0musage_example\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    824\u001b[0m             )\n\u001b[1;32m    825\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mMissingBeamOptions\u001b[0m: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/\nIf you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). \nExample of usage: \n\t`load_dataset('wikipedia', '20200501.pt', beam_runner='DirectRunner')`"
     ]
    }
   ],
   "source": [
    "# source: https://huggingface.co/nlp/viewer/?dataset=wikipedia&config=20200501.pt\n",
    "import nlp\n",
    "from nlp import load_dataset\n",
    "dataset = load_dataset('wikipedia','20200501.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2"
    }
   },
   "source": [
    "### Create text and csv files of Wikipedia in Portuguese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The text file (all the articles in one file) will allow the training of the Portuguese tokenizer and the csv one will facilitate the tests of the study."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dest = path_data/'docs'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1000\n",
      "2000\n",
      "3000\n",
      "4000\n",
      "5000\n",
      "6000\n",
      "7000\n",
      "8000\n",
      "9000\n",
      "10000\n",
      "11000\n",
      "12000\n",
      "13000\n",
      "14000\n",
      "15000\n",
      "16000\n",
      "17000\n",
      "18000\n",
      "19000\n",
      "20000\n",
      "21000\n",
      "22000\n",
      "23000\n",
      "24000\n",
      "25000\n",
      "26000\n",
      "27000\n",
      "28000\n",
      "29000\n",
      "30000\n",
      "31000\n",
      "32000\n",
      "33000\n",
      "34000\n",
      "35000\n",
      "36000\n",
      "37000\n",
      "38000\n",
      "39000\n",
      "40000\n",
      "41000\n",
      "42000\n",
      "43000\n",
      "44000\n",
      "45000\n",
      "46000\n",
      "47000\n",
      "48000\n",
      "49000\n",
      "50000\n",
      "51000\n",
      "52000\n",
      "53000\n",
      "54000\n",
      "55000\n",
      "56000\n",
      "57000\n",
      "58000\n",
      "59000\n",
      "60000\n",
      "61000\n",
      "62000\n",
      "63000\n",
      "64000\n",
      "65000\n",
      "66000\n",
      "67000\n",
      "68000\n",
      "69000\n",
      "70000\n",
      "71000\n",
      "72000\n",
      "73000\n",
      "74000\n",
      "75000\n",
      "76000\n",
      "77000\n",
      "78000\n",
      "79000\n",
      "80000\n",
      "81000\n",
      "82000\n",
      "83000\n",
      "84000\n",
      "85000\n",
      "86000\n",
      "87000\n",
      "88000\n",
      "89000\n",
      "90000\n",
      "91000\n",
      "92000\n",
      "93000\n",
      "94000\n",
      "95000\n",
      "96000\n",
      "97000\n",
      "98000\n",
      "99000\n",
      "100000\n",
      "101000\n",
      "102000\n",
      "103000\n",
      "104000\n",
      "105000\n",
      "106000\n",
      "107000\n",
      "108000\n",
      "109000\n",
      "110000\n",
      "111000\n",
      "112000\n",
      "113000\n",
      "114000\n",
      "115000\n",
      "116000\n",
      "117000\n",
      "118000\n",
      "119000\n",
      "120000\n",
      "121000\n",
      "122000\n",
      "123000\n",
      "124000\n",
      "125000\n",
      "126000\n",
      "127000\n",
      "128000\n",
      "129000\n",
      "130000\n",
      "131000\n",
      "132000\n",
      "133000\n",
      "134000\n",
      "135000\n",
      "136000\n",
      "137000\n",
      "138000\n",
      "139000\n",
      "140000\n",
      "141000\n",
      "142000\n",
      "143000\n",
      "144000\n",
      "145000\n",
      "146000\n",
      "147000\n",
      "148000\n",
      "149000\n",
      "150000\n",
      "151000\n",
      "152000\n",
      "153000\n",
      "154000\n",
      "155000\n",
      "156000\n",
      "157000\n",
      "158000\n",
      "159000\n",
      "160000\n",
      "161000\n",
      "162000\n",
      "163000\n",
      "164000\n",
      "165000\n",
      "166000\n",
      "167000\n",
      "168000\n",
      "169000\n",
      "170000\n",
      "171000\n",
      "172000\n",
      "173000\n",
      "174000\n",
      "175000\n",
      "176000\n",
      "177000\n",
      "178000\n",
      "179000\n",
      "180000\n",
      "181000\n",
      "182000\n",
      "183000\n",
      "184000\n",
      "185000\n",
      "186000\n",
      "187000\n",
      "188000\n",
      "189000\n",
      "190000\n",
      "191000\n",
      "192000\n",
      "193000\n",
      "194000\n",
      "195000\n",
      "196000\n",
      "197000\n",
      "198000\n",
      "199000\n",
      "200000\n",
      "201000\n",
      "202000\n",
      "203000\n",
      "204000\n",
      "all texts from wikipedia pt in the file /mnt/home/pierre/.fastai/data/pt_wiki/all_texts_ptwiki.txt\n",
      "\n",
      "CPU times: user 49 s, sys: 15.1 s, total: 1min 4s\n",
      "Wall time: 3min 48s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "get_one_clean_file(dest,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2G\t/mnt/home/pierre/.fastai/data/pt_wiki/all_texts_ptwiki.txt\r\n"
     ]
    }
   ],
   "source": [
    "# size\n",
    "fname = f'all_texts_{lang}wiki.txt'\n",
    "!du -hs {dest.parent/fname}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all texts from wikipedia pt in the file /mnt/home/pierre/.fastai/data/pt_wiki/all_texts_ptwiki.csv\n",
      "\n",
      "CPU times: user 1min 14s, sys: 11 s, total: 1min 25s\n",
      "Wall time: 3min 56s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "get_one_clean_csv_file(dest,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2G\t/mnt/home/pierre/.fastai/data/pt_wiki/all_texts_ptwiki.csv\r\n"
     ]
    }
   ],
   "source": [
    "# size\n",
    "fname = f'all_texts_{lang}wiki.csv'\n",
    "!du -hs {dest.parent/fname}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fotografia (do grego φως [\"fós\"] (\"luz\"), e γραφις [\"grafis\"] (\"estilo\", \"pincel\") ou γραφη \"grafê\", e significa \"desenhar com luz e contraste\"), por definição, é essencialmente a \"técnica de criação\" de imagens por meio de exposição luminosa, fixando-as em uma superfície sensível. A primeira fotografia reconhecida remonta ao ano de 1826 e é atribuída ao francês Joseph Nicéphore Niépce. Contudo, a invenção da fotografia não é obra de um só autor, mas um processo de acúmulo de avanços por parte de muitas pessoas, trabalhando, juntas ou em paralelo, ao longo de muitos anos. Se por um lado os...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Espadanedo é uma antiga freguesia portuguesa do concelho de Macedo de Cavaleiros, com 17,90 km² de área e 188 habitantes (2011). A sua densidade populacional era 10,5 hab/km².\\nFoi extinta (agregada) pela reorganização administrativa de 2012/2013, sendo o seu território integrado na União de Freguesias de Espadanedo, Edroso, Murçós e Soutelo Mourisco.\\n\\nA antiga freguesia de S. Miguel de Espadanedo e Valongo pertenceu ao antigo concelho de Torre D. Chama, extinto a 24 de Outubro de 1855, em 1839 aparece agregada à comarca de Bragança e em 1852 à de Mirandela. Passa a pertencer definitivam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo. O seu trabalho mais conhecido é, sem dúvida, o Panthéon (Panteão) de Paris, construído a partir de 1755, inicialmente uma igreja dedicada a Santa Genoveva.\\n\\nSoufflot nasceu em Irancy, perto de Auxerre, na França. Com 18 anos, entrou para a Academia Francesa de Roma, onde os jovens estudantes da década de 1750 se tornariam na primeira geração de criativos pleanamente neoclássicos. Ficará na Itália de 1731 a 1738. Quando volto...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A Faculdade de Medicina da Universidade de São Paulo (FMUSP) é uma escola médica da Universidade de São Paulo. Foi fundada em 1912 com o nome de \"Faculdade de Medicina e Cirurgia de São Paulo\" por Arnaldo Vieira de Carvalho (1867-1920), médico formado em 1888 pela Faculdade de Medicina do Rio de Janeiro. Em homenagem ao ilustre fundador, a Faculdade é, ainda hoje, chamada de a \"Casa de Arnaldo\" por seus alunos e ex-alunos. Em 1925 teve seu nome alterado para \"Faculdade de Medicina de São Paulo\" e em 1934, foi incorporada à recém-criada Universidade de São Paulo, passando a ter a atual desi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A Escola do Teatro Bolshoi no Brasil é uma tradicional escola de balé existente na cidade de Joinville, no estado de Santa Catarina. Fundada em 2000, é a única filial do Teatro Bolshoi de Moscou e possui alunos de vários estados brasileiros. Tem como missão formar artistas cidadãos, promover e difundir a arte-educação.\\n\\nA instituição foi fundada em 15 de março de 2000, é a única filial do Teatro Bolshoi. \\n\\nUm orgulho para o Brasil e para Joinville, cidade sede. A Escola do Teatro Bolshoi no Brasil, com professores russos e brasileiros, forma bailarinos com a mesma precisão, técnica e q...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      text\n",
       "0  Fotografia (do grego φως [\"fós\"] (\"luz\"), e γραφις [\"grafis\"] (\"estilo\", \"pincel\") ou γραφη \"grafê\", e significa \"desenhar com luz e contraste\"), por definição, é essencialmente a \"técnica de criação\" de imagens por meio de exposição luminosa, fixando-as em uma superfície sensível. A primeira fotografia reconhecida remonta ao ano de 1826 e é atribuída ao francês Joseph Nicéphore Niépce. Contudo, a invenção da fotografia não é obra de um só autor, mas um processo de acúmulo de avanços por parte de muitas pessoas, trabalhando, juntas ou em paralelo, ao longo de muitos anos. Se por um lado os...\n",
       "1  Espadanedo é uma antiga freguesia portuguesa do concelho de Macedo de Cavaleiros, com 17,90 km² de área e 188 habitantes (2011). A sua densidade populacional era 10,5 hab/km².\\nFoi extinta (agregada) pela reorganização administrativa de 2012/2013, sendo o seu território integrado na União de Freguesias de Espadanedo, Edroso, Murçós e Soutelo Mourisco.\\n\\nA antiga freguesia de S. Miguel de Espadanedo e Valongo pertenceu ao antigo concelho de Torre D. Chama, extinto a 24 de Outubro de 1855, em 1839 aparece agregada à comarca de Bragança e em 1852 à de Mirandela. Passa a pertencer definitivam...\n",
       "2  Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo. O seu trabalho mais conhecido é, sem dúvida, o Panthéon (Panteão) de Paris, construído a partir de 1755, inicialmente uma igreja dedicada a Santa Genoveva.\\n\\nSoufflot nasceu em Irancy, perto de Auxerre, na França. Com 18 anos, entrou para a Academia Francesa de Roma, onde os jovens estudantes da década de 1750 se tornariam na primeira geração de criativos pleanamente neoclássicos. Ficará na Itália de 1731 a 1738. Quando volto...\n",
       "3  A Faculdade de Medicina da Universidade de São Paulo (FMUSP) é uma escola médica da Universidade de São Paulo. Foi fundada em 1912 com o nome de \"Faculdade de Medicina e Cirurgia de São Paulo\" por Arnaldo Vieira de Carvalho (1867-1920), médico formado em 1888 pela Faculdade de Medicina do Rio de Janeiro. Em homenagem ao ilustre fundador, a Faculdade é, ainda hoje, chamada de a \"Casa de Arnaldo\" por seus alunos e ex-alunos. Em 1925 teve seu nome alterado para \"Faculdade de Medicina de São Paulo\" e em 1934, foi incorporada à recém-criada Universidade de São Paulo, passando a ter a atual desi...\n",
       "4  A Escola do Teatro Bolshoi no Brasil é uma tradicional escola de balé existente na cidade de Joinville, no estado de Santa Catarina. Fundada em 2000, é a única filial do Teatro Bolshoi de Moscou e possui alunos de vários estados brasileiros. Tem como missão formar artistas cidadãos, promover e difundir a arte-educação.\\n\\nA instituição foi fundada em 15 de março de 2000, é a única filial do Teatro Bolshoi. \\n\\nUm orgulho para o Brasil e para Joinville, cidade sede. A Escola do Teatro Bolshoi no Brasil, com professores russos e brasileiros, forma bailarinos com a mesma precisão, técnica e q..."
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(dest.parent/fname)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Byte-level BPE (BBPE) tokenizers from Transformers and Tokenizers (Hugging Face libraries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are following 3 steps in order to get 2 identical GPT2 tokenizers, one trained on an English corpus and the other on Wikipedia in Portuguese\n",
    "1. **Get the pre-trained GPT2 Tokenizer (pre-training with an English corpus) from the Transformers library (Hugging Face)**: it will give us the tokenizer stucture we need and the pre-trained English tokenizer.\n",
    "2. **Train a Byte-level BPE (BBPE) Tokenizer on the Portuguese wikipedia corpus by using the Tokenizers library (Hugging Face)**: this will give us the vocabulary files of our GPT2 tokenizer in Portuguese.\n",
    "3. **Import the tokenizer config files in Portuguese into the pre-trained GPT2 Tokenizer**: it will give us a tokenizer structure with the vocab in Portuguese."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Get the pre-trained GPT2 Tokenizer (pre-training with an English corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, will need to install the transformers library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transformers==3.0.0\r\n"
     ]
    }
   ],
   "source": [
    "# ! pip install transformers\n",
    "!pip freeze | grep transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can get the English pre-trained GPT2 tokenizer ([GPT2Tokenizer](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizer) or [GPT2TokenizerFast](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizerfast))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 260 ms, sys: 63 ms, total: 323 ms\n",
      "Wall time: 1.53 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from transformers import GPT2TokenizerFast\n",
    "\n",
    "pretrained_weights = 'gpt2'\n",
    "tokenizer_en = GPT2TokenizerFast.from_pretrained(pretrained_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To correct the warning about token_pad (GPT2TokenizerFast), run the following code\n",
    "# source: https://github.com/huggingface/transformers/issues/2648#issuecomment-616177044\n",
    "tokenizer_en.pad_token = tokenizer_en.eos_token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the GPT2 tokenizer vocab. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------- vocab ----------\n",
      "\n",
      "vocab_files_names: {'vocab_file': 'vocab.json', 'merges_file': 'merges.txt'}\n",
      "\n",
      "vocab_file\n",
      "-  gpt2 : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json\n",
      "-  gpt2-medium : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-medium-vocab.json\n",
      "-  gpt2-large : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-large-vocab.json\n",
      "-  gpt2-xl : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-xl-vocab.json\n",
      "-  distilgpt2 : https://s3.amazonaws.com/models.huggingface.co/bert/distilgpt2-vocab.json\n",
      "\n",
      "merges_file\n",
      "-  gpt2 : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt\n",
      "-  gpt2-medium : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-medium-merges.txt\n",
      "-  gpt2-large : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-large-merges.txt\n",
      "-  gpt2-xl : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-xl-merges.txt\n",
      "-  distilgpt2 : https://s3.amazonaws.com/models.huggingface.co/bert/distilgpt2-merges.txt\n",
      "\n",
      "vocab_size: 50257\n",
      "\n",
      "First 50 items of the vocab: {'ormon': 10615, 'Ġ13': 1511, '659': 36445, 'Ġbecome': 1716, 'Ġinvari': 25275, 'Ġsearch': 2989, 'Ġzo': 40565, 'YL': 45448, 'Ġreception': 16307, 'Ġseminars': 44160, 'anta': 4910, 'ĠAmos': 49104, 'ĠDefinition': 30396, 'Ġ3000': 20343, 'ĠDemocrat': 9755, 'ploy': 1420, 'ãĥİ': 25053, 'ĠECO': 39031, 'ï': 171, 'ĠKING': 32957}\n"
     ]
    }
   ],
   "source": [
    "# source: https://huggingface.co/transformers/_modules/transformers/tokenization_utils_fast.html\n",
    "\n",
    "print('---------- vocab ----------')\n",
    "print()\n",
    "\n",
    "print('vocab_files_names:',tokenizer_en.vocab_files_names)\n",
    "print()\n",
    "\n",
    "for k,v in tokenizer_en.pretrained_vocab_files_map.items():\n",
    "    print(k)\n",
    "    for kk,vv in v.items():\n",
    "        print('- ',kk,':',vv)\n",
    "    print()\n",
    "    \n",
    "print('vocab_size:',tokenizer_en.vocab_size)\n",
    "print()\n",
    "#print(tokenizer_en.get_vocab())\n",
    "\n",
    "num = 50\n",
    "print(f'First {num} items of the vocab: {dict(itertools.islice(tokenizer_en.get_vocab().items(), 20))}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Train a Byte-level BPE (BBPE) tokenizer on the Portuguese Wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the [Tokenizers](https://github.com/huggingface/tokenizers) library from Hugging Face in order to train a **Byte-level BPE (BBPE) Tokenizer** on the Portuguese Wikipedia with the objective to get the vocab files `vocab.json`, which is a list of the most frequent tokens ranked by frequency, and the `merges.txt` which is a list of merges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokenizers==0.8.0\r\n"
     ]
    }
   ],
   "source": [
    "# !pip install tokenizers\n",
    "!pip freeze | grep tokenizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50257"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get GPT2 tokenizer_en vocab size\n",
    "ByteLevelBPE_tokenizer_pt_vocab_size = tokenizer_en.vocab_size\n",
    "ByteLevelBPE_tokenizer_pt_vocab_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 25min 17s, sys: 5min 4s, total: 30min 21s\n",
      "Wall time: 2min 7s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# ByteLevelBPETokenizer Represents a Byte-level BPE as introduced by OpenAI with their GPT-2 model\n",
    "from tokenizers import ByteLevelBPETokenizer\n",
    "\n",
    "ByteLevelBPE_tokenizer_pt = ByteLevelBPETokenizer()\n",
    "\n",
    "# Get list of paths to corpus files\n",
    "paths = [str(path_data/'all_texts_ptwiki.txt')]\n",
    "\n",
    "# Customize training with <|endoftext|> special GPT2 token\n",
    "ByteLevelBPE_tokenizer_pt.train(files=paths, \n",
    "                                vocab_size=ByteLevelBPE_tokenizer_pt_vocab_size, \n",
    "                                min_frequency=2, \n",
    "                                special_tokens=[\"<|endoftext|>\"])\n",
    "\n",
    "# Get sequence length max of 1024\n",
    "ByteLevelBPE_tokenizer_pt.enable_truncation(max_length=1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/mnt/home/pierre/.fastai/data/pt_wiki/ByteLevelBPE_tokenizer_pt/vocab.json',\n",
       " '/mnt/home/pierre/.fastai/data/pt_wiki/ByteLevelBPE_tokenizer_pt/merges.txt']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save tokenizer\n",
    "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n",
    "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n",
    "if not (path_to_ByteLevelBPE_tokenizer_pt_rep).exists():\n",
    "    path_to_ByteLevelBPE_tokenizer_pt_rep.mkdir(exist_ok=True, parents=True)\n",
    "ByteLevelBPE_tokenizer_pt.save_model(str(path_to_ByteLevelBPE_tokenizer_pt_rep))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have both a `vocab.json`, which is a list of the most frequent tokens ranked by frequency, and a `merges.txt` list of merges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check our tokenizer pre-trained in Portuguese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50257, ['<|endoftext|>', '!', '\"', '#', '$'])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get vocab as a list\n",
    "ByteLevelBPE_tokenizer_pt_vocab = ByteLevelBPE_tokenizer_pt.get_vocab() \n",
    "ByteLevelBPE_tokenizer_pt_vocab_ls = [k for k, v in sorted(ByteLevelBPE_tokenizer_pt_vocab.items(), key=lambda item: item[1])]\n",
    "len(ByteLevelBPE_tokenizer_pt_vocab_ls),ByteLevelBPE_tokenizer_pt_vocab_ls[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([39, 1120, 298, 22335, 258, 11001, 14],\n",
       " ['G', 'osto', 'Ġdo', 'Ġqueijo', 'Ġe', 'Ġvinho', '.'],\n",
       " [(0, 1), (1, 5), (5, 8), (8, 15), (15, 17), (17, 23), (23, 24)])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"Gosto do queijo e vinho.\"\n",
    "output = ByteLevelBPE_tokenizer_pt.encode(text)\n",
    "output.ids,output.tokens,output.offsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input text: Gosto do queijo e vinho.\n",
      "tokens ids: [39, 1120, 298, 22335, 258, 11001, 14]\n",
      "back to text: Gosto do queijo e vinho.\n"
     ]
    }
   ],
   "source": [
    "back_to_text = ByteLevelBPE_tokenizer_pt.decode(ByteLevelBPE_tokenizer_pt.encode(text).ids)\n",
    "\n",
    "print('input text:', text)\n",
    "print('tokens ids:', output.ids)\n",
    "print('back to text:', back_to_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Import the tokenizer config files in Portuguese into the pre-trained GPT2 Tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to get exactly the same tokenizer model but with different vocabs (English and Portuguese), we import the Portuguese vocab files (`vocab.json` and `merges.txt`) into the `GPT2TokenizerFast` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the path to ByteLevelBPE_tokenizer_pt config files\n",
    "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n",
    "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n",
    "\n",
    "# import the pre-trained GPT2TokenizerFast tokenizer with the tokenizer_pt config files\n",
    "tokenizer_pt = GPT2TokenizerFast.from_pretrained(\n",
    "    str(path_to_ByteLevelBPE_tokenizer_pt_rep), \n",
    "    pad_token='<|endoftext|>')\n",
    "\n",
    "# Get sequence length max of 1024\n",
    "tokenizer_pt.model_max_length = 1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input text: Gosto do queijo e vinho.\n",
      "tokens ids: [39, 1120, 298, 22335, 258, 11001, 14]\n",
      "back to text: Gosto do queijo e vinho.\n"
     ]
    }
   ],
   "source": [
    "# Check\n",
    "text = \"Gosto do queijo e vinho.\"\n",
    "tokens_ids = tokenizer_pt.encode(text)\n",
    "back_to_text = tokenizer_pt.decode(tokenizer_pt.encode(text))\n",
    "\n",
    "print('input text:', text)\n",
    "print('tokens ids:', tokens_ids)\n",
    "print('back to text:', back_to_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## English pre-trained tokenizer on a text in 3 languages (en, pt, fr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's tokenize by an English tokenizer a text in 3 different languages ​​(English, Portuguese, French) in order to observe the number of tokens required in each case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(en) Jacques-Germain Soufflot (Irancy, July 22, 1713 - Paris, August 29, 1780) was a French architect, initiator of the architectural style of Neoclassicism.\n",
      "\n",
      "(pt) Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo.\n",
      "\n",
      "(fr) Jacques-Germain Soufflot (Irancy, 22 juillet 1713 - Paris, 29 août 1780) était un architecte français, initiateur du style architectural du néoclassicisme.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# text in 3 languages to be tokenized\n",
    "text_en = 'Jacques-Germain Soufflot (Irancy, July 22, 1713 - Paris, August 29, 1780) was a French architect, initiator of the architectural style of Neoclassicism.'\n",
    "text_pt = 'Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo.'\n",
    "text_fr = 'Jacques-Germain Soufflot (Irancy, 22 juillet 1713 - Paris, 29 août 1780) était un architecte français, initiateur du style architectural du néoclassicisme.'\n",
    "\n",
    "langs = ['en', 'pt', 'fr']\n",
    "texts = [text_en,text_pt,text_fr]\n",
    "\n",
    "for lang,text in zip(*[langs,texts]):\n",
    "    print(f'({lang}) {TitledStr(text)}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(en - 22 tokens) ['Jacques-Germain', 'Soufflot', '(Irancy,', 'July', '22,', '1713', '-', 'Paris,', 'August', '29,', '1780)', 'was', 'a', 'French', 'architect,', 'initiator', 'of', 'the', 'architectural', 'style', 'of', 'Neoclassicism.']\n",
      "\n",
      "(pt - 25 tokens) ['Jacques-Germain', 'Soufflot', '(Irancy,', '22', 'de', 'julho', 'de', '1713', '—', 'Paris,', '29', 'de', 'agosto', 'de', '1780)', 'foi', 'um', 'arquitecto', 'francês,', 'iniciador', 'do', 'estilo', 'arquitectónico', 'do', 'Neoclassicismo.']\n",
      "\n",
      "(fr - 21 tokens) ['Jacques-Germain', 'Soufflot', '(Irancy,', '22', 'juillet', '1713', '-', 'Paris,', '29', 'août', '1780)', 'était', 'un', 'architecte', 'français,', 'initiateur', 'du', 'style', 'architectural', 'du', 'néoclassicisme.']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# number and list of classical tokens (ie, tokens separated by a blank)\n",
    "for lang,text in zip(*[langs,texts]):\n",
    "    print(f'({lang} - {len(text.split())} tokens) {TitledStr(text.split(\" \"))}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(en - 44 tokens) ['Jac', 'ques', '-', 'G', 'erm', 'ain', 'ĠSou', 'ff', 'lot', 'Ġ(', 'Iran', 'cy', ',', 'ĠJuly', 'Ġ22', ',', 'Ġ17', '13', 'Ġ-', 'ĠParis', ',', 'ĠAugust', 'Ġ29', ',', 'Ġ17', '80', ')', 'Ġwas', 'Ġa', 'ĠFrench', 'Ġarchitect', ',', 'Ġiniti', 'ator', 'Ġof', 'Ġthe', 'Ġarchitectural', 'Ġstyle', 'Ġof', 'ĠNe', 'oc', 'lass', 'icism', '.']\n",
      "\n",
      "(pt - 62 tokens) ['Jac', 'ques', '-', 'G', 'erm', 'ain', 'ĠSou', 'ff', 'lot', 'Ġ(', 'Iran', 'cy', ',', 'Ġ22', 'Ġde', 'Ġj', 'ul', 'ho', 'Ġde', 'Ġ17', '13', 'ĠâĢĶ', 'ĠParis', ',', 'Ġ29', 'Ġde', 'Ġag', 'ost', 'o', 'Ġde', 'Ġ17', '80', ')', 'Ġfo', 'i', 'Ġum', 'Ġar', 'qu', 'itect', 'o', 'Ġfranc', 'ê', 's', ',', 'Ġin', 'ici', 'ador', 'Ġdo', 'Ġest', 'ilo', 'Ġar', 'qu', 'itect', 'ón', 'ico', 'Ġdo', 'ĠNe', 'oc', 'lass', 'icism', 'o', '.']\n",
      "\n",
      "(fr - 53 tokens) ['Jac', 'ques', '-', 'G', 'erm', 'ain', 'ĠSou', 'ff', 'lot', 'Ġ(', 'Iran', 'cy', ',', 'Ġ22', 'Ġju', 'illet', 'Ġ17', '13', 'Ġ-', 'ĠParis', ',', 'Ġ29', 'Ġa', 'o', 'û', 't', 'Ġ17', '80', ')', 'Ġ', 'ét', 'ait', 'Ġun', 'Ġarchitect', 'e', 'Ġfr', 'an', 'ç', 'ais', ',', 'Ġiniti', 'ateur', 'Ġdu', 'Ġstyle', 'Ġarchitectural', 'Ġdu', 'Ġn', 'é', 'oc', 'lass', 'icism', 'e', '.']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# number and list of tokens \n",
    "# after the text tokenization by imported BPE GPT2TokenizerFast (trained with an English corpus...)\n",
    "for lang,text in zip(*[langs,texts]):\n",
    "    toks = tokenizer_en.tokenize(text)\n",
    "    print(f'({lang} - {len(toks)} tokens) {TitledStr(toks)}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(en - 44 tokens) [28821, 13281, 12, 38, 7780, 391, 22862, 487, 26487, 357, 23798, 948, 11, 2901, 2534, 11, 1596, 1485, 532, 6342, 11, 2932, 2808, 11, 1596, 1795, 8, 373, 257, 4141, 7068, 11, 5383, 1352, 286, 262, 27070, 3918, 286, 3169, 420, 31172, 11965, 13]\n",
      "\n",
      "(pt - 62 tokens) [28821, 13281, 12, 38, 7780, 391, 22862, 487, 26487, 357, 23798, 948, 11, 2534, 390, 474, 377, 8873, 390, 1596, 1485, 851, 6342, 11, 2808, 390, 556, 455, 78, 390, 1596, 1795, 8, 11511, 72, 23781, 610, 421, 5712, 78, 46754, 25792, 82, 11, 287, 44070, 7079, 466, 1556, 18526, 610, 421, 5712, 18840, 3713, 466, 3169, 420, 31172, 11965, 78, 13]\n",
      "\n",
      "(fr - 53 tokens) [28821, 13281, 12, 38, 7780, 391, 22862, 487, 26487, 357, 23798, 948, 11, 2534, 7544, 32512, 1596, 1485, 532, 6342, 11, 2808, 257, 78, 42324, 83, 1596, 1795, 8, 220, 25125, 4548, 555, 7068, 68, 1216, 272, 16175, 15152, 11, 5383, 15093, 7043, 3918, 27070, 7043, 299, 2634, 420, 31172, 11965, 68, 13]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# number and list of tokens ids\n",
    "# after the text tokenization + numerization by imported BPE GPT2TokenizerFast (trained with an English corpus...)\n",
    "for lang,text in zip(*[langs,texts]):\n",
    "    toks_ids = tokenizer_en.encode(text)\n",
    "    print(f'({lang} - {len(toks_ids)} tokens) {TitledStr(toks_ids)}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(en) Jacques-Germain Soufflot (Irancy, July 22, 1713 - Paris, August 29, 1780) was a French architect, initiator of the architectural style of Neoclassicism.\n",
      "\n",
      "(pt) Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo.\n",
      "\n",
      "(fr) Jacques-Germain Soufflot (Irancy, 22 juillet 1713 - Paris, 29 août 1780) était un architecte français, initiateur du style architectural du néoclassicisme.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# decode (back to the text)\n",
    "for lang,text in zip(*[langs,texts]):\n",
    "    toks_ids = tokenizer_en.encode(text)\n",
    "    text_decoded = tokenizer_en.decode(toks_ids)\n",
    "    print(f'({lang}) {TitledStr(text_decoded)}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd7wU5dn/8c9XlICIGhQNiBSDIYjAoYgxEluIP7E8mthixQaPiTWmiC2WFJPYSXiMNUFjDTYeHhsqxC5FUSDEjoogAqIiRQWv3x8z57gcTplzOHPWc/b7fr32tTv3zNxzzczutffeMzujiMDMzErHesUOwMzMGpcTv5lZiXHiNzMrMU78ZmYlxonfzKzEOPGbmZUYJ/4GIunvkn5bpGVL0t8kLZE0uQHqO0bSkw0RWz2X31VSSFq/CMu+QNI/cqj3bEnX51DvXyWd19D15q0hPy957bNqljVH0pBqxhUtB9RVs0386Q5aIKlNQdkJkiYVMay8DAZ+AHSKiEGVRxY7kRdTYyaFmkTE7yPihHWpo6r9GBEnRsRv1i26fJXy+++rqtkm/tT6wGnFDqKuJLWo4yxdgDkRsSyPeMyseWnuif8S4BeSNq08oqruBEmTJJ2Qvj5G0lOSrpD0oaQ3JH03LX9H0vuShlWqdnNJEyQtlfQvSV0K6v52Ou4DSS9LOqRg3N8lXS3pfknLgN2riLejpHHp/K9JGp6WHw9cD+wk6RNJF1aaryfw14LxH6blm0i6SdJCSW9JOldSle8HSZdIelLSJunwcZJmp11LD1Vaz5B0oqRX0/GjJSkd1z3dLh9JWiTpjir32peOkzRP0nxJP0/r+Iak5ZI2K1jmgHQ9NqgU917A2cCh6bq/WNO2rGK9N5B0m6S7JLWUtJ6kkZJel7RY0p2S2qXTlr+fhkl6O12/cwrqqvjlIekvaTzlj1WSLkjHlde/VNK/Jf2wlv24RveCpOHpOn2QrmPHLPuminW/QNI/Jf0jjWWGpG9JOit9778jac+C6TeRdEO6r96V9FtJLaqLO/V1Sf+X1v+cpG8W1PddSVPS98oUSd8tGNctfR8tlTQB2LyqdUin/bqk8en7Y0n6ulPB+EmSfqPks75U0sOSNi8Yf5SSz8fiwv1ZmwZY7tEFyz1PNXQx1UtENMsHMAcYAtwN/DYtOwGYlL7uCgSwfsE8k4AT0tfHAKuAY4EWwG+Bt4HRwNeAPYGlwEbp9H9Ph3dJx18FPJmOawO8k9a1PtAfWAT0Kpj3I2Bnki/jVlWsz7+A/wFaAWXAQuD7BbE+WcO2WGs8cBNwH9A23RavAMcXTp/Gch3wELBhOu4A4DWgZ7ou5wJPF9QbwHhgU6BzGude6bjbgHPK1xEYXE285fvmtnTb9U7rGZKOvx/4ScH0VwB/rqauC4B/1GFbXgD8A2gN/F+6b1qk404HngU6pfv4GuC2SjFfl87bF/gU6FldHGl5+fL7pcMHAx3TbXQosAzoUMN+/Dtfvr/3IHlf9U/j+zPweJZ9U812Wwn8v3Q/3wS8me6/DYDhwJsF09+bbo82wBbAZOC/a4n7A2BQWv8twO3puHbAEuCodNxh6fBm6fhngMvTddyF5HO31rZNp90MOBDYkOS9/k/g3kqf+deBb6X7bRLwh3TcdsAnfPmZvpwkJwypZlmF+6IhljsYaAlcCnxe3XLrlR8bOuF+VR58mfi3J0mq7al74n+1YFzvdPotC8oWA2UFO/32gnEbAauBrUk+wE9Uiu8a4PyCeW+qYV22TutqW1B2MfD36j5YleZfYzzJF9mnwHYFZf9dsG2OAZ4D7gDuAloWTPcA6RdEOrwesBzokg4HBQkduBMYmb6+CbiW5FhETfuufN98u6DsT8AN6etDgacK1uU9YFA1dV1AQVLIsC0vAMaRfDmMAlQw3WzSL4h0uAPJB3L9gpg7FYyfDPy4qjjSsvYk79Mf17AtpgP7V7efWTPZ3AD8qdJ78HOga237pprtNqFgeD+SZFT+Jdg2rW9TYMv0/dS6YPrDgIm1xH19wfDewH/S10cBkytN/0xaT2eS5NumYNytlbdtDduzDFhS6TN/bsHwT4EH09e/Zs3PdBvgMzIk/gZY7m0F4zasabn1eTT3rh4iYiZJK2dkPWZfUPB6RVpf5bKNCobfKVjuJyQtmo4kffA7Kuky+jD9uXsE8I2q5q1CR+CDiFhaUPYWsFUd1qXQ5iQtibdqqK87sD9wYUR8VlDeBbiqYD0+AFRp3vcKXi/ny230q3TayZJmSTquljgLt8lbJNsBkl8q20nahuSg9kcRkfVspizb8jtAH5IWWOFVDLsA9xSs+2ySL5EtC6apbt3XoKRbaixwa0TcXlB+tKTpBcvYnhq6MqpYt4p9mr4HF5Nt31Sl8nt9UUSsLhgmnb8Lya+A+QVxX0PS8q9JdbGssR6p8n3UkSSBLqs0rkqSNpR0Tdpt8jHwOLCp1jyOVlMchZ/pZSTbs1YNvNzlWZebVbNP/KnzSX6aFn4Ayt84GxaUFSbi+ti6/IWkjUh+ss4j2Yn/iohNCx4bRcRPCuYNqjcPaCepbUFZZ+DdjHFVrnsRSUuwSw31zSbpmnpAUo+C8ndIfsIXrkvriHi61iAi3ouI4RHRkeQXxv9I6l7DLFsXvO5Msh2IiJUkrdUjSFqHN9e02ErDWbblwyS/Ah6VVJjU3wGGVlr3VhGRdT8U+jNJF8W55QVKjpVcB5xM0q2xKTCT5MuyqnWpbB4F+1TJGW2bkf19Ul/vkLT4Ny/YLhtHRK90fG1xV7bGeqTK99F8kmMDbSqNq87PgR7AjhGxMUm3DXy5TWsynzU/0xuSbM8s1nW5hccDWtdhuZmUROKPiNdIui1OLShbSPJGOjI9CHUc8M1qqshqb0mDJbUEfgM8FxHvkPzi+FZ6oGiD9LFDeuArS/zvAE8DF0tqJakPcDxJv2gWC4BOaVykrbY7gd9JapsmnDNI+rYLl3sbycHRRwoOvP0VOEtSL6g4qHdwliAkHVxwgGsJSUJYXcMs56Utp14kX0KFB4NvIvnp/1+V465kAdBV6YHrrNsyIv5E0oXwaMFBt7+SbLMu6fq0l7R/Lau9Fkn/DewKHB4RXxSMakOyTRam0x1L0uIvXJeK/ViFW4FjJZVJ+hrwe5L34Jy6xlgXETGf5MvyMkkbKzkI/k1Ju2aMu7L7ST4vh0taX9KhJP3e4yPiLWAqcKGSA+6DSbqhqtOW5NfJh0oOxJ9fh1UbC+xb8Jm+iOw5c12Xu5+SA9wtgQvJ9oWRWUkk/tRFJB+sQsOBX5L8jOpFkhDWxa0kO/gDYABJi5S0W2FP4MckrZn3gD+SHDDK6jCSfuR5wD0kxwcmZJz3MWAW8J6kRWnZKSS/et4gOZB7K3Bj5RkjYgzJtntMUteIuCeN/fb0J+xMYGjGOHYAnpP0CUk/+mkR8WYN0/+L5EDyo8ClEfFwQVxPAV8Az9eS2P6ZPi+W9Hz6OtO2jOT8+HtJvvjakRywHwc8LGkpyYHeHWtc46odBmwDzNOXZ/acHRH/Bi4j6c9eQHJc6amC+araj4XxPgqcR3JcZj5JQ+bH9YivPo4m6T78N8mX+liSYyBQS9yVRcRiYF+SVvNiki7CfSOifN7DSbb7BySft5tqqO5KkoOni0j214NZVygiZgEnkXw25qfrNTfj7Ou63FOA29PlLgXeJ/lV1SC0ZhemWdMh6TGSPvIG/0es2VdF2m38IbBtLQ2lzEqpxW/NiKQdSE5brO2/AGZNjqT90m7ONiSnc84gOQOsQTjxW5MjaQzwCHB6pbNzzJqL/Um6IucB25Kc8ttg3TPu6jEzKzFu8ZuZlZhGv+xtfWy++ebRtWvXYodhZtakTJs2bVFEtK9c3iQSf9euXZk6dWqxwzAza1IkVfmvZnf1mJmVGCd+M7MS48RvZlZimkQff1U+//xz5s6dy8qVK4sdijVxrVq1olOnTmywwQa1T2zWDDTZxD937lzatm1L165dUdU3ETKrVUSwePFi5s6dS7du3YodjlmjaLJdPStXrmSzzTZz0rd1IonNNtvMvxytpDTZxA846VuD8PvISk2TTvxmZlZ3zSbxSw37aGgXXHABl156KQC//vWveeSRRwC48sorWb58ecV0EcEee+zBxx9/DCR/XpszZw677bbbWnWWl0+aNIljjjkGgPHjx3P++XW554OZlZome3C3KbvooosqXl955ZUceeSRbLhhcgfI+++/n759+7LxxhvXq+599tmH8847jzPPPLOiTsuPLmxa3URxvi/KaM2oxd/Yli1bxj777EPfvn3ZfvvtueOO5LLwXbt25cwzz2TQoEEMGjSI1157ba15jznmGMaOHcuoUaOYN28eu+++O7vvvjsAt9xyC/vv/+Xd/Nq3b0+LFi1o167dWvWUl7ds2ZJNNtkESPqrd9ttN8aPH5/HaptZM+DEX08PPvggHTt25MUXX2TmzJnstddeFeM23nhjJk+ezMknn8zpp59ebR2nnnoqHTt2ZOLEiUycOBGAp556igEDBlRMM2XKFLbeemvuvvvuteYvL//ud7/LVVddVVE+cOBAnnjiiYZYTTNrhnJN/JI2lTRW0n8kzZa0k6R2kiZIejV9/nqeMeSld+/ePPLII5x55pk88cQTFS1ugMMOO6zi+ZlnnqlTvR988AFt27Zdp9i22GIL5s2bt051mFnzlXeL/yrgwYj4NtAXmA2MBB6NiG1JbqI9MucYcvGtb32LadOm0bt3b84666w1+u0LTw+s66mC66+/Pl988cU6xbZy5Upat269TnWYWfOVW+KXtDGwC3ADQER8FhEfktxSbEw62RjggLxiyNO8efPYcMMNOfLII/nFL37B888/XzGuvL//jjvuYKeddqqxnrZt27J06Zd3D+zRowdvvPHGOsX2yiuvsP32269THWbWfOV5Vs82wELgb5L6AtOA04AtI2I+QETMl7RFVTNLGgGMAOjcuXOtC2vsO0jOmDGDX/7yl6y33npssMEGXH311RXjPv30U3bccUe++OILbrvtthrrGTFiBEOHDqVDhw5MnDiRffbZh0mTJtG9e/d6xzZx4kQuvvjies9vZs1bbvfclTQQeBbYOSKek3QV8DFwSkRsWjDdkoiosZ9/4MCBUflGLLNnz6Znz545RL5uym8as/nmm9dr/vnz53P00UczYcKEes2/YMECDj/8cB599NF6zV+q6vt+8umc9lUmaVpEDKxcnmcf/1xgbkQ8lw6PBfoDCyR1SIPqALyfYwxNTocOHRg+fHjFH7jq6u233+ayyy5r4KjMrDnJrasnIt6T9I6kHhHxMvB94N/pYxjwh/T5vrxiKIY5c+ascx2HHHJIvefdYYcd1nn5Zta85f3P3VOAWyS1BN4AjiX5lXGnpOOBt4GDc47BzMwK5Jr4I2I6sFb/Eknr38zMisD/3DUzKzFO/GZmJabZXJ2zoU+r82lvZtZcucW/DsrPmd9mm20YMGAAO+20E/fccw8AkyZNYpNNNqFfv3707NmTCy+8kIceeoiysjLKysrYaKON6NGjB2VlZXTo0KHK8qOPPrraZV9//fU1XgAuq9WrV/O9731vnesBGDx4cEXsZWVlFduiLm688Ubee++9aseffPLJPP300/WKb/r06Zxwwgn1mtesOWk2Lf7GFhEccMABDBs2jFtvvRWAt956i3HjxlVM873vfY/x48ezbNkyysrK2HfffZk+fToAu+22G5deeikDB6557Lu68ry0aNGiQa7kuXr1aiC5TEVZWVm967nxxhvp378/3/jGN9Yat3DhQl544QX+8pe/1KvusrIyXn/9dd5991222mqresdo1tS5xV9Pjz32GC1btuTEE0+sKOvSpQunnHLKWtO2adOGAQMG8Prrr9d5OStWrGDYsGH07t2b/v378/jjj681zbhx49h555354IMPWLBgAT/60Y8YOHAggwYN4tlnnwXg3HPP5fjjj2fXXXdlm222YfTo0QCsWrWKTTdN/kh9zjnnVLTWO3bsyPDhwwEYM2YMgwYNoqysjJ/+9Kd88cUXFfOde+65DBo0iMmTJ1e7Dvvttx8DBgygV69eXH/99RXLPeqoo+jduzfbb789o0aN4o477mD69OkceuihlJWV8dlnn61Rzz//+U+GDh1aMTxlyhR23XVXBgwYwNChQ1mwYAGQ/PIYOXIkgwYNokePHmv8Qth3330rrqVkVqqc+Otp1qxZ9O/fP9O0ixcv5tlnn6VXr151Xs6oUaNo2bIlM2bM4Oabb+aoo45aIyGOHTuWSy+9lPvvv5927dpx6qmn8qtf/YqpU6dy5513rtG18corrzBhwgSeffZZfv3rX1e00sv97ne/Y/r06UycOJF27dpx0kknMXPmTO655x6efvpppk+fzqpVq7j99tsB+Oijj+jfvz+TJ0+uuBhdedIuKyvjww8/BJIvjmnTpjFlyhQuv/xylixZwrRp01i0aBEzZsxg5syZHH300RXzln8BtGzZco34Cu9V8Omnn3Laaadx1113MW3aNI488kjOO++8imkjgsmTJ3PJJZesceVU36vAzF09Deakk07iySefpGXLlkyZMgWAJ554gn79+rHeeusxcuTIeiX+J598kl/+8pcA9OrVi44dO1bc1WvChAlMnjyZhx9+mI022giARx55hJdffrli/iVLlrBixQogae22bNmSLbbYgnbt2rFw4cK1rin0xRdfcPjhh3PmmWdSVlbGlVdeyZQpUyq6nlasWMHWW28NQMuWLfnhD3+4xvxVdfVcccUVFV1gc+fO5fXXX6d79+68/PLLnHbaaey9997sueeetW6L+fPn0759eyC5ts6sWbMYMmQIkHQ1derUqWLaH/3oRwAMGDBgjX9T+14FZk789darVy/uuuuuiuHRo0ezaNGiNfrmy/v410VNF9Hr3r07r732Gq+++ir9+vWrmH7y5MlrtZYBvva1r1W8btGiBatWrVprmvPOO49vfvObHHXUURX1HXfccfzmN79ZY7pVq1bRunXrWu838Mgjj/D444/z7LPP0rp1awYPHszKlSvZbLPNeOmll3jggQcYNWoUd911F9dee22NdbVu3ZqVK1dWxNWnT59qW+/l61p5PX2vArNmlPgb+/TLPfbYg7PPPpurr76an/zkJwAsX768wZezyy67cMstt7DLLrswe/Zs5s+fT/fu3Xn66afp1q0bF198MQcddBB33XUXPXv2ZMiQIYwePZqf/exnQHImS9aDrffeey+PP/44jz32WEXZkCFDOOiggzjttNPYfPPNWbx4McuWLaNjx46Z6vzoo49o164drVu3ZtasWRW/hhYuXEirVq04+OCD6datW8Wxksr3JyjUs2dPXnvtNQYPHsx2223Hu+++y+TJkxk0aBCfffYZr776aq2/qnyvAjP38debJO69917+9a9/0a1bNwYNGsSwYcP44x//2KDLOeWUU1ixYgW9e/fmiCOO4KabblqjNb/ddttx8803c+CBB/Lmm28yevRonnrqKfr06cN2223Hddddl3lZl112GXPnzmWHHXagrKyMiy66iN69e3P++eczZMgQ+vTpw5577llxEDWLffbZh+XLl9O3b18uuugidtxxRwDeeecddtllF8rKyhg+fDi///3vATj22GM54YQTqjy4W36vAkha9GPHjuWMM86gb9++9OvXj+eee47alN/zwKyU5XY9/obUlK7Hb/mJCAYPHswDDzzAxhtvXOf5V6xYwe67785TTz1FixYt1hjn6/Fbc1SM6/GbNShJXHrppbz99tv1mv/tt9/mT3/601pJ36zUNOk+/oio883MrWmr7R7GNenRowc9evRYq7wp/Oo1a0hNtsXfqlUrFi9e7A+trZOIYPHixbRq1arYoZg1mibb4u/UqRNz585l4cKFxQ7FmrhWrVqt8R8As+auySb+DTbYgG7duhU7DDOzJqfJdvWYmVn9NNkWv5lZfTSlU3DzOv3WLX4zsxLjxG9mVmKc+M3MSowTv5lZiXHiNzMrMU78ZmYlJtfTOSXNAZYCq4FVETFQUjvgDqArMAc4JCKW5BmHmZl9qTFa/LtHRFnBpUFHAo9GxLbAo+mwmZk1kmJ09ewPjElfjwEOKEIMZmYlK+/EH8DDkqZJGpGWbRkR8wHS5y2qmlHSCElTJU31hdjMzBpO3pds2Dki5knaApgg6T9ZZ4yIa4FrIbkDV14BmpmVmlxb/BExL31+H7gHGAQskNQBIH1+P88YzMxsTbklfkltJLUtfw3sCcwExgHD0smGAfflFYOZma0tz66eLYF70lsjrg/cGhEPSpoC3CnpeOBt4OAcYzAzs0pyS/wR8QbQt4ryxcD381qumZnVzP/cNTMrMU78ZmYlxonfzKzEOPGbmZUYJ34zsxLjxG9mVmKc+M3MSowTv5lZiXHiNzMrMU78ZmYlptbEL2nn9CJrSDpS0uWSuuQfmpmZ5SFLi/9qYLmkvsCvgLeAm3KNyszMcpMl8a+KiCC5ZeJVEXEV0DbfsMzMLC9Zrs65VNJZwJHALpJaABvkG5aZmeUlS+I/FDgcOD4i3pPUGbgk37BKly5UsUPILM73HTHNmqJaE39EvAdcXjD8Nu7jNzNrsrKc1fMjSa9K+kjSx5KWSvq4MYIzM7OGl6Wr50/AfhExO+9gzMwsf1nO6lngpG9m1nxkafFPlXQHcC/waXlhRNydW1RmZpabLIl/Y2A5sGdBWQBO/GZmTVCWs3qObYxAzMyscWQ5q+dbkh6VNDMd7iPp3PxDMzOzPGQ5uHsdcBbwOUBEvAT8OM+gzMwsP1kS/4YRMblS2aqsC5DUQtILksanw90kPZf+N+AOSS3rErCZma2bLIl/kaRvkhzQRdJBwPw6LOM0oPB00D8CV0TEtsAS4Pg61GVmZusoS+I/CbgG+Lakd4HTgZ9kqVxSJ2Af4Pp0WMAewNh0kjHAAXWM2czM1kGW0znfjYgh6c1Y1ouIpZLaZaz/SpJr+Jdfxnkz4MOIKO8qmgtsVdWMkkYAIwA6d+6ccXFmZlabLC3+uyWtHxHL0qT/DWBCbTNJ2hd4PyKmFRZXMWmVl3iMiGsjYmBEDGzfvn2GMM3MLIssLf57gbGSDgS2BsYBv8gw387Af0naG2hF8kewK4FN0y+SVUAnYF69Ijczs3qptcUfEdeRtPDvBf4XODEiHs4w31kR0SkiupKc/vlYRBwBTAQOSicbBtxXz9jNzKweqm3xSzqjcJCktT8d+I6k70TE5VXPWaszgdsl/RZ4AbihnvWYmVk91NTVU/m+uvdUU16riJgETEpfvwEMqmsdZmbWMKpN/BFxYeGwpLZJcXySe1RmZpabLNfq2V7SC8BMYJakaZJ65R+amZnlIcvpnNcCZ0REl4joAvyc5Po9ZmbWBGVJ/G0iYmL5QNpf3ya3iMzMLFdZzuN/Q9J5wM3p8JHAm/mFZGZmecrS4j8OaE9yx627gc2BY3KMyczMcpSlxT8kIk4tLJB0MPDPfEIyM7M8ZWnxn5WxzMzMmoCa/rk7FNgb2ErSqIJRG1OHG7GYmdlXS01dPfOAqcB/AYVX2FwK/CzPoMzMLD81/XP3ReBFSbdGxOeNGJOZmeUoy9U5nfTNzJqRLAd3zcysGak28Uu6OX0+rfHCMTOzvNXU4h8gqQtwnKSvS2pX+GisAM3MrGHVdFbPX4EHgW1IzuopvF9upOVmZtbEVNvij4hREdETuDEitomIbgUPJ30zsyaq1ks2RMRPJPUFvpcWPR4RL+UblpmZ5SXLjVhOBW4Btkgft0g6Je/AzMwsH1ku0nYCsGNELAOQ9EfgGeDPeQZmZmb5yHIev4DVBcOrWfNAr5mZNSFZWvx/A56TdE86fABwQ34hmZlZnrIc3L1c0iRgMElL/9iIeCHvwMzMLB9ZWvxExPPA8znHYmZmjSC3a/VIaiVpsqQXJc2SdGFa3k3Sc5JelXSHpJZ5xWBmZmvL8yJtnwJ7RERfoAzYS9J3gD8CV0TEtsAS4PgcYzAzs0pqTPySWkh6pD4VR+KTdHCD9BHAHsDYtHwMycFiMzNrJDUm/ohYDSyXtEl9Kk+/OKYD7wMTgNeBDyOi/NaNc4Gt6lO3mZnVT5aDuyuBGZImAMvKCyPi1NpmTL84yiRtCtwD9KxqsqrmlTQCGAHQuXPnDGGamVkWWRL//6WPeouID9NTQr8DbCpp/bTV34nk3r5VzXMtcC3AwIEDq/xyMDOzustyHv8YSa2BzhHxctaKJbUHPk+TfmtgCMmB3YnAQcDtwDDgvnpFbmZm9ZLlIm37AdNJrs2PpDJJ4zLU3QGYKOklYAowISLGA2cCZ0h6DdgM/wvYzKxRZenquQAYBEwCiIjpkrrVNlN66eZ+VZS/kdZnZmZFkOU8/lUR8VGlMve5m5k1UVla/DMlHQ60kLQtcCrwdL5hmZlZXrK0+E8BepH8E/c24GPg9DyDMjOz/GQ5q2c5cE56A5aIiKX5h2VmZnnJclbPDpJmAC+R/JHrRUkD8g/NzMzykKWP/wbgpxHxBICkwSQ3Z+mTZ2BmZpaPLH38S8uTPkBEPAm4u8fMrImqtsUvqX/6crKka0gO7AZwKOk5/WZm1vTU1NVzWaXh8wte+zx+M7MmqtrEHxG7N2YgZmbWOGo9uJteUvlooGvh9Fkuy2xmZl89Wc7quR94FpgBfJFvOGYgFTuCOrig2AGY1V2WxN8qIs7IPRIzM2sUWU7nvFnScEkdJLUrf+QemZmZ5SJLi/8z4BLgHL48myeAbfIKyszM8pMl8Z8BdI+IRXkHY2Zm+cvS1TMLWJ53IGZm1jiytPhXA9MlTSS5NDPg0znNzJqqLIn/3vRhZmbNQJbr8Y9pjEDMzKxxZPnn7ptUcW2eiPBZPWZmTVCWrp6BBa9bAQcDTeY8/ib1L1DwP0HNLHe1ntUTEYsLHu9GxJXAHo0Qm5mZ5SBLV0//gsH1SH4BtM0tIjMzy1WWrp7C6/KvAuYAh+QSjZmZ5S7LWT31ui6/pK2Bm4BvkFzV89qIuCq9zs8dJJd5ngMcEhFL6rMMMys+H0drerJ09XwNOJC1r8d/US2zrgJ+HhHPSw+mQxcAAAgSSURBVGoLTJM0ATgGeDQi/iBpJDASOLN+4ZuZWV1luWTDfcD+JIl8WcGjRhExPyKeT18vBWYDW6V1lf83YAxwQN3DNjOz+srSx98pIvZal4VI6gr0A54DtoyI+ZB8OUjaopp5RgAjADp37rwuizczswJZWvxPS+pd3wVI2gi4Czg9Ij7OOl9EXBsRAyNiYPv27eu7eDMzqyRLi38wcEz6D95PAQEREX1qm1HSBiRJ/5aIuDstXiCpQ9ra7wC8X8/YzcysHrIk/qH1qViSgBuA2RFxecGoccAw4A/p8331qd/MzOony+mcb9Wz7p2Bo4AZkqanZWeTJPw7JR0PvE1yCQgzM2skWVr89RIRT5J0C1Xl+3kt18zMapbl4K6ZmTUjTvxmZiXGid/MrMQ48ZuZlRgnfjOzEuPEb2ZWYpz4zcxKjBO/mVmJceI3MysxTvxmZiXGid/MrMQ48ZuZlRgnfjOzEuPEb2ZWYpz4zcxKjBO/mVmJceI3MysxTvxmZiXGid/MrMQ48ZuZlRgnfjOzEuPEb2ZWYpz4zcxKjBO/mVmJyS3xS7pR0vuSZhaUtZM0QdKr6fPX81q+mZlVLc8W/9+BvSqVjQQejYhtgUfTYTMza0S5Jf6IeBz4oFLx/sCY9PUY4IC8lm9mZlVr7D7+LSNiPkD6vEV1E0oaIWmqpKkLFy5stADNzJq7r+zB3Yi4NiIGRsTA9u3bFzscM7Nmo7ET/wJJHQDS5/cbeflmZiWvsRP/OGBY+noYcF8jL9/MrOTleTrnbcAzQA9JcyUdD/wB+IGkV4EfpMNmZtaI1s+r4og4rJpR389rmWZmVruv7MFdMzPLhxO/mVmJceI3MysxTvxmZiXGid/MrMQ48ZuZlRgnfjOzEuPEb2ZWYpz4zcxKjBO/mVmJceI3MysxTvxmZiXGid/MrMQ48ZuZlRgnfjOzEuPEb2ZWYpz4zcxKjBO/mVmJceI3MysxTvxmZiXGid/MrMQ48ZuZlRgnfjOzEuPEb2ZWYpz4zcxKTFESv6S9JL0s6TVJI4sRg5lZqWr0xC+pBTAaGApsBxwmabvGjsPMrFQVo8U/CHgtIt6IiM+A24H9ixCHmVlJUkQ07gKlg4C9IuKEdPgoYMeIOLnSdCOAEelgD+DlRg20eDYHFhU7CMvM+6vpKaV91iUi2lcuXL8IgaiKsrW+fSLiWuDa/MP5apE0NSIGFjsOy8b7q+nxPitOV89cYOuC4U7AvCLEYWZWkoqR+KcA20rqJqkl8GNgXBHiMDMrSY3e1RMRqySdDDwEtABujIhZjR3HV1jJdW81cd5fTU/J77NGP7hrZmbF5X/umpmVGCd+M7MS48Rv1gAknS5pw2LHYVWTdKqk2ZJuKXYsXwXu4zdrAJLmAAMjolT+GNSkSPoPMDQi3iwoWz8iVhUxrKIpxh+4LCXpSOBUoCXwHPBT4CPgKmBfYAWwf0QsKFqQtgZJXYEHSfZXP+AV4HGgIzBR0qKI2L1oAdpaJP0V2AYYJ6kzcAfQleTfu4cXMbSicVdPkUjqCRwK7BwRZcBq4AigDfBsRPQlSSjDixelVaMHcG1E9AE+Jvningfs7qT/1RMRJ5LuH+AKYABJg6okkz64xV9M3yd5A06RBNAaeB/4DBifTjMN+EFRorOavBMRT6Wv/0Hyq82ajnERsaLYQRSTE3/xCBgTEWetUSj9Ir488LIa76OvosoHxnygrGlZVuwAis1dPcXzKHCQpC0AJLWT1KXIMVk2nSXtlL4+DHgSWAq0LV5IZtm5NVkkEfFvSecCD0taD/gcOKnIYVk2s4Fhkq4BXgWuJumie0DSfPfz21edT+c0q4P0rJ7xEbF9kUMxqzd39ZiZlRi3+M3MSoxb/GZmJcaJ38ysxDjxm5mVGCd+a7YkfVLsGMy+ipz4zcxKjBO/NXuSNpL0qKTnJc2QtH9a3jW9Rvt1kmZJelhS63TcDpJekvSMpEskzUzLj5H0l4K6x0vaLX19taSpaV0XFkyzt6T/SHpS0ihJ49PyNpJulDRF0gsFcfWSNFnS9DSGbRttY1lJcOK3UrAS+GFE9Ce5QuNlSq+MB2wLjI6IXsCHwIFp+d+AEyNiJ5JrJmVxTkQMBPoAu0rqI6kVcA3JteAHA+0Lpwcei4gd0rgukdQGOBG4Kr1q60Bgbv1W26xqTvxWCgT8XtJLwCPAVsCW6bg3I2J6+noa0FXSpkDbiHg6Lb8143IOkfQ88ALQC9gO+DbwRsENQG4rmH5PYKSk6cAkoBXQGXgGOFvSmUCXUr+SpDU8X6vHSsERJC3tARHxeXq3rFbpuE8LpltNcnlsUb1VrNlgagUgqRvwC2CHiFgi6e/puJrqEnBgRLxcqXy2pOeAfYCHJJ0QEY/VUI9ZnbjFb6VgE+D9NOnvDtR4FdSIWAIslfSdtOjHBaPnAGWS1pO0NTAoLd+Y5HK/H0naEhialv8H2Ca9xg8kN98p9xBwSnm3k6R+6fM2JL8SRgHjSLqOzBqMW/xWCm4B/lfSVGA6STKuzfHAdZKWkXTDfJSWPwW8CcwAZgLPA0TEi5JeAGYBb6TTERErJP0UeFDSImBywTJ+A1wJvJQm/zkkt9w8FDhS0ufAe8BF9Vtts6r5Wj1mVZC0UUR8kr4eCXSIiNPWpa40uY8GXo2IKxowXLM6cVePWdX2SU+nnAl8D/jtOtQ1PD2AO4uk2+mahgjQrL7c4jczKzFu8ZuZlRgnfjOzEuPEb2ZWYpz4zcxKjBO/mVmJ+f+ugb7h+P4FcgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# graph \"Number of tokens by tokenization method and lang\"\n",
    "# source: https://matplotlib.org/3.2.1/gallery/lines_bars_and_markers/barchart.html#sphx-glr-gallery-lines-bars-and-markers-barchart-py\n",
    "text_split = list()\n",
    "toks_split = list()\n",
    "\n",
    "for text in texts:\n",
    "    text_split.append(len(text.split()))\n",
    "    toks_ids = tokenizer_en.encode(text)\n",
    "    toks_split.append(len(toks_ids))\n",
    "    \n",
    "labels = langs\n",
    "xy = list(np.array([1.,2.,3.]) - 0.2)\n",
    "xz = list(np.array([1.,2.,3.]) + 0.2)\n",
    "y = text_split\n",
    "z = toks_split\n",
    "\n",
    "ax = plt.subplot(111)\n",
    "ax.bar(xy, y, width=0.4, color='b', align='center')\n",
    "ax.bar(xz, z, width=0.4, color='g', align='center')\n",
    "\n",
    "ax.set_xlabel('languages')\n",
    "ax.set_xticks(range(1,len(labels)+1))\n",
    "ax.set_xticklabels(labels)\n",
    "ax.set_ylabel('number of tokens')\n",
    "ax.legend(['split(\" \")', 'GPTTokenizerFast (en)'])\n",
    "\n",
    "ax.set_title('Number of tokens by tokenization method and lang')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, **even if a GPT2TokenizerFast trained with an English corpus can tokenize any text in any language, it was optimized for English**: the number of generated tokens is lower for an English text than for the same text in another language with an equivalent number of words.\n",
    "\n",
    "In our example, \n",
    "- English: 44 tokens (22 words)\n",
    "- French: 20% more tokens with 53 (21 words)\n",
    "- Portuguese: 40% more tokens with 62 (25 words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## English vs Portuguese tokenizer on Portuguese Wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this paragraph, we will compare the number of tokens per Portuguese text generated respectively by the English and Portuguese tokenizers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fotografia (do grego φως [\"fós\"] (\"luz\"), e γραφις [\"grafis\"] (\"estilo\", \"pincel\") ou γραφη \"grafê\", e significa \"desenhar com luz e contraste\"), por definição, é essencialmente a \"técnica de criação\" de imagens por meio de exposição luminosa, fixando-as em uma superfície sensível. A primeira fotografia reconhecida remonta ao ano de 1826 e é atribuída ao francês Joseph Nicéphore Niépce. Contudo, a invenção da fotografia não é obra de um só autor, mas um processo de acúmulo de avanços por parte de muitas pessoas, trabalhando, juntas ou em paralelo, ao longo de muitos anos. Se por um lado os...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Espadanedo é uma antiga freguesia portuguesa do concelho de Macedo de Cavaleiros, com 17,90 km² de área e 188 habitantes (2011). A sua densidade populacional era 10,5 hab/km².\\nFoi extinta (agregada) pela reorganização administrativa de 2012/2013, sendo o seu território integrado na União de Freguesias de Espadanedo, Edroso, Murçós e Soutelo Mourisco.\\n\\nA antiga freguesia de S. Miguel de Espadanedo e Valongo pertenceu ao antigo concelho de Torre D. Chama, extinto a 24 de Outubro de 1855, em 1839 aparece agregada à comarca de Bragança e em 1852 à de Mirandela. Passa a pertencer definitivam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo. O seu trabalho mais conhecido é, sem dúvida, o Panthéon (Panteão) de Paris, construído a partir de 1755, inicialmente uma igreja dedicada a Santa Genoveva.\\n\\nSoufflot nasceu em Irancy, perto de Auxerre, na França. Com 18 anos, entrou para a Academia Francesa de Roma, onde os jovens estudantes da década de 1750 se tornariam na primeira geração de criativos pleanamente neoclássicos. Ficará na Itália de 1731 a 1738. Quando volto...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A Faculdade de Medicina da Universidade de São Paulo (FMUSP) é uma escola médica da Universidade de São Paulo. Foi fundada em 1912 com o nome de \"Faculdade de Medicina e Cirurgia de São Paulo\" por Arnaldo Vieira de Carvalho (1867-1920), médico formado em 1888 pela Faculdade de Medicina do Rio de Janeiro. Em homenagem ao ilustre fundador, a Faculdade é, ainda hoje, chamada de a \"Casa de Arnaldo\" por seus alunos e ex-alunos. Em 1925 teve seu nome alterado para \"Faculdade de Medicina de São Paulo\" e em 1934, foi incorporada à recém-criada Universidade de São Paulo, passando a ter a atual desi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A Escola do Teatro Bolshoi no Brasil é uma tradicional escola de balé existente na cidade de Joinville, no estado de Santa Catarina. Fundada em 2000, é a única filial do Teatro Bolshoi de Moscou e possui alunos de vários estados brasileiros. Tem como missão formar artistas cidadãos, promover e difundir a arte-educação.\\n\\nA instituição foi fundada em 15 de março de 2000, é a única filial do Teatro Bolshoi. \\n\\nUm orgulho para o Brasil e para Joinville, cidade sede. A Escola do Teatro Bolshoi no Brasil, com professores russos e brasileiros, forma bailarinos com a mesma precisão, técnica e q...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      text\n",
       "0  Fotografia (do grego φως [\"fós\"] (\"luz\"), e γραφις [\"grafis\"] (\"estilo\", \"pincel\") ou γραφη \"grafê\", e significa \"desenhar com luz e contraste\"), por definição, é essencialmente a \"técnica de criação\" de imagens por meio de exposição luminosa, fixando-as em uma superfície sensível. A primeira fotografia reconhecida remonta ao ano de 1826 e é atribuída ao francês Joseph Nicéphore Niépce. Contudo, a invenção da fotografia não é obra de um só autor, mas um processo de acúmulo de avanços por parte de muitas pessoas, trabalhando, juntas ou em paralelo, ao longo de muitos anos. Se por um lado os...\n",
       "1  Espadanedo é uma antiga freguesia portuguesa do concelho de Macedo de Cavaleiros, com 17,90 km² de área e 188 habitantes (2011). A sua densidade populacional era 10,5 hab/km².\\nFoi extinta (agregada) pela reorganização administrativa de 2012/2013, sendo o seu território integrado na União de Freguesias de Espadanedo, Edroso, Murçós e Soutelo Mourisco.\\n\\nA antiga freguesia de S. Miguel de Espadanedo e Valongo pertenceu ao antigo concelho de Torre D. Chama, extinto a 24 de Outubro de 1855, em 1839 aparece agregada à comarca de Bragança e em 1852 à de Mirandela. Passa a pertencer definitivam...\n",
       "2  Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo. O seu trabalho mais conhecido é, sem dúvida, o Panthéon (Panteão) de Paris, construído a partir de 1755, inicialmente uma igreja dedicada a Santa Genoveva.\\n\\nSoufflot nasceu em Irancy, perto de Auxerre, na França. Com 18 anos, entrou para a Academia Francesa de Roma, onde os jovens estudantes da década de 1750 se tornariam na primeira geração de criativos pleanamente neoclássicos. Ficará na Itália de 1731 a 1738. Quando volto...\n",
       "3  A Faculdade de Medicina da Universidade de São Paulo (FMUSP) é uma escola médica da Universidade de São Paulo. Foi fundada em 1912 com o nome de \"Faculdade de Medicina e Cirurgia de São Paulo\" por Arnaldo Vieira de Carvalho (1867-1920), médico formado em 1888 pela Faculdade de Medicina do Rio de Janeiro. Em homenagem ao ilustre fundador, a Faculdade é, ainda hoje, chamada de a \"Casa de Arnaldo\" por seus alunos e ex-alunos. Em 1925 teve seu nome alterado para \"Faculdade de Medicina de São Paulo\" e em 1934, foi incorporada à recém-criada Universidade de São Paulo, passando a ter a atual desi...\n",
       "4  A Escola do Teatro Bolshoi no Brasil é uma tradicional escola de balé existente na cidade de Joinville, no estado de Santa Catarina. Fundada em 2000, é a única filial do Teatro Bolshoi de Moscou e possui alunos de vários estados brasileiros. Tem como missão formar artistas cidadãos, promover e difundir a arte-educação.\\n\\nA instituição foi fundada em 15 de março de 2000, é a única filial do Teatro Bolshoi. \\n\\nUm orgulho para o Brasil e para Joinville, cidade sede. A Escola do Teatro Bolshoi no Brasil, com professores russos e brasileiros, forma bailarinos com a mesma precisão, técnica e q..."
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lang = 'pt'\n",
    "fname = f'all_texts_{lang}wiki.csv'\n",
    "df = pd.read_csv(path_data/fname)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11h 46min 52s, sys: 2h 52min 55s, total: 14h 39min 48s\n",
      "Wall time: 1h 32min 56s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df2 = df.copy()\n",
    "\n",
    "tokens_en_list = list()\n",
    "num_token_by_word_en_list = list()\n",
    "tokens_pt_list = list()\n",
    "num_token_by_word_pt_list = list()\n",
    "\n",
    "for index, row in df2.iterrows():\n",
    "    text = row['text']\n",
    "    \n",
    "    tokens_en = tokenizer_en.encode(text)\n",
    "    tokens_pt = tokenizer_pt.encode(text)\n",
    "    \n",
    "    tokens_en_list.append(tokens_en)\n",
    "    tokens_pt_list.append(tokens_pt)\n",
    "    \n",
    "    length_text = len(text.split())\n",
    "    tokens_by_word_en = len(tokens_en)/length_text\n",
    "    tokens_by_word_pt = len(tokens_pt)/length_text\n",
    "    \n",
    "    num_token_by_word_en_list.append(tokens_by_word_en)\n",
    "    num_token_by_word_pt_list.append(tokens_by_word_pt)\n",
    "    \n",
    "df2['tokens_en'] = tokens_en_list\n",
    "df2['num_token_by_word_en'] = num_token_by_word_en_list\n",
    "df2['tokens_pt'] = tokens_pt_list\n",
    "df2['num_token_by_word_pt'] = num_token_by_word_pt_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>tokens_en</th>\n",
       "      <th>num_token_by_word_en</th>\n",
       "      <th>tokens_pt</th>\n",
       "      <th>num_token_by_word_pt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fotografia (do grego φως [\"fós\"] (\"luz\"), e γραφις [\"grafis\"] (\"estilo\", \"pincel\") ou γραφη \"grafê\", e significa \"desenhar com luz e contraste\"), por definição, é essencialmente a \"técnica de criação\" de imagens por meio de exposição luminosa, fixando-as em uma superfície sensível. A primeira fotografia reconhecida remonta ao ano de 1826 e é atribuída ao francês Joseph Nicéphore Niépce. Contudo, a invenção da fotografia não é obra de um só autor, mas um processo de acúmulo de avanços por parte de muitas pessoas, trabalhando, juntas ou em paralelo, ao longo de muitos anos. Se por um lado os...</td>\n",
       "      <td>[37, 313, 519, 32188, 544, 357, 4598, 10536, 2188, 18074, 228, 49535, 35558, 14631, 69, 10205, 82, 8973, 5855, 2290, 89, 12340, 304, 7377, 111, 33643, 17394, 139, 228, 29945, 35558, 14631, 70, 32188, 271, 8973, 5855, 395, 18526, 1600, 366, 79, 924, 75, 4943, 267, 84, 7377, 111, 33643, 17394, 139, 228, 138, 115, 366, 70, 32188, 25792, 1600, 304, 2216, 64, 366, 8906, 268, 9869, 401, 300, 10277, 304, 3445, 4594, 12340, 16964, 2730, 72, 16175, 28749, 11, 38251, 3209, 268, 2413, 434, 68, 257, 366, 83, 2634, 31522, 3970, 390, 269, 7496, 16175, 28749, 1, 390, 3590, ...]</td>\n",
       "      <td>2.369519</td>\n",
       "      <td>[9493, 22419, 363, 360, 5712, 46504, 30015, 16691, 33769, 70, 592, 32642, 1293, 21146, 2301, 258, 6970, 112, 17273, 12320, 46645, 17472, 16691, 33769, 4700, 274, 32642, 1293, 38371, 474, 330, 6701, 1996, 1766, 414, 6970, 112, 17273, 12320, 46645, 29825, 330, 4700, 369, 474, 258, 1952, 330, 1202, 1308, 290, 297, 3551, 258, 9592, 2301, 358, 5758, 12, 372, 9420, 259, 330, 38735, 260, 2271, 2, 260, 4318, 358, 1513, 260, 5214, 36162, 12, 34567, 13, 267, 300, 349, 4144, 17706, 14, 315, 806, 9659, 8892, 13110, 443, 750, 260, 22051, 258, 372, 11679, 443, 3081, 7358, 5227, 293, ...]</td>\n",
       "      <td>1.295616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Espadanedo é uma antiga freguesia portuguesa do concelho de Macedo de Cavaleiros, com 17,90 km² de área e 188 habitantes (2011). A sua densidade populacional era 10,5 hab/km².\\nFoi extinta (agregada) pela reorganização administrativa de 2012/2013, sendo o seu território integrado na União de Freguesias de Espadanedo, Edroso, Murçós e Soutelo Mourisco.\\n\\nA antiga freguesia de S. Miguel de Espadanedo e Valongo pertenceu ao antigo concelho de Torre D. Chama, extinto a 24 de Outubro de 1855, em 1839 aparece agregada à comarca de Bragança e em 1852 à de Mirandela. Passa a pertencer definitivam...</td>\n",
       "      <td>[36, 2777, 324, 22739, 78, 38251, 334, 2611, 1885, 13827, 2030, 70, 947, 544, 2493, 1018, 947, 64, 466, 369, 5276, 8873, 390, 25942, 78, 390, 19931, 1000, 72, 4951, 11, 401, 1596, 11, 3829, 1849, 13276, 31185, 390, 6184, 94, 21468, 304, 27778, 7947, 39781, 357, 9804, 737, 317, 424, 64, 29509, 312, 671, 16595, 330, 1538, 6980, 838, 11, 20, 387, 65, 14, 13276, 31185, 13, 198, 37, 23013, 1070, 600, 64, 357, 363, 2301, 4763, 8, 279, 10304, 35459, 23638, 16175, 28749, 6863, 265, 12151, 390, 2321, 14, 6390, 11, 3758, 78, 267, 384, 84, 5771, 10205, ...]</td>\n",
       "      <td>2.392857</td>\n",
       "      <td>[1438, 15237, 1388, 360, 372, 349, 2661, 4766, 3764, 298, 5975, 260, 16023, 260, 19537, 12, 297, 1115, 12, 5699, 638, 2107, 4490, 260, 1430, 258, 2731, 2668, 363, 9381, 609, 315, 450, 7173, 9342, 621, 1194, 12, 21, 8770, 15, 2107, 4490, 14, 199, 2704, 10602, 363, 599, 45513, 9, 534, 19571, 8497, 260, 1762, 15, 11825, 12, 790, 275, 467, 2521, 11558, 347, 2743, 260, 40439, 260, 3158, 271, 1388, 360, 12, 2088, 6129, 12, 6834, 286, 592, 258, 2582, 3026, 6713, 1141, 423, 14, 199, 199, 33, 2661, 4766, 260, 327, 14, 4379, 260, 3158, 271, 1388, ...]</td>\n",
       "      <td>1.486264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo. O seu trabalho mais conhecido é, sem dúvida, o Panthéon (Panteão) de Paris, construído a partir de 1755, inicialmente uma igreja dedicada a Santa Genoveva.\\n\\nSoufflot nasceu em Irancy, perto de Auxerre, na França. Com 18 anos, entrou para a Academia Francesa de Roma, onde os jovens estudantes da década de 1750 se tornariam na primeira geração de criativos pleanamente neoclássicos. Ficará na Itália de 1731 a 1738. Quando volto...</td>\n",
       "      <td>[28821, 13281, 12, 38, 7780, 391, 22862, 487, 26487, 357, 23798, 948, 11, 2534, 390, 474, 377, 8873, 390, 1596, 1485, 851, 6342, 11, 2808, 390, 556, 455, 78, 390, 1596, 1795, 8, 11511, 72, 23781, 610, 421, 5712, 78, 46754, 25792, 82, 11, 287, 44070, 7079, 466, 1556, 18526, 610, 421, 5712, 18840, 3713, 466, 3169, 420, 31172, 11965, 78, 13, 440, 384, 84, 491, 44349, 8873, 285, 15152, 369, 258, 66, 17305, 38251, 11, 5026, 288, 21356, 85, 3755, 11, 267, 5961, 400, 2634, 261, 357, 47, 12427, 28749, 8, 390, 6342, 11, 1500, 622, 8836, 4598, 257, ...]</td>\n",
       "      <td>2.268409</td>\n",
       "      <td>[33134, 13, 35969, 2582, 1738, 26035, 363, 41, 791, 89, 12, 2331, 260, 1637, 260, 45571, 2047, 2807, 12, 2820, 260, 1653, 260, 28277, 9, 379, 342, 17859, 3081, 12, 2333, 629, 298, 2266, 44381, 298, 1247, 5171, 41486, 14, 384, 467, 1313, 447, 1507, 372, 12, 944, 10436, 12, 275, 14982, 37648, 308, 363, 4042, 301, 280, 9, 260, 2807, 12, 4217, 259, 1129, 260, 26110, 12, 3470, 349, 2608, 8761, 259, 1925, 404, 19432, 373, 14, 199, 199, 15533, 1738, 26035, 2765, 300, 419, 791, 89, 12, 3067, 260, 21440, 269, 266, 12, 347, 2020, 14, 991, 664, ...]</td>\n",
       "      <td>1.418052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A Faculdade de Medicina da Universidade de São Paulo (FMUSP) é uma escola médica da Universidade de São Paulo. Foi fundada em 1912 com o nome de \"Faculdade de Medicina e Cirurgia de São Paulo\" por Arnaldo Vieira de Carvalho (1867-1920), médico formado em 1888 pela Faculdade de Medicina do Rio de Janeiro. Em homenagem ao ilustre fundador, a Faculdade é, ainda hoje, chamada de a \"Casa de Arnaldo\" por seus alunos e ex-alunos. Em 1925 teve seu nome alterado para \"Faculdade de Medicina de São Paulo\" e em 1934, foi incorporada à recém-criada Universidade de São Paulo, passando a ter a atual desi...</td>\n",
       "      <td>[32, 13585, 32926, 671, 390, 5786, 1437, 12379, 26986, 312, 671, 390, 311, 28749, 34410, 357, 23264, 2937, 47, 8, 38251, 334, 2611, 3671, 5708, 285, 2634, 67, 3970, 12379, 26986, 312, 671, 390, 311, 28749, 34410, 13, 19434, 72, 1814, 4763, 795, 34463, 401, 267, 299, 462, 390, 366, 47522, 32926, 671, 390, 5786, 1437, 304, 21239, 3686, 544, 390, 311, 28749, 34410, 1, 16964, 16644, 41476, 47154, 8704, 390, 1879, 2100, 8873, 357, 1507, 3134, 12, 40454, 828, 285, 2634, 67, 3713, 1296, 4533, 795, 49584, 279, 10304, 13585, 32926, 671, 390, 5786, 1437, 466, 15338, 390, 42799, ...]</td>\n",
       "      <td>2.244809</td>\n",
       "      <td>[33, 4687, 260, 8138, 305, 1627, 260, 862, 1140, 363, 20640, 21090, 9, 372, 349, 2198, 9867, 305, 1627, 260, 862, 1140, 14, 1376, 4330, 300, 8928, 297, 275, 781, 260, 330, 42326, 260, 8138, 258, 33944, 260, 862, 1140, 2, 358, 22872, 9094, 260, 6171, 363, 1692, 8386, 13, 29477, 521, 4805, 4461, 300, 13789, 534, 4687, 260, 8138, 298, 1124, 260, 1588, 14, 575, 3904, 443, 26215, 5687, 12, 259, 4687, 372, 12, 872, 1913, 12, 2136, 260, 259, 330, 14238, 260, 22872, 2, 358, 660, 4433, 258, 417, 13, 43556, 14, 575, 9703, 1122, 467, 781, 12677, ...]</td>\n",
       "      <td>1.253501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A Escola do Teatro Bolshoi no Brasil é uma tradicional escola de balé existente na cidade de Joinville, no estado de Santa Catarina. Fundada em 2000, é a única filial do Teatro Bolshoi de Moscou e possui alunos de vários estados brasileiros. Tem como missão formar artistas cidadãos, promover e difundir a arte-educação.\\n\\nA instituição foi fundada em 15 de março de 2000, é a única filial do Teatro Bolshoi. \\n\\nUm orgulho para o Brasil e para Joinville, cidade sede. A Escola do Teatro Bolshoi no Brasil, com professores russos e brasileiros, forma bailarinos com a mesma precisão, técnica e q...</td>\n",
       "      <td>[32, 16319, 5708, 466, 1665, 47756, 10797, 1477, 23013, 645, 39452, 346, 38251, 334, 2611, 2083, 291, 1538, 3671, 5708, 390, 3652, 2634, 2152, 21872, 12385, 269, 312, 671, 390, 15251, 4244, 11, 645, 1556, 4533, 390, 8909, 5181, 283, 1437, 13, 7557, 4763, 795, 4751, 11, 38251, 257, 6184, 118, 77, 3970, 1226, 498, 466, 1665, 47756, 10797, 1477, 23013, 390, 5826, 66, 280, 304, 1184, 9019, 435, 403, 418, 390, 410, 6557, 380, 418, 1556, 22484, 48029, 576, 72, 4951, 13, 5825, 401, 78, 2051, 28749, 1296, 283, 6802, 292, 269, 32482, 26102, 418, 11, 1552, 2502, 304, ...]</td>\n",
       "      <td>2.442777</td>\n",
       "      <td>[33, 3183, 298, 4948, 15934, 385, 73, 325, 889, 372, 349, 3898, 2198, 260, 26728, 8050, 347, 786, 260, 23696, 12, 325, 1282, 260, 1925, 5337, 14, 32791, 300, 2424, 12, 372, 259, 2542, 19606, 298, 4948, 15934, 385, 73, 260, 9001, 258, 1893, 4433, 260, 1292, 3606, 4779, 14, 2904, 390, 3829, 4964, 3098, 6267, 12, 5746, 258, 29331, 259, 2432, 13, 45422, 14, 199, 199, 33, 4493, 379, 4330, 300, 1079, 260, 1563, 260, 2424, 12, 372, 259, 2542, 19606, 298, 4948, 15934, 385, 73, 14, 722, 199, 2635, 18621, 341, 275, 889, 258, 341, 23696, 12, 786, ...]</td>\n",
       "      <td>1.358349</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      text  \\\n",
       "0  Fotografia (do grego φως [\"fós\"] (\"luz\"), e γραφις [\"grafis\"] (\"estilo\", \"pincel\") ou γραφη \"grafê\", e significa \"desenhar com luz e contraste\"), por definição, é essencialmente a \"técnica de criação\" de imagens por meio de exposição luminosa, fixando-as em uma superfície sensível. A primeira fotografia reconhecida remonta ao ano de 1826 e é atribuída ao francês Joseph Nicéphore Niépce. Contudo, a invenção da fotografia não é obra de um só autor, mas um processo de acúmulo de avanços por parte de muitas pessoas, trabalhando, juntas ou em paralelo, ao longo de muitos anos. Se por um lado os...   \n",
       "1  Espadanedo é uma antiga freguesia portuguesa do concelho de Macedo de Cavaleiros, com 17,90 km² de área e 188 habitantes (2011). A sua densidade populacional era 10,5 hab/km².\\nFoi extinta (agregada) pela reorganização administrativa de 2012/2013, sendo o seu território integrado na União de Freguesias de Espadanedo, Edroso, Murçós e Soutelo Mourisco.\\n\\nA antiga freguesia de S. Miguel de Espadanedo e Valongo pertenceu ao antigo concelho de Torre D. Chama, extinto a 24 de Outubro de 1855, em 1839 aparece agregada à comarca de Bragança e em 1852 à de Mirandela. Passa a pertencer definitivam...   \n",
       "2  Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo. O seu trabalho mais conhecido é, sem dúvida, o Panthéon (Panteão) de Paris, construído a partir de 1755, inicialmente uma igreja dedicada a Santa Genoveva.\\n\\nSoufflot nasceu em Irancy, perto de Auxerre, na França. Com 18 anos, entrou para a Academia Francesa de Roma, onde os jovens estudantes da década de 1750 se tornariam na primeira geração de criativos pleanamente neoclássicos. Ficará na Itália de 1731 a 1738. Quando volto...   \n",
       "3  A Faculdade de Medicina da Universidade de São Paulo (FMUSP) é uma escola médica da Universidade de São Paulo. Foi fundada em 1912 com o nome de \"Faculdade de Medicina e Cirurgia de São Paulo\" por Arnaldo Vieira de Carvalho (1867-1920), médico formado em 1888 pela Faculdade de Medicina do Rio de Janeiro. Em homenagem ao ilustre fundador, a Faculdade é, ainda hoje, chamada de a \"Casa de Arnaldo\" por seus alunos e ex-alunos. Em 1925 teve seu nome alterado para \"Faculdade de Medicina de São Paulo\" e em 1934, foi incorporada à recém-criada Universidade de São Paulo, passando a ter a atual desi...   \n",
       "4  A Escola do Teatro Bolshoi no Brasil é uma tradicional escola de balé existente na cidade de Joinville, no estado de Santa Catarina. Fundada em 2000, é a única filial do Teatro Bolshoi de Moscou e possui alunos de vários estados brasileiros. Tem como missão formar artistas cidadãos, promover e difundir a arte-educação.\\n\\nA instituição foi fundada em 15 de março de 2000, é a única filial do Teatro Bolshoi. \\n\\nUm orgulho para o Brasil e para Joinville, cidade sede. A Escola do Teatro Bolshoi no Brasil, com professores russos e brasileiros, forma bailarinos com a mesma precisão, técnica e q...   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             tokens_en  \\\n",
       "0            [37, 313, 519, 32188, 544, 357, 4598, 10536, 2188, 18074, 228, 49535, 35558, 14631, 69, 10205, 82, 8973, 5855, 2290, 89, 12340, 304, 7377, 111, 33643, 17394, 139, 228, 29945, 35558, 14631, 70, 32188, 271, 8973, 5855, 395, 18526, 1600, 366, 79, 924, 75, 4943, 267, 84, 7377, 111, 33643, 17394, 139, 228, 138, 115, 366, 70, 32188, 25792, 1600, 304, 2216, 64, 366, 8906, 268, 9869, 401, 300, 10277, 304, 3445, 4594, 12340, 16964, 2730, 72, 16175, 28749, 11, 38251, 3209, 268, 2413, 434, 68, 257, 366, 83, 2634, 31522, 3970, 390, 269, 7496, 16175, 28749, 1, 390, 3590, ...]   \n",
       "1                             [36, 2777, 324, 22739, 78, 38251, 334, 2611, 1885, 13827, 2030, 70, 947, 544, 2493, 1018, 947, 64, 466, 369, 5276, 8873, 390, 25942, 78, 390, 19931, 1000, 72, 4951, 11, 401, 1596, 11, 3829, 1849, 13276, 31185, 390, 6184, 94, 21468, 304, 27778, 7947, 39781, 357, 9804, 737, 317, 424, 64, 29509, 312, 671, 16595, 330, 1538, 6980, 838, 11, 20, 387, 65, 14, 13276, 31185, 13, 198, 37, 23013, 1070, 600, 64, 357, 363, 2301, 4763, 8, 279, 10304, 35459, 23638, 16175, 28749, 6863, 265, 12151, 390, 2321, 14, 6390, 11, 3758, 78, 267, 384, 84, 5771, 10205, ...]   \n",
       "2                                [28821, 13281, 12, 38, 7780, 391, 22862, 487, 26487, 357, 23798, 948, 11, 2534, 390, 474, 377, 8873, 390, 1596, 1485, 851, 6342, 11, 2808, 390, 556, 455, 78, 390, 1596, 1795, 8, 11511, 72, 23781, 610, 421, 5712, 78, 46754, 25792, 82, 11, 287, 44070, 7079, 466, 1556, 18526, 610, 421, 5712, 18840, 3713, 466, 3169, 420, 31172, 11965, 78, 13, 440, 384, 84, 491, 44349, 8873, 285, 15152, 369, 258, 66, 17305, 38251, 11, 5026, 288, 21356, 85, 3755, 11, 267, 5961, 400, 2634, 261, 357, 47, 12427, 28749, 8, 390, 6342, 11, 1500, 622, 8836, 4598, 257, ...]   \n",
       "3  [32, 13585, 32926, 671, 390, 5786, 1437, 12379, 26986, 312, 671, 390, 311, 28749, 34410, 357, 23264, 2937, 47, 8, 38251, 334, 2611, 3671, 5708, 285, 2634, 67, 3970, 12379, 26986, 312, 671, 390, 311, 28749, 34410, 13, 19434, 72, 1814, 4763, 795, 34463, 401, 267, 299, 462, 390, 366, 47522, 32926, 671, 390, 5786, 1437, 304, 21239, 3686, 544, 390, 311, 28749, 34410, 1, 16964, 16644, 41476, 47154, 8704, 390, 1879, 2100, 8873, 357, 1507, 3134, 12, 40454, 828, 285, 2634, 67, 3713, 1296, 4533, 795, 49584, 279, 10304, 13585, 32926, 671, 390, 5786, 1437, 466, 15338, 390, 42799, ...]   \n",
       "4             [32, 16319, 5708, 466, 1665, 47756, 10797, 1477, 23013, 645, 39452, 346, 38251, 334, 2611, 2083, 291, 1538, 3671, 5708, 390, 3652, 2634, 2152, 21872, 12385, 269, 312, 671, 390, 15251, 4244, 11, 645, 1556, 4533, 390, 8909, 5181, 283, 1437, 13, 7557, 4763, 795, 4751, 11, 38251, 257, 6184, 118, 77, 3970, 1226, 498, 466, 1665, 47756, 10797, 1477, 23013, 390, 5826, 66, 280, 304, 1184, 9019, 435, 403, 418, 390, 410, 6557, 380, 418, 1556, 22484, 48029, 576, 72, 4951, 13, 5825, 401, 78, 2051, 28749, 1296, 283, 6802, 292, 269, 32482, 26102, 418, 11, 1552, 2502, 304, ...]   \n",
       "\n",
       "   num_token_by_word_en  \\\n",
       "0              2.369519   \n",
       "1              2.392857   \n",
       "2              2.268409   \n",
       "3              2.244809   \n",
       "4              2.442777   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              tokens_pt  \\\n",
       "0  [9493, 22419, 363, 360, 5712, 46504, 30015, 16691, 33769, 70, 592, 32642, 1293, 21146, 2301, 258, 6970, 112, 17273, 12320, 46645, 17472, 16691, 33769, 4700, 274, 32642, 1293, 38371, 474, 330, 6701, 1996, 1766, 414, 6970, 112, 17273, 12320, 46645, 29825, 330, 4700, 369, 474, 258, 1952, 330, 1202, 1308, 290, 297, 3551, 258, 9592, 2301, 358, 5758, 12, 372, 9420, 259, 330, 38735, 260, 2271, 2, 260, 4318, 358, 1513, 260, 5214, 36162, 12, 34567, 13, 267, 300, 349, 4144, 17706, 14, 315, 806, 9659, 8892, 13110, 443, 750, 260, 22051, 258, 372, 11679, 443, 3081, 7358, 5227, 293, ...]   \n",
       "1                                   [1438, 15237, 1388, 360, 372, 349, 2661, 4766, 3764, 298, 5975, 260, 16023, 260, 19537, 12, 297, 1115, 12, 5699, 638, 2107, 4490, 260, 1430, 258, 2731, 2668, 363, 9381, 609, 315, 450, 7173, 9342, 621, 1194, 12, 21, 8770, 15, 2107, 4490, 14, 199, 2704, 10602, 363, 599, 45513, 9, 534, 19571, 8497, 260, 1762, 15, 11825, 12, 790, 275, 467, 2521, 11558, 347, 2743, 260, 40439, 260, 3158, 271, 1388, 360, 12, 2088, 6129, 12, 6834, 286, 592, 258, 2582, 3026, 6713, 1141, 423, 14, 199, 199, 33, 2661, 4766, 260, 327, 14, 4379, 260, 3158, 271, 1388, ...]   \n",
       "2                                      [33134, 13, 35969, 2582, 1738, 26035, 363, 41, 791, 89, 12, 2331, 260, 1637, 260, 45571, 2047, 2807, 12, 2820, 260, 1653, 260, 28277, 9, 379, 342, 17859, 3081, 12, 2333, 629, 298, 2266, 44381, 298, 1247, 5171, 41486, 14, 384, 467, 1313, 447, 1507, 372, 12, 944, 10436, 12, 275, 14982, 37648, 308, 363, 4042, 301, 280, 9, 260, 2807, 12, 4217, 259, 1129, 260, 26110, 12, 3470, 349, 2608, 8761, 259, 1925, 404, 19432, 373, 14, 199, 199, 15533, 1738, 26035, 2765, 300, 419, 791, 89, 12, 3067, 260, 21440, 269, 266, 12, 347, 2020, 14, 991, 664, ...]   \n",
       "3                                     [33, 4687, 260, 8138, 305, 1627, 260, 862, 1140, 363, 20640, 21090, 9, 372, 349, 2198, 9867, 305, 1627, 260, 862, 1140, 14, 1376, 4330, 300, 8928, 297, 275, 781, 260, 330, 42326, 260, 8138, 258, 33944, 260, 862, 1140, 2, 358, 22872, 9094, 260, 6171, 363, 1692, 8386, 13, 29477, 521, 4805, 4461, 300, 13789, 534, 4687, 260, 8138, 298, 1124, 260, 1588, 14, 575, 3904, 443, 26215, 5687, 12, 259, 4687, 372, 12, 872, 1913, 12, 2136, 260, 259, 330, 14238, 260, 22872, 2, 358, 660, 4433, 258, 417, 13, 43556, 14, 575, 9703, 1122, 467, 781, 12677, ...]   \n",
       "4                                    [33, 3183, 298, 4948, 15934, 385, 73, 325, 889, 372, 349, 3898, 2198, 260, 26728, 8050, 347, 786, 260, 23696, 12, 325, 1282, 260, 1925, 5337, 14, 32791, 300, 2424, 12, 372, 259, 2542, 19606, 298, 4948, 15934, 385, 73, 260, 9001, 258, 1893, 4433, 260, 1292, 3606, 4779, 14, 2904, 390, 3829, 4964, 3098, 6267, 12, 5746, 258, 29331, 259, 2432, 13, 45422, 14, 199, 199, 33, 4493, 379, 4330, 300, 1079, 260, 1563, 260, 2424, 12, 372, 259, 2542, 19606, 298, 4948, 15934, 385, 73, 14, 722, 199, 2635, 18621, 341, 275, 889, 258, 341, 23696, 12, 786, ...]   \n",
       "\n",
       "   num_token_by_word_pt  \n",
       "0              1.295616  \n",
       "1              1.486264  \n",
       "2              1.418052  \n",
       "3              1.253501  \n",
       "4              1.358349  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save\n",
    "fname = \"comparaison_tokenizers_en_pt_with_corpus_pt.csv\"\n",
    "df2.to_csv(path_data/fname, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load\n",
    "fname = \"comparaison_tokenizers_en_pt_with_corpus_pt.csv\"\n",
    "df2 = pd.read_csv(path_data/fname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(en) 1.31\n",
      "(pt) 1.12\n"
     ]
    }
   ],
   "source": [
    "# check min\n",
    "num_token_by_word_en_min = df2.num_token_by_word_en.min()\n",
    "num_token_by_word_pt_min = df2.num_token_by_word_pt.min()\n",
    "print('(en)',round(num_token_by_word_en_min,2))\n",
    "print('(pt)',round(num_token_by_word_pt_min,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(en) 75.29\n",
      "(pt) 75.65\n"
     ]
    }
   ],
   "source": [
    "# check max\n",
    "num_token_by_word_en_max = df2.num_token_by_word_en.max()\n",
    "num_token_by_word_pt_max = df2.num_token_by_word_pt.max()\n",
    "print('(en)',round(num_token_by_word_en_max,2))\n",
    "print('(pt)',round(num_token_by_word_pt_max,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(en) 2.25\n",
      "(pt) 1.36\n"
     ]
    }
   ],
   "source": [
    "# check mean\n",
    "num_token_by_word_en_mean = df2.num_token_by_word_en.mean()\n",
    "num_token_by_word_pt_mean = df2.num_token_by_word_pt.mean()\n",
    "print('(en)',round(num_token_by_word_en_mean,2))\n",
    "print('(pt)',round(num_token_by_word_pt_mean,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rate of increase: 65.87 %\n",
      "Multiplier coefficient: 1.66\n",
      "Multiplier coefficient in %: 165.87 %\n"
     ]
    }
   ],
   "source": [
    "# check increase rate and Multiplier coefficient\n",
    "increase = 0.\n",
    "multiplier = 0.\n",
    "\n",
    "for tok_en,tok_pt in zip(*(tokens_en_list,tokens_pt_list)):\n",
    "    increase += (len(tok_en)-len(tok_pt))/len(tok_pt)\n",
    "    multiplier += len(tok_en)/len(tok_pt)\n",
    "    \n",
    "# Rate of increase in % from pt to en\n",
    "increase_pct = increase / len(tokens_en_list)\n",
    "print('Rate of increase:',round(increase_pct*100,2),'%')\n",
    "\n",
    "# Multiplier coefficient = (Rate of increase in %, converted to number) + 1\n",
    "multiplier_coef = round(increase_pct+1,2)\n",
    "print('Multiplier coefficient:',multiplier_coef)\n",
    "\n",
    "# Multiplier coefficient in % = Multiplier coefficient, converted to %\n",
    "multiplier_pct = round((multiplier/len(tokens_en_list))*100,2)\n",
    "print('Multiplier coefficient in %:',multiplier_pct,'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 51s, sys: 1.63 s, total: 1min 53s\n",
      "Wall time: 1min 53s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "len_tokens_text_list = list()\n",
    "for index, row in df2.iterrows():\n",
    "    text = row['text']\n",
    "    length_text = len(text.split())\n",
    "    len_tokens_text_list.append(length_text)\n",
    "\n",
    "tokens_en_list = df2.tokens_en.tolist()\n",
    "len_tokens_en_list = [len(t) for t in tokens_en_list]\n",
    "\n",
    "tokens_pt_list = df2.tokens_pt.tolist()\n",
    "len_tokens_pt_list = [len(t) for t in tokens_pt_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/mnt/home/pierre/.conda/envs/fastai2/lib/python3.7/site-packages/IPython/core/pylabtools.py:132: UserWarning: Creating legend with loc=\"best\" can be slow with large amounts of data.\n",
      "  fig.canvas.print_figure(bytes_io, **kw)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEWCAYAAACqitpwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOydd3xUVfbAvyeTCYQioakQQFARRVEUVFxcu4IN7F2xrLi7rnV/KLjuCoqKZW27NqxYARUjFhZZBRUrIE1AVmyQgPROgJTz++PdgZfJzGSSTEk5389nPpk57977zpuZvDP3nnPPEVXFMAzDMBJJRroVMAzDMOoeZlwMwzCMhGPGxTAMw0g4ZlwMwzCMhGPGxTAMw0g4ZlwMwzCMhGPGxYiKiLwoIsPTdG4RkRdEZK2IfJOA8S4XkamJ0K2K5+8oIioimWk491AReSUJ494mIs8mYdynROTviR432STy/yVZn1kqMeNSixCRX0RkuYg09sn+ICJT0qhWsjgSOBFop6qHhR9Mt7FIJzXlxqOq96jqH6ozRqTPUVX/qKp3VU+75FKfv3/xYsal9pEJ3JBuJSqLiAQq2WUP4BdV3ZwMfQzDSC5mXGofDwD/JyI54QciLb2IyBQR+YN7frmIfC4iD4vIOhH5SUR+5+RLRGSFiAwIG7aViEwSkY0i8omI7OEbe193bI2ILBSR83zHXhSRJ0XkAxHZDBwbQd+2IjLe9V8kIlc7+VXAs8ARIrJJRIaF9dsPeMp3fJ2TNxORl0RkpYj8KiK3i0jE77iIPCAiU0WkmXt9pYgscMtwE8OuU0XkjyLygzv+uIiIO7a3e1/Wi8gqERkT8VPbyZUislRElonIX90Yu4vIFhFp6TtnD3cdwTC9+wK3Aee7a58d672McN1BEXldRN4SkSwRyRCRwSLyo4isFpGxItLCtQ19nwaIyGJ3fX/zjbVjBiUi/3b6hB7FIjLUHQuNv1FE5ovImRV8jmWWl0TkandNa9w1to3ns4lw7UNF5A0RecXpMldE9hGRIe67v0RETvK1byYiz7nPqkBEhotIIJrejuYi8r4b/2sR2cs33u9EZJr7rkwTkd/5jnVy36ONIjIJaBXpGmoVqmqPWvIAfgFOAMYBw53sD8AU97wjoECmr88U4A/u+eVAMXAFEACGA4uBx4EGwEnARqCJa/+ie32UO/4oMNUdawwscWNlAocAq4D9fX3XA73xfsQ0jHA9nwBPAA2B7sBK4HifrlNjvBfljgMvAe8ATd178T/gKn97p8szwESgkTt2BrAI2M9dy+3AF75xFXgPyAE6OD37umOvA38LXSNwZBR9Q5/N6+696+bGOcEd/wD4k6/9w8C/oow1FHilEu/lUOAVIBt43302AXfsRuAroJ37jJ8GXg/T+RnX9yBgG7BfND2cPHT+g93rc4G27j06H9gMtInxOb7Izu/3cXjfq0Ocfv8CPo3ns4nyvm0F+rjP+SXgZ/f5BYGrgZ997fPc+9EY2BX4BrimAr3XAIe58V8FRrtjLYC1wKXu2IXudUt3/EvgIXeNR+H935V7b2vTI+0K2KMSH9ZO43IA3o27NZU3Lj/4jnVz7XfzyVYD3d3zF0P/HO51E6AEaO9uEp+F6fc0cIev70sxrqW9G6upT3Yv8KJP17iNC56x3AZ09cmu8b03lwNfA2OAt4AsX7sJOCPkXmcAW4A93GvFZzSAscBg9/wlYCSebyjWZxf6bPb1ye4HnnPPzwc+913Lb8BhUcYa6r/xxPFeDgXG4xmgxwDxtVuAM0LudRugCO8GGNK5ne/4N8AFkfRwstZ439MLYrwXs4D+0T5nyhqX54D7w76DRUDHij6bKO/bJN/r04FN7DS0Td14OcBu7vuU7Wt/ITC5Ar2f9b0+BfjePb8U+Cas/ZdunA54P/oa+469Fv7e1raHLYvVQlT1O7xfa4Or0H2573mhGy9c1sT3eonvvJvwfpm1xfOJHC7e8to6tzRwMbB7pL4RaAusUdWNPtmvQG4lrsVPKyDLjRFtvL2B/sAwVd3uk+8BPOq7jjWAhPX9zfd8Czvfo1tc229EZJ6IXFmBnv735Fe89wG8GVdXEdkTL5BhvarGGyUXz3vZCzgQGKHu7uXYA3jbd+0L8AzVbr420a69DG4J703gNVUd7ZNfJiKzfOc4gPiXfdri+0zdd3A18X02kQj/rq9S1RLfa1z/PfBmM8t8ej+NN4OJRTRdylyHI/QZtQXWaln/YnjbWkfKwyKNhHEH8C3wT58s9OVsBGxwz/03+6rQPvRERJrgTe+X4t0kP1HVE2P0jZVyeynQQkSa+m6KHYCCOPUKH3sV3i/aPYD5UcZbgLcEOEFEjlPVhU6+BLhbVV+N89w7lVD9DW85BRE5EviviHyqqouidGkPfO/Tb6kbZ6uIjMUz0PsCL8c6bdjreN7LD4E5wEcicozvB8US4EpV/Tz8JCLSMYYOkfgX3nLO7b4x9sBbVjse+FJVS0RkFp5BjnQt4SzF+0xD4zUGWhL/96SqLMGbubRS1eIIxyubTr7MdTg6AP8BluH5ahr7DEyHKpyjRmEzl1qKu3mNAa73yVbi/dNd4hyPVwJ7RRkiXk4RkSNFJAu4C/haVZfgzZz2EZFLnZM4KCKHOmdnPPovAb4A7hWRhiJyIHAV3jp1PCwH2jm9cL8+xwJ3i0hTd1O7Gc/X4D/v63gO8f/6nK1PAUNEZH/Y4cg9Nx4lRORcEWnnXq7FuyGUxOjydxFp5M51Bd5nGOIlvGWSfuF6h7Ec6CguWCHe91JV78dbbvlIREIzh6fw3rM93PW0FpH+FVx2OUTkGuBo4CJVLfUdaoz3nqx07a7Am7n4r2XH5xiB14ArRKS7iDQA7sH7Dv5SWR0rg6ouwzPI/xSRXcQLfNhLRI6OU+9wPsD7f7lIRDJF5HygK/Ceqv4KTAeGiRdkcSTekl2txoxL7eZOvH9eP1cDg/CWDvbHu+lUh9fwZklrgB54v6xxv5BPAi7A+1X2G3AfnkMyXi7EW9dfCryN56+ZFGffj4F5wG8issrJrsObvf2E57x/DXg+vKOqjsJ77z4WkY6q+rbTfbSIbAC+A06OU49Dga9FZBOeX+MGVf05RvtP8IIHPgIeVNUPfXp9DpQC31Zw83zD/V0tIt+653G9l+rtH8nDM64t8II0xgMfishGPOf+4TGvODIXAnsCS2VnxNhtqjofb3b9Jd4NuRvgnyVF+hz9+n4E/B3PT7YM78fSBVXQrypchrfUOh/vh8ObeD4pqEDvcFR1NXAa8Fe8/81bgNNUNdT3Irz3fQ3e/9tLibuM9CBll18Nw0gnIvIxns8i4TvfDSOVmHExjBqCiBwKTALahznnDaPWkfRlMbf2P1NE3nOvO7nNRT+IyJjQmqWINHCvF7njHX1jDHHyhSLSxyfv62SLRGSwTx7xHIZRUxGRUcB/gRvNsBh1gVT4XG7Ai9IJcR/wsKp2xlvHvMrJr8ILx9sbbwPZfQAi0hVvjXV/oC/whDNYAbzIn5PxHGMXuraxzmEYNRJVHaCqzVT1xXTrYhiJIKnGxUXRnIqXygMREbwdt2+6JqPwdkeDt/9glHv+JnC8a98fbyPfNucoXYS3A/YwYJGq/uT2LIwG+ldwDsMwDCMFJHufyyN4URFN3euWwDpf3Hg+OzdD5eI2mKlqsYisd+1z8SJYiNBnSZj88ArOUQYRGQgMBGjcuHGPfffdtwqXaBiGUX+ZMWPGKlVtHS5PmnERkdOAFao6Q0SOCYkjNNUKjkWTR5p1xWpfXqg6Ei91Bz179tTp06dHamYYhmFEQUQiZhNI5sylN9BPRE7BS6a3C95MJkdEMt3Moh1uhzLeDKM9kC9eVt9meDHfIXkIf59I8lUxzmEYhmGkgKT5XFR1iKq2U9WOeA75j1X1YmAycI5rNgAvpxJ4G7kGuOfnuPbq5Be4aLJOQGe85HnTgM4uMizLnWO86xPtHIZhGEYKSMcO/VuBm0VkEZ5/5Dknfw5o6eQ345Iyquo8vLQe8/Hy8FyrqiVuVvIXvNTpC4Cxrm2scxiGYRgpwDZROiL5XIqKisjPz2fr1q1p0qrqNGzYkHbt2hEMBitubBiGUUVEZIaq9gyXW1bkGOTn59O0aVM6duyIRC5uVyNRVVavXk1+fj6dOnVKtzqGYdRDLHFlDLZu3UrLli1rlWEBEBFatmxZK2dchmHUDcy4VEBtMywhaqvehmHUDcy4GIZh1CM2bi3irvfms3rTtqSex3wuhmEY9YTJ36/gtrfnsnzDVg5s14z+3ataVbxizLgYhmHUcdZt2c6d785n3MwC9tmtCU9e0pvu7XOSek5bFqvhvPLKKxx22GF0796da665hpKSEpo0acLf/vY3DjroIHr16sXy5csrHsgwjHrJhLnLOOGhTxk/eynXH9+Zd687MumGBWzmEjfD3p3H/KUbEjpm17a7cMfp+0c9vmDBAsaMGcPnn39OMBjkz3/+M6+++iqbN2+mV69e3H333dxyyy0888wz3H777QnVzTCM2s3Kjdu4Y/x3fDD3Nw7I3YWXrjyMrm13Sdn5zbjUYD766CNmzJjBoYceCkBhYSG77rorWVlZnHbaaQD06NGDSZPiLTtvGEZdR1XJm1XAsHfns2V7Cbf07cLA3+9JZiC1C1VmXOIk1gwjWagqAwYM4N577y0jf/DBB3eEGgcCAYqLiyN1NwyjnrFsfSG3jZvL5IUr6bFHc+47+0D23rVJWnQxn0sN5vjjj+fNN99kxYoVAKxZs4Zff42Y3dowjHqMqvLa14s56aFP+eqnNdxxelfGXnNE2gwL2MylRtO1a1eGDx/OSSedRGlpKcFgkMcffzzdahmGUYNYvHoLV42axg8rNgGwW9MGNG+URSAjvRupLXGlI1LiygULFrDffvulSaPqU9v1NwwjOiWlyqgvfmHEhO/ZXlJa5lh2MMC9Z3XjjIOTt48lRLTElbYsZhiGUctYtGIT5z71BXe+N59ImZ4Ki0p4YOLC1Cvmw4yLYRhGLaG4pJTHJy/ilMc+46dVm3n4/IPYVlwasW3BukLyZhakWMOdmM/FMAyjFjB/6QZueWs23xVs4JRuuzOs3wG0btqAByf+j4J1hRH7DBk3FyAly2Ph2MzFMAyjBrOtuISHPlxIv39P5bf123jy4kN44uIetG7aAIBBfbqQHQxE7JvO5bGkGRcRaSgi34jIbBGZJyLDnPxFEflZRGa5R3cnFxF5TEQWicgcETnEN9YAEfnBPQb45D1EZK7r85i4zR8i0kJEJrn2k0SkebKu0zAMI1nMXLyW0x6bymMfL6Jf97b89+ajOLlbmzJtzjg4l3vP6hZ1jKVRZjXJJpkzl23Acap6ENAd6CsivdyxQara3T1mOdnJQGf3GAg8CZ6hAO4ADgcOA+7wGYsnXdtQv75OPhj4SFU7Ax+514ZhGLWCwu0l3PPBAs5+8gs2bSvmhcsP5aHzupPTKCti+zMOziU3JzvisbZR5MkmacZFPTa5l0H3iBX33B94yfX7CsgRkTZAH2CSqq5R1bXAJDxD1QbYRVW/VC+e+iXgDN9Yo9zzUT55neGRRx5hy5Yt6VbDMOoteTML6D3iYzoNfp/eIz5OmPP8659Wc/KjnzLy05+44LAOfHjTURy7764V9ou0PJYdDDCoT5eE6FVZkupzEZGAiMwCVuAZiK/dobvd0tfDItLAyXKBJb7u+U4WS54fQQ6wm6ouA3B/I34yIjJQRKaLyPSVK1dW+TrTgRkXw0gfeTMLGDJuLgXrClG8yKwh4+ZWy8Bs2lbM3/O+4/yRX1Gq8NrVh3PPmd1o2jAYV//Q8lhuTjYC5OZkp2yvSySSGi2mqiVAdxHJAd4WkQOAIcBvQBYwErgVuBOItJ1UqyCvjH4jnQ707NmzRu4m/eWXX+jbty+HH344M2fOZJ999uGoo45i6dKlHHvssbRq1YrJkyenW03DqFc8MHEhhUUlZWQh53lVbuaf/G8lt42by9L1hVx1ZCf+etI+NMqq/O35jINz02ZMwklJKLKqrhORKUBfVX3QibeJyAvA/7nX+UB7X7d2wFInPyZMPsXJ20VoD7BcRNqo6jK3fLai2hcxYTD8Nrfaw5Rh925w8ogKmy1cuJDnnnuO3r17c+WVV7J9+3batm3L5MmTadWqVWJ1MgyjQqI5ySvrPF+/pYjh78/njRn57NW6MW/+8Xf02KNuxB8lM1qstZuxICLZwAnA9+5mj4vsOgP4znUZD1zmosZ6AevdktZE4CQRae4c+ScBE92xjSLSy411GfCOb6xQVNkAn7xW0r59e3r37g3AJZdcwtSpU9OskWHUb6I5ySvjPP9w3m+c8PAnjJtZwLXH7sX71/++zhgWSO7MpQ0wSkQCeEZsrKq+JyIfi0hrvGWtWcAfXfsPgFOARcAW4AoAVV0jIncB01y7O1V1jXv+J+BFIBuY4B4AI4CxInIVsBg4t9pXE8cMI1lIWH6H8NeGYaSWQX26MGTc3DJLY/E6z1dv2sYd4+fx3pxltM1pSItGWTwx+UfyZi5lUJ8uNWZZq7okzbio6hzg4Ajy46K0V+DaKMeeB56PIJ8OHBBBvho4vpIq11gWL17Ml19+yRFHHMHrr7/OkUceyaJFi9i4caMtixlGCsibWcADExeydF0hbXOyGdSnC/ee1a2cLJZhUFXGz17K0PHz2LythFO67c7HC1aw1aVvCQUFQHJ31Ee6lmScz9K/1AL2228/Ro0axTXXXEPnzp3505/+RFZWFieffDJt2rQxh75hJJFQZFholhIyAvee1Y3PB0f8rVyO5Ru28re3v+O/C5ZzUPscHjjnQK54YdoOwxKiOkEB8RDtWiDxBs2MSy0gIyODp556qozsuuuu47rrrkuTRoZRf6hOZJiq8sb0fO56fz5FJaXcfup+XNG7E4EMSVhQQGVIdJRbLMy4GIZhxKCqRmDJmi3c9vZcPvthFYd3asF9Zx9Ix1aNdxxvm5MdMeFkMnfUp9KgmXGp4XTs2JHvvvuu4oaGUY9Jph+hskagtFR5+atfue8/3yPA8DMO4KLDOpARVhmyOkEBVSWVBs2yIldAba3UWVv1NozKkozd8n4qk1blp5WbOH/kl9wxfh49O7bgw5uP5pJee5QzLJCeHfWpTBFjM5cYNGzYkNWrV9OyZctaFf6rqqxevZqGDRumWxXDSDrJ9iOExog1MyouKeW5qT/z0KT/0SAzgwfPPYizD8mt8L6R6h318VxLojDjEoN27dqRn59Pbcs7Bp5hbNeuXcUNDaOWkwo/Qiwj8P1vG7jlzTnMyV9Pn/13467+B7DrLjX3h12qDJoZlxgEg0E6deqUbjUMw4hBOhzjANuLS3liyiIen7yIXRoGefyiQzil2+61apUjmZjPxTCMWk06Us3PyV9Hv39P5ZH//sAp3dow6eajOfXANmZYfNjMxTCMWk0q/Qhbi0p45L8/MPLTH2ndtAHPXtaTE7rulvDz1AXMuBiGUetJhR9h+i9ruOXNOfy0ajMXHNqeIafsR7Ps+Gqt1EfMuBiGYcRg87ZiHpi4kFFf/kJuTjavXHU4R3a2nH4VYcbFMAwjClN/WMXgcXMoWFfIgCM6MqhPFxo3sNtmPFT4LolIb2CWqm4WkUuAQ4BHVfXXpGtnGIaRBjZsLeKe9xcwetoS9mzVmLHXHMGhHVukW61aRTwm+EngIBE5CLgFeA54CTg6mYoZhmGkg48WLOe2t+eycuM2rjl6T246YR8ahkWjGRUTj3EpVlUVkf54M5bnRGRAhb0MwzBqEWs2b+fOd+eRN2sp++7elGcu68mB7XLSrVatJR7jslFEhgCXAEe5ypIWImEYRp1AVXl/7jLueGceG7YWceMJnfnzMXuTlWnbAKtDPO/e+cA24CpV/Q3IBR6oqJOINBSRb0RktojME5FhTt5JRL4WkR9EZIyIZDl5A/d6kTve0TfWECdfKCJ9fPK+TrZIRAb75BHPYRiG4WfFhq388ZUZ/OW1meQ2z+bd647kxhP2McOSACp8B1X1N1V9SFU/c68Xq+pLcYy9DThOVQ8CugN9RaQXcB/wsKp2BtYCV7n2VwFrVXVv4GHXDhHpClwA7A/0BZ4QkYCbQT0OnAx0BS50bYlxDsMwDFSVN2fkc8JDnzB54UoGn7wv4/70O/bdfZd0q1ZnqNC4iMhZbgawXkQ2iMhGEdlQUT/12OReBt1DgeOAN518FHCGe97fvcYdP168XAr9gdGquk1VfwYWAYe5xyJV/UlVtwOjgf6uT7RzGIZRzylYV8jlL0zj/96YTZfdm/KfG37PH4/ei8yAzVYSSTw+l/uB01V1QWUHd7OLGcDeeLOMH4F1qlrsmuTjLbPh/i4BUNViEVkPtHTyr3zD+vssCZMf7vpEO0e4fgOBgQAdOnSo7OUZhlGLKC1VXv1mMSM+WIACw/rtz6VRaq0Y1Sce47K8KoYFQFVLgO4ikgO8DewXqZn7G+kT1hjySD8zYrWPpN9IYCRAz549rbqWYdQgKqouWZnqk7+s2sytb83h65/XcOTerbj3rG60b9EoVZdSL4nHuEwXkTFAHp4fBQBVHRfvSVR1nYhMAXoBOSKS6WYW7YClrlk+0B7IF5FMoBmwxicP4e8TSb4qxjkMw6gFhKpLhoqAFawr5KYxs7hxzCxyc7I5dt/WvDWjoMzxIePmApQxMCWlyguf/8yDHy4kGMjgvrO7cV7P9pa9OAXEs8i4C7AFOAk43T1Oq6iTiLR2MxZEJBs4AVgATAbOcc0GAO+45+Pda9zxj9Wr1TseuMBFk3UCOgPfANOAzi4yLAvP6T/e9Yl2DsMwagGRqkuGlhYK1hXy6leLo1afDPHD8o2c/eQXDH9/AUfu3YpJNx3N+Yd2MMOSIiqcuajqFVUcuw0wyvldMoCxqvqeiMwHRovIcGAm3o5/3N+XRWQR3ozlAnf+eSIyFpgPFAPXuuU2ROQvwEQgADyvqvPcWLdGOYdhGLWAiqpIRlvDXrqukKKSUp6a8iP/+ngRjRsEePSC7vQ7qK0ZlRQj3g/9GA1E9sFLAbObqh4gIgcC/VR1eCoUTBU9e/bU6dOnp1sNwzCA3iM+jlhdsiJaN2lA66YNmL9sA6cd2Iah/fanVZMGSdDQCCEiM1S1Z7g8nmWxZ4AhQBGAqs7BzSoMwzCSQaTqkuGEz0MyM4TVm7exctM2nr60B/++6BAzLGkkHod+I1X9JmxKWRytsWEYRjzcnjeX179eQokqAREuPLw9w8/oBpStLlmwrhCh7FJYdjDA2T1ymfz9SgrWFZKZIRSXKuf0aMffT+1Ks0aWoSrdxGNcVonIXrjPVkTOAZYlVSvDMOo0t+fN5ZWvFu94XaLKK18t5pWvFpPrCysOGZlIYcd99t+dBpkLef7zn9ltl4bcc1Y3jt6ndbouyQgjHuNyLd5ekH1FpAD4GS+JpWEYRpV4/eslUY9FCisOL2P8xY+r6PPIpyxes4VLe+3BrSfvSxMr4lWjiOfTKFDVE0SkMZChqhtFxKrmGIZRZUoqCCQKhRWHb4rcuLWIeyd8z2tfL6Zjy0aMHtiLXnu2TKaqRhWJx7iME5H+qroZQER2B94HeiRVM8MwaiXx7JwPiFRoYMLDkScvXMFt4+ayfMNWrv59J24+sQvZWVbEq6YST7RYHvCmy0TcEfgQL3rMMAyjDKGd9QXrClG8Ja4bx8zi4Ds/JG9mwY52Fx7ePvogjrY52QCs27Kdm8fO4ooXptGkQSZv/el3/O3UrmZYajjxbKJ8xu2AzwM6Ateo6hfJVswwjNpHpJ31AGu3FJXxo4SiwkLRYuFkBwMM6tOFCXOX8fd35rFuy3auP25vrj1ubxpkmlGpDUTdRCkiN/tfApcCc/F2vKOqDyVduxRimygNo/p0Gvx+1N3zALk52Xw++Lhy8vCltD8evSdf/rSaD+b+xv5td+GBcw6ia1urtVITibaJMtbMpWnY67ejyA3DqMf4DUNGBb6UaGldQtFgqkrerAKGvTufLdtKGNSnCwOP2pOg1VqpdUQ1Lqo6zP9aRJp64h0FwAzDqOeEZy+uyEkf8qNEYtn6Qv729nd8/P0KDumQw/3nHMjeu9pv2dpKhT4XETkAeBlo4V6vAi7zJYk0DKOeEs3HIgLhdibkRwlHVRk9bQn3vL+A4lLl76d15fLfdSRgRbxqNfGEIo8EblbVyQAicgxevrHfJVEvwzBqAVGzFys8cn73CkOSF6/ewuBxc/jix9UcsWdL7jv7QDq0tCJedYF4jEvjkGEBUNUpbkOlYRj1nLY52RGzF7fNyS63q95PSaky6otfeGDiQgIZwj1nduPCw6yIV10iHuPyk4j8HW9pDLzULz8nTyXDMGoLg/p0KeNzgejLXyEWrdjErW/NYcavazm2S2vuPrNbTF+MUTuJx7hcCQwDQmWNPwUuT5ZChmHUHvzZi2Mtf+XNLOD+/3zP0vVbAWiUFeCh8w7izINzbbZSR4nHuJygqtf7BSJyLvBGrE4i0h54CdgdKAVGquqjIjIUuBpY6ZrepqofuD5DgKuAEuB6VZ3o5H2BR/EqTj6rqiOcvBMwGi/Y4FvgUlXdLiIN3Ll7AKuB81X1lziu1TCMCoiU3iXS3hV/+1vfmsO24tIdstJSJUPEDEsdJp7g8UipXuJJ/1IM/FVV9wN6AdeKSFd37GFV7e4eIcPSFa8I2f5AX+AJl3ImADwOnAx0BS70jXOfG6szsBbPMOH+rlXVvYGHXTvDMKpB3swCug/7kBvHzCqT3mXIuLllUrv42VZcwt/zvitjWAC2FpeWqXdv1D2izlxE5GTgFCBXRB7zHdqFOIqFqeoyXN0Xl0l5ARDZu+fRHxitqtuAn0VkEXCYO7ZIVX9yeo0G+rvxjgMucm1GAUPxSjL3d88B3gT+LSKiFdV0NgwjIuH7WfxEy2A8a8k6bnlzNhu3Rb5dRI00M+oEsWYuS4HpwFZghu8xHuhTmZO4hJcHA1870V9EZI6IPC8izZ0sF/AXech3smjylsA6VS0Ok5cZyx1f79qH6zVQRKaLyPSVK1eGHzYMwxFtP0uIgnWFO2YvW4tKuOeDBZz1xOds3FpMy8Xt/BYAACAASURBVMZZEfuYE79uE2uH/mxgtoi8pqpFVT2BiDQB3gJuVNUNIvIkcBdeZcu7gH/iBQ1EWnxVIhtAjdGeCo7tFKiOxNvHQ8+ePW1WYxjs9KkUrCuMKzV+iCHj5vLjik28O2cpv6zewoWHdWDIKfvy8YIVlY4oM2o/8WRFro5hCeIZlldVdZwbb7nv+DPAe+5lPuDPw90Ob/ZEFPkqIEdEMt3sxN8+NFa+iGQCzYA1Vb0Ow6gvVDadi5/CohL+NXkRHVo04rWrD+d3e7UC4o8oM+oWSasLKl4YyHPAAn8GZRFp4/wxAGcC37nn44HXROQhoC3QGfgGbxbS2UWGFeA5/S9SVRWRycA5eBFjA4B3fGMNAL50xz82f4thlMcf+ZXTKMi6wqJyaVsqy39u/D2NssreWmJtqDTqJvHkFjtXVd+oSBaB3rg0/SIyy8luw4v26o63TPULcA2Aqs4TkbHAfLyAgWtVtcSd7y/ARLxQ5Od9ec1uBUaLyHC8UgDPOflzwMsuKGANnkEyjHpLpPBhoMwsZe2WKi9S7CA3J7ucYTHqJ1HruexoIPKtqh5Skay2Y/VcjLpK3swCBr05m6KSnf/rwYDQpEFmQgxKiOxggHvP6mYzlHpGpeu5VDcU2TCMmsGwd+eVMSwARSVabcMSEAgNm5MdZGi//c2wGDuINX8NhSL3wwtBDrERuCmZShmGUXXCl8ASOTsJkeEzLEC5TZKGEVcoMp5TfV88P8lCVd2eIv0Mw6gE4dFekTIWV4eACI0bBNiwteziRbSNlEb9JZ70LycCPwKPAf8GFrklM8MwahgVbXaMhT/NV/NGQS4+vD052cEdsoaZGZSoljMsIWzHveEnnrCOh4BjVXURgIjsBbwPTEimYoZhVJ7q3OD9sT1btpcw/Zd1rCssYq/WjclfW8jWCpa+bMe94SeemcuKkGFx/ASsSJI+hmFUg0Td4LcVl/K/5Ru564wDKNxeUqFPxXbcG+HEY1zmicgHInK5iAwA3gWmichZInJWkvUzDKMSHLtv64SNpcClvfZgmavBEo3cnGwLQTbKEc+yWENgOXC0e70Sr37K6Xjfv3FR+hmGkWImf5+4BKwBEToNfp+MGPnFcnOyY9ZyMWoohWshqykEkrfhNZ7cYlck7eyGYSSMvJkFVYoOEyJkdWVnXrFY+cUSHY1mJJHSEvj5E5j5Cix4D84bBV2SF5tleRoMoxZS1czFkWjcIJOzD8klb2YB66NEgkUjYJUkawZzxsJHd8L6fGjWDo7/Bxx4nndszc8w6zXvsSEfsptDj8uhZeekqmTGxTBqGbfnzeXVrxbvmG1UxrCEDFFmhlBcqvTv3pY7Tt+fT/+3krHT8yutS3WMmpEg5oyFd6+HIjeLXL8Exl8Hi7+EVT/AL58BAnsfD32GQ5dTILNB0tUy42IYtYi8mQVlDEtlaNusIf265/LMZz/RonEWd5/ZjRO77gZUfX9MroUfp5+P7txpWEIUb4Xpz0Pj1nDc7XDQhd6MJoXEyi12c6yO/jT6hmGkhgcmLqySYckKZFBcqjz1yY+c37M9t526H5O/X0HvER+zdF1hhWMGAwIKRaU7W1r4cQ1h/ZLox7ZthJw9yhqWWEtoCSTWzKWp+9sFOBSvRgp4UWKfJlwTwzAqpKqbJLeXlBIMZPDyVYfx+86ty6WJiUWuL0W/FfyqIZQUwQ8fes75WBRv9QxJyHhEWkJ793rveYINTKzcYsMARORD4BBV3eheDwUqquViGEaC8CeirCqNswJ8eNNRNG7g/cvHswwWKYW+GZM0s+J7mPkyzBkDm1dCk91gn77w02Qo3ha5z3qfLy3SElpRYVkDlCDi8bl0APyJKrcDHROqhWEY5cibWcDQ8fNYV1j9rMZbtpfsMCxQ8QwoIGIbI2sKW9fDd2/BzFehYDpkZHohxN0vgb1P8PaqzBkLb/8RNMIPBv+S2PooQRvR5NUgHuPyMvCNiLyNFw5/JvBSwjUxDGMHlVm2iofwtDBtc7Jj7lEpVTXDkk5KS+HXqd6y1/zxUFwIu3aFPvdAt/OgSVgmhtCsw7/kBRDM9nwqIZq1i+yjSYKzP55NlHeLyH+AI53oClWdWVE/EWmPZ4R2B0qBkar6qIi0AMbgzX5+Ac5T1bUiIsCjeAXKtgCXq+q3bqwBwO1u6OGqOsrJewAvAtnAB8ANqqrRzlGRzoaRTsKXvxIV5BvJ8T6oT5eYxsuSUKaIcOf6EdfC1g0w61VY9ys0aAbdL4KDL4G2B5dNXR1OyMDEctYf/4+KDVCCqLDMMYCIBIDd8BkjVV1cQZ82QBtV/VZEmuIVHDsDuBxYo6ojRGQw0FxVbxWRU4Dr8IzL4cCjqnq4MxTTgZ54/28zgB7OIH0D3AB8hWdcHlPVCSJyf6RzxNLXyhwb6STRM5UQuTEc73kzCxj27rxyxcSsXHGKCHeu+9nzGDj4Utj3VO/mn+jzJjBaLFqZ4wqNi4hcB9yBl1+sBJctQlUPrKQC7+DVg/k3cIyqLnMGaIqqdhGRp93z1137hcAxoYeqXuPkTwNT3GOyqu7r5BeG2oX6hp8jln5mXIx00n3YhwnxrYSojIEIr1xpUWApQBX+2QU2LS9/rGkb+Ov3qdepikQzLvH4XG4Auqjq6mqcvCNwMPA1sJuqLgNwN/9dXbNcwL8YmO9kseT5EeTEOEe4XgOBgQAdOnSo4tUZRvW4+JkvE2JYQjnCYs1WInHGwblmTKpCVWYAm1d5/Wa+EtmwAGz8LfG6poF4jMsSYH1VTyAiTYC3gBtVdYNEXzOMdECrII8bVR0JjARv5lKZvoaRCG7Pm8vnP65JyFghw2JZilNAZfaLlBTDjx95IcQL/wOlRZDbAxo2h60RXMEp3kmfLOIxLj8BU0TkfWBHIHU8O/RFJIhnWF5V1VBq/uUi0sa3ZBUqPJYPtPd1bwcsdfJjwuRTnLxdhPaxzmEYNYbb8+byylcxXZeVxkoNp4h49ousWgSzXoFZr8Om36BRKzj8Gs85v+t+kX0uSXKup4N4jMti98hyj7hw0V/PAQvCDNF4YAAwwv19xyf/i4iMxnPor3fGYSJwj4g0d+1OAoao6hoR2SgivfCW2y4D/lXBOQwjZfh9Gc2yg4jA2i1F1c5iHItmvpr3RhKJul9kCXz7srfsteQrkADs0we6X+z9Dfg+n3iiu2oxcUWLVWlgkSOBz4C5eKHIALfhGYKxeJszFwPnOkMheM7+vnihyFeo6nQ31pWuL8DdqvqCk/dkZyjyBOA6F4rcMtI5YulrDn0jGhU5vCOlv2/eKMimrcVlcnGlgmBAeOCcg+qfDyXZ+bLCx9++GQoj3VKc56vVPt4M5cALoOluidOjBlKdaLHWwC3A/nhVKQFQ1Tq1sGvGxYhEpBBhfyRWskKIq0O987tEW146/bHEGJhI4weyvI2OGlb/puPvPcPW7tDYe1LqENGMS0YcfV8Fvgc6AcPwNiVOS6h2hlFDiZSDq7CohAcmLox6PN0UrCskb2ZButVIHbH8H8kav2R7WcOS3QL6/Rsufw/aH1ZvDEss4vG5tFTV50TkBlX9BPhERD5JtmKGUROI5iAPyWuqA33IuLlAPUk0mex8WbHGuX4mtNgzMeepY8QzcwkF4C8TkVNF5GDKRmkZRp0lWhqUDBE6DX4/ckB8DcA/u6rzRAvdrW5I75Y18PVIL1FkxPHbm2GJQTzGZbiINAP+Cvwf8CxwU1K1MowawqA+XcgOBsrJS1RRvI3WiSaef8p4qKmzqoRz/D/Kp0ipakhvaQks+i+8cbm3g37CINilDWSEReHVoZDhZBFP4sr33NP1wLHJVccwag6hKLDCopIdUWDJDCMGCGbAD/ecSu8RH8fMWuwnmk71JvlkIkJ61/zkpbSf/TpsKPB8KD2vgoMvht27pax6Y10iaaHItQ2LFjP8RIoCCwaEopLk/788cn53gPLnzxAQyuiQHQxwdo9c3ppREDWizYjC9s0w/x3PqPw6FSQD9jreCyHucjJkNki3hrWC6uQWM6rAtPFP0/7bB9hVV7JCWrPkkEEc2u+adKtlxEmkKLBUGBaAm8bM4uJeHbj3rG7l9teEdAvfc9Nzjxb1L/lkVWYTqpA/zUvF8t3bsH2j5zc5/h9w0IWwS9vU6F4PsJmLI5Ezl2njn+aAGbeTLTsLeBZqFt/1GG4GppbQafD7CaunUhUEePj87nXfQFSVyu5t2fgbzB7t1UlZ9T8INob9z/RmKR16WehwNajyzEVEGgBn4xXe8tdzSVAQed2j/bcPlDEsANmynfbfPgBmXGoFFVVqTDaKN0Mx4xKFeHJ7FW+HHyZ6y14/fOiVAO5whLcfZf8zoEHT1Otdj4hnWewdPGf+DHyJK43o7KorI4ao7qqrUq+MUSUG9enCjWNmpVWHehPtVRVi7W1ZPt+bocweDVtWQZPdoff1Xs35VnunVs96TDzGpZ2q9k26JnWIFdKa3VkZQd6K3dOgj1F5zjg4N+3GpdZGe6UisipaLfhAEJ48wgsd7nKyV81xr+MgYO7lVBNPSP0XItIt6ZrUIZYcMohCLZtAulCzWHLIoDRpZFSFdK7CR6p7XysI+ULWLwF0Z52TOWMTe57j/wGZDcvLG7eGviPgrwvh/Jdhn5PMsKSJeN71I4HLReRnvGWxKpU5rk8c2u8apoGLFlvFCmnFkh4WLVYTiJbhOFJ6/HQ59CtbSbJGEY8vpCr4Z0NNd4d2PSGrMRRv9Y5nNYHeN8JR/2fO+RpCPFmR94gkV9Vfk6JRmrB9LnWfaBmOI+0TSQd1Ym/K0Bwim2WBoevKiuJdPpszFsZft9OQhNi1K/z+r7DvaRCMMIsxUkKVo8VU9VdXm6Wzqr7gUvA3SYaShpFMomU4TnQ1yHjJyQ7SuEFm3dqbEs0XEp7nK54ywbPHwIe3w+YohWS3bYRu5yRGbyPhxBOKfAfQE+gCvAAEgVeA3slVzTASS02KvgoGhKH99q/9xiSc4/8RX+neaMtn466GSXd4xih/GjEXJxOV9dhICvE49M8E+gGbAVR1KWAB4katI6dRzSgB3LxRsO5WizzwPG8jY7P2gHh/I21sjDS7CbFxKeR/Q4Ver+pmPTaSSjwO/e2udLACiEjjeAYWkeeB04AVqnqAkw0FroYdcbq3qeoH7tgQ4CqgBLheVSc6eV/gUSAAPKuqI5y8EzAaaAF8C1yqqtvdps+XgB7AauB8Vf0lHp2Nuk0qk1GEfDmTv19Zd5a94vWRHHheeXl431A54KpiWYlrPPEYl7Ei8jSQIyJXA1cCz8TR70Xg33g3ej8Pq+qDfoGIdAUuwCul3Bb4r4js4w4/DpwI5APTRGS8qs4H7nNjjRaRp/AM05Pu71pV3VtELnDtzo9DX6MOkzezgHWFRRU3TBC13jEfTjw+ksr0rQ7N2ltW4lpAPA79B0XkRGADnt/lH6o6KY5+n4pIxzj16A+MVtVtwM8isgg4zB1bpKo/AYjIaKC/iCwAjgMucm1GAUPxjEt/9xzgTeDfIiJqSdTqFf7Q4obBDAqLSlN6/jplWAAm3Fr5EOMds5WqGpOw2U2s3GFGjSOu3UXOmFRoUOLkLyJyGTAd+KuqrgVyga98bfKdDGBJmPxwoCWwTnVHEWt/+9xQH1UtFpH1rn253CsiMhAYCNChQ4fqX5lRIwgPOU61YbmkVx35LsVjHNbnR14ug/KO/coQzIaDLvJyglkNlVpJqreuPgnchfdz5C7gn3jLbJF2PSmRAw40RnsqOFZWqDoSGAnePpdYihu1g7yZBdw0ZlbaNkA2CmYw/Iw6kNAiUtbhSGQ1Kr/kNf46kEDFfbNbeBsh1+dDdnNPVrjWDEkdIaXGRVWXh56LyDNAqMplPtDe17QdsNQ9jyRfhecDynSzF3/70Fj5IpIJNAPWJPhSjDQRbYd96NjNaTQsAtxzVh1JXBEpVDgS2zeXl4VvdoxEMBtOvs8MSB0maiiyiHzk/t6XqJOJSBvfyzOB79zz8cAFItLARYF1Br4BpgGdRaSTiGThOf3HO//JZCC0g2oAXvbm0FgD3PNzgI/N31I3CC13FawrRIGCdYUMGTeXvJkFAAwdP4/ULoDtRICLe3Wo3b6WOWPh4QO8XfbVdbrHIlp4slGniDVzaSMiRwP9nCO9zHKTqn4ba2AReR04BmglIvnAHcAxItIdb5nqF+AaN9Y8ERkLzAeKgWtVtcSN8xdgIl4o8vOqOs+d4lZgtIgMB2YCzzn5c8DLLihgDZ5BMuoA0XbYh+qepDIaDLza9aWqtTfMeM5Yz1FfmISJfXYLKC6Mv5iXUeeIZVz+AQzGW3J6KOyY4kVrRUVVL4wgfi6CLNT+buDuCPIPgA8iyH9iZ0SZX74VODeWbkbtJNoO+2TvvO+XMZVbMsfSVlaxVFtxf/F5jC89klJVfh5xalLPnRDCjUh2C68K47cvQWkSDHJoyQuSn3rfqLFENS6q+ibwpoj8XVXvSqFOhhGRRlkBNm8vn1xSgY6D30/KOftlTGVE8Fkaucqi7WQVI4LPQhHM2OXEpJwzocwZC+9cCyW+yqiFa2B61N951UMCZWcnZkzqLfHsc7lLRPoBRznRFFV9L1Yfw0g0t+fNjWhYks0tmWN3GJYQjWQ7twbHMq3PX1KuT6ULcX10Z1nDkmzOfMoMigHEkVtMRO4FbsDzh8wHbnAyw0gZr32d+szFvfdqQVuJXJq6raxOvY+lKoW4UpncMbuFGRZjB/GEIp8KdFfVUgARGYXnQB+STMUMw09piuP9crKDvHr1EWy5rw2NCpeVOy7pSJoYLZPw23+EcQPLbmAMzW4kAzQFMz6/n8UwiC8rMkCO73mzZChiGNEIhRqniuxggKH99geg0cl3ejdOP+lKmhhtFqIl7JjJ5P3ZMzah2U1CDItAp6PLvw8hLLTYiEA8M5d7gZkiMhkvHPkobNZiJBn/ZslUUq7EcOiGWROinqIV4vKT6OgvCez0o1TW32PUayoscww7Nj8eimdcvlbV35KtWKqxMsc1h0jliFNBTnaQWXeclNJzVoj/hp7d3Ku+mIzwYfBmJrYvxagk0cocx7UspqrLVHW8qr5TFw2LUbOItFkyFaxP9CZM/473hw+I7XiP1t/vwC9cA8nKQRBa2qqoyJdhxEmqE1caRkzyZhZQkKZyxG1zovgUqkK89U+iZRSOlo24NAlGN+RDilTkyzCqiBkXI62EfCupNCgZ4qVuKfKFoGUHAwzq0yVxJ4kW2eWvfxLJAOX9GUSSszcl2BiKtlgGYiMlxGVcRCQA7OZvr6qp33hg1CnyZhYw6M3ZFJWkNs74ofO6A0TNrpwQokV2+eWRDFDS/CmN4W9LK25nGAmiQuMiItfhJZ1czs4FXwXqSG5xI1WEp8v/bX0hKbYrCDurRFbLmIQ72qHsTCBWZNfQFEfzB7Ph9EdSe06j3hPPzOUGoIuqrk62MkbdJTwCrFb7VcKXs/xZhdcvgXFXQ6BBlM4ptqZWb95IE/EYlyXA+mQrYtRthr07Ly0RYOEkxK8STyGtkm3VP08iuOm7itsYRhKIalxE5Gb39Cdgioi8D+z4j1HV8DT8hhGR2/PmsnZLamutRKJ5o2Bi/CqpzNdVHZq1r7iNYSSJWDOXpu7vYvfIcg9I+dzeqK3cnjeXV75Kf+xHdjDAHafvX7lO0Xakx7NTPqkIoK4g1zYoilBqOF0pagzDEaueyzAAETlXVd/wHxORCotxicjzwGnAClU9wMlaAGOAjniVKM9T1bUiIsCjwCnAFuDyUKVLERkA3O6GHa6qo5y8B/AikI1XTOwGVdVo56hIXyPx5M0s4NU0GxaBqkWDRQoTHjcQFn/l3bT9x1JJsLHnnA/3oVhqFqOGUWH6FxH5VlUPqUgWod9RwCbgJZ9xuR9Yo6ojRGQw0FxVbxWRU4Dr8IzL4cCjqnq4MxTTgZ54s6UZQA9nkL7BCzb4Cs+4PKaqE6Kdo6I3wtK/JI507F0JJ5ghPHDuQeUNSvhNuPNJ8MOH5W/KDx8QZXYi0PNKmD068owhmfS8Ck6z1WijZhEt/Ussn8vJeDf7XBF5zHdoF7w69zFR1U9FpGOYuD9wjHs+CpgC3OrkL6ln6b4SkRyXz+wYYJKqrnE6TQL6isgUYBdV/dLJXwLOACbEOIeRAtKRFyw3J5tj923N5O9Xxt63Emk24q/I6N9FH9Wvosmr4hgJf+JIw6hFxPK5LMWbNfTDmzGE2AjcVMXz7aaqy8DLVyYiuzp5Ll5UWoh8J4slz48gj3WOcojIQGAgQIcOHap4ScbteXN5/esllMSRBDWRXNKrA8PP6BZ/h3iivEK76LMawfYUz0zCCWRB/8fNsBi1klg+l9nAbBF5TVWTHeojkVSogrxSqOpIYCR4y2KV7V8fCd8I2bFlNp//uKbijgmm914tKmdYIP4or7Q66x3ZLbziW2ZYjFpKPPtcvhWR8BvverxZzfBKbq5cLiJt3IyiDbDCyfMBf9xkO7yZUz47l7hC8ilO3i5C+1jnMKpJpI2QqfarlKu3UhlqwmwkJgJnjTSDYtQJ4km5PwF4H7jYPd4FPgN+w4vWqgzjgQHu+QDgHZ/8MvHoBax3S1sTgZNEpLmINAdOAia6YxtFpJeLNLssbKxI5zCqSbpS4YfIzcnm88HHVc2wvHdzzTYs2S3MsBh1inhmLr1Vtbfv9VwR+VxVe4vIJdE6icjreLOOViKSj5efbAQwVkSuwts7Ewpp/gAveGARXijyFQCqukZE7gKmuXZ3hpz7wJ/YGYo8wT2IcQ6jmqS6KqSfHVmL4wm5DW/TYk/4+ZP0KB4NW/Yy6jjxhCLPBgaq6tfu9WHAM6p6kIjMVNWDU6Bn0rFQ5PKE+1e2bC9Oy077HUthgc/L7y8Jr5YYHhFW08hqDKdF2KdiGLWUSoci+/gD8LyINMFzpG8A/iAijYF7E6umUVNId6LJnOwgQ/vtX3YJ7OE4aqRMuLVmGRYLJTbqKRUaF1WdBnQTkWZ4M511vsOVrNtq1ATCZySRHOTp9K88cn73svrsWOaKEsUVigKbM7ZshuJ0IRmgpZaR2KjXxFPPpQFwNl46lUzPfw6qemdSNTOSQqQZyZBxc4Gy9U3S5V/Jzckub1gqXObS1NdIiYTtoDeMHcSzLPYOXujxDHxZkY3aSaQZSWFRCQ9MXFjmpt4oK8Dm7amduQQDUj4lfjwbH9OFZMCZT9vMxDAiEI9xaaeqfZOuiZESos1ICtYV0nvExzuWylJtWDIEHjgnQi6wmpre3qK9DCMm8RiXL0Skm6rOTbo2RtJpm5Md0Tkv7HTap9p5nx0McO9Z3SLvX0l7evswMoJwxhNmVAyjAuLZRHkkMENEForIHBGZKyJzkq2YkRwG9elCdjBQTp6u3Dc52cGyhmXOWC8j8dAc72+LPdOkWQSyW5hhMYw4iWfmcnLStTBSRugmnuiU+P0ypnJL5ljayiqWaivuL/ZuwOGy8aVHAl5VyDtODws1fu/m8lmK0zFrsfBhw6g2FW6iBBCRI4HOqvqCiLQGmqjqz0nXLoXUt02UiawQ2S9jKiOCz9JItu+QbdMAgpAlO6szbJcGZJ3578g37TljYdzVCdGnWljEl2FUimibKOPZoX8HXrGuLqq6j4i0Bd4ISwlT66kPxiVvZgF/e3tuwp31U7Oup13Gqsp1ymrsrcXtKLjlSvemA8mAHleYUTGMKlCdHfpnAgcD3wKo6lIRaZpg/YwkkzezgL++MZuS0sTewPtlTCVXKmlYIEISyRQbllb7wl++Tu05DaMeEY9x2e5q0yuAS/ti1DIemLhwh2GJ5B8J+UIqQ7+MqTwQfBqJVF2nJtKgGQxJzFKgYRixice4jBWRp4EcEbkauBJ4JrlqGYnkxIem7HDeh/tH2skqRgSfhSLiMjB+w1RKBplSmlTdq02no2HA+HRrYRj1jnhyiz0oIifiJazsAvxDVSclXTOj0oTnDDt239a89tVi/Lf/WzLHlnG8AzSS7QwKjK3QuAzLfJ5LA/8lw81UMqjBhsWMimGklXhmLjhjYgalBhMpZ1ikaLC2Efwjv2lz7ii5vJzcP0tZRxOas6nmL4FZtJdh1AiiGhcR2UhkL6sAqqq7JE2rekY8WYorIp4sxv0yplJKxo4ZhyqMKTmGu4svYXvYVyF8+awFmyqlT8qxdCyGUaOIalxU1SLCUkDezAKmvv0EYxhN2warWLqlFY+8fQHw50oZmIqyGIeMRchHsqS0NYOLr+bz0gM4TBbQK+M7zgt8tsPJ30i2lls+q5HYTMUwaiRxLYslGhH5BdgIlADFqtpTRFoAY/BS+/8CnKeqa8XL8f8oXhnkLcDlqvqtG2cAcLsbdriqjnLyHuwsgfwBcIPGs1s0Dcx6fyR3ysgyDvY7dST3v5/JGQcPi9gn0kwnp1EwZpXIkK+lVIWXSk7kvuILCFDKXZnPU6xwfuanZXSoke+WZEDDHChcG73EsWEYNYK0GBfHsarqdwAMBj5S1REiMti9vhUv/Uxn9zgceBI43Bmj0AZPxct/Nl5V17o2A4Gv8IxLX2BCai6rcvxh+ys0yijvYP/D9leA8sYlkm/lxjGzgNghxm1lFT+WtuHWooFM1y4ckzGLe4LPsjtrWauNy81SapRvxfakGEatI53GJZz+wDHu+ShgCp5x6Q+85GYeX4lIjoi0cW0nqeoaABGZBPQVkSnALqr6pZO/BJxBDTUubTNWV0oezbfSL2MqDwZH7ki30k5W8WBw5I4Q4/uLL+D5kr5ks51/Bp/krIzPEIE12oQWEtmfoppmI2OVHA2j1pIu46LAh25j5tOqOhLYTVWXAajqMhHZ1bXNBfzZC/OdLJY8P4K8HCIyEG+GQ4cOHap7TVVia/buNCpcFlkeoX0038rQ4Etl8ngBZEkxFwU+4pPSg3iqpB99M77hzuAL7CrrAShVlYKFLwAADxpJREFUyKIoqgFJi2Exx7xh1AnSZVx6uzQyuwKTROT7GG0j3eK0CvLyQs+ojQQvt1hslZNDo5PvpPid68gs2bpDVhxoyLz9buJGV7wrp1EQVVhfWESGCKfKZzuWv9ZqE0SgeVg013YN8HjxGTxe0p9mbOaJ4COcEvimTJsMgcZaE4qLCpw10gyKYdQh0mJcVHWp+7tCRN4GDgOWi0gbN2tpA6xwzfOB9r7u7YClTn5MmHyKk7eL0L5mcuB53ofw0Z1e1cVm7Zi513VcNm0PCl15X7+j/lT5rEyIcMsIS1qzS/fklqKBLNQOnJnxGf8IvkzzKEtfaV32so2OhlFnSblxcbnJMlR1o3t+EnAnMB4YAIxwf99xXcYDfxGR0XgO/fXOAE0E7hGR5q7dScAQVV0jIhtFpBfwNXAZ8K9UXV+VOPC8Mr/abxzx8Q7DEk6kHfYhtmqQh4vP4ZmSU9mVtTwXfIDjAzOTonKVsdBhw6gXpGPmshvwthdhTCbwmqr+R0Sm4eUxuwpYDJzr2n+AF4a8CC8U+QoAZ0TuAqa5dneGnPvAn9gZijyBGurM9xMKL66ogFekHfYA35R24daigfysbbgw8DFDMl9lF0ltueKYmC/FMOoVKTcuqvoTcFAE+Wrg+AhyBa6NMtbzwPMR5NOBA6qtbIrIm1nAoDdnU1Si5VKuqEJz2bQjtHiptqKdz8Bs1gbcX3wBo0r60F5W8GrwbnoH5qXxanzYspdh1FtqUihyvWXYu/N2GJZyKVecT6SdrOLR4BNspiFFKgRF+azkAAYXXc1SWnJFYAKDMsfSSGqAg95CiA2j3mPGpQawdksRwzKf57LAf2M62EWgCVtZX9qIu4svZmzJsewpS3kjeCc9M/6XOoUjYRsdDcPwYcalBhCPYQkxqeQQ/lZ0FavZhT8F3uGGzHE0lOhpX5KOBKDH5eakNwyjDGZc0szFz3zJqMDHFRqW1dqUYUWXMb60N/vKrzwXfJBuGT+nRsky2J4UwzAqxoxLEomWSj9vZgFDx89jXaE34wg0iF50SxXeK+3FHUWXs5FG3Jz5Bn8MjCdLYqfXTzjmnDcMoxKYcUkSkVLp//Ot87lt3O/ZUlTqRYVljSU3SmgxwHLN4faiK5lU2pOD5EfuDz5Nl4z8qO0Tju1JMQyjiphxSRKRUunfLc9QUqSQQZmosHBU4Y2So7mr+BK2E+S2zFe5MjAhNfXqJQh3RDd4hmEY8WDGJUlES6X/aPAJIHralXxtxZCiP/BZ6YEcJgsYEXyGPTN+S7a6Fu1lGEZCMeOSDOaMJTcj8q//aEalVIVXS45nRPGFKMJdmc9zceAjMiSJ+TSbtIH/i5Uz1DAMo2qYcUk0c8ZS8vafCFSiy8+lu3Nr0dV8o/vx+4w53JP5LO2jGKeEMHR98sY2DMPAjEvC2TLhHzTS4oobAiUqPFdyCv8sPpcsirg/82nODXySnEzFFu1lGEYKMeOSYBoWxucf+V9pLoOKrmG27s0JGdO5O/g8u8m6xCt01jO2J8UwjJRjxiXBFCtkxZh5FGmAJ0v68a/iM2nKFh4L/ovTM75M7Gzl/9u79xipyjOO49/f7gKaahEFqSIWNDTKHw2lYLG1XlqLl7ZBW6tIjVpNMFovvWiC0dht//BSo4lGUwut9X6vRuslQCu0piAiCohSBFErSsQrQk2twNM/3nd1GGdmYTmzM7v8PsnJnHnPmXOe992Zefa858x73O1lZg3m5FKgH0+by22Vb3oJwJJNw7jg48ksjWF8v2UO7X1uZjetK2bnPjlvZk3EyaUg97Yfz20xveKy/0YfrtnwA6Zu/B678QFT+1zF+NYFxezYRylm1oScXAow99pTOS6mV+zaWrBpBBd8fAYrY08mtc7k4rY7tn1YfJ+cN7Mm5+RSgLHvPPiZxPJh9OPKDcdz08bxHKpF3Nr3Uoa0vFt5A1UEn9zOxb+cN7MepdcmF0lHAtcArcAfIuLyeu2rlc2HZZmzcSSXbpjE2JZlzO13Ll/Qe13arsDdXmbWI/XK5CKpFbge+A6wCpgv6aGIeKEe+9tIC21s4oPYgfs3HsIgvccDfX9Fn66OXOyEYmY9XK9MLsABwIqIWAkg6S5gAlCX5DJ/twmMe+cBlsdQTm2bzofRD9W4aqzDZt1etEB7145wzMyaTW9NLkOA10qerwK+Vr6SpMnAZIC99967yzs78NybWNs+g4Gb3uflTYMZ3vJm1XUjPh1fzN1eZtZb9dbkUukniZ85lIiIqcBUgDFjxmzTCJH9299g5/b+VBtnMvJhioQTipn1er01uawChpY83wt4o947bWlfC+39Ky7TDz0Mi5ltP3prcpkPjJA0HHgdmAhM6pY9+6jEzKx3JpeI2CDpbGA66VLkGyPi+QaHZWa23eiVyQUgIh4FHm10HGZm26OWRgdgZma9j5OLmZkVzsnFzMwK5+RiZmaFc3IxM7PCObmYmVnhFLFNo570GpLeAl4tYFMDgZ564xXH3hiOvTEcezG+GBGDygudXAom6emIGNPoOLrCsTeGY28Mx15f7hYzM7PCObmYmVnhnFyKN7XRAWwDx94Yjr0xHHsd+ZyLmZkVzkcuZmZWOCcXMzMrnJNLQSQdKWmZpBWSpjQ6ng6SXpH0nKSFkp7OZbtKmilpeX4ckMsl6dpch8WSRpds55S8/nJJp9Qp1hslrZG0pKSssFglfTW3xYr82kq3wy4y9nZJr+e2Xyjp6JJlF+Y4lkk6oqS84vtI0nBJ83Kd7pbUt8DYh0qaJWmppOclnZfLm77ta8Te9G0vaQdJT0lalGP/da39SeqXn6/Iy4d1tU7dIiI8beNEuiHZS8A+QF9gETCy0XHl2F4BBpaV/RaYkuenAFfk+aOBxwAB44B5uXxXYGV+HJDnB9Qh1oOB0cCSesQKPAUcmF/zGHBUnWNvB86vsO7I/B7pBwzP753WWu8j4B5gYp6/ATizwNj3AEbn+Z2BF3OMTd/2NWJv+rbPbbFTnu8DzMvtWXF/wFnADXl+InB3V+vUHZOPXIpxALAiIlZGxP+Au4AJDY6plgnAzXn+ZuCYkvJbInkS2EXSHsARwMyIeDci3gNmAkcWHVRE/AN4tx6x5mWfj4i5kT6Rt5Rsq16xVzMBuCsiPoqIl4EVpPdQxfdR/i//W8B9+fWl7VBE7Ksj4pk8vw5YCgyhB7R9jdiraZq2z+23Pj/tk6eosb/Sv8d9wLdzfFtVpyJi3xJOLsUYArxW8nwVtd/g3SmAGZIWSJqcywZHxGpIH05g91xerR6NrF9RsQ7J8+Xl9XZ27jq6saNbqZMYK5XvBrwfERvKyguXu1q+Qvovuke1fVns0APaXlKrpIXAGlIyfqnG/j6JMS9fm+Nrxs+tk0tBKvUfN8s13t+IiNHAUcBPJR1cY91q9WjG+m1trI2ow++AfYFRwGrgqlzelLFL2gn4M/CziPig1qpV4mlY/BVi7xFtHxEbI2IUsBfpSGP/Gvtrqtg74+RSjFXA0JLnewFvNCiWzUTEG/lxDfAA6Q38Zu6qID+uyatXq0cj61dUrKvyfHl53UTEm/nLYxMwjdT2dBJjpfK3SV1PbfWKXVIf0pfz7RFxfy7uEW1fKfae1PY53veB2aRzLtX290mMeXl/UldsM35unVwKMh8Yka/y6Es62fZQg2NC0uck7dwxD4wHlpBi67iS5xTgwTz/EHByvhpoHLA2d4dMB8ZLGpC7F8bnsu5QSKx52TpJ43I/9ckl26qLji/m7FhS23fEPjFf/TMcGEE64V3xfZTPU8wCjsuvL22HIuIU8EdgaURcXbKo6du+Wuw9oe0lDZK0S57fETicdM6o2v5K/x7HAY/n+LaqTkXEvkW668qB3j6RrqB5kdRnelGj48kx7UO6QmQR8HxHXKR+2r8By/PjrrlcwPW5Ds8BY0q2dRrpROEK4Cd1ivdOUhfGx6T/uk4vMlZgDOlL5iXgOvIIFXWM/dYc22LSh3qPkvUvynEso+TKqWrvo/y3fCrX6V6gX4GxH0TqLlkMLMzT0T2h7WvE3vRtD3wZeDbHuAS4pNb+gB3y8xV5+T5drVN3TB7+xczMCuduMTMzK5yTi5mZFc7JxczMCufkYmZmhXNyMTOzwjm5mFUgaX3na231Nkdp89F52yWdvw3b+5HSaMCzysqHSZpUVJxmXeHkYtZ9RpF+d1CU04GzIuKwsvJhQJeTC8XHadshJxezTki6QNL8PAhixz03huWjhmlK9+KYkX9ljaSxed25kq6UtCT/Qvo3wAlK9xc5IW9+pKTZklZKOrfK/k9UuhfKEklX5LJLSD8gvEHSlWUvuRz4Zt7Pz/PgiFeW1OGMvI1jJf01/9J+D0kvStq7PE5Jh+jT+6I82zHqg1lN3fmLTU+eesoErM+P44GppF+ltwAPk+7dMgzYAIzK690DnJTnlwBfz/OXk+/xApwKXFeyj3ZgDuk+HAOBd4A+ZXHsCfwbGAS0AY8Dx+Rlsyn5dXzJaw4FHi55Phm4OM/3A54GhufntwFn53qdWCXOv5AGQAXYCWhr9N/HU/NPPnIxq218np4FngH2I43dBPByRCzM8wuAYXmsqJ0jYk4uv6OT7T8S6T4cb5MGhhxctnwsMDsi3oo0zPrtpOS2tXU4WWlo93mkYV066nAOcCHwUUTcWeX1/wSuzkdWu8Snw8GbVdXW+Spm2zUBl0XE7zcrTPcO+aikaCOwI5WHOa+lfBvln8kibgcs4JyIqDTY6BBgEzBYUkukUYQ3ExGXS3qEdB7mSUmHR8S/CojLejEfuZjVNh04Tel+IUgaImn3aitHugPjujxaMKSRaDusI92Kd2vMAw6RNFBSK3Ai8PdOXlO+n+nAmUpD0yPpS0ojZrcBfyKd/F8K/KLS6yXtGxHPRcQVpC61/bayDrYd8pGLWQ0RMUPS/sDcNLo764GTSEcZ1ZwOTJP0H9J5kbW5fBYwJXdPXbaF+18t6cL8WgGPRkRnQ74vBjZIWgTcBFxDOkf0TB6i/i3SrXN/CTwREU/kmObnI5TyOA+SdFiu8wvAY1sSu23fPCqyWcEk7RT53uiSppCGez+vwWGZdSsfuZgV77v5aKMNeJV09ZXZdsVHLmZmVjif0Dczs8I5uZiZWeGcXMzMrHBOLmZmVjgnFzMzK9z/AZbkNtrvyFRpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sorted_len_tokens_text_list = sorted(len_tokens_text_list)\n",
    "y_len_tokens_en_list = (12*np.array(sorted_len_tokens_text_list)).tolist() \n",
    "y_len_tokens_pt_list = (7*np.array(sorted_len_tokens_text_list)).tolist()                             \n",
    "\n",
    "ax = plt.subplot(111)\n",
    "ax.scatter(len_tokens_text_list, len_tokens_en_list)\n",
    "ax.plot(sorted_len_tokens_text_list, y_len_tokens_en_list)\n",
    "ax.scatter(len_tokens_text_list, len_tokens_pt_list)\n",
    "ax.plot(sorted_len_tokens_text_list, y_len_tokens_pt_list)\n",
    "\n",
    "ax.set_xlabel('length of texts')\n",
    "ax.set_ylabel('length of en and pt tokens')\n",
    "ax.legend(['en', 'pt'])\n",
    "\n",
    "ax.set_title('Number of tokens by tokenization method')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the use of a BBPE tokenizer trained with an English corpus on a corpus of another language (here, Portuguese) **requires on average almost 70% of additional tokens** (66% exactly) than a BBPE tokenizer trained with the same language of the corpus of application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## END"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}