{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c2231055-6a63-4425-9248-c5aae455396e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/logo_final.png\"  align=\"left\"/> \n",
    "</center>   \n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4ce5373-7375-4e11-872a-d58813d66c24",
   "metadata": {},
   "source": [
    "# <span style=\"color:red\"><center>BERT-Español</center></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30115409-9d8e-45df-a9cd-072858c02397",
   "metadata": {},
   "source": [
    "<center>Explorando modelos pre-entrenados</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23484740-ab13-447a-bc3f-01686f528f49",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Autores</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405456a4-e556-41d2-bf60-593a5055f8b4",
   "metadata": {
    "tags": []
   },
   "source": [
    "1. Alvaro Mauricio Montenegro Díaz, ammontenegrod@unal.edu.co\n",
    "2. Daniel Mauricio Montenegro Reyes, dextronomo@gmail.com "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7aaec87-e77d-4152-bb83-bc9e62d4a94c",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Diseño gráfico y Marketing digital</span>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aac3be8e-e519-43bf-9bd9-9903a27b570b",
   "metadata": {},
   "source": [
    "1. Maria del Pilar Montenegro Reyes, pmontenegro88@gmail.com "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80e4663a-f0f2-42d8-a044-6e3ab6d16e3e",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Asistentes</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b038974-8718-4142-a66e-895520f68ca7",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0a15511c-982d-4ff7-96fd-a6f1c95f39a1",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Referencias</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "190d6ac6-19de-4dcf-8904-a9e158c84d34",
   "metadata": {},
   "source": [
    "1. [HuggingFace BERT model](https://huggingface.co/transformers/model_doc/bert.html)\n",
    "1. [Getting Started with Google BERT: Build and train state-of-the-art natural language processing models using BERT](http://library.lol/main/A0CA3A1276D07957FD7B28F843C299BA)\n",
    "1. [Transformers for Natural Language Processing: Build innovative deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, RoBERTa, and more](http://library.lol/main/A8C97E552646B3F194ECA333221CEE88)\n",
    "1. [HuggingFace. Transformers ](https://huggingface.co/transformers/)\n",
    "1. [HuggingFace. Intro pipeline](https://huggingface.co/course/chapter1/3?fw=pt)\n",
    "1. [Tutorial Transformer de Google](https://www.tensorflow.org/text/tutorials/transformer)\n",
    "1. [Transformer-chatbot-tutorial-with-tensorflow-2](https://blog.tensorflow.org/2019/05/transformer-chatbot-tutorial-with-tensorflow-2.html) \n",
    "1. [Transformer Architecture: The positional encoding](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/)\n",
    "1. [Illustrated Auto-attención](https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a)\n",
    "1. [Illustrated Attention](https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3#0458)\n",
    "1. [Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et. al, 2015)](https://arxiv.org/pdf/1409.0473.pdf)\n",
    "1. [Effective Approaches to Attention-based Neural Machine Translation (Luong et. al, 2015)](https://arxiv.org/pdf/1508.04025.pdf)\n",
    "1. [Attention Is All You Need (Vaswani et. al, 2017)](https://arxiv.org/pdf/1706.03762.pdf)\n",
    "1. [Self-Attention GAN (Zhang et. al, 2018)](https://arxiv.org/pdf/1805.08318.pdf)\n",
    "1. [Sequence to Sequence Learning with Neural Networks (Sutskever et. al, 2014)](https://arxiv.org/pdf/1409.3215.pdf)\n",
    "1. [TensorFlow’s seq2seq Tutorial with Attention (Tutorial on seq2seq+attention)](https://github.com/tensorflow/nmt)\n",
    "1. [Lilian Weng’s Blog on Attention (Great start to attention)](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html#a-family-of-attention-mechanisms)\n",
    "1. [Jay Alammar’s Blog on Seq2Seq with Attention (Great illustrations and worked example on seq2seq+attention)](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/)\n",
    "1. [Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (Wu et. al, 2016)](https://arxiv.org/pdf/1609.08144.pdf)\n",
    "1. [Adam: A method for stochastic optimization](https://arxiv.org/pdf/1412.6980.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17e9be15-8ae8-4715-b57f-a6e8a8111c0c",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Contenido</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a382773-c33f-4247-b44c-6e3806b9af20",
   "metadata": {},
   "source": [
    "* [Introducción](#Introducción)\n",
    "* [Extracción de incrustamientos de un modelo BERT pre-entrenado](#Extracción-de-incrustamientos-de-un-modelo-BERT-pre-entrenado)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3c46451-72e1-4588-b271-855af1898df1",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Introducción</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "192cd6c6-0240-4bd0-b2e6-427aff064ebb",
   "metadata": {},
   "source": [
    "Usaremos la implementación de HuggingFace in en Pytorch. \n",
    "\n",
    "+ BERT es un modelo con incrustaciones de posición absoluta, por lo que generalmente se recomienda rellenar (padding) las entradas a la derecha en lugar de a la izquierda.\n",
    "\n",
    "+ BERT fue entrenado con el modelado de lenguaje enmascarado (MLM) y los objetivos de predicción de la siguiente oración (NSP). Es eficiente para predecir tokens enmascarados y en NLU en general, pero no es óptimo para la generación de texto."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b86ab415-c6d7-4d89-bbae-098b30f95578",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Extracción de incrustamientos de un modelo BERT pre-entrenado</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cd69e7e-8b83-4a1b-ad59-8d5f01381baa",
   "metadata": {},
   "source": [
    "La tarea de PLN es análisis de sentimiento. El primer experimento lo hacemos en  Español. Para esta tarea esta bién usar el modelo uncase (eliminando mayúsculas). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d93f08d4-ed19-46ef-88cb-f79056f932de",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertModel, BertTokenizer\n",
    "import torch\n"
   ]
  },
  {
   "cell_type": "raw",
   "id": "090081a2-b0e4-4345-aff4-b6cd79b6eb3c",
   "metadata": {},
   "source": [
    "## Tensorflow\n",
    "from transformers import TBertModel, TBertTokenizer\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e483a0f6-52b5-439c-96fc-ba7bc99cc4e4",
   "metadata": {},
   "source": [
    "### Cargamos el modelo pre-entrenado 'bert-base-uncase' y su respectivo tokenizador"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b0d6e7aa-c090-4cff-80ce-f85de557c3cc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "15602b925db14a84b4dde755aa021139",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/650 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0486ec8faa6c4f4b9adc29394e84dd8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/440M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at dccuchile/bert-base-spanish-wwm-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.bias']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertModel were not initialized from the model checkpoint at dccuchile/bert-base-spanish-wwm-uncased and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ccedf76f949f4f7d89185d8179c79334",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/248k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc026ad7a9244e0aa3cb071f985b329b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/134 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "36f7a65ec3494521ac81accce6d341b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/310 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "21f0922616554c2992181a22b623321d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/486k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = BertModel.from_pretrained('dccuchile/bert-base-spanish-wwm-uncased')\n",
    "                                 \n",
    "tokenizer = BertTokenizer.from_pretrained('dccuchile/bert-base-spanish-wwm-uncased')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f85fb650-c2c7-4287-aa32-fa84acb2c658",
   "metadata": {},
   "source": [
    "### Preprocesamiento de la entrada"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "40e00063-d712-494c-9bc5-39cf1bd332bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['yo', 'amo', 'a', 'bogotá']\n"
     ]
    }
   ],
   "source": [
    "# sentencia\n",
    "sentence = 'Yo amo a Bogotá'\n",
    "\n",
    "# tokenización\n",
    "tokens = tokenizer.tokenize(sentence)\n",
    "\n",
    "# print\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77d5985b-8bfd-4fa0-8255-0dce67066768",
   "metadata": {},
   "source": [
    "### Agregamos los tokens [CLS] al comienzo y [SEP] al final de la lista de tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f48fc4af-645a-4dc2-a73d-07c25f3623a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'yo', 'amo', 'a', 'bogotá', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "tokens = ['[CLS]'] + tokens + ['[SEP]']\n",
    "\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "396b2885-baed-4daf-8504-da040e19e927",
   "metadata": {},
   "source": [
    "###  Relleno y máscara para la sentencia"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df8ec922-bb09-4452-b387-9a44e2a6cdc5",
   "metadata": {},
   "source": [
    "El tamaño de la lista de tokens es 5. Supongamos que hemos decidido que el tamaño máximo se las sentencias será 7. BERT está constuido para aceptar sentencias hasta de tamaño 512. Todas las sentencias deben tener el mismo tamaño."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e144cfa0-e684-4f55-8fc7-d30317821fae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'yo', 'amo', 'a', 'bogotá', '[SEP]', '[PAD]']\n"
     ]
    }
   ],
   "source": [
    "## Relleno\n",
    "max_sentence_size = 7\n",
    "pad_size = max_sentence_size - len(tokens)\n",
    "\n",
    "for i in range(pad_size): \n",
    "    tokens = tokens + ['[PAD]'] \n",
    "\n",
    "print(tokens)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "315fd135-f97a-4453-a658-8972d56d5c75",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 1, 1, 1, 1, 1, 0]\n"
     ]
    }
   ],
   "source": [
    "## máscara de atención\n",
    "attention_mask = [1 if i!= '[PAD]' else 0 for i in tokens]\n",
    "print(attention_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad09897-d398-4f44-8431-a554a2d83972",
   "metadata": {},
   "source": [
    "### Convertimos la lsita de tokens en la lista de ID de los tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6fbea2a3-320d-4dd7-8521-d1af470d3d45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4, 1252, 4017, 1012, 14548, 5, 1]\n"
     ]
    }
   ],
   "source": [
    "token_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
    "print(token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f15280fd-00da-45ca-a450-a7a3ad1d69f5",
   "metadata": {},
   "source": [
    "### Convertimos token_ids y attention_mask a tensores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6e6e7bd2-4899-4cb9-9d8c-5b9c8d8ed639",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[    4,  1252,  4017,  1012, 14548,     5,     1]])\n"
     ]
    }
   ],
   "source": [
    "token_ids = torch.tensor(token_ids).unsqueeze(0) # unsuezze es para agregar una dimensión al comienzo (varias sentencias)\n",
    "attention_mask = torch.tensor(attention_mask).unsqueeze(0)\n",
    "print(token_ids) # tensor([[ 101, 1045, 2293, 3000,  102,    0,    0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "675844ed-0528-44c2-86eb-b3f6c59742df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[    4,  1252,  4017,  1012, 14548,     5,     1]])\n",
      "tensor([[1, 1, 1, 1, 1, 1, 0]])\n"
     ]
    }
   ],
   "source": [
    "print(token_ids)\n",
    "print(attention_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86126d72-8263-4d14-93b2-37d269bc10fb",
   "metadata": {},
   "source": [
    "### Pregunta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5c9ca04-b524-4655-aa73-899edad4d1f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "¿Cómo hace esto con tensorflow?"
   ]
  },
  {
   "cell_type": "raw",
   "id": "a9a18715-4f3b-4759-b455-1c6339eb17e8",
   "metadata": {},
   "source": [
    "# Consigne aquí su respuesta\n",
    "import tensorflow as tf\n",
    "token_ids = tf.expand_dims(tf.constant(token_ids), axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a587dc32-086e-4b90-aaa5-e786350cd427",
   "metadata": {},
   "source": [
    "### Extracción del incrustamiento final"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb47b7a9-de74-4348-8792-8e27c6725df3",
   "metadata": {},
   "source": [
    "model regresa una lista con dos objetos: \n",
    "\n",
    "* El primer valor, *last_hidden_state*, contiene la representación de todos los tokens obtenidos solo de la capa del codificador final (codificador 12).\n",
    "* A continuación, *pooler_output* indica la representación del token [CLS] de la capa codificadora final, que se procesa posteriormente mediante una capa lineal y una activación *tanh*. La capa lineal es entrenada cuando se entrena el modelo BERT para la tarea NSP (Next sequence prediction).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0613566b-a5a5-410e-86ec-4a8b021315d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "out = model(token_ids, attention_mask = attention_mask)\n",
    "last_hidden_state, pooler_output = out.last_hidden_state, out.pooler_output # out[0], out[1]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6d52ab29-9b0d-416e-b0f0-823ec5e74168",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 7, 768])\n"
     ]
    }
   ],
   "source": [
    "print(last_hidden_state.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4b228c0-26b1-4371-8ca1-d9a4371f2c03",
   "metadata": {},
   "source": [
    "* Tamaño batch = 1\n",
    "* Tamaño secuencia = 7\n",
    "* tamaño del emebdding = 768"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e6061579-ecac-4724-a27b-c3d74896cc42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['last_hidden_state', 'pooler_output'])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# out es un diccionario. Podemos obtener las claves  así:\n",
    "out.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1287a8b-9a09-4b2f-8fc2-5c6d4ffe5f77",
   "metadata": {},
   "source": [
    "### Extracción de los incrustamientos de todas las capas codificadoras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5e43f16-437b-419c-b6e5-61f1ca3f1333",
   "metadata": {},
   "source": [
    "En esta sección revisamos como extraer las incrustaciones (embeddings) que salen de cada una de las capas codificadoras (12 por ejemplo en el modelo base). Algunos veces estop se hace para extraer diferentes features de las sentencias. \n",
    "\n",
    "Por ejemplo en la tarea NER (name entity recognition) los investigadores han usado las incrustaciones de las diferentes capas, para hacr promedios pesados de algunas de ellas y con esto han podido mejorar la exactitud en la precisión.\n",
    "\n",
    "Para hacer esto, es necesario instanciar el modelo preentrenado con la opción *output_hidden_states=True*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8b14fb0-e508-4164-b9a5-cbae7e9db8ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = BertModel.from_pretrained('dccuchile/bert-base-spanish-wwm-uncased', output_hidden_states=True)\n",
    "tokenizer = BertTokenizer.from_pretrained('dccuchile/bert-base-spanish-wwm-uncased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9cd791bc-ec6a-43f3-8deb-e6fa50c7df6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "out = model(token_ids, attention_mask=attention_mask)\n",
    "\n",
    "last_hidden_state, pooler_output, hidden_states = \\\n",
    "        out.last_hidden_state, out.pooler_output, out.hidden_states\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2916d80b-0f59-48fb-b349-b7c986ab065c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 7, 768])\n",
      "torch.Size([1, 768])\n",
      "13\n"
     ]
    }
   ],
   "source": [
    "print(last_hidden_state.shape)\n",
    "print(pooler_output.shape)\n",
    "print(len(hidden_states)) # esta es una lista conteniendo las\n",
    "                          # incrutaciones de todas las capas codificadoras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc6d576f-40d0-4a9a-a56d-159b0d01c42f",
   "metadata": {},
   "source": [
    "Observe que *hidden_states* tiene 13 elementos. La capa 0 corresponde a la incrustación de la capa de entrada, luego los elementos 1 a 12 corresponden a las incrustaciones de de salida de cada una de las 12 capas codificadoras.\n",
    "\n",
    "\n",
    "La representación de los token de la última capa oculta (codificadora) pueden ser obtenidos así:\n",
    "\n",
    "* *last_hidden_state[0][0]*: entrega la representación del prime token, es decir, *[CLS]*.\n",
    "* *last_hidden_state[0][1]*: entrega la representación del Token *I*.\n",
    "* *last_hidden_state[0][2]*: entrega la representación del Token *love*. \n",
    "\n",
    "Esta es la representación contextual final de los token. \n",
    "\n",
    "Las incrustaciones de cada capa *i*, se obtienen mediante *hidden_states[i]:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "dc65d589-ec50-468f-b440-10fdf46a4932",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 7, 768])\n",
      "torch.Size([1, 7, 768])\n"
     ]
    }
   ],
   "source": [
    "# Incrutaciones de la capa de entrada\n",
    "input_embedding = hidden_states[0]\n",
    "print(input_embedding.shape)\n",
    "\n",
    "# incrustaciones de la capa codificadora 11\n",
    "embedding_11 = hidden_states[11]\n",
    "print(embedding_11.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5555e4d2-9e50-4592-9ae7-e7ccc6df3645",
   "metadata": {},
   "outputs": [],
   "source": [
    "help(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "372442e1-0221-4d21-95da-9d0058ea7070",
   "metadata": {},
   "source": [
    "### Recuperando los pesos de atención"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "176cad80-144e-486a-a805-8dc09569c275",
   "metadata": {},
   "source": [
    "Los pesos de  atención después de la atención softmax, se utilizan para calcular el promedio ponderado en las cabezas de  autoatención. \n",
    "Son obtenidos pasando al modelo *output_attentions=True*\n",
    "\n",
    "+ *output attention* es una tupla. Cada elemento coresponde a los pesos de atención de cada capa codificadora."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d18b8954-dae6-4721-a0ec-7a0ca3ce5d9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = BertModel.from_pretrained('dccuchile/bert-base-spanish-wwm-uncased',\\\n",
    "                                  output_hidden_states=True, output_attentions=True)\n",
    "tokenizer = BertTokenizer.from_pretrained('dccuchile/bert-base-spanish-wwm-uncased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0efd6e94-5101-473f-a9b3-24efa41b12a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12\n"
     ]
    }
   ],
   "source": [
    "out = model(token_ids, attention_mask=attention_mask)\n",
    "\n",
    "last_hidden_state, pooler_output, hidden_states, attentions = \\\n",
    "        out.last_hidden_state, out.pooler_output, out.hidden_states, \\\n",
    "        out.attentions\n",
    "print(len(attentions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "be0b076f-cadd-4c72-96f7-12dc08906dd1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 12, 7, 7])\n"
     ]
    }
   ],
   "source": [
    "print(attentions[11].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6f09a46-8acc-46b4-aef3-b92b1fe0c2f9",
   "metadata": {},
   "source": [
    "La salida se explica así:\n",
    "\n",
    "- El tamaño del batch es 1. Una sentencia.\n",
    "- Son 12 cabezas de atención.\n",
    "- La sentencia viene de tamaño 7.\n",
    "\n",
    "Por lo tanto tenemos la salida de las 12 cabezas de atención para la sentencia.\n",
    "\n",
    "Vamos a darle una mirada a los pesos de atención de la última capa codificadora\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "58fabdd1-b720-4734-8acc-f10010d37a62",
   "metadata": {},
   "outputs": [],
   "source": [
    "attention11 = attentions[11].squeeze()#elimina la dimensión de batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c414704d-b8f9-4271-93db-721319905899",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([12, 7, 7])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attention11.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf6013b9-45bc-4d5f-b1cc-aa04837e9450",
   "metadata": {},
   "source": [
    "### Función para graficar pesos de atención de una cabeza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b99ba957-08ff-4553-a4e2-3d1aca5472a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.axes_grid1 import make_axes_locatable\n",
    "\n",
    "# versión con decode utf-8\n",
    "def plot_attention_head_cp(in_tokens, translated_tokens, attention):\n",
    "  # The plot is of the attention when a token was generated.\n",
    "  # The model didn't generate `<START>` in the output. Skip it.\n",
    "  translated_tokens = translated_tokens[1:]\n",
    "\n",
    "  ax = plt.gca()\n",
    "  ax.matshow(attention)\n",
    "  ax.set_xticks(range(len(in_tokens)))\n",
    "  ax.set_yticks(range(len(translated_tokens)))\n",
    "\n",
    "  labels = [label.decode('utf-8') for label in in_tokens.numpy()]\n",
    "  ax.set_xticklabels(\n",
    "      labels, rotation=90)\n",
    "\n",
    "  labels = [label.decode('utf-8') for label in translated_tokens.numpy()]\n",
    "  ax.set_yticklabels(labels)\n",
    "\n",
    "\n",
    "\n",
    "def plot_attention_head(in_tokens, translated_tokens, attention):\n",
    "  # The plot is of the attention when a token was generated.\n",
    "  # The model didn't generate `<START>` in the output. Skip it.\n",
    "  #translated_tokens = translated_tokens[1:]\n",
    "\n",
    "  ax = plt.gca()\n",
    "  pcm = ax.matshow(attention)\n",
    "  ax.set_xticks(range(len(in_tokens)))\n",
    "  ax.set_yticks(range(len(translated_tokens)))\n",
    "\n",
    "  labels = [label for label in in_tokens]\n",
    "  ax.set_xticklabels(\n",
    "      labels, rotation=90)\n",
    "\n",
    "  labels = [label for label in translated_tokens]\n",
    "  ax.set_yticklabels(labels)\n",
    "  \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f9c4698e-32b6-42ba-a90e-a68023d49177",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([7, 7])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "head = attention11[0]\n",
    "head.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a25408a7-86c1-4da8-9045-e2235583e2c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "head = head.detach().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0e583ceb-8a8a-47df-9633-b8857b7de318",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAREAAAEUCAYAAAAMWQI9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAW+ElEQVR4nO3de5RfZX3v8feH3EkggCB6jkBUblouQYYKFhQE8eABPdQWRGoRjuaorEUPPUo95awuBO1FbeP9kqoltSyKYkGrFimQAGq9RIEECBdBbkWRcJXcM/mcP/YeGcbJZJJn79+ezHxea83K/J7f3vv7TGbmM8/e+9l7yzYREVtru647EBHbtoRIRBRJiEREkYRIRBRJiEREkYRIRBRJiEREkYRIRBRJiEREkclddyCiFyTtBvwZ8HJg+kC77dd21qlxIiORGLckvUfScfXLS4C7gN2AC4H7gR931bfxJCES49lC4CRJfwDsansBsNr2YttnAod3273xISES45btlbb/BPgWsK5uflLS6yQdAuzRXe/GD+Uq3pgIJJ0I3Aj8V+BTwE7Ahbav7LBb40IOrMZE8YTtp4CngNcCSPq9brs0PmQkEhOCpJ/afsXm2sYTSbuMYrGNtp8sqZORSIxrko4AXgXsJulPB721IzCpm171zMP1h0ZYZhKwZ0mRCRcikr4xisUet/32tvsSPTEVmEX1s77DoPangT9os7CkT4xisadt/7+WurDc9iEjLSDpptIiE253RtLdwDtGWgT4tO3f6VGXogck7WX7fkk7ALb9TA9q3g/8xWYWe7/tl7VUf7rtNaXLbM6EG4kA59u+fqQFJH2gV52Jntmh/qu7C4CkFcAZtm9tseZ82wtHWkDSzm0VHwgHSQcC+9fNywd/zaUBAhNwJDKc+hv5pPOfMW5J+j7VH5BF9eujgb+0/aoe1N7V9oq26wxTdzbwdar5MEupRtkHAg8Ab7L9dBN1JtxkM0l/IWn/+vNpkhYB9wCPDJoiPW5JmirpgPpjStf96aGZAwECYHsxMLPNgpJOkvQosEzSQ5JaD6whLgKWAPvYPtn2/wD2oZru/6Gmiky4kYik24ADbFvSPOA04DhgX2Ch7d/ttIMtqv/6LgTuo/qrtAfVkP6G7nrVG5KuAH4KfLlu+iOgr/7FaqvmUuAU23dIeiXwYduvaaveMPVvBw6yvWFI+2RgWVPHYibiMZF1g3ZbXg/8s+1+YHn9nzue/S1wvO07ASTtC1wKHNppr3rjLOADwL9QBegNwJkt19xg+w4A2z+sD+r20rqhAVL3ZYOktU0VGe+/NMNZK+kA4BHgGOC9g95rdXg7BkwZCBAA23dNlF0a208A5/S47POHzE15zmvbf9dy/en1NUJD54kImNZUkYkYIn8CXE51Sfh82z8HkPQGquHueLZE0hd5dkh/OvCTDvvTM5L+FRi67/4U1TGDzzdxlmIYf89z56YMfd22XwCbCqpfNlVkwh0TGYmkN9v+Wtf9aIukacDZwJE8O6T/jO3GhrZjlaSPU/3huLRuOpXqF2kGsKPtt3XVty5ImmJ7fSPbSog8S9IDtoumAMfYJOkG268erk3SbW1MLpT0Fdun1J//je0/G/Te1baPb7rmZvojql34twIn2d69ie1OuFO8mzHSNQbbPEknSrpJ0uOSnpb0a0mNzBXYBuwm6Td/IOrPd61frht+lWL7DPr8dUP701LN3yLplfVI7H7gG1S3RNh/5LVGbyIeExnJeB+WfQz4farTe+P9ax3q/wDflXQP1R+LFwPvkTST6rR3G0b6P279/1/Sh4BTqCaXXUp1W8glm5tFu6UmXIhIWsbw30ABjQzvxrAHgVsnYIBg+9uS9qH6CyzgjkEHUz/WUtnt67Mj2wEzBp0pEdWxmLbNA+4EPgt80/YaSY1/7yfcMRFJe430vu37e9CH3YHD6pc/sv2rtmvWdQ+jmsV4PfCbg6k9ONXYufpU9ruBgeMii6nOyjRycHETNRczwojD9jFt1a7rTwKOp5pQ+VpgEdXEyj2Gmz+y1XUmYIjsDexu+3tD2o8CHrZ9T8v1TwE+QvVDLOAo4H22L2+zbl37auAZYBmwcaDd9ri/4FDSF4ApPLvr8jag3/ZIV3SPG5KmAydSBcqRwLW239rEtifc7gzV0PXPh2lfXb93Usv1zwcOGxh91M9DuYZq7krbdun1GYEx5DDbBw96fZ2kW9osWI/8HrT9y/r1HwNvpjrAeYHtx1uuPx14F7A31QV4X7J9uaQdgZObqjMRz87Msb10aKPtJcCcHtTfbsjuy2P07vtwjaSJGiL9kl468ELSS4D+lmt+nvrMj6RXA38N/CPVJLcFLdeGatTVRzXyfAPwUQDbTzd5cHUijkSmj/BeLw52/Zuk7/DcSU/f7kFdqCaanVdfN7GeanfKtnfsUf0uvQ9YJOne+vUc2r92ZtKg0capwIJ6MuPXJN3ccm2Al9s+EKCeqfyjNopMxBD5saR32v77wY2S/ie9mQL+EPAfVMdCRPWDdUUP6mJ7h/rmvfswcpi2or5vy3Nq9/AK4u9RjQyOrV9/nur70KZJkibXBzGPpTpbMqAXv3u/OWhcX3TXSpGJeGB1d+AKqmHmQGj0Ud2L8+SB/dcW638QeAvVdTpfAr7Tq1Oukt5Bde3Qi4CbqZ4A933bx460Xou1/8M9ehaupK9Q3Vf1krrpNGBn23/YYs3zqXYjVlDdDPkV9S0o9qa67USrj6yQ1A+sHHhJNdJeRcMj0AkXIgMkHQMcUL+8zfZ1PawtqlNvZ1IF2FeAL/bgzNAyqlPLP7A9t7450wdsn9pm3a5r1/VvGXJgddi2FuoeDrwQuNr2yrptX2CW7XFxweeEO7Aq6acAthfZ/mT9cd1wy7SlHnn8sv7YAOwMXC7pw23WBdb42ftuTqvvdbFfyzXHQm2Am+pfaOo+vJJqF6c1qp5r8wPbVwwECFS3YBgIkDZ/1kaz7SbqT8RjIi+r7zi1KQJmt1Vc0jnAGVRD3C9QzRFZL2k74G7gvLZqAw9J2gm4Evh3SU9QPZekFzqpPWiG8hTgjyU9UL/eC7i95fKd/qz1qv6E253Z3IzVWr/th1qqfyHVrstvzYyV9DLby9uoO0yt11D9AF1lu60L0Dqv3eUM5THws9aT+hMuRCKiWRPumEhENGvCh4iqO76n9gSqn9rNmvAhwnMnAKX2xKif2g1KiEREkXFxYHXqdjM8Y9LW3UR73cbVTN2u4JKZyZO2etV1/auYOmn7rV5/2ou3/sTG6ifWMGPnrZ/5vuoXZZcZbVi7ksnTtv4JHf3Ttn4Kd/+qlUzafutre+u/5fSvXMmkmVtfe+rDKze/0CasZy1TtvJJEWtYyTqvHfY/fVzME5kxaQeO2LW12csj27W15zFv1t4L7938Qi25+aJDOqsN8OTe3f3oru/1I6gG2fOC73dS94e+dpPvZXcmIookRCKiSEIkIookRCKiSEIkIookRCKiSEIkIookRCKiSEIkIookRCKiSEIkIooUh4ikOZJWDzyMR9ILJP2zpHsk3S7p25L2rZe7dZj1D5f0Q0k3S1ou6YK6/VRJP5P0zdI+RkR7mrqK6Z76MQCieqbLQttvAZA0F9gdeHAT6y4ETrF9S/0U8/0AbF8m6RHgvQ31MSJa0PSlkMcA621/bqDB9s1QjVg2sc7zgV/Uy/Yzyjtw13dpmgcwfbtZW93hiCjT9DGRA9jyR1HOB+6UdIWk/1U/yXyzbC+w3We7r+h+IBFRpPMDq7YvpHoK3NXAW4Gruu1RRGyJpkPkNuDQLV3J9j22P0v10OODJT2v4X5FREuaDpHrgGmS3jnQIOmw+mFFw5L03/Xs48r3AfqBJxvuV0S0pNEDq/UTz08GPibp/cAa4D7gf9eL7Cdp8NO2zgXeDMyXtIrqubSn1wdYI2Ib0PiNKm0/DJyyibenDNP21ab7EBG908TuTD8we2CyWVMknQp8Bniiye1GRLOKRyK2HwT2aKAvQ7d7GXBZ09uNiGZ1foo3IrZtCZGIKJIQiYgiCZGIKJIQiYgiCZGIKDIuHui9YfY0Hnv9Szup/diBnZQF4N5vdXeJ0ezZ7qx211582aOd1R6LU7kzEomIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCjS2a0AJF0ErLD98fr1h4BfAS8CTgAMfLC+63tEjFFdjkS+CJwBIGk74C3AQ8Bc4GDgOOAjkl443MqS5klaImnJhjUre9PjiPgtnYWI7fuAxyQdAhwP3AQcCVxqu9/2I8D1wGGbWH+B7T7bfZOnz+xVtyNiiK7vbPYF4O3AC4AvUYVJRGxDuj6wegXw36hGG98BbgBOlTRJ0m7Aq4Efddi/iNiMTkcittdJWgQ8abtf0hXAEcAtVAdWz7P9yy77GBEj6zRE6gOqhwN/CGDbwPvqj4jYBnS2OyPp5cDPgGtt391VPyKiTGcjEdu3Ay/pqn5ENKPrA6sRsY1LiEREkYRIRBRJiEREkYRIRBRJiEREkYRIRBTp+gK8Rmy3wUx/vL+T2me9/vpO6gJce+6RndX2JHVWG6B/ypTOaj+z386d1Z6xvLPSm5SRSEQUSYhERJGESEQUSYhERJGESEQUSYhERJGESEQUSYhERJGESEQUSYhERJGESEQUSYhERJGESEQUSYhERJHWQkTSlZJ+Iuk2SfPqtmck/U3dfo2k35W0WNK9kt5YLzNd0j9IWibpJknHtNXHiCjX5kjkLNuHAn3AOZKeB8wEFtftvwY+CLwOOBm4sF7vbADbBwKnAQslTW+xnxFRoM2bEp0j6eT68z2AfYB1wFV12zJgre31kpYBc+r2I4FPAti+Q9L9wL7A0sEbr0c38wCmzdipva8iIkbUykhE0tHAccARtg8GbgKmA+vr5+0CbATWAtjeyLOBNqpbZtleYLvPdt+UqTMb7H1EbIm2dmdmA0/YXiVpf6qHdo/WDcDpAJL2BfYE7my+ixHRhLZC5CpgsqSlwEXAD7Zg3c8Ak+pdnMuAt9te20IfI6IBrRwTqX/pTxjmrVmDlrlgyDqz6n/XAG9vo18R0bzME4mIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIm1exds7G2Hyqv5OSt+96vmd1AX4z9dM7az2C364obPaALssX9VZ7bW7TOus9liUkUhEFEmIRESRhEhEFEmIRESRhEhEFEmIRESRhEhEFEmIRESRhEhEFEmIRESRhEhEFEmIRESRhEhEFEmIRESRhEhEFBmTISLpSkk/kXSbpHld9yciNm2s3pToLNuPS5oB/FjS12w/NniBOlzmAUybvlMHXYwIGKMjEeAcSbdQPQh8D2CfoQvYXmC7z3bflCkze97BiKiMuZGIpKOB44AjbK+StBiY3mWfImLTxuJIZDbwRB0g+wOHd92hiNi0sRgiVwGTJS0FLqLapYmIMWrM7c7YXguc0HU/ImJ0xuJIJCK2IQmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIImNuxurWUP9Gpjy1ppPa37vugE7qAtz1js92Vvt1/35mZ7UBJt9+X2e1Nx7y0s5qj0UZiUREkYRIRBRJiEREkYRIRBRJiEREkYRIRBRJiEREkYRIRBRJiEREkYRIRBRJiEREkTEbIpJmSTq7635ExMg2GyKS5ki6ta0OSDpa0quGeesvgbvbqhsRzRgLI5GjgeeEiKTtgRttX91JjyJi1EYbIpMlLZS0VNLlkraXdKykmyQtk/QlSdMAJL1B0h2SvivpE5K+WbfvIunKehs/kHSQpDnAu4BzJd0s6ShJJwGLgPMlXSNp9za+8IhoxmhDZD9gge2DgKeBPwUuBk61fSDVfUneLWk68HngBNtHArsN2sYHgJvqbfw58I+27wM+B8y3Pdf2jcB3gcNtzwW+Cpw3XIckzZO0RNKS9RtWbcnXHBENGm2IPGj7e/Xn/wQcC/zc9l1120Lg1cD+wL22f163XzpoG0cCXwawfR3wPEmzh6n1X4BvSLoRmAf8znAdsr3Adp/tvimTtx/llxERTRttiHiUy2kL3xtuu58CPm37KOC9wPRR1o6IDow2RPaUdET9+WnANcAcSXvXbW8DrgfuAF5SH+sAOHXQNm4ATofqjAywwvbTwK+BHQYttzPwaP35GaP9QiKiG6MNkeXAGZKWArsA84Ezga9KWgZsBD5nezXwHuAqSd8FHgGeqrdxAdBXb+OveTYg/hU4eeDAKnAhcHm9OzMQJhExRm32Rs31wc+XD/PWtcAhw7Qvsr2/JAGfBpbU23kceNMw278LOGhI879srl8RMTa0MU/knZJuBm4DZlOdrYmIcarxR0bYnk+1uxMRE8BYmLEaEduwhEhEFEmIRESRhEhEFEmIRESRhEhEFEmIRESRxueJdGHD9pN49NAdO6m917+t6aQuQN+97+6s9o7T1nVWG+CeT7y0s9ozlnd3TeiLFnVWepMyEomIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIsUhImmOpNX1YyKQdL6k2yQtrR9I9cq6fbGkO+u2myVdXrdfIOk/67ZbJb2xbj9X0gOSPlXax4hoT1O3ArjH9tz6UZsnAq+wvVbSrsDUQcudbnvJMOvPt/1RSS8DbpT0fNvzJT0B9DXUx4hoQdP3E3kh1TN21wLYXrElK9teLmkDsCvwq5GWlTQPmAcwZdbOW9fbiCjW9DGRq4E9JN0l6TOSXjPk/UsG7c58ZOjK9a7PRkbxDF7bC2z32e6bPH1mM72PiC3W6EjE9jOSDgWOAo4BLpP0ftsX14tsanfmXEl/BPwaONW2m+xXRLSnjcdo9gOLgcWSlgFnABdvZrX5tj/adF8ion2N7s5I2k/SPoOa5gL3N1kjIsaWpkcis4BPStoJ2AD8jPrgZ+0SSavrz1fYPq7h+hHRY00fE/kJ8KpNvHf0JtovaLIPEdFbTezO9AOzByabNUXSucD/BZ5ucrsR0azikYjtB4E9GujL0O3OB+Y3vd2IaFaunYmIIgmRiCiSEImIIgmRiCiSEImIIgmRiCjS+LUzXZi03sx6uL+T2isOnNFJXYCzzv5WZ7W//OE3dFYbYL+/fbKz2v0zu/lZG6syEomIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIgmRiCiSEImIIkUhImmOpNUDd3qX1F8/Z/dWSV+VtH3dPlnSCkl/NWT9xZLulLRU0h2SPlU/swZJM+ptrZO0a0k/I6I9TYxE7rE9t/58te25tg8A1gHvqtuPB+4ETpGkIeufbvsg4CBgLfB1ANur6+0+3EAfI6Ilbe7O3AjsXX9+GvBx4AHg8OEWtr0OOA/YU9LBm9u4pHmSlkhasn7dyoa6HBFbqpUQkTQZOAFYJmkGcCzwTeBSqkAZVv0w8FuA/TdXw/YC2322+6ZMndlMxyNiizUdIjPq4yNLqEYdXwROBBbZXgV8DThZ0qQRtjF0dycixrCmb4+4etDxEQAknQb8nqT76qbnAccA1wxduQ6XA4HlDfcrIlrS6ileSTsCRwJ72p5jew5wNsPs0kiaAvwV8KDtpW32KyKa0/Y8kd8HrrO9dlDb14E3SppWv75E0lLgVmAm8KaW+xQRDWp0d8b2rCGvLwYuHtL2OLBb/fLoJutHRO+VjkT6gdkDk82aNDDZDJgCbGx6+xHRjKKRiO0HgT0a6svQba8G5rax7YhoTq6diYgiCZGIKJIQiYgiCZGIKJIQiYgiCZGIKCLbXfehmKRHgfu3cvVdgRUNdie1x3791N5ye9nebbg3xkWIlJC0xHZfak+c+qndrOzORESRhEhEFEmIwILUnnD1U7tBE/6YSESUyUgkIookRCKiSEIkIookRCKiSEIkIor8f5fDD5S7g1Q2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_attention_head(in_tokens=tokens, translated_tokens=tokens, attention=head)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b603500-d79b-4b20-99e3-0996c8454d41",
   "metadata": {},
   "source": [
    "### Visualizando los pesos de todas las cabezas de atención"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "864f57d2-01d6-41b5-872c-402b44a9e7e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_attention_weights(sentence, translated_tokens, attention_heads):\n",
    "  in_tokens = sentence\n",
    "  #in_tokens = tokenizers.pt.tokenize(in_tokens).to_tensor()\n",
    "  #in_tokens = tokenizers.pt.lookup(in_tokens)[0]\n",
    "  #in_tokens\n",
    "\n",
    "  fig = plt.figure(figsize=(16, 8))\n",
    "\n",
    "  for h, head in enumerate(attention_heads):\n",
    "    ax = fig.add_subplot(3, 4, h+1)\n",
    "\n",
    "    plot_attention_head(in_tokens, translated_tokens, head)\n",
    "\n",
    "    ax.set_xlabel(f'Head {h+1}')\n",
    "\n",
    "  plt.tight_layout()\n",
    "  plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "51b75032-bdf9-4b9f-a069-373bc7afc6fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/EAAAI4CAYAAAAiUWldAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABy3ElEQVR4nO3deZxkdX3v/9dnenp2ZgEG3ICJsqmAo44KigqKJnpdQkxEQgxozCRqromJGnO9j/wUNSaaXOISjUQTMddrVAhuMUhUBlBjdIBhhn2TTRTZl1l7ej6/P+qMtpPu6qo+31r79Xw8+kHVqXM+9TnFqffUt86pcyIzkSRJkiRJ/W9OrxuQJEmSJEmtcRAvSZIkSdKAcBAvSZIkSdKAcBAvSZIkSdKAcBAvSZIkSdKAcBAvSZIkSdKAcBAvSZIkSdKAcBAvSZIkSdKAcBAvSZIkSdKAmNvrBtRdEbES+FPgCcCC3dMz83k9a0rSwDJTJJVkpkgqaVgzxT3xs0BEvCEiTqjufga4DlgJnA7cAvygV71JGjxmiqSSzBRJJc2GTHEQPzucBbw0In4d2DczzwS2Zua6zHwNcHRv25M0YMwUSSWZKZJKGvpMcRA/C2Tm5sz8Q+DfgB3V5Psj4gUR8WTggN51J2nQmCmSSjJTJJU0GzIlMrPXPaiLIuIlwMXAo4GPAMuB0zPziz1sS9KAMlMklWSmSCppWDPFE9vNPvdl5gPAA8DzACLiWb1tSdIAM1MklWSmSCppKDPFPfGzTERcmplPmW5aF/rYu4XZdmXm/Z3uRdLMmSmSSjJTJJU0rJninvhZIiKOAZ4JrIyIP57w0FJgpAct3VH9RZN5RoADu9OOpHaYKZJKMlMklTTsmTKUg/iI+HILs92bmad1upc+Mg9YQuP/+V4Tpj8I/HqrRSLiQy3M9mBm/u9p5rk6M588zXNd1mpfUieZKZMyU6QZMlMmZaZIM2SmTKpIpkCxXCmaKUN5OH1EXA+8rtkswN9l5hO71FLfiIiDMvOWiNgLyMx8uM3lbwH+fJrZ3p6Zj5+mzoLM3FZ3HqkbzJSpmSlS+8yUqZkpUvvMlKnVzZSqRu1cKZ0pQ7knHnhHZl7YbIaIeFe3mukze1Xf8uwNEBF3A6dm5hUtLn9GZp7VbIaIWDFdkd0baEQcCRxeTb56Yh/+w6g+YqZMzUyR2memTM1MkdpnpkytbqZAgVwpnSlDuSd+MtULe3/OlhWeQkR8l8Yb/YLq/nHAX2TmM9uss29m3l2jj2XAl2hcp3EjjW8IjwRuBV6emQ/OtLbUDWZKg5kilWGmNJgpUhlmSkOpTKmWnXGulM6UOTNpot9FxJ9HxOHV7fkRcQFwI3BnRJzQo57mRcQR1d9oL3qoLN69EQNk5jpgcasLR8RLI+IuYFNE3B4Rbb8BKu8G1gOHZOaJmfmrwCHAD4D3zrCm1BFmSlNmitQmM6UpM0Vqk5nSVK1MgWK5UjRThnJPfERcCRyRmRkRa4GTgROAQ4GzMvPpXe7nOOAs4GYa37ocQOMwjou62UfVy7nApcA/V5N+C1hTbUitLL8ReGVmXhMRzwDen5nPnUEfVwFHZebOPabPBTZN91s1qZvMlKa9mClSm8yUpr2YKVKbzJSmvdTKlKpG7VwpnSnD+pv4HRMOHfll4F8ycxy4unqhuu1vgBdm5rUAEXEo8FngqT3o5bXAu4B/pfGmugh4TRvL78zMawAy87+icZKImdix50Zc1dwZEdtnWFPqFDNlamaK1D4zZWpmitQ+M2VqdTMFyuRK0UwZ1kH89og4ArgTOB54y4TH2jp8opDR3RsxQGZe16vDSjLzPuBNNUrsF794rcVfuJ+Z/6fFOgsi4snw366VGMD8Gv1JnWCmTMFMkWbETJmCmSLNiJkyhQKZAmVypWimDOsg/g+Bs4GVNM4m+EOAiHgxjcMpum19RHySnx/GcQpwSQ/6ICK+Auz5G4oHaPxG4+M5/VkR/4FfvNbinvdb9WNgqg3+JzOoJ3WSmTIFM0WaETNlCmaKNCNmyhQKZAqUyZWimTKUv4lvJiJekZnndPk55wNvBI7l54dxfDQzu344VkR8kMYb/LPVpJNobDgLgaWZ+epu97SniBjNzLFe9yG1wkwxU6SSzBQzRSrJTBnOTJmNg/hbM/PAXvfRKxFxUWY+Z7JpEXFlZj5xmuU/n5mvrG7/VWb+6YTHzs/MF86wr6Bx+M9vAi/NzP1nUkfqNjPFTJFKMlPMFKkkM6VeplTzF8+VupkylJeYm8aev0Po/BNGvCQiLouIeyPiwYh4KCJ6dX3RlRHxszdydXvf6u6OFpY/ZMLtF+xZu91mIuIZ1TdktwBfBi4GDm+3jtRDZoqZIpVkppgpUklmSr1MgYK5UipThvU38c304tCDvwV+jcblA3p96MOfAN+OiBtpvKl/CXhDRCymcSmI6TTrv+V1i4j3Aq8EbqVxeMvpwPrMbKUHqZ+YKWaKVJKZYqZIJZkp9TIFCuRK6UwZykF8RGxi8hc0gF4c/nQbcEUfbMRk5tci4hAa3/gEcM2EEzr8bQslFlVnVpwDLJxwlsWg8duSVq0FrgU+Bnw1M7dFRM9fH2kyZsrUzBSpfWbK1MwUqX1mytQKZAqUyZWimTKUv4mPiIOaPZ6Zt7RRa3/gadXd72fmT2fQz9OAdwMXAj87oUMblzkpJhqXd3g9sPu3IetonJmxpZMpRMQ6mnzjlJnHt1hnBHghcDLwPOAC4ATggMmuoSj1kpnStBczRWqTmdK0FzNFapOZ0rSXWplS1VhHzVwpnSnDOog/GNg/M7+zx/RnA3dk5o0t1nkl8AEa/7MDeDbw1sw8u81+zgceBjYBu3ZPz8x3tVOnhIj4BDDKzw8feTUwnpmv63YvE3paALyExkZ9LPDNzPzNXvUj7clMadqLmSK1yUxp2ouZIrXJTGnay3BmSmYO3R/wVeCoSaavAb7SRp3Lgf0m3F8JXD6Dftb3+jWZuE6tTGuy/NOAR0y4/9vAl4APAXu3UWcB8EfAR2gcXjK3mr4UOLXXr5N//k38M1Oar1Mr05osb6b4N+v+zJTm69TKtCbLmyn+zbo/M6X5OrUybZoatXOldKYM69npV2Xmxj0nZuZ6YFUbdebkLx5Ccg8zO6P/NyJiRpc06YDxiHjc7jsR8VhgvI3lP051JseIeA7wl8CngQeAM9uocxaNYNkEvBj4a4DMfDA9aYz6j5kyNTNFap+ZMjUzRWqfmTK1upkCZXKlaKYM5YntaHzTMZV2Tmry7xHxdRpnEAQ4CfjaDPp5I/C2iNgOjNE4PCUzc+kMatX1VuCCiLipur8KeE0by49k5r3V7ZOAMzPzHOCciNjQRp0nZOaRABHxSeD7bSwrdZuZMjUzRWqfmTI1M0Vqn5kytbqZAmVypWimDOsg/gcR8buZ+Q8TJ0bE7wCXtFHnduA/afweJGj8Dzu33WYyc6+I2JvGNQabvcmaiogVe9bIzIvaLPMdGt8mPb+6/3Ea69iqkYiYm40TMDyfxuEgu7WzPf3sZBKZuTOi65ewlNphpkzNTJHaZ6ZMzUyR2jeUmQJFcqVupkCZXCmaKcN6Yrv9gXNpHPawe8NdA8wDTszMn7RY5z3Aq4BLgX8Evp4zeMEi4nXAHwKPATYARwPfzcznN1uuhRr/mZnPa7OXzwMPAp+pJp0MrMjM32hx+XfQOATkbuBA4CmZmdUJNc7KzGe1WGcc2Lz7Lo1vCbfQ22/qpEmZKU3rmClSm8yUpnXMFKlNw5gpTeq0lSt1M6WqUTtXSmfKUA7id4uI44EjqrtXZua3ZlAjaFwO4DU03gyfBz6ZLZ7lsaqxicYJEb6Xmasj4nDgXZl5UjdrVHUuz8wnTTdtmhpHA48Ezs/MzdW0Q4ElmXlpO/1Ig8RMmbSOmSLNkJkyaR0zRZqhYcqUUnVKZEq1TF/lylCe2C4iLgXIzAsy88PV37cmm2c61bdPP6n+dgIrgLMj4v1ttLQtM7dVzzs/M68BDmtj+VI1AC6rNkKqWs+gcZhJSyLi0sz8Xmaeu3sDBsjM63ZvwK28tqXmkbrBTGnKTJHaZKY0ZaZIbRrSTClVp1amVMvUzpXSmTKsv4l/fET8tzM0ThDAsumKRMSbgFNpHDrxCRrXSRyLiDnA9cDbWuzn9ohYDnwR+I+IuA+4o8Vli9SovslKGtdJ/O2IuLW6fxBwVRt9FHltC9aRusFM2YOZItVipuzBTJFqGcZMqVWnYKZAmde3aKYM5eH0EXFQC7ONZ+bt09Q5ncbhI7dM8tjjM/PqGfT2XBr/g87LzB3tLj/TGtO9JpOt40zqVFp5bYvUkbrBTJl0GTNFmiEzZdJlzBRphoY9U2ZSp1SmtFKr0vT1LZ0pQzmIlyRJkiRpGA3lb+IlSZIkSRpGs2YQHxFrp59rMGrYi9R7/bLdD+P7uJ96kbqlX7b7YXwf91MvUrf0y3Y/jO/jfuhl1gzigRL/0/qlRqk6w9iL1C39st0P4/u4n3qRuqVftvthfB/3Uy9St/TLdj+M7+Oe9zKbBvGSJEmSJA20oTix3bw5C3PhyF5N59mxayvz5iyceoa5I9M+z47xLcwbWdR0nvm/1PxkiVvv28bCFQuazrPlx036rOzcvpm58xc3nWd8fjR/fMtmRhY1r5HTvyyMb97MyOLmdebdsbnp42NsZ5T5Ted5iPvuzsyV03ck1TNvzoIWMmUb8+ZM/V7O8fFpn2cstzMazbf7xhVHmtXYxmg0z5Rdy5vnFsDY9ocZnb+k6TwrH31f08cfuneMvfYebTrPvddMn2/TvbYAubP562umqJ/MiwW5IJr/Oznte7mFz2utbPclasTc6a9QPO3nLuBxT3ig6eN33zPOvvs0/yByw6bmuQWt5eR0r6+Zon4yb2RhLpzb/Gpk045bWvicsiO3Mi+av49z1zT/HrfwWSfmTZ9brYzDVhz8cNPHH7pvjL1WNP+c8uOHp7/K2/hDmxnZq3mmz79lS9PH62bKUFwnfuHIXhyz72/UK7LviiK9HHzWTbVrbHj3kwt0AvcfXP9/71jzcUzLDnznd2vX+Eae3fKlIKQ6Fo7sxTHLf61WjV0PPFimmZEWvkmbxkMvWF2/D+AN7zm7do3PPGt1/UaA8bvvqV3DTFG3LIjFHD36K7Vq5NiMr8xU3Mi++xWpc+55/1a7xssfe2yBTiC3b69dw0xRtyycu4xnPvqUWjXy/uZforVq18PNd9S1Ys6qVq6+Nr1XnPOd2jXec/FLC3QCh679Qe0azTLFw+klSZIkSRoQDuIlSZIkSRoQDuIlSZIkSRoQDuIlSZIkSRoQtQfxEbEqIrZGxIbq/iMi4l8i4saIuCoivhYRh1bzXTHJ8kdHxH9FxIaIuDoi3llNPykiboiIr9btUdLgMFMklWSmSCrJTFE/KHV2+hszc3VEBHAucFZmvgogIlYD+wO3TbHsWcArM/PyiBgBDgPIzM9FxJ3AWwr1KGlwmCmSSjJTJJVkpqinSl9i7nhgLDP/fveEzNwAjW+tplhmP+DH1bzjwFWFe5I0uMwUSSWZKZJKMlPUE6V/E38EcEmby5wBXBsR50bE70XEglYWioi1EbE+Itbv2LW17UYlDYQeZcq2thuVNBB6kiljaaZIQ6o3n1PGt7TdqIZLz09sl5mnA2uA84HfBM5rcbkzM3NNZq6ZN2dhJ1uUNEDKZEpL/55KmgVKZMpoa5/RJc0CRT6njCzqZIsaAKUH8VcCT213ocy8MTM/BjwfeFJE7FO4L0mDyUyRVJKZIqkkM0U9UXoQ/y1gfkT87u4JEfG0iHjuVAtExP+oTgoBcAgwDtxfuC9Jg8lMkVSSmSKpJDNFPVH0xHaZmRFxIvC3EfF2YBtwM/BH1SyHRcTtExZ5M/AK4IyI2ALsBE6pTvIgaZYzUySVZKZIKslMUa+UPjs9mXkH8MopHh6dZNoXSvcgaXiYKZJKMlMklWSmqBdKHE4/DiyLiA0Fav1MRJwEfBS4r2RdSX3PTJFUkpkiqSQzRT1Xe098Zt4GHFCglz3rfg74XOm6kvqbmSKpJDNFUklmivpB8cPpe2Hnsvnc88uPq1XjniPL9HLTv9U/ueSyZVmgkzJ+6XN3FanjD300UEZGYMWyWiV++MeHFWnlse++rHaNvb5YvwbAnz/zN2rXeNwhWwt0AnH3PUXqSN2Qey1k+7OeVKvG2JIy5yJe+o1ratfIhzcX6ASOfu8f1q6x79PKXC97zrc3FKkjDYotn19RpM6D2x5Ru8b+p9xRoBP41+etrl3j8cvvr10DOj/26fl14iVJkiRJUmscxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCB6dp34iHg3cHdmfrC6/17gp8BjgBcBCbwnMz/Xqx4lDQ4zRVJJZoqkkswUldTLPfGfBE4FiIg5wKuA24HVwJOAE4APRMQjJ1s4ItZGxPqIWL9z2+budCypnxXLlB3jW7vTsaR+VixTxnb4OUVSyc8pW7rTsfpWzwbxmXkzcE9EPBl4IXAZcCzw2cwcz8w7gQuBp02x/JmZuSYz18xdsLhbbUvqUyUzZd7Iwm61LalPlcyU0Xl+TpFmu7KfUxZ1q231qZ4dTl/5BHAa8AjgH2ls0JI0U2aKpJLMFEklmSkqotcntjsX+BUa3zh9HbgIOCkiRiJiJfAc4Ps97E/SYDFTJJVkpkgqyUxRET3dE5+ZOyLiAuD+zByPiHOBY4DLaZzc4W2Z+ZNe9ihpcJgpkkoyUySVZKaolJ4O4quTOhwN/AZAZibw1upPktpipkgqyUyRVJKZolJ6djh9RDwBuAH4ZmZe36s+JA0HM0VSSWaKpJLMFJXUsz3xmXkV8NhePb+k4WKmSCrJTJFUkpmiknp9YjtJkiRJktSiXl9irog5O5MF947XqvHaX76wSC/ffPOxtWvkSBToBMZHR2vXePiwFQU6gYVXFykjdcf4Lnhoc60SHz7pE0VaOeO9k14utj07d9avARz+N7fVrrH9kP0LdAIjRapI3TFn204WXfvTWjXuf9oji/Ryz0ufULvG3l+4rEAnsO/GLbVr7Fxc5qPsvCJVpC7JhJ31xj5/e8gXirTyv178W/WLPOYR9WsAO/ZZXL/G8jJpsKDDYx/3xEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCA6NoiPiC9GxCURcWVErK2mPRwRf1VN/0ZEPD0i1kXETRHxsmqeBRHxTxGxKSIui4jjO9WjpMFhpkgqyUyRVJKZom7q5J7412bmU4E1wJsiYh9gMbCumv4Q8B7gBcCJwOnVcm8EyMwjgZOBsyJiQQf7lDQYzBRJJZkpkkoyU9Q1cztY+00RcWJ1+wDgEGAHcF41bROwPTPHImITsKqafizwYYDMvCYibgEOBTZOLF59w7UWYP7C5Z1bC0n9omuZsmDOkg6uhqQ+0b1MmbtXB1dDUp/oXqaMmCmzXUf2xEfEccAJwDGZ+STgMmABMJaZWc22C9gOkJm7+PkXCtHKc2TmmZm5JjPXjM5bXLB7Sf2m25kyb87Cgt1L6jfdz5RFBbuX1G/8nKJu69Th9MuA+zJzS0QcDhzdxrIXAacARMShwIHAteVblDRAzBRJJZkpkkoyU9RVnRrEnwfMjYiNwLuB77Wx7EeBkeowk88Bp2Xm9g70KGlwmCmSSjJTJJVkpqirOvKb+GrDe9EkDy2ZMM8791hmSfXfbcBpnehL0mAyUySVZKZIKslMUbd5nXhJkiRJkgaEg3hJkiRJkgaEg3hJkiRJkgZEJ68T3z27YO6W8Volrt+yX5FWfvTcebVrPOK/dhboBPa+ekvtGtv3nl+gE2nAzB2BfZbXKvH+3311kVbu/e3678FHXHBXgU5g26OX1a7x8KPrZyTA8iJVpC7ZuZNdP727Voml/1Zv+d0Wn1f/0lQPXX9ogU5g3i3112n0oYcKdAL1PkVKXTYnyAX1/j39s5eeWqSVnfssqF3jhlePFugERh4eqV3joK+NFeik89wTL0mSJEnSgHAQL0mSJEnSgHAQL0mSJEnSgHAQL0mSJEnSgHAQL0mSJEnSgHAQL0mSJEnSgHAQL0mSJEnSgOjLQXxEfDEiLomIKyNiba/7kTTYzBRJJZkpkkoyU9Suub1uYAqvzcx7I2Ih8IOIOCcz75k4Q7WBrwWYv2B5D1qUNEDaypQFo0t70aOkwdFepsTiXvQoaXC0lylz/Zwy2/XlnnjgTRFxOfA94ADgkD1nyMwzM3NNZq4ZHfUfR0lNtZUp80YWdb1BSQOlvUyJBV1vUNJAafNzysKuN6j+0nd74iPiOOAE4JjM3BIR6wD/9ZM0I2aKpJLMFEklmSmaiX7cE78MuK/aiA8Hju51Q5IGmpkiqSQzRVJJZora1o+D+POAuRGxEXg3jcNKJGmmzBRJJZkpkkoyU9S2vjucPjO3Ay/qdR+ShoOZIqkkM0VSSWaKZqIf98RLkiRJkqRJOIiXJEmSJGlAOIiXJEmSJGlA9N1v4mcixncx+sC2WjW+860jivRy3es+VrvGC/7jNQU6gblX3Vy7xq4nP65+I9KgySS27ahVYvQ/ryrSysJ9n1S7xvJ/vKdAJ/DAq+rlLMD8TdsLdALjRapI3ZGZ5I6xejV21lt+t4ePr/8e/MANXy3QCbz9l3+rdo1dP/5JgU6kwTK+aJQHV+9Xq8ay79xSpJe5Nz1Yu8avf2hrgU7gitMOq13jjuftXaATeMQ3i5SZknviJUmSJEkaEA7iJUmSJEkaEA7iJUmSJEkaEH07iI+IJRHxxl73IWk4mCmSSjJTJJVkpqgd0w7iI2JVRFzRqQYi4riIeOYkD/0FcH2nnldSb5gpkkoyUySVZKZoEPTDnvjjgF/YkCNiEXBxZp7fk44kDbLjMFMklXMcZoqkco7DTFFNrQ7i50bEWRGxMSLOjohFEfH8iLgsIjZFxD9GxHyAiHhxRFwTEd+OiA9FxFer6XtHxBerGt+LiKMiYhXw+8CbI2JDRDw7Il4KXAC8IyK+ERH7d2LFJfWUmSKpJDNFUklmivpaq4P4w4AzM/Mo4EHgj4FPASdl5pE0rjf/+ohYAHwceFFmHgusnFDjXcBlVY3/BXw6M28G/h44IzNXZ+bFwLeBozNzNfAF4G2TNRQRayNifUSsH9u5pZ11ltR7fZ0pO8bNFGnA9HWmjOW24issqaP6O1O2P1x8hTVYWh3E35aZ36lu/1/g+cAPM/O6atpZwHOAw4GbMvOH1fTPTqhxLPDPAJn5LWCfiFg2yXM9CvhyRFwMrAWeOFlDmXlmZq7JzDWjcxe1uBqS+kRfZ8q8ETNFGjB9nSmjsaDGqknqgf7OlPlLaqyahkGrg/hscb5o87HJ6n4E+LvMfDbwFsB/+aThY6ZIKslMkVSSmaK+1uog/sCIOKa6fTLwDWBVRBxcTXs1cCFwDfDY6vceACdNqHERcAo0zsoI3J2ZDwIPAXtNmG8FcFd1+9RWV0TSQDFTJJVkpkgqyUxRX2t1EH81cGpEbAT2Bs4AXgN8ISI2AbuAv8/MrcAbgPMi4tvAncADVY13AmuqGn/JzzfSrwAn7j65A3A6cHZ1SMnuDVrScDFTJJVkpkgqyUxRX5s73QzVCRieMMlD3wSePMn0CzLz8IgI4O+A9VWde4GXT1L/OuCoPSb/63R9SRpMZoqkkswUSSWZKRoEnbhO/O9GxAbgSmAZjTM2StJMmSmSSjJTJJVkpqjrpt0T367MPIPGISeSVJuZIqkkM0VSSWaKeqETe+IlSZIkSVIHFN8T3ws7F41w11OX1qpx0L9vK9LLmpteX7vG0vk7CnQCN37ocbVrLLy6zFUuHnNBkTJSd4yPkw88WKvEru3bi7Sy9Lyrate47/oDCnQC//5f/692jRcd9uwCnUiDJebMYc7ihbVqjN9f5rNB7txZu8bbDj62QCdw3q1n167xooOfWaCTMq+L1C2xKxnZ3upV8Ca39YmPLtLLgpvrZRvAZX+yokAncNfpW2vX2Oufxwt00nnuiZckSZIkaUA4iJckSZIkaUA4iJckSZIkaUA4iJckSZIkaUA4iJckSZIkaUA4iJckSZIkaUDUHsRHxKqI2BoRG6r774iIKyNiY0RsiIhnVNPXRcS11bQNEXF2Nf2dEfGjatoVEfGyavqbI+LWiPhI3R4lDQ4zRVJJZoqkkswU9YNS14m/MTNXR8QxwEuAp2Tm9ojYF5g3Yb5TMnP9JMufkZl/HRGPBy6OiP0y84yIuA9YU6hHSYPDTJFUkpkiqSQzRT1VahC/2yOBuzNzO0Bm3t3Owpl5dUTsBPYFftps3ohYC6wFGF2yYmbdSup3PcmUBXOWzKxbSf2uR5myeGbdSup3PcmU+QuXz6hZDY/Sv4k/HzggIq6LiI9GxHP3ePwzEw4p+cCeC1eHn+wC7pruiTLzzMxck5lr5i7wH0dpSPUkU+bNWVCme0n9pjeZEgvLdC+p3/QkU0bnOfaZ7Yruic/MhyPiqcCzgeOBz0XE2zPzU9UsUx1S8uaI+C3gIeCkzMySfUkaTGaKpJLMFEklmSnqldKH05OZ48A6YF1EbAJOBT41zWJnZOZfl+5F0uAzUySVZKZIKslMUS8UPZw+Ig6LiEMmTFoN3FLyOSTNHmaKpJLMFEklmSnqldJ74pcAH46I5cBO4AaqEzBUPhMRW6vbd2fmCYWfX9JwMVMklWSmSCrJTFFPlP5N/CXAM6d47Lgppr+zZA+ShoeZIqkkM0VSSWaKeqXE4fTjwLKI2FCg1s9ExJuBPwMeLFlXUt8zUySVZKZIKslMUc/V3hOfmbcBBxToZc+6ZwBnlK4rqb+ZKZJKMlMklWSmqB8UPzt9L4yMJUvuGK9V4+4jy1zD9bVv/LfaNf75/S8u0Akc9jf3164xvrje6yoNopw/j52H1vv3ec4PynyRHo95RO0a1566tEAncMSH3lC7xkFLbirQCex66KEidaSuGBkhli+rV+OBQpkyMlK7xrUfe3KBTuBJ73967RqPXnhNgU6ALVvK1JG6YM7m7Sz+7g31iuy9vEgv8XD99843PnNugU7gmD/5/do1ltxc5vNFp68ZWPTs9JIkSZIkqXMcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCBqDeIjYlVEbN19ncSIGI+IDRFxRUR8ISIWVdPnRsTdEfG+PZZfFxHXRsTGiLgmIj4SEcurxxZWtXZExL51+pQ0GMwUSSWZKZJKMlPUL0rsib8xM1dXt7dm5urMPALYAew+z/8LgWuBV0ZE7LH8KZl5FHAUsB34EkBmbq3q3lGgR0mDw0yRVJKZIqkkM0U918nD6S8GDq5unwx8ELgVOHqymTNzB/A24MCIeNJ0xSNibUSsj4j1Yzs2F2pZUh/rXqaMmSnSLNC1TNkx7jXIpVmge5mya1uhljWoOjKIj4i5wIuATRGxEHg+8FXgszQ26kll5jhwOXD4dM+RmWdm5prMXDM6b3GZxiX1pa5nyqiZIg2zbmfKvJFFZRqX1Je6nilzFpRpXAOr9CB+YfUbkfU0vnn6JPAS4ILM3AKcA5wYESNNaux5yImk2ctMkVSSmSKpJDNFPTG3cL3dv+X4mYg4GXhWRNxcTdoHOB74xp4LVxv4kcDVhfuSNJjMFEklmSmSSjJT1BMdvcRcRCwFjgUOzMxVmbkKeCOTHFYSEaPA+4DbMnNjJ/uSNJjMFEklmSmSSjJT1C2dvk78rwHfysztE6Z9CXhZRMyv7n8mIjYCVwCLgZd3uCdJg8tMkVSSmSKpJDNFXVH0cPrMXLLH/U8Bn9pj2r3AyurucSWfX9JwMVMklWSmSCrJTFGv1N0TPw4sq07oUFRE7D5RxCiwq3R9SX3JTJFUkpkiqSQzRX2h1p74zLwNOKBQL3vW3gqs7kRtSf3JTJFUkpkiqSQzRf0iMrPXPdQWEXcBt0wz277A3TWfql9qzNZeDsrMldPMI9VmpsyaXswUdYWZMmt6MVPUFWbKrOllykwZikF8KyJifWauGYYa9iL1Xr9s98P4Pu6nXqRu6Zftfhjfx/3Ui9Qt/bLdD+P7uB966fTZ6SVJkiRJUiEO4iVJkiRJGhCzaRB/5hDVKFVnGHuRuqVftvthfB/3Uy9St/TLdj+M7+N+6kXqln7Z7ofxfdzzXmbNb+L7SUQ8PPG6khFxGrAmM/+gQO11wFsyc/0e0/8A+CPgccDKzCxxUgdJfaBHmfIZYA0wBnwf+L3MHKv7fJJ6r0eZ8kkamRLAdcBpmflw3eeT1Hu9yJQJj38YeM3E5x8Gs2lP/Gz3HeAEpj+TpSS14jPA4cCRwELgdb1tR9KAe3NmPikzjwJuBWp/uJc0u0XEGmB5r/voBAfxfSYiVkbEORHxg+rvWdX0p0fEdyPisuq/h1XTF0bEv0TExoj4HI0P0/9NZl6WmTd3b00k9YMOZsrXskJjT/xjurZSknqmg5nyYDV/VPN4qKg0C3QqUyJiBPgA8LaurUwXze11A7PUwojYMOH+3sCXq9sfBM7IzG9HxIHA14HHA9cAz8nMnRFxAvAXwCuA1wNbMvOoiDgKuLRbKyGpb/QsUyJiFHg18IclV0hST/UkUyLin4AXA1cBf1J4nST1Ti8y5Q+AL2fmjxvfDQ4XB/G9sTUzV+++s/t3IdXdE4AnTNjYlkbEXsAy4KyIOITGt9Oj1ePPAT4EkJkbI2Jjx7uX1G96mSkfBS7KzIsLrIek/tCTTMnM11R7zz4MnAT8U6kVktRTXc2UiHgU8BvAcaVXpF84iO8/c4BjMnPrxInVSRkuyMwTI2IVsG7Cwx5yJmkqHcuUiPj/gJXA75VpVdIA6OjnlMwcrw6RfSsO4qXZoBOZ8mTgYOCG6suBRRFxQ2YeXKzrHvM38f3nfCaczCUiVlc3lwE/qm6fNmH+i4BTqnmPAI7qeIeSBklHMiUiXgf8MnByZu4q2rGkflY8U6Lh4N23gZfSOJRW0vArnimZ+W+Z+YjMXJWZq2gcfj80A3hwEN+P3gSsqU7WcBXw+9X09wPvi4jvACMT5v8YsKQ6lORtNE4w9d9ExJsi4nYaJ5/aGBGf6NgaSOonHckU4O+B/YH/jIgNEfHnnWlfUp/pRKYEjcNmNwGbgEcCp3dqBST1lU59ThlqXidekiRJkqQB4Z54SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGxNxeN6DuioiVwJ8CTwAW7J6emc/rWVOSBpaZIqkkM0VSScOaKe6JnwUi4g0RcUJ19zPAdcBK4HTgFuAHvepN0uAxUySVZKZIKmk2ZIqD+NnhLOClEfHrwL6ZeSawNTPXZeZrgKN7256kAWOmSCrJTJFU0tBnioP4WSAzN2fmHwL/BuyoJt8fES+IiCcDB/SuO0mDxkyRVJKZIqmk2ZApkZm97kFdFBEvAS4GHg18BFgOnJ6ZX+xhW5IGlJkiqSQzRVJJw5opnthu9rkvMx8AHgCeBxARz+ptS5IGmJkiqSQzRVJJQ5kp7omfZSLi0sx8ynTTutDH3i3Mtisz7+90L5JmzkyRVJKZIqmkYc0U98TPEhFxDPBMYGVE/PGEh5YCIz1o6Y7qL5rMMwIc2J12JLXDTJFUkpkiqaRhz5ShHMRHxJdbmO3ezDyt0730kXnAEhr/z/eaMP1B4NdbLRIRH2phtgcz839PM8/VmfnkaZ7rslb7kjrJTJmUmSLNkJkyKTNFmiEzZVJFMgWK5UrRTBnKw+kj4nrgdc1mAf4uM5/YpZb6RkQclJm3RMReQGbmw20ufwvw59PM9vbMfPw0dRZk5ra680jdYKZMzUyR2memTM1MkdpnpkytbqZUNWrnSulMGco98cA7MvPCZjNExLu61Uyf2av6lmdvgIi4Gzg1M69ocfkzMvOsZjNExIrpiuzeQCPiSODwavLVE/vwH0b1ETNlamaK1D4zZWpmitQ+M2VqdTMFCuRK6UwZyj3xk6le2PtztqzwFCLiuzTe6BdU948D/iIzn9lmnX0z8+4afSwDvkTjOo0baXxDeCRwK/DyzHxwprWlbjBTGswUqQwzpcFMkcowUxpKZUq17IxzpXSmzJlJE/0uIv48Ig6vbs+PiAuAG4E7I+KEHvU0LyKOqP5Ge9FDZfHujRggM9cBi1tdOCJeGhF3AZsi4vaIaPsNUHk3sB44JDNPzMxfBQ4BfgC8d4Y1pY4wU5oyU6Q2mSlNmSlSm8yUpmplChTLlaKZMpR74iPiSuCIzMyIWAucDJwAHAqclZlP73I/xwFnATfT+NblABqHcVzUzT6qXs4FLgX+uZr0W8CaakNqZfmNwCsz85qIeAbw/sx87gz6uAo4KjN37jF9LrBput+qSd1kpjTtxUyR2mSmNO3FTJHaZKY07aVWplQ1audK6UwZ1t/E75hw6MgvA/+SmePA1dUL1W1/A7wwM68FiIhDgc8CT+1BL68F3gX8K4031UXAa9pYfmdmXgOQmf8VjZNEzMSOPTfiqubOiNg+w5pSp5gpUzNTpPaZKVMzU6T2mSlTq5spUCZXimbKsA7it0fEEcCdwPHAWyY81tbhE4WM7t6IATLzul4dVpKZ9wFvqlFiv/jFay3+wv3M/D8t1lkQEU+G/3atxADm1+hP6gQzZQpmijQjZsoUzBRpRsyUKRTIFCiTK0UzZVgH8X8InA2spHE2wR8CRMSLaRxO0W3rI+KT/PwwjlOAS3rQBxHxFWDP31A8QOM3Gh/P6c+K+A/84rUW97zfqh8DU23wP5lBPamTzJQpmCnSjJgpUzBTpBkxU6ZQIFOgTK4UzZSh/E18MxHxisw8p8vPOR94I3AsPz+M46OZ2fXDsSLigzTe4J+tJp1EY8NZCCzNzFd3u6c9RcRoZo71ug+pFWaKmSKVZKaYKVJJZspwZspsHMTfmpkH9rqPXomIizLzOZNNi4grM/OJ0yz/+cx8ZXX7rzLzTyc8dn5mvnCGfQWNw39+E3hpZu4/kzpSt5kpZopUkplipkglmSn1MqWav3iu1M2UobzE3DT2/B1C558w4iURcVlE3BsRD0bEQxHRq+uLroyIn72Rq9v7Vnd3tLD8IRNuv2DP2u02ExHPqL4huwX4MnAxcHi7daQeMlPMFKkkM8VMkUoyU+plChTMlVKZMqy/iW+mF4ce/C3wazQuH9DrQx/+BPh2RNxI4039S8AbImIxjUtBTKdZ/y2vW0S8F3glcCuNw1tOB9ZnZis9SP3ETDFTpJLMFDNFKslMqZcpUCBXSmfKUA7iI2ITk7+gAfTi8KfbgCv6YCMmM78WEYfQ+MYngGsmnNDhb1sosag6s+IcYOGEsywGjd+WtGotcC3wMeCrmbktInr++kiTMVOmZqZI7TNTpmamSO0zU6ZWIFOgTK4UzZSh/E18RBzU7PHMvKWNWvsDT6vufj8zfzqDfp4GvBu4EPjZCR3auMxJMdG4vMPrgd2/DVlH48yMLZ1MISLW0eQbp8w8vsU6I8ALgZOB5wEXACcAB0x2DUWpl8yUpr2YKVKbzJSmvZgpUpvMlKa91MqUqsY6auZK6UwZ1kH8wcD+mfmdPaY/G7gjM29ssc4rgQ/Q+J8dwLOBt2bm2W32cz7wMLAJ2LV7ema+q506JUTEJ4BRfn74yKuB8cx8Xbd7mdDTAuAlNDbqY4FvZuZv9qofaU9mStNezBSpTWZK017MFKlNZkrTXoYzUzJz6P6ArwJHTTJ9DfCVNupcDuw34f5K4PIZ9LO+16/JxHVqZVqT5Z8GPGLC/d8GvgR8CNi7jToLgD8CPkLj8JK51fSlwKm9fp3882/in5nSfJ1amdZkeTPFv1n3Z6Y0X6dWpjVZ3kzxb9b9mSnN16mVadPUqJ0rpTNlWM9OvyozN+45MTPXA6vaqDMnf/EQknuY2Rn9vxERM7qkSQeMR8Tjdt+JiMcC420s/3GqMzlGxHOAvwQ+DTwAnNlGnbNoBMsm4MXAXwNk5oPpSWPUf8yUqZkpUvvMlKmZKVL7zJSp1c0UKJMrRTNlKE9sR+Objqm0c1KTf4+Ir9M4gyDAScDXZtDPG4G3RcR2YIzG4SmZmUtnUKuutwIXRMRN1f1VwGvaWH4kM++tbp8EnJmZ5wDnRMSGNuo8ITOPBIiITwLfb2NZqdvMlKmZKVL7zJSpmSlS+8yUqdXNFCiTK0UzZVgH8T+IiN/NzH+YODEifge4pI06twP/SeP3IEHjf9i57TaTmXtFxN40rjHY7E3WVESs2LNGZl7UZpnv0Pg26fnV/Y/TWMdWjUTE3GycgOH5NA4H2a2d7elnJ5PIzJ0RXb+EpdQOM2VqZorUPjNlamaK1L6hzBQokit1MwXK5ErRTBnWE9vtD5xL47CH3RvuGmAecGJm/qTFOu8BXgVcCvwj8PWcwQsWEa8D/hB4DLABOBr4bmY+v9lyLdT4z8x8Xpu9fB54EPhMNelkYEVm/kaLy7+DxiEgdwMHAk/JzKxOqHFWZj6rxTrjwObdd2l8S7iF3n5TJ03KTGlax0yR2mSmNK1jpkhtGsZMaVKnrVypmylVjdq5UjpThnIQv1tEHA8cUd29MjO/NYMaQeNyAK+h8Wb4PPDJbPEsj1WNTTROiPC9zFwdEYcD78rMk7pZo6pzeWY+abpp09Q4GngkcH5mbq6mHQosycxL2+lHGiRmyqR1zBRphsyUSeuYKdIMDVOmlKpTIlOqZfoqV4byxHYRcSlAZl6QmR+u/r412TzTqb59+kn1txNYAZwdEe9vo6Vtmbmtet75mXkNcFgby5eqAXBZtRFS1XoGjcNMWhIRl2bm9zLz3N0bMEBmXrd7A27ltS01j9QNZkpTZorUJjOlKTNFatOQZkqpOrUypVqmdq6UzpRh/U384yPiv52hcYIAlk1XJCLeBJxK49CJT9C4TuJYRMwBrgfe1mI/t0fEcuCLwH9ExH3AHS0uW6RG9U1W0rhO4m9HxK3V/YOAq9roo8hrW7CO1A1myh7MFKkWM2UPZopUyzBmSq06BTMFyry+RTNlKA+nj4iDWphtPDNvn6bO6TQOH7llkscen5lXz6C359L4H3ReZu5od/mZ1pjuNZlsHWdSp9LKa1ukjtQNZsqky5gp0gyZKZMuY6ZIMzTsmTKTOqUypZValaavb+lMGcpBvCRJkiRJw2gofxMvSZIkSdIwmjWD+IhYO/1cg1HDXqTe65ftfhjfx/3Ui9Qt/bLdD+P7uJ96kbqlX7b7YXwf90Mvs2YQD5T4n9YvNUrVGcZepG7pl+1+GN/H/dSL1C39st0P4/u4n3qRuqVftvthfB/3vJfZNIiXJEmSJGmgDcWJ7ebNWZAL5+zVdJ4duY15sWDKx8f3mj/t8+zcvpm58xc3nWfkoe21+gDIXbum7WUstzE6TR2m+X87xnZGab7eeei86Xu5fwujyxc1nSeua34SyVZ6eYj77s7MldM2JNU0snhxji7fu+k845s3M7J46jyYMzb98+zcupm5C5tnClG/xpwd0+d8K/m2a17zZlrpZWT59C+MmaJhM29kYS4cbX7loB3jW5g30mS7b+Hj2rQ1gPFFza8u3EoWjM+fJpiA8S2bGVnUvM7cLdN8ThnbzOjoNPn2yOkzZccDW5m3bGHTefK65nXMFPWT0XmLc8HCFU3nGduxmdF5U79/xqf5Nx1g57bNzF3Q/D2Y01ywfOeWzcydJgti+qFPS58xcprd0630smt0+l7GH36YkSVLms4z/7bNTR+vmylDcZ34hXP24phlJ9aq8fBzDinSy5KLrq9dY9fDzf+ntyrHZnwVh5/Z+bEDC3QCc0+4tXaNb+TZLV8KQqpjdPneHPj6N9eqsfAn0//j2Irp/nFsxdJbd9YvAjx4QP1mVrzsRwU6gXkvqB8HZoq6ZeHoMp75mFfXK9LCF/yteGj1I2rXuP9xZT4+7nt58x0frVjwjh8X6ATGj5/JJax/kZmiblmwcAVPeeb/rFXjwYNaGK22YPve9T/vzC0z9GGs+bi6JVsfNV6/CHDI//yv2jWaZYqH00uSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCBqD+IjYlVEbI2IDdX9R0TEv0TEjRFxVUR8LSIOrea7YpLlj46I/4qIDRFxdUS8s5p+UkTcEBFfrdujpMFhpkgqyUyRVJKZon5Q6hJzN2bm6ogI4FzgrMx8FUBErAb2B26bYtmzgFdm5uURMQIcBpCZn4uIO4G3FOpR0uAwUySVZKZIKslMUU+Vvk788cBYZv797gmZuQEa31pNscx+wI+receBq1p5oohYC6wFWDCnwEUBJfWjnmTK3GUrZtywpL7Wm88pc/eaccOS+lpPMmX+guUz7VdDovRv4o8ALmlzmTOAayPi3Ij4vYhY0MpCmXlmZq7JzDXzWltE0uDpSaaMLF7cdqOSBkJvPqeMLGq7UUkDoSeZMjrPzymzXc9PbJeZpwNrgPOB3wTO621HkgaZmSKpJDNFUklmikooPYi/Enhquwtl5o2Z+THg+cCTImKfwn1JGkxmiqSSzBRJJZkp6onSg/hvAfMj4nd3T4iIp0XEc6daICL+R3VSCIBDgHHg/sJ9SRpMZoqkkswUSSWZKeqJoie2y8yMiBOBv42ItwPbgJuBP6pmOSwibp+wyJuBVwBnRMQWYCdwSnWSB0mznJkiqSQzRVJJZop6pfTZ6cnMO4BXTvHw6CTTvlC6B0nDw0yRVJKZIqkkM0W9UOJw+nFgWURsKFDrZyLiJOCjwH0l60rqe2aKpJLMFEklmSnqudp74jPzNuCAAr3sWfdzwOdamnd8nPH76m3vF33szFrL7/Yrv/SM2jVybEeBTsqY97K7i9TZVaSKZoN+yJTRzcn+P6h3ZNutv1pmq1/xg8m+xG/PT58yUqATWLmh/tF+d9y7rEAnsKpIFc0G/ZApADknpp+pidha5rPB+Gi9PgAedeGDBTqBa984v3aNuOWRBTqBQ7ijSB0Nv37IlDk7xll46wO1nu/mV5b593j5vg/XrrHw/y2v3whw12Pq59vBn91WoJPO6/kl5iRJkiRJUmscxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCBqXyd+piLi3cDdmfnB6v57gZ8CjwFeBCTwnuqaiZLUlJkiqSQzRVJJZopK6uWe+E8CpwJExBzgVcDtwGrgScAJwAci4pG9alDSQDFTJJVkpkgqyUxRMT0bxGfmzcA9EfFk4IXAZcCxwGczczwz7wQuBJ422fIRsTYi1kfE+jG2d6ttSX2qaKbs2NyttiX1qZKZsmN8a7faltSnymbKlm61rT7V69/EfwI4DXgN8I9AtLpgZp6ZmWsyc80o8zvUnqQBUyZT5i3uUHuSBkyRTJk3srBD7UkaMIUyZVGH2tOg6PUg/lzgV2h84/R14CLgpIgYiYiVwHOA7/ewP0mDxUyRVJKZIqkkM0VF9OzEdgCZuSMiLgDuz8zxiDgXOAa4nMbJHd6WmT/pZY+SBoeZIqkkM0VSSWaKSunpIL46qcPRwG8AZGYCb63+JKktZoqkkswUSSWZKSqlZ4fTR8QTgBuAb2bm9b3qQ9JwMFMklWSmSCrJTFFJPdsTn5lXAY/t1fNLGi5miqSSzBRJJZkpKqnXJ7aTJEmSJEkt6ulv4vvJCb/52iJ1fvmSC2vX+OaT9ynQCeTYjgI1dhboRBosY0uCHx8zUqvGohvLxOvRv3Np7RpXvuuoAp3A4hvvq13jpzfsW6ATacDs2kVsrnet+PHHrCzTymjLV7Sa0tjSMpf2PeBL9XIWYPX/vqxAJ3BtkSpSl+xKYlu9z/kLfzivSCsnPLn+u2fd4qMLdAIL7qmfbzuWl3ldOn0BdPfES5IkSZI0IBzES5IkSZI0IBzES5IkSZI0IBzES5IkSZI0IBzES5IkSZI0IBzES5IkSZI0IBzES5IkSZI0IDo2iI+IL0bEJRFxZUSsraY9HBF/VU3/RkQ8PSLWRcRNEfGyap4FEfFPEbEpIi6LiOM71aOkwWGmSCrJTJFUkpmiburknvjXZuZTgTXAmyJiH2AxsK6a/hDwHuAFwInA6dVybwTIzCOBk4GzImLBnsUjYm1ErI+I9WNs7+BqSOoTXcuU8c2bO782knqta5myY9fWzq+NpF7rYqZs6fzaqK91chD/poi4HPgecABwCLADOK96fBNwYWaOVbdXVdOPBf4ZIDOvAW4BDt2zeGaemZlrMnPNKPM7uBqS+kTXMmVk8eJOroek/tC1TJk3Z2En10NSf+hipizq5HpoAMztRNGIOA44ATgmM7dExDpgATCWmVnNtgsau9Azc1dE7O4lOtGTpMFlpkgqyUyRVJKZom7r1J74ZcB91UZ8OHB0G8teBJwCEBGHAgcC15ZvUdIAMVMklWSmSCrJTFFXdWoQfx4wNyI2Au+mcVhJqz4KjETEJuBzwGmZ6Y/epdnNTJFUkpkiqSQzRV3VkcPpqw3vRZM8tGTCPO/cY5kl1X+3Aad1oi9Jg8lMkVSSmSKpJDNF3eZ14iVJkiRJGhAO4iVJkiRJGhAdOZx+EM27+vYidS448Um1axx/6aYCncC3nrS0do05S5dMP1MLxu+5t0gdqRtiJyy8q97JYpfcPl6kl+9++im1azz83Jx+phbsu3yf+jWeemeBTqQBM2cOuaTeJaHm3FDmc8ryHY+oXeO2X1lRoBOYM1a/xg/uOrB+EWApNxapI3XDziWj3PPMR9aqMf/+Mr185/3PqF1jZG6ZzylbV2+tXeOn88tcvu+ArxUpMyX3xEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCD6chAfEV+MiEsi4sqIWNvrfiQNNjNFUklmiqSSzBS1a26vG5jCazPz3ohYCPwgIs7JzHsmzlBt4GsBFrCoFz1KGhxtZcroXit60aOkwdHe55S5S3vRo6TB0VamzFvs55TZri/3xANviojLge8BBwCH7DlDZp6ZmWsyc80o87veoKSB0lamjCxa3PUGJQ2UtjJl3sjCrjcoaaC0lSlz5/s5Zbbruz3xEXEccAJwTGZuiYh1wIJe9iRpcJkpkkoyUySVZKZoJvpxT/wy4L5qIz4cOLrXDUkaaGaKpJLMFEklmSlqWz8O4s8D5kbERuDdNA4rkaSZMlMklWSmSCrJTFHb+u5w+szcDryo131IGg5miqSSzBRJJZkpmol+3BMvSZIkSZIm4SBekiRJkqQB4SBekiRJkqQB0Xe/iZ+xiHrL77uiTB8/+kntEuf8zQkFGoE8rX6NhfeO1y8CLPzi94vUkbph9OFdPOI7D9WqMWfLWJFelv9gS+0aDx/5iAKdwIUfP7N2jQ/fd1CBTuCrFMpsqRvGx+G+B2qV2LWlfhYAzPnhj2rXeMz5ZfYBvfOcT9eucfKFawt0AkuLVJG6Y+eS5M5jd9Wqsd9/1hw7VZZvvLd2jfGlZa6od+f99evsOHxrgU46zz3xkiRJkiQNCAfxkiRJkiQNCAfxkiRJkiQNCAfxkiRJkiQNiL4dxEfEkoh4Y6/7kDQczBRJJZkpkkoyU9SOaQfxEbEqIq7oVAMRcVxEPHOSh/4CuL5TzyupN8wUSSWZKZJKMlM0CPphT/xxwC9syBGxCLg4M8/vSUeSBtlxmCmSyjkOM0VSOcdhpqimVgfxcyPirIjYGBFnR8SiiHh+RFwWEZsi4h8jYj5ARLw4Iq6JiG9HxIci4qvV9L0j4otVje9FxFERsQr4feDNEbEhIp4dES8FLgDeERHfiIj9O7HiknrKTJFUkpkiqSQzRX2t1UH8YcCZmXkU8CDwx8CngJMy80hgLvD6iFgAfBx4UWYeC6ycUONdwGVVjf8FfDozbwb+HjgjM1dn5sXAt4GjM3M18AXgbZM1FBFrI2J9RKwfY3s76yyp9/o7U8Y2F19hSR3V15myY9e24issqaP6OlPGH/ZzymzX6iD+tsz8TnX7/wLPB36YmddV084CngMcDtyUmT+spn92Qo1jgX8GyMxvAftExLJJnutRwJcj4mJgLfDEyRrKzDMzc01mrhllfourIalP9HemjC6usWqSeqCvM2XenAU1Vk1SD/R1pows8XPKbNfqID5bnC/afGyyuh8B/i4znw28BfBfPmn4mCmSSjJTJJVkpqivtTqIPzAijqlunwx8A1gVEQdX014NXAhcAzy2+r0HwEkTalwEnAKNszICd2fmg8BDwF4T5lsB3FXdPrXVFZE0UMwUSSWZKZJKMlPU11odxF8NnBoRG4G9gTOA1wBfiIhNwC7g7zNzK/AG4LyI+DZwJ/BAVeOdwJqqxl/y8430K8CJu0/uAJwOnF0dUrJ7g5Y0XMwUSSWZKZJKMlPU1+ZON0N1AoYnTPLQN4EnTzL9gsw8PCIC+DtgfVXnXuDlk9S/Djhqj8n/Ol1fkgaTmSKpJDNFUklmigZBJ64T/7sRsQG4ElhG44yNkjRTZoqkkswUSSWZKeq6affEtyszz6BxyIkk1WamSCrJTJFUkpmiXig+iB9U9x21okidFT+9t3aNvT9zSYFOIMd21K6x/fxV9RsB+GKZMlI3jO01hx8dv9f0Mzax8Ketnti2uSU/rtcHwLYVIwU6gaf/2etr1/jQ//eRAp3AV3lqkTpSN+xcuoD7XnhIrRorvnZ1kV7GDz2wdo0cKXMg5+//7f+sXePxJ95cvxFgvEgVqTvm35sc8n/rfc6/60mLivRy7Z/Vv9zdfv9e5nLhcx9sdrGA1iy6bjCGx504nF6SJEmSJHWAg3hJkiRJkgaEg3hJkiRJkgaEg3hJkiRJkgaEg3hJkiRJkgaEg3hJkiRJkgZE7UF8RKyKiK0RsaG6/46IuDIiNkbEhoh4RjV9XURcW03bEBFnV9PfGRE/qqZdEREvq6a/OSJujYgy1yOSNBDMFEklmSmSSjJT1A9KXQjvxsxcHRHHAC8BnpKZ2yNiX2DehPlOycz1kyx/Rmb+dUQ8Hrg4IvbLzDMi4j5gTaEeJQ0OM0VSSWaKpJLMFPVU6avZPxK4OzO3A2Tm3e0snJlXR8ROYF/gp83mjYi1wFqABSyaWbeS+l1PMmXu0hUz61ZSv+tJpsxbZKZIQ6o3Y5/5y2bWrYZG6d/Enw8cEBHXRcRHI+K5ezz+mQmHlHxgz4Wrw092AXdN90SZeWZmrsnMNaPML9O9pH7Tk0yZu2hxme4l9ZveZMoCM0UaUr0Z+4yaKbNd0T3xmflwRDwVeDZwPPC5iHh7Zn6qmmWqQ0reHBG/BTwEnJSZWbIvSYPJTJFUkpkiqSQzRb1S+nB6MnMcWAesi4hNwKnAp6ZZ7IzM/OvSvUgafGaKpJLMFEklmSnqhaKH00fEYRFxyIRJq4FbSj6HpNnDTJFUkpkiqSQzRb1Sek/8EuDDEbEc2AncQHUChspnImJrdfvuzDyh8PNLGi5miqSSzBRJJZkp6onSv4m/BHjmFI8dN8X0d5bsQdLwMFMklWSmSCrJTFGvlDicfhxYFhEbCtT6mYh4M/BnwIMl60rqe2aKpJLMFEklmSnqudp74jPzNuCAAr3sWfcM4IzSdSX1NzNFUklmiqSSzBT1g+Jnp++ZqHdQwbIbNpdpY9GC2jV2HfyoAp3AnB07a9f40aVLC3QCj+XmInWkbhh9aBePvvDhWjW27Vs/CwAW/KR+Ns19eF6BTuDVn/hq7Ro/2rmiQCfSYJl7/1aWf+XKWjV2bd06/Uyt9HLbtJejntbOA1YW6AQ2vP2jtWsc9k+vL9AJrOKOInWkbtg1dw5b95tfq8Y+V28r0su+G+tfHW9s2XiBTuD63/6H2jWO+ps3FOik84qenV6SJEmSJHWOg3hJkiRJkgaEg3hJkiRJkgaEg3hJkiRJkgaEg3hJkiRJkgZErUF8RKyKiK27r5MYEeMRsSEiroiIL0TEomr63Ii4OyLet8fy6yLi2ojYGBHXRMRHImJ59djCqtaOiNi3Tp+SBoOZIqkkM0VSSWaK+kWJPfE3Zubq6vbWzFydmUcAO4Dfr6a/ELgWeGVExB7Ln5KZRwFHAduBLwFk5taqrtf8kGYXM0VSSWaKpJLMFPVcJw+nvxg4uLp9MvBB4Fbg6MlmzswdwNuAAyPiSR3sS9JgMlMklWSmSCrJTFHXdGQQHxFzgRcBmyJiIfB84KvAZ2ls1JPKzHHgcuDwFp5jbUSsj4j1Y2wv07ikvtT1TBnbXKZxSX2p25myI7eVaVxSX+r655TtD5dpXAOr9CB+YfUbkfU0vnn6JPAS4ILM3AKcA5wYESNNaux5yMmkMvPMzFyTmWtGmV+zbUl9qjeZMrq4ZtuS+lRPMmVeLKjZtqQ+1ZvPKfOX1Gxbg25u4Xq7f8vxMxFxMvCsiLi5mrQPcDzwjT0XrjbwI4GrC/claTCZKZJKMlMklWSmqCc6eom5iFgKHAscmJmrMnMV8EYmOawkIkaB9wG3ZebGTvYlaTCZKZJKMlMklWSmqFs6fZ34XwO+lZkTf7T+JeBlEbH7GPjPRMRG4ApgMfDyDvckaXCZKZJKMlMklWSmqCuKHk6fmUv2uP8p4FN7TLsXWFndPa7k80saLmaKpJLMFEklmSnqlbp74seBZdUJHYqKiN0nihgFdpWuL6kvmSmSSjJTJJVkpqgv1NoTn5m3AQcU6mXP2luB1Z2oLak/mSmSSjJTJJVkpqhfRGb2uofaIuIu4JZpZtsXuLvmU/VLjdnay0GZuXKaeaTazJRZ04uZoq4wU2ZNL2aKusJMmTW9TJkpQzGIb0VErM/MNcNQw16k3uuX7X4Y38f91IvULf2y3Q/j+7ifepG6pV+2+2F8H/dDL50+O70kSZIkSSrEQbwkSZIkSQNiNg3izxyiGqXqDGMvUrf0y3Y/jO/jfupF6pZ+2e6H8X3cT71I3dIv2/0wvo973sus+U18P4mIhydeVzIiTgPWZOYfFKi9DnhLZq7fY/qngOcCD1STTsvMDXWfT1Lv9ShTAngP8Bs0Lrnzscz8UN3nk9R7PcqUi4G9qrv7Ad/PzF+t+3ySeq9HmfJ84AM0dlo/TGPsc0Pd5+sXtS4xp4Hz1sw8u9dNSBoKp9G4zM7hmbkrIvbrcT+SBlhmPnv37Yg4B/hSD9uRNPg+Brw8M6+OiDcA/5vGZ5ehMJsOpx8IEbEyIs6JiB9Uf8+qpj89Ir4bEZdV/z2smr4wIv4lIjZGxOeAhT1dAUl9pYOZ8nrg9MzcBZCZP+3KCknqqU5/TomIvYDnAV/s9LpI6r0OZkoCS6vby4A7Or4yXeSe+N5YGBEbJtzfG/hydfuDwBmZ+e2IOBD4OvB44BrgOZm5MyJOAP4CeAWND9JbMvOoiDgKuLTJ8743Iv4c+Cbw9szcXnStJPVKLzLlccBJEXEicBfwpsy8vvSKSeqJXn1OATgR+GZmPlhudST1WC8y5XXA1yJiK/AgcHTpleolB/G9sTUzV+++s/t3IdXdE4AnNH5uCsDS6lvpZcBZEXEIjW+WRqvHnwN8CCAzN0bExime88+AnwDzaJxI4U+B0wutj6Te6kWmzAe2ZeaaiPg14B+BZ08xr6TB0otM2e1k4BMF1kFS/+hFprwZeHFm/ldEvBX4PzQG9kPBQXz/mQMck5lbJ06MiA8DF2TmiRGxClg34eFpz06YmT+ubm6PiH8C3lKmXUl9riOZAtwOnFPdPhf4p/qtShoAncoUImIf4Ok09sZLmh2KZ0pErASelJn/VU36HHBesY77gL+J7z/nAz87U2NErK5uLgN+VN0+bcL8FwGnVPMeARw1WdGIeGT13wB+FbiiXMuS+lhHMoXG71WfV91+LnBdiWYl9b1OZQo0rnbx1czcVqhXSf2vE5lyH7AsIg6t7r8AuLpYx33AQXz/eROwpjpZw1XA71fT3w+8LyK+A4xMmP9jwJLqUJK3Ad+fou5nImITsAnYl8aloSQNv05lyl8Cr6hy5X0M0SFqkprqVKYAvAr4bAd6ltS/imdKZu4Efhc4JyIuB14NvLWD69B1XidekiRJkqQB4Z54SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGhIN4SZIkSZIGxNxeN6DuioiVwJ8CTwAW7J6emc/rWVOSBpaZIqkkM0VSScOaKe6JnwUi4g0RcUJ19zPAdcBK4HTgFuAHvepN0uAxUySVZKZIKmk2ZIqD+NnhLOClEfHrwL6ZeSawNTPXZeZrgKN7256kAWOmSCrJTJFU0tBnioP4WSAzN2fmHwL/BuyoJt8fES+IiCcDB/SuO0mDxkyRVJKZIqmk2ZApkZm97kFdFBEvAS4GHg18BFgOnJ6ZX+xhW5IGlJkiqSQzRVJJw5opnthu9rkvMx8AHgCeBxARz+ptS5IGmJkiqSQzRVJJQ5kp7omfZSLi0sx8ynTTutDH3i3Mtisz7+90L5JmzkyRVJKZIqmkYc0U98TPEhFxDPBMYGVE/PGEh5YCIz1o6Y7qL5rMMwIc2J12JLXDTJFUkpkiqaRhz5ShHMRHxJdbmO3ezDyt0730kXnAEhr/z/eaMP1B4NdbLRIRH2phtgcz839PM8/VmfnkaZ7rslb7kjrJTJmUmSLNkJkyKTNFmiEzZVJFMgWK5UrRTBnKw+kj4nrgdc1mAf4uM5/YpZb6RkQclJm3RMReQGbmw20ufwvw59PM9vbMfPw0dRZk5ra680jdYKZMzUyR2memTM1MkdpnpkytbqZUNWrnSulMGco98cA7MvPCZjNExLu61Uyf2av6lmdvgIi4Gzg1M69ocfkzMvOsZjNExIrpiuzeQCPiSODwavLVE/vwH0b1ETNlamaK1D4zZWpmitQ+M2VqdTMFCuRK6UwZyj3xk6le2PtztqzwFCLiuzTe6BdU948D/iIzn9lmnX0z8+4afSwDvkTjOo0baXxDeCRwK/DyzHxwprWlbjBTGswUqQwzpcFMkcowUxpKZUq17IxzpXSmzJlJE/0uIv48Ig6vbs+PiAuAG4E7I+KEHvU0LyKOqP5Ge9FDZfHujRggM9cBi1tdOCJeGhF3AZsi4vaIaPsNUHk3sB44JDNPzMxfBQ4BfgC8d4Y1pY4wU5oyU6Q2mSlNmSlSm8yUpmplChTLlaKZMpR74iPiSuCIzMyIWAucDJwAHAqclZlP73I/xwFnATfT+NblABqHcVzUzT6qXs4FLgX+uZr0W8CaakNqZfmNwCsz85qIeAbw/sx87gz6uAo4KjN37jF9LrBput+qSd1kpjTtxUyR2mSmNO3FTJHaZKY07aVWplQ1audK6UwZ1t/E75hw6MgvA/+SmePA1dUL1W1/A7wwM68FiIhDgc8CT+1BL68F3gX8K4031UXAa9pYfmdmXgOQmf8VjZNEzMSOPTfiqubOiNg+w5pSp5gpUzNTpPaZKVMzU6T2mSlTq5spUCZXimbKsA7it0fEEcCdwPHAWyY81tbhE4WM7t6IATLzul4dVpKZ9wFvqlFiv/jFay3+wv3M/D8t1lkQEU+G/3atxADm1+hP6gQzZQpmijQjZsoUzBRpRsyUKRTIFCiTK0UzZVgH8X8InA2spHE2wR8CRMSLaRxO0W3rI+KT/PwwjlOAS3rQBxHxFWDP31A8QOM3Gh/P6c+K+A/84rUW97zfqh8DU23wP5lBPamTzJQpmCnSjJgpUzBTpBkxU6ZQIFOgTK4UzZSh/E18MxHxisw8p8vPOR94I3AsPz+M46OZ2fXDsSLigzTe4J+tJp1EY8NZCCzNzFd3u6c9RcRoZo71ug+pFWaKmSKVZKaYKVJJZspwZspsHMTfmpkH9rqPXomIizLzOZNNi4grM/OJ0yz/+cx8ZXX7rzLzTyc8dn5mvnCGfQWNw39+E3hpZu4/kzpSt5kpZopUkplipkglmSn1MqWav3iu1M2UobzE3DT2/B1C558w4iURcVlE3BsRD0bEQxHRq+uLroyIn72Rq9v7Vnd3tLD8IRNuv2DP2u02ExHPqL4huwX4MnAxcHi7daQeMlPMFKkkM8VMkUoyU+plChTMlVKZMqy/iW+mF4ce/C3wazQuH9DrQx/+BPh2RNxI4039S8AbImIxjUtBTKdZ/y2vW0S8F3glcCuNw1tOB9ZnZis9SP3ETDFTpJLMFDNFKslMqZcpUCBXSmfKUA7iI2ITk7+gAfTi8KfbgCv6YCMmM78WEYfQ+MYngGsmnNDhb1sosag6s+IcYOGEsywGjd+WtGotcC3wMeCrmbktInr++kiTMVOmZqZI7TNTpmamSO0zU6ZWIFOgTK4UzZSh/E18RBzU7PHMvKWNWvsDT6vufj8zfzqDfp4GvBu4EPjZCR3auMxJMdG4vMPrgd2/DVlH48yMLZ1MISLW0eQbp8w8vsU6I8ALgZOB5wEXACcAB0x2DUWpl8yUpr2YKVKbzJSmvZgpUpvMlKa91MqUqsY6auZK6UwZ1kH8wcD+mfmdPaY/G7gjM29ssc4rgQ/Q+J8dwLOBt2bm2W32cz7wMLAJ2LV7ema+q506JUTEJ4BRfn74yKuB8cx8Xbd7mdDTAuAlNDbqY4FvZuZv9qofaU9mStNezBSpTWZK017MFKlNZkrTXoYzUzJz6P6ArwJHTTJ9DfCVNupcDuw34f5K4PIZ9LO+16/JxHVqZVqT5Z8GPGLC/d8GvgR8CNi7jToLgD8CPkLj8JK51fSlwKm9fp3882/in5nSfJ1amdZkeTPFv1n3Z6Y0X6dWpjVZ3kzxb9b9mSnN16mVadPUqJ0rpTNlWM9OvyozN+45MTPXA6vaqDMnf/EQknuY2Rn9vxERM7qkSQeMR8Tjdt+JiMcC420s/3GqMzlGxHOAvwQ+DTwAnNlGnbNoBMsm4MXAXwNk5oPpSWPUf8yUqZkpUvvMlKmZKVL7zJSp1c0UKJMrRTNlKE9sR+Objqm0c1KTf4+Ir9M4gyDAScDXZtDPG4G3RcR2YIzG4SmZmUtnUKuutwIXRMRN1f1VwGvaWH4kM++tbp8EnJmZ5wDnRMSGNuo8ITOPBIiITwLfb2NZqdvMlKmZKVL7zJSpmSlS+8yUqdXNFCiTK0UzZVgH8T+IiN/NzH+YODEifge4pI06twP/SeP3IEHjf9i57TaTmXtFxN40rjHY7E3WVESs2LNGZl7UZpnv0Pg26fnV/Y/TWMdWjUTE3GycgOH5NA4H2a2d7elnJ5PIzJ0RXb+EpdQOM2VqZorUPjNlamaK1L6hzBQokit1MwXK5ErRTBnWE9vtD5xL47CH3RvuGmAecGJm/qTFOu8BXgVcCvwj8PWcwQsWEa8D/hB4DLABOBr4bmY+v9lyLdT4z8x8Xpu9fB54EPhMNelkYEVm/kaLy7+DxiEgdwMHAk/JzKxOqHFWZj6rxTrjwObdd2l8S7iF3n5TJ03KTGlax0yR2mSmNK1jpkhtGsZMaVKnrVypmylVjdq5UjpThnIQv1tEHA8cUd29MjO/NYMaQeNyAK+h8Wb4PPDJbPEsj1WNTTROiPC9zFwdEYcD78rMk7pZo6pzeWY+abpp09Q4GngkcH5mbq6mHQosycxL2+lHGiRmyqR1zBRphsyUSeuYKdIMDVOmlKpTIlOqZfoqV4byxHYRcSlAZl6QmR+u/r412TzTqb59+kn1txNYAZwdEe9vo6Vtmbmtet75mXkNcFgby5eqAXBZtRFS1XoGjcNMWhIRl2bm9zLz3N0bMEBmXrd7A27ltS01j9QNZkpTZorUJjOlKTNFatOQZkqpOrUypVqmdq6UzpRh/U384yPiv52hcYIAlk1XJCLeBJxK49CJT9C4TuJYRMwBrgfe1mI/t0fEcuCLwH9ExH3AHS0uW6RG9U1W0rhO4m9HxK3V/YOAq9roo8hrW7CO1A1myh7MFKkWM2UPZopUyzBmSq06BTMFyry+RTNlKA+nj4iDWphtPDNvn6bO6TQOH7llkscen5lXz6C359L4H3ReZu5od/mZ1pjuNZlsHWdSp9LKa1ukjtQNZsqky5gp0gyZKZMuY6ZIMzTsmTKTOqUypZValaavb+lMGcpBvCRJkiRJw2gofxMvSZIkSdIwmjWD+IhYO/1cg1HDXqTe65ftfhjfx/3Ui9Qt/bLdD+P7uJ96kbqlX7b7YXwf90Mvs2YQD5T4n9YvNUrVGcZepG7pl+1+GN/H/dSL1C39st0P4/u4n3qRuqVftvthfB/3vJfZNIiXJEmSJGmgDcWJ7ebNWZgLR/ZqOs+OXVuZN2fh1DO08DrsyG3MiwVN58mF85rXGNvMvNHFzZ9o1/S9jO3cwujcRU3niW3NT9zYyvrEIdN/z7Pj/q3MW97ktQV2Xbuz6eNjbGeU+U3neYj77s7MldM2JNU0LxbkwjlLms4z3ftn1+Lm7y2AsbHNjE6TBzv329X08fEHtzCytHkW5JaRaXsZ37KZkUXNexnd3DybxnZsZnRe8xpztjXPAoAdu7Ywb84067Sjeb6ZKeoncxcuztGlezedZ3zrZkYWTv3+mXfv9mmfZ9rPOsCOfZu/L3Zu2czcabJg3v0tvI/HtzBvZJr38dzmnzFa+cx06GPvmbaXu+4ZZ+U+zXPwuo3NezVT1E/mjS7OBfOXN51n2s8YLYx9WhlvNK6M1qzGZkbnNn8fjy+cfryxc9tm5i5oXufxj76r6eOtZMGVd07/Fp4urwFG79zc9PG6mTIU14lfOLIXx+z96/WK7Jz+H6RWjD2xlasHNDfSwofcVsRVN9WuMf8fmn850qqtz72zdo1v5NktXwpCqmPhnCUcvegltWpsP/rwIr3c9cattWvs3LC8fiPA/t8fq11j8TU/LdAJ7Pxh/TgwU9Qto0v35uDf/ONaNR79/64v0sstv3NI7RoHffHuAp3A2N7TDQ6m9x+f/1T9RoBfftTq2jXMFHXLgvnLecYRv1erRrSw07C1Qs0H8a249wnNd5y06vvv+1jtGkee8YYCncCjPvDd2jWaZYqH00uSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCAcxEuSJEmSNCBqD+IjYlVEbI2IDdX9R0TEv0TEjRFxVUR8LSIOrea7YpLlj46I/4qIDRFxdUS8s5p+UkTcEBFfrdujpMFhpkgqyUyRVJKZon5Q6uz0N2bm6ogI4FzgrMx8FUBErAb2B26bYtmzgFdm5uURMQIcBpCZn4uIO4G3FOpR0uAwUySVZKZIKslMUU+VvsTc8cBYZv797gmZuQEa31pNscx+wI+receBqwr3JGlwmSmSSjJTJJVkpqgnSv8m/gjgkjaXOQO4NiLOjYjfi4gFrSwUEWsjYn1ErN+xq/51lCX1pd5kSm5ru1FJA6EnmTK+dXPbjUoaCD3JlLExM2W26/mJ7TLzdGANcD7wm8B5LS53Zmauycw18+Ys7GSLkgZIkUxp7d9TSbNAiUwZWbi4ky1KGiAlMmV01EyZ7UoP4q8EntruQpl5Y2Z+DHg+8KSI2KdwX5IGk5kiqSQzRVJJZop6ovQg/lvA/Ij43d0TIuJpEfHcqRaIiP9RnRQC4BBgHLi/cF+SBpOZIqkkM0VSSWaKeqLoie0yMyPiROBvI+LtwDbgZuCPqlkOi4jbJyzyZuAVwBkRsQXYCZxSneRB0ixnpkgqyUyRVJKZol4pfXZ6MvMO4JVTPDw6ybQvlO5B0vAwUySVZKZIKslMUS+UOJx+HFgWERsK1PqZiDgJ+ChwX8m6kvqemSKpJDNFUklminqu9p74zLwNOKBAL3vW/RzwudJ1JfU3M0VSSWaKpJLMFPWD4ofT90Lu3Mn4PffWqnH/bz29SC9bV8b0M03jMZ+6pkAnML51a+0a16xbXb8R4CDuLFJH6op5o8RBj65VYuH1Py3SyoqzHlm7xm3/Y6xAJ/DT8cmOCmzPo7aVOQHvyA9vKVJH6oq9xtn5nAdqlcgLVxZpZXz1Q7Vr5Ad/VKAT2PqEI2rXOPTTry/QCfwS/1mkjtQN25fP4aZXLKlV41EXl/kZfoxn7Rr7XFLm4IOLttWvsd8l2+sX6YKeXydekiRJkiS1xkG8JEmSJEkDwkG8JEmSJEkDwkG8JEmSJEkDwkG8JEmSJEkDwkG8JEmSJEkDwkG8JEmSJEkDomfXiY+IdwN3Z+YHq/vvBX4KPAZ4EZDAezLzc73qUdLgMFMklWSmSCrJTFFJvdwT/0ngVICImAO8CrgdWA08CTgB+EBEPHKyhSNibUSsj4j1Y2zvTseS+lmxTNkxvqU7HUvqZ8UyZeeDZoqkcpmya/Pm7nSsvtWzQXxm3gzcExFPBl4IXAYcC3w2M8cz807gQuBpUyx/Zmauycw1o8zvVtuS+lTJTJk3sqhbbUvqUyUzZe5SM0Wa7UpmypzFi7vVtvpUzw6nr3wCOA14BPCPNDZoSZopM0VSSWaKpJLMFBXR6xPbnQv8Co1vnL4OXAScFBEjEbESeA7w/R72J2mwmCmSSjJTJJVkpqiInu6Jz8wdEXEBcH9mjkfEucAxwOU0Tu7wtsz8SS97lDQ4zBRJJZkpkkoyU1RKTwfx1UkdjgZ+AyAzE3hr9SdJbTFTJJVkpkgqyUxRKT07nD4ingDcAHwzM6/vVR+ShoOZIqkkM0VSSWaKSurZnvjMvAp4bK+eX9JwMVMklWSmSCrJTFFJvT6xnSRJkiRJalGvLzFXzq7xWovv/fnLirTx7zd9r3aNF//Dcwp0AmTWLvHYz99ToBGo939H6q6cO4edK+pd13nsgKVFellw147aNfb53sICncD8B+u/k2994fwCncAvXVCkjNQVu3YF27bOq1Vjxd/dXKSXn551eP0i42X+VV9yy5baNXYsWVKgE2mwzBmHeQ9ErRq3vqz+OAFg3p31h5OP++exAp3A6//hDbVrrFxYppdOD7LdEy9JkiRJ0oBwEC9JkiRJ0oBwEC9JkiRJ0oBwEC9JkiRJ0oBwEC9JkiRJ0oBwEC9JkiRJ0oDo2CA+Ir4YEZdExJURsbaa9nBE/FU1/RsR8fSIWBcRN0XEy6p5FkTEP0XEpoi4LCKO71SPkgaHmSKpJDNFUklmirqpk3viX5uZTwXWAG+KiH2AxcC6avpDwHuAFwAnAqdXy70RIDOPBE4GzoqIBXsWj4i1EbE+ItaPsb2DqyGpT3QvU8Y2d35tJPVa1zJl/CEzRZoFupYpO7eYKbNdJwfxb4qIy4HvAQcAhwA7gPOqxzcBF2bmWHV7VTX9WOCfATLzGuAW4NA9i2fmmZm5JjPXjDK/g6shqU90L1NGF3dyPST1h65lysheZoo0C3QtU+YuMlNmu7mdKBoRxwEnAMdk5paIWAcsAMYyM6vZdkFjF3pm7oqI3b1EJ3qSNLjMFEklmSmSSjJT1G2d2hO/DLiv2ogPB45uY9mLgFMAIuJQ4EDg2vItShogZoqkkswUSSWZKeqqTg3izwPmRsRG4N00Ditp1UeBkYjYBHwOOC0z/dG7NLuZKZJKMlMklWSmqKs6cjh9teG9aJKHlkyY5517LLOk+u824LRO9CVpMJkpkkoyUySVZKao27xOvCRJkiRJA8JBvCRJkiRJA8JBvCRJkiRJA6Ijv4nviah5dYbR0SJt/MrLfqt2jZs+PlKgE3jc71xfu8Ydz9unQCew/5VFykhdMbZkDnccu6hWjYM+fVORXnLHjto19r9pXoFOYGzV/rVrvOE9Xy3QCXz6zw4oUkfqhpGH5rDiwgW1anz/J4cV6eWXrtpau8b4U8r08qSPbKxdY8ObV9dvRBowGbCr5tDlEevKjDfm37+zdo3Nh5UZb2zfd1ftGrcfV2Z4/Lh/K1JmSu6JlyRJkiRpQDiIlyRJkiRpQDiIlyRJkiRpQDiIlyRJkiRpQDiIlyRJkiRpQDiIlyRJkiRpQDiIlyRJkiRpQPTlID4ivhgRl0TElRGxttf9SBpsZoqkkswUSSWZKWpXmavZl/fazLw3IhYCP4iIczLznokzVBv4WoAFLOpFj5IGR1uZMnfpil70KGlwtJUpo0vMFElNtfc5ZZmZMtv15Z544E0RcTnwPeAA4JA9Z8jMMzNzTWauGWV+1xuUNFDaypS5ixZ3vUFJA6W9TFlgpkhqqq1MGfFzyqzXd3viI+I44ATgmMzcEhHrgAW97EnS4DJTJJVkpkgqyUzRTPTjnvhlwH3VRnw4cHSvG5I00MwUSSWZKZJKMlPUtn4cxJ8HzI2IjcC7aRxWIkkzZaZIKslMkVSSmaK29d3h9Jm5HXhRr/uQNBzMFEklmSmSSjJTNBP9uCdekiRJkiRNwkG8JEmSJEkDwkG8JEmSJEkDou9+Ez8jEcTc0Xo1xsaKtDLnxh/VrvG49+5foBO47X+url1jyxO21W8EKLNGUneMbIflN4zXqrFrvxVFeoltBbJp2/b6NYCRjTfUrnHKXvcU6AQ+zQFF6kjdML4kuffYeu/DQz+8o0wvC2t+XgLm3re5QCfwlRuOqF1j7OXzC3QCB19YpIzUFZEwUvMj+pJby3zGj6xf44HHLaxfBBhfXO+zGwDzC9ToAvfES5IkSZI0IBzES5IkSZI0IBzES5IkSZI0IPp2EB8RSyLijb3uQ9JwMFMklWSmSCrJTFE7ph3ER8SqiLiiUw1ExHER8cxJHvoL4PpOPa+k3jBTJJVkpkgqyUzRIOiHPfHHAb+wIUfEIuDizDy/Jx1JGmTHYaZIKuc4zBRJ5RyHmaKaWh3Ez42IsyJiY0ScHRGLIuL5EXFZRGyKiH+MiPkAEfHiiLgmIr4dER+KiK9W0/eOiC9WNb4XEUdFxCrg94E3R8SGiHh2RLwUuAB4R0R8IyK8Opk0fMwUSSWZKZJKMlPU11odxB8GnJmZRwEPAn8MfAo4KTOPpHG9+ddHxALg48CLMvNYYOWEGu8CLqtq/C/g05l5M/D3wBmZuTozLwa+DRydmauBLwBvm6yhiFgbEesjYv1YlrnOoaSu6e9M2f5w8RWW1FF9nSnjD5W5rrqkrunrTNm5xUyZ7VodxN+Wmd+pbv9f4PnADzPzumraWcBzgMOBmzLzh9X0z06ocSzwzwCZ+S1gn4hYNslzPQr4ckRcDKwFnjhZQ5l5Zmauycw1o7GgxdWQ1Cf6O1PmL6mxapJ6oK8zZWSvxTVWTVIP9HWmzF1kpsx2rQ7is8X5os3HJqv7EeDvMvPZwFsAR+jS8DFTJJVkpkgqyUxRX2t1EH9gRBxT3T4Z+AawKiIOrqa9GrgQuAZ4bPV7D4CTJtS4CDgFGmdlBO7OzAeBh4C9Jsy3Arirun1qqysiaaCYKZJKMlMklWSmqK+1Ooi/Gjg1IjYCewNnAK8BvhARm4BdwN9n5lbgDcB5EfFt4E7ggarGO4E1VY2/5Ocb6VeAE3ef3AE4HTi7OqRk9wYtabiYKZJKMlMklWSmqK/NnW6G6gQMT5jkoW8CT55k+gWZeXhEBPB3wPqqzr3Ayyepfx1w1B6T/3W6viQNJjNFUklmiqSSzBQNgk5cJ/53I2IDcCWwjMYZGyVppswUSSWZKZJKMlPUddPuiW9XZp5B45ATSarNTJFUkpkiqSQzRb3QiT3xkiRJkiSpA4rvie+JTHJ8vFaJGC3zUsTypfWL3HlP/RrAo//qmto1vvajSwt0Ai/mKUXqSN0wZ2ey4N6xWjW2PbLMteZ3Lqz/XeucsVavlNPcwh/Xz7fHnn94gU7gEC4pUkfqhpWLHuL3nnJxrRrrrn1EkV62/vJkP/Vtz7zbdxToBFb+y961a+TanxboRBosu+Ylm1ftrFVj/JKRIr1EgY8Yy67fUr8IEK/aXLvGlm/uV6CTznNPvCRJkiRJA8JBvCRJkiRJA8JBvCRJkiRJA8JBvCRJkiRJA8JBvCRJkiRJA6L2ID4iVkXE1ojYUN1/R0RcGREbI2JDRDyjmr4uIq6tpm2IiLOr6e+MiB9V066IiJdV098cEbdGxEfq9ihpcJgpkkoyUySVZKaoH5S6xNyNmbk6Io4BXgI8JTO3R8S+wLwJ852SmesnWf6MzPzriHg8cHFE7JeZZ0TEfcCaQj1KGhxmiqSSzBRJJZkp6qnS14l/JHB3Zm4HyMy721k4M6+OiJ3AvoAX/pRkpkgqyUyRVJKZop4o/Zv484EDIuK6iPhoRDx3j8c/M+GQkg/suXB1+Mku4K7pnigi1kbE+ohYP8b2Mt1L6jc9yZQdOzaX6V5Sv+lJpmy+b0eZ7iX1m55kyvjDfk6Z7Yruic/MhyPiqcCzgeOBz0XE2zPzU9UsUx1S8uaI+C3gIeCkzMwWnutM4EyApbH3tPNLGjw9y5SljzFTpCHUq0x5zBOXmSnSEOpVpsw/yM8ps13pw+nJzHFgHbAuIjYBpwKfmmaxMzLzr0v3ImnwmSmSSjJTJJVkpqgXih5OHxGHRcQhEyatBm4p+RySZg8zRVJJZoqkkswU9UrpPfFLgA9HxHJgJ3ADsHbC45+JiK3V7bsz84TCzy9puJgpkkoyUySVZKaoJ0r/Jv4S4JlTPHbcFNPfWbIHScPDTJFUkpkiqSQzRb1S4nD6cWBZRGwoUOtnIuLNwJ8BD5asK6nvmSmSSjJTJJVkpqjnau+Jz8zbgAMK9LJn3TOAM0rXldTfzBRJJZkpkkoyU9QPSl8nXpIkSZIkdUjxS8wNqhzbWabO3JH6ReZE/RrAva89pnaN5256bIFOYDE3FakjdUOMjTPvjnpHs9331H2L9PLQAfW/a11x/XiBTiC2168z75b5BTqRBsu9dy/lXz7+glo1HrX4xiK9bFtWP1OW7L2kQCdw8Nuuql3jou8+sUAncLCfUzRA5mwPltxUdxhX5rPB2F71xz7bl5cZks4bqf9LhC1ZoJEucE+8JEmSJEkDwkG8JEmSJEkDwkG8JEmSJEkDwkG8JEmSJEkDwkG8JEmSJEkDotYgPiJWRcTWiNhQ3R+PiA0RcUVEfCEiFlXT50bE3RHxvj2WXxcR10bExoi4JiI+EhHLq8cWVrV2RESZ0zxL6mtmiqSSzBRJJZkp6hcl9sTfmJmrq9tbM3N1Zh4B7AB+v5r+QuBa4JURsef1007JzKOAo4DtwJcAMnNrVfeOAj1KGhxmiqSSzBRJJZkp6rlOHk5/MXBwdftk4IPArcDRk82cmTuAtwEHRsSTpiseEWsjYn1ErB9je6GWJfWxrmXKjp1bCrUsqY91LVN2bt1cqGVJfaxrmTJupsx6HRnER8Rc4EXApohYCDwf+CrwWRob9aQycxy4HDh8uufIzDMzc01mrhllfpnGJfWlbmfKvLmLyjQuqS91O1PmLlxcpnFJfanbmTJipsx6pQfxC6vfiKyn8c3TJ4GXABdk5hbgHODEiBhpUmPPQ04kzV5miqSSzBRJJZkp6om5hevt/i3Hz0TEycCzIuLmatI+wPHAN/ZcuNrAjwSuLtyXpMFkpkgqyUyRVJKZop7o6CXmImIpcCxwYGauysxVwBuZ5LCSiBgF3gfclpkbO9mXpMFkpkgqyUyRVJKZom7p9HXifw34VmZOPPPcl4CXRcTuH7J/JiI2AlcAi4GXd7gnSYPLTJFUkpkiqSQzRV1R9HD6zFyyx/1PAZ/aY9q9wMrq7nEln1/ScDFTJJVkpkgqyUxRr9TdEz8OLKtO6FBUROw+UcQosKt0fUl9yUyRVJKZIqkkM0V9odae+My8DTigUC971t4KrO5EbUn9yUyRVJKZIqkkM0X9IjKz1z3UFhF3AbdMM9u+wN01n6pfaszWXg7KzJXTzCPVZqbMml7MFHWFmTJrejFT1BVmyqzpZcpMGYpBfCsiYn1mrhmGGvYi9V6/bPfD+D7up16kbumX7X4Y38f91IvULf2y3Q/j+7gfeun02eklSZIkSVIhDuIlSZIkSRoQs2kQf+YQ1ShVZxh7kbqlX7b7YXwf91MvUrf0y3Y/jO/jfupF6pZ+2e6H8X3c815mzW/i+0lEPDzxupIRcRqwJjP/oEDtdcBbMnP9HtOfB/w1MA+4BPidzNxZ9/kk9UaPcuQPgD8CHgeszMy7q+kBfBB4MbAFOC0zL63bh6Tu6bNMORz4J+ApwDsy86/r9iCpu/osU04B/rSa7WHg9Zl5ed0+emk27YmftSJiDnAW8KrMPILG2SxP7W1XkgbQd4AT+O9nxH0RcEj1txb4WJf7kjSYpsqUe4E30dj5IEmtmipTfgg8NzOPAt7NEBxZ4yC+z0TEyog4JyJ+UP09q5r+9Ij4bkRcVv33sGr6woj4l4jYGBGfAxZOUnYfYHtmXlfd/w/gFV1ZIUld16EcITMvy8ybJ3no5cCns+F7wPKIeGSHVk9Sl3U7UzLzp5n5A2Csg6slqUd6kCnfzcz7qrvfAx7TmTXrnrm9bmCWWhgRGybc3xv4cnX7g8AZmfntiDgQ+DrweOAa4DmZuTMiTgD+gsZA/PXAlsw8KiKOAiY7hPVuYDQi1lSHnfw6cEAnVkxS13Q7R5p5NHDbhPu3V9N+3GYdSb3TT5kiafD1a6b8DvDvNZbvCw7ie2NrZq7efWf3b0SquycAT2j8xBSApRGxF7AMOCsiDgESGK0efw7wIYDM3BgRG/d8sszMiHgVcEZEzAfOB/w9vDTYupoj04hJpnnCFWmw9FOmSBp8fZcpEXE8jUH8sTNZvp84iO8/c4BjMnPrxIkR8WHggsw8MSJWAesmPDzth+XM/E/g2VWtFwKHlmpYUt/pSI40cTu/eHTPY4A7atST1F+6nSmShlvXM6Xag/8J4EWZeU+dWv3A38T3n/OBn521MSJWVzeXAT+qbp82Yf6LgFOqeY8AjpqsaETsV/13Po2zM/59wZ4l9ZeO5EgTXwZ+OxqOBh7ITA+ll4ZHtzNF0nDraqZUh+z/K/DqCecIG2gO4vvPm4A11YkbrgJ+v5r+fuB9EfEdYGTC/B8DllSHlbwN+P4Udd8aEVcDG4GvZOa3OtO+pD7QkRyJiDdFxO009rRvjIhPVA99DbgJuAH4B+ANpVdIUk91NVMi4hHV9D8G/ndE3B4RSzuyZpJ6odufU/6cxom+PxoRGyJi/WTLDxKvEy9JkiRJ0oBwT7wkSZIkSQPCQbwkSZIkSQPCQbwkSZIkSQPCQbwkSZIkSQPCQbwkSZIkSQPCQbwkSZIkSQPCQbwkSZIkSQPi/wfRTUYJu1x7SQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x576 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "heads = attention11.detach().numpy()\n",
    "\n",
    "plot_attention_weights(sentence=tokens, translated_tokens=tokens, \n",
    "                      attention_heads=heads)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16bf982c-06c4-4492-8964-a5836c033eeb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}