{
"cells": [
{
"cell_type": "markdown",
"id": "c2231055-6a63-4425-9248-c5aae455396e",
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"<figure>\n",
"<center>\n",
"<img src=\"../Imagenes/logo_final.png\" align=\"left\"/> \n",
"</center> \n",
"</figure>"
]
},
{
"cell_type": "markdown",
"id": "f4ce5373-7375-4e11-872a-d58813d66c24",
"metadata": {},
"source": [
"# <span style=\"color:red\"><center>BERT</center></span>"
]
},
{
"cell_type": "markdown",
"id": "30115409-9d8e-45df-a9cd-072858c02397",
"metadata": {},
"source": [
"<center>Explorando modelos pre-entrenados</center>"
]
},
{
"cell_type": "markdown",
"id": "23484740-ab13-447a-bc3f-01686f528f49",
"metadata": {},
"source": [
"## <span style=\"color:blue\">Autores</span>"
]
},
{
"cell_type": "markdown",
"id": "405456a4-e556-41d2-bf60-593a5055f8b4",
"metadata": {
"tags": []
},
"source": [
"1. Alvaro Mauricio Montenegro Díaz, ammontenegrod@unal.edu.co\n",
"2. Daniel Mauricio Montenegro Reyes, dextronomo@gmail.com "
]
},
{
"cell_type": "markdown",
"id": "c7aaec87-e77d-4152-bb83-bc9e62d4a94c",
"metadata": {},
"source": [
"## <span style=\"color:blue\">Diseño gráfico y Marketing digital</span>\n",
" "
]
},
{
"cell_type": "markdown",
"id": "aac3be8e-e519-43bf-9bd9-9903a27b570b",
"metadata": {},
"source": [
"1. Maria del Pilar Montenegro Reyes, pmontenegro88@gmail.com "
]
},
{
"cell_type": "markdown",
"id": "80e4663a-f0f2-42d8-a044-6e3ab6d16e3e",
"metadata": {},
"source": [
"## <span style=\"color:blue\">Asistentes</span>"
]
},
{
"cell_type": "markdown",
"id": "7b038974-8718-4142-a66e-895520f68ca7",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"id": "0a15511c-982d-4ff7-96fd-a6f1c95f39a1",
"metadata": {},
"source": [
"## <span style=\"color:blue\">Referencias</span> "
]
},
{
"cell_type": "markdown",
"id": "190d6ac6-19de-4dcf-8904-a9e158c84d34",
"metadata": {},
"source": [
"1. [HuggingFace BERT model](https://huggingface.co/transformers/model_doc/bert.html)\n",
"1. [Getting Started with Google BERT: Build and train state-of-the-art natural language processing models using BERT](http://library.lol/main/A0CA3A1276D07957FD7B28F843C299BA)\n",
"1. [Transformers for Natural Language Processing: Build innovative deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, RoBERTa, and more](http://library.lol/main/A8C97E552646B3F194ECA333221CEE88)\n",
"1. [HuggingFace. Transformers ](https://huggingface.co/transformers/)\n",
"1. [HuggingFace. Intro pipeline](https://huggingface.co/course/chapter1/3?fw=pt)\n",
"1. [Tutorial Transformer de Google](https://www.tensorflow.org/text/tutorials/transformer)\n",
"1. [Transformer-chatbot-tutorial-with-tensorflow-2](https://blog.tensorflow.org/2019/05/transformer-chatbot-tutorial-with-tensorflow-2.html) \n",
"1. [Transformer Architecture: The positional encoding](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/)\n",
"1. [Illustrated Auto-attención](https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a)\n",
"1. [Illustrated Attention](https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3#0458)\n",
"1. [Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et. al, 2015)](https://arxiv.org/pdf/1409.0473.pdf)\n",
"1. [Effective Approaches to Attention-based Neural Machine Translation (Luong et. al, 2015)](https://arxiv.org/pdf/1508.04025.pdf)\n",
"1. [Attention Is All You Need (Vaswani et. al, 2017)](https://arxiv.org/pdf/1706.03762.pdf)\n",
"1. [Self-Attention GAN (Zhang et. al, 2018)](https://arxiv.org/pdf/1805.08318.pdf)\n",
"1. [Sequence to Sequence Learning with Neural Networks (Sutskever et. al, 2014)](https://arxiv.org/pdf/1409.3215.pdf)\n",
"1. [TensorFlow’s seq2seq Tutorial with Attention (Tutorial on seq2seq+attention)](https://github.com/tensorflow/nmt)\n",
"1. [Lilian Weng’s Blog on Attention (Great start to attention)](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html#a-family-of-attention-mechanisms)\n",
"1. [Jay Alammar’s Blog on Seq2Seq with Attention (Great illustrations and worked example on seq2seq+attention)](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/)\n",
"1. [Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (Wu et. al, 2016)](https://arxiv.org/pdf/1609.08144.pdf)\n",
"1. [Adam: A method for stochastic optimization](https://arxiv.org/pdf/1412.6980.pdf)"
]
},
{
"cell_type": "markdown",
"id": "17e9be15-8ae8-4715-b57f-a6e8a8111c0c",
"metadata": {},
"source": [
"## <span style=\"color:blue\">Contenido</span>"
]
},
{
"cell_type": "markdown",
"id": "1a382773-c33f-4247-b44c-6e3806b9af20",
"metadata": {},
"source": [
"* [Introducción](#Introducción)\n",
"* [Extracción de incrustamientos de un modelo BERT pre-entrenado](#Extracción-de-incrustamientos-de-un-modelo-BERT-pre-entrenado)"
]
},
{
"cell_type": "markdown",
"id": "c3c46451-72e1-4588-b271-855af1898df1",
"metadata": {},
"source": [
"## <span style=\"color:blue\">Introducción</span>"
]
},
{
"cell_type": "markdown",
"id": "192cd6c6-0240-4bd0-b2e6-427aff064ebb",
"metadata": {},
"source": [
"Usaremos la implementación de HuggingFace in en Pytorch. \n",
"\n",
"+ BERT es un modelo con incrustaciones de posición absoluta, por lo que generalmente se recomienda rellenar (padding) las entradas a la derecha en lugar de a la izquierda.\n",
"\n",
"+ BERT fue entrenado con el modelado de lenguaje enmascarado (MLM) y los objetivos de predicción de la siguiente oración (NSP). Es eficiente para predecir tokens enmascarados y en NLU en general, pero no es óptimo para la generación de texto."
]
},
{
"cell_type": "markdown",
"id": "b86ab415-c6d7-4d89-bbae-098b30f95578",
"metadata": {},
"source": [
"## <span style=\"color:blue\">Extracción de incrustamientos de un modelo BERT pre-entrenado</span>"
]
},
{
"cell_type": "markdown",
"id": "2cd69e7e-8b83-4a1b-ad59-8d5f01381baa",
"metadata": {},
"source": [
"La tarea de PLN es análisis de sentimiento. El primer experimento lo hacemos en inglés y luego en Español. Para esta tarea esta bién usar el modelo uncase (eliminando mayúsculas). "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d93f08d4-ed19-46ef-88cb-f79056f932de",
"metadata": {},
"outputs": [],
"source": [
"from transformers import BertModel, BertTokenizer\n",
"import torch\n"
]
},
{
"cell_type": "raw",
"id": "090081a2-b0e4-4345-aff4-b6cd79b6eb3c",
"metadata": {},
"source": [
"## Tensorflow\n",
"from transformers import TBertModel, TBertTokenizer\n",
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"id": "e483a0f6-52b5-439c-96fc-ba7bc99cc4e4",
"metadata": {},
"source": [
"### Cargamos el modelo pre-entrenado 'bert-base-uncase' y su respectivo tokenizador"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b0d6e7aa-c090-4cff-80ce-f85de557c3cc",
"metadata": {},
"outputs": [],
"source": [
"model = BertModel.from_pretrained('bert-base-uncased')\n",
" \n",
"tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
]
},
{
"cell_type": "markdown",
"id": "f85fb650-c2c7-4287-aa32-fa84acb2c658",
"metadata": {},
"source": [
"### Preprocesamiento de la entrada"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "40e00063-d712-494c-9bc5-39cf1bd332bc",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['i', 'love', 'paris']\n"
]
}
],
"source": [
"# sentencia\n",
"sentence = 'I love París'\n",
"\n",
"# tokenización\n",
"tokens = tokenizer.tokenize(sentence)\n",
"\n",
"# print\n",
"print(tokens)"
]
},
{
"cell_type": "markdown",
"id": "77d5985b-8bfd-4fa0-8255-0dce67066768",
"metadata": {},
"source": [
"### Agregamos los tokens [CLS] al comienzo y [SEP] al final de la lista de tokens"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f48fc4af-645a-4dc2-a73d-07c25f3623a0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['[CLS]', 'i', 'love', 'paris', '[SEP]']\n"
]
}
],
"source": [
"tokens = ['[CLS]'] + tokens + ['[SEP]']\n",
"\n",
"print(tokens)"
]
},
{
"cell_type": "markdown",
"id": "396b2885-baed-4daf-8504-da040e19e927",
"metadata": {},
"source": [
"### Relleno y máscara para la sentencia"
]
},
{
"cell_type": "markdown",
"id": "df8ec922-bb09-4452-b387-9a44e2a6cdc5",
"metadata": {},
"source": [
"El tamaño de la lista de tokens es 5. Supongamos que hemos decidido que el tamaño máximo se las sentencias será 7. BERT está constuido para aceptar sentencias hasta de tamaño 512. Todas las sentencias deben tener el mismo tamaño."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "e144cfa0-e684-4f55-8fc7-d30317821fae",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['[CLS]', 'i', 'love', 'paris', '[SEP]', '[PAD]', '[PAD]']\n"
]
}
],
"source": [
"## Relleno\n",
"max_sentence_size = 7\n",
"pad_size = max_sentence_size - len(tokens)\n",
"\n",
"for i in range(pad_size): \n",
" tokens = tokens + ['[PAD]'] \n",
"\n",
"print(tokens)\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "315fd135-f97a-4453-a658-8972d56d5c75",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1, 1, 1, 1, 1, 0, 0]\n"
]
}
],
"source": [
"## máscara de atención\n",
"attention_mask = [1 if i!= '[PAD]' else 0 for i in tokens]\n",
"print(attention_mask)"
]
},
{
"cell_type": "markdown",
"id": "7ad09897-d398-4f44-8431-a554a2d83972",
"metadata": {},
"source": [
"### Convertimos la lista de tokens en la lista de ID de los tokens"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "6fbea2a3-320d-4dd7-8521-d1af470d3d45",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[101, 1045, 2293, 3000, 102, 0, 0]\n"
]
}
],
"source": [
"token_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
"print(token_ids)"
]
},
{
"cell_type": "markdown",
"id": "f15280fd-00da-45ca-a450-a7a3ad1d69f5",
"metadata": {},
"source": [
"### Convertimos token_ids y attention_mask a tensores"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "6e6e7bd2-4899-4cb9-9d8c-5b9c8d8ed639",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[ 101, 1045, 2293, 3000, 102, 0, 0]])\n"
]
}
],
"source": [
"token_ids = torch.tensor(token_ids).unsqueeze(0) # unsuezze es para agregar una dimensión al comienzo (varias sentencias)\n",
"attention_mask = torch.tensor(attention_mask).unsqueeze(0)\n",
"print(token_ids) # tensor([[ 101, 1045, 2293, 3000, 102, 0, 0]])"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "675844ed-0528-44c2-86eb-b3f6c59742df",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[ 101, 1045, 2293, 3000, 102, 0, 0]])\n",
"tensor([[1, 1, 1, 1, 1, 0, 0]])\n"
]
}
],
"source": [
"print(token_ids)\n",
"print(attention_mask)"
]
},
{
"cell_type": "markdown",
"id": "86126d72-8263-4d14-93b2-37d269bc10fb",
"metadata": {},
"source": [
"### Pregunta"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b5c9ca04-b524-4655-aa73-899edad4d1f2",
"metadata": {},
"outputs": [],
"source": [
"¿Cómo hace esto con tensorflow?"
]
},
{
"cell_type": "raw",
"id": "a9a18715-4f3b-4759-b455-1c6339eb17e8",
"metadata": {},
"source": [
"# Consigne aquí su respuesta\n",
"import tensorflow as tf\n",
"token_ids = tf.expand_dims(tf.constant(token_ids), axis=0)"
]
},
{
"cell_type": "markdown",
"id": "a587dc32-086e-4b90-aaa5-e786350cd427",
"metadata": {},
"source": [
"### Extracción del incrustamiento final"
]
},
{
"cell_type": "markdown",
"id": "eb47b7a9-de74-4348-8792-8e27c6725df3",
"metadata": {},
"source": [
"model regresa una lista con dos objetos: \n",
"\n",
"* El primer valor, *last_hidden_state*, contiene la representación de todos los tokens obtenidos solo de la capa del codificador final (codificador 12).\n",
"* A continuación, *pooler_output* indica la representación del token [CLS] de la capa codificadora final, que se procesa posteriormente mediante una capa lineal y una activación *tanh*. La capa lineal es entrenada cuando se entrena el modelo BERT para la tarea NSP (Next sequence prediction).\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "0613566b-a5a5-410e-86ec-4a8b021315d7",
"metadata": {},
"outputs": [],
"source": [
"out = model(token_ids, attention_mask = attention_mask)\n",
"last_hidden_state, pooler_output = out.last_hidden_state, out.pooler_output # out[0], out[1]\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "6d52ab29-9b0d-416e-b0f0-823ec5e74168",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([1, 7, 768])\n"
]
}
],
"source": [
"print(last_hidden_state.shape)"
]
},
{
"cell_type": "markdown",
"id": "c4b228c0-26b1-4371-8ca1-d9a4371f2c03",
"metadata": {},
"source": [
"* Tamaño batch = 1\n",
"* Tamaño secuencia = 7\n",
"* tamaño del emebdding = 768"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "e6061579-ecac-4724-a27b-c3d74896cc42",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['last_hidden_state', 'pooler_output'])"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# out es un diccionario. Podemos obtener las claves así:\n",
"out.keys()"
]
},
{
"cell_type": "markdown",
"id": "b1287a8b-9a09-4b2f-8fc2-5c6d4ffe5f77",
"metadata": {},
"source": [
"### Extracción de los incrustamientos de todas las capas codificadoras"
]
},
{
"cell_type": "markdown",
"id": "d5e43f16-437b-419c-b6e5-61f1ca3f1333",
"metadata": {},
"source": [
"En esta sección revisamos como extraer las incrustaciones (embeddings) que salen de cada una de las capas codificadoras (12 por ejemplo en el modelo base). Algunos veces estop se hace para extraer diferentes features de las sentencias. \n",
"\n",
"Por ejemplo en la tarea NER (name entity recognition) los investigadores han usado las incrustaciones de las diferentes capas, para hacr promedios pesados de algunas de ellas y con esto han podido mejorar la exactitud en la precisión.\n",
"\n",
"Para hacer esto, es necesario instanciar el modelo preentrenado con la opción *output_hidden_states=True*:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c8b14fb0-e508-4164-b9a5-cbae7e9db8ab",
"metadata": {},
"outputs": [],
"source": [
"model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)\n",
"tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "9cd791bc-ec6a-43f3-8deb-e6fa50c7df6b",
"metadata": {},
"outputs": [],
"source": [
"out = model(token_ids, attention_mask=attention_mask)\n",
"\n",
"last_hidden_state, pooler_output, hidden_states = \\\n",
" out.last_hidden_state, out.pooler_output, out.hidden_states\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "2916d80b-0f59-48fb-b349-b7c986ab065c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([1, 7, 768])\n",
"torch.Size([1, 768])\n",
"13\n"
]
}
],
"source": [
"print(last_hidden_state.shape)\n",
"print(pooler_output.shape)\n",
"print(len(hidden_states)) # esta es una lista conteniendo las\n",
" # incrutaciones de todas las capas codificadoras"
]
},
{
"cell_type": "markdown",
"id": "dc6d576f-40d0-4a9a-a56d-159b0d01c42f",
"metadata": {},
"source": [
"Observe que *hidden_states* tiene 13 elementos. La capa 0 corresponde a la incrustación de la capa de entrada, luego los elementos 1 a 12 corresponden a las incrustaciones de de salida de cada una de las 12 capas codificadoras.\n",
"\n",
"\n",
"La representación de los token de la última capa oculta (codificadora) pueden ser obtenidos así:\n",
"\n",
"* *last_hidden_state[0][0]*: entrega la representación del prime token, es decir, *[CLS]*.\n",
"* *last_hidden_state[0][1]*: entrega la representación del Token *I*.\n",
"* *last_hidden_state[0][2]*: entrega la representación del Token *love*. \n",
"\n",
"Esta es la representación contextual final de los token. \n",
"\n",
"Las incrustaciones de cada capa *i*, se obtienen mediante *hidden_states[i]:"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "dc65d589-ec50-468f-b440-10fdf46a4932",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([1, 7, 768])\n",
"torch.Size([1, 7, 768])\n"
]
}
],
"source": [
"# Incrutaciones de la capa de entrada\n",
"input_embedding = hidden_states[0]\n",
"print(input_embedding.shape)\n",
"\n",
"# incrustaciones de la capa codificadora 11\n",
"embedding_11 = hidden_states[11]\n",
"print(embedding_11.shape)\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "5555e4d2-9e50-4592-9ae7-e7ccc6df3645",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Help on BaseModelOutputWithPoolingAndCrossAttentions in module transformers.modeling_outputs object:\n",
"\n",
"class BaseModelOutputWithPoolingAndCrossAttentions(transformers.file_utils.ModelOutput)\n",
" | BaseModelOutputWithPoolingAndCrossAttentions(last_hidden_state: torch.FloatTensor = None, pooler_output: torch.FloatTensor = None, hidden_states: Union[Tuple[torch.FloatTensor], NoneType] = None, past_key_values: Union[Tuple[Tuple[torch.FloatTensor]], NoneType] = None, attentions: Union[Tuple[torch.FloatTensor], NoneType] = None, cross_attentions: Union[Tuple[torch.FloatTensor], NoneType] = None) -> None\n",
" | \n",
" | Base class for model's outputs that also contains a pooling of the last hidden states.\n",
" | \n",
" | Args:\n",
" | last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):\n",
" | Sequence of hidden-states at the output of the last layer of the model.\n",
" | pooler_output (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, hidden_size)`):\n",
" | Last layer hidden-state of the first token of the sequence (classification token) further processed by a\n",
" | Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence\n",
" | prediction (classification) objective during pretraining.\n",
" | hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):\n",
" | Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)\n",
" | of shape :obj:`(batch_size, sequence_length, hidden_size)`.\n",
" | \n",
" | Hidden-states of the model at the output of each layer plus the initial embedding outputs.\n",
" | attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):\n",
" | Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads,\n",
" | sequence_length, sequence_length)`.\n",
" | \n",
" | Attentions weights after the attention softmax, used to compute the weighted average in the self-attention\n",
" | heads.\n",
" | cross_attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_attentions=True`` and ``config.add_cross_attention=True`` is passed or when ``config.output_attentions=True``):\n",
" | Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads,\n",
" | sequence_length, sequence_length)`.\n",
" | \n",
" | Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the\n",
" | weighted average in the cross-attention heads.\n",
" | past_key_values (:obj:`tuple(tuple(torch.FloatTensor))`, `optional`, returned when ``use_cache=True`` is passed or when ``config.use_cache=True``):\n",
" | Tuple of :obj:`tuple(torch.FloatTensor)` of length :obj:`config.n_layers`, with each tuple having 2 tensors\n",
" | of shape :obj:`(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if\n",
" | ``config.is_encoder_decoder=True`` 2 additional tensors of shape :obj:`(batch_size, num_heads,\n",
" | encoder_sequence_length, embed_size_per_head)`.\n",
" | \n",
" | Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if\n",
" | ``config.is_encoder_decoder=True`` in the cross-attention blocks) that can be used (see\n",
" | :obj:`past_key_values` input) to speed up sequential decoding.\n",
" | \n",
" | Method resolution order:\n",
" | BaseModelOutputWithPoolingAndCrossAttentions\n",
" | transformers.file_utils.ModelOutput\n",
" | collections.OrderedDict\n",
" | builtins.dict\n",
" | builtins.object\n",
" | \n",
" | Methods defined here:\n",
" | \n",
" | __eq__(self, other)\n",
" | \n",
" | __init__(self, last_hidden_state: torch.FloatTensor = None, pooler_output: torch.FloatTensor = None, hidden_states: Union[Tuple[torch.FloatTensor], NoneType] = None, past_key_values: Union[Tuple[Tuple[torch.FloatTensor]], NoneType] = None, attentions: Union[Tuple[torch.FloatTensor], NoneType] = None, cross_attentions: Union[Tuple[torch.FloatTensor], NoneType] = None) -> None\n",
" | \n",
" | __repr__(self)\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Data and other attributes defined here:\n",
" | \n",
" | __annotations__ = {'attentions': typing.Union[typing.Tuple[torch.Float...\n",
" | \n",
" | __dataclass_fields__ = {'attentions': Field(name='attentions',type=typ...\n",
" | \n",
" | __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...\n",
" | \n",
" | __hash__ = None\n",
" | \n",
" | attentions = None\n",
" | \n",
" | cross_attentions = None\n",
" | \n",
" | hidden_states = None\n",
" | \n",
" | last_hidden_state = None\n",
" | \n",
" | past_key_values = None\n",
" | \n",
" | pooler_output = None\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Methods inherited from transformers.file_utils.ModelOutput:\n",
" | \n",
" | __delitem__(self, *args, **kwargs)\n",
" | Delete self[key].\n",
" | \n",
" | __getitem__(self, k)\n",
" | x.__getitem__(y) <==> x[y]\n",
" | \n",
" | __post_init__(self)\n",
" | \n",
" | __setattr__(self, name, value)\n",
" | Implement setattr(self, name, value).\n",
" | \n",
" | __setitem__(self, key, value)\n",
" | Set self[key] to value.\n",
" | \n",
" | pop(self, *args, **kwargs)\n",
" | od.pop(k[,d]) -> v, remove specified key and return the corresponding\n",
" | value. If key is not found, d is returned if given, otherwise KeyError\n",
" | is raised.\n",
" | \n",
" | setdefault(self, *args, **kwargs)\n",
" | Insert key with a value of default if key is not in the dictionary.\n",
" | \n",
" | Return the value for key if key is in the dictionary, else default.\n",
" | \n",
" | to_tuple(self) -> Tuple[Any]\n",
" | Convert self to a tuple containing all the attributes/keys that are not ``None``.\n",
" | \n",
" | update(self, *args, **kwargs)\n",
" | D.update([E, ]**F) -> None. Update D from dict/iterable E and F.\n",
" | If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]\n",
" | If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v\n",
" | In either case, this is followed by: for k in F: D[k] = F[k]\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Methods inherited from collections.OrderedDict:\n",
" | \n",
" | __ge__(self, value, /)\n",
" | Return self>=value.\n",
" | \n",
" | __gt__(self, value, /)\n",
" | Return self>value.\n",
" | \n",
" | __iter__(self, /)\n",
" | Implement iter(self).\n",
" | \n",
" | __le__(self, value, /)\n",
" | Return self<=value.\n",
" | \n",
" | __lt__(self, value, /)\n",
" | Return self<value.\n",
" | \n",
" | __ne__(self, value, /)\n",
" | Return self!=value.\n",
" | \n",
" | __reduce__(...)\n",
" | Return state information for pickling\n",
" | \n",
" | __reversed__(...)\n",
" | od.__reversed__() <==> reversed(od)\n",
" | \n",
" | __sizeof__(...)\n",
" | D.__sizeof__() -> size of D in memory, in bytes\n",
" | \n",
" | clear(...)\n",
" | od.clear() -> None. Remove all items from od.\n",
" | \n",
" | copy(...)\n",
" | od.copy() -> a shallow copy of od\n",
" | \n",
" | items(...)\n",
" | D.items() -> a set-like object providing a view on D's items\n",
" | \n",
" | keys(...)\n",
" | D.keys() -> a set-like object providing a view on D's keys\n",
" | \n",
" | move_to_end(self, /, key, last=True)\n",
" | Move an existing element to the end (or beginning if last is false).\n",
" | \n",
" | Raise KeyError if the element does not exist.\n",
" | \n",
" | popitem(self, /, last=True)\n",
" | Remove and return a (key, value) pair from the dictionary.\n",
" | \n",
" | Pairs are returned in LIFO order if last is true or FIFO order if false.\n",
" | \n",
" | values(...)\n",
" | D.values() -> an object providing a view on D's values\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Class methods inherited from collections.OrderedDict:\n",
" | \n",
" | fromkeys(iterable, value=None) from builtins.type\n",
" | Create a new ordered dictionary with keys from iterable and values set to value.\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Data descriptors inherited from collections.OrderedDict:\n",
" | \n",
" | __dict__\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Methods inherited from builtins.dict:\n",
" | \n",
" | __contains__(self, key, /)\n",
" | True if the dictionary has the specified key, else False.\n",
" | \n",
" | __getattribute__(self, name, /)\n",
" | Return getattr(self, name).\n",
" | \n",
" | __len__(self, /)\n",
" | Return len(self).\n",
" | \n",
" | get(self, key, default=None, /)\n",
" | Return the value for key if key is in the dictionary, else default.\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Static methods inherited from builtins.dict:\n",
" | \n",
" | __new__(*args, **kwargs) from builtins.type\n",
" | Create and return a new object. See help(type) for accurate signature.\n",
"\n"
]
}
],
"source": [
"help(out)"
]
},
{
"cell_type": "markdown",
"id": "372442e1-0221-4d21-95da-9d0058ea7070",
"metadata": {},
"source": [
"### Recuperando los pesos de atención"
]
},
{
"cell_type": "markdown",
"id": "176cad80-144e-486a-a805-8dc09569c275",
"metadata": {},
"source": [
"Los pesos de atención después de la atención softmax, se utilizan para calcular el promedio ponderado en las cabezas de autoatención. \n",
"Son obtenidos pasando al modelo *output_attentions=True*\n",
"\n",
"+ *output attention* es una tupla. Cada elemento coresponde a los pesos de atención de cada capa codificadora."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d18b8954-dae6-4721-a0ec-7a0ca3ce5d9e",
"metadata": {},
"outputs": [],
"source": [
"model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True, output_attentions=True)\n",
"tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "0efd6e94-5101-473f-a9b3-24efa41b12a6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"12\n"
]
}
],
"source": [
"out = model(token_ids, attention_mask=attention_mask)\n",
"\n",
"last_hidden_state, pooler_output, hidden_states, attentions = \\\n",
" out.last_hidden_state, out.pooler_output, out.hidden_states, \\\n",
" out.attentions\n",
"print(len(attentions))"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "be0b076f-cadd-4c72-96f7-12dc08906dd1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([1, 12, 7, 7])\n"
]
}
],
"source": [
"print(attentions[11].shape)"
]
},
{
"cell_type": "markdown",
"id": "a6f09a46-8acc-46b4-aef3-b92b1fe0c2f9",
"metadata": {},
"source": [
"La salida se explica así:\n",
"\n",
"- El tamaño del batch es 1. Una sentencia.\n",
"- Son 12 cabezas de atención.\n",
"- La sentencia viene de tamaño 7.\n",
"\n",
"Por lo tanto tenemos la salida de las 12 cabezas de atención para la sentencia.\n",
"\n",
"Vamos a darle una mirada a los pesos de atención de la última capa codificadora\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "58fabdd1-b720-4734-8acc-f10010d37a62",
"metadata": {},
"outputs": [],
"source": [
"attention11 = attentions[11].squeeze()#elimina la dimensión de batch."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "c414704d-b8f9-4271-93db-721319905899",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([12, 7, 7])"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"attention11.shape"
]
},
{
"cell_type": "markdown",
"id": "cf6013b9-45bc-4d5f-b1cc-aa04837e9450",
"metadata": {},
"source": [
"### Función para graficar pesos de atención de una cabeza"
]
},
{
"cell_type": "code",
"execution_count": 68,
"id": "b99ba957-08ff-4553-a4e2-3d1aca5472a3",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"from mpl_toolkits.axes_grid1 import make_axes_locatable\n",
"\n",
"# versión con decode utf-8\n",
"def plot_attention_head_cp(in_tokens, translated_tokens, attention):\n",
" # The plot is of the attention when a token was generated.\n",
" # The model didn't generate `<START>` in the output. Skip it.\n",
" translated_tokens = translated_tokens[1:]\n",
"\n",
" ax = plt.gca()\n",
" ax.matshow(attention)\n",
" ax.set_xticks(range(len(in_tokens)))\n",
" ax.set_yticks(range(len(translated_tokens)))\n",
"\n",
" labels = [label.decode('utf-8') for label in in_tokens.numpy()]\n",
" ax.set_xticklabels(\n",
" labels, rotation=90)\n",
"\n",
" labels = [label.decode('utf-8') for label in translated_tokens.numpy()]\n",
" ax.set_yticklabels(labels)\n",
"\n",
"\n",
"\n",
"def plot_attention_head(in_tokens, translated_tokens, attention):\n",
" # The plot is of the attention when a token was generated.\n",
" # The model didn't generate `<START>` in the output. Skip it.\n",
" #translated_tokens = translated_tokens[1:]\n",
"\n",
" ax = plt.gca()\n",
" pcm = ax.matshow(attention)\n",
" ax.set_xticks(range(len(in_tokens)))\n",
" ax.set_yticks(range(len(translated_tokens)))\n",
"\n",
" labels = [label for label in in_tokens]\n",
" ax.set_xticklabels(\n",
" labels, rotation=90)\n",
"\n",
" labels = [label for label in translated_tokens]\n",
" ax.set_yticklabels(labels)\n",
" \n",
" "
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "f9c4698e-32b6-42ba-a90e-a68023d49177",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([7, 7])"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"head = attention11[0]\n",
"head.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a25408a7-86c1-4da8-9045-e2235583e2c7",
"metadata": {},
"outputs": [],
"source": [
"head = head.detach().numpy()"
]
},
{
"cell_type": "code",
"execution_count": 64,
"id": "0e583ceb-8a8a-47df-9633-b8857b7de318",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAQoAAAEOCAYAAABxWlnfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUbklEQVR4nO3df5BdZX3H8fcnvwMhQSCibQNxRgQqhCBBVED5VRREkB+CiIN1rAF1tNpRpNVxrJ2OrVDjDwQMOIMOFKgwFMqoUAoR1IoGDEkkRmQEI/4iIgVhkyWbT/+4Z2VZdvfZvXvvubt3P6+Zndz73HPO97m7m88+59znnCPbRESMZFqnOxARE1+CIiKKEhQRUZSgiIiiBEVEFCUoIqIoQRERRQmKiCia0ekOREx2knYZxWLbbT8+WWsrMzMjxkfSFuBXgEZYbLrtPSZr7a4cUUi6aRSLPWb7r9vdl6iHpC+MYrEnbH+8DeU32D5wpAUk/agNdWur3ZUjCkkPAH8z0iLAl2y/vKYuTUmSpgHzbD9RQ62HgU8UFjvf9r5tqD3H9pbxLjORa3fliAL4mO1vj7SApH+sqzNTiaR/B84F+oB7gAWSPmv7gjaXXmH7q4W+vaAdhfv/E0raH9inat5ge/3gZSZr7a4cUQyl+iV53FPlDXeIpDW2l0o6CzgI+Chwj+0lNdXfzfbmOmoNqLkAuBFYBKylMWLdH/gFcFI7R1R11e7Kj0clfULSPtXj2ZLuAB4EfivpmM72ruvNlDQTeDNwo+1ngLaHs6Q3SXoUWCfpl5Je0+6aA/wTsBrYy/bJtt8M7AX8EPjnrqhtu+u+gB/z7GhpOXAHMB3YF/hBp/tX0/dgT+CY6vFcYKea6n4AeAT4Bo2/bnsCd9VQdy2wT/X4EODbNX6v7wdmDNE+g8ZuwKSv3ZUjCqDX1XcLeD1wje0+2xvo3uMyfyLp3cB1wJerpr8A/rOO2ra/YPvPbR/vhoeBI2sovc32T6o+3A3sVEPNfr22tw1urNq2dkPtbv1Ps1XSfsBvafySfnjAazt2pku1eh/wSuBuANsPSHphOwtKervtKyX93TCLfLad9YEXDqr9nOe221l/jqQDef5cBgGz21i3ttrdGhR/S+Mv6kIaR8N/DiDpeODeTnasJltt90qN3x1JM2j/cYL+AK7zL/lAlw2qPfh5O/2a4YPwN91Qe8p86tFP0qm2r+90P9pJ0meAx4GzgfcD7wXut/2xNtedDnzA9op21plMJM1044DupK7drccoRjIVfonPBx4F1gHn0Diw2I4Zic9huw84sd11hiLpPwY8/tdBr91ac18k6ShJlwO/7IbaUzEoRpoT3y1OAr5m+y22T7N9mesbOn5P0kWSDpf0iv6vGuruNeDxXw16bWEN9ZF0iKTPAw8DNwF38ewkqEldu1uPUYxkKuxrnQh8TtKdwDXALUMdGW+T/vkLnxrQZuCoNtcd6efa1p+5pH8GTqcxyelqGu99tQszRSdT7a4MCknrGPqXQ8DuNXendrbfWU16Og54G3CxpP+2PdL5L62qXcdHoUPZoTr6Pw2YO+CTANGYR9JOy4GNwCXAzba3SKrrD1IttbvyYKakPUd6vfpsv+tVYfEG4J3A4bbrGoK/EXg5MKe/zfanhl+jJTVXMcLIoZ0BVh3EPRY4k8bI6Q7gGGBRu0dyddXuyhEFMBPY3fZ3BzZKOpzGuftdTdIbgLfSmEOyCricxvC0jtqXAjtUtS8HTgN+0O66to9od40RavcB3wS+KWkOcAKN78Ejkv7H9tsmfe12Ti/t1BdwM7BkiPZlwH91un81vP9raJxrMbsDtdcO+ncecGsNdQ8GXjTg+dk0Tpb6ArBLm2vPAT4IXERjV2BG1T4feEc31O7WXY/1tvcb5rV1tvevu091k7Q7jf880Di/5Xc11b3b9iGSvg+cAvweWG97r8Kq4617L41zWx6T9FoaYfl+YCmwr+3T2lj7WuAZGp80HAc8ZPuD7arXidrduusxZ4TX2n1gq+MkvQW4kMZuh4AvSvqI7etqKH+zpJ2Bz9C4HgU0dkHabbrtx6rHZwAr3ZhYd72kNW2u/Zf9f3wkfYUadrXqrt2tQfFDSe+2fdnARknv4tlf3m72ceDg/lGEpIXAbTSmtbfbhcB7gMOB/6Xxl+6SGupOlzTDjQN4R9MYhvdr9+/5n2Y/2t7WP3W+JrXU7tZdj92BG4Beng2GZcAs4GTb7Z5/31GDd6+qS9LdV8cuVzVD8kngyqrpTGBn2209mCrpY8DxwGZgD+AVti3ppcBXbR/axtp9wFP9T2mMWp+uHtv2/MleuyuDop+kI4H+YxU/tn17J/tTF0kXAEtoTMCBxlB8re2P1lD7PtsHlNraVPtVwItpHDx9qmp7GY3rdk6FkwHbpiuDQtK9tkecNjyaZSYzSacCh9L4y3Kn7RtqqnsFcKnt71fPD6Fx9P29ba7bsZ/5VKjdrUHRAzww0iLAArfhPgtTnaQNwN40phRDYzdgA7CdxlC4LdfO7OTPfCrU7taDmaM5GaavHYUlfcf2YZKe5LkzBevYXx1cs7baA7yhhhpD6djPfCrU7soRRUS01lQ8zTwixihBERFFUyYoJC0vL5XaqZ3aQ5kyQcFzZ+qldmqn9hhMpaCIiCZNmk89ps/f0TMX7tz0+n1PPMX0+c3d0mPWz8d3j9dnvIWZGuk8tfYZT23NmD6u2r3be5g1rflz8Lbt1PxtKbZtfYoZs5u/hcv0x54qLzSMZ9jKzLbfzqM9tZ/kD5s9xAWOJs08ipkLd2bRv5zbkdovOXtjR+oCsL1zQT59t106Vhvg90cv7ljtBVd+v2O1O+k2Xzfk1d+y6xERRQmKiChKUEREUYIiIooSFBFRlKCIiKIERUQUJSgioihBERFFCYqIKEpQRETRqIJC0mJJPf13XJL0IknXSHpQ0v2SviHpZdVy64dY/1WS7pa0RtIGSZ+s2s+Q9DNJN7fyTUVEa43lpLAHbS9V41ZEN9C4qcpbASQtBXYHNg2z7leB023fV92mfW8A29dK+i3w4WbfQES0XzNnjx4JPGP70v4G22ugMfIYZp0XAr+ulu0D7h9NoepqPcsBZuy2oImuRkQrNHOMYj/Gfv/OFcBGSTdIOkca3QUSbK+0vcz2smavJRER41fLwUzbn6Jx789bgbcB36qjbkS0RjNB8WPgoLGuZPtB25fQuNP0AZJ2baJ2RHRAM0FxOzBb0rv7GyQdLOl1w60g6Y169n7se9G4c9HjTdSOiA4Y88HM6lbyJwOfk3Q+sAV4CPhgtcjekn45YJUPAacCKyQ9DWwDzqoOakbEJNDUNTNt/wo4fZiXZw7R9vVm6kTExDDaXY8+YEH/hKtWkXQGcDHwh1ZuNyJaa1QjCtubgEWtLm77WuDaVm83Ilor53pERFGCIiKKEhQRUZSgiIiiBEVEFCUoIqJo0tzNfL528SHTjulM8UnyPWq5P82674xbHvlRx2q//s+Wdqx2J93m6+6xvWxwe0YUEVGUoIiIogRFRBQlKCKiKEEREUUJiogoSlBERFGCIiKKEhQRUZSgiIiiBEVEFCUoIqJoQgSFpO91ug8RMbwJERS2X9PpPkTE8CZEUEj6Y6f7EBHDa+oGQHWRtBxYDjCHHTrcm4ipa0KMKIZje6XtZbaXzWR2p7sTMWVN6KCIiIkhQRERRQmKiCiaEEFhe16n+xARw5sQQRERE1uCIiKKEhQRUZSgiIiiBEVEFCUoIqIoQRERRQmKiChKUERE0YQ+zTwAqYO1O/t3pM/bO1o/npURRUQUJSgioihBERFFCYqIKEpQRERRgiIiihIUEVGUoIiIogRFRBQlKCKiKEEREUUJiogoGldQ5ObCEVNDRhQRUdSSoFDDBZLWS1on6Yyq/VpJxw9Y7gpJp0qaXi3/Q0lrJZ3Tin5ERHu0akRxCrAUOAA4BrhA0ouBa4D+0JgFHA18A3gX8H+2DwYOBt4t6SUt6ktEtFirguIw4GrbfbZ/C3ybRgB8EzhK0mzgOOBO2z3AscDZktYAdwO7AnsN3qik5ZJWS1r9DFtb1NWIGKtWXeFqyMsw2d4iaRXwehoji6sHLP9+27eMtFHbK4GVAPO1i1vU14gYo1aNKO4EzqiOPSwEXgv8oHrtGuCdwOFAfzDcArxH0kwASS+TtGOL+hIRLdaqEcUNwKuB+wAD59n+TfXarcDXgJts91ZtlwOLgXslCXgUeHOL+hIRLTauoLA9r/rXwEeqr8HLPEPjGMTAtu3AP1RfETHBZR5FRBQlKCKiKEEREUUJiogoSlBERFGCIiKKEhQRUZSgiIiiBEVEFLVqCnfbado0ps2d25Ha7u0tL9SFpu20U0frfz8nDE8YGVFERFGCIiKKEhQRUZSgiIiiBEVEFCUoIqIoQRERRQmKiChKUEREUYIiIooSFBFRlKCIiKJag0LSuZLOrrNmRIxfbWePSpph+9K66kVE64wpKCQtBr5F48bCBwI/Bc4GPgy8CZgLfA84x7ar+45+DzgUuEnSTsAfbV8o6QPAucA24H7bb23JO4qIlmtm12NvYKXtJcATwHuBi2wfbHs/GmFxwoDld7b9Otv/Nmg75wMHVts5t4l+RERNmgmKTba/Wz2+EjgMOFLS3ZLWAUcBLx+w/LXDbGctcJWkt9MYVTyPpOWSVkta3estTXQ1IlqhmaDwEM8vBk6zvT9wGTBnwOtPDbOdNwJfAg4C7pH0vN0g2yttL7O9bJbmPG8DEVGPZoJiD0mvrh6fCXynerxZ0jzgtNIGJE0DFtm+AzgP2BmY10RfIqIGzXzqsQF4h6QvAw8AlwAvANYBDwE/HMU2pgNXSloACFhh+/Em+hIRNWgmKLbbHnzw8ePV13PYPmLQ808OeHpYE7UjogMyMzMiisY0orD9ELBfe7oSERNVRhQRUZSgiIiiBEVEFCUoIqIoQRERRQmKiChKUEREUYIiIopqu8LVeHn7drb39HSo+OATZqeG7U8+2dH6r5w9Nb/vE1FGFBFRlKCIiKIERUQUJSgioihBERFFCYqIKEpQRERRgiIiihIUEVGUoIiIogRFRBQlKCKiaNRBIWmxpB5Ja6rnH5P0Y0lrJa2RdEjVvkrSxqptjaTrqvZPSnqkalsv6cSq/UOSfiHpoja8v4hogbGePfqg7aXVLQVPAF5he6uk3YBZA5Y7y/bqIdZfYftCSfsCd0l6oe0Vkv4ALGvuLUREuzV7mvmLgc22twLY3jyWlW1vkLQN2A34XZN9iIiaNHuM4lZgkaSfSrpY0usGvX7VgF2PCwavXO2mbAceHamIpOWSVkta/Qxbm+xqRIxXUyMK23+UdBBwOHAkcK2k821fUS0y3K7HhyS9HXgSOMMe+YowtlcCKwHma5dcxSSiQ5q+wpXtPmAVsErSOuAdwBWF1VbYvrDZmhHRGU3tekjaW9JeA5qWAg+3pEcRMeE0O6KYB3xR0s7ANuBnwPIBr18lqf8Cl5ttH9N8FyOi05o9RnEP8JphXjtimPZPNlMrIjpvLLsefcCC/glXrSLpQ8DfA0+0crsR0TqjHlHY3gQsanUHbK8AVrR6uxHROjnXIyKKEhQRUZSgiIiiBEVEFCUoIqIoQRERRQmKiChq+qSwumnaNKbNnduR2tt7esoLdaHpL9q9o/Xv6JnT0frxrIwoIqIoQRERRQmKiChKUEREUYIiIooSFBFRlKCIiKIERUQUJSgioihBERFFCYqIKEpQRERRMSgkLZbU03/1bUl91T1F10v6uqQdqvYZkjZL+vSg9VdJ2ihpraSfSLqouh8IkuZW2+qt7ogeERPQaEcUD9peWj3usb3U9n5AL3Bu1X4ssBE4XZIGrX+W7SXAEmArcCOA7Z5qu79q/i1ERLuNd9fjLuCl1eMzgc8DvwBeNdTCtnuB84A9JB0wztoRUZOmg0LSDOA4YJ2kucDRwM3A1TRCY0jVzY3vA/YZRY3lklZLWt3rLc12NSLGqZmgmFsdr1hNY/TwFeAE4A7bTwPXAydLmj7CNgbvmgzJ9krby2wvm6VcxCSiU5q5wlXPgOMVAEg6EzhU0kNV067AkcBtg1euAmR/YEMTtSOiA8b98aik+cBhwB62F9teDLyPIXY/JM0EPg1ssr12vLUjoh6tmEdxCnC77a0D2m4ETpQ0u3p+laS1wHpgR+CkFtSNiJqMedfD9rxBz68ArhjU9hiwsHp6RHNdi4iJYjQjij5gQf+Eq1bqn3AFzAS2t3r7EdEaxRGF7U3AonYUt90DLG3HtiOidXKuR0QUJSgioihBERFFCYqIKEpQRERRgiIiihIUEVHUzElhnfO86+HUVXeK5umsmR0t/1jfvPJCUYsp+j8gIsYiQRERRQmKiChKUEREUYIiIooSFBFRlKCIiKIERUQUJSgioihBERFFCYqIKEpQRERRMSgkLZbU038Vbkl9ktZIWi/p65J2qNpnSNos6dOD1l8laaOktZJ+IukiSTtXr82tttUrabfWv72IaIXRjigeHHAbwR7bS23vB/QC51btxwIbgdOl553meZbtJcASYCuNGwRhu//2hL9q/i1ERLuNd9fjLuCl1eMzgc/TuHHxq4Za2HYvcB6wh6QDxlk7ImrSdFBImgEcB6yTNBc4GrgZuJoh7jvaz3YfcB+wzyhqLJe0WtLqXm9ptqsRMU7NBEX/3b1W0xg9fAU4AbjD9tPA9cDJ1V3LhzOqK9DYXml7me1lszSnia5GRCs0c4WrngHHKwCQdCZwqKSHqqZdgSOB2wavXAXI/sCGJmpHRAeM++NRSfOBw4A9bC+2vRh4H0PsfkiaCXwa2GR77XhrR0Q9WjGP4hTgdttbB7TdCJwoaXb1/CpJa4H1wI7ASS2oGxE1GfOuh+15g55fAVwxqO0xYGH19IjmuhYRE8VoRhR9wIL+CVet1D/hCpgJbG/19iOiNYojCtubgEXtKG67B1jajm1HROvkXI+IKEpQRERRgiIiihIUEVGUoIiIogRFRBQlKCKiSLY73YdRkfQo8PA4NrEbsLlF3Unt1O7W2nvaXji4cdIExXhJWm17WWqndmqPXXY9IqIoQRERRVMpKFamdmqndnOmzDGKiGjeVBpRRESTEhQRUZSgiIiiBEVEFCUoIqLo/wHN7xO0Gz12GwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plot_attention_head(in_tokens=tokens, translated_tokens=tokens, attention=head)"
]
},
{
"cell_type": "markdown",
"id": "6b603500-d79b-4b20-99e3-0996c8454d41",
"metadata": {},
"source": [
"### Visualizando los pesos de todas las cabezas de atención"
]
},
{
"cell_type": "code",
"execution_count": 65,
"id": "864f57d2-01d6-41b5-872c-402b44a9e7e0",
"metadata": {},
"outputs": [],
"source": [
"def plot_attention_weights(sentence, translated_tokens, attention_heads):\n",
" in_tokens = sentence\n",
" #in_tokens = tokenizers.pt.tokenize(in_tokens).to_tensor()\n",
" #in_tokens = tokenizers.pt.lookup(in_tokens)[0]\n",
" #in_tokens\n",
"\n",
" fig = plt.figure(figsize=(16, 8))\n",
"\n",
" for h, head in enumerate(attention_heads):\n",
" ax = fig.add_subplot(3, 4, h+1)\n",
"\n",
" plot_attention_head(in_tokens, translated_tokens, head)\n",
"\n",
" ax.set_xlabel(f'Head {h+1}')\n",
"\n",
" plt.tight_layout()\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 66,
"id": "51b75032-bdf9-4b9f-a069-373bc7afc6fb",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA/EAAAI4CAYAAAAiUWldAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABnmUlEQVR4nO39e7gkdXnv/b/vdZojMyNHT+AkEcGIMOrgEVSUaDRGRY2IZHu4VESNbs3jaUcvH+N+st1b3SEeogb1F8yjW0hkuzH8jBojE8/ooMOAjqgYTiIIIgJzWjNr3c8fXSNrljNr1er+ru6qnvfruvqa7urqu+5aq9Zn+ttVXRWZiSRJkiRJar6RQTcgSZIkSZLqcRAvSZIkSVJLOIiXJEmSJKklHMRLkiRJktQSDuIlSZIkSWoJB/GSJEmSJLWEg3hJkiRJklrCQbwkSZIkSS0xNugG1B4RcXCN2aYz8/Z+1JHUbmaKpJLMFEklNTlTIjPrzqsDXETsAG4EYo7ZRjPzqH7UkdRuZoqkkswUSSU1OVNavSc+Ij5bY7bbMvNFi91Lk0XE+2rMdkdmvnWeebZk5kPmWdb3aiyrVB2pKDOlHjNFqsdMqcdMkeoxU+o5EDKl1XviI+LHwEvnmgX428x8UJ9aWjQRMQKszMw7unjttcDb5pntzZn5wHnqLM3MHU2ZRyrNTKn9WjNFqsFMqf1aM0WqwUyp/dqhz5RW74kH3pKZ/z7XDBHxl/1qprSI+F/A2cAUcBmwOiL+OjPfvcBS52Tmx+dZ1j3mK7Jnw4qIBwPHVpO3ZOaVs+fpRx1pEZgp9ZgpUj1mSj1milSPmVLP0GdKq/fE70v1C7k9h2DFImJTZq6LiDOBhwFvAi7LzOO7rHdoZt7aQz+rgYuAI4HNdD7tezBwHfCMup+Ulaoj9YOZMmc9M0VaIDNlznpmirRAZsqc9YY2U1p9ibmIeFtEHFvdXxIRlwBXAzdHxKmD7a6I8YgYB54JXJSZu4AF/4FGxB9HxC3AFRFxQ0Q8ust+/iuwETg6M0/LzGcCRwPfAf5qAHWkosyUeswUqR4zpR4zRarHTKnngMiUzGztDfg+dx9NcBZwCTAKPBD49oB7ux9wanV/GXBQFzVeA/wM+BydT2zuB3y1izqbgWOr+48A/r3LdfoBMLaP6WN0Dgnpax1v3krfzJTadcwUb95q3MyU2nXMFG/eatzMlNp1hj5TWr0nHpjM6icAPBk4PzOnMnMLA/y+f0S8DPg08HfVpPsC/2ehdTLzfZl5n8x8anZcC5zSRUu7M/OHVc1LgYO6qAGdn/fuffS5G9g5gDpSaWZKPWaKVI+ZUo+ZItVjptQz9JnS9hPb7YyI44Cb6fyCXz/juRWDaQmAVwEPBy4FyMwfR8ThdV8cEX+amZ+IiD/fzyx/vcB+Dp9Va6/HmVm33tKIeAi/fY3DAJYsoJ9SdaTSzJR6zBSpHjOlHjNFqsdMqWfoM6Xtg/j/TOdTn8PonIXwPwAi4qnAdwfY187MnIzo/J4iYoyFfZ9jzx9ht58azfaRWbVmP67r5+z/j+imAdSRSjNT6jFTpHrMlHrMFKkeM6Weoc+UoTs7/R4R8ezMvHBAy34XcDvwAuDVwCuBH2TmWxZQYxR4TWaesyhNFhYR49k5+UQj6kilmSn9ZaZo2Jkp/WWmaNiZKf016Exp+3fi5zLIDeDNwC3AFcDL6Zyc4a0LKZCZU8DTSzQTEf844/7/mPXcF3uoGxHxhIj4KHDDoOtIi8xMqZgpUhFmSsVMkYowUyoHQqYM8yB+9ncO+ukZwD9k5p9k5nMy8yPZ3SEP34iID0TEyRHx0D23LuocPeP+H8x67rCFFouIR0TEe4Frgc8CXwWOHVQdqU/MlLuZKVLvzJS7mSlS78yUuw19prT9O/FzGeT3BJ4O/E1EfAU4H/jCvs5IWMOeaxq+Y8a0BJ6wwDpz/Sxq/5wi4q+A5wLXAZ+q+tqYmR9fSDOl6kh9Zqbs/ZpuntuLmaIDnJmy92u6eW4vZooOcGbK3q/p5rm9NDlTWj2Ij4gr2PcvIoAj+tzOb2TmiyNiHHgK8HzggxHxr5n50gXW6eaSCvuyPDpnRBwBlsXdZ0cMOtdxrOss4CrgQ8DFmbkjIroJjFJ1pKLMlNrMFKkGM6U2M0WqwUypbegzpdUntouI+831fHVtwYGpNuY/BF4MnJyZ3Ry+8UfAg4Cle6Zl5jv2/4p91tjAHJ861f2Dic4JJ54EnEHnE7FLgFOBIxfyaVupOlJpZkrtGhswU6R5mSm1a2zATJHmZabUrrGBIc+Utg/i7w8ckZlfnzX9ZODGzLx6QH39IfA8Otdv3ABcAHxxob+kiPgwsLyq81HgOcC3M/MlRRvuQkQsBZ5GZ2M8Cfi3zHz+oOpIJZgpg2OmaBiZKYNjpmgYmSmD07hMyczW3oCLgeP3MX098M8D7Ot84JnAkh7rbJ7170o6fxALrXMicM8Zj18AXAS8Dzh4AXWWAq8FPkDnsJCxavoq4IX9ruPNW+mbmVK7jpnizVuNm5lSu46Z4s1bjZuZUrvO0GdK2/fEX5mZx+3nuSsy88H97mnG8o+gswFB5xOkX3RR49LMfEREfAt4FvBL4MrMPHqel86u813g1My8LSIeS+cP7dXAOuCBmfmcmnUuAHbROZPiU4BrMvO1C+mlZB2pNDOldh0zRarBTKldx0yRajBTatcZ+kxp9YntmPFdiX1YyEkLioqIPwHeQ+dwkgDeHxFvyMxPL7DUxRGxBngXcFk17aNdtDSambdV908Hzs3MC4ELI2LTAur8/p5wiIiPAd/uopeSdaTSzJR6zBSpHjOlHjNFqsdMqWfoM6Xtg/jvRMTLMvMjMydGxEu4+xc/CG8FTtzzCVREHAZ8CVjohvwe4BXAycA36Xx686Eu+hmNiLHsfC/liXQO49hjIdvArj13MnN3RNeXoyxVRyrNTKnHTJHqMVPqMVOkesyUeoY+U9p+OP0RwGeASe7ecNcDE8BpmXnTgPra63CWiBgBLl/oIS4R8Y/AncAnqklnAGsy87kLrPMW4KnArcBRwEMzM6uTY3w8Mx9Ts84UsHXPQzqf+G2r7mdmrupnHak0M6V2HTNFqsFMqV3HTJFqMFNq1xn6TGn1IH6PiDgF2PP9kO9n5pcH3M+7geOBT1WTTqdzgoY3LbDO5Zl5wnzTatZ6JHAvOieH2FpNewCwMjO/u9B60jAzU2rVMlOkmsyUWrXMFKkmM6VWraHOlFYP4iPiu5n50F7nWQwR8WzgMXQ+YflKZn6mixrnAR/OzG9Vjx9B5wyGr1xgnSI/p6bNI5XW5G3TTFn8eaTSmrxtmimLP49UWpO3TTNl8efZa/6WD+K3Az+eaxZgdWYe1aeWioqILcAxwHXVpKOALcA0nUMvjq9Zp8jPqWl1pNKGfds0U9r5e1N7Dfu2aaa08/em9hr2bdNMqf97a/uJ7Y6tMc/UfDNExNcy86SIuBOY+anGQr/vMPv1XdWZ4Q8XOP/+FPk5NbCOVJqZUk/TssBMUVOZKfU0LQvMFDWVmVJP07KgeKa0ek+8JEmSJEkHkpFBNyBJkiRJkuoZqkF8RJw1/1z9rdW0OiVrNbEnqaQmbuNNq1OyVhN7kkpq4jbetDolazWxJ6mkJm7jw9xTk9ZtqAbxQMmALVWraXVK1mpiT1JJTdzGm1anZK0m9iSV1MRtvGl1StZqYk9SSU3cxoe5p8as27AN4iVJkiRJGlqtObHd6KoVOX7YmjnnmbpjK6OrVsw5z8R/7Ki1vF25g/FYWre9nurE2Oi8dSantzMxsmze+XYftGT+eXZuZWzJ3D+n0du2zltnFzsZZ/7l1VGn1p386tbMPKzIAnXAG1u6IicOOnjOeXbv2MrY0rn/VsbvmKy1vMmp7UyMzv03nDWyYNfubYyPLZ+7zvj8n8/umtzK+MTc69YpNv8sdWrFHdvmr2OmqMUmxlfk0qVr5pxnctdWJsbn+buLqLW8ycmtTMzzd7dr+fy1prZtZXT53HXGts8fBHUzZWTbznnnmZzewcTI3O+dcvf8J3I2U9RmEyNLc9noQXPOU+dvhdH531tAvfcpuw4an7dOnfdOUzX/LKe3bmVkxdy1lt5cJ1PmH0fl7t3z1mlSprTmEnPjh63hyP9+ds91fucFVxXopjJd5gOQ0UPnHkgsxC+fuLZIndWf+FaROiV9KT997aB70PCYOOhgjnn263quc89/ub5ANx1Th68uUmf7vWsMzmuK3WVybsm/fKdInZLMFJW0dOkaTlz/qp7r5Ei9QXwdv3ho7zsjAA69ot6HlXUsveynRepM/fK2InVKMlNU0rLRg3jUmmf1XujgNb3XqPzicUcUqfPro4uUAeDov766SJ2pm39RpE5Jc2WKh9NLkiRJktQSDuIlSZIkSWoJB/GSJEmSJLWEg3hJkiRJklrCQbwkSZIkSS1RaxAfEWsjYntEbKoe3zMizo+IqyPiBxHxuYh4QDXflft4/SMj4tKI2BQRWyLi7dX00yPiJxFxccmVktRsZoqk0swVSSWZKWqyhVxi7urMXBcRAXwG+HhmPg8gItYBRwD7u9bSx4HnZublETEKHAOQmRdExM3A67tdAUmtZaZIKs1ckVSSmaJG6uY68acAuzLzw3smZOYm6HxitZ/XHA78vJp3CvhBnQVFxFnAWQBjh5a5frKkxhlIpoyvvEfXDUtqvL7kysxMWbLE9ynSEOt7piwdWdlTwxpu3Xwn/jjgsgW+5hzgqoj4TES8PCKW1nlRZp6bmeszc/3oqhULblRSKwwkU8aWminSEOtLrszMlIlxM0UaYv3PlJFab210gOrLie0y8x3AeuCLwPOBz/djuZKGk5kiqTRzRVJJZooWUzeD+O8DD1voizLz6sz8EPBE4ISIOKSLZUsaPmaKpNLMFUklmSlqlG4G8V8GlkTEy/ZMiIgTI+Jx+3tBRPxRdUIIgKOBKeD2LpYtafiYKZJKM1cklWSmqFEWfGK7zMyIOA34m4h4M7ADuAZ4bTXLMRFxw4yXvA54NnBORGwDdgNnVid4kHSAM1MklWauSCrJTFHTdHN2ejLzRuC5+3l6fB/T/qmb5Ug6MJgpkkozVySVZKaoSeoeTj8FrI6ITSUXHhGnAx8EflWyrqTGM1MklWauSCrJTFFj1doTn5nXA0eWXnhmXgBcULqupGYzUySVZq5IKslMUZN1dTj9IEz8dDtrn3dFz3Uys0A3Ze2+6eZitb71rjJXr3jyJ9YVqSM11ditWzn0o9/uuc7uAr38xs9+XqTMip+uKlIH4MOX/3OROi876qQidaSmiq07GLt0S+91Vpf7+z3yJxNlCo2WuyLx/T6/rUidq08sUkZqrNw9xdRtBXbW//K23mtUDrn62jJ1ilTp+NwNlxWp8+R7rytSp1/6cp14SZIkSZLUOwfxkiRJkiS1hIN4SZIkSZJawkG8JEmSJEkt4SBekiRJkqSWcBAvSZIkSVJLNGIQHxHfGHQPkoaHmSKpJDNFUklminrViEF8Zj560D1IGh5miqSSzBRJJZkp6lUjBvERcdege5A0PMwUSSWZKZJKMlPUq7FBNzCXiDgLOAtgKcsH3I2ktjNTJJW0V6bEigF3I6ntfJ+iuhqxJ35/MvPczFyfmevHWTLodiS1nJkiqaSZmTJhpkjqke9TVFejB/GSJEmSJOluDuIlSZIkSWoJB/GSJEmSJLVEIwbxmbly0D1IGh5miqSSzBRJJZkp6lUjBvGSJEmSJGl+DuIlSZIkSWoJB/GSJEmSJLXE2KAbaLWIQnXKfZYyldPFaklqp5ycLFbroIL5JA2zBDKz5zoxuav3ZipTRx5epM70xGiROgBf+9mKInXuxZYidSTVFyOFxj6j5TJl23S59zxt4rszSZIkSZJawkG8JEmSJEkt4SBekiRJkqSWcBAvSZIkSVJLOIiXJEmSJKklehrER8RdpRqRJDNFUklmiqSSzBQ1hXviJUmSJElqiSKD+Oh4d0RcGRFXRMTp1fQLIuKpM+Y7LyKeHRGj1fzfiYjNEfHyEn1IGg5miqSSzBRJJZkpGrRSe+KfBawDTgBOBd4dEfcCzgf2bNQTwBOBzwEvAX6dmScCJwIvi4jfmV00Is6KiI0RsXEXOwu1KqkFzBRJJS1+puSOvqyIpEbwfYoGqtQg/iTgU5k5lZk3A/9OZwP9F+AJEbEEeArwlczcDjwJeEFEbAIuBQ4Bjp5dNDPPzcz1mbl+nCWFWpXUAmaKpJIWP1NiaZ9WRVID+D5FAzVWqE7sa2Jm7oiIDcCT6Xwq9akZ8786M79QaPmShouZIqkkM0VSSWaKBqrUnvivAKdX3/c4DHgs8O3qufOBFwMnA3s23C8Ar4iIcYCIeEBErCjUi6T2M1MklWSmSCrJTNFAldoT/xngUcDlQAJvzMybque+CPwD8NnMnKymfRRYC3w3IgK4BXhmoV4ktZ+ZIqkkM0VSSWaKBqqnQXxmrqz+TeAN1W32PLvofO9j5rRp4C+qmyQBZoqksswUSSWZKWoKrxMvSZIkSVJLOIiXJEmSJKklHMRLkiRJktQSpU5st+hiZISRZct6rpOTk/PP1GcjBx1UrNa3dhYrJQ21GB1ldNXK3gsdfmjvNSqxe6pMocldZeoAz/3R6UXqjHB9kTpSU8XICCPLl/dcZ3rbtgLddMSW/yhSZ+xehxepA/Cyo79bpM7F3KNIHampYnyMscOO6LlO7ig3OMij7lmkzq+OW12kDsAJn3hokTq/yzeL1OkX98RLkiRJktQSDuIlSZIkSWoJB/GSJEmSJLWEg3hJkiRJklrCQbwkSZIkSS3R10F8RJwdES/o5zIlDS8zRVJJZoqkkswULZa+XWIuIsYy88P9Wp6k4WamSCrJTJFUkpmixbSgQXxErAU+D1wKPAT4EfAC4PXAHwPLgG8AL8/MjIgN1ePHAJ+NiIOAuzLzPRHxGuBsYDfwg8x8XpE1ktQaZoqkkswUSSWZKWqqbg6nPwY4NzOPB+4AXgl8IDNPzMzj6GzMT5sx/5rMfFxm/s9Zdd4MPKSqc3YXfUgaDmaKpJLMFEklmSlqnG4G8ddn5ter+58ATgJOiYhLI+IK4AnAg2bMf8F+6mwGPhkRf0rnE6nfEhFnRcTGiNg4mTu6aFVSCwwoU7YXal9Sw/g+RVJJg8mUad+naP+6GcTnPh5/EHhOZj4Y+AiwdMbzW/dT54+AvwUeBlwWEb91aH9mnpuZ6zNz/UQs/a0CkobCgDJlWe+dS2oi36dIKmkwmTLi+xTtXzeD+KMi4lHV/TOAr1X3b42IlcBz5isQESPAkZl5CfBGYA2wsoteJLWfmSKpJDNFUklmihqnm7PTbwFeGBF/B/wY+BBwD+AK4BrgOzVqjAKfiIjVQADnZObtXfQiqf3MFEklmSmSSjJT1DjdDOKnM3P2yRjeWt32kpmPn/X47TMentTFsiUNHzNFUklmiqSSzBQ1TjeH00uSJEmSpAFY0J74zLwGOG5xWpF0oDFTJJVkpkgqyUxRU7knXpIkSZKklnAQL0mSJElSS3RzYruByOlpprdvL1Bo9qUeB2/6zjuL1Xr4kuatn9REOTXF1B139V7o13f0XqOwGB0tVutfH7ixSJ0ns65IHampcnqqyP/nOTVVoJuq1uRkkTojN/y8SB2Al6z+cZE6F/PwInWkpspdu9l98y29F5oulyncfnuRMquvKLcf+VvvvqxInSe/eV2ROv3innhJkiRJklrCQbwkSZIkSS3hIF6SJEmSpJZwEC9JkiRJUks4iJckSZIkqSUcxEuSJEmS1BK1B/ERsTYitkfEpurxWyLi+xGxOSI2RcQjqukbIuKqatqmiPh0Nf3tEfGzatqVEfH0avrrIuK6iPjAIqyfpIYyUySVZKZIKslMUZMt9DrxV2fmuoh4FPA04KGZuTMiDgUmZsx3Zmbu6+LC52TmeyLigcBXI+LwzDwnIn4FrO9uFSS1mJkiqSQzRVJJZooaaaGD+D3uBdyamTsBMvPWhbw4M7dExG7gUOAX+5svIs4CzgJYyvIuW5XUAmaKpJLMFEklmSlqlG6/E/9F4MiI+FFEfDAiHjfr+U/OOKTk3bNfXB1+Mg3cMtdCMvPczFyfmevHWdJlq5JawEyRVFL/MyXMFGmI+T5FjdLVnvjMvCsiHgacDJwCXBARb87M86pZ9ndIyesi4k+BO4HTMzO7Wb6k4WKmSCrJTJFUkpmipun2cHoycwrYAGyIiCuAFwLnzfOyczLzPd0uU9LwMlMklWSmSCrJTFGTdHU4fUQcExFHz5i0Dri2SEeSDjhmiqSSzBRJJZkpappu98SvBN4fEWuA3cBPqE7CUPlkRGyv7t+amad236KkA4CZIqkkM0VSSWaKGqXb78RfBjx6P889fj/T397NsiQNPzNFUklmiqSSzBQ1zUIOp58CVkfEppINRMTrgP8C3FGyrqTGM1MklWSmSCrJTFFj1d4Tn5nXA0eWbiAzzwHOKV1XUrOZKZJKMlMklWSmqMm6Pjt9v8XICCPLlvVcZ3r79vln6rPRex5RrNYl25cWqyUNs5gYZ+ze9y5QKHqvUck77ypSZ/exRxWpA/D73zi2SJ0jubJIHamxEnJqqkCdglegKpRPOTVdpA7A8pGJYrWkYdYZ+/T+vn56+44C3XSMHXWfInVybLRIHYA33Vwgd1uoq7PTS5IkSZKk/nMQL0mSJElSSziIlyRJkiSpJRzES5IkSZLUEg7iJUmSJElqCQfxkiRJkiS1xLyD+IhYGxHbI2JT9XgqIjZFxJUR8U8RsbyaPhYRt0bEO2e9fkNEXBURmyPihxHxgYhYUz23rKo1GRGHll89SU1jpkgqyUyRVJKZojaouyf+6sxcV93fnpnrMvM4YBI4u5r+JOAq4LkRv3Vh0jMz83jgeGAncBFAZm6v6t7Y/SpIaiEzRVJJZoqkkswUNVqvh9N/Fbh/df8M4L3AdcAj9zVzZk4CbwSOiogT5iseEWdFxMaI2DiZO3psVVIL9C9TprYXallSg/UtU3axs1DLkhrMsY8aoetBfESMAU8BroiIZcATgYuBT9HZqPcpM6eAy4Fj51tGZp6bmeszc/1ELO22VUkt0PdMGV1WpnFJjdTvTBlnSZnGJTWSYx81STeD+GXVd0Q20vnk6WPA04BLMnMbcCFwWkSMzlFj9iEnkg5cZoqkkswUSSWZKWqcsS5es+e7HL8REWcAj4mIa6pJhwCnAF+a/eJqA38wsKWLZUsaPmaKpJLMFEklmSlqnJ4vMRcRq4CTgKMyc21mrgVexT4OK4mIceCdwPWZubnXZUsaPmaKpJLMFEklmSlqghLXiX8W8OXMnHlGl4uAp0fEni+IfTIiNgNXAiuAZxRYrqThZKZIKslMkVSSmaKBW/Dh9Jm5ctbj84DzZk27DTisevj47lqTdCAwUySVZKZIKslMURPV2RM/BayuTuhQVETsOVHEODBdur6kRjJTJJVkpkgqyUxR4827Jz4zrweOXIyFZ+Z2YN1i1JbUTGaKpJLMFEklmSlqgxLfiZckSZIkSX3QzSXmBicKXGIxGvi5xcR4sVK3Ta2cfyZJkMBU70eyTa85qPdeKiO7p4rUGd06WaQOwK7JiWK1pKFX4j1GlskBoNx7npFyl7jeVXL9pGEWQUz0/n9w7Nw5/0w1Ta9cXqTO1OqlReoAfPlnDyhS52B+VKROvzRwRCtJkiRJkvbFQbwkSZIkSS3hIF6SJEmSpJZwEC9JkiRJUks4iJckSZIkqSUcxEuSJEmS1BLzDuIjYm1EbI+ITdXjqYjYFBFXRsQ/RcTyavpYRNwaEe+c9foNEXFVRGyOiB9GxAciYk313LKq1mREHFp+9SQ1jZkiqSQzRVJJZoraoO6e+Kszc111f3tmrsvM44BJ4Oxq+pOAq4DnRvzWBd3PzMzjgeOBncBFAJm5vap7Y/erIKmFzBRJJZkpkkoyU9RovR5O/1Xg/tX9M4D3AtcBj9zXzJk5CbwROCoiTpiveEScFREbI2LjZO7osVVJLdC/TJneXqhlSQ3Wt0zZxc5CLUtqMMc+aoSuB/ERMQY8BbgiIpYBTwQuBj5FZ6Pep8ycAi4Hjp1vGZl5bmauz8z1E7G021YltUDfM2VkWZnGJTVSvzNlnCVlGpfUSI591CTdDOKXVd8R2Ujnk6ePAU8DLsnMbcCFwGkRMTpHjdmHnEg6cJkpkkoyUySVZKaocca6eM2e73L8RkScATwmIq6pJh0CnAJ8afaLqw38wcCWLpYtafiYKZJKMlMklWSmqHF6vsRcRKwCTgKOysy1mbkWeBX7OKwkIsaBdwLXZ+bmXpctafiYKZJKMlMklWSmqAlKXCf+WcCXM3PmGV0uAp4eEXu+IPbJiNgMXAmsAJ5RYLmShpOZIqkkM0VSSWaKBm7Bh9Nn5spZj88Dzps17TbgsOrh47trTdKBwEyRVJKZIqkkM0VNVGdP/BSwujqhQ1ERsedEEePAdOn6khrJTJFUkpkiqSQzRY037574zLweOHIxFp6Z24F1i1FbUjOZKZJKMlMklWSmqA0iMwfdQy0RcQtw7TyzHQrcWmiRpWo1rU7JWv3u6X6Zedg880i1mClDvW51a5kpKsZMGep1q1vLTFExZkrRWk2rU7fWfjOlNYP4OiJiY2aub1KtptUZ9p6kkpq4jTetzrD3JJXUxG28aXWGvSeppCZu48PcU5PWrcTZ6SVJkiRJUh84iJckSZIkqSWGbRB/bgNrNa1OyVpN7EkqqYnbeNPqlKzVxJ6kkpq4jTetTslaTexJKqmJ2/gw99SYdRuq78Q3WUTcNfM6kxHxImB9Zv5ZgdobgNdn5sZZ0/8MeC3we8BhmVnqRAySBmxAmfJJYD2wC/g28PLM3NXr8iQN3oAy5WN0MiWAHwEvysy7el2epMEbRKbMeP79wItnLn/YDNueeO3t68CpzH9mS0mq45PAscCDgWXASwfbjqSWe11mnpCZxwPXAT2/uZd0YIuI9cCaQfex2BzEN0BEHBYRF0bEd6rbY6rpD4+Ib0TE96p/j6mmL4uI8yNic0RcQOfN9G/JzO9l5jX9WxNJTbCImfK5rNDZE3/fvq2UpIFZxEy5o5o/qnk8PFQ6ACxWpkTEKPBu4I19W5kBGRt0AweQZRGxacbjg4HPVvffC5yTmV+LiKOALwAPBH4IPDYzd0fEqcB/A54NvALYlpnHR8TxwHf7tRKSGmNgmRIR48B/Av5zyRWSNFADyZSI+HvgqcAPgP+r8DpJGpxBZMqfAZ/NzJ93PhscXg7i+2d7Zq7b82DP90Kqh6cCvz9jY1sVEQcBq4GPR8TRdD6dHq+efyzwPoDM3BwRmxe9e0lNM8hM+SDwlcz8aoH1kNQMA8mUzHxxtffs/cDpwN+XWiFJA9XXTImIewN/Ajy+9Io0kYP4ZhgBHpWZ22dOrE7KcElmnhYRa4ENM572kDNJ+7NomRIR/zdwGPDyMq1KaoFFfZ+SmVPVIbJvwEG8dCBYjEx5CHB/4CfVhwPLI+InmXn/Yl03iN+Jb4YvMuNkLhGxrrq7GvhZdf9FM+b/CnBmNe9xwPGL3qGkNlmUTImIlwJPBs7IzOmiHUtqsuKZEh3333Mf+GM6h9JKGn7FMyUz//+Zec/MXJuZa+kcfj+UA3hwEN8UrwHWVydr+AFwdjX9XcA7I+LrwOiM+T8ErKwOJXkjnRNM/ZaIeE1E3EDn5FObI+Kji7YGkppkUTIF+DBwBPDNiNgUEW9bnPYlNcxiZErQOWz2CuAK4F7AOxZrBSQ1ymK9TzlgeJ14SZIkSZJawj3xkiRJkiS1hIN4SZIkSZJawkG8JEmSJEkt4SBekiRJkqSWcBAvSZIkSVJLOIiXJEmSJKklHMRLkiRJktQSDuIlSZIkSWoJB/GSJEmSJLWEg3hJkiRJklrCQbwkSZIkSS3hIF6SJEmSpJZwEC9JkiRJUkuMDboBtUdEHFxjtunMvL0fdSS1m5kiqSQzRVJJTc6UyMy68+oAFxE7gBuBmGO20cw8qh91JLWbmSKpJDNFUklNzpRW74mPiM/WmO22zHzRYvfSZBHxvhqz3ZGZb51nni2Z+ZB5lvW9GssqVUcqykypx0yR6jFT6jFTpHrMlHoOhExp9Z74iPgx8NK5ZgH+NjMf1KeWFk1EjAArM/OOLl57LfC2eWZ7c2Y+cJ46SzNzR1PmkUozU2q/1kyRajBTar/WTJFqMFNqv3boM6XVe+KBt2Tmv881Q0T8Zb+aKS0i/hdwNjAFXAasjoi/zsx3L7DUOZn58XmWdY/5iuzZsCLiwcCx1eQtmXnl7Hn6UUdaBGZKPWaKVI+ZUo+ZItVjptQz9JnS6j3x+1L9Qm7PIVixiNiUmesi4kzgYcCbgMsy8/gu6x2ambf20M9q4CLgSGAznU/7HgxcBzyj7idlpepI/WCmzFnPTJEWyEyZs56ZIi2QmTJnvaHNlFZfYi4i3hYRx1b3l0TEJcDVwM0RcepguytiPCLGgWcCF2XmLmDBf6AR8ccRcQtwRUTcEBGP7rKf/wpsBI7OzNMy85nA0cB3gL8aQB2pKDOlHjNFqsdMqcdMkeoxU+o5IDIlM1t7A77P3UcTnAVcAowCDwS+PeDe7gecWt1fBhzURY3XAD8DPkfnE5v7AV/tos5m4Njq/iOAf+9ynX4AjO1j+hidQ0L6Wsebt9I3M6V2HTPFm7caNzOldh0zxZu3GjczpXadoc+UVu+JByaz+gkATwbOz8ypzNzCAL/vHxEvAz4N/F016b7A/1loncx8X2beJzOfmh3XAqd00dLuzPxhVfNS4KAuakDn5717H33uBnYOoI5UmplSj5ki1WOm1GOmSPWYKfUMfaa0/cR2OyPiOOBmOr/g1894bsVgWgLgVcDDgUsBMvPHEXF43RdHxJ9m5ici4s/3M8tfL7Cfw2fV2utxZtattzQiHsJvX+MwgCUL6KdUHak0M6UeM0Wqx0ypx0yR6jFT6hn6TGn7IP4/0/nU5zA6ZyH8D4CIeCrw3QH2tTMzJyM6v6eIGGNh3+fY80fY7adGs31kVq3Zj+v6Ofv/I7ppAHWk0syUeswUqR4zpR4zRarHTKln6DNl6M5Ov0dEPDszLxzQst8F3A68AHg18ErgB5n5lgXUGAVek5nnLEqThUXEeHZOPtGIOlJpZkp/mSkadmZKf5kpGnZmSn8NOlPa/p34uQxyA3gzcAtwBfByOidneOtCCmTmFPD0Es1ExD/OuP8/Zj33xR7qRkQ8ISI+Ctww6DrSIjNTKmaKVISZUjFTpCLMlMqBkCnDPIif/Z2DfnoG8A+Z+SeZ+ZzM/Eh2d8jDNyLiAxFxckQ8dM+tizpHz7j/B7OeO2yhxSLiERHxXuBa4LPAV4FjB1VH6hMz5W5mitQ7M+VuZorUOzPlbkOfKW3/TvxcBvk9gacDfxMRXwHOB76wrzMS1rDnmobvmDEtgScssM5cP4vaP6eI+CvgucB1wKeqvjZm5scX0kypOlKfmSl7v6ab5/ZipugAZ6bs/ZpuntuLmaIDnJmy92u6eW4vTc6UVg/iI+IK9v2LCOCIPrfzG5n54ogYB54CPB/4YET8a2a+dIF1urmkwr4sj84ZEUeAZXH32RGDznUc6zoLuAr4EHBxZu6IiG4Co1QdqSgzpTYzRarBTKnNTJFqMFNqG/pMafWJ7SLifnM9X11bcGCqjfkPgRcDJ2dmN4dv/BHwIGDpnmmZ+Y79v2KfNTYwx6dOdf9gonPCiScBZ9D5ROwS4FTgyIV82laqjlSamVK7xgbMFGleZkrtGhswU6R5mSm1a2xgyDOl7YP4+wNHZObXZ00/GbgxM68eUF9/CDyPzvUbNwAXAF9c6C8pIj4MLK/qfBR4DvDtzHxJ0Ya7EBFLgafR2RhPAv4tM58/qDpSCWbK4JgpGkZmyuCYKRpGZsrgNC5TMrO1N+Bi4Ph9TF8P/PMA+zofeCawpMc6m2f9u5LOH8RC65wI3HPG4xcAFwHvAw5eQJ2lwGuBD9A5LGSsmr4KeGG/63jzVvpmptSuY6Z481bjZqbUrmOmePNW42am1K4z9JnS9j3xV2bmcft57orMfHC/e5qx/CPobEDQ+QTpF13UuDQzHxER3wKeBfwSuDIzj57npbPrfBc4NTNvi4jH0vlDezWwDnhgZj6nZp0LgF10zqT4FOCazHztQnopWUcqzUypXcdMkWowU2rXMVOkGsyU2nWGPlNafWI7ZnxXYh8WctKCoiLiT4D30DmcJID3R8QbMvPTCyx1cUSsAd4FXFZN+2gXLY1m5m3V/dOBczPzQuDCiNi0gDq/vyccIuJjwLe76KVkHak0M6UeM0Wqx0ypx0yR6jFT6hn6TGn7IP47EfGyzPzIzIkR8RLu/sUPwluBE/d8AhURhwFfAha6Ib8HeAVwMvBNOp/efKiLfkYjYiw730t5Ip3DOPZYyDawa8+dzNwd0fXlKEvVkUozU+oxU6R6zJR6zBSpHjOlnqHPlLYfTn8E8Blgkrs33PXABHBaZt40oL72OpwlIkaAyxd6iEtE/CNwJ/CJatIZwJrMfO4C67wFeCpwK3AU8NDMzOrkGB/PzMfUrDMFbN3zkM4nftuq+5mZq/pZRyrNTKldx0yRajBTatcxU6QazJTadYY+U1o9iN8jIk4B9nw/5PuZ+eUB9/Nu4HjgU9Wk0+mcoOFNC6xzeWaeMN+0mrUeCdyLzskhtlbTHgCszMzvLrSeNMzMlFq1zBSpJjOlVi0zRarJTKlVa6gzpdWD+Ij4bmY+tNd5FkNEPBt4DJ1PWL6SmZ/posZ5wIcz81vV40fQOYPhKxdYp8jPqWnzSKU1eds0UxZ/Hqm0Jm+bZsrizyOV1uRt00xZ/Hn2mr/lg/jtwI/nmgVYnZlH9amloiJiC3AMcF016ShgCzBN59CL42vWKfJzalodqbRh3zbNlHb+3tRew75tmint/L2pvYZ92zRT6v/e2n5iu2NrzDM13wwR8bXMPCki7gRmfqqx0O87zH59V3Vm+MMFzr8/RX5ODawjlWam1NO0LDBT1FRmSj1NywIzRU1lptTTtCwonimt3hMvSZIkSdKBZGTQDUiSJEmSpHocxEuSJEmS1BJDNYiPiLOaVqtpdUrWamJPUklN3MabVqdkrSb2JJXUxG28aXVK1mpiT1JJTdzGh7mnJq3bUA3igZIBW6pW0+qUrNXEnqSSmriNN61OyVpN7EkqqYnbeNPqlKzVxJ6kkpq4jQ9zT41Zt2EbxEuSJEmSNLRac3b6iViay2LFnPNMspMJlsxdaHy81vImp7cxMbJ8znl23Gf+K/RN3bGV0VVz9z3265i3zu4dWxlbOncdgLHbt887z2TuYCKWzjlPTk/PW2cXOxmf7+ddU51ad/KrWzPzsCIL1AFvIpbk0nkyZVfuZDzm3i5jrN6VOientzMxsmzOeXJ8/lq7dm9jfGzubBo9cte8dXbevoMla+bOAYDpa0bnnWdyahsTo3P3lDsn561jpqjNRg9akWOH3mPOeabu3MroQXPnztLrdtRaXp3/y6mRT3X+fqeX1simya2MT8z/PmVkcv6rKNXKlB075+/JTFGLTYyvyKVL1sw5z65dWxkfn/vvLrbP/7cCdTOlznuC7UyMzv1+Z2r5RK2edu28i/ElK+eeZ/7YYXrrVkZWzD3jkp9tnb+fBmVKa64TvyxW8MilT+25zsi9jijQTceWvzykSJ0jPl9vQ65jzUVXFKkzvXX+DbnfvpSfvnbQPWh4LI0VPHLsyT3XGT3s0ALddEzds0ymrHr/TUXqAGx9wUFF6uz+6TVF6pRkpqiksUPvwb3+8s96rnPMq35YoJuOkcPKZMrWB5Z777T8ujuK1Jn6/lVF6pRkpqikpUvW8PATXtFzndHNPynQTcfIoQcXqXPHQ+9dpA7ATY8oc2D57775m0XqlDRXpng4vSRJkiRJLeEgXpIkSZKklnAQL0mSJElSSziIlyRJkiSpJRzES5IkSZLUErUG8RGxNiK2R8Sm6vE9I+L8iLg6In4QEZ+LiAdU8125j9c/MiIujYhNEbElIt5eTT89In4SEReXXClJzWamSCrNXJFUkpmiJlvIJeauzsx1ERHAZ4CPZ+bzACJiHXAEcP1+Xvtx4LmZeXlEjALHAGTmBRFxM/D6bldAUmuZKZJKM1cklWSmqJG6uU78KcCuzPzwngmZuQk6n1jt5zWHAz+v5p0CflBnQRFxFnAWdK7pLGkoDSZTWN51w5Iary+5MjNTRg9Z00u/kpqt75mydGJ1Tw1ruHXznfjjgMsW+JpzgKsi4jMR8fKIWFrnRZl5bmauz8z1EyxZcKOSWmEgmTIeZoo0xPqSKzMzZfQgdzZIQ6zvmTI+bqZo//pyYrvMfAewHvgi8Hzg8/1YrqThZKZIKs1ckVSSmaLF1M0g/vvAwxb6osy8OjM/BDwROCEiDuli2ZKGj5kiqTRzRVJJZooapZtB/JeBJRHxsj0TIuLEiHjc/l4QEX9UnRAC4GhgCri9i2VLGj5miqTSzBVJJZkpapQFn9guMzMiTgP+JiLeDOwArgFeW81yTETcMOMlrwOeDZwTEduA3cCZ1QkeJB3gzBRJpZkrkkoyU9Q03Zydnsy8EXjufp4e38e0f+pmOZIODGaKpNLMFUklmSlqkrqH008BqyNiU8mFR8TpwAeBX5WsK6nxzBRJpZkrkkoyU9RYkZmD7qGW1RNH5KOPeF7PdaZu/WWBbjpidLRInZFVBxWpA/AH/3ZVkTr/8qA1ReqU9KX89GWZuX7QfWg4rIqD8xGjT+q5zsjEvj5877ZYmQuGxIpyl6V5/8bPFKnzyvudVKROSWaKSlo9cUQ++p5n9Fxn9403Feimo9z7lJVF6gC85tKvF6lzzv0fWKROSWaKSloVB+cj4omDbmNvv/mKf3N84WffK1LnyfdeV6ROSXNlSl8uMSdJkiRJknrnIF6SJEmSpJZwEC9JkiRJUks4iJckSZIkqSUcxEuSJEmS1BIO4iVJkiRJaolGDOIj4huD7kHS8DBTJJVkpkgqyUxRrxoxiM/MRw+6B0nDw0yRVJKZIqkkM0W9asQgPiLuGnQPkoaHmSKpJDNFUklmino1NugG5hIRZwFnASwdPWjA3Uhqu70yheUD7kZS2/k+RVJJvk9RXY3YE78/mXluZq7PzPUTI8sG3Y6klpuZKeMsGXQ7klrO9ymSSvJ9iupq9CBekiRJkiTdzUG8JEmSJEkt4SBekiRJkqSWaMQgPjNXDroHScPDTJFUkpkiqSQzRb1qxCBekiRJkiTNz0G8JEmSJEkt4SBekiRJkqSWGBt0AwsSMegO9hJLmnf9xl/vXj7oFqR2iCBGRwfdxV5uOfOEInUOv+SmInUAnvy1Vxep83t8r0gdqbHGRpk+dHXPZUZ+dXvvveypteqgQoXK7fP56M9PLlTp1kJ1JNVV7H1TlMuUbdOTxWq1iXviJUmSJElqCQfxkiRJkiS1hIN4SZIkSZJawkG8JEmSJEkt4SBekiRJkqSW6GkQHxF3lWpEkswUSSWZKZJKMlPUFO6JlyRJkiSpJYoM4qPj3RFxZURcERGnV9MviIinzpjvvIh4dkSMVvN/JyI2R8TLS/QhaTiYKZJKMlMklWSmaNBK7Yl/FrAOOAE4FXh3RNwLOB/Ys1FPAE8EPge8BPh1Zp4InAi8LCJ+p1AvktrPTJFUkpkiqSQzRQNVahB/EvCpzJzKzJuBf6ezgf4L8ISIWAI8BfhKZm4HngS8ICI2AZcChwBHzy4aEWdFxMaI2Dg5vb1Qq5JaYNEzZVfu6NOqSGqAxX+fsntbn1ZFUgMs/vsUdvZpVdRGY4XqxL4mZuaOiNgAPJnOp1KfmjH/qzPzC3MVzcxzgXMBVk8ckYV6ldR8i54pq0YOMVOkA8fiv09Zfm8zRTpwLP77lDjYTNF+ldoT/xXg9Or7HocBjwW+XT13PvBi4GRgz4b7BeAVETEOEBEPiIgVhXqR1H5miqSSzBRJJZkpGqhSe+I/AzwKuBxI4I2ZeVP13BeBfwA+m5mT1bSPAmuB70ZEALcAzyzUi6T2M1MklWSmSCrJTNFA9TSIz8yV1b8JvKG6zZ5nF53vfcycNg38RXWTJMBMkVSWmSKpJDNFTeF14iVJkiRJagkH8ZIkSZIktYSDeEmSJEmSWqLUie0W3eQhE1z3/Pv1XOfID99ZoJuOWLG8SJ1csaxIHYBPf/LxRercm28UqSM1VQAx2vvnmLl7d+/NVI743LVF6vz0ZWuL1AGY/uVUsVrSMMuxEXatWdpznbFj1/beTGXXkjJv88Z+cUeROgDf/9e1Reocxa1F6khNFSMjjCzrfayRu8q9TxlZs7pMoely7y3+8a77FqvVJu6JlyRJkiSpJRzES5IkSZLUEg7iJUmSJElqCQfxkiRJkiS1hIN4SZIkSZJawkG8JEmSJEkt0ddBfEScHREv6OcyJQ0vM0VSSWaKpJLMFC2Wvl0nPiLGMvPD/VqepOFmpkgqyUyRVJKZosW0oEF8RKwFPg9cCjwE+BHwAuD1wB8Dy4BvAC/PzIyIDdXjxwCfjYiDgLsy8z0R8RrgbGA38IPMfF6RNZLUGmaKpJLMFEklmSlqqm4Opz8GODczjwfuAF4JfCAzT8zM4+hszE+bMf+azHxcZv7PWXXeDDykqnP2vhYUEWdFxMaI2Di1bWsXrUpqgYFkyiQ7y6+JpCYYTKZM+j5FGlKDyZTcUX5NNDS6GcRfn5lfr+5/AjgJOCUiLo2IK4AnAA+aMf8F+6mzGfhkRPwpnU+kfktmnpuZ6zNz/ejyFV20KqkFBpIpEywp1L6khhlMpkz4PkUaUoPJlFhaqH0No24G8bmPxx8EnpOZDwY+Aszc6vb30fQfAX8LPAy4LCL69v18SY1ipkgqyUyRVJKZosbpZhB/VEQ8qrp/BvC16v6tEbESeM58BSJiBDgyMy8B3gisAVZ20Yuk9jNTJJVkpkgqyUxR43TzCdAW4IUR8XfAj4EPAfcArgCuAb5To8Yo8ImIWA0EcE5m3t5FL5Laz0yRVJKZIqkkM0WN080gfjozZ5+M4a3VbS+Z+fhZj98+4+FJXSxb0vAxUySVZKZIKslMUeN0czi9JEmSJEkagAXtic/Ma4DjFqcVSQcaM0VSSWaKpJLMFDWVe+IlSZIkSWqJ1lzaYPzmrdznr7/dc52p3fu8LGN37rijTJ2R0TJ1gCte++kidZ78rnVF6khNlZlMT+7quU6MRIFuOqZu/kWROmv/+21F6gD8y0+/VaTOk1+zrkgdqbHu2sbY1zb3XCanpgo001EqnaYnJgpVgi1nf6ZInSe/Y12ROlJT5fQ009u2DbqNvUzdckuZQgXHPi9aVea906e4d5E6/eKeeEmSJEmSWsJBvCRJkiRJLeEgXpIkSZKklnAQL0mSJElSSziIlyRJkiSpJRzES5IkSZLUErUH8RGxNiK2R8Sm6vFbIuL7EbE5IjZFxCOq6Rsi4qpq2qaI+HQ1/e0R8bNq2pUR8fRq+usi4rqI+MAirJ+khjJTJJVkpkgqyUxRky30OvFXZ+a6iHgU8DTgoZm5MyIOBWZeRPTMzNy4j9efk5nviYgHAl+NiMMz85yI+BWwvrtVkNRiZoqkkswUSSWZKWqkhQ7i97gXcGtm7gTIzFsX8uLM3BIRu4FDgV/sb76IOAs4C2Apy7tsVVILmCmSSjJTJJVkpqhRuv1O/BeBIyPiRxHxwYh43KznPznjkJJ3z35xdfjJNHDLXAvJzHMzc31mrh+PJV22KqkF+p8pmCnSEPN9iqSSfJ+iRulqT3xm3hURDwNOBk4BLoiIN2fmedUs+zuk5HUR8afAncDpmZndLF/ScDFTJJVkpkgqyUxR03R7OD2ZOQVsADZExBXAC4Hz5nnZOZn5nm6XKWl4mSmSSjJTJJVkpqhJujqcPiKOiYijZ0xaB1xbpCNJBxwzRVJJZoqkkswUNU23e+JXAu+PiDXAbuAnVCdhqHwyIrZX92/NzFO7b1HSAcBMkVSSmSKpJDNFjdLtd+IvAx69n+cev5/pb+9mWZKGn5kiqSQzRVJJZoqaZiGH008BqyNiU8kGIuJ1wH8B7ihZV1LjmSmSSjJTJJVkpqixau+Jz8zrgSNLN5CZ5wDnlK4rqdnMFEklmSmSSjJT1GRdn52+36YOXsFtTz+x5zqHfv3mAt1UxkaLlNlxn1VF6gA86v/q/WcEsIpvFakjNVZAjETPZXK63NViRleXyYLrX/bAInUAjt5wbJE6v8umInWkpoqlS4ljjp5/xvnq/PSGAt1UtY44tEydqekidQBO2vysInVW8NMidaSmivExxg49ouc603feVaCbjpGD71Gkzo1/fFSROgBH/78PL1Lnd/lmkTr90tXZ6SVJkiRJUv85iJckSZIkqSUcxEuSJEmS1BIO4iVJkiRJagkH8ZIkSZIktYSDeEmSJEmSWsJBvCRJkiRJLTHvID4i1kbE9ojYVD2eiohNEXFlRPxTRCyvpo9FxK0R8c5Zr98QEVdFxOaI+GFEfCAi1lTPLatqTUZEmYuZSmo0M0VSSWaKpJLMFLVB3T3xV2fmuur+9sxcl5nHAZPA2dX0JwFXAc+NiJj1+jMz83jgeGAncBFAZm6v6t7Y/SpIaiEzRVJJZoqkkswUNVqvh9N/Fbh/df8M4L3AdcAj9zVzZk4CbwSOiogT5iseEWdFxMaI2Lh7x9YeW5XUAn3LlF25s1DLkhqsb5kyuXtboZYlNVj/MmV6e6GWNYy6HsRHxBjwFOCKiFgGPBG4GPgUnY16nzJzCrgcOHa+ZWTmuZm5PjPXjy1d0W2rklqg35kyHkvKNC6pkfqdKRNjy8s0LqmR+p4pI8vKNK6h1M0gfln1HZGNdD55+hjwNOCSzNwGXAicFhGjc9SYfciJpAOXmSKpJDNFUklmihpnrIvX7Pkux29ExBnAYyLimmrSIcApwJdmv7jawB8MbOli2ZKGj5kiqSQzRVJJZooap+dLzEXEKuAk4KjMXJuZa4FXsY/DSiJiHHgncH1mbu512ZKGj5kiqSQzRVJJZoqaoMR14p8FfDlzr7NEXQQ8PeI3Xzr9ZERsBq4EVgDPKLBcScPJTJFUkpkiqSQzRQO34MPpM3PlrMfnAefNmnYbcFj18PHdtSbpQGCmSCrJTJFUkpmiJqqzJ34KWF2d0KGoiNhzoohxYLp0fUmNZKZIKslMkVSSmaLGm3dPfGZeDxy5GAvPzO3AusWoLamZzBRJJZkpkkoyU9QG3ZydfiCmx2HrvXu/OsOhBXr5jShztYiR3VmkDsDKG3bOP5MkICB6Py3InBeUWaixMpG84+BymZI/X1qsljTUMoldU73XWVrub27XPVcXqTP+izuL1AG48aY1ReocXaSK1HAFxhoxMVGgkY48aHmROpHl3qeM33VgXr2vxIntJEmSJElSHziIlyRJkiSpJRzES5IkSZLUEg7iJUmSJElqCQfxkiRJkiS1hIN4SZIkSZJaYt5BfESsjYjtEbGpejwVEZsi4sqI+KeIWF5NH4uIWyPinbNevyEiroqIzRHxw4j4QESsqZ5bVtWajIiiV3+T1ExmiqSSzBRJJZkpaoO6e+Kvzsx11f3tmbkuM48DJoGzq+lPAq4CnhvxWxc1PDMzjweOB3YCFwFk5vaq7o3dr4KkFjJTJJVkpkgqyUxRo/V6OP1XgftX988A3gtcBzxyXzNn5iTwRuCoiDhhvuIRcVZEbIyIjbu3be2xVUkt0LdM2ZU7CrUsqcH6limTU9sKtSypwfqXKdPbC7WsYdT1ID4ixoCnAFdExDLgicDFwKfobNT7lJlTwOXAsfMtIzPPzcz1mbl+bPmKbluV1AL9zpTxWFqmcUmN1O9MmRhdXqZxSY3U90wZWVamcQ2lbgbxy6rviGyk88nTx4CnAZdk5jbgQuC0iBido8bsQ04kHbjMFEklmSmSSjJT1DhjXbxmz3c5fiMizgAeExHXVJMOAU4BvjT7xdUG/mBgSxfLljR8zBRJJZkpkkoyU9Q4PV9iLiJWAScBR2Xm2sxcC7yKfRxWEhHjwDuB6zNzc6/LljR8zBRJJZkpkkoyU9QEJa4T/yzgy5m5c8a0i4CnR8SS6vEnI2IzcCWwAnhGgeVKGk5miqSSzBRJJZkpGrgFH06fmStnPT4POG/WtNuAw6qHj++uNUkHAjNFUklmiqSSzBQ1UZ098VPA6uqEDkVFxJ4TRYwD06XrS2okM0VSSWaKpJLMFDXevHviM/N64MjFWHhmbgfWLUZtSc1kpkgqyUyRVJKZojaIzBx0D7VExC3AtfPMdihwa6FFlqrVtDola/W7p/tl5mHzzCPVYqYM9brVrWWmqBgzZajXrW4tM0XFmClFazWtTt1a+82U1gzi64iIjZm5vkm1mlZn2HuSSmriNt60OsPek1RSE7fxptUZ9p6kkpq4jQ9zT01atxJnp5ckSZIkSX3gIF6SJEmSpJYYtkH8uQ2s1bQ6JWs1sSeppCZu402rU7JWE3uSSmriNt60OiVrNbEnqaQmbuPD3FNj1m2ovhPfZBFx18zrTEbEi4D1mflnBWpvAF6fmRtnTT8PeBzw62rSizJzU6/LkzR4A8qUAP4f4E/oXILnQ5n5vl6XJ2nwBpQpXwUOqh4eDnw7M5/Z6/IkDd6AMuWJwLvp7Ki+i87Y5ye9Lq+J5r3EnFrvDZn56UE3IWkovIjOZXeOzczpiDh8wP1IarHMPHnP/Yi4ELhogO1Iar8PAc/IzC0R8UrgrXTeuwydYTucvpUi4rCIuDAivlPdHlNNf3hEfCMivlf9e0w1fVlEnB8RmyPiAmDZQFdAUqMsYqa8AnhHZk4DZOYv+rJCkgZqsd+nRMRBwBOA/7PY6yJp8BYxUxJYVd1fDdy46CszIO6J759lEbFpxuODgc9W998LnJOZX4uIo4AvAA8Efgg8NjN3R8SpwH8Dnk3njfS2zDw+Io4HvjvHcv8qIt4G/Bvw5szcWXStJA3KIDLl94DTI+I04BbgNZn549IrJmkgBvU+BeA04N8y845yqyNpwAaRKS8FPhcR24E7gEeWXqmmcBDfP9szc92eB3u+F1I9PBX4/c7XTQFYVX0qvRr4eEQcTeeTpfHq+ccC7wPIzM0RsXk/y/wvwE3ABJ2TJ7wJeEeh9ZE0WIPIlCXAjsxcHxHPAv5/wMn7mVdSuwwiU/Y4A/hogXWQ1ByDyJTXAU/NzEsj4g3AX9MZ2A8dB/HNMAI8KjO3z5wYEe8HLsnM0yJiLbBhxtPznpEwM39e3d0ZEX8PvL5Mu5IablEyBbgBuLC6/xng73tvVVILLFamEBGHAA+nszde0oGheKZExGHACZl5aTXpAuDzxTpuGL8T3wxfBH5zpsaIWFfdXQ38rLr/ohnzfwU4s5r3OOD4fRWNiHtV/wbwTODKci1LarBFyRQ631d9QnX/ccCPSjQrqfEWK1Ogc7WLizNzR6FeJTXfYmTKr4DVEfGA6vEfAFuKddwwDuKb4TXA+upkDT8Azq6mvwt4Z0R8HRidMf+HgJXVoSRvBL69n7qfjIgrgCuAQ+lcGkrS8FusTPnvwLOrXHknQ3qImqTfsliZAvA84FOL0LOk5iqeKZm5G3gZcGFEXA78J+ANi7gOA+V14iVJkiRJagn3xEuSJEmS1BIO4iVJkiRJagkH8ZIkSZIktYSDeEmSJEmSWsJBvCRJkiRJLeEgXpIkSZKklnAQL0mSJElSSziIlyRJkiSpJRzES5IkSZLUEg7iJUmSJElqCQfxkiRJkiS1hIN4SZIkSZJawkG8JEmSJEkt4SBekiRJkqSWGBt0A2qPiDi4xmzTmXl7P+pIajczRVJJZoqkkpqcKZGZdefVAS4idgA3AjHHbKOZeVQ/6khqNzNFUklmiqSSmpwprd4THxGfrTHbbZn5osXupcki4n01ZrsjM986zzxbMvMh8yzrezWWVaqOVJSZUo+ZItVjptRjpkj1mCn1HAiZ0uo98RHxY+Clc80C/G1mPqhPLS2aiBgBVmbmHV289lrgbfPM9ubMfOA8dZZm5o6mzCOVZqbUfq2ZItVgptR+rZki1WCm1H7t0GdKq/fEA2/JzH+fa4aI+Mt+NVNaRPwv4GxgCrgMWB0Rf52Z715gqXMy8+PzLOse8xXZs2FFxIOBY6vJWzLzytnz9KOOtAjMlHrMFKkeM6UeM0Wqx0ypZ+gzpdV74vel+oXcnkOwYhGxKTPXRcSZwMOANwGXZebxXdY7NDNv7aGf1cBFwJHAZjqf9j0YuA54Rt1PykrVkfrBTJmznpkiLZCZMmc9M0VaIDNlznpDmymtvsRcRLwtIo6t7i+JiEuAq4GbI+LUwXZXxHhEjAPPBC7KzF3Agv9AI+KPI+IW4IqIuCEiHt1lP/8V2AgcnZmnZeYzgaOB7wB/NYA6UlFmSj1milSPmVKPmSLVY6bUc0BkSma29gZ8n7uPJjgLuAQYBR4IfHvAvd0POLW6vww4qIsarwF+BnyOzic29wO+2kWdzcCx1f1HAP/e5Tr9ABjbx/QxOoeE9LWON2+lb2ZK7TpmijdvNW5mSu06Zoo3bzVuZkrtOkOfKa3eEw9MZvUTAJ4MnJ+ZU5m5hQF+3z8iXgZ8Gvi7atJ9gf+z0DqZ+b7MvE9mPjU7rgVO6aKl3Zn5w6rmpcBBXdSAzs979z763A3sHEAdqTQzpR4zRarHTKnHTJHqMVPqGfpMafuJ7XZGxHHAzXR+wa+f8dyKwbQEwKuAhwOXAmTmjyPi8Lovjog/zcxPRMSf72eWv15gP4fPqrXX48ysW29pRDyE377GYQBLFtBPqTpSaWZKPWaKVI+ZUo+ZItVjptQz9JnS9kH8f6bzqc9hdM5C+B8AEfFU4LsD7GtnZk5GdH5PETHGwr7PseePsNtPjWb7yKxasx/X9XP2/0d00wDqSKWZKfWYKVI9Zko9ZopUj5lSz9BnytCdnX6PiHh2Zl44oGW/C7gdeAHwauCVwA8y8y0LqDEKvCYzz1mUJguLiPHsnHyiEXWk0syU/jJTNOzMlP4yUzTszJT+GnSmtP078XMZ5AbwZuAW4Arg5XROzvDWhRTIzCng6SWaiYh/nHH/f8x67os91I2IeEJEfBS4YdB1pEVmplTMFKkIM6VipkhFmCmVAyFThnkQP/s7B/30DOAfMvNPMvM5mfmR7O6Qh29ExAci4uSIeOieWxd1jp5x/w9mPXfYQotFxCMi4r3AtcBnga8Cxw6qjtQnZsrdzBSpd2bK3cwUqXdmyt2GPlPa/p34uQzyewJPB/4mIr4CnA98YV9nJKxhzzUN3zFjWgJPWGCduX4WtX9OEfFXwHOB64BPVX1tzMyPL6SZUnWkPjNT9n5NN8/txUzRAc5M2fs13Ty3FzNFBzgzZe/XdPPcXpqcKa0exEfEFez7FxHAEX1u5zcy88URMQ48BXg+8MGI+NfMfOkC63RzSYV9WR6dMyKOAMvi7rMjBp3rONZ1FnAV8CHg4szcERHdBEapOlJRZkptZopUg5lSm5ki1WCm1Db0mdLqE9tFxP3mer66tuDAVBvzHwIvBk7OzG4O3/gj4EHA0j3TMvMd+3/FPmtsYI5Pner+wUTnhBNPAs6g84nYJcCpwJEL+bStVB2pNDOldo0NmCnSvMyU2jU2YKZI8zJTatfYwJBnStsH8fcHjsjMr8+afjJwY2ZePaC+/hB4Hp3rN24ALgC+uNBfUkR8GFhe1fko8Bzg25n5kqINdyEilgJPo7MxngT8W2Y+f1B1pBLMlMExUzSMzJTBMVM0jMyUwWlcpmRma2/AxcDx+5i+HvjnAfZ1PvBMYEmPdTbP+nclnT+IhdY5EbjnjMcvAC4C3gccvIA6S4HXAh+gc1jIWDV9FfDCftfx5q30zUypXcdM8eatxs1MqV3HTPHmrcbNTKldZ+gzpe174q/MzOP289wVmfngfvc0Y/lH0NmAoPMJ0i+6qHFpZj4iIr4FPAv4JXBlZh49z0tn1/kucGpm3hYRj6Xzh/ZqYB3wwMx8Ts06FwC76JxJ8SnANZn52oX0UrKOVJqZUruOmSLVYKbUrmOmSDWYKbXrDH2mtPrEdsz4rsQ+LOSkBUVFxJ8A76FzOEkA74+IN2TmpxdY6uKIWAO8C7ismvbRLloazczbqvunA+dm5oXAhRGxaQF1fn9POETEx4Bvd9FLyTpSaWZKPWaKVI+ZUo+ZItVjptQz9JnS9kH8dyLiZZn5kZkTI+Il3P2LH4S3Aifu+QQqIg4DvgQsdEN+D/AK4GTgm3Q+vflQF/2MRsRYdr6X8kQ6h3HssZBtYNeeO5m5O6Lry1GWqiOVZqbUY6ZI9Zgp9ZgpUj1mSj1DnyltP5z+COAzwCR3b7jrgQngtMy8aUB97XU4S0SMAJcv9BCXiPhH4E7gE9WkM4A1mfncBdZ5C/BU4FbgKOChmZnVyTE+npmPqVlnCti65yGdT/y2VfczM1f1s45UmplSu46ZItVgptSuY6ZINZgptesMfaa0ehC/R0ScAuz5fsj3M/PLA+7n3cDxwKeqSafTOUHDmxZY5/LMPGG+aTVrPRK4F52TQ2ytpj0AWJmZ311oPWmYmSm1apkpUk1mSq1aZopUk5lSq9ZQZ0qrB/ER8d3MfGiv8yyGiHg28Bg6n7B8JTM/00WN84APZ+a3qsePoHMGw1cusE6Rn1PT5pFKa/K2aaYs/jxSaU3eNs2UxZ9HKq3J26aZsvjz7DV/ywfx24EfzzULsDozj+pTS0VFxBbgGOC6atJRwBZgms6hF8fXrFPk59S0OlJpw75tmint/L2pvYZ92zRT2vl7U3sN+7ZpptT/vbX9xHbH1phnar4ZIuJrmXlSRNwJzPxUY6Hfd5j9+q7qzPCHC5x/f4r8nBpYRyrNTKmnaVlgpqipzJR6mpYFZoqaykypp2lZUDxTWr0nXpIkSZKkA8nIoBuQJEmSJEn1DNUgPiLOmn+u/tZqWp2StZrYk1RSE7fxptUpWauJPUklNXEbb1qdkrWa2JNUUhO38WHuqUnrNlSDeKBkwJaq1bQ6JWs1sSeppCZu402rU7JWE3uSSmriNt60OiVrNbEnqaQmbuPD3FNj1m3YBvGSJEmSJA2t1pzYbmzV8hw/fM2c80zdsY3RVcvnnGfihulay5uc2s7E6LK5ZxqZ/zOQyd1bmRhbMfdM0/P3NDm1jYnRudcNgKn5T2w4Ob2DiZGlc86Tu+evs4udjLNk/p5qqFPrTn51a2YeVmSBOuCNLVuR46sOnnOeqe1bGV0299/vxO27ai2vzt/wrjXj89bZvX0rY/P0NDI5f67v3rmVsSXzZBMw+uvt884zyU4m5vn7rfN/jZmiNpuIJbmUuf+m6myXMTFRa3m13heMxPx1dm9jYmzuOjsOnf9iRlNbtzK6okam7Jh3Fnbv2MrY0rlrjd26dd46ZorarFimjM//3gJgcno7EyNzj31yyfxZMLlrKxPjc/c9taTefuQ6WTA195CmM0+NfJr4WbsypTWXmBs/fA2/+z9f1nOdI98w/xvSunJ5mV/iyF01/kerKX99R5E6U7+8rUidkr6Un7520D1oeIyvOpj7P//Pe65z34tuKNBNx41Pu2+ROgfdsLtIHYAVn99cpM70jnI5V4qZopKWsoJHjD6p5zpj97lPgW46ckm9DwTm86OXlhuXrvlhmTqHfOybZQoVZKaopKWs4BEjp/ZcZ+yIexXopmPydw4vUufX959nR+kC3P6AMnV+5y/alSkeTi9JkiRJUks4iJckSZIkqSUcxEuSJEmS1BIO4iVJkiRJagkH8ZIkSZIktUStQXxErI2I7RGxqXp8z4g4PyKujogfRMTnIuIB1XxX7uP1j4yISyNiU0RsiYi3V9NPj4ifRMTFJVdKUrOZKZJKM1cklWSmqMkWcom5qzNzXUQE8Bng45n5PICIWAccAVy/n9d+HHhuZl4eEaPAMQCZeUFE3Ay8vtsVkNRaZoqk0swVSSWZKWqkbq4TfwqwKzM/vGdCZm6CzidW+3nN4cDPq3mngB/UWVBEnAWcBTB+2OouWpXUAoPJlIPu0XXDkhqvL7kyM1OWsrynhiU1mpmiRunmO/HHAZct8DXnAFdFxGci4uURsbTOizLz3Mxcn5nrR1e5IUtDajCZsmzFghuV1Bp9yZWZmTLOkq4aldQKZooapS8ntsvMdwDrgS8Czwc+34/lShpOZoqk0swVSSWZKVpM3Qzivw88bKEvysyrM/NDwBOBEyLikC6WLWn4mCmSSjNXJJVkpqhRuhnEfxlYEhEv2zMhIk6MiMft7wUR8UfVCSEAjgamgNu7WLak4WOmSCrNXJFUkpmiRlnwie0yMyPiNOBvIuLNwA7gGuC11SzHRMQNM17yOuDZwDkRsQ3YDZxZneBB0gHOTJFUmrkiqSQzRU3Tzdnpycwbgefu5+nxfUz7p26WI+nAYKZIKs1ckVSSmaImqXs4/RSwOiI2lVx4RJwOfBD4Vcm6khrPTJFUmrkiqSQzRY1Va098Zl4PHFl64Zl5AXBB6bqSms1MkVSauSKpJDNFTdbV4fSDMHHNJEe+8Ib5Z5xHrr13gW4qI2Wu0JdbtxWpA/Czjx1RpM49n3lbkTpSU03csZv7/OutvRcqlAMAB92wu0ydzb8oUgdgy3tOKFLn6D+7tEgdqbECYiTmn28eu6/7WYFmOkr0A3D//3JNkToAn77ma0XqPPtjjyxSR2q0zJ5L7P7ZjQUa6Ri9ucz7i4MvLffe6dvXfrtInSf/xboidfqlL9eJlyRJkiRJvXMQL0mSJElSSziIlyRJkiSpJRzES5IkSZLUEg7iJUmSJElqiUYM4iPiG4PuQdLwMFMklWSmSCrJTFGvGjGIz8xHD7oHScPDTJFUkpkiqSQzRb1qxCA+Iu4adA+ShoeZIqkkM0VSSWaKetWIQbwkSZIkSZrf2KAbmEtEnAWcBbA0Vgy4G0ltt1emjK8acDeS2m6vTGH5gLuR1HZmiupq9J74zDw3M9dn5vqJkaWDbkdSy+2VKaP+5yipNzMzZTyWDLodSS23V6Zgpmj/Gj2IlyRJkiRJd3MQL0mSJElSSzRiEJ+ZKwfdg6ThYaZIKslMkVSSmaJeNWIQL0mSJEmS5ucgXpIkSZKklnAQL0mSJElSSziIlyRJkiSpJcYG3UBdOTXN1J139lwnrvqPAt10/PcffqVInTcf/ZgidQDu8/IydabKlJGaa/cU/PL2nsvkdPbeS2XFf/SecQB5621F6gA8+9HXFqmzuUgVqblidJSR1at6LzTVwP+BDz+0WKm/uCkKVdpVqI7UYCOjg+5gL6P3PKJIndy6rUgdgM9vW1KsVpu4J16SJEmSpJZwEC9JkiRJUks4iJckSZIkqSUcxEuSJEmS1BIO4iVJkiRJaomeBvERcVepRiTJTJFUkpkiqSQzRU3hnnhJkiRJklqiyCA+Ot4dEVdGxBURcXo1/YKIeOqM+c6LiGdHxGg1/3ciYnNEFLq6uaRhYKZIKslMkVSSmaJBK7Un/lnAOuAE4FTg3RFxL+B8YM9GPQE8Efgc8BLg15l5InAi8LKI+J3ZRSPirIjYGBEbd7GzUKuSWmDRM2VyentfVkRSI/QhU3b0ZUUkNYJjHw1UqUH8ScCnMnMqM28G/p3OBvovwBMiYgnwFOArmbkdeBLwgojYBFwKHAIcPbtoZp6bmeszc/04Swq1KqkFFj1TJkaW9WlVJDVAHzJlaZ9WRVIDOPbRQI0VqhP7mpiZOyJiA/BkOp9KfWrG/K/OzC8UWr6k4WKmSCrJTJFUkpmigSq1J/4rwOnV9z0OAx4LfLt67nzgxcDJwJ4N9wvAKyJiHCAiHhARKwr1Iqn9zBRJJZkpkkoyUzRQpfbEfwZ4FHA5kMAbM/Om6rkvAv8AfDYzJ6tpHwXWAt+NiABuAZ5ZqBdJ7WemSCrJTJFUkpmigeppEJ+ZK6t/E3hDdZs9zy463/uYOW0a+IvqJkmAmSKpLDNFUklmiprC68RLkiRJktQSDuIlSZIkSWoJB/GSJEmSJLVEqRPbLboYG2P04EN7L5TTvdeovOn0lxapc/2bDypSB2D5TVmkziEf/UWROlJT5ZJxpn7nnj3XGfv5rwp00zE9VuZz1TiiQFZW/vf3f69InfvzvSJ1pKbK3VNM/erXBQqVe58So6Nl6uzaXaQOwNuO2FCkzpk8pkgdqaliZISRpb1fK356x84C3XTkXXeVqXPUvYrUAbhix32L1WoT98RLkiRJktQSDuIlSZIkSWoJB/GSJEmSJLWEg3hJkiRJklrCQbwkSZIkSS3R10F8RJwdES/o5zIlDS8zRVJJZoqkkswULZa+XWIuIsYy88P9Wp6k4WamSCrJTJFUkpmixbSgQXxErAU+D1wKPAT4EfAC4PXAHwPLgG8AL8/MjIgN1ePHAJ+NiIOAuzLzPRHxGuBsYDfwg8x8XpE1ktQaZoqkkswUSSWZKWqqbg6nPwY4NzOPB+4AXgl8IDNPzMzj6GzMT5sx/5rMfFxm/s9Zdd4MPKSqc/a+FhQRZ0XExojYODm9vYtWJbXAQDJl166t5ddEUhMMJlPYWX5NJDXBYMY+uaP8mmhodDOIvz4zv17d/wRwEnBKRFwaEVcATwAeNGP+C/ZTZzPwyYj4UzqfSP2WzDw3M9dn5vqJkWVdtCqpBQaSKePjKwq1L6lhBpMpLCnUvqSGGczYJ5YWal/DqJtBfO7j8QeB52Tmg4GPADO3uv3t7voj4G+BhwGXRUTfvp8vqVHMFEklmSmSSjJT1DjdDOKPiohHVffPAL5W3b81IlYCz5mvQESMAEdm5iXAG4E1wMouepHUfmaKpJLMFEklmSlqnG4+AdoCvDAi/g74MfAh4B7AFcA1wHdq1BgFPhERq4EAzsnM27voRVL7mSmSSjJTJJVkpqhxuhnET2fm7JMxvLW67SUzHz/r8dtnPDypi2VLGj5miqSSzBRJJZkpapxuDqeXJEmSJEkDsKA98Zl5DXDc4rQi6UBjpkgqyUyRVJKZoqZyT7wkSZIkSS3hIF6SJEmSpJZozfUJc/dupm65pfdCEb3X2OOXtxUpc9R3R4vUAfj8dRuL1HnyR9cVqSM1VWzfycjmn/RcZ/eOnQW66Yif/bxInVy2rEgdgKuf+L+L1Hky64rUkRotpwfdwV5yaqpMne07itQBOHR0RbFa0jDL6Wmmt23rvVDBsc/01u1lCm35aZk6wJ/f48dF6nyJhxap0y/uiZckSZIkqSUcxEuSJEmS1BIO4iVJkiRJagkH8ZIkSZIktYSDeEmSJEmSWsJBvCRJkiRJLVF7EB8RayNie0Rsqh6/JSK+HxGbI2JTRDyimr4hIq6qpm2KiE9X098eET+rpl0ZEU+vpr8uIq6LiA8swvpJaigzRVJJZoqkkswUNdlCrxN/dWaui4hHAU8DHpqZOyPiUGBixnxnZua+Llh+Tma+JyIeCHw1Ig7PzHMi4lfA+u5WQVKLmSmSSjJTJJVkpqiRFjqI3+NewK2ZuRMgM29dyIszc0tE7AYOBX6xv/ki4izgLIClLO+yVUkt0P9MiRXddyup6XyfIqkkM0WN0u134r8IHBkRP4qID0bE42Y9/8kZh5S8e/aLq8NPpoFb5lpIZp6bmeszc/04S7psVVIL9D1TJmJpue4lNY3vUySVZKaoUbraE5+Zd0XEw4CTgVOACyLizZl5XjXL/g4peV1E/ClwJ3B6ZmY3y5c0XMwUSSWZKZJKMlPUNN0eTk9mTgEbgA0RcQXwQuC8eV52Tma+p9tlShpeZoqkkswUSSWZKWqSrg6nj4hjIuLoGZPWAdcW6UjSAcdMkVSSmSKpJDNFTdPtnviVwPsjYg2wG/gJ1UkYKp+MiO3V/Vsz89TuW5R0ADBTJJVkpkgqyUxRo3T7nfjLgEfv57nH72f627tZlqThZ6ZIKslMkVSSmaKmWcjh9FPA6ojYVLKBiHgd8F+AO0rWldR4ZoqkkswUSSWZKWqs2nviM/N64MjSDWTmOcA5petKajYzRVJJZoqkkswUNVnXZ6fvuwhifKL3OiPRe43C7nz6umK1/mDLvYrUGeH6InWkxhoJYlnv14qPqakCzXSMrFpVptCuyTJ1gG/tKLd+0jCLiXHG7n3fnutM3/LLAt2UFStWFKv1P3559PwzSSImJhi7z1E915n62U0FuukYWdt7xgHcdOoRReoAHPv/PqxInd/lm0Xq9EtXZ6eXJEmSJEn95yBekiRJkqSWcBAvSZIkSVJLOIiXJEmSJKklHMRLkiRJktQSDuIlSZIkSWqJeQfxEbE2IrZHxKbq8VREbIqIKyPinyJieTV9LCJujYh3znr9hoi4KiI2R8QPI+IDEbGmem5ZVWsyIg4tv3qSmsZMkVSSmSKpJDNFbVB3T/zVmbmuur89M9dl5nHAJHB2Nf1JwFXAcyNi9sXYz8zM44HjgZ3ARQCZub2qe2P3qyCphcwUSSWZKZJKMlPUaL0eTv9V4P7V/TOA9wLXAY/c18yZOQm8ETgqIk6Yr3hEnBURGyNi467c0WOrklqgb5kyOW2mSAeA/mXK1PZCLUtqsD5myrZCLWsYdT2Ij4gx4CnAFRGxDHgicDHwKTob9T5l5hRwOXDsfMvIzHMzc31mrh+Ppd22KqkF+p0pEyNmijTM+p4po8vKNC6pkfqfKcvLNK6h1M0gfln1HZGNdD55+hjwNOCSzNwGXAicFhGjc9SYfciJpAOXmSKpJDNFUklmihpnrIvX7Pkux29ExBnAYyLimmrSIcApwJdmv7jawB8MbOli2ZKGj5kiqSQzRVJJZooap+dLzEXEKuAk4KjMXJuZa4FXsY/DSiJiHHgncH1mbu512ZKGj5kiqSQzRVJJZoqaoMR14p8FfDkzd86YdhHw9IhYUj3+ZERsBq4EVgDPKLBcScPJTJFUkpkiqSQzRQO34MPpM3PlrMfnAefNmnYbcFj18PHdtSbpQGCmSCrJTJFUkpmiJqqzJ34KWF2d0KGoiNhzoohxYLp0fUmNZKZIKslMkVSSmaLGm3dPfGZeDxy5GAvPzO3AusWoLamZzBRJJZkpkkoyU9QGJb4TL0mSJEmS+qCbS8y1WozOdQnHhcmpqSJ1di8t91nK7duXFalzcJEqUsNFwy7bOtKwfoBJymWmNNSmpsk77uq5TO7aXaCZSqFMKZlMxyz9eZE6X+b+RepIjRVBToz3Xma03DgjVywtUmd6rFyqLL2tWKlWcU+8JEmSJEkt4SBekiRJkqSWcBAvSZIkSVJLOIiXJEmSJKklHMRLkiRJktQS8w7iI2JtRGyPiE3V46mI2BQRV0bEP0XE8mr6WETcGhHvnPX6DRFxVURsjogfRsQHImJN9dyyqtZkRBxafvUkNY2ZIqkkM0VSSWaK2qDunvirM3NddX97Zq7LzOOASeDsavqTgKuA50b81nWbzszM44HjgZ3ARQCZub2qe2P3qyCphcwUSSWZKZJKMlPUaL0eTv9V+M2FOs8A3gtcBzxyXzNn5iTwRuCoiDihx2VLGj5miqSSzBRJJZkpaoSuB/ERMQY8BbgiIpYBTwQuBj5FZ6Pep8ycAi4Hjq2xjLMiYmNEbNyVO7ptVVIL9DtTJqfNFGmY9T1TfJ8iDbW+Z8rUtjKNayh1M4hfVn1HZCOdT54+BjwNuCQztwEXAqdFxOgcNWYfcrJPmXluZq7PzPXjsbSLViW1wEAyZWLETJGG1GAyxfcp0rAaTKaMLu+xbQ2zsS5es+e7HL8REWcAj4mIa6pJhwCnAF+a/eJqA38wsKWLZUsaPmaKpJLMFEklmSlqnJ4vMRcRq4CTgKMyc21mrgVexT4OK4mIceCdwPWZubnXZUsaPmaKpJLMFEklmSlqghLXiX8W8OXM3Dlj2kXA0yNiSfX4kxGxGbgSWAE8o8ByJQ0nM0VSSWaKpJLMFA3cgg+nz8yVsx6fB5w3a9ptwGHVw8d315qkA4GZIqkkM0VSSWaKmqjOnvgpYHV1QoeiImLPiSLGgenS9SU1kpkiqSQzRVJJZooab9498Zl5PXDkYiw8M7cD6xajtqRmMlMklWSmSCrJTFEblPhOvCRJkiRJ6oPIzEH3UEtE3AJcO89shwK3FlpkqVpNq1OyVr97ul9mHjbPPFItZspQr1vdWmaKijFThnrd6tYyU1SMmVK0VtPq1K2130xpzSC+jojYmJnrm1SraXWGvSeppCZu402rM+w9SSU1cRtvWp1h70kqqYnb+DD31KR183B6SZIkSZJawkG8JEmSJEktMWyD+HMbWKtpdUrWamJPUklN3MabVqdkrSb2JJXUxG28aXVK1mpiT1JJTdzGh7mnxqzbUH0nvski4q7MXDnj8YuA9Zn5ZwVqbwBen5kbZ01/AvAeYAK4DHhJZu7udXmS+m9AGfJnwGuB3wMOy8xbq+kBvBd4KrANeFFmfrfXPiT1T8My5Vjg74GHAm/JzPf02oOk/mtYrpwJvKma7S7gFZl5ea99NMWw7YlXJSJGgI8Dz8vM4+ic3fKFg+1KUst8HTiV3z477lOAo6vbWcCH+tyXpHbaX6bcBryGzo4HSVqI/eXKfwCPy8zjgf/KkB1N4yC+ASLisIi4MCK+U90eU01/eER8IyK+V/17TDV9WUScHxGbI+ICYNk+yh4C7MzMH1WP/xV4dl9WSFJfLVKGkJnfy8xr9vHUM4B/yI5vAWsi4l6LtHqS+qzfmZKZv8jM7wC7FnG1JA3QAHLlG5n5q+rht4D7Ls6aDcbYoBs4gCyLiE0zHh8MfLa6/17gnMz8WkQcBXwBeCDwQ+Cxmbk7Ik4F/hudgfgrgG2ZeXxEHA/s6zDWW4HxiFhfHXbyHODIxVgxSX3R7wyZy32A62c8vqGa9vMF1pE0OE3KFEnDoam58hLgX3p4feM4iO+f7Zm5bs+DPd8RqR6eCvx+52umAKyKiIOA1cDHI+JoIIHx6vnHAu8DyMzNEbF59sIyMyPiecA5EbEE+CLg9+Gl9uprhswj9jHNE6xI7dKkTJE0HBqXKxFxCp1B/EndvL6pHMQ3wwjwqMzcPnNiRLwfuCQzT4uItcCGGU/P+4Y5M78JnFzVehLwgFINS2qURcmQOdzA3kf23Be4sYd6kpql35kiafj1PVeqPfgfBZ6Smb/spVbT+J34Zvgi8JuzNkbEuuruauBn1f0XzZj/K8CZ1bzHAcfvq2hEHF79u4TO2Rk/XLBnSc2xKBkyh88CL4iORwK/zkwPpZeGR78zRdLw62uuVIfs/2/gP804R9jQcBDfDK8B1lcnbvgBcHY1/V3AOyPi68DojPk/BKysDit5I/Dt/dR9Q0RsATYD/5yZX16c9iUN2KJkSES8JiJuoLOnfXNEfLR66nPAT4GfAB8BXll6hSQNVF8zJSLuWU3/c+CtEXFDRKxalDWTNCj9fq/yNjon+v5gRGyKiI37en1beZ14SZIkSZJawj3xkiRJkiS1hIN4SZIkSZJawkG8JEmSJEkt4SBekiRJkqSWcBAvSZIkSVJLOIiXJEmSJKklHMRLkiRJktQS/x/1QtzRnbOyJAAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 1152x576 with 12 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"heads = attention11.detach().numpy()\n",
"\n",
"plot_attention_weights(sentence=tokens, translated_tokens=tokens, \n",
" attention_heads=heads)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}