{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "HpgOtduwsIty" }, "source": [ "BERT: Bidirectional Encoder Representations from Transformers\n", "=============================================================" ] }, { "cell_type": "markdown", "metadata": { "id": "S4TzvJfllalU" }, "source": [ "Introducción\n", "------------" ] }, { "cell_type": "markdown", "metadata": { "id": "7lSoQeF7lalV" }, "source": [ "word2vec es un modelo de tipo \"context-free\", lo que significa que cada palabra recibe un único vector que la representa. Esto implica que por ejemplo la palabra \"banco\" recibirá la misma representación en las oraciones *Los domingos no abre el banco* y *Estabamos tan cansados que nos sentamos en un banco*.\n", "\n", "BERT, sin embargo, es un modelo contextual, lo que significa que la representación que se genera de una palabra depende del contexto en el que aparece." ] }, { "cell_type": "markdown", "metadata": { "id": "pwKjjVLXsIuO" }, "source": [ "### Como funciona" ] }, { "cell_type": "markdown", "metadata": { "id": "Z3zWLYU0sIuP" }, "source": [ "Si recordamos de cuando introducimos word2vec, vimos que las representaciones de las palabras se obtenian al entrenar una red neuronal en una tarea \"falsa\" que era predecir una palabra dado el contexto en el que aparece. Este contexto lo especificabamos como una ventana de palabras. Los modelos basados en lenguaje, llevan esta tarea un paso más adelante y tratan de predecir la siguiente palabra dada una secuencia de tokens.\n", "\n", "En el caso de BERT, está pre-entrenado utilizando 2 tareas distintas:\n", "\n", " - **Masked LM:** BERT está basado en una técnica llamada Masked LM (MLM) la cual, en lugar de intentar predecir la siguiente palabra dada una secuencia de palabras, aleatoriamente enmascara palabras en la oración para luego intentar predecirlas desde el contexto. Para hacerlo utiliza el contexto completo de la oración, tanto hacia adelante como hacía atras (por esto se llama bidireccional). En practica, BERT enmascara aproximadamente el 15% de los tokens en una secuencia.\n", " - **NSP (Next Sentence Prediction):** Muchas tareas en NLP requieren el entendimiento de las relaciones entre varias oraciones o secuencias. BERT captura estas relaciones al estar entrenado para predecir la siguiente oración en el cuerpo. En realidad BERT utiliza 50% del tiempo efectivamente la siguiente oración para la tarea de NSP y la taguea con el token IsNext, mientras que el otro 50% utiliza una oración aleatoria del texto y la taguea con el token NoNext." ] }, { "cell_type": "markdown", "metadata": { "id": "Dcyc_TQ6dis7" }, "source": [ "### Para ejecutar este notebook" ] }, { "cell_type": "markdown", "metadata": { "id": "qFxPoP-UlalY" }, "source": [ "Para ejecutar este notebook, instale las siguientes librerias:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "-jsVBbKvlalZ", "outputId": "ab042bd0-840b-4c72-851b-55a4d5ede782", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\u001b[K |████████████████████████████████| 3.1 MB 2.1 MB/s \n", "\u001b[K |████████████████████████████████| 831.4 MB 2.5 kB/s \n", "\u001b[K |████████████████████████████████| 163 kB 50.5 MB/s \n", "\u001b[K |████████████████████████████████| 880 kB 57.5 MB/s \n", "\u001b[K |████████████████████████████████| 3.3 MB 46.6 MB/s \n", "\u001b[?25h Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n", "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", "torchvision 0.13.1+cu113 requires torch==1.12.1, but you have torch 1.9.0 which is incompatible.\n", "torchtext 0.13.1 requires torch==1.12.1, but you have torch 1.9.0 which is incompatible.\n", "torchaudio 0.12.1+cu113 requires torch==1.12.1, but you have torch 1.9.0 which is incompatible.\u001b[0m\n" ] } ], "source": [ "!wget https://raw.githubusercontent.com/santiagxf/M72109/master/NLP/Datasets/mascorpus/tweets_marketing.csv \\\n", " --quiet --no-clobber --directory-prefix ./Datasets/mascorpus/\n", " \n", "!wget https://raw.githubusercontent.com/santiagxf/M72109/master/docs/nlp/neural/BERT.txt \\\n", " --quiet --no-clobber\n", "!pip install -r BERT.txt --quiet" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "Ntcs1AlpfckX" }, "outputs": [], "source": [ "import warnings\n", "warnings.filterwarnings('ignore')" ] }, { "cell_type": "markdown", "metadata": { "id": "CC7vGpjqditL" }, "source": [ "Cargamos el set de datos" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "pmJpenUkditM" }, "outputs": [], "source": [ "import pandas as pd\n", "\n", "tweets = pd.read_csv('Datasets/mascorpus/tweets_marketing.csv')" ] }, { "cell_type": "markdown", "metadata": { "id": "ailv1y8usIuQ" }, "source": [ "## Explorando un modelo pre-entrenado con BERT" ] }, { "cell_type": "markdown", "metadata": { "id": "mAWVIpOasIuR" }, "source": [ "Una de las formas más sencillas de trabajar con el modelo BERT es utilizando la libreria transformers de [HuggingFace](https://huggingface.co/) la cual ofrece una forma muy conveniente de acceder a modelos de NLP en diferentes lenguajes e incluso entrenados para tareas especificas." ] }, { "cell_type": "markdown", "metadata": { "id": "-G9VhuAosIuS" }, "source": [ "Podemos instalar esta libreria desde pip de la siguiente forma. Este paso ya lo realizamos en la sección de instalación de este notebook\n", "\n", "```\n", "pip install transformers\n", "```\n", "\n", "> Nota: Esta librería ya fué instalada en la preparación del notebook." ] }, { "cell_type": "markdown", "metadata": { "id": "XxeO0eUSsIuT" }, "source": [ "### BETO: BERT en español" ] }, { "cell_type": "markdown", "metadata": { "id": "DdLkgbBPsIuU" }, "source": [ "Al igual que con word2vec, entrenar un modelo de lenguaje requiere de una gran cantidad de datos sumado a un poder de computo interesante (cuando BERT fué publicado en 2018, tomó 4 días entrenar el modelo usando 16 TPUs. Si se hubiera entrenado en 8 GPUs hubiera tomado entre 40–70 días).Por este motivo, utilizaremos un modelo pre-entrenado para un cuerpo de texto en español. Este modelo, BETO, fué entrenado sobre un gran corpora de texto. Pueden encontrar más información sobre el autor de este modelo en: https://github.com/dccuchile/beto " ] }, { "cell_type": "markdown", "metadata": { "id": "hTDeAPaysIuV" }, "source": [ "### Tokenizers " ] }, { "cell_type": "markdown", "metadata": { "id": "fTn2Z7gssIuW" }, "source": [ "BERT utiliza su propio tokenizer que está basado en WordPiece. Este tokenizer tiene un vocabulario de 30.000 tokens donde cada secuencia comienza con un token especial [CLS]. Exploremos como funciona este tokenizer" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 177, "referenced_widgets": [ "14a6b8e497714459a7b6d2323109e1b2", "dd94c421ebbc4020971fc158ec4cd89d", "b1d93194510c473cb367a9ebba1dfb1f", "f558851095354e8682ef6a9f935c6ce5", "57651078af7747b29fc9ae06840ea3b4", "5a1a8837b4ce4be6a168c8cf3389ec6e", "beda77da37184d16929e79dc2df49bbe", "53c467a59f274f7290938e59cb3eacf8", "dcecf886942743018a777c1e321ccd82", "12ee09892fe241b685c95712657bb356", "7bbc9bbb73e44e17a5f48b119cae7dab", "0c0226a748764079abedc128deb20d50", "29b56c3525864524bce71a1e33292aaf", "8e65c860b5774a588f534f4abf36590b", "e930b59988e14b31a76a37bd612cb92a", "e4c8ae51fefe428691042e2e7f50da20", "e369996330ea4c17bdd9357a221f7ccf", "cc852bf267644d41bf206d547ca3275b", "9032875bde024e92b73c364c7ca16d57", "43e022acb95b4bf7b173d1f9673f32a5", "46f1fde039c1432497c0e1677678791a", "c070a8c91c474074a7dc9a3738a0bc25", "37b8d480daa343d2b8cdeb9b6977e260", "dca18edaf4284c46bf77a98154f3e457", "c3c85511d3f44fc4b700bdd99950c1e4", "b0330f1bf0f34111a17c3bb0bf1ab03d", "d082f93b52d14343a3dce2bb54cf6fe5", "bc8cc36ba82e42a8bc0eee34c5fe7c27", "39037bdd86ce4a6484afae2086ce4d6c", "91999d8d6077459bb7afb35d6f439027", "a67dea09f38b4f80a7d031909bce9e39", "1a096ab884c44d498f55fa2ee0e90227", "5998e66c6f104689b878d09e6e4fe135", "4acc06f995c04f3ca5831bb7215fb4f2", "1b6b9e54f1654962956afa2827170c4b", "5e977569c71e43d48fe496a2881a5a12", "b44cdfebb4f34d72b321996a6345d071", "f574d11c30974b4fbda3087400250b30", "7a85c4e3b443405e849275c28e0d638a", "bdd2268ead98423f90d2f713b3284c9a", "f738e0bacc0042349157250693c501af", "a29b1968ac194294b120772772641d3b", "b00bc95035c94eb8b1cb34c64ccba2f0", "19dd19da874945219efb93f087847c53", "0ddddbf858e84538975483814330348c", "ce9f837c0e9c400c85bfa5932ac18d12", "dc992a44b6e3445eb39c0188a4b48654", "fd926dd9ed9348cea553a79bfa8a34a8", "bd671210aacb4c8f8044b338a60d8bc5", "d31a0c0dc34c4e1ca9f9630ab3e050bc", "96ab179da94949eaa2bb701e85e50510", "a974bb63d755406a963e8793d249eb54", "b0532d26be0f4ffa896984fd5997599e", "9790af08057c4990993dbdd9852d62fd", "4c20c621a9124d678511944ae80e2629" ] }, "id": "yxWFx0HosIuY", "outputId": "6e9e1d82-10f0-4e55-9432-1b5d6aeabb67" }, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "Downloading: 0%| | 0.00/310 [00:00 *Importante: `AutoTokenizer` automáticamente detecta el tipo de tokenizer que el modelo require para poder ejecutarse. En este caso, es un `BertTokenizer`. Puede inspeccionar el tipo de dato que es `tokenizer` para verificarlo. Noten que el tokenizer depende del modelo que estamos utilizando.*" ] }, { "cell_type": "markdown", "metadata": { "id": "afpvu4chsIuf" }, "source": [ "Exploremos los tokens que genera:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "F9iDb340sIug" }, "outputs": [], "source": [ "text = tweets['TEXTO'][5]" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "id": "HmN6b_tJsIun", "outputId": "2da30539-c574-4747-8488-fff3d12ac67d" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'. @PoliciadeBurgos @PCivilBurgos @Aytoburgos Mismo peligro c/ Rio Viejo junto Mercadona Villimar'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 6 } ], "source": [ "text" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "NLG_OhQDsIup" }, "outputs": [], "source": [ "tokens = tokenizer.encode(text)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WGPYvhKAsIut", "outputId": "4f444889-b3d6-430c-c18a-1dc48b53d950" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[4,\n", " 1008,\n", " 985,\n", " 14666,\n", " 1114,\n", " 5232,\n", " 30958,\n", " 985,\n", " 9419,\n", " 1211,\n", " 1123,\n", " 5232,\n", " 30958,\n", " 985,\n", " 1457,\n", " 1049,\n", " 5232,\n", " 30958,\n", " 1665,\n", " 4615,\n", " 1013,\n", " 989,\n", " 10552,\n", " 3379,\n", " 2689,\n", " 2915,\n", " 1316,\n", " 17400,\n", " 24981,\n", " 5]" ] }, "metadata": {}, "execution_count": 8 } ], "source": [ "tokens" ] }, { "cell_type": "markdown", "metadata": { "id": "wH2F4qrHsIuz" }, "source": [ "¿Notan algo raro en los tokens generados?" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "id": "OuQwdQk5sIu1", "outputId": "5e15a1a9-462c-4fd2-fa70-12fdd662e699" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'[CLS]. @ policiadeburgos @ pcivilburgos @ aytoburgos mismo peligro c / rio viejo junto mercadona villimar [SEP]'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 9 } ], "source": [ "tokenizer.decode(tokens)" ] }, { "cell_type": "markdown", "metadata": { "id": "QQwFFhsXsIu3" }, "source": [ "¿Siguen notando algo raro?" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-TruYQtvsIu3", "outputId": "4bc6ff40-846e-434e-9247-47a0cbf87d57" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['[CLS]',\n", " '.',\n", " '@',\n", " 'policia',\n", " '##de',\n", " '##burgo',\n", " '##s',\n", " '@',\n", " 'pc',\n", " '##iv',\n", " '##il',\n", " '##burgo',\n", " '##s',\n", " '@',\n", " 'ay',\n", " '##to',\n", " '##burgo',\n", " '##s',\n", " 'mismo',\n", " 'peligro',\n", " 'c',\n", " '/',\n", " 'rio',\n", " 'viejo',\n", " 'junto',\n", " 'mercado',\n", " '##na',\n", " 'vill',\n", " '##imar',\n", " '[SEP]']" ] }, "metadata": {}, "execution_count": 10 } ], "source": [ "[tokenizer.convert_ids_to_tokens(idx) for idx in tokens]" ] }, { "cell_type": "markdown", "metadata": { "id": "i8W01ColsIu6" }, "source": [ "### Cargando nuestro modelo de BERT para español" ] }, { "cell_type": "markdown", "metadata": { "id": "xifxnZc5sIu6" }, "source": [ "Para cargar nuestro modelo desde el repositorio de modelos de HuggingFace basta con utilizar el método `from_pretrained` de igual manera que hicimos con el tokenizer. Este método descargará automaticamente el modelo desde el directorio de modelos de HuggingFace. Pueden ver el listado de modelos que están disponibles en este directorio en [https://huggingface.co/models]. Sin embargo, es necesario que indiquemos la tarea que necesitamos resolver para que la libraría pueda generar el objeto adecuado en Python.\n", "\n", "La libraría `transformers` dispone de varias tareas:\n", "\n", "- Causal Language Model\n", "- Masked Language\n", "- Multiple Choice\n", "- Next sentence predicción\n", "- Question answering\n", "- Seq2Seq Language Model\n", "- Sequence Classification\n", "- Sequence Regression\n", "- Token Classification\n", "\n", "En nuestro caso, comenzaremos explorando el poder predictivo del modelo y por lo tanto utilizaremos un **Masked Language Model**, la cual es una de las tareas que BERT fué diseñado especificamente para resolver." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 49, "referenced_widgets": [ "467c0b0c181744afad5e791d864cfb55", "dce4266d563d4077992c7eff3c4c8dee", "35a67f24b8ab4506adf5b90a6ba7e7ef", "ac1bb2ef20764619b21cd5c13028f594", "c29be7b752ee4d1e91f680d5a7e393ce", "78eb7328b622450eade7fae3c34f1aea", "da7320102b1d4f049a1d79878cff1235", "63a3527d70ff4e6b9876821574197707", "22e6d7d9b7fe4b2d825db92b961941ac", "ad53740b5ee14d10b6748c9ebad1c121", "c0699be79d924c6586e181b8f98e8e1b" ] }, "id": "QUKFFSmAsIu7", "outputId": "72abe21b-d66e-4c3c-920b-fa8eb0980f0b" }, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "Downloading: 0%| | 0.00/419M [00:00 Notas: `AutoModelForMaskedLM` es una utilidad de HuggingFace que permite cargar un modelo de lenguage para la tarea de enmascaramiento de lenguage. Esta clase automaticamente detectará el tipo de modelo que estamos utilizando, en nuestro caso un modelo basado en la arquitectura BERT. El parámetro `return_dict=True` hará que el modelo retorne un diccionario con los resultados en lugar de un tupla. Esto solo hará que sea más sencillo interpretar los resultados cuando trabajemos más adelante.." ] }, { "cell_type": "markdown", "metadata": { "id": "B-gb7c9KsIu-" }, "source": [ "Veamos como se comporta nuestro modelo en la tarea de predecir una palabra de un texto:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "P8Vgq8GesIvB" }, "outputs": [], "source": [ "text = \"[CLS] Cuando [MASK] contaron lo que sucedia nos quedamos helados. [SEP]\"\n", "tokens = tokenizer.tokenize(text)" ] }, { "cell_type": "markdown", "metadata": { "id": "4MOk8vhpsIvD" }, "source": [ "Noten que el token [MASK] es la palabra que estamos intentando predecir" ] }, { "cell_type": "markdown", "metadata": { "id": "ZuxmDIWbsIvD" }, "source": [ "Necesitamos saber cual de todos los tokens que generamos es exactamente el que enmascaramos:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "id": "4vXP0940sIvE" }, "outputs": [], "source": [ "masked_indxs = [idx for idx in range(0, len(tokens)) if tokens[idx] == '[MASK]']" ] }, { "cell_type": "markdown", "metadata": { "id": "jW9CXBpssIvG" }, "source": [ "Corremos nuestro modelo:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "id": "xlBcKmvgsIvG" }, "outputs": [], "source": [ "import torch\n", "\n", "indexed_tokens = tokenizer.convert_tokens_to_ids(tokens)\n", "tokens_tensor = torch.tensor([indexed_tokens])\n", "\n", "predictions = model(tokens_tensor).logits" ] }, { "cell_type": "markdown", "metadata": { "id": "iRcwI7OHlalk" }, "source": [ "**Nota:** Ejecutar el modelo sobre nuestro texto devolverá diferente cantidad de objetos dependiendo de como se configuró la carga del modelo en el método `from_pretrained`. En nuestro caso, hemos especificado `return_dic=True` y por lo tanto la salida del médoto es un diccionario con los resultados (si no lo hubieramos especificado hubiera retornado una tupla). Dentro de lo que nos interesa a nosotros está:\n", " - **logits:** Retorna la secuencia de hidden-states en la última capa del modelo. Esto tiene tamaño (batch_size, sequence_length, hidden_size).\n", " - **hidden_states:** Retorna los hidden-states de todas las capas del modelo. 12 en el caso de BERT. Esta información solo se retorna cuando se indica `output_hidden_states=True`\n", "\n", "Para más información sobre que objetos se retornan de la ejecución del modelo pueden ver la documentación del modelo BERT: https://huggingface.co/transformers/model_doc/bert.html#bertmodel" ] }, { "cell_type": "markdown", "metadata": { "id": "RaHu-uY5sIvI" }, "source": [ "Verificamos cuales son las palabras más probables:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "P2q1fyi2sIvI", "outputId": "7835c0b8-39d1-4689-bb19-b6a5d5476b40" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Las 5 palabras más probables para la mascara 0 son: ['nos', 'me', 'les', 'le', 'supi']\n" ] } ], "source": [ "for i,midx in enumerate(masked_indxs):\n", " idxs = torch.argsort(predictions[0,midx], descending=True)\n", " predicted_token = tokenizer.convert_ids_to_tokens(idxs[:5])\n", " print('Las 5 palabras más probables para la mascara',i,'son:',predicted_token)" ] }, { "cell_type": "markdown", "source": [ "### Explorando las representaciones de BERT" ], "metadata": { "id": "5glLiHMUpO3Y" } }, { "cell_type": "markdown", "metadata": { "id": "JQp9yqFjlall" }, "source": [ "En esta sección exploraremos las representaciones que genera BERT. Como se mencionó, BERT generá representaciones que son dependientes del contexto, algo que lo diferencia de Word2Vec. Esto implica que por ejemplo la palabra \"banco\" recibirá la misma representación en las oraciones *Los domingos no abre el banco y salimos a caminar. Estabamos tan cansados que nos sentamos en un banco a ver gente pasar*. Veamos si esto es así como mencionamos explorando estas representaciones" ] }, { "cell_type": "markdown", "metadata": { "id": "83NopPytlall" }, "source": [ "Para hacer esto, volveremos a cargar el modelo, ahora especificando el parametro `output_hidden_states=True`. Esto hará que el modelo retorne las representaciones de todos los estados intermedios como parte de la salida:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "id": "ASmbpVXglall" }, "outputs": [], "source": [ "model = AutoModelForMaskedLM.from_pretrained('dccuchile/bert-base-spanish-wwm-uncased', \n", " return_dict=True, \n", " output_hidden_states=True)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "id": "u853Px1Tlall" }, "outputs": [], "source": [ "text = \"Queriamos retirar dinero del banco. Sin embargo, los domingos el banco no está abierto. Estabamos tan cansados que nos sentamos en un banco a ver gente pasar\"" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "id": "QB8P0qfXlalm" }, "outputs": [], "source": [ "indexed_tokens = tokenizer.encode(text)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "id": "7xwMzYXwlalm" }, "outputs": [], "source": [ "tokens_tensor = torch.tensor([indexed_tokens]) #Al igual que antes, siempre convertimos primero el input en un tensor\n", "\n", "hidden_states = model(tokens_tensor).hidden_states\n", "token_embeddings = torch.stack(hidden_states, dim=0) #hidden_states retorna una lista, transformemos esto en un tensor :)" ] }, { "cell_type": "markdown", "metadata": { "id": "uHatu_9Blalm" }, "source": [ "Veamos las dimensiones de este objeto:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "id": "RYitW_k6lalm", "outputId": "5bb55524-abac-4b74-fa76-116f14da51ea", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "torch.Size([13, 1, 33, 768])" ] }, "metadata": {}, "execution_count": 24 } ], "source": [ "token_embeddings.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "GaOWjc-Alaln" }, "source": [ "¿Que significan?\n", " - 13 es la cantidad de capas dentro del modelo. Si bien BERT posee 12 capas, la capa numero 0 corresponde a los inputs (entradas) del modelo y por eso vemos 13 capas finalmente\n", " - 1 es la cantidad de muestra en el lote (ie. batch size)\n", " - 33 es la cantidad de tokens que se inputaron al modelo\n", " - 768 es la cantidad de unidades de la red neuronal (units) en cada capa" ] }, { "cell_type": "markdown", "metadata": { "id": "gKnUKHBslaln" }, "source": [ "Como nuestro lote/batch solo contiene una oración, podemos deshacernos de la segunda dimensión de este tensor:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "id": "lub15G7Slaln", "outputId": "b5ef34a8-bd35-48eb-d241-0f7f34d6c69c", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "torch.Size([13, 33, 768])" ] }, "metadata": {}, "execution_count": 25 } ], "source": [ "token_embeddings = torch.squeeze(token_embeddings, dim=1)\n", "token_embeddings.size()" ] }, { "cell_type": "markdown", "metadata": { "id": "8jRSP8sVlaln" }, "source": [ "Finalmente, para facilitar el entendimiento de la salida, vamos a cambiar el orden de los valores de este tensor. Recordaran que actualmente tenemos un tensor del tamaño (numero_de_capas, numero_de_tokens, features). Para el análisis que queremos realizar, sería mucho más interesante tener algo del tipo (numero_de_tokens, numero_de_capas, features). De esta forma podriamos revisar todas las representaciones de cada token más facilmente:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "id": "G1mNeVGElaln", "outputId": "e3eab68a-7898-43eb-9c0a-a3c35b3394c6", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "torch.Size([33, 13, 768])" ] }, "metadata": {}, "execution_count": 26 } ], "source": [ "# Cambiamos las dimensiones 0 y 1\n", "token_embeddings = token_embeddings.permute(1,0,2)\n", "token_embeddings.size()" ] }, { "cell_type": "markdown", "metadata": { "id": "e5o9_j41lalo" }, "source": [ "#### ¿Entonces cuales son las representaciones?" ] }, { "cell_type": "markdown", "metadata": { "id": "xfvMuQfXlalo" }, "source": [ "Las representaciones son tensores del tamaño (12,768). Claramente esto es un gran progreso desde las 100 dimensiones en Word2Vec! Veamos si efectivamente las representaciones de banco son diferentes:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "id": "XrjsphP8lalo", "outputId": "a2ee1456-27d9-4c4e-bb80-ff1baabc6663", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "0 [CLS]\n", "1 queria\n", "2 ##mos\n", "3 retirar\n", "4 dinero\n", "5 del\n", "6 banco\n", "7 .\n", "8 sin\n", "9 embargo\n", "10 ,\n", "11 los\n", "12 domingos\n", "13 el\n", "14 banco\n", "15 no\n", "16 está\n", "17 abierto\n", "18 .\n", "19 estabamos\n", "20 tan\n", "21 cansados\n", "22 que\n", "23 nos\n", "24 sentamos\n", "25 en\n", "26 un\n", "27 banco\n", "28 a\n", "29 ver\n", "30 gente\n", "31 pasar\n", "32 [SEP]\n" ] } ], "source": [ "for i, token_str in enumerate(indexed_tokens):\n", " print (i, tokenizer.convert_ids_to_tokens(token_str))" ] }, { "cell_type": "markdown", "metadata": { "id": "jmSafdaslalo" }, "source": [ "En este ejemplo tenemos la palabra **banco** en las posiciones 6, 14 y 27. De estas posiciones, la palabra **banco** en las posiciones 6 y 14 debería de tener un significado similar, mientra que su versión en la posición 27 debería ser distinta. Para medir esto necesitariamos comparar la similaridad de estas representaciones. Para esto podríamos utilizar una métrica como la Similaridad de coseno (cosine similarity), sin embargo necesitamos contar con un vector unidimensional. Si bien no hay una única forma de resolver este problema, una forma podría ser tomar el promedio de los valores a lo largo de las 13 capas para generar un vector unidimensional por cada token:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "id": "BU7-3SdYlalo" }, "outputs": [], "source": [ "sentence_embedding = torch.mean(token_embeddings, dim=1)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "id": "bQEENuAUlalo", "outputId": "af5e4e71-1628-440d-cf97-83447dc43b77", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "torch.Size([33, 768])" ] }, "metadata": {}, "execution_count": 29 } ], "source": [ "sentence_embedding.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "Qq91_h9zlalp" }, "source": [ "Computemos ahora las similaridad de las palabras:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "id": "iMxu6U64lalp" }, "outputs": [], "source": [ "from scipy.spatial.distance import cosine" ] }, { "cell_type": "markdown", "metadata": { "id": "WO3O6nHwlalp" }, "source": [ "*(...) retirar dinero del **banco**. Sin embargo, los domingos el **banco** no está abierto (...)*" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "id": "mdgIQW5Dlalp", "outputId": "59bc296d-dc71-4248-e073-c20de6e2f78d", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "0.8966357111930847" ] }, "metadata": {}, "execution_count": 31 } ], "source": [ "1 - cosine(sentence_embedding[6].detach().numpy(), sentence_embedding[14].detach().numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "ywKzwUJFlalq" }, "source": [ "*(...) retirar dinero del **banco** (...) nos sentamos en un **banco** a (...)*" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "id": "e59yx93Mlalq", "outputId": "0d1a1d18-5e73-44a6-fc1e-6f0d71af3728", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "0.7544149160385132" ] }, "metadata": {}, "execution_count": 32 } ], "source": [ "1 - cosine(sentence_embedding[6].detach().numpy(), sentence_embedding[27].detach().numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "zGMCra6_lalr" }, "source": [ "Podemos ver que la similaridad entre cada una de las representaciones es distinta. Tengan en cuenta que este método no es exacto, pero de alguna forma nos da una idea y una intuición de que tan cercanas pueden ser estas representaciones. BERT es un modelo donde las representaciones dependen del contexto y por lo tanto el concepto de \"similaridad\" aquí es distinto e incluso podría carecer un poco de sentido." ] }, { "cell_type": "markdown", "metadata": { "id": "KuehBDwvlalr" }, "source": [ "#### Otra formar de ver las representaciones (de los autores de BERT)" ] }, { "cell_type": "markdown", "metadata": { "id": "QYCAwayelalr" }, "source": [ "Los autores de BERT proponen una solución un poco distinta, en base a diferentes experimentos que realizaron. Una de las formas que generaron las representaciones con los mejores resultados resultó de concatenar los valores de las últimas 4 capas." ] }, { "cell_type": "markdown", "metadata": { "id": "VKncGN6blalr" }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "id": "t57PUqQGlals" }, "source": [ "Algo interesante que decanta de este ultimo experimento es que claramente cada uno de las diferentes capas dentro de BERT codifican diferentes aspectos de las palabras y que por lo tanto, la estrategia que mejor se adapta para generar estas representaciones más compactas depende mucho de la tarea que se esté realizando. A continuación intentaremos aplicarlo (noten que en las lineas subsiguientes habrá bastante manipulación de las formas de los vectores)" ] }, { "cell_type": "markdown", "metadata": { "id": "V55jnodulals" }, "source": [ "Nos quedamos con las ultimas capas de cada token:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "id": "fpmOTGXXlals" }, "outputs": [], "source": [ "slice_ = token_embeddings.narrow(1, 9, 4)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "id": "8O_hbLRLlals", "outputId": "ff3d558f-9b95-4306-f9db-926634f4e090", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "torch.Size([33, 4, 768])" ] }, "metadata": {}, "execution_count": 34 } ], "source": [ "slice_.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "PfbHxj4klalt" }, "source": [ "Concatenamos los valores de las últimas 4 capas:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "id": "4fPgNMOvlalt" }, "outputs": [], "source": [ "concatenated_tensor = slice_.reshape(33, 4*768)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "id": "T30pdLazlalt", "outputId": "27aeb31a-7a1e-47ce-f86c-53ae539cccf9", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "torch.Size([33, 3072])" ] }, "metadata": {}, "execution_count": 36 } ], "source": [ "concatenated_tensor.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "W_1RzlBblalt" }, "source": [ "Calculamos la similaridad:" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "id": "d_lGUgmhlalu", "outputId": "c4d1ff1b-6918-44c8-dbf8-c96964ba8ee5", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "0.8337029814720154" ] }, "metadata": {}, "execution_count": 37 } ], "source": [ "1 - cosine(concatenated_tensor[6].detach().numpy(), concatenated_tensor[14].detach().numpy())" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "id": "l4wBDTUhlalu", "outputId": "2324d9d7-202b-49e4-b3af-19ff21e15f9c", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "0.6618704795837402" ] }, "metadata": {}, "execution_count": 38 } ], "source": [ "1 - cosine(concatenated_tensor[6].detach().numpy(), concatenated_tensor[27].detach().numpy())" ] } ], "metadata": { "colab": { "name": "BERT.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.11" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "14a6b8e497714459a7b6d2323109e1b2": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_dd94c421ebbc4020971fc158ec4cd89d", "IPY_MODEL_b1d93194510c473cb367a9ebba1dfb1f", "IPY_MODEL_f558851095354e8682ef6a9f935c6ce5" ], "layout": "IPY_MODEL_57651078af7747b29fc9ae06840ea3b4" } }, "dd94c421ebbc4020971fc158ec4cd89d": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_5a1a8837b4ce4be6a168c8cf3389ec6e", "placeholder": "​", "style": "IPY_MODEL_beda77da37184d16929e79dc2df49bbe", "value": "Downloading: 100%" } }, "b1d93194510c473cb367a9ebba1dfb1f": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_53c467a59f274f7290938e59cb3eacf8", "max": 310, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_dcecf886942743018a777c1e321ccd82", "value": 310 } }, "f558851095354e8682ef6a9f935c6ce5": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_12ee09892fe241b685c95712657bb356", "placeholder": "​", "style": "IPY_MODEL_7bbc9bbb73e44e17a5f48b119cae7dab", "value": " 310/310 [00:00<00:00, 6.18kB/s]" } }, "57651078af7747b29fc9ae06840ea3b4": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "5a1a8837b4ce4be6a168c8cf3389ec6e": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "beda77da37184d16929e79dc2df49bbe": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "53c467a59f274f7290938e59cb3eacf8": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "dcecf886942743018a777c1e321ccd82": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "12ee09892fe241b685c95712657bb356": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "7bbc9bbb73e44e17a5f48b119cae7dab": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "0c0226a748764079abedc128deb20d50": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_29b56c3525864524bce71a1e33292aaf", "IPY_MODEL_8e65c860b5774a588f534f4abf36590b", "IPY_MODEL_e930b59988e14b31a76a37bd612cb92a" ], "layout": "IPY_MODEL_e4c8ae51fefe428691042e2e7f50da20" } }, "29b56c3525864524bce71a1e33292aaf": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_e369996330ea4c17bdd9357a221f7ccf", "placeholder": "​", "style": "IPY_MODEL_cc852bf267644d41bf206d547ca3275b", "value": "Downloading: 100%" } }, "8e65c860b5774a588f534f4abf36590b": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_9032875bde024e92b73c364c7ca16d57", "max": 650, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_43e022acb95b4bf7b173d1f9673f32a5", "value": 650 } }, "e930b59988e14b31a76a37bd612cb92a": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_46f1fde039c1432497c0e1677678791a", "placeholder": "​", "style": "IPY_MODEL_c070a8c91c474074a7dc9a3738a0bc25", "value": " 650/650 [00:00<00:00, 17.3kB/s]" } }, "e4c8ae51fefe428691042e2e7f50da20": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "e369996330ea4c17bdd9357a221f7ccf": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "cc852bf267644d41bf206d547ca3275b": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "9032875bde024e92b73c364c7ca16d57": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "43e022acb95b4bf7b173d1f9673f32a5": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "46f1fde039c1432497c0e1677678791a": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "c070a8c91c474074a7dc9a3738a0bc25": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "37b8d480daa343d2b8cdeb9b6977e260": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_dca18edaf4284c46bf77a98154f3e457", "IPY_MODEL_c3c85511d3f44fc4b700bdd99950c1e4", "IPY_MODEL_b0330f1bf0f34111a17c3bb0bf1ab03d" ], "layout": "IPY_MODEL_d082f93b52d14343a3dce2bb54cf6fe5" } }, "dca18edaf4284c46bf77a98154f3e457": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_bc8cc36ba82e42a8bc0eee34c5fe7c27", "placeholder": "​", "style": "IPY_MODEL_39037bdd86ce4a6484afae2086ce4d6c", "value": "Downloading: 100%" } }, "c3c85511d3f44fc4b700bdd99950c1e4": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_91999d8d6077459bb7afb35d6f439027", "max": 247723, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_a67dea09f38b4f80a7d031909bce9e39", "value": 247723 } }, "b0330f1bf0f34111a17c3bb0bf1ab03d": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_1a096ab884c44d498f55fa2ee0e90227", "placeholder": "​", "style": "IPY_MODEL_5998e66c6f104689b878d09e6e4fe135", "value": " 242k/242k [00:00<00:00, 325kB/s]" } }, "d082f93b52d14343a3dce2bb54cf6fe5": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "bc8cc36ba82e42a8bc0eee34c5fe7c27": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "39037bdd86ce4a6484afae2086ce4d6c": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "91999d8d6077459bb7afb35d6f439027": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "a67dea09f38b4f80a7d031909bce9e39": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "1a096ab884c44d498f55fa2ee0e90227": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "5998e66c6f104689b878d09e6e4fe135": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "4acc06f995c04f3ca5831bb7215fb4f2": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_1b6b9e54f1654962956afa2827170c4b", "IPY_MODEL_5e977569c71e43d48fe496a2881a5a12", "IPY_MODEL_b44cdfebb4f34d72b321996a6345d071" ], "layout": "IPY_MODEL_f574d11c30974b4fbda3087400250b30" } }, "1b6b9e54f1654962956afa2827170c4b": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_7a85c4e3b443405e849275c28e0d638a", "placeholder": "​", "style": "IPY_MODEL_bdd2268ead98423f90d2f713b3284c9a", "value": "Downloading: 100%" } }, "5e977569c71e43d48fe496a2881a5a12": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_f738e0bacc0042349157250693c501af", "max": 486125, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_a29b1968ac194294b120772772641d3b", "value": 486125 } }, "b44cdfebb4f34d72b321996a6345d071": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b00bc95035c94eb8b1cb34c64ccba2f0", "placeholder": "​", "style": "IPY_MODEL_19dd19da874945219efb93f087847c53", "value": " 475k/475k [00:00<00:00, 638kB/s]" } }, "f574d11c30974b4fbda3087400250b30": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "7a85c4e3b443405e849275c28e0d638a": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "bdd2268ead98423f90d2f713b3284c9a": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "f738e0bacc0042349157250693c501af": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "a29b1968ac194294b120772772641d3b": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "b00bc95035c94eb8b1cb34c64ccba2f0": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "19dd19da874945219efb93f087847c53": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "0ddddbf858e84538975483814330348c": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_ce9f837c0e9c400c85bfa5932ac18d12", "IPY_MODEL_dc992a44b6e3445eb39c0188a4b48654", "IPY_MODEL_fd926dd9ed9348cea553a79bfa8a34a8" ], "layout": "IPY_MODEL_bd671210aacb4c8f8044b338a60d8bc5" } }, "ce9f837c0e9c400c85bfa5932ac18d12": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_d31a0c0dc34c4e1ca9f9630ab3e050bc", "placeholder": "​", "style": "IPY_MODEL_96ab179da94949eaa2bb701e85e50510", "value": "Downloading: 100%" } }, "dc992a44b6e3445eb39c0188a4b48654": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_a974bb63d755406a963e8793d249eb54", "max": 134, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_b0532d26be0f4ffa896984fd5997599e", "value": 134 } }, "fd926dd9ed9348cea553a79bfa8a34a8": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_9790af08057c4990993dbdd9852d62fd", "placeholder": "​", "style": "IPY_MODEL_4c20c621a9124d678511944ae80e2629", "value": " 134/134 [00:00<00:00, 3.69kB/s]" } }, "bd671210aacb4c8f8044b338a60d8bc5": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "d31a0c0dc34c4e1ca9f9630ab3e050bc": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "96ab179da94949eaa2bb701e85e50510": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "a974bb63d755406a963e8793d249eb54": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "b0532d26be0f4ffa896984fd5997599e": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "9790af08057c4990993dbdd9852d62fd": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "4c20c621a9124d678511944ae80e2629": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "467c0b0c181744afad5e791d864cfb55": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_dce4266d563d4077992c7eff3c4c8dee", "IPY_MODEL_35a67f24b8ab4506adf5b90a6ba7e7ef", "IPY_MODEL_ac1bb2ef20764619b21cd5c13028f594" ], "layout": "IPY_MODEL_c29be7b752ee4d1e91f680d5a7e393ce" } }, "dce4266d563d4077992c7eff3c4c8dee": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_78eb7328b622450eade7fae3c34f1aea", "placeholder": "​", "style": "IPY_MODEL_da7320102b1d4f049a1d79878cff1235", "value": "Downloading: 100%" } }, "35a67f24b8ab4506adf5b90a6ba7e7ef": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_63a3527d70ff4e6b9876821574197707", "max": 439621341, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_22e6d7d9b7fe4b2d825db92b961941ac", "value": 439621341 } }, "ac1bb2ef20764619b21cd5c13028f594": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_ad53740b5ee14d10b6748c9ebad1c121", "placeholder": "​", "style": "IPY_MODEL_c0699be79d924c6586e181b8f98e8e1b", "value": " 419M/419M [00:23<00:00, 19.8MB/s]" } }, "c29be7b752ee4d1e91f680d5a7e393ce": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "78eb7328b622450eade7fae3c34f1aea": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "da7320102b1d4f049a1d79878cff1235": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "63a3527d70ff4e6b9876821574197707": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "22e6d7d9b7fe4b2d825db92b961941ac": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "ad53740b5ee14d10b6748c9ebad1c121": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "c0699be79d924c6586e181b8f98e8e1b": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } } } } }, "nbformat": 4, "nbformat_minor": 0 }