{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IJUUYSe3-GFl"
      },
      "source": [
        "RAG con índice documental y un SLM\n",
        "==================================\n",
        "\n",
        "En el ejemplo anterior vimos el corazón de RAG: embeddings, similitud y contexto. Sin embargo, utilizamos documentos de texto sencillos y generamos los embeddings de forma manual almacenandonos en una lista en memoria. Tal técnica no puede escalar a grandes repositorios de documentos como los que podemos encontrar en una organización.\n",
        "\n",
        "En este ejemplo, vamos a dar un paso más hacia una arquitectura de trabajo más parecida a la que encontraríamos en una organización: documentos con metadatos, chunking con solapamiento, un índice vectorial y un modelo generativo que responde usando los fragmentos recuperados."
      ],
      "id": "IJUUYSe3-GFl"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ur7oyqPv-GFu"
      },
      "source": [
        "## Preparación del ambiente\n",
        "\n",
        "Instalemos las librerías necesarias desde el archivo de requerimientos asociado. En Google Colab puede ser necesario reiniciar la sesión después de la instalación si el entorno lo solicita."
      ],
      "id": "ur7oyqPv-GFu"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1yEudRNg-GFv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e6dddfc7-18a9-45d6-ac7a-2b87a748ea65"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[33mWARNING: huggingface-hub 1.16.1 does not provide the extra 'inference'\u001b[0m\u001b[33m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.9/11.9 MB\u001b[0m \u001b[31m95.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m95.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m76.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.2/121.2 kB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m51.0/51.0 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.4/142.4 kB\u001b[0m \u001b[31m19.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "ipython 7.34.0 requires jedi>=0.16, which is not installed.\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ],
      "source": [
        "!wget -q https://raw.githubusercontent.com/santiagxf/M72109/master/docs/document-understanding/rag-index-slm.txt\n",
        "%pip install -r rag-index-slm.txt --quiet"
      ],
      "id": "1yEudRNg-GFv"
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Sobre LlamaIndex\n",
        "\n",
        "LlamaIndex es un projecto de código abierto diseñado para conectar modelos de lenguaje (LLM) con tus propias fuentes de información. Permite a los desarrolladores crear aplicaciones avanzadas de IA que pueden buscar, resumir y razonar sobre documentos privados o empresariales, superando la limitación de conocimiento público de los modelos.\n",
        "\n",
        "En este notebook, utilizaremos esta librería para construir una solución de RAG facilmente. No es necesario utilizar esta librería, simplemente la utilizamos porque facilita la utilización de la técnica."
      ],
      "metadata": {
        "id": "CFrWnQ_UHFzV"
      },
      "id": "CFrWnQ_UHFzV"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z-Kk26rr-GFz"
      },
      "source": [
        "### Documentos de ejemplo\n",
        "\n",
        "En una organización, los documentos no suelen vivir como una lista de textos dentro del código. Normalmente llegan como archivos en una carpeta compartida, un data lake, un bucket de objetos o un repositorio documental. Para acercarnos a ese patrón, descargaremos algunos documentos en español al sistema de archivos y luego los cargaremos con un lector de documentos de LlamaIndex.\n",
        "\n",
        "Usaremos páginas públicas del curso como sustituto de documentos internos. La idea pedagógica es la misma: separar la ingesta documental del índice y conservar metadatos de procedencia."
      ],
      "id": "Z-Kk26rr-GFz"
    },
    {
      "cell_type": "code",
      "source": [
        "!mkdir repositorio_documental"
      ],
      "metadata": {
        "id": "vogUj47PHmUW"
      },
      "id": "vogUj47PHmUW",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LU_nyi6n-GF0"
      },
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "from urllib.request import urlretrieve\n",
        "\n",
        "repo_documental = Path(\"repositorio_documental\")\n",
        "repo_documental.mkdir(exist_ok=True)\n",
        "\n",
        "documentos_fuente = {\n",
        "    \"lg-user-manual-23476.pdf\": {\n",
        "        \"fuente\": \"lg-user-manual-23476.pdf\",\n",
        "        \"marca\": \"lg\",\n",
        "    },\n",
        "    \"samsumg-user-manual-15223.pdf\": {\n",
        "        \"fuente\": \"samsumg-user-manual-15223.pdf\",\n",
        "        \"marca\": \"samsung\",\n",
        "    }\n",
        "}"
      ],
      "id": "LU_nyi6n-GF0"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aUoTCJKt-GF3"
      },
      "source": [
        "## Chunking con metadatos\n",
        "\n",
        "En documentos largos, un único embedding puede mezclar muchas ideas. Por eso dividimos cada documento en chunks más pequeños y con solapamiento. El solapamiento ayuda a no cortar una explicación justo en el límite entre dos fragmentos.\n",
        "\n",
        "En este notebook usaremos una estrategia de chunking basada en oraciones. A grandes rasgos, intenta respetar límites naturales del texto, como oraciones o párrafos, y luego agrupa contenido hasta aproximarse al tamaño indicado. Sin embargo, vamos a repetir una pequeña porción del fragmento anterior en el siguiente para conservar contexto local de un fragmento al otro. Podemos pensar que esto evita que una definición, una advertencia o una instrucción quede separada de la frase que la explica.\n",
        "\n",
        "Existen otras estrategias:\n",
        "\n",
        "1. Una opción simple es cortar por cantidad fija de caracteres o tokens; es rápida y predecible, pero puede romper frases importantes.\n",
        "2. Podemos cortar por estructura del documento, por ejemplo títulos, secciones, páginas, tablas o encabezados; suele producir chunks más interpretables, aunque requiere que el documento tenga estructura confiable.\n",
        "3. También hay estrategias semánticas, donde se agrupan oraciones según similitud o cambios de tema; pueden mejorar la calidad de retrieval, pero agregan costo computacional y más parámetros que ajustar.\n",
        "\n",
        "El trade-off principal es entre granularidad y contexto.\n",
        "\n",
        "- Chunks muy pequeños pueden recuperar evidencia muy precisa, pero quizás no contienen suficiente información para responder.\n",
        "- Chunks muy grandes conservan más contexto, pero pueden diluir la señal semántica del embedding y ocupar más espacio en la ventana de contexto del modelo.\n",
        "\n",
        "El solapamiento ayuda, aunque también aumenta la cantidad de chunks, el costo de embeddings y la posibilidad de recuperar fragmentos repetidos."
      ],
      "id": "aUoTCJKt-GF3"
    },
    {
      "cell_type": "markdown",
      "source": [
        "### El role de los metadatos\n",
        "\n",
        "Los metadatos son tan importantes como el texto. En este ejemplo conservamos `fuente` y `marca`, pero en un escenario real podríamos guardar página, sección, fecha de actualización, permisos de acceso, tipo de documento o área responsable. Estos metadatos permiten explicar de dónde salió una respuesta, aplicar filtros antes del retrieval, auditar resultados y evitar que un usuario reciba información de documentos que no debería consultar."
      ],
      "metadata": {
        "id": "Sgq-5oqOItzH"
      },
      "id": "Sgq-5oqOItzH"
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Proceso\n",
        "\n",
        "Utilizaremos LlamaIndex para cargar todo el contenido dentro de un directorio y puego crear un indice vectorial.\n",
        "\n",
        "Algunas cosas a notar:\n",
        "\n",
        "- Utilizamos diferentes *Readers* para leer diferentes extensiones de archivos.\n",
        "- Podemos filtrar por extensiones especificas.\n",
        "- Podemos agregar metadatos especificos que necesitemos."
      ],
      "metadata": {
        "id": "PV5lYijIIP3u"
      },
      "id": "PV5lYijIIP3u"
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.core import SimpleDirectoryReader\n",
        "from llama_index.core.node_parser import SentenceSplitter\n",
        "from llama_index.readers.file import PDFReader\n",
        "\n",
        "def metadata_documento(file_path):\n",
        "    archivo = Path(file_path).name\n",
        "    return documentos_fuente.get(archivo, {\"fuente\": archivo, \"marca\": \"sin clasificar\"})\n",
        "\n",
        "lector = SimpleDirectoryReader(\n",
        "    input_dir=str(repo_documental),\n",
        "    required_exts=[\".txt\", \".pdf\"],\n",
        "    file_extractor={\".pdf\": PDFReader()},\n",
        "    file_metadata=metadata_documento,\n",
        ")\n",
        "documentos_llama = lector.load_data()"
      ],
      "metadata": {
        "id": "IkCsjpHBmtH1"
      },
      "id": "IkCsjpHBmtH1",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Utilizamos un `SentenceSplitter` para realizar la separación."
      ],
      "metadata": {
        "id": "1IMrW3wEIwj9"
      },
      "id": "1IMrW3wEIwj9"
    },
    {
      "cell_type": "code",
      "source": [
        "splitter = SentenceSplitter(chunk_size=256, chunk_overlap=30)"
      ],
      "metadata": {
        "id": "UMaEn_-Jmv0r"
      },
      "id": "UMaEn_-Jmv0r",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finalmente, LlamaIndex trabaja con la idea de nodos, los cuales son *chunks* con metadatos."
      ],
      "metadata": {
        "id": "vchWtJp8I2Ny"
      },
      "id": "vchWtJp8I2Ny"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bF7EaOX8-GF3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "378e599e-9252-4cb1-ad79-6c4c85f2378c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:pypdf._reader:invalid pdf header: b'<!== '\n",
            "WARNING:pypdf._reader:incorrect startxref pointer(1)\n",
            "WARNING:pypdf._reader:parsing for Object Streams\n",
            "WARNING:pypdf._reader:invalid pdf header: b'<!== '\n",
            "WARNING:pypdf._reader:incorrect startxref pointer(3)\n",
            "WARNING:pypdf._reader:parsing for Object Streams\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(136, 325)"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "nodos = splitter.get_nodes_from_documents(documentos_llama)\n",
        "\n",
        "len(documentos_llama), len(nodos)\n"
      ],
      "id": "bF7EaOX8-GF3"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Podemos inspeccionar algunos de los nodos para que puede entender como se representan. Esto es a fines educativos solo:"
      ],
      "metadata": {
        "id": "MAHzXi8cI9CQ"
      },
      "id": "MAHzXi8cI9CQ"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e8X9d5Eu-GF4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "010217ea-c95c-4067-9024-7528220dad1f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                     fuente marca  \\\n",
              "0  lg-user-manual-23476.pdf    lg   \n",
              "1  lg-user-manual-23476.pdf    lg   \n",
              "2  lg-user-manual-23476.pdf    lg   \n",
              "3  lg-user-manual-23476.pdf    lg   \n",
              "4  lg-user-manual-23476.pdf    lg   \n",
              "\n",
              "                                               texto  \n",
              "0  Antes de utilizar la unidad, lea detenidamente...  \n",
              "1  1 Guía de inicio\\nGuía de inicio2\\nGuía de ini...  \n",
              "2  El signo de exclamación dentro de \\nun triángu...  \n",
              "3  Los orificios no deben obstruirse. El producto...  \n",
              "4  Para \\nevitar la exposición directa al rayo lá...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7ad14dc8-751c-4884-a80a-f514567eff5e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fuente</th>\n",
              "      <th>marca</th>\n",
              "      <th>texto</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>lg-user-manual-23476.pdf</td>\n",
              "      <td>lg</td>\n",
              "      <td>Antes de utilizar la unidad, lea detenidamente...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>lg-user-manual-23476.pdf</td>\n",
              "      <td>lg</td>\n",
              "      <td>1 Guía de inicio\\nGuía de inicio2\\nGuía de ini...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>lg-user-manual-23476.pdf</td>\n",
              "      <td>lg</td>\n",
              "      <td>El signo de exclamación dentro de \\nun triángu...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>lg-user-manual-23476.pdf</td>\n",
              "      <td>lg</td>\n",
              "      <td>Los orificios no deben obstruirse. El producto...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>lg-user-manual-23476.pdf</td>\n",
              "      <td>lg</td>\n",
              "      <td>Para \\nevitar la exposición directa al rayo lá...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7ad14dc8-751c-4884-a80a-f514567eff5e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7ad14dc8-751c-4884-a80a-f514567eff5e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7ad14dc8-751c-4884-a80a-f514567eff5e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "chunks",
              "summary": "{\n  \"name\": \"chunks\",\n  \"rows\": 325,\n  \"fields\": [\n    {\n      \"column\": \"fuente\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"samsumg-user-manual-15223.pdf\",\n          \"lg-user-manual-23476.pdf\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"marca\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"samsung\",\n          \"lg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"texto\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 325,\n        \"samples\": [\n          \"74  Manual del usuario\\nPara dejar de utilizar un dispositivo de presentaci\\u00f3n externo\\n1. Haga clic con el bot\\u00f3n derecho en el escritorio y seleccione Opciones gr\\u00e1ficas > \\nSalida hacia > Equipo port\\u00e1til.\\n2. Desconecte el monitor externo o el televisor del ordenador.\",\n          \"\\u2022 Prep\\u00e1rese para apagar el ordenador durante el despegue y el aterrizaje.\\nBloqueo del ordenador\\nExtreme las precauciones cuando viaje o utilice su ordenador en zonas poco seguras, \\npara conservar el ordenador de la forma m\\u00e1s segura posible. Puede utilizar la opci\\u00f3n \\ndel Sistema de bloqueo de seguridad. Siga las instrucciones del Sistema de bloqueo \\nde seguridad que proporciona el fabricante para su instalaci\\u00f3n y uso espec\\u00edficos. La \\nilustraci\\u00f3n siguiente muestra c\\u00f3mo utilizar el bloqueo de forma general.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "chunks = pd.DataFrame(\n",
        "    [\n",
        "        {\n",
        "            \"fuente\": nodo.metadata[\"fuente\"],\n",
        "            \"marca\": nodo.metadata[\"marca\"],\n",
        "            \"texto\": nodo.get_content(metadata_mode=\"none\"),\n",
        "        }\n",
        "        for nodo in nodos\n",
        "    ]\n",
        ")\n",
        "\n",
        "chunks.head()"
      ],
      "id": "e8X9d5Eu-GF4"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OCUU8vd0-GF5"
      },
      "source": [
        "## Creando el índice vectorial\n",
        "\n",
        "Ahora configuramos un modelo de embeddings multilingüe y construimos un índice vectorial.\n",
        "\n",
        "Un índice vectorial almacena representaciones numéricas de los chunks para poder buscar por similitud semántica. En vez de preguntar si dos textos comparten exactamente las mismas palabras, preguntamos si sus embeddings apuntan a una zona parecida del espacio vectorial. Esto resulta especialmente útil en documentos reales, donde el usuario puede escribir \"modo suspensión\" y el manual puede hablar de \"suspender el sistema\" o de \"estado de bajo consumo\".\n",
        "\n",
        "En este caso los embeddings se generan con `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, un modelo multilingüe liviano. Cada chunk se transforma en un vector denso de números reales; LlamaIndex conserva ese vector asociado al texto original y a sus metadatos. Cuando luego llega una consulta, la consulta también se convierte en un embedding y se compara contra los vectores almacenados para seleccionar los chunks más similares."
      ],
      "id": "OCUU8vd0-GF5"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Rg26vStJ-GF5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f7805e63-efed-4406-e054-5928db5e4a3b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "from llama_index.core import Settings, VectorStoreIndex\n",
        "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
        "\n",
        "Settings.embed_model = HuggingFaceEmbedding(\n",
        "    model_name=\"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2\", device=\"cuda\"\n",
        ")"
      ],
      "id": "Rg26vStJ-GF5"
    },
    {
      "cell_type": "markdown",
      "source": [
        "En este notebook el índice vive en memoria para mantener el ejemplo liviano. Esto significa que los vectores y las referencias a los nodos existen mientras dura la sesión de Python; si reiniciamos el runtime, debemos reconstruir el índice. Para aprender resulta cómodo porque no requiere servicios externos, credenciales ni configuración adicional.\n",
        "\n",
        "En una implementación empresarial, el mismo patrón suele conectarse a un vector database o a un motor de búsqueda híbrida para persistencia, escalabilidad y control operativo. Plataformas como estas agregan capacidades que un índice en memoria no resuelve por sí solo: almacenamiento persistente, índices aproximados eficientes, filtros por metadatos, control de acceso, replicación, monitoreo, versionado y actualización incremental cuando cambian los documentos. El trade-off es que aumentan la complejidad de operación y obligan a pensar en seguridad, costos, latencia y gobierno del dato.\n",
        "\n",
        "Note que el concepto no cambia: seguimos guardando pares del tipo `(embedding, chunk, metadatos)`. Lo que cambia es dónde viven esos pares y qué garantías necesitamos alrededor de ellos."
      ],
      "metadata": {
        "id": "ikt_zORr-Bmv"
      },
      "id": "ikt_zORr-Bmv"
    },
    {
      "cell_type": "code",
      "source": [
        "indice = VectorStoreIndex(nodos, show_progress=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "82b6eee895954ae48bac1dd8cecb6081",
            "a79634cd62af44d3a6c7937f98c2b7a0",
            "dbfa2e10097742d491b28ac6ff0d9032",
            "fcd4418802214618b9ba7b87ce598a18",
            "145528e6f5bb4eb09263a5ad7cf9c571",
            "52366a488bf549ed906838d77322ae26",
            "c99285c2318a41838c42a868026130cb",
            "7777efb361f94d65b8b5f9e86e0eb4f6",
            "7a70d65508a94865909b65cf87ee83c6",
            "56ca23ad92764f8fa370a458893ba522",
            "18bd20084a3045a7b772491c68038fba"
          ]
        },
        "id": "oCB0oNi695sf",
        "outputId": "587e7625-acb7-46fe-8215-071f1aa44775"
      },
      "id": "oCB0oNi695sf",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating embeddings:   0%|          | 0/325 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "82b6eee895954ae48bac1dd8cecb6081"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "retriever = indice.as_retriever(similarity_top_k=3)"
      ],
      "metadata": {
        "id": "H3SajDx_m5ZH"
      },
      "id": "H3SajDx_m5ZH",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EgTxUSqH-GF5"
      },
      "source": [
        "## Recuperando evidencia\n",
        "\n",
        "Antes de pedirle al modelo que redacte una respuesta, inspeccionemos qué recupera el índice. Este paso es clave: si el retrieval trae mala evidencia, el generador tendrá muy pocas chances de responder bien.\n",
        "\n",
        "Esta idea está directamente conectada con dos conceptos importantes en aplicaciones empresariales: *hallucinations* y *groundedness*. Una alucinación ocurre cuando el modelo produce una respuesta plausible, pero no sustentada por la información disponible. Groundedness, en cambio, mide hasta qué punto la respuesta está apoyada en la evidencia recuperada. No alcanza con que la respuesta suene correcta; necesitamos poder señalar los fragmentos que justifican cada afirmación relevante.\n",
        "\n",
        "¿Por qué resulta tan importante medirlo? Porque muchas aplicaciones de negocio no toleran respuestas creativas sobre políticas internas, manuales técnicos, contratos o procedimientos. Una respuesta grounded permite auditar el sistema, mostrar citas al usuario, detectar documentos faltantes y decidir cuándo conviene responder \"no tengo evidencia suficiente\". En general, una métrica de groundedness compara la respuesta final con los chunks usados como contexto y penaliza afirmaciones que no aparecen o no se infieren razonablemente de esa evidencia."
      ],
      "id": "EgTxUSqH-GF5"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Lmg6imT-GF6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "outputId": "af21a951-4e0c-4dbc-df83-f7d80d41d9d6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   score                         fuente    marca  \\\n",
              "0  0.351       lg-user-manual-23476.pdf       lg   \n",
              "1  0.328       lg-user-manual-23476.pdf       lg   \n",
              "2  0.314  samsumg-user-manual-15223.pdf  samsung   \n",
              "\n",
              "                                               texto  \n",
              "0  16\\nFuncionamiento\\n4\\nFuncionamiento \\nDismin...  \n",
              "1  Funcionamiento 15\\nFuncionamiento\\n4\\nFunciona...  \n",
              "2  En las próximas secciones, se tratarán los mét...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e9b17c8d-5fd1-40ae-a975-4705b354728e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>score</th>\n",
              "      <th>fuente</th>\n",
              "      <th>marca</th>\n",
              "      <th>texto</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.351</td>\n",
              "      <td>lg-user-manual-23476.pdf</td>\n",
              "      <td>lg</td>\n",
              "      <td>16\\nFuncionamiento\\n4\\nFuncionamiento \\nDismin...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.328</td>\n",
              "      <td>lg-user-manual-23476.pdf</td>\n",
              "      <td>lg</td>\n",
              "      <td>Funcionamiento 15\\nFuncionamiento\\n4\\nFunciona...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.314</td>\n",
              "      <td>samsumg-user-manual-15223.pdf</td>\n",
              "      <td>samsung</td>\n",
              "      <td>En las próximas secciones, se tratarán los mét...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e9b17c8d-5fd1-40ae-a975-4705b354728e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-e9b17c8d-5fd1-40ae-a975-4705b354728e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-e9b17c8d-5fd1-40ae-a975-4705b354728e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \")\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.01868154169226939,\n        \"min\": 0.314,\n        \"max\": 0.351,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          0.351,\n          0.328,\n          0.314\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"fuente\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"samsumg-user-manual-15223.pdf\",\n          \"lg-user-manual-23476.pdf\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"marca\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"samsung\",\n          \"lg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"texto\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"16\\nFuncionamiento\\n4\\nFuncionamiento \\nDisminuir la \\nvelocidad de \\nreproducci\\u00f3n \\n(s\\u00f3lo hacia \\ndelante) \\nMientras la reproducci\\u00f3n \\npermanece pausada, pulse v \\nrepetidas veces para reproducir \\na distintas velocidades lentas. \\nPulse PLAY (z) para continuar \\nla reproducci\\u00f3n a velocidad \\nnormal.\\nVer una \\nimagen \\nde v\\u00eddeo \\nampliada.\\nDurante el modo de \\nreproducci\\u00f3n o de pausa, pulse \\nZOOM repetidas veces para \\nseleccionar ese modo. \\nZOOM: 100% : 200% : 300% \\n: 400% : 100% \\nw/s/a/d: Podr\\u00e1 desplazarse \\na trav\\u00e9s de la imagen ampliada \\nmediante los botones.\\nAudio tu\\nPara Hacer esto\\nReproducir \\nrepetidamente\\nPulse REPEAT durante la \\nreproducci\\u00f3n. Para seleccionar \\nuna parte a repetir, pulse REPEAT \\nrepetidas veces.\",\n          \"Funcionamiento 15\\nFuncionamiento\\n4\\nFuncionamiento general\\nPara Hacer esto\\nReproducir Pulse PLAY (z)\\nPausar Pulse PAUSE/STEP (M)\\nDetener Pulse STOP (Z)\\nSaltar hacia \\ndelante o hacia \\natr\\u00e1s \\nPulse C o V durante la \\nreproducci\\u00f3n\\nAdelantar o \\nrebobinar\\nPulse c o v durante la \\nreproducci\\u00f3n.\\nReanudar la \\nreproducci\\u00f3n\\nPulse STOP (Z) durante la \\nreproducci\\u00f3n para guardar el \\npunto de parada.\\n y Pulse STOP (Z) una vez: \\nMuestra MZ en la pantalla \\n(Reanudar Stop)\\n y Pulse STOP (Z) dos veces: \\nMuestra Z en la pantalla \\n(Completar Stop)\\nProtector de \\npantalla\\nEl salvapantallas aparece cuando \\nusted deja el lector de la unidad \\nen modo Stop durante cerca de \\ncinco minutos.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "pregunta = \"¿Que función cumple el Modo Suspensión?\"\n",
        "nodos_recuperados = retriever.retrieve(pregunta)\n",
        "\n",
        "pd.DataFrame(\n",
        "    [\n",
        "        {\n",
        "            \"score\": round(resultado.score or 0, 3),\n",
        "            \"fuente\": resultado.metadata[\"fuente\"],\n",
        "            \"marca\": resultado.metadata[\"marca\"],\n",
        "            \"texto\": resultado.node.get_content(metadata_mode=\"none\"),\n",
        "        }\n",
        "        for resultado in nodos_recuperados\n",
        "    ]\n",
        ")"
      ],
      "id": "1Lmg6imT-GF6"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EXb2ZAHJ-GF6"
      },
      "source": [
        "## Conectando un SLM para responder\n",
        "\n",
        "Usaremos un modelo pequeño de tipo text-to-text. No esperamos la misma calidad que en un LLM grande, pero resulta suficiente para demostrar la separación de responsabilidades: el índice trae evidencia y el SLM redacta una respuesta condicionada por esa evidencia.\n",
        "\n",
        "El rol del SLM (por las siglas en inglés de *Small Language Model*) no es memorizar todo el conocimiento de la organización. En un pipeline RAG bien diseñado, el conocimiento relevante llega en el contexto recuperado. El modelo se ocupa principalmente de leer esos fragmentos, seleccionar la información útil, resolver pequeñas ambigüedades y redactar una respuesta clara para el usuario.\n",
        "\n",
        "Por eso, si el grounding es bueno, no siempre necesitamos un modelo con razonamiento extremadamente fuerte. Para preguntas factuales sobre manuales, políticas o procedimientos, la parte difícil suele ser encontrar la evidencia correcta. Un modelo más grande puede ayudar cuando la pregunta exige múltiples pasos de razonamiento, síntesis compleja o manejo de instrucciones muy largas, pero también implica mayor costo, latencia y riesgo operativo. En general, conviene empezar preguntando: ¿la respuesta está en los documentos y el retrieval la encuentra? Si la respuesta es sí, un SLM puede ser suficiente."
      ],
      "id": "EXb2ZAHJ-GF6"
    },
    {
      "cell_type": "code",
      "source": [
        "from huggingface_hub import notebook_login\n",
        "\n",
        "notebook_login()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17,
          "referenced_widgets": [
            "68d5f0e462ad47b3954e41741a734c44",
            "7772eabebf8249d88c9441fa24f48921",
            "6a57f167f8b74907913ad1c953bb5c9f",
            "8ef368e3dddd41888f4bc15acaf8103a",
            "c1d86babe4044342bbdb5ac5de24e2cf",
            "3fded1b8bc0c47efa3e151696e8989e0",
            "368ce21165cb4fe593e115dca9863854",
            "05af12d618144e8489b267655147caac",
            "cd62142c9b314753b7068335aa71adf3",
            "94ba62f9278948dfb6c561a9ce71cbf9",
            "0ff3ca7d3ce643f98952e79fd40e8d60",
            "fb81e6a9ebbb467a8e8a943071de7218",
            "b3bb8e33a89c42b1b9a92e9a80fb3276",
            "31e963ad7f0e4283afbe83c5c50c3866",
            "c64400b423ed45379fb16ff60f5f790f",
            "78df67275bf74f499eb4392b6a7ad323",
            "bf2d3115ba894dbe9ca63e87207dd553",
            "0731c15a95114a31855ef182937851c0",
            "94c8789a91924e4ea48e6be0c2edbcd8",
            "9500950589414514b1f3a44d9b36320e"
          ]
        },
        "id": "NmVdUIe4FHxz",
        "outputId": "20fe0ae5-05d7-4e91-9c8a-46713f631ff3"
      },
      "id": "NmVdUIe4FHxz",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "VBox(children=(HTML(value='<center> <img\\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "68d5f0e462ad47b3954e41741a734c44"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "En este caso, estamos usando Gemma 2B:"
      ],
      "metadata": {
        "id": "w2FeORcchS3h"
      },
      "id": "w2FeORcchS3h"
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from llama_index.llms.huggingface import HuggingFaceLLM\n",
        "from transformers import BitsAndBytesConfig\n",
        "\n",
        "Settings.llm = HuggingFaceLLM(\n",
        "    tokenizer_name=\"google/gemma-2b-it\",\n",
        "    model_name=\"google/gemma-2b-it\",\n",
        "    model_kwargs={\n",
        "        \"dtype\": torch.bfloat16,\n",
        "    },\n",
        "    device_map=\"auto\"\n",
        ")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81,
          "referenced_widgets": [
            "0b267bf9e1084733825abe0217a0cb1c",
            "28fa2515858f441e851aeec3271e76e6",
            "d53879b5660447cda3d1bd94da28aa1a",
            "a6fbe67af03b436da4aa5724567bf18a",
            "5b28a246ae294153b55a113c697ab3cd",
            "3905a1fa9d9749d282478ec814dff3fd",
            "1cabb167d7d446b1a0ab86e92373ea7e",
            "9d2e8f9288834de0af7c2d57786ff289",
            "f9f6ba3b4a0e40adbb9a40a142772a79",
            "1d2dce7cd4b04234888e6765d7d1d46c",
            "b9edd164912f40aca1380029d29759d5",
            "60afe0fabefd4a60b00d1a08f7138a42",
            "b79ea979b76148d1a6c8ee0d38fedaf9",
            "e7998009bc1a4aed886bbe9c9479f530",
            "a37c03b166fa4bd09d68dd36178b38f9",
            "ed4b8dee2c044080adf7df836f94c680",
            "6f17612bc07440bcbd1ae73cfcb2a630",
            "a58bc62e38bf48f180f5ab1bb1884f1c",
            "652f4c7d01434963aad88e2fb58f53e3",
            "087fbffa0afb4d38ac1fbc39019e10f0",
            "103516eeffa643ef847fcd824adc4439",
            "ca2680e4eda44b2d85d29cbf4da34e14"
          ]
        },
        "id": "srzPIKcWZvL-",
        "outputId": "422cd4f8-b8a5-4aa6-cafe-bd34bb202245"
      },
      "id": "srzPIKcWZvL-",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0b267bf9e1084733825abe0217a0cb1c"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.model:   0%|          | 0.00/4.24M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "60afe0fabefd4a60b00d1a08f7138a42"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Creamos un `QueryEngine` a partir del indice que creamos anteriormente. Este `QueryEngine` funciona de forma similar a un retriever, pero ademos administra la generación de embeddings de las consultas via el modelo de embeddings que configuramos.\n",
        "\n",
        "Configuramos ademas el parámetro `top_k` de igual forma que configuramos el retriever anteriormente."
      ],
      "metadata": {
        "id": "_-KVmBdtgzYO"
      },
      "id": "_-KVmBdtgzYO"
    },
    {
      "cell_type": "code",
      "source": [
        "query_engine = indice.as_query_engine(similarity_top_k=3)\n"
      ],
      "metadata": {
        "id": "NYx8-OiAgrsR"
      },
      "id": "NYx8-OiAgrsR",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Verificamos que funciona."
      ],
      "metadata": {
        "id": "BoGUuHHHhPR4"
      },
      "id": "BoGUuHHHhPR4"
    },
    {
      "cell_type": "code",
      "source": [
        "pregunta_llama_index = \"¿Que función cumple el Modo Suspensión?\"\n",
        "respuesta_llama_index = query_engine.query(pregunta_llama_index)\n",
        "\n",
        "print(\"Pregunta:\", pregunta_llama_index)\n",
        "print(\"Respuesta:\", respuesta_llama_index)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X5JdyqC4hOPj",
        "outputId": "40fadce3-0eba-4ee5-8dd7-2b3afdde94cc"
      },
      "id": "X5JdyqC4hOPj",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pregunta: ¿Que función cumple el Modo Suspensión?\n",
            "Respuesta: \n",
            "El modo de suspensión almacena la información que no haya sido guardada en el disco duro, solo se almacena en la memoria del ordenador. Si se interrumpe el suministro de energía, se perderá la información.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Podemos inspeccionar los nodos que fueron utilizados para la respuesta:"
      ],
      "metadata": {
        "id": "knRBUKeBph68"
      },
      "id": "knRBUKeBph68"
    },
    {
      "cell_type": "code",
      "source": [
        "pd.set_option('display.max_colwidth', None)\n",
        "\n",
        "references_data = []\n",
        "for i, node in enumerate(respuesta_llama_index.source_nodes, 1):\n",
        "    references_data.append({\n",
        "        \"Score\": f\"{node.score:.3f}\",\n",
        "        \"Source\": node.metadata['fuente'],\n",
        "        \"Content\": node.get_content(metadata_mode='none')\n",
        "    })\n",
        "\n",
        "references_df = pd.DataFrame(references_data)\n",
        "display(references_df)"
      ],
      "metadata": {
        "id": "0GaTM2eRpg-b"
      },
      "id": "0GaTM2eRpg-b",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ¿Como relaciona RAG la evidencia con la consulta?\n",
        "\n",
        "Si recuerdan de cuando vimos *few-shot learning*, los grandes modelos de lenguaje son grandes generados de texto condicionado. Cuando generamos texto condicionando sobre ejemplos, obtenemos *few-shot learning*. Sin embargo, también podemos condicionar la generación de contenido basado en evidencia que requerimos que el modelo utilice para responde.\n",
        "\n",
        "En el caso de la libraria llama-index, este condicionamiento está codificado dentro de la misma. Podemos inspeccionarlo:"
      ],
      "metadata": {
        "id": "Yzbu7UWDhxhL"
      },
      "id": "Yzbu7UWDhxhL"
    },
    {
      "cell_type": "code",
      "source": [
        "print(query_engine._response_synthesizer.get_prompts()[\"text_qa_template\"].default_template.template)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wWAT7dJdhZ7J",
        "outputId": "eaefa5c6-28dc-41a3-fdea-85519742ffff"
      },
      "id": "wWAT7dJdhZ7J",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Context information is below.\n",
            "---------------------\n",
            "{context_str}\n",
            "---------------------\n",
            "Given the context information and not prior knowledge, answer the query.\n",
            "Query: {query_str}\n",
            "Answer: \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A6dlYwnM-GF8"
      },
      "source": [
        "## Una pregunta fuera del alcance\n",
        "\n",
        "Un sistema empresarial no debería inventar respuestas cuando la base documental no contiene evidencia. Un mecanismo simple es revisar el score de recuperación y responder explícitamente cuando no hay soporte suficiente. En producción este control suele complementarse con evaluación, reglas de negocio y monitoreo.\n",
        "\n",
        "¿Cómo podríamos medir este comportamiento? Una práctica común es construir un conjunto de preguntas de evaluación con dos tipos de casos: preguntas dentro del alcance, donde sabemos qué documento contiene la respuesta, y preguntas fuera del alcance, donde esperamos una negativa explícita. Para las primeras medimos si el sistema recupera evidencia relevante y si la respuesta está grounded. Para las segundas medimos si el sistema evita responder con información inventada.\n",
        "\n",
        "En forma simple, podemos observar el score de similitud de los chunks recuperados y definir umbrales. Sin embargo, los scores dependen del modelo de embeddings, del índice y del dominio, por lo que no conviene interpretarlos como probabilidades universales. Resulta mejor calibrarlos con ejemplos reales: revisar distribuciones de scores para preguntas correctas e incorrectas, medir falsos positivos cuando el sistema responde sin evidencia y falsos negativos cuando rechaza preguntas que sí tenían soporte.\n",
        "\n",
        "También podemos evaluar a nivel de respuesta: pedir que el sistema cite fuentes, verificar que cada afirmación importante aparezca en los chunks recuperados y registrar casos donde el modelo dice \"no hay información suficiente\". En aplicaciones empresariales, esta medición es clave porque convierte el problema de confianza en algo observable: no solo miramos si la respuesta es fluida, sino si está justificada por evidencia autorizada."
      ],
      "id": "A6dlYwnM-GF8"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Sdr0LPUs-GF8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "584180a6-08f5-468d-de08-e7c43024421e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pregunta: ¿Cuál es el menú de la cafetería de la oficina?\n",
            "Respuesta: \n",
            "No hay información en el contexto sobre el menú de la cafetería de la oficina.\n"
          ]
        }
      ],
      "source": [
        "pregunta_fuera_de_alcance = \"¿Cuál es el menú de la cafetería de la oficina?\"\n",
        "respuesta_llama_index = query_engine.query(pregunta_fuera_de_alcance)\n",
        "\n",
        "print(\"Pregunta:\", pregunta_fuera_de_alcance)\n",
        "print(\"Respuesta:\", respuesta_llama_index)\n"
      ],
      "id": "Sdr0LPUs-GF8"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oRbggVgh-GF8"
      },
      "source": [
        "## ¿Qué faltaría para producción?\n",
        "\n",
        "Este ejemplo sigue siendo pequeño, pero ya muestra las piezas principales de una arquitectura RAG más robusta:\n",
        "\n",
        "- Documentos almacenados en el sistema de archivos antes de la ingesta.\n",
        "- Carga documental con un procesador como `SimpleDirectoryReader` de LlamaIndex.\n",
        "- Ingesta separada de la consulta en línea.\n",
        "- Chunking explícito con solapamiento.\n",
        "- Conservación de metadatos de procedencia.\n",
        "- Índice vectorial para retrieval semántico.\n",
        "- SLM desacoplado del índice para generar la respuesta.\n",
        "- Guardas simples para evitar respuestas sin evidencia.\n",
        "\n",
        "En un escenario real agregaríamos persistencia del índice, control de permisos por documento, versionado, monitoreo de latencia, evaluación de respuestas, reranking, búsqueda híbrida y pruebas con preguntas representativas del dominio."
      ],
      "id": "oRbggVgh-GF8"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dg1bSgYg-GF8"
      },
      "source": [
        "## Cierre\n",
        "\n",
        "RAG no es solo llamar a un LLM con un texto adicional. Podemos pensarlo como un pipeline de información: preparar documentos, representarlos como embeddings, recuperar evidencia, construir un prompt controlado y generar una respuesta verificable. La calidad final depende tanto del generador como del índice y de la forma en que partimos los documentos."
      ],
      "id": "Dg1bSgYg-GF8"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "pygments_lexer": "ipython3"
    },
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "toc_visible": true
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "82b6eee895954ae48bac1dd8cecb6081": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a79634cd62af44d3a6c7937f98c2b7a0",
              "IPY_MODEL_dbfa2e10097742d491b28ac6ff0d9032",
              "IPY_MODEL_fcd4418802214618b9ba7b87ce598a18"
            ],
            "layout": "IPY_MODEL_145528e6f5bb4eb09263a5ad7cf9c571"
          }
        },
        "a79634cd62af44d3a6c7937f98c2b7a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_52366a488bf549ed906838d77322ae26",
            "placeholder": "​",
            "style": "IPY_MODEL_c99285c2318a41838c42a868026130cb",
            "value": "Generating embeddings: 100%"
          }
        },
        "dbfa2e10097742d491b28ac6ff0d9032": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7777efb361f94d65b8b5f9e86e0eb4f6",
            "max": 325,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_7a70d65508a94865909b65cf87ee83c6",
            "value": 325
          }
        },
        "fcd4418802214618b9ba7b87ce598a18": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_56ca23ad92764f8fa370a458893ba522",
            "placeholder": "​",
            "style": "IPY_MODEL_18bd20084a3045a7b772491c68038fba",
            "value": " 325/325 [00:02&lt;00:00, 256.86it/s]"
          }
        },
        "145528e6f5bb4eb09263a5ad7cf9c571": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "52366a488bf549ed906838d77322ae26": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c99285c2318a41838c42a868026130cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7777efb361f94d65b8b5f9e86e0eb4f6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7a70d65508a94865909b65cf87ee83c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "56ca23ad92764f8fa370a458893ba522": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "18bd20084a3045a7b772491c68038fba": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "68d5f0e462ad47b3954e41741a734c44": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "VBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "VBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "VBoxView",
            "box_style": "",
            "children": [],
            "layout": "IPY_MODEL_368ce21165cb4fe593e115dca9863854"
          }
        },
        "7772eabebf8249d88c9441fa24f48921": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_05af12d618144e8489b267655147caac",
            "placeholder": "​",
            "style": "IPY_MODEL_cd62142c9b314753b7068335aa71adf3",
            "value": "<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.svg\nalt='Hugging Face'> <br> Copy a token from <a\nhref=\"https://huggingface.co/settings/tokens\" target=\"_blank\">your Hugging Face\ntokens page</a> and paste it below. <br> Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file. </center>"
          }
        },
        "6a57f167f8b74907913ad1c953bb5c9f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "PasswordModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "PasswordModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "PasswordView",
            "continuous_update": true,
            "description": "Token:",
            "description_tooltip": null,
            "disabled": false,
            "layout": "IPY_MODEL_94ba62f9278948dfb6c561a9ce71cbf9",
            "placeholder": "​",
            "style": "IPY_MODEL_0ff3ca7d3ce643f98952e79fd40e8d60",
            "value": ""
          }
        },
        "8ef368e3dddd41888f4bc15acaf8103a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "CheckboxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "CheckboxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "CheckboxView",
            "description": "Add token as git credential?",
            "description_tooltip": null,
            "disabled": false,
            "indent": true,
            "layout": "IPY_MODEL_fb81e6a9ebbb467a8e8a943071de7218",
            "style": "IPY_MODEL_b3bb8e33a89c42b1b9a92e9a80fb3276",
            "value": true
          }
        },
        "c1d86babe4044342bbdb5ac5de24e2cf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ButtonModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ButtonView",
            "button_style": "",
            "description": "Login",
            "disabled": false,
            "icon": "",
            "layout": "IPY_MODEL_31e963ad7f0e4283afbe83c5c50c3866",
            "style": "IPY_MODEL_c64400b423ed45379fb16ff60f5f790f",
            "tooltip": ""
          }
        },
        "3fded1b8bc0c47efa3e151696e8989e0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_78df67275bf74f499eb4392b6a7ad323",
            "placeholder": "​",
            "style": "IPY_MODEL_bf2d3115ba894dbe9ca63e87207dd553",
            "value": "\n<b>Pro Tip:</b> If you don't already have one, you can create a dedicated\n'notebooks' token with 'write' access, that you can then easily reuse for all\nnotebooks. </center>"
          }
        },
        "368ce21165cb4fe593e115dca9863854": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": "center",
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "flex",
            "flex": null,
            "flex_flow": "column",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "50%"
          }
        },
        "05af12d618144e8489b267655147caac": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cd62142c9b314753b7068335aa71adf3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "94ba62f9278948dfb6c561a9ce71cbf9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0ff3ca7d3ce643f98952e79fd40e8d60": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fb81e6a9ebbb467a8e8a943071de7218": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b3bb8e33a89c42b1b9a92e9a80fb3276": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "31e963ad7f0e4283afbe83c5c50c3866": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c64400b423ed45379fb16ff60f5f790f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ButtonStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "button_color": null,
            "font_weight": ""
          }
        },
        "78df67275bf74f499eb4392b6a7ad323": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bf2d3115ba894dbe9ca63e87207dd553": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0731c15a95114a31855ef182937851c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "LabelModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "LabelModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "LabelView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_94c8789a91924e4ea48e6be0c2edbcd8",
            "placeholder": "​",
            "style": "IPY_MODEL_9500950589414514b1f3a44d9b36320e",
            "value": "Connecting..."
          }
        },
        "94c8789a91924e4ea48e6be0c2edbcd8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9500950589414514b1f3a44d9b36320e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0b267bf9e1084733825abe0217a0cb1c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_28fa2515858f441e851aeec3271e76e6",
              "IPY_MODEL_d53879b5660447cda3d1bd94da28aa1a",
              "IPY_MODEL_a6fbe67af03b436da4aa5724567bf18a"
            ],
            "layout": "IPY_MODEL_5b28a246ae294153b55a113c697ab3cd"
          }
        },
        "28fa2515858f441e851aeec3271e76e6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3905a1fa9d9749d282478ec814dff3fd",
            "placeholder": "​",
            "style": "IPY_MODEL_1cabb167d7d446b1a0ab86e92373ea7e",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "d53879b5660447cda3d1bd94da28aa1a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9d2e8f9288834de0af7c2d57786ff289",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_f9f6ba3b4a0e40adbb9a40a142772a79",
            "value": 2
          }
        },
        "a6fbe67af03b436da4aa5724567bf18a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1d2dce7cd4b04234888e6765d7d1d46c",
            "placeholder": "​",
            "style": "IPY_MODEL_b9edd164912f40aca1380029d29759d5",
            "value": " 2/2 [00:22&lt;00:00,  9.43s/it]"
          }
        },
        "5b28a246ae294153b55a113c697ab3cd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3905a1fa9d9749d282478ec814dff3fd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1cabb167d7d446b1a0ab86e92373ea7e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9d2e8f9288834de0af7c2d57786ff289": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f9f6ba3b4a0e40adbb9a40a142772a79": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1d2dce7cd4b04234888e6765d7d1d46c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b9edd164912f40aca1380029d29759d5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "60afe0fabefd4a60b00d1a08f7138a42": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b79ea979b76148d1a6c8ee0d38fedaf9",
              "IPY_MODEL_e7998009bc1a4aed886bbe9c9479f530",
              "IPY_MODEL_a37c03b166fa4bd09d68dd36178b38f9"
            ],
            "layout": "IPY_MODEL_ed4b8dee2c044080adf7df836f94c680"
          }
        },
        "b79ea979b76148d1a6c8ee0d38fedaf9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6f17612bc07440bcbd1ae73cfcb2a630",
            "placeholder": "​",
            "style": "IPY_MODEL_a58bc62e38bf48f180f5ab1bb1884f1c",
            "value": "tokenizer.model: 100%"
          }
        },
        "e7998009bc1a4aed886bbe9c9479f530": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_652f4c7d01434963aad88e2fb58f53e3",
            "max": 4241003,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_087fbffa0afb4d38ac1fbc39019e10f0",
            "value": 4241003
          }
        },
        "a37c03b166fa4bd09d68dd36178b38f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_103516eeffa643ef847fcd824adc4439",
            "placeholder": "​",
            "style": "IPY_MODEL_ca2680e4eda44b2d85d29cbf4da34e14",
            "value": " 4.24M/4.24M [00:00&lt;00:00, 5.04MB/s]"
          }
        },
        "ed4b8dee2c044080adf7df836f94c680": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6f17612bc07440bcbd1ae73cfcb2a630": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a58bc62e38bf48f180f5ab1bb1884f1c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "652f4c7d01434963aad88e2fb58f53e3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "087fbffa0afb4d38ac1fbc39019e10f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "103516eeffa643ef847fcd824adc4439": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca2680e4eda44b2d85d29cbf4da34e14": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}