{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "188b48ed5b49e64b",
      "metadata": {
        "collapsed": false,
        "id": "188b48ed5b49e64b"
      },
      "source": [
        "# UD03 · Notebook 2 — Clasificador de noticias\n",
        "\n",
        "En esta práctica, crearemos un clasificador de noticias utilizando las técnicas de procesamiento del lenguaje natural que hemos visto en clase, centrándose en la representación del texto.\n",
        "\n",
        "Usaremos el `dataset` [AG News](https://www.kaggle.com/amananandrai/ag-news-classification-dataset) que contiene 1,000,000 noticias de 4 categorías diferentes.\n",
        "\n",
        "## Dataset\n",
        "\n",
        "Para cargar el conjunto de datos, usaremos la librería `datasets`. Esta librería nos permitirá cargar muchos conjuntos de datos diferentes de una manera simple.En este caso, cargaremos el conjunto de datos AG News."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a731df9a4a91fd19",
      "metadata": {
        "collapsed": false,
        "id": "a731df9a4a91fd19"
      },
      "source": [
        "## Preparación del dataset\n",
        "\n",
        "Para instalar las librerías necesarias, ejecutaremos la siguiente celda.\n",
        "\n",
        "Usaremos `pytorch` (una libreria de deep learning), `pipeline` (una libreria de tratamiento de datos), `scikit-learn` (una libreria de machine learning) y `transformers` (una libreria de modelos de lenguaje)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "9987ceaeb8f560be",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:06:47.825392Z",
          "start_time": "2024-01-17T21:06:09.205588Z"
        },
        "id": "9987ceaeb8f560be",
        "outputId": "c5c34496-e88b-4fac-d4a5-86999f4e8292",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: torch in /usr/local/lib/python3.12/dist-packages (2.9.0+cpu)\n",
            "Collecting torch\n",
            "  Downloading torch-2.9.1-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (30 kB)\n",
            "Requirement already satisfied: datasets in /usr/local/lib/python3.12/dist-packages (4.0.0)\n",
            "Collecting datasets\n",
            "  Downloading datasets-4.4.1-py3-none-any.whl.metadata (19 kB)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.12/dist-packages (1.6.1)\n",
            "Collecting scikit-learn\n",
            "  Downloading scikit_learn-1.8.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (11 kB)\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.12/dist-packages (4.57.3)\n",
            "Collecting gensim\n",
            "  Downloading gensim-4.4.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (8.4 kB)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from torch) (3.20.0)\n",
            "Requirement already satisfied: typing-extensions>=4.10.0 in /usr/local/lib/python3.12/dist-packages (from torch) (4.15.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.12/dist-packages (from torch) (75.2.0)\n",
            "Requirement already satisfied: sympy>=1.13.3 in /usr/local/lib/python3.12/dist-packages (from torch) (1.14.0)\n",
            "Requirement already satisfied: networkx>=2.5.1 in /usr/local/lib/python3.12/dist-packages (from torch) (3.6.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.12/dist-packages (from torch) (3.1.6)\n",
            "Requirement already satisfied: fsspec>=0.8.5 in /usr/local/lib/python3.12/dist-packages (from torch) (2025.3.0)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.8.93 (from torch)\n",
            "  Downloading nvidia_cuda_nvrtc_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting nvidia-cuda-runtime-cu12==12.8.90 (from torch)\n",
            "  Downloading nvidia_cuda_runtime_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting nvidia-cuda-cupti-cu12==12.8.90 (from torch)\n",
            "  Downloading nvidia_cuda_cupti_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting nvidia-cudnn-cu12==9.10.2.21 (from torch)\n",
            "  Downloading nvidia_cudnn_cu12-9.10.2.21-py3-none-manylinux_2_27_x86_64.whl.metadata (1.8 kB)\n",
            "Collecting nvidia-cublas-cu12==12.8.4.1 (from torch)\n",
            "  Downloading nvidia_cublas_cu12-12.8.4.1-py3-none-manylinux_2_27_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting nvidia-cufft-cu12==11.3.3.83 (from torch)\n",
            "  Downloading nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting nvidia-curand-cu12==10.3.9.90 (from torch)\n",
            "  Downloading nvidia_curand_cu12-10.3.9.90-py3-none-manylinux_2_27_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting nvidia-cusolver-cu12==11.7.3.90 (from torch)\n",
            "  Downloading nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_x86_64.whl.metadata (1.8 kB)\n",
            "Collecting nvidia-cusparse-cu12==12.5.8.93 (from torch)\n",
            "  Downloading nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.8 kB)\n",
            "Collecting nvidia-cusparselt-cu12==0.7.1 (from torch)\n",
            "  Downloading nvidia_cusparselt_cu12-0.7.1-py3-none-manylinux2014_x86_64.whl.metadata (7.0 kB)\n",
            "Collecting nvidia-nccl-cu12==2.27.5 (from torch)\n",
            "  Downloading nvidia_nccl_cu12-2.27.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.0 kB)\n",
            "Collecting nvidia-nvshmem-cu12==3.3.20 (from torch)\n",
            "  Downloading nvidia_nvshmem_cu12-3.3.20-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.1 kB)\n",
            "Collecting nvidia-nvtx-cu12==12.8.90 (from torch)\n",
            "  Downloading nvidia_nvtx_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.8 kB)\n",
            "Collecting nvidia-nvjitlink-cu12==12.8.93 (from torch)\n",
            "  Downloading nvidia_nvjitlink_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting nvidia-cufile-cu12==1.13.1.3 (from torch)\n",
            "  Downloading nvidia_cufile_cu12-1.13.1.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting triton==3.5.1 (from torch)\n",
            "  Downloading triton-3.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.7 kB)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.12/dist-packages (from datasets) (2.0.2)\n",
            "Collecting pyarrow>=21.0.0 (from datasets)\n",
            "  Downloading pyarrow-22.0.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.2 kB)\n",
            "Requirement already satisfied: dill<0.4.1,>=0.3.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.3.8)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (from datasets) (2.2.2)\n",
            "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.12/dist-packages (from datasets) (2.32.4)\n",
            "Requirement already satisfied: httpx<1.0.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.28.1)\n",
            "Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.12/dist-packages (from datasets) (4.67.1)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.12/dist-packages (from datasets) (3.6.0)\n",
            "Requirement already satisfied: multiprocess<0.70.19 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.70.16)\n",
            "Requirement already satisfied: huggingface-hub<2.0,>=0.25.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.36.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.12/dist-packages (from datasets) (25.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.12/dist-packages (from datasets) (6.0.3)\n",
            "Requirement already satisfied: scipy>=1.10.0 in /usr/local/lib/python3.12/dist-packages (from scikit-learn) (1.16.3)\n",
            "Requirement already satisfied: joblib>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from scikit-learn) (1.5.2)\n",
            "Requirement already satisfied: threadpoolctl>=3.2.0 in /usr/local/lib/python3.12/dist-packages (from scikit-learn) (3.6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.12/dist-packages (from transformers) (2025.11.3)\n",
            "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.22.1)\n",
            "Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.7.0)\n",
            "Requirement already satisfied: smart_open>=1.8.1 in /usr/local/lib/python3.12/dist-packages (from gensim) (7.5.0)\n",
            "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.12/dist-packages (from fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (3.13.2)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (4.12.0)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (2025.11.12)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (1.0.9)\n",
            "Requirement already satisfied: idna in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (3.11)\n",
            "Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1.0.0->datasets) (0.16.0)\n",
            "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<2.0,>=0.25.0->datasets) (1.2.0)\n",
            "Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests>=2.32.2->datasets) (3.4.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests>=2.32.2->datasets) (2.5.0)\n",
            "Requirement already satisfied: wrapt in /usr/local/lib/python3.12/dist-packages (from smart_open>=1.8.1->gensim) (2.0.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy>=1.13.3->torch) (1.3.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.12/dist-packages (from jinja2->torch) (3.0.3)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2.9.0.post0)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2025.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2025.2)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (2.6.1)\n",
            "Requirement already satisfied: aiosignal>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (1.4.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (25.4.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (1.8.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (6.7.0)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (0.4.1)\n",
            "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (1.22.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)\n",
            "Downloading torch-2.9.1-cp312-cp312-manylinux_2_28_x86_64.whl (899.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m899.7/899.7 MB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cublas_cu12-12.8.4.1-py3-none-manylinux_2_27_x86_64.whl (594.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m594.3/594.3 MB\u001b[0m \u001b[31m2.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_cupti_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (10.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.2/10.2 MB\u001b[0m \u001b[31m31.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_nvrtc_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl (88.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.0/88.0 MB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_runtime_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (954 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m954.8/954.8 kB\u001b[0m \u001b[31m26.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cudnn_cu12-9.10.2.21-py3-none-manylinux_2_27_x86_64.whl (706.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m706.8/706.8 MB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (193.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m193.1/193.1 MB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cufile_cu12-1.13.1.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (1.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m72.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_curand_cu12-10.3.9.90-py3-none-manylinux_2_27_x86_64.whl (63.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.6/63.6 MB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_x86_64.whl (267.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.5/267.5 MB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (288.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m288.2/288.2 MB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusparselt_cu12-0.7.1-py3-none-manylinux2014_x86_64.whl (287.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m287.2/287.2 MB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nccl_cu12-2.27.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (322.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m322.3/322.3 MB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nvjitlink_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl (39.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m39.3/39.3 MB\u001b[0m \u001b[31m20.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nvshmem_cu12-3.3.20-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (124.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.7/124.7 MB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nvtx_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m90.0/90.0 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading triton-3.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (170.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m170.5/170.5 MB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading datasets-4.4.1-py3-none-any.whl (511 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m511.6/511.6 kB\u001b[0m \u001b[31m32.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading scikit_learn-1.8.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (8.9 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.9/8.9 MB\u001b[0m \u001b[31m100.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading gensim-4.4.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (27.9 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m27.9/27.9 MB\u001b[0m \u001b[31m69.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pyarrow-22.0.0-cp312-cp312-manylinux_2_28_x86_64.whl (47.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m47.7/47.7 MB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: nvidia-cusparselt-cu12, triton, pyarrow, nvidia-nvtx-cu12, nvidia-nvshmem-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, scikit-learn, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, gensim, nvidia-cusolver-cu12, torch, datasets\n",
            "  Attempting uninstall: pyarrow\n",
            "    Found existing installation: pyarrow 18.1.0\n",
            "    Uninstalling pyarrow-18.1.0:\n",
            "      Successfully uninstalled pyarrow-18.1.0\n",
            "  Attempting uninstall: nvidia-nccl-cu12\n",
            "    Found existing installation: nvidia-nccl-cu12 2.28.9\n",
            "    Uninstalling nvidia-nccl-cu12-2.28.9:\n",
            "      Successfully uninstalled nvidia-nccl-cu12-2.28.9\n",
            "  Attempting uninstall: scikit-learn\n",
            "    Found existing installation: scikit-learn 1.6.1\n",
            "    Uninstalling scikit-learn-1.6.1:\n",
            "      Successfully uninstalled scikit-learn-1.6.1\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 2.9.0+cpu\n",
            "    Uninstalling torch-2.9.0+cpu:\n",
            "      Successfully uninstalled torch-2.9.0+cpu\n",
            "  Attempting uninstall: datasets\n",
            "    Found existing installation: datasets 4.0.0\n",
            "    Uninstalling datasets-4.0.0:\n",
            "      Successfully uninstalled datasets-4.0.0\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "torchvision 0.24.0+cpu requires torch==2.9.0, but you have torch 2.9.1 which is incompatible.\n",
            "torchaudio 2.9.0+cpu requires torch==2.9.0, but you have torch 2.9.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed datasets-4.4.1 gensim-4.4.0 nvidia-cublas-cu12-12.8.4.1 nvidia-cuda-cupti-cu12-12.8.90 nvidia-cuda-nvrtc-cu12-12.8.93 nvidia-cuda-runtime-cu12-12.8.90 nvidia-cudnn-cu12-9.10.2.21 nvidia-cufft-cu12-11.3.3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.5 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvshmem-cu12-3.3.20 nvidia-nvtx-cu12-12.8.90 pyarrow-22.0.0 scikit-learn-1.8.0 torch-2.9.1 triton-3.5.1\n"
          ]
        }
      ],
      "source": [
        "# Instalamos las librerías necesarias en las versiones correctas\n",
        "\n",
        "%pip install --upgrade torch datasets scikit-learn transformers gensim"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "a6b60a6618dbd12a",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:07:05.113346Z",
          "start_time": "2024-01-17T21:06:47.834079Z"
        },
        "id": "a6b60a6618dbd12a",
        "outputId": "26ec802b-340f-4bc3-b0f4-098efc81fbe8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 471,
          "referenced_widgets": [
            "56d72d09d1c44d1d9d164feaf00f8af5",
            "532d8fde516142fc83cdea0465373b1f",
            "865bdf3b7e374ec3b79e5f709eab102b",
            "76ce06d2bebd4255ba1c0ec4f54484b2",
            "0fc8b0d73dfe4b29a348603b0a925a80",
            "270d58076f5a4d75a4370e5bba9446d8",
            "7e56a7cbad524a4c914044a3012c2157",
            "de1bd213ae9347649601dda8e97a3b37",
            "15f4061437f94aba8a1649cf4851db62",
            "b640fd8d5cd94e298902a906ed6ccfee",
            "3a1a42e20705464ea482df25a8f104e2",
            "ae6f092619b74359a37ae1e36b70e365",
            "d9eb974cc04c47be9165f0fa5176d0ed",
            "65af4478f05f4ff39a9f8ad9773836cf",
            "6b7d7b5109994017b05f818de389d04d",
            "037b7551b385424d91cec8e49170346a",
            "7f7619dc0c9842b9932f892dd86d72ad",
            "e495df792c2f4612b74e77ced1ad16c3",
            "af1bd882e01049f88567d7b840ffa26e",
            "e4e556f090644f38bad0aaa98163ee7b",
            "ec31211b37f04a61bb02e575db34d2f5",
            "e9081f5e3e6f47889b392235f486b95a",
            "f18a839d9172477c90be67215743df2c",
            "883c027905ff4da7b7a40712763c7241",
            "b5b5b61cdb534a7aae322785efcf6441",
            "20f2b302137449b9a7fd11f51ca6048d",
            "652b36d50a3b44a9804bc10fc401f211",
            "b4dd80991d4947048e91a62c03b0b680",
            "104d4449aa85426cb0cb291b2fb2d13d",
            "90667122980a48cc9222551bb62e7870",
            "85e47e688e1c4a7b9a9d88bd41e2e05b",
            "a27d93c6ab944ad2ba130fcad58a1713",
            "43f0c87dc5164293a1dbb9296ba5896c",
            "e1ce142b95f34322a6b920a6fb91388e",
            "a71a6e5d195d464e99aeb911a3628bef",
            "25a454103c6d40a3afec3d45cccdf152",
            "148ca1351e6144e78ea077b92a8c2124",
            "48b2d5727cd54ab4a0236cb53fb09ae2",
            "70b95e3bcad54421a4a559daa0470606",
            "2cbdc678d64f4ec9b93ef24e68a0664f",
            "df571bcd5ffb42cc932327f799bea0a1",
            "fbc31f4c81c44f9d8e5e7406eb8b2df0",
            "716a7eff902943f1a8d1a0f894a58354",
            "078610d46254480d8563f31d90110f0a",
            "b5014fe9a9a24f11a3673253f99ebe30",
            "62ad95d9fee64204a7003049cc676809",
            "581e4ba997dc417fb7929522903991ad",
            "00a2957a63264a37a3afc0642bb8f0b9",
            "921054ad4fac4bce876c075b5213879a",
            "8f88c015fe3d40139b7b60c4db49f516",
            "9f53fba3d2f54dddad18fb44192b422b",
            "cde74a6f3b32447f8bc0d33df06bf1e5",
            "21758c64f99a446b87245aa7bea88fd8",
            "d3215c7e914449f59d7832e9e0ba7ec3",
            "e3ea2bf632f74685b8b71951d346a7da"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "README.md: 0.00B [00:00, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "56d72d09d1c44d1d9d164feaf00f8af5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "data/train-00000-of-00001.parquet:   0%|          | 0.00/18.6M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ae6f092619b74359a37ae1e36b70e365"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "data/test-00000-of-00001.parquet:   0%|          | 0.00/1.23M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f18a839d9172477c90be67215743df2c"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating train split:   0%|          | 0/120000 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e1ce142b95f34322a6b920a6fb91388e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating test split:   0%|          | 0/7600 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b5014fe9a9a24f11a3673253f99ebe30"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['text', 'label'],\n",
              "        num_rows: 120000\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['text', 'label'],\n",
              "        num_rows: 7600\n",
              "    })\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "# Cargamos el conjunto de datos. Se descargará y almacenará automáticamente en local.\n",
        "# Este conjunto de datos contiene noticias de diferentes categorías. En este caso\n",
        "# usaremos las categorías de mundo, deportes, negocios y ciencia ficción/tecnología.\n",
        "\n",
        "# 'ag_news' a secas ya no resuelve: el Hub exige el formato namespace/name\n",
        "dataset = load_dataset('fancyzhx/ag_news')\n",
        "\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "28e10ddd",
      "metadata": {
        "id": "28e10ddd",
        "outputId": "363f41ed-0194-4701-b24c-3b5154a84b4d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'text': \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\", 'label': 2}\n",
            "{'text': Value('string'), 'label': ClassLabel(names=['World', 'Sports', 'Business', 'Sci/Tech'])}\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['World', 'Sports', 'Business', 'Sci/Tech']"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "print(dataset['train'][0])\n",
        "\n",
        "print(dataset['train'].features)\n",
        "\n",
        "classes = dataset['train'].features[\"label\"].names\n",
        "classes"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ba463cc1c41c1298",
      "metadata": {
        "collapsed": false,
        "id": "ba463cc1c41c1298"
      },
      "source": [
        "Automáticamente, la función `load` ha dividido el conjunto de datos en dos conjuntos: uno de train y uno para test. Para acceder a estos conjuntos, usaremos los atributos `train` i `test` del objeto `dataset`. Estos atributos son objetos `data.Dataset` que contiene los ejemplos y etiquetas del conjunto de capacitación y prueba. Para acceder a ejemplos y etiquetas, utilizaremos los atributos `data` y `label` del objeto `data.Dataset`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "3ca34052cf84fbd8",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:07:05.194181Z",
          "start_time": "2024-01-17T21:07:05.122941Z"
        },
        "id": "3ca34052cf84fbd8",
        "outputId": "5a1082b4-f4ea-4402-916a-d84debf126a8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Número de ejemplos de train: 120000\n",
            "Número de ejemplos de test: 7600\n"
          ]
        }
      ],
      "source": [
        "# Separar el conjunto de datos en entrenamiento y test\n",
        "ds_train = dataset['train']\n",
        "ds_test = dataset['test']\n",
        "# Veamos cuántos ejemplos hay en cada set\n",
        "print('Número de ejemplos de train:', len(ds_train))\n",
        "print('Número de ejemplos de test:', len(ds_test))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f3645de832fbdacf",
      "metadata": {
        "collapsed": false,
        "id": "f3645de832fbdacf"
      },
      "source": [
        "Imprimimos los primeros 5 ejemplos del conjunto de entrenamiento.Como podemos ver, cada ejemplo es una noticia y su etiqueta."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "60e7bededaa593f7",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:07:05.480814Z",
          "start_time": "2024-01-17T21:07:05.152188Z"
        },
        "id": "60e7bededaa593f7",
        "outputId": "f89ee862-e12c-4680-8a2c-9d61b8db5f36",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2 (Business) -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
            "2 (Business) -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
            "2 (Business) -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
            "2 (Business) -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
            "2 (Business) -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
          ]
        }
      ],
      "source": [
        "# Imprimimos los primeros 5 ejemplos del conjunto de entrenamiento\n",
        "for w in ds_train.take(5):\n",
        "    print(f\"{w['label']} ({classes[w['label']]}) -> {w['text']}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c1e77e47",
      "metadata": {
        "id": "c1e77e47"
      },
      "source": [
        "## Tokenización\n",
        "\n",
        "La representación del texto en un modelo de idioma requiere que el texto se convierta en números. Si queremos una representación de nivel de palabra, necesitamos hacer dos cosas:\n",
        "* Utilizar un **tokenizador** para dividir el texto en **tokens**.\n",
        "* Construir un **vocabulario** con estos tokens."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "54f8c05f",
      "metadata": {
        "id": "54f8c05f",
        "outputId": "79070fff-3505-41a0-f0fd-3ecc63c47933",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 182,
          "referenced_widgets": [
            "341bb640332d4f3d8ce30461bc7db394",
            "ff8231c6f55d4e28993326f33ae0953f",
            "6d65083f505a4e1292d3a0945f934645",
            "f9d7963140d84ae8ad13b34ab8c3c057",
            "ff566ba3395f49b5ad360d127972a7b1",
            "1ec4e95839b9423f862c73e3ba389908",
            "38d196281f174c1d94ae7048143810a7",
            "8c2f2e3717a1458d871f800ec1a2f66c",
            "73a4d555bc9048b8b7a86e86dd95251f",
            "622c03c617694987b3e5e57aea675a37",
            "651269ac7d2945b6b4f5798d558cd7a5",
            "31a94173eb5148e4bb8618858c1fb59f",
            "b4a5307dbb9a42879f43bab4b47ee288",
            "a2e72a4e10794dcbbd01809b865bac38",
            "d7677041381f4a24a047bc347f3353b5",
            "fe78ed573686418885dceea8fe6aa155",
            "3d161870b8af458383c347dcd5048095",
            "805840e276fc4fa4afa80e1e406ff8e2",
            "6f4f61a619ec4aa695d46b79c5acfce2",
            "a784e5a8171441398d5840cf2f171298",
            "3804ed24021a4b099e864f81d1fb4dd8",
            "497c81c8fbd14e7aaaa4173af8f96937",
            "9140a3ffb12840b1a27c47cf5cc6f4da",
            "0cb7834486b84e1c994f3ff3cdfce8bd",
            "9e2efae12b784f49ab354c034c0f7d60",
            "8a6f82a9bb36445c887f0027a6ef22a2",
            "f2665abfcc1c459c8bfd76b0b0d98491",
            "4b383b77b9694cb0895d78f3bbd30f6f",
            "9fb4be2ed9e948989883401127bbd8d8",
            "7688e7e8597c47ef9fae0e828de6b63c",
            "f4e113d7f3bf4fa9ba4c040df24001d6",
            "383523127c0c468f91c216d24e2fdc69",
            "77c9f365635d4a3588966e3ab11f8979",
            "bfbc78015274456dbc6636d6d98169d8",
            "47a1e6e583bb4d07a54baa1b762af19e",
            "cd655bd27f204438855e6290c08455b5",
            "72424bb76bf44070b40afbe3b9f30d31",
            "1e5d0e40b07544d5b3cbec2a0e291938",
            "acd006f0dd6a4766babe5d7c3fc3fb27",
            "e2a166d9b7004fab940a98a5d8999db9",
            "263ed60471bc43769bee5c972f959516",
            "2b5ea1c77e6e4591b50bef4883fd20f6",
            "8f7492ea6c2d4d599bdac75b40836e48",
            "804bbae253494a2085f42b5132943bf3"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/48.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "341bb640332d4f3d8ce30461bc7db394"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vocab.txt: 0.00B [00:00, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "31a94173eb5148e4bb8618858c1fb59f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json: 0.00B [00:00, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9140a3ffb12840b1a27c47cf5cc6f4da"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/570 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bfbc78015274456dbc6636d6d98169d8"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['-', '-', 'hello', ',', 'how', 'are', 'you', 'doing', 'today', '?']\n",
            "30522\n"
          ]
        }
      ],
      "source": [
        "# Utilizamos el tokenizador de Bert (uno de los primeros modelos de lenguaje basados ​​en transformación) para tokenizar las oraciones\n",
        "\n",
        "from transformers import BertTokenizer\n",
        "\n",
        "tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n",
        "\n",
        "print(tokenizer.tokenize(\"-- Hello, how are you doing today?\"))\n",
        "\n",
        "# Podiamos ver el vocabulario de tokenización\n",
        "vocab = tokenizer.get_vocab()\n",
        "print(len(vocab))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8bc6424e",
      "metadata": {
        "id": "8bc6424e"
      },
      "source": [
        "Usando el tokenizer, también podemos convertir nuestra cadena tokenizada en un conjunto de números:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "169e003b",
      "metadata": {
        "id": "169e003b",
        "outputId": "e6f7c39c-cc57-46f8-8154-4f4385b49f99",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1011, 1011, 7592, 1010, 2129, 2024, 2017, 2725, 2651, 1029]\n"
          ]
        }
      ],
      "source": [
        "tokenizada = tokenizer.tokenize(\"-- Hello, how are you doing today?\")\n",
        "\n",
        "def encode(text):\n",
        "    tk = tokenizer.tokenize(text)\n",
        "    return tokenizer.convert_tokens_to_ids(tk)\n",
        "\n",
        "print(encode(\"-- Hello, how are you doing today?\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8990e6d999d35898",
      "metadata": {
        "collapsed": false,
        "id": "8990e6d999d35898"
      },
      "source": [
        "## Representación del texto\n",
        "\n",
        "Para entrenar un modelo de redes neuronales, necesitamos representar el texto como números. En esta práctica, usaremos la representación de la representación de la bolsa de palabras (BoW) que consiste en representar cada palabra como un número. Esta representación es muy simple y no tiene en cuenta el orden de las palabras o su semántica. Pero es una representación que funciona lo suficientemente bien en muchos casos."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "89259e6a26a7e3ac",
      "metadata": {
        "collapsed": false,
        "id": "89259e6a26a7e3ac"
      },
      "source": [
        "## Representación de la bolsa de las palabras\n",
        "\n",
        "Aunque el significado de las palabras no es fácil de deducir sin poder acceder al contexto, en algunos casos, la representación de la bolsa de palabras puede ser útil.Por ejemplo, en el texto de una noticia, la palabra `covid` puede ser un buen indicador de que las noticias hablan sobre Covid-19 y la palabra `snow` puede ser un buen indicador de que las noticias hablan sobre el tiempo atmosférico.\n",
        "\n",
        "De las técnicas clásicas de vectorización de texto, la más simple es la representación de la bolsa de las palabras (BoW). En esta representación, cada palabra se representa como un número. Para convertir un texto en una representación de BoW, primero creamos un vector con tantos ceros como las palabras están en el vocabulario. Luego, para cada palabra del texto, aumentamos el valor de la posición correspondiente al vector por 1. Por ejemplo, si el texto es `this sentence is a test sentence`, el vector resultante seria `[1, 2, 1, 1, 0, 0, 0, 0, 0, 0, ...]`.\n",
        "\n",
        "Si recordamos la representación one-hot, veremos que la representación de BoW es muy similar. La diferencia es que la representación one-hot será una serie de vectores con un solo 1 y el resto de los valores en 0. En cambio, la representación de BoW será un vector con tantas veces que aparezca cada palabra. Podemos considerar que la representación de BoW sería la suma de vectores únicos.\n",
        "\n",
        "Por ejemplo, si el texto es `this sentence is a test sentence`, el vector one-hot de la primera palabra seria `[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...]` y el vector one-hot de la segunda palabra seria `[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, ...]`. La representación de BoW sería la suma de estos dos vectores: `[1, 1, 0, 0, 0, 0, 0, 0, 0, 0, ...]`.\n",
        "\n",
        "Para generar una representación de BoW, usaremos esta técnica para convertir cada palabra en un vector único y luego agregar todos los vectores. Para hacer esto, usaremos la función `to_bow` que crearemos a continuación. Esta función recibe un texto y devuelve un vector con la representación BoW del texto."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "9d57f10bba32951f",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:07:07.472087Z",
          "start_time": "2024-01-17T21:07:07.133342Z"
        },
        "id": "9d57f10bba32951f",
        "outputId": "0621f4e1-1421-4cc4-fd91-717b07a4dbd6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "vectorizer = CountVectorizer()\n",
        "corpus = [\n",
        "        'I like hot dogs.',\n",
        "        'The dog ran fast.',\n",
        "        'Its hot outside.',\n",
        "    ]\n",
        "vectorizer.fit_transform(corpus)\n",
        "\n",
        "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "616ad0ef",
      "metadata": {
        "id": "616ad0ef"
      },
      "source": [
        "Para calcular el vector BoW de una noticia de nuestro dataset AG_NEWS, podemos usar la siguiente función:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "7990a193",
      "metadata": {
        "id": "7990a193",
        "outputId": "0f4f23ac-70a3-4c97-8262-4f4f4825e39a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'text': \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\", 'label': 2}\n",
            "tensor([0., 0., 0.,  ..., 0., 0., 0.])\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "\n",
        "len_vocab = len(vocab)\n",
        "\n",
        "def to_bow(text, tamano_vocabulario=len_vocab):\n",
        "    res = torch.zeros(tamano_vocabulario, dtype=torch.float32)\n",
        "\n",
        "    for i in encode(text):\n",
        "        if i<tamano_vocabulario:\n",
        "            res[i] += 1\n",
        "    return res\n",
        "\n",
        "print(ds_train[0])\n",
        "print(to_bow(ds_train[0][\"text\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f83bb2a026a54a15",
      "metadata": {
        "collapsed": false,
        "id": "f83bb2a026a54a15"
      },
      "source": [
        "### Entrenamiento de modelos de clasificación BoW\n",
        "\n",
        "Nuestro primer modelo será un clasificador de noticias utilizando la representación de BoW. Para hacer esto, crearemos un modelo de redes neuronales con una capa de entrada con tantas neuronas como las palabras están en nuestro vocabulario y una capa de salida con tantas neuronas como categorías que hay en nuestro conjunto de datos.\n",
        "\n",
        "#### Representación BoW\n",
        "\n",
        "Primero necesitamos convertir el texto en la representación de BoW utilizando la función `to_bow` que hemos creado antes. Esta función recibe un texto y devuelve un vector con la representación BoW del texto.\n",
        "\n",
        "En pytorch se utilizan los `DataLoaders`, para cargar los datos en lotes y convertirlos en tensores de Pytorch. Aprovecharemos esta funcionalidad para convertir los datos de BoW en tensores de PyTorch, Usando el parámetro `collate_fn` del `DataLoader` y proporcionando una función que convierta los datos textuales en tensores de BoW."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "0fd6b204",
      "metadata": {
        "id": "0fd6b204"
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "def bowify(batch):\n",
        "    '''\n",
        "    Esta característica recibe una lista de noticias y devuelve un tensor con las etiquetas\n",
        "    (vector de floats) y otro con las noticias codificadas como BoW (matriz de floats donde cada fila\n",
        "    es un vector de BoW).\n",
        "    '''\n",
        "\n",
        "    # Las etiquetas son 0, 1, 2 o 3.\n",
        "    # Usamos Longtensor porque son enteros.\n",
        "\n",
        "    etiquetas = torch.LongTensor([noticia[\"label\"] for noticia in batch])\n",
        "\n",
        "    # La noticias son tensores de BoW\n",
        "    noticias = torch.stack([to_bow(noticia[\"text\"]) for noticia in batch])\n",
        "\n",
        "    return (\n",
        "            etiquetas,\n",
        "            noticias\n",
        "    )\n",
        "\n",
        "\n",
        "train_loader = DataLoader(ds_train, batch_size=16, collate_fn=bowify)\n",
        "test_loader = DataLoader(ds_test, batch_size=16, collate_fn=bowify)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8e85d6ad8ccb956f",
      "metadata": {
        "collapsed": false,
        "id": "8e85d6ad8ccb956f"
      },
      "source": [
        "### Modelo de classificación\n",
        "\n",
        "Ahora definamos una red neuronal clasificadora simple que contiene una capa lineal. El tamaño del vector de entrada es igual a `vocab_size`, y el tamaño de salida corresponde al número de clases (4). Debido a que estamos resolviendo la tarea de clasificación, la función de activación final es `LogSoftmax()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "9f785d77",
      "metadata": {
        "id": "9f785d77"
      },
      "outputs": [],
      "source": [
        "net = torch.nn.Sequential(\n",
        "    torch.nn.Linear(len(vocab), 4),\n",
        "    torch.nn.LogSoftmax(dim=1)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "803d5e68",
      "metadata": {
        "id": "803d5e68"
      },
      "source": [
        "### Entrenamiento del modelo\n",
        "\n",
        "Ahora definiremos el bucle de entrenamiento estándar de Pytorch. Debido a que nuestro conjunto de datos es bastante grande, con nuestro propósito de enseñanza, entrenaremos solo para una epoca, y a veces incluso por menos de una epoca (especificando el parámetro `epoch_size` nos permite limitar el entrenamiento). También informaremos la precisión del entrenamiento acumulado durante el entrenamiento; La frecuencia de notificación se especifica utilizando el parámetro `report_freq`.\n",
        "\n",
        "Para entrenar el modelo, usaremos el optimizador `Adam`(ya que es uno de los optimizadores más utilizados) y la función de costo `CrossEntropyLoss` (ya que tenemos un problema de calificación con más de dos clases)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "4e46fdbd",
      "metadata": {
        "id": "4e46fdbd",
        "outputId": "d44cc4ea-b74b-4dad-b54d-e59282afc97c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3200: acc=0.7403125\n",
            "6400: acc=0.80359375\n",
            "9600: acc=0.8258333333333333\n",
            "12800: acc=0.841171875\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.03097911379230556, 0.84761460554371)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "def train_epoch(\n",
        "    net,\n",
        "    dataloader,\n",
        "    lr=0.01,\n",
        "    optimizer=None,\n",
        "    loss_fn=torch.nn.NLLLoss(),\n",
        "    epoch_size=None,\n",
        "    report_freq=200,\n",
        "):\n",
        "    # Si no se especifica un optimizador, usamos Adam\n",
        "    optimizer = optimizer or torch.optim.Adam(net.parameters(), lr=lr)\n",
        "\n",
        "    # Ponemos la red en modo de entrenamiento.Esto activa el comportamiento de las capas de DropOut, por ejemplo.\n",
        "    net.train()\n",
        "\n",
        "    # Inicializar las variables que nos servirán para calcular la precisión\n",
        "    total_loss, acc, count, i = 0, 0, 0, 0\n",
        "\n",
        "    # Iteremamos sobre el dataloader\n",
        "    for labels, features in dataloader:\n",
        "\n",
        "        # Ponemos los gradientes a cero\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # calculamos la salida de la red\n",
        "        out = net(features)\n",
        "\n",
        "        # Calculamos la pérdida. Esta función ya se aplica a Softmax a la salida.\n",
        "        loss = loss_fn(out, labels)  # cross_entropy(out,labels)\n",
        "\n",
        "        # Propagamos la pérdida de regreso. Esto hará que se calculen los gradientes .\n",
        "        loss.backward()\n",
        "\n",
        "        # Actualizamos los pesos de la red. Esto toma un paso de optimización.\n",
        "        optimizer.step()\n",
        "\n",
        "        # Actualizamos variables para calcular la precisión.\n",
        "        total_loss += loss\n",
        "\n",
        "        # Calculamos la precisión. Para hacer esto, debemos convertir la salida de red en etiquetas.\n",
        "        # La clase con la mayor probabilidad es la que predecimos como etiqueta.\n",
        "        _, predicted = torch.max(out, 1)\n",
        "        acc += (predicted == labels).sum()\n",
        "\n",
        "        # Actualizamos el contador de muestras\n",
        "        count += len(labels)\n",
        "\n",
        "        # Mostramos la precisión cada report_freq muestras\n",
        "        i += 1\n",
        "        if i % report_freq == 0:\n",
        "            print(f\"{count}: acc={acc.item()/count}\")\n",
        "\n",
        "        # Si se especifica epoch_size y ya hemos procesado este número de muestras, dejamos el bucle.\n",
        "        if epoch_size and count > epoch_size:\n",
        "            break\n",
        "    return total_loss.item() / count, acc.item() / count\n",
        "\n",
        "\n",
        "train_epoch(net, train_loader, epoch_size=15000)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "af3230bbc2532904",
      "metadata": {
        "collapsed": false,
        "id": "af3230bbc2532904"
      },
      "source": [
        "El modelo ha logrado una precisión cercano a `0.85` en el conjunto de entrenamiento; Un número suficientemente aceptable considerando que hemos simplificado el problema para reducir el tiempo de ejecución del tutorial. En un caso real, usaríamos todas las noticias del conjunto de entrenamiento y el modelo sería más preciso."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e5fe580202e44097",
      "metadata": {
        "collapsed": false,
        "id": "e5fe580202e44097"
      },
      "source": [
        "## Representación de Word2Vec\n",
        "\n",
        "La representación de Word2Vec es una representación ampliamente utilizada en el procesamiento del lenguaje natural. Esta representación tiene en cuenta el contexto de las palabras y permite operaciones con las palabras. Por ejemplo, si restamos la palabra vector `king` y sumamos el vector de la palabra `woman`, obtendremos un vector que será muy similar al vector de la plabra `queen`.\n",
        "\n",
        "Para generar representación de Word2Vec, usaremos la librería `gensim`. Esta librería contiene muchos modelos de representación de palabras. En este caso usaremos el modelo `word2vec-google-news-300` que contiene la representación de Word2Vec de 3 millones de palabras y frases.\n",
        "\n",
        "> La primera vez que esta celda se está ejecutando, la función `load` Descargará el modelo de 1.5GB. Esto puede tomar unos minutos.\n",
        "> Esta función devuelve un objeto `KeyedVectors` que contiene la representación Word2Vec."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "12929ab110408820",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:14:40.504566Z",
          "start_time": "2024-01-17T21:12:31.589189Z"
        },
        "id": "12929ab110408820",
        "outputId": "1de761bf-da00-4c3e-8c59-6d42c94abfb0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[==================================================] 100.0% 1662.8/1662.8MB downloaded\n"
          ]
        }
      ],
      "source": [
        "import gensim.downloader as api\n",
        "\n",
        "w2v = api.load('word2vec-google-news-300')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b4b9fa1231614bce",
      "metadata": {
        "collapsed": false,
        "id": "b4b9fa1231614bce"
      },
      "source": [
        "Ahora podemos acceder a la representación de Word2Vec de cada palabra. Por ejemplo, para acceder a la representación de la palabra `king`, usaremos la función `get_vector` del objeto `KeyedVectors`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "fee703eb5efc219a",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:14:41.336216Z",
          "start_time": "2024-01-17T21:14:40.524698Z"
        },
        "id": "fee703eb5efc219a",
        "outputId": "bc5fe455-3395-4b89-9e6c-7291d4a3b4c8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 1.25976562e-01,  2.97851562e-02,  8.60595703e-03,  1.39648438e-01,\n",
              "       -2.56347656e-02, -3.61328125e-02,  1.11816406e-01, -1.98242188e-01,\n",
              "        5.12695312e-02,  3.63281250e-01, -2.42187500e-01, -3.02734375e-01,\n",
              "       -1.77734375e-01, -2.49023438e-02, -1.67968750e-01, -1.69921875e-01,\n",
              "        3.46679688e-02,  5.21850586e-03,  4.63867188e-02,  1.28906250e-01,\n",
              "        1.36718750e-01,  1.12792969e-01,  5.95703125e-02,  1.36718750e-01,\n",
              "        1.01074219e-01, -1.76757812e-01, -2.51953125e-01,  5.98144531e-02,\n",
              "        3.41796875e-01, -3.11279297e-02,  1.04492188e-01,  6.17675781e-02,\n",
              "        1.24511719e-01,  4.00390625e-01, -3.22265625e-01,  8.39843750e-02,\n",
              "        3.90625000e-02,  5.85937500e-03,  7.03125000e-02,  1.72851562e-01,\n",
              "        1.38671875e-01, -2.31445312e-01,  2.83203125e-01,  1.42578125e-01,\n",
              "        3.41796875e-01, -2.39257812e-02, -1.09863281e-01,  3.32031250e-02,\n",
              "       -5.46875000e-02,  1.53198242e-02, -1.62109375e-01,  1.58203125e-01,\n",
              "       -2.59765625e-01,  2.01416016e-02, -1.63085938e-01,  1.35803223e-03,\n",
              "       -1.44531250e-01, -5.68847656e-02,  4.29687500e-02, -2.46582031e-02,\n",
              "        1.85546875e-01,  4.47265625e-01,  9.58251953e-03,  1.31835938e-01,\n",
              "        9.86328125e-02, -1.85546875e-01, -1.00097656e-01, -1.33789062e-01,\n",
              "       -1.25000000e-01,  2.83203125e-01,  1.23046875e-01,  5.32226562e-02,\n",
              "       -1.77734375e-01,  8.59375000e-02, -2.18505859e-02,  2.05078125e-02,\n",
              "       -1.39648438e-01,  2.51464844e-02,  1.38671875e-01, -1.05468750e-01,\n",
              "        1.38671875e-01,  8.88671875e-02, -7.51953125e-02, -2.13623047e-02,\n",
              "        1.72851562e-01,  4.63867188e-02, -2.65625000e-01,  8.91113281e-03,\n",
              "        1.49414062e-01,  3.78417969e-02,  2.38281250e-01, -1.24511719e-01,\n",
              "       -2.17773438e-01, -1.81640625e-01,  2.97851562e-02,  5.71289062e-02,\n",
              "       -2.89306641e-02,  1.24511719e-02,  9.66796875e-02, -2.31445312e-01,\n",
              "        5.81054688e-02,  6.68945312e-02,  7.08007812e-02, -3.08593750e-01,\n",
              "       -2.14843750e-01,  1.45507812e-01, -4.27734375e-01, -9.39941406e-03,\n",
              "        1.54296875e-01, -7.66601562e-02,  2.89062500e-01,  2.77343750e-01,\n",
              "       -4.86373901e-04, -1.36718750e-01,  3.24218750e-01, -2.46093750e-01,\n",
              "       -3.03649902e-03, -2.11914062e-01,  1.25000000e-01,  2.69531250e-01,\n",
              "        2.04101562e-01,  8.25195312e-02, -2.01171875e-01, -1.60156250e-01,\n",
              "       -3.78417969e-02, -1.20117188e-01,  1.15234375e-01, -4.10156250e-02,\n",
              "       -3.95507812e-02, -8.98437500e-02,  6.34765625e-03,  2.03125000e-01,\n",
              "        1.86523438e-01,  2.73437500e-01,  6.29882812e-02,  1.41601562e-01,\n",
              "       -9.81445312e-02,  1.38671875e-01,  1.82617188e-01,  1.73828125e-01,\n",
              "        1.73828125e-01, -2.37304688e-01,  1.78710938e-01,  6.34765625e-02,\n",
              "        2.36328125e-01, -2.08984375e-01,  8.74023438e-02, -1.66015625e-01,\n",
              "       -7.91015625e-02,  2.43164062e-01, -8.88671875e-02,  1.26953125e-01,\n",
              "       -2.16796875e-01, -1.73828125e-01, -3.59375000e-01, -8.25195312e-02,\n",
              "       -6.49414062e-02,  5.07812500e-02,  1.35742188e-01, -7.47070312e-02,\n",
              "       -1.64062500e-01,  1.15356445e-02,  4.45312500e-01, -2.15820312e-01,\n",
              "       -1.11328125e-01, -1.92382812e-01,  1.70898438e-01, -1.25000000e-01,\n",
              "        2.65502930e-03,  1.92382812e-01, -1.74804688e-01,  1.39648438e-01,\n",
              "        2.92968750e-01,  1.13281250e-01,  5.95703125e-02, -6.39648438e-02,\n",
              "        9.96093750e-02, -2.72216797e-02,  1.96533203e-02,  4.27246094e-02,\n",
              "       -2.46093750e-01,  6.39648438e-02, -2.25585938e-01, -1.68945312e-01,\n",
              "        2.89916992e-03,  8.20312500e-02,  3.41796875e-01,  4.32128906e-02,\n",
              "        1.32812500e-01,  1.42578125e-01,  7.61718750e-02,  5.98144531e-02,\n",
              "       -1.19140625e-01,  2.74658203e-03, -6.29882812e-02, -2.72216797e-02,\n",
              "       -4.82177734e-03, -8.20312500e-02, -2.49023438e-02, -4.00390625e-01,\n",
              "       -1.06933594e-01,  4.24804688e-02,  7.76367188e-02, -1.16699219e-01,\n",
              "        7.37304688e-02, -9.22851562e-02,  1.07910156e-01,  1.58203125e-01,\n",
              "        4.24804688e-02,  1.26953125e-01,  3.61328125e-02,  2.67578125e-01,\n",
              "       -1.01074219e-01, -3.02734375e-01, -5.76171875e-02,  5.05371094e-02,\n",
              "        5.26428223e-04, -2.07031250e-01, -1.38671875e-01, -8.97216797e-03,\n",
              "       -2.78320312e-02, -1.41601562e-01,  2.07031250e-01, -1.58203125e-01,\n",
              "        1.27929688e-01,  1.49414062e-01, -2.24609375e-02, -8.44726562e-02,\n",
              "        1.22558594e-01,  2.15820312e-01, -2.13867188e-01, -3.12500000e-01,\n",
              "       -3.73046875e-01,  4.08935547e-03,  1.07421875e-01,  1.06933594e-01,\n",
              "        7.32421875e-02,  8.97216797e-03, -3.88183594e-02, -1.29882812e-01,\n",
              "        1.49414062e-01, -2.14843750e-01, -1.83868408e-03,  9.91210938e-02,\n",
              "        1.57226562e-01, -1.14257812e-01, -2.05078125e-01,  9.91210938e-02,\n",
              "        3.69140625e-01, -1.97265625e-01,  3.54003906e-02,  1.09375000e-01,\n",
              "        1.31835938e-01,  1.66992188e-01,  2.35351562e-01,  1.04980469e-01,\n",
              "       -4.96093750e-01, -1.64062500e-01, -1.56250000e-01, -5.22460938e-02,\n",
              "        1.03027344e-01,  2.43164062e-01, -1.88476562e-01,  5.07812500e-02,\n",
              "       -9.37500000e-02, -6.68945312e-02,  2.27050781e-02,  7.61718750e-02,\n",
              "        2.89062500e-01,  3.10546875e-01, -5.37109375e-02,  2.28515625e-01,\n",
              "        2.51464844e-02,  6.78710938e-02, -1.21093750e-01, -2.15820312e-01,\n",
              "       -2.73437500e-01, -3.07617188e-02, -3.37890625e-01,  1.53320312e-01,\n",
              "        2.33398438e-01, -2.08007812e-01,  3.73046875e-01,  8.20312500e-02,\n",
              "        2.51953125e-01, -7.61718750e-02, -4.66308594e-02, -2.23388672e-02,\n",
              "        2.99072266e-02, -5.93261719e-02, -4.66918945e-03, -2.44140625e-01,\n",
              "       -2.09960938e-01, -2.87109375e-01, -4.54101562e-02, -1.77734375e-01,\n",
              "       -2.79296875e-01, -8.59375000e-02,  9.13085938e-02,  2.51953125e-01],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "w2v.get_vector('king')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ade1217873948a0",
      "metadata": {
        "collapsed": false,
        "id": "ade1217873948a0"
      },
      "source": [
        "También podemos acceder a las palabras más similares a una palabra. Por ejemplo, para acceder a las palabras más similares a la palabra `king`, usaremos la función `most_similar` del objeto `KeyedVectors`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "bffdc3f5aef651db",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:14:57.407515Z",
          "start_time": "2024-01-17T21:14:40.936113Z"
        },
        "id": "bffdc3f5aef651db",
        "outputId": "ccbfba34-96e1-4e8c-a6d0-4b301466f80f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "kings -> 0.7138045430183411\n",
            "queen -> 0.6510956883430481\n",
            "monarch -> 0.6413194537162781\n",
            "crown_prince -> 0.6204220056533813\n",
            "prince -> 0.6159993410110474\n",
            "sultan -> 0.5864824056625366\n",
            "ruler -> 0.5797567367553711\n",
            "princes -> 0.5646552443504333\n",
            "Prince_Paras -> 0.5432944297790527\n",
            "throne -> 0.5422105193138123\n"
          ]
        }
      ],
      "source": [
        "for w, p in w2v.most_similar('king'):\n",
        "    print(f\"{w} -> {p}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "94fa38ee95af3d11",
      "metadata": {
        "collapsed": false,
        "id": "94fa38ee95af3d11"
      },
      "source": [
        "Lo más interesante de la representación de Word2Vec es que los vectores tienen una estructura matemática que nos permite realizar operaciones con las palabras. Por ejemplo, si restamos el vector de la palabra `man` al vector de la palabra `king` y sumamos el vector de la palabra `woman`, obtendremos un vector que será muy similar al vector de la palabra `queen`.\n",
        "\n",
        "$$ KING - MAN + WOMAN = QUEEN $$\n",
        "\n",
        "Para hacer esta operación, usaremos la función `most_similar` del objeto `KeyedVectors` y pasaremos los vectores de las palabras `king`, `woman` y `man`. Esta característica devolverá una lista con las palabras más similares al vector resultante.Como podemos ver, la palabra más similar es `queen`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "4f79029055681603",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-01-17T21:14:57.817533Z",
          "start_time": "2024-01-17T21:14:57.418430Z"
        },
        "id": "4f79029055681603",
        "outputId": "0c97b985-a60e-4653-d844-23de4def2d0f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('queen', 0.7118193507194519)"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "w2v.most_similar(positive=['king', 'woman'], negative=['man'])[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "28a76b6e",
      "metadata": {
        "id": "28a76b6e"
      },
      "source": [
        "### Clasificador de Word2Vec\n",
        "\n",
        "Ahora crearemos un clasificador de noticias utilizando la representación Word2Vec. Primero tendremos que obtener la representación de cada palabra para convertir el texto en vectores. Luego agregaremos todos los vectores para obtener un vector para cada noticia. Este vector será la representación de las noticias.\n",
        "\n",
        "Para convertir un texto en un vector, usaremos la función `to_w2v` que crearemos a continuación. Esta función recibe un texto y devuelve un vector con la representación Word2Vec del texto."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "ce374416",
      "metadata": {
        "id": "ce374416",
        "outputId": "9f11a287-fe1c-40f7-d695-34ffce736c8a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([-17.0809,  11.0404,  -0.9337,  12.4042,  -6.2286,   3.0224, -10.0442,\n",
            "         -8.5156,  -5.9407,   1.1501,  -3.8471,  -8.0006, -18.2444,   4.3982,\n",
            "        -14.2061,  11.0110,  11.2352,  14.8521,  -2.5686,   2.8961, -22.3914,\n",
            "         -3.2182,   9.7872,   0.3238,  -8.6214,   4.2367, -21.9348,   5.7704,\n",
            "         -0.6942,  -1.7075,  -2.4800,   2.1805,  -7.0602, -12.3824, -11.6949,\n",
            "          8.2563, -18.9995,  11.3932,  -7.3198,   7.3370,  -6.1129,  -3.6244,\n",
            "          5.8519,   8.3060,   3.9137,  -1.8091,  -3.2730, -15.8203,  -9.6418,\n",
            "          8.9092, -16.8270,  24.5614,  -2.5387,  21.7112,   6.0571,  14.3324,\n",
            "        -17.4978, -12.2693,   1.1129, -15.9192, -12.1886,  -9.5650, -19.0873,\n",
            "         -7.7948,  -4.9111, -18.4653, -10.2332,  11.3437,  -6.0452,   5.4705,\n",
            "          3.7500,  -9.5068,   4.4747,  -0.2912,  -3.9221,   0.3543,  13.0927,\n",
            "          2.3088,   3.5300, -11.2126, -14.8031,  -2.9008,  -3.4219,  -0.3365,\n",
            "         13.8353,   7.0914,  -5.2219,  22.0132,   4.2657,   5.8488,  -0.5776,\n",
            "         -1.5022,  -5.0004, -13.3813,   4.8757,  10.3992,  -9.8992,  10.6411,\n",
            "         25.6584,  -3.4937,  -6.5989,  -1.0960,  -6.7775,   0.1842,  -6.0798,\n",
            "         13.2260,  -6.2332,   4.9711,   0.8566,   4.1002, -11.5986, -16.8590,\n",
            "         -6.4362,  -3.6979,   4.9203,  15.2933,   7.6364,  -5.8566,  -1.6903,\n",
            "         -2.3312,  12.3486,   7.5709,  -0.6597,   2.7831,  12.6196, -15.9392,\n",
            "         -9.4420,  -1.7229,   7.7839,  10.5602,  -5.9280,  -2.6489,  -6.4361,\n",
            "         -3.8383, -16.0124,   8.0287,  -3.4375,   2.8186,  22.9197,  13.0072,\n",
            "         20.2472,  -6.0054,   4.0575,  -4.7046,  -6.8406, -12.1006,  -4.0645,\n",
            "         18.0959,  -4.8794,   1.5283,   8.0677, -28.4229,  -6.3982,  -4.6095,\n",
            "         -8.2329,  -7.8615,   8.5930,  14.3553,  -2.6136,  -0.7672,   0.5814,\n",
            "          6.9687,   0.7667,   0.1969,  -1.1499,  -4.6281,  15.1071,   5.1883,\n",
            "        -10.1836,   6.9755,  -9.1791,  -8.8102,  -6.4574,  -8.9768,  -1.3514,\n",
            "         20.5255,  22.8086, -22.4160,  -2.1751,  -5.2745,   0.4971,   1.9747,\n",
            "          5.5252,  -4.8856,  -2.1867,   5.9344,  -5.4659,   5.1147,  -3.3837,\n",
            "          5.4895,  12.1746,  -4.1896, -27.1298,  -4.4509,  10.7126,   4.8896,\n",
            "         -6.0110,  -0.7719,   7.8879, -10.4668,  -9.2913,   2.1059, -19.3102,\n",
            "        -10.8101,   5.4989,  -7.4446,  -4.7968,   9.8521,  -3.9826,  14.8542,\n",
            "         16.3674,   7.4929, -11.3996,   1.8357,  -8.1945,   6.2330,  15.2261,\n",
            "         -3.4122, -16.1802,  -2.0000, -12.0552,  11.2962,   5.6537,  -0.8348,\n",
            "         -0.8463,  -6.4080,   5.8111,   2.4668,   1.0925, -14.5064,   1.1021,\n",
            "         -4.3229,  -8.5156,   1.3596,   0.2417,   1.4028,   7.4663,   8.9206,\n",
            "          7.3249,   8.0591,   7.7924,   6.9987,  27.3159,  -4.7353,   0.7053,\n",
            "          6.7754, -12.8845,  13.8699,   5.8623,  -6.8129,   5.8627,   2.7595,\n",
            "          6.3065,   9.9255,  -2.8854, -10.1693,  -7.0736,   7.9216,   2.5093,\n",
            "         -9.5866,   7.4031,  -0.9011,   9.9832,  -2.0049,  -6.3317,   0.4062,\n",
            "          0.0936,   1.1288,  -2.5539,  -2.7307,  -5.4014,  -2.8721,   1.3374,\n",
            "          0.2924,   3.5125,  -8.9189, -15.5585, -13.4326,  -9.4054,   3.7766,\n",
            "        -12.4168,  12.1424,  -1.4987,   0.1738,  -0.9734,   6.0570,  -0.8122,\n",
            "         -3.2520,  -5.7413,  -4.4579,   4.8879,  -0.8176,  -9.8232,   8.6069,\n",
            "         -3.3508, -15.2089, -10.0510,  -1.9859, -10.7878,  18.1031])\n"
          ]
        }
      ],
      "source": [
        "def to_w2v(text):\n",
        "    res = torch.zeros(300, dtype=torch.float32)\n",
        "    for word in text:\n",
        "        if word in w2v:\n",
        "            res += torch.tensor(w2v.get_vector(word))\n",
        "    return res\n",
        "\n",
        "print(to_w2v(ds_train[0][\"text\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5c1b76ac",
      "metadata": {
        "id": "5c1b76ac"
      },
      "source": [
        "Como lo hicimos con la representación de BoW, usaremos el `DataLoaders` de PyTorch para convertir los datos en vectores Word2Vec en tensores Pytorch. Aprovecharemos el parámetro `collate_fn` del `DataLoader` para proporcionar una función que convierta los datos textuales en tensores Word2Vec."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "5a6c88fb",
      "metadata": {
        "id": "5a6c88fb"
      },
      "outputs": [],
      "source": [
        "def w2vify(batch):\n",
        "    etiquetas = torch.LongTensor([noticia[\"label\"] for noticia in batch])\n",
        "    noticias = torch.stack([to_w2v(tokenizer.tokenize(noticia[\"text\"])) for noticia in batch])\n",
        "    return etiquetas, noticias\n",
        "\n",
        "train_loader = DataLoader(ds_train, batch_size=16, collate_fn=w2vify)\n",
        "test_loader = DataLoader(ds_test, batch_size=16, collate_fn=w2vify)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3dffc153",
      "metadata": {
        "id": "3dffc153"
      },
      "source": [
        "### Modelo de clasificación\n",
        "\n",
        "Ahora crearemos el modelo de clasificación usando Pytorch. Definiremos un modelo simple con una capa lineal. El tamaño del vector de entrada será `300` (el tamaño de la representación Word2Vec) y el tamaño de la salida será el número de clases (4). Como estamos resolviendo una tarea de clasificación, la función de activación final será `LogSoftmax()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "fd0fd6b4",
      "metadata": {
        "id": "fd0fd6b4"
      },
      "outputs": [],
      "source": [
        "net = torch.nn.Sequential(\n",
        "    torch.nn.Linear(300, 4),\n",
        "    torch.nn.LogSoftmax(dim=1)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "649418d1",
      "metadata": {
        "id": "649418d1"
      },
      "source": [
        "Finalmente, entrenamos al modelo utilizando el mismo procedimiento que hemos realizado con la representación de BoW."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "55612930",
      "metadata": {
        "id": "55612930",
        "outputId": "0b3b8b97-96b3-4305-b63c-9ca7d4a7a579",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3200: acc=0.73375\n",
            "6400: acc=0.76828125\n",
            "9600: acc=0.7792708333333334\n",
            "12800: acc=0.792578125\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.08423855258966051, 0.7985740938166311)"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "train_epoch(net, train_loader, epoch_size=15000)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7754f4cd2c374fd4",
      "metadata": {
        "collapsed": false,
        "id": "7754f4cd2c374fd4"
      },
      "source": [
        "El resultado no es muy bueno. Esto se debe a que el modelo Word2Vec que utilizamos no tiene las palabras que aparecen en el conjunto de datos. Por ejemplo, si buscamos la palabra `covid`, veremos que no aparece en el modelo.\n",
        "\n",
        "Para resolver este problema, tendremos que usar un modelo Word2Vec entrenado con las palabras del conjunto de datos. Pero esto es muy lento y no lo haremos en este tutorial."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": ".venv",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.11"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}