{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "iKENeWEEwKIb"
},
"source": [
"\n",
"BERT en un problema de modelado de tópicos\n",
"=========================================="
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LV5DTexIybKw"
},
"source": [
"Introducción\n",
"------------"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sT_t9OxYwKIc"
},
"source": [
"Los modelos basados en transformers nos pueden ayudar a resolver varios tipos de problemas. Desde problemas de clasificación y regresión hasta tareas más complejas como resumen de textos o generación de leguaje condicionado. Veremos como aplicar las técnicas de modelado de tópicos (Topic Modeling) utilizando un modelo basado en BERT para nuestro típico problema de clasificación de tweets."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Dcyc_TQ6dis7"
},
"source": [
"### Para ejecutar este notebook"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YWGfRkiHybK1"
},
"source": [
"Para ejecutar este notebook, instale las siguientes librerias:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "YXE4ZBrlybK1",
"outputId": "b991dd62-125c-4a7c-bc63-217fa7fbb4d5"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[K |████████████████████████████████| 10.4 MB 4.6 MB/s \n",
"\u001b[K |████████████████████████████████| 235 kB 45.2 MB/s \n",
"\u001b[K |████████████████████████████████| 184 kB 46.9 MB/s \n",
"\u001b[K |████████████████████████████████| 1.0 MB 42.3 MB/s \n",
"\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
"en-core-web-sm 3.4.0 requires spacy<3.5.0,>=3.4.0, but you have spacy 2.3.5 which is incompatible.\n",
"confection 0.0.2 requires srsly<3.0.0,>=2.4.0, but you have srsly 1.0.5 which is incompatible.\u001b[0m\n",
"\u001b[K |████████████████████████████████| 3.1 MB 5.3 MB/s \n",
"\u001b[K |████████████████████████████████| 831.4 MB 2.5 kB/s \n",
"\u001b[K |████████████████████████████████| 306 kB 57.5 MB/s \n",
"\u001b[K |████████████████████████████████| 90 kB 8.5 MB/s \n",
"\u001b[K |████████████████████████████████| 163 kB 57.8 MB/s \n",
"\u001b[K |████████████████████████████████| 3.3 MB 36.7 MB/s \n",
"\u001b[K |████████████████████████████████| 880 kB 53.6 MB/s \n",
"\u001b[K |████████████████████████████████| 5.2 MB 47.1 MB/s \n",
"\u001b[?25h Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
" Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
" Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
"\u001b[K |████████████████████████████████| 88 kB 5.8 MB/s \n",
"\u001b[K |████████████████████████████████| 85 kB 4.3 MB/s \n",
"\u001b[K |████████████████████████████████| 636 kB 56.0 MB/s \n",
"\u001b[K |████████████████████████████████| 1.3 MB 68.6 MB/s \n",
"\u001b[K |████████████████████████████████| 1.1 MB 59.0 MB/s \n",
"\u001b[K |████████████████████████████████| 19.1 MB 1.2 MB/s \n",
"\u001b[K |████████████████████████████████| 19.1 MB 94.5 MB/s \n",
"\u001b[K |████████████████████████████████| 21.0 MB 1.2 MB/s \n",
"\u001b[K |████████████████████████████████| 23.2 MB 1.4 MB/s \n",
"\u001b[K |████████████████████████████████| 23.3 MB 1.5 MB/s \n",
"\u001b[K |████████████████████████████████| 23.3 MB 15.3 MB/s \n",
"\u001b[K |████████████████████████████████| 22.1 MB 61.0 MB/s \n",
"\u001b[K |████████████████████████████████| 22.1 MB 12.5 MB/s \n",
"\u001b[?25h Building wheel for hdbscan (PEP 517) ... \u001b[?25l\u001b[?25hdone\n",
" Building wheel for sentence-transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Building wheel for umap-learn (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Building wheel for pynndescent (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
"\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
"torchtext 0.13.1 requires torch==1.12.1, but you have torch 1.9.0 which is incompatible.\n",
"torchaudio 0.12.1+cu113 requires torch==1.12.1, but you have torch 1.9.0 which is incompatible.\u001b[0m\n"
]
}
],
"source": [
"!wget https://raw.githubusercontent.com/santiagxf/M72109/master/NLP/Datasets/mascorpus/tweets_marketing.csv \\\n",
" --quiet --no-clobber --directory-prefix ./Datasets/mascorpus/\n",
"\n",
"!wget https://raw.githubusercontent.com/santiagxf/M72109/master/m72109/nlp/normalization.py \\\n",
" --quiet --no-clobber --directory-prefix ./m72109/nlp/\n",
"\n",
"!wget https://raw.githubusercontent.com/santiagxf/M72109/master/docs/nlp/neural/bertopic.txt \\\n",
" --quiet --no-clobber\n",
"\n",
"!wget https://raw.githubusercontent.com/santiagxf/M72109/master/docs/nlp/preprocessing/Normalization.txt \\\n",
" --quiet --no-clobber\n",
"\n",
"!pip install -r Normalization.txt --quiet\n",
"!pip install -r bertopic.txt --quiet"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ntcs1AlpfckX"
},
"outputs": [],
"source": [
"import warnings\n",
"warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_gBXNzwYwKIu"
},
"source": [
"Cargamos el set de datos"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "I8vqJD9JwKIv"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"tweets = pd.read_csv('Datasets/mascorpus/tweets_marketing.csv')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "osRGJ85Lx_3S"
},
"outputs": [],
"source": [
"from m72109.nlp.normalization import TweetTextNormalizer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NxJjAT-RyLKV"
},
"outputs": [],
"source": [
"normalizer = TweetTextNormalizer(lemmatize=False, stem=False, reduce_len=True, strip_handles=True, strip_stopwords=False, strip_urls=True, strip_accents=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tlwyfrwaMDAy"
},
"outputs": [],
"source": [
"docs = normalizer.transform(tweets['TEXTO'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ExV73PU4Ak1I"
},
"source": [
"### Verificando el hardware disponible"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"gather": {
"logged": 1604083146665
},
"id": "ULqO3R6_Am2I",
"outputId": "bd12fc79-7572-4592-83f1-fc6b6a1b490c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Este notebook se está ejecutando en cpu\n"
]
}
],
"source": [
"import torch\n",
"device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
"\n",
"print(\"Este notebook se está ejecutando en\", device)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Hqf5VvQKVko_"
},
"source": [
"## Clustering\n",
"\n",
"Los modelos como BERT son capaces de generar representaciones o embeddings contextualizados para las secuencias de texto que se introducen. En uno de los ejemplos anteriores, entrenamos un modelo de clasificación de tweets. El mismo consistia de una arquitectura basada en transformers + un clasificador.\n",
"\n",
"En este ejemplo, no utilizaremos la capa de clasificación sino que solo nos quedaremos con los embeddings. Tenga en cuenta que, a pesar de que no utilizamos el clasificador (MLP), los embeddings de BERT fueron ajustados (fine-tune) al problema de clasificación de tweets puntualmente. Esto hace que nuestra capacidad de clustering sea más acorde al conjunto de datos.\n",
"\n",
"Para mostrar como funciona, carguemos el modelo de clasificación de tweets. El mismo está publicado en HuggingFace bajo la cuenta de esta materia:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "w9JNa2s5wKI5",
"outputId": "bc554fcc-5960-4832-bc27-4a15989e370d"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of the model checkpoint at fce-m72109/mascorpus-bert-classifier were not used when initializing BertModel: ['classifier.bias', 'classifier.weight']\n",
"- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
"- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
]
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"embedding_model = pipeline(\"feature-extraction\", model=\"fce-m72109/mascorpus-bert-classifier\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pA8jWrPvWVjP"
},
"source": [
"> Note como la tarea que indicamos en el pipeline es `feature-extration`."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A90unty6WzCc"
},
"source": [
"### Clusters según palabras"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Nx4FkysvWa_o"
},
"source": [
"Utilizando la libraria `BERTopic`, podemos computar los embeddings para su visualización:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "s1f5MNLUcqFJ"
},
"outputs": [],
"source": [
"from bertopic import BERTopic\n",
"\n",
"topic_model = BERTopic(language='spanish', embedding_model=embedding_model)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "earA7Mc1cqv_"
},
"outputs": [],
"source": [
"topics, probs = topic_model.fit_transform(docs)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uRFsU9R5WjU6"
},
"source": [
"Podemos ver que la libraría a detectado 28 diferentes tópicos:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "l7aecwz_jxP8",
"outputId": "d7398585-3033-466f-989d-f4e8a4b2f0ce"
},
"outputs": [
{
"data": {
"text/plain": [
"array([-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,\n",
" 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27])"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"\n",
"np.unique(np.asarray(topics))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lDfLvAPbWp3c"
},
"source": [
"Veamos como estos tópicos se agrupan:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 667
},
"id": "HNsU9kokj9hM",
"outputId": "95adf71b-3136-4660-dfef-3e087adf3472"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"topic_model.visualize_topics()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zWT72EosW3C7"
},
"source": [
"### Clusters según oraciones\n",
"\n",
"Podemos también computar los embeddings de todo un documento u oración. La libraría `sentence-transformers` nos da esta facilidad. De igual forma, indicaremos el modelo que entrenamos para la clasificación de tweets:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 394,
"referenced_widgets": [
"2b87eb6e967f463daa7d8d0295e75106",
"b45412b7583d47249636d432e403a958",
"c0975a1bdb53473f812f48238a11d2b2",
"85816b020dc446f88f7ad61586a99155",
"a03f4f73e65c4deb99aa7f7d35b3fcb2",
"6ad15651036a40f8926494da07553b20",
"9c25d9bcb41e4ee3b5c7f32f6d77372d",
"52e7630f83d54184a0c3aeca776d7d14",
"a7a9ab1a3ff84632a071d40f5de4dc4c",
"07b1393bac604efabfbaa9d6e24b7b4e",
"7c40002d60c34373ba2437daa6448904",
"e30d5e61fc374f45932499034a77ab53",
"4e7bf00744b94b7d90121b5b5d3bfdc6",
"74b5a6a79a864510bdd246377d09c6e4",
"bdc9d989d3a047f88668a7e5268d9441",
"62d0f4f5a23d4c71a91f68a24728dc0d",
"16c1ef7628ba4ccebefff13c5ca7e93f",
"8cc92a0c34d749ae92d6af8c7d5bbed4",
"398c1fc9ab0a45f9af098a47f9064054",
"34a94730405146c29a408532c529b28e",
"671a749a69fe484d8f888990f3844569",
"ece524392073472a90e48ab75a45ec63",
"6a18204debe14692a74755612eebbe8b",
"c47c26606903444d95dd76e09aa53966",
"78eb5e47fd7444ba911aac685b988805",
"114c2f0e9f254994875e3ff90d69d363",
"96fcf45bd608416db122f1ba08ac0456",
"fbf2acee4a1140b2af7ca79621e6762d",
"14f94dac266f435a9b480856e409be9c",
"9c4aebb2720143a584e41c064b2ef32c",
"f5b3880bbdb240aebff60f2c3e1669b8",
"7e904f521c4248ea85fe98191ae4cb14",
"eddea1f36ade4e92b18894a2f08f1028",
"f0c68c6f8e134f74bab87625694f1244",
"7451b0bf775b4b7996005f98470a4a31",
"6ebd34c1b43741bdadf6c984e069f7d2",
"40e1311f7e024811ab7da2a8140f9c61",
"25f6b585386243e1b0c10768ac70ca8f",
"2f49397eeb124ac1b13a5a46fdcf529d",
"ba56255f678141e7a218a513c72875b5",
"2bb9ce27eac0453b9d6bfca3e069100c",
"2b04854b3d3a41b78d43f8e126fd4d9f",
"153674f8cd6343b18a6f60ab18eee058",
"7494666469fd436b968c0b1f809aa7a2",
"20dbfbdc0c7f47b2a324b65913624708",
"231ea305b77e47f2829d8f4306244cfe",
"91ec36b738984abdb5d660bba746788a",
"f7d06370b65445f69ad075fb475ba5ca",
"46bf8f4920f845dd8200f5e1a0e4a904",
"822c185215e94a1b9e5ee15173cf313b",
"c9d7088da2124dc1bd0ebeff1d895e51",
"82c90405f19d44c8856e9f269ea32c91",
"68cfad8b2f5947d8bdbf245f78141de1",
"e3a59c29b31e46a4b2aa3b45963993d0",
"f406fdf9e0d345e89536063bcda37606",
"10b46a1400e043b4a980a289d19cce3b",
"349c6ab8e3b24375b30b349a809cb8ad",
"25622fc399574a33841916f23782baa7",
"341ee6f414de4bd8a0911876952dc98f",
"a5fe4475a0274a02a16ea681ded9915c",
"7357d44abda74c019c50d026a9deb71e",
"cc3daab5a0aa425e965b0d01d90a872d",
"e8053d311c24401e85c94f111376484b",
"40b8f11d311844caa0c7a888482d30da",
"e46e2789e6fb4f55b9f48eb7e96fbfad",
"b5dfdf099e694e08a29cae7e0b840812",
"fe7869cf1e71439c8b7cacfef9a0e1dc",
"184ce563f61b4f16b8773b309217ed60",
"f4d8818950934401bd318a0e5f0d9dc9",
"5257cd15caa94e4badb60bbe9c7c866a",
"ced80f5a14f749ff984c1d170db3ea73",
"27079be73cee4c868f1ec079f02fd6ca",
"bd96e524520a4b049006eb620b059e00",
"212454e6b5124fcaabb14825550f6b7d",
"efe5ea412d4046b4be3643d871a31345",
"2b30a4760ad1467eb58bf5ea8dc37918",
"cf05488caa7c4a4898e9191cece0313d",
"d2d875cb832f43ee972e037f4be6892e",
"baa720a1b3bb4a608312f46a9a33c583",
"94ab5577fe4e4db9ac4de405fe5ee396",
"b724340602af4c63b1a82fbe9b5ade6f",
"853110aaea514d99aaf1cfaab922f3d8",
"fb073fb4f0a24ac1943a38c2cccc2cd0",
"d750a270ad3440478ed6e9f0c452419d",
"6c617922e1e64b45883586371d818a9f",
"4f49156ebabd4cf98623f9c47ae95b87",
"7f8e1fadf4e6461f8b5c81c19c62fc33",
"a3aa2671d04a44a39b7761f7b4ce6db1",
"71dd5c60441d41acb301b6e393b78d95",
"3c3eb4569e10491b8ab5b52e3eedc867",
"ee9f17941f4243dca71286af48cd198d",
"846c6eafaf264d4e830e27e916e8e58b",
"a2a92de308f64f05a0949f843f8fd864",
"f61eac81c8874cb9951b14b251bf9a44",
"a2a12bfbaf1342e4bec9199fcf2393be",
"983213fdff5243a38f8b6cc1d8fdbfba",
"793f3e0c7f6b4ee7b29cf95e153050f4",
"02d3822cf6334df2be35e95c866862d2",
"d86ba32492cb4233846b63ac16d81e03"
]
},
"id": "6WIDBqAAtwH8",
"outputId": "5dcb6469-7a06-403d-d8a2-0e0573aec121"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2b87eb6e967f463daa7d8d0295e75106",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/1.17k [00:00\n",
"\n",
"\n",
" \n",
"\n",
"