{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "accelerator": "GPU", "colab": { "name": "Copy of transformers_joint_intent_classification_slot_filling.ipynb", "provenance": [], "collapsed_sections": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "bBb1hsPFt8lV" }, "source": [ "# Joint Intent Classification and Slot Filling with Transformers\n", "\n", "The goal of this notebook is to fine-tune a pretrained transformer-based neural network model to convert a user query expressed in English into\n", "a representation that is structured enough to be processed by an automated service.\n", "\n", "Here is an example of interpretation computed by such a Natural Language Understanding system:\n", "\n", "```python\n", ">>> nlu(\"Book a table for two at Le Ritz for Friday night\",\n", " tokenizer, joint_model, intent_names, slot_names)\n", "```\n", "```\n", "{\n", " 'intent': 'BookRestaurant',\n", " 'slots': {\n", " 'party_size_number': 'two',\n", " 'restaurant_name': 'Le Ritz',\n", " 'timeRange': 'Friday night'\n", " }\n", "}\n", "```\n", "\n", "Intent classification is a simple sequence classification problem. The trick is to treat the structured knowledge extraction part (\"Slot Filling\") as token-level classification problem using BIO-annotations:\n", "\n", "```python\n", ">>> show_predictions(\"Book a table for two at Le Ritz for Friday night!\",\n", "... tokenizer, joint_model, intent_names, slot_names)\n", "```\n", "```\n", "## Intent: BookRestaurant\n", "## Slots:\n", " Book : O\n", " a : O\n", " table : O\n", " for : O\n", " two : B-party_size_number\n", " at : O\n", " Le : B-restaurant_name\n", " R : I-restaurant_name\n", " ##itz : I-restaurant_name\n", " for : O\n", " Friday : B-timeRange\n", " night : I-timeRange\n", " ! : O\n", "```\n", "\n", "We will show how to train a such as join \"sequence classification\" and \"token classification\" joint model on a [voice command dataset]() published by snips.ai.\n", "\n", "\n", "This notebook is a partial reproduction of some of the results presented in this paper:\n", "\n", "BERT for Joint Intent Classification and Slot Filling\n", "Qian Chen, Zhu Zhuo, Wen Wang\n", "\n", "https://arxiv.org/abs/1902.10909" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "id": "meK0JpbEPAAA", "outputId": "7762859c-18c7-42c4-b22e-5bd940500da0" }, "source": [ "import tensorflow as tf\n", "tf.__version__" ], "execution_count": 1, "outputs": [ { "output_type": "execute_result", "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'2.4.1'" ] }, "metadata": { "tags": [] }, "execution_count": 1 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0pualy2lUWAs", "outputId": "bfcf9fcd-a573-4e4b-93ef-985137442fd7" }, "source": [ "!nvidia-smi" ], "execution_count": 2, "outputs": [ { "output_type": "stream", "text": [ "Wed Mar 17 23:11:37 2021 \n", "+-----------------------------------------------------------------------------+\n", "| NVIDIA-SMI 460.56 Driver Version: 460.32.03 CUDA Version: 11.2 |\n", "|-------------------------------+----------------------+----------------------+\n", "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", "| | | MIG M. |\n", "|===============================+======================+======================|\n", "| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |\n", "| N/A 56C P8 10W / 70W | 0MiB / 15109MiB | 0% Default |\n", "| | | N/A |\n", "+-------------------------------+----------------------+----------------------+\n", " \n", "+-----------------------------------------------------------------------------+\n", "| Processes: |\n", "| GPU GI CI PID Type Process name GPU Memory |\n", "| ID ID Usage |\n", "|=============================================================================|\n", "| No running processes found |\n", "+-----------------------------------------------------------------------------+\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "S7meRWCzubcW" }, "source": [ "# TODO: update this notebook to work with the latest version of transformers\n", "%pip install -q transformers==2.11.0" ], "execution_count": 3, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "GmHZ3QTAcvOt" }, "source": [ "## The Data\n", "\n", "We will use a speech command dataset collected, annotated and published by French startup SNIPS.ai (bought in 2019 by Audio device manufacturer Sonos).\n", "\n", "The original dataset comes in [YAML format with inline markdown annotations](https://snips-nlu.readthedocs.io/en/latest/dataset.html).\n", "\n", "Instead we will use a preprocessed variant with token level B-I-O annotations closer the representation our model will predict. This variant of the SNIPS\n", "dataset was prepared by [Su Zhu](https://github.com/sz128).\n", "\n" ] }, { "cell_type": "code", "metadata": { "id": "xuJFBFbWOW7t" }, "source": [ "from urllib.request import urlretrieve\n", "from pathlib import Path\n", "\n", "\n", "SNIPS_DATA_BASE_URL = (\n", " \"https://github.com/ogrisel/slot_filling_and_intent_detection_of_SLU/blob/\"\n", " \"master/data/snips/\"\n", ")\n", "for filename in [\"train\", \"valid\", \"test\", \"vocab.intent\", \"vocab.slot\"]:\n", " path = Path(filename)\n", " if not path.exists():\n", " print(f\"Downloading {filename}...\")\n", " urlretrieve(SNIPS_DATA_BASE_URL + filename + \"?raw=true\", path)" ], "execution_count": 4, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "-Xo3f5tNjJCL" }, "source": [ "Let's have a look at the first lines from the training set:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "o4mCRg2UQr1f", "outputId": "3988ad09-b04e-4de2-864e-b80eb5afb74d" }, "source": [ "lines_train = Path(\"train\").read_text(\"utf-8\").strip().splitlines()\n", "lines_train[:5]" ], "execution_count": 5, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['Add:O Don:B-entity_name and:I-entity_name Sherri:I-entity_name to:O my:B-playlist_owner Meditate:B-playlist to:I-playlist Sounds:I-playlist of:I-playlist Nature:I-playlist playlist:O <=> AddToPlaylist',\n", " 'put:O United:B-entity_name Abominations:I-entity_name onto:O my:B-playlist_owner rare:B-playlist groove:I-playlist playlist:O <=> AddToPlaylist',\n", " 'add:O the:O tune:B-music_item by:O misato:B-artist watanabe:I-artist to:O the:O Trapeo:B-playlist playlist:O <=> AddToPlaylist',\n", " 'add:O this:O artist:B-music_item to:O my:B-playlist_owner this:B-playlist is:I-playlist miguel:I-playlist bosé:I-playlist playlist:O <=> AddToPlaylist',\n", " 'add:O heresy:B-entity_name and:I-entity_name the:I-entity_name hotel:I-entity_name choir:I-entity_name to:O the:O evening:B-playlist acoustic:I-playlist playlist:O <=> AddToPlaylist']" ] }, "metadata": { "tags": [] }, "execution_count": 5 } ] }, { "cell_type": "markdown", "metadata": { "id": "ezTIAec5jNlq" }, "source": [ "Some remarks:\n", "\n", "- The class label for the voice command appears at the end of each line (after the \"<=>\" marker).\n", "- Each word-level token is annotated with B-I-O labels using the \":\" separator.\n", "- B/I/O stand for \"Beginning\" / \"Inside\" / \"Outside\"\n", "- \"Add:O\" means that the token \"Add\" is \"Outside\" of any annotation span\n", "- \"Don:B-entity_name\" means that \"Don\" is the \"Beginning\" of an annotation of type \"entity-name\".\n", "- \"and:I-entity_name\" means that \"and\" is \"Inside\" the previously started annotation of type \"entity-name\".\n", "\n", "\n", "Let's write a parsing function and test it on the first line:" ] }, { "cell_type": "code", "metadata": { "id": "PMzOqdGRWP2u" }, "source": [ "def parse_line(line):\n", " utterance_data, intent_label = line.split(\" <=> \")\n", " items = utterance_data.split()\n", " words = [item.rsplit(\":\", 1)[0]for item in items]\n", " word_labels = [item.rsplit(\":\", 1)[1]for item in items]\n", " return {\n", " \"intent_label\": intent_label,\n", " \"words\": \" \".join(words),\n", " \"word_labels\": \" \".join(word_labels),\n", " \"length\": len(words),\n", " }" ], "execution_count": 6, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-Sg-hZOJXUPG", "outputId": "635c3bc4-e208-4569-bfed-6f46cefa0458" }, "source": [ "parse_line(lines_train[0])" ], "execution_count": 7, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "{'intent_label': 'AddToPlaylist',\n", " 'length': 12,\n", " 'word_labels': 'O B-entity_name I-entity_name I-entity_name O B-playlist_owner B-playlist I-playlist I-playlist I-playlist I-playlist O',\n", " 'words': 'Add Don and Sherri to my Meditate to Sounds of Nature playlist'}" ] }, "metadata": { "tags": [] }, "execution_count": 7 } ] }, { "cell_type": "markdown", "metadata": { "id": "hQAwISaFX0zZ" }, "source": [ "This utterance is a voice command of type \"AddToPlaylist\" with to annotations:\n", "\n", "- an entity-name: \"Don and Sherri\",\n", "- a playlist: \"Medidate to Sounds of Nature\".\n", "\n", "\n", "The goal of this project is to build a baseline Natural Understanding model to analyse such voice commands and predict:\n", "\n", "- the intent of the speaker: the sentence level class label (\"AddToPlaylist\");\n", "- extract the interesting \"slots\" (typed named entities) from the sentence by performing word level classification using the B-I-O tags as target classes. This second task is often referred to as \"NER\" (Named Entity Recognition) in the Natural Language Processing literature. Alternatively this is also known as \"slot filling\" when we expect a fixed set of named entity per sentence of a given class.\n", "\n", "The list of possible classes for the sentence level and the word level classification problems are given as:\n", "\n", "\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "q98BNyqOmM1b", "outputId": "16911831-ea54-4895-92f5-cc2c07042b1a" }, "source": [ "print(Path(\"vocab.intent\").read_text(\"utf-8\"))" ], "execution_count": 8, "outputs": [ { "output_type": "stream", "text": [ "AddToPlaylist\n", "BookRestaurant\n", "GetWeather\n", "PlayMusic\n", "RateBook\n", "SearchCreativeWork\n", "SearchScreeningEvent\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "u3h7lLfDmXyn", "outputId": "8afa77a3-d7ed-4d23-8074-e6b960457d7a" }, "source": [ "print(Path(\"vocab.slot\").read_text(\"utf-8\"))" ], "execution_count": 9, "outputs": [ { "output_type": "stream", "text": [ "B-album\n", "B-artist\n", "B-best_rating\n", "B-city\n", "B-condition_description\n", "B-condition_temperature\n", "B-country\n", "B-cuisine\n", "B-current_location\n", "B-entity_name\n", "B-facility\n", "B-genre\n", "B-geographic_poi\n", "B-location_name\n", "B-movie_name\n", "B-movie_type\n", "B-music_item\n", "B-object_location_type\n", "B-object_name\n", "B-object_part_of_series_type\n", "B-object_select\n", "B-object_type\n", "B-party_size_description\n", "B-party_size_number\n", "B-playlist\n", "B-playlist_owner\n", "B-poi\n", "B-rating_unit\n", "B-rating_value\n", "B-restaurant_name\n", "B-restaurant_type\n", "B-served_dish\n", "B-service\n", "B-sort\n", "B-spatial_relation\n", "B-state\n", "B-timeRange\n", "B-track\n", "B-year\n", "I-album\n", "I-artist\n", "I-city\n", "I-country\n", "I-cuisine\n", "I-current_location\n", "I-entity_name\n", "I-facility\n", "I-genre\n", "I-geographic_poi\n", "I-location_name\n", "I-movie_name\n", "I-movie_type\n", "I-music_item\n", "I-object_location_type\n", "I-object_name\n", "I-object_part_of_series_type\n", "I-object_select\n", "I-object_type\n", "I-party_size_description\n", "I-playlist\n", "I-playlist_owner\n", "I-poi\n", "I-restaurant_name\n", "I-restaurant_type\n", "I-served_dish\n", "I-service\n", "I-sort\n", "I-spatial_relation\n", "I-state\n", "I-timeRange\n", "I-track\n", "O\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "aIflBkyomf9R" }, "source": [ "\"POI\" stands for \"Point of Interest\".\n", "\n", "Let's parse all the lines and store the results in pandas DataFrames:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 424 }, "id": "rI3gFdEvQvbG", "outputId": "6b6341cd-c008-422a-c0f1-316566f2693d" }, "source": [ "import pandas as pd\n", "\n", "parsed = [parse_line(line) for line in lines_train]\n", "\n", "df_train = pd.DataFrame([p for p in parsed if p is not None])\n", "df_train" ], "execution_count": 10, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", " | intent_label | \n", "words | \n", "word_labels | \n", "length | \n", "
---|---|---|---|---|
0 | \n", "AddToPlaylist | \n", "Add Don and Sherri to my Meditate to Sounds of... | \n", "O B-entity_name I-entity_name I-entity_name O ... | \n", "12 | \n", "
1 | \n", "AddToPlaylist | \n", "put United Abominations onto my rare groove pl... | \n", "O B-entity_name I-entity_name O B-playlist_own... | \n", "8 | \n", "
2 | \n", "AddToPlaylist | \n", "add the tune by misato watanabe to the Trapeo ... | \n", "O O B-music_item O B-artist I-artist O O B-pla... | \n", "10 | \n", "
3 | \n", "AddToPlaylist | \n", "add this artist to my this is miguel bosé play... | \n", "O O B-music_item O B-playlist_owner B-playlist... | \n", "10 | \n", "
4 | \n", "AddToPlaylist | \n", "add heresy and the hotel choir to the evening ... | \n", "O B-entity_name I-entity_name I-entity_name I-... | \n", "11 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
13079 | \n", "SearchScreeningEvent | \n", "find a Consolidated Theatres showing The Good ... | \n", "O O B-location_name I-location_name O B-movie_... | \n", "10 | \n", "
13080 | \n", "SearchScreeningEvent | \n", "where can i see animated movies in the neighbo... | \n", "O O O O B-movie_type I-movie_type B-spatial_re... | \n", "9 | \n", "
13081 | \n", "SearchScreeningEvent | \n", "Showtimes for animated movies in the area . | \n", "O O B-movie_type I-movie_type B-spatial_relati... | \n", "8 | \n", "
13082 | \n", "SearchScreeningEvent | \n", "Which animated movies are playing at Megaplex ... | \n", "O B-movie_type I-movie_type O O O B-location_n... | \n", "11 | \n", "
13083 | \n", "SearchScreeningEvent | \n", "What movie schedules start at sunset ? | \n", "O B-object_type I-object_type O O B-timeRange O | \n", "7 | \n", "
13084 rows × 4 columns
\n", "\n", " | intent_label | \n", "words | \n", "word_labels | \n", "length | \n", "
---|---|---|---|---|
0 | \n", "AddToPlaylist | \n", "Add Don and Sherri to my Meditate to Sounds of... | \n", "O B-entity_name I-entity_name I-entity_name O ... | \n", "12 | \n", "
1 | \n", "AddToPlaylist | \n", "put United Abominations onto my rare groove pl... | \n", "O B-entity_name I-entity_name O B-playlist_own... | \n", "8 | \n", "
2 | \n", "AddToPlaylist | \n", "add the tune by misato watanabe to the Trapeo ... | \n", "O O B-music_item O B-artist I-artist O O B-pla... | \n", "10 | \n", "
3 | \n", "AddToPlaylist | \n", "add this artist to my this is miguel bosé play... | \n", "O O B-music_item O B-playlist_owner B-playlist... | \n", "10 | \n", "
4 | \n", "AddToPlaylist | \n", "add heresy and the hotel choir to the evening ... | \n", "O B-entity_name I-entity_name I-entity_name I-... | \n", "11 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
13079 | \n", "SearchScreeningEvent | \n", "find a Consolidated Theatres showing The Good ... | \n", "O O B-location_name I-location_name O B-movie_... | \n", "10 | \n", "
13080 | \n", "SearchScreeningEvent | \n", "where can i see animated movies in the neighbo... | \n", "O O O O B-movie_type I-movie_type B-spatial_re... | \n", "9 | \n", "
13081 | \n", "SearchScreeningEvent | \n", "Showtimes for animated movies in the area . | \n", "O O B-movie_type I-movie_type B-spatial_relati... | \n", "8 | \n", "
13082 | \n", "SearchScreeningEvent | \n", "Which animated movies are playing at Megaplex ... | \n", "O B-movie_type I-movie_type O O O B-location_n... | \n", "11 | \n", "
13083 | \n", "SearchScreeningEvent | \n", "What movie schedules start at sunset ? | \n", "O B-object_type I-object_type O O B-timeRange O | \n", "7 | \n", "
13084 rows × 4 columns
\n", "\n", " | words | \n", "word_labels | \n", "length | \n", "
---|---|---|---|
intent_label | \n", "\n", " | \n", " | \n", " |
AddToPlaylist | \n", "1842 | \n", "1842 | \n", "1842 | \n", "
BookRestaurant | \n", "1873 | \n", "1873 | \n", "1873 | \n", "
GetWeather | \n", "1900 | \n", "1900 | \n", "1900 | \n", "
PlayMusic | \n", "1900 | \n", "1900 | \n", "1900 | \n", "
RateBook | \n", "1856 | \n", "1856 | \n", "1856 | \n", "
SearchCreativeWork | \n", "1854 | \n", "1854 | \n", "1854 | \n", "
SearchScreeningEvent | \n", "1859 | \n", "1859 | \n", "1859 | \n", "