{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Assignment 3 Ungraded Sections - Part 2: T5 SQuAD Model\n", "\n", "Welcome to the part 2 of testing the models for this week's assignment. This time we will perform decoding using the T5 SQuAD model. In this notebook we'll perform Question Answering by providing a \"Question\", its \"Context\" and see how well we get the \"Target\" answer. \n", "\n", "## Colab\n", "\n", "Since this ungraded lab takes a lot of time to run on coursera, as an alternative we have a colab prepared for you.\n", "\n", "[T5 SQuAD Model Colab](https://drive.google.com/file/d/1c-8KJkTySRGqCx_JjwjvXuRBTNTqEE0N/view?usp=sharing)\n", "\n", "- If you run into a page that looks similar to the one below, with the option `Open with`, this would mean you need to download the `Colaboratory` app. You can do so by `Open with -> Connect more apps -> in the search bar write \"Colaboratory\" -> install`\n", "\n", " \n", "\n", "- After installation it should look like this. Click on `Open with Google Colaboratory`\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Outline\n", "\n", "- [Overview](#0)\n", "- [Part 1: Resuming the assignment (T5 SQuAD Model)](#1)\n", "- [Part 2: Fine-tuning on SQuAD](#2)\n", " - [2.1 Loading in the data and preprocessing](#2.1)\n", " - [2.2 Decoding from a fine-tuned model](#2.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### Overview\n", "\n", "In this notebook you will:\n", "* Implement the Bidirectional Encoder Representation from Transformer (BERT) loss. \n", "* Use a pretrained version of the model you created in the assignment for inference." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# Part 1: Getting ready\n", "\n", "Run the code cells below to import the necessary libraries and to define some functions which will be useful for decoding. The code and the functions are the same as the ones you previsouly ran on the graded assignment." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:tokens_length=568 inputs_length=512 targets_length=114 noise_density=0.15 mean_noise_span_length=3.0 \n" ] } ], "source": [ "import string\n", "import t5\n", "import numpy as np\n", "import trax \n", "from trax.supervised import decoding\n", "import textwrap \n", "\n", "wrapper = textwrap.TextWrapper(width=70)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "PAD, EOS, UNK = 0, 1, 2\n", "\n", "\n", "def detokenize(np_array):\n", " return trax.data.detokenize(\n", " np_array,\n", " vocab_type='sentencepiece',\n", " vocab_file='sentencepiece.model',\n", " vocab_dir='.')\n", "\n", "\n", "def tokenize(s):\n", " return next(trax.data.tokenize(\n", " iter([s]),\n", " vocab_type='sentencepiece',\n", " vocab_file='sentencepiece.model',\n", " vocab_dir='.'))\n", " \n", " \n", "vocab_size = trax.data.vocab_size(\n", " vocab_type='sentencepiece',\n", " vocab_file='sentencepiece.model',\n", " vocab_dir='.')\n", "\n", "\n", "def get_sentinels(vocab_size, display=False):\n", " sentinels = {}\n", " for i, char in enumerate(reversed(string.ascii_letters), 1):\n", " decoded_text = detokenize([vocab_size - i]) \n", " # Sentinels, ex: - \n", " sentinels[decoded_text] = f'<{char}>' \n", " if display:\n", " print(f'The sentinel is <{char}> and the decoded token is:', decoded_text)\n", " return sentinels\n", "\n", "\n", "sentinels = get_sentinels(vocab_size, display=False) \n", "\n", "\n", "def pretty_decode(encoded_str_list, sentinels=sentinels):\n", " # If already a string, just do the replacements.\n", " if isinstance(encoded_str_list, (str, bytes)):\n", " for token, char in sentinels.items():\n", " encoded_str_list = encoded_str_list.replace(token, char)\n", " return encoded_str_list\n", " \n", " # We need to decode and then prettyfy it.\n", " return pretty_decode(detokenize(encoded_str_list))" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "HEoSSKNwgDVA" }, "source": [ "\n", "# Part 2: Fine-tuning on SQuAD\n", "\n", "Now let's try to fine tune on SQuAD and see what becomes of the model.For this, we need to write a function that will create and process the SQuAD `tf.data.Dataset`. Below is how T5 pre-processes SQuAD dataset as a text2text example. Before we jump in, we will have to first load in the data. \n", "\n", "\n", "### 2.1 Loading in the data and preprocessing\n", "\n", "You first start by loading in the dataset. The text2text example for a SQuAD example looks like:\n", "\n", "```json\n", "{\n", " 'inputs': 'question: context:
',\n", " 'targets': '',\n", "}\n", "```\n", "\n", "The squad pre-processing function takes in the dataset and processes it using the sentencePiece vocabulary you have seen above. It generates the features from the vocab and encodes the string features. It takes on question, context, and answer, and returns \"question: Q context: C\" as input and \"A\" as target." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": {}, "colab_type": "code", "id": "RcdR5Dh9UVEw" }, "outputs": [], "source": [ "# Retrieve Question, C, A and return \"question: Q context: C\" as input and \"A\" as target.\n", "def squad_preprocess_fn(dataset, mode='train'):\n", " return t5.data.preprocessors.squad(dataset)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(b'question: What linguistic event in English could we parallel with the Dutch lowering of diphthongs ? context: This change is interesting from a sociolinguistic point of view because it has apparently happened relatively recently , in the 1970s , and was pioneered by older well - educated women from the upper middle classes . The lowering of the diphthongs has long been current in many Dutch dialects , and is comparable to the English Great Vowel Shift , and the diphthongisation of long high vowels in Modern High German , which centuries earlier reached the state now found in Polder Dutch . Stroop theorizes that the lowering of open - mid to open diphthongs is a phonetically \" natural \" and inevitable development and that Dutch , after having diphthongised the long high vowels like German and English , \" should \" have lowered the diphthongs like German and English as well . ',\n", " b'English Great Vowel Shift')" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# train generator, this takes about 1 minute\n", "train_generator_fn, eval_generator_fn = trax.data.tf_inputs.data_streams(\n", " 'squad/plain_text:1.0.0',\n", " data_dir='data/',\n", " bare_preprocess_fn=squad_preprocess_fn,\n", " input_name='inputs',\n", " target_name='targets'\n", ")\n", "\n", "train_generator = train_generator_fn()\n", "next(train_generator)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": {}, "colab_type": "code", "id": "QGQsExH8xv40" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "question: When did the hyperinflation of 1923 occur ? \n", "\n", "context: Europe spent these years rebuilding and coming to terms with the vast human cost of the conflict . The economy of the United States became increasingly intertwined with that of Europe . In Germany , the Weimar Republic gave way to episodes of political and economic turmoil , which culminated with the German hyperinflation of 1923 and the failed Beer Hall Putsch of that same year . When Germany could no longer afford war payments , Wall Street invested heavily in European debts to keep the European economy afloat as a large consumer market for American mass - produced goods . By the middle of the decade , economic development soared in Europe , and the Roaring Twenties broke out in Germany , Britain and France , the second half of the decade becoming known as the \" Golden Twenties \" . In France and francophone Canada , they were also called the \" ann é es folles \" ( \" Crazy Years \" ) . \n", "\n", "target: 1923\n" ] } ], "source": [ "#print example from train_generator\n", "(inp, out) = next(train_generator)\n", "print(inp.decode('utf8').split('context:')[0])\n", "print()\n", "print('context:', inp.decode('utf8').split('context:')[1])\n", "print()\n", "print('target:', out.decode('utf8'))" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "cC3JaiSMpWma" }, "source": [ "\n", "### 2.2 Decoding from a fine-tuned model\n", "\n", "You will now use an existing model that we trained for you. You will initialize, then load in your model, and then try with your own input. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# Initialize the model \n", "model = trax.models.Transformer(\n", " d_ff = 4096,\n", " d_model = 1024,\n", " max_len = 2048,\n", " n_heads = 16,\n", " dropout = 0.1,\n", " input_vocab_size = 32000,\n", " n_encoder_layers = 24,\n", " n_decoder_layers = 24,\n", " mode='predict') # Change to 'eval' for slow decoding." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# load in the model\n", "# this will take a minute\n", "shape11 = trax.shapes.ShapeDtype((1, 1), dtype=np.int32)\n", "model.init_from_file('model_squad.pkl.gz',\n", " weights_only=True, input_signature=(shape11, shape11))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": {}, "colab_type": "code", "id": "FdGy_pHJGEF6" }, "outputs": [], "source": [ "# create inputs\n", "# a simple example \n", "# inputs = 'question: She asked him where is john? context: John was at the game'\n", "\n", "# an extensive example\n", "inputs = 'question: What are some of the colours of a rose? context: A rose is a woody perennial flowering plant of the genus Rosa, in the family Rosaceae, or the flower it bears.There are over three hundred species and tens of thousands of cultivars. They form a group of plants that can be erect shrubs, climbing, or trailing, with stems that are often armed with sharp prickles. Flowers vary in size and shape and are usually large and showy, in colours ranging from white through yellows and reds. Most species are native to Asia, with smaller numbers native to Europe, North America, and northwestern Africa. Species, cultivars and hybrids are all widely grown for their beauty and often are fragrant.'" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ 822 10 363 33 128 13 8 6548 13 3 9 4659\n", " 58 2625 10 71 4659 19 3 9 1679 63 24999 5624\n", " 53 1475 13 8 3 729 302 15641 6 16 8 384\n", " 15641 8433 15 6 42 8 5624 34 4595 7 5 7238\n", " 33 147 386 6189 3244 11 3 324 7 13 2909 13\n", " 10357 291 7 5 328 607 3 9 563 13 2677 24\n", " 54 36 3 15 12621 21675 7 6 11908 6 42 5032\n", " 53 6 28 6269 7 24 33 557 3 8715 28 4816\n", " 3 2246 19376 7 5 20294 5215 16 812 11 2346 11\n", " 33 1086 508 11 504 63 6 16 6548 3 6836 45\n", " 872 190 4459 7 11 1131 7 5 1377 3244 33 4262\n", " 12 3826 6 28 2755 2302 4262 12 1740 6 1117 1371\n", " 6 11 3457 24411 2648 5 3 7727 725 6 10357 291\n", " 7 11 9279 7 33 66 5456 4503 21 70 2790 11\n", " 557 33 29346 5]\n" ] } ], "source": [ "# tokenizing the input so we could feed it for decoding\n", "print(tokenize(inputs))\n", "test_inputs = tokenize(inputs) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Run the cell below to decode.\n", "\n", "### Note: This will take some time to run" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "c_CwYjXHIQOJ" }, "outputs": [], "source": [ "# Temperature is a parameter for sampling.\n", "# # * 0.0: same as argmax, always pick the most probable token\n", "# # * 1.0: sampling from the distribution (can sometimes say random things)\n", "# # * values inbetween can trade off diversity and quality, try it out!\n", "output = decoding.autoregressive_sample(model, inputs=np.array(test_inputs)[None, :],\n", " temperature=0.0, max_length=5) # originally max_length=10\n", "print(wrapper.fill(pretty_decode(output[0])))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should also be aware that the quality of the decoding is not very good because max_length was downsized from 10 to 5 so that this runs faster within this environment. The colab version uses the original max_length so check that one for the actual decoding." ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "name": "C4W3-solutions.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }