{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "2653fc4e43550c6c822c716ac5ae5402", "grade": false, "grade_id": "cell-d6207702596e12c3", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "# Part 4: Convolutional Sequence Classification" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "f5684b617ecf276425277bfd0443bf60", "grade": false, "grade_id": "cell-c64c5a9c41bae198", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "__Before starting, we recommend you enable GPU acceleration if you're running on Colab.__" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "99c911e0f995f0e739924832d8f7ae6a", "grade": false, "grade_id": "cell-0ff33f04036d9e18", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "# Execute this code block to install dependencies when running on colab\n", "try:\n", " import torch\n", "except:\n", " from os.path import exists\n", " from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n", " platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n", " cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\\.\\([0-9]*\\)\\.\\([0-9]*\\)$/cu\\1\\2/'\n", " accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'\n", "\n", " !pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision\n", "\n", "try: \n", " import torchbearer\n", "except:\n", " !pip install torchbearer\n", "\n", "try:\n", " import torchtext\n", "except:\n", " !pip install torchtext\n", " \n", "try:\n", " import spacy\n", "except:\n", " !pip install spacy\n", " \n", "try:\n", " spacy.load('en')\n", "except:\n", " !python -m spacy download en" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "4205da1995e90b0f195240b8abb6d550", "grade": false, "grade_id": "cell-6d109dcd5c757094", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "# 4 - Convolutional Sentiment Analysis\n", "\n", "In the previous part of the lab we used RNNs to build a simple text classification model. In this final part of the lab, we will be using a *convolutional neural network* (CNN) to conduct sentiment analysis, implementing the model from [Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882).\n", "\n", "Why use CNNs on text? In the same way that a 3x3 filter can look over a patch of an image, a 1x2 filter can look over 2 sequential words in a piece of text, i.e. a bi-gram. In this CNN model we will use multiple filters of different sizes which will look at the bi-grams (a 1x2 filter), tri-grams (a 1x3 filter) and/or n-grams (a 1x$n$ filter) within the text. The intuition here is that the appearance of certain bi-grams, tri-grams and n-grams within the review will be a good indication of the final sentiment.\n", "\n", "In the past couple of years there has been a real move away from recurrent models for sequence tasks, and with the advent of models like WaveNet and self-attention in transformer models performance for real world problems like language translation and text classification has exceeded that of recurrent models." ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "cfdb609fdf2efe6a11212c3eb43230c3", "grade": false, "grade_id": "cell-9089b12940c07b64", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "## Preparing Data\n", "\n", "As in the previous part, we'll prepare the IMDB data by tokenising with spacy. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "bd418603e3153adf7316ba0b070382ae", "grade": false, "grade_id": "cell-e0319042c512ca1c", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "import torch\n", "from torchtext import data\n", "from torchtext import datasets\n", "\n", "TEXT = data.Field(tokenize='spacy')\n", "LABEL = data.LabelField(dtype=torch.float)\n", "\n", "train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)\n", "train_data, valid_data = train_data.split()" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "677b35e079ff6e0b74f4489dc15350d1", "grade": false, "grade_id": "cell-ccb2988e76995250", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "We'll build the vocab and load pre-trained GloVe word embeddings, rather than starting with random embeddings which we must learn." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "761dcdc5fcfeeacb2d6d8ed9469aa7a0", "grade": false, "grade_id": "cell-32616067b259247a", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "TEXT.build_vocab(train_data, max_size=25000, vectors=\"glove.6B.100d\")\n", "LABEL.build_vocab(train_data)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "54af62d5bb2044913edf5e581a1aa9f5", "grade": false, "grade_id": "cell-3b3018b69f18bfbd", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "As before, we create the iterators (this time we don't need to worry about sorting by length within the batch)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "ea75324b593c38921117bb26748a2d4f", "grade": false, "grade_id": "cell-f2e19bb0447e5ef3", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "BATCH_SIZE = 64\n", "\n", "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "\n", "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n", " (train_data, valid_data, test_data), \n", " batch_size=BATCH_SIZE, \n", " device=device)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "6ee3f348721c2b829fda0e0a5507966b", "grade": false, "grade_id": "cell-f4fe1af026ed6c4c", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "## Build the Model\n", "\n", "Now to build our model.\n", "\n", "The first major hurdle is visualizing how CNNs are used for text. Images are typically 2 dimensional (we'll ignore the fact that there is a third \"colour\" dimension for now) whereas text is 1 dimensional. However, we know that the first step in almost all of our previous tutorials (and pretty much all NLP pipelines) is converting the words into word embeddings. This is how we can visualize our words in 2 dimensions, each word along one axis and the elements of vectors aross the other dimension. Consider the 2 dimensional representation of the embedded sentence below:\n", "\n", "![](http://comp6248.ecs.soton.ac.uk/labs/lab7/assets/sentiment9.png)\n", "\n", "We can then use a filter that is **[n x emb_dim]**. This will cover $n$ sequential words entirely, as their width will be `emb_dim` dimensions. Consider the image below, with our word vectors represented in green. Here we have 4 words with 5 dimensional embeddings, creating a [4x5] \"image\" tensor. A filter that covers two words at a time (i.e. bi-grams) will be **[2x5]** filter, shown in yellow, and each element of the filter with have a _weight_ associated with it. The output of this filter (shown in red) will be a single real number that is the weighted sum of all elements covered by the filter.\n", "\n", "![](http://comp6248.ecs.soton.ac.uk/labs/lab7/assets/sentiment12.png)\n", "\n", "The filter then moves \"down\" the image (or across the sentence) to cover the next bi-gram and another output (weighted sum) is calculated. \n", "\n", "![](http://comp6248.ecs.soton.ac.uk/labs/lab7/assets/sentiment13.png)\n", "\n", "Finally, the filter moves down again and the final output for this filter is calculated.\n", "\n", "![](http://comp6248.ecs.soton.ac.uk/labs/lab7/assets/sentiment14.png)\n", "\n", "In our case (and in the general case where the width of the filter equals the width of the \"image\"), our output will be a vector with number of elements equal to the height of the image (or length of the sentence) minus the height of the filter plus one, $4-2+1=3$ in this case.\n", "\n", "This example showed how to calculate the output of one filter. Our model (and pretty much all CNNs) will have lots of these filters. The idea is that each filter will learn a different feature to extract. In the above example, we are hoping each of the **[2 x emb_dim]** filters will be looking for the occurence of different bi-grams. \n", "\n", "In our model, we will also have different sizes of filters, heights of 3, 4 and 5, with 100 of each of them. The intuition is that we will be looking for the occurence of different tri-grams, 4-grams and 5-grams that are relevant for analysing sentiment of movie reviews.\n", "\n", "The next step in our model is to use *pooling* (specifically *max pooling*) on the output of the convolutional layers. This is similar to the FastText model where we performed the average over each of the word vectors, implemented by the `F.avg_pool2d` function, however instead of taking the average over a dimension, we are taking the maximum value over a dimension. Below an example of taking the maximum value (0.9) from the output of the convolutional layer on the example sentence (not shown is the activation function applied to the output of the convolutions).\n", "\n", "![](http://comp6248.ecs.soton.ac.uk/labs/lab7/assets/sentiment15.png)\n", "\n", "The idea here is that the maximum value is the \"most important\" feature for determining the sentiment of the review, which corresponds to the \"most important\" n-gram within the review. How do we know what the \"most important\" n-gram is? Luckily, we don't have to! Through backpropagation, the weights of the filters are changed so that whenever certain n-grams that are highly indicative of the sentiment are seen, the output of the filter is a \"high\" value. This \"high\" value then passes through the max pooling layer if it is the maximum value in the output. \n", "\n", "As our model has 100 filters of 3 different sizes, that means we have 300 different n-grams the model thinks are important. We concatenate these together into a single vector and pass them through a linear layer to predict the sentiment. We can think of the weights of this linear layer as \"weighting up the evidence\" from each of the 300 n-grams and making a final decision. \n", "\n", "### Implementation Details\n", "\n", "We implement the convolutional layers with `nn.Conv2d`. The `in_channels` argument is the number of \"channels\" in your image going into the convolutional layer. In actual images this is usually 3 (one channel for each of the red, blue and green channels), however when using text we only have a single channel, the text itself. The `out_channels` is the number of filters and the `kernel_size` is the size of the filters. Each of our `kernel_size`s is going to be **[n x emb_dim]** where $n$ is the size of the n-grams.\n", "\n", "In PyTorch, RNNs want the input with the batch dimension second, whereas CNNs want the batch dimension first. Thus, the first thing we do to our input is `permute` it to make it the correct shape. We then pass the sentence through an embedding layer to get our embeddings. The second dimension of the input into a `nn.Conv2d` layer must be the channel dimension. As text technically does not have a channel dimension, we `unsqueeze` our tensor to create one. This matches with our `in_channels=1` in the initialization of our convolutional layers. \n", "\n", "We then pass the tensors through the convolutional and pooling layers, using the `ReLU` activation function after the convolutional layers. Another nice feature of the pooling layers is that they handle sentences of different lengths. The size of the output of the convolutional layer is dependent on the size of the input to it, and different batches contain sentences of different lengths. Without the max pooling layer the input to our linear layer would depend on the size of the input sentence (not what we want). One option to rectify this would be to trim/pad all sentences to the same length, however with the max pooling layer we always know the input to the linear layer will be the total number of filters. **Note**: there is an exception to this if your sentence(s) are shorter than the largest filter used. You will then have to pad your sentences to the length of the largest filter. In the IMDb data there are no reviews only 5 words long so we don't have to worry about that, but you will if you are using your own data.\n", "\n", "Finally, we perform dropout on the concatenated filter outputs and then pass them through a linear layer to make our predictions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "513d77c9316914e07fdf1a1d4b1ac386", "grade": false, "grade_id": "cell-db763b1b029c9f4b", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "import torch.nn as nn\n", "import torch.nn.functional as F\n", "\n", "class CNN(nn.Module):\n", " def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, dropout):\n", " super().__init__()\n", " \n", " self.embedding = nn.Embedding(vocab_size, embedding_dim)\n", " self.conv_0 = nn.Conv2d(in_channels = 1, out_channels = n_filters, \n", " kernel_size = (filter_sizes[0], embedding_dim))\n", " self.conv_1 = nn.Conv2d(in_channels = 1, out_channels = n_filters, \n", " kernel_size = (filter_sizes[1], embedding_dim))\n", " self.conv_2 = nn.Conv2d(in_channels = 1, out_channels = n_filters, \n", " kernel_size = (filter_sizes[2], embedding_dim))\n", " self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)\n", " self.dropout = nn.Dropout(dropout)\n", " \n", " def forward(self, text):\n", " text = text.permute(1, 0)\n", " embedded = self.embedding(text)\n", " embedded = embedded.unsqueeze(1)\n", " \n", " conved_0 = F.relu(self.conv_0(embedded).squeeze(3))\n", " conved_1 = F.relu(self.conv_1(embedded).squeeze(3))\n", " conved_2 = F.relu(self.conv_2(embedded).squeeze(3))\n", " \n", " pooled_0 = F.max_pool1d(conved_0, conved_0.shape[2]).squeeze(2)\n", " pooled_1 = F.max_pool1d(conved_1, conved_1.shape[2]).squeeze(2)\n", " pooled_2 = F.max_pool1d(conved_2, conved_2.shape[2]).squeeze(2)\n", " \n", " cat = self.dropout(torch.cat((pooled_0, pooled_1, pooled_2), dim=1))\n", " \n", " return self.fc(cat)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "ba2ff6225f80c3ef5722eebf915f1ce8", "grade": false, "grade_id": "cell-7add4674e7e89b00", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "Currently the `CNN` model can only use 3 different sized filters, but we can actually improve the code of our model to make it more generic and take any number of filters.\n", "\n", "We do this by placing all of our convolutional layers in a `nn.ModuleList`, a function used to hold a list of PyTorch `nn.Module`s. If we simply used a standard Python list, the modules within the list cannot be \"seen\" by any modules outside the list which will cause us some errors.\n", "\n", "We can now pass an arbitrarily sized list of filter sizes and the list comprehension will create a convolutional layer for each of them. Then, in the `forward` method we iterate through the list applying each convolutional layer to get a list of convolutional outputs, which we also feed through the max pooling in a list comprehension before concatenating together and passing through the dropout and linear layers." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "56facdcdf29482a1d9a4e0c16a026c71", "grade": false, "grade_id": "cell-010641cdfd13da58", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "class CNN(nn.Module):\n", " def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, dropout):\n", " super().__init__()\n", " \n", " self.embedding = nn.Embedding(vocab_size, embedding_dim)\n", " self.convs = nn.ModuleList([\n", " nn.Conv2d(in_channels = 1, out_channels = n_filters, \n", " kernel_size = (fs, embedding_dim)) \n", " for fs in filter_sizes\n", " ])\n", " self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)\n", " self.dropout = nn.Dropout(dropout)\n", " \n", " def forward(self, text):\n", " text = text.permute(1, 0)\n", " embedded = self.embedding(text)\n", " embedded = embedded.unsqueeze(1)\n", "\n", " conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]\n", " pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]\n", " \n", " cat = self.dropout(torch.cat(pooled, dim=1))\n", " \n", " return self.fc(cat)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "41fe8028f4b3aa4c3de86ed489b5211b", "grade": false, "grade_id": "cell-dcb1625a83252a09", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "We create an instance of our `CNN` class." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "ce3759efd80bc24d1cec542efc1095a8", "grade": false, "grade_id": "cell-ca875e268b81e8fc", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "INPUT_DIM = len(TEXT.vocab)\n", "EMBEDDING_DIM = 100\n", "N_FILTERS = 100\n", "FILTER_SIZES = [3,4,5]\n", "OUTPUT_DIM = 1\n", "DROPOUT = 0.5\n", "\n", "model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "794fea6068c2df8e919ff16e625e505e", "grade": false, "grade_id": "cell-0e5f8e5031bec226", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "And load the pre-trained embeddings" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "4c944398a724a5962b2b3751913ff541", "grade": false, "grade_id": "cell-7d81d24e269f5d0e", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "pretrained_embeddings = TEXT.vocab.vectors\n", "\n", "model.embedding.weight.data.copy_(pretrained_embeddings)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "e17cc6fae19f0a21c802087c5815fc7e", "grade": false, "grade_id": "cell-b3cea01c4ac79540", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "## Train the Model" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "2935cd16fcd2980579e5e7973315d210", "grade": false, "grade_id": "cell-4c701bea2531cb1c", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "__Use the following block to train (for 5 epochs) and evaluate the model using Torchbearer. Use Adam (with the default learning rate of 0.001) as the optimiser and Binary Cross Entropy loss (bear in mind the model does not have a Sigmoid at the end):__" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "checksum": "60bd09b6992b1cfbac98e8bb0f963075", "grade": true, "grade_id": "cell-341166393367aef1", "locked": false, "points": 10, "schema_version": 1, "solution": true } }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "d38a233f4cfb19bc9fcddb394a5a3922", "grade": false, "grade_id": "cell-5d57ce05de6b868b", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "__Use the following block to note any observations you can make about this model. How does it compare to the recurrent ones in the previous part of the lab?__" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "nbgrader": { "checksum": "ecd8d2cd3208fbc6daf466a6018c565b", "grade": true, "grade_id": "cell-d97f8c8c005e1baa", "locked": false, "points": 5, "schema_version": 1, "solution": true } }, "source": [ "YOUR ANSWER HERE" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "5dbaedc4c6117779d69699b7c1710011", "grade": false, "grade_id": "cell-5788e25f5eb9dc65", "locked": true, "schema_version": 1, "solution": false } }, "source": [ "## User Input\n", "\n", "And again, as a sanity check we can check some input sentences\n", "\n", "**Note**: As mentioned in the implementation details, the input sentence has to be at least as long as the largest filter height used. We modify our `predict_sentiment` function to also accept a minimum length argument. If the tokenized input sentence is less than `min_len` tokens, we append padding tokens (``) to make it `min_len` tokens. Note also that we don't need to pass in the sequences lengths for this model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "checksum": "7b7ae9b1599eca80395040fdeb8c7fff", "grade": false, "grade_id": "cell-5951884849fb206d", "locked": true, "schema_version": 1, "solution": false } }, "outputs": [], "source": [ "import spacy\n", "nlp = spacy.load('en')\n", "\n", "def predict_sentiment(sentence, min_len=5):\n", " tokenized = [tok.text for tok in nlp.tokenizer(sentence)]\n", " if len(tokenized) < min_len:\n", " tokenized += [''] * (min_len - len(tokenized))\n", " indexed = [TEXT.vocab.stoi[t] for t in tokenized]\n", " tensor = torch.LongTensor(indexed).to(device)\n", " tensor = tensor.unsqueeze(1)\n", " prediction = torch.sigmoid(model(tensor))\n", " return prediction.item()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Use the final code block below to test your own positive and negative reviews:__" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "checksum": "4bb862031b32ef8409af83f1f0998d4e", "grade": true, "grade_id": "cell-3275e9ee123cf3bf", "locked": false, "points": 2, "schema_version": 1, "solution": true } }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "raise NotImplementedError()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" } }, "nbformat": 4, "nbformat_minor": 2 }