![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/logo.png\"){kind=logo}
\n",
"\n",
"When working with sequential data (time-series, sentences, etc.) the order of the inputs is crucial for the task at hand. Recurrent neural networks (RNNs) process sequential data by accounting for the current input and also what has been learned from previous inputs. In this notebook, we'll learn how to create and train RNNs on sequential data.\n",
"\n",
"![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/rnn.png\")
\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "VoMq0eFRvugb"
},
"source": [
"# Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "qWro5T5qTJJL"
},
"source": [
"* **Objective:** Process sequential data by accounting for the currend input and also what has been learned from previous inputs.\n",
"* **Advantages:** \n",
" * Account for order and previous inputs in a meaningful way.\n",
" * Conditioned generation for generating sequences.\n",
"* **Disadvantages:** \n",
" * Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations. \n",
" * Processing long sequences can yield memory and computation issues.\n",
" * Interpretability is difficult but there are few [techniques](https://arxiv.org/abs/1506.02078) that use the activations from RNNs to see what parts of the inputs are processed. \n",
"* **Miscellaneous:** \n",
" * Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "rsHeBbehrKzl"
},
"source": [
"![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/rnn2.png\")
\n",
"\n",
"RNN forward pass for a single time step $X_t$:\n",
"\n",
"$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$\n",
"\n",
"$y_t = W_{hy}h_t + b_y $\n",
"\n",
"$ P(y) = softmax(y_t) = \\frac{e^y}{\\sum e^y} $\n",
"\n",
"*where*:\n",
"* $X_t$ = input at time step t | $\\in \\mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)\n",
"* $W_{hh}$ = hidden units weights| $\\in \\mathbb{R}^{HXH}$ ($H$ is the hidden dim)\n",
"* $h_{t-1}$ = previous timestep's hidden state $\\in \\mathbb{R}^{NXH}$\n",
"* $W_{xh}$ = input weights| $\\in \\mathbb{R}^{EXH}$\n",
"* $b_h$ = hidden units bias $\\in \\mathbb{R}^{HX1}$\n",
"* $W_{hy}$ = output weights| $\\in \\mathbb{R}^{HXC}$ ($C$ is the number of classes)\n",
"* $b_y$ = output bias $\\in \\mathbb{R}^{CX1}$\n",
"\n",
"You repeat this for every time step's input ($X_{t+1}, X_{t+2}, ..., X_{N})$ to the get the predicted outputs at each time step.\n",
"\n",
"**Note**: At the first time step, the previous hidden state $h_{t-1}$ can either be a zero vector (unconditioned) or initialize (conditioned). If we are conditioning the RNN, the first hidden state $h_0$ can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "dIXlGMExJD6w"
},
"source": [
"Let's see what the forward pass looks like with an RNN for a synthetic task such as processing reviews (a sequence of words) to predict the sentiment at the end of processing the review."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "RcWE5cw0_cKA",
"outputId": "a44156b9-b43f-409c-f0ce-4a4bd871d6a0"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.0.0)\n"
]
}
],
"source": [
"# Load PyTorch library\n",
"!pip3 install torch"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "o6eEK1wM_dXG"
},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Qi9hIEV6COLF"
},
"outputs": [],
"source": [
"batch_size = 5\n",
"seq_size = 10 # max length per input (masking will be used for sequences that aren't this max length)\n",
"x_lengths = [8, 5, 4, 10, 5] # lengths of each input sequence\n",
"embedding_dim = 100\n",
"rnn_hidden_dim = 256\n",
"output_dim = 4"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "bLEzfxjhB94C",
"outputId": "f2feefbf-8635-4b23-ef53-b5713cf2cdb2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([5, 10, 100])\n"
]
}
],
"source": [
"# Initialize synthetic inputs\n",
"x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
"x_lengths = torch.tensor(x_lengths)\n",
"print (x_in.size())"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "dr6oLqtXB98N",
"outputId": "9817e88d-6e73-414a-dfa6-2386f40db0d9"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([5, 256])\n"
]
}
],
"source": [
"# Initialize hidden state\n",
"hidden_t = torch.zeros((batch_size, rnn_hidden_dim))\n",
"print (hidden_t.size())"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "ryZMOLLgB9-v",
"outputId": "14ec0a2a-bf37-4e03-b69b-099180f8f149"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"RNNCell(100, 256)\n"
]
}
],
"source": [
"# Initialize RNN cell\n",
"rnn_cell = nn.RNNCell(embedding_dim, rnn_hidden_dim)\n",
"print (rnn_cell)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "rlbZ7ujxExXb",
"outputId": "6c83ba2b-94c5-4f76-c8fb-ef0c1ccdeb37"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([5, 10, 256])\n"
]
}
],
"source": [
"# Forward pass through RNN\n",
"x_in = x_in.permute(1, 0, 2) # RNN needs batch_size to be at dim 1\n",
"\n",
"# Loop through the inputs time steps\n",
"hiddens = []\n",
"for t in range(seq_size):\n",
" hidden_t = rnn_cell(x_in[t], hidden_t)\n",
" hiddens.append(hidden_t)\n",
"hiddens = torch.stack(hiddens)\n",
"hiddens = hiddens.permute(1, 0, 2) # bring batch_size back to dim 0\n",
"print (hiddens.size())"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"colab_type": "code",
"id": "3TTL-jmg-MHa",
"outputId": "3fae323f-c37d-4dac-c8a8-7fea7a45c95c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"out: torch.Size([5, 10, 256])\n",
"h_n: torch.Size([1, 5, 256])\n"
]
}
],
"source": [
"# We also could've used a more abstracted layer\n",
"x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
"rnn = nn.RNN(embedding_dim, rnn_hidden_dim, batch_first=True)\n",
"out, h_n = rnn(x_in) #h_n is the last hidden state\n",
"print (\"out: \", out.size())\n",
"print (\"h_n: \", h_n.size())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "iAsyRNnbHwcT"
},
"outputs": [],
"source": [
"def gather_last_relevant_hidden(hiddens, x_lengths):\n",
" x_lengths = x_lengths.long().detach().cpu().numpy() - 1\n",
" out = []\n",
" for batch_index, column_index in enumerate(x_lengths):\n",
" out.append(hiddens[batch_index, column_index])\n",
" return torch.stack(out)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "PVhp1KLqHqpA",
"outputId": "d04be3ef-c2d6-48b9-f0f5-a93f619ec594"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([5, 256])\n"
]
}
],
"source": [
"# Gather the last relevant hidden state\n",
"z = gather_last_relevant_hidden(hiddens, x_lengths)\n",
"print (z.size())"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 119
},
"colab_type": "code",
"id": "yGk_iZ5cITZl",
"outputId": "84749ff2-1e45-4599-a38d-8c83cee116a9"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([5, 4])\n",
"tensor([[0.3030, 0.2351, 0.2168, 0.2452],\n",
" [0.2614, 0.1912, 0.2617, 0.2858],\n",
" [0.2428, 0.2600, 0.2254, 0.2717],\n",
" [0.2379, 0.2226, 0.1901, 0.3494],\n",
" [0.2629, 0.2854, 0.2146, 0.2371]], grad_fn=![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/seq2seq.jpeg\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "tnxUIEMdukYY"
},
"source": [
"# Issues with vanilla RNNs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "uMx2s93VLUTt"
},
"source": [
"There are several issues with the vanilla RNN that we've seen so far. \n",
"\n",
"1. When we have an input sequence that has many time steps, it becomes difficult for the model to retain information seen earlier as we process more and more of the downstream timesteps. The goals of the model is to retain the useful components in the previously seen time steps but this becomes cumbersome when we have so many time steps to process. \n",
"\n",
"2. During backpropagation, the gradient from the loss has to travel all the way back towards the first time step. If our gradient is larger than 1 (${1.01}^{1000} = 20959$) or less than 1 (${0.99}^{1000} = 4.31e-5$) and we have lot's of time steps, this can quickly spiral out of control.\n",
"\n",
"To address both these issues, the concept of gating was introduced to RNNs. Gating allows RNNs to control the information flow between each time step to optimize on the task. Selectively allowing information to pass through allows the model to process inputs with many time steps. The most common RNN gated varients are the long short term memory ([LSTM](https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM)) units and gated recurrent units ([GRUs](https://pytorch.org/docs/stable/nn.html#torch.nn.GRU)). You can read more about how these units work [here](http://colah.github.io/posts/2015-08-Understanding-LSTMs/).\n",
"\n",
"![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/gates.png\")
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "tirko0kwp-9J"
},
"outputs": [],
"source": [
"# GRU in PyTorch\n",
"gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
" batch_first=True)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "UZjUhh4VBWxM",
"outputId": "9fe275fe-c8d9-42f0-e5d0-0295268ed83d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([5, 10, 100])\n"
]
}
],
"source": [
"# Initialize synthetic input\n",
"x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
"print (x_in.size())"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"colab_type": "code",
"id": "xJ_SE7AvBfa4",
"outputId": "b9411aaa-fab1-4104-aee7-8f9a423332ab"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"out: torch.Size([5, 10, 256])\n",
"h_n: torch.Size([1, 5, 256])\n"
]
}
],
"source": [
"# Forward pass\n",
"out, h_n = gru(x_in)\n",
"print (\"out:\", out.size())\n",
"print (\"h_n:\", h_n.size())"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ij_GA2Rr9BbA"
},
"source": [
"**Note**: Choosing whether to use GRU or LSTM really depends on the data and empirical performance. GRUs offer comparable performance with reduce number of parameters while LSTMs are more efficient and may make the difference in performance for your particular task."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "9agJw4gwK1LC"
},
"source": [
"# Bidirectional RNNs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Xck0n-KpmXkV"
},
"source": [
"There have been many advancements with RNNs ([attention](https://www.oreilly.com/ideas/interpretability-via-attentional-and-memory-based-interfaces-using-tensorflow), Quasi RNNs, etc.) that we will cover in later lessons but one of the basic and widely used ones are bidirectional RNNs (Bi-RNNs). The motivation behind bidirectional RNNs is to process an input sequence by both directions. Accounting for context from both sides can aid in performance when the entire input sequence is known at time of inference. A common application of Bi-RNNs is in translation where it's advantageous to look at an entire sentence from both sides when translating to another language (ie. Japanese → English).\n",
"\n",
"![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/birnn.png\")
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "gSk_5XrvApCd"
},
"outputs": [],
"source": [
"# BiGRU in PyTorch\n",
"bi_gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
" batch_first=True, bidirectional=True)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"colab_type": "code",
"id": "Fx7-GTptBCtZ",
"outputId": "f0242cc5-534a-460b-ebe0-4e8c504fab22"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"out: torch.Size([5, 10, 512])\n",
"h_n: torch.Size([2, 5, 256])\n"
]
}
],
"source": [
"# Forward pass\n",
"out, h_n = bi_gru(x_in)\n",
"print (\"out:\", out.size()) # collection of all hidden states from the RNN for each time step\n",
"print (\"h_n:\", h_n.size()) # last hidden state from the RNN"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "k5lvJirLBjI6"
},
"source": [
"Notice that the output for each sample at each timestamp has size 512 (double the hidden dim). This is because this includes both the forward and backward directions from the BiRNN. "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "mJSknbofK2S9"
},
"source": [
"# Document classification with RNNs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "JgYdEZmHlmft"
},
"source": [
"Let's apply RNNs to the document classification task from the [emebddings notebook](https://colab.research.google.com/drive/1yDa5ZTqKVoLl-qRgH-N9xs3pdrDJ0Fb4) where we want to predict an article's category given its title."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "eIvXqvPQEiDC"
},
"source": [
"## Set up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "muTcvMynlmAu"
},
"outputs": [],
"source": [
"import os\n",
"from argparse import Namespace\n",
"import collections\n",
"import copy\n",
"import json\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n",
"import re\n",
"import torch"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "00ESjecep-_y"
},
"outputs": [],
"source": [
"# Set Numpy and PyTorch seeds\n",
"def set_seeds(seed, cuda):\n",
" np.random.seed(seed)\n",
" torch.manual_seed(seed)\n",
" if cuda:\n",
" torch.cuda.manual_seed_all(seed)\n",
" \n",
"# Creating directories\n",
"def create_dirs(dirpath):\n",
" if not os.path.exists(dirpath):\n",
" os.makedirs(dirpath)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "m67THDvxEl1e",
"outputId": "7118c77b-cbf9-4d7e-ff7a-b9dc1fb63cbb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using CUDA: True\n"
]
}
],
"source": [
"# Arguments\n",
"args = Namespace(\n",
" seed=1234,\n",
" cuda=True,\n",
" shuffle=True,\n",
" data_file=\"news.csv\",\n",
" split_data_file=\"split_news.csv\",\n",
" vectorizer_file=\"vectorizer.json\",\n",
" model_state_file=\"model.pth\",\n",
" save_dir=\"news\",\n",
" train_size=0.7,\n",
" val_size=0.15,\n",
" test_size=0.15,\n",
" pretrained_embeddings=None,\n",
" cutoff=25, # token must appear at least \n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" category title\n",
"0 Business Wall St. Bears Claw Back Into the Black (Reuters)\n",
"1 Business Carlyle Looks Toward Commercial Aerospace (Reu...\n",
"2 Business Oil and Economy Cloud Stocks' Outlook (Reuters)\n",
"3 Business Iraq Halts Oil Exports from Main Southern Pipe...\n",
"4 Business Oil prices soar to all-time record, posing new..."
]
},
"execution_count": 25,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"# Raw data\n",
"df = pd.read_csv(args.data_file, header=0)\n",
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 85
},
"colab_type": "code",
"id": "ICl2MNK4EwrL",
"outputId": "d2073597-71e5-40b1-a845-90bf4913ea7a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Business: 30000\n",
"Sci/Tech: 30000\n",
"Sports: 30000\n",
"World: 30000\n"
]
}
],
"source": [
"# Split by category\n",
"by_category = collections.defaultdict(list)\n",
"for _, row in df.iterrows():\n",
" by_category[row.category].append(row.to_dict())\n",
"for category in by_category:\n",
" print (\"{0}: {1}\".format(category, len(by_category[category])))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "76PwKQHLEww5"
},
"outputs": [],
"source": [
"# Create split data\n",
"final_list = []\n",
"for _, item_list in sorted(by_category.items()):\n",
" if args.shuffle:\n",
" np.random.shuffle(item_list)\n",
" n = len(item_list)\n",
" n_train = int(args.train_size*n)\n",
" n_val = int(args.val_size*n)\n",
" n_test = int(args.test_size*n)\n",
"\n",
" # Give data point a split attribute\n",
" for item in item_list[:n_train]:\n",
" item['split'] = 'train'\n",
" for item in item_list[n_train:n_train+n_val]:\n",
" item['split'] = 'val'\n",
" for item in item_list[n_train+n_val:]:\n",
" item['split'] = 'test' \n",
"\n",
" # Add to final list\n",
" final_list.extend(item_list)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 85
},
"colab_type": "code",
"id": "CQeS0KHOEwzm",
"outputId": "93c9aadb-25c4-4029-f002-8a43f3956045"
},
"outputs": [
{
"data": {
"text/plain": [
"train 84000\n",
"val 18000\n",
"test 18000\n",
"Name: split, dtype: int64"
]
},
"execution_count": 28,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"# df with split datasets\n",
"split_df = pd.DataFrame(final_list)\n",
"split_df[\"split\"].value_counts()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "pPJDyVusEw3-"
},
"outputs": [],
"source": [
"# Preprocessing\n",
"def preprocess_text(text):\n",
" text = ' '.join(word.lower() for word in text.split(\" \"))\n",
" text = re.sub(r\"([.,!?])\", r\" \\1 \", text)\n",
" text = re.sub(r\"[^a-zA-Z.,!?]+\", r\" \", text)\n",
" text = text.strip()\n",
" return text\n",
" \n",
"split_df.title = split_df.title.apply(preprocess_text)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"colab_type": "code",
"id": "IAetKendEw6b",
"outputId": "d5946f7e-840e-4a0b-e492-d3da68cefd44"
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | category | \n", "title | \n", "
|---|---|---|
| 0 | \n", "Business | \n", "Wall St. Bears Claw Back Into the Black (Reuters) | \n", "
| 1 | \n", "Business | \n", "Carlyle Looks Toward Commercial Aerospace (Reu... | \n", "
| 2 | \n", "Business | \n", "Oil and Economy Cloud Stocks' Outlook (Reuters) | \n", "
| 3 | \n", "Business | \n", "Iraq Halts Oil Exports from Main Southern Pipe... | \n", "
| 4 | \n", "Business | \n", "Oil prices soar to all-time record, posing new... | \n", "
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" category split title\n",
"0 Business train general electric posts higher rd quarter profit\n",
"1 Business train lilly to eliminate up to us jobs\n",
"2 Business train s amp p lowers america west outlook to negative\n",
"3 Business train does rand walk the talk on labor policy ?\n",
"4 Business train housekeeper advocates for changes"
]
},
"execution_count": 30,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"# Save to CSV\n",
"split_df.to_csv(args.split_data_file, index=False)\n",
"split_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "NHzGXAI3E7lF"
},
"source": [
"## Vocabulary"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "ZIRUjX0MEw88"
},
"outputs": [],
"source": [
"class Vocabulary(object):\n",
" def __init__(self, token_to_idx=None):\n",
"\n",
" # Token to index\n",
" if token_to_idx is None:\n",
" token_to_idx = {}\n",
" self.token_to_idx = token_to_idx\n",
"\n",
" # Index to token\n",
" self.idx_to_token = {idx: token \\\n",
" for token, idx in self.token_to_idx.items()}\n",
"\n",
" def to_serializable(self):\n",
" return {'token_to_idx': self.token_to_idx}\n",
"\n",
" @classmethod\n",
" def from_serializable(cls, contents):\n",
" return cls(**contents)\n",
"\n",
" def add_token(self, token):\n",
" if token in self.token_to_idx:\n",
" index = self.token_to_idx[token]\n",
" else:\n",
" index = len(self.token_to_idx)\n",
" self.token_to_idx[token] = index\n",
" self.idx_to_token[index] = token\n",
" return index\n",
"\n",
" def add_tokens(self, tokens):\n",
" return [self.add_token[token] for token in tokens]\n",
"\n",
" def lookup_token(self, token):\n",
" return self.token_to_idx[token]\n",
"\n",
" def lookup_index(self, index):\n",
" if index not in self.idx_to_token:\n",
" raise KeyError(\"the index (%d) is not in the Vocabulary\" % index)\n",
" return self.idx_to_token[index]\n",
"\n",
" def __str__(self):\n",
" return \"| \n", " | category | \n", "split | \n", "title | \n", "
|---|---|---|---|
| 0 | \n", "Business | \n", "train | \n", "general electric posts higher rd quarter profit | \n", "
| 1 | \n", "Business | \n", "train | \n", "lilly to eliminate up to us jobs | \n", "
| 2 | \n", "Business | \n", "train | \n", "s amp p lowers america west outlook to negative | \n", "
| 3 | \n", "Business | \n", "train | \n", "does rand walk the talk on labor policy ? | \n", "
| 4 | \n", "Business | \n", "train | \n", "housekeeper advocates for changes | \n", "
![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/layernorm.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "oAhAHcgZBMFe"
},
"source": [
"The most useful location to apply layernorm will be inside the RNN on the activations before the non-linearities. However, this is a bit involved and though PyTorch has a [LayerNorm](https://pytorch.org/docs/stable/nn.html#torch.nn.LayerNorm) class, they do not have an RNN that has built in layernorm yet. You could implement the RNN yourself and manually add layernorm by following a similar setup like below.\n",
"\n",
"```python\n",
"# Layernorm\n",
"for t in range(seq_size):\n",
" # Normalize over hidden dim\n",
" layernorm = nn.LayerNorm(args.hidden_dim)\n",
" # Activating the module\n",
" a = layernorm(x)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1YHneO3SStOp"
},
"source": [
"# TODO"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "gGHaKTe1SuEk"
},
"source": [
"- interpretability with task to see which words were most influential"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "13_Recurrent_Neural_Networks",
"provenance": [],
"toc_visible": true,
"version": "0.3.2"
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}