{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Recurrent Neural networks\n", "=====" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### RNN " ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "```python\n", "keras.layers.recurrent.SimpleRNN(units, activation='tanh', use_bias=True, \n", " kernel_initializer='glorot_uniform', \n", " recurrent_initializer='orthogonal', \n", " bias_initializer='zeros', \n", " kernel_regularizer=None, \n", " recurrent_regularizer=None, \n", " bias_regularizer=None, \n", " activity_regularizer=None, \n", " kernel_constraint=None, recurrent_constraint=None, \n", " bias_constraint=None, dropout=0.0, recurrent_dropout=0.0)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Arguments:\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Backprop Through time " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Contrary to feed-forward neural networks, the RNN is characterized by the ability of encoding longer past information, thus very suitable for sequential models. The BPTT extends the ordinary BP algorithm to suit the recurrent neural\n", "architecture." ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "scrolled": true }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Reference**: [Backpropagation through Time](http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "import theano\n", "import theano.tensor as T\n", "import keras \n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "from sklearn.preprocessing import LabelEncoder\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split\n", "# -- Keras Import\n", "from keras.models import Sequential\n", "from keras.layers import Dense, Activation\n", "from keras.preprocessing import image\n", "\n", "from keras.datasets import imdb\n", "from keras.datasets import mnist\n", "\n", "from keras.models import Sequential\n", "from keras.layers import Dense, Dropout, Activation, Flatten\n", "from keras.layers import Conv2D, MaxPooling2D\n", "\n", "from keras.utils import np_utils\n", "from keras.preprocessing import sequence\n", "from keras.layers.embeddings import Embedding\n", "from keras.layers.recurrent import LSTM, GRU, SimpleRNN\n", "\n", "from keras.layers import Activation, TimeDistributed, RepeatVector\n", "from keras.callbacks import EarlyStopping, ModelCheckpoint" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### IMDB sentiment classification task" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. \n", "\n", "IMDB provided a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. \n", "\n", "There is additional unlabeled data for use as well. Raw text and already processed bag of words formats are provided. \n", "\n", "http://ai.stanford.edu/~amaas/data/sentiment/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Data Preparation - IMDB" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading data...\n", "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n", "25000 train sequences\n", "25000 test sequences\n", "Example:\n", "[ [1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 19193, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]]\n", "Pad sequences (samples x time)\n", "X_train shape: (25000, 100)\n", "X_test shape: (25000, 100)\n" ] } ], "source": [ "max_features = 20000\n", "maxlen = 100 # cut texts after this number of words (among top max_features most common words)\n", "batch_size = 32\n", "\n", "print(\"Loading data...\")\n", "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=max_features)\n", "print(len(X_train), 'train sequences')\n", "print(len(X_test), 'test sequences')\n", "\n", "print('Example:')\n", "print(X_train[:1])\n", "\n", "print(\"Pad sequences (samples x time)\")\n", "X_train = sequence.pad_sequences(X_train, maxlen=maxlen)\n", "X_test = sequence.pad_sequences(X_test, maxlen=maxlen)\n", "print('X_train shape:', X_train.shape)\n", "print('X_test shape:', X_test.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Model building " ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Build model...\n", "Train...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/valerio/anaconda3/envs/deep-learning-pydatait-tutorial/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:2094: UserWarning: Expected no kwargs, you passed 1\n", "kwargs passed to function are ignored with Tensorflow backend\n", " warnings.warn('\\n'.join(msg))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train on 25000 samples, validate on 25000 samples\n", "Epoch 1/1\n", "25000/25000 [==============================] - 104s - loss: 0.7329 - val_loss: 0.6832\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print('Build model...')\n", "model = Sequential()\n", "model.add(Embedding(max_features, 128, input_length=maxlen))\n", "model.add(SimpleRNN(128)) \n", "model.add(Dropout(0.5))\n", "model.add(Dense(1))\n", "model.add(Activation('sigmoid'))\n", "\n", "# try using different optimizers and different optimizer configs\n", "model.compile(loss='binary_crossentropy', optimizer='adam')\n", "\n", "print(\"Train...\")\n", "model.fit(X_train, y_train, batch_size=batch_size, epochs=1, \n", " validation_data=(X_test, y_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### LSTM " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A LSTM network is an artificial neural network that contains LSTM blocks instead of, or in addition to, regular network units. A LSTM block may be described as a \"smart\" network unit that can remember a value for an arbitrary length of time. \n", "\n", "Unlike traditional RNNs, an Long short-term memory network is well-suited to learn from experience to classify, process and predict time series when there are very long time lags of unknown size between important events." ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "scrolled": true }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "```python\n", "keras.layers.recurrent.LSTM(units, activation='tanh', recurrent_activation='hard_sigmoid', use_bias=True, \n", " kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', \n", " bias_initializer='zeros', unit_forget_bias=True, kernel_regularizer=None, \n", " recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None, \n", " kernel_constraint=None, recurrent_constraint=None, bias_constraint=None, \n", " dropout=0.0, recurrent_dropout=0.0)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Arguments\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### GRU " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Gated recurrent units are a gating mechanism in recurrent neural networks. \n", "\n", "Much similar to the LSTMs, they have fewer parameters than LSTM, as they lack an output gate." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "```python\n", "keras.layers.recurrent.GRU(units, activation='tanh', recurrent_activation='hard_sigmoid', use_bias=True, \n", " kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', \n", " bias_initializer='zeros', kernel_regularizer=None, recurrent_regularizer=None, \n", " bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, \n", " recurrent_constraint=None, bias_constraint=None, \n", " dropout=0.0, recurrent_dropout=0.0)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Your Turn! - Hands on Rnn" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "print('Build model...')\n", "model = Sequential()\n", "model.add(Embedding(max_features, 128, input_length=maxlen))\n", "\n", "# !!! Play with those! try and get better results!\n", "#model.add(SimpleRNN(128)) \n", "#model.add(GRU(128)) \n", "#model.add(LSTM(128)) \n", "\n", "model.add(Dropout(0.5))\n", "model.add(Dense(1))\n", "model.add(Activation('sigmoid'))\n", "\n", "# try using different optimizers and different optimizer configs\n", "model.compile(loss='binary_crossentropy', optimizer='adam')\n", "\n", "print(\"Train...\")\n", "model.fit(X_train, y_train, batch_size=batch_size, \n", " epochs=4, validation_data=(X_test, y_test))\n", "score, acc = model.evaluate(X_test, y_test, batch_size=batch_size)\n", "print('Test score:', score)\n", "print('Test accuracy:', acc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Sentence Generation using RNN(LSTM)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from keras.models import Sequential\n", "from keras.layers import Dense, Activation, Dropout\n", "from keras.layers import LSTM\n", "from keras.optimizers import RMSprop\n", "from keras.utils.data_utils import get_file\n", "\n", "import numpy as np\n", "import random\n", "import sys\n", "\n", "path = get_file('nietzsche.txt', origin=\"https://s3.amazonaws.com/text-datasets/nietzsche.txt\")\n", "text = open(path).read().lower()\n", "print('corpus length:', len(text))\n", "\n", "chars = sorted(list(set(text)))\n", "print('total chars:', len(chars))\n", "char_indices = dict((c, i) for i, c in enumerate(chars))\n", "indices_char = dict((i, c) for i, c in enumerate(chars))\n", "\n", "# cut the text in semi-redundant sequences of maxlen characters\n", "maxlen = 40\n", "step = 3\n", "sentences = []\n", "next_chars = []\n", "for i in range(0, len(text) - maxlen, step):\n", " sentences.append(text[i: i + maxlen])\n", " next_chars.append(text[i + maxlen])\n", "print('nb sequences:', len(sentences))\n", "\n", "print('Vectorization...')\n", "X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)\n", "y = np.zeros((len(sentences), len(chars)), dtype=np.bool)\n", "for i, sentence in enumerate(sentences):\n", " for t, char in enumerate(sentence):\n", " X[i, t, char_indices[char]] = 1\n", " y[i, char_indices[next_chars[i]]] = 1\n", "\n", "\n", "# build the model: a single LSTM\n", "print('Build model...')\n", "model = Sequential()\n", "model.add(LSTM(128, input_shape=(maxlen, len(chars))))\n", "model.add(Dense(len(chars)))\n", "model.add(Activation('softmax'))\n", "\n", "optimizer = RMSprop(lr=0.01)\n", "model.compile(loss='categorical_crossentropy', optimizer=optimizer)\n", "\n", "\n", "def sample(preds, temperature=1.0):\n", " # helper function to sample an index from a probability array\n", " preds = np.asarray(preds).astype('float64')\n", " preds = np.log(preds) / temperature\n", " exp_preds = np.exp(preds)\n", " preds = exp_preds / np.sum(exp_preds)\n", " probas = np.random.multinomial(1, preds, 1)\n", " return np.argmax(probas)\n", "\n", "# train the model, output generated text after each iteration\n", "for iteration in range(1, 60):\n", " print()\n", " print('-' * 50)\n", " print('Iteration', iteration)\n", " model.fit(X, y, batch_size=128, nb_epoch=1)\n", "\n", " start_index = random.randint(0, len(text) - maxlen - 1)\n", "\n", " for diversity in [0.2, 0.5, 1.0, 1.2]:\n", " print()\n", " print('----- diversity:', diversity)\n", "\n", " generated = ''\n", " sentence = text[start_index: start_index + maxlen]\n", " generated += sentence\n", " print('----- Generating with seed: \"' + sentence + '\"')\n", " sys.stdout.write(generated)\n", "\n", " for i in range(400):\n", " x = np.zeros((1, maxlen, len(chars)))\n", " for t, char in enumerate(sentence):\n", " x[0, t, char_indices[char]] = 1.\n", "\n", " preds = model.predict(x, verbose=0)[0]\n", " next_index = sample(preds, diversity)\n", " next_char = indices_char[next_index]\n", "\n", " generated += next_char\n", " sentence = sentence[1:] + next_char\n", "\n", " sys.stdout.write(next_char)\n", " sys.stdout.flush()\n", " print()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }