{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "RecurrentNeuralNetworksTuning.ipynb",
      "provenance": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qqDNVAuZa03l",
        "colab_type": "text"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Go to https://colab.research.google.com and choose the \\\"Upload\\\" option to upload this notebook file.\n",
        "1. In the Edit menu, choose \\\"Notebook Settings\\\" and then set the \\\"Hardware Accelerator\\\" dropdown to GPU.\n",
        "1. Read through the code in the following sections:\n",
        "  * [IMDB Dataset](#scrollTo=mXcb24B6a03_)\n",
        "  * [Define model](#scrollTo=kAz68ipVa05_)\n",
        "  * [Train model](#scrollTo=kIynp1v_a06Y)\n",
        "  * [Assess model](#scrollTo=ALyNCqx4a06r)\n",
        "1. Complete at least one of these exercises. Remember to keep notes about what you do!\n",
        "  * [Exercise Option #1 - Standard Difficulty](#scrollTo=_9dsjJwya06_)\n",
        "  * [Exercise Option #2 - Advanced Difficulty](#scrollTo=nyZbljLAa09z)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "POc2jClCa03y",
        "colab_type": "text"
      },
      "source": [
        "## Documentation/Sources\n",
        "* [Class Notes](https://jennselby.github.io/MachineLearningCourseNotes/#recurrent-neural-networks)\n",
        "* [https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/](https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/) for information on sequence classification with keras\n",
        "* [https://keras.io/](https://keras.io/) Keras API documentation\n",
        "* [Keras recurrent tutorial](https://github.com/Vict0rSch/deep_learning/tree/master/keras/recurrent)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h04E5miUb8wh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1bd4c14d-c6d1-4b59-a950-597a007e35b0"
      },
      "source": [
        "# upgrade tensorflow to tensorflow 2\n",
        "%tensorflow_version 2.x\n",
        "# display matplotlib plots\n",
        "%matplotlib inline\n",
        "from tensorflow import test\n",
        "from tensorflow import device"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 2.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mXcb24B6a03_",
        "colab_type": "text"
      },
      "source": [
        "# IMDB Dataset\n",
        "The [IMDB dataset](https://keras.io/datasets/#imdb-movie-reviews-sentiment-classification) consists of movie reviews (x_train) that have been marked as positive or negative (y_train). See the [Word Vectors Tutorial](https://github.com/jennselby/MachineLearningTutorials/blob/master/WordVectors.ipynb) for more details on the IMDB dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u2kXcDIia04D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.datasets import imdb\n",
        "from tensorflow.keras.preprocessing import sequence"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KWuzcUyua04f",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "81a744d0-dc6e-4a2a-ac86-8639610972b6"
      },
      "source": [
        "(imdb_x_train, imdb_y_train), (imdb_x_test, imdb_y_test) = imdb.load_data()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hYQ3yPO4a04x",
        "colab_type": "text"
      },
      "source": [
        "For a standard keras model, every input has to be the same length, so we need to set some length after which we will cutoff the rest of the review. (We will also need to pad the shorter reviews with zeros to make them the same length)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QtYp3G31a040",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cutoff = 500\n",
        "imdb_x_train_padded = sequence.pad_sequences(imdb_x_train, maxlen=cutoff)\n",
        "imdb_x_test_padded = sequence.pad_sequences(imdb_x_test, maxlen=cutoff)\n",
        "\n",
        " # see https://stackoverflow.com/questions/42821330/restore-original-text-from-keras-s-imdb-dataset\n",
        "imdb_index_offset = 3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iOGRpn25a05o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Embedding, LSTM, Dense"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kAz68ipVa05_",
        "colab_type": "text"
      },
      "source": [
        "# Define model\n",
        "\n",
        "Unlike last time, when we used convolutional layers, we're going to use an LSTM, a special type of recurrent network.\n",
        "\n",
        "Using recurrent networks means that rather than seeing these reviews as one input happening all at once, with the convolutional layers taking into account which words are next to each other, we are going to see them as a sequence of inputs, with one word occurring at each timestep."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4u9ZArrxa06G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "db3bf5cd-e935-472c-e700-c4124d0573b3"
      },
      "source": [
        "imdb_lstm_model = Sequential()\n",
        "imdb_lstm_model.add(Embedding(input_dim=len(imdb.get_word_index()) + imdb_index_offset,\n",
        "                              output_dim=100,\n",
        "                              input_length=cutoff))\n",
        "# return_sequences tells the LSTM to output the full sequence, for use by the next LSTM layer. The final\n",
        "# LSTM layer should return only the output sequence, for use in the Dense output layer\n",
        "imdb_lstm_model.add(LSTM(units=32, return_sequences=True))\n",
        "imdb_lstm_model.add(LSTM(units=32))\n",
        "imdb_lstm_model.add(Dense(units=1, activation='sigmoid')) # because at the end, we want one yes/no answer\n",
        "imdb_lstm_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['binary_accuracy'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
            "1646592/1641221 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kIynp1v_a06Y",
        "colab_type": "text"
      },
      "source": [
        "# Train model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-ktpTQmpa06Z",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "73bfd77c-005b-49a1-94b7-f233f3685240"
      },
      "source": [
        "# Train using GPU acceleration\n",
        "# (see https://colab.research.google.com/notebooks/gpu.ipynb#scrollTo=Y04m-jvKRDsJ)\n",
        "device_name = test.gpu_device_name()\n",
        "if device_name != '/device:GPU:0':\n",
        "  print(\n",
        "      '\\n\\nThis error most likely means that this notebook is not '\n",
        "      'configured to use a GPU.  Change this in Notebook Settings via the '\n",
        "      'command palette (cmd/ctrl-shift-P) or the Edit menu.\\n\\n')\n",
        "  raise SystemError('GPU device not found')\n",
        "\n",
        "with device('/device:GPU:0'):\n",
        "  imdb_lstm_model.fit(imdb_x_train_padded, imdb_y_train, epochs=1, batch_size=64)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 25000 samples\n",
            "25000/25000 [==============================] - 64s 3ms/sample - loss: 0.3914 - binary_accuracy: 0.8204\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ALyNCqx4a06r",
        "colab_type": "text"
      },
      "source": [
        "# Assess model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fzNKy7fCa06y",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "87123078-55bf-4280-a1de-85b436fec55a"
      },
      "source": [
        "with device('/device:GPU:0'):\n",
        "  imdb_lstm_scores = imdb_lstm_model.evaluate(imdb_x_test_padded, imdb_y_test)\n",
        "  print('loss: {} accuracy: {}'.format(*imdb_lstm_scores))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "25000/25000 [==============================] - 15s 589us/sample - loss: 0.3151 - binary_accuracy: 0.8690\n",
            "loss: 0.31508431223869326 accuracy: 0.8689600229263306\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_9dsjJwya06_",
        "colab_type": "text"
      },
      "source": [
        "# Exercise Option #1 - Standard Difficulty\n",
        "\n",
        "Experiment with different model configurations from the one above. Try other recurrent layers, different numbers of layers, change some of the defaults. See [Keras Recurrent Layers](https://keras.io/layers/recurrent/)\n",
        "\n",
        "__Keep notes on what you try and what results you get.__"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CGDXolUia07G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ltzqj4Kaa07U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DO6zsWBja07t",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dz9hy7moa078",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nyZbljLAa09z",
        "colab_type": "text"
      },
      "source": [
        "# Exercise Option #2 - Advanced Difficulty\n",
        "\n",
        "Set up your own RNN model for the Reuters Classification Problem\n",
        "\n",
        "Take the model from exercise 1 (imdb_lstm_model) and modify it to classify the [Reuters data](https://keras.io/datasets/#reuters-newswire-topics-classification).\n",
        "\n",
        "Think about what you are trying to predict in this case, and how you will have to change your model to deal with this."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zI9p998Ra090",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.datasets import reuters\n",
        "from tensorflow.keras.preprocessing import sequence\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Embedding, LSTM, Dense"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-OR-J3MWa095",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "a1c602a4-c78b-4142-d93b-1143f057141d"
      },
      "source": [
        "(reuters_x_train, reuters_y_train), (reuters_x_test, reuters_y_test) = reuters.load_data()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/reuters.npz\n",
            "2113536/2110848 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MiW4Vhgpa098",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uYmb0IySa0-B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}