{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "JL3ZPJPgC4wC"
},
"source": [
"# Shakespeare Text Generation (using RNN LSTM)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "eoWGtkL8PPJ3"
},
"source": [
"> - 🤖 See [full list of Machine Learning Experiments](https://github.com/trekhleb/machine-learning-experiments) on **GitHub**
\n",
"> - ▶️ **Interactive Demo**: [try this model and other machine learning experiments in action](https://trekhleb.github.io/machine-learning-experiments/)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "bui0MyTjv1Mp"
},
"source": [
"## Experiment overview\n",
"\n",
"In this experiment we will use character-based [Recurrent Neural Network](https://en.wikipedia.org/wiki/Recurrent_neural_network) (RNN) to generate a Shakespeare's-like text based on the Shakespeare dataset from [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) blog post.\n",
"\n",
"For this experiment we will use [Tensorflow v2](https://www.tensorflow.org/) with its [Keras API](https://www.tensorflow.org/guide/keras).\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "3XxEZuRNIzHH"
},
"source": [
"_Inspired by [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation)_"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "IDJctDhhEDTD"
},
"source": [
"## Import dependencies"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1347,
"status": "ok",
"timestamp": 1584519798294,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Z2xZnxncD2Ep",
"outputId": "9e110fb9-a6d1-4bd6-818c-2198114e3133"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TensorFlow 2.x selected.\n"
]
}
],
"source": [
"# Selecting Tensorflow version v2 (the command is relevant for Colab only).\n",
"# %tensorflow_version 2.x"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 68
},
"colab_type": "code",
"executionInfo": {
"elapsed": 2811,
"status": "ok",
"timestamp": 1584519800384,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "SpueB6zADYgE",
"outputId": "3f4e9997-8ce8-4ef8-f8f9-320c9a44e03b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Python version: 3.6.9\n",
"Tensorflow version: 2.1.0\n",
"Keras version: 2.2.4-tf\n"
]
}
],
"source": [
"import tensorflow as tf\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import platform\n",
"import time\n",
"import pathlib\n",
"import os\n",
"\n",
"print('Python version:', platform.python_version())\n",
"print('Tensorflow version:', tf.__version__)\n",
"print('Keras version:', tf.keras.__version__)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ciI_JnnNEGCw"
},
"source": [
"## Download the dataset"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 68
},
"colab_type": "code",
"executionInfo": {
"elapsed": 2533,
"status": "ok",
"timestamp": 1584519800396,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Vk2HldshEOph",
"outputId": "45342704-eb1c-4863-a4e4-8a94d9cc764b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt\n",
"1122304/1115394 [==============================] - 0s 0us/step\n",
"/content/tmp/datasets/shakespeare.txt\n"
]
}
],
"source": [
"cache_dir = './tmp'\n",
"dataset_file_name = 'shakespeare.txt'\n",
"dataset_file_origin = 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt'\n",
"\n",
"dataset_file_path = tf.keras.utils.get_file(\n",
" fname=dataset_file_name,\n",
" origin=dataset_file_origin,\n",
" cache_dir=pathlib.Path(cache_dir).absolute()\n",
")\n",
"\n",
"print(dataset_file_path)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "eKTy6YS5Gx-g"
},
"source": [
"## Analyze the dataset"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 2199,
"status": "ok",
"timestamp": 1584519800398,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "CzfX7hK8G0bn",
"outputId": "dbcd9c2e-967d-404f-ebd4-387ee0b629d7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Length of text: 1115394 characters\n"
]
}
],
"source": [
"# Reading the database file.\n",
"text = open(dataset_file_path, mode='r').read()\n",
"\n",
"print('Length of text: {} characters'.format(len(text)))"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 272
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1774,
"status": "ok",
"timestamp": 1584519800402,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "nknOAOuoH-Av",
"outputId": "7030d3f8-c6a8-43e0-81a0-c1e87d677314"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"First Citizen:\n",
"Before we proceed any further, hear me speak.\n",
"\n",
"All:\n",
"Speak, speak.\n",
"\n",
"First Citizen:\n",
"You are all resolved rather to die than to famish?\n",
"\n",
"All:\n",
"Resolved. resolved.\n",
"\n",
"First Citizen:\n",
"First, you know Caius Marcius is chief enemy to the people.\n",
"\n"
]
}
],
"source": [
"# Take a look at the first 250 characters in text.\n",
"print(text[:250])"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 71
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1590,
"status": "ok",
"timestamp": 1584519800403,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "VPXkB5bOITqi",
"outputId": "69fdb374-4b35-442d-a934-7017245ea18e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"65 unique characters\n",
"vocab: ['\\n', ' ', '!', '$', '&', \"'\", ',', '-', '.', '3', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
]
}
],
"source": [
"# The unique characters in the file\n",
"vocab = sorted(set(text))\n",
"\n",
"print('{} unique characters'.format(len(vocab)))\n",
"print('vocab:', vocab)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "GqpuKh9HMNnf"
},
"source": [
"## Process the dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "6dj4e-AGMaV4"
},
"source": [
"### Vectorize the text\n",
"\n",
"Before feeding the text to our RNN we need to convert the text from a sequence of characters to a sequence of numbers. To do so we will detect all unique characters in the text, form a vocabulary out of it and replace each character with its index in the vocabulary."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 408
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1510,
"status": "ok",
"timestamp": 1584519800417,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "xFFpuXfGMPq2",
"outputId": "530395b0-62e2-41e5-c0bf-63b2ca9da7cf"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" '\\n': 0,\n",
" ' ' : 1,\n",
" '!' : 2,\n",
" '$' : 3,\n",
" '&' : 4,\n",
" \"'\" : 5,\n",
" ',' : 6,\n",
" '-' : 7,\n",
" '.' : 8,\n",
" '3' : 9,\n",
" ':' : 10,\n",
" ';' : 11,\n",
" '?' : 12,\n",
" 'A' : 13,\n",
" 'B' : 14,\n",
" 'C' : 15,\n",
" 'D' : 16,\n",
" 'E' : 17,\n",
" 'F' : 18,\n",
" 'G' : 19,\n",
" ...\n",
"}\n"
]
}
],
"source": [
"# Map characters to their indices in vocabulary.\n",
"char2index = {char: index for index, char in enumerate(vocab)}\n",
"\n",
"print('{')\n",
"for char, _ in zip(char2index, range(20)):\n",
" print(' {:4s}: {:3d},'.format(repr(char), char2index[char]))\n",
"print(' ...\\n}')"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 85
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1466,
"status": "ok",
"timestamp": 1584519800417,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "lQB33zI7NkRo",
"outputId": "ef5bc93f-7bad-437f-f70a-9eb53d895613"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['\\n' ' ' '!' '$' '&' \"'\" ',' '-' '.' '3' ':' ';' '?' 'A' 'B' 'C' 'D' 'E'\n",
" 'F' 'G' 'H' 'I' 'J' 'K' 'L' 'M' 'N' 'O' 'P' 'Q' 'R' 'S' 'T' 'U' 'V' 'W'\n",
" 'X' 'Y' 'Z' 'a' 'b' 'c' 'd' 'e' 'f' 'g' 'h' 'i' 'j' 'k' 'l' 'm' 'n' 'o'\n",
" 'p' 'q' 'r' 's' 't' 'u' 'v' 'w' 'x' 'y' 'z']\n"
]
}
],
"source": [
"# Map character indices to characters from vacabulary.\n",
"index2char = np.array(vocab)\n",
"print(index2char)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1367,
"status": "ok",
"timestamp": 1584519800418,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "DXUAlYmvN_Rj",
"outputId": "ef3412c4-68e0-4362-e093-feba628b7079"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"text_as_int length: 1115394\n",
"'First Citizen:\\n' --> array([18, 47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 0])\n"
]
}
],
"source": [
"# Convert chars in text to indices.\n",
"text_as_int = np.array([char2index[char] for char in text])\n",
"\n",
"print('text_as_int length: {}'.format(len(text_as_int)))\n",
"print('{} --> {}'.format(repr(text[:15]), repr(text_as_int[:15])))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "CHv5HhUuTQYS"
},
"source": [
"## Create training sequences"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1227,
"status": "ok",
"timestamp": 1584519800418,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "9tHT8IXGTTtZ",
"outputId": "489b7e07-e898-49bc-e099-a9e7c4791c8c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"examples_per_epoch: 11043\n"
]
}
],
"source": [
"# The maximum length sentence we want for a single input in characters.\n",
"sequence_length = 100\n",
"examples_per_epoch = len(text) // (sequence_length + 1)\n",
"\n",
"print('examples_per_epoch:', examples_per_epoch)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 102
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1331,
"status": "ok",
"timestamp": 1584519800779,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Pf5tMMDnUP3q",
"outputId": "f8e191f6-df0b-426b-9905-af62be4d605c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"F\n",
"i\n",
"r\n",
"s\n",
"t\n"
]
}
],
"source": [
"# Create training dataset.\n",
"char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)\n",
"\n",
"for char in char_dataset.take(5):\n",
" print(index2char[char.numpy()])"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 156
},
"colab_type": "code",
"executionInfo": {
"elapsed": 3082,
"status": "ok",
"timestamp": 1584519802697,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Ap71VjB2Vuct",
"outputId": "0c05de4a-d66c-4f20-83a7-e584d853e98d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sequences count: 11043\n",
"\n",
"'First Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou '\n",
"'are all resolved rather to die than to famish?\\n\\nAll:\\nResolved. resolved.\\n\\nFirst Citizen:\\nFirst, you k'\n",
"\"now Caius Marcius is chief enemy to the people.\\n\\nAll:\\nWe know't, we know't.\\n\\nFirst Citizen:\\nLet us ki\"\n",
"\"ll him, and we'll have corn at our own price.\\nIs't a verdict?\\n\\nAll:\\nNo more talking on't; let it be d\"\n",
"'one: away, away!\\n\\nSecond Citizen:\\nOne word, good citizens.\\n\\nFirst Citizen:\\nWe are accounted poor citi'\n"
]
}
],
"source": [
"# Generate batched sequences out of the char_dataset.\n",
"sequences = char_dataset.batch(sequence_length + 1, drop_remainder=True)\n",
"\n",
"# Sequences size is the same as examples_per_epoch.\n",
"print('Sequences count: {}'.format(len(list(sequences.as_numpy_iterator()))));\n",
"print()\n",
"\n",
"# Sequences examples.\n",
"for item in sequences.take(5):\n",
" print(repr(''.join(index2char[item.numpy()])))"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Y8spPCfe-iTn"
},
"outputs": [],
"source": [
"# sequences shape:\n",
"# - 11043 sequences\n",
"# - Each sequence of length 101\n",
"#\n",
"#\n",
"# 101 101 101\n",
"# [(.....) (.....) ... (.....)]\n",
"#\n",
"# <---------- 11043 ----------->"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "HdcrcUs4Xxso"
},
"source": [
"For each sequence, duplicate and shift it to form the input and target text. For example, say `sequence_length` is `4` and our text is `Hello`. The input sequence would be `Hell`, and the target sequence `ello`."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "9fxvXsP0XFDh"
},
"outputs": [],
"source": [
"def split_input_target(chunk):\n",
" input_text = chunk[:-1]\n",
" target_text = chunk[1:]\n",
" return input_text, target_text"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 6915,
"status": "ok",
"timestamp": 1584519807164,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "454rWIQYXXRY",
"outputId": "de3a70e1-49e8-4f68-d988-77d1856a2e60"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dataset size: 11043\n"
]
}
],
"source": [
"dataset = sequences.map(split_input_target)\n",
"\n",
"# Dataset size is the same as examples_per_epoch.\n",
"# But each element of a sequence is now has length of `sequence_length`\n",
"# and not `sequence_length + 1`.\n",
"print('dataset size: {}'.format(len(list(dataset.as_numpy_iterator()))))"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 122
},
"colab_type": "code",
"executionInfo": {
"elapsed": 6548,
"status": "ok",
"timestamp": 1584519807165,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Kuoh4tCdYCck",
"outputId": "b03368eb-4c92-4662-95fe-d64f6bec308d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input sequence size: 100\n",
"Target sequence size: 100\n",
"\n",
"Input: 'First Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou'\n",
"Target: 'irst Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou '\n"
]
}
],
"source": [
"for input_example, target_example in dataset.take(1):\n",
" print('Input sequence size:', repr(len(input_example.numpy())))\n",
" print('Target sequence size:', repr(len(target_example.numpy())))\n",
" print()\n",
" print('Input:', repr(''.join(index2char[input_example.numpy()])))\n",
" print('Target:', repr(''.join(index2char[target_example.numpy()])))"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "cp0tl0sN807l"
},
"outputs": [],
"source": [
"# dataset shape:\n",
"# - 11043 sequences\n",
"# - Each sequence is a tuple of 2 sub-sequences of length 100 (input_text and target_text)\n",
"#\n",
"#\n",
"# 100 100 100\n",
"# /(.....)\\ /(.....)\\ ... /(.....)\\ <-- input_text\n",
"# \\(.....)/ \\(.....)/ \\(.....)/ <-- target_text\n",
"#\n",
"# <----------- 11043 ------------->"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "BDYHEJ0pY1ai"
},
"source": [
"Each index of these vectors are processed as one time step. For the input at time step 0, the model receives the index for \"F\" and trys to predict the index for \"i\" as the next character. At the next timestep, it does the same thing but the RNN considers the previous step context in addition to the current input character."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 272
},
"colab_type": "code",
"executionInfo": {
"elapsed": 6469,
"status": "ok",
"timestamp": 1584519807584,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "C-0zpv53Y2o4",
"outputId": "05fb1543-308c-42f9-d2ee-003c2454772b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Step 0\n",
" input: 18 ('F')\n",
" expected output: 47 ('i')\n",
"Step 1\n",
" input: 47 ('i')\n",
" expected output: 56 ('r')\n",
"Step 2\n",
" input: 56 ('r')\n",
" expected output: 57 ('s')\n",
"Step 3\n",
" input: 57 ('s')\n",
" expected output: 58 ('t')\n",
"Step 4\n",
" input: 58 ('t')\n",
" expected output: 1 (' ')\n"
]
}
],
"source": [
"for i, (input_idx, target_idx) in enumerate(zip(input_example[:5], target_example[:5])):\n",
" print('Step {:2d}'.format(i))\n",
" print(' input: {} ({:s})'.format(input_idx, repr(index2char[input_idx])))\n",
" print(' expected output: {} ({:s})'.format(target_idx, repr(index2char[target_idx])))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1iDlp40lC5YB"
},
"source": [
"## Split training sequences into batches\n",
"\n",
"We used `tf.data` to split the text into manageable sequences. But before feeding this data into the model, we need to shuffle the data and pack it into batches."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 6256,
"status": "ok",
"timestamp": 1584519807585,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "eDq-wa5EC3wW",
"outputId": "d13224e8-062a-4df8-c232-04f60d572e8d"
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 19,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"# Batch size.\n",
"BATCH_SIZE = 64\n",
"\n",
"# Buffer size to shuffle the dataset (TF data is designed to work\n",
"# with possibly infinite sequences, so it doesn't attempt to shuffle\n",
"# the entire sequence in memory. Instead, it maintains a buffer in\n",
"# which it shuffles elements).\n",
"BUFFER_SIZE = 10000\n",
"\n",
"dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)\n",
"\n",
"dataset"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 8229,
"status": "ok",
"timestamp": 1584519809742,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "1x4puZiiOlyl",
"outputId": "1debcd12-46ba-4b5f-aa56-a162fbe76f1a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Batched dataset size: 172\n"
]
}
],
"source": [
"print('Batched dataset size: {}'.format(len(list(dataset.as_numpy_iterator()))))"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 306
},
"colab_type": "code",
"executionInfo": {
"elapsed": 9749,
"status": "ok",
"timestamp": 1584519811500,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "b_kYgvQGBO0U",
"outputId": "2dfda8dc-a501-4e5f-8835-6c87b8f77051"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1st batch: input_text: tf.Tensor(\n",
"[[39 58 59 ... 57 58 53]\n",
" [47 52 1 ... 1 63 53]\n",
" [53 61 1 ... 46 57 6]\n",
" ...\n",
" [ 1 57 43 ... 52 42 8]\n",
" [58 46 43 ... 43 1 58]\n",
" [56 43 47 ... 53 59 1]], shape=(64, 100), dtype=int64)\n",
"\n",
"1st batch: target_text: tf.Tensor(\n",
"[[58 59 56 ... 58 53 41]\n",
" [52 1 63 ... 63 53 59]\n",
" [61 1 50 ... 57 6 1]\n",
" ...\n",
" [57 43 43 ... 42 8 0]\n",
" [46 43 43 ... 1 58 53]\n",
" [43 47 45 ... 59 1 61]], shape=(64, 100), dtype=int64)\n"
]
}
],
"source": [
"for input_text, target_text in dataset.take(1):\n",
" print('1st batch: input_text:', input_text)\n",
" print()\n",
" print('1st batch: target_text:', target_text)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "UkDCH15v_2I6"
},
"outputs": [],
"source": [
"# dataset shape:\n",
"# - 172 batches\n",
"# - 64 sequences per batch\n",
"# - Each sequence is a tuple of 2 sub-sequences of length 100 (input_text and target_text)\n",
"#\n",
"#\n",
"# 100 100 100 100 100 100\n",
"# |/(.....)\\ /(.....)\\ ... /(.....)\\| ... |/(.....)\\ /(.....)\\ ... /(.....)\\| <-- input_text\n",
"# |\\(.....)/ \\(.....)/ \\(.....)/| ... |\\(.....)/ \\(.....)/ \\(.....)/| <-- target_text\n",
"#\n",
"# <------------- 64 ----------------> <------------- 64 ---------------->\n",
"#\n",
"# <--------------------------------- 172 ----------------------------------->"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ghB-VwLlD-Oz"
},
"source": [
"## Build the model\n",
"\n",
"Use [tf.keras.Sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) to define the model. For this simple example three layers are used to define our model:\n",
"\n",
"- [tf.keras.layers.Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding): The input layer. A trainable lookup table that will map the numbers of each character to a vector with `embedding_dim` dimensions;\n",
"- [tf.keras.layers.LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM): A type of RNN with size units=rnn_units (You can also use a GRU layer here.)\n",
"- [tf.keras.layers.Dense](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense): The output layer, with vocab_size outputs."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 425
},
"colab_type": "code",
"executionInfo": {
"elapsed": 9280,
"status": "ok",
"timestamp": 1584519811503,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "0cg8DlO3QjuT",
"outputId": "21a9daf0-1aca-4923-958e-317293023462"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tmp_input_array shape: (2, 8)\n",
"tmp_input_array:\n",
"[[9 0 2 1 4 6 8 8]\n",
" [3 7 9 1 0 4 8 6]]\n",
"\n",
"tmp_output_array shape: (2, 8, 5)\n",
"tmp_output_array:\n",
"[[[-0.01348125 0.04649678 0.03382711 0.03225223 0.00077888]\n",
" [ 0.01548834 0.02294052 -0.03144439 -0.02458044 -0.01775809]\n",
" [-0.01713194 0.02779663 0.02824989 -0.01959893 -0.01335952]\n",
" [-0.00085108 -0.04381631 0.04934248 0.01773458 -0.0059096 ]\n",
" [-0.04360831 0.02180418 0.04860454 -0.04669809 0.0012654 ]\n",
" [-0.04296128 0.00703833 -0.01924504 -0.03515936 -0.00462629]\n",
" [ 0.03757649 0.01776652 0.0135636 -0.00759625 0.01923021]\n",
" [ 0.03757649 0.01776652 0.0135636 -0.00759625 0.01923021]]\n",
"\n",
" [[-0.00344205 -0.04512044 -0.04339879 -0.04655287 0.04151651]\n",
" [-0.01463156 -0.04065967 -0.03041167 -0.03633127 -0.04055873]\n",
" [-0.01348125 0.04649678 0.03382711 0.03225223 0.00077888]\n",
" [-0.00085108 -0.04381631 0.04934248 0.01773458 -0.0059096 ]\n",
" [ 0.01548834 0.02294052 -0.03144439 -0.02458044 -0.01775809]\n",
" [-0.04360831 0.02180418 0.04860454 -0.04669809 0.0012654 ]\n",
" [ 0.03757649 0.01776652 0.0135636 -0.00759625 0.01923021]\n",
" [-0.04296128 0.00703833 -0.01924504 -0.03515936 -0.00462629]]]\n"
]
}
],
"source": [
"# Let's do a quick detour and see how Embeding layer works.\n",
"# It takes several char indices sequences (batch) as an input.\n",
"# It encodes every character of every sequence to a vector of tmp_embeding_size length.\n",
"tmp_vocab_size = 10\n",
"tmp_embeding_size = 5\n",
"tmp_input_length = 8\n",
"tmp_batch_size = 2\n",
"\n",
"tmp_model = tf.keras.models.Sequential()\n",
"tmp_model.add(tf.keras.layers.Embedding(\n",
" input_dim=tmp_vocab_size,\n",
" output_dim=tmp_embeding_size,\n",
" input_length=tmp_input_length\n",
"))\n",
"# The model will take as input an integer matrix of size (batch, input_length).\n",
"# The largest integer (i.e. word index) in the input should be no larger than 9 (tmp_vocab_size).\n",
"# Now model.output_shape == (None, 10, 64), where None is the batch dimension.\n",
"tmp_input_array = np.random.randint(\n",
" low=0,\n",
" high=tmp_vocab_size,\n",
" size=(tmp_batch_size, tmp_input_length)\n",
")\n",
"tmp_model.compile('rmsprop', 'mse')\n",
"tmp_output_array = tmp_model.predict(tmp_input_array)\n",
"\n",
"print('tmp_input_array shape:', tmp_input_array.shape)\n",
"print('tmp_input_array:')\n",
"print(tmp_input_array)\n",
"print()\n",
"print('tmp_output_array shape:', tmp_output_array.shape)\n",
"print('tmp_output_array:')\n",
"print(tmp_output_array)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "I7ZuvZHBD_pS"
},
"outputs": [],
"source": [
"# Length of the vocabulary in chars.\n",
"vocab_size = len(vocab)\n",
"\n",
"# The embedding dimension.\n",
"embedding_dim = 256\n",
"\n",
"# Number of RNN units.\n",
"rnn_units = 1024"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "-sojdDCAICWO"
},
"outputs": [],
"source": [
"def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
" model = tf.keras.models.Sequential()\n",
"\n",
" model.add(tf.keras.layers.Embedding(\n",
" input_dim=vocab_size,\n",
" output_dim=embedding_dim,\n",
" batch_input_shape=[batch_size, None]\n",
" ))\n",
"\n",
" model.add(tf.keras.layers.LSTM(\n",
" units=rnn_units,\n",
" return_sequences=True,\n",
" stateful=True,\n",
" recurrent_initializer=tf.keras.initializers.GlorotNormal()\n",
" ))\n",
"\n",
" model.add(tf.keras.layers.Dense(vocab_size))\n",
" \n",
" return model"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "XoPwxyAPEg6z"
},
"outputs": [],
"source": [
"model = build_model(vocab_size, embedding_dim, rnn_units, BATCH_SIZE)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 255
},
"colab_type": "code",
"executionInfo": {
"elapsed": 8897,
"status": "ok",
"timestamp": 1584519812083,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "iLnlZFgU55bQ",
"outputId": "eeece968-3a2a-4652-a0b7-d8f6944cdbc2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"sequential_1\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"embedding_1 (Embedding) (64, None, 256) 16640 \n",
"_________________________________________________________________\n",
"lstm (LSTM) (64, None, 1024) 5246976 \n",
"_________________________________________________________________\n",
"dense (Dense) (64, None, 65) 66625 \n",
"=================================================================\n",
"Total params: 5,330,241\n",
"Trainable params: 5,330,241\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 422
},
"colab_type": "code",
"executionInfo": {
"elapsed": 8704,
"status": "ok",
"timestamp": 1584519812084,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "CcaO_rO_8-GH",
"outputId": "21830f00-dc89-417a-81d8-8988354235f4"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAb0AAAGVCAIAAADls7hIAAAABmJLR0QA/wD/AP+gvaeTAAAgAElE\nQVR4nOzdeVhT17ow8LUhhBCGADJIwSCDIJPidDUIeihHa0UZnEBrW3qetor6BdSjFNAKKGirRS4W\n7FPL4bRVwQEvqMix14EKVtSKCMXWAoqonDKIDJJAEpLvj3W7by5DyE5CEuD9/eWe1lp7N7zdw1rv\nIiQSCQIAACA3HU03AAAARhmImwAAQA3ETQAAoAbiJgAAUEOTXrh161ZqaqqmmgIAANqJw+Fs27aN\nXPw/95vPnj07e/as2psEwCh29uzZ58+fa7oVI66srKysrEzTrdCMsrKyW7duSa+hDdzpzJkz6moP\nAKMeQRBbt25ds2aNphsyslavXo3Ga3DA5y4N3m8CAAA1EDcBAIAaiJsAAEANxE0AAKAG4iYAAFAD\ncRMADbh06RKLxbpw4YKmG6JiGzduJP60fv166U1XrlyJjY3F/xYKhSkpKc7OznQ63dTU1NPTs76+\nfmBpPT09U6dO3bVrlzxVi8Xi0NBQNpvNYDBsbW2Dg4MrKysRQufPn//ss8/6+vrIPfPz88lGWlhY\nKHCaEDcB0IAxnIfM3Ny8qKjo0aNHWVlZ5Mo9e/akp6fHxcXhxbCwsO++++7EiRM8Hu/XX391cnJ6\n/fr1wKLi4+MfPXokZ71isbikpOTkyZNtbW2lpaV8Pn/BggWNjY1BQUEMBiMgIKC9vR3vGRwc/Pz5\n8xs3bixdulSxc4S4CYAGBAYGdnR0LF++fKQr4vP5Pj4+I12LNAMDgyVLlri4uOjr6+M1Bw4cyM3N\nPX36tLGxMUIoNzc3Pz//zJkzc+fOpdFoNjY2BQUFnp6e/cr56aeffvnlF0pVczgcX19fJpPp4OCQ\nnJzc0dHxz3/+EyEUFRU1ffr0pUuXikQihBBBELa2tn5+flOmTFHsHCFuAjCWZWVlNTc3a7ABtbW1\nu3fvTkxMZDAYeM3Ro0dnzpzp5eUl4yg+n79jx460tDT5K6LRaNLvPRwdHRFCdXV1eDEhIaGiooJS\ngTJA3ARA3UpLS9lsNkEQX375JUIoMzPT0NCQyWQWFBS8/fbbJiYmdnZ2OTk5eOf09HQGg2FlZbVx\n40YbGxsGg+Hj43P79m28lcvl0un0iRMn4sXNmzcbGhoSBNHa2ooQio6O3r59e11dHUEQzs7OCKF/\n/etfJiYmycnJajvZ9PR0iUQSFBSEFwUCQVlZmbe3t+yj4uPjN2/ebGlpqXC9fD4fIWRiYoIXzczM\nFi5cmJaWppI3JBA3AVA3X1/fn376iVzctGnT1q1b+Xy+sbHxqVOn6urqHB0dP/roI6FQiBDicrkR\nERE8Hi8qKqq+vr68vFwkEi1atOjZs2cIofT0dOkhnhkZGYmJieRiWlra8uXLnZycJBJJbW0tQgh/\nHhGLxWo72cLCQldXVyaTiRcbGxsFAsG9e/f8/f3x/wbc3NwyMjKkw9nNmzfr6urWrVunTL137txB\nCPn6+pJrZsyY8eLFiwcPHihTLAZxEwBt4ePjY2JiYmlpGR4e3t3d3dDQQG6i0Whubm76+vru7u6Z\nmZldXV3Z2dkKVBEYGNjZ2bl7927VtVqW7u7uJ0+eODk5kWvw9x9LS8vk5OTq6uqmpqaQkJAtW7ac\nPHkS78Dn86OjozMzMxWutKmpKTc3NyoqisPhkPe5CCH8NrOqqkrhkkkQNwHQOnQ6HSGE7zcHmj17\nNpPJ/O2339TbKEU0NzdLJBLyZhMhhD8WeXh4+Pj4mJubs1isxMREFov19ddf4x3i4uI+/vhjW1tb\nhSvlcDhRUVEhISFFRUV6enrketyMpqYmhUsmDZIPCQCg5fT19VtaWjTdiuH19PSgP2MlZmNjgxDC\nr18xOp1ub2+PP+CUlpZWVVUpmQXYysoqKyvLw8Oj33oDAwOySUqC+00ARhmhUNje3m5nZ6fphgwP\nhyrpPudGRkZTpkx5+PCh9G4ikYjFYiGEsrKyrl69qqOjgzul4+9CycnJBEH8/PPPclZqaWlpamo6\ncL1AICCbpCSImwCMMsXFxRKJZN68eXiRRqMN9USvcVZWVgRBdHR0SK8MCwu7f//+48eP8SKPx3v6\n9CnulpSdnS2Rgu+p4+PjJRLJ7Nmz5az0woULgz7m42ZYW1src0YYxE0ARgGxWPzq1SuRSFRZWRkd\nHc1msyMiIvAmZ2fntra2/Px8oVDY0tLy9OlT6QPNzc0bGxvr6+u7urqEQmFRUZE6+yExmUxHR8d+\n+fC3bdtmb28fERHR0NDw8uXLmJgYPp//ySefyFNgeHi4tbV1eXn5UDvU1tZaW1uHhYUN3ISbIbvf\nqJwgbgKgbl9++eWcOXMQQjExMcHBwZmZmYcPH0YITZs27fHjx8eOHdu+fTtCaMmSJTU1NfiQnp4e\nLy8vAwMDPz8/FxeX69evky8NN23a5O/vv3btWldX17179+LnUA6HgzsqRUZGWllZubu7L126tK2t\nTf0nGxgYWF1djXtTYmZmZiUlJXZ2dt7e3ra2tnfu3CksLBy2RycmEAiam5sLCgqG2kFG98y7d+/a\n2tpOmzaNUvuHrIZ06tSpfmsAALIhhE6dOjWiVWzYsMHc3HxEqxjWqlWrVq1aNexuGzZssLW1lV5T\nU1NDo9G+//57lTSjr6/Pz88vKyuL6oGtra0MBuPQoUPSK6OioiZMmDDssQPPHe43ARgFpD+taDk+\nn3/58uWamhr8HcbZ2TkpKSkpKWnQzB2U9PX15efnd3V1hYeHUz02ISHB29uby+UihCQSSWNjY2lp\nKR4LoACImwAAVWpra8N5Pf72t7/hNbGxsatXrw4PD+/3gYiq4uLivLy8oqIi6Q6h8khNTa2oqLh0\n6RLuzllQUIDzehQWFirWEs3EzTlz5ujq6sr5RmMoH374obGxMUEQFRUV8mxVc8ZDsVh8+PBhqqlo\ntDAtY1lZmZubG+4aYm1tvW/fPrVVnZeX5+joiLukTJw4sV8+x3EiLi4uOzu7o6PDwcFB+6fp/uqr\nr8iH2ePHj5Prk5OTuVzu/v37lSk8ICDgxIkT5GB8ORUUFPT29hYXF5uZmeE1ISEh0s/vijRF+qFd\nne83AwICpk+frmQhOPfB/fv35dl68eJFExOT8+fPK1mpPH7//ff58+cjhKieozobSclbb72FEHr1\n6pX6q3ZycmKxWOqvV05o5N9vagM532+OSQPPXZPjhQiCUGd1OOOhGip68OBBUlJSZGRkd3e3hGLy\nFbU1ks/nBwQESGeX0BJa2zAASJp8vyk9dFQxsiOvCuOyRCI5c+YMOYRWtunTp+fl5b3zzjvSw8u0\njcbTMg5FaxsGAEmRuNnX1/fpp5+y2WwDA4Np06bhp/u0tDRDQ0MdHZ1Zs2ZZW1vr6ekZGhrOnDnT\nz89v0qRJDAbD1NR0586d0uXU1tZOnTrV0NAQ90orLS2VXQVCSCKRHDx40NXVVV9fn8Vi7dixQ7pA\nGVspZTzEDUhJSXF1dTUwMLCwsHBwcEhJSZFO2DUSRktaRnU2TB4lJSXu7u4sFovBYHh5eV2+fBkh\n9OGHH+IXo05OTvfv30cIffDBB0wmk8VinT9/Hg3xG/v888+ZTKaxsXFzc/P27dttbW3ln6cBjCPS\nD+1yvt/8+9//rq+vf/bs2VevXsXFxeno6Ny9e1cikezZswchdPv27e7u7tbW1iVLliCECgsLW1pa\nuru7cQ+AiooKXEhAQICjo+OTJ0+EQuEvv/wyd+5cBoPx+++/y64iPj6eIIgvvvji1atXPB4vIyMD\nSb3BlL0VdwM+cuQIuTNC6OrVqx0dHc3NzX5+foaGhgKBAG9NTk7W1dUtKCjg8Xj37t2ztrb+y1/+\nQvW1yNy5c6m+36TUyA0bNhgaGj58+LCnp6e6unrOnDnGxsYNDQ146zvvvGNtbU2WfPDgQYRQS0sL\nXly5ciVOy4hdvHjR2Ng4KSlpqIb1e7+ptoZJ5Hi/eebMmYSEhLa2tpcvX86bN4/skbdy5UpdXd0X\nL16Qe65bt458dyzjN4YQioqKOnLkyIoVK3799VcZVUvg/eY4oIL+mz09PZmZmaGhoStXrjQ1Nd21\na5eenp50KkB3d3cmkzlhwoS1a9cihNhstoWFBZPJxB9DpZNfGRsbT548mUajeXh4HDt2rKenBz8I\nD1UFn88/fPjwX//6123btpmamhoYGJibm5Olyd46lKEyHubn58+aNSsoKMjAwGDmzJnBwcE3btzA\n/dHUT2vTMqqhYfJYtWrVnj17zMzMzM3Ng4KCXr58icc1R0ZG9vX1kfV2dnbevXsXT8U17M/4wIED\nW7ZsycvLmzp16gg1G4xelL8LPXr0iMfjkZMoGRgYTJw4cdBUgDiHIJ4ICf35NnOoBAReXl4sFgvP\n2zlUFbW1tTweLyAgYNASZG8dVr+Mhz09PeR0KAihvr4+PT09XV1dxQpXFa1Ny6g9DcM/M9xL/M03\n33RxcfnHP/4RFxdHEERubm54eDj+jyj/z1geYWFhgw6IHnvU/C1Xe6xatUp6kXLc7O7uRgjt2rVL\nelJjnFNPSXp6evgPb6gq8LD8oaYckb2VqqVLlx48eLCgoGDx4sXV1dX5+fnLli3TeNwcltamZRzR\nhhUWFh48eLC6urqzs1M6dhMEsXHjxm3btl29evWvf/0rnngWb1Ltzzg6OprD4ShxBqMAHkG/detW\nTTdEA/C5S6McN3FgOnz4cHR0tGoahRBCSCQStbW1sdlsGVVcv34dIdTb2ztoCfj2cKitVCUkJNy7\ndy8iIuL169c2NjZr1qxR51RWitHatIwj0bAbN27cu3dv69atDQ0NoaGhK1as+Mc//vHGG28cOXJE\n+vNjREREXFzcN998M2nSJBMTE3t7e7xetT9jDocz0t8MNe7MmTMIoTF/moPC5y6NctzEH8cHHaKj\njOvXr4vF4pkzZ8qowtPTU0dH58cff4yMjBxYguytVFVXV9fV1bW0tNBooyYlvtamZRyJht27d8/Q\n0BAhVFVVJRQKN23ahOd97fcgaWZmFhYWlpuba2xs/NFHH5HrR+hnDMYJyt+FGAzGBx98kJOTk5mZ\n2dnZ2dfX9/z583//+98K1C0QCDo6OkQiUXl5OZfLxSn5ZFRhaWm5cuXKs2fPZmVldXZ2VlZWSneo\nlL2Vqi1btrDZbOUzEYw0rU3LqKqGDSxZKBQ2NTUVFxfjuImfUa5cudLT01NTU0N2eCJFRkb29vZe\nvHhx+fLl5EoV/ozBeCT9cV3Ofki9vb0xMTFsNptGo+FoVV1dnZaWhgfbT548uaSk5MCBAzjxvbW1\n9YkTJ3Jzc3GaZTMzs5ycHIlEkp2d7e/vb2VlRaPR8Mf3p0+fyq5CIpF0dXV9+OGHEyZMMDIy8vX1\n/fTTTxFCdnZ2Dx48kL31yJEjuM8gk8kMCgrKyMjArZ0yZUpdXd3XX3+N51m2t7fHfaGuXbs2YcIE\n8irp6em5ubnl5eXJ02vh1q1b8+fPJ1+WTZw40cfH58cffxz2QKqN3LBhg56enq2tLY1GMzExCQkJ\nqaurI0t7+fKlv78/g8FwcHD4f//v/+HerM7Ozrg/UHl5ub29vYGBga+v7x9//HHp0iVjY+N9+/YN\nbFVZWZmHh4eOjg4+l+TkZLU17OjRo9JTIfZz7tw5XGBMTIy5ubmpqenq1atx11cnJyey25NEIpkx\nY0ZsbGy/8xr0N/bZZ5/h/JWTJk2SM/UZgn5IY93Ac4f8m4PLyMiIjo4mF3t7e7du3aqvr8/j8TTY\nqn60IS3joLStYUuXLn38+PEIFQ5xc8zTrvHpWuuPP/7gcrnSL7/odDqbzRYKhUKhUCXzOqmK1qZl\n1HjDhEIh7pNUWVmJ72012x4wlkD+zUEYGBjo6ellZWU1NTUJhcLGxsZvvvnm008/DQ8Pb2xsJIYm\nO53qb7/9pvCxgKqYmJiamprff//9gw8+2Lt3r6abM15s3LiR/En3y/t35cqV2NhY/G+hUJiSkuLs\n7Eyn001NTT09Pevr6weW1tPTM3XqVOm+YjKIxeLQ0FA2m81gMGxtbYODg3F/8PPnz3/22WfS/yPP\nz88nG2lhYaHIeUrffMJzOunGjRt//etfTUxMdHV1WSyWj49PRkaGUCjUdLv+V2xsLO5tPnny5DNn\nzmi6Of9LSxoWHx+vo6MzadKkkU7Kh+A5XQp+RVNUVPTo0aOenh5y/aeffrp8+fLOzk68GBoa6urq\nWlZWhu9LgoKCqqqqBpa2bds29Od8lsMSCoUTJkwoKSnp7u5+/PjxokWLWCwWHmWblpa2cOFCcpSw\nWCx+/vz5jRs3li5dqtg8GRA3AVDKSMdNHo/H4XA0XpTC8wtJJJL9+/e7uLjw+Xy8mJOTQxBEZWWl\n7KJu3ry5ePFiSnFz2bJl5OKdO3cQQsnJyXiRy+VyOJx+tz4wvxAAY5MKE+tpJEdfbW3t7t27ExMT\nyYHLR48enTlzpuz5ePl8/o4dO9LS0uSviEajSU+UgPvz1tXV4cWEhISKigpKBcoAcROAESeRSFJT\nU3GiEzMzs5CQEHIsPKXEehpMHqiw9PR0iUQSFBSEFwUCQVlZ2bBz5MTHx2/evFmZYdN45mHcQw4h\nZGZmtnDhwrS0NAnFVOKDgrgJwIhLSEiIjY2Nj49vbm6+cePGs2fP/Pz8mpqaEELp6enSgxczMjIS\nExPJxbS0tOXLl+PEerW1tVwuNyIigsfjRUVF1dfXl5eXi0SiRYsW4fSDlIpCf/Z5EIvFI3ruhYWF\nrq6u5ExqjY2NAoHg3r17/v7+OPS7ubllZGRIh7ObN2/W1dWtW7dOmXrxc7qvry+5ZsaMGS9evHjw\n4IEyxWIQNwEYWXw+PzU1dcWKFevXr2exWF5eXl999VVra6vC49k0mzyQku7u7idPnkgPXsBj8Cwt\nLZOTk6urq5uamkJCQrZs2XLy5Em8A5/Pj46OzszMVLjSpqam3NzcqKgoDodD3ucihKZMmYIQqqqq\nUrhkEsRNAEZWdXX169evZ8+eTa6ZM2cOnU4fOCRUARpMHiiP5uZmiUQiPW0vnjzGw8PDx8fH3Nyc\nxWIlJiayWCzy/yJxcXEff/yxra2twpVyOJyoqKiQkJCioiLpyXhwM/BtvpKg3zsAI6u9vR0hZGRk\nJL3S1NS0q6tLJeVrbfJAhFBPTw/6M1ZiePyx9Oy7dDrd3t4ef8ApLS2tqqpKTU1VplIrK6usrCwP\nD49+6/GIFdwkJcH9JgAjy9TUFCHUL0qqKrGe1iYPxHCoku5zbmRkNGXKlIcPH0rvJhKJcDqLrKys\nq1ev6ujo4E7p+LtQcnIyQRA///yznJVaWlria94Pnq9BJeP9IG4CMLI8PT2NjIyk/+xv374tEAhm\nzZqFF5VJrKe1yQMxKysrgiD6TW0dFhZ2//79x48f40Uej/f06VPcLSk7O1u6myS+j8b9N6VfdMh2\n4cKFQR/zcTNwgiElQdwEYGQxGIzt27efO3fu+PHjnZ2dVVVVkZGRNjY2GzZswDtQTayntckDB2Iy\nmY6OjnguBtK2bdtw0siGhoaXL1/GxMTw+fxPPvlEngLDw8Otra3Ly8uH2qG2ttba2nrQaUtwM2T3\nG5UTxE0ARtyePXtSUlKSkpIsLCwWLlw4efJkMn8oQmjTpk3+/v5r1651dXXdu3cvfpDkcDi4d1Fk\nZKSVlZW7u/vSpUvb2toQQj09PV5eXnj2bBcXl+vXr5MvEKkWpQaBgYHV1dW4NyVmZmZWUlJiZ2fn\n7e1ta2t7586dwsLCYXt0YgKBoLm5uaCgYKgdZHTPvHv3rq2t7bRp0yi1f8hqSDDOEgCqkHrHp2sq\nR5/C4yxrampoNJqcyUyH1dfX5+fnl5WVRfXA1tZWBoNx6NAh6ZUwzhKA8ULjOfpk4/P5ly9frqmp\nwd9hnJ2dk5KSkpKSlJ89oa+vLz8/v6urS4HkYQkJCd7e3lwuFyEkkUgaGxtLS0tx/38FQNwEAKhS\nW1vbkiVLXFxc/va3v+E1sbGxq1evDg8P7/eBiKri4uK8vLyioiLpDqHySE1NraiouHTpEu7OWVBQ\nYGtr6+fnV1hYqFhLIG4CMGrExcVlZ2d3dHQ4ODicPXtW080ZxFdffUU+zB4/fpxcn5yczOVy9+/f\nr0zhAQEBJ06cIAfgy6mgoKC3t7e4uNjMzAyvCQkJkX5+V6Al0O8dgFEjJSUlJSVF061Q0OLFi3Fe\nODULDg4ODg5WbZlwvwkAANRA3AQAAGogbgIAADUQNwEAgJpBvgudPn1a/e0AYPS6deuWppsw4vAg\nxfEZHJ4/f94/c4p0J3g8XggAAIC0fuOFCIkqZtsAQOUIgjh16pT0xA8AaAl4vwkAANRA3AQAAGog\nbgIAADUQNwEAgBqImwAAQA3ETQAAoAbiJgAAUANxEwAAqIG4CQAA1EDcBAAAaiBuAgAANRA3AQCA\nGoibAABADcRNAACgBuImAABQA3ETAACogbgJAADUQNwEAABqIG4CAAA1EDcBAIAaiJsAAEANxE0A\nAKAG4iYAAFADcRMAAKiBuAkAANRA3AQAAGogbgIAADUQNwEAgBqImwAAQA3ETQAAoAbiJgAAUANx\nEwAAqIG4CQAA1BASiUTTbQAAIYQ2bNjw6NEjcrG8vNzBwcHMzAwv6urqfvvtt3Z2dhpqHQD/i6bp\nBgDwP6ytrb/++mvpNZWVleS/HR0dIWgCLQHP6UBbrFu3bqhNdDo9IiJCjW0BQBZ4TgdaxNPT8+HD\nh4P+Jh89euTi4qL+JgEwENxvAi3y3nvv6erq9ltJEMT06dMhaALtAXETaJG1a9f29fX1W6mrq/v+\n++9rpD0ADAqe04F28fHxuX37tlgsJtcQBPHs2TNbW1sNtgoAaXC/CbTLu+++SxAEuaijo+Pr6wtB\nE2gViJtAu6xevVp6kSCI9957T1ONAWBQEDeBdrGwsAgICCC/DhEEERoaqtkmAdAPxE2gddavX49f\nu+vq6r711lsTJkzQdIsA+D8gbgKts2LFCjqdjhCSSCTr16/XdHMA6A/iJtA6hoaGy5YtQwjR6fTl\ny5drujkA9AdxE2ijd955ByEUGhpqaGio6bYAMIBEPqtWrdJ0SwEAYGTJGQ8p5EOaN2/e1q1bR67F\nAEg7fvx4eHg4jUY5Zdfhw4cRQmP+t3rr1q20tLRTp05puiFjBL6ecu5M4UdpZ2e3Zs0ahZoEAGVB\nQUEMBkOBA8+cOYMQGg+/1bS0tPFwmmojf9yE95tASykWNAFQA4ibAABADcRNAACgBuImAABQA3ET\nAACogbgJAEIIXbp0icViXbhwQdMN0UZXrlyJjY3F/xYKhSkpKc7OznQ63dTU1NPTs76+fuAhPT09\nU6dO3bVrlzzli8Xi0NBQNpvNYDBsbW2Dg4PxlHznz5//7LPPBqay1jiImwAghBAk8B7Knj170tPT\n4+Li8GJYWNh333134sQJHo/366+/Ojk5vX79euBR8fHx0rM6yyYWi0tKSk6ePNnW1lZaWsrn8xcs\nWNDY2Ij7ogUEBLS3t6vsfFQB4iYACCEUGBjY0dGhhuHwfD7fx8dnpGtRlQMHDuTm5p4+fdrY2Bgh\nlJubm5+ff+bMmblz59JoNBsbm4KCAk9Pz35H/fTTT7/88gulijgcjq+vL5PJdHBwSE5O7ujo+Oc/\n/4kQioqKmj59+tKlS0UikYrOSQUgbgKgVllZWc3NzZpuhVxqa2t3796dmJhI9qU9evTozJkzvby8\nZBzF5/N37Nghfx9yhBCNRpN+Q+Lo6IgQqqurw4sJCQkVFRWUChxpEDcBQKWlpWw2myCIL7/8EiGU\nmZlpaGjIZDILCgrefvttExMTOzu7nJwcvHN6ejqDwbCystq4caONjQ2DwcBzIuGtXC6XTqdPnDgR\nL27evNnQ0JAgiNbWVoRQdHT09u3b6+rqCIJwdnZGCP3rX/8yMTFJTk7WwGkPJz09XSKRBAUF4UWB\nQFBWVubt7S37qPj4+M2bN1taWipcL5/PRwiZmJjgRTMzs4ULF6alpWnPuxSImwAgX1/fn376iVzc\ntGnT1q1b+Xy+sbHxqVOn6urqHB0dP/roI6FQiBDicrkRERE8Hi8qKqq+vr68vFwkEi1atOjZs2cI\nofT0dOmxjxkZGYmJieRiWlra8uXLnZycJBJJbW0tQgh/9JCeh057FBYWurq6MplMvNjY2CgQCO7d\nu+fv74//h+Hm5paRkSEdzm7evFlXV7du3Tpl6r1z5w5CyNfXl1wzY8aMFy9ePHjwQJliVQjiJgBD\n8vHxMTExsbS0DA8P7+7ubmhoIDfRaDQ3Nzd9fX13d/fMzMyurq7s7GwFqggMDOzs7Ny9e7fqWq0a\n3d3dT548cXJyItfg7z+WlpbJycnV1dVNTU0hISFbtmw5efIk3oHP50dHR2dmZipcaVNTU25ublRU\nFIfDIe9zEUJTpkxBCFVVVSlcsmpB3ARgeDj/PL7fHGj27NlMJvO3335Tb6NGVnNzs0QiIW82EUL6\n+voIIQ8PDx8fH3NzcxaLlZiYyGKxvv76a7xDXFzcxx9/rMzkoxwOJyoqKiQkpKioSE9Pj1yPm9HU\n1KRwyapFOUkXAGAgfX39lpYWTbdClXp6etCfsRKzsbFBCOEXtRidTre3t8cfcEpLS6uqqlJTU5Wp\n1MrKKisry8PDo996AwMDsknaAO43AVCWUChsb2+3s7PTdENUCYcq6T7nRkZGU6ZMefjwofRuIpGI\nxWIhhLKysq5evaqjo0MQBEEQ+LtQcnIyQRA///yznJVaWlqampoOXC8QCMgmaQOImwAoq7i4WCKR\nzJs3Dy/SaLShnuhHESsrK4IgOjo6pFeGhYXdv3//8ePHeJHH4z19+hR3S8rOzpbOiI7vvuPj4yUS\nyezZs+Ws9MKFC4M+5uNmWFtbK3NGKgRxEwBFiMXiV69eiUSiysrK6OhoNpsdERGBNzk7O7e1teXn\n5wuFwpaWlqdPn0ofaG5u3tjYWF9f39XVJRQKi4qKtLMfEpPJdHR0fP78ucULfuUAACAASURBVPTK\nbdu22dvbR0RENDQ0vHz5MiYmhs/nf/LJJ/IUGB4ebm1tXV5ePtQOtbW11tbWYWFhAzfhZsjuN6pO\nEDcBQF9++eWcOXMQQjExMcHBwZmZmXiyjWnTpj1+/PjYsWPbt29HCC1ZsqSmpgYf0tPT4+XlZWBg\n4Ofn5+Licv36dfJV4KZNm/z9/deuXevq6rp37178dMnhcHBHpcjISCsrK3d396VLl7a1tWnkfOUU\nGBhYXV2Ne1NiZmZmJSUldnZ23t7etra2d+7cKSwsHLZHJyYQCJqbmwsKCobaQUb3zLt379ra2k6b\nNo1S+0eQ/POyrVq1Ss6dAdAgNfxWN2zYYG5uPqJVDAvPLDSiVdTU1NBotO+//14lpfX19fn5+WVl\nZVE9sLW1lcFgHDp0SCXNGAql6wn3mwAoQguT9Kics7NzUlJSUlLSoJk7KOnr68vPz+/q6goPD6d6\nbEJCgre3N5fLVbINKqR1cXPOnDm6urpy3vkP5cMPPzQ2NiYIoqKiQp6tas4hJhaLDx8+TCm5Q15e\nnqOjIzGYyZMnK9CG8XCdgfJiY2NXr14dHh7e7wMRVcXFxXl5eUVFRdIdQuWRmppaUVFx6dIl6e6c\nGqd1cfPu3bv+/v5KFvLNN98cO3ZM/q0SNY57rampWbBgwbZt23g8nvxHrVy58vHjx05OTiwWCz8p\niEQiHo/X1NRE9YeIjfnrPHLi4uKys7M7OjocHBzOnj2r6eaMuOTkZC6Xu3//fmUKCQgIOHHiBDls\nX04FBQW9vb3FxcVmZmbK1K5yWtrvnSAIdVaHc4ipoaIHDx4kJSVFRkZ2d3crGUR0dXUNDAwMDAxc\nXFwULmSsXucRlZKSkpKSoulWqNXixYsXL16s/nqDg4ODg4PVX++wtO5+E1P+nlx2RFBhvJBIJGfO\nnCGHmsk2ffr0vLy8d955R3oYhpLy8/MVPnasXmcARpSK42ZfX9+nn37KZrMNDAymTZuGP1GlpaUZ\nGhrq6OjMmjXL2tpaT0/P0NBw5syZfn5+kyZNYjAYpqamO3fulC6ntrZ26tSphoaGuJ9HaWmp7CoQ\nQhKJ5ODBg66urvr6+iwWa8eOHdIFythKKYcYbkBKSoqrq6uBgYGFhYWDg0NKSop0ChyFKZlSDK4z\nAGoi53d3Oft2/P3vf9fX1z979uyrV6/i4uJ0dHTu3r0rkUj27NmDELp9+3Z3d3dra+uSJUsQQoWF\nhS0tLd3d3fhLWUVFBS4kICDA0dHxyZMnQqHwl19+mTt3LoPB+P3332VXER8fTxDEF1988erVKx6P\nl5GRgRC6f/8+Pkr2Vtyx7siRI+TOCKGrV692dHQ0Nzf7+fkZGhoKBAK8NTk5WVdXt6CggMfj3bt3\nz9ra+i9/+Yucl5E0d+7c6dOn91t58eJFY2PjpKSkoY6Sfr8pkUiioqKqqqqkd4DrLBk3febU0A9p\nXKF0PVUZN/l8PpPJDA8Px4s8Hk9fX3/Tpk2SP/+eu7q68KZvv/0WIUT+zeN0e7m5uXgxICBAOqbg\nGZr+/ve/y6iCx+MxmcxFixaRR+E7F/wXK3urZIi/Zz6fjxfxH39tbS1enDNnzn/8x3+QRX388cc6\nOjq9vb1yXMX/NWjcHJZ0Ui9s0Lg5zq8zxE2gAErXU5XfhR49esTj8cjJRgwMDCZOnDhoci2clYuc\nMAS/ZRtqSK+XlxeLxcJ/1UNVUVtby+PxAgICBi1B9tZh9csh1tPTQ04bgBDq6+vT09PT1dVVrHCq\nWCwWOUdVdHS07J3H7XV+/vz56dOnFWvGaHHr1i2E0Jg/TbXB11NOqoyb3d3dCKFdu3ZJT/6Jc08p\nSU9PD/85DVUFHr46VGp+2VupWrp06cGDBwsKChYvXlxdXZ2fn79s2TK1xU1pKp9xZcxc57KyskGH\nOY894+Q0tY0qvwvhP5jDhw9L39BSiuKDEolEbW1tbDZbRhX4xqS3t3fQEmRvpSohIeHNN9+MiIgw\nMTFZsWLFmjVrZPRhHEXG0nWG53RAFfnlUx6qjJv4o+2gQ0eUcf36dbFYPHPmTBlVeHp66ujo/Pjj\nj4OWIHsrVdXV1XV1dS0tLUKhsKGhITMzU7Odcv/9739/8MEHypcD1xkAOakybjIYjA8++CAnJycz\nM7Ozs7Ovr+/58+f//ve/FShKIBB0dHSIRKLy8nIul4tTV8mowtLScuXKlWfPns3Kyurs7KysrJTu\n6Cd7K1Vbtmxhs9nKj9gdiGpKMYlEwufz8/LyyJn/qBqf1xkAZcl5EyvnN8re3t6YmBg2m02j0fBf\nUXV1dVpaGh4LOHny5JKSkgMHDuAE0dbW1idOnMjNzcXpSM3MzHJyciQSSXZ2tr+/v5WVFY1GmzBh\nwtq1a58+fSq7ColE0tXV9eGHH06YMMHIyMjX1/fTTz9FCNnZ2T148ED21iNHjuDhX0wmMygoKCMj\nA7d2ypQpdXV1X3/9NY5K9vb2uI/OtWvXJkyYQF5APT09Nze3vLw8eS7jrVu35s+fT77znThxoo+P\nz48//oi3Xrp0ydjYeN++fQMPPHfu3MCP6aRdu3ZJJBK4zhh8TwcK0Fg/pHEiIyMjOjqaXOzt7d26\ndau+vj6Px9Ngq8Yeha/zOPmtQtxULY31QxoP/vjjDy6XK/3ij06ns9lsoVAoFAq1Z/6T0Q6uM9Bm\nWjo+XWsZGBjo6ellZWU1NTUJhcLGxsZvvvnm008/DQ8Pb2xsHDTPG6ZA2sHxTMZ1VvhlLgCqAnGT\nGhaL9cMPP/zyyy8uLi4GBgbu7u7Z2dkHDhz49ttvp06dKuPGPjc3V9NtH01kXGdNN220unLlSmxs\nLP63UChMSUlxdnam0+mmpqaenp719fUDD+np6Zk6dap0H14ZxGJxaGgom81mMBi2trbBwcF4DMWw\nkpKS3N3dTUxM9PX1nZ2dd+7cKf0xcN++ff1uQcjhGLLP5fz585999tkIpZeG53TK/Pz8/vu//1vT\nrRj74Dqr0J49e+7fv3/ixAm8GBYW9vDhwxMnTsyaNaulpWXjxo2D9luIj49/9OiRnFWIxeKSkpL8\n/PyZM2c2NTVt2LBhwYIFDx8+fOONN2QfeO3atS1btoSHh+vp6RUVFa1fv76qqqqoqEjOeoc6l6Cg\noCdPngQEBOTn5w86t7BS5HwPOk7etYMxQA2/VR6Px+FwNFuU/N8x9u/f7+LiQuYByMnJIQiisrJS\n9lE3b97EOTfxXL7DEgqFy5YtIxdxMoTk5ORhDwwMDBSJROQiTnnV0NCAF/fu3StjgqNhz4XL5XI4\nHKFQOGwzYH4hAEZWVlZWc3OzthU1qNra2t27dycmJpKD/Y8ePTpz5kzZc+ry+fwdO3ZQGshLo9Gk\nZ0BxdHRECNXV1Q174MWLF6WHz1pYWCCE5JwNYdhzSUhIqKioUPmIZIibYJySSCSpqalubm76+vpm\nZmYhISFkDhoul0un08lJHTZv3mxoaEgQRGtrK0IoOjp6+/btdXV1BEE4Ozunp6czGAwrK6uNGzfa\n2NgwGAwfH5/bt28rUBRSOgfrQOnp6RKJJCgoCC8KBIKysrJh55WKj4/fvHmzMqkG8OzBCnzEe/Hi\nhYGBgYODw7B7ynMuZmZmCxcuTEtLk6h0jhaIm2CcSkhIiI2NjY+Pb25uvnHjxrNnz/z8/JqamhBC\n6enp0gmSMzIyEhMTycW0tLTly5c7OTlJJJLa2loulxsREcHj8aKiourr68vLy0Ui0aJFi3DSPEpF\noT+nyRSLxao6zcLCQldXV3ISqsbGRoFAcO/ePX9/fxzl3dzcMjIypMPKzZs36+rq1q1bp0y9+Dnd\n19eX0lE8Hu/atWsfffQRTo6FxcbGmpmZ0el0BweHkJCQu3fvyn8uCKEZM2a8ePHiwYMHypxOPxA3\nwXjE5/NTU1NXrFixfv16Fovl5eX11Vdftba2KjwwlEaj4VtXd3f3zMzMrq6u7OxsBcoJDAzs7Ozc\nvXu3Ys3op7u7+8mTJ9IjzfA3E0tLy+Tk5Orq6qamppCQkC1btpw8eRLvwOfzo6OjMzMzFa60qakp\nNzc3KiqKw+GQ97lySklJsbGx2bdvH7nm/fffP3/+/LNnz16/fp2Tk9PQ0LBw4cLq6mp5zgWbMmUK\nQqiqqkrhMxoI4iYYj6qrq1+/fj179mxyzZw5c+h0Ovl8rYzZs2czmcxBM8+qWXNzs0QikZ7xFE9s\n5eHh4ePjY25uzmKxEhMTWSwW+T+MuLi4jz/+2NbWVuFKORxOVFRUSEhIUVERpQmszp07d/r06cuX\nLxsbG5MrJ02aNGPGDCMjIzqdPm/evOzsbD6fj1NcD3suGD59/CShKtAPCYxHOPezkZGR9EpTU9Ou\nri6VlK+vr9/S0qKSopTR09OD/owvGM6NgN+uYnQ63d7eHn/AKS0traqqSk1NVaZSKyurrKwsDw8P\nSkfl5uampqYWFxfL7rfk5eWlq6v7+++/o+HOhYRHl+FLoSpwvwnGI9yhr1+UbG9vt7OzU75woVCo\nqqKUhEOGdN9vIyOjKVOmPHz4UHo3kUiEU8BkZWVdvXpVR0cH9zDH34WSk5MJgvj555/lrNTS0pJq\nf8kjR44cP3782rVrw3b2FIvFYrEY/59A9rmQBAIB+vNSqArETTAeeXp6GhkZSceC27dvCwSCWbNm\n4UUajTbUhCLDKi4ulkgk8+bNU74oJVlZWREE0W/O+rCwsPv37z9+/Bgv8ni8p0+f4q482dnZ0r0U\n8S0z7r8p/U5DtgsXLsj/mC+RSGJiYqqqqvLz8/vd/mNvvfWW9CKeHJDD4Qx7LiR8+jgZmKpA3ATj\nEYPB2L59+7lz544fP97Z2VlVVRUZGWljY7Nhwwa8g7Ozc1tbW35+vlAobGlpefr0qfTh5ubmjY2N\n9fX1XV1dOCaKxeJXr16JRKLKysro6Gg2m40zmVItimoOVtmYTKajoyOev4S0bds2nGi1oaHh5cuX\nMTExfD7/k08+kafA8PBwa2vr8vLyoXaora21trbuN3uHjKMePnz4+eefHzt2TE9PT3ow5aFDh/AO\nL168yM3NbW9vFwqFt27d+vDDD9lsdmRkpPzngk9fdn9VqiBugnFqz549KSkpSUlJFhYWCxcunDx5\ncnFxsaGhId66adMmf3//tWvXurq67t27Fz/lcTgc3LsoMjLSysrK3d196dKlbW1tCKGenh4vLy88\nDb2Li8v169fJt4pUi1KtwMDA6upq3JsSMzMzKykpsbOz8/b2trW1vXPnTmFh4bA9OjGBQNDc3FxQ\nUDDUDoN2k5Rx1LDdKpcsWbJr1y47Ozsmk7lmzZr58+eXlZWRiVnlOZe7d+/a2tpOmzZt+NOTn5zj\nimCcJRgt1P9b3bBhg7m5uTprlMg9LrCmpoZGo8kYqkhJX1+fn59fVlaWGo5SidbWVgaDcejQoWH3\nhHGWAKjbCOXdUZ6zs3NSUlJSUpLyM4709fXl5+d3dXVRSoqo2FGqkpCQ4O3tzeVyVVssxE0AxrjY\n2NjVq1eHh4f3+0BEVXFxcV5eXlFRkXSH0BE6SiVSU1MrKiouXbpEqRupPCBuAqCUuLi47Ozsjo4O\nBweHs2fParo5g0tOTuZyufv371emkICAgBMnTpBj7Uf0KOUVFBT09vYWFxePxDSo0O8dAKWkpKSk\npKRouhXDW7x4Mc4LN04EBwcHBwePUOFwvwkAANRA3AQAAGogbgIAADUQNwEAgBoK34XKyspWr149\nck0BQCXKysoQQmP+t4qHD47501SbfqNRZSMk8qWPT01NvXXrlqJNAoCyoqKiGTNmqL//ChjPzpw5\nI89u8sZNANSMIIhTp05JTzIBgJaA95sAAEANxE0AAKAG4iYAAFADcRMAAKiBuAkAANRA3AQAAGog\nbgIAADUQNwEAgBqImwAAQA3ETQAAoAbiJgAAUANxEwAAqIG4CQAA1EDcBAAAaiBuAgAANRA3AQCA\nGoibAABADcRNAACgBuImAABQA3ETAACogbgJAADUQNwEAABqIG4CAAA1EDcBAIAaiJsAAEANxE0A\nAKAG4iYAAFADcRMAAKiBuAkAANRA3AQAAGogbgIAADUQNwEAgBqaphsAwP9ob2+XSCTSa7q7u1+9\nekUuGhkZ6enpqb1dAPRH9PulAqApb7755vXr14faqqur++LFC2tra3U2CYBBwXM60BZr164lCGLQ\nTTo6OgsWLICgCbQExE2gLVatWkWjDf7iiCCI9957T83tAWAoEDeBtjAzM1u8eLGuru7ATTo6OqGh\noepvEgCDgrgJtMj69evFYnG/lTQaLTAwkMViaaRJAAwEcRNokaCgIH19/X4r+/r61q9fr5H2ADAo\niJtAizCZzNDQ0H6djQwMDJYuXaqpJgEwEMRNoF3WrVsnFArJRT09vVWrVhkYGGiwSQD0A3ETaJe3\n3npL+lWmUChct26dBtsDwEAQN4F20dPTCw8Pp9PpeNHU1DQgIECzTQKgH4ibQOusXbtWIBAghPT0\n9NavXz9Up04ANAXGWQKtIxaL33jjjaamJoRQaWnp/PnzNd0iAP4PuN8EWkdHR+fdd99FCNnY2Pj4\n+Gi6OQD0N+qfgE6fPq3pJgDVs7CwQAjNnTv3zJkzmm4LUD0fHx87OztNt0Jxo/45fahMEAAArXXq\n1Kk1a9ZouhWKG/X3m2j0/zfQWgRBaPDanj17dtWqVWqoaPXq1QghuLFVmzFwrwPvN4GWUk/QBEAB\nEDcBAIAaiJsAAEANxE0AAKAG4iYAAFADcRMAAKiBuAlU7NKlSywW68KFC5puiJpcuXIlNjYW/1so\nFKakpDg7O9PpdFNTU09Pz/r6+oGH9PT0TJ06ddeuXfKULxaLQ0ND2Ww2g8GwtbUNDg6urKyU58Ck\npCR3d3cTExN9fX1nZ+edO3e+fv2a3Lpv3z7i//L09JQ+fKhzOX/+/GeffdbX1ydPG8YqiJtAxUb7\nSApK9uzZk56eHhcXhxfDwsK+++67EydO8Hi8X3/91cnJSTpUkeLj4x89eiRnFWKxuKSk5OTJk21t\nbaWlpXw+f8GCBY2NjcMeeO3atS1bttTX17e2tqakpKSlpeGeqnIa6lyCgoIYDEZAQEB7e7v8pY01\nklEOIXTq1ClNt2Js0vJry+PxOByO8uWsWrVq1apVChy4f/9+FxcXPp+PF3NycgiCqKyslH3UzZs3\nFy9ejBCKj4+XpxahULhs2TJy8c6dOwih5OTkYQ8MDAwUiUTkIh6/0NDQgBf37t37/fffD3XssOfC\n5XI5HI5QKJTnFPrR8t+VPOB+E4xWWVlZzc3Nmqq9trZ29+7diYmJDAYDrzl69OjMmTO9vLxkHMXn\n83fs2JGWliZ/RTQaTfqlh6OjI0Korq5u2AMvXrwoPTkoHvLP4/HkqXTYc0lISKioqKB0ImMJxE2g\nSqWlpWw2myCIL7/8EiGUmZlpaGjIZDILCgrefvttExMTOzu7nJwcvHN6ejqDwbCystq4caONjQ2D\nwfDx8bl9+zbeyuVy6XT6xIkT8eLmzZsNDQ0JgmhtbUUIRUdHb9++va6ujiAIZ2dnhNC//vUvExOT\n5ORk9Zxpenq6RCIJCgrCiwKBoKyszNvbW/ZR8fHxmzdvtrS0VLhePp+PEDIxMaF64IsXLwwMDBwc\nHIbdU55zMTMzW7hwYVpammQ8vZYhQdwEquTr6/vTTz+Ri5s2bdq6dSufzzc2Nj516lRdXZ2jo+NH\nH32EZxDicrkRERE8Hi8qKqq+vr68vFwkEi1atOjZs2cIofT0dOmh8RkZGYmJieRiWlra8uXLnZyc\nJBJJbW0tQgh/qRg4jfAIKSwsdHV1ZTKZeLGxsVEgENy7d8/f3x//P8DNzS0jI0M6rNy8ebOurk7J\naT/wc7qvry+lo3g83rVr1z766CMykT5CKDY21szMjE6nOzg4hISE3L17V/5zQQjNmDHjxYsXDx48\nUOZ0RimIm0AdfHx8TExMLC0tw8PDu7u7GxoayE00Gs3NzU1fX9/d3T0zM7Orqys7O1uBKgIDAzs7\nO3fv3q26Vg+pu7v7yZMnTk5O5Br8zcTS0jI5Obm6urqpqSkkJGTLli0nT57EO/D5/Ojo6MzMTIUr\nbWpqys3NjYqK4nA45H2unFJSUmxsbPbt20euef/998+fP//s2bPXr1/n5OQ0NDQsXLiwurpannPB\npkyZghCqqqpS+IxGL4ibQK3w/Y70jJXSZs+ezWQyf/vtN/U2irLm5maJRELebCKE8LTvHh4ePj4+\n5ubmLBYrMTGRxWJ9/fXXeIe4uLiPP/7Y1tZW4Uo5HE5UVFRISEhRUVG/qZJlO3fu3OnTpy9fvmxs\nbEyunDRp0owZM4yMjOh0+rx587Kzs/l8fkZGhjznguHTx2n5x5uxkEcOjCX6+votLS2absUwenp6\n0J/xBbOxsUEI4XevGJ1Ot7e3xx9wSktLq6qqUlNTlanUysoqKyvLw8OD0lG5ubmpqanFxcVvvPGG\njN28vLx0dXV///13NNy5kPDkzPhSjDdwvwm0iFAobG9v1/5M4DhkSPf9NjIymjJlysOHD6V3E4lE\neE7jrKysq1ev6ujo4B7m+LtQcnIyQRA///yznJVaWlqamppSaueRI0eOHz9+7do12UETISQWi8Vi\nMf4/gexzIeG588bn1PYQN4EWKS4ulkgk8+bNw4s0Gm2oJ3rNsrKyIgiio6NDemVYWNj9+/cfP36M\nF3k83tOnT3FXnuzsbOnef/iGGvffnD17tpyVXrhwQf7HfIlEEhMTU1VVlZ+fb2RkNHCHt956S3rx\n7t27EomEw+EMey4kfPrW1tZyNmksgbgJNEwsFr969UokElVWVkZHR7PZ7IiICLzJ2dm5ra0tPz9f\nKBS2tLQ8ffpU+kBzc/PGxsb6+vquri6hUFhUVKS2fkhMJtPR0fH58+fSK7dt22Zvbx8REdHQ0PDy\n5cuYmBg+n//JJ5/IU2B4eLi1tXV5eflQO9TW1lpbW4eFhcl51MOHDz///PNjx47p6elJD6Y8dOgQ\n3uHFixe5ubnt7e1CofDWrVsffvghm82OjIyU/1zw6cvurzpWQdwEqvTll1/OmTMHIRQTExMcHJyZ\nmXn48GGE0LRp0x4/fnzs2LHt27cjhJYsWVJTU4MP6enp8fLyMjAw8PPzc3FxuX79OvnecNOmTf7+\n/mvXrnV1dd27dy9+JORwOLijUmRkpJWVlbu7+9KlS9va2tR8poGBgdXV1bg3JWZmZlZSUmJnZ+ft\n7W1ra3vnzp3CwsJhe3RiAoGgubm5oKBgqB0G7SYp46hhu1UuWbJk165ddnZ2TCZzzZo18+fPLysr\nmzBhgvzncvfuXVtb22nTpg1/emOPGscmjQg0+sdsaS01XNsNGzaYm5uPaBXDUmycZU1NDY1GkzFU\nkZK+vj4/P7+srCw1HKUSra2tDAbj0KFDChw7Bv5m4X4TaNgozazj7OyclJSUlJQ0aOYOSvr6+vLz\n87u6usLDw0f6KFVJSEjw9vbmcrnqr1objP24eejQIfwW/6uvvlJ/7Xl5eY6OjvjV0sSJE9evXz/U\nng8ePAgPD3dwcNDX17ewsJg+fTrZSzk8PJyQ6eLFi9IVDdX3OzU1lSAIHR2dqVOn3rhxY0ROeDyJ\njY1dvXp1eHh4vw9EVBUXF+fl5RUVFUl3CB2ho1QiNTW1oqLi0qVLlLqRjimavuFVFpLjnh+/Sjt6\n9Kh6mjSQk5MTi8WSsUNlZSWTyYyKinry5Amfz3/06NHOnTsDAgLw1rCwsB9++AG/wv/3v/+NEAoK\nChIIBN3d3c3NzR999NGFCxfIihBCEydOFAgE/aoQiUT29vYIIbLYYclzbZURGxuLu8FPnjz5zJkz\nI1eRbArnQ8IuX74cExOjwvZoufz8/JSUFOlMS1SN9O9KDcb+/aac+Hy+j4+Ppmo/dOiQqalpWlra\n5MmTGQyGi4sL+RkEIUQQxPz581ksFo1GI9fo6ekxmUxLS8tZs2ZJFzVr1qw//vgjPz+/XxV5eXnK\nDFYZCSkpKb29vRKJ5MmTJ6N31t/FixcfOHBA061Qn+Dg4NjYWOlMS+MQxM3/odmkZC9fvuzo6JD+\nKEyn08nsYTk5OTKexTZs2LBs2TJycdOmTQiho0eP9tstNTUVf8sGAChpPMbNH3/88T/+4z+YTKaJ\niYmXl1dnZ2e/pGRpaWmGhoY6OjqzZs2ytrbW09MzNDScOXOmn5/fpEmTGAyGqanpzp07yQKVz2A2\nZ86c7u7uN9988+bNm0qe3Ztvvunm5nb9+nXpjOI3b97k8Xg4XS4AQEnjLm52d3cHBQWtWrWqra2t\npqbGxcVFIBD0S0oWHR29Y8cOiURy9OjRJ0+e/PHHHwsWLLh//35sbOz9+/fb2tref//9gwcPkhm0\nlM9gtnPnztmzZz948MDX19fDw+Pzzz9XpkPixo0bEULS38G++OKLbdu2KVwgAEDauIub9fX1nZ2d\nHh4eDAbD2to6Ly8P58EelLu7O5PJnDBhwtq1axFCbDbbwsKCyWTiz+Jk2h7lM5gZGBj89NNP//mf\n/zl16tSHDx/GxMS4ubn9+OOPipX2/vvvGxoafvvtt7hX9uPHj+/evatk2kcAAGnc5UNydHS0srJa\nv359VFRURETE5MmT5TkKf/YViUR4EXe/UO3QaT09PS6Xy+Vyb9++feDAgfz8/NWrVz969MjMzIxq\nUSwWa926dceOHcvNzf3ggw8OHz68adMmOp2OEzFQcvjw4TNnzlA9anQpKytDCFGaswyMc+PuftPA\nwODatWu+vr7JycmOjo7h4eHSQ+W0wdy5c//rv/4rMjKypaXl+vXrihWCvw599dVX7e3tZ86cwU/u\nAACVGHf3mwghDw+PCxcutLS0pKamHjhwwMPDQz1Jwvu5cePGvXv3tm7dihBauXLlqVOnyG5GCKF3\n33336NGjcs6iNZC3t/e8efPKyso2bNiwevVqBW5asa1bt0pPVjEmqh5d5QAAIABJREFU4TvNMX9b\nrT0IgtB0E5Q17u43GxsbcWJBS0vL/fv3z5w5s1+eQbW5d++eoaEh/ndvb2+/ZuCv4cokTcC3nGfP\nnsWhGQCgKuMxbm7cuPG3334TCAT3799/+vQpzvbYLykZpTKpZjATCoVNTU3FxcVk3EQIhYaGnj59\nur29vaOjo6Cg4JNPPgkODlYmbq5Zs8bCwiI0NBTPHAsAUBkNj1dSGhpuzNYXX3yBU6saGhquWLGi\nvr7ex8fHzMxMV1f3jTfeiI+PxyPGysvL7e3tDQwMfH19Y2NjcT/zyZMnl5SUHDhwAGe6tra2PnHi\nRG5uLi7QzMwsJydHIpFcunTJ2Nh43759A2s/d+6c9Oxd/Zw7dw7v9sMPP4SFhTk5Oenr69PpdFdX\n14SEhJ6eHumiOjs7FyxYYG5ujhDS0dFxdnZOTk4eWJGFhcWWLVvwyp07d/7000/437t27cJz6uro\n6Li7u5eUlCh/bccGJcdZAqrGwO+KkIzy6Y8Jgjh16tSYfwenEePk2sL7TTUbA7+rcfecDgAASoK4\nCQA1V65ciY2Nxf8WCoUpKSnOzs50Ot3U1NTT07O+vn7gIT09PVOnTt21a5c85YvF4tDQUDabzWAw\nbG1tg4ODKysr5W+eWCw+fPjwoElqSktL58+fz2QybWxsYmJient7yU1JSUnu7u4mJib6+vrOzs47\nd+4cKq9ov3M5f/78Z599NkqTqCoM4iYAFOzZsyc9PT0uLg4vhoWFfffddydOnODxeL/++quTk9Og\n4SY+Pl46XYBsYrG4pKTk5MmTbW1tpaWlfD5/wYIFjY2N8hxbU1OzYMGCbdu2DezBVl1dvXjx4oCA\ngJaWlnPnzv3jH/8gZxNCCF27dm3Lli319fWtra0pKSlpaWlDDQTody5BQUEMBiMgIKC9vV3OExwL\nNP2CVVlo9L9j1lojfW15PB6Hw9F4UfJ/F9q/f7+Liwufz8eLOTk5BEFUVlbKPurmzZs4owqewHJY\nQqFw2bJl5OKdO3cQQtLfAIdSUVGxYsWK48ePe3t7T58+vd/WsLAwBwcHsViMFw8ePEgQxK+//ooX\nAwMDpVNq4pePDQ0Ncp4Ll8vlcDhCoVCeExwDf7Nwvwk0RoW5+9SQBrC2tnb37t2JiYkMBgOvOXr0\n6MyZM2VP6Mjn83fs2JGWliZ/RTQajUwhiBDC3cjq6uqGPXD69Ol5eXnvvPMOObEdSSQSFRYWLly4\nkOxz/vbbb0skEnJOt4sXL0qn1MRJG/rdtMo4l4SEhIqKCkqnOapB3ARKkUgkqampbm5u+vr6ZmZm\nISEhZLoTLpdLp9Nx5yeE0ObNmw0NDQmCaG1tRQj1y92Xnp7OYDCsrKw2btxoY2PDYDB8fHxu376t\nQFFIFZn9BkpPT5dIJEFBQXhRIBCUlZUNO11lfHz85s2bLS0tFa4XjwM2MTFRuASE0OPHj1+/fs1m\ns8k1uNfaUG9OX7x4YWBg4ODgIL1SxrmYmZktXLgwLS1NMsr758gJ4iZQSkJCQmxsbHx8fHNz840b\nN549e+bn59fU1IQQSk9Pl+5rkpGRkZiYSC72y93H5XIjIiJ4PF5UVFR9fX15eblIJFq0aBGe8pdS\nUUgVmf0GKiwsdHV1JRNINzY2CgSCe/fu+fv740Dv5uaWkZEhHThu3rxZV1enZCYq/Jzu6+urTCF/\n/PEHQsjY2Jhcw2AwDAwM8H+pfng83rVr1z766COczgYb9lxmzJjx4sULMrni2AZxEyiOz+enpqau\nWLFi/fr1LBbLy8vrq6++am1t/frrrxUrkEaj4VtXd3f3zMzMrq6u7OxsBcpRPrNfP93d3U+ePJEe\nwoC//1haWiYnJ1dXVzc1NYWEhGzZsuXkyZN4Bz6fHx0dnZmZqXClTU1Nubm5UVFRHA6HvM9VDP50\n3m9yCz09vUGT2qSkpNjY2JDTAiL5zmXKlCkIoaqqKmXaOVpA3ASKq66ufv369ezZs8k1c+bModPp\n5PO1MmbPns1kMsmnfs1qbm6WSCTSs5Xgd4geHh4+Pj7m5uYsFisxMZHFYpH/z4iLi/v444+VmdOJ\nw+FERUWFhIQUFRUpOXMkfidLJkLEBAIBOYcV6dy5c6dPn758+bL0zak854IvzqA3sGPPeMyHBFQF\ndz0xMjKSXmlqatrV1aWS8vX19VtaWlRSlJJ6enrQn7ESs7GxQQjhF6wYnU63t7fHH3BKS0urqqpS\nU1OVqdTKyiorK8vDw0OZQjD8arizs5Ncw+Pxenp68FmQcnNzU1NTi4uL33jjDXKlnOeCQzC+UGMe\n3G8CxZmamiKE+kXJ9vZ2Ozs75QsXCoWqKkp5OChI9+42MjKaMmVKvyxWIpEIpzLIysq6evWqjo4O\nntEef0tJTk4mCOLnn3+Ws1JLS0t8hZXn4OBgbGz89OlTcg1+ESydOObIkSPHjx+/du2adNBEcp8L\nzoo98AZ2TIK4CRTn6elpZGQk/cdz+/ZtgUBATk1Mo9EUzopfXFwskUhwtioli1KelZUVQRAdHR3S\nK8PCwu7fv//48WO8yOPxnj59irslZWdnS3f3w3fNuM+j9GsN2S5cuKCqqZtpNNrSpUtv3LhBfisr\nKioiCAK/NpVIJDExMVVVVfn5+f2eHuQ/F3xxcMqbMQ/iJlAcg8HYvn37uXPnjh8/3tnZWVVVFRkZ\naWNjs2HDBryDs7NzW1tbfn6+UChsaWmRvt9Bg+XuE4vFr169EolElZWV0dHRbDY7IiJCgaKoZvYb\nFpPJdHR0fP78ufTKbdu22dvbR0RENDQ0vHz5MiYmhs/nf/LJJ/IUGB4ebm1tXV5ePtQOtbW11tbW\nYWFhlI6SYffu3U1NTXv27Onu7r5169bBgwcjIiJcXV0RQg8fPvz888+PHTump6dHSDl06JD85eOL\nI7s365gBcRMoZc+ePSkpKUlJSRYWFgsXLpw8ebJ0XtFNmzb5+/uvXbvW1dV17969+CGOw+Hg3kWR\nkZFWVlbu7u5Lly7F83f29PR4eXkZGBj4+fm5uLhcv36dfKVItSiVCwwMrK6ulv4AbWZmVlJSYmdn\n5+3tbWtre+fOncLCwmF7dGICgaC5uZnsdj7QoB0hZR9VVlbm6+v7xhtv3L59+8GDBzY2NvPnz79x\n4wbe6uHhcfny5R9++GHChAkrV67829/+dvToURl1UXX37l1bW1tlMsaOJiM7HGnkodE/Zktrqfna\nbtiwwdzcXG3VkeQcZ1lTU0Oj0b7//nuVVNrX1+fn55eVlaWGo9SgtbWVwWAcOnRInp3HwN8s3G8C\nLaLNaXWcnZ2TkpKSkpKGShQkv76+vvz8/K6urvDw8JE+Sj0SEhK8vb25XK6mG6ImEDcBkFdsbOzq\n1avDw8P7fSCiqri4OC8vr6ioSLpD6AgdpQapqakVFRWXLl1SspPpKAJxE2iFuLi47Ozsjo4OBweH\ns2fParo5Q0pOTuZyufv371emkICAgBMnTpDD7Uf0qJFWUFDQ29tbXFys8JypoxH0ewdaISUlJSUl\nRdOtkMvixYtxLjWAEAoODg4ODtZ0K9QN7jcBAIAaiJsAAEANxE0AAKAG4iYAAFADcRMAAKghJKM8\nrz05XwoAYLQ4deqUdAL/UWfU90M6deqUppsARkRYWFh0dDSHw9F0Q4DqDTq9+ygy6u83wVhFEMRo\nvysBYxW83wQAAGogbgIAADUQNwEAgBqImwAAQA3ETQAAoAbiJgAAUANxEwAAqIG4CQAA1EDcBAAA\naiBuAgAANRA3AQCAGoibAABADcRNAACgBuImAABQA3ETAACogbgJAADUQNwEAABqIG4CAAA1EDcB\nAIAaiJsAAEANxE0AAKAG4iYAAFADcRMAAKiBuAkAANRA3AQAAGogbgIAADUQNwEAgBqImwAAQA3E\nTQAAoAbiJgAAUANxEwAAqIG4CQAA1NA03QAA/kdOTk5XV5f0mitXrrS3t5OLoaGhlpaWam8XAP0R\nEolE020AACGEIiIivv32Wz09PbyIf5kEQSCE+vr6jIyMmpub9fX1NdlEABBC8JwOtMfatWsRQsI/\niUQikUiE/62rq7t69WoImkBLwP0m0BYikcja2rqtrW3QrVevXn3zzTfV3CQABgX3m0Bb0Gi0tWvX\nks/p0iwsLBYuXKj+JgEwKIibQIusXbtWKBT2W6mnp/fuu+/q6upqpEkADATP6UCLSCQSNpv9/Pnz\nfuvv3LkzZ84cjTQJgIHgfhNoEYIg1q9f3+9RfdKkSbNnz9ZUkwAYCOIm0C79HtX19PQiIiJwbyQA\ntAQ8pwOtM3Xq1EePHpGLv/zyi4eHhwbbA0A/cL8JtM67775LPqq7u7tD0ATaBuIm0Drr168XiUQI\nIT09vffff1/TzQGgP3hOB9po9uzZ9+7dIwiivr6ezWZrujkA/B9wvwm00XvvvYcQmjt3LgRNoIVG\nZT6kW7dupaamaroVYAT19PQQBNHb27t69WpNtwWMIA6Hs23bNk23grJReb/57Nmzs2fParoV40hZ\nWVlZWZk6a2QwGNbW1nZ2duqs9Pnz5/C7UqeysrJbt25puhWKGJX3m9iZM2c03YTxAt/0qfmC19bW\nOjs7q7PG06dPh4WFwe9KbUbvw8SovN8E44GagyYA8oO4CQAA1EDcBAAAaiBuAgAANRA3AQCAGoib\nYKRcunSJxWJduHBB0w0ZKVeuXImNjcX/FgqFKSkpzs7OdDrd1NTU09Ozvr5+4CE9PT1Tp07dtWuX\nPOWLxeLQ0FA2m81gMGxtbYODgysrK+VvnlgsPnz4sI+Pz8BNpaWl8+fPZzKZNjb/v71zD2rq+B74\nXkzz4hGCEkQQ5CUKgmhtSxBEh5FWaYX6Aq0zpY4v1AlUaxEiCqlBrRYzVLBTm6ZjfSA+BlREO1VR\n8V0VofgEBFQqAUFeCSQh9/fHjvnmhxJuAglJ2M9f7t7ds2cz18Pe3bPnOCYmJnZ1dake8Xg8Hx8f\nGxsbCoXi6en5/ffft7e3v1d+j7mcPHlyx44d3d3dxDU0XZDdROgL877Cu2XLlszMzOTkZFiMjo7e\nv3//wYMHJRLJw4cPPTw83mtuuFyueqgnzSiVyitXrhw6dKipqam4uFgqlU6bNq2uro5I36dPn06b\nNm3dunUSiaTHo/Ly8vDw8LCwsIaGhhMnTvz+++9xcXGqpxcuXFi7dm11dXVjY2N6erpAIOjNW6jH\nXObMmUOlUsPCwtRTN5stuAly5MgRE9XcRJk/f/78+fMHW4tekUgkbDa7/3KIv1fbtm0bO3asVCqF\nxcOHD2MYVlpaqrnX1atXw8PDAQBcLpfIKHK5/PPPP1cVb926BQDg8/l9diwpKZk7d+6BAwcCAgIm\nTpzY42l0dLSbm5tSqYTFnTt3Yhj28OFDWIyIiFAoFKrGCxcuBADU1tYSnAuHw2Gz2XK5nMgEjfy9\n0gBabyJMHqFQKBaLDTZcRUVFSkpKWloalUqFNXv37p08ebKfn5+GXlKpdMOGDQKBgPhAJBJJfZfD\n3d0dAFBZWdlnx4kTJx4/fvyrr756N3OyQqEoKCgIDQ1VhYKeNWsWjuP5+fmwePr0afVUTiNGjAAA\n9Fi0aphLampqSUmJVtM0RZDdROiF4uJiFxcXDMP27NkDAMjOzra0tKTT6fn5+bNmzbKxsXF2dj58\n+DBsnJmZSaVSWSzWqlWrHB0dqVRqUFDQzZs34VMOh0Mmk0eOHAmLa9assbS0xDCssbERAJCQkLB+\n/frKykoMw6Cr/NmzZ21sbPh8vp6mlpmZieP4nDlzYFEmk924cSMgIEBzLy6Xu2bNGnt7e53HlUql\nAAAbGxudJQAAqqqq2tvb1aOleHh4AAB62zl9+fIljUZzc3NTr9QwFyaTGRoaKhAIcLPepUF2E6EX\ngoODr127piquXr3622+/lUql1tbWR44cqaysdHd3X758OUyJweFwYmNjJRJJfHx8dXX13bt3FQrF\nzJkznz9/DgDIzMyEX4uQrKystLQ0VVEgEHzxxRceHh44jldUVAAA4NGEUqnU09QKCgq8vb3pdDos\n1tXVyWSyO3fuzJgxAxr98ePHZ2VlqRuOq1evVlZWLl68uD/jwu/04ODg/gh59eoVAMDa2lpVQ6VS\naTRafX39u40lEsmFCxeWL19OJpNVlX3OZdKkSS9fvrx//35/9DRykN1EGJSgoCAbGxt7e/uYmJiO\njo7a2lrVIxKJNH78eAqF4uPjk52d3dbWJhKJdBgiIiKitbU1JSVl4LT+Hx0dHc+ePYNrNAg8/7G3\nt+fz+eXl5fX19VFRUWvXrj106BBsIJVKExISsrOzdR60vr4+JycnPj6ezWar1rm6AY/OeyRV/uCD\nD+Bitgfp6emOjo5bt25V1RCZi5eXFwCgrKysP3oaOchuIgYHuIR5N1s6ZMqUKXQ6/dGjR4ZVqm/E\nYjGO46rFJgAA7iH6+voGBQXZ2dkxGIy0tDQGg/Hrr7/CBsnJyStWrHByctJ5UDabHR8fHxUVVVhY\n2CPZp7bAPVkYTl+FTCaj0Wg9Wp44cSI3N/fcuXPqi1Mic4E/znsXsGaDCcdDQpg3FAqloaFhsLXo\nSWdnJ3hrKyGOjo4AALjZCiGTya6urvAAp7i4uKysrJ/hYlksllAoHJA8S3CbuLW1VVUjkUg6Ozvh\nLFTk5ORkZGQUFRWNGjVKVUlwLtAEwx/KXEHrTYQxIpfL37x5Y+D4m0SARkHdu9vKysrLy+vBgwfq\nzRQKBYPBAAAIhcLz589bWFhgGIZhGDxL4fP5GIb9888/BAe1t7e3tbUdEP3d3Nysra1rampUNXBT\n2N/fX1Xz888/Hzhw4MKFC+pGExCei0wmA29/KHMF2U2EMVJUVITjeGBgICySSKTevugNDIvFwjCs\npaVFvTI6OvrevXtVVVWwKJFIampqoFuSSCRS9/uDK2jo8zhlyhSCg546dao/n/nqkEik2bNnX758\nWXVuVlhYiGEY3DbFcTwxMbGsrCwvL8/KyqpHX4JzgT+Og4PDgChsnCC7iTAWlEplc3OzQqEoLS1N\nSEhwcXGJjY2Fjzw9PZuamvLy8uRyeUNDg/pyCQBgZ2dXV1dXXV3d1tYml8sLCwv154dEp9Pd3d1f\nvHihXrlu3TpXV9fY2Nja2trXr18nJiZKpdKNGzcSERgTE+Pg4HD37t3eGlRUVDg4OERHR2vVSwMp\nKSn19fVbtmzp6Oi4fv36zp07Y2Njvb29AQAPHjz48ccf9+3b98EHH2Bq7Nq1i7h8+ONo9mY1dZDd\nROiFPXv2fPTRRwCAxMTEyMjI7Ozs3bt3AwD8/f2rqqr27du3fv16AMBnn3329OlT2KWzs9PPz49G\no4WEhIwdO/bixYuqbcTVq1fPmDFj0aJF3t7eP/zwA/wGZLPZ0FEpLi6OxWL5+PjMnj27qalJ31OL\niIgoLy9XP4BmMplXrlxxdnYOCAhwcnK6detWQUFBnx6dEJlMJhaLVW7n7/JeR0jNvW7cuBEcHDxq\n1KibN2/ev3/f0dFx6tSply9fhk99fX3PnTv3119/DR8+fN68eUuXLt27d6+GsbTl9u3bTk5O6h/+\nZoi+LyTpA3TP0sAY4D7cypUr7ezs9DpEnxB8r54+fUoikf78888BGbS7uzskJEQoFBqglwFobGyk\nUqm7du0i0hjds0Qg+ouphNLx9PTk8Xg8Hq+3QEHE6e7uzsvLa2tri4mJ0Xcvw5CamhoQEMDhcAZb\nEf2C7CYCoTVJSUkLFiyIiYnpcUCkLUVFRcePHy8sLFR3CNVTLwOQkZFRUlJy5syZfjqZGj9DxW4u\nW7bM2toaw7CSkpLB1gUAAI4fP+7u7q6+9U4mk1ks1vTp03fu3Nnc3DzYChqU5ORkkUjU0tLi5uZm\nKpl4+Xw+h8PZtm1bf4SEhYUdPHhQdfVer730TX5+fldXV1FREZPJHGxd9M5QsZu//fbbvn37BluL\n/zFv3ryqqioPDw8Gg4HjuFKpFIvFubm5bm5uiYmJvr6+xJ37zID09PSuri4cx589ezZ//vzBVoco\n4eHh27dvH2wtjIXIyMikpKQeNzjNlaFiN40cDMNsbW2nT58uEolyc3Pr6+sjIiL6+Q2IQCD0xBCy\nm6qAg0bO/PnzY2NjxWLxL7/8Mti6IBCI92DOdhPH8Z07d3p7e1MoFAaDsWHDBvWn3d3dmzdvdnFx\nodFo/v7+0AdFc5hIAMClS5c+/vhjOp1uY2Pj5+cH7/m+VxToRyBI6O9dWFhoMFURCIQWDLIflE4Q\n9LPjcrkYhv3000/Nzc0SiSQrKwsAcO/ePfj0u+++o1Aox44da25uTk5OtrCwuH37NuwFADh//nxL\nS4tYLA4JCbG0tJTJZDiOt7e329jY7NixQyqVvnr1au7cuQ0NDRpEnT592tramsfj9aahan+zB9DG\njR492mCqasZ0/ey0AvkFGxjTfa9M8i0h8n5LJBI6nT5z5kxVDVyLQbsplUrpdHpMTIyqMYVCWb16\nNf7WGKlSx0BrW1FRgeP4v//+CwA4ffq0+kAaRPVJb3YTx3G442kkqpru+60VyG4aGNN9r8w2jlxF\nRYVEIgkLC3vv08ePH0skkgkTJsAijUYbOXLke6M9qoeJdHd3Z7FYS5YsiY+Pj42NHTNmjFaiiNPR\n0YHjOMyIYCSqHjt2zFQ2iPvJEJmmkWBC7hPqmK3dhMEFekvn0tHRAQDYtGmTeibrHiEI34VGo124\ncGHjxo18Pp/H4y1cuFAkEukmSjNPnjwBAIwbN854VA0MDPz222+1n4opcf36dYFAgPZ8DQYMWWCK\nmK3dhHGtYVaAd4H2dPfu3QkJCVqJ9fX1PXXqVENDQ0ZGxvbt2319feFdNx1EaeDs2bMAgFmzZhmP\nqs7OzupJfswVgUAwFKZpJBw9enSwVdARsz1PnzBhgoWFxaVLl977dPTo0VQqVdu7Q3V1dTA8rb29\n/bZt2yZPnvzgwQPdRGng1atXu3fvdnZ2Xrp0qZGrikAMTczWbtrb28+bN+/YsWNCobC1tbW0tFSV\n7wUAQKVSv/nmm8OHD2dnZ7e2tnZ3d7948eK///7TLLOurm7VqlWPHj2SyWT37t2rqakJDAzUIIpI\nIEgcx9vb25VKJY7jDQ0NR44cmTp16rBhw/Ly8uD+pmFURSAQWjDI51I6QfDcs62tbdmyZcOHD7ey\nsgoODt68eTMAwNnZ+f79+ziOd3V1JSYmuri4kEgkaGTLy8uzsrJgrAQvL6/Kyspff/0VGi9XV9cn\nT55UV1cHBQUxmcxhw4aNGjWKy+UqFIreROE4fubMGWtr661bt76r28mTJ/39/el0OplMtrCwAG+v\nDH388cc8Hu/169fqjQ2gqmZM99xTK9B5uoEx3fcKw00wPXxubm50dLQpam6iLFiwAJjybhRB0Htl\nYEz3vTLb73QEAoHQE8huIhB64e+//05KSoL/lsvl6enpnp6eZDLZ1tZ2woQJ1dXV73bp7OwcN26c\nupeYBpRK5Zdffuni4kKlUp2cnCIjI0tLSwnqpkGfrVu3Yv8flcPvyZMnd+zYYSrhpfUKspsIxMCz\nZcuWzMzM5ORkWIyOjt6/f//BgwclEsnDhw89PDzeGyuey+U+fvyY4BBKpfLKlSuHDh1qamoqLi6W\nSqXTpk2rq6sj0pegPj2YM2cOlUoNCwt78+YNQSXNFWQ3EUaBVCoNCgoyNlG6sX379pycnNzcXGtr\nawBATk5OXl7e0aNHP/nkExKJ5OjomJ+fr1rEqbh27Rq8HUscNpsdHBxMp9Pd3Nz4fH5LS8sff/zR\nZ68+9emROkldq/j4+IkTJ86ePVuhUGilqpmB7CbCKBAKhWKx2NhE6UBFRUVKSkpaWhq8eQEA2Lt3\n7+TJkzXnxZVKpRs2bBAIBMQHIpFIp06dUhXd3d0BAJWVlX12JKKPBlJTU0tKSrRS1fxAdhMxYOA4\nnpGRMX78eAqFwmQyo6KiVJffORwOmUxWpXZYs2aNpaUlhmGNjY0AgISEhPXr11dWVmIY5unpmZmZ\nSaVSWSzWqlWrHB0dqVRqUFDQzZs3dRAF+hHNTzcyMzNxHJ8zZw4symSyGzdu9JkTmMvlrlmzprdr\nwUSAeYmhL5oGCOqjASaTGRoaKhAIhrLjAbKbiAEjNTU1KSmJy+WKxeLLly8/f/48JCSkvr4eAJCZ\nmal+fzErKystLU1VFAgEX3zxhYeHB47jFRUVHA4nNjZWIpHEx8dXV1ffvXtXoVDMnDkTZkvXShR4\nmyZTqVTq/wcAAICCggJvb29VxrS6ujqZTHbnzp0ZM2bAvwHjx4/PyspSNzpXr16trKxcvHhxf8a9\ndesWACA4OFhzMyL6JCUlMZlMMpns5uYWFRV1+/btHkImTZr08uXL+/fv90dhkwbZTcTAIJVKMzIy\n5s6du2TJEgaD4efn98svvzQ2Nqpf09IKEokEl64+Pj7Z2dltbW0ikUgHOREREa2trSkpKbqpoRUd\nHR3Pnj3z8PBQ1cDzFnt7ez6fX15eXl9fHxUVtXbt2kOHDsEGUqk0ISEhOztb50Hr6+tzcnLi4+PZ\nbLZqndsbferz9ddfnzx58vnz5+3t7YcPH66trQ0NDS0vL1cX4uXlBQAoKyvTWWdTB9lNxMBQXl7e\n3t4+ZcoUVc1HH31EJpNV39f9YcqUKXQ6vZ/R+QyAWCzGcVw9PS+FQgEA+Pr6BgUF2dnZMRiMtLQ0\nBoOh+nOSnJy8YsUKJycnnQdls9nx8fFRUVGFhYV9JuDtU5/Ro0dPmjTJysqKTCYHBgaKRCKpVApD\nu6qAE4RfEkMTs42HhDAw0DfFyspKvdLW1ratrW1A5FMolIaGhgERpT86OzvBW9sEgWH64N4rhEwm\nu7q6wgOc4uLisrKyjIyM/gzKYrGEQqGvry+Rxpr1eRc/P79hw4bBwIYqaDQaeDvZoQlabyIGBltb\nWwBADyv55s0bZ2fn/guXy+UDJUqvQIOi7hluZWXl5eUFY1N8YA+sAAADvklEQVSpUCgUDAYDACAU\nCs+fP29hYQE9zOG5EJ/PxzCMeCJoe3t7+OMTQbM+76JUKpVKpfpfAgCATCYDbyc7NEF2EzEwTJgw\nwcrKSv1/+82bN2Uy2YcffgiLJBIJxqLXgaKiIhzHAwMD+y9Kr7BYLAzDeiRwjo6OvnfvXlVVFSxK\nJJKamhroBiQSidQ9JeGCmsvl4jiuvuOhmVOnTmn1ma9BHwDAp59+qt4Ypp9is9nqlXCCDg4OxAc1\nM5DdRAwMVCp1/fr1J06cOHDgQGtra1lZWVxcnKOj48qVK2EDT0/PpqamvLw8uVze0NBQU1Oj3t3O\nzq6urq66urqtrQ3aRKVS2dzcrFAoSktLExISXFxcYJpPbUURieY3UNDpdHd3d5hrQMW6detcXV1j\nY2Nra2tfv36dmJgolUo3btxIRGBMTIyDg8Pdu3d7a1BRUeHg4BAdHU28l2Z9Xr58mZOT8+bNG7lc\nfv369WXLlrm4uMTFxalLgBPU2QPUDEB2EzFgbNmyJT09ncfjjRgxIjQ0dMyYMUVFRZaWlvDp6tWr\nZ8yYsWjRIm9v7x9++AF+5bHZbOhdFBcXx2KxfHx8Zs+e3dTUBADo7Oz08/Oj0WghISFjx469ePGi\n6mtRW1GGJCIiory8HHpTQphM5pUrV5ydnQMCApycnG7dulVQUEDQg1Imk4nF4vz8/N4avNeJUnMv\nzfp89tlnmzZtcnZ2ptPpCxcunDp16o0bN4YPH64u4fbt205OTv7+/kSmYJ7oN0ydfkBxEg2M4eMk\nrly50s7OzpAj4gP0Xj19+pREIvW4qqgz3d3dISEhQqHQAL0I0tjYSKVSd+3a1X9Rpht/E603EUaK\nicbd8fT05PF4PB6PSKQMzXR3d+fl5bW1tcHEUHrtRZzU1NSAgAAOh6MP4aYCspsIxACTlJS0YMGC\nmJiYHgdE2lJUVHT8+PHCwkJ1h1A99SJIRkZGSUnJmTNn+nQUNW+Q3UQYHcnJySKRqKWlxc3N7dix\nY4Otji7w+XwOh7Nt27b+CAkLCzt48KDqJr5eexEhPz+/q6urqKiIyWQOuHDTAvm9I4yO9PT09PT0\nwdaiv4SHh4eHhw+2FgNJZGRkZGTkYGthFKD1JgKBQGgHspsIBAKhHchuIhAIhHYgu4lAIBDaYcLn\nQrm5uYOtwlAB3qsz+x/8+vXrYAhM03h48eKF8cdqeT+D7XivC/BeBwKBMHVM9L4Qhg/hJCEIBAKh\nA2h/E4FAILQD2U0EAoHQDmQ3EQgEQjuQ3UQgEAjt+D+k2lIo1gYIxwAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"execution_count": 28,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"tf.keras.utils.plot_model(\n",
" model,\n",
" show_shapes=True,\n",
" show_layer_names=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "JpL9idwZV6fL"
},
"source": [
"For each character the model looks up the embedding, runs the GRU one timestep with the embedding as input, and applies the dense layer to generate logits predicting the log-likelihood of the next character:\n",
"\n",
"\n",
"\n",
"Image source: [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation) notebook."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Npruiy2RAPkt"
},
"source": [
"## Try the model"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 11144,
"status": "ok",
"timestamp": 1584519814917,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "E4DCLA0GASL1",
"outputId": "eba67358-d6b9-4b42-a2f1-dc6ed3e0708c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(64, 100, 65) # (batch_size, sequence_length, vocab_size)\n"
]
}
],
"source": [
"for input_example_batch, target_example_batch in dataset.take(1):\n",
" example_batch_predictions = model(input_example_batch)\n",
" print(example_batch_predictions.shape, \"# (batch_size, sequence_length, vocab_size)\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "MWebJXU9CEPd"
},
"source": [
"To get actual predictions from the model we need to sample from the output distribution, to get actual character indices. This distribution is defined by the logits over the character vocabulary."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 340
},
"colab_type": "code",
"executionInfo": {
"elapsed": 11020,
"status": "ok",
"timestamp": 1584519814919,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Y4Jgo-iECFWI",
"outputId": "a59cb406-3813-4fb0-8297-6862a73f2499"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Prediction for the 1st letter of the batch 1st sequense:\n",
"tf.Tensor(\n",
"[-5.5658985e-03 -5.6167855e-03 2.3333444e-03 -5.4010577e-03\n",
" -1.2658490e-03 -2.0685978e-03 -1.7119508e-03 -1.9059415e-03\n",
" -1.8099162e-03 1.8549486e-03 -6.1538210e-03 2.1649192e-03\n",
" 5.9574037e-03 3.1515316e-03 -3.3867492e-03 6.0820174e-03\n",
" 2.0375988e-03 1.1445739e-03 1.3204901e-03 -2.5559508e-03\n",
" -6.7244383e-04 2.0627938e-03 -5.1490301e-03 5.0887186e-03\n",
" -2.1955089e-03 -5.3793588e-03 3.5228319e-03 -1.3459967e-03\n",
" 1.5698730e-03 -3.6733383e-03 -2.6844436e-04 1.2893401e-03\n",
" -2.9499468e-03 -2.0524827e-03 4.9254433e-03 -1.4868007e-03\n",
" -3.2468680e-03 -3.1228375e-03 1.4175450e-03 2.6028175e-03\n",
" -1.9702244e-04 -1.5901095e-04 -5.2650859e-05 -2.0922027e-03\n",
" -1.1370353e-03 -1.3396480e-03 2.7639134e-04 2.6140050e-03\n",
" 3.0784088e-04 -7.5585191e-04 2.4966525e-03 1.0233605e-03\n",
" 1.4258339e-03 -1.6010429e-04 5.1720534e-03 9.4948718e-03\n",
" -4.6309843e-03 5.5265515e-03 -1.2462666e-04 -1.3864412e-03\n",
" -2.6601439e-03 2.0305226e-03 1.0555825e-03 1.6011465e-03\n",
" 2.9543908e-03], shape=(65,), dtype=float32)\n"
]
}
],
"source": [
"print('Prediction for the 1st letter of the batch 1st sequense:')\n",
"print(example_batch_predictions[0, 0])"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 10898,
"status": "ok",
"timestamp": 1584519814922,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "0dOr0MwFHlRb",
"outputId": "f8dc2264-a19a-4887-f964-9086a95b14a4"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tf.Tensor([[2 2 2 1 2]], shape=(1, 5), dtype=int64)\n"
]
}
],
"source": [
"# Quick overview of how tf.random.categorical() works.\n",
"\n",
"# logits is 2-D Tensor with shape [batch_size, num_classes].\n",
"# Each slice [i, :] represents the unnormalized log-probabilities for all classes.\n",
"# In the example below we say that the probability for class \"0\" is low but the\n",
"# probability for class \"2\" is much higher.\n",
"tmp_logits = [\n",
" [-0.95, 0, 0.95],\n",
"];\n",
"\n",
"# Let's generate 5 samples. Each sample is a class index. Class probabilities \n",
"# are being taken into account (we expect to see more samples of class \"2\").\n",
"tmp_samples = tf.random.categorical(\n",
" logits=tmp_logits,\n",
" num_samples=5\n",
")\n",
"\n",
"print(tmp_samples)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 10502,
"status": "ok",
"timestamp": 1584519814922,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "JPzr0r4zCgS3",
"outputId": "23e7516b-1753-402a-bc61-8a8988e49f18"
},
"outputs": [
{
"data": {
"text/plain": [
"TensorShape([100, 1])"
]
},
"execution_count": 32,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"sampled_indices = tf.random.categorical(\n",
" logits=example_batch_predictions[0],\n",
" num_samples=1\n",
")\n",
"\n",
"sampled_indices.shape"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 10276,
"status": "ok",
"timestamp": 1584519814923,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "YaA7DclID8dz",
"outputId": "08a3c58b-e331-4f91-8e3f-be2649a85f48"
},
"outputs": [
{
"data": {
"text/plain": [
"(100,)"
]
},
"execution_count": 33,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"sampled_indices = tf.squeeze(\n",
" input=sampled_indices,\n",
" axis=-1\n",
").numpy()\n",
"\n",
"sampled_indices.shape"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 119
},
"colab_type": "code",
"executionInfo": {
"elapsed": 10212,
"status": "ok",
"timestamp": 1584519814924,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "_ubGQ0gVENhB",
"outputId": "4ab910cf-06e6-4fd1-c751-746d79d895d0"
},
"outputs": [
{
"data": {
"text/plain": [
"array([62, 61, 40, 20, 4, 44, 17, 43, 28, 18, 50, 48, 0, 11, 58, 17, 41,\n",
" 22, 62, 46, 12, 24, 25, 54, 10, 6, 18, 5, 4, 23, 55, 51, 38, 9,\n",
" 50, 42, 42, 9, 49, 4, 48, 54, 59, 4, 30, 7, 20, 4, 53, 27, 21,\n",
" 5, 16, 27, 25, 47, 21, 7, 24, 25, 46, 21, 6, 56, 1, 21, 24, 28,\n",
" 55, 8, 23, 0, 45, 29, 4, 29, 5, 49, 25, 45, 3, 4, 22, 11, 13,\n",
" 30, 43, 0, 62, 0, 48, 29, 33, 58, 46, 51, 12, 59, 55, 62])"
]
},
"execution_count": 34,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"sampled_indices"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 122
},
"colab_type": "code",
"executionInfo": {
"elapsed": 9992,
"status": "ok",
"timestamp": 1584519814925,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Gi9HOzw9EajS",
"outputId": "aa7dc69f-63ad-4273-b2df-e6cc8eada7cd"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input:\n",
" 'is service.\\n\\nBAPTISTA:\\nA thousand thanks, Signior Gremio.\\nWelcome, good Cambio.\\nBut, gentle sir, met'\n",
"\n",
"Next char prediction:\n",
" \"xwbH&fEePFlj\\n;tEcJxh?LMp:,F'&KqmZ3ldd3k&jpu&R-H&oOI'DOMiI-LMhI,r ILPq.K\\ngQ&Q'kMg$&J;ARe\\nx\\njQUthm?uqx\"\n"
]
}
],
"source": [
"print('Input:\\n', repr(''.join(index2char[input_example_batch[0]])))\n",
"print()\n",
"print('Next char prediction:\\n', repr(''.join(index2char[sampled_indices])))"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 272
},
"colab_type": "code",
"executionInfo": {
"elapsed": 9825,
"status": "ok",
"timestamp": 1584519814926,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "b87e0lsYMTsv",
"outputId": "75c7e1bf-b2d5-4f15-cba7-0ae5f8765327"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Prediction 0\n",
" input: 47 ('i')\n",
" next predicted: 1 ('x')\n",
"Prediction 1\n",
" input: 57 ('s')\n",
" next predicted: 1 ('w')\n",
"Prediction 2\n",
" input: 1 (' ')\n",
" next predicted: 1 ('b')\n",
"Prediction 3\n",
" input: 57 ('s')\n",
" next predicted: 1 ('H')\n",
"Prediction 4\n",
" input: 43 ('e')\n",
" next predicted: 1 ('&')\n"
]
}
],
"source": [
"for i, (input_idx, sample_idx) in enumerate(zip(input_example_batch[0][:5], sampled_indices[:5])):\n",
" print('Prediction {:2d}'.format(i))\n",
" print(' input: {} ({:s})'.format(input_idx, repr(index2char[input_idx])))\n",
" print(' next predicted: {} ({:s})'.format(target_idx, repr(index2char[sample_idx])))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "LqcBufKEE_p6"
},
"source": [
"## Train the model\n",
"\n",
"At this point the problem can be treated as a standard classification problem. Given the previous RNN state, and the input this time step, predict the class of the next character."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "l4s0-PvrFub5"
},
"source": [
"### Attach an optimizer, and a loss function"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"colab_type": "code",
"executionInfo": {
"elapsed": 9644,
"status": "ok",
"timestamp": 1584519814927,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "UOEUUm6JE95a",
"outputId": "ba94797c-e3c6-4d71-ac44-2cf76f7b2843"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Prediction shape: (64, 100, 65) # (batch_size, sequence_length, vocab_size)\n",
"scalar_loss: 4.17424\n"
]
}
],
"source": [
"# An objective function.\n",
"# The function is any callable with the signature scalar_loss = fn(y_true, y_pred).\n",
"def loss(labels, logits):\n",
" return tf.keras.losses.sparse_categorical_crossentropy(\n",
" y_true=labels,\n",
" y_pred=logits,\n",
" from_logits=True\n",
" )\n",
"\n",
"example_batch_loss = loss(target_example_batch, example_batch_predictions)\n",
"\n",
"print(\"Prediction shape: \", example_batch_predictions.shape, \" # (batch_size, sequence_length, vocab_size)\")\n",
"print(\"scalar_loss: \", example_batch_loss.numpy().mean())"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "SXhJsB6eFgrJ"
},
"outputs": [],
"source": [
"adam_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
"model.compile(\n",
" optimizer=adam_optimizer,\n",
" loss=loss\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "MK3Cf-xZFwL4"
},
"source": [
"### Configure checkpoints"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "LUhXnHPJFy5q"
},
"outputs": [],
"source": [
"# Directory where the checkpoints will be saved.\n",
"checkpoint_dir = 'tmp/checkpoints'\n",
"os.makedirs(checkpoint_dir, exist_ok=True)\n",
"\n",
"# Name of the checkpoint files\n",
"checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt_{epoch}')\n",
"\n",
"checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(\n",
" filepath=checkpoint_prefix,\n",
" save_weights_only=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "oFg9MFJoGZWf"
},
"source": [
"### Execute the training"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "AVk-pARPGaja"
},
"outputs": [],
"source": [
"EPOCHS=40"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1197604,
"status": "ok",
"timestamp": 1584521003714,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "y0rveBdAGeEz",
"outputId": "2449548c-5ca4-4340-df15-9cdf1714dcbc"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train for 172 steps\n",
"Epoch 1/40\n",
"172/172 [==============================] - 31s 179ms/step - loss: 2.5607\n",
"Epoch 2/40\n",
"172/172 [==============================] - 29s 171ms/step - loss: 1.8654\n",
"Epoch 3/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 1.6191\n",
"Epoch 4/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 1.4893\n",
"Epoch 5/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 1.4108\n",
"Epoch 6/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 1.3550\n",
"Epoch 7/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 1.3101\n",
"Epoch 8/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 1.2705\n",
"Epoch 9/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 1.2335\n",
"Epoch 10/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 1.1980\n",
"Epoch 11/40\n",
"172/172 [==============================] - 29s 171ms/step - loss: 1.1610\n",
"Epoch 12/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 1.1230\n",
"Epoch 13/40\n",
"172/172 [==============================] - 30s 174ms/step - loss: 1.0843\n",
"Epoch 14/40\n",
"172/172 [==============================] - 30s 174ms/step - loss: 1.0446\n",
"Epoch 15/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 1.0031\n",
"Epoch 16/40\n",
"172/172 [==============================] - 30s 174ms/step - loss: 0.9617\n",
"Epoch 17/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 0.9211\n",
"Epoch 18/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 0.8810\n",
"Epoch 19/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 0.8413\n",
"Epoch 20/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.8018\n",
"Epoch 21/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.7692\n",
"Epoch 22/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.7346\n",
"Epoch 23/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 0.7056\n",
"Epoch 24/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.6777\n",
"Epoch 25/40\n",
"172/172 [==============================] - 30s 174ms/step - loss: 0.6528\n",
"Epoch 26/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.6282\n",
"Epoch 27/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 0.6077\n",
"Epoch 28/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.5897\n",
"Epoch 29/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 0.5735\n",
"Epoch 30/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.5566\n",
"Epoch 31/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.5447\n",
"Epoch 32/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.5318\n",
"Epoch 33/40\n",
"172/172 [==============================] - 30s 174ms/step - loss: 0.5198\n",
"Epoch 34/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.5103\n",
"Epoch 35/40\n",
"172/172 [==============================] - 30s 172ms/step - loss: 0.4999\n",
"Epoch 36/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.4930\n",
"Epoch 37/40\n",
"172/172 [==============================] - 30s 174ms/step - loss: 0.4858\n",
"Epoch 38/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.4782\n",
"Epoch 39/40\n",
"172/172 [==============================] - 30s 173ms/step - loss: 0.4719\n",
"Epoch 40/40\n",
"172/172 [==============================] - 29s 170ms/step - loss: 0.4672\n"
]
}
],
"source": [
"history = model.fit(\n",
" x=dataset,\n",
" epochs=EPOCHS,\n",
" callbacks=[\n",
" checkpoint_callback\n",
" ]\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "mLdnOyvzMggJ"
},
"outputs": [],
"source": [
"def render_training_history(training_history):\n",
" loss = training_history.history['loss']\n",
" plt.title('Loss')\n",
" plt.xlabel('Epoch')\n",
" plt.ylabel('Loss')\n",
" plt.plot(loss, label='Training set')\n",
" plt.legend()\n",
" plt.grid(linestyle='--', linewidth=1, alpha=0.5)\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 295
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1197218,
"status": "ok",
"timestamp": 1584521003724,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "4Ghveem_OQBV",
"outputId": "f4fcc0aa-4509-4330-a88b-1f5863c354d2"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deXxU5b348c+TzCQTMiFDMkA2SCAs\nYQ+CIK6AWlvcamvrdq1t7RWt1i62bl20/dVq19ta7bVaW7XutvVq1VoVUBFUCgqyRQkwQAIBBpyQ\nQCbr8/tjJjFAEjInmTlP5nzfr1dezJxZ8jlnSJ7MOWfOUVprhBBCOFeK3QFCCCHsJQOBEEI4nAwE\nQgjhcDIQCCGEw8lAIIQQDicDgRBCOJwMBEII4XAyEAjRA6VUQCl1ht0dQsSTDARCCOFwMhAIESOl\nVLpS6rdKqZ3Rr98qpdKjt/mVUi8opUJKqf1KqaVKqZTobTcppaqVUnVKqQ+VUqfbOydCRLjsDhBi\nAPo+cAJQDmjgOeAHwA+BG4AqYGj0vicAWik1HrgOOF5rvVMpVQKkJjZbiK7JOwIhYncZ8BOt9R6t\n9V7gx8Dl0duagXygWGvdrLVeqiMH9GoF0oGJSim31jqgtd5sS70QR5CBQIjYFQDbOl3fFp0G8Eug\nEnhFKbVFKXUzgNa6EvgWcDuwRyn1pFKqACEMIAOBELHbCRR3uj4yOg2tdZ3W+gat9WjgPOA77dsC\ntNaPa61Pjj5WAz9PbLYQXZOBQIhjcyulPO1fwBPAD5RSQ5VSfuBHwKMASqlzlFJjlFIKqCWySqhN\nKTVeKTU/ulE5DDQAbfbMjhCHk4FAiGN7icgv7vYvD7AS+ABYC7wH/DR637HAa0A98DbwB631EiLb\nB+4CgkANMAy4JXGzIET3lJyYRgghnE3eEQghhMPJQCCEEA4nA4EQQjicDARCCOFwA+4QE36/X5eU\nlFh6bEtLCy6XmbMsbdaY3AZm90mbNQO1bdWqVUGt9dCubjNzbnpQUlLCypUrLT22oaGBjIyMfi7q\nH9JmjcltYHaftFkzUNuUUtu6vAFZNSSEEI7nqIFg27ZuB0TbSZs1JreB2X3SZk0ytjlqIBBCCHG0\nAbeNQAgxcDQ3N1NVVUU4HI75cRs3boxTVd+Y3rZ161aKiopwu929fpyjBgK/3293QrekzRqT28Ds\nvkS0VVVVkZWVRUlJCZHj8PVOc3NzTL/IEsnktqamJg4cOEBVVRWjRo3q9eMctWrI6T+UVkmbdSb3\nJaItHA6Tm5sb0yAAGPuLFsxuS0tLIzc3N+Z3YI4aCCorK+1O6Ja0WWNyG5jdl6i2WAcBIOZfZIlk\nepuV5e2YgaCi5gAPvLub2kPNdqd0qaWlxe6EbkmbdSb3mdxm8lGRk7HNMQPB9n2HeGptiG37D9qd\nIoRIkH379lFeXk55eTl5eXkUFhZ2XG9qaurxsStXruT6668/5vc48cQT+ys3Jj/72c/67bkcs7G4\ncEjk03Y7Qw1MLfLZXHM0j8djd0K3pM06k/tMbktJ6Z+/UXNzc1m9ejUAt99+O16vl+9+97sdt/d0\nSIaZM2cyc+bMY7YtX768X1pj9bOf/Yxbb731sGlWl5tj3hEU+iIDQdXHDTaXdM3q8ZMSQdqsM7nP\n5Lb09PS4PfeXv/xlrr76ambPns2NN97IihUrmDNnDtOnT+fEE0/kww8/BOD111/nnHPOASKDyFe/\n+lXmzp3LhAkTuPvuuzuez+v1dtx/7ty5XHjhhZSVlXHZZZd1rKp56aWXKCsrY8aMGVx//fUdz9vZ\n+vXrmTVrFuXl5UydOpVNmzYB8Oijj3ZMX7hwIa2trdx88800NDRQXl7OZZdd1vEcVpdb3N4RKKVG\nAI8Aw4mcqPt+rfXvjrjPXOA5YGt00j+01j+JR092hpsMdwo7Q2Zu6KmpqSEvL8/ujC5Jm3Um9yW6\n7cf/XM+GnQd6dV+tda82ek4sGMxt506KuaWqqorly5eTmprKgQMHWLp0KS6Xi9dee41bb72Vv//9\n70c9pqKigiVLlrB//34mT57MNddcc9QeRO+//z7r16+noKCAk046iWXLljFz5kwWLlzIm2++yahR\no7jkkku6bLrvvvv45je/yWWXXUZTUxOtra1s3LiRp556imXLluF2u/n617/OY489xl133cU999zT\n8W6nndVdW+O5aqgFuEFr/Z5SKgtYpZR6VWu94Yj7LdVaHz089jOlFEMzU6kOHYr3t7IkFAoZ+wtD\n2qwzuc/ktt4OBFZ94QtfIDU1FYDa2lquuOIKNm3ahFKK5uaudyg5++yzSU9Px+fzMWzYMHbv3k1R\nUdFh95k1a1bHtPLycgKBAF6vl9GjR3fs13/JJZdw//33H/X8c+bM4Y477qCqqorPfe5zjB07lkWL\nFrFq1SqOP/54IHJQuWHDhnU7Xy0tLWYNBFrrXcCu6OU6pdRGoBA4ciBImGGZLmPfEQiR7GL5yz3e\nR/jMzMzsuPzDH/6QefPm8eyzzxIIBJg7d26Xj+m82iU1NbXLva56c5/uXHrppcyePZsXX3yRBQsW\n8Mc//hGtNVdccQV33nlnr5/HioRsLFZKlQDTgXe7uHmOUmoNsBP4rtZ6fRePvwq4CqCoqIiKioqO\n24qLi4HDD7bk9/vx+/1UVlZ2vBAej4dhmW7e3nHwsMeXlpYSDoeprq7umJaXl4fP5zvsfl6vl6Ki\nIqqqqqivr++YXlZWRigUoqampmNaYWEhHo+HzZs3d0zz+Xzk5eURCAQ69kN2uVyMGTOGYDBIMBjs\n+H6xzFNJSQk1NTWEQqG4zVMwGKSurs7SPFl5nWKZJyDhr1Ms8xQMBgkEAgl5nWKdJyDur1Nzc3PH\nBtmGhk+2z6WmppKWltaxCqRdRkYGLS0ttLS0dNw/LS0NpRSNjY0d93O5XLjdbhobG2lrawMi7/o9\nHk/H92zX/su5ubmZ5uZmWltbO24Ph8Ps37+foUOH0tjYyEMPPQREBqLGxkZaW1tpa2ujra2N5uZm\nGhoaDnvu9saGhoaOdxLt89TS0kJTUxPjx49ny5YtVFRUUFxczOOPP47Wmra2tsPmafv27YwbN46F\nCxeyZcsWVq1axRlnnMEXv/hFrrvuOnJycti/fz/19fWMHTsWt9vNgQMHOt4BtL+m4XCY5uZmKioq\nDnudeqS1jusX4AVWAZ/r4rbBgDd6eQGw6VjPN2PGDG3V716t0MU3vaAPNbZYfo54aWpqsjuhW9Jm\nncl9iWjbsGGDpce1trb2c4nWt912m/7lL3+pr7jiCv3MM890TF++fLkeO3asLi8v19///vd1cXGx\n1lrrJUuW6LPPPvuwx7a3TZo0SW/dulVrrXVmZuZR99da62uvvVb/5S9/0Vpr/fzzz+vx48fr4447\nTi9cuFBfeumlR/XdeeedeuLEiXratGn6rLPO0vv27dNaa/3kk0/qadOm6SlTpujjjjtOv/3221pr\nrW+88UZdVlZ22HO1L7euljuwUnfze1XpOH44QinlBl4A/q21/k0v7h8AZmqtg93dZ+bMmdrqiWke\nX17Jrc9/yKIbTqN0qNfSc8RLXV0dWVlZdmd0SdqsM7kvEW0bN25kwoQJMT+utbW1Yx2+aay01dfX\n4/V60Vpz7bXXMnbsWL797W/Hra2r5a6UWqW1Pnp/WOK4+6iKbOl5ENjY3SCglMqL3g+l1Kxoz754\nNaU11wFQbeAupJ1XD5hG2qwzuc/ktmN92MtOVtoeeOABysvLmTRpErW1tSxcuDAOZdaXWzy3EZwE\nXA6sVUq17+N0KzASQGt9H3AhcI1SqgVoAC7WcXyLMjQzMrs7Q+YNBEKI5PXtb387Lu8A+ks89xp6\nC+hx/y+t9T3APfFqOJJ/kIsUBdUyEAiRMDrOu4KKw1n5W9oxnywGKCzIJ2+wx8iBwNT9uUHa+sLk\nvkS0eTwe9u3bF/MvJ5MP9Wxym8vlYt++fTEfPsQxxxqCyG50hUMyjFw15POZd/yjdtJmncl9iWhr\n3+117969cf9eIsLj8Rz1QbdjcdRAUFFRQYEvg/e2f2x3ylEqKiooKyuzO6NL0madyX2JaHO73TGd\nKaud05ebVRUVFZaWt6NWDQEU+DKoqQ3T2mbuMcWFECKRHDcQFPoyaG7V7K1rPPadhRDCARw1EHi9\n3o7DUZu2wbj9ULYmkjbrTO6TNmuSsc1RA0FRUVHHCWpMGwhi3biTSNJmncl90mZNMrY5aiCoqqqi\nwPfJmcpMUlVVZXdCt6TNOpP7pM2aZGxz1EBQX1+PN91FdobbuIGg81ElTSNt1pncJ23WJGObowaC\ndgW+DCOPNySEEHZw5EBQ6DPz08VCCGGHuB6GOh76chjqdrc9t45/vF/N2tvP6qcqIYQwmy2HoTZR\n+5mUCnwZ1IVbOBDu+tykduh8lifTSJt1JvdJmzXJ2OaogaD9dG3tu5CatMH4mKeSs5G0WWdyn7RZ\nk4xtjhoI2pm6C6kQQtjBkQNBUceni8M2lwghhP0cNRAUFhYC4Pem405VRu1C2t5mImmzzuQ+abMm\nGdscNRC0n6whJUWRn23WeQliPZFEIkmbdSb3SZs1ydjmqIFg8+bNHZcLfRlGfZagc5tppM06k/uk\nzZpkbHPUQNBZgc+sdwRCCGEXxw4EhUMy2H0gTHNrm90pQghhK0cNBJ3P0Vro89CmoabWjD2HnH5u\nW6tMbgOz+6TNmmRsc9RAkJeX13G50DcIMOezBJ3bTCNt1pncJ23WJGObowaCQCDQcbnAF9m6bsoG\n485tppE260zukzZrkrHNUQNBOPzJaiDTPl3cuc000madyX3SZk0ytjlqIOjM407F700z5h2BEELY\nxVEDgcvlOux6gS/DmMNMHNlmEmmzzuQ+abMmGdsceT6Cdtc8uoqPdtex6Ia5/fJ8QghhKjkfQVQw\nGDzseuRDZWFMGAyPbDOJtFlncp+0WZOMbY4fCBqaW/n4kP0nqEnG/1yJYHIbmN0nbdYkY5ujBoIj\nFRq255AQQthBBgLM+SyBEELYwVEDQXFx8WHX209ZacJ5CY5sM4m0WWdyn7RZk4xtjhoIjjRkkBuP\nO0VWDQkhHM1RA8G2bdsOu66UMua8BEe2mUTarDO5T9qsScY2Rw0EXZHzEgghnM7xA4Ep7wiEEMIu\njhoI/H7/UdMKfRkE65sIN7faUPSJrtpMIW3WmdwnbdYkY1vcBgKl1Ail1BKl1Aal1Hql1De7uI9S\nSt2tlKpUSn2glDouXj3Q9UJqPwrpLptPUJOM/7kSweQ2MLtP2qxJxrZ4viNoAW7QWk8ETgCuVUpN\nPOI+nwHGRr+uAv43jj1UVlYeNc2UXUi7ajOFtFlncp+0WZOMbXEbCLTWu7TW70Uv1wEbgcIj7nY+\n8IiOeAfwKaXy49XU0tJy1DRTPl3cVZsppM06k/ukzZpkbEvI8VSVUiXAdODdI24qBHZ0ul4Vnbbr\niMdfReQdA0VFRVRUVHTc1v4Bis67Tfn9fvx+P5WVlR0LxuOJnJGspqaGUCjUcd+RJaNQwJrKHUzN\nOghETvfm8/kO+z5er5eioiKqqqqor6/vmF5WVkYoFKKmpuaTmSosxOPxsHnz5o5pPp+PvLw8AoFA\nx8kjXC4XY8aMIRgMEgwGO75fLPNUUlJy1DyVlpYSDoeprq7umNaXeQoGg9TV1VmaJyuvUyzzBCT8\ndYplnoLBIIFAICGvU6zzBCTsdYp1njr/PCTidYplnoLBIKFQyJbfEceaJ6DbeeqR1jquX4AXWAV8\nrovbXgBO7nR9ETCzp+ebMWOGtmrr1q1dTp99x2v6hqdXW37e/tBdmwmkzTqT+6TNmoHaBqzU3fxe\njeteQ0opN/B34DGt9T+6uEs1MKLT9aLotLgoKSnpcnqBz2P7NoLu2kwgbdaZ3Cdt1iRjWzz3GlLA\ng8BGrfVvurnb88CXonsPnQDUaq13dXPfPuvu7VHhkEHsrLV3IDjmWzcbSZt1JvdJmzXJ2BbPdwQn\nAZcD85VSq6NfC5RSVyulro7e5yVgC1AJPAB8PY49h63366zA52FXKExbm30nqOmuzQTSZp3JfdJm\nTTK2xW1jsdb6LUAd4z4auDZeDb1V5MugqbWNYH0jwwZ77M4RQoiEctQni7tTIOclEEI4mKMGgtLS\n0i6nd3yozMaBoLs2E0ibdSb3SZs1ydjmqIGgfd/cIxUY8KGy7tpMIG3WmdwnbdYkY5ujBoLOH3Lp\nbLDHTVa6y9ZdSLtrM4G0WWdyn7RZk4xtjhoIelI4JIPqkLkjvRBCxIsMBFFyghohhFM5aiBoPzZN\nVwp9GVR9fKj9UBcJ11Ob3aTNOpP7pM2aZGxz1EDg8/m6va18hI8D4RZW77DnwyI9tdlN2qwzuU/a\nrEnGNkcNBJ2PFHikMyYOJy01hRc/iNsRLnrUU5vdpM06k/ukzZpkbHPUQNCT7Aw3p47z89LaXbYe\nakIIIRJNBoJOzp6az87aMO/btHpICCHs4KiBwOv19nj7GROGk+ayZ/XQsdrsJG3WmdwnbdYkY5uy\nay8Zq2bOnKlXrlwZt+f/70dWsraqluU3zyclpcdj5gkhxIChlFqltZ7Z1W2OekdQVVV1zPucMzWf\nmgNh3t/xcQKKPtGbNrtIm3Um90mbNcnY5qiBoPN5RLtzenT10AsJXj3Umza7SJt1JvdJmzXJ2Oao\ngaA3vOku5o4bKnsPCSEcQwaCLpw9NZ/dBxpZtT2xq4eEEMIOsrG4C/WNLcz4f69yyayR3H7epLh+\nLyGESATZWBzV2/N5etNdzBs/jJfW7qI1QauHkvE8qIlgchuY3Sdt1iRjm6MGgpqaml7f9+yp+eyp\na2RlYH8ciz4RS1uiSZt1JvdJmzXJ2OaogSAW88uG4XGn8OJae449JIQQiSIDQTcy013MLxvGS2tr\nErZ6SAgh7OCogaCwsDCm+589pYBgfSMrtsZ/9VCsbYkkbdaZ3Cdt1iRjm6MGAo/HE9P955UNJcOd\nyotrd8ap6BOxtiWStFlncp+0WZOMbY4aCDZv3hzT/QeluZg/YRgvr4v/6qFY2xJJ2qwzuU/arEnG\nNkcNBFacMyWfYH0T727dZ3eKEELEhQwExzB3/DAGpaXaduYyIYSIN0cNBFbO55mRlsr8ssjqoZbW\ntjhURSTjeVATweQ2MLtP2qxJxjZHDQR5eXmWHnfO1Hz2HWzi3TjuPWS1LRGkzTqT+6TNmmRsc9RA\nEAgELD2uffXQP9fEb+8hq22JIG3WmdwnbdYkY5ujBoJwOGzpcR53KudNK+Dv71Xx0e66fq6KsNqW\nCNJmncl90mZNMrY5aiDoi++dNR5vuotb/rFWzlMghEgqvRoIlFKZSqmU6OVxSqnzlFLu+Kb1P5fL\nZfmxud50fnD2RFZt+5jH3t3Wj1URfWmLN2mzzuQ+abMmGdt6dT4CpdQq4BRgCLAM+A/QpLW+zNJ3\n7YNEnI+gO1prvvTnFby/PcSr3zmV/OwMWzqEECJW/XE+AqW1PgR8DviD1voLwIA7Y0swGOzT45VS\n3PHZKbS0tXHbc+v7qSqir23xJG3WmdwnbdYkY1uvBwKl1BzgMuDF6LRUS9/RRv3xAo7MHcS3zhjH\nKxt28/K6/vuQWTL+50oEk9vA7D5psyYZ23o7EHwLuAV4Vmu9Xik1Glhi6Tsmga+dPIqJ+YP50XPr\nqW1otjtHCCH6pFcDgdb6Da31eVrrn0c3Gge11tfHuc1YrtQUfv75qQTrG/n5yxV25wghRJ/0dq+h\nx5VSg5VSmcA6YINS6nvHeMyflVJ7lFLrurl9rlKqVim1Ovr1o9jzY1NcXNxvzzWlKJuvnjSKx9/d\n3i/nK+jPtv4mbdaZ3Cdt1iRjW29XDU3UWh8APgv8CxgFXH6MxzwEfPoY91mqtS6Pfv2kly3G+M6n\nxlE0JINb/vEBjS2tducIIYQlvR0I3NHPDXwWeF5r3Qz0uN+p1vpNIDFnfu+lbdv6d///QWku7rhg\nCpv3HuTeJX07Rnl/t/UnabPO5D5psyYZ23r76YM/AgFgDfCmUqoYOGDpOx5ujlJqDbAT+K7Wust9\nMpVSVwFXARQVFVFR8cl6+fa3Qp0XgN/vx+/3U1lZSUtLC/DJmXtqamoIhUId9y0tLSUcDlNdXd0x\nLS8vD5/Pd9j38Xq9FBUVUVVVRX19fcf008rKWDBpKH9YsomJ3jAlQ9IoLCzE4/EcdpIIn89HXl4e\ngUCg42PgLpeLMWPGEAwGCQaDHd8vlnkqKSnp93kqKysjFApRU1MDRPZEqKurszRPVl6nWOYJsDRP\ngOXXKZZ5CgaDBAKBhLxOsc4TkLDXKdZ56vzzkIjXKZZ5CgaDhEKhhL1OscwT0O089UhrbekLcPXi\nPiXAum5uGwx4o5cXAJt6831nzJihrdq4caPlx/YkWBfW03/yij7xzkV66956S88Rr7b+IG3Wmdwn\nbdYM1DZgpe7m92pvNxZnK6V+o5RaGf36NZDZm8f2MAAd0FrXRy+/RGT1k78vz3ks7SNmf8v1pvPI\nV2dxqKmFL/zxbT6sif3AdPFq6w/SZp3JfdJmTTK29XYbwZ+BOuCL0a8DwF8sfccopVSeUkpFL8+K\ntsT1fJDxfAEnF2bz9MI5KOCi+9/mg6rQMR/TWTL+50oEk9vA7D5psyYZ23o7EJRqrW/TWm+Jfv0Y\nGN3TA5RSTwBvA+OVUlVKqSuVUlcrpa6O3uVCYF10G8HdwMXRty9xU1lZGc+nZ+zwLJ65eg6ZaS4u\nfeBd/hPo/bbyeLf1hbRZZ3KftFmTjG29HQgalFInt19RSp0ENPT0AK31JVrrfK21W2tdpLV+UGt9\nn9b6vujt92itJ2mtp2mtT9BaL7c0BzFo34AST8W5mfztmjkMG5zO5Q++y9JNe3v1uES0WSVt1pnc\nJ23WJGNbbweCq4F7lVIBpVQAuAdYaOk7OkB+dgZPL5xDSW4mVz60kn+vP8YWeyGEsFFvDzGxRms9\nDZgKTNVaTwfmx7UsDtp3IU0EvzedJ686gYkFg/n6Y+/xf+9X93j/RLbFStqsM7lP2qxJxrZenY+g\nywcqtV1rPdLSg/vAzvMRWFHf2MLXHv4P727dzzfmjeG6+WNJc8mJ4YQQidUf5yPo8nn78FhbHPND\nFXHgTXfx0FdmcUF5IXcvruSz9y5j466jP4tnR1tvSZt1JvdJmzXJ2NaXgWDAnbi386cFE8njTuU3\nF5Xzx8tnsKcuzHn3vMU9izfR0tpme1tvSJt1JvdJmzXJ2NbjISaUUnV0/QtfAXKexhidNSmP40ty\n+OFz6/jVKx/x6obd/PqL0xgzLMvuNCGEg/X4jkBrnaW1HtzFV5bW2twzOBssJzONey89jnsunc72\n/YdYcPdb3P/mZlrbBtwbLCFEkrC8sdgufdlY3NzcjNvt7uci6/bWNfL9Z9fyyobdHDfSx4/Pm8yU\nomy7s45i2nLrzOQ2MLtP2qwZqG3x2lg84LQf0c8UQ7PS+ePlM/jtReVs2VvPufe8xfVPvM+O/Yfs\nTjuMacutM5PbwOw+abMmGdscNRB0PjSuKZRSfHZ6IQ9eUMS180p5ZUMN83/9Oj/+53r2H2yyOw8w\nc7m1M7kNzO6TNmuSsc1RA4HJMtNS+d5ZZbz+3Xl8/rgiHl4e4LRfLOHeJZU0NMnZz4QQ8SMDgWHy\nsj3c9fmp/PtbpzJ7dC6//PeHzP3VEp5YsZ2mlrZjP4EQQsTIUQNB+xmtTHRk29jhWfzpipk8vXAO\nBb4MbvnHWub+cgkPLdtKuDmx7xAG0nIzjcl90mZNMrY5aq+hgUprzRsf7eXeJZX8J/Axfm8aXz15\nFJefUEyWx8y9F4QQZpG9hqI6n1/UND21KaWYO34Yz1x9Ik9ddQITC7L5xcsfcuJdi/n1Kx/GfaPy\nQF1uJjC5T9qsScY2+VDYADN7dC6zR+fyQVWIPyzZzO8XV/KnpVu5ZNZIvnJSCSNyBtmdKIQYYGQg\nGKCmFvm47/IZbNpdxx9e38wjbwd4aPlWPjM5n6+dMorpI4fYnSiEGCAcNRB4vV67E7pltW3s8Cz+\n56JyvnfWeB5eHuDxFdt5ce0uZhYP4WunjOLMiXmkpvTtQLHJuNwSxeQ+abMmGdtkY3GSqW9s4en/\n7ODPy7ZS9XEDI3MG8dWTSvjCzBFkpjtq3BdCdCIbi6OqqqrsTuhWf7V501189eRRvP7dufzhsuPI\n9aZx+z83cMKdi/jpCxvYvi/2w1c4YbnFi8l90mZNMrY56k/E+vp6uxO61d9trtQUFkzJZ8GUfFZt\n+5iHlgd4aHmAB5dt5fSy4XzlpBJOLM1FqWOvNnLScutvJvdJmzXJ2OaogcCpZhQPYUbxEGoWTOCx\nd7fx+LvbeW3jbsYN93LFiSVcML2QQWnyX0EIp3LUqiGny8v2cMOnxrPs5vn88sKpuFNT+P6z6zjh\nZ4u4818b2VXbYHeiEMIGsrHYwbTWrNz2MX9ZtpWX19WQohTnTivgypNHMbnQvPMiCCGsk43FUcl4\nrtG+UEpxfEkOf7hsBm98bx5fmlPCK+trOOf3b3Hx/W+zaONu2tq0LLc+MLlP2qxJxjZHDQQ1NTV2\nJ3TL7rYROYP40bkTWX7L6dy6oIxt+w5x5cMrOeN/3uDBNz5M+IHuesvu5XYsJvdJmzXJ2OaogUAc\nW3aGm6tOLeXNG+fxu4vLyUxz8fu3g5x012J+v2gToUNmnCxHCNF/ZFcR0SV3agrnlxdy3rQCnnlj\nNS8HWvj1qx/xh9c3c9HxI7jy5FFyXCMhkoSjNhbX1dWRlZXVz0X9YyC0fVhTx/1vbuG51dVo4Jyp\n+Vx16mgmFdi3Ydnk5QZm90mbNQO1raeNxY4aCJqbm3G7zTx+/0Bq21XbwF+WBXj83e3UN7Zwylg/\n15xWypxefkAtnm2mMblP2qwZqG2y11DU5s2b7U7o1kBqy8/O4NYFE1h283xu+nQZFTV1XPqnd/ns\nvct4ed0uWtsS98eFycsNzO6TNmuSsc1RA4HoX9kZbq6ZW8rSG+fxswumEGpo5upH3+PM37zBU//Z\nTmOLmXsaCSEOJwOB6DOPO5VLZ49k8Q1zuefS6WSkpXLT39dy6i+W8MCbW6hvbLE7UQjRA0cNBD6f\nz+6EbiVDW2qK4pypBbzwjathMTsAABeTSURBVJP565WzKB3q5Y6XNnLSXYv5n1c/isuupyYvNzC7\nT9qsScY2R20sFom3ekeIe5dU8uqG3WSmpfJfJxRz5SmjGJblsTtNCEeRjcVRgUDA7oRuJWtb+Qgf\nD3xpJi9/6xROnzCcB5Zu4eSfL+FHz62jOtT3g9yZvNzA7D5psyYZ2xw1EITDYbsTupXsbWV5g7n7\nkuksvmEun5teyBMrtnPaL5bwvWfWsDV40Na2eDK5T9qsScY2Rw0Ewn4l/kzu+vxU3vjePP7rhGL+\n+cFOTv/163zn6dUE+jAgCCGsi9tAoJT6s1Jqj1JqXTe3K6XU3UqpSqXUB0qp4+LV0s7lMveIGk5r\nK/BlcPt5k1h643yuPHkUL63dxem/eYPvPrOGbft6PyCYvNzA7D5psyYZ2+K2sVgpdSpQDzyitZ7c\nxe0LgG8AC4DZwO+01rOP9byysTg57akL88c3tvDoO9toadN8bnoh35g/lpG5cjwjIfqDLRuLtdZv\nAvt7uMv5RAYJrbV+B/AppfLj1QMQDAbj+fR94vS2YVkefnjORJbeOI8vzSnmuTU7mf/r17npbx+w\nY/8hW9v6wuQ+abMmGdvs3EZQCOzodL0qOi1ukvEFTIREtg0b7OG2cyex9MbINoRnV1cz71evc/Pf\nux4QTF5uYHaftFmTjG3mruzqRCl1FXAVQFFRERUVFR23FRcXA7Bt27aOaX6/H7/fT2VlJS0tkU+1\nejyR/dZramoOO4tPaWkp4XCY6urqjml5eXn4fL7Dvo/X66WoqIiqqirq6+s7ppeVlREKhQ47IURh\nYSEej+ew4374fD7y8vIIBAIdW/ZdLhdjxowhGAwSDAY7vl8s81RSUhL3eQoGg9TV1VmaJyuvU0lJ\nCfpQiIvHpXJG4QieXhfiH+9X87dVVZxR6uXiqUPIy3KTl5cHkPDXKZZ5CgaDBAKBhLxOsc4T0OfX\nKV7z1PnnIRGvUyzzFAwGCYVCtvyOONY8Ad3OU0/i+oEypVQJ8EI32wj+CLyutX4iev1DYK7WeldP\nz9mXbQQVFRWUlZVZemy8SVvPamrD3PfGZh5fsZ22Ns2FM4q4dt4YDu7ZbntbT0xYdt2RNmsGaptt\nh6E+xkBwNnAdn2wsvltrPetYz9mXgaChoYGMjAxLj403aeudIweE86fl8c0zyozdqGzSsjuStFkz\nUNt6GgjitmpIKfUEMBfwK6WqgNsAN4DW+j7gJSKDQCVwCPhKvFpE8sjL9nD7eZO4+rTSyIDw7nb+\nb00NF0wv5Np5Yxjlz7Q7UYgBJ557DV2itc7XWru11kVa6we11vdFBwGiewtdq7Uu1VpP0VrHfZ/Q\nzuvTTCNtsWkfEP7y+RFcMaeEf66JfjDtqdVs3lt/7CdIEBOXXTtpsyYZ2+STxWJAyx3k4kfnTmTp\nTfP42imj+de6Gs74zRtc/8T7fLS7zu48IQYEGQhEUhiW5eHWBRN466Z5XH1aKYs27uas377J1x9b\nxfqdtXbnCWG0AbH7aH9p373KRNJmzZFtud50bvp0GVedMpoH39rKw8sDvLS2hjMmDOPaeWOYPnKI\nrX0mkTZrkrFNzkcgklptQzOPLA/w4LKthA41c8pYP9fNG8Ps0bl2pwmRUHI+gqjKykq7E7olbdYc\nqy07w803Th/Lspvmc+uCMjbuquOi+9/hC/ct542P9hLvP4QG8rKzk7RZY7XNUQNB+6ftTCRt1vS2\nLTPdxVWnlvLWTfP48XmTqPq4gSv+vILz713Gv9fX0NYWnwEhGZadHaTNGqttjhoIhPC4U7nixBLe\n+N487vzcFEKHmln411V8+ndv8tzqalpa2+xOFCLhHDUQtB9vyETSZo3VtjRXCpfMGsniG07jtxeV\nozV888nVnP6bN3hyxXaaWvpnQEjGZZcI0maN1TbZWCwE0NameWXDbu5dUsna6lrysz1cdepoLj5+\nJBlpqXbnCdFnsrE46lhH4LOTtFnTX20pKYpPT87j+etO4uGvzmLEkEH8+J8bOPnni7l3SSUHws22\n9sWDtFmTjG2OGgg6H1rWNNJmTX+3KaU4bdxQnr56Dk8vnMPkwmx++e8POemuxfzq3x+y/2CTrX39\nSdqsScY2R32gTIhYzBqVw6xRs1hXXcu9Syq59/VKHnxrK5fMGsl/nzqK/Gwzj0ApRKxkIBDiGCYX\nZvO//zWDyj11/O/rW3j47QB/fSfAhTOK+MpJoxg3PMvuRCH6xFEbi5ubm3G73f1c1D+kzRo72nbs\nP8T9b27hqZU7aGpp47iRPi6eNZJzpuYzKO3wv61k2Vkjbdb01CYbi6PaT/9mImmzxo62ETmD+H+f\nncw7t5zOD86eQG1DMzf+7QNm37GI7z+7lnXVnxzkTpadNdJmjdU2Rw0Enc+jahpps8bOtpzMNL52\nymhe+85pPHP1HM6cNJy/rarinN+/xTm/X8qj72xjU2CHbX3HIq+rNcnYJtsIhOgjpRTHl+RwfEkO\nt507iedWV/PEih384P/WMcituHBLG5efUMxY2ZYgDCUDgRD9KDvDzZfmlHD5CcWsqarlnpfX8OSK\nHTzy9jbmjM7lS3OKOXPicFypjnozLgznqIEgLy/P7oRuSZs1prYppSgf4eNXX5hGqyuDp1bu4LF3\ntnPNY++RN9jDpbNHcvGsEQzLsu9wBaYuO5A2q6y2OWqvISHs1NqmWVyxh0feDrB0U5DUFMUpY/2c\nX17AmRPz8KY76u8ykWCy11BURUWF3QndkjZrTG6Dw/tSUxRnThzOX6+czeIbTuO/TxnNpt31fPup\nNcz86atc9/h7vLphd78d8C6WNtNImzVW2+RPECFsMHqol5s/U8aNZ43nve0f89zqnby4dhcvfLCL\n7Aw3C6bkce60AmaPyiU1RdmdK5KcDARC2CglRTGzJIeZJTn86NyJvFUZ5PnVO3lu9U6eWLEDvzed\nBVPyOHtKPseX5JAig4KIA0cNBF6v1+6EbkmbNSa3QWx97tQU5o0fxrzxw2hoamVxxR5eXLuTp1dG\n9joaPjidz0zO59xp+UwfMaTPg4LJy07arLHaJhuLhTDcwcYWFlXs4YU1O3n9o700tbRRkO3hU5Py\nOGPCcGaNyiHN5ajNfcKCnjYWO2ogqKqqoqioqJ+L+oe0WWNyG/R/X124mdc27uaFNbtYWhmkqaWN\nrHQXp44byukTIu8mhmSm2dLWn6TNmp7aehoIHLVqqL6+3u6EbkmbNSa3Qf/3ZXncXDC9iAumF3Go\nqYVllftYtHE3iyr28OLaXaQomFE8hNMnDOeMCcMoHepFqa5XIZm87KTNGqttjhoIhEgmg9JcnDlx\nOGdOHE5bm2ZtdS2LNu7mtY17uOtfFdz1rwpG5gxiftkwTp8wjFmjckh3yWk3xdFkIBAiCaSkKKaN\n8DFthI/vfGo8O0MNLK7Yw+KKPTyxYjsPLQ+QmZbKqeOGMr9sGPPKhtmdLAziqG0EQjhRQ1MryzcH\neW3jHhZX7Gb3gUYAyvKyOGF0LrNH5TBrVA653nSbS0U8ycbiqFAohM/n6+ei/iFt1pjcBub1aa1Z\nv/MAr3+4h7c+2sOa6joamlsBGDfcGx0Ycpk1KoehWfYNDKYtt84GaptsLI6qqakx9gWUNmtMbgPz\n+pRSTC7MZnJhNmcUtDJ6zBzWVod4Z8t+3t26n7+tquKRt7cBMDJnENNH+pg+wsf0kUOYkD84Ybup\nmrbcOkvGNkcNBEKIw6W5UphRnMOM4hyunQfNrW2sq65lxdb9rN4R4p0t+3hu9c6O+04pzGZ6dFvE\n5MJsinMGyaedk4AMBEKIDu7UFKaPHML0kUM6pu2qbeD97SHe3/4x728P8dd3tvGnt7YCkJXuYmLB\n4Oi7jMFMLshm9FCvHB9pgHHUQFBYWGh3QrekzRqT28Dsvt625WdnkD8lgwVT8gFoamnjo911rN9Z\ny7rqA6zbWcuj72yjMXrU1Ax3KhMLBjO1KJupRdlMKfQx2p8Z0zuHZFhudrDa5qiNxc3Nzbjd7n4u\n6h/SZo3JbWB2X3+2tbS2sXnvQdZV17K2upZ11bWs33mgY0O0N93F5MLBTC3yMaUwm7HDvZTkZuJx\nd/25Bqcst/7WU5tsLI7avHkzZWVldmd0SdqsMbkNzO7rzzZXagrj87IYn5fF52dEDnHQPjh8UBVi\nbXUta6pqeWh54LDzLeRneyjJzaTEn8ko/yCKczMZ7c+kIVjF1EkT+qWtvyXja+qogUAIkTidB4cv\nzBwBRFYrbdpTx+a9BwkEI19b9x3k5XW7+PhQc8djUxSMzKmhdKiX0mFeSodmMnqol9KhXnJ6eSwl\n0XsyEAghEibNlcKkgmwmFWQfdVvtoWYC+w6yNXiQFRXbqG3zsHlvfcfB9doNGeSOvIPIzWSUv/3d\nRORfOd2nNXFdakqpTwO/A1KBP2mt7zri9i8DvwSqo5Pu0Vr/KV49pu77C9JmlcltYHafaW3Zg9xM\nGxTZNfWE/NSOE7G3tmmqP25gc7CezXvqO95NvL1lH/94v/qw5xialU5xziAKfBnk+zwUZGeQn+2h\nwJdBgS+DIYPc3R6Er7dMW26dWW2L28ZipVQq8BFwJlAF/Ae4RGu9odN9vgzM1Fpf19vnlUNMCCHa\nNTS1EtgXGRi2RFc1bd9/iF21YWpqwzS1Hn7+Z487hQJfBkVDBjFiSOTfoiEZjMiJ/JubmdbngcJU\ndm0sngVUaq23RCOeBM4HNvT4qDgKBAKUlJTY9e17JG3WmNwGZvclQ1tGWioT8gczIX/wUbe1tWmC\nBxvZFQqzq7aBndF/q0MN7NjfwNqq0GHbJSCy6+vQrHRyvWnkZqaRk5lGTmY6fm/kcq43naYDQaaM\nHUVOZppxJwSy+prGcyAoBHZ0ul4FzO7ifp9XSp1K5N3Dt7XWO468g1LqKuAqgKKiIioqKjpuKy4u\nBmDbtm0d0/x+P36/n8rKSlpaWgDweDyEw2FqamoIhUId9y0tLSUcDlNd/clbzLy8PHw+32Hfx+v1\nUlRURFVV1WHH/C4rKyMUClFTU/PJjBcW4vF42Lx5c8c0n89HXl4egUCAcDgMgMvlYsyYMQSDQaqq\nqjqmxzJPJSUlcZ+nYDBIbm5uzPMUDAYtvU6xzFM4HE7o6xTrPLXfPxGvU6zz1N6XiNcp1nnq/PPQ\n19dpXHEx4/zpkXnypwCZ+P3F+P1+PtjwEdW1Deyub2F/GA6qDHbsrWXvgQa21Bzi/cZWasNttLQd\nueZkS2Q+0lLweVLxZaTi96YzYqiPtLYwmaltDMlIZUhGKjMmjcPd2kBo315Lr1Ms//fC4XC3r1NP\n4rlq6ELg01rrr0WvXw7M7rwaSCmVC9RrrRuVUguBi7TW83t63r6sGqqoqDB2ty9ps8bkNjC7T9p6\nR2vNgXAL+w82sa++kTUfbsHjG8q++sj14MEmgnWN7DvYRLC+kdAR7zLaZaW7yPWmMSQzjSGD2r/c\nHddzMt0MznAz2OPGm+4iy+Miy+OO6V1HT8vNrlVD1cCITteL+GSjMABa632drv4J+EUce3C5zN2j\nQNqsMbkNzO6Ttt5RSpGd4SY7w80ofya+lv2MGVPc7f2bWtrYd7CRYF0Te+vD0X8b2VvX2DFQ7D4Q\npmLXAT4+1NzxobvupLlSGOxx4U138V8nFPO1U0Z3e1+ryy2e7whcRFb3nE5kAPgPcKnWen2n++Rr\nrXdFL18A3KS1PqGn55WNxUKIZBJubuXjQ03sP9jEgYYW6sLN1De2UBeOXK6LXq4Pt3D6hGGcX27x\nMBJ2vCPQWrcopa4D/k1k99E/a63XK6V+AqzUWj8PXK+UOg9oAfYDX45XD0TWdfv9/nh+C8ukzRqT\n28DsPmmzpr/bPO7UyPGcsjP6/FxW2+K6yVtr/ZLWepzWulRrfUd02o+igwBa61u01pO01tO01vO0\n1hU9P2PfdN7YYhpps8bkNjC7T9qsScY2s/Z9EkIIkXAyEAghhMM5aiBo3/fWRNJmjcltYHaftFmT\njG2OGgiEEEIczVEDQedP4ZlG2qwxuQ3M7pM2a5KxzVEDgRBCiKPJQCCEEA434M5ZrJTaC1h9b+YH\nTN0JWNqsMbkNzO6TNmsGalux1npoVzcMuIGgL5RSK7v7iLXdpM0ak9vA7D5psyYZ22TVkBBCOJwM\nBEII4XBOGwjutzugB9JmjcltYHaftFmTdG2O2kYghBDiaE57RyCEEOIIMhAIIYTDOWYgUEp9Win1\noVKqUil1s909nSmlAkqptUqp1UopW0+/ppT6s1Jqj1JqXadpOUqpV5VSm6L/DjGo7XalVHV02a1W\nSi2wqW2EUmqJUmqDUmq9Uuqb0em2L7se2mxfdkopj1JqhVJqTbTtx9Hpo5RS70Z/Xp9SSqUZ1PaQ\nUmprp+VWnui2To2pSqn3lVIvRK9bW25a66T/InKGtM3AaCANWANMtLurU18A8NvdEW05FTgOWNdp\n2i+Am6OXbwZ+blDb7cB3DVhu+cBx0ctZRE7TOtGEZddDm+3LDlCAN3rZDbwLnAA8DVwcnX4fcI1B\nbQ8BF9r9fy7a9R3gceCF6HVLy80p7whmAZVa6y1a6ybgSeB8m5uMpLV+k8hpQzs7H3g4evlh4LMJ\njYrqps0IWutdWuv3opfrgI1AIQYsux7abKcj6qNX3dEvDcwH/hadbtdy667NCEqpIuBs4E/R6wqL\ny80pA0EhsKPT9SoM+UGI0sArSqlVSqmr7I7pwnCt9a7o5RpguJ0xXbhOKfVBdNWRLautOlNKlQDT\nifwFadSyO6INDFh20dUbq4E9wKtE3r2HtNYt0bvY9vN6ZJvWun253RFdbv+jlEq3ow34LXAj0Ba9\nnovF5eaUgcB0J2utjwM+A1yrlDrV7qDu6Mh7TmP+KgL+FygFyoFdwK/tjFFKeYG/A9/SWh/ofJvd\ny66LNiOWnda6VWtdDhQRefdeZkdHV45sU0pNBm4h0ng8kAPclOgupdQ5wB6t9ar+eD6nDATVwIhO\n14ui04ygta6O/rsHeJbID4NJdiul8gGi/+6xuaeD1np39Ie1DXgAG5edUspN5BftY1rrf0QnG7Hs\numozadlFe0LAEmAO4FNKuaI32f7z2qnt09FVbVpr3Qj8BXuW20nAeUqpAJFV3fOB32FxuTllIPgP\nMDa6RT0NuBh43uYmAJRSmUqprPbLwKeAdT0/KuGeB66IXr4CeM7GlsO0/5KNugCbll10/eyDwEat\n9W863WT7suuuzYRlp5QaqpTyRS9nAGcS2YaxBLgweje7lltXbRWdBnZFZB18wpeb1voWrXWR1rqE\nyO+zxVrry7C63Oze6p2oL2ABkb0lNgPft7unU9doInsxrQHW290GPEFkNUEzkXWMVxJZ97gI2AS8\nBuQY1PZXYC3wAZFfuvk2tZ1MZLXPB8Dq6NcCE5ZdD222LztgKvB+tGEd8KPo9NHACqASeAZIN6ht\ncXS5rQMeJbpnkV1fwFw+2WvI0nKTQ0wIIYTDOWXVkBBCiG7IQCCEEA4nA4EQQjicDARCCOFwMhAI\nIYTDyUAgxBGUUq2djiy5WvXj0WqVUiWdj54qhAlcx76LEI7ToCOHFRDCEeQdgRC9pCLnjfiFipw7\nYoVSakx0eolSanH0IGSLlFIjo9OHK6WejR7Pfo1S6sToU6UqpR6IHuP+leinVoWwjQwEQhwt44hV\nQxd1uq1Waz0FuIfI0R8Bfg88rLWeCjwG3B2dfjfwhtZ6GpHzKKyPTh8L3Ku1ngSEgM/HeX6E6JF8\nsliIIyil6rXW3i6mB4D5Wust0YO41Witc5VSQSKHZ2iOTt+ltfYrpfYCRTpycLL25yghcjjjsdHr\nNwFurfVP4z9nQnRN3hEIERvdzeVYNHa63IpsqxM2k4FAiNhc1Onft6OXlxM5AiTAZcDS6OVFwDXQ\ncYKT7ERFChEL+UtEiKNlRM9K1e5lrXX7LqRDlFIfEPmr/pLotG8Af1FKfQ/YC3wlOv2bwP1KqSuJ\n/OV/DZGjpwphFNlGIEQvRbcRzNRaB+1uEaI/yaohIYRwOHlHIIQQDifvCIQQwuFkIBBCCIeTgUAI\nIRxOBgIhhHA4GQiEEMLh/j/M747rOPVgTgAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"render_training_history(history)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "X-dhNP2OG2EM"
},
"source": [
"## Generate text"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "SU_YfP6sG3NC"
},
"source": [
"### Restore the latest checkpoint\n",
"\n",
"To keep this prediction step simple, use a batch size of 1.\n",
"\n",
"Because of the way the RNN state is passed from timestep to timestep, the model only accepts a fixed batch size once built.\n",
"\n",
"To run the model with a different `batch_size`, we need to rebuild the model and restore the weights from the checkpoint."
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1197040,
"status": "ok",
"timestamp": 1584521003726,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "BlG8o3wiG6f2",
"outputId": "031b45e0-fcdf-4ab6-b7e9-e167b29800c9"
},
"outputs": [
{
"data": {
"text/plain": [
"'tmp/checkpoints/ckpt_40'"
]
},
"execution_count": 44,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"tf.train.latest_checkpoint(checkpoint_dir)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "l7evN0LvH01P"
},
"outputs": [],
"source": [
"simplified_batch_size = 1\n",
"\n",
"model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)\n",
"\n",
"model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))\n",
"\n",
"model.build(tf.TensorShape([simplified_batch_size, None]))"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 255
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1196901,
"status": "ok",
"timestamp": 1584521004053,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Y3eduDtZI9zQ",
"outputId": "95ccdb2f-f059-4be2-ee85-3541f422c7ea"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"sequential_2\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"embedding_2 (Embedding) (1, None, 256) 16640 \n",
"_________________________________________________________________\n",
"lstm_1 (LSTM) (1, None, 1024) 5246976 \n",
"_________________________________________________________________\n",
"dense_1 (Dense) (1, None, 65) 66625 \n",
"=================================================================\n",
"Total params: 5,330,241\n",
"Trainable params: 5,330,241\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "euNvAtr4JC3A"
},
"source": [
"### The prediction loop\n",
"\n",
"The following code block generates the text:\n",
"\n",
"- It Starts by choosing a start string, initializing the RNN state and setting the number of characters to generate.\n",
"\n",
"- Get the prediction distribution of the next character using the start string and the RNN state.\n",
"\n",
"- Then, use a categorical distribution to calculate the index of the predicted character. Use this predicted character as our next input to the model.\n",
"\n",
"- The RNN state returned by the model is fed back into the model so that it now has more context, instead than only one character. After predicting the next character, the modified RNN states are again fed back into the model, which is how it learns as it gets more context from the previously predicted characters.\n",
"\n",
"\n",
"\n",
"Image source: [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation) notebook."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "bOqdqGouJFf_"
},
"outputs": [],
"source": [
"# num_generate\n",
"# - number of characters to generate.\n",
"#\n",
"# temperature\n",
"# - Low temperatures results in more predictable text.\n",
"# - Higher temperatures results in more surprising text.\n",
"# - Experiment to find the best setting.\n",
"def generate_text(model, start_string, num_generate = 1000, temperature=1.0):\n",
" # Evaluation step (generating text using the learned model)\n",
"\n",
" # Converting our start string to numbers (vectorizing).\n",
" input_indices = [char2index[s] for s in start_string]\n",
" input_indices = tf.expand_dims(input_indices, 0)\n",
"\n",
" # Empty string to store our results.\n",
" text_generated = []\n",
"\n",
" # Here batch size == 1.\n",
" model.reset_states()\n",
" for char_index in range(num_generate):\n",
" predictions = model(input_indices)\n",
" # remove the batch dimension\n",
" predictions = tf.squeeze(predictions, 0)\n",
"\n",
" # Using a categorical distribution to predict the character returned by the model.\n",
" predictions = predictions / temperature\n",
" predicted_id = tf.random.categorical(\n",
" predictions,\n",
" num_samples=1\n",
" )[-1,0].numpy()\n",
"\n",
" # We pass the predicted character as the next input to the model\n",
" # along with the previous hidden state.\n",
" input_indices = tf.expand_dims([predicted_id], 0)\n",
"\n",
" text_generated.append(index2char[predicted_id])\n",
"\n",
" return (start_string + ''.join(text_generated))"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 578
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1202383,
"status": "ok",
"timestamp": 1584521010077,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "Z-8e8P60J5Pg",
"outputId": "25240489-3063-4bdd-b203-ef14886d1e60"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ROMEO: God do not burden me.\n",
"\n",
"GREMIO:\n",
"I want the king that murders writ upon your tongue\n",
"And born too supplict, but thanks\n",
"We should to him and nothing can I never my\n",
"brothers forbeard our words:\n",
"Confess which have wine, and provide thee\n",
"And see how did bestride our nobising duke,\n",
"Whose souls more pleased, that men are worst challenge into a farmet together\n",
"Let hate not so, for these own with him.\n",
"\n",
"Tailors:\n",
"But now the blood of two,\n",
"You and your accused find your wars.\n",
"\n",
"MENENIUS:\n",
"Part not chastisement; hidest me well arm'd\n",
"With eterpity of the mon poor house.\n",
"\n",
"Third Servant:\n",
"When you are gentlemen, some interr'd\n",
"As is my goods, God says King Richard say,\n",
"As is the sin that she doth aim at you\n",
"As I could pick my daughter woful side!\n",
"Thou follow'st me in his life, against your husband's land:\n",
"My lawful king is dead! wis gone to ET:\n",
"Where is my woful? Was never man so bath\n",
"Lord Her further.\n",
"\n",
"BRUTUS:\n",
"Let me prepare from fair, nor ambition\n",
"That spucish him our absence, like a grave is no more drea\n"
]
}
],
"source": [
"# Generate the text with default temperature (1.0).\n",
"print(generate_text(model, start_string=u\"ROMEO: \"))"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 561
},
"colab_type": "code",
"executionInfo": {
"elapsed": 1208506,
"status": "ok",
"timestamp": 1584521016280,
"user": {
"displayName": "Oleksii Trekhleb",
"photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
"userId": "03172675069638383074"
},
"user_tz": -60
},
"id": "wq_NlwWJSdix",
"outputId": "408ecde5-0646-4eda-902e-071e4f123417"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ROMEO: hexen his virginallike.\n",
"\n",
"BRUTUS:\n",
"For thyself\n",
"Look, sirr; let not this innocent mine friends\n",
"And hateful dogs as appeared, for it exced\n",
"Exelling clouds with gentle knterpleaking:\n",
"Their liberal dogs not his;\n",
"For Yorkome help, ho! ay, I trow:\n",
"it will we meet us king E:\n",
"Needs friends, father, in that mad bessisking in\n",
"Whom I might thrive is rich'd die by thee?\n",
"\n",
"CORIOLANUS:\n",
"Brak how, gnow roaring with Sicils\n",
"Up Jusul's dee Signior Katharina,\n",
"And thyself shalt hear of that which the ear o' Lord:\n",
"You are the first gape on me at first,\n",
"Thou lovest me wash:\n",
"By heaven, I'll haughthere.\n",
"O, gentle marvantio; and beginn pluck'd\n",
"But the christs and Riciards a\n",
"LUUMEOfoke fray,\n",
"And yet whick, in my doesder differences be runed by the king,\n",
"Who in his monthry glorious regeal death.\n",
"O, that she smooth thy cause?\n",
"Was e turn be so\n",
"to thine as thee won that Edward, king, widdwisk\n",
"Ere te goil: deserve not thy mother leave for reason\n",
"many an hourable fought, as they do,\n",
"If you should esear mon-judgment,\n",
"Nor \n"
]
}
],
"source": [
"# Generate the text with higher temperature to get more unexpected results.\n",
"print(generate_text(model, start_string=u\"ROMEO: \", temperature=1.5))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "0hh80MqEO_XI"
},
"source": [
"## Save the model"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "VPE98xa8PA-u"
},
"outputs": [],
"source": [
"model_name = 'text_generation_shakespeare_rnn.h5'\n",
"model.save(model_name, save_format='h5')"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "WYP08xbbTNKp"
},
"source": [
"## Converting the model to web-format\n",
"\n",
"To use this model on the web we need to convert it into the format that will be understandable by [tensorflowjs](https://www.tensorflow.org/js). To do so we may use [tfjs-converter](https://github.com/tensorflow/tfjs/tree/master/tfjs-converter) as following:\n",
"\n",
"```\n",
"tensorflowjs_converter --input_format keras \\\n",
" ./experiments/text_generation_shakespeare_rnn/text_generation_shakespeare_rnn.h5 \\\n",
" ./demos/public/models/text_generation_shakespeare_rnn\n",
"```\n",
"\n",
"You find this experiment in the [Demo app](https://trekhleb.github.io/machine-learning-experiments) and play around with it right in you browser to see how the model performs in real life."
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "text_generation_shakespeare_rnn.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 1
}