{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Vanilla LSTM with CuPy\n", "\n", "This is a port of [Vanilla LSTM with numpy](http://blog.varunajayasiri.com/numpy_lstm.html) that shows how to run the numpy-based machine learning code on GPU using [CuPy](https://github.com/cupy/cupy). The all contents below is basically copied from the article: [Vanilla LSTM with numpy](http://blog.varunajayasiri.com/numpy_lstm.html).\n", "\n", "This is inspired from [Minimal character-level language model with a Vanilla Recurrent Neural Network](https://gist.github.com/karpathy/d4dee566867f8291f086), in Python/numpy by [Andrej Karpathy](https://github.com/karpathy).\n", "\n", "The model usually reaches an error of about 45 after 5000 iterations when tested with [100,000 character sample from Shakespeare](http://cs.stanford.edu/people/karpathy/char-rnn/shakespear.txt). However it sometimes get stuck in a local minima; reinitialize the weights if this happens.\n", "\n", "You need to place the input text file as `input.txt` in the same folder as the python code." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download data" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " % Total % Received % Xferd Average Speed Time Time Time Current\n", " Dload Upload Total Spent Left Speed\n", "\r", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r", "100 99993 100 99993 0 0 70182 0 0:00:01 0:00:01 --:--:-- 149k\n" ] } ], "source": [ "%%bash\n", "curl -L http://cs.stanford.edu/people/karpathy/char-rnn/shakespear.txt -o input.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import numpy as np\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "from IPython import display\n", "\n", "np.random.seed(2017)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above imports are from the original article as is. But in this article, you can run almost all computation on GPU by just replacing `np` with `cp`. Well, `cp` is just another name of `cupy`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import cupy as cp\n", "\n", "cp.random.seed(2017)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To show the difference of computational time by switching CPU and GPU simply, let's use `xp` instead of `np` and switch the referenced package between `numpy` and `cupy`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# If you use CPU\n", "# xp = np\n", "\n", "# If you use GPU: almost 2 times faster\n", "xp = cp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Because CuPy has high compatibility with NumPy, the same code using NumPy is easily converted for CuPy by just replacing `numpy` with `cupy`**." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:**\n", "\n", "If the `H_size` is larger than 1000 or so, training on GPU, namely, using CuPy is several times faster than NumPy. But when the array size is small, the speed gain is often small." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Read and process data" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": true }, "outputs": [], "source": [ "data = open('input.txt', 'r').read()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Process data and calculate indexes" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "data has 99993 characters, 62 unique\n" ] } ], "source": [ "chars = list(set(data))\n", "data_size, X_size = len(data), len(chars)\n", "print(\"data has %d characters, %d unique\" % (data_size, X_size))\n", "char_to_idx = {ch:i for i,ch in enumerate(chars)}\n", "idx_to_char = {i:ch for i,ch in enumerate(chars)}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parameters" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "H_size = 100 # Size of the hidden layer\n", "T_steps = 25 # Number of time steps (length of the sequence) used for training\n", "learning_rate = 1e-1 # Learning rate\n", "weight_sd = 0.1 # Standard deviation of weights for initialization\n", "z_size = H_size + X_size # Size of concatenate(H, X) vector" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Activation Functions and Derivatives\n", "\n", "### Sigmoid\n", "\n", "$$\n", "\\begin{eqnarray}\n", "\\sigma(x) &=& \\frac{1}{1 + e^{-x}} \\\\\n", "\\frac{d \\sigma(x)}{d x} &=& \\sigma(x) \\cdot (1 - \\sigma(x))\n", "\\end{eqnarray}\n", "$$\n", "\n", "### Tanh\n", "\n", "$$\n", "\\frac{d \\tanh(x)}{dx} = 1 - \\tanh^2(x)\n", "$$" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def sigmoid(x):\n", " return 1 / (1 + xp.exp(-x))\n", "\n", "\n", "def dsigmoid(y):\n", " return y * (1 - y)\n", "\n", "\n", "def tanh(x):\n", " return xp.tanh(x)\n", "\n", "\n", "def dtanh(y):\n", " return 1 - y * y" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Initialize weights\n", "\n", "We use random weights with normal distribution `(0, weight_sd)` for `tanh` activation function and `(0.5, weight_sd)` for `sigmoid` activation function.\n", "\n", "Biases are initialized to zeros.\n", "\n", "Formulae for LSTM are shown below." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": true }, "outputs": [], "source": [ "W_f = (xp.random.randn(H_size, z_size) * weight_sd + 0.5).astype(xp.float32)\n", "b_f = xp.zeros((H_size, 1), dtype=xp.float32)\n", "\n", "W_i = (xp.random.randn(H_size, z_size) * weight_sd + 0.5).astype(xp.float32)\n", "b_i = xp.zeros((H_size, 1), dtype=xp.float32)\n", "\n", "W_C = (xp.random.randn(H_size, z_size) * weight_sd).astype(xp.float32)\n", "b_C = xp.zeros((H_size, 1), dtype=xp.float32)\n", "\n", "W_o = (xp.random.randn(H_size, z_size) * weight_sd + 0.5).astype(xp.float32)\n", "b_o = xp.zeros((H_size, 1), dtype=xp.float32)\n", "\n", "#For final layer to predict the next character\n", "W_y = (xp.random.randn(X_size, H_size) * weight_sd).astype(xp.float32)\n", "b_y = xp.zeros((X_size, 1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Gradients" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dW_f = xp.zeros_like(W_f, dtype=xp.float32)\n", "dW_i = xp.zeros_like(W_i, dtype=xp.float32)\n", "dW_C = xp.zeros_like(W_C, dtype=xp.float32)\n", "\n", "dW_o = xp.zeros_like(W_o, dtype=xp.float32)\n", "dW_y = xp.zeros_like(W_y, dtype=xp.float32)\n", "\n", "db_f = xp.zeros_like(b_f, dtype=xp.float32)\n", "db_i = xp.zeros_like(b_i, dtype=xp.float32)\n", "db_C = xp.zeros_like(b_C, dtype=xp.float32)\n", "\n", "db_o = xp.zeros_like(b_o, dtype=xp.float32)\n", "db_y = xp.zeros_like(b_y, dtype=xp.float32)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Forward pass\n", "\n", "![](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png)\n", "\n", "Image taken from [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/). Please read the article for a good explanation of LSTMs.\n", "\n", "### Concatenation of $h_{t-1}$ and $x_t$\n", "\n", "$$\n", "z = [h_{t-1}, x_t]\n", "$$\n", "\n", "### LSTM functions\n", "\n", "$$\n", "\\begin{eqnarray}\n", "f_t &=& \\sigma(W_f \\cdot z + b_f) \\\\\n", "i_t &=& \\sigma(W_i \\cdot z + b_i) \\\\\n", "\\bar{C}_t &=& \\tanh(W_C \\cdot z + b_C) \\\\\n", "C_t &=& f_t \\ast C_{t-1} + i_t \\ast \\bar{C}_t \\\\\n", "o_t &=& \\sigma(W_O \\cdot z + b_t) \\\\\n", "h_t &=& o_t \\ast \\tanh(C_t)\n", "\\end{eqnarray}\n", "$$\n", "\n", "### Logits\n", "\n", "$$\n", "y_t = W_y \\cdot h_t + b_y\n", "$$\n", "\n", "### Softmax\n", "\n", "$$\n", "\\hat{p}_t = {\\rm softmax}(y_t)\n", "$$\n", "\n", "$\\hat{p}_t$ is `p` in code and $p_t$ is `targets`." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def forward(x, h_prev, C_prev):\n", " assert x.shape == (X_size, 1)\n", " assert h_prev.shape == (H_size, 1)\n", " assert C_prev.shape == (H_size, 1)\n", "\n", " z = xp.concatenate((h_prev, x))\n", " f = sigmoid(xp.dot(W_f, z) + b_f)\n", " i = sigmoid(xp.dot(W_i, z) + b_i)\n", " C_bar = tanh(xp.dot(W_C, z) + b_C)\n", "\n", " C = f * C_prev + i * C_bar\n", " o = sigmoid(xp.dot(W_o, z) + b_o)\n", " h = o * tanh(C)\n", "\n", " y = xp.dot(W_y, h) + b_y\n", " y -= y.max()\n", " p = xp.exp(y) / xp.sum(xp.exp(y))\n", "\n", " return z, f, i, C_bar, C, o, h, y, p" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Backward pass\n", "\n", "### Loss\n", "\n", "$$\n", "\\mathcal{L} = - \\sum p_{t,j}log \\hat{p}_{t,j}\n", "$$\n", "\n", "### Gradients\n", "\n", "$$\n", "\\begin{eqnarray}\n", "dy_t &=& \\hat{p}_t - p_t \\\\\n", "dh_t &=& dh'_{t+1} + W_y^T \\cdot d_y \\\\\n", "do_t &=& dh_t \\ast \\tanh (C_t) \\\\\n", "dC_t &=& dC'_{t+q} + dh_t \\ast o_t \\ast (1 - \\tanh^2(C_t)) \\\\\n", "d \\bar{C}_t &=& d C_t \\ast i_t \\\\\n", "d i_t &=& d C_t \\ast \\bar{C}_t \\\\\n", "d f_t &=& d C_t \\ast C_{t-1} \\\\\n", "d z_t &=& W_f^T \\cdot df_t + W_i^T \\cdot di_t + W_C^T \\cdot \\bar{C}_t + W^T_o \\cdot do_t \\\\\n", "[dh'_t, dx_t] &=& dz_t \\\\\n", "dC'_t &=& f \\ast dC_t\n", "\\end{eqnarray}\n", "$$\n", "\n", "- `target` is target character index $p_t$\n", "- `dh_next` is $dh_{t+1}$ (size $H \\times 1$)\n", "- `dC_next` is $dC_{t+1}$ (size $H \\times 1$)\n", "- `C_prev` is $C_{t-1}$ (size $H \\times 1$)\n", "- Returns $dh_t$ and $dC_t$" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def backward(target, dh_next, dC_next, C_prev, z, f, i, C_bar, C, o, h, y, p):\n", "\n", " global dW_f, dW_i, dW_C, dW_o, dW_y\n", " global db_f, db_i, db_C, db_o, db_y\n", "\n", " assert z.shape == (X_size + H_size, 1)\n", " assert y.shape == (X_size, 1)\n", " assert p.shape == (X_size, 1)\n", "\n", " for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:\n", " assert param.shape == (H_size, 1)\n", "\n", " dy = xp.copy(p)\n", " dy[target] -= 1\n", "\n", " dW_y += xp.dot(dy, h.T)\n", " db_y += dy\n", "\n", " dh = xp.dot(W_y.T, dy)\n", " dh += dh_next\n", " do = dh * tanh(C)\n", " do = dsigmoid(o) * do\n", " dW_o += xp.dot(do, z.T)\n", " db_o += do\n", "\n", " dC = xp.copy(dC_next)\n", " dC += dh * o * dtanh(tanh(C))\n", " dC_bar = dC * i\n", " dC_bar = dC_bar * dtanh(C_bar)\n", " dW_C += xp.dot(dC_bar, z.T)\n", " db_C += dC_bar\n", "\n", " di = dC * C_bar\n", " di = dsigmoid(i) * di\n", " dW_i += xp.dot(di, z.T)\n", " db_i += di\n", "\n", " df = dC * C_prev\n", " df = dsigmoid(f) * df\n", " dW_f += xp.dot(df, z.T)\n", " db_f += df\n", "\n", " dz = xp.dot(W_f.T, df) \\\n", " + xp.dot(W_i.T, di) \\\n", " + xp.dot(W_C.T, dC_bar) \\\n", " + xp.dot(W_o.T, do)\n", " dh_prev = dz[:H_size, :]\n", " dC_prev = f * dC\n", "\n", " return dh_prev, dC_prev" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Forward Backward Pass\n", "\n", "Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.\n", "\n", "- `input`, `target` are list of integers, with character indexes.\n", "- `h_prev` is the array of initial `h` at $h_1$ (size $H \\times 1$)\n", "- `C_prev` is the array of initial `C` at $C_1$ (size $H \\times 1$)\n", "- Returns loss, final $h_T$ and $C_T$" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def forward_backward(inputs, targets, h_prev, C_prev):\n", " # To store the values for each time step\n", " x_s, z_s, f_s, i_s, C_bar_s, C_s, o_s, h_s, y_s, p_s = {}, {}, {}, {}, {}, {}, {}, {}, {}, {}\n", "\n", " # Values at t - 1\n", " h_s[-1] = xp.copy(h_prev)\n", " C_s[-1] = xp.copy(C_prev)\n", "\n", " loss = 0\n", " # Loop through time steps\n", " assert len(inputs) == T_steps\n", " for t in range(len(inputs)):\n", " x_s[t] = xp.zeros((X_size, 1), dtype=xp.float32)\n", " x_s[t][inputs[t]] = 1 # Input character\n", "\n", " z_s[t], f_s[t], i_s[t], C_bar_s[t], C_s[t], o_s[t], h_s[t], y_s[t], p_s[t] \\\n", " = forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass\n", "\n", " loss += -xp.log(p_s[t][targets[t], 0]) # Loss for at t\n", "\n", "\n", " for dparam in [dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y]:\n", " dparam.fill(0)\n", "\n", " dh_next = xp.zeros_like(h_s[0], dtype=xp.float32) #dh from the next character\n", " dC_next = xp.zeros_like(C_s[0], dtype=xp.float32) #dh from the next character\n", "\n", " for t in reversed(range(len(inputs))):\n", " # Backward pass\n", " dh_next, dC_next = backward(target = targets[t], dh_next = dh_next, dC_next = dC_next, C_prev = C_s[t-1],\n", " z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t], C = C_s[t], o = o_s[t],\n", " h = h_s[t], y = y_s[t], p = p_s[t])\n", "\n", " # Clip gradients to mitigate exploding gradients\n", " for dparam in [dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y]:\n", " xp.clip(dparam, -1, 1, out=dparam)\n", "\n", " return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sample the next character" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def sample(h_prev, C_prev, first_char_idx, sentence_length):\n", " x = xp.zeros((X_size, 1))\n", " x[first_char_idx] = 1\n", "\n", " h = h_prev\n", " C = C_prev\n", "\n", " indexes = []\n", " for t in range(sentence_length):\n", " _, _, _, _, C, _, h, _, p = forward(x, h, C)\n", " assert xp.all(p.ravel() >= 0)\n", " idx = xp.random.choice(xp.arange(X_size), size=(1,), p=p.ravel())[0]\n", " x = xp.zeros((X_size, 1), dtype=xp.float32)\n", " x[idx] = 1\n", " indexes.append(int(idx))\n", "\n", " return indexes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training (Adagrad)\n", "\n", "$$\n", "w = w - \\eta \\frac{dw}{\\sum dw^2_{\\tau}}\n", "$$" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def update_status(inputs, h_prev, C_prev):\n", " #initialized later\n", " global plot_iter, plot_loss\n", " global smooth_loss\n", "\n", " # Get predictions for 200 letters with current model\n", " display.clear_output(wait=True)\n", "\n", " sample_idx = sample(h_prev, C_prev, inputs[0], 200)\n", " txt = ''.join(idx_to_char[idx] for idx in sample_idx)\n", "\n", " # Clear and plot\n", " plt.clf()\n", " plt.plot(plot_iter, plot_loss)\n", " display.display(plt.gcf())\n", "\n", " #Print prediction and loss\n", " print(\"----\\n %s \\n----\" % (txt, ))\n", " print(\"iter %d, loss %f\" % (iteration, smooth_loss))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Memory variables for Adagrad" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "mW_f = xp.zeros_like(W_f, dtype=xp.float32)\n", "mW_i = xp.zeros_like(W_i, dtype=xp.float32)\n", "mW_C = xp.zeros_like(W_C, dtype=xp.float32)\n", "mW_o = xp.zeros_like(W_o, dtype=xp.float32)\n", "mW_y = xp.zeros_like(W_y, dtype=xp.float32)\n", "\n", "mb_f = xp.zeros_like(b_f, dtype=xp.float32)\n", "mb_i = xp.zeros_like(b_i, dtype=xp.float32)\n", "mb_C = xp.zeros_like(b_C, dtype=xp.float32)\n", "mb_o = xp.zeros_like(b_o, dtype=xp.float32)\n", "mb_y = xp.zeros_like(b_y, dtype=xp.float32)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Exponential average of loss\n", "# Initialize to a error of a random model\n", "smooth_loss = (-xp.log(1.0 / X_size) * T_steps).astype(xp.float32)\n", "\n", "iteration, p = 0, 0\n", "\n", "# For the graph\n", "plot_iter = np.zeros((0))\n", "plot_loss = np.zeros((0))" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "scrolled": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8FGX+B/DPN53eEooECCV0qZEuglQBxbPCqYeCx09O\nPe84DuGwoCeK5dTTU++wYgNRUBCUIlJEapBeQi+hht4JSZ7fHzu72TK7szWbmXzerxcvdmdnZ58n\n2Xznme9TRpRSICIi64qJdgGIiCiyGOiJiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMjiGOiJ\niCyOgZ6IyOLiol0AAEhOTlZpaWnRLgYRkamsXbv2hFIqxWi/YhHo09LSkJmZGe1iEBGZiojs92c/\npm6IiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMjiGOiJiCzO9IF+5vpDOHflWrSLQURUbJk6\n0GcdPY8npq7H6K83RrsoRETFlqkD/aXcPADAkXNXolwSIqLiy9SBnoiIjJk60KtoF4CIyARMHejt\nJNoFICIqxiwR6PefvBjtIhARFVuWCPSnL3F4JRGRN6YO9IpJeiIiQ6YO9EREZMzUgV7YC0tEZMgw\n0IvIRyJyXEQ2u21/XES2i8gWEXnFaftYEdklIlki0icShSYiIv/5c8/YTwD8B8Cn9g0i0h3AQAAt\nlVJXRaSqtr0pgEEAmgG4DsBPItJQKZUf7oIDzNETEfnDsEWvlFoK4JTb5hEAJiqlrmr7HNe2DwQw\nVSl1VSm1F8AuAO3CWF4iIgpQsDn6hgBuFJFVIrJERG7QttcEcNBpv2xtmwcRGS4imSKSmZOTE2Qx\niIjISLCBPg5AZQAdAPwdwDSRwLpGlVKTlFIZSqmMlJSUIItBRERGgg302QBmKJvVAAoAJAM4BKCW\n036p2rYIYZKeiMhIsIH+OwDdAUBEGgJIAHACwCwAg0QkUUTqAkgHsDocBSUiouAYjroRkSkAugFI\nFpFsAM8C+AjAR9qQy1wAQ5RSCsAWEZkGYCuAPACPRmrEjVa6yB2aiMgiDAO9Umqwl5fu97L/BAAT\nQikUERGFj6lnxjJHT0RkzOSBnoiIjDDQExFZnKkDPZdAICIyZupAT0RExhjoiYgsztSBnuvRExEZ\nM3WgZ46eiMiYqQM9EREZY6AnIrI4BnoiIoszdaBnip6IyJipAz0RERljoCcisjhTB3oOoyciMmbq\nQM8cPRGRMVMHeiIiMsZAT0RkcaYO9FwCgYjImGGgF5GPROS4diNw99f+JiJKRJKdto0VkV0ikiUi\nfcJdYCIiCow/LfpPAPR13ygitQD0BnDAaVtTAIMANNPe866IxIalpEREFBTDQK+UWgrglM5LbwAY\nDdfBLwMBTFVKXVVK7QWwC0C7cBSUiIiCE1SOXkQGAjiklNrg9lJNAAednmdr24iIKEriAn2DiJQG\n8A/Y0jZBE5HhAIYDQO3atUM5FBER+RBMi74+gLoANojIPgCpAH4TkeoADgGo5bRvqrbNg1JqklIq\nQymVkZKSEkQxiIjIHwEHeqXUJqVUVaVUmlIqDbb0TBul1FEAswAMEpFEEakLIB3A6rCWmIiIAuLP\n8MopAFYAaCQi2SIyzNu+SqktAKYB2ApgLoBHlVL54SqszudF6tBERJZhmKNXSg02eD3N7fkEABNC\nKxYREYWLqWfGEhGRMQZ6IiKLY6AnIrI4BnoiIotjoCcisjhTB3oOriQiMmbqQE9ERMYY6ImILI6B\nnojI4hjoiYgsztSBnkvdEBEZM3WgJyIiYwz0REQWZ+pArziSnojIkKkDPeM8EZExcwd6IiIyxEBP\nRGRxDPRERBZn6kDPFD0RkTFTB3oiIjJmGOhF5CMROS4im522vSoi20Vko4h8KyIVnV4bKyK7RCRL\nRPpEquAAZ8YSEfnDnxb9JwD6um1bAKC5UqoFgB0AxgKAiDQFMAhAM+0974pIbNhK64bj6ImIjBkG\neqXUUgCn3LbNV0rlaU9XAkjVHg8EMFUpdVUptRfALgDtwlheIiIKUDhy9EMB/Kg9rgngoNNr2do2\nDyIyXEQyRSQzJycnDMWInGPnriBtzBysO3A62kUhIgpYSIFeRMYByAPwRaDvVUpNUkplKKUyUlJS\ngvr8osrR/7rrBADg0xX7i+YDiYjCKC7YN4rIgwAGAOihlCPkHgJQy2m3VG1bRDBDT0RkLKgWvYj0\nBTAawG1KqUtOL80CMEhEEkWkLoB0AKtDLyYREQXLsEUvIlMAdAOQLCLZAJ6FbZRNIoAFIgIAK5VS\njyiltojINABbYUvpPKqUyo9U4SVSByYishDDQK+UGqyz+UMf+08AMCGUQvmLqRsiImOmnhmrOGOK\niMiQqQM9EREZM3WgZ3ueiMiYqQM9Iz0RkTFzB3oiIjLEQE9EZHEM9EREFmfqQM9liomIjJk60BMR\nkTFTB3rOlyIiMsZAT0RkcaYO9EREZMzUgZ4NeiIiY6YO9EREZIyBnojI4hjoiYgsztSBnuvRExEZ\nM3WgJyIiY6YO9M7t+e1Hz+FqXsRuT0tEZFqGgV5EPhKR4yKy2WlbZRFZICI7tf8rOb02VkR2iUiW\niPSJVMEB1wlTfd/8BeO+3ex9ZyKiEsqfFv0nAPq6bRsDYKFSKh3AQu05RKQpgEEAmmnveVdEYsNW\nWgNr958GAMxcfwgHTl4qqo8lIirWDAO9UmopgFNumwcCmKw9ngzgdqftU5VSV5VSewHsAtAuTGX1\nUDrB9Rxi75x9Yup69H/7l0h9LBGRqQSbo6+mlDqiPT4KoJr2uCaAg077ZWvbIiK9Wlmvr52/khep\njyUiMpWQO2OVrRkd8DhHERkuIpkikpmTkxNqMWxlAbDj2PmwHIuIyCqCDfTHRKQGAGj/H9e2HwJQ\ny2m/VG2bB6XUJKVUhlIqIyUlJchieOr9xtKwHYuIyAqCDfSzAAzRHg8BMNNp+yARSRSRugDSAawO\nrYj+i/T8KU7QIiIz8md45RQAKwA0EpFsERkGYCKAXiKyE0BP7TmUUlsATAOwFcBcAI8qpSI2uN09\n7hboBOLNh84iv4ABmohKrjijHZRSg7281MPL/hMATAilUMHKPn3Z5fnWw+cw4O1leKx7A4zq0yjk\n44tIyMcgIipqpp4Za+TY+SsAgE2Hzka5JERE0WPqQF8uyfCChIioxDN5oI8PaP9j567gs5X7I1Qa\nsrKCAoW0MXPw3yW7o10UooCZOtAHatjkNXj6u804cvay8c5ETvK1jv7X5mVFuSREgbN0oH/o4zUu\nz09dyAUAjsKhoPGbQ2Zk6UDvDUfPUKD4jSEzK5GBnihYnDRHZsRAT0RkcSUq0LMtRkQlUYkK9CXR\nz9uP4XIub7FIVJIx0FtY1tHzGPpJJsZ9tynaRbEMXhWSGZk+0I/r1yTg95SUERTnr1wDAOznbRVD\nxgBPZmb6QF8mkcsgeMPgRESABQJ9TADNc18j4y7l5jlawEREVmKBQB94IkbvLZ0m/ozrx88PQ4mK\nj5KSoipKHEZPZmT6QB+uSa5nLlmvNc+YFD4M8GRmFgj0gUf6w2euYNWekwGnasw6K5Ite6KSzfQ9\nmYHk6I+es92I5M73lgMAbmqYgslD20WiWERExYbpW/T+5OiX7MjBhat5Htu3Hz0X0GfZTxRERGZi\n+kC/cs9Jv/bbonM7QTFIaizafhxpY+bgpLa88co9pwIvYBSZNNNERGEWUqAXkb+KyBYR2SwiU0Qk\nSUQqi8gCEdmp/V8pXIXVc+BU8JOBjp67gk9+3ev19Y+017Yc1r/n7J6cCyiw2Nr2B05eQs75q2E5\nVm5eAS7qXEmZkWLXNrkZ+dV6/LDpSLSL4ZegA72I1ATwZwAZSqnmAGIBDAIwBsBCpVQ6gIXa84gJ\nZnils/HfbzXcR6/Dd8vhs7j5X0sw6Zc9AX2eUgqbi+hm5cH8aLq+ugg3TPgpLJ9/z/9WoNmz88Jy\nLG/yCxTOXrbeiKmisvnQWWw4eCbaxTClGesO4U9f/BbtYvgl1NRNHIBSIhIHoDSAwwAGApisvT4Z\nwO0hfoZP4RhemZdfEPB7Dp6y3Y7wt/2nA3rfrA2HMeDtZZiz8QiUUliUdTxiVwXRTt2sL4IAMvHH\nbWj53HxOdgvSgLeXYeA7v0a7GBRhQQd6pdQhAK8BOADgCICzSqn5AKoppezXM0cBVNN7v4gMF5FM\nEcnMyckJthh+u3fSSq+vzd1yVHe7PVCGc3jiruMXAAC7cy5g3pajeOjjNfhwmff0UaRtO3IOn63Y\nF7XPD9Xsjbav2rkr1kgREUVCKKmbSrC13usCuA5AGRG533kfZRt4rtuuVEpNUkplKKUyUlJSgi1G\nWFqti7P0TzSRzssePWsbxZN9OnqLjt3y71/w9MwtUfv8UBXVHIFoXx0RhSKU1E1PAHuVUjlKqWsA\nZgDoBOCYiNQAAO3/46EXM7K8pU7sf9znDNICpy/mhpR+iXQMKQm3yDXrZDbyz8FTl/DAh6t0h0mT\nsVAC/QEAHUSktNh6K3sA2AZgFoAh2j5DAMwMrYi+5RUEnl/34BYI8wuUS+D4aVvhuWrsjE1IH/eD\n4/n8rcfQ+p8L0O+tX5BvEOzz8gtwLd/7PsfOXfE6SuWHTUfwvyW7fR7fXUkIfrzRe8nw2vws/LLz\nBH7aeizaRTGlUHL0qwB8A+A3AJu0Y00CMBFALxHZCVurf2IYyulVaqXSIR/jrNM6NycuXEX9f/yA\nT5bv071cn7L6gG6w3n70PD72MVQTsKVJ/usjWLd/cSHueHe57mt/+uI3vPTjdp/Ht5u35Sg6vrQQ\neRYb+ulLCTinEQUtpFE3SqlnlVKNlVLNlVIPKKWuKqVOKqV6KKXSlVI9lVIRnWV0X/vaIR9j4fbC\nFnv2adtomglztmGFn5Ox7HbnXPD5+s7jrq9fzfO8Gsk6dj6gz7TbdfwCTl6wjX9/+rvNOHL2Ck5e\nzPX7/Veued5ucPOhs45jUmC+WnMA244ENvOajHE+Q3BMPzM2Uo3WYFrDU1YfdNyfdcex81i++4TX\nfV9fsMPRQg9Ha7Tn60tw87+W6L62Zp/xENDGT8/12Dbg7WW47T++h95tOHgGI79ab7mJY6F6cvom\n3PLvX6JdDCIAFgj0bWpXDOvxQs1rX8y15dh7v7EUv39/VTiK5OL1+VleX4vExKFDZ2xXOHtyLmDt\nfs+Ls2GT12DGukMBXT2Q/y5ezcNLP27D1Tze4B0wXraE9Jk+0Be3zrhIl+atn3f5fP1ybj6Oa0sY\nhLMsN/9rCe58b4Xf+xdFCz/r6HnHiai45+gnzNmqe6I08s6iXfjfkj34ctWBCJQqfLJPX8LnK/dH\n/HOYugmO6QN9uP3OS2eov8Jx4kkbMwc7gszVN3nGMwUTKm9XORPmbMUJbcE39z/Au/4b+M/x7OVr\nhiOXnPV5c2nAn+GPvPwCxxwHABj19YaQ1zR5/5e9AZ0o7XK1fpw8H6O1ioPB76/EU99tjtgM5eLV\nnDMfBvowm7n+kMvzRduP4+ftx7DvxMWAjuPvqpy+hCs0eIu97//ifZTRbwcCW/7gyrV8tHxuPp77\nPvqTt56fvRUdXlqIM5dsJ7Fv1mZj5LQNEfmsggKF5bu89+WEw6wNhzH6m8iU3+7MRVuA9/ad23ns\nPD4IcF0oZ8X7NFf8MdCH2XNui6Q99MkaDP0kE91eW+z1Pb/uPoFF2wOfV/bMzM3o/YZ+BywQvlZQ\nUYzHt3diz9pwOKj3KyiXEULHz13B8E8z/Zpg897i3Xh3cWFK7Gftd3HucuQn53z06178/oNVXseH\nh+Mn/+cp6zAtMzsMR9KXm1eA8wY/5wFvL8MLc7Zh5Z6TSBszx5Fy88el3DzMXB/c9yKc8vILHFdY\nZmOJQP9gp7RoFyEke3IuYsQXa122+QrS9pm6n67Yjx3HvA/pDFf3xSfL97k8X7H7JEZOW++yzdsy\nEr7szrmAGb+FJwDN+O0Q2r7wk2MlxtcX7MD8rcfwvR8njpfnbscrcws7ue0nh3Dlg32dKPdoV3pH\n/Lipzfkr1wLulC2Kk/R7i40n8tmHEk9ZbetrWLPXd39F9ulLjvTZZysin/v3x+/eXY6GT/0Y1HuV\nUlGdwGiJQB/qUsXh9uQ3GwN+z5Vr/rcU/J0hq/e9WrIjx7He/FkfN0T/ak1h598Lc7Y5Hn+6Yh8G\nv78SM35zTVGN/majz3Xs7/7vcjQc9yP2OM016Pn6EoyctgEz1x8KeXLXit22VFfWUde+jeW7A0+B\n2W8UH66/S3+OY/8GK6XwwuytjlSf870Qrh8/H3f/Vz/Pf/byNaSNmYP3l7qmRxYUwUzS05cCH3Fl\ndBLt8vIidHhpIQAgv4gC5Nr9p3wG403a8uJ7A0zDArYrmrpjf8DZy9fw3bpDxm8IM0sE+p5Nqhb5\nZ7750w6vr32VeTAsn3HqYi6u6Syh/HVmtstCaHqTnQDPFv3pi7kY8tFqx3rz7q1yZ09O36S7/Rkf\nC6DN2ei99bxm32nk5hc4xvrPXH/IEQCfmLre7/ytUgqLs4773WnrT4ve62cFuP9nK/ZhzHTPk3wg\nx9l1/AI+WLbXkeqz39XssvY73pitfy+DnPO21u8U7QQ96usNSBszJ6ggrCc3r8BwQmCkFEWcn7/l\nKO58bwW+8GN0U3cfaVhvthy2TZ67/Z1f8Zev1ns0SCLNEoG+U4PkIv/M7RH+RRUooM0/F2D0Nxs9\nLtePn7+KLi8vcjzXm+wEAC86tcSnZR7EFbfjHDkb3nvgBvL3+MRU15OMfUjoGR9XGe8u3oWMF37C\ngx+v8bqURDiH3wXagfn0zC2YusbzJK/XSjxzKVd31VJvpTcOdq5n9W/W2lJi9vsmhGrct5vQ419L\ncCqI+RLhGFgQafY71e3JCby1Hgj71YC3xlmkWCLQW5G9xfr9hsNBt2gOOwXy0d9s9DiO+2EjNTRu\n4bbA0gfuefuc81dxKTcPr8zNckzMOuh2C8nV+2wt35e1XLvz1cz6g2ew63jgJ2Z/ZhTr+drtik7v\n19fttcXo8vIij99JIEnIn7Yaj+aavGJfAEfU9/iUdfhaO3Fc8GPd/7z8ApeT26KswoEGgX6XjVaO\nJf9YJtC/eW+raBchrKZrwS6cC5O5H8m9pXn9+Plh+yxnwyZnujz/VWc4oXOAGznNlnZ4a+FOAMAN\nE37CQLelGPRazgB0W5y3v/Mrer7uOea+4bgfA14R1B9/d+uj0QtuhVcutheNupk2ZHsOV334U53R\nXEF+XZRSyC9QuqunBpr+ajDuRzz13ebgCuKmxfj5OHCy8KTu7UTx7bpspI2ZE/Ds8PwCVeSt62iw\nTKAvZv2xIbPn9IDCy/BQRbrX/1Juvl9XBfd94Lk0hHN97SY7jfZxXxAuHHLzC/xeEdSXq3n5juGh\nepzTSfY+Bnfjvt3scxG0nw2G39q//8H+hg+cuoSnZ25Gs2fnBTRpzZ39K/bFqgMuk87cXblW4Pfs\n6X0n9a9ajpy9jNfnZ0EphUlLbXM63K/0vPl2XTY6vrQQf/pirUvq88Cp0FM3D3y4Cn/4aHXIxwkn\nywT64jbyJpzC1TpyjvO2NffDcliHV+dl4frx83HMj6GC7o6fD29/QVHOpezzxlKfM5Kdf85fZ2bj\nwY/X6O7nawlrd5dyC1vewdzz2J1ShSknfwO9UgppY+Z4DL+1s4+acbZWu8fy2Bmb8NTM0L7Xj3+5\nDm/9vAtbDp9zNGL8DQNjpm/CkbNXMG+La1rR+d4Tgcg6et4xdPSXnSewdEeOz4ZVUQ+0tEyg79ow\nBTUqJEW7GKZx/PyViK0b0v+twFdt1Lvna1H/MZzWSfv4sxTFvpPeW5HnrlxzSb9lu00UmrK6MAUl\n8D9QNX1mnuPx3f8rHHK598RFrHLq/Dzv9HO9lJvnNU2hULhgmK/vhYLC5OX7MHbGJt0Tgm75nXZz\nnig1VQuM2acv4ZW5273f6c39uVIY/c0GZGonDed4GuyiZ+5Lhuecv4q0MXNcfpa+9HlzKcbO2IRp\nTinFb9cdwpVr+Y4Z1tFkmUBfoVQ8VoztEe1iFGt/+7pwFMl9H6zyOdkqFPb1bwKhFzTOXb7m9x+a\nP75acwBnLuV6Xbdm8PsrPS79e7/hmdt/fMo6DPHz0rzF+Plo/uw84x0BfBfk7M91B864hLd7J63U\n3a/pM/NcRms5u3g1z3ERZHSl9+ysLZiy+oDuiqXBXCV2eXkR3l28G8u8LQXhdszL1/JdZvo6n1yC\nvbB/dZ7rqrCZWuf+x7/uC+g4o52G146ctgGNn56LVs8vCK5QYRQX7QJEkkjxX9WwKK12mo0Y6WFk\n4ZBXoLwGrWA8OX0T1h88gymrD+LHJ270eH370fN+debZOycLChSGf5ZpsHdg3vxpp+Oxt3SW3to4\n/i6md8LLjWT+PHWd06Stwu2/HfA+8qj9i56pmdk+5lK4cy9zKLcFtZfZfkj7bN0R3eoDAD5fuR/l\nkuIwsFVN/47n9nyjTme4mVimRW83fURHx+Mm1ctHsSQULScvXHXkS93ZUyXebgqi11HqzZQ1B4LO\n6Xoze2Ph1catby/T3efrEDvnF247ht05F5A2Zo5j256ci04durYwV1CgPG5teclHpzNg61R29v7S\nPX7n/Id+kokvVu13mQ3sXB47vfSMfZ8Jc7Zh/8mLeHnudrw819bRvvXwOTz13WaPuRv+sP9MfN2A\nJ5i5BUXNcoG+bZ3K6FCvMgDg3htqRbk0FA17gpiibvfafO8znt25B7VwO3ZOv/Ud6uipYZMz8foC\nz3raA+gzM7dg/cEzeGeR570P3JdYMDLhh20BnZjGfbsZ/d9yPcE5z2cYOW2Dbsvf/iP5ZecJ3PTq\nYpfXbv1P4fHOXr6mO4Q0FG3+Gf3UjJGQUjciUhHABwCaw3a1MxRAFoCvAKQB2AfgHqVUcDNPQpRe\nrSz2Tezv0nIh64vGWiL+cl/GOlz8HVZop9dBaG+9frM2G99vOIyWqZ53b5sRxM/WWzos2HFR7yzy\nHJ3k69TnfEXR8rn5KJ/kO+yt2nPS59BQMwq1Rf9vAHOVUo0BtASwDcAYAAuVUukAFmrPi1RalTIA\ngPJJ8QCAzg2qFHURKIr8Wa8kWvb7GKHjL71O20DHbeulP5y3iAAnLhbPG8O7n6RsfXH+X+XojfBy\ndu+klXh+9laf+4Rq06GzLgv8RVrQgV5EKgDoCuBDAFBK5SqlzgAYCGCytttkALeHWshAjb+tGT4c\nkoHmNSsAAF65q2VRF4HIdC465d8FUmw77N37nX/ddQK7i2lZvXn6u82OBf6KQigt+roAcgB8LCLr\nROQDESkDoJpSyt6jdBRAtVALGaik+Fj0aFL4sddVSMKdbVKLuhhEVARe/MH77ObGT+uvH+9v+//H\nzUd1t4dr2YSiupFJKIE+DkAbAO8ppVoDuAi3NI2yXU/p/kxFZLiIZIpIZk5O4DetCISIYMLvmkf0\nM4jMxGgm8uUiWv9lzsbA78UbyBINgdznIRDeVowNlF6HdyRIsD34IlIdwEqlVJr2/EbYAn0DAN2U\nUkdEpAaAxUqpRr6OlZGRoTIzwzse2V1efgEajAvu7jBEZC2JcTEes2Gj5cMhGS4ZiECIyFqlVIbR\nfkG36JVSRwEcFBF7EO8BYCuAWQCGaNuGAJgZ7GeEU2yMddfCIaLAFJcgDwBv+LiJUbiEOjP2cQBf\niEgCgD0AHoLt5DFNRIYB2A/gnhA/Iyz8nTlIRFSUNh/yvmppuIQU6JVS6wHoXTZw0RkiomLCcjNj\nfVny924uz2c/3iU6BSEiKkIlKtDXqVIGpeJjHc9FgKf6N4liiYiIIq9EBXrAdbKFUsADHetErzBE\nREWgxAV6d8HeqICIyCxKdKAXsd69ZomI3JXoQA8AsYz0RGRxJT7Qx8QI9k3sH+1iEBFFTIkL9C5L\nsTI/T0QlQIkL9H2aVQ9o/yY1yiM+1nZCMLphARFRcVTiAv3Ld7XAm/e2Qq3KpVAvpYxj+/8eaKu7\nv/NNpGc+5jrB6tHu9SNTSCKiMCpxgT4+Nga3t66JX0bfjCSnyVPOLf0143rqvtd9XbSKpRL8+sxX\n7mqBoZ3rBl5YIqIwKHGB3h8p5RJ1t7vn9NvUqeTzODUrlgIA3JNRC/k6NzQmIioKTDo7+eLh9sjN\ntwXkH5+40ZGbt7OPxKxaLhFLR3dHUnws6iaXwd4Tnrcxe6x7A/yhYx3s0u4LGakbIETTsC518eGy\nvdEuBhEZYIveSecGyejeqCoAWydsg6rlABSmdaqUtaVq/tqroSPt88yApgCATvWroK+233v3tcGo\nPo1QtXwSOtVPBgBcKqI79hSlHk2qRrsIROQHBno/vH5PK6z6Rw+UTojDvon9MbhdbcdrLWtVBAA8\nfKPvHPyfutVHWpXSLu8lIioKDPR+SIiLQbXySbqvVS6TgH0T++Pmxr5vBdakRnks/nt3/HNgs6DK\nYM/3FydxMfz6EJkB/1LDyH5D5YQ47z/WuNgYvHtfm4CPfWfb1KDLFSlcPYLIHBjow2h413oonRCL\njDqVfe7X7/oaAR/73htqBbR/zzDmzxtULRu2YxFR0WOgD6PODZKx9fm+qFA6PizHcx7PX7NiKXw2\nrJ3j+RcPt3c8rq6bVpKwzeQd0ML/E1MwVyvh4HxDmXDwNsSWyIwY6IuJ7o1SAAC1K5c23Ld+Shl0\nbpDsmLVbJtEzyHlLq/Rs4tqX8K+7Wxp+nlL62/U+olSCZ1kSfaSywmHPi/2w9fk+hvuVC+DEN+fP\nvM0kWUfIf4EiEisi60Rktva8sogsEJGd2v++ZxURAODjh9ph+ohO+PZPnfDFw+0x4XfNHUHT3hFr\nD7g1Ktiely9lu3Kom+w7tTKw1XWOxx8Mcb2XezC5/4TYGAxuVwuta/v3q133TC+M6t3QcL+bG1f1\nuZLo7U71cBYTIxA/Ogzi3Kc2e5EQF+P3rGerua6C/qADMrdwNLWeALDN6fkYAAuVUukAFmrPyYtX\n72qB17RWdds6lVClbCI6N0jGfe3roGxiHBaP6oafR93k8h57TKtZsRQ+fugGvHFvS7x5bytULuMa\nnDrWrwJzf+YkAAAPeUlEQVTA8yphcLta+L+u9fDz32zHva2lfgAFbC1b9wb9jgm34KU7WiDWj8CZ\nEBeD0glxePjGeniwU5pj+196pnvs+/o9vq8u9IamDuvi/9ISBV6uTNzteOEWnx3qzj50O3FGyh8N\nhu+Gy9AAfp5kHiEFehFJBdAfwAdOmwcCmKw9ngzg9lA+w+ruzqiFu3y0qtOSyyAxznv+uXujqiiX\nFI/bW9fEsie7O7YLgH8Pao0Ff+2KCqVc+wxeuqMFxvZrgnopnlcCbw1u7Xj89SMd0ey6Co5LiTpV\nSuPJvo39rZqLpPhYjO1X+N6/9GyIzKdc1xSqWNp7K3rfxP6opZPWelqbsOYP5ZaDapfm2Wk+7y9d\nPba5z5Cu6NQHU1Qjj8om2j5zeNd62P1iP+yacIvuft8/1gUbx/cO+nP8veohcwm1Rf8mgNEAnOf3\nV1NKHdEeHwWgO8BcRIaLSKaIZObk5IRYDHJXv2pZJMXHIr1aOcMboNuD1Zv3tnJp3d+gBUJ7eLyj\ndSpGdPO9YucNaZWRXrUspo/o6PGarxOWs3VP98LkobaO54c6p2Hmo539ep+31nVy2UTExwrKJrrm\n6Kc94lnGRtXLeWx79z7XlU1Xju3h6OhOjIvF58Pae7wnUhLjYhAbI4iLjcE7v3ft+I6LEVyfWgHl\nk4IfDJDg5+/ISh7qnBbtIkRc0IFeRAYAOK6UWuttH2VrQuleMCulJimlMpRSGSkpKcEWw7Tm/uXG\nsHf4lYqPxV1tU/G3Xg0xsldhTtwowN7awhbc7bN83dlb0rUqe5+0Nbhdbfxf13oomxiHBSNvsl0J\nGPDWdqxUJgE3NUzBnD93wdP9mzrKZdR67tGkGhrrBOrHutfHzgn9EBsbXGu1V9PCtsrnw9ojKT4W\n9bUhp0nxseiSnhzUcf3l3EJ3rkF/t9FQf+1l3A9i5M62NUM+Rri4X4lGSmUfV5KRdGebopsbE8r4\nu84AbhORfgCSAJQXkc8BHBORGkqpIyJSA8DxcBTUahpXLx/we4xSzCLiyPcHomfTai6doOWT4nDu\nSp7j+d1tU5FaqRQ61qvi8d6RvRoiNkbwaPcGARfauQN11T96eLzufrLwNvrH2Vwt9ZI2Zg4A27pD\n9rWKfC0g2qhaOb9+dvagfmebVKw7cAZ1k8v43P/BTmn4ZPk+44LrKJ0Qi0u5+SifFA9l8Nt378SO\njxVcy/ezU0ITFyNIjItFzYqlcOjM5YDLGw7JZRNw4kIuAGBcvyYYPX1jwMeoUCoeZy9fC3fR/FKr\ncikcPOXfz64oJxwG3aJXSo1VSqUqpdIADALws1LqfgCzAAzRdhsCYGbIpSQAQBWtszW9qmerNZwW\njLzJJfUiIuhUP1l3ZMufe6R7D/I6Zj7aGYtGdbMd12m7tyUmQnXL9TUQo+Wd83Qi/VfDO6BFagV8\nPaIjrk81vgqxu79DHex9qZ9HB7i7GjqjWAa0qIHFo7qhTe3CKyj3voatz/fBr0/ejF/H3AygsHVb\n3q2VW6VMAh65yTOdtvk54+Gm7uynhS4NPK9QXrmrRUDHMlrqIyG2MPTsfrGf4/HXj3RyPI6PCy4S\nfvV/HQLaP5wB19vf5t6X+nlsc+/7iaRIDHCeCKCXiOwE0FN7TmHQvGYFTB3eAWNuCbxDdO1TPbF6\nnGerWU+18kloazC7N1gta1V0tIJD/QNrGUBgBoB8nWE37etVwazHugSV1/ZnSKfeqKAneqQjLbmM\nox8C8BzWWDohDpXKJDiG1j7QoQ7+eXtzl5FLALD26V6634fEuFjsfakfRnSrj9mPB5Yi1KvWPRm1\n8PjNthN6p/qFV3aj+zby2LdUfCy6NkzR3qefnnC+QomNEWwc3xurx/VA3eQymP14F1xXIQldGgSX\n0m1cvTz+3qewXGlVjOemhMNfe+qnzpLiYzy+K18+XHT9OkCYAr1SarFSaoD2+KRSqodSKl0p1VMp\ndSocn0E2HepV8Xvon7MqZRNRtVzxGiMdTD2cud/a0dmiUd1cZg8DCDiVEarB7WojLta1ji/dcT3S\nq9lafeWS4rF8zM149tamqG4wfj0uNgYPdKjjcTxfRARP9m2M5jX9OyH21voivKXI/ta7ESbecT3e\nu78t6iWXwZN9G2No57oY0a0+bkgrnFPRsHo51KlSBnte7IdX7vJMh7WtUwkNtZ+BfcJe+aR4x/ez\nec0KWD62B5LLJnic2LzxtV/H+p4pR2fOQdh56ZCsF/r69dkA0LdZdTzRM91jZJc3nRok44831gMA\n9A9iSZRAcWYsRU3phDhMH9ExpOGA3tRNts0edpaXH7mbv7xyVwt8+XB7l45bPT0au65BdF3FUnio\nc120TK2IR26qj5fuuB5v3tsqImV0XwF12ZPdHSfbxaO64c1Bts+1t7b1UjiD2tVGhVLx+HlUN4zo\nVh9J8bF4sm9jR4d/i9QKjmUw7Ckze99OsnY/h8S4GEz43fVoXbuiR4eyMxHB+Nv0U0B/vLGuS3rx\nbi9XDrc0r+71GHomPVA4cisxLhZtalfEf37f2sc7XDlfNNrvSe0t9tdLKYsNz/Z2GdIcKbzDFEWE\nv2mZQFJE1con4cb0ZIzoVh+tvIwQ8mVg65r4ctWBgN8HANNHdESMj0rdk2FbdK5Tg2R8vnI/nvpu\ns8c+vmb9xsRIUCk5f/00siuSyyai1fMLHNtSK5V29JNUr5DkCNa3trwO0zKzMf62Zuj5+pKAPufv\nfRp5nFAmD22HK3n52HjwLO7/cBVEgFa1KuLbP/k3bNbdp0PbOVJDdp4d97boWi+ljEt/gJ4HO6Xh\n1XlZAGy/h43jeyNPu/qboZXxsS/X+TyG/avhHNPtP1tfjfyiGlnEFj1FREJsDNrWqYS3A2gNGYmN\nEXw2rD061U9G6YTA2yj/HNgcG8f39po39qVtncp+L/lQHDWoWk53Qlp6NdswUedz2I3pKdg3sX9A\nq5b6GhWUEBeD8knxKF/K9jurXdn3SCUj7kFetzxacQT6y2M8e6ut87t93coo4za/onxSvEcn+1P9\nm+C525oZ5vv/2jPdMelsQIvrUKtyKbx81/Uu+6wc619fWTixRU8RISKYPqKT8Y5FKDZGUD4pHq/c\n1VI3d+zu60c6Irls4KtYmmmd/k+HtsfmQ2f9nszmjXNg9aZFakV88IeMkOYd3Nc+uDu0tUurjNX7\nCrsL7RPj7L+r1EqlkH3a+7DIh7V8+se/2u6RvGhUN9SsWAoNn/oRQGHqqHXtStj1Yj+cuZSL8knx\nLnMbWteuiHZ1Kxv2x0QCAz2RFzfoLJFgdrMf74IBby9zPK9cJsFnC3lcvyZ4dX6W4XH7NKuO5btP\noo5Bi7enQR+Gng71KqNJjfJ49lbPXPuXD7dHvh8doE2vK+8I9Hc7LTlif+vsx7s4xu/7Ukq7khTY\nrlRWjL0Zl3LzUd9tORG9q6dgU1XhwEBPFGb22bkZdYpfqsffETh2f+xaD3/sWs9wvz90rIM726Z6\nLDMRDlOHey5VYdfJqcP4H/0a48Nltha3t9D/zICmGNqlLpbvPuGyvWLpBJ9rLdm9/4e2mLn+sOOE\nZl9JtrhjoCcKs7Z1KmPl2B4ul+jF7UYmwaSkfBHxXEuoqA3vWh/Du7pOHnNPoxk9N5JaqXRAEwSL\nCwZ6oghwDvK/Pd0r5DkD4fT9Y12ikicuSvHaSBv7DeztN7+xb7f/H0ynvhmVjFoSRZHRMglFLZCl\nHszqoc5pOHMpF8O1tNOfe6QjNkYcw2Az6lTCqN4Nde9xYEXi70yuSMrIyFCZmZnRLgYRkamIyFql\nlOHdb4rP9SQREUUEAz0RkcUx0BMRWRwDPRGRxTHQExFZHAM9EZHFMdATEVkcAz0RkcUViwlTIpID\nYH8Ih0gGcMJwr+LPKvUAWJfiyCr1AFgXuzpKKcMF+otFoA+ViGT6MzusuLNKPQDWpTiySj0A1iVQ\nTN0QEVkcAz0RkcVZJdBPinYBwsQq9QBYl+LIKvUAWJeAWCJHT0RE3lmlRU9ERF6YOtCLSF8RyRKR\nXSIyJtrl0SMiH4nIcRHZ7LStsogsEJGd2v+VnF4bq9UnS0T6OG1vKyKbtNfeEgn0Jmgh16OWiCwS\nka0iskVEnjBxXZJEZLWIbNDq8pxZ66KVIVZE1onIbJPXY59WhvUikmnyulQUkW9EZLuIbBORjlGt\ni1LKlP8AxALYDaAegAQAGwA0jXa5dMrZFUAbAJudtr0CYIz2eAyAl7XHTbV6JAKoq9UvVnttNYAO\nsN2A/kcAtxRxPWoAaKM9Lgdgh1ZeM9ZFAJTVHscDWKWVx3R10cowEsCXAGab9fullWEfgGS3bWat\ny2QAD2uPEwBUjGZdirTyYf5BdgQwz+n5WABjo10uL2VNg2ugzwJQQ3tcA0CWXh0AzNPqWQPAdqft\ngwH8L8p1mgmgl9nrAqA0gN8AtDdjXQCkAlgI4GYUBnrT1UP73H3wDPSmqwuACgD2QusDLQ51MXPq\npiaAg07Ps7VtZlBNKXVEe3wUQDXtsbc61dQeu2+PChFJA9AatpawKeuipTvWAzgOYIFSyqx1eRPA\naAAFTtvMWA8AUAB+EpG1IjJc22bGutQFkAPgYy2l9oGIlEEU62LmQG8JynaqNs3QJxEpC2A6gL8o\npc45v2amuiil8pVSrWBrEbcTkeZurxf7uojIAADHlVJrve1jhno46aL9Tm4B8KiIdHV+0UR1iYMt\nXfueUqo1gIuwpWocirouZg70hwDUcnqeqm0zg2MiUgMAtP+Pa9u91emQ9th9e5ESkXjYgvwXSqkZ\n2mZT1sVOKXUGwCIAfWG+unQGcJuI7AMwFcDNIvI5zFcPAIBS6pD2/3EA3wJoB3PWJRtAtnaVCADf\nwBb4o1YXMwf6NQDSRaSuiCQAGARgVpTL5K9ZAIZoj4fAlu+2bx8kIokiUhdAOoDV2uXeORHpoPW6\n/8HpPUVC+9wPAWxTSr3u9JIZ65IiIhW1x6Vg62vYDpPVRSk1VimVqpRKg+37/7NS6n6z1QMARKSM\niJSzPwbQG8BmmLAuSqmjAA6KSCNtUw8AWxHNuhR1h0uYOz36wTb6YzeAcdEuj5cyTgFwBMA12M70\nwwBUga0DbSeAnwBUdtp/nFafLDj1sAPIgO2LvxvAf+DW0VME9egC26XmRgDrtX/9TFqXFgDWaXXZ\nDOAZbbvp6uJUjm4o7Iw1XT1gGz23Qfu3xf73bMa6aGVoBSBT+459B6BSNOvCmbFERBZn5tQNERH5\ngYGeiMjiGOiJiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMji/h9rDZqjFxSujQAAAABJRU5E\nrkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "----\n", " outhant her abnuck a casthantes Pandors,\n", "Are you corpore; surt is. in\n", "Is boo all you\n", "\n", "KLUS:\n", "Goke to knowt but, he have be pedaen lost'd Cold led. Ravert was, soperen, but brous.\n", "\n", "KIN:\n", "-flak\n", "Thym;,\n", "Wit \n", "----\n", "iter 6000, loss 49.692995\n", "CPU times: user 12min 4s, sys: 15.6 s, total: 12min 19s\n", "Wall time: 12min 21s\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8FGX+B/DPN53eEooECCV0qZEuglQBxbPCqYeCx09O\nPe84DuGwoCeK5dTTU++wYgNRUBCUIlJEapBeQi+hht4JSZ7fHzu72TK7szWbmXzerxcvdmdnZ58n\n2Xznme9TRpRSICIi64qJdgGIiCiyGOiJiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMjiGOiJ\niCyOgZ6IyOLiol0AAEhOTlZpaWnRLgYRkamsXbv2hFIqxWi/YhHo09LSkJmZGe1iEBGZiojs92c/\npm6IiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMjiGOiJiCzO9IF+5vpDOHflWrSLQURUbJk6\n0GcdPY8npq7H6K83RrsoRETFlqkD/aXcPADAkXNXolwSIqLiy9SBnoiIjJk60KtoF4CIyARMHejt\nJNoFICIqxiwR6PefvBjtIhARFVuWCPSnL3F4JRGRN6YO9IpJeiIiQ6YO9EREZMzUgV7YC0tEZMgw\n0IvIRyJyXEQ2u21/XES2i8gWEXnFaftYEdklIlki0icShSYiIv/5c8/YTwD8B8Cn9g0i0h3AQAAt\nlVJXRaSqtr0pgEEAmgG4DsBPItJQKZUf7oIDzNETEfnDsEWvlFoK4JTb5hEAJiqlrmr7HNe2DwQw\nVSl1VSm1F8AuAO3CWF4iIgpQsDn6hgBuFJFVIrJERG7QttcEcNBpv2xtmwcRGS4imSKSmZOTE2Qx\niIjISLCBPg5AZQAdAPwdwDSRwLpGlVKTlFIZSqmMlJSUIItBRERGgg302QBmKJvVAAoAJAM4BKCW\n036p2rYIYZKeiMhIsIH+OwDdAUBEGgJIAHACwCwAg0QkUUTqAkgHsDocBSUiouAYjroRkSkAugFI\nFpFsAM8C+AjAR9qQy1wAQ5RSCsAWEZkGYCuAPACPRmrEjVa6yB2aiMgiDAO9Umqwl5fu97L/BAAT\nQikUERGFj6lnxjJHT0RkzOSBnoiIjDDQExFZnKkDPZdAICIyZupAT0RExhjoiYgsztSBnuvRExEZ\nM3WgZ46eiMiYqQM9EREZY6AnIrI4BnoiIoszdaBnip6IyJipAz0RERljoCcisjhTB3oOoyciMmbq\nQM8cPRGRMVMHeiIiMsZAT0RkcaYO9FwCgYjImGGgF5GPROS4diNw99f+JiJKRJKdto0VkV0ikiUi\nfcJdYCIiCow/LfpPAPR13ygitQD0BnDAaVtTAIMANNPe866IxIalpEREFBTDQK+UWgrglM5LbwAY\nDdfBLwMBTFVKXVVK7QWwC0C7cBSUiIiCE1SOXkQGAjiklNrg9lJNAAednmdr24iIKEriAn2DiJQG\n8A/Y0jZBE5HhAIYDQO3atUM5FBER+RBMi74+gLoANojIPgCpAH4TkeoADgGo5bRvqrbNg1JqklIq\nQymVkZKSEkQxiIjIHwEHeqXUJqVUVaVUmlIqDbb0TBul1FEAswAMEpFEEakLIB3A6rCWmIiIAuLP\n8MopAFYAaCQi2SIyzNu+SqktAKYB2ApgLoBHlVL54SqszudF6tBERJZhmKNXSg02eD3N7fkEABNC\nKxYREYWLqWfGEhGRMQZ6IiKLY6AnIrI4BnoiIotjoCcisjhTB3oOriQiMmbqQE9ERMYY6ImILI6B\nnojI4hjoiYgsztSBnkvdEBEZM3WgJyIiYwz0REQWZ+pArziSnojIkKkDPeM8EZExcwd6IiIyxEBP\nRGRxDPRERBZn6kDPFD0RkTFTB3oiIjJmGOhF5CMROS4im522vSoi20Vko4h8KyIVnV4bKyK7RCRL\nRPpEquAAZ8YSEfnDnxb9JwD6um1bAKC5UqoFgB0AxgKAiDQFMAhAM+0974pIbNhK64bj6ImIjBkG\neqXUUgCn3LbNV0rlaU9XAkjVHg8EMFUpdVUptRfALgDtwlheIiIKUDhy9EMB/Kg9rgngoNNr2do2\nDyIyXEQyRSQzJycnDMWInGPnriBtzBysO3A62kUhIgpYSIFeRMYByAPwRaDvVUpNUkplKKUyUlJS\ngvr8osrR/7rrBADg0xX7i+YDiYjCKC7YN4rIgwAGAOihlCPkHgJQy2m3VG1bRDBDT0RkLKgWvYj0\nBTAawG1KqUtOL80CMEhEEkWkLoB0AKtDLyYREQXLsEUvIlMAdAOQLCLZAJ6FbZRNIoAFIgIAK5VS\njyiltojINABbYUvpPKqUyo9U4SVSByYishDDQK+UGqyz+UMf+08AMCGUQvmLqRsiImOmnhmrOGOK\niMiQqQM9EREZM3WgZ3ueiMiYqQM9Iz0RkTFzB3oiIjLEQE9EZHEM9EREFmfqQM9liomIjJk60BMR\nkTFTB3rOlyIiMsZAT0RkcaYO9EREZMzUgZ4NeiIiY6YO9EREZIyBnojI4hjoiYgsztSBnuvRExEZ\nM3WgJyIiY6YO9M7t+e1Hz+FqXsRuT0tEZFqGgV5EPhKR4yKy2WlbZRFZICI7tf8rOb02VkR2iUiW\niPSJVMEB1wlTfd/8BeO+3ex9ZyKiEsqfFv0nAPq6bRsDYKFSKh3AQu05RKQpgEEAmmnveVdEYsNW\nWgNr958GAMxcfwgHTl4qqo8lIirWDAO9UmopgFNumwcCmKw9ngzgdqftU5VSV5VSewHsAtAuTGX1\nUDrB9Rxi75x9Yup69H/7l0h9LBGRqQSbo6+mlDqiPT4KoJr2uCaAg077ZWvbIiK9Wlmvr52/khep\njyUiMpWQO2OVrRkd8DhHERkuIpkikpmTkxNqMWxlAbDj2PmwHIuIyCqCDfTHRKQGAGj/H9e2HwJQ\ny2m/VG2bB6XUJKVUhlIqIyUlJchieOr9xtKwHYuIyAqCDfSzAAzRHg8BMNNp+yARSRSRugDSAawO\nrYj+i/T8KU7QIiIz8md45RQAKwA0EpFsERkGYCKAXiKyE0BP7TmUUlsATAOwFcBcAI8qpSI2uN09\n7hboBOLNh84iv4ABmohKrjijHZRSg7281MPL/hMATAilUMHKPn3Z5fnWw+cw4O1leKx7A4zq0yjk\n44tIyMcgIipqpp4Za+TY+SsAgE2Hzka5JERE0WPqQF8uyfCChIioxDN5oI8PaP9j567gs5X7I1Qa\nsrKCAoW0MXPw3yW7o10UooCZOtAHatjkNXj6u804cvay8c5ETvK1jv7X5mVFuSREgbN0oH/o4zUu\nz09dyAUAjsKhoPGbQ2Zk6UDvDUfPUKD4jSEzK5GBnihYnDRHZsRAT0RkcSUq0LMtRkQlUYkK9CXR\nz9uP4XIub7FIVJIx0FtY1tHzGPpJJsZ9tynaRbEMXhWSGZk+0I/r1yTg95SUERTnr1wDAOznbRVD\nxgBPZmb6QF8mkcsgeMPgRESABQJ9TADNc18j4y7l5jlawEREVmKBQB94IkbvLZ0m/ozrx88PQ4mK\nj5KSoipKHEZPZmT6QB+uSa5nLlmvNc+YFD4M8GRmFgj0gUf6w2euYNWekwGnasw6K5Ite6KSzfQ9\nmYHk6I+es92I5M73lgMAbmqYgslD20WiWERExYbpW/T+5OiX7MjBhat5Htu3Hz0X0GfZTxRERGZi\n+kC/cs9Jv/bbonM7QTFIaizafhxpY+bgpLa88co9pwIvYBSZNNNERGEWUqAXkb+KyBYR2SwiU0Qk\nSUQqi8gCEdmp/V8pXIXVc+BU8JOBjp67gk9+3ev19Y+017Yc1r/n7J6cCyiw2Nr2B05eQs75q2E5\nVm5eAS7qXEmZkWLXNrkZ+dV6/LDpSLSL4ZegA72I1ATwZwAZSqnmAGIBDAIwBsBCpVQ6gIXa84gJ\nZnils/HfbzXcR6/Dd8vhs7j5X0sw6Zc9AX2eUgqbi+hm5cH8aLq+ugg3TPgpLJ9/z/9WoNmz88Jy\nLG/yCxTOXrbeiKmisvnQWWw4eCbaxTClGesO4U9f/BbtYvgl1NRNHIBSIhIHoDSAwwAGApisvT4Z\nwO0hfoZP4RhemZdfEPB7Dp6y3Y7wt/2nA3rfrA2HMeDtZZiz8QiUUliUdTxiVwXRTt2sL4IAMvHH\nbWj53HxOdgvSgLeXYeA7v0a7GBRhQQd6pdQhAK8BOADgCICzSqn5AKoppezXM0cBVNN7v4gMF5FM\nEcnMyckJthh+u3fSSq+vzd1yVHe7PVCGc3jiruMXAAC7cy5g3pajeOjjNfhwmff0UaRtO3IOn63Y\nF7XPD9Xsjbav2rkr1kgREUVCKKmbSrC13usCuA5AGRG533kfZRt4rtuuVEpNUkplKKUyUlJSgi1G\nWFqti7P0TzSRzssePWsbxZN9OnqLjt3y71/w9MwtUfv8UBXVHIFoXx0RhSKU1E1PAHuVUjlKqWsA\nZgDoBOCYiNQAAO3/46EXM7K8pU7sf9znDNICpy/mhpR+iXQMKQm3yDXrZDbyz8FTl/DAh6t0h0mT\nsVAC/QEAHUSktNh6K3sA2AZgFoAh2j5DAMwMrYi+5RUEnl/34BYI8wuUS+D4aVvhuWrsjE1IH/eD\n4/n8rcfQ+p8L0O+tX5BvEOzz8gtwLd/7PsfOXfE6SuWHTUfwvyW7fR7fXUkIfrzRe8nw2vws/LLz\nBH7aeizaRTGlUHL0qwB8A+A3AJu0Y00CMBFALxHZCVurf2IYyulVaqXSIR/jrNM6NycuXEX9f/yA\nT5bv071cn7L6gG6w3n70PD72MVQTsKVJ/usjWLd/cSHueHe57mt/+uI3vPTjdp/Ht5u35Sg6vrQQ\neRYb+ulLCTinEQUtpFE3SqlnlVKNlVLNlVIPKKWuKqVOKqV6KKXSlVI9lVIRnWV0X/vaIR9j4fbC\nFnv2adtomglztmGFn5Ox7HbnXPD5+s7jrq9fzfO8Gsk6dj6gz7TbdfwCTl6wjX9/+rvNOHL2Ck5e\nzPX7/Veued5ucPOhs45jUmC+WnMA244ENvOajHE+Q3BMPzM2Uo3WYFrDU1YfdNyfdcex81i++4TX\nfV9fsMPRQg9Ha7Tn60tw87+W6L62Zp/xENDGT8/12Dbg7WW47T++h95tOHgGI79ab7mJY6F6cvom\n3PLvX6JdDCIAFgj0bWpXDOvxQs1rX8y15dh7v7EUv39/VTiK5OL1+VleX4vExKFDZ2xXOHtyLmDt\nfs+Ls2GT12DGukMBXT2Q/y5ezcNLP27D1Tze4B0wXraE9Jk+0Be3zrhIl+atn3f5fP1ybj6Oa0sY\nhLMsN/9rCe58b4Xf+xdFCz/r6HnHiai45+gnzNmqe6I08s6iXfjfkj34ctWBCJQqfLJPX8LnK/dH\n/HOYugmO6QN9uP3OS2eov8Jx4kkbMwc7gszVN3nGMwUTKm9XORPmbMUJbcE39z/Au/4b+M/x7OVr\nhiOXnPV5c2nAn+GPvPwCxxwHABj19YaQ1zR5/5e9AZ0o7XK1fpw8H6O1ioPB76/EU99tjtgM5eLV\nnDMfBvowm7n+kMvzRduP4+ftx7DvxMWAjuPvqpy+hCs0eIu97//ifZTRbwcCW/7gyrV8tHxuPp77\nPvqTt56fvRUdXlqIM5dsJ7Fv1mZj5LQNEfmsggKF5bu89+WEw6wNhzH6m8iU3+7MRVuA9/ad23ns\nPD4IcF0oZ8X7NFf8MdCH2XNui6Q99MkaDP0kE91eW+z1Pb/uPoFF2wOfV/bMzM3o/YZ+BywQvlZQ\nUYzHt3diz9pwOKj3KyiXEULHz13B8E8z/Zpg897i3Xh3cWFK7Gftd3HucuQn53z06178/oNVXseH\nh+Mn/+cp6zAtMzsMR9KXm1eA8wY/5wFvL8MLc7Zh5Z6TSBszx5Fy88el3DzMXB/c9yKc8vILHFdY\nZmOJQP9gp7RoFyEke3IuYsQXa122+QrS9pm6n67Yjx3HvA/pDFf3xSfL97k8X7H7JEZOW++yzdsy\nEr7szrmAGb+FJwDN+O0Q2r7wk2MlxtcX7MD8rcfwvR8njpfnbscrcws7ue0nh3Dlg32dKPdoV3pH\n/Lipzfkr1wLulC2Kk/R7i40n8tmHEk9ZbetrWLPXd39F9ulLjvTZZysin/v3x+/eXY6GT/0Y1HuV\nUlGdwGiJQB/qUsXh9uQ3GwN+z5Vr/rcU/J0hq/e9WrIjx7He/FkfN0T/ak1h598Lc7Y5Hn+6Yh8G\nv78SM35zTVGN/majz3Xs7/7vcjQc9yP2OM016Pn6EoyctgEz1x8KeXLXit22VFfWUde+jeW7A0+B\n2W8UH66/S3+OY/8GK6XwwuytjlSf870Qrh8/H3f/Vz/Pf/byNaSNmYP3l7qmRxYUwUzS05cCH3Fl\ndBLt8vIidHhpIQAgv4gC5Nr9p3wG403a8uJ7A0zDArYrmrpjf8DZy9fw3bpDxm8IM0sE+p5Nqhb5\nZ7750w6vr32VeTAsn3HqYi6u6Syh/HVmtstCaHqTnQDPFv3pi7kY8tFqx3rz7q1yZ09O36S7/Rkf\nC6DN2ei99bxm32nk5hc4xvrPXH/IEQCfmLre7/ytUgqLs4773WnrT4ve62cFuP9nK/ZhzHTPk3wg\nx9l1/AI+WLbXkeqz39XssvY73pitfy+DnPO21u8U7QQ96usNSBszJ6ggrCc3r8BwQmCkFEWcn7/l\nKO58bwW+8GN0U3cfaVhvthy2TZ67/Z1f8Zev1ns0SCLNEoG+U4PkIv/M7RH+RRUooM0/F2D0Nxs9\nLtePn7+KLi8vcjzXm+wEAC86tcSnZR7EFbfjHDkb3nvgBvL3+MRU15OMfUjoGR9XGe8u3oWMF37C\ngx+v8bqURDiH3wXagfn0zC2YusbzJK/XSjxzKVd31VJvpTcOdq5n9W/W2lJi9vsmhGrct5vQ419L\ncCqI+RLhGFgQafY71e3JCby1Hgj71YC3xlmkWCLQW5G9xfr9hsNBt2gOOwXy0d9s9DiO+2EjNTRu\n4bbA0gfuefuc81dxKTcPr8zNckzMOuh2C8nV+2wt35e1XLvz1cz6g2ew63jgJ2Z/ZhTr+drtik7v\n19fttcXo8vIij99JIEnIn7Yaj+aavGJfAEfU9/iUdfhaO3Fc8GPd/7z8ApeT26KswoEGgX6XjVaO\nJf9YJtC/eW+raBchrKZrwS6cC5O5H8m9pXn9+Plh+yxnwyZnujz/VWc4oXOAGznNlnZ4a+FOAMAN\nE37CQLelGPRazgB0W5y3v/Mrer7uOea+4bgfA14R1B9/d+uj0QtuhVcutheNupk2ZHsOV334U53R\nXEF+XZRSyC9QuqunBpr+ajDuRzz13ebgCuKmxfj5OHCy8KTu7UTx7bpspI2ZE/Ds8PwCVeSt62iw\nTKAvZv2xIbPn9IDCy/BQRbrX/1Juvl9XBfd94Lk0hHN97SY7jfZxXxAuHHLzC/xeEdSXq3n5juGh\nepzTSfY+Bnfjvt3scxG0nw2G39q//8H+hg+cuoSnZ25Gs2fnBTRpzZ39K/bFqgMuk87cXblW4Pfs\n6X0n9a9ajpy9jNfnZ0EphUlLbXM63K/0vPl2XTY6vrQQf/pirUvq88Cp0FM3D3y4Cn/4aHXIxwkn\nywT64jbyJpzC1TpyjvO2NffDcliHV+dl4frx83HMj6GC7o6fD29/QVHOpezzxlKfM5Kdf85fZ2bj\nwY/X6O7nawlrd5dyC1vewdzz2J1ShSknfwO9UgppY+Z4DL+1s4+acbZWu8fy2Bmb8NTM0L7Xj3+5\nDm/9vAtbDp9zNGL8DQNjpm/CkbNXMG+La1rR+d4Tgcg6et4xdPSXnSewdEeOz4ZVUQ+0tEyg79ow\nBTUqJEW7GKZx/PyViK0b0v+twFdt1Lvna1H/MZzWSfv4sxTFvpPeW5HnrlxzSb9lu00UmrK6MAUl\n8D9QNX1mnuPx3f8rHHK598RFrHLq/Dzv9HO9lJvnNU2hULhgmK/vhYLC5OX7MHbGJt0Tgm75nXZz\nnig1VQuM2acv4ZW5273f6c39uVIY/c0GZGonDed4GuyiZ+5Lhuecv4q0MXNcfpa+9HlzKcbO2IRp\nTinFb9cdwpVr+Y4Z1tFkmUBfoVQ8VoztEe1iFGt/+7pwFMl9H6zyOdkqFPb1bwKhFzTOXb7m9x+a\nP75acwBnLuV6Xbdm8PsrPS79e7/hmdt/fMo6DPHz0rzF+Plo/uw84x0BfBfk7M91B864hLd7J63U\n3a/pM/NcRms5u3g1z3ERZHSl9+ysLZiy+oDuiqXBXCV2eXkR3l28G8u8LQXhdszL1/JdZvo6n1yC\nvbB/dZ7rqrCZWuf+x7/uC+g4o52G146ctgGNn56LVs8vCK5QYRQX7QJEkkjxX9WwKK12mo0Y6WFk\n4ZBXoLwGrWA8OX0T1h88gymrD+LHJ270eH370fN+debZOycLChSGf5ZpsHdg3vxpp+Oxt3SW3to4\n/i6md8LLjWT+PHWd06Stwu2/HfA+8qj9i56pmdk+5lK4cy9zKLcFtZfZfkj7bN0R3eoDAD5fuR/l\nkuIwsFVN/47n9nyjTme4mVimRW83fURHx+Mm1ctHsSQULScvXHXkS93ZUyXebgqi11HqzZQ1B4LO\n6Xoze2Ph1catby/T3efrEDvnF247ht05F5A2Zo5j256ci04durYwV1CgPG5teclHpzNg61R29v7S\nPX7n/Id+kokvVu13mQ3sXB47vfSMfZ8Jc7Zh/8mLeHnudrw819bRvvXwOTz13WaPuRv+sP9MfN2A\nJ5i5BUXNcoG+bZ3K6FCvMgDg3htqRbk0FA17gpiibvfafO8znt25B7VwO3ZOv/Ud6uipYZMz8foC\nz3raA+gzM7dg/cEzeGeR570P3JdYMDLhh20BnZjGfbsZ/d9yPcE5z2cYOW2Dbsvf/iP5ZecJ3PTq\nYpfXbv1P4fHOXr6mO4Q0FG3+Gf3UjJGQUjciUhHABwCaw3a1MxRAFoCvAKQB2AfgHqVUcDNPQpRe\nrSz2Tezv0nIh64vGWiL+cl/GOlz8HVZop9dBaG+9frM2G99vOIyWqZ53b5sRxM/WWzos2HFR7yzy\nHJ3k69TnfEXR8rn5KJ/kO+yt2nPS59BQMwq1Rf9vAHOVUo0BtASwDcAYAAuVUukAFmrPi1RalTIA\ngPJJ8QCAzg2qFHURKIr8Wa8kWvb7GKHjL71O20DHbeulP5y3iAAnLhbPG8O7n6RsfXH+X+XojfBy\ndu+klXh+9laf+4Rq06GzLgv8RVrQgV5EKgDoCuBDAFBK5SqlzgAYCGCytttkALeHWshAjb+tGT4c\nkoHmNSsAAF65q2VRF4HIdC465d8FUmw77N37nX/ddQK7i2lZvXn6u82OBf6KQigt+roAcgB8LCLr\nROQDESkDoJpSyt6jdBRAtVALGaik+Fj0aFL4sddVSMKdbVKLuhhEVARe/MH77ObGT+uvH+9v+//H\nzUd1t4dr2YSiupFJKIE+DkAbAO8ppVoDuAi3NI2yXU/p/kxFZLiIZIpIZk5O4DetCISIYMLvmkf0\nM4jMxGgm8uUiWv9lzsbA78UbyBINgdznIRDeVowNlF6HdyRIsD34IlIdwEqlVJr2/EbYAn0DAN2U\nUkdEpAaAxUqpRr6OlZGRoTIzwzse2V1efgEajAvu7jBEZC2JcTEes2Gj5cMhGS4ZiECIyFqlVIbR\nfkG36JVSRwEcFBF7EO8BYCuAWQCGaNuGAJgZ7GeEU2yMddfCIaLAFJcgDwBv+LiJUbiEOjP2cQBf\niEgCgD0AHoLt5DFNRIYB2A/gnhA/Iyz8nTlIRFSUNh/yvmppuIQU6JVS6wHoXTZw0RkiomLCcjNj\nfVny924uz2c/3iU6BSEiKkIlKtDXqVIGpeJjHc9FgKf6N4liiYiIIq9EBXrAdbKFUsADHetErzBE\nREWgxAV6d8HeqICIyCxKdKAXsd69ZomI3JXoQA8AsYz0RGRxJT7Qx8QI9k3sH+1iEBFFTIkL9C5L\nsTI/T0QlQIkL9H2aVQ9o/yY1yiM+1nZCMLphARFRcVTiAv3Ld7XAm/e2Qq3KpVAvpYxj+/8eaKu7\nv/NNpGc+5jrB6tHu9SNTSCKiMCpxgT4+Nga3t66JX0bfjCSnyVPOLf0143rqvtd9XbSKpRL8+sxX\n7mqBoZ3rBl5YIqIwKHGB3h8p5RJ1t7vn9NvUqeTzODUrlgIA3JNRC/k6NzQmIioKTDo7+eLh9sjN\ntwXkH5+40ZGbt7OPxKxaLhFLR3dHUnws6iaXwd4Tnrcxe6x7A/yhYx3s0u4LGakbIETTsC518eGy\nvdEuBhEZYIveSecGyejeqCoAWydsg6rlABSmdaqUtaVq/tqroSPt88yApgCATvWroK+233v3tcGo\nPo1QtXwSOtVPBgBcKqI79hSlHk2qRrsIROQHBno/vH5PK6z6Rw+UTojDvon9MbhdbcdrLWtVBAA8\nfKPvHPyfutVHWpXSLu8lIioKDPR+SIiLQbXySbqvVS6TgH0T++Pmxr5vBdakRnks/nt3/HNgs6DK\nYM/3FydxMfz6EJkB/1LDyH5D5YQ47z/WuNgYvHtfm4CPfWfb1KDLFSlcPYLIHBjow2h413oonRCL\njDqVfe7X7/oaAR/73htqBbR/zzDmzxtULRu2YxFR0WOgD6PODZKx9fm+qFA6PizHcx7PX7NiKXw2\nrJ3j+RcPt3c8rq6bVpKwzeQd0ML/E1MwVyvh4HxDmXDwNsSWyIwY6IuJ7o1SAAC1K5c23Ld+Shl0\nbpDsmLVbJtEzyHlLq/Rs4tqX8K+7Wxp+nlL62/U+olSCZ1kSfaSywmHPi/2w9fk+hvuVC+DEN+fP\nvM0kWUfIf4EiEisi60Rktva8sogsEJGd2v++ZxURAODjh9ph+ohO+PZPnfDFw+0x4XfNHUHT3hFr\nD7g1Ktiely9lu3Kom+w7tTKw1XWOxx8Mcb2XezC5/4TYGAxuVwuta/v3q133TC+M6t3QcL+bG1f1\nuZLo7U71cBYTIxA/Ogzi3Kc2e5EQF+P3rGerua6C/qADMrdwNLWeALDN6fkYAAuVUukAFmrPyYtX\n72qB17RWdds6lVClbCI6N0jGfe3roGxiHBaP6oafR93k8h57TKtZsRQ+fugGvHFvS7x5bytULuMa\nnDrWrwJzf+YkAAAPeUlEQVTA8yphcLta+L+u9fDz32zHva2lfgAFbC1b9wb9jgm34KU7WiDWj8CZ\nEBeD0glxePjGeniwU5pj+196pnvs+/o9vq8u9IamDuvi/9ISBV6uTNzteOEWnx3qzj50O3FGyh8N\nhu+Gy9AAfp5kHiEFehFJBdAfwAdOmwcCmKw9ngzg9lA+w+ruzqiFu3y0qtOSyyAxznv+uXujqiiX\nFI/bW9fEsie7O7YLgH8Pao0Ff+2KCqVc+wxeuqMFxvZrgnopnlcCbw1u7Xj89SMd0ey6Co5LiTpV\nSuPJvo39rZqLpPhYjO1X+N6/9GyIzKdc1xSqWNp7K3rfxP6opZPWelqbsOYP5ZaDapfm2Wk+7y9d\nPba5z5Cu6NQHU1Qjj8om2j5zeNd62P1iP+yacIvuft8/1gUbx/cO+nP8veohcwm1Rf8mgNEAnOf3\nV1NKHdEeHwWgO8BcRIaLSKaIZObk5IRYDHJXv2pZJMXHIr1aOcMboNuD1Zv3tnJp3d+gBUJ7eLyj\ndSpGdPO9YucNaZWRXrUspo/o6PGarxOWs3VP98LkobaO54c6p2Hmo539ep+31nVy2UTExwrKJrrm\n6Kc94lnGRtXLeWx79z7XlU1Xju3h6OhOjIvF58Pae7wnUhLjYhAbI4iLjcE7v3ft+I6LEVyfWgHl\nk4IfDJDg5+/ISh7qnBbtIkRc0IFeRAYAOK6UWuttH2VrQuleMCulJimlMpRSGSkpKcEWw7Tm/uXG\nsHf4lYqPxV1tU/G3Xg0xsldhTtwowN7awhbc7bN83dlb0rUqe5+0Nbhdbfxf13oomxiHBSNvsl0J\nGPDWdqxUJgE3NUzBnD93wdP9mzrKZdR67tGkGhrrBOrHutfHzgn9EBsbXGu1V9PCtsrnw9ojKT4W\n9bUhp0nxseiSnhzUcf3l3EJ3rkF/t9FQf+1l3A9i5M62NUM+Rri4X4lGSmUfV5KRdGebopsbE8r4\nu84AbhORfgCSAJQXkc8BHBORGkqpIyJSA8DxcBTUahpXLx/we4xSzCLiyPcHomfTai6doOWT4nDu\nSp7j+d1tU5FaqRQ61qvi8d6RvRoiNkbwaPcGARfauQN11T96eLzufrLwNvrH2Vwt9ZI2Zg4A27pD\n9rWKfC0g2qhaOb9+dvagfmebVKw7cAZ1k8v43P/BTmn4ZPk+44LrKJ0Qi0u5+SifFA9l8Nt378SO\njxVcy/ezU0ITFyNIjItFzYqlcOjM5YDLGw7JZRNw4kIuAGBcvyYYPX1jwMeoUCoeZy9fC3fR/FKr\ncikcPOXfz64oJxwG3aJXSo1VSqUqpdIADALws1LqfgCzAAzRdhsCYGbIpSQAQBWtszW9qmerNZwW\njLzJJfUiIuhUP1l3ZMufe6R7D/I6Zj7aGYtGdbMd12m7tyUmQnXL9TUQo+Wd83Qi/VfDO6BFagV8\nPaIjrk81vgqxu79DHex9qZ9HB7i7GjqjWAa0qIHFo7qhTe3CKyj3voatz/fBr0/ejF/H3AygsHVb\n3q2VW6VMAh65yTOdtvk54+Gm7uynhS4NPK9QXrmrRUDHMlrqIyG2MPTsfrGf4/HXj3RyPI6PCy4S\nfvV/HQLaP5wB19vf5t6X+nlsc+/7iaRIDHCeCKCXiOwE0FN7TmHQvGYFTB3eAWNuCbxDdO1TPbF6\nnGerWU+18kloazC7N1gta1V0tIJD/QNrGUBgBoB8nWE37etVwazHugSV1/ZnSKfeqKAneqQjLbmM\nox8C8BzWWDohDpXKJDiG1j7QoQ7+eXtzl5FLALD26V6634fEuFjsfakfRnSrj9mPB5Yi1KvWPRm1\n8PjNthN6p/qFV3aj+zby2LdUfCy6NkzR3qefnnC+QomNEWwc3xurx/VA3eQymP14F1xXIQldGgSX\n0m1cvTz+3qewXGlVjOemhMNfe+qnzpLiYzy+K18+XHT9OkCYAr1SarFSaoD2+KRSqodSKl0p1VMp\ndSocn0E2HepV8Xvon7MqZRNRtVzxGiMdTD2cud/a0dmiUd1cZg8DCDiVEarB7WojLta1ji/dcT3S\nq9lafeWS4rF8zM149tamqG4wfj0uNgYPdKjjcTxfRARP9m2M5jX9OyH21voivKXI/ta7ESbecT3e\nu78t6iWXwZN9G2No57oY0a0+bkgrnFPRsHo51KlSBnte7IdX7vJMh7WtUwkNtZ+BfcJe+aR4x/ez\nec0KWD62B5LLJnic2LzxtV/H+p4pR2fOQdh56ZCsF/r69dkA0LdZdTzRM91jZJc3nRok44831gMA\n9A9iSZRAcWYsRU3phDhMH9ExpOGA3tRNts0edpaXH7mbv7xyVwt8+XB7l45bPT0au65BdF3FUnio\nc120TK2IR26qj5fuuB5v3tsqImV0XwF12ZPdHSfbxaO64c1Bts+1t7b1UjiD2tVGhVLx+HlUN4zo\nVh9J8bF4sm9jR4d/i9QKjmUw7Ckze99OsnY/h8S4GEz43fVoXbuiR4eyMxHB+Nv0U0B/vLGuS3rx\nbi9XDrc0r+71GHomPVA4cisxLhZtalfEf37f2sc7XDlfNNrvSe0t9tdLKYsNz/Z2GdIcKbzDFEWE\nv2mZQFJE1con4cb0ZIzoVh+tvIwQ8mVg65r4ctWBgN8HANNHdESMj0rdk2FbdK5Tg2R8vnI/nvpu\ns8c+vmb9xsRIUCk5f/00siuSyyai1fMLHNtSK5V29JNUr5DkCNa3trwO0zKzMf62Zuj5+pKAPufv\nfRp5nFAmD22HK3n52HjwLO7/cBVEgFa1KuLbP/k3bNbdp0PbOVJDdp4d97boWi+ljEt/gJ4HO6Xh\n1XlZAGy/h43jeyNPu/qboZXxsS/X+TyG/avhHNPtP1tfjfyiGlnEFj1FREJsDNrWqYS3A2gNGYmN\nEXw2rD061U9G6YTA2yj/HNgcG8f39po39qVtncp+L/lQHDWoWk53Qlp6NdswUedz2I3pKdg3sX9A\nq5b6GhWUEBeD8knxKF/K9jurXdn3SCUj7kFetzxacQT6y2M8e6ut87t93coo4za/onxSvEcn+1P9\nm+C525oZ5vv/2jPdMelsQIvrUKtyKbx81/Uu+6wc619fWTixRU8RISKYPqKT8Y5FKDZGUD4pHq/c\n1VI3d+zu60c6Irls4KtYmmmd/k+HtsfmQ2f9nszmjXNg9aZFakV88IeMkOYd3Nc+uDu0tUurjNX7\nCrsL7RPj7L+r1EqlkH3a+7DIh7V8+se/2u6RvGhUN9SsWAoNn/oRQGHqqHXtStj1Yj+cuZSL8knx\nLnMbWteuiHZ1Kxv2x0QCAz2RFzfoLJFgdrMf74IBby9zPK9cJsFnC3lcvyZ4dX6W4XH7NKuO5btP\noo5Bi7enQR+Gng71KqNJjfJ49lbPXPuXD7dHvh8doE2vK+8I9Hc7LTlif+vsx7s4xu/7Ukq7khTY\nrlRWjL0Zl3LzUd9tORG9q6dgU1XhwEBPFGb22bkZdYpfqsffETh2f+xaD3/sWs9wvz90rIM726Z6\nLDMRDlOHey5VYdfJqcP4H/0a48Nltha3t9D/zICmGNqlLpbvPuGyvWLpBJ9rLdm9/4e2mLn+sOOE\nZl9JtrhjoCcKs7Z1KmPl2B4ul+jF7UYmwaSkfBHxXEuoqA3vWh/Du7pOHnNPoxk9N5JaqXRAEwSL\nCwZ6oghwDvK/Pd0r5DkD4fT9Y12ikicuSvHaSBv7DeztN7+xb7f/H0ynvhmVjFoSRZHRMglFLZCl\nHszqoc5pOHMpF8O1tNOfe6QjNkYcw2Az6lTCqN4Nde9xYEXi70yuSMrIyFCZmZnRLgYRkamIyFql\nlOHdb4rP9SQREUUEAz0RkcUx0BMRWRwDPRGRxTHQExFZHAM9EZHFMdATEVkcAz0RkcUViwlTIpID\nYH8Ih0gGcMJwr+LPKvUAWJfiyCr1AFgXuzpKKcMF+otFoA+ViGT6MzusuLNKPQDWpTiySj0A1iVQ\nTN0QEVkcAz0RkcVZJdBPinYBwsQq9QBYl+LIKvUAWJeAWCJHT0RE3lmlRU9ERF6YOtCLSF8RyRKR\nXSIyJtrl0SMiH4nIcRHZ7LStsogsEJGd2v+VnF4bq9UnS0T6OG1vKyKbtNfeEgn0Jmgh16OWiCwS\nka0iskVEnjBxXZJEZLWIbNDq8pxZ66KVIVZE1onIbJPXY59WhvUikmnyulQUkW9EZLuIbBORjlGt\ni1LKlP8AxALYDaAegAQAGwA0jXa5dMrZFUAbAJudtr0CYIz2eAyAl7XHTbV6JAKoq9UvVnttNYAO\nsN2A/kcAtxRxPWoAaKM9Lgdgh1ZeM9ZFAJTVHscDWKWVx3R10cowEsCXAGab9fullWEfgGS3bWat\ny2QAD2uPEwBUjGZdirTyYf5BdgQwz+n5WABjo10uL2VNg2ugzwJQQ3tcA0CWXh0AzNPqWQPAdqft\ngwH8L8p1mgmgl9nrAqA0gN8AtDdjXQCkAlgI4GYUBnrT1UP73H3wDPSmqwuACgD2QusDLQ51MXPq\npiaAg07Ps7VtZlBNKXVEe3wUQDXtsbc61dQeu2+PChFJA9AatpawKeuipTvWAzgOYIFSyqx1eRPA\naAAFTtvMWA8AUAB+EpG1IjJc22bGutQFkAPgYy2l9oGIlEEU62LmQG8JynaqNs3QJxEpC2A6gL8o\npc45v2amuiil8pVSrWBrEbcTkeZurxf7uojIAADHlVJrve1jhno46aL9Tm4B8KiIdHV+0UR1iYMt\nXfueUqo1gIuwpWocirouZg70hwDUcnqeqm0zg2MiUgMAtP+Pa9u91emQ9th9e5ESkXjYgvwXSqkZ\n2mZT1sVOKXUGwCIAfWG+unQGcJuI7AMwFcDNIvI5zFcPAIBS6pD2/3EA3wJoB3PWJRtAtnaVCADf\nwBb4o1YXMwf6NQDSRaSuiCQAGARgVpTL5K9ZAIZoj4fAlu+2bx8kIokiUhdAOoDV2uXeORHpoPW6\n/8HpPUVC+9wPAWxTSr3u9JIZ65IiIhW1x6Vg62vYDpPVRSk1VimVqpRKg+37/7NS6n6z1QMARKSM\niJSzPwbQG8BmmLAuSqmjAA6KSCNtUw8AWxHNuhR1h0uYOz36wTb6YzeAcdEuj5cyTgFwBMA12M70\nwwBUga0DbSeAnwBUdtp/nFafLDj1sAPIgO2LvxvAf+DW0VME9egC26XmRgDrtX/9TFqXFgDWaXXZ\nDOAZbbvp6uJUjm4o7Iw1XT1gGz23Qfu3xf73bMa6aGVoBSBT+459B6BSNOvCmbFERBZn5tQNERH5\ngYGeiMjiGOiJiCyOgZ6IyOIY6ImILI6BnojI4hjoiYgsjoGeiMji/h9rDZqjFxSujQAAAABJRU5E\nrkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%time\n", "while True:\n", " # Try catch for interruption\n", " try:\n", " # Reset\n", " if p + T_steps >= len(data) or iteration == 0:\n", " g_h_prev = xp.zeros((H_size, 1), dtype=xp.float32)\n", " g_C_prev = xp.zeros((H_size, 1), dtype=xp.float32)\n", " p = 0\n", "\n", " inputs = [char_to_idx[ch] for ch in data[p: p + T_steps]]\n", " targets = [char_to_idx[ch] for ch in data[p + 1: p + T_steps + 1]]\n", "\n", " loss, g_h_prev, g_C_prev = forward_backward(inputs, targets, g_h_prev, g_C_prev)\n", " smooth_loss = smooth_loss * 0.999 + loss * 0.001\n", "\n", " # Print every hundred steps\n", " if iteration % 100 == 0:\n", " update_status(inputs, g_h_prev, g_C_prev)\n", "\n", " # Update weights\n", " for param, dparam, mem in zip([W_f, W_i, W_C, W_o, W_y, b_f, b_i, b_C, b_o, b_y],\n", " [dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y],\n", " [mW_f, mW_i, mW_C, mW_o, mW_y, mb_f, mb_i, mb_C, mb_o, mb_y]):\n", " mem += dparam * dparam # Calculate sum of gradients\n", " #print(learning_rate * dparam)\n", " param += -(learning_rate * dparam / xp.sqrt(mem + 1e-8))\n", "\n", " plot_iter = np.append(plot_iter, [iteration])\n", " if isinstance(loss, cp.ndarray):\n", " loss = loss.get()\n", " plot_loss = np.append(plot_loss, [loss])\n", "\n", " p += T_steps\n", " iteration += 1\n", " if iteration > 6000:\n", " break\n", " except KeyboardInterrupt:\n", " update_status(inputs, g_h_prev, g_C_prev)\n", " break" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Gradient Check\n", "\n", "Approximate the numerical gradients by changing parameters and running the model. Check if the approximated gradients are equal to the computed analytical gradients (by backpropagation).\n", "\n", "Try this on `num_checks` individual paramters picked randomly for each weight matrix and bias vector." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from random import uniform\n", "\n", "\n", "def gradient_check(inputs, target, h_prev, C_prev):\n", " global W_f, W_i, W_C, W_o, W_y, b_f, b_i, b_C, b_o, b_y\n", " global dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y\n", "\n", " num_checks = 10 # Number of parameters to test\n", " delta = 1e-5 # The change to make on the parameter\n", "\n", " # To calculate computed gradients\n", " _, _, _ = forward_backward(inputs, targets, h_prev, C_prev)\n", "\n", " for param, dparam, name in zip([W_f, W_i, W_C, W_o, W_y, b_f, b_i, b_C, b_o, b_y],\n", " [dW_f, dW_i, dW_C, dW_o, dW_y, db_f, db_i, db_C, db_o, db_y],\n", " ['W_f', 'W_i', 'W_C', 'W_o', 'W_y', 'b_f', 'b_i', 'b_C', 'b_o', 'b_y']):\n", " assert param.shape == dparam.shape\n", " dparam_copy = xp.copy(dparam) #Make a copy because this will get modified\n", "\n", " # Test num_checks times\n", " for i in range(num_checks):\n", " # Pick a random index\n", " rnd_idx = int(uniform(0,param.size))\n", "\n", " # evaluate cost at [x + delta] and [x - delta]\n", " old_val = xp.ravel(param)[rnd_idx]\n", " xp.ravel(param)[rnd_idx] = old_val + delta\n", " loss_plus_delta, _, _ = forward_backward(inputs, targets, h_prev, C_prev)\n", " xp.ravel(param)[rnd_idx] = old_val - delta\n", " loss_mins_delta, _, _ = forward_backward(inputs, targets, h_prev, C_prev)\n", " xp.ravel(param)[rnd_idx] = old_val\n", "\n", " grad_analytical = xp.ravel(dparam_copy)[rnd_idx]\n", " grad_numerical = (loss_plus_delta - loss_mins_delta) / (2 * delta)\n", " # Clip numerical error because grad_analytical is clipped\n", " grad_numerical = xp.clip(grad_numerical, -1, 1)\n", "\n", "\n", " err_sum = abs(grad_numerical + grad_analytical) + 1e-09\n", " rel_error = abs(grad_analytical - grad_numerical) / err_sum\n", "\n", " # If relative error is greater than 1e-06\n", " if rel_error > 1e-06:\n", " print('%s (%e, %e) => %e' % (name, grad_numerical, grad_analytical, rel_error)) " ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "W_f (-5.788818e-03, -1.894525e-03) => 5.068488e-01\n", "W_f (-4.424725e-03, -2.570756e-03) => 2.650237e-01\n", "W_f (-1.574892e-02, 3.109146e-04) => 1.040279e+00\n", "W_f (2.522772e-02, -2.049634e-05) => 1.001626e+00\n", "W_f (-2.772584e-02, -3.596031e-04) => 9.743922e-01\n", "W_f (-9.696586e-02, -3.551726e-02) => 4.638221e-01\n", "W_f (-1.919746e-02, -4.452238e-05) => 9.953723e-01\n", "W_i (-2.128421e-02, -1.987336e-03) => 8.292046e-01\n", "W_i (-9.618537e-03, -1.770927e-03) => 6.890236e-01\n", "W_i (-5.882980e-02, 1.477409e-03) => 1.051520e+00\n", "W_i (-4.811230e-02, -8.973669e-03) => 6.856085e-01\n", "W_i (-2.069689e-02, 2.397710e-03) => 1.262057e+00\n", "W_C (-9.150008e-03, -4.846476e-03) => 3.074723e-01\n", "W_C (-5.384253e-02, 3.501519e-03) => 1.139112e+00\n", "W_C (0.000000e+00, 1.724286e-05) => 9.999420e-01\n", "W_C (-3.551228e-02, -4.914250e-03) => 7.568799e-01\n", "W_C (0.000000e+00, 2.751392e-05) => 9.999637e-01\n", "W_C (-2.206686e-02, 1.656087e-04) => 1.015123e+00\n", "W_C (-4.734427e-03, 3.258538e-02) => 1.339983e+00\n", "W_C (5.200395e-03, 4.812866e-02) => 8.049695e-01\n", "W_o (1.664602e-02, 1.216984e-02) => 1.553371e-01\n", "W_o (-3.094253e-03, -1.786075e-02) => 7.046765e-01\n", "W_y (5.069651e-05, 9.411758e-05) => 2.998380e-01\n", "W_y (1.354294e-06, 3.190190e-06) => 4.038944e-01\n", "W_y (-1.707993e-01, -3.533963e-01) => 3.483377e-01\n", "W_y (-3.616201e-05, -6.439822e-05) => 2.807863e-01\n", "W_y (1.045780e-04, 2.551565e-04) => 4.185810e-01\n", "W_y (-7.008616e-05, -6.961373e-05) => 3.381729e-03\n", "W_y (-1.320173e-03, -3.049548e-03) => 3.957632e-01\n", "W_y (-2.135536e-05, -5.128610e-05) => 4.120281e-01\n", "W_y (-3.736993e-05, -3.475788e-05) => 3.621374e-02\n", "W_y (4.692321e-03, 1.760141e-02) => 5.790457e-01\n", "b_f (9.625265e-02, 1.281972e-01) => 1.423236e-01\n", "b_f (-1.711278e-02, -8.734714e-03) => 3.241346e-01\n", "b_f (-6.363861e-02, -9.571880e-02) => 2.013097e-01\n", "b_f (-1.353493e-01, -2.709278e-01) => 3.337093e-01\n", "b_f (1.295448e-01, 2.294842e-01) => 2.783604e-01\n", "b_f (0.000000e+00, 1.190781e-05) => 9.999160e-01\n", "b_f (-1.097183e-02, 2.446750e-02) => 2.625979e+00\n", "b_f (6.660086e-03, 1.548321e-01) => 9.175182e-01\n", "b_f (-3.765070e-02, -1.929383e-03) => 9.025073e-01\n", "b_f (8.172772e-03, -1.733225e-02) => 2.784549e+00\n", "b_i (-1.797925e-02, 3.960431e-03) => 1.565016e+00\n", "b_i (2.327358e-04, 1.566012e-02) => 9.707118e-01\n", "b_i (3.910756e-05, 1.577477e-02) => 9.950540e-01\n", "b_i (-1.699091e-01, -2.288668e-01) => 1.478467e-01\n", "b_i (-7.337276e-02, -9.197736e-02) => 1.125164e-01\n", "b_i (-3.058013e-02, 9.644181e-03) => 1.921303e+00\n", "b_i (6.721953e-02, 1.486464e-01) => 3.772104e-01\n", "b_i (-8.904879e-02, -5.259708e-02) => 2.573440e-01\n", "b_i (-8.196457e-02, -5.133832e-02) => 2.297493e-01\n", "b_i (-1.127150e-02, 2.934633e-03) => 1.704013e+00\n", "b_C (-8.878277e-02, -9.282753e-02) => 2.227166e-02\n", "b_C (-1.053689e-01, -2.198471e-01) => 3.520066e-01\n", "b_C (-1.254594e-01, -1.306037e-01) => 2.008990e-02\n", "b_C (-8.069183e-03, 4.495833e-02) => 1.437483e+00\n", "b_C (1.315191e-01, 3.286199e-01) => 4.283507e-01\n", "b_C (-2.857717e-01, -5.199677e-01) => 2.906598e-01\n", "b_C (-3.367491e-01, -5.167727e-01) => 2.109186e-01\n", "b_C (-8.336149e-02, -2.002720e-01) => 4.121887e-01\n", "b_C (-2.030515e-01, -2.748539e-01) => 1.502439e-01\n", "b_C (-2.615591e-01, -5.192239e-01) => 3.300082e-01\n", "b_o (-1.828830e-01, -3.566602e-01) => 3.220820e-01\n", "b_o (-6.164580e-02, -6.660104e-02) => 3.863832e-02\n", "b_o (3.400145e-03, 4.486300e-02) => 8.590997e-01\n", "b_o (-8.065856e-02, 1.755721e-02) => 1.556476e+00\n", "b_o (-3.057445e-02, -1.059484e-02) => 4.853036e-01\n", "b_o (-6.164580e-02, -6.660104e-02) => 3.863832e-02\n", "b_o (-7.878454e-03, 8.931704e-03) => 1.596026e+01\n", "b_o (-3.220456e-02, 8.119028e-03) => 1.674183e+00\n", "b_o (-1.044374e-01, -1.323533e-01) => 1.178925e-01\n", "b_o (-1.055425e-01, -1.247781e-01) => 8.351679e-02\n", "b_y (4.384235e-02, 8.768427e-02) => 3.333312e-01\n", "b_y (7.502261e-03, 1.500445e-02) => 3.333311e-01\n", "b_y (-1.380362e-01, -2.760797e-01) => 3.333449e-01\n", "b_y (2.693329e-01, 5.386631e-01) => 3.333312e-01\n", "b_y (3.615790e-03, 7.231543e-03) => 3.333311e-01\n", "b_y (-2.748632e-01, -5.497285e-01) => 3.333350e-01\n", "b_y (-1.380362e-01, -2.760797e-01) => 3.333449e-01\n", "b_y (-1.380362e-01, -2.760797e-01) => 3.333449e-01\n", "b_y (8.233201e-05, 1.646628e-04) => 3.333287e-01\n", "b_y (8.127466e-02, 1.625486e-01) => 3.333313e-01\n" ] } ], "source": [ "gradient_check(inputs, targets, g_h_prev, g_C_prev)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 2 }