{ "cells": [ { "cell_type": "markdown", "id": "3378c30f", "metadata": {}, "source": [ "# Introduction to RNNs: learning the alphabet\n", "\n", "A simple example to understand how Recurrent Neural Networks learn sequential patterns.\n", "\n", "## Key concepts\n", "\n", "- **Sequence learning**: RNNs process input one step at a time, maintaining a \"memory\" (hidden state) of what they've seen.\n", "- **Hidden state**: The internal memory that carries information from previous timesteps (or indexes) in a sequence.\n", "- **Next-character prediction**: Given a sequence like `['a', 'b', 'c']`, predict the next character (`'d'`).\n", "\n", "## How the RNN processes a sequence\n", "\n", "Given input `['a', 'b', 'c']`, the RNN predicts `'d'`:\n", "\n", "```text\n", "\n", " ┌─◀──────┐ 'h₁', then 'h₂'\n", " │ ▲ \n", " ▼ |\n", " ┌──────────┐\n", " | RNN | ┌───────┐\n", "'a', then 'b', ────▶| (hidden |──'h₃'──▶│ Dense │───▶ 'd'\n", " then 'c' | state h) | └───────┘ \n", " └──────────┘\n", "\n", "\n", "h₁ = RNN('a', h₀) # h₁ \"knows\" about 'a'\n", "h₂ = RNN('b', h₁) # h₂ \"knows\" about 'a', 'b' \n", "h₃ = RNN('c', h₂) # h₃ \"knows\" about 'a', 'b', 'c'\n", "output = Dense(h₃) → 'd' # Final hidden state predicts next char\n", "```\n", "\n", "The same RNN weights are used at each step within a sequence, only the hidden state changes.\n", "\n", "## Setup\n", "\n", "### Imports" ] }, { "cell_type": "code", "execution_count": 1, "id": "8b23c200", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import tensorflow as tf\n", "\n", "from keras.models import Sequential\n", "from keras.layers import Input,SimpleRNN, Dense\n", "from keras.utils import to_categorical" ] }, { "cell_type": "markdown", "id": "b40f911a", "metadata": {}, "source": [ "### Configuration" ] }, { "cell_type": "code", "execution_count": 2, "id": "80f5d901", "metadata": {}, "outputs": [], "source": [ "epochs = 200\n", "batch_size = 4" ] }, { "cell_type": "markdown", "id": "79b7f84e", "metadata": {}, "source": [ "## 1. Data preparation\n", "\n", "### 1.1. Create the alphabet dataset" ] }, { "cell_type": "code", "execution_count": 3, "id": "2168c2bd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Vocabulary: abcdefghijklmnopqrstuvwxyz\n", "\n", "Mappings:\n", "\n", " a -> 0\n", " b -> 1\n", " c -> 2\n", " d -> 3\n", " e -> 4\n", " f -> 5\n", " g -> 6\n", " h -> 7\n", " i -> 8\n", " j -> 9\n", " k -> 10\n", " l -> 11\n", " m -> 12\n", " n -> 13\n", " o -> 14\n", " p -> 15\n", " q -> 16\n", " r -> 17\n", " s -> 18\n", " t -> 19\n", " u -> 20\n", " v -> 21\n", " w -> 22\n", " x -> 23\n", " y -> 24\n", " z -> 25\n" ] } ], "source": [ "# Create alphabet mapping\n", "alphabet = 'abcdefghijklmnopqrstuvwxyz'\n", "char_to_idx = {c: i for i, c in enumerate(alphabet)}\n", "idx_to_char = {i: c for i, c in enumerate(alphabet)}\n", "vocab_size = len(alphabet)\n", "\n", "print(f'Vocabulary: {alphabet}')\n", "\n", "print('\\nMappings:\\n')\n", "\n", "for key, val in char_to_idx.items():\n", " print(f' {key} -> {val}')" ] }, { "cell_type": "markdown", "id": "381f58f6", "metadata": {}, "source": [ "### 1.2. Training 'freatures' & labels" ] }, { "cell_type": "code", "execution_count": 4, "id": "20df9dca", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Input sequence -> label\n", "('a', 'b', 'c') -> d\n", "('b', 'c', 'd') -> e\n", "('c', 'd', 'e') -> f\n", "('d', 'e', 'f') -> g\n", "('e', 'f', 'g') -> h\n", "('f', 'g', 'h') -> i\n", "('g', 'h', 'i') -> j\n", "('h', 'i', 'j') -> k\n", "('i', 'j', 'k') -> l\n", "('j', 'k', 'l') -> m\n", "('k', 'l', 'm') -> n\n", "('l', 'm', 'n') -> o\n", "('m', 'n', 'o') -> p\n", "('n', 'o', 'p') -> q\n", "('o', 'p', 'q') -> r\n", "('p', 'q', 'r') -> s\n", "('q', 'r', 's') -> t\n", "('r', 's', 't') -> u\n", "('s', 't', 'u') -> v\n", "('t', 'u', 'v') -> w\n", "('u', 'v', 'w') -> x\n", "('v', 'w', 'x') -> y\n", "('w', 'x', 'y') -> z\n", "('x', 'y', 'z') -> a\n", "X shape: (24, 3), y shape: (24,)\n" ] } ], "source": [ "seq_length = 3\n", "\n", "y = alphabet[seq_length:] + 'a' # Wrap around: 'xyz' -> 'a'\n", "x = list(zip(*[alphabet[i:] for i in range(seq_length)]))\n", "\n", "print('\\nInput sequence -> label')\n", "\n", "for features, label in zip(x, y):\n", " print(f'{features} -> {label}')\n", "\n", "print(f'X shape: {np.array(x).shape}, y shape: {np.array(list(y)).shape}')" ] }, { "cell_type": "markdown", "id": "84e9b9c8", "metadata": {}, "source": [ "### 1.3. One-hot encoding" ] }, { "cell_type": "code", "execution_count": 5, "id": "61098b28", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "First sequence (one-hot):\n", " 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0...\n", " 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0...\n", " 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0...\n", "\n", "First label (one-hot):\n", " 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0...\n", "\n", "X shape (one-hot): (24, 3, 26)\n", "y shape (one-hot): (24, 26)\n" ] } ], "source": [ "X_indices = np.array([[char_to_idx[c] for c in seq] for seq in x])\n", "y_indices = np.array([char_to_idx[c] for c in y])\n", "\n", "X_onehot = to_categorical(X_indices, num_classes=vocab_size)\n", "y_onehot = to_categorical(y_indices, num_classes=vocab_size)\n", "\n", "print('First sequence (one-hot):')\n", "\n", "for ele in X_onehot[0][:, :10]:\n", " print(f' {(\", \").join(map(str, ele))}...')\n", "\n", "print('\\nFirst label (one-hot):')\n", "print(f' {(\", \").join(map(str, y_onehot[0][:10]))}...')\n", "\n", "print(f'\\nX shape (one-hot): {X_onehot.shape}')\n", "print(f'y shape (one-hot): {y_onehot.shape}')" ] }, { "cell_type": "markdown", "id": "1d2a5f3c", "metadata": {}, "source": [ "## 2. Recurrent model\n", "\n", "A simple architecture:\n", "1. **Input**: One-hot encoded characters (26-dimensional vectors)\n", "2. **SimpleRNN**: Process the sequence, output final hidden state \n", "3. **Dense**: Predict the next character\n", "\n", "With one-hot encoding, each input character is a vector of 26 zeros with a single 1 at the character's index position.\n", "\n", "### 2.1. Define RNN" ] }, { "cell_type": "code", "execution_count": 6, "id": "81b8060e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Model: \"sequential\"\n",
       "
\n" ], "text/plain": [ "\u001b[1mModel: \"sequential\"\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ simple_rnn (SimpleRNN)          │ (None, 8)              │           280 │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense (Dense)                   │ (None, 26)             │           234 │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "
\n" ], "text/plain": [ "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n", "│ simple_rnn (\u001b[38;5;33mSimpleRNN\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m8\u001b[0m) │ \u001b[38;5;34m280\u001b[0m │\n", "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", "│ dense (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m26\u001b[0m) │ \u001b[38;5;34m234\u001b[0m │\n", "└─────────────────────────────────┴────────────────────────┴───────────────┘\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
 Total params: 514 (2.01 KB)\n",
       "
\n" ], "text/plain": [ "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m514\u001b[0m (2.01 KB)\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
 Trainable params: 514 (2.01 KB)\n",
       "
\n" ], "text/plain": [ "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m514\u001b[0m (2.01 KB)\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
 Non-trainable params: 0 (0.00 B)\n",
       "
\n" ], "text/plain": [ "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Model parameters\n", "hidden_dim = 8 # RNN hidden state size\n", "\n", "model = Sequential([\n", " Input(shape=(seq_length, vocab_size)),\n", " SimpleRNN(hidden_dim),\n", " Dense(vocab_size, activation='softmax')\n", "])\n", "\n", "model.compile(\n", " optimizer='adam',\n", " loss='categorical_crossentropy',\n", " metrics=['accuracy']\n", ")\n", "\n", "model.summary()" ] }, { "cell_type": "markdown", "id": "de0fba5a", "metadata": {}, "source": [ "### 2.2. Train RNN" ] }, { "cell_type": "code", "execution_count": 11, "id": "cb5d1af5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "CPU times: user 7.74 s, sys: 958 ms, total: 8.7 s\n", "Wall time: 6.83 s\n" ] } ], "source": [ "%%time\n", "\n", "history = model.fit(\n", " X_onehot, y_onehot,\n", " epochs=epochs,\n", " batch_size=batch_size,\n", " verbose=0\n", ")\n", "\n", "print()" ] }, { "cell_type": "markdown", "id": "46732ca6", "metadata": {}, "source": [ "### 2.3. Learning curves" ] }, { "cell_type": "code", "execution_count": 8, "id": "8e2fd47d", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAEiCAYAAADksOZKAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAWixJREFUeJzt3XlYVNUbwPHvDMsAsqqsiuKKO+6IayaKZubS4laoWZZhadhG5pL9EltcSk3TUrM0TUvLNEvJJRV3cUvNHRdAEWXfZO7vD2JqAhQRuDPwfp5nnoc5c+6d9wBzeDn33HM0iqIoCCGEEEKIu9KqHYAQQgghhDmQpEkIIYQQoggkaRJCCCGEKAJJmoQQQgghikCSJiGEEEKIIpCkSQghhBCiCCRpEkIIIYQoAkmahBBCCCGKQJImIYQQQogikKRJlJjhw4fj4+NTrGOnTJmCRqMp2YCK6EHiFkI8OHPtO0TFI0lTBaDRaIr02LZtm9qhCiFMiPQdQhjTyN5z5d8333xj9HzZsmVs3ryZr7/+2qi8e/fuuLu7F/t9srOz0ev16HS6+z72zp073LlzBxsbm2K/f3ENHz6cbdu2cfHixTJ/byFMmfQdQhiTpKkCGjNmDPPmzeNeP/q0tDTs7OzKKCr1SNIkRNFI32G+FEUhIyMDW1tbtUMxa3J5TgDw0EMP0aRJEw4ePEjnzp2xs7Pj7bffBuDHH3+kd+/eeHl5odPpqFOnDu+99x45OTlG5/jvvISLFy+i0Wj4+OOPWbhwIXXq1EGn09GmTRv2799vdGxB8xI0Gg1jxoxh3bp1NGnSBJ1OR+PGjdm0aVO++Ldt20br1q2xsbGhTp06fP755w801yE1NZXx48fj7e2NTqfD19eXjz/+ON8fi82bN9OxY0ecnZ2xt7fH19fX8H3LM2fOHBo3boydnR0uLi60bt2aFStWFCsuIUxNRe47/vjjD5588klq1KiBTqfD29ubV199lfT09Hx1T506xVNPPYWrqyu2trb4+voyYcIEozpXr15l5MiRhu9XrVq1GD16NFlZWYW2FWDp0qVoNBqjf/x8fHx49NFH+fXXX2ndujW2trZ8/vnnACxZsoSHH34YNzc3dDodjRo1Yv78+QW28ZdffqFLly44ODjg6OhImzZtDP3X5MmTsbKy4saNG/mOGzVqFM7OzmRkZNzz+2hOLNUOQJiOmzdv0qtXLwYNGsTTTz9tGG5funQp9vb2hIaGYm9vz++//86kSZNISkrio48+uud5V6xYQXJyMi+88AIajYYPP/yQAQMGcP78eaysrO567M6dO/nhhx946aWXcHBw4NNPP+Xxxx8nOjqaKlWqAHD48GF69uyJp6cn7777Ljk5OUydOhVXV9difR8UReGxxx5j69atjBw5kubNm/Prr7/y+uuvc/XqVWbNmgXAiRMnePTRR2nWrBlTp05Fp9Nx9uxZdu3aZTjXokWLeOWVV3jiiScYO3YsGRkZHD16lL179zJkyJBixSeEqamofcfq1atJS0tj9OjRVKlShX379jFnzhyuXLnC6tWrDfWOHj1Kp06dsLKyYtSoUfj4+HDu3DnWr1/P+++/D8C1a9do27Ytt2/fZtSoUTRo0ICrV6+yZs0a0tLSsLa2LlJM/3b69GkGDx7MCy+8wPPPP4+vry8A8+fPp3Hjxjz22GNYWlqyfv16XnrpJfR6PSEhIYbjly5dyrPPPkvjxo0JCwvD2dmZw4cPs2nTJoYMGcIzzzzD1KlTWbVqFWPGjDEcl5WVxZo1a3j88cfL32VTRVQ4ISEhyn9/9F26dFEAZcGCBfnqp6Wl5St74YUXFDs7OyUjI8NQNmzYMKVmzZqG5xcuXFAApUqVKkpCQoKh/Mcff1QAZf369YayyZMn54sJUKytrZWzZ88ayo4cOaIAypw5cwxlffr0Uezs7JSrV68ays6cOaNYWlrmO2dB/hv3unXrFED53//+Z1TviSeeUDQajSGeWbNmKYBy48aNQs/dt29fpXHjxveMQQhzIH3HvdsXHh6uaDQa5dKlS4ayzp07Kw4ODkZliqIoer3e8HVwcLCi1WqV/fv35ztnXr2C2qooirJkyRIFUC5cuGAoq1mzpgIomzZtKlLcQUFBSu3atQ3Pb9++rTg4OCj+/v5Kenp6oXEHBAQo/v7+Rq//8MMPCqBs3bo13/uYO7k8Jwx0Oh0jRozIV/7va+DJycnEx8fTqVMn0tLSOHXq1D3PO3DgQFxcXAzPO3XqBMD58+fveWxgYCB16tQxPG/WrBmOjo6GY3NyctiyZQv9+vXDy8vLUK9u3br06tXrnucvyMaNG7GwsOCVV14xKh8/fjyKovDLL78A4OzsDORegtDr9QWey9nZmStXruS7pCBEeVJR+45/ty81NZX4+Hjat2+PoigcPnwYgBs3brBjxw6effZZatSoYXR83qU2vV7PunXr6NOnD61bt873PsWdZlCrVi2CgoLuGndiYiLx8fF06dKF8+fPk5iYCOROPUhOTuatt97KN1r073iCg4PZu3cv586dM5QtX74cb29vunTpUqy4TZkkTcKgWrVqBQ4Bnzhxgv79++Pk5ISjoyOurq48/fTTAIYP2N38t6PI6wRv3bp138fmHZ937PXr10lPT6du3br56hVUVhSXLl3Cy8sLBwcHo/KGDRsaXofcDr1Dhw4899xzuLu7M2jQIL777jujBOrNN9/E3t6etm3bUq9ePUJCQowu3wlRHlTUviM6Oprhw4dTuXJl7O3tcXV1NSQKee3LS9KaNGlS6Hlu3LhBUlLSXesUR61atQos37VrF4GBgVSqVAlnZ2dcXV0N89Dy4s5Lgu4V08CBA9HpdCxfvtxw/M8//8zQoUPL5fpZkjQJg4Luqrh9+zZdunThyJEjTJ06lfXr17N582Y++OADgEJHWP7NwsKiwHKlCDduPsixpc3W1pYdO3awZcsWnnnmGY4ePcrAgQPp3r27YaJrw4YNOX36NCtXrqRjx458//33dOzYkcmTJ6scvRAlpyL2HTk5OXTv3p0NGzbw5ptvsm7dOjZv3szSpUuBorXvfhWWhPx3Yn2egn4u586do1u3bsTHxzNz5kw2bNjA5s2befXVV4H7j9vFxYVHH33UkDStWbOGzMxMQ3Jc3shEcHFX27Zt4+bNm/zwww907tzZUH7hwgUVo/qHm5sbNjY2nD17Nt9rBZUVRc2aNdmyZQvJyclGo015lxNq1qxpKNNqtXTr1o1u3boxc+ZMpk2bxoQJE9i6dSuBgYEAVKpUiYEDBzJw4ECysrIYMGAA77//PmFhYeVvkqQQfyvvfcexY8f466+/+OqrrwgODjaUb9682ahe7dq1ATh+/Hih53J1dcXR0fGudeCfkbbbt28bpgfAP6PfRbF+/XoyMzP56aefjEbjtm7dalQv79Lm8ePH7znyFhwcTN++fdm/fz/Lly+nRYsWNG7cuMgxmRMZaRJ3lfff2r//O8vKyuKzzz5TKyQjFhYWBAYGsm7dOq5du2YoP3v2rGHu0f165JFHyMnJYe7cuUbls2bNQqPRGOY7JCQk5Du2efPmAGRmZgK5dxX9m7W1NY0aNUJRFLKzs4sVnxDmoLz3HQW1T1EUPvnkE6N6rq6udO7cmcWLFxMdHW30Wt6xWq2Wfv36sX79eg4cOJDvvfLq5SUyO3bsMLyWmprKV199dc947xZ3YmIiS5YsMarXo0cPHBwcCA8Pz7dswH9H63r16kXVqlX54IMP2L59e7kdZQIZaRL30L59e1xcXBg2bBivvPIKGo2Gr7/+2iQuj+WZMmUKv/32Gx06dGD06NGGhKdJkyZERUXd9/n69OlD165dmTBhAhcvXsTPz4/ffvuNH3/8kXHjxhk6rqlTp7Jjxw569+5NzZo1uX79Op999hnVq1enY8eOQG7H4+HhQYcOHXB3d+fkyZPMnTuX3r1755szJUR5Ut77jgYNGlCnTh1ee+01rl69iqOjI99//32B860+/fRTOnbsSMuWLRk1ahS1atXi4sWLbNiwwfA+06ZN47fffqNLly6MGjWKhg0bEhMTw+rVq9m5cyfOzs706NGDGjVqMHLkSF5//XUsLCxYvHgxrq6u+RKywvTo0QNra2v69OnDCy+8QEpKCosWLcLNzY2YmBhDPUdHR2bNmsVzzz1HmzZtGDJkCC4uLhw5coS0tDSjRM3KyopBgwYxd+5cLCwsGDx4cJFiMUeSNIm7qlKlCj///DPjx4/nnXfewcXFhaeffppu3boVeFeGGlq1asUvv/zCa6+9xsSJE/H29mbq1KmcPHmySHfo/JdWq+Wnn35i0qRJrFq1iiVLluDj48NHH33E+PHjDfUee+wxLl68yOLFi4mPj6dq1ap06dKFd999FycnJwBeeOEFli9fzsyZM0lJSaF69eq88sorvPPOOyXWfiFMUXnvO6ysrFi/fj2vvPIK4eHh2NjY0L9/f8aMGYOfn59RXT8/P/bs2cPEiROZP38+GRkZ1KxZk6eeespQp1q1auzdu5eJEyeyfPlykpKSqFatGr169TKsrm5lZcXatWt56aWXmDhxIh4eHowbNw4XF5cC714siK+vL2vWrOGdd97htddew8PDg9GjR+Pq6sqzzz5rVHfkyJG4ubkxffp03nvvPaysrGjQoIFh/tO/BQcHM3fuXLp164anp2eRYjFHso2KKLf69evHiRMnOHPmjNqhCCHMiPQd9+/IkSM0b96cZcuW8cwzz6gdTqmROU2iXPjvtgVnzpxh48aNPPTQQ+oEJIQwC9J3lIxFixZhb2/PgAED1A6lVMnlOVEu1K5dm+HDh1O7dm0uXbrE/Pnzsba25o033lA7NCGECZO+48GsX7+eP//8k4ULFzJmzBgqVaqkdkilSi7PiXJhxIgRbN26ldjYWHQ6HQEBAUybNo2WLVuqHZoQwoRJ3/FgfHx8iIuLIygoiK+//rrc3+AiSZMQQgghRBHInCYhhBBCiCKQpEkIIYQQoggq3ERwvV7PtWvXcHBwKJebCQphihRFITk5GS8vL7Ra+V/t36RPEqLsFbdPqnBJ07Vr1/D29lY7DCEqpMuXL1O9enW1wzAp0icJoZ777ZMqXNKUN7P/8uXLODo6qhyNEBVDUlIS3t7e5f7OmuKQPkmIslfcPqnCJU15w9+Ojo7SQQlRxuTyU37SJwmhnvvtk2RygRBCCCFEEUjSJIQQQghRBJI0CSHE33bs2EGfPn3w8vJCo9Gwbt26ex6zbds2WrZsiU6no27duixdurTU4xRCqEOSJiGE+Ftqaip+fn7MmzevSPUvXLhA79696dq1K1FRUYwbN47nnnuOX3/9tZQjFUKoocJNBBdCiML06tWLXr16Fbn+ggULqFWrFjNmzACgYcOG7Ny5k1mzZhEUFFRaYQohVCJJUyGu3k5n3tazTO7TCJ2lhdrhCCFMUGRkJIGBgUZlQUFBjBs3Tp2ARLmSnJHNL8djycjOUTsUs9Lc25lm1Z1L5dySNBUgR6/w9Bd7uRCfiqONFW/1aqB2SEIIExQbG4u7u7tRmbu7O0lJSaSnp2Nra5vvmMzMTDIzMw3Pk5KSSj1OYZ7mbzvHZ9vOqR2G2XmtR31JmsqShVbDW70a8MLXB1m44xyd61elfZ2qaoclhCgHwsPDeffdd9UOQ5iBU7HJAPh5O1PdOX8CLgpW182+1M4tSVMhghp78HjL6nx/6AovLDvIdy8G0NBTFp4TQvzDw8ODuLg4o7K4uDgcHR0LHGUCCAsLIzQ01PA8b2ViIf7rckIakDty0qmeq8rRCJC75+7q/f5NaOtTmeTMOwxfso8rt9LUDkkIYUICAgKIiIgwKtu8eTMBAQGFHqPT6Qyrf8sq4KIwiqJw+e+/OTUq26kcjcgjSdNd2FhZsCi4NfXd7YlLyiR48T5upWapHZYQopSkpKQQFRVFVFQUkLukQFRUFNHR0UDuKFFwcLCh/osvvsj58+d54403OHXqFJ999hnfffcdr776qhrhi3LkRkomGdl6tBrwkktzJkOSpntwsrPiq2fb4ulkw/kbqTz71X7Ssu6oHZYQohQcOHCAFi1a0KJFCwBCQ0Np0aIFkyZNAiAmJsaQQAHUqlWLDRs2sHnzZvz8/JgxYwZffPGFLDcgHljepTlPJ1usLORPtamQOU1F4Olky7Jn2/LEgkgOR98mZPkhFga3ll9kIcqZhx56CEVRCn29oNW+H3roIQ4fPlyKUYmK6HJCOgDelWWUyZTIX/0iqufuwJfDWqOz1LL19A3eXHMUvb7wzlUIIYS4X6djkxm78jALtucuNSDzmUyLJE33obVPZT4b2hILrYYfDl9l2saTd/2vVAghhLgfn28/x49R1wzLDTTwkBsFTIkkTfepW0N3Pny8GQBf7LzA5zvOqxyREEKI8uLizVQAnm5Xg08GNWeIfw2VIxL/JnOaiuHxVtVJSM3i/Y0nmf7LKSrbWfNUG1lnRQghxIOJ/nsu06A2NWhSzUnlaMR/yUhTMT3fuTYvdKkNwFs/HGXzn3H3OEIIIYQoXHpWDvEpuVvseLvIXCZTJEnTA3irZwOeaFUdvQJjVhxi34UEtUMSQghhpvIWs3S0scTJzkrlaERBJGl6ABqNhukDmtKtgRuZd/SM/Go/J2Nk800hhBD3L29tJm+5Y85kSdL0gCwttMwd0pI2Pi4kZ9whePE+wy++EEIIUVTRCbJtiqmTieAlwNbagi+C2zBwYSSnYpN55su9rBndnqr2OrVDE0IIUQSKojD6m0NoNPDZ0JZoNBoAXl99hNikDJaOaIuFNrfs3fUn+P7glRKPIeOOHpCRJlMmSVMJydtu5fH5u7l4M43hS/bx7fPtcLCR69JCCGHqYhIz2HQiFoDryZm4O9qQlnWH1X8nRyeuJdKsujN6vcLyPdFk5ehLLZZ2tSuX2rnFg5GkqQS5O9qw7Nm2PLkgkuNXkxi17CBLRrTBxspC7dCEEELcxZVb6YavLyek4e5oY9jKBOBGcu5dbXHJGWTl6LHQavh1XGf+HnwqMfY2lrg52JTsSUWJkaSphNV2tWfpiLYMWhhJ5PmbvLoqirlDWhqGdYUQQpief89FvXwrjdY+lY3L/v46L5Gq5mxLXTf7sg1SqE7VieDz58+nWbNmODo64ujoSEBAAL/88stdj1m9ejUNGjTAxsaGpk2bsnHjxjKKtuiaVndiUXBrrC20/HI8lok/HpftVoQQwoRF/ytBir6Znr8swbhMNtKtmFRNmqpXr8706dM5ePAgBw4c4OGHH6Zv376cOHGiwPq7d+9m8ODBjBw5ksOHD9OvXz/69evH8ePHyzjye2tftyqzBzVHo4EVe6OZteWM2iEJIYQoRN4aSf/+usAyucOtQlM1aerTpw+PPPII9erVo379+rz//vvY29uzZ8+eAut/8skn9OzZk9dff52GDRvy3nvv0bJlS+bOnVvGkRfNI009ea9vEwA+jTjDV7svqhuQEEKIAl02GlVKy1d2+T9l1WXF7grJZOY05eTksHr1alJTUwkICCiwTmRkJKGhoUZlQUFBrFu3rgwiLJ6n29XkZkoWs7b8xZT1J3CpZM1jfl5qhyWEEOVeQmoW126n37sicPHmPwnSxfhUjl9N5PyNVENZdEIax68mcuZ6CiAjTRWV6knTsWPHCAgIICMjA3t7e9auXUujRo0KrBsbG4u7u7tRmbu7O7GxsYWePzMzk8zMTMPzpKSyX7H7lW51uZmaybLIS4z/LgoXOys61XMt8ziEEKKiiE/JpOMHv5ORff9LA1xPzuTROTuNytKycozKZC2likn1pMnX15eoqCgSExNZs2YNw4YNY/v27YUmTvcrPDycd999t0TOVVwajYYpfRqTkJrFz0djeOHrg3z7fDv8vJ1VjUsIIcqrs9dTyMjWY6nVFHmh4U71qqIAO8/EG8pa1HDG3dGGTcf/+efc18OBxl6OJR2yMAOqJ03W1tbUrVsXgFatWrF//34++eQTPv/883x1PTw8iIuLMyqLi4vDw8Oj0POHhYUZXdJLSkrC29u7hKIvOq1Ww8ynmpOYns0fZ+IZvmQfq19sL7esCiFEKUhMzwZy72Ze+1KHBz7flMcaP/A5hPkzub3n9Hq90eW0fwsICCAiIsKobPPmzYXOgQLQ6XSGJQ3yHmqxttQy/+lWNKvuxK20bIYt3kdMYtGutwshhCi6pL+TJidb2ZVBlBxVk6awsDB27NjBxYsXOXbsGGFhYWzbto2hQ4cCEBwcTFhYmKH+2LFj2bRpEzNmzODUqVNMmTKFAwcOMGbMGLWacN/sdZYsGd6G2lUrcfV2OiOW7Cc5I1vtsIQQolxJlKRJlAJVk6br168THByMr68v3bp1Y//+/fz66690794dgOjoaGJiYgz127dvz4oVK1i4cCF+fn6sWbOGdevW0aRJE7WaUCxV7HUsG9kWVwcdp2KTGbPiMHdKcR8jIYSoaGSkSZQGVec0ffnll3d9fdu2bfnKnnzySZ588slSiqjsVHex48thrXnq80i2/3WDKetP8F7fJoadtYUQQhRf3kiTo2yaLkqQyc1pqkiaVXfmk0Et0Gjgmz3RfLnzgtohCSFEuZCUcQeQkSZRsiRpUllQYw8mPNIQgPc3nuTXE4WvOSWEEKJoZE6TKA2SNJmAkR1rMdS/BooC41ZGcfTKbbVDEkIIs2a4PCdJkyhBkjSZAI1Gw7uPNaZLfVfSs3MY+dUBrhZx6X8hRMmaN28ePj4+2NjY4O/vz759++5af/bs2fj6+mJra4u3tzevvvoqGRkZZRStKMw/SZPqyxGKckSSJhNhaaFl7pAWNPBw4EZyJiOXylIEQpS1VatWERoayuTJkzl06BB+fn4EBQVx/fr1AuuvWLGCt956i8mTJ3Py5Em+/PJLVq1axdtvv13GkYv/krvnRGmQpMmEONhY8eXwNoalCF7+9jA5ekXtsISoMGbOnMnzzz/PiBEjaNSoEQsWLMDOzo7FixcXWH/37t106NCBIUOG4OPjQ48ePRg8ePA9R6dE6ZM5TaI0SNJkYqo52/LlsNbYWGnZdvoGH/92Wu2QhKgQsrKyOHjwIIGBgYYyrVZLYGAgkZGRBR7Tvn17Dh48aEiSzp8/z8aNG3nkkUfKJGZRsIzsHDLv5K59J0mTKElysdcENavuzAePN2PsyijmbztHYy9HHm3mpXZYQpRr8fHx5OTk4O7ublTu7u7OqVOnCjxmyJAhxMfH07FjRxRF4c6dO7z44ot3vTyXmZlptFVUUlJSyTRAGORdmtNqoJK1/JkTJUd+m0xU3+bV+PNaEp/vOM/rq49Su6o9jWRXbSFMyrZt25g2bRqfffYZ/v7+nD17lrFjx/Lee+8xceLEAo8JDw/n3XffLeNIy5+M7Bym/HSCuKT8k+7TsnKA3DvntFpZMFiUHEmaTNgbPRvwZ0wSf5yJ5/llB1j/ckcqV7JWOywhyqWqVatiYWFBXFycUXlcXBweHh4FHjNx4kSeeeYZnnvuOQCaNm1Kamoqo0aNYsKECWi1+WdAhIWFERoaanielJSEt7d3CbakYvj91HVW7r981zo1K9uVUTSiopCkyYRZaDXMGdyCvvN2celmGiHLD7FsZFusLGQqmhAlzdramlatWhEREUG/fv0A0Ov1REREFLopeFpaWr7EyMLCAgBFKfgmDp1Oh06nK7nAK6hLN9MAaOtTmSdbV8/3ukajoX2dKmUdlijnJGkycc521ix8pjX9P9tF5PmbvL/hJFMea6x2WEKUS6GhoQwbNozWrVvTtm1bZs+eTWpqKiNGjAAgODiYatWqER4eDkCfPn2YOXMmLVq0MFyemzhxIn369DEkT6J0RCfkJk3t6lThydYyUifKhiRNZsDXw4GZTzXnxW8OsnT3RZpWc+LxVvn/sxJCPJiBAwdy48YNJk2aRGxsLM2bN2fTpk2GyeHR0dFGI0vvvPMOGo2Gd955h6tXr+Lq6kqfPn14//331WpChXHlVm7S5O1iq3IkoiLRKIWNIZdTSUlJODk5kZiYiKOjeU2snvnbaT79/SzWllrWvBhAs+rOaockRJGY8+eutMn3pni6fLSVSzfTWDWqHf615TKcuD/F/dzJ5BgzMi6wPt0auJF1R88LXx/kRnLmvQ8SQohyJkevcPVW7lZT3jLZW5QhSZrMiFarYdag5tR2rURMYgYhKw5xJ0evdlhCCFGmYhLTuaNXsLbQ4u5oo3Y4ogKRpMnMONpYsfCZ1tjrLNl3IYGPfpUVw4UQ5duF+FQ6f7iVryMvAv9MAq/mYouFrMMkypAkTWaorps9Hz3RDIDPd5xn0/FYlSMSQojSM/2Xk0QnpDHxxxMAXEmQS3NCHZI0maleTT15rmMtAF5ffYQL8akqRySEEKXjVmq20fO8kSa5c06UNUmazNibvRrQxseF5Mw7jP7mIOl/bx0ghBDlVVrWHS7/vdxADRlpEmVMkiYzZmWhZe6QllS113EqNpkJ644VugqxEEKYq9vpWYavLyek/zPSJEmTKGOSNJk5d0cb5gxugVYDPxy6yrf77r4XkxBCmBNFUbj89xwmgMsJaYbnMtIkypokTeVAQJ0qvB7UAIAp609w/GqiyhEJIUTJiE/JIj37n6kHp+OSiU/JXaPO20WSJlG2ZBuVcuLFLrU5eCmBLSevE7LiED+/3BEHGyu1wxJCiCJRFIVzN1JITL9jVH7ueorR899PXQfA0cYSJzvp40TZkqSpnNBoNHz8pB+9P93JpZtpvPX9MeYOaYFGI2uYCCFM38ZjsYSsOHTPegcv3QJkPpNQh1yeK0ec7ayZM6QFlloNG47F8M2eS2qHJIQQRZKXDDnZWlGzip3Ro45rJab0aUTLGs7UrGJHbddKPNuhlsoRi4pI1ZGm8PBwfvjhB06dOoWtrS3t27fngw8+wNfXt9Bjli5dyogRI4zKdDodGRkZpR2uWWhZw4W3ejXgfxtO8t7PJ2nu7ULT6k5qhyWEEHeVd0fca0G+PNOuZoF1hkuiJFSm6kjT9u3bCQkJYc+ePWzevJns7Gx69OhBaurdF2p0dHQkJibG8Lh0SUZU/m1kx1p0b+ROVo6ekBWHSMrIvvdBQgihoiu3ZMFKYfpUHWnatGmT0fOlS5fi5ubGwYMH6dy5c6HHaTQaPDw8Sjs8s6XRaPj4CT8e+fQPohPSeHPNUT4b2lLmNwkhTJKiKIaRJllGQJgyk5rTlJiYe6t85cqV71ovJSWFmjVr4u3tTd++fTlx4kRZhGdWnOysmDe0JVYWGn45HsuySBmNE0KYpoTULNKyctBocjfhFcJUmUzSpNfrGTduHB06dKBJkyaF1vP19WXx4sX8+OOPfPPNN+j1etq3b8+VK1cKrJ+ZmUlSUpLRo6Jo7u1MWK+GAPxvw58cuyLrNwkhTM/lW7mLVXo42qCztFA5GiEKZzJJU0hICMePH2flypV3rRcQEEBwcDDNmzenS5cu/PDDD7i6uvL5558XWD88PBwnJyfDw9vbuzTCN1kjOvjQo5E72TkKY749RLLMbxJCmJh/NuCVS3PCtJlE0jRmzBh+/vlntm7dSvXq1e/rWCsrK1q0aMHZs2cLfD0sLIzExETD4/LlirXNiEaj4aMn/KjmbMulm2mE/SD70wkhTMtl2UtOmAlVkyZFURgzZgxr167l999/p1at+7+dNCcnh2PHjuHp6Vng6zqdDkdHR6NHReNkZ2VYv+nnozGs3F+xEkchhGn7J2mS+UzCtKmaNIWEhPDNN9+wYsUKHBwciI2NJTY2lvT0fzZnDA4OJiwszPB86tSp/Pbbb5w/f55Dhw7x9NNPc+nSJZ577jk1mmA2WtZw4fWg3PWvpvx0gtOxySpHJETJ8PHxYerUqURHR6sdiiimy7fk8pwwD6omTfPnzycxMZGHHnoIT09Pw2PVqlWGOtHR0cTExBie37p1i+eff56GDRvyyCOPkJSUxO7du2nUqJEaTTArz3eqzUO+rmTeyV2/KS3rzr0PEsLEjRs3jh9++IHatWvTvXt3Vq5cSWZmptphiftgWG6giiRNwrRplAo2wSUpKQknJycSExMr5KW6mymZPPLpH8QlZfJkq+p89KSf2iGJCqAsPneHDh1i6dKlfPvtt+Tk5DBkyBCeffZZWrZsWSrvV1Iqep90J0eP78RN5OgV9oR1w8PJRu2QRAVQ3M+dSUwEF2Wnir2OTwa1QKuB1Qev8MOhgpdqEMLctGzZkk8//ZRr164xefJkvvjiC9q0aUPz5s1ZvHix3ABhomISM8jRK1hbanFz0KkdjhB3JUlTBdSudhXGdqsPwDvrjnPuRorKEQnx4LKzs/nuu+947LHHGD9+PK1bt+aLL77g8ccf5+2332bo0KFqhygKkDcJvLqLLVqt7FogTJuq26gI9Yx5uC57zt8k8vxNxqw4zNqX2mNjJYvKCfNz6NAhlixZwrfffotWqyU4OJhZs2bRoEEDQ53+/fvTpk0bFaOs2E7GJPHxr6dJz87J91p8Su78M9k+RZgDGWmqoCy0Gj4Z1Jwqlaw5GZPE+xtOqh2SEMXSpk0bzpw5w/z587l69Soff/yxUcIEUKtWLQYNGlSk882bNw8fHx9sbGzw9/dn3759d61/+/ZtQkJC8PT0RKfTUb9+fTZu3Fjs9pRHX+2+SMSp6+w+dzPf46+43JHuxl4Vbz6XMD8y0lSBuTnaMHNgc4Yt3sfXey4RUKcKjzQteL0rIUzV+fPnqVmz5l3rVKpUiSVLltzzXKtWrSI0NJQFCxbg7+/P7NmzCQoK4vTp07i5ueWrn5WVRffu3XFzc2PNmjVUq1aNS5cu4ezsXNzmlEuXbuZeghve3oeWNV3yvW5jqaVzfdeyDkuI+yZJUwXXpb4rox+qw/xt53hzzVGaeDnJbb/CrFy/fp3Y2Fj8/f2Nyvfu3YuFhQWtW7cu8rlmzpzJ888/z4gRIwBYsGABGzZsYPHixbz11lv56i9evJiEhAR2796NlZUVkLtulDCWt6RAHz9PWtW8+4bsQpgyuTwnCO1en1Y1XUjOvEPIikNk3sk/70AIUxUSElLg9khXr14lJCSkyOfJysri4MGDBAYGGsq0Wi2BgYFERkYWeMxPP/1EQEAAISEhuLu706RJE6ZNm0ZOjnyG8mTn6IlJzF2wWBavFOZOkiaBlYWWOYNb4GJnxbGriUyT+U3CjPz5558FrsXUokUL/vzzzyKfJz4+npycHNzd3Y3K3d3diY2NLfCY8+fPs2bNGnJycti4cSMTJ05kxowZ/O9//yv0fTIzM0lKSjJ6lGfXbqejV0BnqcVVlhQQZk6SJgGAl7MtM59qDsBXkZfYeCzm7gcIYSJ0Oh1xcXH5ymNiYrC0LN0ZCHq9Hjc3NxYuXEirVq0YOHAgEyZMYMGCBYUeEx4ejpOTk+Hh7e1dqjGq7XLC36NMle3QaGRJAWHeJGkSBl0buPFilzoAvLnmKJdupqockRD31qNHD8LCwkhMTDSU3b59m7fffpvu3bsX+TxVq1bFwsIiXwIWFxeHh4dHgcd4enpSv359LCz+Wa6jYcOGxMbGkpWVVeAxebHmPQq6tFieGLZIkSUFRDkgE8GFkdd61OfAxQQOXLrFS8sP8f1oWb9JmLaPP/6Yzp07U7NmTVq0aAFAVFQU7u7ufP3110U+j7W1Na1atSIiIoJ+/foBuSNJERERjBkzpsBjOnTowIoVK9Dr9Wi1uf+D/vXXX3h6emJtbV3gMTqdDp2u/F6mWnPwCm+vPUbWHb1RubeLrUoRCVFyZKRJGLG00DJnSAsqV7LmxDVZv0mYvmrVqnH06FE+/PBDGjVqRKtWrfjkk084duzYfV/6Cg0NZdGiRXz11VecPHmS0aNHk5qaaribLjg4mLCwMEP90aNHk5CQwNixY/nrr7/YsGED06ZNu68J6OXN+iPX8iVMFlqNLCkgygUZaRL5eDrZMvMpP4Yv2c/Xey7RtlZl+vh5qR2WEIWqVKkSo0aNeuDzDBw4kBs3bjBp0iRiY2Np3rw5mzZtMkwOj46ONowoAXh7e/Prr7/y6quv0qxZM6pVq8bYsWN58803HzgWc3X5Vu7luAVPt6S1T+7yAjZWFtjr5M+NMH8apYLtYlnRdxS/Hx/9eop5W89hr7Nk/csdqVW1ktohCTNVFp+7P//8k+jo6HxziR577LFSeb+SUp76JL1eocGkTWTd0fPHG13xlnlMwkQV93Mnqb8o1KuB9dl/8Rb7LiTw0vJDsj+dMEnnz5+nf//+HDt2DI1GQ97/gXl3asmaSWXnenImWXf0WGg1eDrZqB2OECWuWHOaLl++zJUrVwzP9+3bx7hx41i4cGGJBSbUZ/n3+k15+9NN/bnoa94IUVbGjh1LrVq1uH79OnZ2dpw4cYIdO3bQunVrtm3bpnZ4FUrepTkvZxssLWTKrCh/ivVbPWTIELZu3QpAbGws3bt3Z9++fUyYMIGpU6eWaIBCXe6ONswa2ByNBlbsjebHqKtqhySEkcjISKZOnUrVqlXRarVotVo6duxIeHg4r7zyitrhVSjRf+8xJyt/i/KqWEnT8ePHadu2LQDfffcdTZo0Yffu3SxfvpylS5eWZHzCBHSu78qYrnUBePuHY5y7kaJyREL8IycnBwcHByB3raVr164BULNmTU6fPq1maBVO3kiTJE2ivCpW0pSdnW1YZ2TLli2GiZYNGjQgJkZWki6PxgXWp13tyqRm5RCy/BDpWTJPRJiGJk2acOTIEQD8/f358MMP2bVrF1OnTqV27doqR1exGBaylE2/RTlVrKSpcePGLFiwgD/++IPNmzfTs2dPAK5du0aVKlVKNEBhGiy0Gj4d1IKq9tacik1mwrpjVLAbL4WJeuedd9Drc9cFmjp1KhcuXKBTp05s3LiRTz/9VOXoKpYrf2+ZUl0WshTlVLHunvvggw/o378/H330EcOGDcPPzw/I3fE777KdKH/cHG34dHALnv5iLz8cukoLb2eeCfBROyxRwQUFBRm+rlu3LqdOnSIhIQEXFxfZ66yMyZYporwrVtL00EMPER8fT1JSEi4uLobyUaNGYWcnH5byrH2dqrzVqwHTNp5i6s9/0sjLkVY1K6sdlqigsrOzsbW1JSoqiiZNmhjKK1eW38mylpGdQ1xyBoCszyTKrWJdnktPTyczM9OQMF26dInZs2dz+vRp3NzcSjRAYXqe71Sb3k09yc5RGP3NIa7/3VEKUdasrKyoUaOGrMVkAq7eTkdRwM7agiqVCt53TwhzV6ykqW/fvixbtgzI3U3c39+fGTNm0K9fP+bPn1+iAQrTo9Fo+OCJZtRzs+d6ciZjlh8mO0d/7wOFKAUTJkzg7bffJiEhQe1QKrTLCf/cOSeXRUV5Vayk6dChQ3Tq1AmANWvW4O7uzqVLl1i2bJlMvKwg7HWWLHimFfY6S/ZdTCB84ym1QxIV1Ny5c9mxYwdeXl74+vrSsmVLo4coG4akSS7NiXKsWHOa0tLSDOui/PbbbwwYMACtVku7du24dOlSiQYoTFcdV3tmPOXHC18fZPGuC/h5O9G3eTW1wxIVTL9+/dQOocK5GJ/K2sNXydH/cwft3gs3AfCuLHfOifKrWElT3bp1WbduHf379zfs8A1w/fp1s99wUtyfoMYehHStw7yt53jr+2P4ejjQwEN+B0TZmTx5stohVDjvrj/B1tM3Cnyttqt9GUcjRNkp1uW5SZMm8dprr+Hj40Pbtm0JCAgAckedWrRoUeTzhIeH06ZNGxwcHHBzc6Nfv35FWsF39erVNGjQABsbG5o2bcrGjRuL0wxRQkK7+9KpXlXSs3N44euDJKZnqx2SEKIUnbmeuyvAo808Gd7ex/AY260eA1rIaLMov4o10vTEE0/QsWNHYmJiDGs0AXTr1o3+/fsX+Tzbt28nJCSENm3acOfOHd5++2169OjBn3/+SaVKlQo8Zvfu3QwePJjw8HAeffRRVqxYQb9+/Th06JDRLcei7OQtfPnonJ1cuplG6KooFgW3RquVyaCi9Gm12rtOPJY760pWdo6emMTcO2bf6d0IDycblSMSouxolAdc1vnKlSsAVK9e/YGDuXHjBm5ubmzfvp3OnTsXWGfgwIGkpqby888/G8ratWtH8+bNWbBgwT3fIykpCScnJxITE+VSYgk7fjWRx+fvJvOOnlcD6zM2sJ7aIQkTUZqfux9//NHoeXZ2NocPH+arr77i3XffZeTIkSX6fiXN3Pqk6JtpdP5oK9aWWk5N7Sn/HAmzVNzPXbFGmvR6Pf/73/+YMWMGKSm5w7QODg6MHz+eCRMmoNUW66ofiYmJwN0XpouMjCQ0NNSoLCgoiHXr1hVYPzMzk8zMTMPzpKSkYsUm7q1JNSf+168Jr685yuyIv2hSzZFuDd3VDkuUc3379s1X9sQTT9C4cWNWrVpl8kmTucnblLe6i60kTKLCKVZ2M2HCBObOncv06dM5fPgwhw8fZtq0acyZM4eJEycWKxC9Xs+4cePo0KHDXS+zxcbG4u5u/IfY3d2d2NjYAuuHh4fj5ORkeHh7excrPlE0T7b25ul2NVAUeOXbw5yOTVY7JFFBtWvXjoiICLXDKHdkqxRRkRUrafrqq6/44osvGD16NM2aNaNZs2a89NJLLFq0iKVLlxYrkJCQEI4fP87KlSuLdXxhwsLCSExMNDwuX75coucX+U3u05h2tSuTmpXDc8v2k5CapXZIooJJT0/n008/pVo1mZRc0v69iKUQFU2xLs8lJCTQoEGDfOUNGjQo1qq8Y8aM4eeff2bHjh33nBvl4eFBXFycUVlcXBweHh4F1tfpdOh0uvuOSRSflYWW+UNb0e+zXVy6mcaL3xzkm5H+WFsW77KtEHfz3415FUUhOTkZOzs7vvnmGxUjK39+OxHLZ9vOATLSJCqmYiVNfn5+zJ07N9/q33PnzqVZs2ZFPo+iKLz88susXbuWbdu2UatWrXseExAQQEREBOPGjTOUbd682bDsgTANLpWs+XJYa/rP282+CwlMXHec6Y83le0VRImbNWuW0e+VVqvF1dUVf39/ow3FxYObu/Ws4esm1ZxUjEQIdRQrafrwww/p3bs3W7ZsMSQrkZGRXL58+b7WTAoJCWHFihX8+OOPODg4GOYlOTk5YWubu6pscHAw1apVIzw8HICxY8fSpUsXZsyYQe/evVm5ciUHDhxg4cKFxWmKKEV13Rz4dEgLRi7dz6oDl6nnbs9znWqrHZYoZ4YPH652CBXGpZu5l+amD2hKu9qF37AjRHlVrOslXbp04a+//qJ///7cvn2b27dvM2DAAE6cOMHXX39d5PPMnz+fxMREHnroITw9PQ2PVatWGepER0cTExNjeN6+fXtWrFjBwoUL8fPzY82aNaxbt07WaDJRXX3dePuRhgBM23iSraevqxyRKG+WLFnC6tWr85WvXr2ar776SoWIyqfE9GzDwrV9/Lxk1FhUSA+8TtO/HTlyhJYtW5r0YnLmtiZKeaAoCm99f4xVBy5jr7Pkh5faU9/dQe2wRBkqzc9d/fr1+fzzz+natatR+fbt2xk1alSRdhlQk7n0ScevJvLonJ1UtbfmwDvd1Q5HiAdS3M+dzMwVpU6j0fBevya0rVWZlMw7PLt0PzeSM+99oBBFEB0dXeB8yJo1axIdHa1CROXTFcP6TDIBXFRckjSJMmFtqWXB062oWcWOK7fSeX7ZATKyTXdEUpgPNzc3jh49mq/8yJEjVKlSRYWIyidZn0kISZpEGapcyZolw9vgZGtF1OXbhH4XhV5fYleHRQU1ePBgXnnlFbZu3UpOTg45OTn8/vvvjB07lkGDBqkdXrlx7noqAN6VbVWORAj13NfdcwMGDLjr67dv336QWEQFUNvVnoXPtOLpL/ey8VgsH1U5zZs986/5JURRvffee1y8eJFu3bphaZnbpen1eoKDg5k2bdp9n2/evHl89NFHxMbG4ufnx5w5c2jbtu09j1u5ciWDBw+mb9++hW7rZK4+/vU0qw7kLgwsI02iIruvpMnJ6e7rcjg5OREcHPxAAYnyz792FT58ohmvrjrC/G3nqO5iy1D/mmqHJcyUtbU1q1at4n//+x9RUVHY2trStGlTata8/9+pVatWERoayoIFC/D392f27NkEBQVx+vRp3NzcCj3u4sWLvPbaa3Tq1OlBmmKytpz8Z0HhdrXlkqeouEr07jlzYC53qlQEszb/xScRZ9BoYM7gFjzazEvtkEQpMZfPnb+/P23atGHu3LlA7oiVt7c3L7/8Mm+99VaBx+Tk5NC5c2eeffZZ/vjjD27fvn1fI03m8L0JCI8gJjGD70e3p1VNWTBUmD+5e06YnXGB9Qyb+766KoptsoaTKIbHH3+cDz74IF/5hx9+yJNPPlnk82RlZXHw4EECAwMNZVqtlsDAQCIjIws9burUqbi5uTFy5MgivU9mZiZJSUlGD1OXtz5TVXtrlSMRQl2SNAnVaDQapj7WhMf8vMjOUXjxm4McuHj/exeKim3Hjh088sgj+cp79erFjh07inye+Ph4cnJycHd3Nyp3d3c37FbwXzt37uTLL79k0aJFRX6f8PBwnJycDA9vb+8iH6uG7Bw9aVm5d7o62lipHI0Q6pKkSahKq9Uw4yk/uvq6kpGtZ8TS/fx5zfT/8xamIyUlBWvr/CMgVlZWpTqKk5yczDPPPMOiRYuoWrVqkY8LCwsjMTHR8Lh8+XKpxVgSkv4eZQJwtJWkSVRskjQJ1VlZaPlsaCva+lQmOeMOwYv3ciE+Ve2whJlo2rSp0dZLeVauXEmjRo2KfJ6qVatiYWFBXFycUXlcXBweHh756p87d46LFy/Sp08fLC0tsbS0ZNmyZfz0009YWlpy7ty5At9Hp9Ph6Oho9DBleZfmHHSWWGhl6xRRsRVrw14hSpqttQVfDG/NoM/38GdMEk9/sZc1owPwdJI1YcTdTZw4kQEDBnDu3DkefvhhACIiIlixYgVr1qwp8nmsra1p1aoVERER9OvXD8idCB4REcGYMWPy1W/QoAHHjh0zKnvnnXdITk7mk08+MfnLbkWVlzTJKJMQkjQJE+JoY8WykW15akEk5+NTeebLfXz3QgCVK8nkU1G4Pn36sG7dOqZNm8aaNWuwtbXFz8+P33//ncqVK9/XuUJDQxk2bBitW7embdu2zJ49m9TUVEaMGAFAcHAw1apVIzw8HBsbm3wbhTs7OwOUqw3EkzLuAJI0CQFyeU6YmKr2Or5+zh9PJxvOXk9h+JJ9JGdk3/tAUaH17t2bXbt2kZqayvnz53nqqad47bXX8PPzu6/zDBw4kI8//phJkybRvHlzoqKi2LRpk2FyeHR0NDExMaXRBJOVN9LkZCv/Ywsh6zQJk3T2egpPfR5JQmoW7WpXZumItthYWagdliimsvjc7dixgy+//JLvv/8eLy8vBgwYwOOPP06bNm1K5f1Kiqn3SV/vucTEdccJauzO58+0VjscIUqErNMkypW6bvZ8NaIt9jpL9pxPYMyKQ2Tn6NUOS5iY2NhYpk+fTr169XjyySdxdHQkMzOTdevWMX36dJNPmMxB3t1zstyAEJI0CRPWtLoTXwxrjc5Sy5aT13ljzVHZ4FcY9OnTB19fX44ePcrs2bO5du0ac+bMUTuscifJcHlOkiYhJGkSJq1d7Sp8NrQlFloNaw9f5Z0fj0viJAD45ZdfGDlyJO+++y69e/fGwkIu35aGREmahDCQpEmYvG4N3ZnxpB8aDazYG81bPxwlRxKnCm/nzp0kJyfTqlUr/P39mTt3LvHx8WqHVe4YkiY7SZqEkKRJmIV+Laox8yk/tBr47sAVQr+L4o7McarQ2rVrx6JFi4iJieGFF15g5cqVeHl5odfr2bx5M8nJyWqHaNaOXUkkfONJjl5JBGROkxAgSZMwI/1bVGfO4JZYajX8GHWNl789TNYdSZwqukqVKvHss8+yc+dOjh07xvjx45k+fTpubm489thjaodntt78/iif7zjP1dvpALg72qgckRDqk6RJmJXezTyZ/3QrrC20/HI8lpeWHyTzTo7aYQkT4evry4cffsiVK1f49ttv1Q7HbCmKwvn4FACebleDKX0a4V/r/hYKFaI8kqRJmJ3ujdxZGNzKcFfd88sOkpEtiZP4h4WFBf369eOnn35SOxSzdCMlk4xsPVoNTO7TmOEdaqGVfeeEkKRJmKeHfN1YMrwNtlYW7PjrBiOW7Cct647aYQlRLlxOSAPA08kWKwv5MyFEHvk0CLPVvm5Vlo3MXQAz8vxNhi2WLVeEKAmXE3LnMdWobKdyJEKYFkmahFlr41OZr0e2xcHGkv0Xb/H0l/tITJPESYgHEf33SJN3ZVuVIxHCtEjSJMxeixoufPt8O5ztrDhy+TYDF0YSl5ShdlhCmK28y3My0iSEMVWTph07dtCnTx+8vLzQaDSsW7furvW3bduGRqPJ94iNjS2bgIXJalLNiZWj2lHVXsep2GQGfLabczdS1A5LCLPy9Z5LPDrnDzYdz+1TvSVpEsKIqklTamoqfn5+zJs3776OO336NDExMYaHm5tbKUUozEkDD0d+GN0enyp2XL2dzhPzd3M4+pbaYQlhNj7ZcobjV5NIzryDRpP7z4gQ4h+War55r1696NWr130f5+bmhrOzc8kHJMxejSp2rBndnpFL93PkSiJDFu1l3tAWPNzAXe3QhDBpaVl3iE/JBGDB062o41qJOq72KkclhGkxyzlNzZs3x9PTk+7du7Nr16671s3MzCQpKcnoIcq3qvY6Vjzfjs71XUnPzuH5ZQdZvveS2mEJYdKu3Mq9Y87RxpKeTTyo5+6gckRCmB6zSpo8PT1ZsGAB33//Pd9//z3e3t489NBDHDp0qNBjwsPDcXJyMjy8vb3LMGKhlko6S74c1ponWlUnR68wYe1xpv9yCr1s9CtEgaJv/j35u4rMYxKiMKpenrtfvr6++Pr6Gp63b9+ec+fOMWvWLL7++usCjwkLCyM0NNTwPCkpSRKnCsLKQstHTzSjRmU7Zm7+iwXbz3H5VhoznvTDxspC7fCEMCmXb/29zICLJE1CFMasRpoK0rZtW86ePVvo6zqdDkdHR6OHqDg0Gg2vdKvHzKf8sLLQsOFoDAM/j+S6LEkghJFoWWZAiHsy+6QpKioKT09PtcMQJm5Ay+p8PdI/dy2nK4n0nbeL41cT1Q5LCNUpikJiejYX41MBqC5JkxCFUvXyXEpKitEo0YULF4iKiqJy5crUqFGDsLAwrl69yrJlywCYPXs2tWrVonHjxmRkZPDFF1/w+++/89tvv6nVBGFG2tWuwo8hHXh26X7O3UjliQW7+fAJPx7z81I7NCFU8/K3h/n5aIzhuYw0CVE4VUeaDhw4QIsWLWjRogUAoaGhtGjRgkmTJgEQExNDdHS0oX5WVhbjx4+nadOmdOnShSNHjrBlyxa6deumSvzC/NSsUokfXupAp3pVycjW88q3h5m28SR3cvRqhyaEKrafvmH4urqLLc29ndULRggTp1EUpULdTpSUlISTkxOJiYkyv6kCy9ErfPzbaeZvOwdAh7pVmDu4JS6VrFWOrHySz13h1Pze5OgV6ry9EYA9Yd1wd9Sh0WjKNAYh1FDcz53Zz2kSojgstBre7NmAeUNaYmdtwa6zN+kzdycnrsk8J1FxJGf8s7l1FXtrSZiEuAdJmkSF1ruZJ2tf6kDNKnZcuZXO4/N382PUVbXDEqJMJKbnJk121hZYWcifAyHuRT4losLz9XDgp5COdKnvSka2nrEro3h3/Qmy7sg8J1G+5SVNjjZWKkcihHmQpEkIwMnOisXD2xDStQ4AS3ZdZMD8XZy/kaJyZKKszZs3Dx8fH2xsbPD392ffvn2F1l20aBGdOnXCxcUFFxcXAgMD71rf1CSl3wHAyVaSJiGKQpImIf5modXwelADFgW3xtnOiuNXk3h0zk6+O3CZCna/RIW1atUqQkNDmTx5MocOHcLPz4+goCCuX79eYP1t27YxePBgtm7dSmRkJN7e3vTo0YOrV83jEm/eSJMkTUIUjSRNQvxH90bubBrbmYDaVUjLyuGNNUcZ8+1hwx8YUX7NnDmT559/nhEjRtCoUSMWLFiAnZ0dixcvLrD+8uXLeemll2jevDkNGjTgiy++QK/XExERUcaRF4/h8pwkTUIUiSRNQhTAw8mGb57z542evlhqc7dfeeSTPzhwMUHt0EQpycrK4uDBgwQGBhrKtFotgYGBREZGFukcaWlpZGdnU7ly5ULrZGZmkpSUZPRQyz9Jk1ltQyqEaiRpEqIQFloNLz1UlzWj21Ojsh1Xb6fz1OeRfLLljCyGWQ7Fx8eTk5ODu7u7Ubm7uzuxsbFFOsebb76Jl5eXUeL1X+Hh4Tg5ORkeam4gnpQhl+eEuB+SNAlxD829ndnwSkf6t6iGXoFZW/5i8KI9XL2drnZowoRMnz6dlStXsnbtWmxsbAqtFxYWRmJiouFx+fLlMozSmMxpEuL+SNIkRBE42Fgxa2BzZg30w15nyf6Lt+g1ewcbj8Xc+2BhFqpWrYqFhQVxcXFG5XFxcXh4eNz12I8//pjp06fz22+/0axZs7vW1el0ODo6Gj3UIkmTEPdHkiYh7kP/FtXZ8EpH/LydScq4w0vLD/HmmqNGKysL82RtbU2rVq2MJnHnTeoOCAgo9LgPP/yQ9957j02bNtG6deuyCLXEJMk6TULcF0mahLhPNatUYs2LAYR0rYNGA6sOXCZo1g62/3Xj3gcLkxYaGsqiRYv46quvOHnyJKNHjyY1NZURI0YAEBwcTFhYmKH+Bx98wMSJE1m8eDE+Pj7ExsYSGxtLSop5rO+VJCNNQtwXuWVCiGKwstDyelADOtVz5Y01R4lOSGPY4n081bo6E3o3kj9CZmrgwIHcuHGDSZMmERsbS/Pmzdm0aZNhcnh0dDRa7T//a86fP5+srCyeeOIJo/NMnjyZKVOmlGXoRfL7qTi2nf4nub8QnwrkLu4qhLg3jVLBVu2T3dZFSUvLusNHv55m6e6LKAp4ONowbUATHm7gfu+DKwj53BWurL43OXqFplN+JS0rJ99rf7zRFe/KdqX23kKYmuJ+7mSkSYgHZGdtyeQ+jXmkqSdvrDnKhfhUnl16gAEtqjGpTyOc7azVDlEI4pIySMvKwVKr4aWudQ3l9d3tJWESoogkaRKihLTxqczGVzoxc/Npvtx5gR8OX2XHmXgm9WlEn2aeaDQatUMUFVh0QhoA1VxsCe1eX+VohDBPMhFciBJka23BhN6NWDO6PXVcKxGfkskr3x4mePE+Lv49f0QINVz+O2nydpFRJSGKS5ImIUpByxoubBzbidDu9bG21PLHmXh6zN7BpxFnyLyTf06JEKXNkDTJpTghik2SJiFKic7Sgle61eO3cZ3pVK8qWXf0zNz8F71m/8Hus/FqhycqmMu3clew965sq3IkQpgvSZqEKGU+VSux7Nm2zBncAlcHHefjUxnyxV7GrTxMXFKG2uGJCiJvTlMNGWkSotgkaRKiDGg0Gvr4eRExvgvDAmqi0cC6qGt0/Xgb87aeJSNbLtmJ0vH+hj8JnLmdo1duAzKnSYgHIUmTEGXI0caKd/s24ceQDrSs4UxaVg4f/XqabjO2s+FoDBVs2TRRypIzsln0xwXOXk8hO0fBQWdJHTd7tcMSwmxJ0iSECppVd+b70e35ZFBzPJ1suHo7nZAVhxj4+R6OX01UOzxRTuRdknOytWLlqHZEjO+CvU5WmhGiuCRpEkIlGo2Gvs2rETG+C2O71cPGSsu+iwn0mbuTN9Yc4XqyzHcSD+ZyQu7kb58qdrSrXQU3RxuVIxLCvEnSJITK7KwtebV7fX4f/xB9m3uhKPDdgSt0/Wgbs7f8RXJGttohCjMlywwIUbIkaRLCRHg52/LJoBZ8P7o9ftWdSM3KYfaWM3T+cCuLdpyXyeLivl2+JUmTECVJ1aRpx44d9OnTBy8vLzQaDevWrbvnMdu2baNly5bodDrq1q3L0qVLSz1OIcpSq5ourH2pA/OGtKS2ayVupWXz/saTdPloK8v3XiI7R692iMJMyDIDQpQsVZOm1NRU/Pz8mDdvXpHqX7hwgd69e9O1a1eioqIYN24czz33HL/++mspRypE2dJqNfRu5slv4zrz4RPNqOZsS1xSJhPWHidw5nbWHb5Kjl7utBN3J1unCFGyVL2NolevXvTq1avI9RcsWECtWrWYMWMGAA0bNmTnzp3MmjWLoKCg0gpTCNVYWmh5qrU3fZt78e3eaOZuPculm2mMWxXF/G3nGN+jPt0buctmwCIfvV7hyt+rgMtIkxAlw6zmNEVGRhIYGGhUFhQURGRkpEoRCVE2dJYWDO9Qi+2vd+X1IF8cbSw5HZfMqK8P0v+z3bIti8jnRkommXf0aDXg6Sx3zQlREswqaYqNjcXd3d2ozN3dnaSkJNLT0ws8JjMzk6SkJKOHEOaqks6SkK51+eONhwnpWgdbKwuiLt9myBd7GbJoD/svJqgdojAReZfmvJxtsbIwq65eCJNV7j9J4eHhODk5GR7e3t5qhyTEA3Oys+L1oAbseKMrw9v7YG2hZfe5mzy5IJKBn0ey80y8rC5ewUXLfCYhSpxZJU0eHh7ExcUZlcXFxeHo6IitbcE7d4eFhZGYmGh4XL58uSxCFaJMuDromPJYY35/rQuD23pjZaFh74UEnv5yL/0/282WP+Mkeaqg8ha29K5ccN8ohLh/ZpU0BQQEEBERYVS2efNmAgICCj1Gp9Ph6Oho9BCivKnuYkf4gGaGkSedpZaoy7d5btkBHvl0JxuOxsjddhWMLDcgRMlTNWlKSUkhKiqKqKgoIHdJgaioKKKjo4HcUaLg4GBD/RdffJHz58/zxhtvcOrUKT777DO+++47Xn31VTXCF8LkeDrZMuWxxux882Fe7FKHStYWnIxJImTFIQJnbuebPZdIz5JFMisCWdhSiJKn6pIDBw4coGvXrobnoaGhAAwbNoylS5cSExNjSKAAatWqxYYNG3j11Vf55JNPqF69Ol988YUsNyDEf7g66HirVwNe7FKbJbsusnT3RS7Ep/LOuuPM3PwXT7erSXBATara69QOVZSgv+KSORmTe7PL+RspgCRNQpQkjVLBJjwkJSXh5OREYmKiXKoTFUZq5h1WH7jMFzsvGNbusbbU8njL6jzXqRZ1XO1L9f3lc1e4kvrepGXdoc3/tpD6n5HEA+8ESnIsxH8U93On6kiTEKJsVNJZMrxDLZ5uV5NfT8SxcMc5jlxJ5Nt90Xy7L5rAhu4829GHgNpVZKFMM3UxPo3UrBx0llra+FQGwL9WZUmYhChBkjQJUYFYWmjp3cyTR5p6sP/iLRbuOM+Wk3GGRz03e54JqEn/FtVwsLFSO1xxH/ImfjfwdOSb5/xVjkaI8kmSJiEqII1GQ9talWlbqzLnbqSweOcF1h6+ypnrKUz68QQf/HKK/i2rERzgQ313B7XDFUVw5ZbcLSdEaTOrJQeEECWvjqs97/dvyp63uzGlTyPquFYiNSuHb/ZE02PWDgZ+HsmGozFk5+jVDrVMzJs3Dx8fH2xsbPD392ffvn13rb969WoaNGiAjY0NTZs2ZePGjWUUqbF/FrOUdZmEKC2SNAkhAHC0sWJ4h1psCe3C8uf86dnYAwtt7mKZISsO0WH678za/BdxSRlqh1pqVq1aRWhoKJMnT+bQoUP4+fkRFBTE9evXC6y/e/duBg8ezMiRIzl8+DD9+vWjX79+HD9+vIwj/2fbFBlpEqL0yN1zQohCxSSms2JvNN/uu0x8SiYAlloNQY09GNquRpEnjpvL587f3582bdowd+5cAPR6Pd7e3rz88su89dZb+eoPHDiQ1NRUfv75Z0NZu3btaN68OQsWLCjSe5bU96bbjG2cu5HK8uf86VC3arHPI0RFIHfPCSFKnKeTLeN7+PLyw/X45XgM3+y5xP6Lt9hwLIYNx2LwqWLHU228ebpdTRzNfOJ4VlYWBw8eJCwszFCm1WoJDAwkMjKywGMiIyMN68vlCQoKYt26dYW+T2ZmJpmZmYbnRdlE/NrtdD7ZcuaudfK2TZGRJiFKjyRNQoh7srbU0rd5Nfo2r8af15L4Zu8lfjx8lYs30/hkyxmGtq2pdogPLD4+npycHNzd3Y3K3d3dOXXqVIHHxMbGFlg/Nja20PcJDw/n3Xffva/Ybqdls+rAvffNrGRtgaeTzX2dWwhRdJI0CSHuSyMvR6b1b8qERxqy4VgMN5IzcbIz71GmshQWFmY0OpWUlIS3t/ddj3F10PF6kO89z+1fqzKWFjJVVYjSIkmTEKJYKuksear13f/Ym5OqVatiYWFBXFycUXlcXBweHh4FHuPh4XFf9SF3E3Gd7v4WnHR10BHSte59HSOEKHnyL4kQQgDW1ta0atWKiIgIQ5leryciIoKAgIACjwkICDCqD7B58+ZC6wshzJuMNAkhxN9CQ0MZNmwYrVu3pm3btsyePZvU1FRGjBgBQHBwMNWqVSM8PByAsWPH0qVLF2bMmEHv3r1ZuXIlBw4cYOHChWo2QwhRSiRpEkKIvw0cOJAbN24wadIkYmNjad68OZs2bTJM9o6Ojkar/WeAvn379qxYsYJ33nmHt99+m3r16rFu3TqaNGmiVhOEEKVI1mkSQpQ6+dwVTr43QpS94n7uZE6TEEIIIUQRSNIkhBBCCFEEFW5OU97VyKKswiuEKBl5n7cKNhugSKRPEqLsFbdPqnBJU3JyMsA9F5MTQpS85ORknJyc1A7DpEifJIR67rdPqnATwfV6PdeuXcPBweGuG43mrdJ7+fJls5+cKW0xPeWlHVC0tiiKQnJyMl5eXkZ3nwnpk8y5LeWlHVB+2lLUdhS3T6pwI01arZbq1asXub6jo6NZ/wL9m7TF9JSXdsC92yIjTAWTPsn821Je2gHlpy1FaUdx+iT5l08IIYQQoggkaRJCCCGEKAJJmgqh0+mYPHnyfW+saYqkLaanvLQDyldbTFl5+j6Xl7aUl3ZA+WlLabejwk0EF0IIIYQoDhlpEkIIIYQoAkmahBBCCCGKQJImIYQQQogikKSpEPPmzcPHxwcbGxv8/f3Zt2+f2iHd1ZQpU9BoNEaPBg0aGF7PyMggJCSEKlWqYG9vz+OPP05cXJyKEf9jx44d9OnTBy8vLzQaDevWrTN6XVEUJk2ahKenJ7a2tgQGBnLmzBmjOgkJCQwdOhRHR0ecnZ0ZOXIkKSkpZdiKXPdqy/Dhw/P9nHr27GlUxxTaEh4eTps2bXBwcMDNzY1+/fpx+vRpozpF+Z2Kjo6md+/e2NnZ4ebmxuuvv86dO3fKsinlgrn1RyB9kil8jqU/Kvn+SJKmAqxatYrQ0FAmT57MoUOH8PPzIygoiOvXr6sd2l01btyYmJgYw2Pnzp2G11599VXWr1/P6tWr2b59O9euXWPAgAEqRvuP1NRU/Pz8mDdvXoGvf/jhh3z66acsWLCAvXv3UqlSJYKCgsjIyDDUGTp0KCdOnGDz5s38/PPP7Nixg1GjRpVVEwzu1RaAnj17Gv2cvv32W6PXTaEt27dvJyQkhD179rB582ays7Pp0aMHqamphjr3+p3Kycmhd+/eZGVlsXv3br766iuWLl3KpEmTyrQt5s5c+yOQPkntz7H0R6XQHykin7Zt2yohISGG5zk5OYqXl5cSHh6uYlR3N3nyZMXPz6/A127fvq1YWVkpq1evNpSdPHlSAZTIyMgyirBoAGXt2rWG53q9XvHw8FA++ugjQ9nt27cVnU6nfPvtt4qiKMqff/6pAMr+/fsNdX755RdFo9EoV69eLbPY/+u/bVEURRk2bJjSt2/fQo8x1bZcv35dAZTt27crilK036mNGzcqWq1WiY2NNdSZP3++4ujoqGRmZpZtA8yYOfZHiiJ9kql9jqU/Kpn+SEaa/iMrK4uDBw8SGBhoKNNqtQQGBhIZGaliZPd25swZvLy8qF27NkOHDiU6OhqAgwcPkp2dbdSmBg0aUKNGDZNv04ULF4iNjTWK3cnJCX9/f0PskZGRODs707p1a0OdwMBAtFote/fuLfOY72Xbtm24ubnh6+vL6NGjuXnzpuE1U21LYmIiAJUrVwaK9jsVGRlJ06ZNcXd3N9QJCgoiKSmJEydOlGH05suc+yOQPsnUPscFkf7o/vojSZr+Iz4+npycHKNvLIC7uzuxsbEqRXVv/v7+LF26lE2bNjF//nwuXLhAp06dSE5OJjY2Fmtra5ydnY2OMfU2AYb47vbziI2Nxc3Nzeh1S0tLKleubHLt69mzJ8uWLSMiIoIPPviA7du306tXL3JycgDTbIter2fcuHF06NCBJk2aABTpdyo2NrbAn1vea+LezLU/AumTTO1zXBDpj+6/P6pwG/aWV7169TJ83axZM/z9/alZsybfffcdtra2KkYm/m3QoEGGr5s2bUqzZs2oU6cO27Zto1u3bipGVriQkBCOHz9uNB9FiHuRPsn0SX90/2Sk6T+qVq2KhYVFvln3cXFxeHh4qBTV/XN2dqZ+/fqcPXsWDw8PsrKyuH37tlEdc2hTXnx3+3l4eHjkmxR7584dEhISTL59tWvXpmrVqpw9exYwvbaMGTOGn3/+ma1bt1K9enVDeVF+pzw8PAr8ueW9Ju6tvPRHIH2SOfRJ0h/dmyRN/2FtbU2rVq2IiIgwlOn1eiIiIggICFAxsvuTkpLCuXPn8PT0pFWrVlhZWRm16fTp00RHR5t8m2rVqoWHh4dR7ElJSezdu9cQe0BAALdv3+bgwYOGOr///jt6vR5/f/8yj/l+XLlyhZs3b+Lp6QmYTlsURWHMmDGsXbuW33//nVq1ahm9XpTfqYCAAI4dO2bU6W7evBlHR0caNWpUNg0xc+WlPwLpk8yhT5L+qGjBiP9YuXKlotPplKVLlyp//vmnMmrUKMXZ2dlo1r2pGT9+vLJt2zblwoULyq5du5TAwEClatWqyvXr1xVFUZQXX3xRqVGjhvL7778rBw4cUAICApSAgACVo86VnJysHD58WDl8+LACKDNnzlQOHz6sXLp0SVEURZk+fbri7Oys/Pjjj8rRo0eVvn37KrVq1VLS09MN5+jZs6fSokULZe/evcrOnTuVevXqKYMHDzaptiQnJyuvvfaaEhkZqVy4cEHZsmWL0rJlS6VevXpKRkaGSbVl9OjRipOTk7Jt2zYlJibG8EhLSzPUudfv1J07d5QmTZooPXr0UKKiopRNmzYprq6uSlhYWJm2xdyZY3+kKNInmcLnWPqjku+PJGkqxJw5c5QaNWoo1tbWStu2bZU9e/aoHdJdDRw4UPH09FSsra2VatWqKQMHDlTOnj1reD09PV156aWXFBcXF8XOzk7p37+/EhMTo2LE/9i6dasC5HsMGzZMUZTcW3wnTpyouLu7KzqdTunWrZty+vRpo3PcvHlTGTx4sGJvb684OjoqI0aMUJKTk02qLWlpaUqPHj0UV1dXxcrKSqlZs6by/PPP5/vjZwptKagNgLJkyRJDnaL8Tl28eFHp1auXYmtrq1StWlUZP368kp2dXaZtKQ/MrT9SFOmTTOFzLP1RyfdHmr8DEkIIIYQQdyFzmoQQQgghikCSJiGEEEKIIpCkSQghhBCiCCRpEkIIIYQoAkmahBBCCCGKQJImIYQQQogikKRJCCGEEKIIJGkSQgghhCgCSZpEuafRaFi3bp3aYQghBCB9kjmTpEmUquHDh6PRaPI9evbsqXZoQogKSPok8SAs1Q5AlH89e/ZkyZIlRmU6nU6laIQQFZ30SaK4ZKRJlDqdToeHh4fRw8XFBcgdpp4/fz69evXC1taW2rVrs2bNGqPjjx07xsMPP4ytrS1VqlRh1KhRpKSkGNVZvHgxjRs3RqfT4enpyZgxY4xej4+Pp3///tjZ2VGvXj1++umn0m20EMJkSZ8kikuSJqG6iRMn8vjjj3PkyBGGDh3KoEGDOHnyJACpqakEBQXh4uLC/v37Wb16NVu2bDHqgObPn09ISAijRo3i2LFj/PTTT9StW9foPd59912eeuopjh49yiOPPMLQoUNJSEgo03YKIcyD9EmiUIoQpWjYsGGKhYWFUqlSJaPH+++/ryiKogDKiy++aHSMv7+/Mnr0aEVRFGXhwoWKi4uLkpKSYnh9w4YNilarVWJjYxVFURQvLy9lwoQJhcYAKO+8847heUpKigIov/zyS4m1UwhhHqRPEg9C5jSJUte1a1fmz59vVFa5cmXD1wEBAUavBQQEEBUVBcDJkyfx8/OjUqVKhtc7dOiAXq/n9OnTaDQarl27Rrdu3e4aQ7NmzQxfV6pUCUdHR65fv17cJgkhzJj0SaK4JGkSpa5SpUr5hqZLiq2tbZHqWVlZGT3XaDTo9frSCEkIYeKkTxLFJXOahOr27NmT73nDhg0BaNiwIUeOHCE1NdXw+q5du9Bqtfj6+uLg4ICPjw8RERFlGrMQovySPkkURkaaRKnLzMwkNjbWqMzS0pKqVasCsHr1alq3bk3Hjh1Zvnw5+/bt48svvwRg6NChTJ48mWHDhjFlyhRu3LjByy+/zDPPPIO7uzsAU6ZM4cUXX8TNzY1evXqRnJzMrl27ePnll8u2oUIIsyB9kig2tSdVifJt2LBhCpDv4evrqyhK7oTIefPmKd27d1d0Op3i4+OjrFq1yugcR48eVbp27arY2NgolStXVp5//nklOTnZqM6CBQsUX19fxcrKSvH09FRefvllw2uAsnbtWqP6Tk5OypIlS0qlzUII0yV9kngQGkVRFDWSNSEg9zr+2rVr6devn9qhCCGE9EnirmROkxBCCCFEEUjSJIQQQghRBHJ5TgghhBCiCGSkSQghhBCiCCRpEkIIIYQoAkmahBBCCCGKQJImIYQQQogikKRJCCGEEKIIJGkSQgghhCgCSZqEEEIIIYpAkiYhhBBCiCKQpEkIIYQQogj+D1FPBim3xfHZAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig, axes = plt.subplots(1, 2, figsize=(6, 3))\n", "\n", "axes[0].plot(history.history['loss'])\n", "axes[0].set_title('Training loss')\n", "axes[0].set_xlabel('Epoch')\n", "axes[0].set_ylabel('Loss')\n", "\n", "axes[1].plot(history.history['accuracy'])\n", "axes[1].set_title('Training accuracy')\n", "axes[1].set_xlabel('Epoch')\n", "axes[1].set_ylabel('Accuracy')\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "25bcbaad", "metadata": {}, "source": [ "## 3. Model evaluation" ] }, { "cell_type": "code", "execution_count": 9, "id": "b947b44f", "metadata": {}, "outputs": [], "source": [ "def predict_next(chars, model):\n", " '''Predict the next character given a sequence.'''\n", "\n", " indices = np.array([[char_to_idx[c] for c in chars]])\n", " onehot = to_categorical(indices, num_classes=vocab_size)\n", " pred = model.predict(onehot, verbose=0)\n", " print(f' Logits: {pred}')\n", " next_idx = np.argmax(pred)\n", "\n", " return idx_to_char[next_idx]" ] }, { "cell_type": "code", "execution_count": 10, "id": "3519832f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Predictions:\n", " Logits: [[0.01264508 0.00741717 0.00480832 0.4269097 0.02531913 0.00292493\n", " 0.01553659 0.06030714 0.00229446 0.00290143 0.00359587 0.01262618\n", " 0.00353318 0.00144151 0.00928345 0.00776676 0.07203305 0.00402647\n", " 0.03111149 0.05916719 0.09306646 0.0837661 0.0180071 0.00439138\n", " 0.02711562 0.00800418]]\n", " abc -> d\n", "\n", " Logits: [[0.00313391 0.00366649 0.00605078 0.01086417 0.0188335 0.00351843\n", " 0.4486534 0.00798101 0.00504046 0.0076088 0.02736166 0.06016673\n", " 0.00899895 0.01515133 0.04818705 0.05538888 0.00997268 0.00113339\n", " 0.00906708 0.09369843 0.00614864 0.00053398 0.12510487 0.01431022\n", " 0.00073657 0.00868867]]\n", " def -> g\n", "\n", " Logits: [[0.00784801 0.00407867 0.00498702 0.00964788 0.00912741 0.00202328\n", " 0.01949766 0.01245422 0.01269589 0.00172639 0.00372224 0.01604388\n", " 0.00449361 0.00175741 0.03844464 0.56684625 0.06987028 0.003383\n", " 0.01054794 0.05695849 0.0158059 0.00069931 0.01520741 0.09765687\n", " 0.00941778 0.00505864]]\n", " mno -> p\n", "\n", " Logits: [[6.6283241e-02 5.5090124e-03 3.0880210e-03 2.8868102e-02 3.0716448e-03\n", " 3.7106201e-02 9.9118345e-04 5.0325293e-02 3.8348646e-03 1.9698305e-02\n", " 8.1956163e-03 3.2462701e-03 8.1025176e-03 1.5421070e-02 5.8209221e-03\n", " 4.1968067e-04 4.4395551e-03 2.3337429e-02 1.8353451e-02 4.5291316e-03\n", " 1.0655003e-02 6.3336039e-01 1.4202254e-03 1.2996721e-03 3.8333397e-02\n", " 4.2897584e-03]]\n", " stu -> v\n", "\n", " Logits: [[0.00384184 0.00700759 0.00964573 0.01538874 0.04503155 0.00781558\n", " 0.01777511 0.00121006 0.02534541 0.02995089 0.00394947 0.0067529\n", " 0.01260536 0.02045139 0.00387523 0.00607849 0.05042443 0.15998141\n", " 0.01242924 0.00636617 0.01154838 0.00659586 0.06323423 0.01203608\n", " 0.03680959 0.42384925]]\n", " wxy -> z\n", "\n" ] } ], "source": [ "# Test predictions\n", "test_sequences = ['abc', 'def', 'mno', 'stu', 'wxy']\n", "\n", "print('Predictions:')\n", "\n", "for seq in test_sequences:\n", " if len(seq) == seq_length and all(c in char_to_idx for c in seq):\n", " pred = predict_next(seq, model)\n", " print(f' {seq} -> {pred}\\n')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 5 }