{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# DeepHit\n", "\n", "In this notebook we show an example of how we can fit a [DeepHit](http://medianetlab.ee.ucla.edu/papers/AAAI_2018_DeepHit) model.\n", "\n", "We will: \n", "- use SUPPORT as an example dataset,\n", "- use entity embeddings for categorical variables,\n", "- use the [AdamWR optimizer](https://arxiv.org/pdf/1711.05101.pdf) with cyclical learning rates,\n", "- use the scheeme proposed by [Smith 2017](https://arxiv.org/pdf/1506.01186.pdf) to find a suitable learning rate." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import torch\n", "from pycox import datasets\n", "from pycox.models import DeepHitSingle\n", "from pycox.evaluation import EvalSurv\n", "from torchtuples import optim\n", "from torchtuples import callbacks as cb\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn_pandas import DataFrameMapper\n", "from pycox.preprocessing.feature_transforms import OrderedCategoricalLong\n", "from pycox.preprocessing.label_transforms import LabTransDiscreteSurv\n", "from torchtuples.practical import MixedInputMLP" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`sklearn_pandas` can be installed with `! pip install sklearn-pandas`" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "np.random.seed(123456)\n", "_ = torch.manual_seed(123456)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Dataset\n", "\n", "We load the SUPPORT data set and split in train, test and validation." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "df_train = datasets.support.read_df()\n", "df_test = df_train.sample(frac=0.2)\n", "df_train = df_train.drop(df_test.index)\n", "df_val = df_train.sample(frac=0.2)\n", "df_train = df_train.drop(df_val.index)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
x0x1x2x3x4x5x6x7x8x9x10x11x12x13durationevent
082.7099611.02.01.00.00.00.0160.055.016.038.195309142.019.0000001.09985430.01
179.6609501.00.01.00.00.01.054.067.016.038.000000142.010.0000000.8999021527.00
471.7949830.01.01.00.00.00.065.0135.040.038.593750146.00.0999910.3999637.01
549.9329800.01.01.00.00.00.070.0105.033.038.195309127.05.2998051.19995150.01
662.9429890.05.02.01.00.01.0116.0130.035.038.195309133.014.0996090.799927381.00
\n", "
" ], "text/plain": [ " x0 x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 \\\n", "0 82.709961 1.0 2.0 1.0 0.0 0.0 0.0 160.0 55.0 16.0 38.195309 \n", "1 79.660950 1.0 0.0 1.0 0.0 0.0 1.0 54.0 67.0 16.0 38.000000 \n", "4 71.794983 0.0 1.0 1.0 0.0 0.0 0.0 65.0 135.0 40.0 38.593750 \n", "5 49.932980 0.0 1.0 1.0 0.0 0.0 0.0 70.0 105.0 33.0 38.195309 \n", "6 62.942989 0.0 5.0 2.0 1.0 0.0 1.0 116.0 130.0 35.0 38.195309 \n", "\n", " x11 x12 x13 duration event \n", "0 142.0 19.000000 1.099854 30.0 1 \n", "1 142.0 10.000000 0.899902 1527.0 0 \n", "4 146.0 0.099991 0.399963 7.0 1 \n", "5 127.0 5.299805 1.199951 50.0 1 \n", "6 133.0 14.099609 0.799927 381.0 0 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_train.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Feature transforms\n", "We have 14 covariates, in addition to the durations and event indicators.\n", "\n", "We will standardize the 8 numerical covariates, and leave the 4 binary variables as is. \n", "The 3 categorical variables will be transformed to `int64` integers giving the category. The category 0 is reserved for `None` and very small categories that are set to `None`. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "cols_standardize = ['x0', 'x7', 'x8', 'x9', 'x10', 'x11', 'x12', 'x13']\n", "cols_leave = ['x1', 'x4', 'x5']\n", "cols_categorical = ['x2', 'x3', 'x6']\n", "\n", "standardize = [([col], StandardScaler()) for col in cols_standardize]\n", "leave = [(col, None) for col in cols_leave]\n", "categorical = [(col, OrderedCategoricalLong()) for col in cols_categorical]\n", "\n", "x_mapper_float = DataFrameMapper(standardize + leave)\n", "x_mapper_long = DataFrameMapper(categorical) # we need a separate mapper to ensure the data type 'int64'" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "x_fit_transform = lambda df: (x_mapper_float.fit_transform(df), x_mapper_long.fit_transform(df))\n", "x_transform = lambda df: (x_mapper_float.transform(df), x_mapper_long.transform(df))" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "x_train = x_fit_transform(df_train)\n", "x_val = x_transform(df_val)\n", "x_test = x_transform(df_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Label transforms\n", "\n", "DeepHit is a discret time method, so we need labels with index of the duration, and an event indicator.\n", "\n", "SUPPORT is a contiuous time data set, so we need to discretize it. We choose 25 equidistant time points between the min and max of the training durations.\n", "This is handled by `LabTransDiscreteSurv`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "get_target = lambda df: (df['duration'].values, df['event'].values)\n", "labtrans = LabTransDiscreteSurv(25)\n", "y_train = labtrans.fit_transform(*get_target(df_train))\n", "y_val = labtrans.transform(*get_target(df_val))\n", "durations_test, events_test = get_target(df_test)\n", "val = (x_val, y_val)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can look at the discrete durations with the attribte `cuts`" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([ 0. , 84.541664, 169.08333 , 253.625 , 338.16666 ,\n", " 422.70834 , 507.25 , 591.7917 , 676.3333 , 760.875 ,\n", " 845.4167 , 929.9583 , 1014.5 , 1099.0416 , 1183.5834 ,\n", " 1268.125 , 1352.6666 , 1437.2084 , 1521.75 , 1606.2916 ,\n", " 1690.8334 , 1775.375 , 1859.9166 , 1944.4584 , 2029. ],\n", " dtype=float32)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "labtrans.cuts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Neural net\n", "\n", "We make a torch net. If this is new to you, we would recommend [the tutorials by PyTroch](https://pytorch.org/tutorials/).\n", "\n", "For simplicity, we will just use the implementation of `torchtuples.practical.MixedInputMLP`.\n", "By mixed input we refer to that we have both entity embeddings and regular numerical covariates." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We make embeddings half the size of the number of categories." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "num_embeddings = x_train[1].max(0) + 1\n", "embedding_dims = num_embeddings // 2" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(array([8, 7, 4]), array([4, 3, 2]))" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "num_embeddings, embedding_dims" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "in_features = x_train[0].shape[1]\n", "out_features = len(labtrans.cuts)\n", "num_nodes = [32, 32, 32, 32]\n", "batch_norm = True\n", "dropout = 0.1\n", "net = MixedInputMLP(in_features, num_embeddings, embedding_dims, num_nodes,\n", " out_features, batch_norm, dropout)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "MixedInputMLP(\n", " (embeddings): EntityEmbeddings(\n", " (embeddings): ModuleList(\n", " (0): Embedding(8, 4)\n", " (1): Embedding(7, 3)\n", " (2): Embedding(4, 2)\n", " )\n", " )\n", " (mlp): MLPVanilla(\n", " (net): Sequential(\n", " (0): DenseVanillaBlock(\n", " (linear): Linear(in_features=20, out_features=32, bias=True)\n", " (activation): ReLU()\n", " (batch_norm): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", " (dropout): Dropout(p=0.1)\n", " )\n", " (1): DenseVanillaBlock(\n", " (linear): Linear(in_features=32, out_features=32, bias=True)\n", " (activation): ReLU()\n", " (batch_norm): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", " (dropout): Dropout(p=0.1)\n", " )\n", " (2): DenseVanillaBlock(\n", " (linear): Linear(in_features=32, out_features=32, bias=True)\n", " (activation): ReLU()\n", " (batch_norm): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", " (dropout): Dropout(p=0.1)\n", " )\n", " (3): DenseVanillaBlock(\n", " (linear): Linear(in_features=32, out_features=32, bias=True)\n", " (activation): ReLU()\n", " (batch_norm): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", " (dropout): Dropout(p=0.1)\n", " )\n", " (4): Linear(in_features=32, out_features=25, bias=True)\n", " )\n", " )\n", ")" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "net" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fitting model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "DeepHit has a loss that is a combinatin of a negative log-likelihood and a ranking loss. `alpha` is a parameter that controls the linear combination between the two, and `sigma` is a parameter used by the ranking loss. `alpha = 1` give a loss only containing the negative log-likelihood and `alpha = 0` give a pure ranking loss. Note that this is different than the original paper.\n", "\n", "We make an AdamWR optimizer where we multiply the learning rate wtih 0.8 at the start of every cycle." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "optimizer = optim.AdamWR(decoupled_weight_decay=0.01, cycle_eta_multiplier=0.8)\n", "model = DeepHitSingle(net, optimizer, alpha=0.2, sigma=0.1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can use `lr_finder` to find a suitable initial learning rate. This method also sets the learning rate for the net." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.02420128264794396" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEKCAYAAADn+anLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3zU9f3A8dc7g1wSMiAJkEFI2HtoGIKKi4qIYB0IdeG2VWut1tJl1dpqW7W/2rpwoKJCFUdRoTgRFBSCssIOK2GGESAkIev9++MuGELGhdzlLpf38/G4B/dd931/uAf35jO+n4+oKsYYY0xNgnwdgDHGGP9lScIYY0ytLEkYY4yplSUJY4wxtbIkYYwxplaWJIwxxtQqxNs3EJHRwD+BYOBFVX2s2vFU4FUg1nXOFFWdIyJDgKmVpwEPqup7dd0rPj5e09LSPFwCY4wJbMuWLdunqgk1HRNvPichIsHABmAUkAssBSap6poq50wFvlfVZ0WkNzBHVdNEJAIoUdUyEUkEVgBJqlpW2/0yMjI0MzPTa+UxxphAJCLLVDWjpmPebm4aAmxS1c2qWgLMBMZXO0eBaNf7GGAngKoWVkkIDtd5xhhjmpC3k0QykFNlO9e1r6oHgWtEJBeYA9xVeUBEhopIFrAKuL2uWoQxxhjP84eO60nAK6qaAowBpotIEICqfquqfYDBwG9ExFH9YhG5VUQyRSQzLy+vSQM3xphA5+2O6x1AxyrbKa59Vd0EjAZQ1cWuRBAP7K08QVXXikgB0Bc4odNBVafi6uDOyMiwJiljAkxpaSm5ubkUFxf7OpRmz+FwkJKSQmhoqNvXeDtJLAW6iUg6zuQwEfhJtXO2A+cDr4hIL5z9D3mua3JcHdedgJ7AVi/Ha4zxM7m5uURFRZGWloaI+DqcZktV2b9/P7m5uaSnp7t9nVebm1x9CHcC84C1wFuqmiUiD4vIONdp9wK3iMgKYAYwWZ1Drs4EVojIcuA94Gequs+b8Rpj/E9xcTFxcXGWIBpJRIiLi2twjczrz0mo6hycHdJV9z1Q5f0aYEQN100HpjfkXoeKSikrryAk2B+6WowxnmIJwjNO5e8xoH5Ntx8oZM+RY74OwxhjAkZAJQmAXflFvg7BGBNg8vPzeeaZZxp83ZgxY8jPz2/wdZMnT2bWrFkNvs4bAi5J7LAkYYzxsNqSRFlZ3Y9uzZkzh9jYWG+F1SS83ifR1HYdsmFyxgSqhz7IYs3Owx79zN5J0fzxkj51njNlyhSys7MZOHAgoaGhOBwO2rRpw7p169iwYQOXXnopOTk5FBcXc/fdd3PrrbcCkJaWRmZmJgUFBVx00UWceeaZLFq0iOTkZP773/8SHh5eb3yfffYZ9913H2VlZQwePJhnn32WsLAwpkyZwuzZswkJCeFHP/oRjz/+OG+//TYPPfQQwcHBxMTEsGDBgkb//QRUkggSYafVJIwxHvbYY4+xevVqli9fzvz587n44otZvXr18aGkL7/8Mm3btqWoqIjBgwdz+eWXExcXd8JnbNy4kRkzZvDCCy8wYcIE3nnnHa655po671tcXMzkyZP57LPP6N69O9dddx3PPvss1157Le+99x7r1q1DRI43aT388MPMmzeP5OTkU2rmqklAJYlWwUHszLeahDGBqr7/8TeVIUOGnPCswVNPPcV77zknqc7JyWHjxo0nJYn09HQGDhwIwOmnn87WrVvrvc/69etJT0+ne/fuAFx//fU8/fTT3HnnnTgcDm666SbGjh3L2LFjARgxYgSTJ09mwoQJXHbZZZ4oamD1SYQGC7sOWU3CGONdkZGRx9/Pnz+fTz/9lMWLF7NixQoGDRpU47MIYWFhx98HBwfX259Rl5CQEJYsWcIVV1zBhx9+yOjRowF47rnneOSRR8jJyeH0009n//79p3yP4/dq9Cf4kdCQIGtuMsZ4XFRUFEeOHKnx2KFDh2jTpg0RERGsW7eOb775xmP37dGjB1u3bmXTpk107dqV6dOnM3LkSAoKCigsLGTMmDGMGDGCzp07A5Cdnc3QoUMZOnQoc+fOJScn56QaTUMFVpIIDuJgYSlFJeWEtwr2dTjGmAARFxfHiBEj6Nu3L+Hh4bRv3/74sdGjR/Pcc8/Rq1cvevTowbBhwzx2X4fDwbRp07jyyiuPd1zffvvtHDhwgPHjx1NcXIyq8uSTTwLwq1/9io0bN6KqnH/++QwYMKDRMXh10aGm1rV3fy0b9yif3TuSLgmtfR2OMcYD1q5dS69evXwdRsCo6e/Tl4sONalQ13Qcu6zz2hhjPCLgmpuKgJ3WeW2MaQbuuOMOvv766xP23X333dxwww0+iuhkAZckRLDOa2MCjKoG5CR/Tz/9dJPe71S6FwKquUkE4luHWXOTMQHE4XCwf//+U/qBMz+oXE/C4Thpgc86BVRNAiApNtyam4wJICkpKeTm5mLLEzde5cp0DRF4SSLGwYY9NY9nNsY0P6GhoQ1aSc14VkA1N4GrJpFfbFVTY4zxgIBLEokxDopKyzlUVOrrUIwxptkLuCSRHOucetcm+jPGmMYLuCSReDxJWOe1McY0VsAliaQY5/Aumw3WGGMaz+tJQkRGi8h6EdkkIlNqOJ4qIl+IyPcislJExrj2jxKRZSKyyvXnee7cL751GKHBwg5rbjLGmEbz6hBYEQkGngZGAbnAUhGZraprqpz2e+AtVX1WRHoDc4A0YB9wiaruFJG+wDwgub57BgUJHWIcVpMwxhgP8HZNYgiwSVU3q2oJMBMYX+0cBaJd72OAnQCq+r2q7nTtzwLCRSQMNyTFhNtT18YY4wHeThLJQE6V7VxOrg08CFwjIrk4axF31fA5lwPfqeqx6gdE5FYRyRSRzMonMpNiw9lhHdfGGNNo/tBxPQl4RVVTgDHAdBE5HpeI9AH+CtxW08WqOlVVM1Q1IyEhAYCkWAd7DhdTXmEP1BljTGN4O0nsADpW2U5x7avqJuAtAFVdDDiAeAARSQHeA65T1Wx3b5oYE05ZhZJ35KSKhzHGmAbwdpJYCnQTkXQRaQVMBGZXO2c7cD6AiPTCmSTyRCQW+AiYoqpf0wBJsc5hsDbRnzHGNI5Xk4SqlgF34hyZtBbnKKYsEXlYRMa5TrsXuEVEVgAzgMnqnHjpTqAr8ICILHe92rlz3yR7oM4YYzzC67PAquocnB3SVfc9UOX9GmBEDdc9AjxyKvdMjHEmCRvhZIwxjeMPHdceF+0IoXVYiDU3GWNMIwVkkhAREmMcJzQ3qao1PxljTAMFZJIAZ7/ErkPO5qbS8gp++95qhj/2OfOydvs4MmOMaT4COEk4axKHCku5/uUlzFiynYhWwbz01RZfh2aMMc1GwC1fWikxJpx9BSX8+JmvyTlYyBNXDmD/0WP8Zc461uw8TO+k6Po/xBhjWrgArkk4RzgdLCzhjZuHcfnpKVyVkUp4aDCvLtrq2+CMMaaZCNgkcV7PdvxkaCrv3zGCIeltAYiJCOXSQcm8v3wHB4+W+DhCY4zxfwGbJNpGtuIvP+5Hp7jIE/ZfP7wTx8oqmLk0p5YrjTHGVArYJFGbnh2iOaNzHK9/s42y8ooTjh0qLMX5sLcxxhhogUkC4PrhaezIL+LTtXsBOFxcyoOzsxj0p495+eutvg3OGGP8SMCObqrLBb3akRwbzrSvt3CsrJxHPlrLvoJjxEWG8dLCzVx/RidCgltk/jTGmBO0yF/CkOAgrj2jE99uOcDdM5eTGOPgv3eM4NHL+rHzUDHzsvb4OkRjjPELLbImATBpcCpLthzg3J7t+MmQVIKDhPIKJbVtBC9/vYWL+yf6OkRjjPG5FlmTAOdw2JcnD+baYZ0IDhIAgoOE64ensWzbQVbm5vs4QmOM8b0WmyRqMyEjhdZhIUyzDmxjjLEkUV2UI5QrTk/hw5U72XvY1qMwxrRsbiUJEQkSkUEicrGInOfuCnHN1eThaZRVKK9/u93XoRhjjE/VmSREpIuITAU2AY8Bk4CfAZ+KyDcicoOIBFxtJC0+kvN6tOONb7ZRXFru63CMMcZn6vuBfwR4Heiiqheq6jWqeoWq9gfGATHAtd4O0hduGJHO/qMlPDg7i237j/o6HGOM8QkJpGkoMjIyNDMz0yOfpar8atZK3v0ulwqFM7vGM3FIRy7s04FQe9DOGBNARGSZqmbUeKyuJCEil9X1war6rhs3Hw38EwgGXlTVx6odTwVeBWJd50xR1TkiEgfMAgYDr6jqnfXdy5NJotKuQ0W8nZnLf5bmsCO/iKuHpvLnH/fz6D2MMcaXGpMkprnetgOGA5+7ts8FFqnq2HpuHAxsAEYBucBSYJKqrqlyzlTge1V9VkR6A3NUNU1EIoFBQF+gr6+SRKXyCmXKOyv574qdLJ5yHnGtw7xyH2OMaWp1JYk6201U9QZVvQEIBXqr6uWqejnQx7WvPkOATaq6WVVLgJnA+Oq3ASqXiYsBdrrufVRVvwL8YhxqcJBw28jOlNg048aYFsTdxvWOqrqryvYeINWN65KBqr+oua59VT0IXCMiucAc4C43YwJARG4VkUwRyczLy2vIpQ3WtV0UZ3WLZ/ribZRWm2bcGGMCkbtJ4jMRmScik0VkMvAR8KmHYpiEs88hBRgDTG/IsFpVnaqqGaqakZCQ4KGQajd5eBq7DxczL2u31+9ljDG+5taPsas/4DlggOs1VVXd+R//DqBjle0U176qbgLect1nMeAA4t2JyxfO7dGOTnERvGLTdhhjWoCGjOX8DvhIVe8B5olIlBvXLAW6iUi6iLQCJgKzq52zHTgfQER64UwS3m03aoSgIOG6M9LI3HaQVbmHfB2OMcZ4lbvTctyCczjq865dycD79V2nqmXAncA8YC3wlqpmicjDIjLOddq9wC0isgKYAUxW15ArEdkKPAlMFpFc1+gnn7syI4WIVsG8smirr0Mxxhivcnc9iTtwjlT6FkBVN7o7f5OqzsHZIV113wNV3q8BRtRybZqb8TWpaNckgDOX5PCbMT2Jt+GwxpgA5W5z0zHXEFYARCQE59DVFuu6M9IoKa/gTZsE0BgTwNxNEl+KyG+BcBEZBbwNfOC9sPxf13atGZreljmrdtV/sjHGNFPuJokpODuTVwG34Ww++r23gmouzunRjnW7j7DH1p0wxgQod4fAVqjqC6p6pWsW2BcqO5dbspHdnc9lLNjgt4OxjDGmUdwd3TRCRD4RkQ0isllEtojIZm8H5+96JUaREBXGgo37fB2KMcZ4hbujm14C7gGWAbYKj4uIcFa3eL5Yt5fyCiU4SHwdkjHGeJS7fRKHVHWuqu5V1f2VL69G1kyM7J7AwcJSVu+wB+uMMYGnzpqEiJzmevuFiPwdeBc4VnlcVb/zYmzNwpld4xFx9ksM6Bjr63CMMcaj6mtueqLadtX5xhU4z7PhND9xrcPomxTDgo153HV+N1+HY4wxHlVnklDVcwFEpLOqntBRLSKdvRlYc3J293ie+3Izh4tLiXa4s8xGy6WqbNtfSKe4CESsD8cYf+dun8SsGva97clAmrOzuyVQXqEs2mTdNHVZs/MwE6d+wzmPz+ee/yynuNTGQBjj7+rrk+iJcxW6mGrrXUfjnK3VAKd1akPrsBAWbMxjdN8ONZ6z61ARby3N5UhxKfeP7kmrkIZMwOu/8gtLyDtyjIiwEFq3CiEyLJiQ4BPLtr/gGE98soGZS7YTE+6c9+qd73LZlFfA89dmkBwb7qPojTH1qa9PogcwFogFLqmy/whwi7eCam5Cg4M4o0scCzbkoarHm1HKyiv4Yn0eM5ds54v1e6lwPX64Zd9Rnr76NByhwT6MuuEKS8r4OGsPi7P3s3lfAdl5RzlwtOSk81qFBBEWHERYaBCtgoPILyqlpKyCycPTufv8bsREhDKmXwfunrGcS/71FU//5DTO6BLngxIZY+oj7jw4LSJnuBYE8msZGRmamZnpk3tP/2Ybf3h/NZ/fO5K0uEjmrN7FEx9vYMu+oyREhTEhI4WrMlJZuCmP3723mrO6xfPCdRl+nyjKyitYuGkf73+/g4+z9lBUWk6biFC6tYuiS7tIuiS0pl20g6KSMgqOlXP0WBlHS8ooKaugpKyCY2UVtAoJ4sYRaXRtd+ISJNl5Bdz6WiZb9xfy6GX9mJDRsZYojDHeJCLLVDWjpmPuPkyXIyLv8cOU3guBu1U11xMBBoKR3ZxTdDwzP5t1uw+zesdherSP4pmrT2NU7/aEuppgro7rRGhwEL9+ZyU3TFvKS5MziGjl7tfQtI4eK+OGV5ayZMsBYsJDuXRQMpcOTGJwWluCPPDgYJeE1rx/xwh+9sZ33D9rJQXFZdx4ZroHIjfGeIq7v07TgDeBK13b17j2jfJGUM1RalwEaXERzFqWS0qbcJ6cMIDxA5NrfAp7QkZHQoOFe99awZXPLebHg5IZ3iWenh2iPPLj6wlHj5Vxw7SlLNt+kL/8uB+Xn55MWIjnaz1RjlBevD6DX8xczsMfruFwcSl3n9/NRj4Z4yfcbW5aoaoDqu1brqoDvRbZKfBlcxPAVxv3kXOwkMtOc+8H9X+rd/HY3HVs3V8IQJuIUM7qlsD9o3uQ0ibC2+HWqrCkjMnTlpK59QD/nDiISwYkef2eZeUVTHl3FbOW5XLTmen8/uJeliiMaSKeaG7aJyLX4FxeFGASYOM9qzmzW3yDzh/dN5HRfRPZdaiIxdn7WZS9n7mrdvH5ur08cElvrjw9pUE/lIeLS1mVe4iS8gpaBQcRGhxEWEgQvRKj3R5NVVjirEFkbj3AP64a2CQJAiAkOIi/Xd6f1mEhvPTVFlqHhXDPqO5Ncm9jTO3crUl0Av4FnOHa9TXwc1X1q2XZfF2T8IScA4Xc9/YKvt1ygAt6teMvl/WjXVTNo43zjhzj6037WLr1AMu2HWT9niPU9HX2TY52a6jpoaJSbnk1k8xtzgQxfmCyJ4rUIKrKvW+t4P3lO/jPbWcwOK1tk8dgTEtTV03CrSTRXARCkgCoqFBe/noLf5u3nrDgIAZ0jKV7+yh6dGhN+2gHy7YdZP76PFa5JhVsHRbCaZ3akNGpDaeltiEyLJjScqW0vILcg4X86cO1hIUE8ew1pzMkveYf3d2Hirn+5SVs3lfAExMGMq6JahA1KThWxph/LqS8Qplz91nEhNtT7MZ4U6OThIik4KxJNHh0k4iMBv4JBAMvqupj1Y6nAq/ifBYjGJiiqnNcx34D3IRzevKfq+q8uu4VKEmi0qa9R5i6YDPrdh9hw54jFJdWABAkcFpqG87pkcDI7u3onRRd5zTlm/Y6h5puP1DIH8f14ZqhqSc0Y23ae4TrXlrCoaJSnr82o8HNZt6wPCefK55dxIV9O/DvSYOaff/E0q0H2LDnCHsPHyOv4Bj7jhyjT1IMVw9LJb51mK/DMy2cJ5LEJzhHN0137boGuFpV6xzdJCLBwAaco6BygaXAJFVdU+WcqcD3qvqsiPQG5qhqmuv9DGAIkAR8CnRX1Vrncgi0JFFVRYWSc7CQHQeL6JMUQ0xEw/53fbi4lF/MXM7n6/bStV1ruiREkhYfSbsoB099tpHQ4CBeuWEwfZNjvFSChntm/ib+9r/1/O2K/n71DMWi7H08NHsNFapEOUKIcoQSGxHKRX07MKp3hxMSds6BQh76IItP1+49vq9tZCtiI0LZnHeUViFBXDowiRvPTKdLQmt25hex/UAh2w8U0q1dVK01P3fYGifGXZ7ouE5Q1WlVtl8RkV+4cd0QYFPl5IAiMhMYD6ypco7inOYDIAbY6Xo/HpipqseALSKyyfV5fv9QnzcEBQmd4iLpFBd5StdHO0J58boMpi3ayjeb95Odd5TP1+2ltFxJi4vgtRuHkhrnuxFVNbnt7C4s3LCPB2dnoaokxYbTPtpB+ygH0eEhPqldfLFuL7e/vozEGAe9EqM5UlxGflEpa3Yd5r/Ld9KxbTg3DE/nx4OSeXPJdv71+UYEYcpFPRk/MIn41mHHn5nZtLeAVxZtYdayXN7KzCU4SCivOPE/bbee3ZlfXdjj+DXuyDlQyJ8+XMPi7P3838SBnN+rvUf/DkzL4m5N4jOcz0VUHd10g6qeX891VwCjVfVm1/a1wFBVvbPKOYnAx0AbIBK4QFWXici/gW9U9XXXeS8Bc1V1VrV73ArcCpCamnr6tm3b6i+1AZz/09yZX0S76DCvPAPhCbsPFTP+6a/Yc/jYCft7J0Zz45npXDIgsclin7tqFz+f+T09OkTx2o1DaRvZ6vix8grl46zdvPTVFjK3HTy+/6K+HfjD2N4k1TFoIL+whFnLcskvLCU1LoLUthEkx4bz/IJsXv9mOwM7xvKvSYPo2LbuJF5cWs5zX2bz7PxsgkRIinWwZd9RHhrfl2uHdWr8X4AJWJ5obqo6ukmBRbgxusnNJPFLVxxPiMgZOJdK7Qs8hRtJoqpAbm5qyY6VlbMrv5i9R46x53AxuQeLePe7XDbuLSAhKoxrh3Xi2mGdaFPlR9vT3v0ul/veXsGg1DZMu2FwnVPCr8jJ58OVOxnRNZ5zerRr1H3nrNrFr2etBIHfjenFRf0ST+rIP3C0hI9W7uT5BZvJPVjE2P6J/HZML2IjQvn5jO/5dO1ebju7M78e3dNvHtY0/sVno5tcP/oPquqFru3fAKjqo1XOycKZSHJc25uBYTg7rI+fKyLzXJ9Va3OTJYmWQ1VZuHEfL321hS835BEX2Yq/XNaPC/vUPAtvY8xa5kwQw7vE8cJ1GUSGNe00Ktv3F3LXzO9ZkZNPSJAwJL0to3q3p01EK2av2MmCDXmUVSi9E6P5/dheDO/yw8CD8grlwdlZTP9mG+f2SKBru9bH59gCZ3OWP/VDGd/wRE0iHbgLSKNKP4aqjqvnuhCcHdfnAztwdlz/RFWzqpwzF/iPqr4iIr2Az4BkoDfOzvLKjuvPgG4ttePa1C5r5yHun7WSrJ2Hufy0FP44rrfHFn+au2oXd7z5HcO7xPPi9b6bkLGiQvk+J59P1uzhkzW7yc47CkBijINxA5MYPyCZXolRNfbTqCovLNzMEx9vIEiEyDDnlO75haUcPVbGHed25c7zujao38MEFk8kiRU4m4FWARWV+1X1SzeuHQP8H87hrS+r6p9F5GEgU1Vnu0YxvQC0xtmUdb+qfuy69nfAjUAZ8AtVnVvXvSxJtFwlZRX86/ONPDM/m/ZRYVw1OJWjJWXkF5aQX1hKaEgQvROj6Z0UTZ+k6FofUKxq/vq93PJaJv1TYpl+0xC/mohxc14B+UWlDEyJdbsJqeo09uDsC3nogzW89/0O+iRF88SEAfTsEF3HJ5hA5Ykk8a2qDvV4ZB5mScIsz8nn3reWk513FEdoELHhzuGmR0vKyDlQdPy85NhwftSnPaP7dCAjre1JQ0WXbDnAdS9/S+f41sy4dVhAP9D3v9W7+f37qzhUVMqvLuzBzWd2tr6LFsYTSeInQDeco5CODzNR1e88FaQnWJIw4GyaKSmvOKlp6FBRKet2HSZr52EWZe9nwcY8SsoqiG8dxtDObQkJEioUKlRZsD6PhOgw3rrtjBbxsNv+gmP89r1VzMvaw1nd4nliwgC3alsmMHgiSTwKXAtk80Nzk6rqeR6L0gMsSZiGKDhWxvz1e5m7ejerdxxCgCARRKBdlIMnrxpAYkzLWVpVVXlzyXb+9OEaIluF8PiVAzi3Z+NGZxn/t3rHIfqlxDb6Yborgc6qevJalcY0U63DQhjbP4mx/X03T5U/ERGuHtqJIWltuWvG99zwylIGp7Xhwj4duLBPh3qf0zDN0xvf1j1Pq7vDGVbjnFvJGBPgurWP4v07RvCrC3twpLiMRz5ay1l/+4KL/rmQhRvzfB2e8bDCkrI6j7tbk4gF1onIUk7sk6hzCKwxpnlyhAZzx7lduePcrmzfX8jHa3bz5pLt3PxqJtMmD2Z4V99PAmk8o7Ck1qcKAPeTxB8bH4oxpjlKjYvg5rM6c/lpKUyc+g03v5bJazcOIcPW+ggIRfUkCXebmzKBha7nInbhnIhvUeNCM8Y0J20iWzH95iF0iHZww7SlrMzN93VIxgPqa25yN0ksABwikoxzGOy1wCuNiswY0+y0i3Lwxi1DiY0M5dqXlvBx1m6KS+v+n6jxb55qbhJVLRSRm4BnVPVvrqewjTEtTGJMOG/ePIyJU7/h1unLcIQGcWbXeM7r2Z7IsGA27ilg494jbNxbQM8OUTz64/4NXv/ENJ2iepK820nCNVnf1bgm3sP9WogxJsB0bBvBZ/eO5NstB/h87R4+Xbv3+MJKwUFCWlwE6XGRfLJmD2t2fsWL12fQtV2Uj6M2NTl6zDNJ4m7gN8B7qpolIp2BLxoZmzGmGXOEBjOyewIjuyfw4DglO6+AsgolPT7y+BofmVsPcPvry7j06UX831UDuaC3LYDkb4o80SehqgtUdZyq/tW1vVlVf+6B+IwxAUBE6Nouip4dok9YBCojrS2z7zyT9PhIbpmeyd/nraPgWN0/SqbpqCqF9TQ31ZkkROQFEelXy7FIEblRRK5uRIzGmACXFBvO27efweWnpfD0F9mM/NsXvLhws3V4+4FjZRXUNzNTfTWJp4E/iMhaEXlbRJ4RkZdFZCHOIbBRQK0rxRljDDibph6/cgDv/Ww4vRKjeeSjtZzz9/nMWLL9pHW9TdOpb2QT1NMnoarLgQki0hrIABKBImCtqq73RJDGmJZjUGobXr95KIuy9/H4vPX85t1VvLpoKw9c0vuEFfVM0zjqRtOfWx3XqloAzG9kPMYYA8DwLvG889M45qzazV/mrOUnL3zLhX3a85uLepEWH+nr8FqM+oa/gvujm4wxxqNEhIv7J3J+r3a8uHAzz8zPZl7WfPokRXNuj3ac0yOBgR1jKatQDheXcrioDFC6JLSucZlW03CNbm4yxhhvc4QGc+d53bgyoyOzluXy5fo8nv0ym39/sYkggepdFr0So5k8vBPjByb7bM3xQFHflBxwCklCRIKA1qp6+FSCMsaYmrSPdhyfefZQUSlfbdzHml2HiAwLIcoRSrQjhMNFpbzx7XZ+/c4qHp27jomDU/npOV0CenlZb6pvcj9wM0mIyJvA7aeGIoUAABfGSURBVEA5sBSIFpF/qurfGxWhMcbUICY8lIv7J3Jx/8STjl0zrBPfbjnAq4u2MnVBNu9/v4PHLu/HOT1sFb2Gcqe5yd2pNXq7ag6XAnOBdJyT/NVLREaLyHoR2SQiU2o4/g8RWe56bRCR/CrH/ioiq12vq9yM1RgTwESEYZ3jePaa03n/jhFEOUKYPG0p989aweHiUl+H16x4srkpVERCcSaJf6tqqYjUO7hZRIJxPmsxCsgFlorIbFVdU3mOqt5T5fy7gEGu9xcDpwEDgTBgvojMtWYuY0yl/imxfPjzM/nnpxt57stsFm7cx+ThaYwbmNSi1ic/VZ6sSTwPbAUigQUi0glw58d6CLDJNY1HCTATGF/H+ZOAGa73vYEFqlqmqkeBlcBoN+M1xrQQYSHB3D+6J+/+bASJMQ4enbuO4Y99zsSpi5m5ZLs92V0HjyUJVX1KVZNVdYw6bQPOdePSZCCnynaua99JXIknHfjctWsFMFpEIkQk3nW/jjVcd6uIZIpIZl6erb9rTEs1sGMs7/5sBF/cdw6/OL87ew8fY8q7q7jq+cXsOVzs6/D8UlFJOUH1jCZ2K0mIyN0iEi1OL4nId8B5HoixqonALFUtB1DVj4E5OKf/mAEsxtlxfgJVnaqqGaqakZCQ4OGQjDHNTXp8JHdf0I3P7h3Js1efxsa9BVzyr69YnmMr6VVXWFJORKu6ex3cbW660dUX8COgDc5O68fcuG4HJ/7vP8W1ryYT+aGpCQBV/bOqDlTVUYAAG9yM1xjTwokIF/VL5N2fDadVSBATnl/M+9/X9vPTMhWWlBHRqu5nTdxNEpUVkjHAdFXNqrKvLkuBbiKSLiKtcCaC2Sd9uEhPnMlncZV9wSIS53rfH+iPc+lUY4xxW88O0cy+80wGdYzlF/9ZzrPzs30dkt9w1iQ8kySWicjHOJPEPBGJAirqu0hVy4A7gXnAWuAt16JFD4vIuCqnTgRmqp4waW0osFBE1gBTgWtcn2eMMQ3SNrIVr988lHEDkvjr/9bxxrfbfB2SXygsKSe8nuYmd4fA3oRzKOpm11rXccAN7lyoqnNw9i1U3fdAte0Ha7iuGOcIJ2OMabTQ4CCemDCAI8Wl/P791cSEhzK2f5Kvw/KpolIPNTepagXO/oTfi8jjwHBVXdn4EI0xpumEBgfxzNWnk9GpDff8ZzlfbmjZIyI91twkIo/hXOd6jev1cxH5S6MjNMaYJhbeKpgXrx9M13ZR3D59GfPX7/V1SD5T5ME+iTHAKFV9WVVfxvlQ29hGxmeMMT4REx7KazcOISnWweRpS7n51Uw25xX4Oqwmd7SkzGNDYAFiq7yPOaWIjDHGTyREhfHRz8/i/tE9+Gbzfn70jwU89EEWhwpbzvxPRSXlhNdTk3C34/pR4HsR+QLn0NezgZMm6zPGmObEERrMz87pypWnd+TJT9bz6qKtfL1pH2/ddgaxEa18HZ7XFZaUE1HPmhzudlzPAIYB7wLvAGeo6n8aHaExxviBhKgwHr2sP9NvGsrWfYXc8MpSt2ZIbc5UlaLSRvZJiMhplS8gEefcS7lAkmufMcYEjBFd43lq0iBW5OTz09e/o6Ss3sfBmq3i0gpUafRzEk/UcUzx/PxNxhjjU6P7duDRy/rx63dWcd/bK/i/qwYSVN8seM3QUVdNKTKsEX0SqurOTK+IyChV/cTd4Iwxxp9dNTiVA0dL+ev/1pEY4+A3Y3r5OiSPq1y6NNwTfRJu+KuHPscYY/zC7SM7M2lIKi8s3MzqHYd8HY7HVa4l4ckhsHUJvLqYMaZFExGmXNSTNhGt+OPsLE6cWq75q+yY99TDdPUJrL89Y4zB+dDdr0f3ZNm2g7wXYNOMH29uaqIkYYwxAemK01MY0DGWR+eu40hx4Dxo90NzU9Mkia0e+hxjjPErQUHCw+P6sK/gGE99ttHX4XjM0ePNTZ6ZKhwRGQ6kVb1GVV9z/XnZKcRojDHNwoCOsVyV0ZFpX29lQkZHurWP8nVIjVbkyZqEiEwHHgfOBAa7XhmNitAYY5qRX13Yg4hWwTz84Rpfh+IR7jY3uVuTyAB6a6B17xtjjJviWofx8/O78chHa1mcvZ8zusT5OqRGKSr1bMf1aqBD40Iyxpjm7ZphnWgfHcaTn6xv9kNiC0vKCA4SWgXXnQbqm7vpAxGZDcQDa0RknojMrnx5MF5jjPF7jtBg7jyvG0u3HmTBxn2+DqdRjh5zzgArUvdjbvU1Nz3uuZCMMab5uyqjI8/Nz+aJj9dzdrf4en9k/VVRSTkR9czbBPXUJFT1S1X9EtgOfFtlewmwzZ1ARGS0iKwXkU0ictIaFCLyDxFZ7nptEJH8Ksf+JiJZIrJWRJ6S5vptGGMCRquQIO6+oBsrcw/xyZo9vg7nlBWWltc7/BXc75N4G6g6Z265a1+dRCQYeBq4COgNTBKR3lXPUdV7VHWgqg4E/oVzzYrKIbcjgP5AX5wjqka6Ga8xxnjNZYOSSY+P5MlPNlBR0Tz7JopKyuqd3A/cTxIhqlpSueF6786yTUOATaq62XXNTGB8HedPAmZU3gZwuO4TBoQCzTdtG2MCRkhwEL+4oBvrdh/ho1W7fB3OKSksqX/BIXA/SeSJyLjKDREZD7jTa5MM5FTZznXtO4mIdALSgc8BVHUx8AWwy/Wap6pr3YzXGGO86pL+SfRoH8WTn2xolosTFbqxvjW4nyRuB34rIttFJAf4NXBbI+KryURglqqWA4hIV6AXkIIzsZwnImdVv0hEbhWRTBHJzMvL83BIxhhTs6AgYcqYnmzZd5QXFm72dTgNVlhS5rmahKpmq+ownP0KvVR1uKpucuPSHUDHKtsprn01mcgPTU0APwa+UdUCVS0A5gJn1BDbVFXNUNWMhIQEd4pjjDEecW6PdlzUtwNPfbaRnAOFvg6nQQpLyon0YMc1InIx8DPglyLygIg84MZlS4FuIpIuIq1wJoKTnq8QkZ5AG2Bxld3bgZEiEiIioTg7ra25yRjjVx64pDfBQdLs1pwo8mRzk4g8B1wF3IVzgaErgU71XaeqZcCdwDycP/BvqWqWiDxctY8DZ/KYWW3aj1lANrAKWAGsUNUP3InXGGOaSmJMOPdc0J3P1+1lXlbzGVvjbse1u3M3DVfV/iKyUlUfEpEncDb/1EtV5wBzqu17oNr2gzVcV47n+z2MMcbjJo9I453vcnnogyzO6hZPZJjbE2z7REWFUlRaTrgHm5uKXH8WikgSUAoknmJ8xhgTUEKDg3jk0r7sOlTMP5vBmhOVk/t5cgjshyISC/wd+A7nIkNvnlp4xhgTeDLS2jIhI4WXv9ri953Y7k4TDu6PbvqTquar6js4+yJ6Vm8yMsaYlu6Xo3oQJMLTX7gz+NN3flhwyEPNTSLiEJFfisi7OGsQN4qIozFBGmNMoOkQ42DSkI7MWpbr17WJwtLKpUs919z0GtAH59xK/8b5vMT0UwvPGGMC10/P6er3tYnK5iZ3hsC62wXfV1WrTsz3hYgExhp+xhjjQZW1iTe+3c4d53alY9sIX4d0kuPNTR6c4O87ERlWuSEiQ4HMUwnOGGMC3U/P6UpQkP/WJo4eq2xuamSfhIisEpGVwOnAIhHZKiJbcD4ZndH4UI0xJvB0iHHwkyGpfts34e761lB/TWIscAkwGucMrSOBc1zvL2pEjMYYE9BuH9nFb2sTlX0SkW6sTFdnXUNV3Vp9zhhjzIkqaxOvf7ONX1zQnQ4x/jMg9PhzEqEenODPGGNMw9w4Ip2yCmXm0u2+DuUERSXOPglPridhjDGmgVLjIji7ewIzl+RQVu4/CxMdLSknJEhoFVJ/CrAkYYwxXnTN0FR2Hy7m83V7fR3Kce5OEw6WJIwxxqvO69mODtEOXv/Wf5qc3F2VDixJGGOMV4UEBzFxSEcWbMhj+37/GA7r7qp0YEnCGGO8buLgVIKDhDeX+EdtwpqbjDHGj3SIcXB+z3a8nZnDsbJyX4fj9qp0YEnCGGOaxDXDOrH/aAn/W73b16FQWFLm1qp0YEnCGGOaxJld40ltG8EbftCBXVhS7tbkfmBJwhhjmkRQkPCToaks2XKAzK0HfBpLYUk5EW5MyQGWJIwxpslcM6wTybHh3P/OSopLfdc3UVTqR30SIjJaRNaLyCYRmVLD8X+IyHLXa4OI5Lv2n1tl/3IRKRaRS70drzHGeEvrsBAeu7wfm/OO8o9PN/gsDudzEu71Sbi76NApEZFg4GlgFJALLBWR2ap6fMEiVb2nyvl3AYNc+78ABrr2twU2AR97M15jjPG2s7olMGlIR15YsJnRfTowKLVNk96/vEIpLq0g3E/6JIYAm1R1s6qWADOB8XWcPwmYUcP+K4C5quofT6IYY0wj/GZML9pHO7h/1somHxJbuZaEvzQ3JQM5VbZzXftOIiKdcK5T8XkNhydSc/JARG4VkUwRyczLy2tkuMYY433RjlAevawfG/cW8NRnG5v03oUllavS+UeSaIiJwCxVPSGtikgi0A+YV9NFqjpVVTNUNSMhIaEJwjTGmMY7p0c7rjg9hee+3MzXm/Y12X2Pr2/tJ89J7AA6VtlOce2rSW21hQnAe6pa6uHYjDHGp/4wtjddE1pzy2uZLNvWNMNijy845Cc1iaVANxFJF5FWOBPB7OoniUhPoA3OtbOrq62fwhhjmrWY8FCm3zyE9tEOJk9byuodh7x+z8ok4RdzN6lqGXAnzqaitcBbqpolIg+LyLgqp04EZqqqVr1eRNJw1kS+9GacxhjjK+2iHLx+81CiHaFc9/ISNu454tX7/dAn4R/NTajqHFXtrqpdVPXPrn0PqOrsKuc8qKonPUOhqltVNVlV/WdJJ2OM8bDk2HDeuHkowUHC1S9+y65DRV67l781NxljjHFDWnwkb9w8lPzCUp75Ittr9ynyp+YmY4wx7uvePopLByXx9rIc9hcc88o9KmsStuiQMcY0Q7ee3Zni0gpeW7zNK59f2SdhNQljjGmGuraL4oJe7Xht8dbjTUOeZH0SxhjTzN02sgsHC0t5e1lO/Sc3UGFJOaHBQmiwez//liSMMcbPZHRqw6DUWF5YuJmycs8O7iwqKXN7cj+wJGGMMX5HRLjt7C7kHCjif1meXe7Uub61+xOAW5Iwxhg/NKp3ezrHR/L8l5up9pxxoxwuLnW7PwIsSRhjjF8KDhJuPqszq3YcYv56z8xwnZ1XwOfr9jI4ra3b11iSMMYYP3XZacl0SYjkl28tJ+dA45bTUVUenJ2FIzSY+y7s4fZ1liSMMcZPOUKDefH6wVQo3PTqUo4Un/pk2POydrNw4z5+Oao7CVFhbl9nScIYY/xYenwkz1x9Gtl5R7nnP8spr2h4/0RRSTl/+nAtPTtEce2wTg261pKEMcb4uRFd4/njJb35dO1eHv94fYOvf/qLTezIL+Lh8X0JcfP5iEruj4MyxhjjM9cO68S63Ud4dn42fZNiuLh/olvXbdl3lKkLNnPpwCSGpLvfYV3JahLGGNMMiAgPjevDgJQY/jg7i0NF7vVP/OnDNbQKCeK3Y3qd0n0tSRhjTDMRGhzEI5f248DRYzzhRrPTVxv38fm6vdx1XlfaRTtO6Z6WJIwxphnplxLDtcM6Mf2bbazKrX250/IK5ZGP1pDSJpzrh6ed8v0sSRhjTDNz74U9iIsM43fvr6p1tNM73+WybvcRfj26J44GzNVUnSUJY4xpZqIdofxhbC9W5h7izSXbTzpeWFLG4/PWM7BjLGPd7OCujSUJY4xphsYNSGJ4lzj+9r915B05cRW7FxZsYe+RY/xhbC9EpFH38XqSEJHRIrJeRDaJyJQajv9DRJa7XhtEJL/KsVQR+VhE1orIGhFJ83a8xhjTHIgID4/vS3FpORf+3wIenJ3Fytx89h4u5vkF2Yzp14HTOzV8yOtJ9/Hk7IInfbhIMLABGAXkAkuBSaq6ppbz7wIGqeqNru35wJ9V9RMRaQ1UqGqtE5hkZGRoZmamh0thjDH+a+nWA7yyaCufrNlDSVkFUWEhFJeV8+kvR9IpLtKtzxCRZaqaUdMxbz9MNwTYpKqbXYHMBMYDNSYJYBLwR9e5vYEQVf0EQFULvByrMcY0O4PT2jI4rS2HikqZs2oXs5fvZGSPBLcTRH28nSSSgarr7+UCQ2s6UUQ6AenA565d3YF8EXnXtf9TYIqqlle77lbgVoDU1FSPBm+MMc1FTHgok4akMmmIZ38H/anjeiIwq0oSCAHOAu4DBgOdgcnVL1LVqaqaoaoZCQkJTRWrMca0CN5OEjuAjlW2U1z7ajIRmFFlOxdYrqqbVbUMeB84zStRGmOMqZG3k8RSoJuIpItIK5yJYHb1k0SkJ9AGWFzt2lgRqawenEftfRnGGGO8wKtJwlUDuBOYB6wF3lLVLBF5WETGVTl1IjBTqwy1cjU73Qd8JiKrAAFe8Ga8xhhjTuTVIbBNzYbAGmNMw9U1BNafOq6NMcb4GUsSxhhjamVJwhhjTK0Cqk9CRI4ADV8A9gcxQO0TtNd9vKZj1fdV3a7vfTywz62oGxZnfec0tBzVtyvfe6IcdcXpzjn2nZz43r4T92J05xxPfyfg27LEqmrND5qpasC8gMxGXj/1VI/XdKz6vqrb9b1vTFnqK0dd5zS0HHXE3+hyNHVZ7Dvxv7K0lO/E38pS9WXNTSf6oBHHazpWfd8HDXx/qtz5jNrOaWg5qm9/UMs5p6opy2LfiXvsOwn87+S4QGtuytRahnE1N4FSlkApBwROWQKlHGBlaQqBVpOY6usAPChQyhIo5YDAKUuglAOsLF4XUDUJY4wxnhVoNQljjDEeZEnCGGNMrSxJGGOMqVWLSRIicpaIPCciL4rIIl/H0xgiEiQifxaRf4nI9b6O51SJyDkistD1vZzj63gaS0QiRSRTRMb6OpZTJSK9XN/HLBH5qa/jaQwRuVREXhCR/4jIj3wdz6kSkc4i8pKIzPLF/ZtFkhCRl0Vkr4isrrZ/tIisF5FNIjKlrs9Q1YWqejvwIfCqN+OtiyfKgnOd8BSgFOfiTE3OQ+VQoABw4KNygMfKAvBr4C3vRFk/D/07Wev6dzIBGOHNeOviobK8r6q3ALcDV3kz3tp4qBybVfUm70Zau2YxuklEzsb5Y/KaqvZ17QsGNgCjcP7ALAUmAcHAo9U+4kZV3eu67i3gJlU90kThn8ATZXG9Dqrq8yIyS1WvaKr4K3moHPtUtUJE2gNPqurVTRV/VR4qywAgDmfC26eqHzZN9D/w1L8T11ovPwWmq+qbTRV/VR7+N/8E8IaqftdE4R/n4XL45N96SFPf8FSo6gIRSau2ewiwSVU3A4jITGC8qj4K1FjdF5FU4JCvEgR4piwikguUuDbLqx9vCp76TlwOAmHeiNMdHvpOzgEigd5AkYjMUdUKb8Zdnae+E1WdDcwWkY8AnyQJD30nAjwGzPVFggCP/zvxiWaRJGqRDORU2c4FhtZzzU3ANK9FdOoaWpZ3gX+JyFnAAm8G1kANKoeIXAZcCMQC//ZuaA3WoLKo6u8ARGQyrhqSV6NzX0O/k3OAy3Am7TlejazhGvrv5C7gAiBGRLqq6nPeDK4BGvqdxAF/BgaJyG9cyaTJNOck0WCq+kdfx+AJqlqIM+E1a6r6Ls6EFzBU9RVfx9AYqjofmO/jMDxCVZ8CnvJ1HI2lqvtx9qv4RLPouK7FDqBjle0U177mKFDKEijlgMApS6CUAwKnLM2qHM05SSwFuolIuoi0AiYCs30c06kKlLIESjkgcMoSKOWAwClL8yrHqc5f3pQvYAawix+GfN7k2j8G5yiBbOB3vo6zJZUlUMoRSGUJlHIEUlkCoRzNYgisMcYY32jOzU3GGGO8zJKEMcaYWlmSMMYYUytLEsYYY2plScIYY0ytLEkYY4yplSUJY7xIRAp8HYMxjWFJwpgmJiItas4007xZkjCmCcgPq/DNBtb4Oh5j3GX/ozGm6ZwG9FXVLb4OxBh3WU3CmKazxBKEaW4sSRjTdI76OgBjGsqShDHGmFpZkjDGGFMrmyrcGGNMrawmYYwxplaWJIwxxtTKkoQxxphaWZIwxhhTK0sSxhhjamVJwhhjTK0sSRhjjKmVJQljjDG1+n+3K/ymriq1IwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "batch_size = 256\n", "lrfind = model.lr_finder(x_train, y_train, batch_size, tolerance=1.5)\n", "lrfind.plot()\n", "lrfind.get_best_lr()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.02420128264794396" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.optimizer.param_groups[0]['lr']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have found that `get_best_lr` sometimes gives a little high learning rate, so we instead set it to 0.01" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.01" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.optimizer.set_lr(0.01)\n", "model.optimizer.param_groups[0]['lr']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we have introduced shrinkage to the loss function, we also want to monitor the negative log-likelihood (nll) duraing trainig. We can add this to metrics.\n", "\n", "For early stopping, we will use a callback that will stop at the end of the cycle if the current best model was not obtained in the current cycle.\n", "We use the negative log-likelihood of the validation set for early stopping." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "epochs = 512\n", "callbacks = [cb.EarlyStoppingCycle()]\n", "verbose = False # set to True if you want printout" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1min 35s, sys: 11.4 s, total: 1min 46s\n", "Wall time: 38.6 s\n" ] } ], "source": [ "%%time\n", "log = model.fit(x_train, y_train, batch_size, epochs, callbacks, verbose,\n", " val_data=val)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3hUVfrA8e/JZNIoSSDUhBJKqKH3jggioqAiVYoNCyi66i66P11FXdta1hVBxC6KiA0URESK0kM31ISWkADpvc6c3x9ngEkhCUk0DLyf58nDzL3n3jkT4Lz3dKW1RgghxNXHraozIIQQompIABBCiKuUBAAhhLhKSQAQQoirlAQAIYS4SrlXdQYuRUBAgG7atGlVZ0MIIVzKjh074rXWdQofd6kA0LRpU8LCwqo6G0II4VKUUieKOy5NQEIIcZWSACCEEFcpCQBCCHGVcqk+ACHElScvL4/o6Giys7OrOisuz8vLi6CgIKxWa5nSSwAQQlSp6OhoatSoQdOmTVFKVXV2XJbWmoSEBKKjowkODi7TNdIEJISoUtnZ2dSuXVsK/wpSSlG7du1LqklJABBCVDkp/CvHpf4eXSoApGXnVXUWhBDiiuFSASA1O7+qsyCEEFcMlwoAdtm8RghRyZKTk3nnnXcu+boRI0aQnJx8yddNmzaNpUuXXvJ1fwaXCgBS/gshKtvFAkB+fsktDitWrMDPz+/PytZfokzDQJVSw4H/AhZgodb6pULn3wAGO976AHW11n6OczZgn+PcSa31TY7jwcBioDawA5istc4tKR82u0QAIa5kzy4PZ39MaqXes23DmvzrxnYXPT979mwiIyPp1KkTVqsVLy8v/P39OXjwIIcPH2b06NFERUWRnZ3NrFmzmD59OnBhbbL09HSuv/56+vXrx6ZNmwgMDOT777/H29u71LytWbOGxx57jPz8fLp37868efPw9PRk9uzZLFu2DHd3d4YNG8Z//vMfvvrqK5599lksFgu+vr5s2LChwr+bUgOAUsoCzAWGAtHAdqXUMq31/nNptNaPOKV/EOjsdIssrXWnYm79MvCG1nqxUmo+cBcwr6S8yP7FQojK9tJLL/HHH3+we/du1q1bxw033MAff/xxfiz9Bx98QK1atcjKyqJ79+7ceuut1K5du8A9jhw5whdffMF7773H2LFj+frrr7n99ttL/Nzs7GymTZvGmjVrCAkJYcqUKcybN4/Jkyfz7bffcvDgQZRS55uZ5syZw6pVqwgMDCxX01NxylID6AFEaK2PAiilFgOjgP0XST8B+FdJN1RmrNI1wETHoY+BZyglANik/BfiilbSk/pfpUePHgUmUr311lt8++23AERFRXHkyJEiASA4OJhOncxzbteuXTl+/Hipn3Po0CGCg4MJCQkBYOrUqcydO5eZM2fi5eXFXXfdxciRIxk5ciQAffv2Zdq0aYwdO5ZbbrmlMr5qmfoAAoEop/fRjmNFKKWaAMHAr06HvZRSYUqpLUqp0Y5jtYFkrfW5RraS7jndcX1YXiltckIIUVHVqlU7/3rdunX88ssvbN68mT179tC5c+diJ1p5enqef22xWErtPyiJu7s727ZtY8yYMfzwww8MHz4cgPnz5/P8888TFRVF165dSUhIKPdnnP+sCt+hoPHAUq21zelYE631KaVUM+BXpdQ+IKWsN9RaLwAWAFQLDJE6gBCiUtWoUYO0tLRiz6WkpODv74+Pjw8HDx5ky5Ytlfa5rVq14vjx40RERNCiRQs+/fRTBg4cSHp6OpmZmYwYMYK+ffvSrFkzACIjI+nZsyc9e/Zk5cqVREVFFamJXKqyBIBTQCOn90GOY8UZD8xwPqC1PuX486hSah2mf+BrwE8p5e6oBZR0z/OkD1gIUdlq165N3759ad++Pd7e3tSrV+/8ueHDhzN//nzatGlDq1at6NWrV6V9rpeXFx9++CG33Xbb+U7g++67j8TEREaNGkV2djZaa15//XUAHn/8cY4cOYLWmiFDhtCxY8cK50GV1rGqlHIHDgNDMIX0dmCi1jq8ULrWwE9AsHbcVCnlD2RqrXOUUgHAZmCU1nq/Uuor4GunTuC9WusSB+N6Nmips2MOy7RxIa4gBw4coE2bNlWdjStGcb9PpdQOrXW3wmlL7QNwPKHPBFYBB4AlWutwpdQcpdRNTknHA4t1wYjSBghTSu0B1gIvOY0e+gfwN6VUBKZP4P2yfLmcfHtZkgkhhChFmfoAtNYrgBWFjj1d6P0zxVy3CQi9yD2PYkYYXZKsXBteVsulXiaEEH+pGTNmsHHjxgLHZs2axR133FFFOSrK5fYDyMyz4V/VmRBCiFLMnTu3qrNQKpdaCqIGmWTmyFBQIYSoDK4XAHJtpScUQghRKpcKAG5oCQBCCFFJXCsAKE1mrjQBCSFEZXCpAKCwSw1ACFGlqlevftFzx48fp3379n9hbirGpQKAaQKSGoAQQlQGlxoGKn0AQlzhVs6G0/tKT3cp6ofC9S9d9PTs2bNp1KgRM2aYVWyeeeYZ3N3dWbt2LUlJSeTl5fH8888zatSoS/rY7Oxs7r//fsLCwnB3d+f1119n8ODBhIeHc8cdd5Cbm4vdbufrr7+mYcOGjB07lujoaGw2G0899RTjxo2r0NcuC5cKAEoCgBCiko0bN46HH374fABYsmQJq1at4qGHHqJmzZrEx8fTq1cvbrrppktahmbu3Lkopdi3bx8HDx5k2LBhHD58mPnz5zNr1iwmTZpEbm4uNpuNFStW0LBhQ3788UfALEL3V3CpACBNQEJc4Up4Uv+zdO7cmbNnzxITE0NcXBz+/v7Ur1+fRx55hA0bNuDm5sapU6c4c+YM9evXL/N9f//9dx588EEAWrduTZMmTTh8+DC9e/fmhRdeIDo6mltuuYWWLVsSGhrKo48+yj/+8Q9GjhxJ//79/6yvW4CL9QFIJ7AQovLddtttLF26lC+//JJx48axaNEi4uLi2LFjB7t376ZevXrF7gNQHhMnTmTZsmV4e3szYsQIfv31V0JCQti5cyehoaH83//9H3PmzKmUzyqNS9UAFJrMHAkAQojKNW7cOO655x7i4+NZv349S5YsoW7dulitVtauXcuJEycu+Z79+/dn0aJFXHPNNRw+fJiTJ0/SqlUrjh49SrNmzXjooYc4efIke/fupXXr1tSqVYvbb78dPz8/Fi5c+Cd8y6JcKgC4ocnMkwAghKhc7dq1Iy0tjcDAQBo0aMCkSZO48cYbCQ0NpVu3brRu3fqS7/nAAw9w//33Exoairu7Ox999BGenp4sWbKETz/9FKvVSv369XnyySfZvn07jz/+OG5ublitVubNK3F33EpT6n4Al5NuDd11p39vYuG0S15EVAhxmZL9ACpXpe4HcHnR5ORUTjucEEJc7VyqCQjAlptV1VkQQlzl9u3bx+TJkwsc8/T0ZOvWrVWUo/IpUwBQSg0H/gtYgIVa65cKnX8DGOx46wPU1Vr7KaU6AfOAmoANeEFr/aXjmo+AgVzYIH6a1np3aXmx52SUJctCCBeitXaprV5DQ0PZvbvU4uovd6lN+qUGAKWUBZgLDAWige1KqWVOWzuitX7EKf2DmI3fATKBKVrrI0qphsAOpdQqrXWy4/zjWuull5Jhu9QAhLiieHl5kZCQQO3atV0qCFxutNYkJCTg5eVV5mvKUgPoAUQ4tnBEKbUYGAXsv0j6CcC/HBk67JS5GKXUWaAOkHyRa0uXLwFAiCtJUFAQ0dHRxMXFVXVWXJ6XlxdBQUFlTl+WABAIRDm9jwZ6FpdQKdUECAZ+LeZcD8ADiHQ6/IJS6mlgDTBba51TzHXTgekAXRu4SQ1AiCuM1WolODi4qrNxVarsUUDjgaVa6wKD9ZVSDYBPgTu01nbH4SeA1kB3oBbwj+JuqLVeoLXudm4Ik7s9G5vddYauCiHE5aosAeAU0MjpfZDjWHHGA184H1BK1QR+BP6ptd5y7rjWOlYbOcCHmKamUnmRK+sBCSFEJShLANgOtFRKBSulPDCF/LLCiZRSrQF/YLPTMQ/gW+CTwp29jloByvT6jAb+KEuGvcglS9YDEkKICiu1D0Brna+UmgmswgwD/UBrHa6UmgOEaa3PBYPxwGJdcBzSWGAAUFspNc1x7Nxwz0VKqTqAAnYD95Ulw6YGIAFACCEqqkzzALTWK4AVhY49Xej9M8Vc9xnw2UXueU2Zc+nES+WSIU1AQghRYS62FAR4kyNNQEIIUQlcLgB4kSdNQEIIUQlcLAAovFSOjAISQohK4FoBQCnpBBZCiEriYgHADW9yyZAAIIQQFeZyAcDMA5AmICGEqCiXCwDeKkeagIQQohK4WABQ+LjJKCAhhKgMLhYA3Kim8mQUkBBCVAKXCwDeKpfMHKkBCCFERblcAPBR0gQkhBCVweUCgJfKJTNPAoAQQlSU6wUAcsnMkT4AIYSoKBcLAApPLcNAhRCiMrhYAHDDQ8taQEIIURnKFACUUsOVUoeUUhFKqdnFnH9DKbXb8XNYKZXsdG6qUuqI42eq0/GuSql9jnu+5dgZrJSMOAKANAEJIUSFlbohjFLKAswFhgLRwHal1DKt9f5zabTWjzilfxDo7HhdC/gX0A3QwA7HtUnAPOAeYCtms5nhwMpSMoMbdvLyci/lOwohhChGWWoAPYAIrfVRrXUusBgYVUL6CVzYGP46YLXWOtFR6K8Ghjv2A66ptd7i2ELyE8y+wCVTJrv23AwK7jwphBDiUpUlAAQCUU7vox3HilBKNQGCgV9LuTbQ8brUexaXXQ+dS06+vQxZF0IIcTGV3Qk8Hliqta60YTpKqelKqTClVFhaegaAmQ0sI4GEEKJCyhIATgGNnN4HOY4VZzwXmn9KuvaU43Wp99RaL9Bad9Nad6tR0xfAsSmMdAQLIURFlCUAbAdaKqWClVIemEJ+WeFESqnWgD+w2enwKmCYUspfKeUPDANWaa1jgVSlVC/H6J8pwPel5sTRB2D2BJAagBBCVESpo4C01vlKqZmYwtwCfKC1DldKzQHCtNbngsF4YLF26p3VWicqpZ7DBBGAOVrrRMfrB4CPAG/M6J+SRwABOEaKepNLugwFFUKICik1AABorVdghmo6H3u60PtnLnLtB8AHxRwPA9qXNaPAhRqAyiUlK++SLhVCCFGQy80EBvAiRwKAEEJUkIsGgFySMyUACCFERbhkAPBWuSRlymxgIYSoCBcLAKYT2Nc9X2oAQghRQS4WAEx2/T3ypQ9ACCEqyCUDgK+7TZqAhBCiglwrAAC4e1PTPU+agIQQooJcLwBYvahhkSYgIYSoKBcMAD5Uc8uXJiAhhKgg1wsA7l5UczMzge122RNACCHKy/UCgNUHb5WL1pCWLesBCSFEeblgAPDCS5vmH2kGEkKI8nPBAOCNJzkAJEtHsBBClJvrBQB3b6zaEQCkBiCEEOXmegHA6oXVfi4ASA1ACCHKywUDgA8WWzYgNQAhhKiIMgUApdRwpdQhpVSEUmr2RdKMVUrtV0qFK6U+dxwbrJTa7fSTrZQa7Tj3kVLqmNO5TmXKsbsXbucCgPQBCCFEuZW6I5hSygLMBYYC0cB2pdQyrfV+pzQtgSeAvlrrJKVUXQCt9VqgkyNNLSAC+Nnp9o9rrZdeUo6tPqi8LGp4uUsTkBBCVEBZagA9gAit9VGtdS6wGBhVKM09wFytdRKA1vpsMfcZA6zUWmdWJMNYvSAvCz9vd2kCEkKICihLAAgEopzeRzuOOQsBQpRSG5VSW5RSw4u5z3jgi0LHXlBK7VVKvaGU8izuw5VS05VSYUqpsLi4OLB6g7ZRx8dNmoCEEKICKqsT2B1oCQwCJgDvKaX8zp1USjUAQoFVTtc8AbQGugO1gH8Ud2Ot9QKtdTetdbc6deqAuzcAAV6aJGkCEkKIcitLADgFNHJ6H+Q45iwaWKa1ztNaHwMOYwLCOWOBb7XW50tsrXWsNnKADzFNTaWzngsAdlKkCUgIIcqtLAFgO9BSKRWslPLANOUsK5TmO8zTP0qpAEyT0FGn8xMo1PzjqBWglFLAaOCPMuXYEQDqeNqkCUgIISqg1FFAWut8pdRMTPONBfhAax2ulJoDhGmtlznODVNK7QdsmNE9CQBKqaaYGsT6QrdepJSqAyhgN3BfmXLsCAC1PGykZGlsdo3FTZXpUiGEEBeUGgAAtNYrgBWFjj3t9FoDf3P8FL72OEU7jdFaX3OJeTU8qgMQYMlCay/SsvPw8/Eo162EEOJq5nozgRt0BKBJ5j5AloMQQojycr0AUL0u1GlDg6QwQJaEFkKI8nK9AAAQ3B+/+DCs5EtHsBBClJOLBoABWPKz6KgiSJEmICGEKBfXDABN+qJR9HHbL01AQghRTq4ZAHxqQYMO9LGESyewEEKUk2sGAEA17U8XtyNkZKRXdVaEEMIluWwAIHggHuTjn7CzqnMihBAuyXUDQJPe2HCjUUpYVedECCFckusGAM8aHPVoTUjmrqrOiRBCuCTXDQBATI32NM0/irblV3VWhBDC5bh0AHBvEIoXuZw+vr/0xEIIIQpw6QDQIKQrANEHt1dxToQQwvW4dABo0roLNq3IOLmnqrMihBAux6UDgMXDm9PWRngmHqrqrAghhMtx6QAAkObXisDcSDJypCNYCCEuRZkCgFJquFLqkFIqQik1+yJpxiql9iulwpVSnzsdtymldjt+ljkdD1ZKbXXc80vHdpOXzDMwlMYqjj8io8pzuRBCXLVKDQBKKQswF7geaAtMUEq1LZSmJfAE0Fdr3Q542Ol0lta6k+PnJqfjLwNvaK1bAEnAXeX5AnVbmI7gk4d2lOdyIYS4apWlBtADiNBaH9Va5wKLgVGF0twDzNVaJwForc+WdEPHRvDXAEsdhz7GbAx/yao16gBAZpR0BAshxKUoSwAIBJzbV6IpusdvCBCilNqolNqilBrudM5LKRXmOH6ukK8NJGutzzXcF3dPAJRS0x3Xh8XFxRVN4NuIbLdqeCUexG7XZfg6QgghoPI6gd2BlsAgYALwnlLKz3Guida6GzAReFMp1fxSbqy1XqC17qa17lanTp2iCZQi3a8VzezHiYyTlUGFEKKsyhIATgGNnN4HOY45iwaWaa3ztNbHgMOYgIDW+pTjz6PAOqAzkAD4KaXcS7hnmXkEhtJKRbHjeGJ5byGEEFedsgSA7UBLx6gdD2A8sKxQmu8wT/8opQIwTUJHlVL+SilPp+N9gf1aaw2sBcY4rp8KfF/eL1GjcSdqqiwiIw6W9xZCCHHVKTUAONrpZwKrgAPAEq11uFJqjlLq3KieVUCCUmo/pmB/XGudALQBwpRSexzHX9Jan1u45x/A35RSEZg+gffL+yVU/fYAZEZLR7AQQpSVe+lJQGu9AlhR6NjTTq818DfHj3OaTUDoRe55FDPCqOLqtkGjqJ8WTmJGLrWqlWtKgRBCXFVcfiYwAJ41SKvfixFuW9l1QvoBhBCiLK6MAAB4d76N5m6xRB3YVtVZEUIIl3DFBABr+9HkY8Hv6PKqzooQQriEKyYAUK02x2r2oHv6r+Tl26o6N0IIcdm7cgIAkNFyFIEqnhN711d1VoQQ4rJ3RQWA+j1vIUdbydn9VVVnRQghLntXVgCoW4/Nli4EnfoJ7NIMJIQQJbmiAgDA4foj8bUlwu5FVZ0VIYS4rF1xAcCj7Ui220PIX/0sZKdUdXaEEOKydcUFgBEdGvK8bSqWrATY8GpVZ0cIIS5bV1wAqFvTi6B2ffiOQegt8yE+omCCjASIlTWDhBCiTGsBuZrJvZowc+9YRlbfhvWTUdD2JmjUE478DPuWgi0HRs+DThOrOqtCCFFlrsgA0DO4FrXqBfKsfTbP1V2P2v4+bHkHrNWgy2SIPwLfzwCPatC28O6WQghxdbgiA4BSism9mvDU9+ncettkOo/1hJhdUL89ePlCbgZ8ejMsvQsmVoMW11Z1loUQ4i93xfUBnHNzlyCqe7rzwcbj4OEDTfuawh/Mk//EJVCnFXx9D6SXuIe9EEJcmuQo2PEx7PwU9nwJp/eV/Vq7zTRVr3/VtFb8icpUA3Bs8v5fwAIs1Fq/VEyascAzgAb2aK0nKqU6AfOAmoANeEFr/aUj/UfAQODcWM1pWuvdFfo2Tqp7ujOldxPeWRfJhO6N6NMioGACbz+49X14dwAsnwXjPwelKuvjhRCuyG6H7GTzk58DAa3AzfGcHB8Ba56B+h2g78PgXsy+I6f3waa34Y+lYM8veK5JP+h1PzTuDT61THmTlw3JJyArCfKyzOtN/4MEx+CVtc9DYDdoOdR8bt3W4FEDLFaweIC7Jyg3SDsNiZGQmQgBLaF2C5OmFMrs5VJCAqUsmD1+h2L2/t0OTHDa2QulVEtgCXCN1jpJKVVXa31WKRWC2S/miFKqIbADaKO1TnYEgB+01ktLzaVDt27ddFhYWFmTk51nY/ibG7BrWPXwALw9LEUTbXobfv4njHoHOk8q872FEGWUEAlfTYMe90CXKcWnSY+D3DRwcxRsFqv58aj+1z2YxeyGpXdA4tELx/yaQNepYMuH314zhW1+FtQLhZFvgGcNSD8NJ7dA+HcQd8D0NXadCl2ngdUH8rPh8E+w9V1IiTL39ahuftJPF81HvfYw8O8Q1MMEkr1fwuk/MM/WxVFFz7lZod3NcNP/wOqFUmqH1rpbkSvLEAB6A89ora9zvH8CQGv9olOaV4DDWuuFpdxrDzDGERA+4k8OAACbIxOY8N4Wpg9oxpMj2hRNYLfDxyNN5L5rtYmwQriarQsg4heYsPjCE+tfKSPeFHYePgWPxx+Bj0aags7iAXeugsAuBdPsWQzfPQC6mOVbagZCqxHQZiQEDyw9GGQkwMHlcGS1KZRbj4DrXgTP6kXT5qSBLc88Re/7Clb8HXxqQ+8Z5k97HuxdAsd/M+nb3QLDX4JTYfDDI5B+xulmCpr0gbajIXSMecIvzJYPR9dBwhFIOgE5qSbA1AqGagHg7mWCQr32Rf8OczPgTDjEHzY1BVueGc1oywNbLlSvB7Wagbe/qT1EbYPt70HT/jD+c5S3b7kDwBhguNb6bsf7yUBPrfVMpzTfYWoJfTHNRM9orX8qdJ8ewMdAO6213REAegM5wBpgttY6p5jPnw5MB2jcuHHXEydOlJjf4jzxzV6+3B7Fspn9aB/oWzRB0nFYONRE6jEfQkvpFBYuJDUW/tcF8jLhto+h3ejK/4yzB82Tb/2ORQun7FT4b0fzutf90P1uU2CdPQDLZoK2m3x9Mx3cLHDvBtMEC7DjI1j+MAT3h44TTaFryzUFW34ORG+HiDXms5sNghvfAv8mRfOXnwNb5sGG/5iahG8j02RyaIUpGEfNNX1+Fg84sQnCPoAjq0zezmk2GG5daApjZ/ERprB2DlyZibD/O/CsCTXqQ+2WUKNeBX/JlWzvEvjufqjXDnXfb39qAPgByAPGAkHABiBUa53sON8AWAdM1VpvcTp2GvAAFgCRWus5JeWlPDUAgJSsPAa8spYewbV4b0qR34GRHAVfTICz4XDN/0HP+0xnsRCXu+8eME+w1eubgQ73/VZ5zSZ2G6x/BTa8YgrLGg2g9Q0w5OkLgyrONaM27X/hafmcanVh6nJTs47aBh9eD036QuNekBJt1uxqOQzGfgJW7+LzkJtp0v3yDGgNvR8whbpPAKTFmNr7kZ8h+SSEDDf/f+u1N7+D47+bwJN6qmi+Ok0w3yc/B6rVgY7jTYC6khz+GZZMQT115k9tApoPbNVaf+h4f+6JfrtSqiam8P/3xZp7lFKDgMe01iNLykt5AwDA66sP89aaI/z8yABC6tUoPlFOuomYB5aBl59pw+txD/gGXfzGdhuEf2v+cQ+dA1avcuVPXKXysk278ZlwyEyANjeZJoGyitkNCwZBn5lQtx18d59pBmp1vXlKPbIa2t9Spg7B87Q2hfPZA7DxTTixETpOgOABcGglHPzRFJ6j5pon9f92Av+mcMePpq364A8XmiQadroQKAC2zIdVT5hg4uULIdfDTW+ZZpjSJJ80TS8RvxQ87lHDfE6/R6DFkKLXZSWZPOdmmOaTWsHmc4vrxL0SJUeh/BuXOwC4Y5p3hgCnMJ3AE7XW4U5phmM6hqcqpQKAXUAnIA1YCSzXWr9Z6L4NtNaxSikFvAFka61nl5SXigSApIxc+rz0K9e3r8/r4zpdPKHWELXVTBw7sBxQZrJY97vM04LFwzQVpcaY9rit75o2PYDr/m3aD4Uoi5Nb4YvxkJVY8HizQTDwH6ZN+Zy00+YpOCvZtF171oDazWH3F+bf34M7TY31f11N+/X1r5gOzZQo82DSd5a5j90G6182zSOtbyhaUzgTDp/dCmmx5r21Gtzwminwz1n9LxMYpi6HtDPwzd0w4UtoNbxs3zs3w7R3l/dpOyfNDN3OiIfqdcCvadX0e7iQcncCOy4eAbyJad//QGv9glJqDhCmtV7mKMRfA4ZzYbjnYqXU7cCHQLjT7aZprXcrpX4F6mC6sHcD92mt00vKR0UCAMCc5fv5ePNx1j8+iCB/n1LTk3QCti0wY3lzLrKyaL32MOBx06Z4Jhxm7Sm+w0kIZxG/wJeTzUPFkKfMqBJ3T9jzhWkXz0qG6eugToipJbx/rWnqcPcyhX92qukEBFNAd7/bvN7xkRnWrNxMzbVmkFn7auZ28A2EtS/Cesco7ka9THBo3NO8TzkF7w81T+cDHoe6baFeO/CqWTDveVnwTm/zGVYfk48HtkohfBmrUAC4XFQ0AMSmZDHglbVM7NGYZ0e1L/uFOWmm9z4303RQWTygZkPz4x9s/uFHh8HCIXDNUzDgsQvXam2GgEVtg/6Puk5w2Pmp6Rwf8lRV58T1nNlvOi87Tii+meGPr+Gbe027+O3fQPW6Bc+nxsD8fqYZ5e41sOpJ2PFhwadsu80002TEQWDXC0/y+bmw8BrT/HLjf82S6HN7miahrnfAJ6Ogwzho0hvW/tuMZAkeAD3vh7UvmIeeO1dC/dCSv2PkWvjU0dl841tm2KO4bEkAcPj70j18vzuGLU8Mwb9aJbcBfj4eTm6CWXvNKIe4Q+Y/77k2yzptYPwiU1CEBSMAACAASURBVHVPjYWYnWZo2+UWFLKS4M0OZuTDgztNfkXxUmPN0Mdz7dyZiTC/P6RGQ63mplkw5DpTQNvtpvll/UtmMtCExRdGwxQW8Qt8Nsa0bcfsMhOPhj5bvjyuf8UU7l6+Jqjcs9b8m8tJNzXXLe+YJh83d5j0FTS/pmz3Xf6weTB6YIv0fV3mJAA4HDqdxnVvbuCJ61tz78BKLthO7zNPbnVamyp8+mkzTGzQExAQAt/cY2YH1mllnhDB1CBunm9GRVwuzjUTuLlD93vg+iITvytPVrJpy/VvUrCj0pYPlkIT1WN2QdxhE5BqNzdjnksSs8u0V0dvN8MC/Zs6/TQxf/o1KX8AProOFk8y+bj9a/N3vGQyHPoJrnsBti80/UR+Tczfb3aKqQ12mmQmEZXW8bnmOfjtPyZYTP2h6O+jrPKy4Z1eppC/e41ZE8tZfg788Y1pT7/UdbHyc6+ezlQXJgHAybh3N3MqOYv1jw/G4lbJswx/ehKObzB9A/XaQ4exF6r4SSfMKqQ5aWZiS+2WsPopU5Xv7pglWa9d0Y652D2mWaBRz+InmJRHfq4ZN174CTQ7Bd4INeOyrT5m1MejB0y78zkp0bDrM1Pwdb/7Qmfeic1m2F3vGUUnBDnLzYRvp8Ox38yUezBt2w06mqfUuINmWG7IcFNQ1qhvxnj//H8FJwv5B5uCtUFHM2rLq6a5d1oMnNppxmn71DYzItPPmCatxONmnLizuu1gwKNmEs+575KfazpQk46ZJp2YXaYwb9zL/D0lHYev7zZNLZmJpmmwwzjY9i4Mex76PGhGyOz6DCLXmCbAjHi49hlzrizDNG35puO31QhTOFdE8kmTz4YlDIAQVywJAE5+3BvLjM93snBKN65tayZvZOfZ8LJWwRjgnDRY9U9TUGibqT00G2SCh2cN8xR5fmy1MsdDx5ghqt5+Zgr93i9NbcOvCfg1Nvfwa3yh2SH9tBkh4uVr+iQOrTBNU5lJcNcqqOs0Q3r9q2b9kXs3mAJo4TVw/avQc7qZ0v/LM2ZI3bmCuEk/GPW26Xzc+F9AmzyO/eRC05HdfqGD0JYHiyeaJo7Ot5sg6FPbdKCfCjMjROq0MmO8d35snpLP7eXQeqSpTSWfhPhDpt8laqtpBy/M0xd63G1GvzgPQ9TaNHElHXMEhKNm4a24g+YJvlpdczz1FAWm1/s2MoX9yS0XOl8b9TTNONkpZuRMYiQ0HwKTlhbtENXaBImyDHcUopJJAHCSZ7PT7+VfaVW/Jh/f0Z33fjvKKz8d4o1xnbixY8NKyGk5pMfBge/NeiKndkJehjleo6GZXRnYxTxhR/wCUVvMlPFGPcxTtD3PdEzbci/cz8vXjDBJOmFmUYIpwLx8zdNsQCvz9G3xhHvWmFpK2hmY28M0OUxcbK55b4hJ1+8RM1XezR26TYNud5nx4T8+diGvXaZAi6Gw/CHTSdmkr5lYlxJtjve4x3SA7vkCRr4J3e4o+XcSH2FqTFFbYNCTZmRKcQVrZoLpr8hONZOJajQoOnKlJHa7qS1smWdGthRuKgoIuTA7NCvJzLBMOgHX/PPCZMGMBBOsu99VdCapEFVMAkAhb605wuurD3NDaAN+3BeLj4cFb6uFNY8OxM+nits07XbzhJoWa54yC0/iid1rOu5ObDJPxV2mmEIqI848vZ4NN/0RaWdMAVarmXlKjd1tzneaZJpuTu+DD0eYGkDTfqYAs+XB3b9caCrY+5UZ5w1mpufN75rhhOckRJrRJO1vNeuugGm+WTbTdJDWa2eCS/i3F9ZOGfxPs9hVWX8X6afNiCshRLlIACjkbFo2fV/6lTybZsbg5owIbcBNb2/k1i6BvDKmY6V8hks48AN8ebt5HXqbmYAU0OLC+fxcWDIFgrqZWkB5J+/k55oZ1tkp0O1OWXpbiL+QBIBifBUWhY+HOzd0aADAiysP8O76oyye3otezWpX2udc9k5sMm3udUKqOidCiD+BBIAyyMq1MezN9Xi6W1j18IDKHyEkhBBV4GIBQOZuO/H2sDB7eBsizqazKryYjRqEEOIKIgGgkOHt6xMcUI356yM5Vzv6/Ug8Q15bR8TZtFKuFkII1yEBoBCLm+Ke/s3YG53C5sgEEtJzeGTJbiLjMnjjl4tv0JydV8xuRkIIcRmTAFCMW7oEElDdk3nrI5n9zT5SMvMYEVqfFftiOXKmaC1g3rpIujy3mgOxqVWQWyGEKB8JAMXwslq4s19TfjsSz+r9Z/j78Fa8MDoUH6uFt36NKJB2U2Q8r646SGaujX8tC8eVOtWFEFc3CQAXcXuvJvj5WOnfMoA7+wbjX82DKX2a8sPemPN9AWfTspm1eDfBAdX454g2bDuWyPK9sVWccyGEKBsZBlqC+PQcfL2tWC0mTiZm5NLv5V9p5O9Dy3rVOXwmjZOJmXw/ox8t6lZn1NzfiU/LZc2jA6nmWc6VG4UQopJVaBioUmq4UuqQUipCKVXsto1KqbFKqf1KqXCl1OdOx6cqpY44fqY6He+qlNrnuOdbjl3FLisB1T3PF/4Atap58Ph1rbBpzf6YVGx2zWu3daJV/RpY3BTP3tSO06nZTHhvC+MXbGbMvE1sjkyowm8ghBAXV5Y9gS2YPYGHAtGYPYEnaK33O6VpCSwBrtFaJyml6mqtzyqlagFhQDfM0oo7gK6ONNuAh4CtwArgLa31ypLy8lfXAMrjpZUHWXvwLL7eVk4lZ5Gance3D/SlRd3LbNMXIcRVoyI1gB5AhNb6qNY6F1gMjCqU5h5grtY6CUBrfdZx/DpgtdY60XFuNTBcKdUAqKm13qJNBPoEGF2ub3aZmX19a1Y9MoAl9/Xmy3t74enuxl0fbycpI7f0i4UQ4i9UlgAQCEQ5vY92HHMWAoQopTYqpbYopYaXcm2g43VJ9wRAKTVdKRWmlAqLiytm3ffLWJC/D+9O7kZsSjZ3frydHSeSZJSQEOKyUVmjgNyBlsAgYALwnlLqIpudXhqt9QKtdTetdbc6dSq4K1IV6NrEn9fHdiTiTDq3ztvEyP/9zuJtJ8nKlYljQoiqVZYAcApo5PQ+yHHMWTSwTGudp7U+hukzaFnCtaccr0u65xVjZIeGbHlyCC/c3B6bXTP7m330/PcvPP/DflKz86o6e0KIq1RZAsB2oKVSKlgp5QGMB5YVSvMd5ukfpVQApknoKLAKGKaU8ldK+QPDgFVa61ggVSnVyzH6ZwrwfWV8octVNU93JvVswspZ/Vlyb28GtqrLh5uOc/vCrSRnSv+AEOKvV2oA0FrnAzMxhfkBYInWOlwpNUcpdZMj2SogQSm1H1gLPK61TtBaJwLPYYLIdmCO4xjAA8BCIAKIBEocAXSlUErRI7gW/5vQmQWTu3LwdBrjF2whLi3nT/9sm12zcl+sBBwhBCATwarc70fiueeTMGp4uXNzl0Bu7NCQhn7epGfn4+ZmOpIry5KwKP6+dC++3lYeubYlk3o1ISMnn7NpOTSu5YOXtZy7fQkhLmuyIcxlbNfJJP675gi/H4kn317w7+POvsE8MaJ1gQlp5ZFvszPk9fV4uVsIqOHBxogELG4Km+PzxnYLurq2whTiKnKxACDrFVwGOjf256M7epCUkcvqA2fIyMmnuqc7e6KT+WDjMfbHpvD2xC4EVPcscm1KVh6/H4mnvq8nXZvUAkBrzWs/H2ZV+Gnen9qdxrV9WLYnhhMJmSyY3JWhbevx68GzbD2WSN0anmw/nsi3u07x6LBW1Kvp9Vd/fSFEFZEawGXum53RPPHNPqp5unP/wOZM7t2E7DwbP+yN5Ye9MWw/nnT+Kf7ufsH8bVgIzywLZ0lYNFaLoqGfN0vu7c2EBVvwtFpY8VA/Cq+6cSIhg8H/Wce9A5vzj+Gtq+JrCiH+RNIE5ML2x6Ty4soD/HYknlrVPEjLziPPpmlZtzrD2tVjUKu6LN8TwyebT1DDy5207HxmDWnJwFZ1mPTeVnw8LCRk5DJvUheuD21Q7Gfc/9kONkbEs/mJIbKQnRBXGGkCcmFtG9bk07t6suVoAh9uPEaQvw+3dAmkbYOa55/muzetxYCWdfj3ygM8fl0rpvRuCsC827tw98dhhNSrznXt6l/0M+4Z0IyVf5xmSVgUd/QN/iu+lhCiikkN4CqwNzoZfx8PGtUqeUTRmHmbOJ2azfcz+lK7mP4GIYRrqtBy0MK1dQjyK7XwB5gxuAXRSVl0f+EXJizYwudbT5KRk19s2k+3nOCTzccrN6NCiL+U1ABEAQdPp7Jibyw/7oslMi6DGl7ujOkaxAODWlCnhqkVbIqIZ+LCrQDMv70Lw9sX369QmNYarcHN7bLb+kGIK5p0AotLorVm58kkPt50gpV/xNLQz5vP7upJTW8r17+5AS+rhRpe7hyNy2D5g/1oGlCt2PskZeTy6s+HCI9J5ejZdPyrefDDQ/2o6WX9i7+REFcvCQCi3HadTGLah9vxsrrRtkFNNhyJ5+v7+xBQ3YMb3vqdBr5ejOveiGPxGSjgwSEtCajuSVp2HpMWbuVgbBrdg/1p5O/DkrAobuvaiJfHdKjqryXEVUNGAYly69zYny/v7cXk97ex9lAcD1/bkk6NzGrfb47rxF0fb+fZ5fup4elOTr6dH/bGMmdUez7edJz9Mam8O7krQ9rUA8DPx4P56yO5oUMDBoS43vLeQlxJpAYgyiwqMZOf959hSu8mBZamOJuajVKKgOoeHD6TzsNf7uZAbCpuCt6a0JmRHRqeT5udZ+OGt34jK9fGdzP6Ut3LHQ+LG+4VXOrinLi0HH45cIZdJ5OqfGZzbr6dez4JY0RofcZ1b1xsGrtdS5+I+NNJE5D4y+Tk21j42zGaBVQrduLZrpNJ3DpvE+eWPbK4KdoH+tIruBYB1T2JOJvO0fh0LG4Kfx8P6tX0olezWvRpEXDRvgO7XfPoV3v4bvcpzv2THt2pIW+O7/xnfc1SzVsXycs/HaRZQDXWPDqwyAzsY/EZjF+wmb8NDblogBCiMkgAEJeVzZEJhMekkGfTpGTlseNEIrujksmzaQKqe9KsTjXQkJiZS0xyFpm5Nixuitt7NubZUe2L3G/hb0d5/scDTOvTlHHdG7FsTwzz1kWybGZfOgRdfHO6PJudU0nm/nk2O20a1MTD/dJrI9FJmew4kcTg1nWp6WUlOimToa9vwNvDQmJGbpF8ZObmc/PcTRw6k0azOtVY87eiAUKIyiJ9AOKy0rt5bXo3r13gWHaejew8G34+HgWO59ns7DqZzOLtJ/l48wm6NPFnVKcLW0gfiE3llZ8OMbRtPf51Y1uUUgT5e7NkexQv/HiAxdN7FSlctdas3n+G537cT1Ri1vnjXZv4s2By1zJPhPv14BneXX+UrcfMNheBft68PrYj7/9+DIDP7urJ6Lkb+W5XzPkAoLXmiW/2cfhsGrd2CeLrndGEnUiie9NaZfztCVE5ZCKYuGx4WS1FCn8Aq8WNHsG1eOXWDnRp7MdT3/1BbIoptLPzbMxavAtfHysv3RJ6vqCv4WXl4WtbsvVYImsOnC1wv7i0HKZ+uJ3pn+7Ay93Ci7eEMm9SF54b1Y4/TqVw8zubiIxLLzW/EWfTuffTHcSmZPPo0BDen9oNq0Ux/r0t/Lz/DLOubUnbhjUZ3LoOy/bEkG+zA/DhxuN8vzuGx4a14rnR7aju6c7ibVEV/fUJccnKFACUUsOVUoeUUhFKqdnFnJ+mlIpTSu12/NztOD7Y6dhupVS2Umq049xHSqljTuc6Ve5XE1cad4sbr4/tRL5d8+iSPby15ghDXlvP4TPp/Oe2jkWe2sf3aEyzgGo8szyco44CPTEjl9sXbmX7sUSeHtmWFbP6M6FHY64PbcDk3k35YnovMnLyuXnuRr4Ki+JiTaRaa578dh/eVgtf39+HB4e0ZEibevz4UH8m9WzMwJA63OlYU+nmzoHEp+ewKTKBDYfjeP7H/QxtW4/7BzbHx8Odmzo15Md9MbI/tPjLldoHoJSyYDZ5H4rZ/H07MEFrvd8pzTSgm9Z6Zgn3qYXZ/jFIa52plPoI+EFrvbSsmZU+AAHwxbaTPPHNPgD6tQjgjr5Nzw8zLWznySTu/jiMvHw7z41uz3u/HSXibDofTutOnxYBxV4TlZjJI1/uJuxEEv1aBPDiLaFFltI4t7vai7eEMqFHyR242Xk2erzwC20b1iQ8JpVAP2++vr/P+VVX90Ync9PbG3ludHsm9WjM1mOJ+HhY6Njo4n0XQlyKivQB9AAitNZHHTdaDIwC9pd4VVFjgJVa68xLvE6IAsZ3b4S/jwdtGtSgSe3iZyCf06WxP8tm9uX+z3by8Je7sVoU703pdtHCH6BRLR+W3NubRdtO8vLKg9z49u+8N6Xb+Tb6uLQc/r3iAN2b+jOuW6NS8+tltTAitAGLt0cRUN2DhVO7FVhyOzTQl9b1azB/XSQf/H6MY/EZAIwIrc8T17cp0zpOF3PkTBr7TqVw6HQaPh7uTOzZ+PySHkKUpQYwBhiutT7XrDMZ6On8tO+oAbwIxGFqC49oraMK3edX4HWt9Q+O9x8BvYEcYA0wW2tdZGd0pdR0YDpA48aNu544caJcX1Rc3bLzbMxdG0G3prUYeAkT0E4mZDLtw21EJ2fx4s2hxKZk8d5vx8jMzefHh/oTUq9Gme4THpPCw4t389Ktoed3bnO2aOsJ/vntH3Rp7Mfk3k04mZDF/PWR2LTmzXGdGHGRfRzOybfZSc3Op1a1C30oP+yNYebnuwDwsLiRZ7djtbgxpmsQ0/s3u+jyHeLKU+5hoGUMALWBdK11jlLqXmCc1voap/MNgL1AQ611ntOx04AHsACI1FrPKSkv0gQkqkJSRi7TPw1j+/EkAIa0rssjQ0NoH+hbaZ+hteZsWk6BiWuxKVnM/HwX+6JT+OjO7vRpfvFay9++3M3qA2dY9fAAGvp5k2ezM+S19fh4WHh7Ymea1q7GycRM3vvtKF/vOEW+3c717RswqVfj83MrGtXywddb1mi6ElUkAPQGntFaX+d4/wSA1vrFi6S3AIlaa1+nY7OAdlrr6Re5ZhDwmNZ6ZEl5kQAgqkp2no3PtpygZ3BtQoMqr+AvTUpmHmPmb+J0SjZL7utNmwY1i6SJOJvG0Dc2oLUJTgunduPzbSf557d/8P7UbkX6R86mZvPhpuN8tuUEadkXlvv29bYy+/rWjOvWqMjs5GV7Ymjk703nxv5/zhcVf6qKBAB3TLPOEOAUphN4otY63ClNA611rOP1zcA/tNa9nM5vAZ7QWq8tfI0y4/beALK11kVGGDmTACCuRjHJWdzyzibsWvPdjL409PMucH7W4l2s3n+GaX2a8s66SF4d04H//HzofGfzxSaYpWXnse1YIja7xq41H248ztZjiXRp7MezN7UnNMgXrTUv/XSQd9cfJaC6J78+NrDUlVzPpmZTtwqX4BBFlXtDGK11PjATWAUcAJZorcOVUnOUUjc5kj2klApXSu0BHgKmOX1wU6ARsL7QrRcppfYB+4AA4PlL/VJCXA0a+nnz8Z09yMy1cffHYQU26YmMS2f5nhgm927Co8Na0THIl79/vZczqTk8fl3rEmcX1/CyMqRNPYa1q8/w9g1YPL0Xr93WkRMJmdz49u88+MUuHv1qD++uP8rwdvVJyMjh9Z8Pn79+X3QK3+06VeCeC387So9/r2HtobOFP+5PcyIhg7s/3s7ZtOy/7DOvFGWaB6C1XqG1DtFaN9dav+A49rTWepnj9RNa63Za645a68Fa64NO1x7XWgdqre2F7nmN1jpUa91ea3271rr0mTdCXKVa1a/B2xM7c/B0KrMW78LmWEjp7V8j8HS3ML1/Myxuihdv6YBFKfq1CCgy07o0Silu7RrEuscHMXNwC1bvP803O08xa0hL5t3ehUk9G/PJZrPC66rw04yZv4mHv9zNku1mvEfE2TReWXUIgBdXHDifR2dbjyYw+f2tnEjIKNfvYeW+WJ5ZFl5gfsZ7vx3llwNnCwQngLNp2diLyYO4QNYCEsKFfLzpOP9aFk69mmYo59m0HO7p34wnR7Q5nyY8JoUg/4p36J5JzeZoXMb5QJKcmcs1r63H22ohNiWL0CA/qnta2HYskU/v6smLKw5wMjGTR4aG8PT34bxyawfGdr8wTDY7z8bQN9YTlZhFQ18vFk/vTePaxQ9xtds1Mz7fSdOAavxjeGvzXVOzGfLaetJy8vnkzh4MCKlDek4+PV/4Be24/4pZ/WldvyYbI+KZ8sE2Zgxuwd+Ghpy/76nkLBQUaUa70smewEJcAab2acqcUe3o2zyAQSF1uaNPMA8Mal4gTbuGvpUymqdeTa8CtQg/Hw9mX9+aU8lZDG5Vly/u6ck7E7vSyN+H2xduZU90Cs+Nbs/kXk3o1MiP11YfIivXdv76//16hKjELJ4e2ZaMXBsT3tty0ZrAoq0nWPnHaeati+T73aaZ6cWVB8nJtxNQ3ZO310YA8P3uU2Tk2pg7sQvVPd3594qDHIvP4IFFO7HZNZ9uPk52nslDns3OhAVbuH3h1mJrJ1eqc9+/OBIAhHAxU3o35fVxnXh5TAeevrFtsesn/Vlu6xrEDw/2493JXfHxcMfXx8p7joltN3VsyMgODVFK8eSINpxJzeH11YfIzM3nyJk0Fmw4yi2dA7mzXzCL7u5Jek4+g/+zjgkLtvDp5uPn+zbOpGbzyk+H6NuiNt2b+vPkN/v4YttJvt11insHNuOBQc3ZdiyR7ccT+WzLSdo0qMmgVnV4aEhLNhyOY+y7m3FT8PKtoSRl5rF8TwwA3+48xcnETI7GZ7BiX+wlf3etNcv2xHAy4c+by5qcmcs3O6O599MwHvtqT4Fzx+MzGPfuZj7aeKxAYC3J2dRsBryy9qLnpQlICFFh6Tn5+FgtBYaPzvh8Jz/ujcXT3Q1fbys5+XbWPDqQAMeaTScTMvlqRxQr9sUSGZdBA18vnh7ZluV7Y/jlwFl+fngAnlY3Rvz3N5Iy8wj08+aXvw0EoN/Lv1LT28qx+AxeuLk9k3o2ISffxrWvryc2OZvP7u5Jz+BaXPfmBjzc3fj2gb4MeW09vt5WMnPzsVrcWPFQ/4tuxqO1JjwmlRZ1q+NltZCZm8/jS/fy495Y+jSvzef39Cr2unMSM0xBfl27+mWeyb3+cBzTPwkjJ9+Op7sbOfl2tjwxhPq+ZkTVW2uO8Ppq089Rq5oHjw4LYVLPJiXec8ainaw+cIYjL4yQJiAhxJ+juqd7kcL0v+M68fndPZnQozE1va3MGdXufOEP0Li2D48Oa8WaRwfx1X298fW2cv+inazYd5qHrmlB04BqNPD15o1xnajh5c6cUe3w9rDg7WHhrv7BHIvPoJqH5fzS4J7uFj6+owdf3tuLXs1qo5RiSu+m/HEqlae/D+dkYiazhrRkxuAWHDydxpqDFx+pNG99JCP/9ztdn1vNrMW7GDNvMyv2xdKjaS02RSZwIDa1xN/H3LURPP/jAQa8upapH2xjT1RyiekPnk5lxqKdNKtTne9m9OXr+/sAsDEi/nya3yPiaR9Yk6/u601Iver889s/Shxt9cv+M/y4L5ZZQ1peNI3UAIQQl4V8m51PNp9gf2wq/745tMDGPPk2e4FtQ9Oy8xj46jpu6tiQZ25qd9F7ZuTk0+vFNaRl59M+sCbLZ/bDZtcMfm0dtXw8+G5G3yJDZVfsi+WBRTsZ2rYeAdU9WPnHaWx2zVvjO9OlsT+9XlzDyA4NePW2jsV+Zm6+nV4vrqF9oC+dG/mxaOtJPCyKDX8fXOzWp2dTsxk9dyM2xzyPBr7e2O2abi/8wqCQOrw+rhOZufl0fPZn7uwbzBMj2pCdZ2P03I2cTcthxUP9z9cSnH8/w97YQE0vK8sf7Ien1SI1ACHE5cvd4sad/YL5z20di+zKVrjgrOFlZe2jg/jnDW0oSTVPd27rakYizRoSglIKd4sb9w9swZ7oFH53esIGs13pI1/upmsTf/43oTMv3tKB7f+8lp1PDWVw67r4+lgZ0zWI73fHEJ9eZOkywGwSlJiRyx19mvLI0BBevCWUmJRsVoWfKZI2KjGTKR9sIzkrj/endqeBrxmd5Oam6N28Nhsj49Fas+1YInk2TV/HIoZeVgtvT+xCVq6NhxbvOr/XBJhg+a9l4ZxOzebFW0NL3OFOAoAQwiX5+lixFvNEXdisIS15Y1xHrm1T9/yxW7sGUr+mF//7NeL8sew8GzMW7aRuTU8WTO6Kl9UCmA2JnD9nWt+m5NrsLNpykrOp2bzy00He/OXw+bkJX4VFU6+mJ/1bmsL6mtZ1aVzLhw82HiuQrw2H47jx7d+JSc5iweRuRdaW6ts8gDOpOUTGZbApMgEPi1uBXeNa1K3O86Pbs+1YIpPf38a+6BQS0nOY8sE2vtl5igevaUmXUpbukC0hhRBXNF8fKzd3DipwzNPdwvQBzZjzw362HUukR3At3v/9GDEp2Sye3qvELUGb16nO4FZ1eHdDJHPXRpDrePr2dLdwS5dA1h46y30Dm5+vtVjcFNP6NGXOD/vZE5VMx0Z+fLTxGHN+2E9IvRrMv71rsSuz9nM87W+MiOf3I/F0aeKHt4elQJpbuwaRlWfj9dWHufHt3/HzsZKZa+M/t3VkTNegIvcsTGoAQoir0oQejaldzYO310aQkJ7DvHWRXNumLr2alT6D+oHBLfCyWritWxDrHhvEjR0b8vJPB3nsqz3YNUUK39u6BVHd050PNh7jjdWHeWb5fq5tU49vHuhz0WW5G9f2Icjfmx/2xrA/NvV8QCjs9l5NWPf4IGYMbk7T2tVYel/vMhX+IDUAIcRV6txoold+OsRDi3eRlWdj9vWty3Rt96a12PnU0PPvXx3TgeikTH47Ek/3pv40q1O9QPoaXlZu6xbEhxuP/3979x4jZXXGcfz7C0t3RaAoS4kCheUSFRVvrq8uQAAABtdJREFUqCiUEkojULM2tYmoUamoTbQCxlglxD9s+oe9pLVNWlqK1moJGPHS1aQXRE0bE9D1Espt64qNYLCsKGhsU0Uf/zhn8A2748zszuzsmXk+yWRmzjuX8+RMzjPvO++cBwgJ4u5vnd7jj8JZsyY3sz4utfF5RYyGNw3mtotO5raLiur+Eb4H4JyrW1fNGM/wpgae6zzAZeeOY/KXiivwc7SmwYNYfdV0Zk1u5ua5PZ92+Z0LWxje1MD1X2nhx5dOKzj5w2eT/rDGBqaVsf5Eju8BOOfq1rCmwXz3q5P43T92s3xe/vPlizFqWCN/vO78vNu/PHIIL9359aIm/pwL41Ic508cWdLziuUJwDlX126cM4kls1qOnPVTSaVO4s1DG1m58BTOmVCZQjyeAJxzdU1Sv0z+vXX97IkVe+2i0pGk+ZI6JHVK6la1S9JiSV2SXomX6zLbPs60t2XaWyRtia/5kKT+W9HKOedc4QQQa/z+ClgATAUulzS1h4c+ZGZnxsuaTPv/Mu2tmfYfAT83s8nAu8CS3ofhnHOuVMXsAZwHdJrZbjP7EFgPXNKXN411gOcCG2LTH4Bv9uU1nXPOlaaYBDAG2JO5vze2He1SSVslbZA0LtPeJKld0mZJuUl+JHAw1hv+vNdE0g3x+e1dXV1FdNc551wxynVe0RPABDObBmwkfKPPGR9XobsCuEfSpJ5eIB8zW21m081s+qhRo8rUXeecc8UkgDeB7Df6sbHtCDM7YGa5pfHWAOdktr0Zr3cDzwJnAQeAEZJyZyF1e03nnHOVVUwCeAGYEs/a+QKwCGjLPkDSCZm7rcDO2H6cpMZ4uxmYCeywsGzeM8C343OuAf7Ul0Ccc86VpuD/AMzssKTvAX8FBgH3mdl2ST8A2s2sDVgqqRU4DLwDLI5PPwX4raRPCMnmbjPbEbfdDqyX9EPgZeDeMsblnHOugKQqgkl6H+iodj/KrBl4u+Cj0lFr8YDHlIJaiwfKG9N4M+v2I2pq/wTu6KmsWcoktddSTLUWD3hMKai1eKB/YvLVQJ1zrk55AnDOuTqVWgJYXe0OVECtxVRr8YDHlIJaiwf6IaakfgR2zjlXPqntATjnnCsTTwDOOVenkkgAheoRpEDSOEnPSNohabukZbH9eEkbJb0arytT+qdCJA2S9LKkJ+P9pOs8SBoRFzTcJWmnpAtqYIxuiZ+5bZLWSWpKbZwk3Sdpv6RtmbYex0XBL2NsWyWdXb2e55cnpp/Ez95WSY9JGpHZtiLG1CGpxPLvPRvwCaCEegQD3WHgVjObCswAbopx3AFsMrMpwKZ4PyXLiEt/RKnXefgF8BczOxk4gxBbsmMkaQywFJhuZqcR/s2/iPTG6X5g/lFt+cZlATAlXm4AVvVTH0t1P91j2gicFhfW/BewAiDOFYuAU+Nzfh3nxj4Z8AmACtQjqAYz22dmL8Xb7xMmljGEWHKrpyZVF0HSWOAbhAUAk6/zIOmLwGzisiRm9qGZHSThMYoagGPi4otDgH0kNk5m9nfCMjNZ+cblEuABCzYTFp48gQGmp5jM7G+ZZfI3ExbKhBDTejP7v5m9DnQS5sY+SSEBFFuPIBmSJhBWRd0CjDazfXHTW8DoKnWrN+4Bvg98Eu8XXedhgGoBuoDfx8NaayQdS8JjFFfj/SnwBmHiPwS8SNrjlJNvXGplzrgW+HO8XZGYUkgANUXSUOARYLmZvZfdFldJTeK8XEkXA/vN7MVq96WMGoCzgVVmdhbwAUcd7klpjCCsyEv49tgCnAgcS/fDDslLbVwKkbSScNh4bSXfJ4UEULAeQSokDSZM/mvN7NHY/J/c7mm83l+t/pVoJtAq6d+Ew3JzCcfPU67zsBfYa2Zb4v0NhISQ6hgBzANeN7MuM/sIeJQwdimPU06+cUl6zpC0GLgYuNI++6NWRWJKIQEUrEeQgnh8/F5gp5n9LLOpjVAPARKqi2BmK8xsrJlNIIzJ02Z2JQnXeTCzt4A9kk6KTV8DdpDoGEVvADMkDYmfwVxMyY5TRr5xaQOujmcDzQAOZQ4VDWiS5hMOq7aa2X8zm9qARZIaJbUQfuB+vs9vaGYD/gIsJPwi/hqwstr96WUMswi7qFuBV+JlIeG4+SbgVeAp4Phq97UXsc0Bnoy3J8YPZifwMNBY7f6VGMuZQHscp8eB41IfI+AuYBewDXgQaExtnIB1hN8wPiLsqS3JNy6ACGcOvgb8k3AGVNVjKDKmTsKx/twc8ZvM41fGmDqABeXogy8F4ZxzdSqFQ0DOOecqwBOAc87VKU8AzjlXpzwBOOdcnfIE4JxzdcoTgHPO1SlPAM45V6c+BT6BUy0ACCB3AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "_ = log.to_pandas()[['train_loss', 'val_loss']].plot()" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "surv = pd.DataFrame(model.predict_survival_function(x_test), index = labtrans.cuts)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "ev = EvalSurv(surv, durations_test, events_test, censor_surv='km')" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.6192133993854209" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ev.concordance_td()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.205767795738803" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "time_grid = np.linspace(durations_test.min(), durations_test.max(), 100)\n", "ev.integrated_brier_score(time_grid)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "-0.6075163720586912" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ev.integrated_mbll(time_grid)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }