{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Hook callbacks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This provides both a standalone class and a callback for registering and automatically deregistering [PyTorch hooks](https://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks), along with some pre-defined hooks. Hooks can be attached to any [`nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), for either the forward or the backward pass.\n", "\n", "We'll start by looking at the pre-defined hook [`ActivationStats`](/callbacks.hooks.html#ActivationStats), then we'll see how to create our own." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [], "source": [ "from fastai.gen_doc.nbdoc import *\n", "from fastai.callbacks.hooks import * \n", "from fastai import *\n", "from fastai.train import *\n", "from fastai.vision import *" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class ActivationStats[source]

\n", "\n", "> ActivationStats(`learn`:[`Learner`](/basic_train.html#Learner), `modules`:`Sequence`\\[[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)\\]=`None`, `do_remove`:`bool`=`True`) :: [`HookCallback`](/callbacks.hooks.html#HookCallback)\n", "\n", "Callback that record the activations. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[`ActivationStats`](/callbacks.hooks.html#ActivationStats) saves the layer activations in `self.stats` for all `modules` passed to it. By default it will save activations for *all* modules. For instance:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HBox(children=(IntProgress(value=0, max=1), HTML(value='0.00% [0/1 00:00<00:00]'))), HTML(value…" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Total time: 00:13\n", "epoch train loss valid loss\n", "0 0.077055 0.049985 (00:13)\n", "\n" ] } ], "source": [ "path = untar_data(URLs.MNIST_SAMPLE)\n", "data = ImageDataBunch.from_folder(path)\n", "learn = create_cnn(data, models.resnet18, callback_fns=ActivationStats)\n", "learn.fit(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The saved `stats` is a `FloatTensor` of shape `(2,num_modules,num_batches)`. The first axis is `(mean,stdev)`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(194, 44)" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(learn.data.train_dl),len(learn.activation_stats.modules)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "torch.Size([2, 44, 194])" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "learn.activation_stats.stats.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So this shows the standard deviation (`axis0==1`) of 5th last layer (`axis1==-5`) for each batch (`axis2`):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsvXmUHGd97/15uqu7ep99RvviTbYxsoyFN4wxO0kIW3IT/OYGMCSOE5JwIZdAwn2zHJaE90BekpiE4wAhkDcmARzAGAK2Cd4XZFubJWux1pFGs8/03tXV/bx/VD3V1T3dMz2jkUYjPZ9zcq5U3V1dLXPrW9/fKqSUaDQajUajCCz1BWg0Go3m3EILg0aj0Wjq0MKg0Wg0mjq0MGg0Go2mDi0MGo1Go6lDC4NGo9Fo6tDCoNFoNJo6tDBoNBqNpg4tDBqNRqOpw1jqC5gPvb29csOGDUt9GRqNRrOsePbZZ8eklH3tvn9ZCcOGDRvYtm3bUl+GRqPRLCuEEEfn834dStJoNBpNHVoYNBqNRlOHFgaNRqPR1KGFQaPRaDR1aGHQaDQaTR1aGDQajUZThxYGjUaj0dRxwQiDlJLvPDtIrmQv9aVoNBrNOc0FIwyHx3L80bd2cN+Ok0t9KRqNRnNOc8EIw1jWAuBUurjEV6LRaDTnNheMMEzkSgCMZEpLfCUajUZzbnPBCMN4znEMI2ktDBqNRjMbF4wwTLrCMJrRoSSNRqOZjQtGGDzHoENJGo1GMyttCYMQ4qtCiBEhxO4Wr3cIIe4TQuwQQrwghLjdPb5eCPGcEGK7e/xO32duE0LsEkLsFEL8lxCid3F+UnMmPMdQolqVZ/KrNBqNZlnTrmP4GvCWWV7/ILBHSnk1cCvweSFEGBgCbpRSbgGuBz4uhFglhDCAvwVeK6XcDOwEfn9hP6E9lDDYVclk3jqTX6XRaDTLmraEQUr5CDAx21uApBBCAAn3vbaU0pJSqtiN6fs+4f5f3P1MCjijDQYTOQshnD/rcJJGo9G0ZrFyDHcBV+Dc3HcBH5JSVgGEEGuFEDuB48BnpZQnpZRl4Hfd954ErgS+0uzEQog7hBDbhBDbRkdHF3yBEzmLDT1xQAuDRqPRzMZiCcObge3AKmALcJcQIgUgpTzuhosuAd4rhBgQQoRwhOEa9zM7gT9pdmIp5d1Syq1Syq19fW2vLG08B+M5i8tXJAEY0U1uGo1G05LFEobbgXulw0HgMHC5/w1SypPAbuDVOOKBlPIlKaUE/gO4aZGuZQY5q4JlV9mkhEE7Bo1Go2nJYgnDMeD1AEKIAWATcEgIsUYIEXWPdwE3A/uAE8CVQghlAd4I7F2ka5nBhDsOY3VnlKRpaMeg0Wg0s9Buueo9wJPAJiHEoBDiA0KIO33lp58EbhJC7AIeAj4mpRzDyTs8LYTYATwMfE5Kuct1D38JPOLmH7YAn1ncn1Zjwq1C6o6H6UuZp+0YhtNFrvv0gxwYzizG5Wk0Gs05hdHOm6SUt83x+kngTU2OPwBsbvGZLwFfauf7Txc1J6k7HqY/efrCcGQsx0imxL7hDJcOJBfjEjUajeac4YLofB53Q0k9cZP+ZISR0xyLkbcqAGSLereDRqM5/7gghEE1t3XFQ6zsjDA8XcKyq/M6x3i2xJcfPYSUkqy77Cerl/5oNJrzkAtDGPIW4WCAhGnw8tUdWJUqL55Kz+scn/7hXj51/15eGs2RtxxByGjHoNFozkMuDGHIWnTHwwgh2LK2E4Adx6fa/vyLp9L85/MnAMcl5EoV788ajUZzvnFhCEPOEQZwSlZ7EybPz0MYPvfj/Uh37l6uZHuOQecYNBrN+UhbVUnLnXdft867mTuuoYPtbQpDtSp5eP8IN1zUzVOHJsgUbXKWdgwajeb85YJwDG+8coC3b1nt/X3L2k4OjeaYLpSbvn+6UObGv3qIn+0bYSxXolyRXLWqA3AdgysIGS0MGo3mPOSCEIZGrnbzDDsHm7uGJ18aZ2i6yHNHJxmackpbLx1IAG6OwXUMOS0MGo3mPOSCFIbNa5QwTHvHhtNF3vkPj3N0PMdTh8YBGJwsMDTtCMMl/TVhWE45hhdOTvPgnuGlvgyNRrOMuCByDI10REP0JkyOT+S9Y7tPTPP8sSn+5YmjPPlSTRhOTRcAWN8TJxQUy64q6a6fHmTn4DRvuHJgqS9Fo9EsEy5IYQBY2RHx3ADUmuC+te04mZKNEDA4mWdoukg4GKAnHiZuGuRKthdCyhSb5yjOJQYnC8tCwDQazbnDBRlKAkcYTvmEQa37VAnlV1/ax6l0kWMTeVZ0RBBCkDCNuhxDtmQj5bm9P3pwMu+FvjQajaYdLmhhGHLDRAATuTKhoGB1Z5SEafALV62gKuH5Y1Os6IgAOMJQrOUYqtKZm/SFB/dzaDR7Rq93IQKUK9lM5suUK3LeI0A0Gs2FywUrDCs6oqSLtbDQVN6iKxbmU++4ij9765Ws74kBcCpdZJVPGHKWk2MIuPujD43m+MKDB7zO6DPFe776DJ++f8+8PnNiqiZ82jVoNJp2uaBzDODc+C/uS3jd0a+9vB+gLjG9oiMKQNw0mMpb5C2bvqTJcLrEgRFnJ8PR8TxniqHpAo8eGCOo1KhNTkz6haFCZ2yxr0yj0ZyPXMCOwRUGN88w6ToG/+vqPqxEJBExyBRt8laF/qRz7MCIE0I66hOSkl3hvh0nFy3/8ODeEQCK5cq8Pjc4Wbsm7Rg0Gk27XLDCoG72qjJpImfRFQ95r4eCAVakInXvTYQNRt0lPwMpE4ADw44wHBvPeZ99aO8If3DP82w7Orko1/qA24dQLM8vTzDoCyWpEluNRqOZiwtWGAZSyjE4N8+pfLnOMQCs6XJiLyvdUFIiYnhVS/3u5w+6oaTJfJm0W7467pa++hvoFkqmWObJl8aAhTgGnzBox6DRaNpkTmEQQnxVCDEihNjd4vUOIcR9QogdQogXhBC3u8fXCyGeE0Jsd4/f6ftMWAhxtxBivxDiRSHEryzeT2qPSChIdzzM0HSRalUyma9NYFWs6XIEYWWnIwJxs5aSGXBDScd8IaRjbp5h2i193X3i9IXhsQNjlCuSVR0RSvOsLBqcLJCKONdcsLRj0Gg07dGOY/ga8JZZXv8gsEdKeTVwK/B5IUQYGAJulFJuAa4HPi6EWOV+5hPAiJTyMuBK4OGFXf7psSLl9DKki2WqkhmO4arVHfQnTbrd4wkz6L2mQklVCUn35qtEYirvOIddPmGQUi4o53BozAlRXbuhe96O4cRkgU0rnJ3UOS0MGo2mTeYUBinlI8DEbG8BkkIIASTc99pSSktKWXLfYzZ81/uBv3LPX5VSji3k4k8X1f3sX/3p5703beDhj76WgJuFTpi11/tdYQC4bkM3UKtMmnKntr40mvXKYe/66UHe/sXH532Np6aLdMZCdEZD8xKGYrnCWLbEZQOOMOR197NGo2mTxcgx3AVcAZwEdgEfklJWAYQQa4UQO4HjwGellCeFEJ3u5z7phpq+JYRYkkE+KzoinEoXmXSf8BsdQzAgiIZrLiHucwydsTBhw/nnu6Q/QXc8zLEJ5+leOQYp4YWTzgrRpw6P89LI/JvghqaLrEhFiIQC80o+qx4GJQzaMWg0mnZZDGF4M7AdWAVsAe4SQqQApJTHpZSbgUuA97oCYABrgCeklK8AngQ+1+rkQog7hBDbhBDbRkdHF+Fya6zsiDCRs7yS1cYcQyMqZAQQDxsk3ZzDio4I67pjnmOYLlhc3BcHauGkA8NZ8uUK1er8wkmn0gVWdkSIhIIU7Urb4ajDo45IXbEyBUBBJ581Gk2bLIYw3A7cKx0OAoeBy/1vkFKeBHYDrwbGgTxwr/vyt4BXtDq5lPJuKeVWKeXWvr6+RbjcGht6nZv3k4ecSFajY2gkHq4JQywcJOEKxcqOCOt7YnU5hssGkvQnTXafmCZdLDOSKSElFO35Pbmfmi6ywhUGKcGqtOcaDrojOjatSBIOBrRj0Gg0bbMYwnAMeD2A6wg2AYeEEGuEEFH3eBdwM7BPOo+89+EkqnE/O79ZD4vE9Rt7APiv3acA6JrDMSR8jiFhGiRcxzCQirC+O8bJqQKWXWWqUKYzFuLqtZ08e3SSg74Q0nz6CSy7yljWYkUqiumGrdoNJx0cydKXNOmIhoiZQZ1j0Gg0bTPnSAwhxD04N/FeIcQg8OdACEBK+SXgk8DXhBC7AAF8TEo5JoR4I06FknSPf05Kucs97ceAbwghvgCM4riOs05f0uSygQT7h7OEgwHivnxCMxK+ctWYGfT+vrIjyqrOKFXpLPyZzpfpiIa5pD/JA3uGeXR/LbfudCCbjaduynC66J4/4jmFUrkC0dBsHwMcYbikz1kuFA8b2jFoNJq2mVMYpJS3zfH6SeBNTY4/AGxu8ZmjwC1tXuMZ5aaLe9k/nKUrHsIprGqNEgIjIAgHAyQjBsGAoC9psrLT6Xk4NJbDqlTpjIW44SKnWumbPz/mnWM+juGUKwwDHRHG3I7rdhyDlJKXRrK84xpnz3U0HNR9DBqNpm0u2M5nxY0XO+GkufILUGtwi4WDCCHojodZ1RkhGBDeBNa9Q04VUmc0xBUrUnREQ3ULgebTgaw+p5LP0F6OYiRTIlOyvXWk8XBQdz5rNJq2ueCF4YaNPQjRnjCYRoBQUHgC8Udv2sSX3/NKoDaU70UlDLEQgYDguo2Oa1DC4d/+9qNdQ163dDOGXWFwks8qxzC3MKichhKGWNggr2claTSaNrnghaEjFuI1l/Vx1erUnO8VwhGFmJuLGEhFvM7iZCRE0jTYO+TMTuqIOkJzw0WOI7l6rdO+kbcqPHpglGs/9SC/+/89xxce3F/3Hd/bfsJbxTk0XSQWDpI0jZpjaCOU1CgMcVM7Bo1G0z4XvDAAfO326/jEL13Z1nvjYaNuZpKflZ0RXnLLRDtjToL4lkt7EaImELmSzfZjU1h2ldWdUW/wHjg7ID70ze186WcvAU4Pg1orqhxDoVzhob3D/MsTR1pe48GRLEnToD/pJLmjYUPnGDQaTdtoYZgnyUjNMTSyoiOK7TawKWG4dCDJo3/8Wt66eSXgOIbpQploKMiqzkhdMrrghom+/ewgdqXK0HTRG/ltGsoxVLjnmWN88gd7GM+WaMaBkQwX9ye8ZLrOMWg0mvmghWGevOGKAW7d1N/0NZVHAOiM1nIWa7pinsvIWTbpYtnpLwgb5H05A7WX+VS6yMP7Rzk1XfTGg9dCSRXSRRu7Kvne9pMzrqFYrrD9+BRb1nZ6x3SOQaPRzIcLdrXnQvnfb97U8jW1tyFsBLzQj8I0AgQDgnzJcQypqOM8TvqW6fjHav/hPc+Tsypc6Y60UOcrlatkis7T/7efHeT9N2+s+57njk1SLFe5+ZJe75jKMUgp5yzJ1Wg0Gu0YFhG1t6EzOrMnQghBzA3pTBd8jsEX+y+7TWw3XdyDEQzwl297Ge+7aQNAXblqplgmbATYM5TmhZP1Ox8eP+jshr7e7aEAp4+hKmm6z+FvfrKP//PdXTOOazSaCxftGBYRlQ9Q+YVG4m5IJ12wWdUZIRYO1u1iVqGkj7zxMq5d31UnLv5QUqZo88YrB7h/5xCPHRjjZas6vPc9dnCca9Z2kozUrkHNeMpbFe88ikcPjvH8sSlef8UAr20RItNoNBcW2jEsIiqU5M8v+ImZNceQioTcv8/MMYSNwAzHEXFnJRWsKtmSzYaeGF2xEEd8fRDT+TK7Bqd4lS+MBHjJ8lyTeUmqWukvvv/CvBcBaTSa8xMtDIuIcgypFrOM4m7oKF0ok4qGiIcNLLuK7YaQ1DwktefBjxEMYAQEk3mLSlWSjIRY3xPn6HjOe8/Th8epSmYIg0p855uUrOYsmxWpCEfH8/xs38gCfrVGoznf0MKwiMRNg65YiJ4WU1pj4SCZYplMyXZzDM6TvKpM8hxDsPl/lkgoyKhbopqMGGzoqe2AANg/7DTXNTbrqWVD+SYlqwWrwstWOe9XC4sWypGxHDsHp+a9c0Kj0Zxb6BzDIvOP//NaVrkhpUbipsGBEdUZHfLi/flShVQkVBdKakYkFGA0o4TBcQzf23GSkl3BNIIcGsuxsiNCLFz/n9WfY2gkV6p4a0qzxdPrdfjgvz3HCyfTrO6M8h933sjqzub/DhqN5txGO4ZF5oaLeljXE2v6Wiwc9LbFpfyOwX2SL80SSgKnyW0s43MMvTGkhOMTTsnrodEcG93lQ43fCzNzDNWqpFCu0JdwhCFzmjsbhqaLrO2OcmKqwP5TmdM6l0ajWTq0MJxFEqZBueKEWepCSVZ9KMkMNu+sjoaDNcdgGqzrdkTg2EQOKSWHRrNNhaFVjkF1WsfdpUOn4xjsSpXJvMXVa5zGuvmIzL///Bh//r3dC/5ujUazuGhhOIv4Qzwd0VCtG9q9ibYTSlI33GQkxAbXmRwZyzORs0gXbS5yl/PUf2+9ACnUmIyYEobSwnMMEzkLKWFDT7zuN7XDd549wQ/dLXoajWbp0cJwFombNSeQihq1pHBj8rmVMBi1zycjBt3xMEnT4Oh4jsNjTnXSRU0cg1owNJm36o6rUtVYKEgyYngd1QthLOucW4XR2hUGKSV7h9JndPXovc8Nev8+Go1mbrQwnEVmOAaVFHbnGFmVCsGAIBhoPrbC35yWjBgIIVjfG+PIeJ5Do64w9DUPJV3UG+f5Y5N1x9UAv7gZJBExvHHfC2HMrZZa3+0IQ7vnGpwskCnZ5MsVnHXgi0u1Kvnot3fy1ccOL/q5NZrzFS0MZxG/Y/DnGFRIx7KrLUtVoTYvSYhapdH6njjHJvIcGssRCoqWlUDXbezmmcMTdaWkhbLzvdGwE0pq5RgGJ/O87a7HvJt/M8Zzzmv9qQjRULBtx7DHXWwkZXu7JubLdKFMpSo5OtF6IZJGo6mnLWEQQnxVCDEihGiaIRRCdAgh7hNC7BBCvCCEuN09vl4I8ZwQYrt7/M4mn/1+q/OebyjHEAoKoqGgJwwFX/K5VRgJwHQdQ8I0CLiu4tL+BEfHc/z4hVOs645htBCW6y/qJl202Tec4chYjul8ueYYwk4oqdVT/u4TaXYOTnt9Es0YyzihpN5EuM59TOasWfsa9pxMe38+E6PBJ9zw2bFxHUrSaNqlXcfwNeAts7z+QWCPlPJq4Fbg80KIMDAE3Cil3AJcD3xcCLFKfUgI8S4gu4DrXpbEXSFIRULeNjjwOYbK7MKgcgwp3xyk21+1kcsGkhweyzVNPCuu2+gsCvre9pP84t89yt88sM9LRkfDwVmrktRNPlu0sewq/+ubz3OkIWY/li0RNgIkvER2hXSxzE1//VPu3zXU8rrUjmzgjIwGn3KFYXCy4HWYazSa2WlLGKSUjwATs70FSApnwE/Cfa8tpbSklCr+YPq/TwiRAD4CfGohF74ciblC0OGOzDCNAELUHEOpzVBSwqzPVfzL+6/j4r44N13c0/KzqzujrO6M8qWHXyJvVTiVLnr9E/GwQcIMtXQMWXfLXLZkc2wix3e3n+SJl8br3jOWtehLmK7gOaGkkXSJQrnCsVnCOHtPpTFdMcyXz4BjyDnXblclQ24PiUajmZ3FyjHcBVwBnAR2AR+SUlYBhBBrhRA7gePAZ6WUarvMJ4HPAxdM8NdzDK4wCCGIhw0vpGPZVe8m2QyVfE5G6jubB1IRHvzIa7j9VRubfczj+o21UdxT+bLnGGLhWvK5WdhHDfrLlWzSrqtQovK2ux7jnmeOMZYt0ZsIu7/TOdd0wbkp+9eX+kkXyxyfKHi9D7kz4Bj8lVj+8SEajaY1iyUMbwa2A6uALcBdQogUgJTyuJRyM3AJ8F4hxIAQYgtwsZTyP+c6sRDiDiHENiHEttHR0UW63KVB5Rj8Q/b8o7ctu0qoDcfQKAxAWwt4fu2Va/mll6/kNZf1MV0oe98bMw1S7jmzTeL8KimdKdnen/NWhUpVsnNwmp+8cIqxbIket4M6YRrkSjbTBavu840cHHGiiNesc4ThTOylnsz5hGFC5xk0mnZYLGG4HbhXOhwEDgOX+9/gOoXdwKuBG4GtQogjwGPAZUKInzU7sZTybinlVinl1r6+vkW63KUh0RBKAiUMqly1vRyDf9fCfLjhoh6++BuvYCBl1jmGaCjoXVuzPINqfMuVbDLu03/eqnjCsmNwut4xeMLgOoZCc8cwknZCOxf3O7mRM5V8DgcDhIMBjmnHoNG0xWIJwzHg9QBCiAFgE3BICLFGCBF1j3cBNwP7pJT/KKVcJaXc4B7bL6W8dZGu5Zwl5pardkRrT/zOFjfnhlieSxhahJLmS2cszFTBchf3OCtHE8oxuHmGglXh2aNO34MSi2zR9v6ct2xPWCZyFsPpEr3KMbhhqSl3WmsrxzCcdtJPaoxHs+mvrZjKz17t5L0vV6YrHmJNd1SHkjSaNmm3XPUe4ElgkxBiUAjxASHEnb7y008CNwkhdgEPAR+TUo7h5B2eFkLsAB4GPielvGD3SKreA79jcBK1vnLVtkJJC3MMio5oiGK5ykTO8q5JOYbaPunj/I8vPcF0vkzWvb7GUFJjr4I/lNROjmE4XSQYEKzpinrnbIcfv3CK6z7zEF974sic753IW3TFwqzvjuleBo2mTdp69JRS3jbH6yeBNzU5/gCweY7PHgGuauc6ljuRUID33bSBN1wx4B2Lhg3vBmrZVWKx1v9JzEVzDI6wDE0XvLEcyQbHMJIpUZUwlis1DSUVrMqMG7k/+ayEB2Z3DP1J0xMlf7nqowdG+fKjh9k7lOb+P3w1fUlHdH78wil+91+fpSrh5FRhzt86mXOFoSfOM4cnkFK2lY/RaC5k9D6Gs4gQgr9428vqjsXDQYbcG1xpjgY3FUpKna4wuKtHT04VfY7BEQsVKlJiNZUv1/oYSrY3xC9n2Z5jCAYElar0xnerDm91426ZY8gU6U/V9kf4cwwf+Y8dTBfKWHaVo+M5Txi+8thhNvTEmS6U25rtNJm3uHxFivU9MXJWhaPjeTY0mSel0Whq6JEYS0x0HsnnqOp8XiTHcHKq5hhqOQbnJq6EYbpg1eUY/KEkdd2b13QA9aEkcJrKoLVjGEmXGEiaBAMC0wh4VUmVqmQsW+KGi3rqPj+dL/Ps0Ul+4eUr6IqHybQxDXYy7+QY3nLVCkwjwN88sL+NfyGN5sJGC8MSE/clny27itlOjsE8/RwDOA5FPd035hiUMEzmajmGbF1Vku094b/rmtVc0p9gbbeTK1Ad3Sdcx1AoVyg36ToezhS97XFx0/DON5l3RnhvdCe1qhzFIwdGqVQlr7u8n1TEIF2Y3TFUq5IpN8ewsiPKHbdcxPd3nGT78SnvPSW7wrvvfpJnj87Wv6nRXFhoYVhiYmbQayCba1bSZQNJrlyZ4spVqZbvaQflGACioebJZy+UVCh7LiLbkHxWOYHXXTHAgx95jRcSajxX458BiuUKU/kyA8kIUF+2O+6O8FYhH9VU998vjtAZC7FlbRfJSMgTqVaki2WqErpiTujsd15zMX1Jkzu+vo1HDzg9MUfG8jx1aIKf7VvePTIazWKihWGJiYUMLLuKXanOGUoaSEX44YdezarT3KXc6d4ooZYPCAYE8XDQyycoYRjLlrypp9miXVfOmvNGatRvnPOHupTLacwzqE10AymfMJSUMNSXsWaKZapVyc/2j/Kay/oIBgSpaMgTjFao5Hd33Pm9CdPg6++/jlQ0xG9+5Rn2DqU5MeVUKh3RpawajYcWhiVG3Zjz5cqc5aqL9p3hIIY7nTXmu6knIrVBetNuD8IJN0+QihjkrIonGLmS7RupUZ/ziPv+vrbLCQc1OoZht7lNhZJi4VooadQVhtWdUUJBQaZoc2gsy0TO4uZLegHcxUKzOwY1DsPvkK5YmeIr790KwM7BKS8PoqevajQ1tDAsMd4Wt1JlzlDSYiGE8G6W/pu66j+QUnoCcHzSeZJe0eE82asbeqHs9DGEgmLGNfuH/K3trs8TKEYaHEPcDHrJZxVK6k2YJCMh0oWytyFOuaVkGzmGSXeAnnIMCiU4R8bznvBpx6DR1NDCsMSop+tsycauyrMiDFBLQNc7hpCzTc2qYLtdxeqJWt3AVVipXHHEo9EtQP1CorVu81pjKEkJjDpvNGR4uZbxXIlgQNARDXkrR9XMI5UvSEVCWJUqRTex3WykttrF0BWrFwYjGGBtd4wjYzkG3QT5dKHsuSSN5kJHC8MSo27Mam/A2RIGlWfw39iTpkG2WPbcAtRyAStdxwDQ5bqNsWxpRn4BalVJUHMMM0NJJUJB4Z0rbtaGCY5nLXriYQIBQcpNMk+6N+2uuPN+1cuRKdq8++6n+PPvvzDjOiYbcgx+NvTEOTKeZ3CygNqkqofsaTQOWhiWGBV2GXdvYmcjxwC1m3udY3DXeyphiPp2TK9I1YRBPeWPZS1vx4Qf0wh4OYw1XS1CSeki/cmI14Xsr0oay1peT4TnGBqe/tWE2nSxzItDae7bcXJGSawaoBdrIl4beuIcHc9xYjLPZnfstw4naTQOWhiWGHWDU/uUz14oSTmG2k1T3YSVMKxzn/YBBnyOod8VhtFMc8cgRG0on5qD1FhBNDRd62FwrsMg71Y8jedqk1qTEYN0scxkziIaCs4YJDg8XSRnVUgXbbYdmaz7jmH3O5qNwNjQGyNvVRjLWtzoLjjSCWiNxkELwxKjYv0qZHO2HEOnF8KpPfH3Jk3GsiWvzHN9T00Y/KGkAXc8xVi25CXPG1G5k654mKRp1OUYjk/k+fmRCa5Z2+V7f5B8uYKUTtdzT1wJQ4hM0WYib9WFhNR605dGa5thH9gzXHcNQ9NFVnU0L+3d0FMbi3Fpf4L+pKmnr2o0LloYlpilcgyd7vf6b+wbe+LYVentYfYLw0CTUFLeqtSVpvrx755IRUN1OYYv/vdBAkJwxy0XeceiYQMpneT2uC+UlHKFYSpfris7TXrCkHP/bvDA3lNIWRvFPTRd9KqpGvELw+rOKOt7YloYNBoXLQxLTNI0EMLnGM5a8tl1DL4buxICNTJive/mWScMvpttsxwDOMlkw22aU+EggAPDGb797CC3Xbe27qatKpnGsiUCY2qLAAAgAElEQVTyVoUeXygpW7IZy5bqHIMKJaktcO+6ZjXHJwqeUEgpOTVdZGVnc2FY1Rmp5UG6Y6zvievks0bjooVhiQkEBEnTWIJQ0swcg+o03jk4jRC1/EAsHPRCNwD9yVpuoFmOAZwQVUc0hBC1yqIf7Rrinf/wBMmIwe/eeknd+1V1lOqb6PUln8Epm/V3bCunpUJJ17sD99S/43jOwqpUWZlqLgxGMMC67hjBgGAgabKuO8ZwukSxvPjrRTWa5YYWhnOAjljIa+A6W47hdZf3839+6QquXFmbu9SXNImFg0wXyqQiobpREmEjgOlem989NOtjAOhLmF6SOhkxODSa4w+/+TyX9Cf4wR++ekaIRwnU8QklDLV+BXDGW3T7QknxcJCAcMJF4WCg1mdhOzf2oSmnT2LlLONDNvbGHecQDHg5FNVfodFcyOh9DOcAHdEQh9wQyNkShrhp8FuvvqjumBCC9T1x9g6l6YiGvNJQVWGUMA1KtsWAr5rI38zm509+8QqvkzkVDTGScfoWvvgbr2B1k5u1EoZjrjD0xN0cg28Nqt8xCCFIRkJMF8r0JU2vtLbkPvEPTTuNa62SzwB//JbLvTJY1VF9YqpQF0LTaC5EtDCcA6QiIa+G3zxLwtCKDT0xTxg63Cd0lUhWu5z9q0lbOgZfuEmFg/7H1rVNRQFq1VHHJ5wbei3HUPuuxka1ZMTwhEEN6yt4wuA8+bdKPgNsWpH0/qyEQTkNjeZCRoeSzgH8N9pwsPkT+NlCjbrujIVImgbBgKgJg2mQjBhEjCCqNaCVY/AzkIoQDgb4vVsvbvke9cT//PFJggExI8egrsmPCjP1J02vukqN7Dg5XSAcDHhlr3OhQknKaWg0FzJzCoMQ4qtCiBEhxO4Wr3cIIe4TQuwQQrwghLjdPb5eCPGcEGK7e/xO93hMCHG/EOJF9/hfL+5PWn7UCcM54BjACf8IIeiMhjxhiJsGyUiIQEB4N/JWjsHP+1+1kQc+covXBd0Mv2N45zWrfY1sszsGcCa0RgwlDI5jODVdZKDDJBBob79zJBSkOx7m5LR2DBpNO3ehrwFvmeX1DwJ7pJRXA7cCnxdChIEh4EYp5RbgeuDjQohV7mc+J6W8HLgGeJUQ4hcWeP3nBalzShgcx6DE6lWX9HLteqcRbX13zCtpVYLQqirJTzQcnDNur84TCgo+9PpLveP+/daNw/CSnmOIeEKiHMPQVJGVs+QXmrGyI+Lt315sMsUyJVtXPGmWB3M+7kkpHxFCbJjtLUBSOHMHEsAEYEsp/YNrTFwRklLmgf92/2wJIZ4D1izo6s8TzinH0FsvDH932zXea59651Wo/jGVLG7VxzBfkpEQoaDg/7punTd4Tx1XdDU4BpWY7k+aXm7GyzGkC1y7rov5sLIjyuDk3E1uecvm0GiOq1Z3tH3u3/jy0/QmTL76vlfO65o0mqVgMe5CdwFXACeBXcCHlCgIIdYKIXYCx4HPSilP+j8ohOgEfhl4qNXJhRB3CCG2CSG2jY6en+sX/U/FZ6uPoRX9SZNf37qW113eP+M106jNKlLC0I5jaIdoOMh9f3Azn/ilK+uO+8tku1rkGAZSEQIBgWkEKJUrVKtOc9uKeTqGVZ0RTs7hGEp2hdv/+ee88x8e96bBtsPhsRw/fXGEn+0bmdc1aTRLwWLchd4MbAdWAVuAu4QQKQAp5XEp5WbgEuC9QogB9SEhhAHcA/ydlPJQq5NLKe+WUm6VUm7t6+tbhMs99ziXQklCCD77q5t55YbuWd/nOYY2cgztcvmKVNPfn4yEMI1A3bRXqAmqqoCKhIIUyxXGcxbliqyb79QOqzqjpIs2OXeY36HRLEcbBuv96b27efrwBOWK9Cqf5qJkV7yRIJ/54d6muyM0mnOJxbgL3Q7cKx0OAoeBy/1vcJ3CbuDVvsN3AweklF9YhGtY1vhDSUtdrtouXo6hjaqk0yUVNeiKhWdMSVV9DaokNRIKUCxXvemwjaGnuWisTPr4d3bxy3//GLtPTANwZCzHd54b5LqNjmiealMY1Ea6127qY/9wlq89cWRe19WM6UKZLz96iGpVzv1mjWaeLMZd6BjwegDXEWwCDgkh1gghou7xLuBmYJ/7908BHcD/WoTvX/b4HUNoiUNJ7RI9A46hFclIaEapKsCvXLuGr7x3q1faGgkFKZQrXohnvmGuWpObc8OfzFukizbv+eozDE7mefTgGAC/65bdtusYlDDcdt06Xn95P5/7yT6OnebAvp++OMyn7t/LvuHMaZ1Ho2lGO+Wq9wBPApuEEINCiA8IIe5U5afAJ4GbhBC7cHIFH5NSjuHkHZ4WQuwAHsapRNolhFgDfAK4ElDlrL91Bn7bskE5hmBAEGyzvHKpUTfds+EYbr6kh9dcNjOM2BEN8forvOgkUTeUlCs5Cej5ipbnGNw8Q65kc93GbqYLZb7x5FEe3T/K6s4oN7pzmU612fMwlnPmN/UmTT71zqswAgE+88O987q2RvIN+7E1msWknaqk2+Z4/STwpibHHwA2Nzk+CCyPu99ZQiVRlzrxPB+iYWcqrOofOJN89M2Xz/0mwAwFKdrVmmOYp2gNpCIIgdfLkLMqXLkyRWc0xHeeG6RkV3nr5pVEQkG6YiFONcxVklLyg51DvPHKAS9JD7Wbd2/cZGVHlDdc0c+zx+qXCrViJFMkV6p4Aw4Vqix33BUdjWYxWT53ovMY5RiWOvE8HwZSJv3J9hvIzgYRI+A4BmthjiEUDNAdCzOWLSGlJFeyiZtB3n3dWsayFpmizc2XOM5lRUd0Ro7hpdEsf3DP8/xo91DdcbVrQ4356E9FGEmX6nZHtOIz9+/ld76xbcZx1cg3ph2D5gywfO5E5zFht+JmOQnD79xyMd/94KuW+jLqUFVJakXoQsJcyYhBrmRTsqvYVUksbHDLpX0MpEyEgFdd4oSRVnZEGJouMpGz+PT9e7DsKlN5J+l9YrI+xDSeLRENBb3u7r6EScmukinNXe46NF3k6Hh+hoioYYHjWe0YNIvP8rkTneekosYyCyUF591ZfKbxcgwLdAzgDArMFG0vhp8wDYxggD960ybed9OGukqoU9NFvvv8Cf7p0cPsGUp7N/rGpLSzka5WIaXKa9XuiNmYzFuUfKKjKNpV79wazWKjp6ueI3REQ9gVXXp4OqhyVeUYYgtovkuaIbK+XgZ1jl/burbufStTEcZzFk+85FQq+T/TKAyj2ZK3qhTqheHivsSs16P2b59KF+vKb1UoSecYNGeC5fOIep7TEQ0tq1DSuUjE5xjCRmBBpb+JiEGmZJN1b/KJFiM/VO/EI/sdYcgUy2SLrR1Db3z+jqFalUy6TqExn1Fyk886x6A5E+g70TnCphXJGZUnmvnh72NY6KiOpGmQLZW9yqZWs6BUGM1yu5j9YtJYxjqeK3m9FlBbjToyhzBkijYVt4GtUWzUprpGxyCl5NP372HfKd3foFk4OpR0jvCpd7x8qS9h2RMJBSmVq+RKlQU33qkcQ7akcgzNBaZxAZDzGUcYJvNlClaFaDhItSpn5Bg6os7AwLkcg/+m3yg2XiipwTGkCzb/9OhhuuLhukVEGs180I5Bc94QCQWwKlUyxfKCG++SEaMuXxCfI5SUdF/PFm0vlAR4PQ7pYhm7KutyDEII+hLmnMKg1o5CE8fghpLyVqVumJ8a7a3zVZrTQQuD5rxBNZVN5q2FOwYzhF2VjLtJ33iL86htdq9Y30UsHHRyDL7yU9U9rXIAvYn6uU19SZPROUpNJ3JOfiFsBGY00ynHAPWuQQmGHtSnOR20MGjOGyJu8n48ay3YMSTcia3D7hN6K8cA8Be//DI+9IZLSXrhJ5tQ0Gn4U0/4qrnNn2MAVxjmcgyuOF2+Itkkx1D11qsqEXOOO4JR1sP1NKeBFgbNeYMa7DeeW7hjUKEh9YQ+W8nrr1y7hles6yJhGmTd5LPagKc+r57me2Y4hgijmdmH8Kkb/pUrU02qkipeEtvf5KacxHwcw/GJPP/vA/vb6sTWXBhoYdCcN6hQ0nShvPCqJOUY0kUMd/nP3J8JkXbLVfuSJl2xkLfwZ8S9+ffEZzqG8Zw16w18Mm9hGgE29sbJlmwyxVqTW7FcYbU7DbZZKKk8jxzDt7Yd528fOjBnlZTmwkELg+a8wfQN9FvoylHVtzCcLhILB2fsgGhGMlJzDHHTqJujtOdkmp54uGmOQcpaA1szJnIWPfEwK10B8LuGYrnqjQkfyzVxDNX2HcOBkSxAnfAslGpVYtk6v7Hc0cKgOW+IhGr/c16oY1A5hlPTxZbNbY34cwxJ02BlR8Sb0LrrxDQvX9MxQ2D6EnP3MkzkLLriYVak1AIhnzDYFbpiYWLhYINjmH9V0kFXGKYL7a8qbcU9Pz/Gq/+fn+qw1DJHC4PmvMG/+nPhOQZn0m26aM+aeG78TNYVhkTEYE1XlGPjOaYLZfYPZ9i8pnPGZ7zu51kqkyZyFt3xsLcn4lvPDnrJ7GK5QiQUoCcRrs8x2PMLJZUrVQ6POetLM8Uyll3l608eqQtx3fPMMd5+12Ntne/waI7hdMmbV6VZnmhh0Jw3+HcgnG5VErQfjnKa4pwcQ9x0prHmrIqzelPC5tUdMz6jHMPYLI5hMu8Iw5quKO+7aQP37zzJO//hcaSUlOwqkVCQ/mSEQd801/mGko6O57DdCqZM0ebxg2P82fde4NmjtX0RLw6l2TOUbut8qmQ3XTj9sJRm6dDCoDlviCyCY/CHj1p1PTeSjBjkrAp2VZIwDW6+tJd4OMhXHjsMwOY1M4VBCVBultHbEznL23X9F297GR9542UcnygwXSgjpfN7t27oYvvxKe+GXJollJQplmfkEQ4MZ70/p4tlL+cx6ZvmmrcqlCuyrf3SasJspnj6YSnN0qGFQXPeUJdjWKBjCBsBrxKpXXHxi0kyYhAJBXndFQPkrQorUhH6U5EZn1FlsPly85CLZVfJFG26fcP31HRV1f9gGgFec1kfdlXyhLuPulaVNNMxfPjft/Phf99ed0wlnsG5mU+5T/rThVreouBeo9VGCazq/k4vQiJbs3RoYdCcNyxGjgFqJavtJp/ValaodUr/wlUrAHh5E7cAzk09ICBfai4MU+44DL8wJN3vUcIQCQXZur6bWDjIw/tHAd9IjCZP988fm5qR7D4wkmV1Z5RgQJAplr3v9e9/UOGpZmLTiA4lnR+0JQxCiK8KIUaEELtbvN4hhLhPCLFDCPGCEOJ29/h6IcRzQojt7vE7fZ+5VgixSwhxUAjxd6KdukCNZhZMf47hNIRBCUK7+xySvryEChHduqmP3kSYV1/a2/QzQgjiYYOc1TzkMtFUGJxzq4R1xN36d9PFvTy8fxQpZUvHMJopMZ6zZpSSHhjOcNlAglTEIF2wPUGYLtSHkoC2ylCVY9ChpOVNu47ha8BbZnn9g8AeKeXVwK3A54UQYWAIuFFKuQW4Hvi4EGKV+5l/BH4buNT9v9nOr9HMiT+UFFtgKAlqT+btOgZ/wjrpiYrBEx9/Pb95w/qWn4uZQQotqnfUDbozWnMjqRmOwfm9r9nUx+BkgUNjuZblqvuHnTHcJd/NvVqVHBrLcUl/gmQk5DgGVxCmfMIwr1BSSYeSzgfaEgYp5SPAxGxvAZLuU3/Cfa8tpbSklMq7mur7hBArgZSU8inpFDx/HXjHAn+DRgNAOBjw5gcthmNou1zVF0ryi0TYCMzaIBcLGy3LOpvtg0gpx6CEwW3ou2atUw67/1SmNivJvYk/sGeYvUNpXnT3M/if+qcLTnnqqs4oyYhBumh7oSS/Y1DiVbbbSD67gqBDScubxdrHcBfwfeAkkAR+XUpZBRBCrAXuBy4BPiqlPCmE2AoM+j4/CKxudmIhxB3AHQDr1q1bpMvVnI8IIYiGguStyoLWeirUzb3dc/idRbtios5faBFKynt7q2vX0JhjMF3H0OG6imzJroWS3BzD//3d3aztjnJRr7NCVOUgoDbgrydhknIdg/r8dL6ZY5i9N0FK6TkGHUpa3ixW8vnNwHZgFbAFuEsIkQKQUh6XUm7GEYb3CiEG5nNiKeXdUsqtUsqtfX19i3S5mvMVVbI6nxt0Iyoc1H7yeWYoqR3iYYNci+Rzc2GYmWOA2m91hKF+iF7JrvDzI5M8ekAlp2uOwRsJHg973dtTbjXSlL8qyb2W0hw5hrxVQeW8dShpebNYwnA7cK90OAgcBi73v0FKeRLYDbwaOAGs8b28xj2m0ZwWEa/U9HRyDK5jOM1Q0lxEw8G6JTt+8u6Tt7+6KhYOEgyIGaEkVZqb8zkGlWNQHdBqRIdVJwzuSPCkSSoaIl0oN00+e6GkObqp/fso0oswXkOzdCyWMBwDXg/gOoJNwCEhxBohRNQ93gXcDOyTUg4BaSHEDW5e4j3A9xbpWjQXMJGQc/NsZypqKxJeuWp74hIJBQgGBAFRXzI7F3Ez6DmDRlR/g1/ghBAkTGNG8tk0goSDATIl2wsVld3OZ78Q9CVNSnbVm2OkRmn0uI5hMl/2QkBTzUJJczgGf/hIO4blTbvlqvcATwKbhBCDQogPCCHu9JWffhK4SQixC3gI+JiUcgy4AnhaCLEDeBj4nJRyl/uZ3wO+DBwEXgJ+tGi/SnPBEgkF256K2oqEOy+p3V4IIQTJiEHcNOb1vdGQ0VIYClaFgGCGwCUjhlfK2jgCJFcXSpJIKbEqVW8I3xY3Sa2e/MdzFgEBXbEwyUjIE4CuWIhM0aZSlZQrVa8nolEYHj84xh/c87wnMMoxCOHMmtIsX9r6X76U8rY5Xj8JvKnJ8QeAzS0+sw24qp3v12jaJRIKnFZFEvgdQ/vnSZhGWyMj/MTNYMs+hlypQiw8U2iSkRBSOrORTF95biLi5Cv8qz3VDf2269axdUMXL5yc5oE9w1iVKmEjwFi2RHfcJBAQdXmS9T1xJvNTpAtlAoHa96tKp2K5wl/et4d7njkGwK2X9fEr167xehj6kyYZXZW0rNGdz5rzikgoeFo9DADrumOEgsLbkNYOyUho3gnvWHgWx1C2vY109d9T+45IQ0NfplhzDGXfXoRoOMCrLun19lWoeUpjWcvbE+Hv3t7QEwOcXgb/bumSXWU6X+bddz/FPc8c445bLiIcDLB/xCmFzZYcMVjZEdWOYZmzWOWqGs05wSp3vMPpcMulvTz9p2+o6zqei46oMa+taeDkDyy7il2pYgTrn9HyVqXpTgn/k33EqK9YypVsr3LIrlS9J/yQe+6wG5ZSjWpj2ZK3izrZ4BjASUD7/yWtSpUfv3CK7cen+Nt3b+HtW1bzyP5R9rs9EirHsLoz2vY0Vs25iRYGzXnFp95xFdXTXBIjhJiXKAD8yS9cQWWe3+sfpJdqEIZcqUK0SUhMVUAJAaFg7bYdNw0mclZdjsFqEAaVryi54abxrMW6dY47SPk6rDf0uo4hbxH2XVfZrnp5hNdc5pSOb1qRZNsRZ0S3em1VZwTLrvL4wTE+9M3t/OTDt8z731OztOhQkua8wkk+n/3nnavXdvKKdV3z+owKPTUbpFco201LbtWTfcSoT7AnTINsXSip6oWSlFNodAzjbTiGgi+UZFWqXme1CktdNpDkxFTB20cBTigJ4P5dQ4xlSxweq01w1SwPtDBoNEuEuvE3S0C36t72hCFU//91E6azd7po1/oYVGhLPfWr/9eyqxSsCjmrQk9DjiEgYE2Xc2OfbsgxWHbVS24r97FpIAnA/uEs2ZKNaQS8cz59aByA0UzrvdbNsCtV/u6hA954Ds3ZRwuDRrNEKGfTbJBevtRKGJwbeKShXyJuOsln5RJsX/LZCyW5nynZlVpzW7zeMXREQ3TFnBv7VL5clxwvV6qU7AphI+BVK21aoYQhQ6Zkk4wYnsi8NOqsDB2bZX1pM547NsXfPLDfGyWuOftoYdBolgjPMTTZ4pYv201DYjXHUC8MCdPwwj6qzFb93QsluQJRsqu+ruewe17nZt4ZCxMKBkiYxoxQUsmuUipXve5ycBLNsXCQ/cMZskWbhGnUhaVg/sLw4iknca1yIZqzj04+azRLhJd8buIYClalRbmqcwNvbHyrX0nqhJXUWA2VpPZyDHbVy2v0uI4hbASIhALeQL6OaIipfLluyJ8TSqrUiVIgILi0P8H+4QymESQRMeoS2TB/Ydg75I4Ib2PMt+bMoB2DRrNEeMnnZqGkFuWqLR1Dk2VBaqR3uLEqya4ynqvNSaqdO0RXrCYM0wWrLsxVrjjCYDbkN65c1cHOwWmmC2USplHXE5GK1EZ4zEa1KtlxfAqoOYbZRnDI06w808yOFgaNZolQc5Uak8/VqiRvNS9XTbVIPscbHAPUdjoop2D6HIOarNrjKyO9alWKl61yVpF2xkJuKMm5OQcDwks++/snAG64qJtM0Wb3iWkSZohU1PCu9arVHd53AQyni+w5ObPH4dvPDvL2Lz7OEwfH2Ndkd4Sf549N8vK/+AnbXSHRLD5aGDSaJULdzBuTz6oktFnyOdUi+ewf+KdchXIitT4G5z2Wm2NImEbdef759uv432/eBEBXPMxY1qJg2QjhiI3lJp8bv/uGi3oAx4kkIwZRd5Dh5StT9CbMulDS5368jzv/9dkZv+uenzvjNb7w0IE5V4l+46mjZEs2n//Jvqava04fLQwazRLhL1e94+vb+OGuIaB2Q28eSmqVY/CN/jabC0PYF0rKFu26LupGBpIRRtJFCuUK0VAQ0wi4oaTqDLcykIpwUW/c+24hBOt7Yly3odsRBl8o6fhkfkYZ6oHhDM8fmyJpGjxzuLYostlioGzJ5ke7TtETD/PogbG69ysOjmT5/X97rqWw7Bqc1qIyB1oYNJolwjQCBAScmCzwkz3DPOKWZ6rEcPPO5+Y5hrjPMXhVSQ2hpFryueL0Scwy22kgZZKzKoxlLaKhIGEjQMl2Gtwavxvgetc1qPzGfb9/Mx96w6X0JsPkrIoX1hpOl+oqnYbTRb76+BFCQcEn3+HM1FRd3c1u7D/cOUShXOHvb7uGvqTJ5368b0a+4alD4/xg5xCn3B0Ujfxo9xB//9OD3sgQzUy0MGg0S4QQgnjYYPeJaQBG3CfrfFkt6Zl5A1bLehrj/EmfY1AhqtmSzznLbupIFAPuqO7DYzkiIWffg8oxNNt1cePFrjD49mWHggH63M7qsYyFlJJT00XKFWec973PDXL9Zx7inmeO8cYrB3jr5pX0Jkw29sSJhY2mwnDv84Nc1Bfnxot7+PAbLuOZIxN8f8fJuveopjy/APlRTqrY4nWNFgaNZkmJmUH2usnW4bTzhNtsradCCMHqzigDHZG64/EmOQaVuwgZzctVZxsd0p9ybuhHxnPEwo5jUA1uZhPHcONFPZhGgFWd9delqp5GsyXSRdu7WRfKFY6M5wG4+zev5a/etRkjGOCv3vVy/vgtlxM2At7oDj8HR7Jcv7EHIQS//sq1bF7Twafv30vGtxhI3fBb3fjnEg6NFgaNZknxPxl7jqFU8V5rxnc/+Cp+79aL6475y1VryWc3lOQ6BiMgEMKZeZSz7DoxaUQ5hql8magrDJbX4Dbzc31Jk8c+9jredvXq+uPKMWRLdaGdglUhX3LmQb3pZSu8/ok3XjnAW65aQTgYmLFjulKVTOQs+tyRG8GA4M/eeiUjmRI/2n2qdu52HYOlQ0mt0A1uGs0S4ncFY9kSdqXq3dBb7a1uNqnUNIKEgoJyRXqJ6JznGBxhEMJZeVqyq+4sptlyDLUn/0goSKXqTGt1GtyaP0/2Ndlf0esTBn9uIu/Oamp1DaYrRH4mchZVWd97cdVqp7x2JF0THTXPqZVjUMKgHUNrtGPQaJYQ/81fSmd5jrphNet8ng2VW0g0hJL8o7NVriBXmt0xJEzDy0HEwk6OoWxLp8GtiWNohRqoN5opMexzDHnLpjCLawk3EQZvjEeiJgyRUJCkadT1ShTaDCWdqRzDvz19bNnPeZpTGIQQXxVCjAghdrd4vUMIcZ8QYocQ4gUhxO3u8S1CiCfdYzuFEL/u+8zrhRDPCSG2CyEeE0Jcsng/SaNZPqgn5kv7EwCMZIrkSqpcdX6GXiV+kw0NbiGfMJihYFuOAWquwatKqlQp2jPLVWcjFAzQGQs5oaR0fShpNsfQLMeghKGnwTH1Jut7JWo3/uahIvXvciYcg5SSz/7Xi3zjySOLfu6zSTv/hb8GvGWW1z8I7JFSXg3cCnxeCBEG8sB7pJQvcz//BSFEp/uZfwR+Q0q5Bfg34P8s7PI1muWNemJWVT3D6ZJ345qvY/BXBIETMgkGRN1Gu3AwQKlcmbMqCWoJ6GgoSCgYoGDZVKqyabnqbDi9DFa9MJSdEtZW16CcjZ/a4L/6kFVvIlwnDGr4Xqsbf2GO10+HkUyJ6UKZ8dzyHhk+pzBIKR8BZnaR+N4CJIWzNSThvteWUu6XUh5wz3ESGAH6fJ9JuX/uAOrrzTSaC4RoyLmJq+7h4XTRCwG1yjG0ImEahILCKyfNW5W6LW/gxO7TxTJSMmsfA/gcQ9hpcFOrO+fjGAAu6o2z68Q0w9NFT6TyVsXdUjePUJK718EfSgJnEOD4LKEku8F5qP6Okk8YFmv2khrnoa7n50cm+OmLw4ty7rPJYuQY7gKuwLm57wI+JKWs+y8hhLgOCAMvuYd+C/ihEGIQ+E3grxfhOjSaZYdyDNdt7EYI54kzZ1UIBwN1IaD2zmUQMYIYrhgU3PP4CRsBJtyn2bkcQ2MoqSYM8xOsN1wxwImpAk8fnmBdd8y7NscxzC+UFA4GZnRs9ybDTUNJhXKFI2M5rvyzH3sD+qB58vkD/7KNP7l317x+VzP2DythcK7nbx88wF/98MXTPu/ZZjGE4c3AdmAVsAW4Swih3ABCiJXAN4DbfYLxYeAXpZRrgH8G/qbVyYUQdwghtgkhto2OLrGqCYsAABtJSURBVO+EjkbTyNYN3bx2Ux+9CZOeuOmMobDseYeRwEk6m27YB5xRG+GGZjTTCDCVd2r+58ox9Lshm1jYqXhSO52bNbjNxmsv70cIZ5zFhh5HGJRjiLVKPjcJJY1mS/QmwnUrTcFxDJP5sucMCr4cw7GJPFal2ryc1S1X3XZkgp++OMLBkcy8flczlGPIWRWK5QrD6SLThfIcnzr3WAxhuB24VzocBA4DlwO4AnE/8Akp5VPusT7gainl0+7n/x24qdXJpZR3Sym3Sim39vX1tXqbRrMsedvVq/jn268DnDEUw+liy7Wec7FlTSfXrOvEUOGaUmWG6wgbASbdWUWzVSU51+M4hojbx6CYr2PoS5rePuyNvU6SPW/ZczuGGTkGa0Z+AWo5B+WE/OWqagnSowdqD5WFBsfwpYdfqvv76bB/pLbfejxnXdDCcAx4PYAQYgDYBBxyE9D/CXxdSvlt3/sngQ4hxGXu398I7F2E69BoljUDqQgjmRL58sKE4bdvuYh/es9WTwysSrWJYwgyVWjPMdSFkoK165lPuariDVcMALCxzxm251UlzVKu2tjgNp4tzcgvAPS6VUqqZNVfjqp6OV44mWY0U6JcqWJXpff6geEMD+4dQYjWVUztUq1KDgxnWNvt7MwenMiTLtrOjKll1jPRTrnqPcCTwCYhxKAQ4gNCiDuFEHe6b/kkcJMQYhfwEPAxKeUY8GvALcD73LLU7UKILVJKG/ht4DtCiB04OYaPnoHfptEsK/qTplOVVGq+1rNdDF/CuZljUHnWuRzD6i7nBtcVC3tjNWD+yWeAt25eSX/S5Np1XRgBJyxl2dWWjsFskWPoTcxs7lOOQeUZvByDb3gfwOMHx+qWIhXLFXYMOnOqXrGua8b481zJplptPyl9YqpA3qpwo1tIsHeotnciXVxermHO//VJKW+b4/WTwJuaHP9X4F9bfOY/cdyERqNx6U9FGM85M4UWkmNQGIHajXuGMPj+Ppf4rO6M8q07b2Tzmg4OjdZCJAtxDGu7YzzziTcATpWTerpv5YwacwzVqmQ8azV3DO4xtZXOyzHYVa8nJGkaPLJ/1Kv+Akc41IylFR0Rjozl6l676a9/yid+6Qp+bevatn6jyi/cdHEv/7Ft0FtRCpAulOlPRlp99JxDdz5rNOcIazqjSAnPHp1cUChJ4S9RnRFK8j3tt9NA98oN3ZhGY47h9G4bsXDQe7qPtyiZbcwxTBfK2FXZVBhUd7UqZ/U7hlzJJiDgpkt6eP74VF0eoVCukHUrrfoSZt1rh8ayTBfKvOTLGcyGlJJv/vwY0VCQm9yelD0+x+DPM7z17x/lX5862tZ5lwo9K0mjOUd4+zWrCAQEzxwe53WX9y/4PEbdCIz6Cp46xzBHKMmP33nMN/ncSDRUE4aWjqEhlNSquQ0cNxA2AozlSkgpvVxByVaNfAb9yQhPH56oCy0Vy1WyJRvTCJCMGBTLFaSUCCF4adRxDxNtNqr9aPcpHtw7wp/+4uX0JU1MI8C+4ZpjUMIgpeSFk2m2H5/if96wvq1zLwVaGDSacwTTCPKr167hV69dc1rnMQKtcwzzdQyK06lKaiQaNnzC0MIxBJ3BfZWqJBgQjHpzkmbmGIQQ9MbDjGWsuoS1M8G1Qtw06IyFSBfKM3IMmZJNMhIiEgpSlU7C3jSCnlNoRxiklHzyB3u4cmWK979qo3M9CZMTUwXvPemCI0hWpYqUtRHrfiZzFofHc1zan/A29S0VOpSk0Zxn+MWgMZSkqouEmF9IaLFDSaozuOVIDN/uCKhVHPU1CSWB4yTGc6W66p+i6xhiZpCOaIiqhJF0rRFOhZLUnmqojeJ+yc2ptDPaomRXGZou8tarV3puTYW31JgS5RiUm/Ffh+Kj397Ju/7hCa7+y5/w0N6l7ZbWwqDRnGcEAwJlGppVJYHjFhobxWajbhDfApLPfmLhoFcy2mosR6MwqMmszUZ7gzNYbyxbvzZU5RjiYYPOmHOjHpp2nuKjoSAFq0K2ZJMwDS/Zrz6vQkmT+bmFQTX+JXy/RY1Gv8QdjqiEQY3hGM7UO4ZMscwj+0d5wxX9VCUcaDO3cabQwqDRnIeoJ9fGkRiqa3m+yW1zER1D1BeKmssxlCrqRp2lOx72bvCN9CaceUnqidwICIrlqjvBNUinuwhoyBWY7niYou04hoRpeL+pUK5QrUqvCmsiO7cwqCY6f2iuJ+4I2OquKLFwkHSDY5jKl+vczU9fHMGqVLnjFmcBU95a2r4HLQwazXlIKFC/zlPhOYY5BujNOJ8/+bwIjsH7c4vrMIP1juHgSNZ7+m5GjysMqhehMxam6E5wTZgGHTElDI5j6I6HKVgV0sUyCX8oqVzhxFSBkl1ldWeUjNtvMRvKMfj/TVUuZCAZoSMaqoWS7NoNfzRTCyf9+IVT9CVNtq7vIhIKeIP+lgotDBrNeYhyDM2mq8L8HUPY50ACgfZDUM3w92i0k2OQUnJgDmHoioWwKlUvWdwZCznCUKoQM40ZjqEr7ghHtmSTNA0voV4oV7z8wtYNzhiPucJJ3v4MX5WXCiUNpExSEZ8w+FyCSkAXrAr//eIob37ZAIGAIBY2tGPQaDSLjxKEGVVJvhzDfFA3avM0w0hQGzUOs1QlKWGoVBnLWkwXyt4yo2Z0NeQQumIhJ7lccnY+KMeg9k73xMPe6/XJ54qXX3jlhm6AupHeiqm8xdvveoy9Q2ly1kzH0OMmyQdS9Y7B3109nC5h2VU+/O/bKZQr/PLmVe6/T3BGF/bZRguDRnMeorqfW4WS5tPDADWBOd3EM9TcSigoZlyfIuwLJR1wp57O5hg6Y/WOoDMWpiphqlAmFjbocB3DcNrZCZGKGE7yuWiTiNQ7hsNjWTqiIe/7mjmGpw6Ns2Nwmh3Hp7wcgz/5vMKdM7W6K0oqGiLtNtIVfWGp4XSRP/3PXfzXC6f4s7deyfVuV3YsHFzyfdRaGDSa8xA1L2lm8tm5AS7UMZxu4hlqoaToLP0Q/lCS6imYNZQUV47BDRW5QmHZVRJmENMIEgs7vQrRUJBIOEimZGNXJQkzVFeVNJGz6E2EvRWi4znLm+2keP6Ys98hXSzXks8+Ybjp4h7++X2vZOv6LlJRw5d8rt3wBycL/GDnSW67bi3vv3mjdzwWDupQkkajWXxCXo6hhWNYYFXS6Ta3+b97tgS4XxgOjmRJmIb3FN6MWg5BhZJq1Usqwa3eEw0HiYaC3jDB+uRzlUzRaXpTeYKJbIl3fvFxPveTfd45lTBkijZZb0d37d8mEBDuHgpRn3z2CcODe4cplqu85rL6dQLRsA4laTSaM4DRqiopeHpVSYvhGJQwzCZOtXLVKgdGslzcn5i170KVsZ7yhZIU6oadcoUh5gqDojH5nC7apKIhOmNhhIB9wxkOjGQ56DqXcqXKzhOuYyg0dwx+OqIhsiUbu1L19lGvSEU4NpEH4LqNPXXvj4UN8mVdlaTRaBYZo4VjUMnjeVclGYuXY1A34VkdQ7DeMcyWeIaZOQYVSoJaglu9JxoK1jkff4Nb0Z24mowYBAOCzmiIn+1zlvyoMR77TmW8foR00fY25bVaxZpyx1tkirZXrrrO3WR32UDCcyaKqA4laTSaM0GtKqn5EL35OobFzDGoG/Vs4qRCV9mizUimxHp3V3QrQsEASdPwQjadPmGIe6Ek5wYcbXQMEYOIUWtwSxdsb690dzzsic2Y23fw/LFJwBGfjJtjSMzy76kS39OFWlOb+j3XN7gFgJiuStJoNGcCFUpq3M+80ByDEpjTbW7zf/dsCXB1nSPuzbgr3rzj2U9n3LkBBwR1Q+hUf4ESi1jYST4rEhEDI/j/t3fmMXbV1x3/fN8+w8zYY8/gbYyNzWIoq3EBszUpVYA0CV1CgTYhOEUoEVFJq6ohQmqookpN01QVpSoiKiIQClESaIlQlLRV2/QPSGuINzZjwBDbYJvVNl5mO/3j3vvmvvHc9954rt+7M3M+0mju/O4y5/3ufb9zzzm/3zk5inlxaCiwGKK3/GgFMwT5msyMX7zxPv3dZU5b0M2+Q8NB7eo6/RlXDFGd6WWhxRCvDxHhwWfHcY4Lia6kY5yVVA6T76URfI7cNkmrnmFMMUSLwOIWQBJRwLkyzlUUWShzYq6k2hhDsXrevkNDHBkepTtmMUCQf2pwZJR9h4fZtvcAqxZ2M6ejyL7DQ9V8S0n0xC2G4RGKebF6WS+L5lRYu/JoxdBRKrjF4DhO+iQtcFvQU6ZSzLEirL3cLGkucBuzGOoEn/ORxRDFDJqwGMJjxg/8VYuh6koq1LjEukIl0FHMVy2UyOKILJU1y4JV0G8fOMLO9w4x0NtJT0eR/YeHOTg4XNc1F1kM+w4HrqRKIc8lK/t46qtXHhVfgKB/BkdGGR6ZWg3qqeCKwXFmIEkL3OZ3lXn+L65mTbiqt1kiRZNG8DkatOuVL626ksL01M1ZDGNv/pUJ6k5EA3RnguKoxBRDT0dwTrSW4cozgsJJO947xDsfDjLQ20F3JVifcCCs+ZBEdK0gxjBKuYHVFSnOg21c5NZQMUi6X9IeSVsS9s+R9CNJGyU9J2ld2H6epKfCtk2Sro+dI0l/KWmrpBck/VF6H8lxnCSLATimXEeFfI7cJGs4JNExiRhDlJ46KatqnDFXUq42H1N53Kyk0pirqVTIVZVdRzHP3tB1FbmXLj2lj19fdSKXrOwDYNMvg2mqS+Z20FMpsv/IMPsPD9W1fuKzko4MjTTsw+piuza6k5pxND4A3AM8mLD/NuB5M/ukpH7gJUkPAweBm8zsZUmLgWck/cTM3gduBpYCq8xsVNKx1zF0HOcoxiyGqSW8i3P+Sb2cuahnytfpLheDqaB1rIDIlbQ7tBh6m7AY5sYthkI8xhC5ko5WDN2xN/1KKV9NoBfFBdaunM/alfOrmVA37ggUw+K5HdUiPrs/OMzqk3oT5eos5cnnxP7DQYyhUZymajFkWTGY2c8kLa93CNCtYPVJF/AuMGxmW2PX2CVpD9APvA98Efh9MxsN9+855k/gOM5RjKXEmLrrJ+KHX7wklet0lPI8euvFrFrYnXiMJEr5HIPDo5QKubrpMyKqA38xX33rzudUnZlVE3wO90dB5qA9Vy0gFG+HIAidE2yILIbeDl5/J0i29+HgSN3gsyS6yoVgHcPQaGOLIUwyeLCNqbfTiDHcA5wB7AI2A7dHA36EpAuBEvBK2LQSuF7Sekk/lnRqCnI4jhNSSki7nRV+dfm8hnWNI3fS3I5iU9XmokBxpZiPZZHNV8+N3FHxlc9dMQVQGbe2IU4+J+adUObtA4Pkc2JBd3nCKbFJdFcCxXBocKShkuvMgCspDcVwFbABWAycB9wjqWpvSloEPASsiymMMnDYzNYA3wbuT7q4pFtDBbJ+7969KYjrODOfyGIoJmQvnQ5EiqGZGUkwNvBXioEyKBdyNUHhhT0VPn3BAJed2jemGMpxi2FswI5cSXGi4jsLeyoU8rlqUBkaLxjsrhSnlSspjadmHfCYBWwDXgNWAYQK4kngTjN7OnbODuCxcPtx4Jyki5vZfWa2xszW9Pf3Jx3mOE6MpNKe04lI9mZmJEF8VlJwXkcpX7PwLJ8Tf3Pduaxa2FO1KLrKY9eOFIMEXRMExqN600t6O4CxoDI0XhfSUymwL3QlNZrZ1TFOMRwZHmm59ZDGU/MGcCWApAXA6cCrkkoEg/6DZvaDcef8C/DRcPvXgK04jpMaSaU9pxNVV1LTimFsHQMEq7ST3uRzYewh7jKKVkN3lQoTztzqC4vvDMydQDE0YTHsOzTU1KykaEHeoTCR3n+9tJez7/oJz+36oO55adIw+CzpEeAjQJ+kHcDXgCKAmd0LfB14QNJmQMBXzOxtSZ8BrgDmS7o5vNzNZrYB+CvgYUl/DBwAbkn1UznOLCdp5fN0YvKupLFZSRC8edd7k+/rKrMglso7msk0kRspOD6Qo2oxxFxJXQ1iDD1hjMHMJu1Keub198hJrOyvn0gwTZqZlXRjg/27gI9N0P5d4LsJ57wP/GaTMjqOM0mqs5Kms8VQdSU1pxi6ygU6S/nqgN3bWWR+V/K53//C2hol0FEK/t/4wHNEZDEsCS2GeHyiscVQYP/hIQr53KTXMazf/i5nD8xJJR1Js0wuYYrjONOCYi7bs5KaYbKuJEk8fMtFnBRmLr37xvPrKsbF4QAfEbmgGiqG0GIo5HN0lQscODKcWLs6orsS1GQoF/INExF2FscshsNDI2zZuY91ly6ve07auGJwnBlIUmnP6cSYK6k5xQDBIryIgd76qbrHE72R9yRMo71oxTyuOK2fcwbmVtu6K4WGSfSi40YtSOvd6M2/kM9Ryuc4ODjC5p0fMDgyygXLkhfQHQ+m71PjOE4iUWxhOruSyoXJuZKmSqWBxTDQ28mDn7+wmnMJxpRIo3UMcZdVM2lFgvKew6zfHtR+aLVicIvBcWYgnzhnEeVCrqGLI8tUYwwJweC0GXMlNf//onhGMxZDRDOxgqgmwzOvv8eKvhOY31VueE6aTN/XCcdxElk2/wRuuXxFu8WYElVXUhNFetIgCvrGZxs1IrIYmokxRDSjGDpKeQ4OjfDsG++xusXWArhicBwno0w2+DxVjsVi6K4UKOVzDV12x2IxvLLnAO9+OMi5S+c2PD5tXDE4jpNJxlxJrbEYoiJESTGGiVg6r5PFcysNj+upUQyNh93OYoEX39oPwFmLp57RdrJMXwek4zgzmkoxT1e50LIAekeDWUkTcdtHT2HdpSc3PK7GldREsaN4dtgzUkh1PllcMTiOk0k+u3YZF62YXKW5qbBqYQ/XXTDAxSuOrsOcxPj60kn0TDLGEK1+PvXErpYubItwxeA4TiY5bUE3py1IrtmQNh2lPN+87tzjcu1KMUchJ4ZHrbnpqqEyOHvJnOMiTyM8xuA4jnOckVSNXTQ7Kwng7AFXDI7jODOWKM7QVPA5VAxnucXgOI4zc5mMxXBid4XOUp4zFrY+8AweY3Acx2kJPZXatOD1+OzaZVx91sKqS6nVuMXgOI7TAiZjMVSKeZbOm1wSwDRxxeA4jtMCqjGGaZDYMPsSOo7jzAC6KwUKOVWr62UZjzE4juO0gE9fMMCy+e1zD00GVwyO4zgt4Kwlc9o2/XSyNLRpJN0vaY+kLQn750j6kaSNkp6TtC5sP0/SU2HbJknXT3Du3ZIOTP1jOI7jOGnRjLPrAeDqOvtvA543s3OBjwDfklQCDgI3mdmvhOf/naRq/lhJa4DWJxp3HMdx6tJQMZjZz4B36x0CdEsS0BUeO2xmW83s5fAau4A9QD+ApDzwTeDPpia+4ziOkzZphMfvAc4AdgGbgdvNbDR+gKQLgRLwStj0JeAJM3uz0cUl3SppvaT1e/fuTUFcx3Ecpx5pKIargA3AYuA84B5J1XXckhYBDwHrzGxU0mLgOuDvm7m4md1nZmvMbE1/f38K4jqO4zj1SEMxrAMes4BtwGvAKoBQQTwJ3GlmT4fHnw+cAmyTtB3olLQtBTkcx3GcFEhjuuobwJXA/0haAJwOvBoGoB8HHjSzH0QHm9mTwMLob0kHzOyUFORwHMdxUqChYpD0CMFsoz5JO4CvAUUAM7sX+DrwgKTNgICvmNnbkj4DXAHMl3RzeLmbzWxD6p/CcRzHSQ2ZWbtlaBpJe4HXj/H0PuDtFMVJkyzLBtmWL8uyQbbly7JskG35pptsy8ys6SDttFIMU0HSejNb0245JiLLskG25cuybJBt+bIsG2RbvpkuW/azOTmO4zgtxRWD4ziOU8NsUgz3tVuAOmRZNsi2fFmWDbItX5Zlg2zLN6NlmzUxBsdxHKc5ZpPF4DiO4zTBrFAMkq6W9JKkbZLuaLMsSyX9p6Tnw5Tkt4ftd0naKWlD+PPxNsm3XdLmUIb1Yds8Sf8m6eXwd1uy4ko6PdY/GyTtk/TldvXdRCnpk/pKAXeHz+AmSavbJN83Jb0YyvB4lPFY0nJJh2J9eG8bZEu8j5K+GvbdS5KuOp6y1ZHvezHZtkvaELa3uu+SxpD0nj0zm9E/QJ4ged8KgkR+G4Ez2yjPImB1uN0NbAXOBO4C/jQD/bUd6BvX9tfAHeH2HcA3MiBnHngLWNauviNYwLka2NKor4CPAz8mWAR6MfDzNsn3MaAQbn8jJt/y+HFtkm3C+xh+PzYCZeDk8Pucb7V84/Z/C/jzNvVd0hiS2rM3GyyGC4FtZvaqmQ0CjwLXtksYM3vTzJ4Nt/cDLwBL2iVPk1wLfCfc/g7wW22UJeJK4BUzO9YFj1PGJk5Jn9RX1xKkhzEL8obNVZBgsqXymdlPzWw4/PNpYOB4ypBEQt8lcS3wqJkdMbPXgG0E3+vjRj35JAn4PeCR4ylDEnXGkNSevdmgGJYAv4z9vYOMDMSSlhMkFfx52PSl0NS7v13uGoL6Gj+V9IykW8O2BTaWIv0tYEF7RKvhBmq/mFnoO0juqyw+h58neJOMOFnSLyT9t6TL2yTTRPcxa313ObDbwnozIW3pu3FjSGrP3mxQDJlEUhfwQ+DLZrYP+EdgJUHq8jcJTNV2cJmZrQauAW6TdEV8pwW2aVunsilI0Pgp4PthU1b6roYs9FUSku4EhoGHw6Y3gZPM7HzgT4B/Vix9fovI5H2cgBupfSlpS99NMIZUmeqzNxsUw05gaezvgbCtbUgqEtzQh83sMQAz221mIxYUOfo2x9lUTsLMdoa/9xBkx70Q2B2ZnuHvPe2QLcY1wLNmthuy03chSX2VmedQQVLLTwB/EA4ghG6ad8LtZwj8+Ke1Uq469zFLfVcAfgf4XtTWjr6baAwhxWdvNiiG/wNOlXRy+KZ5A/BEu4QJ/ZP/BLxgZn8ba4/7/H4b2DL+3BbIdoKk7mibIFC5haC/Phce9jngX1st2zhq3tiy0HcxkvrqCeCmcIbIxcAH1kQFw7SRdDVBSd1PmdnBWHu/gpK7SFoBnAq82mLZku7jE8ANksqSTg5l+99WyhbjN4AXzWxH1NDqvksaQ0jz2WtVJL2dPwRR+a0EmvzONstyGYGJt4mg8t2GUL6HCEqjbgpv5KI2yLaCYPbHRuC5qK+A+cB/AC8D/w7Ma2P/nQC8A8yJtbWl7wiU05vAEIHf9g+T+opgRsg/hM/gZmBNm+TbRuBvjp69e8Njfze85xuAZ4FPtkG2xPsI3Bn23UvANe3ou7D9AeAL445tdd8ljSGpPXu+8tlxHMepYTa4khzHcZxJ4IrBcRzHqcEVg+M4jlODKwbHcRynBlcMjuM4Tg2uGBzHcZwaXDE4juM4NbhicBzHcWr4f5+j1DlHlTw7AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.plot(learn.activation_stats.stats[1][-5].numpy());" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class Hook[source]

\n", "\n", "> Hook(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `hook_func`:`HookFunc`, `is_forward`:`bool`=`True`)\n", "\n", "Create a hook. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hook)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Registers and manually deregisters a [PyTorch hook](https://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks). Your `hook_func` will be called automatically when forward/backward (depending on `is_forward`) for your module `m` is run, and the result of that function is placed in `self.stored`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

remove[source]

\n", "\n", "> remove()" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hook.remove)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Deregister the hook, if not called already." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class Hooks[source]

\n", "\n", "> Hooks(`ms`:`ModuleList`, `hook_func`:`HookFunc`, `is_forward`:`bool`=`True`)\n", "\n", "Create several hooks. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hooks)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Acts as a `Collection` (i.e. `len(hooks)` and `hooks[i]`) and an `Iterator` (i.e. `for hook in hooks`) of a group of hooks, one for each module in `ms`, with the ability to remove all as a group. Use `stored` to get all hook results. `hook_func` and `is_forward` behavior is the same as [`Hook`](/callbacks.hooks.html#Hook). See the source code for [`HookCallback`](/callbacks.hooks.html#HookCallback) for a simple example." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

remove[source]

\n", "\n", "> remove()" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hooks.remove)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Deregister all hooks created by this class, if not previously called." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convenience functions for hooks" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

hook_output[source]

\n", "\n", "> hook_output(`module`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → [`Hook`](/callbacks.hooks.html#Hook)" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(hook_output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Function that creates a [`Hook`](/callbacks.hooks.html#Hook) for `module` that simply stores the output of the layer." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

hook_outputs[source]

\n", "\n", "> hook_outputs(`modules`:`ModuleList`) → [`Hooks`](/callbacks.hooks.html#Hooks)" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(hook_outputs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Function that creates a [`Hook`](/callbacks.hooks.html#Hook) for all passed `modules` that simply stores the output of the layers. For example, the (slightly simplified) source code of [`model_sizes`](/callbacks.hooks.html#model_sizes) is:\n", "\n", "```python\n", "def model_sizes(m, size):\n", " x = m(torch.zeros(1, in_channels(m), *size))\n", " return [o.stored.shape for o in hook_outputs(m)]\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

model_sizes[source]

\n", "\n", "> model_sizes(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `size`:`tuple`=`(256, 256)`, `full`:`bool`=`True`) → `Tuple`\\[`Sizes`, `Tensor`, [`Hooks`](/callbacks.hooks.html#Hooks)\\]\n", "\n", "Pass a dummy input through the model to get the various sizes. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(model_sizes)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It can be useful to get the size of each layer of a model (e.g. for printing a summary, or for generating cross-connections for a [`DynamicUnet`](/vision.models.unet.html#DynamicUnet)), however they depend on the size of the input. This function calculates the layer sizes by passing in a minimal tensor of `size`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class HookCallback[source]

\n", "\n", "> HookCallback(`learn`:[`Learner`](/basic_train.html#Learner), `modules`:`Sequence`\\[[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)\\]=`None`, `do_remove`:`bool`=`True`) :: [`LearnerCallback`](/basic_train.html#LearnerCallback)\n", "\n", "Callback that registers given hooks. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(HookCallback)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For all `modules`, uses a callback to automatically register a method `self.hook` (that you must define in an inherited class) as a hook. This method must have the signature:\n", "\n", "```python\n", "def hook(self, m:Model, input:Tensors, output:Tensors)\n", "```\n", "\n", "If `do_remove` then the hook is automatically deregistered at the end of training. See [`ActivationStats`](/callbacks.hooks.html#ActivationStats) for a simple example of inheriting from this class." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Undocumented Methods - Methods moved below this line will intentionally be hidden" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

remove[source]

\n", "\n", "> remove()" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(HookCallback.remove)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_begin[source]

\n", "\n", "> on_train_begin(`kwargs`)\n", "\n", "To initialize constants in the callback. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(HookCallback.on_train_begin)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_end[source]

\n", "\n", "> on_train_end(`kwargs`)\n", "\n", "Useful for cleaning up things and saving files/models. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(HookCallback.on_train_end)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

hook[source]

\n", "\n", "> hook(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `i`:`Tensors`, `o`:`Tensors`) → `Tuple`\\[`Rank0Tensor`, `Rank0Tensor`\\]" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats.hook)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_batch_end[source]

\n", "\n", "> on_batch_end(`train`, `kwargs`)\n", "\n", "Called at the end of the batch. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats.on_batch_end)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_begin[source]

\n", "\n", "> on_train_begin(`kwargs`)\n", "\n", "To initialize constants in the callback. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats.on_train_begin)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_end[source]

\n", "\n", "> on_train_end(`kwargs`)\n", "\n", "Useful for cleaning up things and saving files/models. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats.on_train_end)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## New Methods - Please document or move to the undocumented section" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

hook_fn[source]

\n", "\n", "> hook_fn(`module`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `input`:`Tensors`, `output`:`Tensors`)" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hook.hook_fn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "jekyll": { "keywords": "fastai", "summary": "Implement callbacks using hooks", "title": "callbacks.hooks" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 2 }