{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Hook callbacks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This provides both a standalone class and a callback for registering and automatically deregistering [PyTorch hooks](https://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks), along with some pre-defined hooks. Hooks can be attached to any [`nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), for either the forward or the backward pass.\n", "\n", "We'll start by looking at the pre-defined hook [`ActivationStats`](/callbacks.hooks.html#ActivationStats), then we'll see how to create our own." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [], "source": [ "from fastai.gen_doc.nbdoc import *\n", "from fastai.callbacks.hooks import * \n", "from fastai.train import *\n", "from fastai.vision import *" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class ActivationStats[source]

\n", "\n", "> ActivationStats(**`learn`**:[`Learner`](/basic_train.html#Learner), **`modules`**:`Sequence`\\[[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)\\]=***`None`***, **`do_remove`**:`bool`=***`True`***) :: [`HookCallback`](/callbacks.hooks.html#HookCallback)\n", "\n", "Callback that record the mean and std of activations. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[`ActivationStats`](/callbacks.hooks.html#ActivationStats) saves the layer activations in `self.stats` for all `modules` passed to it. By default it will save activations for *all* modules. For instance:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "Total time: 00:02

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_loss
10.1123840.083544
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "path = untar_data(URLs.MNIST_SAMPLE)\n", "data = ImageDataBunch.from_folder(path)\n", "#learn = create_cnn(data, models.resnet18, callback_fns=ActivationStats)\n", "learn = Learner(data, simple_cnn((3,16,16,2)), callback_fns=ActivationStats)\n", "learn.fit(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The saved `stats` is a `FloatTensor` of shape `(2,num_modules,num_batches)`. The first axis is `(mean,stdev)`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(193, 3)" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(learn.data.train_dl),len(learn.activation_stats.modules)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "torch.Size([2, 3, 193])" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "learn.activation_stats.stats.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So this shows the standard deviation (`axis0==1`) of 2th last layer (`axis1==-2`) for each batch (`axis2`):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd4XGeV+PHv0aj3XizJKu69xHGcOMUpOI4hnZKQQKgGliwJZSGQXRJgF9gNP9hAAtlsCBCW9AIOqU5i0p1Y7t2W5aLee9fM+/vj3hmPpFGxLGlG0vk8jx6N7r0zOrqauee+XYwxKKWUmpqC/B2AUkop/9EkoJRSU5gmAaWUmsI0CSil1BSmSUAppaYwTQJKKTWFaRJQSqkpTJOAUkpNYZoElFJqCgv2dwC+JCcnm9zcXH+HoZRSE8a2bdtqjDEpp/u8gEwCubm5FBQU+DsMpZSaMETkxEiep9VBSik1hWkSUEqpKUyTgFJKTWGaBJRSagrTJKCUUlOYJgGllJrCNAkopdQUNmQSEJFsEdksIgdEZJ+I3ObjmDUi0igiO+2vH3rtWycih0SkUETuGO0/QCmlApkxhicLiqlr7fJ3KD4NZ7BYD/BtY8x2EYkBtonIJmPM/j7HvW2M+Zj3BhFxAPcDHwFKgK0istHHc5VSalIqa+zgu0/v5tK5qTx0ywpExN8h9TJkScAYU26M2W4/bgYOAJnDfP2VQKExpsgY0wU8Dlw90mCVUmqiqbdLAK8frOKFPeV+jqa/02oTEJFcYBnwgY/d54rILhF5SUQW2NsygWKvY0oYIIGIyAYRKRCRgurq6tMJSymlAlZDWzcAseHBfO/p3fz8pYM0tnf7OapThp0ERCQaeAa43RjT1Gf3diDHGLME+A3wV/fTfLyU8fX6xpgHjTErjDErUlJOew4kpZQKSA3tVkngV59aypo5qTz41lF+8cohP0d1yrCSgIiEYCWAvxhjnu273xjTZIxpsR+/CISISDLWnX+216FZQNkZR62UUhOEuySwKDOO+29azoWzU9hSVOvnqE4ZTu8gAX4PHDDG/HKAY9Lt4xCRlfbr1gJbgVkikiciocANwMbRCl4ppQKdu+onNiIEgBU5CRypaqGhLTB6Cw2nd9Bq4DPAHhHZaW/7ATAdwBjzAPBx4Gsi0gO0AzcYYwzQIyK3Aq8ADuBhY8y+Uf4blFIqYDW2dxMR4iA8xAHAWTmJAGw7Uc+l89L8GRowjCRgjHkH33X73sfcB9w3wL4XgRdHFJ1SSgWghrYubn10Bz+/fhFZCZFDHhsfGeL5eWl2PMFBQoGdBJwuwxNbi7lq6TSiw8Z/iRcdMayUmpD2ljZS1dzhl9+9o7iBdwpr2FJU12/ffW8c4amCU50iG9q6iYs4lQQiQh0syIxj2/F6AN46XM0PntvDA/84OvaB+6BJQCk1IX3uD1u55+X+vWye31XG1fe/yx3P7OZYTeuY/O7S+nYAyhvae23vcbq4b3Mh331mN6/trwSgob13EgCrXWBXSQOdPU5etY975P3jtHb2jEm8g9EkMMkZY3hlXwU9Tpe/Q1Fq1HR0O6lp6WR/ed/e6vD0thIKK5vZuKuMf35sO06Xz17pZ6TUvviXNfYuiRRWt9DR7SIqNJhvPL6DisYOGtu6e1UHAayemURnj4uX9lTw+oFKZqZG09TRw2Mfnhz1WIeiSWCS23q8nq/8eRub7LsNpSaD6uZOAI5UtfS6wXG5DDuLG7hq6TR+dt0i9pY28WRB8UAvM2Il7pJAY++SwO6SRgB+dNUC2rqc7CppoKG9i/iI0F7HrZmdyuy0aO5+fh9VzZ18/eIZrMxL5PfvHKN7nG/YNAlMcgfsO6VDlc0jfo3q5k7ueeUgz+/SIR4qMFQ2WXfgXT0ujteeqvI5VttKY3s3y7ITuGrJNFbmJnLPK4do73KO6u8vrW8DoLyhd0lgb2kj0WHBXDw3FYDiujYafJQEgoKE2y6dTUNbN44g4eI5qXxn7Rx+sH4eQeM8t9D4N0WrceW++BdWtfTb9y9P7aKt28kd6+aSnei7h8P+siau/917tHc7mZcRy5VLpo1pvEoNR2VTp+fxwYpmZqbGALD9hNXYumx6PCLCly/M58uPFLC/vNHTNXM0uEsCZX1KAntKG5k/LZaEyBBiwoM5UtlCZ4+LuD5JAOCKheksmBZLYlQo8ZGhrMwbvfhOh5YEJrnDFb6TQGtnD89sL+GF3eVc9ss32Xq8fy8HgH8crqK928m6BekcrW4Zk/pVpU6XuyQAcKjiVCl3R3EDMWHBzEiJBmBhZixg3cyMxOaDVdz00BbufG4PBfZnpLPHSVVzJ9FhwTR39NBiN+b2OF3sL2ticWYcIkJ2QiS7S63qob7VQWCVBp74yrk8cPNZI4pttGgSmMSMMZ6SQFFNa68L+O6SRlwGfnbdItLjwvnGYzt8jmAsqm4lNSaMS+al0tXj4mRd27jFr9RAKps7CHUEkZ8SxUHvJHCygaXT4wkKsqpU0mPDSYgMYd8Ik8Aj7x9n+4kGNu4s4+MPvM/3n93jKQUsz0kATvUQOlJl3fUvyooDYHpiJIftz1/f6iC36LBgovwwNsCbJoFJrKKpg+aOHhZnxdHV46LY6wK+/aRVbF63IJ3f3LiMmpZO/vWvewGrXvOht4sAKKpuIT8litlpVnH7SGUz7xbWePYr5Q9VTZ2kxoYxLz3WUxJo7ezhUEUTy6YneI4TEeZPi/XZi2go3U4XHx6r4+NnZfHBnZfy2XNzeOzDkzy51WpoXplr/R53D6EP7PmAFmXaSSAp0nPjFR/hOwkEAk0Ck5j7w7F+UQbQu0pox8kG8pOjSIgKZXFWPF+5cAYv7CmnqLqFO5/bw7+/cIDalk6OVreSnxLNzFSreH2kqoX/fu0wv9x0GGtmEKXOzJ6SRlyDVDM2dfSfdrmyqYO02HDmpMdwsq7NU73pMnD+zORex87PiOVgRfNpd5PeXdJIa5eT82YkERkazA/WzyMmLJi/fGB141yRa9Xhlze0U1zXxv/bdJhl0+PJTYoCIDshwvNasZoElD+4i6LrF1pJ4IidBIwx7CyuZ+n0eM+xt5yXS0hQELc/sZNddje31w5U0tjezYyUaKLDgpkWF862E/VsP9lAW5fTUxeq1EjtLmngyvve4entJT737y9rYumPXmWvXbfuZiWBMJbZ7+HfvFHIbzcfZWVuImfnJvQ6dv60WLp6XBSd5sCx94/WALAqPwmA8BAHaxek09LZgyNIWJodj4jVSPyNx3cA8Osblnmqorw7WwxUHRQINAlMUi6XYV9ZE6kxYUxPiiQ1JsxTEiiua6empYvlXsXmlJgwrl46jd0ljSRHhxHiEJ4qsD6Y+SnWnc2stBg2H6ryFHG9e2goNRJvHbYWkBqo+/GB8iZcxup1462qqZPUmHDOn5nMJ87K4oE3j1LR1MFtl83qt3zj/AyresbdOOx0GZ8dHErq27j+d+9xwX+9wVf+XMCmA1XMz4glIepUo+6VS6wbqvTYcMJDHKREh/HohyfZcbKBn1y9sNeFf3qvJNC/YThQaBKYhAqO17H4R6/yt51lzJ9m9Y6YmRrNq/sqOO9nr3PDg+8DeO6i3L54QR5BAl+9KJ/50+IosLvbzbR7WsxKjca7Bsi7h4ZSI/FOoXW3/d7RWp8LsbsbYY973cW3dvbQ3NlDWmw4IsK/X7uQc/OTuHhOCufNSOr3GvkpUYQGB3naBX7y9/2c89PXec++03fberyObSfqmZ0aw+aD1ewqbuj3eqtnJpMQGUKmXdWTER9BXWsXy6bHc/XS3t2nMxMiEIHgICEq1HG6p2bc6DiBSWhLUS0tnT38+zULuXSeNWjlyiXTaOroZmZKNC2dPSzJjmeO3djrNjc9ljf/5WIy4yMoa+hgV3EDocFBTIu33vDuxuF5GbEcKG86rSTws5cOkBwVxpcuyAu4hbbV+Ln8V29xzbJMvrZmBu1dTrafaOCCWcm8faSGl/dW8Olzpvc6vrTB6szgXZVTZY8WTosNAyAs2MGjXz4HY/D53gpxBDEvI5ZtJ+oxxvDS3nJqWjq5+aEP+Mk1C7npnBzrde2S7b03LuNoVQv/+fJBrj8rq99r3XvDMiLti/q0uHB2FcOd6+f1+91hwQ7SY8PpdroC+j2vSWASOl7bRlpsGDevyvFsu3HldG5cOX2QZ1ncxdmzchJ4+N1j5CdH4bDrOOdlxNqvlc0P/7aPCh9JwOkynuPdOnuc/O9bRbgM7C5t5N5PLfXUm6qpo761i0OVzXx4rJavrZnB1uN1dDldfPH8PErr23n43WM0tndz/fJMUmPDAd8lAffNR5p9DFgX/8GusxfNTuG+N46w7UQ9lU2d3Ll+Hu8X1XLnc3tp73LypQvyqWzqJCrUQXRYMEuy43n0y6t8vtaFs08tf3vzqhyWT0/wNBL3lZ0YSU1LYFebDmdlsWwR2SwiB0Rkn4jc5uOYm0Rkt/31nogs8dp3XET2iMhOESkY7T9gKqpu7uQHz+3xTGIFUNHYwXt20fpEbSs5iVFn9DuW51hVRe72AIBFWXE8+uVzuOmcHGLCgj13Tm4HK5qY928vc7S698C0k7VtuIw1cOf5XWW9+nWrqcP9vnDf1b9bWEOIQ1iZl8gXzs+jsqmD/3z5IDc8uIVa+8Lpfo+fqG3D6TJ09bi8kkDYsH/3pXNTcRn46YsHAFi7II0Hbj6Ly+al8tMXD9Dc0U1lc4cn+QzX6pnJfPnC/AH3f/WifG69eOZpveZ4G06bQA/wbWPMPGAV8HURmd/nmGPARcaYxcBPgAf77L/YGLPUGLPijCNWbNxVxqMfnOTGB7d4PiT/89ZRPveHrXR0OzlR20ZO0uALXQwlIy6C9YvSuXxBeq/t581IxhEkpMWF96sO2nGygS6ni4PlvS/yR6utD/01SzMBfNb9qsnvVMeENjp7nGwpqmVZdgKRocHcvCqHPXdfzlNfPZfShna++KcCup0uyhraSYwKpcsejXvWv2/i20/uAjitC/aizDiSo8PYfrKBzPgIpidGEhocxDXLMnEZq7NEdVMnqTHDTyzDccncNK5bnjX0gX40ZBIwxpQbY7bbj5uBA0Bmn2PeM8bU2z9uwVpQXo2RguN1JEaFUt/WxR3P7Aasiay6nC52nGygqrmT3OQzKwkA/Pams7h6aabPfWmxYf2qg9xF9r7JoajG+vCfZY+wrG0N7OKxGj2tnT1c+9t3+fBYnack4DJwpLKF/eVNLMvp3Tnh7NxEfnjlfHYWN/D6gSq6nYbVdr//B948SnNHD+sXZfD51bnEnMZI26Ag4ZK5VjXO6plJnjr6bHtVsJL6NiqbO3pVMU0Vp9U7SERygWXAB4Mc9kXgJa+fDfCqiGwTkQ2nG6DqzRjD1uP1rJmdwtr56Ry1764q7RWWXthjdbWbPsCEcKMlLTa8X3WQu5hf2We1p6LqVlJiwsixB9HUa0lgyvjwWB07TjbwzLYSCqtaCAu2Ljl/311Ot9OwNCu+33M+Mt9ad/dZe+zABXYSeHFvOTlJkdx7w1LuunLBaTe2XjLXet3VXoPJ3G1gxfXtVDZ1jHpJYCIYdhIQkWjgGeB2Y4zPMdgicjFWEvie1+bVxpjlwBVYVUkXDvDcDSJSICIF1dXVw/4DppoTtW3UtHSyIjeRjLhwKps7cbqM54L88t4KAM+oxbGSFmtVB3mP9HSv4tQvOVS3kJ8cRVxECCJQ19Z/BOhk1NbVw/W/e88z8dhEc7K2jZf3lp/Ra2yxp1J4p7CGwuoWz2jev+4oBWBJdv8kkBoTTn5KFG8crAKsrsyRoQ6MgY8uyhhxT5uPzE/j1zcu84ygB0iIDCEq1MH+siY6ul1aEhiIiIRgJYC/GGOeHeCYxcBDwNXGmFr3dmNMmf29CngOWOnr+caYB40xK4wxK1JSUnwdosAz2+eK3ATS48JxugyVTR2etVZrWqy77Oln2CYwlPTYcHpchu0n6/nVpsN0O12crLW68/WtDjpWY0094QgS4iNCpkxJYPuJBradqOetIzVDHxyAfvT8Pr72l+1n1IazpagWEauBt7iunUVZcaTHhlPR1EFKTBgZcb4vuqvyk+ixbzAyEyI8NzXeF/DT5QgSrloyjRDHqcueiJCVEMm2E9bnKvU0Gpsni+H0DhLg98ABY8wvBzhmOvAs8BljzGGv7VEiEuN+DKwF9o5G4FNVwfF64iJCmJkSzbR46wO0v6yJbqchNtyqI02IDOm3puloc/fM+PZTu7j39SO8tLeCLqcLkd5JoL61i/q2bmbYvYwSokKp8zFb6WRUYF9YTtaOzTq3I9HY3t1vNSxfyhvb2XyoCmNOjeodjjcOVnqmeGju6GZPaSNXe61BMTM12tPjbElW/IB39efYc+snRoUSGRrM4qw45qTFsMAe/DiashMjOG7fwKTGaEnAl9XAZ4BL7G6eO0VkvYh8VUS+ah/zQyAJ+G2frqBpwDsisgv4EHjBGPPyaP8RU0nBiTrOykkgKEhIj7UGce0qaQBgzRxrYNj0Ma4KglN9tE/YH57f27OKzk2P7VUd5G4Udn/wEyNDqbNLK80+Jgab6LqdLp7dXkK308U2e8S1+wIzUn/bWcq9rx3pte0vH5zw3L2ejh89v4+b/newJj3L0wXWZGwxYcFsPlTVb39hVXO/CQQLq1rY8Mg2vvv0bowxFJyox2XgEyuyybVLpjNToz1z/S/Njhvw97vn68myR+befdUCnv7auWMy6Cor4VSp+XS6nU4WQzavG2PeAQY988aYLwFf8rG9CFjS/xlqJHqcLk7Utnm6bbqL0juLrSRw6bxUNu4q83zgxpI7CUSHBZOTFOmZdO7c/CQeLm+iqaObX2067GknyE+2PvgJUaEU17VR2tDOhf+1mX/96Dw+vzpvzOMdL6/sq+BbT+6ipbOHHSet/8uZrsHwlw9OsrO4ga9clE94iIPOHid3b9zHFQszTnu1rK3H6yiua6e9y0nEAFMZuFyGJwqKWT0zifTYCF4/WEldaxe7SxpYMyeVA+VNXHHv23zj0ll86YI8frRxP0uz43h1fyU9LsP+8ib2lDay5WgtIQ5h+fQELpydQumHJ8lNijpVEvDRHuCWFhvOvIxYzyj18BAH4SFjM/WC93w/pztOYDLQEcMTSHljBz0u4xkDEB8ZQnhIELvsJLB8egLn5CVy4ayxb1NJiQkjLiKEW87LJSrUwb6yJqJCHZ7i+uaDVfzh3eOAVaR339ElRYWyq7iBQxVNOF2Gn714kJV5iSyYNvBd4UTyQZF1d37PK4do6exhbnoMByuaaeroJjZ84Co6p8uw5hebufXimXzq7N4ju4uqW+jqsbr/njsjiUMVzXQ7zbCqdcAaxBcRYo2ELa6zR+DWtnpGgPf11LZiSurb+cH6ebiM4ZntJaz91VvUtHTy6jcv9Kxb/Zs3jvDSnnKOVLXwjN2T55uXzeaBN49yzyuHKDhezwWzUogIdfCtj8zm6qXTCA9xsG5hOkeqWjh7gFG2bo99+RxCg8d+ejP3lM/RYcFE+3mBF3+Yen/xBOauenHfuYgIGXERnrvt1NgwnvjKueMSS4gjiLe/dzExYcEcrW7hZy8dJC8linS7dOLupfSP76whKTqUYLsxLsEe3+D+WyLDHHzziZ28dNuF/aabGEp7l5Pmzm4+PFbHW4erue2y2WTGRwz9xDG09XgdMfaygwDXLc/kpy8e5GRtGwszB050JfVtFNe1s+NkQ68k0NDW5Wnsf7+olnNnJLHbLnWVNQxv7qZvP7mL4CDh9stme7Ydq/GdBBrbuvmvlw9xdm4CVyxMp6ndmjbZPaf/gfImjla34AgScpMiOVbTygM3n4UxVieBf7p4BiX1bTy1rYSUmDB+ft0iwJpF011qyYiL4KfXLhoy7vGaedP9eZqK3UNBk8CE4q5WyPGq88+IC+dYTSuJUaGEBY/vTIXuO9sZKdHMTotm4bQ4T53qPw5Vkx4b3m/QWmJkKN1Oa5rryFAH/3HNIr7+6Hb+vrtswIFpvuw4Wc/HH3i/15TA+SnRfPWiGaPwl41MQ1sXByuauf2yWTy9rYSObhfnz0wBDnJiiCTgHkjlnivn1HYrwTuCxNPdco+dBCqaOjxzNTldhv/bcoKk6FDOm5FMotf0xxWNHdS2dvH8rjJEwBirdNGX02W4a+Ne6tu6uPuqlYgIcZEhPPKFlaTGhLHu3rc5UtnC0apWchIjeeIr51Ld3OlJJlfYPXc+vzqPbSfr+fl1iydE9Yq7lDoVewaBJoEJ5URdKyEOId3rg+W+8/bnXYyI8PTXziPUEUSXvXpTe7eTNXP6V0u552bfWdzA9MRIrliYzqzUaO7fXMiVi6cNe2K5PaWNOF2G718xl4WZcXz36d0jXkd2tGw9bjUEnzcjmYtmp9Dc0eOpujtR10prZw9hwUEEO4Lo7HEiiKe642iVdbEvqe/dfuBODpfOTeUfh6rp6HZ6Fi93ugzVzZ2kx4WzaX8Fd23cB1jdd9/87hrCgh04XcbTG+u5naXMTo2hsb273wIrnT1O/vnRHby6v5JvXja7V/Wce3BVbpK1Zq67y29ydBjJ0f3fd/OnxfLGt9eM/ESOs5jwEBKjQnt9rqYSXU9gAimuayM7IbJXtYm7cdjfg1xiw0MID3EQExZMhN2A571ojVtilFV6OFrdwvTESIKChFsvmcnhyhZe3V8x7N9XWt9OaHAQX74gn9Uzk5k/LZZ9ZY1DP3EMbT1eR6gjiMVZcSyzG0OjwoJJjg5jT0kjF//iH/znywcB2PDINv75se2e57rn1SltaO81AO9odQuhjiA+sSKbLqeL947WcKSymfn23XeZ3S7wp/dOkBkfwS8+sYSKpg7eOGD16Klr7fKsAWEMLMmOIy85iqLq3kng6W0lvLq/kh9+bD63XTbL5983O81q3zhR28aM1LHvgTae7r1hKbde4vvvnuw0CUwgJ+vaevVkAKt+FQKna5uIeGLpu2gNQIJdz2vMqaktPrZ4GgmRIZ4RogM5UN7EL189hDGGkoZ2MuMjPCWHBdNiOVZj3W37y4fH6liaHd+vF0tOUiQv7a2gqrmT53aUUlLfxpuHq/ngWJ2nm6X7jr/baTzz5YNVQshNjmRVfiIx4cF8+8ld9LiMp4dYWUM7hyubeb+olptX5XDtskxSY8J41h6R656naYU9b9PirHjyU6Ioqm7p1cXzyYIS5qbH8PnVuQP+fbPSrPV8u5wuTzfPyeKCWSmedbSnGk0CE4QxxufsoIFSEvCWGhtOiEN81oEnRZ1KVu6/xREkLMqKZ0/p4NU5z2wr4ddvFFLZ1ElJfbunLhdgwbQ4jLF6wviDMYbDlc2eldy85SSe6iNf09LFv/3VGi/Z0NZNeaPVuHu0usXTqO1dJXS0uoUZKdHEhIfw8OfOprPHqm5bu8CaB6e8oYM/v3+C0OAgPnV2No4g4ZplmWw+WEVdaxc1zVZV0JcuyOfKJdNYOz+N/JRomjp6PCOBD1c2s6u4gY+flTVoP/zZaacukjNSJldJYCrTJDBBNLZ309zR029iuEBoE+jr3PwkLl+Q7rNfd0LUqW6S3oPaFmXGcqSymY5u54Cv6x50daSqmdL69l49gdxdU/3VLlDV3Elbl9PnxXFZTgKpMWH8+YsriQp1sPlQtWe5wf1lTdTZo6rdi5W4G4c7e5ycrGvz3KGenZvII19YyTcumcnc9BiiQh2UNbbz2oFKLpuX6mkMvnZZJj0uwwu7yzwLmsxKi+Y3Ny4jNTacfLux3t2r7KmCYoKDhGuXDd4wP9trJTr3uA818WkSmCDcXSr7JoE5aTH805oZ/eb996dvfmQ29316uc990WHBhDisu03vv2VRZhw9LuPpg+7LCXv6hd0ljdS0dPYqCWTEhZMQGcK+IUoTY8Vdx57n4+L4mVU5vP/9S8mIi2Ct/X/64gX5iFhJy10VdOEsqwG2pL6N+tYu/vDucZwu06vqZUVuIt9aO8fqHhwfwbuFNZQ3dti9kCzzMmKtdojSRk8SSPYqgbkHaxVWtdDR7eSZ7aVcOi+VJB+NvN5yk6IIcQhJUaG9Fl9XE5smgQnC3T2078RwwY4gvrtu7oToigdWm0FCZChBQq87eXfVkXvemfrWLn724gHPpHQul+GEfQ7ePGTNZZPplQREhAXT4thX7p/GYfdddd4A1STuxvxPnzOdzPgIblyZTV5SFPvLGz3TgS/MjCM5OpTiunY++T/v8/OXDpKfHMXKPN+DqjLiwjlcaT33fK/pkQHykiM5XttGTUsXoY4gYiNOdQTMSogkKyGCP285wZMFxdS1dvGFYYzaDg0OYkZKNDOmaN35ZKVdRAPY8ZpW3jpSTXNHD49+cJKw4KAxXydgPCRGhRIaHNRrNGhmfASJUaHsKW3EGMMdz+7mlX2VPFlQzIOfXUFWQgRddn34tpNWV0zvOV8A5mXE8Kf3T/hc5/h0vbC7nOd2lPLQLb4Xw2to6+LV/ZXsKWnkplXTOVbTQnhIEBlDJOOzcxN5945LrHinxbK7pIHY8BDCgoOYFh9BZkIkL+0tp6mjh59dt4gbzs4esJ5+mt0pICshot/NwfTEKN4prCYnMZKk6NBer+EIEr6zdg63P7GT/3jhAEuz4wdMNH394hNLxmUUrxo/mgQC2L/9bS9v29MQL5sezz2fWExk6MT/l83LiO01yAusO/mFmXHsLmnkkfdP8Mq+Sr6wOo83DlZy22M7+MUnrSmo8pKjPHfdfUcH5yVH09XjoqKp44xHDr+yr4LXDlTS2N5NVKiDoppWT534B0W1fOPxHVTaE+W1dvXQ1N5NblLUsMc5AMzPiOWF3eUU15Xw2XNzcAQJWQkR7CpuICkqlOuWZw7aUDvN/hv7lgLA6tP/zPZOiuvbSIruX3Vz1ZJp/O/bRewra+KrF80Y9sRsgw14UxPTxL+iTFItnT1sKarlc+flctuls4iPDBmTGRT94VefWtpvBkqwGofvP1zNXRv3cd6MJO786DxmpUXz/Wf3ePq9f2R+Gg++VURwkPTrEeWeOO9ETasnCbR1WdMeDGc0dW1LJ91OQ3pcOEfc/fbr2ymsbuEbj+1MSOXrAAAcuklEQVTg8Q2riAx18OmHPmB6YiTPfO0s/vTecd48VE1sRAjzMmKG+A29uRuzZ6RE8f0r5gGnRq9+YkX2kDFn2FOJn+cjCeTYjb+7ihs5J7//XX5QkPDz6xbz152lnpW81NSkSSBAvXOkmm6n4YqF6ZOyEc5XQrt2WRZlDR1cPDeVyxek4QgSz13u09tLCHEIF85K4cG3isiID+9X5eO+8B2rbeW8mcl0O11ce/97zEmP4dc3Lhsypu89s4fyxnY23nq+p7G2uL6Ng3Zj9a82HSZIrIVx/vr11cRFhFBS38bGXWXUtnaxftHpNc6vyE1k3YJ0vnHpLM+MnvPSYwkNDuKmc6YP8Wy4aHYK1y3L5JK5qf32uRNie7fT56hegEVZcSzK0jv7qU6TQIB6/UAVseHBnsXZp4KZqdH86lNLe23LTowkJymSE7Vt5CdHMde+2/ZV3ZMRG05ocJCnJ9X/bTnBocpmz+RnQ9lb2khlcwf7yho97Q8l9e0ct3slfXDMmiH07ivnexbtuWh2CkFiLZ7uq2fQYKLDgnngM2f12nbVkmlcMCt5yJ46YI0N+WWf8+WWk3iqgdpXdZBSbtrCE4BcLsPmQ1WsmZPqmX1zKnPPXZOTFElSVChpsWHk+xixGhQk5CRGcrymlYa2Lv77tSOEOITyxg7qWrvo6Hby8t4K7nnloKe7qVtjezcVTR0YA08VlHi2l9S3caymzZ5bP5zsxAg+fU6OZ398ZKhneoy85DMfQBUUJMNKAEOJiwwhPtJKVCmj8Hpq8hrO8pLZIrJZRA6IyD4Ruc3HMSIivxaRQhHZLSLLvfbdIiJH7K9bRvsPmIx2lzZS09Lls5g/FZ3vSQJRiAiPbziXf1k7x+exOUlRnKht44mtxTS2d3OHXde+r6yRuzfu46v/t437Nx/l/s2FvZ53pLLZ8/ivO60pFzLjIyiua+N4TStz0mJ5bMMqHv3Sqn69Y9YtTCfUEcTMAJtKwT3brJYE1GCGc5vZA3zbGDMPWAV8XUTm9znmCmCW/bUB+B2AiCQCdwHnYC0wf5eITJ36jRF640AlQWJVNSg4b0YSMWHBLLGXI8xLjhqwnSQ3KZLjta28ur+SBdNiuX65NQp2d0kjL+4p56OLM7hqyTRe3ltBZ8+p0cnu/vaJUaE0d/SQERfOvIwYtp9soL3bSV5yJHnJUf3mbgL43Hm5bPrWhcRFju26zqfL3S4wUJuAUjCMJGCMKTfGbLcfNwMHgL7jy68GHjGWLUC8iGQAlwObjDF1xph6YBOwblT/gknojUNVnJWTMCkbhEciPjKUD++8jGuGsd5ATnIUnT3W+r6XzUsjPjKUzPgIHv3gJE0dPXxsUQbXLsukqaOHtw/XeJ53uLKZCHvVK7DaJ7ISIj3z6/RdF8FbsCOo1xoPgcIdkyYBNZjTqnAWkVxgGdB3pepMoNjr5xJ720Db1QAqGjvYW9rEJXO12563iFDHsLrI5nldjN1dH+dPi6W0oZ0Qh3D+rGRWz0wmLiKEv+8u8xx7pKqZWWnRntk2Z6XG9JqWYjTq+8fbeTOSBiy9KOU27CQgItHAM8Dtxpi+E7T4+nSaQbb7ev0NIlIgIgXV1dXDDWvS2XzI6g9/6TxtDxgJ98yk0+LCPf3w3d/PyUsiJjyE0OAgrliYzqb9lZ6eQ4crW5iVGsPZuYn2rKaxnhHJocFBntG5E8mq/CQ2f2fNlFw3Vw3fsJKAiIRgJYC/GGOe9XFICZDt9XMWUDbI9n6MMQ8aY1YYY1akpEzduvDXD1SRlRDBLJ2fZUSmxUcQExbM5QvTPSUH9ypZ3g3tN6/KobXLyR/eOU5DWxfVzZ3MTosmOzGSN759EVctyfSUBHLsxW+UmoyGvEUQ65P0e+CAMeaXAxy2EbhVRB7HagRuNMaUi8grwE+9GoPXAt8fhbgnpY5uJ+8W1vCJFYPP664G5ggSnv/n83utF3vBrGT+ac0Mrl+e5dm2MDOOyxek8dDbRRi7cOqeFsJdl+6uRhmsPUCpiW445cTVwGeAPSKy0972A2A6gDHmAeBFYD1QCLQBn7f31YnIT4Ct9vN+bIypG73wJ5ctRbW0dzu1a+gZ6nvRDg9x8N11c/sdd/tls3ll39v892tHOG9GEufOSOq1Py4ihLzkqCk1YE9NPUMmAWPMO/iu2/c+xgBfH2Dfw8DDI4puinnjYBURIQ5W5ScNfbA6Y/MyYvnJNQuJCHFw/QCTtb32rYvQmiA1mWmLUYAwxvD6gSrOn5Xsc0UuNTY+sypn0P1nOiW1UoFO5yQIEIcrWyhtaOdSrQpSSo0jTQIB4t1Ca+DSRXOmbs8opdT40yQQIA5XNpMYFUr6BFkmUik1OWgSCBCHKpuZnRatXUOVUuNKk0AAMMZwpLKFOWmntzKVUkqdKU0CAaCssYOWzh5maRJQSo0zTQIB4HCFNZf9nHRNAkqp8aVJIAAcshc0mZ2qSUApNb40CQSAwxXNpMeGB9yiJEqpyU+TQAA4bM9lr5RS402TgJ91dDu1Z5BSym80CfjZm4er6exxcaGuJ6yU8gNNAn724p5y4iND+k1jrJRS40GTgB91dDt5/UAVl89PJ8Sh/wql1PjTK48fvXW4mpbOHtYvzvB3KEqpKWo4y0s+DHwMqDLGLPSx/1+Am7xebx6QYq8qdhxoBpxAjzFmxWgFPtE5XYb/eauIxKhQztOqIKWUnwynJPBHYN1AO40x9xhjlhpjlmKtH/xmnyUkL7b3awLw8od3j7HtRD3/9rF5WhWklPKbIa8+xpi3gOGuC3wj8NgZRTQF1LR0cs8rh7hsXhrXLM30dzhKqSls1G5BRSQSq8TwjNdmA7wqIttEZMMQz98gIgUiUlBdXT1aYQWkfWVNdPa4+OL5eTp1tFLKr0azHuJK4N0+VUGrjTHLgSuAr4vIhQM92RjzoDFmhTFmRUrK5O4zX1TdAsCM1Cg/R6KUmupGMwncQJ+qIGNMmf29CngOWDmKv2/CKqpuJSYsmJToMH+HopSa4kYlCYhIHHAR8DevbVEiEuN+DKwF9o7G75voimpayE+J0qogpZTfDaeL6GPAGiBZREqAu4AQAGPMA/Zh1wKvGmNavZ6aBjxnX+iCgUeNMS+PXugTV1F1K6vytVuoUsr/hkwCxpgbh3HMH7G6knpvKwKWjDSwyaqtq4fyxg7yk7U9QCnlf9pBfZwVVVuFpfwUnTpaKeV/mgTGWVGNOwloSUAp5X+aBMbZsepWRCBPq4OUUgFAk8A4K6ppYVpcBOEhDn+HopRSmgTGW1F1q1YFKaUChiaBcWSMoai6hRnaKKyUChCaBMZRVXMnrV1OLQkopQKGJoFxdNSeM0gbhZVSgUKTwDjSMQJKqUCjSWAcFVW3Eh4SREZsuL9DUUopQJPAuCqqaSEvOZqgIJ04TikVGDQJjCPtHqqUCjSaBMZJZ4+Tkvo2ZmijsFIqgGgSGCcnattwGW0UVkoFFk0C42TrcWvVzZmpmgSUUoFjyCQgIg+LSJWI+FwVTETWiEijiOy0v37otW+diBwSkUIRuWM0A59Imjq6+dWmIyzNjmd+Rqy/w1FKKY/hlAT+CKwb4pi3jTFL7a8fA4iIA7gfa5H5+cCNIjL/TIKdqO597Qi1rZ38+OoF2jNIKRVQhkwCxpi3gLoRvPZKoNAYU2SM6QIeB64ewetMaC6X4fEPT3LN0kwWZ8X7OxyllOpltNoEzhWRXSLykogssLdlAsVex5TY26aU0oZ2WrucnJ2b6O9QlFKqnyHXGB6G7UCOMaZFRNYDfwVmAb7qPcxALyIiG4ANANOnTx+FsALDkapmAGanaYOwUirwnHFJwBjTZIxpsR+/CISISDLWnX+216FZQNkgr/OgMWaFMWZFSkrKmYYVMA5VWJPGzUqL8XMkSinV3xknARFJFxGxH6+0X7MW2ArMEpE8EQkFbgA2nunvm2iOVDaTFhtGXESIv0NRSql+hqwOEpHHgDVAsoiUAHcBIQDGmAeAjwNfE5EeoB24wRhjgB4RuRV4BXAADxtj9o3JXxHADlc1M1tLAUqpADVkEjDG3DjE/vuA+wbY9yLw4shCm/hcLkNhVQufXpnj71CUUsonHTE8horr2+jodmmjsFIqYGkSGEOHK7VRWCkV2DQJjKED5U0AzNKSgFIqQGkSGEOv7KtgSXY8seHaM0gpFZg0CYyRouoW9pU1ceXiDH+HopRSA9IkMEb+vrscgI9qElBKBTBNAmPAGMPGXWWszE0kIy7C3+EopdSANAmMgZL6dgqrWrhiUbq/Q1FKqUFpEhgDe0obAVg+PcHPkSil1OA0CYyBPaWNhDiEuRk6PkApFdg0CYyBPSWNzEmPISzY4e9QlFJqUJoERpkxhj2ljSzKjPN3KEopNSRNAqOsuK6dxvZuFmXqUpJKqcCnSWCUuRuFtSSglJoINAmMst2lDYQ6gpidrvMFKaUCnyaBUWSM4c1D1SzKitNGYaXUhDBkEhCRh0WkSkT2DrD/JhHZbX+9JyJLvPYdF5E9IrJTRApGM/BAtLukkYMVzVy3PNPfoSil1LAMpyTwR2DdIPuPARcZYxYDPwEe7LP/YmPMUmPMipGFOHE8UVBMeEgQVy6Z5u9QlFJqWIazvORbIpI7yP73vH7cAmSdeVgTT3uXk+d3lrF+YYZOHa2UmjBGu03gi8BLXj8b4FUR2SYiG0b5dwWUrcfraO7s4ZplWhWklJo4hiwJDJeIXIyVBM732rzaGFMmIqnAJhE5aIx5a4DnbwA2AEyfPn20who3JfXtAMxM1V5BSqmJY1RKAiKyGHgIuNoYU+vebowps79XAc8BKwd6DWPMg8aYFcaYFSkpKaMR1rgqbWgjOEhIiw33dyhKKTVsZ5wERGQ68CzwGWPMYa/tUSIS434MrAV89jCaDErr20mPC8cRJP4ORSmlhm3I6iAReQxYAySLSAlwFxACYIx5APghkAT8VkQAeuyeQGnAc/a2YOBRY8zLY/A3BISS+nYy43UBGaXUxDKc3kE3DrH/S8CXfGwvApb0f8bkVNrQzrkzkvwdhlJKnRYdMTwKup0uKps6yNKSgFJqgtEkMAoqGjtwGchKiPR3KEopdVo0CYwCd/fQzAQtCSilJhZNAqOgtMFOAlodpJSaYDQJjIJSuySQEa9jBJRSE4smgVFQ2tBGakyYTh+tlJpwNAmMgtKGdm0PUEpNSJoEzlBJfRs7TzYwI0XnDFJKTTyaBM6Ay2X4zlO7EBFuu3SWv8NRSqnTpkngDDy/u4wtRXX828fmkZ2oYwSUUhOPJoEz8I9D1SRHh/LJFdn+DkUppUZEk8AZ+PBYHSvzErEnyVNKqQlHk8AIldS3UdrQzsrcRH+HopRSI6ZJYIQ+PFYHwDn5OnOoUmri0iQwQh8eqyM2PJg5aTH+DkUppUZMk8AIfXjcag8I0pXElFIT2LCSgIg8LCJVIuJzeUix/FpECkVkt4gs99p3i4gcsb9uGa3A/amxvZui6laW5yT4OxSllDojwy0J/BFYN8j+K4BZ9tcG4HcAIpKItRzlOViLzN8lIhP+yllY1QKgVUFKqQlvWEnAGPMWUDfIIVcDjxjLFiBeRDKAy4FNxpg6Y0w9sInBk8mEcKSyGYDZmgSUUhPcaLUJZALFXj+X2NsG2t6PiGwQkQIRKaiurh6lsMbGkaoWwkOCdP0ApdSEN1pJwFfrqBlke/+NxjxojFlhjFmRkpIySmGNjcOVzcxMjdZGYaXUhDdaSaAE8J47IQsoG2T7hFZY1cKsVK0KUkpNfKOVBDYCn7V7Ca0CGo0x5cArwFoRSbAbhNfa2yas5o5uyhs7mJWmU0crpSa+4OEcJCKPAWuAZBEpwerxEwJgjHkAeBFYDxQCbcDn7X11IvITYKv9Uj82xgzWwBzwjtg9g7QkoJSaDIaVBIwxNw6x3wBfH2Dfw8DDpx9aYCqsdCcBLQkopSY+HTF8Gjq6nby0t5yw4CBdP0ApNSkMqySgoLWzh48/8D4Hypv47ro5OLRnkFJqEtAkMEyv7q/gQHkT9316GR9bPM3f4Sil1KjQ6qBh2rS/ktSYMNYvzPB3KEopNWo0CQxDZ4+TNw9Vc+m8NB0gppSaVDQJDMP7R2tp7XKydn6av0NRSqlRpUlgGDbtryQy1MG5M3QVMaXU5KJJYAjtXU7+vrucS+amEh7i8Hc4Sik1qjQJDOFvO0tpbO/ms+fm+jsUpZQadZoEBmGM4Y/vHWdeRixn5074tXCUUqofTQKD2Hq8noMVzXzuvBxEtFeQUmry0SQwiL/tLCUixMFVS3yug6OUUhOeJoEBOF2GV/ZVcMm8VCJCtUFYKTU5aRIYwIfH6qhp6dIRwkqpSU2TwABe3FNOeEgQF88N7KUulVLqTAwrCYjIOhE5JCKFInKHj/2/EpGd9tdhEWnw2uf02rdxNIMfK8ZYVUEXz0klMlTn2FNKTV5DXuFExAHcD3wEa83grSKy0Riz332MMeabXsf/M7DM6yXajTFLRy/ksVfa0E5Vcyfn6QhhpdQkN5ySwEqg0BhTZIzpAh4Hrh7k+BuBx0YjOH/ZXdIIwOKseD9HopRSY2s4SSATKPb6ucTe1o+I5AB5wBtem8NFpEBEtojINSOOdBztKmkgxCHMzdB1hJVSk9twKrx9jZIyAxx7A/C0McbptW26MaZMRPKBN0RkjzHmaL9fIrIB2AAwffr0YYQ1dnYVNzAvI5awYO0aqpSa3IZTEigBsr1+zgLKBjj2BvpUBRljyuzvRcA/6N1e4H3cg8aYFcaYFSkp/uuR43IZ9pY2sUSrgpRSU8BwksBWYJaI5IlIKNaFvl8vHxGZAyQA73ttSxCRMPtxMrAa2N/3uYGkqKaFls4eFmfF+TsUpZQac0NWBxljekTkVuAVwAE8bIzZJyI/BgqMMe6EcCPwuDHGu6poHvA/IuLCSjg/9+5VFIh2FVuNwkuytSSglJr8htUJ3hjzIvBin20/7PPz3T6e9x6w6AziG1cNbV3c/49CkqPDmJES7e9wlFJqzOmIYVtZQzsbHtlGSV07v71pOQ5dS1gpNQXocFjg0Q9OcvfGfQD84pNLWJmX6OeIlFJqfEz5JNDV4+IXrx5icVYc9964jMz4CH+HpJRS42bKVwe9fqCSutYuvn7JTE0ASqkpZ8ongScKikmPDefCWTpbqFJq6pnSSaCsoZ23Dlfz8bOytCFYKTUlTekk8OvXjxAkwqfOzh76YKWUmoSmbBI4WNHEkwXFfPbcXLITI/0djlJK+cWUTALFdW1875k9xISH8I1LZ/o7HKWU8psp10X03cIavvinrQSJ8F8fX0x8ZKi/Q1JKKb+Zckngl5sOkxwdxhNfOVe7hCqlprwpVR20q7iBbSfq+cLqPE0ASinFFEsCf3j3GNFhwXxiRZa/Q1FKqYAwJZJAj9PFf792mI27yvjkimxiwkP8HZJSSgWESd0mUNnUwfO7yvjLByc5VtPKdcsy+fba2f4OSymlAsakSwLtXU6e2HqSF/aUU3CiHmPgrJwEvrduLusWpvs7PKWUCijDSgIisg64F2tlsYeMMT/vs/9zwD1Aqb3pPmPMQ/a+W4B/tbf/uzHmT6MQt0/NHd184Y9b2Xq8nrnpMdx+6Ww+ujidmakxY/UrlVJqQhsyCYiIA7gf+AjWovNbRWSjj2UinzDG3NrnuYnAXcAKwADb7OfWj0r0Xpo6urn5oQ/YX9bEb25cxpVLpo32r1BKqUlnOA3DK4FCY0yRMaYLeBy4epivfzmwyRhTZ1/4NwHrRhbq4CJDHOQnR/HAzWdpAlBKqWEaTnVQJlDs9XMJcI6P464XkQuBw8A3jTHFAzw3c4SxDirYEcR/37BsLF5aKaUmreGUBHzNsWz6/Pw8kGuMWQy8Brjr/YfzXOtAkQ0iUiAiBdXV1cMISyml1JkaThIoAbznWs4CyrwPMMbUGmM67R//FzhruM/1eo0HjTErjDErUlJ0gRellBoPw0kCW4FZIpInIqHADcBG7wNEJMPrx6uAA/bjV4C1IpIgIgnAWnubUkqpADBkm4AxpkdEbsW6eDuAh40x+0Tkx0CBMWYj8A0RuQroAeqAz9nPrRORn2AlEoAfG2PqxuDvUEopNQJijM8qer9asWKFKSgo8HcYSik1YYjINmPMitN93pSYO0gppZRvmgSUUmoK0ySglFJTWEC2CYhINXBihE9PBmpGMZzRFMixgcZ3JgI5NtD4zkQgxwan4ssxxpx2//qATAJnQkQKRtI4Mh4COTbQ+M5EIMcGGt+ZCOTY4Mzj0+ogpZSawjQJKKXUFDYZk8CD/g5gEIEcG2h8ZyKQYwON70wEcmxwhvFNujYBpZRSwzcZSwJKKaWGadIkARFZJyKHRKRQRO4IgHiyRWSziBwQkX0icpu9/W4RKRWRnfbXej/GeFxE9thxFNjbEkVkk4gcsb8n+CGuOV7nZ6eINInI7f48dyLysIhUicher20+z5VYfm2/F3eLyHI/xXePiBy0Y3hOROLt7bki0u51Hh/wQ2wD/i9F5Pv2uTskIpePZWyDxPeEV2zHRWSnvX28z91A15HRe+8ZYyb8F9bEdkeBfCAU2AXM93NMGcBy+3EM1mI784G7ge/4+5zZcR0Hkvts+y/gDvvxHcB/BsD/tgLI8ee5Ay4ElgN7hzpXwHrgJaz1NFYBH/gpvrVAsP34P73iy/U+zk+x+fxf2p+RXUAYkGd/rh3jHV+f/f8P+KGfzt1A15FRe+9NlpLAmSyBOSaMMeXGmO3242as6bXHZFW1UXY1pxYF+hNwjR9jAbgUOGqMGengwVFhjHkLa4ZcbwOdq6uBR4xlCxDfZ7r1cYnPGPOqMabH/nEL1noe426AczeQq4HHjTGdxphjQCHW53vMDBafiAjwSeCxsYxhIINcR0btvTdZksC4LWM5EiKSCywDPrA33WoX1R72R3WLFwO8KiLbRGSDvS3NGFMO1hsQSPVbdJYb6P0BDJRzBwOfq0B8P34B6w7RLU9EdojImyJygZ9i8vW/DLRzdwFQaYw54rXNL+euz3Vk1N57kyUJDHsZy/EmItHAM8Dtxpgm4HfADGApUI5V1PSX1caY5cAVwNfFWiM6YIi1iNFVwFP2pkA6d4MJqPejiNyJtdbHX+xN5cB0Y8wy4FvAoyISO85hDfS/DKhzB9xI75sQv5w7H9eRAQ/1sW3Q8zdZksCwl7EcTyISgvWP+4sx5lkAY0ylMcZpjHFhLcU5pkXdwRhjyuzvVcBzdiyV7uKj/b3KX/FhJaftxphKCKxzZxvoXAXM+1FEbgE+Btxk7Epju6ql1n68DaveffZ4xjXI/zKQzl0wcB3whHubP86dr+sIo/jemyxJYMglMMebXZf4e+CAMeaXXtu96+euBfb2fe54EJEoEYlxP8ZqRNyLdd5usQ+7BfibP+Kz9boLC5Rz52Wgc7UR+KzdU2MV0Oguuo8nEVkHfA+4yhjT5rU9RUQc9uN8YBZQNM6xDfS/3AjcICJhIpJnx/bheMbm5TLgoDGmxL1hvM/dQNcRRvO9N16t3OPQir4eq+X8KHBnAMRzPlYxbDew0/5aD/wZ2GNv3whk+Cm+fKxeGLuAfe5zBiQBrwNH7O+JfoovEqgF4ry2+e3cYSWjcqAb627riwOdK6wi+f32e3EPsMJP8RVi1Q+7338P2Mdeb//PdwHbgSv9ENuA/0vgTvvcHQKu8Me5s7f/Efhqn2PH+9wNdB0ZtfeejhhWSqkpbLJUBymllBoBTQJKKTWFaRJQSqkpTJOAUkpNYZoElFJqCtMkoJRSU5gmAaWUmsI0CSil1BT2/wHR7xc3djC91gAAAABJRU5ErkJggg==\n", "text/plain": [ "

" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(learn.activation_stats.stats[1][-2].numpy());" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Internal implementation" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

hook[source]

\n", "\n", "> hook(**`m`**:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), **`i`**:`Tensors`, **`o`**:`Tensors`) → `Tuple`\\[`Rank0Tensor`, `Rank0Tensor`\\]\n", "\n", "Take the mean and std of `o`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats.hook)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Callback methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_begin[source]

\n", "\n", "> on_train_begin(**\\*\\*`kwargs`**)\n", "\n", "Initialize stats. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats.on_train_begin)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_batch_end[source]

\n", "\n", "> on_batch_end(**`train`**, **\\*\\*`kwargs`**)\n", "\n", "Take the stored results and puts it in `self.stats` " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats.on_batch_end)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_end[source]

\n", "\n", "> on_train_end(**\\*\\*`kwargs`**)\n", "\n", "Polish the final result. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ActivationStats.on_train_end)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class Hook[source]

\n", "\n", "> Hook(**`m`**:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), **`hook_func`**:`HookFunc`, **`is_forward`**:`bool`=***`True`***, **`detach`**:`bool`=***`True`***)\n", "\n", "Create a hook on `m` with `hook_func`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hook)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Registers and manually deregisters a [PyTorch hook](https://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks). Your `hook_func` will be called automatically when forward/backward (depending on `is_forward`) for your module `m` is run, and the result of that function is placed in `self.stored`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

remove[source]

\n", "\n", "> remove()\n", "\n", "Remove the hook from the model. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hook.remove)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Deregister the hook, if not called already." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class Hooks[source]

\n", "\n", "> Hooks(**`ms`**:`ModuleList`, **`hook_func`**:`HookFunc`, **`is_forward`**:`bool`=***`True`***, **`detach`**:`bool`=***`True`***)\n", "\n", "Create several hooks on the modules in `ms` with `hook_func`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hooks)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Acts as a `Collection` (i.e. `len(hooks)` and `hooks[i]`) and an `Iterator` (i.e. `for hook in hooks`) of a group of hooks, one for each module in `ms`, with the ability to remove all as a group. Use `stored` to get all hook results. `hook_func` and `is_forward` behavior is the same as [`Hook`](/callbacks.hooks.html#Hook). See the source code for [`HookCallback`](/callbacks.hooks.html#HookCallback) for a simple example." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

remove[source]

\n", "\n", "> remove()\n", "\n", "Remove the hooks from the model. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hooks.remove)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Deregister all hooks created by this class, if not previously called." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convenience functions for hooks" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

hook_output[source]

\n", "\n", "> hook_output(**`module`**:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), **`detach`**:`bool`=***`True`***, **`grad`**:`bool`=***`False`***) → [`Hook`](/callbacks.hooks.html#Hook)\n", "\n", "Return a [`Hook`](/callbacks.hooks.html#Hook) that stores activations of `module` in `self.stored` " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(hook_output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Function that creates a [`Hook`](/callbacks.hooks.html#Hook) for `module` that simply stores the output of the layer." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

hook_outputs[source]

\n", "\n", "> hook_outputs(**`modules`**:`ModuleList`, **`detach`**:`bool`=***`True`***, **`grad`**:`bool`=***`False`***) → [`Hooks`](/callbacks.hooks.html#Hooks)\n", "\n", "Return [`Hooks`](/callbacks.hooks.html#Hooks) that store activations of all `modules` in `self.stored` " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(hook_outputs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Function that creates a [`Hook`](/callbacks.hooks.html#Hook) for all passed `modules` that simply stores the output of the layers. For example, the (slightly simplified) source code of [`model_sizes`](/callbacks.hooks.html#model_sizes) is:\n", "\n", "```python\n", "def model_sizes(m, size):\n", " x = m(torch.zeros(1, in_channels(m), *size))\n", " return [o.stored.shape for o in hook_outputs(m)]\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

model_sizes[source]

\n", "\n", "> model_sizes(**`m`**:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), **`size`**:`tuple`=***`(64, 64)`***) → `Tuple`\\[`Sizes`, `Tensor`, [`Hooks`](/callbacks.hooks.html#Hooks)\\]\n", "\n", "Pass a dummy input through the model `m` to get the various sizes of activations. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(model_sizes)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

model_summary[source]

\n", "\n", "> model_summary(**`m`**:[`Learner`](/basic_train.html#Learner), **`n`**:`int`=***`70`***)\n", "\n", "Print a summary of `m` using a output text width of `n` chars " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(model_summary)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

num_features_model[source]

\n", "\n", "> num_features_model(**`m`**:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → `int`\n", "\n", "Return the number of output features for `model`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(num_features_model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It can be useful to get the size of each layer of a model (e.g. for printing a summary, or for generating cross-connections for a [`DynamicUnet`](/vision.models.unet.html#DynamicUnet)), however they depend on the size of the input. This function calculates the layer sizes by passing in a minimal tensor of `size`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

dummy_batch[source]

\n", "\n", "> dummy_batch(**`m`**:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), **`size`**:`tuple`=***`(64, 64)`***) → `Tensor`\n", "\n", "Create a dummy batch to go through `m` with `size`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(dummy_batch)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

dummy_eval[source]

\n", "\n", "> dummy_eval(**`m`**:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), **`size`**:`tuple`=***`(64, 64)`***)\n", "\n", "Pass a [`dummy_batch`](/callbacks.hooks.html#dummy_batch) in evaluation mode in `m` with `size`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(dummy_eval)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class HookCallback[source]

\n", "\n", "> HookCallback(**`learn`**:[`Learner`](/basic_train.html#Learner), **`modules`**:`Sequence`\\[[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)\\]=***`None`***, **`do_remove`**:`bool`=***`True`***) :: [`LearnerCallback`](/basic_train.html#LearnerCallback)\n", "\n", "Callback that can be used to register hooks on `modules`. Implement the corresponding function in `self.hook`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(HookCallback)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For all `modules`, uses a callback to automatically register a method `self.hook` (that you must define in an inherited class) as a hook. This method must have the signature:\n", "\n", "```python\n", "def hook(self, m:Model, input:Tensors, output:Tensors)\n", "```\n", "\n", "If `do_remove` then the hook is automatically deregistered at the end of training. See [`ActivationStats`](/callbacks.hooks.html#ActivationStats) for a simple example of inheriting from this class." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Callback methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_begin[source]

\n", "\n", "> on_train_begin(**\\*\\*`kwargs`**)\n", "\n", "Register the [`Hooks`](/callbacks.hooks.html#Hooks) on `self.modules`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(HookCallback.on_train_begin)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_end[source]

\n", "\n", "> on_train_end(**\\*\\*`kwargs`**)\n", "\n", "Remove the [`Hooks`](/callbacks.hooks.html#Hooks). " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(HookCallback.on_train_end)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Undocumented Methods - Methods moved below this line will intentionally be hidden" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": false }, "outputs": [ { "data": { "text/markdown": [ "

remove[source]

\n", "\n", "> remove()" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(HookCallback.remove)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

hook_fn[source]

\n", "\n", "> hook_fn(**`module`**:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), **`input`**:`Tensors`, **`output`**:`Tensors`)\n", "\n", "Applies `hook_func` to `module`, `input`, `output`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Hook.hook_fn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## New Methods - Please document or move to the undocumented section" ] } ], "metadata": { "jekyll": { "keywords": "fastai", "summary": "Implement callbacks using hooks", "title": "callbacks.hooks" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 2 }