{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#|default_exp conv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Convolutions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#|export\n",
"import torch\n",
"from torch import nn\n",
"\n",
"from torch.utils.data import default_collate\n",
"from typing import Mapping\n",
"\n",
"from miniai.training import *\n",
"from miniai.datasets import *"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pickle,gzip,math,os,time,shutil,torch,matplotlib as mpl, numpy as np\n",
"import pandas as pd,matplotlib.pyplot as plt\n",
"from pathlib import Path\n",
"from torch import tensor\n",
"\n",
"from torch.utils.data import DataLoader\n",
"from typing import Mapping"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mpl.rcParams['image.cmap'] = 'gray'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"path_data = Path('data')\n",
"path_gz = path_data/'mnist.pkl.gz'\n",
"with gzip.open(path_gz, 'rb') as f: ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding='latin-1')\n",
"x_train, y_train, x_valid, y_valid = map(tensor, [x_train, y_train, x_valid, y_valid])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the context of an image, a feature is a visually distinctive attribute. For example, the number 7 is characterized by a horizontal edge near the top of the digit, and a top-right to bottom-left diagonal edge underneath that.\n",
"\n",
"It turns out that finding the edges in an image is a very common task in computer vision, and is surprisingly straightforward. To do it, we use a *convolution*. A convolution requires nothing more than multiplication, and addition."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Understanding the Convolution Equations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To explain the math behind convolutions, fast.ai student Matt Kleinsmith came up with the very clever idea of showing [CNNs from different viewpoints](https://medium.com/impactai/cnns-from-different-viewpoints-fab7f52d159c).\n",
"\n",
"Here's the input:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's our kernel:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since the filter fits in the image four times, we have four results:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x_imgs = x_train.view(-1,28,28)\n",
"xv_imgs = x_valid.view(-1,28,28)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mpl.rcParams['figure.dpi'] = 30"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAGSElEQVR4nO3dP2hUex7G4R0xCBZK0mkhCmJjCrGUoBZqMChWQkDFiFgGLQVba8FGQbCxF5KIigREEPsUaa1iEQWbCMZ/ZLbZvSCb+c46ySTvic9T3pfJHJSPB+6Pc9Jqt9v/AvJs2+wLAFYnTgglTgglTgglTgi1vRpbrZb/lQt91m63W6v9d3dOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCCVOCLV9sy/gb3To0KGO28DAQPnZ48ePl/uDBw/KfWVlpdw30/T0dMdtfHy8/OyPHz/W+3I2nTsnhBInhBInhBInhBInhBInhBInhGq12+3OY6vVefyLHT58uNwnJibK/eLFix23bdvqfy/37t1b7q1Wq9yrv+9kT548Kfdbt26V+9LS0jpezfpqt9ur/qW5c0IocUIocUIocUIocUIocUIoRyk9mJmZKfexsbENupL/tVWPUro5ceJEub97926DruTPOUqBhhEnhBInhBInhBInhBInhBInhPJqzB7Mzs6W+1rOOT99+lTujx8/Lvduj5yt5dWYx44dK/duZ438GXdOCCVOCCVOCCVOCCVOCCVOCCVOCOV5zh5s314fD+/Zs6fnn/3z589yX1xc7Plnr9WuXbvKfX5+vty7vdazMjU1Ve6XLl0q9+/fv/f83f3meU5oGHFCKHFCKHFCKHFCKHFCKHFCKM9z9uDXr1/lvrCwsEFXsrFGR0fLfXBwsG/f/eHDh3JPPsfslTsnhBInhBInhBInhBInhBInhBInhPI8J78ZHx/vuN24caP8bD/fWzs0NFTuS0tLffvufvM8JzSMOCGUOCGUOCGUOCGUOCGUR8a2mG6viLx9+3a5Hzx4sOM2MDDQ0zX9v+bm5jpu3V4ZuhW5c0IocUIocUIocUIocUIocUIocUIo55w92L9/f7lfuXKl3E+dOrWOV/O7kZGRcq8eEVyrbo9tdTtjffHiRcdteXm5p2tqMndOCCVOCCVOCCVOCCVOCCVOCCVOCOXVmKsYHh4u95mZmXLft2/fel7OH2m1Vn3L4j/6ec75/Pnzcr9w4ULfvrvJvBoTGkacEEqcEEqcEEqcEEqcEEqcEMrznD3odpbYbe+nbdvqf29XVlb69t3nzp0r97Nnz5b7y5cv1/NyGs+dE0KJE0KJE0KJE0KJE0KJE0KJE0I551zF/Px8uZ88ebLcL1++XO6vXr3quH379q38bL9dv3694zY5ObmBV4I7J4QSJ4QSJ4QSJ4QSJ4QSJ4Tyakx+s3v37o7b58+f1/Szz58/X+5/6yNjXo0JDSNOCCVOCCVOCCVOCCVOCCVOCOWRMX4zOjq62ZfAf7hzQihxQihxQihxQihxQihxQihxQqgte845MDDQcTtz5kz52devX5f78vJyT9eU4Nq1a+V+//79DboSunHnhFDihFDihFDihFDihFDihFDihFCNPeccGRkp9zt37nTcTp8+XX72wIED5b6wsFDu/TQ0NFTuY2Nj5X7v3r1y37lz5x9f0391O//d7F9v2DTunBBKnBBKnBBKnBBKnBBKnBCqsb8CcG5urtyHh4d7/tkPHz4s9y9fvvT8s9eq2zHQ0aNHy736++7mzZs35d7tz+3p06c9f/dW5lcAQsOIE0KJE0KJE0KJE0KJE0KJE0I559xiWq1Vj8z+8fHjx3J/9uxZx+3mzZvlZz0S1hvnnNAw4oRQ4oRQ4oRQ4oRQ4oRQ4oRQjT3nPHLkSLlPTk523K5evbrOV7N+3r9/X+5fv34t97dv35b7o0ePyn1+fr7cWX/OOaFhxAmhxAmhxAmhxAmhxAmhxAmhGnvO2c2OHTs6bhMTE+Vn7969W+6Dg4PlPjU1Ve6zs7Mdt+np6fKzi4uL5U7zOOeEhhEnhBInhBInhBInhBInhBInhNqy55zQFM45oWHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHECaHKV2MCm8edE0KJE0KJE0KJE0KJE0KJE0L9G+VTMapHtgsFAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"im3 = x_imgs[7]\n",
"show_image(im3);"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"top_edge = tensor([[-1,-1,-1],\n",
" [ 0, 0, 0],\n",
" [ 1, 1, 1]]).float()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We're going to call this our kernel (because that's what fancy computer vision researchers call these)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAADHUlEQVR4nO3ZsW0CQRBA0VuLFhwQuYnriZro6Zpw7CLGDWAJAoS/eC+8nWCSr13p1sxswP/38eoFgPuIFSLEChFihQixQoRYIeL0yPBay38eeLKZWbe+u1khQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFiNMjw+fzebtcLs/aBd7e9Xr988zNChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0Ssmbl7eN/3OY7jievAe9v3fTuOY906c7NChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFiDUz9w+v9bNt2/fz1oG39zUzn7cOHooVeB3PYIgQK0SIFSLEChFihQixQoRYIUKsECFWiPgFV2MhFpDbtPEAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_image(top_edge, noframe=False);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The filter will take any window of size 3×3 in our images, and if we name the pixel values like this:\n",
"\n",
"$$\\begin{matrix} a1 & a2 & a3 \\\\ a4 & a5 & a6 \\\\ a7 & a8 & a9 \\end{matrix}$$\n",
"\n",
"it will return $-a1-a2-a3+a7+a8+a9$."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | \n",
" 0 | \n",
" 1 | \n",
" 2 | \n",
" 3 | \n",
" 4 | \n",
" 5 | \n",
" 6 | \n",
" 7 | \n",
" 8 | \n",
" 9 | \n",
" 10 | \n",
" 11 | \n",
" 12 | \n",
" 13 | \n",
" 14 | \n",
" 15 | \n",
" 16 | \n",
" 17 | \n",
" 18 | \n",
" 19 | \n",
" 20 | \n",
" 21 | \n",
" 22 | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
"
\n",
" \n",
" 1 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
"
\n",
" \n",
" 2 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
"
\n",
" \n",
" 3 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
"
\n",
" \n",
" 4 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
"
\n",
" \n",
" 5 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.15 | \n",
" 0.17 | \n",
" 0.41 | \n",
" 1.00 | \n",
" 0.99 | \n",
" 0.99 | \n",
" 0.99 | \n",
" 0.99 | \n",
" 0.99 | \n",
" 0.68 | \n",
" 0.02 | \n",
" 0.00 | \n",
"
\n",
" \n",
" 6 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.17 | \n",
" 0.54 | \n",
" 0.88 | \n",
" 0.88 | \n",
" 0.98 | \n",
" 0.99 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.62 | \n",
" 0.05 | \n",
"
\n",
" \n",
" 7 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.70 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.99 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.23 | \n",
"
\n",
" \n",
" 8 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.43 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.90 | \n",
" 0.52 | \n",
" 0.52 | \n",
" 0.52 | \n",
" 0.52 | \n",
" 0.74 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.23 | \n",
"
\n",
" \n",
" 9 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.02 | \n",
" 0.11 | \n",
" 0.11 | \n",
" 0.09 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.05 | \n",
" 0.88 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.67 | \n",
" 0.03 | \n",
"
\n",
" \n",
" 10 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.33 | \n",
" 0.95 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.56 | \n",
" 0.00 | \n",
"
\n",
" \n",
" 11 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.34 | \n",
" 0.74 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.05 | \n",
" 0.00 | \n",
"
\n",
" \n",
" 12 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.00 | \n",
" 0.36 | \n",
" 0.83 | \n",
" 0.96 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.98 | \n",
" 0.80 | \n",
" 0.04 | \n",
" 0.00 | \n",
"
\n",
" \n",
"
\n"
],
"text/plain": [
""
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.DataFrame(im3[:13,:23])\n",
"df.style.format(precision=2).set_properties(**{'font-size':'7pt'}).background_gradient('Greys')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor(2.9727)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(im3[3:6,14:17] * top_edge).sum()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor(-2.9570)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(im3[7:10,14:17] * top_edge).sum()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def apply_kernel(row, col, kernel): return (im3[row-1:row+2,col-1:col+2] * kernel).sum()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor(2.9727)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"apply_kernel(4,15,top_edge)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[[(0, 0), (0, 1), (0, 2), (0, 3), (0, 4)],\n",
" [(1, 0), (1, 1), (1, 2), (1, 3), (1, 4)],\n",
" [(2, 0), (2, 1), (2, 2), (2, 3), (2, 4)],\n",
" [(3, 0), (3, 1), (3, 2), (3, 3), (3, 4)],\n",
" [(4, 0), (4, 1), (4, 2), (4, 3), (4, 4)]]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[[(i,j) for j in range(5)] for i in range(5)]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAIjElEQVR4nO3dy09USRzF8eLVAvJoHs1DCdOKINIu1ERd+McZ/yT3ro0uDIlRCAgERWhBIZrmJQo9G5bmd25ywT7I97OcU6nqYXJyk6lbdZvq9XoC4Ke50T8AwJ9RTsAU5QRMUU7AFOUETLVG4bNnz/hfucA5e/r0adOf/jlPTsAU5QRMUU7AFOUETFFOwBTlBExRTsAU5QRMUU7AFOUETFFOwBTlBExRTsAU5QRMUU7AFOUETFFOwBTlBExRTsAU5QRMUU7AFOUETFFOwBTlBEyFl0oju0KhIMdcuXIlzPv7++Uct27dypWPjIzINf6GWq0W5nNzc3IONWZtbS3M3T9/yZMTMEU5AVOUEzBFOQFTlBMwRTkBU5QTMEU5AVO8hHBKvSAwPDwc5pOTk3KNcrkc5gMDA3KO1tb4P1lbW5ucw0F3d3eY3759W86hXiI4OTkJ842NDbmGmuM88eQETFFOwBTlBExRTsAU5QRMUU7AFOUETLHPeWpoaCjMZ2Zmwlwdck5J70Fubm7KOZaWlsJ8eXk5zKvVqlzjb1D7nOrvnVJKnZ2dYa72pre3t+UaBwcHcsx54ckJmKKcgCnKCZiinIApygmYopyAKcoJmGKf89TOzk6YqwuMFxcX5Rr7+/thvrCwIOdQ+5Rfv36Vcyhq/3BwcFDOocY0NTWFeZa/RXt7uxwTaeRZzSx4cgKmKCdginICpignYIpyAqYoJ2CKcgKmKCdgipcQTqmXDD58+JArT0kf3P3165ecQ435/fu3nENpaWkJ8ywXaFcqlTCfnp4O80+fPsk1Dg8P5ZhIT0+PHNPR0ZFrjTx4cgKmKCdginICpignYIpyAqYoJ2CKcgKm2Oc81dXVFebq47rq8HBK+rD1WRxiLhaLcg5FXX5dKpXkHOqSbrWXmmW/9ujoSI6JqI/vNhpPTsAU5QRMUU7AFOUETFFOwBTlBExRTsAU+5yn+vv7w/zRo0dh/uDBA7mG2rsrFApyjtbW+D+Z2qPMcoZxfHw8129IKaUfP36EuTqvmeWsZpa95UjeS6nPG09OwBTlBExRTsAU5QRMUU7AFOUETFFOwBTlBEzxEsIptaGtDlur/Kz09vaG+cTERJjPzMzINbq7u8N8ZWVFzrG9vR3m6uD5WVyOrV4yaG72fjZ5/zrgEqOcgCnKCZiinIApygmYopyAKcoJmGKf81Tei5TL5bJc4/r162GuLrZOKaXOzs5ceZYP9K6urob50tKSnOPt27dhPjs7K+dQRkdHw3xsbCz3Go3EkxMwRTkBU5QTMEU5AVOUEzBFOQFTlBMwRTkBU5fiJYQsB6EnJyfDvFKphLk65JySftEhC3WT+vz8fJjPzc3JNdbX18NcHaROKaWFhYUw//LlS5irlylS0jfT570RvtF4cgKmKCdginICpignYIpyAqYoJ2CKcgKmLsU+ZxbqEPLGxkaYq327lFLa2toK8729PTmHGnNwcBDm6jLnLLJ8Hfvhw4dhfv/+/TA/Pj6Wa6h9Y/dLo5WL/euBfxjlBExRTsAU5QRMUU7AFOUETFFOwNSl2OdUZyBTSun58+dhvrm5GebValWuoT4Iu7u7K+c4OTmRYyJ9fX1yzM2bN8P83r17co4sl2xHarWaHJPl73WR8eQETFFOwBTlBExRTsAU5QRMUU7AFOUETFFOwNSleAkhi58/f4a5eslgeXlZrpHlq9Ln7du3b3KMOsSc5bB1oVDI/Jv+5KJfCH0WeHICpignYIpyAqYoJ2CKcgKmKCdginICpi7FPmd7e7scMz09HealUinM1SXKKemD0t3d3XKOvPt/WQ5rq79Xlo8Rs0+ZH09OwBTlBExRTsAU5QRMUU7AFOUETFFOwBTlBEz9Ey8hqM37s9gQVxvz9Xo99xpnoVgshvnU1FTuNb5//y7HLC4u5l7nsuPJCZiinIApygmYopyAKcoJmKKcgCnKCZhq+D5nZ2enHPPff/+F+czMTJhnucz548ePYf7u3bswPz4+lmsoLS0tckxvb2+Yq0Pjo6Ojcg21j/n582c5h/qKt9LcrJ8bWcZcZP/2vx1wgVFOwBTlBExRTsAU5QRMUU7AFOUETDV8n3NkZESOmZiYCPPh4eEw393dlWuofcrBwUE5R15dXV1yjPodaq90dXVVrvHmzZswf/36tZyjVquF+dWrV8N8bGxMrpHlEu6LjCcnYIpyAqYoJ2CKcgKmKCdginICpignYIpyAqYa/hLC/v6+HDM/P58r7+npkWuMj4+H+ePHj8O8ra1NrqFkObCtDkKrQ+GvXr2Sa7x48SLMFxYW5ByFQiHM79y5E+b/+kHqLPgLAKYoJ2CKcgKmKCdginICpignYIpyAqYavs+pDvamlO1jrZG7d+/KMepA9tbWVphn2Zc7OjoK8+3tbTmH+ijt7OxsmL9//16uoQ5Kl0olOYfaxyyXy2GuDmNfBjw5AVOUEzBFOQFTlBMwRTkBU5QTMEU5AVMN3+c8PDyUY1ZWVsK8Wq2G+fr6ulxjbW0tzNW+m9oHTUnvUarfkFJKOzs7ckykr69PjqlUKmE+NTUl57h27VqYc15T4y8EmKKcgCnKCZiinIApygmYopyAKcoJmKKcgKmGv4SgNquzqNfrYT43NyfnePnyZe7foaiLltvb2+Uc6ovPKldfCU8ppdHR0TDP8juRH09OwBTlBExRTsAU5QRMUU7AFOUETFFOwFTD9zk7OjrkmBs3boR5sVgM84GBAbmG+gDv3t5emA8NDck11CFltUeZUkr9/f1hfhYf8YUHnpyAKcoJmKKcgCnKCZiinIApygmYopyAKcoJmGr4SwhZqNvB1UsGT548kWtkGQP8TTw5AVOUEzBFOQFTlBMwRTkBU5QTMEU5AVNN6kJmAI3BkxMwRTkBU5QTMEU5AVOUEzBFOQFT/wNyIJjq/L44NgAAAABJRU5ErkJggg==\n",
"text/plain": [
"