{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# BinaryNet on CIFAR10\n",
"\n",
" \n",
"\n",
"In this example we demonstrate how to use Larq to build and train BinaryNet on the CIFAR10 dataset to achieve a validation accuracy approximately 83% on laptop hardware.\n",
"On a Nvidia GTX 1050 Ti Max-Q it takes approximately 200 minutes to train. For simplicity, compared to the original papers [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://arxiv.org/abs/1511.00363), and [Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://arxiv.org/abs/1602.02830), we do not impliment learning rate scaling, or image whitening."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pip install larq"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"import larq as lq\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Import CIFAR10 Dataset\n",
"\n",
"We download and normalize the CIFAR10 dataset."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
"170500096/170498071 [==============================] - 38s 0us/step\n"
]
}
],
"source": [
"num_classes = 10\n",
"\n",
"(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()\n",
"\n",
"train_images = train_images.reshape((50000, 32, 32, 3)).astype(\"float32\")\n",
"test_images = test_images.reshape((10000, 32, 32, 3)).astype(\"float32\")\n",
"\n",
"# Normalize pixel values to be between -1 and 1\n",
"train_images, test_images = train_images / 127.5 - 1, test_images / 127.5 - 1\n",
"\n",
"train_labels = tf.keras.utils.to_categorical(train_labels, num_classes)\n",
"test_labels = tf.keras.utils.to_categorical(test_labels, num_classes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Build BinaryNet\n",
"\n",
"Here we build the BinaryNet model layer by layer using the [Keras Sequential API](https://www.tensorflow.org/guide/keras)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# All quantized layers except the first will use the same options\n",
"kwargs = dict(input_quantizer=\"ste_sign\",\n",
" kernel_quantizer=\"ste_sign\",\n",
" kernel_constraint=\"weight_clip\",\n",
" use_bias=False)\n",
"\n",
"model = tf.keras.models.Sequential([\n",
" # In the first layer we only quantize the weights and not the input\n",
" lq.layers.QuantConv2D(128, 3,\n",
" kernel_quantizer=\"ste_sign\",\n",
" kernel_constraint=\"weight_clip\",\n",
" use_bias=False,\n",
" input_shape=(32, 32, 3)),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
"\n",
" lq.layers.QuantConv2D(128, 3, padding=\"same\", **kwargs),\n",
" tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
"\n",
" lq.layers.QuantConv2D(256, 3, padding=\"same\", **kwargs),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
"\n",
" lq.layers.QuantConv2D(256, 3, padding=\"same\", **kwargs),\n",
" tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
"\n",
" lq.layers.QuantConv2D(512, 3, padding=\"same\", **kwargs),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
"\n",
" lq.layers.QuantConv2D(512, 3, padding=\"same\", **kwargs),\n",
" tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
" tf.keras.layers.Flatten(),\n",
"\n",
" lq.layers.QuantDense(1024, **kwargs),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
"\n",
" lq.layers.QuantDense(1024, **kwargs),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
"\n",
" lq.layers.QuantDense(10, **kwargs),\n",
" tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),\n",
" tf.keras.layers.Activation(\"softmax\")\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One can output a summary of the model:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+sequential stats---------------------------------------------------------------------------------------------+\n",
"| Layer Input prec. Outputs # 1-bit # 32-bit Memory 1-bit MACs 32-bit MACs |\n",
"| (bit) x 1 x 1 (kB) |\n",
"+-------------------------------------------------------------------------------------------------------------+\n",
"| quant_conv2d - (-1, 30, 30, 128) 3456 0 0.42 0 3110400 |\n",
"| batch_normalization - (-1, 30, 30, 128) 0 256 1.00 0 0 |\n",
"| quant_conv2d_1 1 (-1, 30, 30, 128) 147456 0 18.00 132710400 0 |\n",
"| max_pooling2d - (-1, 15, 15, 128) 0 0 0 0 0 |\n",
"| batch_normalization_1 - (-1, 15, 15, 128) 0 256 1.00 0 0 |\n",
"| quant_conv2d_2 1 (-1, 15, 15, 256) 294912 0 36.00 66355200 0 |\n",
"| batch_normalization_2 - (-1, 15, 15, 256) 0 512 2.00 0 0 |\n",
"| quant_conv2d_3 1 (-1, 15, 15, 256) 589824 0 72.00 132710400 0 |\n",
"| max_pooling2d_1 - (-1, 7, 7, 256) 0 0 0 0 0 |\n",
"| batch_normalization_3 - (-1, 7, 7, 256) 0 512 2.00 0 0 |\n",
"| quant_conv2d_4 1 (-1, 7, 7, 512) 1179648 0 144.00 57802752 0 |\n",
"| batch_normalization_4 - (-1, 7, 7, 512) 0 1024 4.00 0 0 |\n",
"| quant_conv2d_5 1 (-1, 7, 7, 512) 2359296 0 288.00 115605504 0 |\n",
"| max_pooling2d_2 - (-1, 3, 3, 512) 0 0 0 0 0 |\n",
"| batch_normalization_5 - (-1, 3, 3, 512) 0 1024 4.00 0 0 |\n",
"| flatten - (-1, 4608) 0 0 0 0 0 |\n",
"| quant_dense 1 (-1, 1024) 4718592 0 576.00 4718592 0 |\n",
"| batch_normalization_6 - (-1, 1024) 0 2048 8.00 0 0 |\n",
"| quant_dense_1 1 (-1, 1024) 1048576 0 128.00 1048576 0 |\n",
"| batch_normalization_7 - (-1, 1024) 0 2048 8.00 0 0 |\n",
"| quant_dense_2 1 (-1, 10) 10240 0 1.25 10240 0 |\n",
"| batch_normalization_8 - (-1, 10) 0 20 0.08 0 0 |\n",
"| activation - (-1, 10) 0 0 0 ? ? |\n",
"+-------------------------------------------------------------------------------------------------------------+\n",
"| Total 10352000 7700 1293.75 510961664 3110400 |\n",
"+-------------------------------------------------------------------------------------------------------------+\n",
"+sequential summary---------------------------+\n",
"| Total params 10.4 M |\n",
"| Trainable params 10.4 M |\n",
"| Non-trainable params 7.7 k |\n",
"| Model size 1.26 MiB |\n",
"| Model size (8-bit FP weights) 1.24 MiB |\n",
"| Float-32 Equivalent 39.52 MiB |\n",
"| Compression Ratio of Memory 0.03 |\n",
"| Number of MACs 514 M |\n",
"| Ratio of MACs that are binarized 0.9939 |\n",
"+---------------------------------------------+\n"
]
}
],
"source": [
"lq.models.summary(model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Training\n",
"\n",
"Compile the model and train the model"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"model.compile(\n",
" tf.keras.optimizers.Adam(lr=0.01, decay=0.0001),\n",
" loss=\"categorical_crossentropy\",\n",
" metrics=[\"accuracy\"],\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"outputExpanded": true,
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train on 50000 samples, validate on 10000 samples\n",
"Epoch 1/100\n",
"50000/50000 [==============================] - 131s 3ms/step - loss: 1.5733 - acc: 0.4533 - val_loss: 1.6368 - val_acc: 0.4244\n",
"Epoch 2/100\n",
"50000/50000 [==============================] - 125s 3ms/step - loss: 1.1485 - acc: 0.6387 - val_loss: 1.8497 - val_acc: 0.3764\n",
"Epoch 3/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.9641 - acc: 0.7207 - val_loss: 1.5696 - val_acc: 0.4794\n",
"Epoch 4/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.8452 - acc: 0.7728 - val_loss: 1.5765 - val_acc: 0.4669\n",
"Epoch 5/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.7553 - acc: 0.8114 - val_loss: 1.0653 - val_acc: 0.6928\n",
"Epoch 6/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.6841 - acc: 0.8447 - val_loss: 1.0944 - val_acc: 0.6880\n",
"Epoch 7/100\n",
"50000/50000 [==============================] - 125s 3ms/step - loss: 0.6356 - acc: 0.8685 - val_loss: 0.9909 - val_acc: 0.7317\n",
"Epoch 8/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.5907 - acc: 0.8910 - val_loss: 0.9453 - val_acc: 0.7446\n",
"Epoch 9/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.5610 - acc: 0.9043 - val_loss: 0.9441 - val_acc: 0.7460\n",
"Epoch 10/100\n",
"50000/50000 [==============================] - 125s 3ms/step - loss: 0.5295 - acc: 0.9201 - val_loss: 0.8892 - val_acc: 0.7679\n",
"Epoch 11/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.5100 - acc: 0.9309 - val_loss: 0.8808 - val_acc: 0.7818\n",
"Epoch 12/100\n",
"50000/50000 [==============================] - 126s 3ms/step - loss: 0.4926 - acc: 0.9397 - val_loss: 0.8404 - val_acc: 0.7894\n",
"Epoch 13/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.4807 - acc: 0.9470 - val_loss: 0.8600 - val_acc: 0.7928\n",
"Epoch 14/100\n",
"50000/50000 [==============================] - 126s 3ms/step - loss: 0.4661 - acc: 0.9529 - val_loss: 0.9046 - val_acc: 0.7732\n",
"Epoch 15/100\n",
"50000/50000 [==============================] - 125s 3ms/step - loss: 0.4588 - acc: 0.9571 - val_loss: 0.8505 - val_acc: 0.7965\n",
"Epoch 16/100\n",
"50000/50000 [==============================] - 126s 3ms/step - loss: 0.4558 - acc: 0.9593 - val_loss: 0.8748 - val_acc: 0.7859\n",
"Epoch 17/100\n",
"50000/50000 [==============================] - 126s 3ms/step - loss: 0.4434 - acc: 0.9649 - val_loss: 0.9109 - val_acc: 0.7656\n",
"Epoch 18/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.4449 - acc: 0.9643 - val_loss: 0.8532 - val_acc: 0.7971\n",
"Epoch 19/100\n",
"50000/50000 [==============================] - 126s 3ms/step - loss: 0.4349 - acc: 0.9701 - val_loss: 0.8677 - val_acc: 0.7951\n",
"Epoch 20/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.4351 - acc: 0.9698 - val_loss: 0.9145 - val_acc: 0.7740\n",
"Epoch 21/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.4268 - acc: 0.9740 - val_loss: 0.8308 - val_acc: 0.8065\n",
"Epoch 22/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.4243 - acc: 0.9741 - val_loss: 0.8229 - val_acc: 0.8075\n",
"Epoch 23/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.4201 - acc: 0.9764 - val_loss: 0.8411 - val_acc: 0.8062\n",
"Epoch 24/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.4190 - acc: 0.9769 - val_loss: 0.8649 - val_acc: 0.7951\n",
"Epoch 25/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.4139 - acc: 0.9787 - val_loss: 0.8257 - val_acc: 0.8071\n",
"Epoch 26/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.4154 - acc: 0.9779 - val_loss: 0.8041 - val_acc: 0.8205\n",
"Epoch 27/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.4128 - acc: 0.9798 - val_loss: 0.8296 - val_acc: 0.8115\n",
"Epoch 28/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.4121 - acc: 0.9798 - val_loss: 0.8241 - val_acc: 0.8074\n",
"Epoch 29/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.4093 - acc: 0.9807 - val_loss: 0.8575 - val_acc: 0.7913\n",
"Epoch 30/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.4048 - acc: 0.9826 - val_loss: 0.8118 - val_acc: 0.8166\n",
"Epoch 31/100\n",
"50000/50000 [==============================] - 126s 3ms/step - loss: 0.4041 - acc: 0.9837 - val_loss: 0.8375 - val_acc: 0.8082\n",
"Epoch 32/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.4045 - acc: 0.9831 - val_loss: 0.8604 - val_acc: 0.8091\n",
"Epoch 33/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.4047 - acc: 0.9823 - val_loss: 0.8797 - val_acc: 0.7931\n",
"Epoch 34/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.4023 - acc: 0.9842 - val_loss: 0.8694 - val_acc: 0.8020\n",
"Epoch 35/100\n",
"50000/50000 [==============================] - 125s 3ms/step - loss: 0.3995 - acc: 0.9858 - val_loss: 0.8161 - val_acc: 0.8186\n",
"Epoch 36/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3976 - acc: 0.9859 - val_loss: 0.8495 - val_acc: 0.7988\n",
"Epoch 37/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.4021 - acc: 0.9847 - val_loss: 0.8542 - val_acc: 0.8062\n",
"Epoch 38/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.3939 - acc: 0.9869 - val_loss: 0.8347 - val_acc: 0.8122\n",
"Epoch 39/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.3955 - acc: 0.9856 - val_loss: 0.8521 - val_acc: 0.7993\n",
"Epoch 40/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.3907 - acc: 0.9885 - val_loss: 0.9023 - val_acc: 0.7992\n",
"Epoch 41/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3911 - acc: 0.9873 - val_loss: 0.8597 - val_acc: 0.8010\n",
"Epoch 42/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.3917 - acc: 0.9885 - val_loss: 0.8968 - val_acc: 0.7936\n",
"Epoch 43/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.3931 - acc: 0.9874 - val_loss: 0.8318 - val_acc: 0.8169\n",
"Epoch 44/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3897 - acc: 0.9893 - val_loss: 0.8811 - val_acc: 0.7988\n",
"Epoch 45/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3876 - acc: 0.9888 - val_loss: 0.8453 - val_acc: 0.8094\n",
"Epoch 46/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3876 - acc: 0.9889 - val_loss: 0.8195 - val_acc: 0.8179\n",
"Epoch 47/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3891 - acc: 0.9890 - val_loss: 0.8373 - val_acc: 0.8137\n",
"Epoch 48/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3902 - acc: 0.9888 - val_loss: 0.8457 - val_acc: 0.8120\n",
"Epoch 49/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3864 - acc: 0.9903 - val_loss: 0.9012 - val_acc: 0.7907\n",
"Epoch 50/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3859 - acc: 0.9903 - val_loss: 0.8291 - val_acc: 0.8053\n",
"Epoch 51/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3830 - acc: 0.9915 - val_loss: 0.8494 - val_acc: 0.8139\n",
"Epoch 52/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3828 - acc: 0.9907 - val_loss: 0.8447 - val_acc: 0.8135\n",
"Epoch 53/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3823 - acc: 0.9910 - val_loss: 0.8539 - val_acc: 0.8120\n",
"Epoch 54/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3832 - acc: 0.9905 - val_loss: 0.8592 - val_acc: 0.8098\n",
"Epoch 55/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3823 - acc: 0.9908 - val_loss: 0.8585 - val_acc: 0.8087\n",
"Epoch 56/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3817 - acc: 0.9911 - val_loss: 0.8840 - val_acc: 0.7889\n",
"Epoch 57/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3827 - acc: 0.9914 - val_loss: 0.8205 - val_acc: 0.8250\n",
"Epoch 58/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3818 - acc: 0.9912 - val_loss: 0.8571 - val_acc: 0.8051\n",
"Epoch 59/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3811 - acc: 0.9919 - val_loss: 0.8155 - val_acc: 0.8254\n",
"Epoch 60/100\n",
"50000/50000 [==============================] - 125s 3ms/step - loss: 0.3803 - acc: 0.9919 - val_loss: 0.8617 - val_acc: 0.8040\n",
"Epoch 61/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.3793 - acc: 0.9926 - val_loss: 0.8212 - val_acc: 0.8192\n",
"Epoch 62/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.3825 - acc: 0.9912 - val_loss: 0.8139 - val_acc: 0.8277\n",
"Epoch 63/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.3784 - acc: 0.9923 - val_loss: 0.8304 - val_acc: 0.8121\n",
"Epoch 64/100\n",
"50000/50000 [==============================] - 125s 2ms/step - loss: 0.3809 - acc: 0.9918 - val_loss: 0.7961 - val_acc: 0.8289\n",
"Epoch 65/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3750 - acc: 0.9930 - val_loss: 0.8676 - val_acc: 0.8110\n",
"Epoch 66/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3789 - acc: 0.9928 - val_loss: 0.8308 - val_acc: 0.8148\n",
"Epoch 67/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3783 - acc: 0.9929 - val_loss: 0.8595 - val_acc: 0.8097\n",
"Epoch 68/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3758 - acc: 0.9935 - val_loss: 0.8359 - val_acc: 0.8065\n",
"Epoch 69/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3784 - acc: 0.9927 - val_loss: 0.8189 - val_acc: 0.8255\n",
"Epoch 70/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3786 - acc: 0.9924 - val_loss: 0.8754 - val_acc: 0.8001\n",
"Epoch 71/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3749 - acc: 0.9936 - val_loss: 0.8188 - val_acc: 0.8262\n",
"Epoch 72/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3758 - acc: 0.9932 - val_loss: 0.8540 - val_acc: 0.8169\n",
"Epoch 73/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3740 - acc: 0.9934 - val_loss: 0.8127 - val_acc: 0.8258\n",
"Epoch 74/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3749 - acc: 0.9932 - val_loss: 0.8662 - val_acc: 0.8018\n",
"Epoch 75/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3721 - acc: 0.9941 - val_loss: 0.8359 - val_acc: 0.8213\n",
"Epoch 76/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3746 - acc: 0.9937 - val_loss: 0.8462 - val_acc: 0.8178\n",
"Epoch 77/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3741 - acc: 0.9936 - val_loss: 0.8983 - val_acc: 0.7972\n",
"Epoch 78/100\n",
"50000/50000 [==============================] - 122s 2ms/step - loss: 0.3751 - acc: 0.9933 - val_loss: 0.8525 - val_acc: 0.8173\n",
"Epoch 79/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.3762 - acc: 0.9931 - val_loss: 0.8190 - val_acc: 0.8201\n",
"Epoch 80/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3737 - acc: 0.9940 - val_loss: 0.8441 - val_acc: 0.8196\n",
"Epoch 81/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3729 - acc: 0.9935 - val_loss: 0.8151 - val_acc: 0.8267\n",
"Epoch 82/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3735 - acc: 0.9938 - val_loss: 0.8405 - val_acc: 0.8163\n",
"Epoch 83/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3723 - acc: 0.9939 - val_loss: 0.8225 - val_acc: 0.8243\n",
"Epoch 84/100\n",
"50000/50000 [==============================] - 123s 2ms/step - loss: 0.3738 - acc: 0.9938 - val_loss: 0.8413 - val_acc: 0.8115\n",
"Epoch 85/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.3714 - acc: 0.9947 - val_loss: 0.9080 - val_acc: 0.7932\n",
"Epoch 86/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.3744 - acc: 0.9942 - val_loss: 0.8467 - val_acc: 0.8135\n",
"Epoch 87/100\n",
"50000/50000 [==============================] - 124s 2ms/step - loss: 0.3705 - acc: 0.9948 - val_loss: 0.8491 - val_acc: 0.8163\n",
"Epoch 88/100\n",
"50000/50000 [==============================] - 128s 3ms/step - loss: 0.3733 - acc: 0.9944 - val_loss: 0.8005 - val_acc: 0.8214\n",
"Epoch 89/100\n",
"50000/50000 [==============================] - 134s 3ms/step - loss: 0.3693 - acc: 0.9949 - val_loss: 0.7791 - val_acc: 0.8321\n",
"Epoch 90/100\n",
"50000/50000 [==============================] - 135s 3ms/step - loss: 0.3724 - acc: 0.9942 - val_loss: 0.8458 - val_acc: 0.8124\n",
"Epoch 91/100\n",
"50000/50000 [==============================] - 128s 3ms/step - loss: 0.3732 - acc: 0.9947 - val_loss: 0.8315 - val_acc: 0.8164\n",
"Epoch 92/100\n",
"50000/50000 [==============================] - 127s 3ms/step - loss: 0.3699 - acc: 0.9950 - val_loss: 0.8140 - val_acc: 0.8226\n",
"Epoch 93/100\n",
"50000/50000 [==============================] - 131s 3ms/step - loss: 0.3694 - acc: 0.9950 - val_loss: 0.8342 - val_acc: 0.8210\n",
"Epoch 94/100\n",
"50000/50000 [==============================] - 134s 3ms/step - loss: 0.3698 - acc: 0.9946 - val_loss: 0.8938 - val_acc: 0.8019\n",
"Epoch 95/100\n",
"50000/50000 [==============================] - 133s 3ms/step - loss: 0.3698 - acc: 0.9946 - val_loss: 0.8771 - val_acc: 0.8066\n",
"Epoch 96/100\n",
"50000/50000 [==============================] - 164s 3ms/step - loss: 0.3712 - acc: 0.9946 - val_loss: 0.8396 - val_acc: 0.8211\n",
"Epoch 97/100\n",
"50000/50000 [==============================] - 155s 3ms/step - loss: 0.3689 - acc: 0.9949 - val_loss: 0.8728 - val_acc: 0.8112\n",
"Epoch 98/100\n",
"50000/50000 [==============================] - 133s 3ms/step - loss: 0.3663 - acc: 0.9953 - val_loss: 0.9615 - val_acc: 0.7902\n",
"Epoch 99/100\n",
"50000/50000 [==============================] - 133s 3ms/step - loss: 0.3714 - acc: 0.9944 - val_loss: 0.8414 - val_acc: 0.8188\n",
"Epoch 100/100\n",
"50000/50000 [==============================] - 138s 3ms/step - loss: 0.3682 - acc: 0.9956 - val_loss: 0.8055 - val_acc: 0.8266\n"
]
}
],
"source": [
"trained_model = model.fit(\n",
" train_images, \n",
" train_labels,\n",
" batch_size=50, \n",
" epochs=100,\n",
" validation_data=(test_images, test_labels),\n",
" shuffle=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Output\n",
"\n",
"We can now plot the final validation accuracy and loss:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.9956000019311905\n",
"0.8320999944210052\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VfX9+PHX+47sMMMegooILlDEWeusuLVW3KutqK1VO2y1/VZbW3+1rbVqtY4qaq17o8WF4iqCoOBgCSIjzBDIHne9f398TpKbkHGB3Nwk9/18PPJI7jnnnvM599yc92edz0dUFWOMMQbAl+oEGGOM6TwsKBhjjKlnQcEYY0w9CwrGGGPqWVAwxhhTz4KCMcaYehYUTFoRkUdE5I8JbrtSRI5NdpqM6UwsKBhjjKlnQcGYLkhEAqlOg+meLCiYTsertrlORD4XkUoReUhEBojIayJSLiIzRKR33PanishCESkRkXdFZEzcuvEi8qn3vqeBrCbHOllEFnjvnSUi+yaYxpNEZL6IlInIGhH5XZP1h3v7K/HWX+ItzxaRv4nIKhEpFZEPvWVHikhhM5/Dsd7fvxOR50TkPyJSBlwiIhNF5CPvGOtF5G4RyYh7/14i8paIbBGRjSLyaxEZKCJVItI3brv9RaRIRIKJnLvp3iwomM7qTOA4YA/gFOA14NdAP9z39moAEdkDeBK41ls3HXhFRDK8G+RLwGNAH+BZb7947x0PTAUuB/oC9wPTRCQzgfRVAhcBvYCTgCtF5HRvv7t46f2Hl6ZxwALvfbcBBwCHemn6JRBL8DM5DXjOO+bjQBT4KVAAHAIcA/zIS0M+MAN4HRgM7A68raobgHeByXH7vRB4SlXDCabDdGMWFExn9Q9V3aiqa4EPgDmqOl9Va4AXgfHedmcD/1XVt7yb2m1ANu6mezAQBO5Q1bCqPgfMjTvGFOB+VZ2jqlFVfRSo9d7XKlV9V1W/UNWYqn6OC0zf9lafB8xQ1Se94xar6gIR8QHfB65R1bXeMWepam2Cn8lHqvqSd8xqVf1EVWerakRVV+KCWl0aTgY2qOrfVLVGVctVdY637lHgAgAR8QPn4gKnMRYUTKe1Me7v6mZe53l/DwZW1a1Q1RiwBhjirVurjUd9XBX39y7Az73qlxIRKQGGee9rlYgcJCIzvWqXUuAKXI4dbx9fN/O2Alz1VXPrErGmSRr2EJFXRWSDV6X0/xJIA8DLwFgRGYkrjZWq6sc7mCbTzVhQMF3dOtzNHQAREdwNcS2wHhjiLaszPO7vNcAtqtor7idHVZ9M4LhPANOAYaraE7gPqDvOGmC3Zt6zGahpYV0lkBN3Hn5c1VO8pkMa3wssAUapag9c9Vp8GnZtLuFeaesZXGnhQqyUYOJYUDBd3TPASSJyjNdQ+nNcFdAs4CMgAlwtIkER+S4wMe69/wKu8HL9IiK5XgNyfgLHzQe2qGqNiEzEVRnVeRw4VkQmi0hARPqKyDivFDMVuF1EBouIX0QO8dowvgKyvOMHgf8D2mrbyAfKgAoR2RO4Mm7dq8AgEblWRDJFJF9EDopb/2/gEuBULCiYOBYUTJemqktxOd5/4HLipwCnqGpIVUPAd3E3vy249ocX4t47D7gMuBvYCiz3tk3Ej4CbRaQcuBEXnOr2uxo4ERegtuAamffzVv8C+ALXtrEF+DPgU9VSb58P4ko5lUCj3kjN+AUuGJXjAtzTcWkox1UNnQJsAJYBR8Wt/x+ugftTVY2vUjNpTmySHWPSk4i8Azyhqg+mOi2m87CgYEwaEpEDgbdwbSLlqU6P6Tys+siYNCMij+KeYbjWAoJpykoKxhhj6llJwRhjTL0uN6hWQUGBjhgxItXJMMaYLuWTTz7ZrKpNn33ZRpcLCiNGjGDevHmpToYxxnQpIpJQ12OrPjLGGFPPgoIxxph6FhSMMcbU63JtCs0Jh8MUFhZSU1OT6qQkVVZWFkOHDiUYtLlQjDHJkbSgICJTcWO6b1LVvZtZL8CduDFiqoBLVPXTHTlWYWEh+fn5jBgxgsYDYnYfqkpxcTGFhYWMHDky1ckxxnRTyaw+egSY1Mr6E4BR3s8U3DDAO6Smpoa+fft224AAICL07du325eGjDGplbSgoKrv40aBbMlpwL/VmQ30EpFBO3q87hwQ6qTDORpjUiuVbQpDaDyTVKG3bH3TDUVkCq40wfDhw5uuNsZ0MZFoDBHB7+u4jE5tJMrmihBbK0NkBnzkZAbICfqpDkcpqwlTWRvB7/ORFfSRFfATjsaoDEWpqo2QEfDRMztIz5wgKFTURqiojVAVilIbiRGKxIjG6qbaFoJ+ITvoJyvDT4bfhwgIggjEVFF1v0Pee2ujMapDUSprI9RGYvTNzaB/j0x652RQVF7Lmq3VFG6t4pg9B7DP0J5J/Zy6REOzqj4APAAwYcKETjdYU0lJCU888QQ/+tGPtut9J554Ik888QS9evVKUspMR6oJR1lfWkMkGqNPbga9cjLw+wRVJRxVFCUz4G/2vZFojOqwu8GEozHCESUci+GPu3GWVocpqQpTWh0mporPKzmW14QpqXbLe+cE2aVvLiMLcglHY2worWFDWQ2xmJKTESA3009FbZQ1W6pYs7WKytoIGQE/mQEffhFiqkRVqY3EKPP2WRuO0Ts3SN/cTHrlBN3NLuhHBNaWVFO4pZoNZTUE/EJOhp+sgJ+YKpGYEokqfp+7SQZ8PkqrwxRV1LKlMgRAht/dhDMCfgI+d64ZAR9Bf91vHxl+HxkBH+FojKLyWjZXhKgKRdz5ZPgJBnzUhmPURqKEIjFiCoq78fp9gl8Exd3IuzIRKMjL7NZBYS1u2sQ6Q71lXU5JSQn//Oc/twkKkUiEQKDlj3j69OnJTlra2FxRS+HWaob0yqYgLwMRIRSJsa6kmq1VIfr3yGJAfiZ+n7CutIYv15bydVEFGX4fORkBsjNcTWos1jgnF1Pw+8Dv8xHwCetLa1hRVME3myupqI2g3g1oS2WYzRW1jdIk4m56tZFY/bIMv4/cTD+ZAT+haIxaLxBEYjuf1wn4JOH9iMDAHln0yAoSirrcasQLQiJCZsBHj+wgfXIzCPp9lFSFWLyhjNKqMNXhKDXhKIrbx7A+OYwf3otITKkOuXU+EQJ+IeATojEXFCOxGLv0zWHCiN4U5GXiE6E6HKU6FCEcUyLRGJGoCyahSIxQ1AXI2kiM8poIQb8wemA+h+dlkp0RoCbsctbhaIzMgJ/MoAsiPqE+YNYFJ4A+ORn0y8+kV04G4WiMqpDL6edk+MnLDJKXFSAWUy9NUYIBH3mZfrKDAcLRmAu8VSEQIT8zQF5mgOwMP1lBH5kBv5cBcN+HSFSpCUepCrtA5cYddd8rEalPY0bAV/+Tk+EnNyNA0O+juLKWTeW1bKkIUZCfybDe2Qzpnd1ipqI9pTIoTAOuEpGngINwk4dvU3XUFVx//fV8/fXXjBs3jmAwSFZWFr1792bJkiV89dVXnH766axZs4aamhquueYapkyZAjQM2VFRUcEJJ5zA4YcfzqxZsxgyZAgvv/wy2dnZKT6zjlEbibJycxVfF1VQUROpz632yAoypHc2Q3tlU1wZYt6qrXy6aisVtRGG98lhl745lNdEmLF4IwvWlFA34G920E9+VoCiilriBwH2CeRkBHY6x9gvP5NdC3LZpW9OfZXA+GEurUN6ZRMM+Nji5YZrIzEyAz4yg+6fuaI2QkVNhNpIlMyAv/6GkB30kx10N7YMv7u5Bfxezt0LVD2ygvTKCdIzO1h/AwLIywrQKztIToafsuoIK4srWVlcSYbfx8CeWQzsmUXA56Mq5Ko8cjICDO6VtVM3GPUCp68Dq3/SycCeWeyVomMns0vqk8CRQIGIFAI3AUEAVb0PmI7rjroc1yX10vY47u9fWciidWXtsat6Ywf34KZTWr5Et956K19++SULFizg3Xff5aSTTuLLL7+s7zo6depU+vTpQ3V1NQceeCBnnnkmffv2bbSPZcuW8eSTT/Kvf/2LyZMn8/zzz3PBBRe063kkUygS46uN5SxeX8b60hrWl9ZQXFFLVtBPj+wA+VlB+uRk0Dcvgx5ZQVYWV/LF2lK+XFvKyuIqognmcPvlZ9I7J8gHy4qoCbsc+L5De3LtMXswZlA+60trWL2lirLqMIN7ZTOsTw69soNsKq9lfWk1pdVhRvXPY68hPdljQD4xdbnbqlAUweXeRNzNzufVA8dUvRxsjIL8THpkdd7nRHrmBNkvpxf7DWuuSrKtKZ8TJ97nZLqfpAUFVT23jfUK/DhZx0+liRMnNnqW4K677uLFF18EYM2aNSxbtmyboDBy5EjGjRsHwAEHHMDKlSs7LL3xaiOu6NwjK4jPJ8RiysriSr5cV0bh1qr67UKRGJsratlcHmJtSTVLN5QTijZUkxTkZdA3N5PaSJSymghl1eFtqjYG9shi7yE9OXGfQezeP4/d+uXRKyeITwSfCCXVIdaVVLN2azU5GQEOHNGHYX2yEXH19JvKaxGB/vlZO3XOnfkmb0xH6xINzdujtRx9R8nNza3/+91332XGjBl89NFH5OTkcOSRRzb7rEFmZkMuzu/3U11dnbT0lVaH+XTVVgpLqimvCVNeE6FwazVL1pexYnMl0ZgiAj2zg4QjrgdGc3rnBCnIy2RgzywuPXwE+wzpyV6DezZbNaGqlNdGKK4IUVIVYmjvHPrlt55zHdgziz0H9mh2nYgwoMfOBQNjzLa6XVBIhfz8fMrLm5/VsLS0lN69e5OTk8OSJUuYPXt2h6VrzZYqnpq7mqLyWmIK0ZiydEM5izeUNaprD/jcDXbMoB4cv9dAeudmUFoVYktViIDPx9jBPdhrcA9GFuTWN+AFfELAn/hjLiJCj6yglyvPbXN7Y0xqWFBoB3379uWwww5j7733Jjs7mwEDBtSvmzRpEvfddx9jxoxh9OjRHHzwwe1+/GhMeX9ZEVW1UTICPqKxGC98upYZizciIvTPz6y/mY8oyOHaY/bgwJG92b1fHvlZQbKCPnswzhgDdME5midMmKBNJ9lZvHgxY8aMSVGKOlbTc/109VZuenkhX6wtbbRdr5wg500czoWH7MKgnunRi8kY0zIR+URVJ7S1nZUUupCK2ghbKkNc+9R8cjMDlFSF+e8X6xnQI5M7zh7HmEE9CEVihGMxxgzsQXZG8vs0G2O6FwsKXUAo4p6UdU+XRpm/ppSKmgihaIzLv70rVx89itxMu5TGmJ1nd5JOKKZKVW3EjbvijYcCMKBHFoGeWbx33f4pTqExpruyoNCJRKIxtlSGKK4MEfb6/GcF/fTOzaBfXiYZAR9brEHYGJNEFhQ6gWhM2VRew+aKEKpKXmaAwb2yycv04/fZjKnGmI5jQSGFVJWymjDrSmoIR2P09gbsygpaA7ExJjUsG9oO6kZJ3R7RWIzVW6q45c9/I1RTzW798hjWJ8cCgjEmpSwotIPtDQrVoSjLN1VQVh3hqYfvY3Cez3oPGWM6BbsTtYP4obOPO+44+vfvzzPPPENtbS1nnHEGv//976msrGTy5MmsWr2G2nCEK6+9jmhVCRvWr+foo4+moKCAmTNnpvpUjDFprvsFhdeuhw1ftO8+B+4DJ9za4ur4obPffPNNnnvuOT7++GNUlVNPPZX333+foqIi+vQbwJ/uf5zczAA9/WEK+vTmnrvuZObMmRQUFLRvmo0xZgd0v6CQYm+++SZvvvkm48ePB6CiooKvvvqKPcdP5J23Z5Db42bO/d7p7HbEESlOqTHGbKv7BYVWcvQdQVW54YYbuPzyy+tfry2pZktliDfe+4j5/3uHG3/7W4455hhuvPHGlKbVGGOasobmdhA/dPbxxx/P1KlTqaioAGDBkhUsW7WWaEUxuw3qw4UXXsh1113Hp59+us17jTEm1bpfSSEF4ofOPuGEEzjvvPM45JBDiMaUYFYO9zwwlY0rC7l48hn4fD6CwSD33nsvAFOmTGHSpEkMHjzYGpqNMSlnQ2cnSWl1mNXFleRnBd0E7+00PEVnPFdjTOeX6NDZVn2UBOFojMKtVWRl+BnWp/0CgjHGJJsFhSTYUFpDTGFY7xz8PgsIxpiuo9sEhc5SDVZRG2FrVYh+eRntPmRFZzlHY0z31S2CQlZWFsXFxSm/acZUWVdSTYbfR//8rHbdt6pSXFxMVlb77tcYY+IltfeRiEwC7gT8wIOqemuT9bsAU4F+wBbgAlUt3N7jDB06lMLCQoqKitoh1TuuvCZMaXWEvnkZLC1p/4HtsrKyGDp0aLvv1xhj6iQtKIiIH7gHOA4oBOaKyDRVXRS32W3Av1X1URE5GvgTcOH2HisYDDJy5Mj2SPYOW7axnIvu/pDDdy/gwYv3TmlajDFmRyWz+mgisFxVV6hqCHgKOK3JNmOBd7y/ZzazvkuoCUf5yZPzyc0I8P++u0+qk2OMMTssmUFhCLAm7nWhtyzeZ8B3vb/PAPJFpG/THYnIFBGZJyLzUl1F1Jw/TV/Mkg3l3HbWfu3elmCMMR0p1Q3NvwC+LSLzgW8Da4Fo041U9QFVnaCqE/r169fRaWzVjEUbefSjVfzg8JEctWf/VCfHGGN2SjIbmtcCw+JeD/WW1VPVdXglBRHJA85U1ZIkpqld1YSjXP/CF+w1uAe/nDQ61ckxxpidlsySwlxglIiMFJEM4BxgWvwGIlIgInVpuAHXE6nLeOWzdWyuqOU3J40hM2DTaBpjur6kBQVVjQBXAW8Ai4FnVHWhiNwsIqd6mx0JLBWRr4ABwC3JSk97U1UembWS0QPyOWTXbZpBjDGmS0rqcwqqOh2Y3mTZjXF/Pwc8l8w0JMsnq7aycF0Zt5yxt41tZIzpNlLd0NxlPTJrJT2yApwxvmmHKmOM6bosKOyADaU1vPblBs4+cBg5GTYlhTGm+7CgsAMen7OKmCoXHjwi1Ukxxph2ZUFhO9VGojwxZzXH7Nmf4X1zUp0cY4xpVxYUttMbCzdSXBniwkNGpDopxhjT7iwobKfHZ69iWJ9svrV7QaqTYowx7c6CwnZYvqmcOd9s4byJu+CzGdWMMd2QBYXt8MScNQT9wlkTbE4DY0z3ZEEhQTXhKM99sobj9xpIQV5mqpNjjDFJYUEhQa9+vp6ymgjnH7RLqpNijDFJY0EhQY/PWcWu/XI5eNc+qU6KMcYkjQWFBKzcXMn81SWce+BwG+fIGNOtWVBIwMylmwA4fq+BKU6JMcYklwWFBMxcWsSu/XLtCWZjTLdnQaENVaEIs1cUc/Rom2rTmLRU/DXUlqc6FR3GgkIbPvq6mFAkZvMvm+6hshiqt6Y6FV3HivfgnwfD8z9MdUo6jAWFNryzZBO5GX4mjOid6qSYdKPa/PLKzS2vAyhbB4+eCnPub7w8XA0PHAkPHQ/RyI6lp2zdtstjMVj6GoRrtl23cSHEott/rEStng0f3ePS0N7Wfw5PnQ/ig69eh3UL2v8YiQpXw6s/hS3fJP1QFhRaoaq8u7SIw3YvsDmYTccq/ARuHQ4bvmi8fP3ncNsesOCJ5t+3aTE8eBx88x68+X+weXnDull3Q+lq2LwUFjy+/WmadRfcsS9sWdF4+edPw5PnwFPnQqjKLVOFt/8A9x7q3tfeYlF47y/w8Anwxq/htV82DpSLX4XHz4Ky9Tu2/60r4fHvQVZPmPKe+/3+X1vevmITFM6DmtIdO15rStbA1ONh3sOw8sP2338TFhRasWxTBWtLqtO36mj1bJh9X3JyYaZ1i6dBbZm78cX74G+gUfjw79tel28+cKWAWATOfx4C2fDfn3k5/PXw4e0w5hQYeiC8e2vDDTwRNWXeMcPbBqTPnnA3zRXvuhtxTSn89+fwwW0QzIV5UxunNRZ1OfBnL4Wv3oBouPVjV2+FL5+HD++AuQ/CZ0/DY6fDzFtg7zPhoCth7r/g3T+5c33/Nnj6fFj2pqv22Z5SkSosfAkeOQUitXDhC9B/TzjoCljyqiv5xIvFYM4DcNd4ePAYF8hvHwtv/rb146yZC09fAKVrW99u1SxXutvyDZz3NOx/YeLnsoNs2rBWzFziuqIeObpfilOSAqVrXe6veits+BxOuQv8bXxdYjFYPQuGHwK+7SxZhavhkZNhr9Ph0J/seLpbE6py/9h7n9ly+raugtdvgKEHwAGXQk6KHlZc8S4gLjhsWgz9x7hc/6KXYcDesPFLd9MbPcltv2WFuyH3Gg4XPOd+H/NbmP4L+OI5+PptFyy+80d3bR85ET6+Hw7/aWLp+fh+913ouzsseBKOvMF9hiVrXDA68gbouxu8MMWVJmpK4LBrYeA+8PwPYMU7sPuxbl8LX3TXISMPFr4AOQVw0m2w1xmNj7nsLfjgdlgzxwXCeMEcOO0eGHe+ex0qh/f+DMtnwNpPYJ/JsMuh8Oq18N6tcPT/tX5+0bD7zGfeAuvmQ8FoOOsR6DfarT/oCldN9f5tcNbDbtmGL13QXTMHdjsaDrjENUqv/MCVjkafCLscsu2xQpXuMylZ5a7tJf+F/Ga6u29Z4aoBe4+Ac5+EglGtn0M7saDQiplLN7HnwHwG9cxOdVI6ViwKL17uckoTp8DHD7hc65kPQaCVcZ/mPwavXA17ngzf/RdkbEcX3ll3w9p5ULTE/aMn42b80T0w84/uBjD+/G3Xr/+sIae79L8ul77v2TDqOzBgLPQaAb42CtfffOBKWBk57saV0xcK9oA+u0IgI7F0Vm1xaTnoCvj03+7GeOa/4H93uM//vGfgoePgo7tdUFB19c2+AFz0EvQY7PYz4fsuVz/95+6cDv+pu8H0HgGjjocP/g77Xwxbv4FPH4NwlbuRDxjbOD01pTDrH7DHCbDvZHjuUlc9tdvRruoIdcv7jHTpe+nHcNzNcNg1EAm5m/68h11QiMVcNUy/MXD5e/D1O/DOH+G1X7n9B7PcMWvLXS4/qyd862fuGvQf4wJ7bTlk94bcvg1pPPlOl87Fr8AxN8LhPwMR9516/zaXUdn9mG0/64UvuSD19UyoLYWew+D0e911j8845PSBiZe50srwQ1ywXvmBS8fp98F+57jjgbtud+4H7/4/uPiVbY854/dQshqO+4Mrsf37NBcYcpsMx//F8y6QX/QS9Oy4QTgtKLSgvCbMvJVbueyIXVOdlI73vzvdF/7Uu11xtc9u8Pqv3Jf3yBtg5BEN/wDx5j/m/kmW/BcePQXOfQryEihlla51VRtDD4TCuTD7Xjj6N9tuV/SVqzsOZsPkx9ouucSLRuATL4f37p9caaHuBgQuh/nMxS79U94DjcGc+9xN79NH3TYZeXDKnbDP95o/xtwHYfp17r1Nid/lsocdCMMOhsHjXW4+q8e2237zHqCw93fBH3Q3//0vgs+ecrnRnkPcjeet37rGz02LXS73xNsaAgK4m9rJf4d/HQV5A+BbP29Yd+xNcO9hcPcEqCp2VU3+IHz+jLshHnm9u8mDux41pXDUDS4HndUL5j8Oux7l0jT80IZtx5wCo09qCJ6BDBeAZ93tqrDWzHaB/3tTXQAZfYILnv8+FT57EiZc6t43b6orbVzwgiu11cnMh/wB235m/gCc9SiUFkLvuPHJTvgrrP0UXrgMfjSn8fdx+dvw7MWQNxDGnuoCzx7Ht5zxOeQq13j/2nXu2h37OxdUm2ZgMnLg8GtdW8fKD2HE4Q3rVn7oSl0HXQGHXQ2Dx7mMyL9Pgx+8CRm5DdsufMEFoA4MCACirfVi6IQmTJig8+bNS/px3l68kR88Oo8nLjuIQ3frwhPqVG911Q7l66B8g/vS9R4BvUe6G0jTm3vhJzD1Oy63f9YjDes/exreuMHdQPrt6XKB485reF/RUrhnoque6D3S5fKye7sbWMVGd1M58yEYddy2aXz+h7BoGlz1sWscXfE+/PQLl0sEV7X0wd9cLs2fAeFKOOKXzQeOlix+1dUz15V8jv8THPIjt27pa65+t98YOP9Z6DGo4X2hSti0xFXXfPyA+zyv+czdQOuowjt/cGncYxJ89wG3LFwN5euheDls/so1Eq+Z4252dTJ7uGB49n8aSlavXANfvgC//MZ93nfu64JKtBaunu9uSDWlcPternqicJ6rWrj09eZLMoumQY8hjW+u4Oq9V/3PXcd9znIlxP/d4W58kRoYOtHd5N//q8sInOM1Tv/3Fy4DcM7j8J8z4dR/uKDVkuKv4R/7w1G/cbnyWAR+NLshJ67qAldNKVw1z5Xk7tjHlVguejmx69uaTUtcg/fEy+CEPzcsf/gkV0VzzYLWS8Dxvnnf9bLa/ZjWq0jD1a600HcUXPpftyxU6QIxClfOaggAX70BT0xu/J3ctAT+eZALagdN2e5Tbo6IfKKqE9raLqklBRGZBNwJ+IEHVfXWJuuHA48CvbxtrlfV6clMU6JmrygmI+Bj/+FduCtqJAT/OMDdWJoz6nj3j113g6sshmcvcTmnU+5oHDD2OxvGnuYa/ObcBy9dCbn9YZRXTzz/P676Yt+zIa+/Kw6/9Vu37+GHuJvhqz+DH89pXK20ejZ88SwccZ0LVkdc56oAPn7A/V20FJ65yOUu9z0bvnMLzLjJ3ah2ORR2Oyqxz2LeQ5A/2P3jbV7mGkHHXwCbFrlzHrgPXDRt25x7Rq67mQ49wNX7PjHZ3dj2ndywzfTrXEPn/hfDSbc3lGCye7kAM2T/hm1jMShe5noVla11aZn/mKsmOvgKt82Kd2HEt9x+8ge4/X58P+x3rgsI4ALm/hfB7HvAF3QlmJaqtsae2vzy7/xh22XH3exysQseh4Uvu2sIroRYZ/z57nxf+hEEstz3ojV9d4OR33bVONFaV7UYf0MVcVVbz1zkqmUqN0PlJvjW1Nb3m6j+e7prPW8qHPJj9xmungOrPnTfh0QDArjgmIhgtqvCev1XrkRSttZVA279xv1vxJcI9jjelbY+uscFLn8QFr0ESMvXLplUNSk/uJv818CuQAbwGTC2yTYPAFd6f48FVra13wMOOEA7wsl3faCT75vVIcdKmuXvqN7UQ/WDv6uu/1y1fJNq8deqy99WffsPbt1LP1aNxVQjYdVHTla9uZ9q4bzW9xuuUb1zvOqd49zfkZDqX3ZXfeIYPxUKAAAeO0lEQVTclt+z4n13vLf/2LCsplz13sNVb9tTtbaiYfl/zlK9dYTqJ/9W/eMg1b/sprrsrYb1tRWqd090y8vWNz5OZbHqivdUv57pzktVdfNyd+x3/+xeF37iXj9/meqfhrlzqShq69NUjUZV/3Gg6r2HNex7yWtuX69d37Bse009wX0G4RrV4hVuf7Pvb1hfuk71sTPdtYu3dZXqLUNUZ966Y8dNRNEy1VUfNV4Wi6nec4hL57PfT2w/X77gtr9rf9VoZNv10Yi7Dvcernr73qr/OnbHP8/mlBS67/aLV7rXj09237H47117C1Wr3jZa9aae7tzvO0L1qzeb33bJdLfNZ0+713dPVJ16YrsmB5inCdy7k1lSmAgsV9UVACLyFHAasCg+JgF1WbOeQDNPxnS80uowC9eV8pOjO6a1f4dVbXE5xpaKsV+94XJyE6fE5c77uUbP3Y52L9//q6sPrix2RePT74UhBzS/vzqBTDjxL67qYNY/YMBeLmfXXONtnZHfclUU/7vDNcpl93a57o1felUncTmnI66Dh46FaVe5+vezHm5cV56R66q2HjgK7jvcNWSKz1U/lBU2bLfvOXDy7S6H6As0VHEM2R/Gnu7aC/IGwoUvbtvI1xyfz+U0X7na1fsPGud6t/TfC479ffPtLIk44jrXxXLB44C3j12PbFjfY5DrUdRUr+Hwi6WNP7v2VrA7sHvjZSIu5/3GDa70kojRJ7mG5gMva/776vO7KslXrnavT7ptxz/P5vQc4nLhs//pqn6+et1VZyXzswtmwaRbXSn6oMvd+bd0TqOOd+01/7vL9S4rWuLaiFIgaW0KIvI9YJKq/tB7fSFwkKpeFbfNIOBNoDeQCxyrqp80s68pwBSA4cOHH7Bq1aqkpLlOytsTlr8NM37nGuNa6oa2cRHcd5hrpBu0n6uXPvynrsoCXD3tXePcF+38Z5rfh6prgPviWff6oCsa17m25ekLYNkMGOQ90PSzxY3r2psq3wD/mOAa1yo3u/d8byqMOXnbbV//tSuCH3l9y/tcNsNVvWjM/QRzXIAauI9rsH73VtdjpWydq2Y665GG9275xjUEHvUbGLh34uccroE79nYBIW+Aaxy97G3XcLyjVOHBY11gHbiv6xL504Xte1Nsb5GQ6zm0x/Htl85IrauHzymAKz5o//OvLHb7D1e579ZPv3SZk87i08dcRmj4oa5B/mdLmm9U30Gdok0hAecCj6jq30TkEOAxEdlbtXH3DVV9AFfVxIQJE5LeMj7nmy1k+JPYnqDqGgfrbsYn/LnxP8B7f3HPBjx2huuREJ9LrrPIa4Db7xzXA2XWXa5xsO6mvnmZeyrz0KtbToeI6+tdU+b+/s4ft+88jv+TuzGvmeN6ZrQWEMDVyR/1a5fDzMhzud+W6mgn/b+2jz/q2IY2jaZ2P8YFyud/6Bp2D2wydk2fka7v9/YKZsHEy13XVnD1xjsTEMB99t/+pSs5lax2XXI7c0AA16uo7hmJdttnJlw63ZVuk3H+uX3h0Ktc77MJ3+9cAQFcO9U7f3TP+oz4VrsGhO2RzKCwFhgW93qotyzeD4BJAKr6kYhkAQXApiSmq02zVxQzbngvsoLtPLSFqmtQ/PB2d8MWn8vh7nF8Qx/qDV+4XMK4C1xj03/OdP8oTb/AS6fDsIPgpL+51y9e6fb97V+5LnLL3nDLR32n9TQFMlsuSbSl1zD49nXwzi0wPsEnLSdOcc88jD7BlXCSafdj4IoPofBj2OWw9tvvgT9wPY16DXefd3sY9R1XStjweeOqo3TTJ8ldwA/9ietldfCVyT3Ojghkus4GM37nuiOnSDKHuZgLjBKRkSKSAZwDTGuyzWrgGAARGQNkAUVJTFObymrCfLm2lIN37dv2xtsjXAMvX+XqTHP7w2n/hF8sgx5DXcmgrhpv7kOuz/jxf3Q9g4qXw5PnuuJ6nZI17uYx+oSGZYf+xBWL5z7kXn/1hqub7BUfl5Pg8J+5Lpr990xse3/AVQklOyDU6TnEPSnbnjnPnD6uBHfxK42fddgZIu7Zgd4jYLdmHrIy7SMj13VlTtWT6m2ZOMU9/7Dv2SlLQtKCgqpGgKuAN4DFwDOqulBEbhaRun5WPwcuE5HPgCeBSzRZjRwJmrdyCzGlfedirhvQasF/XP/677/uGmVzC9xDLmtmu4fFakrdw0N7n+lKBrse6R4gW/0RfP5Uw/6+et39Hn1iw7IBY2H341zXxfKNbsyUPY5vv3NoiUjyA09nNGjf9i/e736sC7C57ZwhMV1HRq5rG0xmA3gbkjognqpOV9U9VHU3Vb3FW3ajqk7z/l6kqoep6n6qOk5V30xmehIxe0U7tyeEqlzbwJYV7gnfo3/TuPfF+AtdD5j3/uIeEAtXuuqJOvtOdtUKH97RMATx0unuoZimjdCHXQ2VRfDiFDdWzB7tXOdrjOn2bJTUJmavKGbcsHZsT5jxO/ew0tmPNa7uqRPMcqWFlR+4BrDB+zd+2EnEDU+w5WvXuFxT5sbXaW5fI77lqmVWvOvG3Gmra6kxxjRhQSFOeX17QjtVHX39jjfOyZWtNx7ufzHk9oPqLdv2kgE31EDf3V0D9fK33PDF8VVHdUQaehvtftz2j1RqjEl7FhTiLFhTQkzhwJHtEBSqt7rRIgtGuwbE1mTkuP7yBaOb73Xg87t6xg1fwNs3u1LAsInN72vs6a67Xd2QCcYYsx0sKMRZtK4MgL0H99z5nb3+a/cw0ncfcA/KtGXCpW5AuJa23WeyG9Rs60rXVtBSKcAfcCNj7mzfeWNMWrKgEGfR+jIG9cyid26C4963ZOsq11vooCvc07vtIZDRUDXUXHuCMca0g1Q/0dypLF5fxphBzYxvv70+fgCQ9n9A5sAfun73o09q3/0aY4zHSgqemnCUr4sqGbuzQaG23D1ZvNfp7T85hj/gGp3bmv3LGGN2kN1dPMs2VhCN6c6XFOY/7oZxOPhH7ZMwY4zpQBYUPIvXu0bmMYPyd3wnsaibgGboRBja5mCExhjT6VhQ8CxaX0ZOhp9d+u7E4+Vfve5mVjrESgnGmK4poYZmEXkBeAh4remw1t3FovVljB6Yj9+3nQOnrZ7tJqcpXu6eNO45DPY8JTmJNMaYJEu099E/gUuBu0TkWeBhVV2avGR1LFVl8foyTt2vmXkLWrNqFjxykhv+uscQNxftt37WMEevMcZ0MQndvVR1BjBDRHriJsaZISJrgH8B/1HVcBLTmHRrS6opr4lsXyNzZTE89wM31PEP3+68Q/EaY8x2SLhNQUT6ApcAPwTmA3cC+wNvJSVlHajuSeaEg0IsBi9eDlWb3RSPFhCMMd1Eom0KLwKjgceAU1R1vbfqaRGZl6zEdZTF68sRgT0HJtjzaNZdbmC6E2/ruMlijDGmAyRa+X2Xqs5sbkUiE0F3dovXlzGiby65mQl8HLUVbh7VMac0P6KpMcZ0YYlWH40VkV51L0Skt4h0m36Xi9aXJf58wrpP3dDV+1/c+SdXN8aY7ZRoULhMVUvqXqjqVuCy5CSpY5XXhFm9pSrx4S0K57rfNoGNMaYbSjQo+EUassUi4gd2cijRzmHphnJgOxqZ18x1U2Fa47IxphtKNCi8jmtUPkZEjgGe9JZ1ed9srgRgt355bW+s6koKLU1wY4wxXVyiDc2/Ai4H6saCfgt4MCkp6mDrSmoAGNQrq+2Nt6503VBtXCNjTDeV6MNrMeBe76dbWVdSTb/8TDIDCcxnXNeeMPTA5CbKGGNSJKHqIxEZJSLPicgiEVlR95PA+yaJyFIRWS4i1zez/u8issD7+UpESprbTzKtK61mcK8EpssEFxSCudB/bHITZYwxKZJo9dHDwE3A34GjcOMgtRpQvMboe4DjgEJgrohMU9VFdduo6k/jtv8J0OETC68tqU78obU1H8OQ/VueH9kYY7q4RBuas1X1bUBUdZWq/g5oa07IicByVV2hqiHgKeC0VrY/F9eA3WFUlXUl1QzumUBJIVQFG7+0RmZjTLeWaEmhVkR8wDIRuQpYC7TVXWcIsCbudSFwUHMbisguwEjgnQTT0y62VoWpCccSqz5avwBiEWtPMMZ0a4mWFK4BcoCrgQOAC4CL2zEd5wDPqWq0uZUiMkVE5onIvKKionY76LqSaoDEgkL9Q2vW88gY0321GRS8toGzVbVCVQtV9VJVPVNVZ7fx1rXAsLjXQ71lzTmHVqqOVPUBVZ2gqhP69evXVpITttYLCkOaCwqhSvjP9+D9v0JNmQsKvUdCXvsd3xhjOps2q49UNSoih+/AvucCo0RkJC4YnAOc13QjEdkT6A18tAPH2CkNJYVmnlEoXu5GQl3+Fsy6282/PPqEDk6hMcZ0rETbFOaLyDTgWaCybqGqvtDSG1Q14rU/vAH4gamqulBEbgbmqeo0b9NzgKdUVXfoDHbCupJqMgM++uQ2M2JHqMr9PuYmWDPHzb+865EdmTxjjOlwiQaFLKAYODpumQItBgUAVZ0OTG+y7MYmr3+XYBra3bqSGob0ykaaG+007MW+XQ51U2yWb4RcqzoyxnRviT7RfGmyE5IKa0uqGdK7hUbmsKtaIpjjfucP6JhEGWNMCiU689rDuJJBI6r6/XZPUQdaV1LNUaP7N7+yrvooI7fjEmSMMSmWaPXRq3F/ZwFnAOvaPzkdpzYSZVN5bcvdUeuqj+pKCsYYkwYSrT56Pv61iDwJfJiUFHWQjaW1QAs9jyCupGBBwRiTPhJ9eK2pUUAL9S5dQ6vPKACEvaAQtOojY0z6SLRNoZzGbQobcHMsdFltPs0cqgR/BvgTrWEzxpiuL9HqowSHEe066oLCwJ4tVB+Fq6w9wRiTdhKdT+EMEekZ97qXiJyevGQl39qSagryMskKtjAMdqjKeh4ZY9JOom0KN6lqad0LVS3Bza/QZa0tqWZIa1NwhiutpGCMSTuJBoXmtuvSle3rStqYcS1UBcEEZ2QzxphuItGgME9EbheR3byf24FPkpmwZHKT69S0HhTCVn1kjEk/iQaFnwAh4GncDGo1wI+TlahkK6kKUx2Oth0UrPrIGJNmEu19VAlcn+S0dJiGZxRaaVMIVUGPwR2UImOM6RwS7X30loj0invdW0TeSF6ykiuhGdfClfbgmjEm7SRafVTg9TgCQFW30oWfaC6qcENc9MvPbHmjUJUNcWGMSTuJBoWYiAyveyEiI2hm1NSuorI2AkB+VrDljaxNwRiThhLtVvob4EMReQ8Q4FvAlKSlKskqaqMA5LT04FosZr2PjDFpKdGG5tdFZAIuEMwHXgKqk5mwZKqoiZCb4cfna2bGNYBIkwl2jDEmTSQ6IN4PgWuAocAC4GDgIxpPz9llVNZGyMtq5dRtgh1jTJpKtE3hGuBAYJWqHgWMB0paf0vnVRGKkJvZSlCon2DHnmg2xqSXRINCjarWAIhIpqouAUYnL1nJVVETIa+1oFBXUrDqI2NMmkm0obnQe07hJeAtEdkKrEpespKrsraNoBD22hSs+sgYk2YSbWg+w/vzdyIyE+gJvJ60VCVZRW2EYbmtlAJsfmZjTJra7uk4VfU9VZ2mqqG2thWRSSKyVESWi0izw2SIyGQRWSQiC0Xkie1Nz46oaKukYPMzG2PSVNKGvxYRP3APcBxQCMwVkWmquihum1HADcBhqrpVRDrkKem2q4/qSgpWfWSMSS/bXVLYDhOB5aq6witVPAWc1mSby4B7vGEzUNVNSUxPvcraaOu9j6ykYIxJU8kMCkOANXGvC71l8fYA9hCR/4nIbBGZlMT0AFAbiRKKxsjLbOFpZnBPM4OVFIwxaSfVs6cFgFHAkbgH494XkX3iB98DEJEpeMNqDB8+vOk+tkulN8RF6yUFr/rISgrGmDSTzJLCWmBY3Ouh3rJ4hcA0VQ2r6jfAV7gg0YiqPqCqE1R1Qr9+/XYqUXWD4bXeplAFCARamW/BGGO6oWQGhbnAKBEZKSIZwDnAtCbbvIQrJSAiBbjqpBVJTBMViQSFkDcYnrQwNpIxxnRTSQsKqhoBrgLeABYDz6jqQhG5WURO9TZ7AygWkUXATOA6VS1OVpqgISi0OcyFDXFhjElDSW1TUNXpwPQmy26M+1uBn3k/HaK+pNDWgHj24JoxJg0ls/qoU0q4TcGGuDDGpKG0CwoVNYlUH1lJwRiTntIvKCTc0GxBwRiTftIuKNQ/p5DR2sNrlfbgmjEmLaVdUKioDZMV9BHwt3LqVlIwxqSpNAwK0darjsDaFIwxaSvtgkKbI6RCw8NrxhiTZtIuKFTUtjE/M3htClZSMMakHwsKTUVCEItYm4IxJi2lXVCorI2Qn9AEOxYUjDHpJ+2CQpslhXC1+21BwRiThtIuKFS2FRTqZ12zhmZjTPpJu6BQURshv7XB8Kz6yBiTxtIqKESiMWrCMXIzbH5mY4xpTloFhYapONsY4gJsmAtjTFpKq6BQEXKD4bVafWQlBWNMGkuroFCZ0KxrXlCwNgVjTBpKq6BQnshcCiGv+sh6Hxlj0lBaBYWEZ10DKykYY9KSBYWmQhYUjDHpK62CQnlCJYVK8GeAv41B84wxphtKq6CQWENztZUSjDFpK02DQivPKdhcCsaYNJbUoCAik0RkqYgsF5Hrm1l/iYgUicgC7+eHyUxPeW2EDL+PzEBb8zNbScEYk56SVnEuIn7gHuA4oBCYKyLTVHVRk02fVtWrkpWOeG4wvFYCAtj8zMaYtJbMksJEYLmqrlDVEPAUcFoSj9emytooea09zQze/MxWfWSMSU/JDApDgDVxrwu9ZU2dKSKfi8hzIjKsuR2JyBQRmSci84qKinY4QRW1kdYHwwP38JqVFIwxaSrVDc2vACNUdV/gLeDR5jZS1QdUdYKqTujXr98OH6yiJtJ6d1TwSgoWFIwx6SmZQWEtEJ/zH+otq6eqxapa6718EDggiemhMtTGBDtgvY+MMWktmUFhLjBKREaKSAZwDjAtfgMRGRT38lRgcRLTQ0VtJIE2Bet9ZIxJX0nrfaSqERG5CngD8ANTVXWhiNwMzFPVacDVInIqEAG2AJckKz3gVR+12aZgvY+MMekrqWM5qOp0YHqTZTfG/X0DcEMy0xCvzfmZYzGI2BPNxpj0leqG5g4TiymVoWa6pJZvgJevgooiFxDAgoIxJm2lTVCoDNUNhtfk4bVv3of5j8HzP4DacrfMGpqNMWkqfYJC/fzMTUoKFRvd72/egzf/z/1tJQVjTJpKm6BQ0dKw2RUbIZAF4y+EL551y6yh2RiTpiwoVGyCvP5w4l9h4D5umQ1zYYxJU2kTFFqcS6FiI+QNgGA2TH4MxpwKg8elIIXGGJN6aTO9WKslhT67ur/7jISzH+vglBljTOeRNiWFippW2hTy+qcgRcYY0/mkTVCo65LaqPooGoaqYld9ZIwxJn2CQrPVR5XeMNxWUjDGGCCN2hQuPmQEp+w7mKxgXByse0bBSgrGGAOkUVDIzQw00/Nok/ttQcEYY4A0qj5qVn1JwaqPjDEGLCi437kWFIwxBtI+KBRBVk8IZqU6JcYY0ymkeVDYaO0JxhgTJ82DwiYLCsYYEyd9gkK4GtZ/3niZPc1sjDGNpE9Q+N9dcP8RDRPpgJUUjDGmifQJCoPHAdpQWghVQqjcSgrGGBMnfYLCIG847PUL3G97cM0YY7aRPkEhfwDkD4Z1TYKCPaNgjDH10icogKtCWjff/W1PMxtjzDaSGhREZJKILBWR5SJyfSvbnSkiKiITkpkeBo+H4uWusdkGwzPGmG0kLSiIiB+4BzgBGAucKyJjm9kuH7gGmJOstNQbFNfYXLEJxAe5BUk/rDHGdBXJLClMBJar6gpVDQFPAac1s90fgD8DNUlMi1M39/K6+a6kkFMAPn/SD2uMMV1FMoPCEGBN3OtCb1k9EdkfGKaq/21tRyIyRUTmici8oqKiHU9RXn/oMcT1QLJnFIwxZhspa2gWER9wO/DztrZV1QdUdYKqTujXr9/OHXjQONcDyZ5mNsaYbSQzKKwFhsW9Huotq5MP7A28KyIrgYOBaclvbB4Hxctg6zdWUjDGmCaSGRTmAqNEZKSIZADnANPqVqpqqaoWqOoIVR0BzAZOVdV5SUyT64EEUL3VSgrGGNNE0oKCqkaAq4A3gMXAM6q6UERuFpFTk3XcNtU92QxWUjDGmCaSOkezqk4HpjdZdmML2x6ZzLTUy+sHPYZCWaGVFIwxpon0eqK5Tl3XVCspGGNMI+kZFOqqkKykYIwxjSS1+qjTGncehCuh7+6pTokxxnQq6RkUeg6BY3+X6lQYY0ynk57VR8YYY5plQcEYY0w9CwrGGGPqWVAwxhhTz4KCMcaYehYUjDHG1LOgYIwxpp4FBWOMMfVEVVOdhu0iIkXAqh18ewGwuR2T01Wk43mn4zlDep53Op4zbP9576Kqbc5S1uWCws4QkXmqmtxJfDqhdDzvdDxnSM/zTsdzhuSdt1UfGWOMqWdBwRhjTL10CwoPpDoBKZKO552O5wzped7peM6QpPNOqzYFY4wxrUu3koIxxphWWFAwxhhTL22CgohMEpGlIrJcRK5PdXqSQUSGichMEVkkIgtF5BpveR8ReUtElnm/e6c6re1NRPwiMl9EXvVejxSROd71flpEMlKdxvYmIr1E5DkRWSIii0XkkDS51j/1vt9fisiTIpLV3a63iEwVkU0i8mXcsmavrTh3eef+uYjsvzPHTougICJ+4B7gBGAscK6IjE1tqpIiAvxcVccCBwM/9s7zeuBtVR0FvO297m6uARbHvf4z8HdV3R3YCvwgJalKrjuB11V1T2A/3Pl362stIkOAq4EJqro34AfOoftd70eASU2WtXRtTwBGeT9TgHt35sBpERSAicByVV2hqiHgKeC0FKep3anqelX91Pu7HHeTGII710e9zR4FTk9NCpNDRIYCJwEPeq8FOBp4ztukO55zT+AI4CEAVQ2pagnd/Fp7AkC2iASAHGA93ex6q+r7wJYmi1u6tqcB/1ZnNtBLRAbt6LHTJSgMAdbEvS70lnVbIjICGA/MAQao6npv1QZgQIqSlSx3AL8EYt7rvkCJqka8193xeo8EioCHvWqzB0Ukl25+rVV1LXAbsBoXDEqBT+j+1xtavrbten9Ll6CQVkQkD3geuFZVy+LXqeuD3G36IYvIycAmVf0k1WnpYAFgf+BeVR0PVNKkqqi7XWsArx79NFxQHAzksm01S7eXzGubLkFhLTAs7vVQb1m3IyJBXEB4XFVf8BZvrCtOer83pSp9SXAYcKqIrMRVCx6Nq2vv5VUvQPe83oVAoarO8V4/hwsS3flaAxwLfKOqRaoaBl7AfQe6+/WGlq9tu97f0iUozAVGeT0UMnANU9NSnKZ259WlPwQsVtXb41ZNAy72/r4YeLmj05YsqnqDqg5V1RG46/qOqp4PzAS+523Wrc4ZQFU3AGtEZLS36BhgEd34WntWAweLSI73fa877259vT0tXdtpwEVeL6SDgdK4aqbtljZPNIvIibi6Zz8wVVVvSXGS2p2IHA58AHxBQ/36r3HtCs8Aw3HDjk9W1aaNWF2eiBwJ/EJVTxaRXXElhz7AfOACVa1NZfram4iMwzWuZwArgEtxGb1ufa1F5PfA2bjedvOBH+Lq0LvN9RaRJ4EjccNjbwRuAl6imWvrBce7cdVoVcClqjpvh4+dLkHBGGNM29Kl+sgYY0wCLCgYY4ypZ0HBGGNMPQsKxhhj6llQMMYYU8+CgjEdSESOrBvJ1ZjOyIKCMcaYehYUjGmGiFwgIh+LyAIRud+br6FCRP7ujeX/toj087YdJyKzvbHsX4wb5353EZkhIp+JyKcispu3+7y4eRAe9x4+MqZTsKBgTBMiMgb3xOxhqjoOiALn4wZfm6eqewHv4Z4yBfg38CtV3Rf3NHnd8seBe1R1P+BQ3Kie4EavvRY3t8euuLF7jOkUAm1vYkzaOQY4AJjrZeKzcYOPxYCnvW3+A7zgzWvQS1Xf85Y/CjwrIvnAEFV9EUBVawC8/X2sqoXe6wXACODD5J+WMW2zoGDMtgR4VFVvaLRQ5LdNttvRMWLix+SJYv+HphOx6iNjtvU28D0R6Q/1c+Pugvt/qRuJ8zzgQ1UtBbaKyLe85RcC73kz3xWKyOnePjJFJKdDz8KYHWA5FGOaUNVFIvJ/wJsi4gPCwI9xE9lM9NZtwrU7gBvG+D7vpl83Wim4AHG/iNzs7eOsDjwNY3aIjZJqTIJEpEJV81KdDmOSyaqPjDHG1LOSgjHGmHpWUjDGGFPPgoIxxph6FhSMMcbUs6BgjDGmngUFY4wx9f4/zuy3grJ/hK8AAAAASUVORK5CYII=\n",
"text/plain": [
"