{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AlexNet Training\n",
    "\n",
    "In this notebook we will train the implementation of the AlexNet class provided in the alexnet.py file. We will be using the CIFAR-10 dataset for this task. As, the input dimensions and the amount of data in CIFAR-10 differs from that of ImageNet, some modifications have been made in the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mohit/virtualenvs/tensorflow/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CIFAR-10\n",
    "\n",
    "The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.\n",
    "\n",
    "The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" Get Data \"\"\"\n",
    "\n",
    "# File Path\n",
    "CIFAR_DIR = 'Data/cifar-10-batches-py/'\n",
    "\n",
    "# Load the Data\n",
    "def unpickle(file):\n",
    "    import pickle\n",
    "    with open(file, 'rb') as fo:\n",
    "        cifar_dict = pickle.load(fo, encoding='bytes')\n",
    "    return cifar_dict\n",
    "\n",
    "dirs = ['batches.meta','data_batch_1','data_batch_2','data_batch_3','data_batch_4','data_batch_5','test_batch']\n",
    "all_data = [0,1,2,3,4,5,6]\n",
    "\n",
    "for i,direc in zip(all_data,dirs):\n",
    "    all_data[i] = unpickle(CIFAR_DIR+direc)\n",
    "    \n",
    "batch_meta = all_data[0]\n",
    "data_batch1 = all_data[1]\n",
    "data_batch2 = all_data[2]\n",
    "data_batch3 = all_data[3]\n",
    "data_batch4 = all_data[4]\n",
    "data_batch5 = all_data[5]\n",
    "test_batch = all_data[6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{b'label_names': [b'airplane',\n",
       "  b'automobile',\n",
       "  b'bird',\n",
       "  b'cat',\n",
       "  b'deer',\n",
       "  b'dog',\n",
       "  b'frog',\n",
       "  b'horse',\n",
       "  b'ship',\n",
       "  b'truck'],\n",
       " b'num_cases_per_batch': 10000,\n",
       " b'num_vis': 3072}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([b'labels', b'data', b'batch_label', b'filenames'])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_batch1.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Loaded in this way, each of the batch files contains a dictionary with the following elements:\n",
    "\n",
    "data -- a 10000x3072 numpy array of uint8s. Each row of the array stores a 32x32 colour image. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored in row-major order, so that the first 32 entries of the array are the red channel values of the first row of the image. labels -- a list of 10000 numbers in the range 0-9. The number at index i indicates the label of the ith image in the array data. The dataset contains another file, called batches.meta. It too contains a Python dictionary object. It has the following entries:\n",
    "\n",
    "label_names -- a 10-element list which gives meaningful names to the numeric labels in the labels array described above. For example, label_names[0] == \"airplane\", label_names[1] == \"automobile\", etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.5, 31.5, 31.5, -0.5)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAAEZCAYAAADmAtZNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3cmzHWl63/cnM8883BF3wMUFCijUhEJ1VXVVVxSb7LaaVIhUWOFQKGRq4fDW4Y3/Ai8c4b3/BWkjKaSFLdpBmVJYZLtJdjfZc1fXBKAKMy5w5+HcM58cvKBDEVo8vxfqWw0moO9n++A9J0/mm2/mixvx/KKiKAwAAAAAUB7x3/YBAAAAAAD+U2zUAAAAAKBk2KgBAAAAQMmwUQMAAACAkmGjBgAAAAAlw0YNAAAAAEqGjRoAAAAAlAwbNQAAAAAoGTZqAAAAAFAybNQAAAAAoGQqz/LL/uUv/rBQ9R98d8etdRtvyM9ut+ZkvRrpn9ppV93aufkNOXaxtSnrC/Pzbu3J/gM59s7eR7I+d6Ev68sXBm6tWh/KsaPBsaw3GjW3lkQLcmyepbKeZaeyvjjnn/N6vSXHVkx/9klvIusHO/5cGvf9a21mNpx0ZL0w/xY5OnyiP3uoj7vXPwl8t39Njg71PPsX/8sPI/kPSu7//Nf/VK5NWeqfm/FEn/fZdKTrs6msp+K7Q9R8MjObpTPxvZkcW6kksl6vNWU9iet+LdGfbYX+XZmop4Gxs0z/7ixQV8eW5rkcmub6s4vAsSuhGzT0ybk4dlUzM8sDczgJjK+I+RCa4//z//q/Pddrk5nZJ5/ekj9SrRFRFPj5oTkV6fqvPyPDc9IK/1/kFlgjAkd2tkkROCeFns9nuY9DY89SP/tnh363v76d5ZyExgfXp0A9dGxqfGjs3/297wSnIn9RAwAAAICSYaMGAAAAACXDRg0AAAAASoaNGgAAAACUDBs1AAAAACgZNmoAAAAAUDLPtD1/4ndjNjOz9jm/BfivfvZDOfbi+nuy3m3rVtHjqd/qdXSq22uOFnR3zTTy2+AvbuhL8OpFXR81/EgDM7PT3G+xn/f89vpmZvWsLetF3T8vs0y3/q8kuk390tw5WW/VxHcPunJsb3Be1k8PerL+4NZ9t5bUdZtXq/rt0M3MHm1tu7VuR1+v/qlu7Z2merxqORzoXvvc63T0fFQt2ZstHQeRZ/qz8/zXb79fqej1IQm00E/F7wrFAoRaGocaYEeFOLYzdhTPRPzHeKZ/12ym79HQeZlO/fH5ZCzHWmAuFME20n4tGKdQ1euDajOdhuJWAu3M48CrSL0WWrtebGdqLx64l2J1H5qZBeppIlqTB1r7VzL92ZGYN1kSiLII/m5dFz8r3K490Kb+LH6T7fn/Nr/7N9me/zf52b/p7zbjL2oAAAAAUDps1AAAAACgZNioAQAAAEDJsFEDAAAAgJJhowYAAAAAJcNGDQAAAABK5pm259/aPZD1jSuLbi1JdMv1pc7LgW/XLZe37t5xa3e3nsixFzZ0K/pB4R/7YuVIjk3nbsh63NHndDKrurXTY91Seami247XRIv8uXndkrzb3JT1SaBF9jQVLfRT3Rr3ZGdF1o/u6Nvi1k9/6dbaF/U5vfDKqqw32v716p3q2IDJONDmPfI/28xs/2DPrU1ngbbizznVft/sbO2vo0j/gzhUT/wW1rVA2/Ik0L6/Fjh2JdR2eDzWc2Yynri1Wk1nudSqei6rI2tMA9c6C7RCD7TeVr+7P9DXoz8YyPp0OpV1FUtQr+hzurS4IOtVcc4nE/9amplNAnMh1M1czfM4fvH/vzm0hoTqSpEEPjsQ29DI/WubBI5rVtP3cZz6d3I113NuZIFJFVq3VS10vuNQ5IFo5x6IsogCuQOhKAd17KE1PXSvhTrRn6VVfTgOxneW++NpxqvzEopUeRov/goHAAAAAM8ZNmoAAAAAUDJs1AAAAACgZNioAQAAAEDJsFEDAAAAgJJhowYAAAAAJcNGDQAAAABK5pnmqN26dSrrl1/2862uvH5Jjr3zxZeyPhj2Zb3d9TPDTkcncuwnNz+W9c7Gq25tuatzcdJYZzA8uqNz1Kzwf9dibUMPNZ3L1aj512tpfk2O7Z/obJYbn+vvXmyvu7XunP7/h9myzjgZbPmfbWa2vePnDV3Z1J/d6uhjS3P/ek3Heg5Xavqzjw71/Tcc+Hk4USAW5nmXB3LUUlFPRM6ZWTh7Jg/k4qgMl1C2TJHq+0hlghV5IPMmEE0zGo1k/fjo2K212jrDcX5+XtZV7lYl1gdeBCZ7kgRypUQ2VKOuxzYbOutsOgvkqKX+PK0Esufm53T2pc5R0589rulXjdFQ/y51DzWbTTn2RRDMUVM3Y+A+rRZ6jRh/8pGsj25+5tbmAjmP0YbOU42XL7i12vyS/uy5hqynldC6669/wWisYFyY/91x6IIFPrw4Q2ZYaGQ4ESz0CWc4trP8rjPmqIXz40Qu3hm/24y/qAEAAABA6bBRAwAAAICSYaMGAAAAACXDRg0AAAAASoaNGgAAAACUDBs1AAAAACgZNmoAAAAAUDLPNEft4QOdVVSYn7vTW34ox05jnXWWVWayvrDoZ3K8+voVOXZnV3/3YObnU/3qU52Dlsb6nC2c8zPazMys8LOzqnX/uMzMFpd0Tkmndc6tnfZ0dsT+zkTW86memo25rlvrTRfl2I/HL8v6ZGlZ1uPV+26t1dDX8+j4UNafPPavVzrReTezib6e/UFP1lORudWo6Xyn510RyBIykTcWzFGLAjlqsc6gCmW4SKEotML/7CzV2VZJIFVnNtQ5aqcnfi5glunvbtT1Oakk/vpggVybUARSFLge1Yp/PeNAhlso480inXVmIs8nC/ywauAeTxL/2KJIz+Ek8KaRzvTzOcv8+7Mo9NgXQigLLffXoKIeyAXc1e9W43/2z2Q9vu+Pn1X1hR+3dNZZds7PY62t6xzYxqWLuv6Kfq+LLpx3a5NF/Y4QNXS2Xy4eN6HpHAXCzJJcvzOa+fVApOdT1PU/iMXzRmWRmZ0tyywklEcacpas06fBX9QAAAAAoGTYqAEAAABAybBRAwAAAICSYaMGAAAAACXDRg0AAAAASoaNGgAAAACUDBs1AAAAACiZZ5qjlk501srxrp+dMxseybH1ts5QWFzXmWBF3Q+2WH1FZ9f0cj8PyMysP/J/V9P0cR0c6Gysbm1e1jc2F9zazHbl2JNcf/fgcN+tNRJ9XH0dsWTdOZ3pk9b8+bA7WJVj/+SPdEZTXjyW9as1//OTQmfW7D/WWWbTsT+Pk4rOKBkHsohCGSedrn/NolCAynMuFF8VV8V1FRlrZmaTqb4ulcqvvwyHMlryTNfVqhkFftfuzrasP3gYyGea+d/e00uqpelQ1l966apbq9Z0xlESyL0L5fnkIsdI5YGZmVUCmXxRYKIWuVg/Ar+rVtdrbqXiH1tg6bFKRX931tHZT5OJn7t5lpjB50URmnPxr3/dQxltttSW5eGJfz/NBvpBnxzr3NG48K97JR3IsYNbN2X99K/nZL1Y8bPSauf9jDUzs/qGznhLNv16tq7fX4quPu4o8FqfiUDFYIZkpidLInLSzEI5bGfPG/OE1myVg2YWfs6qDLc48NlP47+AJQ4AAAAAni9s1AAAAACgZNioAQAAAEDJsFEDAAAAgJJhowYAAAAAJcNGDQAAAABK5pm2569Huj3/bOS3g19cX5djt3Z2ZL033pL1Ir7l1t556zU59pt/oI+tXeu6tdnQr5mZ3bql29v2jvZkvdn0Wy5nNd0S+VHvgawvd/2ezBuLNTm2u6RbZNcC/4cwSP12qLcf3Zdj73z/RNanp7dlPbrojx/u6vb7519qyXpzQZy3WMclxIk+562Wvv+mIkaiGuvjft5lgfa7qr1vmuqW66F6qDWwotoCm5kNhrqF9dbWI7c2Heu1ZzbSn314eCjr1YY/p6o1PVcfP/KP28ys3fRbWG9sviTHnuV6mOlWzklVP3JD7flD81TNtUpFn9N6oD2/at49HutnySzV/ftD7bOTxD9vKjbgRRGak+Oqf927gfOzcHFT1rv/0/8g66Mb/rvT6KPP5djJpzdkfSh+96Sj29R3uoE29blel7NT/92qclv/ruGpjg/Jzq34n33hghzbeUO/j1bef0/W8/P++2rfAvdSoP1+HOiwr+Zxnv/mIoBC90/oORoi2/d/Bfkh/EUNAAAAAEqGjRoAAAAAlAwbNQAAAAAoGTZqAAAAAFAybNQAAAAAoGTYqAEAAABAybBRAwAAAICSeaY5aqdHfVmfO+dnGRz0nsixjY7OSegPdGbGLPVzYG58dleOfbKl88a63YZbW1u7KMeuXtbZWMP7Osvo4Z6fCdbs6tCL5RWdU7I45+d6xbHOOarU/HNiZlaL52U9nZ5za/kskMeRH8nyta/pnLU3rvj1bmsixy6u6HM+HLbd2nSq58Lpgc4SzKb6u5s1kZWWnS1npOxkDoqZZZm/PoQyWioVvcyGvns28zOoQt9dr+s5s7m54daOjw7k2MNdPdebDZ3b1R/699HSwqocW6/oXL8nD/3czFrdv8fMzFbW9HeHMr/U9U4Cc6FaDWQdTv2sQzOzmchRC2WwzWahPED/uycTPRdmM33c6UwfWxT55zx0f70IGpFefztt/xw0Ez1fq0Vg/dq4Kuvtc34uV/H+u3Jserwv64NHfn3wcFeOHT7ZlvXpvh5fE8+8QVXnAs4y/Q5R3/YzKvNd/a57/MnHsp7/5CNZX/vH/8ittd9+S449Tc72HhCL4aF1NfScVM6akxZ6zqpjP8tx/8fPP/MnAAAAAAC+UmzUAAAAAKBk2KgBAAAAQMmwUQMAAACAkmGjBgAAAAAlw0YNAAAAAEqGjRoAAAAAlMwzDSCJ8kAWQcWv90fHcuxaIPsmMZ3L9fixn4vRK3TmV+9IZ8RUGntu7WDg18zM5ruLst7oNGV9bnnTrTXr+vKvLZ6X9WY9EdVAzsjMz6X6m7rOcCqq/v8x9I5W5Ng5HQ9n3/l7y7JeNz9/5fx6R46tyXNmdutjP3Pj8Ggox457fjaLmVkhsgLNzObP+ceeBcY+70I5KSqHpV6vy7GhfJhQNlYqsrFCnx2qt9p+Hlm3o9e9aqTzYeJAvS0+v1rRx53EOh9uOvWv5+3bX8qxvf6prF/c9NdUM7NG01+TU5HHZ2ZmsZ6Huek8IFWfTXXWWSjuJ8/9NX0qsv7MzEYjvTZFhb7e7baffZckek19EcQD/f5TP/WvbXNZP/Dyls4knOb6HWNa9d8jioZeQ+pz+lnbeekVt9Y1nfuXHx7K+uyxzh3t3/PzcXufLujvNr3GpDt+Plw01s/56lTfS5OPfirrO5cvuLWL770jx04Dy1dk+h/kgees/OzA2mhnjyv7jQi9WzwN/qIGAAAAACXDRg0AAAAASoaNGgAAAACUDBs1AAAAACgZNmoAAAAAUDJs1AAAAACgZJ5pe/7+qW57nAz8fWNXtIA1M5sNdUvT2HS9Wffb28aRbjHbXdStWrPEbyM7mur2/MMd3br7yoXrsj7fFK3qZ7rV8+xEt5pfFK29raqPezgeyLpVAq13E38+3PmyKscurul26u+9r1sGN+1VtzbL+nLseKBbtaYzv2XwdKTvn3qif1ezreuqy3UUl7T37VekUdXnRnTnt1pFz7c01a3LLdNzvSLaEieBlsWZaO1vZjYe+PdpHOv1YTjS9/hgpNvBx6KVc6ul15440i3Zo8j/3dOhPq6Hd+/K+rDXk/VXX/PXh0bLbzNvZpaqiWYW/K/VRMUaBNpERxZqI+1/9nSi23L3+7qleLOln7GVmr/ex6G23S+A0y2/VbyZ2V//2X9wa9HLL8mx3c0rsn7tpYt6/NKSW0sCrf8tMOciESkR53ptS5YD8SJLOtKp8fobbu3c73xbjs2f+BE+ZmaDO377/tPbX8ixkzv3ZX36yG/9b2Y2EuvueKDfXyLTz7o48CzM1LMsEN9VCdQnsT8fQstqKDrsLEIROU/1GV/BcQAAAAAAvkJs1AAAAACgZNioAQAAAEDJsFEDAAAAgJJhowYAAAAAJcNGDQAAAABKho0aAAAAAJTMM81RS+p6Xzga+3lD/fs6Q2qyr3NaVjd0kEK76econYyO5dhuRefyLK35mT97e4Fsq0znCWUTnSc07vv5cfVIZ/rEic6HO9z3P7vS1rk6B6f6nI36Os/DKv6xPdzS0/r85omsNzo6J6ky9vOjRiOdG1NM9DndvOB/9rzKrTOz7fs6m67dCRxb7H93pONRnnuhLDSVw5LN9FwfDfV1STOdR5aIzMAsD+Xb6TW3mtTc2mSq19RQNlY1kE033/XXnyTW16Na1fW+WPfGI309ikw/K7Yfb8n6sO8/q9557xtybGtOrw9poeeaieuZhDLYCv0PIpHDVqvpzKpGQz9rQjmN6UzcI/EzfY35W9FN9ZzMb/jZf3/yk5/JsTeH+ty/+9brsv7hh/6cvv7Wm3LsxoXzsj7X7bq1JGnKsUUgZy1Xc8rMIpFLmjbm5Njmgs5oa7/pZ7QtpHp9yg/1+0nzrp/FamZ2KqbSvU9uyLHti5uyvnBJX89cPSsz/S5bCS194ncVgZi0YIZkqCzWxjz4jA7jL2oAAAAAUDJs1AAAAACgZNioAQAAAEDJsFEDAAAAgJJhowYAAAAAJcNGDQAAAABKho0aAAAAAJTMMw0giQqda1GM/aCElblzcmwy0p+dnurcnbzun4rpWGe47e8HcnmqfsZCu6rzZVZWN2R9dVmflxWV5zHTuRUqY+lvhvtZZ73Bnhz7aMfPfTEz236ks0AORTmdvC3Hdhf0sW3vfybr85GfR9aq6dyY1Y3XZH3jgp8bE6U6q+j0ms6VmaY6my6L/Oyp4URnZj3v0lTn+s1m/voyC+TxpJkOgKkEMtxUDkua6nUvlGWm8siyXP+u01Od59Nu6/m4uXnBrd2+fVuOjWP9f4zqnJ0c6xzFk55e75NEr5uTiT+Xvryj173Xruu1qwj87qTqr9lxofN8klzPUxUl1OnovM9KJXDOpvoZOp36c3EWyCF8ERSB+7hW8UOkVlt6bbt3dCTrP/i+vl8+/pX/vFxZ0nljb117Vda//sEHbm3tgs70uhzI/Frs6vWpVvPXxiIKrD+BP4HMIv9ZPmrpczatLMr6qK8z93b/ys/V6390S47tXtfvL1HvqqzX11fc2nBB/y4r9HYlETlsielzUsS6HhguhZ5VT/UZZ/4EAAAAAMBXio0aAAAAAJQMGzUAAAAAKBk2agAAAABQMmzUAAAAAKBk2KgBAAAAQMmwUQMAAACAknmmOWo2G8tyTeQJdWqBPKBM/5R0GsiIqfvH1mro7z7Yncl6Jn72tZcvyrEXlq/IeqWis87GA/+cVk3niESJSs4x60/9cImbdx/IsU+OdT2e6euVH/u/a6nQmV+vLer/n0iHep5OK34GSjLbl2OjQKZGrel/99o5nTlzbu6SrPcGOi9nMvPzdtqVZTn2eTca+xlyIZOpni+hnLRGQ+fjnSVHzQp9D0eRXw8ddxHIlgnlxzx6tOXWajW9rqlcrZBWW2dXjif6s+t1/TxoNv119eGDe3LsYKzn0pWrOseoITKYiiyQY5rpvK1c5AGGsuUskOEW+h/jPPUnWyHujxdFvqYzpi7817/t1q490dlYczfuy/r+gc4V3D31M/B2tw/l2B/39DPpZ5/ccGu9wHP6retvyfqHH/oZbWZm16/7maiXLq7JsZ05vcbMIv9+mdzXOa+jv/xY1k9/8gNZL3b9631ZjjRLvu+v2WZm2x/9QtYbv/Mtt9b5zm/JsUVdP48ylW0XWCMi/bqpQyTNrAg9DM+Iv6gBAAAAQMmwUQMAAACAkmGjBgAAAAAlw0YNAAAAAEqGjRoAAAAAlAwbNQAAAAAomWfann9uviXrjbbf1rio6P6Y7YWOrKeB1sNp6reY7Z/o1t1JX7fmrFdEG/yRbjlqo3OyHFVWZD1L/fNSr+pzNst07MCJ6Kxb9K7Jsc3Zkq4X+rzUkwtubfv4p3Ls5cqqrG82dFvfWeyfl9GwL8eeTJ/Ien544taivCfHLrR1PY91W/HTnt+jttbW7aGff4H2vaKNvSj9//T6kIm252a6HXy1qu+TyTjQxv4MXYVbLb2eh9rYj0UkQqj9/myq16beqd9SvN7Q617IaU/fZ8dH/sJYDbSYHpwey/qop1ulzy34z4M81EI60KO60/HP2/Kyju+IIv3d+Uxf74povV0JRQO8ALKJvjZZ6r9jXL2u255fvfKurPe278j6/e1tt3Y41Pfp1oFuzz/q++9t071dOfavv/vvZP0Hf/49WV/d2HRr33rbb91vZvb7774j692x/7u2/+rHeuz9HVlv5/pdt1n163Egkukk0Wv66td+R9aXv/GhWzuMdVzUZKTnUlIVxx56Rseh/vuB8WroV9C6n7+oAQAAAEDJsFEDAAAAgJJhowYAAAAAJcNGDQAAAABKho0aAAAAAJQMGzUAAAAAKBk2agAAAABQMs80Ry2ZBPKEotStzQqdoTAMRBUM+35OmplZteZ/wFwUyAuKdY5LLZ1za+3kJTk2mVyV9Xy0JuvN6oJfzPQ+PQrkO53v+se+vqCzW0aZzgMaHI5k/e7ufbe2WPlUjp0v9PW8tKrP+efbt91aHOm8sWqk5/FU5OWMR/p6jDo/kvWspnNKeuOGWzs91vlv9rV/oOslV637v93MrHdy4I+NdAZbHAham830nIhj/z7NMv3dWSDDJa74nx1bTY5NqvrxcXS8L+u1ij++P9B5hI2Gvl5J1V+TD44P5dhZ5j+HzMz2DvX4dtvPG6sEctRC+XH37/lrj5lZu+vP00ZD3/8qJ83MbEVkpbUDY4dD/fwdT/Q9UBH3UCOQJfgiqA/Gsn7wp3526I9bek7tBfIQP/zG+7J+5WU/M+yqyKc1M3t976Gs73x50//sjr5PT/r6efnoSL9j3Nv+wq19b/tLOfbqR5/L+pVjtb7pd6PGy36GrJmZzfT90Or5vzs/0Ofk3Lc/kPW1f/Tfyvrxkv9+ND7R6+rOnbuyfn7dz8etdcV7sJmlgRzVUMakHhsMWg3iL2oAAAAAUDJs1AAAAACgZNioAQAAAEDJsFEDAAAAgJJhowYAAAAAJcNGDQAAAABKho0aAAAAAJTMM81Ry3d1pk/e9LMMprHOEak1deZPrepnwJiZxVP/u4tU55DkqT6NqxvvurVq9rocu/dYZ99URRaRmVna9LNGsulEjh2N9O9uNP0sozgws+YXzst6bU5n0x2u+Ner1ta5ML3xkazvjD6R9c66//8bjUznqE3GOm8oyTbcWmE6j2P78BeyXq92ZX1p6W23Fs/0cT/vBmN9LxyL7K3FTiBvLJDxmBY6oyVTeYaFvk8s0XMmTvy5nOY6W6bT0ffZ3t49WbfCP7Ys0ydtMtPXa1lkfs2LmpnZ7r6fRWZmNgk8D/r9oVvb39JZQbWankv1el3Wp7m/3ncC9/DauSVZb7f89b7Z1MdVD9RDSUPjoZ8tNZ7p7KcXQdbWOXNJxT8/x0fbcuzPfqbzDn/2l/q5snTBz6967+vX5NhvvBWo/73rbm10oLM9d275OWhmZpf39Hl5I/Pn1empfh+NHul3jFjkAi++84Ycu/ZPdGbpZO9Y1oef+O831V/ekmOX3teZer0Fnes7PfXnaeOWzqYb/egHsl6867+/1K757+BmZlmin6NpoTP7VAxb9BX8PYy/qAEAAABAybBRAwAAAICSYaMGAAAAACXDRg0AAAAASoaNGgAAAACUDBs1AAAAACiZZ9qe/81N3doza/ktfLNqVY49v3BO1hvzc7Ie5X6D4L29B3Ls4UC37kwar7i18XhBjh3NdBvYRvNE1qdTf/xo4LeRNjMbDAayrtqGZ5k+J3Nd3Sq+2dGxBFt7fpvrcaLbhj8Z7Ml650C3Bk8W/WOb9e7Jsa1Yt99ebF52a5WabmKdTvRnt+u6Lfnm+qturWoX5Njn3dGhbqc8FlEVedtvW25mlkSBFvqB9v2zmd/KuSLa65uZxYHvVvdprHoOm9n62oqsnxzpqIread//7jhwUgKGQ39ta3bbcmy3q69nkes1e37eX9uOT/z21GZmR0d6PU9Tva5WRFxLv++fbzOzaWDNbovzFkc6YqJa0c/vdkOf89j8+y8V98eLYtbRz8u1b33brb2+e1OObS3ckfXjRzuyfm/Hb4P/7//oMzn2+/+vXkN+++/8Xbe2uurHApiZXXv7v5L1i6ZjHQaH/u+e7el7KRt/Luu9wZZbe+nSFTm2/d7XZb2ZBV7r635Mx8Ev78uh3fN6TY9mev0qPvm5Wxv/m/9bjt149EjWe7fvurX+Bzqq4dwHvyXrleVLsj4q/OdwbIHW/k+Bv6gBAAAAQMmwUQMAAACAkmGjBgAAAAAlw0YNAAAAAEqGjRoAAAAAlAwbNQAAAAAoGTZqAAAAAFAyzzRH7e13viPrscifiTs6+2ahobOzkrqf0WZmlpif8/LpzZ/KsQcPdM7I3W0/06da0TlpzY7OQarNdC5PMfOztQYnOkckLSb6u2v+ORv29XHduXdb1jsNnQmW5SIvaOZn7piZ7Z0eyPrV2WVZP9zyc3se3NP5KdWpvp4LHX8ubVyel2NPUj9bzswsX9D3yFLVz5fr1HWOz/OuUdPrQ9H0s2eKSM9Vi/U1r1f1+DT188yKQueNRaaz9+Lc/+wk0dlYtcA5e/nyy7J+576f2RPK/AplPB4e+fdCuq1/V5rp/LhWy58LZma9E//Ymy19H0WRfiQfH+u8vzj2/+91PNV5Yx9/pjOvVtf93Kq5Ob22ZKn+7log+7JSFfM41nP8RXA80XNy5cNvubXfzV6XY/fvfirrD27pZ/Xl7V23dnDc0999qHO3PvvBn7m1P35yLMe2VnTm1/Vr12T9t9/9hlt7/5235Ni1d96T9Ud/9O/dWlzVGbKFyOwyM5vU9Pi86+epTlOddzgZ6/e6yXf/g6zv//Efu7Xatn6PbotnlZlZcmffrQ0e6vey3U9uyPrK3/8nst5587pb6wdiVJ8Gf1EDAAAAgJJhowb7QCCaAAAanElEQVQAAAAAJcNGDQAAAABKho0aAAAAAJQMGzUAAAAAKBk2agAAAABQMmzUAAAAAKBknmmO2itvfyDrRbXh1rKKzmGpJDpXJ8n8zzYzi5p+2MHwE527s/VQ53Idjv16t6MzedJt/btbdT1+dcnPvlme07lc/aE+p9OpnwE3G+sss34gX2Wcp7Ie5/7n98cP9XcHPruX66yQKPazq6rRmhz72Zc6k2b+nP/dRxWdwVRt67nSD2TuHRz5+U9X1vxMGTOz99f+e1kvu05T5zQWmb8GTAO5W7OhzkrsdnR2TbUi8goDn21R6P/i/DlTZDpnsYj0716cX5D11RU/X/LkWGckRZHOzhoO/M+epHo97w/0744ivXY16n4m2NHREzlWp+KZzc3NyXqtJjL5Yv24fxzIMfqTf/dv3dq7b+tMqtUlnWnVbupnUSTugam4N18UlULfayfmP9M6HT83y8zsyrt+BpuZ2YW3vinrx0+23Nrug7ty7PaWn6VoZvbkkf8sX+vqu+XBrs5o+/n3/0LWf/wjPz/3jcvrcuz/+E39rjt36h/b4S8Ceaj/h85Ja1x/Tdazgf/dyVDnV/b/5f8l65MDfc6bR4/9sSor0cz0J5s1cn/8Wqqfg8ef3pT1e6N/Jevnq3/o1uJrb8ixT4O/qAEAAABAybBRAwAAAICSYaMGAAAAACXDRg0AAAAASoaNGgAAAACUDBs1AAAAACgZNmoAAAAAUDLPNEetNa+zUtLc3zdmOmLBrKqzsfLCz9UxM2t0/By12WBPjt354jNZLzp+RtPK+nU59subfu6Emdko0pka0WDi1ioXdA5JFEj1efLgnlsbDHXW0DCQ15EEsnGiQmS8NXQGU1HVuVUPt3UO2+K8fz0vXtqUYycTfb1GU/+8TCf6nHWX9O8aT3QWz7TnJ5XUTee/2Vu6XHbTmc79qzX8HMbcAjlqU/8eNDMbTfR3z6YiE2yis/Pagf+Ly3N/fJrqPLFmsy7rUcVfU83M1i+85B9XrOfywf6urI+m/jnNjnQiT6up79HhUF/P075/n7YDuZmvvvqqrHe7Okvx9NTPSswDIW0bGxuyfvv2Lbf2459+JMdeD2QJfe3aiqxXRT7cOHA9XgTzVf2qNsj9OXf/4b4cOyn0y9X8gsjmM7Ns4k+sS5uvy7Fvvacz3G7c/oVbW/34J3Ls1bv63enwVD9PH0/8jMriRL8Tnv7bP5f1cyqXs6Kvx84//99lvdHS69d00c9irAz1XOl9tC3r3c0Lsp43/XW9dhq4jwPzdFTznze9Bb33yFKdGTzZviHrT371Q7d26bL/nHta/EUNAAAAAEqGjRoAAAAAlAwbNQAAAAAoGTZqAAAAAFAybNQAAAAAoGTYqAEAAABAyTzT9vyx7tZsRea32J8F2menmd9K1cwsr+nWn/mp36Y66h/o7+7vyPriyhW3NtnTYwe7ulV8muuWpbO+3yb/IPDdSV1fsNHIbwU9Gun2/KdDfU6TODA1E/96b17RY1fP++1pzcxauuu4FYXfjngw0+1rr1y+JOuVzG9vO5x+KsfGlUeyPs102952x48WEF3cXwh5rHuXVwORDkpS0WNrdT3h4om/do0z3Wp+PNXrYkW0wY8quqXxk4PAdz/21wczs6aIPJhOdeTB/PJFWV8Ry/1sdleOPT+nW+jPz+nzMhHXq9VZkGOvf+1tWZ+KzzYzGwz8NtPLS0tybCg64P5DP4Pju9/7Czn2T7/3Y/3Zd/Xz4Pf/4Hfd2vr583LsC6GqW+RHqV9PEv2OcLSj3wM+ualbtj/6wV+5tQ8L/beAD/7hP5D1a9/8Lbd25VUd+bB3T9/nD+99Ieu7h34ESLqrW/vXf7kl61Hkn5e6iKIwM5uKsWZm2bG+l+zAjxbIQrFIgXf4YV+v+SY+v2b6HT/wWmZF5h/ceKqjZqLAe3S8pONDxh3/nTLXj7Knwl/UAAAAAKBk2KgBAAAAQMmwUQMAAACAkmGjBgAAAAAlw0YNAAAAAEqGjRoAAAAAlAwbNQAAAAAomWeaozYKZPpMR37GQigHISt0PU0PdV1kOAxPdDZEXNcZDJW2f5qP93Xe2P6TQDZWoc9pmg3dWmdB58+kYx2akU/9zx6O/KwOM7Nx5meUmJlFNZ09Van6uVfnNvXveuU1P9fOzGz7QOfK1EQMWxTrsdOBnofri1/zi/GGHFt09Fy6eeNI1s+vrLm1dr0lxz7vplOd4ZIk/r0wm+mQuXanK+s6wc2s1fLPfaWil/BQPkyjJq5rRf8/3miqM71ufqFzik4Oj/3PHunPnk113k+j4f+ul197TY59/bWrsr7Y8fPfzMz6R37u1Cj1s0LNzEZDf0010xltZjqbLk70XJkG5vGFNT9L6A//mz+QY6+eX5X1n//8M1m/ecOvL68sy7Evgrim51xDLCIbGzqBar6r16d6W+eOPv6Vn+9598vbcuzR//Nnsp4c+WvEO++/K8deefubsn75mh5/KHJmJ/v6WWr1n8hy8fNfubWkrZ+1+USvIWlX5zzW1bNuqN8nI9PrbjzU7+G5eKUcxPoZXEwC79lT/yaImjrMrNfWa8jkip4r/ZWX3drDwLPs67L6N/iLGgAAAACUDBs1AAAAACgZNmoAAAAAUDJs1AAAAACgZNioAQAAAEDJsFEDAAAAgJJhowYAAAAAJfNMc9SyQKZPLrJAGjWd9TGbDGR9evxE1g9nfl5Ha3lBjv07v/9tWX889DM3Hh5uybErV3UGSh7pvXY283N5ptaXY9tzOrdr96F/TsdTnaP26rtLsm5NnS51cHLg1hZWm/qzI53RNurrebq00nZraaHzVc6t6YyTlRX/esbxOTn2eKTzV1YW9FypJ/743cc6H+V5F8ojy3M/h0XlnJmZ1Wo1WU8C310U/r0Q+u5A7I31e/760Ahk521sXJD1oxOdGXgy79+Hg76eb/fv6XVzlvnjb96+L8dOAhltb716WdZrkR8W1O3oczoY6d+t5oKZWVXNtUiva1mqf3er7Wd5NQNvEt/+nfdl/epV/ayJa/6xZ5meZy+CSqKfWVbxr11h+rp3Wx1Zf/2Sfp7O/sDP0Pvr4b+RY3/4yU1Zf/DTT9za+x98JMd+65sfyvqbr78q66vn1t1ad+0lOdbmF2X59t27bq0odE7a3IJ+D6jNdCbY8ZafzRvH+rst12tEkgRSQUWOWmF6jk8D351m/rEPl3QW4Mkb78j6zvorsj4Y+O/ptQOdjfk0+IsaAAAAAJQMGzUAAAAAKBk2agAAAABQMmzUAAAAAKBk2KgBAAAAQMmwUQMAAACAkmGjBgAAAAAl80xz1KZTP4vIzCwShxPlgT1lpn9KtaHzyBoLfk5bZ6Az3E7vPJT1b1xfcWtXr4tgCTOzeE2WpyN9Xn7yF/6x7e/r3IpmV//u4cjPYZtf0p/99gc6h+Turs5Xsa6fDbNxyc8/MTNbXDwv6522zvQZpTtu7XQ4kWPzQp+XR/t+bsxSID9lMtQZbfNNne0yG/k5JZOx/l3Pu2pVrx9J4tdVxpqZ2SiQjRXKUZMZbm2dy9WsBjIFhd5JT9ZrTb2mfv3d92R9fs4/9sdPHsuxj7cCOWoiE2xnT/+uyWQq6zdv35b1bts/55sX9dpSrer1IZTJp+pFIIcolNFWqfk5ao25wNpyqHM1l5Z0pufhsZ+b2e8Fsp9eCPo5H0f+GlIEXjGSql6/8kJ/96uvv+nW5v87vUas/+xXsj4Q74xJoo/7888/l/U7d+7I+sULm27t+utvyLHrgSy0WBx69HBXjh0mOjdwPK/f26wQ60ARekfXmXzVhn7eqNjfahr47nm9Pg2a/u96ckm/Rw9ffVnWt4eBLMKBf72rZ49R4y9qAAAAAFA2bNQAAAAAoGTYqAEAAABAybBRAwAAAICSYaMGAAAAACXDRg0AAAAASuaZtufPpro9cDYeu7VKRbfmjCq6BXZ3TrcNzUbHbm3rgW7z+sUnX+rvbvitXMdL23LsaKZbRS83L8l6nPvndGXxNTm23mzL+mTmt1OdP7cgx85S/btOT/dl/cKmH3kQZf5vNjP78+/+SNarLd0mdvWSP49riW5HvP1Yt6meZn4b6sO+bu291Lgg6/OdOVlPK/7/26SBFvTPu8FA99CtiBb6oZbqcaL7Yw+G+rsbdX9OnRz565aZWc90K/pWq+PW0kCcSpzolsXNll4/+n3/819/7ety7PKSvhd6Pf+8VBPdlrvR1M+K6Uy33n744JFbe/zkiRy7vqYjOLqByJQ49u/hItfP0FpNz+OZaJ89ODmVY/tDvd7nEx07MB759TzUf/6FoP9PXV3aqND3aRHr81ep6fpc6q+N3Zcuy7Eb5wPxQ+IdI5/p53x/MJD1vT39jrG777fB/+HPfyLHrmU6zma+7r9DvFzX9+Ek0+vPqKd/VyXzz2m1drYYrEmu51oc+8+y6kSfs9G8fh5VvuG//+QL+j15mOk5Xk/0XJtr+NEl46ped58Gf1EDAAAAgJJhowYAAAAAJcNGDQAAAABKho0aAAAAAJQMGzUAAAAAKBk2agAAAABQMmzUAAAAAKBknmmOWrU6k/VZ388TCmV5jDOdHfF451eyfuOnH7u1buJnDZmZtWd+hoKZ2eff+6Vbq1/WuRMHIlvOzKx1VeeVXd5subVHOzq3IpvqvI5Kzc+2WRNZY2ZmedHX9aHO1WmJPI67N7+QY3/4Iz/nyMxs8019W+Rd//83qumyHJv29O9aWvG/+97d23LsjRM/98XM7Pd/99uyvr7p50cNUj/f7UUQRfo+nAQyXhSdimPWrOs5EcX+sVUSPVenqb6Hp4WfbzW/rHP3QudsOtP5cMc9fw2IEp2Zs7GxKeunp/533/hU52K2WzoLsRXIhzOR79So6bHDQKZeKEdNZfpFkf5/2TjSc2k08LPSej299uSFzhLKcn0PpJF/TQI/64UQmT5/hYl7MQrkzAXuY4sC+XtifFHoTMIokEHZyPz3iCLT713d7rysr6+uy/po6N/H+0f6efhgS2c13nvjsls7aup7fO5EvxNeODiR9erQrw9Mv7ftB/62U+jHjS2O/OdN7VRnIfcyvX/Ye+R/+VGkf1d3Q89TCzz+Y/HMmJk+7qfxX8ASBwAAAADPFzZqAAAAAFAybNQAAAAAoGTYqAEAAABAybBRAwAAAICSYaMGAAAAACXDRg0AAAAASuaZ5qgdzR7K+nTi5ygMdLyM7Rz7OWhmZo+P/lzW97eP3dp69bocuxzIKemN/M+ubuusotpIB1M8ym7J+uu/95JbO8j94zIzO3qsp8fKeT+b4u0P9P8BNNo6A2V//5Ks7+35uT3tjs4huXZNZzDNberJVmT+PM1m+pxtbw1kfXDoj59OdH7KcV/np2xdOyfr7e6qW3uyr3MIn3dpIG8sz/1cr2ZTZ7DEsb4XMpEVZGZWiO+OAzlE1UpgiRcRSfns1z8nZma9Xk/W08wfv/14S3934Ng6bT8/8uorV+XYW7duyHoauF7L5/z7bDr1c4TMzJaWl2R9PNbj49h/FtWqOqtsNNJrU2H+9VL5bWZmRSBHrd/XQUWJyBJUEWIvilBmoczIC8WkBf6/PhbX3cwsj/1rG5mec9VCz5u84n93UQSyWlNdjwJzspr42Vidhn5/WQ7cx7sXr7i1g1Wd8/rk4WNZv3u0J+sLh/593tvdlWNPAnmim/M6c/hYZDEumX6Pbgbugfmpn5uXrZ+XY6eVwE2S6mzNVNxDc19B0CN/UQMAAACAkmGjBgAAAAAlw0YNAAAAAEqGjRoAAAAAlAwbNQAAAAAoGTZqAAAAAFAyz7Y9f/+JrA96224tC7QOPu7flvV87LdUNzObb/mtWocnX8qx7SXdVjTu+C34qw3dznRu5rccNTOL1/w21GZmiyt+G9m5ed2S9MFN3b4/Eu1UD3f0/wFM0n1ZX1vXLfQfbvkt9A/2A22mq7rF9aruvGv1un/eQm2UJxPd6vjJLb+lebuqD+y1d/2Wv2Zm/UD7/v0j/x6o1nWr4+ddt6sjHUKt6M8ydhJoeZwk/n2man/z2Xqum4gOyEOxAYH21nmmW+iPBv59WqnodsiDU7/Ns5lZlvlr1+LSohz7nd/7jqyfO7ci68rBgR8rYma2tKjbeh8e6vHDgf+cizp6TY5MX++KiHqIY702pTM9D5tVvW7Guf/d40BUw4sg17eamXruBJ5JQUVovGihHxgZjB0Qn1DkgViBwNoYB35XImIHZlX96twOxKJcTvz7Zbmr14CdjXVZf/xAt/e/ceCvjfdq+px+/bFefy6P9Tl93Gy7tUFgDZk/1vFEaep/dxyI0OmLddPMLAo8j+pVf64tzOt3i6fBX9QAAAAAoGTYqAEAAABAybBRAwAAAICSYaMGAAAAACXDRg0AAAAASoaNGgAAAACUDBs1AAAAACiZZ5qjNjr1c9LMzKJkz61VuzpDYb4VyK+6o/PGuisztzY7p7MjoqrOvdhYesutPdrS5+TkC5199eaFN2W90/GzQC5u6mybg0Bmxp3P/M8e9XSGSdLyc9DMzGpNnWuxtuGf8+1HOqNtkuucNQvkQ0UiN2ZuQedtXLmqM5z2vnzo1tKZzhnpHeo8ru0nfkabmdlEZE8tn1uQY593oayzVGSKxSKLzMys0dDXLZSj1uv5121uzs9oNAtOZStEBlUaGBs8Z6m/ppqZ1UUWUS2QUxQVOvNrPPTv8Vkg382m+lmSBU5qUfjnZTbTxx3F+ne//sY1Wf/s08/8754Gssyaes0eiyzSNDBZqoF7pFPXv7tW8Y8tH54xJ+x5kOjzk2fi/AdOTxS4z6NYf0BRiGsbWn8CxyZvtSjwdwaRg2ZmVgTC6dShRRV94NXAq3Vh/trYqOp3iLVAJtjqnM7eXTn2MyjXA8/5g+//pax/Fnj3mkZ+jtqnp/pd972FZVmfrfvvVtOh3j9UO/qc1mr6mnQ7/v6i09RjnwZ/UQMAAACAkmGjBgAAAAAlw0YNAAAAAEqGjRoAAAAAlAwbNQAAAAAoGTZqAAAAAFAybNQAAAAAoGSebY7a4Q1ZT+p+ntAk0pk9ta7OKjp/fUPWVb5NWtf72fxEZxn1dv3MsP6xzhMbPdF5Yh//5JasL8/5lziuduTY3/qOzp67fGXNrS2t6GyouVWdLdFc1tczjtfd2v7WFTl29/BLWc/rD2TdZlUxuCaH1lq6HonT0u3oeyDP/XwUM7N+X+dHpbFfbzR0zsjzLpQJVsi6zuM5PdXXJZSjpvKr4lDGkcpXMrNW08+1OTn1c/XMzNJUzyeVJxb6/I3zF+TYcaTPWVLx79FuIHvutK+v187uTuC7xZobyBO7f/++rF/c3NT1i5fc2pe3P5djOx39PGg0/N8VRfpVYjrWOUZ5puuxyJdrt/05/KIoAv+nHomctVDKXOg+LUJhaKIeJ/q4g9+di6OPdO5fKCCuCNQj8btCL85pEshirPnnpZrq3xWbrk/mdP1yw3+vO7ei18bbHX2v/fRPdc7a7XuP3dpRR+cRt9/WGZJLm/57X6els+Wqdf1+0wqsMYvimVKvBubpU+AvagAAAABQMmzUAAAAAKBk2KgBAAAAQMmwUQMAAACAkmGjBgAAAAAlw0YNAAAAAEqGjRoAAAAAlMwzzVFbb+qvG9b97ImK6VytoqL3nLVFnUc2Per6x7Urh9rR5wf6u/t+Ps3cZFmOTav6d02KqaznmZ+ZcbSjs2tOZ/qzX75yzj+umc5YOnyoz1nc1ye90fHPy5Ur78ixaxd0ZsbRWGe87e35OUv5VM/TpKbzVd758LI/NjuSY3MLZPKl+h6IxD0WxaEsnefb8YnODItE3k+9uiDH9no9WR+OdCbY4ryf4TIe6s/unQ5k/Sjxr/lgqOfT3Jy/ZpqZtVo6h7EjsmcOjg/l2Hpd32fVqp+j1ox0Js5CIGctjvQ9nBf+vRKFcu8KkdFoZjdufibrrZZY2yL9LJllur4wv+iPDWTqDUf6WTNJAzlhopym+jn1IhgFzt/xyYlbSwLZfbPZLPDtOutMfX6c6AypucC9VhGZhOGctMC9JqtmeeGPjwv9LhtaI+LYPy9JICetGlgj4sBrvfpdnUR/9vXX3pP1zZWrsv5ob88/rjxwTpt6HjfMz6itz+kctaShnyedts6Y7Lb8d8ZaYM1/GvxFDQAAAABKho0aAAAAAJQMGzUAAAAAKBk2agAAAABQMmzUAAAAAKBk2KgBAAAAQMk80/b851K/va+Z2eS836p195Fun737aEfW05ZugV2Z+u07461Mjm0c6tbEFot276luT9t+Rbe4Xr6qm8wm4nfZrj6n23f0Oc2O/Pbdq1d0O9Q41y1om5Pzsn544rcdr2YP5NjltTVZX196U9az8ZZbe7ilz1mzo6/n4oo/V9KxbiFbqQbawO7ruTI58ef5bByY48+5Tke33z068NsKj4a6BX6lqpfZ9UW9LtbF+Ebdb0lsZlat6SiKvQN/DVhc1LEDzab+7FxEGpiZXdjcdGsPHz6UY9MsMB/FrXC6q6M/VlZWZL1e1/EdsWhXPpnoNuvDwFw6PNSxJt2uH5mQZ/o5NgrERHQ6/vjpVLfID7Xvn051i/hInNOqauH+gqhUAi3bRb0QcRFP89lRpM+v+vw8D7TQD/TIl13uA+35Q3EUofb86nYJHnfgbyAV8cNCUVOZ6XspifV9brG/btdyvbYlif7u1VUdN3V+3V9b01w/L7JATISpmIiqfu+KE/0cjUJRD6EJcUb8RQ0AAAAASoaNGgAAAACUDBs1AAAAACgZNmoAAAAAUDJs1AAAAACgZNioAQAAAEDJsFEDAAAAgJKJftP9/wEAAAAA/3n4ixoAAAAAlAwbNQAAAAAoGTZqAAAAAFAybNQAAAAAoGTYqAEAAABAybBRAwAAAICSYaMGAAAAACXDRg0AAAAASoaNGgAAAACUDBs1AAAAACgZNmoAAAAAUDJs1AAAAACgZNioAQAAAEDJsFEDAAAAgJJhowYAAAAAJcNGDQAAAABKho0aAAAAAJQMGzUAAAAAKBk2agAAAABQMmzUAAAAAKBk2KgBAAAAQMmwUQMAAACAkmGjBgAAAAAl8/8BT+M3Inghzb8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9b90a4f128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the Data Images# Displ \n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "X = data_batch1[b\"data\"] \n",
    "X = X.reshape(10000, 3, 32, 32).transpose(0,2,3,1).astype(\"uint8\")\n",
    "\n",
    "fig = plt.figure(figsize=(15,10))\n",
    "fig.add_subplot(1,3,1)\n",
    "plt.imshow(X[6])\n",
    "plt.axis('off')\n",
    "fig.add_subplot(1,3,2)\n",
    "plt.imshow(X[120])\n",
    "plt.axis('off')\n",
    "fig.add_subplot(1,3,3)\n",
    "plt.imshow(X[360])\n",
    "plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Functions for Dealing With Data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_hot_encode(vec, vals=10):\n",
    "    '''\n",
    "    For use to one-hot encode the 10- possible labels\n",
    "    '''\n",
    "    n = len(vec)\n",
    "    out = np.zeros((n, vals))\n",
    "    out[range(n), vec] = 1\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CifarHelper():\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.i = 0\n",
    "        \n",
    "        # Grabs a list of all the data batches for training\n",
    "        self.all_train_batches = [data_batch1,data_batch2,data_batch3,data_batch4,data_batch5]\n",
    "        # Grabs a list of all the test batches (really just one batch)\n",
    "        self.test_batch = [test_batch]\n",
    "        \n",
    "        # Intialize some empty variables for later on\n",
    "        self.training_images = None\n",
    "        self.training_labels = None\n",
    "        \n",
    "        self.test_images = None\n",
    "        self.test_labels = None\n",
    "    \n",
    "    def set_up_images(self):\n",
    "        \n",
    "        print(\"Setting Up Training Images and Labels\")\n",
    "        \n",
    "        # Vertically stacks the training images\n",
    "        self.training_images = np.vstack([d[b\"data\"] for d in self.all_train_batches])\n",
    "        train_len = len(self.training_images)\n",
    "        \n",
    "        # Reshapes and normalizes training images\n",
    "        self.training_images = self.training_images.reshape(train_len,3,32,32).transpose(0,2,3,1)/255\n",
    "        # One hot Encodes the training labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
    "        self.training_labels = one_hot_encode(np.hstack([d[b\"labels\"] for d in self.all_train_batches]), 10)\n",
    "        \n",
    "        print(\"Setting Up Test Images and Labels\")\n",
    "        \n",
    "        # Vertically stacks the test images\n",
    "        self.test_images = np.vstack([d[b\"data\"] for d in self.test_batch])\n",
    "        test_len = len(self.test_images)\n",
    "        \n",
    "        # Reshapes and normalizes test images\n",
    "        self.test_images = self.test_images.reshape(test_len,3,32,32).transpose(0,2,3,1)/255\n",
    "        # One hot Encodes the test labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
    "        self.test_labels = one_hot_encode(np.hstack([d[b\"labels\"] for d in self.test_batch]), 10)\n",
    "\n",
    "        \n",
    "    def next_batch(self, batch_size):\n",
    "        # Note that the 100 dimension in the reshape call is set by an assumed batch size of 100\n",
    "        x = self.training_images[self.i:self.i+batch_size].reshape(100,32,32,3)\n",
    "        y = self.training_labels[self.i:self.i+batch_size]\n",
    "        self.i = (self.i + batch_size) % len(self.training_images)\n",
    "        return x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the AlexNet Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from alexnet import AlexNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#placeholder for input and dropout rate\n",
    "x = tf.placeholder(tf.float32, shape = [None, 32, 32, 3])\n",
    "y_true = tf.placeholder(tf.float32, shape = [None, 10])\n",
    "keep_prob = tf.placeholder(tf.float32)\n",
    "\n",
    "# Create the AlexNet model\n",
    "model = AlexNet(x = x, keep_prob = keep_prob, num_classes = 10)\n",
    "\n",
    "#define activation of last layer as score\n",
    "score = model.fc8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels = y_true, logits = score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The optimiser used in this implementation is different\n",
    "# to that used in the paper.\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate = 0.0001)\n",
    "train = optimizer.minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize all global variables\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graph Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting Up Training Images and Labels\n",
      "Setting Up Test Images and Labels\n",
      "EPOCH: 0.0\n",
      "ACCURACY \n",
      "0.1\n",
      "\n",
      "\n",
      "EPOCH: 1.0\n",
      "ACCURACY \n",
      "0.2389\n",
      "\n",
      "\n",
      "EPOCH: 2.0\n",
      "ACCURACY \n",
      "0.3206\n",
      "\n",
      "\n",
      "EPOCH: 3.0\n",
      "ACCURACY \n",
      "0.3345\n",
      "\n",
      "\n",
      "EPOCH: 4.0\n",
      "ACCURACY \n",
      "0.3412\n",
      "\n",
      "\n",
      "EPOCH: 5.0\n",
      "ACCURACY \n",
      "0.3572\n",
      "\n",
      "\n",
      "EPOCH: 6.0\n",
      "ACCURACY \n",
      "0.3953\n",
      "\n",
      "\n",
      "EPOCH: 7.0\n",
      "ACCURACY \n",
      "0.4241\n",
      "\n",
      "\n",
      "EPOCH: 8.0\n",
      "ACCURACY \n",
      "0.4486\n",
      "\n",
      "\n",
      "EPOCH: 9.0\n",
      "ACCURACY \n",
      "0.4566\n",
      "\n",
      "\n",
      "EPOCH: 10.0\n",
      "ACCURACY \n",
      "0.4748\n",
      "\n",
      "\n",
      "EPOCH: 11.0\n",
      "ACCURACY \n",
      "0.4869\n",
      "\n",
      "\n",
      "EPOCH: 12.0\n",
      "ACCURACY \n",
      "0.4965\n",
      "\n",
      "\n",
      "EPOCH: 13.0\n",
      "ACCURACY \n",
      "0.5043\n",
      "\n",
      "\n",
      "EPOCH: 14.0\n",
      "ACCURACY \n",
      "0.5127\n",
      "\n",
      "\n",
      "EPOCH: 15.0\n",
      "ACCURACY \n",
      "0.5177\n",
      "\n",
      "\n",
      "EPOCH: 16.0\n",
      "ACCURACY \n",
      "0.5243\n",
      "\n",
      "\n",
      "EPOCH: 17.0\n",
      "ACCURACY \n",
      "0.5345\n",
      "\n",
      "\n",
      "EPOCH: 18.0\n",
      "ACCURACY \n",
      "0.5394\n",
      "\n",
      "\n",
      "EPOCH: 19.0\n",
      "ACCURACY \n",
      "0.5436\n",
      "\n",
      "\n",
      "EPOCH: 20.0\n",
      "ACCURACY \n",
      "0.5468\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# steps = 10,000 will create 20 epochs.\n",
    "# There are a total of 50,000 images in the training set.\n",
    "# (10,000 * 100) / 50,000 = 20\n",
    "steps = 10001\n",
    "\n",
    "ch = CifarHelper()\n",
    "# pre-processes the data.\n",
    "ch.set_up_images()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    \n",
    "    sess.run(init)\n",
    "    \n",
    "    for i in range(steps):\n",
    "        \n",
    "        # get next batch of data.\n",
    "        batch = ch.next_batch(100)\n",
    "        # On training set.\n",
    "        sess.run(train, feed_dict = {x : batch[0], y_true : batch[1], keep_prob : 0.5})\n",
    "        \n",
    "        # Print accuracy after every epoch.\n",
    "        # 500 * 100 = 50,000 which is one complete batch of data.\n",
    "        if i%500 == 0:\n",
    "            \n",
    "            print(\"EPOCH: {}\".format(i / 500))\n",
    "            print(\"ACCURACY \")\n",
    "            \n",
    "            matches = tf.equal(tf.argmax(score, 1), tf.argmax(y_true, 1))\n",
    "            acc = tf.reduce_mean(tf.cast(matches, tf.float32))\n",
    "            \n",
    "            # On valid/test set.\n",
    "            print(sess.run(acc, feed_dict = {x : ch.test_images, y_true : ch.test_labels, keep_prob : 1.0}))\n",
    "            print('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}