{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CIFAR-10: Part 1\n",
    "In this two-part tutorial, we present an end-to-end example of training and using a convolutional neural network for a classic image recognition problem. We will use the CIFAR-10 benchmark dataset, which is a 10-class dataset consisting of 60,000 color images of size 32x32. We will use a .png version of the dataset to emulate the use of a custom dataset that you might find in the wild. The specific items that this tutorial will cover are as follows:\n",
    "\n",
    "**Part 1:**\n",
    "- Download dataset\n",
    "- Write images to lmdbs\n",
    "- Define and train a model with checkpoints\n",
    "- Save the trained model\n",
    "\n",
    "**Part 2:**\n",
    "- Load pre-trained model from Part 1\n",
    "- Run inference on testing lmdb\n",
    "- Continue training to improve test accuracy\n",
    "- Test the retrained model\n",
    "\n",
    "\n",
    "Let's start with some necessary imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.\n",
      "WARNING:root:Debug message: No module named caffe2_pybind11_state_gpu\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Necessities imported!\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "from __future__ import unicode_literals\n",
    "\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import lmdb\n",
    "import shutil\n",
    "from imageio import imread\n",
    "import caffe2.python.predictor.predictor_exporter as pe\n",
    "from caffe2.proto import caffe2_pb2\n",
    "from caffe2.python.predictor import mobile_exporter\n",
    "from caffe2.python import (\n",
    "    brew,\n",
    "    core,\n",
    "    model_helper,\n",
    "    net_drawer,\n",
    "    optimizer,\n",
    "    visualize,\n",
    "    workspace,\n",
    ")\n",
    "\n",
    "# If you would like to see some really detailed initializations,\n",
    "# you can change --caffe2_log_level=0 to --caffe2_log_level=-1\n",
    "core.GlobalInit(['caffe2', '--caffe2_log_level=0'])\n",
    "print(\"Necessities imported!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download and unpack dataset if necessary\n",
    "Now let's download the dataset from [Joseph Redmon](https://pjreddie.com/)'s CIFAR-10 dataset mirror and extract the data from the tarball. Note that this file is fairly large, so it may take a few minutes to download/unpack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import tarfile\n",
    "\n",
    "# Set paths and variables\n",
    "# data_folder is where the data is downloaded and unpacked\n",
    "data_folder = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10')\n",
    "# root_folder is where checkpoint files and .pb model definition files will be outputted\n",
    "root_folder = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_files', 'tutorial_cifar10')\n",
    "\n",
    "url = \"http://pjreddie.com/media/files/cifar.tgz\"   # url to data\n",
    "filename = url.split(\"/\")[-1]                       # download file name\n",
    "download_path = os.path.join(data_folder, filename) # path to extract data to\n",
    "\n",
    "# Create data_folder if not already there\n",
    "if not os.path.isdir(data_folder):\n",
    "    os.makedirs(data_folder)\n",
    "\n",
    "# If data does not already exist, download and extract\n",
    "if not os.path.exists(download_path.strip('.tgz')):\n",
    "    # Download data\n",
    "    r = requests.get(url, stream=True)\n",
    "    print(\"Downloading... {} to {}\".format(url, download_path))\n",
    "    open(download_path, 'wb').write(r.content)\n",
    "    print(\"Finished downloading...\")\n",
    "\n",
    "    # Unpack images from tgz file\n",
    "    print('Extracting images from tarball...')\n",
    "    tar = tarfile.open(download_path, 'r')\n",
    "    for item in tar:\n",
    "        tar.extract(item, data_folder)\n",
    "    print(\"Completed download and extraction!\")\n",
    "    \n",
    "else:\n",
    "    print(\"Image directory already exists. Moving on...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a peek at a few training images to get an idea of what we're dealing with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsEAAACTCAYAAABvaMShAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztvWmQJdl133dubm+vtat67+npnsFgBpjhgAIBEgCBYQg0KRKiZYZALaHNFmXTDpuSLYuWHJSClilK/qCFDn6wglKIJChLtCiZFiUxRBLAhDAAAjtm61m7p7url9q3t+Z6/aFq8vzP7fdqqrsH1VV85xcxMbfey8x3M/Pem9n3/O//GGstKYqiKIqiKMo44T3oCiiKoiiKoijKQaMvwYqiKIqiKMrYoS/BiqIoiqIoytihL8GKoiiKoijK2KEvwYqiKIqiKMrYoS/BiqIoiqIoytihL8FDMMb8ijHm5x90PRRF+YOPjjfKg8AYY40xjzzoeihHG2PMs8aYnxzx3TljTMcY47/Ttg8KfQlWlAeIMeaqMeaTD7oeiqIcPnR8UPbDYXy5JCKy1l631jattfmDrsso9CX4gDDGBA+6DoqijAc63vzBR++xotw/+hJMRMaYDxhjvmmMaRtjfoOIqvDdp4wx3zbGbBpjvmSMeQq+O2WM+dfGmBVjzFvGmJ+G737OGPObxphfN8ZsE9FfONCTUg4cY8xZY8y/2W0Pa8aYXzLGXDTGfG7371VjzD83xkztbv8ZIjpHRL+9GzL6mQd7BspBoOONsh+GjQ+7Eoa/aIy5TkSfM8Y8Y4y54exXzh4bY3xjzP9qjLm8296+YYw5O+S3PmaMWTDGPHMQ56YMxxjz1+FeXTLG/Be7n/+cMebXYbvzu20hMMb8HSL6fiL6pd128ku723zEGPM1Y8zW7v8/Avs/a4z5+d0xpmOM+W1jzOzu82l7d/vzsP3IY+1y0Rjz1d19/z9jzIxbzxHn+18ZY14xxmwYY/6jMeahd+lS7h9r7Vj/R0QREV0jov+RiEIi+uNElBLRzxPRB4homYg+TEQ+Ef15IrpKRBXa+QfEN4job+0e4wIRXSGiH9o97s/tHueP7W5be9Dnqv99R9uRT0TPE9E/JKIG7bzYfIyIHiGiH9xtM3NE9J+I6B/BfleJ6JMPuv7634G1Ex1v9L+7aS/l+EBE54nIEtGv7Y4xNSJ6hohu7LHPXyOiF4noMSIyRPRdRDS7+53dHZ9+mIgWiOhDD/p8x/0/Ivo0EZ3a7cN/goi6RHRyt3//Omz3dlsIdv9+loh+Er6fIaINIvqzRBQQ0Z/a/XsWtn+TiC4S0SQRXSKi14nok7vb/xoR/bO7ONZNInr/brv812/Xda96EtF/vluHx3eP+7NE9KWDvuY6E0z0vbTzMPpH1trUWvubRPS13e/+ayL6x9bar1hrc2vtrxJRvLvP9xDRnLX2b1trE2vtFSL6ZSL6k3DsL1trf8taW1hr+wd3SsoD4EO0M3j9NWtt11o7sNY+Z61901r7e9ba2Fq7QkT/gIg+8WCrqjxAdLxR7pef2x1j9nOPf5KIftZa+5rd4Xlr7Rp8/2ki+sdE9EestV/9jtRW2TfW2n9lrb2124d/g4jeoJ1ny93yo0T0hrX2M9bazFr7L4joVSL6o7DNP7PWXrbWbhHR7xDRZWvt71trMyL6V7Tzj/L9Husz1tqXrLVdIvqbRPQTby+G24OfIqK/a619Zfc3f4GInj7o2WDVFO28uNy0u/802eXa7v8fIqI/b4z5H+C7aHefnIhOGWM24TufiL4Afy98B+qrHE7OEtG13c5cYow5TkS/SDvhqhbt/At/4+CrpxwSdLxR7pe7uc9niejyHt//FSL6NWvtS/dXJeXdwBjz54jof6KdGVQioiYRHbuHQ50iHlfe5hoRnYa/l6DcH/J38y6OteB8F9I71/shIvpFY8zfh8/M7nHd3/uOoTPBRLeJ6LQxxsBn53b/v0BEf8daOwX/1Xf/JbRARG8537WstT8Cx8EHnfIHmwUiOjdE+/QLtNMOnrTWThDRn6Gdjv422kbGCx1vlLth2D3Fz7pEVH/7j93Ztzn4foF2Qt6j+DQR/TFjzF++n0oq98/uDOgvE9F/TztSgykieol2nhfiPhPRCWd3t53cop2XTOQc7cgW7pb9HOus811KRKvvcNwFIvpvnDGtZq390j3U8Z7Rl2CiLxNRRkQ/bYwJjTE/Thx++GUi+iljzIfNDg1jzI8aY1pE9FUiahtj/hdjTG13AcL7jTHf84DOQ3mwfJV2XnD+3m47qRpjPko7s78dItoyxpymHY0eskQ7+k5lPNDxRrkb3ml8eJ2IqrvtJKQdXWUFvv8nRPS/G2Me3W1TTxljZuH7W0T0h4noLxtj/tt3u/LKXdGgnZfZFSIiY8x/STs6WyKibxPRx82O7+4kEf0NZ1+3nfwHInqPMeZP7y6e+xNE9AQR/bt7qNd+jvVnjDFPGGPqRPS3ieg37Tvbov1fRPQ3jDHvIyIyxkwaYz59D/W7L8b+JdhamxDRj9POaup12hGj/5vd775ORH+JiH6JdkLYb+5uR7s3+FNE9DQRvUU7/+r5J7QjMlfGjN328EdpZ6HJdSK6QTtt6X8jou8moi0i+ve027aAv0tEP2t23AD+54OrsfIg0PFGuUvK8YF2FlEKdvWc/x3ttIWbtDNjiG4R/4CI/h8i+l0i2iaif0o7C+rwGNdp50X4r5tD6DU7LlhrLxHR36edfygvEdGTRPTF3e9+j4h+g4heoJ0Fsu7L7C8S0R/fdVn4P3d1358ior9KRGtE9DNE9Clr7TvNzg6r136O9Rki+hUiWqSdReE/Te+Atfb/JaL/g4j+pdlxtHmJiP7I3dbvfjFSmqYoiqIoiqIof/AZ+5lgRVEURVEUZfzQl2BFURRFURRl7NCXYEVRFEVRFGXs0JdgRVEURVEUZezQl2BFURRFURRl7DjQjHHPfuG3SiuKfr/HXxh0qJDWcgZe0z34A10tcG/rsQe9KQpxrGzAZZ+qZbk53eBtMv59YznrX6/PO3uhzAYYePx3FIa8HVSs0+Xz9Xy+7IVTxwKt9WD/Cvym5ydlOUmhXp68nWGF7SLzHK8SX8c//MyPo2n/oSQpuPJ4vWS+gYMB2x3+/uLiYlnudDpl+eLD58pyFnfFsbY2l8tyJeLz6rY5q+mN62+W5V6vXZZXV1bEsdY3OAmdhWuUJOnQzwnOI4i4zRIRFfDdq6+/VpZPn2I/9L/18//i0LebT//kXyxPpFZtlp/3+9x/PNn9KIR7itcBT3Z9Y7ssL69zAreg4gyn8Kfx+Qg58Y96PvfrEMYObFsmk5XMY05KiOMVGv1UqzAmVfmLO8abgr9LU2grwjSI92k0IjiuHAf7KbS7mMsFjJ1f+p3PHup28zO/+lJ55lhRA88VcsadUeMQtp8cL+hejkzwjDM4RwW/YS1cW7g3lnAbOb9VQAbbAp5XIXQAk3O/SLs81rRXZX6FlVtvlOUAho65c0+U5Ynjj5ZlH+qYO5fKs/yB73Gd/YA7zy98+uKhbjNERJ/57DfLm3p7k9v70jaXk1ze9yJ3Bp+3wXebEe88Hsl9A2ifIYw1rQrvf6zB13Smye8/eKTVbXgvI6IcxocKvE/0+pyxO8DxAcbA2QnM7UFUw/EQ+wC+t8CdzlIe59wuFoY8Dll4UcpgDPvUM9+3r3ajM8GKoiiKoijK2HGgM8GbGzzDtXDjSlm2lt/ec+qLfXL4LoAZVA/+1VjglCn+47mQ//JK+/wPg8hrleXpuamy3IUZ20GffzuFGZfC+WfJ1CT71U82+bhxj/8VuLa+VZabLf69wSAWx4pj/td4nvO/hCpVPrE6T2rRYMD/Yg8j/tcREZEn/onD125y4l5SkT84fJi58LzD8+82A/9sxXrhzBC2Wb6bO+QZf9KJefZ4AFGSwYDb0PraOm/fkbPKWcLHwn8Z48wf1lHWV17Tzja31UGX+2MllO3rsHNjeYn/sDxzbgtuT6nT/yjHmbXh4CSG7/E18UM5o+5X+HcwmTbOEPowExxB/82gbaSp03Jg5gQn/MIA+glUJeRJH6rVRJ4ECmCfzU2+70kyfBYGo2Du+eaxHLt5/8PTZ98Jkc8cZlypgFlWZ35p1EywaD84u7fHTLA4EkZIRRAH+jeMjThb7Tt1Ci20py73hdVbl8vyysLrZXlpgWd7NxaviWN11rlf+dAWH//Yj5XlD/3wGd4m4HYtZtSJCGMJ+JU703nY2ejwONIZQKQ2x7FY3ndsXxa+M/iMs8Ojd+ReH/gKIwg5tNsU6tJL+H0mgd/uJzISn8F4OIBIjycGBf69zgCeQ1aOB9UQnpcisALveDAe4enmuawXDj34vEsh8rlfjs7opCiKoiiKoijvEvoSrCiKoiiKoowdByqHwBBHv8cLStY3WHi/0VkS+2QFhxlCUOFjODeHxXR+AKFoI8O3JudQYCVgCUM7ni7LW1u86OX2La5LFHFMsdZgyQMR0e3bXK8ahowhFL20wlKQiYkZ2ESGh5KYzyVOOBRebfCU/+S0D9twfSsVN1zNx5qZOlmWjx2bpaNEMSJsRO/icok9Q5QY+sGfh5gOLiqKQcKAcgQMcRMR9WHBUK/DC9vW12+X5e1tlrv0e9wX0tRd4DS8LmIhFMQe/QDkNY6MptfleoEaSSwUPQrkFq/J8AWnzloVpx1AKBMlDLDICMN3hSOgwJCfCGHDPUE5BLaPPiw8yRN5XPz9rEjhc/htkC0EsDim1pD3Gs+rMcHjYygW33E5CPmRkaQyRIltMk8h/J8endA23g8L7Ue0Cmfcwe/wKwwZS5nFXgvjcB8oG5Q68P0MCr43cZflUusrC+Kwcy2+79ef/3JZ/vpzz5blzjov7u33eDzKMkfmAqvbJqZO8G8cmyvLUZUXm2O42jduWx4uIztKEhoiovUuSyA6CYz5KKNxFqXipRglncFHj8EFtU4bxL6Pi87wcZnCHx3o3z2QPvUG8hklFuPBLQlg4aIHg6gtQGYhhweKYKz0heEBb+h5wxf6ojSUiMgYvt4oM8kylUMoiqIoiqIoyjuiL8GKoiiKoijK2HGgcggDYW3f42nrTof9Um/delPsY8HHEH3qMCxewLp7WKxN9UjKFmzKvnW1gEM8qeUw88J1kGZs8GrpemOiLEdtDlETST+7qRb/Zq/NK/j7IG1IMv69XldO3ycDCC1AaCDq8fT/dh/CIgF/HjjR6jTm35kARwoMZRw5MOS7R8jMjlzbD9uM8Py9FzBknKDfKtTDXeGKIe9Ol9tKF3yG8XMM6RdOqAnDhwZ9F8F3OoWVvn7E28zNcUiTiKi9wb9/LX6rLC8vLtORAuLWHmhB5HV028modgChWm+EBMK5KT5aQkD7yqAdyFXh6PMKn/uyTlkO7csDqQL49s6dZMlVrc71QKkMEVEP2lqlyuMrxleLPoQbob49x1kjAdlDgbFQRzZxmPEhTothVtFK9hgrRo0j3j7HF4NaC5T8JTxWbN5mZ6Wl66+W5RuXXynLK9fZ35uIqALP25Xb/IzbBn9xA+1XSD7uqDvf53af9795g50mHk7BVabGz87QyvbnukXwF+88fh8mNsFJKgEJBKgO7shbEMHz2pgRfR+2N9AKPef5Jnx3LbpDgLsDyKoGKIEA+UaajZYuCdkCjpvwjCmgDfmOCbvngyxmhERG9p/R/uaj5GXom75fdCZYURRFURRFGTv0JVhRFEVRFEUZO/QlWFEURVEURRk7DlQcajGHOFi7DBLOkGVcFyaQiPigHyrg9R0z0gSQlcQ9FOpye22wpNoG7S7YVoUha+T6PdY4JY7lT60KWuM6W8OsLq3ysUCDiZo+8qS2JwfLoyTmulDI16sK+1Sqw61ziIiydLg2yNWmHnZG6aX2o/vd73H3RFjZoF4LdJ6oCU5Yp431zZzr3gW9b6fDOvMeZIzDDDg5WtG4pw46MAMtH88Q7bCmZtgmrzUxSUgHtOwZiNquviUzRx16YmjzqFHNsc/INiCssEAYaUa0taTge12rymxsPtiJWdC9WdDmJQlYQILFIWZ26/a4PRARmZDP5dg0r0FoTbCN4+QxHpMwk1y3K8euvAPWfqAHLKA9DVIeh1KwPssd+SDa9GWgA/aL+9PbHyTeKJ0mZofcc9xAmy/8FDXlwy0XieTjwILN3dI17nuf/7//YVleu8lZ3rpt1nejhpiIyELGOHxKCH07emBBe/WcB6nnczttNLidbiy8XJavXfpKWX7fh3+4LIe5qxPFygy3hzsKdAeY9ZavI1qkBXs8r4SkVzzjGKkDdrLPET4bhq9D6ePnoHeP0cbNqaIZYRmJnd+IdIZcvMMZEdc/gI7YBytX+Uwerg92/8Qqupaz+0FnghVFURRFUZSxQ1+CFUVRFEVRlLHjQOUQODuO+T8yCOPnTuwlgL89sBwq4GgG5BCYVS5w7D4qPlgFGd4uTWCbcHimm2aDt8dQAhFRnPEBNrbZVq3R4JCkB78XQ5jWr/LnREQ+hL9TyELTDFiaEdV5/yCCc0zltWs2ORNeFHFd3DDmYacH4WBPWIGNDn3smZVp2DZ7bC9D5JhBB9ptjjYxGCoanTGo1+tCGeQ2MUoghh/XuvdwRLYlY6CtgTXemXMPleVWhcPob9f6baoVbjfbfSdz1CEnx8x5KV670W1I/GVH/QH2PDCN4NrzJNCXaUSY3YMxqQb3YWKiydsE8mZPzfB3c3NsP+V5PF7EIDEbwG0bQDZDIqIAxktsa90O799pY//j7Q05Yxe0Owytu+H0w4yHkjuUMOBGdww7w8chmQFteNm1eUQb0QIsMjHLZ1hnCcz2JkvuPIthZcdCC+8HlAPIcBqABWkIVnHuuOVXWAJx8tQZ3h/2ef653y3LDz/yZFmeP3FaHCuE8QmvPfaLo0Be4DiN0pPRMhp0VCxGPH88qamB7Y27Ifw+7sN1SUdkr8NjedYdD4dnTRyZJ3EvNcIdD623jwRtbYQsZK8uh1Ilew+ZBnUmWFEURVEURRk79CVYURRFURRFGTsONuaAq09xGSyGyxy3hBBXWNOokDOueuRt6hDeISKyBYd+4g0OHdYhvIThwnqdQ8GYra7d4TA2EVGtxmHMKpTjWGbHeZso4mPFeSa/g9+JIg43hiHIPMApooBVvzaVx/LhdzL4naPmDnH1Kq+M9n1uLBU4vwhW1hMRxRADTkHvItabYtaZERloiKTzA4aRUF5QwDWdbHG4OgOniLgn5QQ9CDnHIH3J0NUAmhBG2907KIL10IdEvZocUp1qcHnp9m1xrCTna9xL+RwH8dFasY3XQZQhBJs7q4lFFHlECFtkecOV4AN5VzwfVkBHIBXAbEu4ehzkXk3I/lZrseRh51jg9tDe5O0iPlYO7a47gKxRMWi/SDqJ2ILLITwaTMaNEB1OMLsaEZGH8jPY7iiNNnhOUlKwR9sf4Wogo98QcvZGbUNkCh7TPJD8HTvG8qUf/fRPleXNJc7StrbwRlmOQilVieo8JtUbXA4jHjcxLJ6l6FIkMwM2Jzj7aAMykfahLa7cZqeIV7/++bJ88Sf+gjhWBa8FSs38I6ShISJMjmZEJlLYyJE8FPuQJRbSmmj07xNKvNDFwQ7/fITUy7jynD3kfLDT6IqJYw0vj9x+v8e9z8eSzgQriqIoiqIoY4e+BCuKoiiKoihjx8HKIXB1IybOIAhXOyuhM4IV3jC1n+EKbVABWAjlthrs9EBEFHf42I0GSyAKCG9iuB3D5X1YGR+FMvROsE+rxcftrW3CJvzvDQzd21yGrYxJYTu+PVivPOfwlIXkGjaTIaQwHL5iFSUUR4Hl5aWyjNcBV7b7bvjMonZgREAWV5iiQbnj/lHAqlYZLYdQNkgjGtDurlzhEOXSLZlsotflBBkiKQYkJLA5hNEhXGmdgJIVq2Whb0DoH/tPYLgNvnXlujjW8y++WpZXl7kN1+tSXnTYCSrgAJNxn8EV+W5XsBg+3MdviIi5m/gA2ocPUrAU2xr8HobJJ1osxeoknASBiGhrix1o0gFLs04cYzcYYyE5D5y778nxJh5A0qI+t8EQnB98kGkkKbZTGSY3YDuDsonQkSodZnx/uMsKYtyV+aNcIEZts8dqes8fIZWA3zz+8ONl+cKTHy3LCbjo+IWUvfgVHpNqDZbX+CCbiBOQzBXwjAnla0Idnp0Y+seEUl7MY9vLX/lsWf7Ixz8hjvXI+Uf4N0F25vn3lwjpoJHVHX6v78jjsI8BBqV5BY2QWZDIQyGlGTBAYbsdJdXZbwKqUe5Lo1xQ7oX73X+/6EywoiiKoiiKMnboS7CiKIqiKIoydhyoHMJAqMd6OM3PoZu0kCE2dJQIcHocjuXBqsfJKQ71VCCkSES0vcnHzsA8PwH3gOlpXu2KYUc0GHfDwlGNzcMxsYOQVkB6kARWaAcVDi0RSdlCCLnj8VgYJkApB7ogEBF5oAYJAlg9us+Qx2GhDQlIAjBRR8cMdzUxms5jwgjJ8NDlfsMwGO5G940OhPWylMOCm1ss6yCSspYs53tXgMOJdUOv+wFuL54LhsUXFliasbLChvtEMqyJ9UrvIS/7g6QCK9/RhCXPUIr17mWOsY4PgkVJDq7expAhJuuAECO27bQt+3Xc4fsTggyn2WA5xOraGu8PrjHGyH6SplzHwQDutWXZA449WB7EMvGGjw4IAbi4VN1kLIeXEM6vGBkylkhHCEhKAW0Lo8eFh2O5PJbvofQKw9/8edzm696EBCvH58+W5bX1FXHc0Oe+UIVkF+hGFIIUcX52DiosNUMxOI9gHfGex32W8GwsvlWWv/bsvxfHeuov/RU+VoDPy6P1jML3AyuSQox2AhFTkKNOF49LmAzFcXEQ26ELxHBPBjviGXFHNUZIAUexnyRV7tFEPhCsy76PdX/oTLCiKIqiKIoyduhLsKIoiqIoijJ2HKgcwoPQXQCh/rAKCTG2ZUgRJRToyuBHGArncq0OK5F9GX/AlbA4t48h9hRXP8MK5whCq64TQQ3kEGJBL8okIDwUQD0iCE0REfVi3i6HGG5RDL9VtTr8diaTc3TAfaA/YJmGOVo+5FSkw836LVwfdGcgclbFjgh5y5AQlz031DTCWLwYsRI2I5RJ8P1EOQKRNC+vVrl9dRNYmT1yFa78e1TkCOvV7rDTw+tvvFKWt7a2xT4dSAYzN8dJNeZOztJRwo6QF+TZO1/fPY8LbQAlD66ZfAbOHhUYozAJDi7ix0Q97Tb33Wwg200Ojg61Jic+WF/n+9aB5CvYWHo9KWFIE66zBwODhX4WgEtNFaQN7rXDxCwEbjjWPzoymghD8iOUMq5CyaLUZUQZHWY8IfGT1xCHZuOzVAH78eYW9+Osz/c5rE6W5WhKjodNuB9hhY/V73GbmZ1mOc2f+pOf5mM5iTd+5Vc/U5aTmPfHxFbNSZYV5mDB8s0vfU4c66Mf/YGy/OTT312W4+wopViR0plRI4o7ZovnDDpn4TgySpLjHExIZzBZhx0ugRhlUFI4fXo/0kkrkoOINyBnOy6jCsz9zf0w4rLsT7PhoDPBiqIoiqIoytihL8GKoiiKoijK2KEvwYqiKIqiKMrYcbAWaaB/ks4dkE3NsdOpw98hWsuArspY/hyTGOVONqh0wDqj6RnWN/b6rGuqVtlWzUCmJN8D3bKjGR30+EdRf9qa5Lr3Etb4VRus46vUpEVaPGA7sAFoYSs5WNzA9l7EF3L6+CQhy7eXy3Jh0Cbp6Gj0iIjyHLLiWbSiGZ4Rj0hIEqUmeIQOGIVFrhTQx/sNlnTVGt8J1HdhdkEDFWk0WCdHRFQHe6LONuv82ptbsBXa3YwQVZGThUoIo8BCL2M96NQUt//HH39CHOuVS5zlbmKC+8P5C+foKIGa/gyy8OHnbl+QtlTD+4n4FHTdQSSH0wj+xmxwQgIHetkErKc2Njb4uI5udLLONpCNJvf5xSW2usvATqna4LGjuEPnitZtoHcn7Fu8UwXarLs2IoX6xwVmH3v3bOi+01TASlI6XcGzaw+towAzE4quy9fWd0abAHTZ129zJsdrV6/y56++XpZXt7lPdy1rd7NAPke7YJ/Z73B50Odn1+QEj7NBhetea0qr0Sl4rh0/Nl+WX7jE9arV2Z+zgAfx6i0+DyKiz/3uvyvL5x59rCzbUGZ7PewkxaispJA1cp96VbQ9xecH2qUVjmBdWItBuxM9FI87wrotCpx5Ufgux/1xTEL7P1EeXUehkUdbQQ+3GX3BihEZXu/l3UZnghVFURRFUZSxQ1+CFUVRFEVRlLHjYC3SoCysZDy04JGWYZUah9/iPod+QrSqgkg4moT55IQ6ISRaa0GIEMLqaLVlwb/IjghtERElMVfAg8xwRZU3bEywBCKqQXjJCSni337I524hTJZBxqutDttb1erSIq02weExE0K45mipISiFzFRoL+SDnZFra1ZAuMY3w8MlGDmxGB5ybrAHbTWHbHDNabAkAgs9C6HHk2cvcp1IsrTI4c7NjQX+DQgfYsa2DEJuxvFpwjac55ARDfbJMENdxsedmeXwOhHRzDRLJbr99bIcx306SqDsATOjYSjNvyPz0nDbO7QBwnB4BLZQrUkpbTLg1liY4aE9lBQUOf8e1v3YzLTYh8B6rQtjTw9s0XLMkghDTJJI2RDKRKKAK+xaDr4NWki6BGD9hg5X+QFlfno3gCFFhKLlWOH2ZPwSn2v8cZqAzWPMdpVFIO3Hri4uluXnX7tUlm9c5/Fhe4G3ycFOMYYQc+rLOtoE7D7Bem3u3Emui8/t5/JbnFHyzEnIHkdEszP8LPvoR58uy29eZhnVzQ7L/8KIf68ayufdN7/MlmlPfPAjZfm7P/af0VFicpL7aA7NPcf3FLcfoCwGM7P1IZsu7JIVYN+auHabcF2hTcUpj9k5ZJrMoT2iJCty2nZtgiV81Sk+R4PZMTOQmkFfyB3ZUICvIHAtEnpnaUPhSKo2Nlkuhhl/A6c/7QedCVYURVEURVHGDn0JVhRFURRFUcaOA5VDuFPab4OZ4BqelEPkEMKNMYyEsgHwS0hinho3hXzH92GqHFfw+7AiMoFgQ5qWAAAgAElEQVQwgw9hgiAEOYTnhssh41zMYYbLl98qy9U6h0pnIg43V6py+t5C+BozMGF0k3I4dwhtdba6hFTFanUIt9jRIc3DSA6he8zSVuQYenTutcjsBuFvaDcjV5K6USuQHqAcIoEMXylk9VpdZleOSu1YWd7cltm6Xn7x5bLseT34BiQMkN0Mok4ibE9EVEBWJpRTCGkFZBREOYQb+m40eWV2P+awUxQd6HBx31iQhXhwU7Ft3NkCRrhsoDsAhC7REeIOtwS4xjnsg8MgRkhxHAphdX8tlKvzB7jSv8flBBpICvc9SXisS51MXAbD9zyUUKU6/F5n0P49z5WSwN+4SnzUyvlDSIDZ7cQKdkwpKfeRziwop+GLUCEM//IBloUTDNG1JZY9TJ3i54SNuC6d5VtcX/jpoMLPkqqTpa+foqQMsq1WwMUBPm9D1sitNXmsHMLyy2srZdmHh1QEsoccrt3UtJT2LC3yuaBTxHue+iBs5ciBDiEnjh8vyyl08AzakCu9inssZVy6ebUsL1+/WZY3VliO1oeMn/lAPutr4DhVA+efa7dYcpd0Qc6GskLIVlpz+vTMyTNl+eSFR8ryNDiGNCMu45jnOmYUIMVCOQfKGVIohyAxzB0ZVtzjTKw+yJaye3Ci0ZlgRVEURVEUZezQl2BFURRFURRl7DjgZBnDV1unEKatOMkyul2eNvdgBWQOYT3j4epEmA53smUYMBMPYFU3holx5W+1xtPxSYLbSMSKaQzBguuFB+4OcY9DGZ4TQq1AvSaaHOIIYbvI5xCWgbCX400twrYYds2dxBKHHqgvmiIU8Icp3KQHsB3uAyEaN5w7Cryu6LzQhvDS+jqvhu72ub43v/ZiWX71NZbHEBFtbd0uy+957FRZrtZAngMhbjwn44aYQeKCsiPIM0KWI02UQ2iqXgEbA5IruNMY+t/BDhf3jYVzNLDq2BN6BNkGLCTekUb1uIIZ+hVs4rogZMlwlw5MdFJACDmDtrW5wfKY+caMOC5KINpt3i4XfQDuFTgDeLk7ekF9wQ3HGBzHhq/SRgcLIiIfzfwLbHhHZ7zBc8VHCUobXAkNRrk9aDPCsQau5/oay6XeePUVcawAevkaSAUWr7Hzgl3jcLlIUAIDlTHyYXBmliUFm1s8Vk14HdiH72ct4n6wtc3bEBG9/sYNrtcSyznW2+CyFMnn+Nv0ExnWnphk94Frr3+9LH/7C7/FGz3zV4ce6zCB7wcoTUNJJQ3kub/0xS+U5W+DS8YayF3iLstYigE/b2wm3SEicNWaPclJjTxI/rUKCXgIxvyoDtKrmpSjBnM89mRr7EqyusZt+6FjLAWZmJIJu5CBB+MmuGj5MO54uA1IeCiWUsJHj4OkFGQTGxvrdLfoTLCiKIqiKIoyduhLsKIoiqIoijJ2HGh8EyUQIYT90R2i70x7+14FtoNjeRDiQ5kDSBDy3FlBj2FMKKdpDGUI6fh8LFwcHCeyjhbCkHVYKYnh48kWhwlyjJKRa14P7hTwo4Fww+CyBalAkaHDgEywYQiWftujFdbGcCqaaWMc0g3yCnN7YbrNZSmHGOEKQDLMjRHgBELLx06eLcvZMq/i/dYL3yjLV6+xAT0RUbfHoZvJWZa4XLjI0ohBD9oaSB4CxwzfE8vx4YsM5DYFtmdcye/KhqBvwfXeBsnHUQDvWwiDB45D+R1GKeCWAM4PmUX5FW9tUY7kuMbkEEIv4P4E2NYg/J5CEpx4wCHC27eXxHFRfpWlfO/Q2aAoYIyAVf/GkX+gvEe4HAh3h+Errl05BLpxFDRcrnbYQQVMATcaE/OIMYiI+j0ed3vb7JawtcGr/9dWWAJx89qbXL7F0gIiovVlljps3eLxojsAFwm4/yjVKuB5dfIEh6iJiALUQsV8rImAE2Gsrq6W5W9+gyV7zYZ0J1m4xo4DOCBWG+A0AdsbuHaYVGi30mUxhefqm89/ETY6/HKI7W24P8J9ia/EC1/6ktjnS//h35bl7jrf96AJUhIYgwxIQItEPutRjrG+wm1qdvZEWT57giUE2O7SHj+vYieZiX+M31vsCXY6SuA96bWFy2V5fou3n5+bF8eqwjtfAY4UJkAZFT5rQc4WSRetAJMagdSsMiLJz17oTLCiKIqiKIoyduhLsKIoiqIoijJ2PDA5BIbbAgiXeLmsUgpxlRBDUpi7GoqFxQTVcmofkwBgXQYgwahUarA9h4F6XQ4/xM4K6xzCmF0IjRG4WQQg+cDfroRSshG22BFiExJvpBAeLSAGFtYxbimvnYHwmG9g1ecRk0NgOFaGZkeb8KMTyX7KAmeVP0oNMpGxgounTp8uyy9e4pXcl6+8XpZDxwlkEPNq3yuXr5blGNrj3Dyv6vYgvJ5bee4ehJQM5mKH7QqQf0zOcmiMAicEBu2mBqu8r791nY4SIayAbrU4WU0GYcVBX0qbMPd8glIBkFZFcNwqhi6dBAVigT60NQ8lF/A5Nq0Y6nVjHVZ1E1ETxrEIQtCYyCXFNgsuKvZObwM+VoXPpVLlOsbQTsMQpBVO/ykwEQeci+uAc5iJfHQt4ut5++qVsvzKq6+Kfa5dZnnD2m12fNna5PsWQ3KDSoQyNSlFWl5gB5lTHsr0+FmwBQ2lBklV6hBibjSkO8P6Gktqbt+4WpY3V3nFfwD3NoZ76YpZ2ussmwhhTBrAOaI7RAOeaTVwPCIi2u6AUwW0az+TySAOOz147vvE92pthdvDt772RbFP2me5TAXGHePzfcygf1p47/Ccdxu0TcrBMWQJ7vV7pp4oyzPgfHUDnj1BCLJJIlqNuG1nk+wUMVnh8TSsw7FW+XwHbT4/IqKzpznxhgdtrcBzxOezDxIII593ax0ek+Tzju4anQlWFEVRFEVRxg59CVYURVEURVHGjoOVQ6A7Q4quBjzVnTkZH1JYXYkhAy+Bld/weS/jMKJ1In85rDwsLIeaClgSjAuep8HRYQoMpTc7MmSwvsYrQ3G19QBXa0Ns9NQsr7IMndWMCYTjugGHWMA0QiY9QIcEKw20qy2Wc/iQBKDIh6/2PqzgKvRCrMzeX+xjP3IIIdVxjuuBpADbRw+Snmxvc+izAW3l5Clefe1EyykBN4/NTW5Di4scbpyf57Zy7hxLGE6emBDHkiv7M/HN2+CnAziR20urhGxschir2eDfqU216ChRr0DIMOe+gT1uquWsVgfaPQ5rhhAmRHN5lKG4bgkVWNGcQqc1qHvArgjjo4H2mKQyZJ7DGImOECHsLxJXeLiyWo43gwFflwaE09F9B/sG9hk32YwFo3tUKlWrwxMnHEZCcF35wu//Tln+7X/5a2V52zHkTxIOzQbghFGt8HWPQGZ37Pj5slyJWAJARJTf5HbyQxd4Zf83brFsYAGkbWENnkURlzMnLpxmeA+5DaxvbsLn0K4ClGzI++z7w1150pjrjpKjPIMHljO21tE1CVb5r62t0FEigaRClHF7uPTCC2UZnT+IiAxKPlK+Rnkb+ztce5BEhqEzboE84uGHzpflT3zi+8ryc198rixfeIjle/0VvtbddSlhSG+zXGbrhZfL8gZIKpeb3M6n4J1jO5H3ugP39PT5R8pyfYqfa2KsgX2NI/+wMB6jHGKEwHFPdCZYURRFURRFGTv0JVhRFEVRFEUZO/QlWFEURVEURRk7DlgTDFmTPMxkxcWk3ycEs82IrE2gi0u7rK1JQBMceVJHkiRoMwZZsXCbAVgmdVkTWMesJo4epwCbs3qTtSoJnFfcY81ogdlPGlJn2QXrDxS4hAH/po9a4xyzqUlFTAIaon6Xj5smd6TJOtSg1tLegwfKKB0jHkuUC6ntzCEDkAWh+eYWawOn2qzpvXDx4bIcwb1+8YXnxXGjZW4rcBspibkuNxc401S/B5Y60UVxrPk5th6yQlePOlHuD8ugA/ZymQkuRg0q6PlmHHujw86546w37PXAUge6yezsJO5CnW6nLEeYpTHEfg1tBaXYjtVcgRphO9zGJwV9vg8dHrNVDu7Qd/L9yUBLGMG4UAUbt1oNrKccHW8GdYwqfI7L0J7RrGqvvpijDhT6EJ7LYSdZvVWWv/U51gQvXX2lLM9OSz3+sRmwe4K+h3abOWQ1HcDzyvPk+O9NTpXlF7e4LW7B+I+ZKrtbrOnNhSJSrvswkBoRrdCClO+NhVUD+Fxxl154kJUVy5i+DmuSgb3bNmiQiYg8OK86jC/VytGx1SMisim/NwzgHKtwfg8/9rjcB7K+yb4Da6cyvnY5tJvcSXWJGeMSyKa2srLG5VvctqvQNj/4zA+U5W98hXXDRES9hNtBI+b6Vpus443Apm+iwvewcYzbMhFRtQH3t8nWn80mW6+hlWCew/quwrVDhbU9mI3T3P16J50JVhRFURRFUcYOfQlWFEVRFEVRxo6DjVPBTLVFyxaY2vadbGw41V2pQ0gyhUw1PbYnwXA1WsEQEVXCGnwHmYE8nvJPINS4tcEhwcYcT9mn2/w5EdGE4VDEuUmwrwn4N9a2OGzZjnnKPw3Rvoio3edj+2B5Y8EiJIMQdQdC5NOz0rotB6usAcg8UBZylNlLGjHK1mmUBAIp3NBLATIaaMPpgLfr9/mePnzhbFm+do2zrC0uStuf9TWWyKQgQZCSDf7BrU0OR12/xpl5iIgmWyzB8KFto6WcD3KZ7iZIIKwMrW1D9iPMzPihc6foKPHBJx8ry+02ny/e92ZT9r8BhPy2O3xdCsNhzdVNvtd9iBr7rt0h2Ca14T7E8PsF9NFKlceOAmyWrBvhg/2zhH8jhJ9HCY8f8H2vVOQY0ajwTlGVyzFx6LI34Lpg23THkQzGTpRD4OeHnVe+weHgwQZLhupgRTYzNy32mW7xfbtylft7QWBLWfD9GMD1mJtnC0UioouPcVavk8fny/IE2BZe+9znynKK9mwg37kjmx/IG9AKrQ7PVN9neythZ1rIcRL7Embw9H3M6ArAH0JuSESrKyz3OgNts4iPmGSvw7KDhdfZSuzkMW4rj77/R+ROdZZi4f0JYBypwEtTFWwW81z2KZScdDsso1m8zhkIZx95qiy3TnH2tqd+4AfL8tkPfb847quvvlaWV1b53cTDTIVgcVYBeWdUl/Z/QZXbV9vj/Ys+jxXVCti6EsiMUtlucmhH4jlu7n5eV2eCFUVRFEVRlLFDX4IVRVEURVGUseNA5RCehexKKYTLYnR9kGGcRo1DTRZkEyGsdO/BPimusC5kSCUC2UK9zuFNdEuICw4vbqYcVpj2OXRx6jSHqYiIwj6HqvyEQwYbyzf4txscMpibO1mWX192MrTAKlEPwh+NBodVkoTPcbvDYZF+X4bxW5Mcmpif5zq3JuTq5sPOqBXpbsgPGS11KIaW5TbyOhY5hKEy3idOuT1vg0RmE1Zsv3KJw0lX3lwQx0WJioEVvRBVpDSF3wP5xeqKXGW9AZl+WhDix2yMRcxtJYQV+/UJ6ZAQ1jhU1Zjh706ePElHiQvnOdScpNwXUDIVhnIeIIXUjFnO32UgJdlq831b24IV3k7GOMzGFs9wmG91neUUbZBy5SARixO4PzUp2QggY5eIO4u+Aa4PIXxunTAq6DmyAZdTOPeqz2NlnKE0yOljML6bgsuDtgxlHma+8fxXynIKmUhRB9XtoF8G0fY6h8I7HW4PkxAytnDdBiCdWluUY8JHP/zBsvypH+Hw+W/9W3aqQAkMZkIcDFCeIsdGkSkT2k+lwv1iaopX86O0wpVD9EAuhWNYEKJrBEq6hteDiKiA/ra2xO4FMxX5jD3sZHAefXCYiRsgN1m+KveB8aIGWRU9zLBYZUmBrcMY5vz+zDHOLHriFMvWHnvivWU56XO9Jqf5fSQEp4Z5R0J6+jGWUKwsLpXl1197tSxfvsqZ8LrL3BdCcKBw/66C9KsBMhiUZzVh3Gs5Y2ATHLnwcZ2NeKbvhc4EK4qiKIqiKGOHvgQriqIoiqIoY8eByiGmJnjavQVT+xGE28gxc+9h6AdW6dfAfLySc+jGS3hufKImQy8zIayK9XhVbRFzmACd9AddDjkvrvLnzZMyVDM/x6GFeJOlFf0NDlpEEIaMIRy2PZCSDS/g0EAVEi0YWCWKl6jV4lXcYUWGOnEleLPJ13tiQhq0H3ZGySE8b3//hsN9Rskh5DauHAKcREDGE8MK5s0NTpwxMcErnleXF8tyEstEMAQuDsIgHYJdOdQrhW3WN6HNEtHtRXaa8E5xeyRICpMNuD31B7x/N5VtcGKKQ3DVCQ5bBUcn5wEREfkB9wdQPwk5hGfck+Lr1WjwTtjWJie4X80f4/6XOXIIdE8w4O6ysQUh4HVeaX9zgdtK0YRwYV2OY2vgWlOAPCfFBEAgqfHgxqGx/c53IFsAacS5eXY42Y5ZavP6VQ6DWke65sGcSmhhTM5Gy5YOG3PHeGxEN5Yr4OiATi5EUrLXaLLUrCC8tjzmG+hvGyscYiYiun2D3SWeffazZfnSyy+U5RDcProggfDgpoeRDB+jU0kAbbkGdd+EJA+YVKXIZYgZXSAmJ1kulUBiBw+2CcEBqV5jWRARUQDtb6YBx63LdnrYKUB+0oKEJ6fPcD+aasokW6+88M2y/OYSt4MQZDgGrqMHn3u+fPbhPpgMJUSpQRUS6MD7wIlzj5TlmeNS8nZull2xHj/B233kfXxeV6+ypOezn/9CWV5clm5ISY/HpyQAJ5wqvI+AHCLz+NnXM3Js3YTx7cxDD/E+zhi8H3QmWFEURVEURRk79CVYURRFURRFGTsONMBpIaQfg9kzhqVTJzRrMgixZRDiyTi8czzgEEujyuGZRiDzjzdgBX57g0PW0TaHqmoFhw9wZf5glX/val+uzO+AvCCFlaGzMxwyoJDDOzdXOQSaF9K8ftDj8FYl5O8MnIsHhtCtKfjtXDpNdAf89yoYv5+Yv0hHiVGyhb2SZSC4Ink/cgh3lT/+ib7/mO99dRnCWeC8EPogwanJ9tjpsTwihbB2gW4nWF+U6qQyRLkGodtGE1YUF1zh9jbUd5Pr+/QHHxXHemj+XFlGg3TjSfPzw46QmEDyCM/wODIYyJX+BdzsKMJkB3wsTOCD21Q9Ge40TQj9wr2bmuT9z5zkcOPFcyyzSsDJYwmSqhARff5ZdjDodYcb6KfoIFPjsatWkdKEWgjnCJKtJ9//nrK8BokAbi69UZYHiewnGbhQWBhrg+DoyCFe/ta3y/IauHgYkJQMBlIOkSeQJIS4H2bgkIEGC9j+ikSOYc99lX//q99+iY8LEowMZAetKW4/j76Hw9XGmd9aW+XnXQyODq0WPz/W11nShc8YcsbZi4/w7zRhrLm1yAl8Qkj0dPo0uxU8/dQHxLHefP2VsnwCcpDY5Ggly1i+cqksr99geUD79ptlOWo0xT6nzvI427nGDgvYD0PD5chyP/YdOYRw/wDplQdS03WQS1UgkcX2Kt/39iaXiYgefpilBs988pNl+czZ02V5xuNxa97/nrL82puXxbGWV3gcsSCBWFrj95RvP8cSoAjkELVIvqour/Gxfuwn/nRZrk/IRDb7QWeCFUVRFEVRlLFDX4IVRVEURVGUseNA5RCdmENF/Rxyqfd5mr9wjNWjnKfzm5DP/gLkXD8+XYcyyyGynjzWW1euluUqSA3SPoeKJuZ4da/tcLi632Y5QVSX+d5j+Jkm5AM/Nv1wWd6AldeBgbDAQK7ytx7fkh64D2SQ+MMSXy8v4HDoxLRceVub4rAIGnB0u45LwSHHlSfcLRgq2o9TRO6shs5A9pBm2dDP222UyKCTAId9qjUpfenBiu085vuYZOgagQkQIBmBkdKKHiRduL3EIa1el+UM211uqM0p7kvH558Ux1q9wee1fQtkBH2ZVOOws93jvmUwUU6D+7ibaj6AMD7Kt3CFO0H7yBIu+74cTg2ssEcJhQ8SmckG7zM/y6E8THwwMydlKC9f4hDywlW+1wmE5QlC8Vtb4EZjuT0QEZ2c5980huu1cJNlD3Mg2fjAkyydWVricYyIqA/jOLoLpImUDxxmCoIV+CCzm5qG1f8VRxZUgSRQEMJttaaGlgna2HYb3I+IqAoJBc6e5pX6FhL2vPD8t8ryiRluy4OE23vbcY+ZanGdK3OcWKFW488xUVMMiV6sM1dWa8E4AKv263VwNqpAwh0Y9xoN6fpQR8eCGv/OH/rIx+gocenLv1+WJyHZ0Fu3OGFW4UnHjvypp8tyAyQQmIwEDVgGYgzZQw7hoXSSj+tZGLfgXr915Ytl+cbl18Vxv9Lg9nhz4VpZ/sFPfaosH5+G97Kzx8tyJZIyqK0OSyj8OrfbHORBCy9/vSxffoUlJhNN6WhVqfFvrq+wDKca7k8iiehMsKIoiqIoijJ26EuwoiiKoiiKMnboS7CiKIqiKIoydhyoJjgDOxgvZb1IPWdNSNaXOpIKiPbqoIOZBl3fMchqlXfYTigdSC1pA3RR/TZoQCFrVALWLNayDmt+jjUpp0+x7oWIaADa3dOnwRbNcr3am6zFi0FTWORSt1yHDD4+sXY3KlDzw79XjfiazDp6q7DG5zUJmYwq/tH6t88oK7O9kJYx72yRJjXBjvWT0AEPL2OWuS3IvFSpsHZpalJqavugA26Dfh2Pi+kBMQuTcS5DH+yq8nW2nBn0WV+WBrzT/ATr2t+6LNvgm8+z/v3MKW73IdiwHQU6PdbCVsGeLoHr65O8kOjmhRpZLHug6Qyh//mB7FfFiCyAcQL3BG51BFm+fKjI9IzU+j/+xPmyvL3J5zgzw/dqANkJt7f5flYiqSWfmeXxCoeFwYDH0UrIx33fY7w9agGJiHp9bkeYtarfPzp2V6ce4jE0WOJ7sLnG5/ZjP/QRsc/F9z9elr/1Imd2u73ItldRlTW2uAYm60vt7iPn2EJsdpp1xLdu8bGaoKMlzCK5weNOnjn6dGh/WYctOsOQj9UC3aUtcF2COBR1utzmQtAEV0JpEVjuD9Z9/Z60JJydYU16EPF4ODUrs7IedkK4vm24vtMznL3zwiOPiX1u3AJLOegvmN1PDvSYLVX2YyPGLejI8G6D6x26HX4fWbvJWQrRspGI6Pgptrd76zJni/z8Z7m+H/5DrG2mczw+1BwdbzTJz5wBcf0DyCL48R/+0bI8Mc1t4H2Pv18ca3WTn3HXr7J2+OaV52Grv0n74Wi9DSmKoiiKoijKu4C+BCuKoiiKoihjx4HKIXyYAo9yLufrkO0qldP8tQpIFSAb282bHB5KOzw1P+FD1iTHWWsbZAvXlxa5LiAViGGbFmTDqYJ9y/YWhzuIpPVUYrle9QbLJrYhxHHsBFvUtGakhCFOeJq/DRZPEVhtZZjVqsXn3o1laK26yd/NQKaoyLHXOuzci0UaSiA8tKoyw7NX4W8IOQLtJaEYXq9Oh+/DAKzxWq0JuR2ERdc2uU2hnZcHtlsoh7BWnscAMowVGJY3bE+UJPx7zSaHpsJAhq18j223pmf4N7PBNTpK5JhdEK2G4DrUA2lblIF1IzYVlEBgdj+CLHFu2DhJOLw76GN2QLBL9Pm4jSbXt1Ll+16TVaTJCb6n09P85Qc+wGHJNOdzvAzWkLWKtOm7eIHDl6srK7y/x3Ws1bhek02wtPLB9otkBkQfwru2ONDHzH1Rr/P9n5/lMG17lcP4p89eEPs8doFDtUu3uR+/+eaVsrwNWfdC4nv+8AkprTs5w8+c5gSXBwOWUlkYg24v8nOsPsH3Y/4E23MSEdUjHqtuXOGw9toqWH+CrVkEGbowwxwREaVg7Qgamk6HrxGONY0GWsrJTjJ/4kRZRhfC5P5cMQ+cj3/iE2X5ueeeK8uzsyyHmHDkcIO3rpblLtqioY0njmGwr/sc8+A++GiR5vOFhCGM0gH31QqMCZgNkEg+y6bgHeT1V17m8kssQZidYTnF+UffK441f4oz5NWgrW7De93iDX7GtEDeefaktKVdBLu2lWtcnmzd/VijM8GKoiiKoijK2KEvwYqiKIqiKMrYcaBxqgDC0gNYYdqqg7uDJ1ccZymHpwIIES6vcxgngLCcV+eYipv56zaE+zY6LDuYO81hignIXlKvwLEgRL3dd4/Lx1oHacTUDIdDKw1eBdtswEpQyARERBTWoAwhDlvnYMjaCq/c7kI4tupLmcOxiH+zu83hqUkn4dFhB6UKKG3AssuoLHHIfmUO+3GnwG0GIH2JByD1qctw2OQEyyOiiNtmr89tIgi4zRdwTlnqxN6hXh6sDrY5rAoHR5ZXX+KsY8kZuar79FleEdyArIMZHZ3MX0TSbSFOuF/GkLmyPifDbLgyO4aQblFwx8xg5XwVxpvCynaTQtg4Sfme+iGH2TPI4jTAhG8p/0aSSflVHcbBx9/7UFl+7L2cYcyH0PKZ8xyijHwph2hWOMTZbrMMJsv4ekURyDQgI1NUlzoNzE5VQMg/CORvHmZsxn0nhOxxFx8+U5Zfv/yi2Kfd4Yyjm1u86v489KNqyLKTEFb81yP5vKvVQG4Cl21qElxaQDeA49y5s+fL8oXznNmPiCgd8PhS9FmCcX2B5XsFSIE64HAwNSHr2KhzmHoLMrwmMbRlyHjYB9eQvJDj1mabn50RZMvL7NGR0BDJe/LQQ9wn++DO0+vKflzA+IDvOSibsDCWF3vIGAsYR6BIMFTJZxTcH8waWLiPFZgnHYBkIwaZBDp+oAvW4hK/oxERZfAsqre4PeM4vXiD22MXjvX1z/+uONYApFfHT7CLxNlTUga0H3QmWFEURVEURRk79CVYURRFURRFGTsONOYQxzxt7kFYzUJULXYirmEdE1xAaCHlEEs7wVXVHKrJE2nub0I+3dk5DjkEdQg59CAZAjhVVGFl5KvX5TT/0gqHOR5//HRZ3li7VZaniUOl6ytcr76VK2/9BtexNclhqG6H9/FhdXp3lQ3S/aZ0H+jCEtuJFqwYjY6WHgLDOANSKhQAAA1YSURBVKPcHVxQtiBW245IiiHLMiaE3+1HDoGrqbcgmYErh5ib5zDO5WsLZRllHiiHiNHA3qmGsbxPAqF0H8zS58CYvg6JVLJUhunaHQ7TtdscBr6xfLSWbAcQxu9uc/hue4tDaaePnxD7pNnwxCqWOFwJeWvIViC5jtNuMMQZggtFBsdNMt6mn0ByHAiLu0HQaoPlFO99gp1mqhHvE4OcoYL2Ek4dfXANOHGKZWUr6+BmgFMl0AZ9T/bFRpXrhYlKPHN02o0fcEh+G8K/tyGxwc3rV8Q+KOert7gc+Hw/6hFrG1pN3mbgsxyAiCgnHgcwgU4MbkppD5JdQCj5Y9/7PbzNQCYo+f3f4xX8x46xY0G9zvdsY4PDz3hvXXnY+hq3jQTqFUBdQnio12r8TE4S+bzrQ1ibPHxe72+cPyx0Qd45CXIGHEMCR66YxXwtuuCsUYXrSOD0kEDfc2cvjZAJ+kPLIfx+YlA+we88xjnyxQvsFnHrNksVlha5PxQZtw/cv7PF7yZE8rkYgSQIx7N4m6WefZDKTLSkg9GpE9yGJ6f5vWd7e4vuFp0JVhRFURRFUcYOfQlWFEVRFEVRxo4DlUNUwG0hgtBdgisbfRkGsbBw3Z+ApBgBT6E3mlCGnOMrizfEsVa3OCTamOZ9Nlc5vHMs4mn3aoXlCJnhkM56W065DyD0WIOQYBPOMcOV4l0OBXSc3PH9ZXCLgLvTbHFdmhGH09qgHwmMXK1tcRUxOl1YZwnoIQflCCiHuBdpxCgJxF7uEHaEiwQeFxOYJDnfk80tDu+0WlIOceIUrOaH8KPvD3fAsHBccs7dQNgLI96YbKPZ4rb53U8/XpaLTNbr0vMconztVW6fyxMyscthx/f5QtRhqT0sgidj5BBoPHTKgPsOchN0nV+H8JvvhHArsJob712nx2PBBoT8VjfYneE4tI1jsyx5ICKqhlxHlHjFGco3YCwAOUKaQ/iZiLbAEaICsrBpy+NgDsk9BhCSjZxEI0UGTiIJOkV06aiwuMrX46GzF8vywg12gLh8XT5XMByMY4Uv3Gv480nohzNT0nkhAZ3T6gbLHr73+76/LL/niSfK8suvXirL1y+/VJZbjjSuNcXhY+NBIhOQVU2AW83p0yzre/PyZXGszTZIXQJui0HE53vhEU6M8PTT7yvLk/CsJiIKILnC+97H2z32mEy0cNjBpBJYXgPpCLooEBF1wYEDywNwlMjRcaZAiYv77IOEUCLBEpcD8SzhY01P8fjf68rxoQP3OoPkPzEkuNiGfYzHv2eM1OwF8IxaucXSisUUxhdw58EERVFNOszk8J64BfKgbizrvx90JlhRFEVRFEUZO/QlWFEURVEURRk7DlQO4YU8HW4CeP+OeGq7uyVXy5qUt2tMcrim4fH0eL8HBs3sCU5L6xzCIiIqMDd6nVdC1+s8HT8NLvM+rGq9gcb5kbSwaE2zPMFAKHtykn/j2jWe/p8EU/BWTa567K+x80QIoaYZDIliDAtCroNYJt4YWL6W87C6PXccKQ47o+QQ+2WUhGGUHMJ1gMBV/xj6zODzFELGHQhn4Yr/SlV2twLCWxg68uAcC5RvgNvAHXWEsJcBR4kUzNa3YMV4Bv3nTgkSbzdoc3m9LdvqYQdVLRGszp+agmQVqVxFX6tx30zhOyFLgcuVgMSqEsqkIznc7iTmY8UQVsRV5Vi+evVqWR6ksl+fPQuJF2AYxXCrCMXDonTPkxKGfp9DiQU4VaBJfxdW8GcggfCqjm9FMdxZI43lNT7MnDnO4fkZyCr02CPny/J0S8qCVlZZ8rQB7SGGsaKC7Qq67tKWfJb4EDJutljCcP7hC2W5Cg5I195iqcLtmyzTmHrve8Rxz59jec2lS2+U5fV1XsGPY0qtwSHyOtSDiGgCJIePPMrJCT749HeV5e96kqUN05Bsw5DjTgLPuGoVnqP3MM4/SHB8QKlnC1wN2o5zwcwsX9cpkIVg38HxKQNHLHzeEBFl8LeU/KWwD2bRgOdVh691ry8lG1egfcUD/q4HzilJHyUId3/fDMrGxOd8rDWQKRER+SDvbDS4n7aaUl60H3QmWFEURVEURRk79CVYURRFURRFGTv0JVhRFEVRFEUZOw5UE9xJQStJrG/pe6BRrUorjEqF9R61OmuCvQFmOeFyFWzU5uusgyIiikFw0oBMORNgQ9IEjVIC+rfOTdb0zp89Lo7bAu1wY5Y1QLNzrOPdBB2gCXj7aiRvwXGwz+mADUkPslzlKHZMWTdjnWx7BFn50HaL6GhZpGWZe2I7jMrettd2UhM8KnucvD4FaoJhu0zog7mObbDIqVS4Pc/NS6srP2AtFGZuWlphnV4GmuIU6mEceyqRJQi1qaDtjKHZXFlgvbwheawENVoht9vcxHSUsPBvfNTLJjH3Ga+QOt4arBsYlSkwRS05CjwDqYfDZpTAPmhvh/c9ALs0Ao1h6mbs6sB20K9RM452RqhtnmzJbJE5WHJloBlEq6MCrmMBgujcyvNNYU0CXq9q7ehY652GbGqYge/4LGtkPcdmbm6G136sbfC9WbjFC1RSAls76F/u0FaBdTNJzOPI4iI/fx57jPW+rSbfz80t1ncvr8j1MGub/N3GJtcRdaIxrHvp9fmZ/PgTbKdIRPTxj39vWX7iCa7LLGRVRTsusYaF3DEb2i/0Ec87WvNzqAMOwZq0CRrV48fle4Ow3kSbWLhEOej0U8i2565lwHUGMdiMJfD5ADK2pbDOoNfl97JbtzljIZG0zdva5OdSArZmYSTHUD4R+acByzN8XgXwzuVDVju8jqiDJyKqwbiJ6ziiUXXZg6PV0hRFURRFURTlXUBfghVFURRFUZSx40DlEN2Yp+PbCU/nN45xmGBm5ozYpxFyiOX4PFuzmD7vH4AcYnqep+8rExymIiIaQNanrU22K6lDmKACFkIphKbef5ylFUEoJRunIMyxtckZYiqQzSkPuV65hexeTsyggL8NxEUiCD0mEO7wwRrLDZsWYCNSg+x3uM9R4F4s0kZtJ63QhmeCwzLRHhnjiuFyCgtZtXrQtq5ek6GmqRn+rttH+xsIt8O/UwOQOaC1EBGRDxZgIVjwoYUehqA22yDPMdKCKyfe34Yc+g3o7kNNDxToMzHaw4HtYl6V9xpDiVhGEhi7UsjYWIlkyA7bYLfPxzLQx1EucwwskyZmpvlAFSlXQRtGzFQYYvY4qHsbwp1uv6hHEJaENpQk0J5xTILyYIPDo0REOUjUPMjWF9WPTrt578Mc3t/e5ut28imwGHtFXsM22HrOz7Bs4oNPP1WWWzP87OoP+N6sb0nbrDbY0W1usPVaFewVMftha4LlEAuQnfLl166K46LEC8PtU9NszfX44yx7eOYHPl6WLzzM2d+IiKan+Jnse3AwGE9zoc0DyR5JPKjMvWQDPSyg5GjUsyQI3OyUMAeZD5cuRiCXrFVRUuNIr4RMb7hdGkojRm3f70upD54XShKOnzhRlkdKV/aQQxiQ6YXwPjXKas79jVFtZb8SSURnghVFURRFUZSxQ1+CFUVRFEVRlLHjQOPioc8r+k4cf7Qsm/BiWW5EdbFP3efQ7EQNVlJDqDOAkEyjxdv7zkpBC6vxGzUOQzUhzFzxeP9eymGrZsFhgiKXU+7HjvGq/5lJXtGb5bz/NEQ3YWElZVaGYzE7VAThkwA2w2wxAZwTOSESC+mk+l0+bq0yQUcJDN0gGGraK1wyKkvcyLKzZFtkiYMy3oc0He5g0Qc5xGtvvCm+8wLO8JTA7iE4FHgBhoogW5cjaUFHiCACOQRIIAzsn+d8vYzvZK0CVxHj8/lO1p0MYYecQR8dEjAsiddUnhO2qWTkimtcsY1hTGc4hayY2MstrPjGdoeZj3D1c1zI+1MDuUuEq6nhXDJoDyHINBKnnUYwfoQQho1h7PNgrsQHeQ3KdoikPCjp8zjoRUen3Zw5yRnQLEd8qVLl+/HwhYu4ixgf6uCEUUFXDLhPYjxyxv8Msu5lGTh3wH3b3OBn19o6Z9LC7G+drgxrFyBJOPMQSw4/9tGPlOWPfh+7PkxPwTPCOuMv1BmfOGgWMsrtwM0oZkZkGLuXsPaDBMd5GlH31HF0wGfWqGeZDxICvA/u9RklwbAj5Cb7dWHA/UfVF9lLxiLrjNu983Hd8303pTM6E6woiqIoiqKMHfoSrCiKoiiKoowd5qiFHRRFURRFURTlftGZYEVRFEVRFGXs0JdgRVEURVEUZezQl2BFURRFURRl7NCXYEVRFEVRFGXs0JdgRVEURVEUZezQl2BFURRFURRl7NCXYEVRFEVRFGXs0JdgRVEURVEUZezQl2BFURRFURRl7NCXYEVRFEVRFGXs0JdgRVEURVEUZezQl2BFURRFURRl7NCXYEVRFEVRFGXs0JdgRVEURVEUZezQl2BFURRFURRl7NCXYEVRFEVRFGXs0JdgRVEURVEUZezQl2BFURRFURRl7NCXYEVRFEVRFGXs0JdgRVEURVEUZezQl2BFURRFURRl7NCXYEVRFEVRFGXs0JdgRVEURVEUZez4/wHfLLhRSyqmbwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import glob\n",
    "\n",
    "# Grab 5 image paths from training set to display\n",
    "sample_imgs = glob.glob(os.path.join(data_folder, \"cifar\", \"train\") + '/*.png')[:5]\n",
    "\n",
    "# Plot images\n",
    "f, ax = plt.subplots(1, 5, figsize=(10,10))\n",
    "plt.tight_layout()\n",
    "for i in range(5):\n",
    "    ax[i].set_title(sample_imgs[i].split(\"_\")[-1].split(\".\")[0])\n",
    "    ax[i].axis('off')\n",
    "    ax[i].imshow(imread(sample_imgs[i]).astype(np.uint8)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create label files and write LMDBs\n",
    "Now that we have our data, we need to write LMDBs for training, validation, and testing. To separate what images we want in each category, we will employ a similar technique to what was often used in the original Caffe framework: creating label files.\n",
    "\n",
    "Label files are text files that map each .png image to its class.\n",
    "\n",
    "    /path/to/im1.png 7\n",
    "    /path/to/im2.png 3\n",
    "    /path/to/im3.png 5\n",
    "    /path/to/im4.png 0\n",
    "    ...\n",
    "    \n",
    "The process of creating these label files will likely be different for every dataset you will encounter. It really depends on how the data is labeled in the original format of the download. In the case of the CIFAR-10 .png download:\n",
    "\n",
    "- cifar/labels.txt is a list of the 10 labels in their string form (airplane, automobile, bird, ...)\n",
    "- cifar/train/ is a directory of 50,000 labeled training images that contain their string label name in the filename (0_frog.png, 1_truck.png, 2_truck.png)\n",
    "- cifar/test/ is a directory of 10,000 testing images that are labeled the same way as the images in cifar/train/\n",
    "\n",
    "Using this information, let's start by creating label files to make life easier before writing to LMDBs.\n",
    "\n",
    "The first step to doing this is to declare our path variables, and create a `classes` dictionary to map string labels to integer labels that the LMDBs will take."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "classes: {'horse': 7, 'automobile': 1, 'deer': 4, 'dog': 5, 'frog': 6, 'cat': 3, 'truck': 9, 'ship': 8, 'airplane': 0, 'bird': 2}\n"
     ]
    }
   ],
   "source": [
    "# Paths to train and test directories\n",
    "training_dir_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'cifar', 'train')\n",
    "testing_dir_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'cifar', 'test')\n",
    "\n",
    "# Paths to label files\n",
    "training_labels_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'training_dictionary.txt')\n",
    "validation_labels_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'validation_dictionary.txt')\n",
    "testing_labels_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'testing_dictionary.txt')\n",
    "\n",
    "# Paths to LMDBs\n",
    "training_lmdb_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'training_lmdb')\n",
    "validation_lmdb_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'validation_lmdb')\n",
    "testing_lmdb_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'testing_lmdb')\n",
    "\n",
    "# Path to labels.txt\n",
    "labels_path = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data', 'cifar10', 'cifar', 'labels.txt')\n",
    "\n",
    "# Open label file handler\n",
    "labels_handler = open(labels_path, \"r\")\n",
    "\n",
    "\n",
    "# Create classes dictionary to map string labels to integer labels\n",
    "classes = {}\n",
    "i = 0\n",
    "lines = labels_handler.readlines()\n",
    "for line in sorted(lines):\n",
    "    line = line.rstrip()\n",
    "    classes[line] = i\n",
    "    i += 1\n",
    "labels_handler.close()\n",
    "\n",
    "print(\"classes:\", classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our `classes` dictionary to map string labels to integer labels, we can write our label files for training, validation, and testing. We will split the data as follows:\n",
    "\n",
    "    - training: 44,000 images (73%)\n",
    "    - validation: 6,000 images (10%)\n",
    "    - testing: 10,000 images (17%)\n",
    "    \n",
    "Note that the validation images are simply a subset of our training images that we will withhold to periodically test with during training. We do this so we can see how well our network is doing on unseen images without exposing our testing images to the model during training; something that makes machine learning experts cringe.\n",
    "\n",
    "To help get a relatively even distribution of each class of image in the training and validation sets, we first read all of the images (full paths) from the training directory into an array called `imgs`, and shuffle this list before iterating over it to write our label files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import shuffle\n",
    "\n",
    "# Open file handlers\n",
    "training_labels_handler = open(training_labels_path, \"w\")\n",
    "validation_labels_handler = open(validation_labels_path, \"w\")\n",
    "testing_labels_handler = open(testing_labels_path, \"w\")\n",
    "\n",
    "\n",
    "# Create training, validation, and testing label files\n",
    "i = 0\n",
    "validation_count = 6000\n",
    "imgs = glob.glob(training_dir_path + '/*.png')  # read all training images into array\n",
    "shuffle(imgs)  # shuffle array\n",
    "for img in imgs:\n",
    "    # Write first 6,000 image paths, followed by their integer label, to the validation label files\n",
    "    if i < validation_count:\n",
    "        validation_labels_handler.write(img + ' ' + str(classes[img.split('_')[-1].split('.')[0]]) + '\\n')\n",
    "    # Write the remaining to the training label files\n",
    "    else:\n",
    "        training_labels_handler.write(img + ' ' + str(classes[img.split('_')[-1].split('.')[0]]) + '\\n')\n",
    "    i += 1\n",
    "print(\"Finished writing training and validation label files\")\n",
    "\n",
    "# Write our testing label files using the testing images\n",
    "for img in glob.glob(testing_dir_path + '/*.png'):\n",
    "    testing_labels_handler.write(img + ' ' + str(classes[img.split('_')[-1].split('.')[0]]) + '\\n')\n",
    "print(\"Finished writing testing label files\")\n",
    "\n",
    "# Close file handlers\n",
    "training_labels_handler.close()\n",
    "validation_labels_handler.close()\n",
    "testing_labels_handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to use these label files to write our LMDBs. The following code is adapted from Caffe2's [lmdb_create_example.py](https://github.com/caffe2/caffe2/blob/master/caffe2/python/examples/lmdb_create_example.py) script. Note that before feeding the image data to the LMDB, we first reorder color channels from RGB --> BGR, and reorder columns from HWC --> CHW.\n",
    "\n",
    "If you have gone through the *Image Pre-Processing Pipeline* tutorial, you know that Caffe2 expects inputs in NCHW format, where N stands for the number of images in a batch. Don't worry, we'll add this N dimension when we define the data layer to our model (see `AddInput` below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_lmdb(labels_file_path, lmdb_path):\n",
    "    labels_handler = open(labels_file_path, \"r\")\n",
    "    # Write to lmdb\n",
    "    print(\">>> Write database...\")\n",
    "    LMDB_MAP_SIZE = 1 << 40\n",
    "    print(\"LMDB_MAP_SIZE\", LMDB_MAP_SIZE)\n",
    "    env = lmdb.open(lmdb_path, map_size=LMDB_MAP_SIZE)\n",
    "\n",
    "    with env.begin(write=True) as txn:\n",
    "        count = 0\n",
    "        for line in labels_handler.readlines():\n",
    "            line = line.rstrip()\n",
    "            im_path = line.split()[0]\n",
    "            im_label = int(line.split()[1])\n",
    "            \n",
    "            # read in image (as RGB)\n",
    "            img_data = imread(im_path).astype(np.float32)\n",
    "            \n",
    "            # convert to BGR\n",
    "            img_data = img_data[:, :, (2, 1, 0)]\n",
    "            \n",
    "            # HWC -> CHW (N gets added in AddInput function)\n",
    "            img_data = np.transpose(img_data, (2,0,1))\n",
    "            \n",
    "            # Create TensorProtos\n",
    "            tensor_protos = caffe2_pb2.TensorProtos()\n",
    "            img_tensor = tensor_protos.protos.add()\n",
    "            img_tensor.dims.extend(img_data.shape)\n",
    "            img_tensor.data_type = 1\n",
    "            flatten_img = img_data.reshape(np.prod(img_data.shape))\n",
    "            img_tensor.float_data.extend(flatten_img)\n",
    "            label_tensor = tensor_protos.protos.add()\n",
    "            label_tensor.data_type = 2\n",
    "            label_tensor.int32_data.append(im_label)\n",
    "            txn.put(\n",
    "                '{}'.format(count).encode('ascii'),\n",
    "                tensor_protos.SerializeToString()\n",
    "            )\n",
    "            if ((count % 1000 == 0)):\n",
    "                print(\"Inserted {} rows\".format(count))\n",
    "            count = count + 1\n",
    "\n",
    "    print(\"Inserted {} rows\".format(count))\n",
    "    print(\"\\nLMDB saved at \" + lmdb_path + \"\\n\\n\")\n",
    "    labels_handler.close()\n",
    "\n",
    "    \n",
    "# Call function to write our LMDBs\n",
    "if not os.path.exists(training_lmdb_path):\n",
    "    print(\"Writing training LMDB\")\n",
    "    write_lmdb(training_labels_path, training_lmdb_path)\n",
    "else:\n",
    "    print(training_lmdb_path, \"already exists!\")\n",
    "if not os.path.exists(validation_lmdb_path):\n",
    "    print(\"Writing validation LMDB\")\n",
    "    write_lmdb(validation_labels_path, validation_lmdb_path)\n",
    "else:\n",
    "    print(validation_lmdb_path, \"already exists!\")\n",
    "if not os.path.exists(testing_lmdb_path):\n",
    "    print(\"Writing testing LMDB\")\n",
    "    write_lmdb(testing_labels_path, testing_lmdb_path)\n",
    "else:\n",
    "    print(testing_lmdb_path, \"already exists!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define our CNN model\n",
    "Now that we have our data formatted in LMDBs, it is time to define our model! \n",
    "\n",
    "First let's set some path variables, define dataset-specific parameters, and declare model training parameters. This is where we will set the number of training iterations that we want, as well as the batch sizes and validation interval to use. Feel free to come back and tinker with these parameters to see how it effects training and efficiency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Paths to the init & predict net output locations\n",
    "init_net_out = 'cifar10_init_net.pb'\n",
    "predict_net_out = 'cifar10_predict_net.pb'\n",
    "\n",
    "# Dataset specific params\n",
    "image_width = 32                # input image width\n",
    "image_height = 32               # input image height\n",
    "image_channels = 3              # input image channels (3 for RGB)\n",
    "num_classes = 10                # number of image classes\n",
    "\n",
    "# Training params\n",
    "training_iters = 2000           # total training iterations\n",
    "training_net_batch_size = 100   # batch size for training\n",
    "validation_images = 6000        # total number of validation images\n",
    "validation_interval = 100       # validate every <validation_interval> training iterations\n",
    "checkpoint_iters = 1000         # output checkpoint db every <checkpoint_iters> iterations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the root_folder directory if it does not already exist. Also, call `workspace.ResetWorkspace(root_folder)` to set the `root_folder` as the working directory of our workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create root_folder if not already there\n",
    "if not os.path.isdir(root_folder):\n",
    "    os.makedirs(root_folder)\n",
    "\n",
    "# Resetting workspace with root_folder argument sets root_folder as working directory\n",
    "workspace.ResetWorkspace(root_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next task is to define some helper functions to modularize our code, and ultimately define our model similarly to the MNIST tutorial. We will use the `ModelHelper` class to define and represent our model, as well as to contain the model's parameter information. The `brew` module will be used to add layers to our CNN model. For more information about the `ModelHelper`+`brew` model creation paradigm, see the [docs](https://caffe2.ai/docs/brew.html).\n",
    "\n",
    ">It is important to note that by calling these functions, we are *NOT* running any computation with our model. Instead, we are constructing the graph of operators that will ultimately dictate the calculations made as our data blobs propagate forward and backward through the network. \n",
    "\n",
    "The first helper function is `AddInput`, which adds the input (data) layer to our model. Note that the image data stored in our LMDBs requires some minor preprocessing before it is fed to our computational layers. First, we read in the raw image data and labels from the LMDB, which is of type `uint8` ([0, 255] pixel values). We then cast the data to type `float` and rescale the data to [0, 1] to promote faster convergence. Finally, we will call `model.StopGradient(data, data)` to prevent the gradient from being calculated any further in the backward pass.\n",
    "\n",
    "One final point about the blob names in quotes:\n",
    "- In the case of `\"data_uint8\"` and `\"label\"`, these are the names of the blobs associated with the DB input\n",
    "- If the name is an *input* blob, this represents the blob name that the operator expects when ran\n",
    "- If the name is an *output* blob, e.g. `\"data\"`, it represents the name of the output blob that the operator creates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def AddInput(model, batch_size, db, db_type):\n",
    "    # load the data\n",
    "    data_uint8, label = brew.db_input(\n",
    "        model,\n",
    "        blobs_out=[\"data_uint8\", \"label\"],\n",
    "        batch_size=batch_size,\n",
    "        db=db,\n",
    "        db_type=db_type,\n",
    "    )\n",
    "    # cast the data to float\n",
    "    data = model.Cast(data_uint8, \"data\", to=core.DataType.FLOAT)\n",
    "    # scale data from [0,255] down to [0,1]\n",
    "    data = model.Scale(data, data, scale=float(1./256))\n",
    "    # don't need the gradient for the backward pass\n",
    "    data = model.StopGradient(data, data)\n",
    "    return data, label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to implement our CNN model definition. The network architecture that we will use is based on the \"quick\" model used in the original Caffe's [cifar10 example](https://github.com/BVLC/caffe/tree/master/examples/cifar10). This model has 3 convolutional/pooling layers, and uses Rectified Linear Unit activations (ReLU). Don't be afraid to come back and alter the model by changing hyperparameters and/or adding and removing layers to see how it affects training convergence.\n",
    "\n",
    "We will use the `update_dims` function as a helper to keep track of the dimensionality shrinkage that the convolutional and pooling layers cause. The dimensionality changes are as follows:\n",
    "\n",
    "$$height_{out}=\\frac{height_{in}-kernel+2*pad}{stride}+1$$\n",
    "\n",
    "---\n",
    "\n",
    "$$width_{out}=\\frac{width_{in}-kernel+2*pad}{stride}+1$$\n",
    "\n",
    "While this function is not necessary, we found that it is an easy strategy to avoid having to hand calculate the dimensionality changes of the data to provide to the penultimate fully connected layer. It also allows us to quickly change hyperparameters such as kernel size and not have to worry about the corresponding dimensionality alterations.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper function for maintaining the correct height and width dimensions after\n",
    "# convolutional and pooling layers downsample the input data\n",
    "def update_dims(height, width, kernel, stride, pad):\n",
    "    new_height = ((height - kernel + 2*pad)//stride) + 1\n",
    "    new_width = ((width - kernel + 2*pad)//stride) + 1\n",
    "    return new_height, new_width\n",
    "\n",
    "\n",
    "def Add_Original_CIFAR10_Model(model, data, num_classes, image_height, image_width, image_channels):\n",
    "    # Convolutional layer 1\n",
    "    conv1 = brew.conv(model, data, 'conv1', dim_in=image_channels, dim_out=32, kernel=5, stride=1, pad=2)\n",
    "    h,w = update_dims(height=image_height, width=image_width, kernel=5, stride=1, pad=2)\n",
    "    # Pooling layer 1\n",
    "    pool1 = brew.max_pool(model, conv1, 'pool1', kernel=3, stride=2)\n",
    "    h,w = update_dims(height=h, width=w, kernel=3, stride=2, pad=0)\n",
    "    # ReLU layer 1\n",
    "    relu1 = brew.relu(model, pool1, 'relu1')\n",
    "    \n",
    "    # Convolutional layer 2\n",
    "    conv2 = brew.conv(model, relu1, 'conv2', dim_in=32, dim_out=32, kernel=5, stride=1, pad=2)\n",
    "    h,w = update_dims(height=h, width=w, kernel=5, stride=1, pad=2)\n",
    "    # ReLU layer 2\n",
    "    relu2 = brew.relu(model, conv2, 'relu2')\n",
    "    # Pooling layer 1\n",
    "    pool2 = brew.average_pool(model, relu2, 'pool2', kernel=3, stride=2)\n",
    "    h,w = update_dims(height=h, width=w, kernel=3, stride=2, pad=0)\n",
    "    \n",
    "    # Convolutional layer 3\n",
    "    conv3 = brew.conv(model, pool2, 'conv3', dim_in=32, dim_out=64, kernel=5, stride=1, pad=2)\n",
    "    h,w = update_dims(height=h, width=w, kernel=5, stride=1, pad=2)\n",
    "    # ReLU layer 3\n",
    "    relu3 = brew.relu(model, conv3, 'relu3')\n",
    "    # Pooling layer 3\n",
    "    pool3 = brew.average_pool(model, relu3, 'pool3', kernel=3, stride=2)\n",
    "    h,w = update_dims(height=h, width=w, kernel=3, stride=2, pad=0)\n",
    "    \n",
    "    # Fully connected layers\n",
    "    fc1 = brew.fc(model, pool3, 'fc1', dim_in=64*h*w, dim_out=64)\n",
    "    fc2 = brew.fc(model, fc1, 'fc2', dim_in=64, dim_out=num_classes)\n",
    "    \n",
    "    # Softmax layer\n",
    "    softmax = brew.softmax(model, fc2, 'softmax')\n",
    "    return softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our next helper function is `AddTrainingOperators`. This function will be called by our train model to add a loss function and an optimization technique for learning. We will use an averaged cross entropy loss function between the model's softmax scores and the ground truth labels. We then add gradient operators to our model with respect to the loss that we previously calculated. Finally, we use the `build_sgd` function from Caffe2's `optimizer` class as our loss minimization function.\n",
    "\n",
    "Feel free to tinker with the hyper-parameters of the `build_sgd` function and observe the change in convergence efficiency during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def AddTrainingOperators(model, softmax, label):\n",
    "    xent = model.LabelCrossEntropy([softmax, label], 'xent')\n",
    "    # Compute the expected loss\n",
    "    loss = model.AveragedLoss(xent, \"loss\")\n",
    "    # Use the average loss we just computed to add gradient operators to the model\n",
    "    model.AddGradientOperators([loss])\n",
    "    # Use stochastic gradient descent as optimization function\n",
    "    optimizer.build_sgd(\n",
    "        model,\n",
    "        base_learning_rate=0.01,\n",
    "        policy=\"fixed\",\n",
    "        momentum=0.9,\n",
    "        weight_decay=0.004\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`AddAccuracy` adds an accuracy layer to a model using the `brew` module. This calculates the percentage of samples in a given batch whose top-1 softmax class matches the ground truth label class (i.e. percentage of samples in batch the model got right)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def AddAccuracy(model, softmax, label):\n",
    "    accuracy = brew.accuracy(model, [softmax, label], \"accuracy\")\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next and final helper function is `AddCheckpoints`, which outputs a checkpoint db at a regular interval of iterations. A checkpoint is essentially a saved state of a model during the training process. Checkpoints are useful for quickly loading a trained or partially trained model in the future, and they are an invaluable insurance policy during very long training processes. Caffe2 checkpoints are akin to Caffe's periodically outputted .caffemodel files. We use `brew`'s `iter` operator to track iterations, and will save them as LMDBs.\n",
    "\n",
    "It is important to note that when using checkpoints, you must be careful about attempting to overwrite checkpoints of the same name from a previous training process. If you attempt to overwrite a checkpoint db, the training process will error out. To deal with this, we will save the checkpoints in a uniquely named directory under our `root_folder`. This directory's name is based on the current system timestamp, to avoid duplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "# Create uniquely named directory under root_folder to output checkpoints to\n",
    "unique_timestamp = str(datetime.datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))\n",
    "checkpoint_dir = os.path.join(root_folder, unique_timestamp)\n",
    "os.makedirs(checkpoint_dir)\n",
    "print(\"Checkpoint output location: \", checkpoint_dir)\n",
    "\n",
    "# Add checkpoints to a given model\n",
    "def AddCheckpoints(model, checkpoint_iters, db_type):\n",
    "    ITER = brew.iter(model, \"iter\")\n",
    "    model.Checkpoint([ITER] + model.params, [], db=os.path.join(unique_timestamp, \"cifar10_checkpoint_%05d.lmdb\"), db_type=\"lmdb\", every=checkpoint_iters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize models with ModelHelper\n",
    "\n",
    "Now that we have created the necessary helper functions, it is time to actually initialize our training and validation models and use our functions to define the models' operator graphs. *Remember that we are not executing the models yet*.\n",
    "\n",
    "First, we define the train model:\n",
    "\n",
    "    (1) Initialize model with ModelHelper class\n",
    "    (2) Add data layer with AddInput function\n",
    "    (3) Add the Cifar10 model, which returns a softmax blob\n",
    "    (4) Add training operators with AddTrainingOperators function; use softmax blob from (3)\n",
    "    (5) Add periodic checkpoints with AddCheckpoints function\n",
    "\n",
    "Next, we define the validation model, which is structurally the same, but is separated because its input data comes from a different LMDB, and uses a different batch size. We will build as follows:\n",
    "\n",
    "    (1) Initialize model with ModelHelper class with init_params=False\n",
    "    (2) Add data layer with AddInput function\n",
    "    (3) Add the Cifar10 model, which returns a softmax blob\n",
    "    (4) Add accuracy layer with AddAccuracy function; use softmax blob from (3)\n",
    "\n",
    "Finally, we define the deploy model:\n",
    "\n",
    "    (1) Initialize model with ModelHelper class with init_params=False\n",
    "    (2) Add the Cifar10 model, which will expect input blob called \"data\"\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training, Validation, and Deploy models all defined!\n"
     ]
    }
   ],
   "source": [
    "arg_scope = {\"order\": \"NCHW\"}\n",
    "\n",
    "# TRAINING MODEL\n",
    "# Initialize with ModelHelper class\n",
    "train_model = model_helper.ModelHelper(\n",
    "    name=\"train_net\", arg_scope=arg_scope)\n",
    "# Add data layer from training_lmdb\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=training_net_batch_size,\n",
    "    db=training_lmdb_path,\n",
    "    db_type='lmdb')\n",
    "# Add model definition, save return value to 'softmax' variable\n",
    "softmax = Add_Original_CIFAR10_Model(train_model, data, num_classes, image_height, image_width, image_channels)\n",
    "# Add training operators using the softmax output from the model\n",
    "AddTrainingOperators(train_model, softmax, label)\n",
    "# Add periodic checkpoint outputs to the model\n",
    "AddCheckpoints(train_model, checkpoint_iters, db_type=\"lmdb\")\n",
    "\n",
    "\n",
    "# VALIDATION MODEL\n",
    "# Initialize with ModelHelper class without re-initializing params\n",
    "val_model = model_helper.ModelHelper(\n",
    "    name=\"val_net\", arg_scope=arg_scope, init_params=False)\n",
    "# Add data layer from validation_lmdb\n",
    "data, label = AddInput(\n",
    "    val_model, batch_size=validation_images,\n",
    "    db=validation_lmdb_path,\n",
    "    db_type='lmdb')\n",
    "# Add model definition, save return value to 'softmax' variable\n",
    "softmax = Add_Original_CIFAR10_Model(val_model, data, num_classes, image_height, image_width, image_channels)\n",
    "# Add accuracy operator\n",
    "AddAccuracy(val_model, softmax, label)\n",
    "\n",
    "\n",
    "# DEPLOY MODEL\n",
    "# Initialize with ModelHelper class without re-initializing params\n",
    "deploy_model = model_helper.ModelHelper(\n",
    "    name=\"deploy_net\", arg_scope=arg_scope, init_params=False)\n",
    "# Add model definition, expect input blob called \"data\"\n",
    "Add_Original_CIFAR10_Model(deploy_model, \"data\", num_classes, image_height, image_width, image_channels)\n",
    "\n",
    "print(\"Training, Validation, and Deploy models all defined!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run training\n",
    "\n",
    "Finally, now that we have our models and their operator graphs defined, it is time to actually run the training process. Under the hood, we have defined our models as operator graphs that are serialized in protobuf format. The final step is to send these protobufs to Caffe2's C++ backend so that model objects can be built and executed. \n",
    "\n",
    "Recall that a `ModelHelper` model object has two nets:\n",
    "\n",
    "- `param_init_net`: Contains parameters and initialization data\n",
    "\n",
    "- `net`: Contains the main network (operator graph) that we just defined\n",
    "\n",
    "Both of these nets must be run, and we must start with the `param_init_net`. Because this net only needs to be run once, we run it with the `workspace.RunNetOnce` function, which instantiates, runs, and immediately destructs the network. If we want to run a network multiple times, as we do in the case of our training and validation nets, we first create the net with `workspace.CreateNet`, and we can then run the net using `workspace.RunNet`. \n",
    "\n",
    "Note that when we call `workspace.RunNet` on the `train_model`, this runs the forward and backward pass with a batch from our training LMDB. Running the `val_model` runs a forward pass with a batch from our validation LMDB (which we set to be all of the images) and adds an accuracy layer that we will use to track model accuracy on our quasi-test data as we train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training iter:  0\n",
      "Loss:  2.311319589614868\n",
      "Validation accuracy:  0.10383333265781403\n",
      "\n",
      "Training iter:  100\n",
      "Loss:  1.9484632015228271\n",
      "Validation accuracy:  0.2861666679382324\n",
      "\n",
      "Training iter:  200\n",
      "Loss:  1.7397210597991943\n",
      "Validation accuracy:  0.3641666769981384\n",
      "\n",
      "Training iter:  300\n",
      "Loss:  1.7527788877487183\n",
      "Validation accuracy:  0.4051666557788849\n",
      "\n",
      "Training iter:  400\n",
      "Loss:  1.3784841299057007\n",
      "Validation accuracy:  0.45116665959358215\n",
      "\n",
      "Training iter:  500\n",
      "Loss:  1.5721114873886108\n",
      "Validation accuracy:  0.4581666588783264\n",
      "\n",
      "Training iter:  600\n",
      "Loss:  1.5422420501708984\n",
      "Validation accuracy:  0.4958333373069763\n",
      "\n",
      "Training iter:  700\n",
      "Loss:  1.3092886209487915\n",
      "Validation accuracy:  0.5076666474342346\n",
      "\n",
      "Training iter:  800\n",
      "Loss:  1.3119572401046753\n",
      "Validation accuracy:  0.5444999933242798\n",
      "\n",
      "Training iter:  900\n",
      "Loss:  1.3184524774551392\n",
      "Validation accuracy:  0.5375000238418579\n",
      "\n",
      "Training iter:  1000\n",
      "Loss:  1.2561535835266113\n",
      "Validation accuracy:  0.5534999966621399\n",
      "\n",
      "Training iter:  1100\n",
      "Loss:  1.1288306713104248\n",
      "Validation accuracy:  0.5805000066757202\n",
      "\n",
      "Training iter:  1200\n",
      "Loss:  1.221421480178833\n",
      "Validation accuracy:  0.5686666369438171\n",
      "\n",
      "Training iter:  1300\n",
      "Loss:  1.1555482149124146\n",
      "Validation accuracy:  0.5920000076293945\n",
      "\n",
      "Training iter:  1400\n",
      "Loss:  1.281171202659607\n",
      "Validation accuracy:  0.5929999947547913\n",
      "\n",
      "Training iter:  1500\n",
      "Loss:  1.0986618995666504\n",
      "Validation accuracy:  0.5846666693687439\n",
      "\n",
      "Training iter:  1600\n",
      "Loss:  1.1475869417190552\n",
      "Validation accuracy:  0.6179999709129333\n",
      "\n",
      "Training iter:  1700\n",
      "Loss:  1.0574977397918701\n",
      "Validation accuracy:  0.6158333420753479\n",
      "\n",
      "Training iter:  1800\n",
      "Loss:  1.2078982591629028\n",
      "Validation accuracy:  0.6358333230018616\n",
      "\n",
      "Training iter:  1900\n",
      "Loss:  0.8897716403007507\n",
      "Validation accuracy:  0.6358333230018616\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "# Initialize and create the training network\n",
    "workspace.RunNetOnce(train_model.param_init_net)\n",
    "workspace.CreateNet(train_model.net, overwrite=True)\n",
    "# Initialize and create validation network\n",
    "workspace.RunNetOnce(val_model.param_init_net)\n",
    "workspace.CreateNet(val_model.net, overwrite=True)\n",
    "# Placeholder to track loss and validation accuracy\n",
    "loss = np.zeros(int(math.ceil(training_iters/validation_interval)))\n",
    "val_accuracy = np.zeros(int(math.ceil(training_iters/validation_interval)))\n",
    "val_count = 0\n",
    "iteration_list = np.zeros(int(math.ceil(training_iters/validation_interval)))\n",
    "\n",
    "# Now, we run the network (forward & backward pass)\n",
    "for i in range(training_iters):\n",
    "    workspace.RunNet(train_model.net)\n",
    "    \n",
    "    # Validate every <validation_interval> training iterations\n",
    "    if (i % validation_interval == 0):\n",
    "        print(\"Training iter: \", i)\n",
    "        loss[val_count] = workspace.FetchBlob('loss')\n",
    "        workspace.RunNet(val_model.net)\n",
    "        val_accuracy[val_count] = workspace.FetchBlob('accuracy')\n",
    "        print(\"Loss: \", str(loss[val_count]))\n",
    "        print(\"Validation accuracy: \", str(val_accuracy[val_count]) + \"\\n\")\n",
    "        iteration_list[val_count] = i\n",
    "        val_count += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the validation accuracy vs. loss over the training iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x2018738ded0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEWCAYAAACdaNcBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VFX6wPHvaxLpTYogHRSkVykiIEVEVBAFBZSmLGtDimJdu6vYWcvPDgoKFpTiAuq66Ao2DAhKkQ4SmhSpoSV5f3+cO2QSUybJTGYyeT/Pc5/M3HLuO3cm75w599xzRVUxxhgTXU4LdwDGGGOCz5K7McZEIUvuxhgThSy5G2NMFLLkbowxUciSuzHGRCFL7gWAiMSIyGERqRHMdU3uicjZIqJ+z78QkWsDWTcX+7pfRF7N7famcLLkHgJecvVNKSJy1O95hgkgK6qarKolVfX3YK6bUyLymIi8Hexyw0FEvhSRBzKYf5WIbBORmJyUp6o9VPW9IMTVXUQ2pyv7UVW9Ma9lZ7NPFZHbQ7UPk/8suYeAl1xLqmpJ4Hfgcr95f0kAIhKb/1EWeu8AgzOYPxh4V1WT8zmecBoK7AOG5PeO7bMfOpbcw8CrAX8gItNF5BBwnYi0F5EfRGS/iOwQkRdEJM5bP9arWdXynr/rLZ8vIodE5HsRqZ3Tdb3ll4jIWhE5ICIvisi3IjIsF6+pkYj8z4v/VxG51G/ZZSKy2tt/goiM9eZXEpF53jb7ROSbTMp+Q0QmpJs3V0Ru8x7fKyLbReSgiPwmIhcGEPInQGUROd+vzPJAL2CK97y3iCzzyv1dRO7P4vUv8h03r2nseRHZKyIbgZ7p1h3hdzw2iMgIb34Z4FOght8vvUrpfzGJSF8RWekdtwUiUt9vWYKIjPPegwPeZ6xIFnGXAq4EbgYaikjzdMs7eZ/LAyKyVUQGe/OLe6/xd2/ZNyJSJKNfHl5MF3qPc/TZ97ZpIu6X1j4R2Skid4pIVRFJFJGyfuu18ZbbFwaAqtoUwgnYDHRPN+8x4ARwOe4LthhwHtAWiAXqAGuBW731YwEFannP3wX2AK2BOOADXG0zp+tWAg4Bfbxl44CTwLBMXstjwNsZzD8d2ATc6ZXTHTgMnO0t3w2c7z0+A2jpPX4aeMnb5nSgUyb77eodR/GelweOAmcCjYAtQGVvWW2gToDvzWTgVb/ntwDx6fbbyHuPmnnH8TJv2dnu3+fUuot8xw24FVgJVPNi/Sbdupd777F4+zgKNPWWdQc2Z3bcgQbese3qHbd7gTVAnLc8AfgBqOztey0wIotjMNzb5jRgPvC837La3r6u9j5XFYDm3rLXgP8CVYAY4AK/9z59/AnAhbn87JcBdgGjgSJAaaCNt+wL4G9++3nRP/7CPlnNPXwWqeqnqpqiqkdV9SdV/VFVk1R1I/A60DmL7WeoaryqngTeA5rnYt3LgGWqOttb9jwugeVUB1xyflpVT6rql7hEMcBbfhJXKyylqvtUdanf/LOAGqp6QlUzrLkDX+MSR3vv+dXAQlXdBSQBRYFGIhKrqpu84xeId4Cr/Wq2Q7x5AKjqAlVd6b1Hy4H3yfo98bkal2QSVHUvkOZXh/e+b1RnAS5Jdgww5gHAHC+2k17ZZXDJ0Weiqu709v1vsv5sDAXeV9UUYBowyK/mex0wX1U/9D6Xe1R1mbjzEcOA21R1h7rzPIu8eAKRk89+b+B3Vf2Xqh5X1YOquthb9o4Xo695ZwAwNcAYop4l9/DZ6v9ERM71mhp2ishB4BFcTSkzO/0eJwIlc7HuWf5xqKv+JAQQe3pn4f4B/XuEbAGqeo/74v2TisjXIuJLRBO89f7rNU+Mz6hwL/F8AAz0Zg3CfUmhqmuA23HH6w/v537lAOP+H3AQuFxE6gEtgOm+hV5zwdcisltEDgAjyPo98UlzXL3XeIrXTPWj18ywH+gRYLm+sk+V5x2bBFKPNQT42RDXdNcJ71gCM711fc1I1YENGWx6Ju7LPKNlgcjJZz+zGHzxNhPXM6wn8IdfxaHQs+QePum7xr0GrMA1ZZQGHsD9bA+lHbimAwBEREibJAK1Hajube9TA9gG4NXKeuOagf6NqwHj1cLGqmot4ArgLhHJrGY8Hegv7nxBS1ybOV4576pqB1wzQgzwRCBBe19GU3A19sHAPFX1/+XyPvAxUF1VywBvEth7sgOXlHxOdUsVkWLADC/GM1W1LK55wVdudl0mtwM1/co7DfcebgsgrvSGePudLyI7gfW4pD3UW74VqJvBdrtwTSsZLTsCFPeLLxbXPOQvJ5/9zGJAVRNx78+1uPfPau1+LLlHjlLAAeCIiDQA/p4P+/w30FJELvf+CUcDFbPZJkZEivpNRYDvcM0jt4tInIh0xZ2Y/EBEionIIBEp7f1sPwSkAHj7ret9KRwAkn3L0lPVn3C17NdxSfiQV0YDEenixXHUmzIsIxNTcLW+6/FrkvGUAvap6jERaUdqM1N2PgTGeCf9ygN3+S0rgkugu4FkEbkM6Oa3fBdQwTvRmVnZvUXkQu+k43jcMf0xwNj8DcEl0uZ+0zW4XzLlcOdreorrHhorIhVEpJm6nkRvAxNFpLK4E8gdvHh+A0qJyMXe8wdxTWpZyeqzPwd3gvlW74RtaRFp47d8Cu69u9SL13gsuUeO23E1pkO4mswHod6h12Z9DfAcsBdXQ/oZOJ7FZteRmkSPAmtU9TjuBFkfXJv9C8AgVV3nbTMU2OL95L7BKwOgPrAAd9LuW+Bfqrowi31Px52wm+Y3rwjwlLffnUA54D4AERkqIsuzOQbrgcVeOXPTLb4JeMLr1XEvLrEG4hVcO/qvwE+4mrpvf/uBsbgmhX1AP9yXrG/5ClxtdLPXe6RSunhX4o7nK7gviJ5A7xy0dwMgIhfgmnhe9trnd6rqTi+uzcA1qroJ977e5cW6FGjiFTEWWA0s8ZY9jjvh/ScwCvdFuc1b5t9MlJFMP/uqegC4CLgK98W3lrTnPb7BnYj9UVVz06QYtXy9D4zBO1G2HeiXTZI1JmKI60I7SVXfDncskcRq7oWciPQUkbJes8b9uB4si7PZzJiI4DWXNQY+CncskcaSu7kA2Ij7iX8x0NdrZjEmoonIe8BnwGhVPRLueCKNNcsYY0wUspq7McZEobCNwVChQgWtVatWuHZvjDEF0pIlS/aoanZdlsOX3GvVqkV8fHy4dm+MMQWSiGzJfi1rljHGmKhkyd0YY6KQJXdjjIlCNqi9MWF28uRJEhISOHbsWLhDMRGkaNGiVKtWjbi47IbmyZgld2PCLCEhgVKlSlGrVi3SDqxpCitVZe/evSQkJFC7du3sN8iANcsYE2bHjh2jfPnyltjNKSJC+fLl8/RrzpK7MRHAErtJL6+fiQKX3NetgzFj4GSOBjg1xpjCpcAl97Vr4V//gndtWH5jgqZkyazu0mgKogKX3Hv1gpYt4Z//hKSkcEdjjDGRqcAldxF44AHYsAGmT89+fWNM7mzevJmuXbvStGlTunXrxu+//w7ARx99ROPGjWnWrBmdOnUCYOXKlbRp04bmzZvTtGlT1q1bl1XRJh+Ebcjf1q1ba27HllGFFi3g6FFYtQpiYoIcnDH5aPXq1TRo0ABw55OWLQtu+c2bw8SJWa9TsmRJDh8+nGbe5ZdfTr9+/Rg6dCiTJk1izpw5zJo1iyZNmvDZZ59RtWpV9u/fT9myZRk1ahTt2rXj2muv5cSJEyQnJ1OsWLHgvpBCyP+z4SMiS1S1dXbbFriaO6TW3teuhQ9CfqdRYwqn77//nkGDBgEwePBgFi1aBECHDh0YNmwYb7zxBsnJyQC0b9+exx9/nCeffJItW7ZYYo8ABfYipiuugMaN4bHH4JprrPZuokN2NexI8Oqrr/Ljjz8yd+5cWrVqxZIlSxg0aBBt27Zl7ty59OrVi9dee42uXbuGO9RCrUDW3AFOOw3uvx9Wr4aPPw53NMZEn/PPP5/3338fgPfee4+OHTsCsGHDBtq2bcsjjzxCxYoV2bp1Kxs3bqROnTrcdttt9OnTh19++SWcoRsKcM0d4KqroEEDePRR6NfPJXxjTM4lJiZSrVq1U8/HjRvHiy++yPDhw3n66aepWLEikydPBmD8+PGsW7cOVaVbt240a9aMJ598kqlTpxIXF0flypW59957w/VSjKdAnlD1N306DBrkau9XXhmEwIzJZxmdNDMGCuEJVX9XXw316sEjj0BKSrijMcaYyFDgk3tMDPzjH7B8OXz6abijMcaYyFDgkzvAwIFQt66rvYeplckYYyJKVCT32Fi47z5YuhTmzQt3NMYYE35RkdwBrrsOate22rsxxkAUJfe4OLj3Xli8GD7/PNzRGGNMeEVNcgcYMgRq1ICHH7bauzGB6tKlC5+nqxFNnDiRm266KcvtfMMEb9++nX79+mW4zoUXXkh2XZ4nTpxIYmLiqee9evVi//79gYQekObNmzNgwICglVdQRFVyP/10uOce+OEH+O9/wx2NMQXDwIEDT12J6vP+++8zcODAgLY/66yzmDFjRq73nz65z5s3j7Jly+a6PH+rV68mOTmZhQsXcuTIkaCUmZGkCBx/PKqSO8Dw4VCtmtXejQlUv379mDt3LidOnADcUL/bt2+nY8eOHD58mG7dutGyZUuaNGnC7Nmz/7L95s2bady4MQBHjx5lwIABNGjQgL59+3L06NFT69100020bt2aRo0a8eCDDwLwwgsvsH37drp06UKXLl0AqFWrFnv27AHgueeeo3HjxjRu3JiJ3sA7mzdvpkGDBvztb3+jUaNG9OjRI81+/E2fPp3BgwfTo0ePNLGvX7+e7t2706xZM1q2bMmGDRsAePLJJ2nSpAnNmjXj7rvvBtL++tizZw+1atUC4O2336Z379507dqVbt26ZXmspkyZQtOmTWnWrBmDBw/m0KFD1K5dm5PeLeUOHjyY5nlQqGpYplatWmmovPSSKqh+9VXIdmFM0KxatSr1yejRqp07B3caPTrbGC699FKdNWuWqqo+8cQTevvtt6uq6smTJ/XAgQOqqrp7926tW7eupqSkqKpqiRIlVFV106ZN2qhRI1VVffbZZ3X48OGqqrp8+XKNiYnRn376SVVV9+7dq6qqSUlJ2rlzZ12+fLmqqtasWVN37959Khbf8/j4eG3cuLEePnxYDx06pA0bNtSlS5fqpk2bNCYmRn/++WdVVe3fv79OnTo1w9dVr1493bJli37++ed62WWXnZrfpk0b/eSTT1RV9ejRo3rkyBGdN2+etm/fXo8cOZIm3s6dO596Dbt379aaNWuqqurkyZO1atWqp9bL7FitWLFCzznnnFOv0bf+sGHDdObMmaqq+tprr+m4ceP+En+az4YHiNcAcmzU1dwBbrgBqlRxtXdjTPb8m2b8m2RUlXvvvZemTZvSvXt3tm3bxq5duzIt55tvvuG6664DoGnTpjRt2vTUsg8//JCWLVvSokULVq5cyapVq7KMadGiRfTt25cSJUpQsmRJrrzyShYuXAhA7dq1ad68OQCtWrVi8+bNf9k+Pj6eChUqUKNGDbp168bPP//Mvn37OHToENu2baNv374AFC1alOLFi/Pll18yfPhwihcvDsAZZ5yR7XG76KKLTq2X2bFasGAB/fv3p0KFCmnKHTFixKnxeiZPnszw4cOz3V9OFOiBwzJTtCjcdZe78cE334B3sxhjIl+Yxvzt06cPY8eOZenSpSQmJtKqVSvAjQa5e/dulixZQlxcHLVq1eLYsWM5Ln/Tpk0888wz/PTTT5QrV45hw4blqhyfIkWKnHocExOTYbPM9OnT+e233041oxw8eJCPP/44xydXY2NjSfHGNkkfc4kSJU49zumx6tChA5s3b+brr78mOTn5VNNWsERlzR1g5Eg480w3YqQxJmslS5akS5cuXH/99WlOpB44cIBKlSoRFxfHV199xZYtW7Isp1OnTkybNg2AFStWnBr69+DBg5QoUYIyZcqwa9cu5s+ff2qbUqVKcejQob+U1bFjR2bNmkViYiJHjhxh5syZp4Ydzk5KSgoffvghv/76K5s3b2bz5s3Mnj2b6dOnU6pUKapVq8asWbMAOH78OImJiVx00UVMnjz51Mndffv2Ae4cwJIlSwCyPHGc2bHq2rUrH330EXv37k1TLsCQIUMYNGhQ0GvtEMXJvVgxuPNO+PJL+O67cEdjTOQbOHAgy5cvT5Pcr732WuLj42nSpAlTpkzh3HPPzbKMm266icOHD9OgQQMeeOCBU78AmjVrRosWLTj33HMZNGgQHTp0OLXNyJEj6dmz56kTqj4tW7Zk2LBhtGnThrZt2zJixAhatGgR0GtZuHAhVatW5ayzzjo1r1OnTqxatYodO3YwdepUXnjhBZo2bcr555/Pzp076dmzJ71796Z169Y0b96cZ555BoA77riDV155hRYtWpw60ZuRzI5Vo0aNuO++++jcuTPNmjVj3Lhxabb5888/A+6ZlBMFfsjfrBw54q5abdUK/CoKxkQUG/K38JoxYwazZ89m6tSpGS7Py5C/Udnm7lOiBNxxh2t/X7wY2rQJd0TGGOOMGjWK+fPnMy9EA2Jl2ywjItVF5CsRWSUiK0VkdAbriIi8ICLrReQXEWkZkmhz4eaboXx5N+aMMcZEihdffJH169dTr169kJQfSJt7EnC7qjYE2gG3iEjDdOtcApzjTSOBV4IaZR6ULAm33w5z54J3TsSYiBOu5lETufL6mcg2uavqDlVd6j0+BKwGqqZbrQ8wxetj/wNQVkSq5CmyILrlFihXznrOmMhUtGhR9u7dawnenKKq7N27l6JFi+a6jBy1uYtILaAF8GO6RVWBrX7PE7x5O9JtPxJXs6dGjRo5izQPSpeGsWPhgQdg2TLwrn0wJiJUq1aNhIQEdu/eHe5QTAQpWrRompuW51TAyV1ESgIfA2NU9WBudqaqrwOvg+stk5sycmvUKHj2WVd7//jj/NyzMVmLi4ujdu3a4Q7DRJmA+rmLSBwusb+nqp9ksMo2oLrf82revIhRtiyMHg2ffAK//hruaIwxJrQC6S0jwFvAalV9LpPV5gBDvF4z7YADqrojk3XDZswYKFUKHnss3JEYY0xoBVJz7wAMBrqKyDJv6iUiN4rIjd4684CNwHrgDeDm0ISbN+XKwW23wUcfwddfhzsaY4wJnai+QjUjBw5A+/awcyd8/z3Ur5/vIRhjTK4FeoVq1I4tk5kyZVyf99hYuPRSyGKoCGOMKbAKXXIHN97MnDmwbRtccQXkYeRRY4yJSIUyuQO0awdTpsC337qbe9j1I8aYaFJokztA//7w+OMwbRo89FC4ozHGmOCJ6lEhA3H33bB+vRtYrG5dGDIk3BEZY0zeFfrkLgKvvgqbN8OIEVCzJnTuHO6ojDEmbwp1s4xPXJwbkuDss6FvX1izJtwRGWNM3lhy95Qta10kjTHRw5K7n9q1YfZsSEiwLpLGmILNkns67dundpG8/nrrImmMKZgK/QnVjFx9NWzYAPfe69rh7RZ9xpiCxpJ7JnxdJB991CV46yJpjClILLlnQgReeSW1i2SNGnDhheGOyhhjAmNt7lk4/XSYMcNd3HTlldZF0hhTcFhyz0a5ctZF0hhT8FizTADq1HFdJLt0cV0kv/wSsrspuSocPAh797pp377Ux8nJcPPN7peBMcaEgiX3APm6SF5zDQwaBBdfnHHi9j3et88l8czExsKtt+Zf/MaYwqXQ3Ykpr554wnWR9CleHMqXhzPOcH99k//z9I/79oXff3e9caz2bozJiUDvxGTJPRd+/93VvM84I/vmmYzMnw+9esHkyTBsWNDDM8ZEMbvNXgjVqAFnnZW7xA7Qsyc0b+5+BWTVdGOMMbllyT0MRFzTztq18Mkn4Y7GGBONLLmHyZVXQv367k5QNn6NMSbYLLmHSUyMG+Jg2TLXBm+MMcFkyT2Mrr3Wtd//859WezfGBJcl9zCKi4M774TvvoOFC8MdjTEmmlhyD7Prr4dKlVzt3RhjgsWSe5gVKwbjxsEXX0AB7fZvjIlAltwjwE03uXu4Pv54uCMxxkQLS+4RoHRpGDUKZs6EVavCHY0xJhpYco8Qt93mxqmZMCHckRhjooEl9whRoQLceCNMmwYbN4Y7GmNMQWfJPYKMG+cubnr66dDtIyUFvv8eVq+GkydDtx9jTHjZeO4RpGpVN0rkpEnwwANQpUpwy1eFsWPhhRfc87g4NwRCw4bQqJGbGjZ0NwSPiwvuvo0x+cuSe4S56y5480149ll45pnglv3YYy6x33ILtGsHK1e6KT4ePvoo9SpZS/rGFHw2nnsEuu46mDULtmxxN/gIhv/7P5fUfb8MRNIuT0yE335LTfi+adOm1HV8Sb9FC3jyyeD/sjDGZM9u1lGArVgBTZq4ppmHH857edOnu3FseveGGTPcjUYCdeRI2qS/apW7h2zr1rBggdXkjclvQUvuIjIJuAz4Q1UbZ7D8QmA24KvjfaKqj2S3Y0vuWevbF77+2t31qVSp3Jczf75L6h06wGef5f4GI/6mTXNfFnfcEdqTv8aYvwrmnZjeBnpms85CVW3uTdkmdpO9e+6B/fvh1VdzX8a338JVV0HTpjBnTnASO7gbhN98szsnMHNmcMo0xgRXtsldVb8B9uVDLMZPmzbQvbs7sXr0aM63/+UXuOwyqF7d1d5Llw5ufM89B+ed59rw160LbtnGmLwLVj/39iKyXETmi0ijzFYSkZEiEi8i8bt37w7SrqPXfffBrl3uRto5sWEDXHwxlCjhBiSrVCn4sRUp4nrYxMRAv365+wIyxoROMJL7UqCmqjYDXgRmZbaiqr6uqq1VtXXFihWDsOvo1rkztG8PTz0V+AVHO3ZAjx5w4oRL7DVrhi6+mjXh3Xfdr4RbbgndfowxOZfn5K6qB1X1sPd4HhAnIhXyHJk5dSPtLVvcSczs/Pmnq7Hv2uWaYho2DH2MvXrBP/7hfl1MmhT6/RljApPn5C4ilUVcr2kRaeOVuTev5Rrn0kvdCdEnnoDk5MzXS0x0bexr1rg+8m3a5F+MDz0E3bq52vuyZfm3X2NM5rJN7iIyHfgeqC8iCSJyg4jcKCI3eqv0A1aIyHLgBWCAhqvzfBTy1d7XrMm8Z8qJE67d+4cfXA2/e/f8jTEmxu23fHnXO2f//vzdvzHmr+wipgIgORkaNICSJWHJkrRXl6akuCtap0+H11+Hv/0tfHF++y1ceKH7tTFz5l+vgjXG5F0w+7mbMIuJgbvvhp9/hs8/T52v6saBnz7djQMfzsQO7kKpp56C2bODPy6OMSZnLLkXENdd5/qs+99I+6GH4OWX3ZWid94ZttDSGDPGNRHdcw988024ozGm8LLkXkCcfjqMHw+LFsHChW50x0cegeuvd7XlSGkCEYG33oI6deCaa2DnznBHZEzhZG3uBUhiItSq5caa2bgRrrjCXUiUk4HA8suvv0Lbtq7XzpdfRmaMxhRE1uYehYoXdzfb2LgRunRxbe2RmjSbNHHj4vzvf3D//eGOxpjCJ0JTg8nMmDGuy+GAAcEbCCxUhgxxzUgTJrgrbXv3DndExhQe1ixjQurYMdeLZuNG142zTp1wR2RMwWbNMiYiFC3qbhAC0L+/S/bGmNCz5G5CrnZtmDIFli51/fKNMaFnbe4mX1x+ubsQa8IE10wzdGjuylGFpCQ3xLBvSkzM+HFmy5KS3Iibl10GxYoF93UaEymszd3km6QkuOgiNwZOo0bueXKy++ub0j9PPy8lJff7j4tzyTwlBQ4fdl1Kr7gCBg504/HY/WBNQRBom7vV3E2+iY113Tdvv90NLhYb66aYmNTHgc4rVix1Kl4848fpn/u6jSYnu/vTTpsGH38MU6dChQrunMDAge6XxWnWYGkKOKu5m0Lt+HF34/Dp0919Zo8edcM8XHONS/QtWkTO1b+RYvVqqFvXXTVt8p/1ljEmAEWKQJ8+8P778Mcf8N570KwZTJwIrVq50TgffhjWrg13pOF3+DCMHOluAtO7t/V8inSW3I3xlCwJgwbBp5+6MXFeew2qVHHJvX59l+yfeQYSEsIdaf778Uf3K+bNN6FvXzc6ab9+7pePiUyW3I3JQPnyrpb61VewdSs895xr8x8/HmrUcMNAFIaaa1KSG6CuQwd3U5ivvoJPPnFffHPnuuarQO/va/KXJXdjslG1qkvmixfDunVw442u2aZNG1ixItzRhc6GDdCxIzz4oBvuYvly14UU3BffSy+5sfsHDrQEH4ksuRuTA2efDf/3f67WumsXtG7tklw03VhS1Q3b3KyZO3k6bRq8+y6ULZt2vVtugeefdz2OBg92tXwTOSy5G5MLvXrBL7+4G4OPGuUuiNq1K7T7PHECXnnFJd1Bg1y7d1Y3Tc+NPXvgyithxAj3y+TXX13NPDNjxrj7CXzwAQwbFvx4TO5Zcjcml848E/79b1dzX7DADXM8d27w95OUBG+/7U7q3nyza/v/7DPo2dN12xw/PjjNQ/Pnu9cwb547cfzll6787Iwf7+4Q9t577kshLxeaRbpDh2DcOPjuu3BHEgBVDcvUqlUrNSZarFih2rSpKqjecotqYmLey0xOVp0+XbVePVduq1aqn32mmpKieuyY6owZqr17q8bGuuUtWqg+/7zqzp0528+RIy5mUG3USHXZstzF+9BDrowRI1zs0WbzZtUmTdxrPPts1ePHwxMHEK8B5FhL7sYEydGjqmPHuv+qhg1znyRTUlRnzUpNJI0bq86c6eZn5I8/VF94QbV1a7d+TIzqpZeqfvihiykrS5aonnuu227s2OzXzy7u++5zZd10U+bxFkTffadaqZJqmTKpr/H558MTiyV3Y8Lk889VK1dWPf101eeeC7wWm5Litj3vPPefec45qtOmqSYlBb7vFStU77pLtWpVV0aZMqojR6ouWpQ22SYlqT7xhKv1n3WW6n/+k7PXmNVruPNOt+9Ro6Ijwb/3nmqRIqp166quXu1eU48equXKqe7dm//xWHI3Jox273ZNJqB60UWq27Zlvf4336h26uTWr1FD9a23VE+ezP3+k5Jcwr7uOtXixV0XumdgAAAXWklEQVS5deuqPvywS/QdO7p5/fsHP0GlpKiOGePKHzeu4Cb45GTV++93r6NzZ9U9e1KX/fKL6mmnqY4enf9xWXI3JsxSUlRffVW1WDHV8uVdU0t6ixerXnyx+0+sXFn1pZdce3owHTyo+vbbql27qoq4fZUqpTplSugSb0qK6q23un3ddVfw9pOYqPrtt6p//hmc8jJz5Ij74gPV66/PuH195Ej3y2fNmtDGkp4ld2MixOrV7mQnqP7976qHD7uaX58+bl758qpPP+0SSqht2aL6xhuqmzaFfl8pKe71guo//pH7Mn75RfWZZ1xTSNGieqq56aGHVPfvD27Mqqrbt7vzFyLufcnsi2nnTtWSJd37mJ8suRsTQY4fd23RIq6GLqJaurTqI4+4mnW0Sk5WveEGl2kefjiwbXbtcu3cQ4aoVqnitvWdpB49WvX991X79nXzypZ15QYryS9d6s5XlCihOnt29us//riLY8GC4Ow/EJbcjYlA//2v69J4zz3hORkXDsnJLlGDS4bpHTvmjstdd6X+wvH9ohkwQHXSJNWtW/+63dKlqb9+ypVTffRR1QMHch/nzJnu/ET16oH3dEpMdOdImjfP2YnvvLDkboyJGElJqoMGuYzz1FOqK1e6roSXXOLOSYBqXJw7cfn446rx8YH3MlqyRPXyy10ZZ5yh+s9/5uzXUEqK6oQJ7tdUmzaqO3bk7LVNm+b2PXlyzrbLrUCTu92swxiTL5KS3LAJH32UOq9+fejRw02dO7tbH+ZWfDw89JC7Srh8ebjjDrj1VjeUc2aOH4e//x3eeceNcDl5cs7vq6sK7dq50UPXrYMSJXL/GgIR6M06LLkbY/LNyZPw8ssu4V50EdSsGfx9LF7skvz8+e72iePHu0HO0idd3zg6Cxe6kS8ffDD3d9369lu44AJXxkMP5fUVZM2SuzGmUPvhB5doP/8cKlZ0Sf7mm12SX7XKDfa2fbsbt2fAgLzv7+qr3VhD69a5YaJDxW6zZ4wp1Nq1cwOsffedu4vUnXdCnTpw113Qvj0kJsL//hecxA4wYYIbFfO++4JTXl5ZcjfGRLX27V3tfdEiaNrUDVFcq5ZrvmnbNnj7qVMHRo+GKVNg6dLglZtb1ixjjClUVqxwibh48eCXvX8/nHMONG7shoHObRt+VqxZxhhjMtC4cWgSO7i7VT38MHz9NcyZE5p9BCrb5C4ik0TkDxHJ8HYA4rwgIutF5BcRaRn8MI0xpmAYORIaNHAncE+cCF8cgdTc3wZ6ZrH8EuAcbxoJvJL3sIwxpmCKjXV3slq3zt0WMVyyTe6q+g2wL4tV+gBTvIunfgDKikiVYAVojDEFzSWXuH78Dz8M+7LKniEUjDb3qsBWv+cJ3ry/EJGRIhIvIvG7d+8Owq6NMSbyiMCzz8KBA/Doo+GJIV9PqKrq66raWlVbV6xYMT93bYwx+apJE7jhBncD9bVr83//wUju2wD/e6RX8+YZY0yh9sgjULSou3AqvwUjuc8Bhni9ZtoBB1R1RxDKNcaYAq1yZbjnHpg1y3WPzE+BdIWcDnwP1BeRBBG5QURuFJEbvVXmARuB9cAbwM0hi9YYYwqYsWOhenUYNw5SUvJvv7HZraCqA7NZrsAtQYvIGGOiSLFibtyZa6+FqVNh6ND82a9doWqMMSE2YAC0aQP33gtHjuTPPi25G2NMiJ12Gjz3nBti+Jln8mmf+bMbY4wp3Dp0gP793aiU2/KhP6Eld2OMyScTJrjbDf7zn6HfV7YnVI0xxgRHnTowY4arxYeaJXdjjMlHl1+eP/uxZhljjIlCltyNMSYKWXI3xpgoZMndGGOikCV3Y4yJQpbcjTEmCllyN8aYKGTJ3RhjopAld2OMiUKW3I0xJgpZcjfGmChkyd0YY6KQJXdjjIlCltyNMSYKWXI3xpgoZMndGGOikCV3Y4yJQpbcjTEmCllyN8aYKGT3UDXGmKwkJcH27ZCQAFu3wrZtcPx43sps1w66dAlOfJmw5G6MKbySkmDnTpe0t25NTeC+v1u3uuUpKcHd7113WXI3xhRiqq7WvGwZLF/u/v76Kxw7BjExEBvr/uZkOu002L3bJfAdOyA5Oe0+ixeH6tXd1KNH6uNq1dzfqlXdOnkRE5O37QNgyd0YExlOnoQ1a1wC903Ll8OePanr1KkDTZtCqVIuKScnu9q373H6KSnJlZt+foUK0K1b2qTt+1u2LIiE7zgEiSV3Y6LdiRPw+++wcSNs2gRbtriar6prbsjob3bLihWDkiX/OpUokfH8kiVdbdeXNA8cSK2J+/6uWOFiBShSBBo3hj59oHlzaNbMJfUyZcJ3HAsYS+7G5IcDB2DtWlcz3bjRJa9y5TKeSpfO2c/2lBTXLrxpk5t8Sdz3eNu2tG3GsbFQtKhrnhD569+M5vn/BfflcPiwm1QDi1PEJf+iRdPWxitUgBYt4LbbXCJv3hzq13dxmlyzo2dMsJw86ZKpL4n7/q5ZA7t2BV6OiEvwWSV/XzLfuBE2b3bJ1t9ZZ7kmjAsvdH9r13ZTnTpu2WlB6gWtCkePpib6QKbERKhZ09XGmzeHKlWiohkk0lhyNyYnkpLgjz9g3bq0yXvtWpdok5JS161Y0dVAL73U/a1fH+rVg7p13Xp//hn4tGoV7N/vHh875pon6tSBhg3hssvSJu+aNV3tOD+IuOaW4sWhUqX82acJiCV3U7Ds2uV6S/h+4vsSi/8UF5ezmuDx4y5h79rlJv/H6ae9e9M2QxQtCuec49qD+/VLm8TLlct8n6ef7mKtWjXnx+DECbe9MVmw5G4i159/Qny8m376yU0JCdlvFxOTcdL3TUWKwL59qQl7//6MyylZEs4800316kHHjqnP69Z1Sbx69eA1cQTKErsJQEDJXUR6Av8CYoA3VXVCuuXDgKeBbd6sl1T1zSDGaaLd4cPw88+pSTw+HtavT11+9tkuuZ53nmunjYlxbbc5nfbscc0a5cq52rYvWaefKlXKe19mY8Io2+QuIjHAy8BFQALwk4jMUdVV6Vb9QFVvDUGMJtocP+66v/mS+E8/werVqT06qld3Sfz6693fVq2ybuIwxvxFIDX3NsB6Vd0IICLvA32A9MndFCbJya57X05OCvqmAwdS260rVnQJvF8/97d1a1dzNsbkSSDJvSqw1e95AtA2g/WuEpFOwFpgrKpuTb+CiIwERgLUqFEj59Ga8Ni2DRYvdjXsxYvdBSd792a9zemnp+2+V7kyNGjgrv4rX941iZx3nqulWzc4Y4IuWCdUPwWmq+pxEfk78A7QNf1Kqvo68DpA69atA7zyweSrAwdcU8nixanT9u1uWWys65t81VWub3K5ci5ZZ9QXu1gxS9rGhFEgyX0bUN3veTVST5wCoKr+1bg3gafyHpoJOf+2b18i/+231OX16kHXrq6G3aaNO5GZX/2njTF5Ekhy/wk4R0Rq45L6AGCQ/woiUkVVd3hPewOrgxqlydjJk9n3Djl69K/z9u2DpUtd88rJk66sM8+Etm3huutcIm/d2k5iGlOAZZvcVTVJRG4FPsd1hZykqitF5BEgXlXnALeJSG8gCdgHDAthzIVPcrKrYS9YAF99BT/8AAcPpr0aMlBxcW5EvaZNYexYl8jbtHEj4lkzijFRQzTQQX+CrHXr1hofHx+WfUc8VXe5+VdfuYT+9deulwnAuefCBRe4XiZZXaiT0VSsmEvuxpgCS0SWqGrr7NazK1Qjgaobl2TBgtTauW+gqdq1oW9f1/bdpYsb9MkYY7JhyT1cEhLSJvPff3fzq1SB7t1Tk3nt2uGN0xhTIFlyzw+qrhfKd9/Bt9/CokVuVEFwfb67dIG773Z/69e3tm9jTJ5Zcg+Fo0dd98Jvv3UJ/bvvXA8VcMn8/PPhpptc7bxJk/wfeMoYE/UsuQfDzp0ukfuS+dKlqV0Mzz0XrrgCOnRwU716VjM3xoScJfecUoWVK13Tii+hb9rklhUt6i74uf12Vztv397dQswYY/KZJfdAHT8OH34IEye6mjm4C386dIBbb3V/W7SwsbaNMRHBknt2/vgDXn0VXnnFNb80aAAvvww9e7qeLNbEYoyJQJbcM7NsGfzrXzBtmrut2SWXwJgxcNFFltCNMRHPkru/5GT49FOX1L/+2l3VOWIE3Hab66JojDEFhCV3cMPcTpoEL77oTo7WqAFPPw033GCDZxljCqTCndzXr3cJfdIkdw/PCy5wSb1PHzd2uTHGFFCFL4Opusv9J06Ef//bJfEBA2D0aHevTmOMiQKFK7mrwj33wJNPulEV//EPd6VolSrhjswYY4KqcCX3Bx5wiX3kSHfS1O4qZIyJUoUnuT/yCDz2mOv98sorNp6LMSaqFY4M98QT8OCDMHQovPaaJXZjTNSL/iz3zDNw771w7bXw1luW2I0xhUJ0Z7qJE2H8eLjmGnj7bYiJCXdExhiTL6I3ub/0krsB9FVXwdSp1m/dGFOoRGdyf+01GDXKXYw0fbrdFNoYU+hEX3J/6y248Ua49FL44ANL7MaYQim6kvs778Df/gYXXwwzZkCRIuGOyBhjwiJ6kvu0aTB8OHTrBjNn2gVKxphCLTqS+4cfwuDB0LkzzJ4NxYqFOyJjjAmrgp/cP/4YBg1yt7n79FM3BrsxxhRyBTu5z57tRnRs2xbmzoWSJcMdkTHGRISCm9znzoX+/aFlS5g/H0qVCndExhgTMQpmcv/8c7jySmja1D0uXTrcERljTEQpeMl9wQK44gpo2BC++ALKlg13RMYYE3EKXnKvUgU6dYL//AfOOCPc0RhjTEQqeAOuNGjgmmKMMcZkquDV3I0xxmTLkrsxxkQhS+7GGBOFLLkbY0wUCii5i0hPEVkjIutF5O4MlhcRkQ+85T+KSK1gB2qMMSZw2SZ3EYkBXgYuARoCA0WkYbrVbgD+VNWzgeeBJ4MdqDHGmMAFUnNvA6xX1Y2qegJ4H+iTbp0+wDve4xlANxGR4IVpjDEmJwJJ7lWBrX7PE7x5Ga6jqknAAaB8+oJEZKSIxItI/O7du3MXsTHGmGzl60VMqvo68DqAiOwWkS25LKoCsCdogQVfpMcHkR+jxZc3Fl/eRHJ8NQNZKZDkvg2o7ve8mjcvo3USRCQWKAPszapQVa0YSIAZEZF4VW2d2+1DLdLjg8iP0eLLG4svbyI9vkAE0izzE3COiNQWkdOBAcCcdOvMAYZ6j/sBC1RVgxemMcaYnMi25q6qSSJyK/A5EANMUtWVIvIIEK+qc4C3gKkish7Yh/sCMMYYEyYBtbmr6jxgXrp5D/g9Pgb0D25oWXo9H/eVG5EeH0R+jBZf3lh8eRPp8WVLrPXEGGOijw0/YIwxUciSuzHGRKECl9yzG+cmn2KoLiJficgqEVkpIqO9+Q+JyDYRWeZNvfy2uceLeY2IXJwPMW4WkV+9OOK9eWeIyH9EZJ33t5w3X0TkBS++X0SkZYhjq+93jJaJyEERGRPO4ycik0TkDxFZ4Tcvx8dLRIZ6668TkaEZ7SuI8T0tIr95McwUkbLe/FoictTvOL7qt00r73Ox3nsNQbmSPJP4cvx+hur/O5P4PvCLbbOILPPm5/vxCwlVLTATrrfOBqAOcDqwHGgYhjiqAC29x6WAtbhxdx4C7shg/YZerEWA2t5riAlxjJuBCunmPQXc7T2+G3jSe9wLmA8I0A74MZ/f0524CzPCdvyATkBLYEVujxdwBrDR+1vOe1wuhPH1AGK9x0/6xVfLf7105Sz2YhbvNVwSwvhy9H6G8v87o/jSLX8WeCBcxy8UU0GruQcyzk3IqeoOVV3qPT4ErOavQzL46wO8r6rHVXUTsB73WvKb/xhA7wBX+M2fos4PQFkRqZJPMXUDNqhqVlcrh/z4qeo3uG686febk+N1MfAfVd2nqn8C/wF6hio+Vf1C3XAfAD/gLjDMlBdjaVX9QV2mmuL3moIeXxYyez9D9v+dVXxe7ftqYHpWZYTy+IVCQUvugYxzk6/EDW/cAvjRm3Wr9zN5ku9nPOGJW4EvRGSJiIz05p2pqju8xzuBM8MYn88A0v5TRcrxg5wfr3Aex+txNUmf2iLys4j8T0Q6evOqejHlZ3w5eT/Ddfw6ArtUdZ3fvEg5frlW0JJ7RBGRksDHwBhVPQi8AtQFmgM7cD/1wuUCVW2JG6r5FhHp5L/Qq3mEtR+suCueewMfebMi6filEQnHKzMich+QBLznzdoB1FDVFsA4YJqIlA5DaBH7fqYzkLQVjEg5fnlS0JJ7IOPc5AsRicMl9vdU9RMAVd2lqsmqmgK8QWrTQb7HrarbvL9/ADO9WHb5mlu8v3+EKz7PJcBSVd3lxRoxx8+T0+OV73GKyDDgMuBa7wsIr7ljr/d4Ca4du54Xi3/TTUjjy8X7GY7jFwtcCXzgF3dEHL+8KmjJPZBxbkLOa6N7C1itqs/5zfdvp+4L+M7MzwEGiLtjVW3gHNyJmVDFV0JESvke4068rSDtGEBDgdl+8Q3xeoG0Aw74NUeEUpoaU6QcPz85PV6fAz1EpJzXBNHDmxcSItITuBPoraqJfvMrirvJDiJSB3e8NnoxHhSRdt5neIjfawpFfDl9P8Px/90d+E1VTzW3RMrxy7Nwn9HN6YTrqbAW9216X5hiuAD3E/0XYJk39QKmAr968+cAVfy2uc+LeQ0hPsOO622w3JtW+o4Tboz9/wLrgC+BM7z5grvb1gYv/tb5cAxL4EYOLeM3L2zHD/clswM4iWtLvSE3xwvX9r3em4aHOL71uDZq32fwVW/dq7z3fRmwFLjcr5zWuCS7AXgJ7yr1EMWX4/czVP/fGcXnzX8buDHduvl+/EIx2fADxhgThQpas4wxxpgAWHI3xpgoZMndGGOikCV3Y4yJQpbcjTEmCllyN/lCRMr7jbK3M91ogacHWMZkEamfzTq3iMi1QYp5srgRLE8L5giFXtnXi0jl9PsK5j5M4WZdIU2+E5GHgMOq+ky6+YL7TKaEJbBMeFcx7lHVsjncLkZVkzNZtgi4VVWXBSNGY9KzmrsJKxE5W9y4+O/hLhypIiKvi0i8uLHyH/Bbd5GINBeRWBHZLyITRGS5iHwvIpW8dR4TkTF+608QkcXixgg/35tfQkQ+9vY7w9tX8wxiW+TNnwCU8n5lTPGWDfXKXSYi/+fV7n1xTRSRX4A2IvKwiPwkIitE5FXvqtZrcOOt+MYTP91vX4jIdeLGDF8hIo978zJ9zcZkxJK7iQTnAs+rakN1Y+LcraqtgWbARSLSMINtygD/U9VmwPe4K0MzIqraBhgP+L4oRgE7VbUh8ChuVM+s3A0cUtXmqjpERBrjLqc/X1Wb4240P8Avrm9Utamqfg/8S1XPA5p4y3qq6ge4qx+v8co8cSpYkWrAY0AXL64OInJZDl+zMZbcTUTYoKrxfs8HishS3KXfDXA3d0jvqKr6hrhdgrvBQkY+yWCdC3BjhaOqviEacqI7cB4QL+7uPZ1xox8CnMAN1ObTTUQW44aC6Aw0yqbstsACVd2jqieBabgbTUDgr9kYYsMdgDHAEd8DETkHGA20UdX9IvIuUDSDbU74PU4m88/y8QDWySkBJqnq/Wlmurb5o+qdyBKR4rjxR1qq6jYReYyMX0ugAn3NxljN3USc0sAh3Oh7vrsbBdu3uDvvICJNyPiXwSnq3e3IS97gBhG7WkQqePPLi0iNDDYtBqQAe8SN0nmV37JDuFs0pvcj0MUr09fc879AX5gxPvbNbyLNUmAV8BuwBZeIg+1FYIqIrPL2tQo4kM02bwG/iEi81+7+MPCliJyGG2nwRmC7/waquldE3vHK30Hq3boAJgNvishR/G4ZqKoJInI/8DXuF8KnqjrX74vFmIBYV0hT6HiJMlZVj3nNQF8A52jq/UiNKfCsNmAKo5LAf70kL8DfLbGbaGM1d2OMiUJ2QtUYY6KQJXdjjIlCltyNMSYKWXI3xpgoZMndGGOi0P8DG4aEoQ/0NvsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title(\"Training Loss vs. Validation Accuracy\")\n",
    "plt.plot(iteration_list, loss, 'b')\n",
    "plt.plot(iteration_list, val_accuracy, 'r')\n",
    "plt.xlabel(\"Training iteration\")\n",
    "plt.legend(('Loss', 'Validation Accuracy'), loc='upper right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save trained model\n",
    "\n",
    "Now that we have the parameters of the trained model in the workspace, we will export the deploy model using the `mobile_exporter` class. In Caffe2, pretrained models are commonly saved as two separate protobuf (.pb) files (init_net and predict_net). Models can also be saved in db formats, but we will save our model as protobuf files, as this is how they commonly appear in the Model Zoo.\n",
    "\n",
    "For consistency, we'll save these in the same unique directory that the checkpoints are in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run init net and create main net\n",
    "workspace.RunNetOnce(deploy_model.param_init_net)\n",
    "workspace.CreateNet(deploy_model.net, overwrite=True)\n",
    "\n",
    "# Use mobile_exporter's Export function to acquire init_net and predict_net\n",
    "init_net, predict_net = mobile_exporter.Export(workspace, deploy_model.net, deploy_model.params)\n",
    "\n",
    "# Locations of output files\n",
    "full_init_net_out = os.path.join(checkpoint_dir, init_net_out)\n",
    "full_predict_net_out = os.path.join(checkpoint_dir, predict_net_out)\n",
    "\n",
    "# Simply write the two nets to file\n",
    "with open(full_init_net_out, 'wb') as f:\n",
    "    f.write(init_net.SerializeToString())\n",
    "with open(full_predict_net_out, 'wb') as f:\n",
    "    f.write(predict_net.SerializeToString())\n",
    "print(\"Model saved as \" + full_init_net_out + \" and \" + full_predict_net_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations!** You have made it through Part 1 of the tutorial. In Part 2, we will load the model that we just trained and do all sorts of fun things like running inference on our testing LMDB, running inference on a given .png, and continuing training for increased performance.\n",
    "\n",
    "Thanks, and see you at Part 2!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}