{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mixed Programing * \n",
    "\n",
    "In this tutorial we show how to combine NDArray and Symbol together to train a neural network from scratch. This mixed programming flavor is one of the unique feature that make MXNet different to other frameworks. The `MX` term in MXNet also often means \"mixed\". \n",
    "\n",
    "Note that [`mx.module`](./module.ipynb) provides all functions will be implemented. So this tutorial is mainly for users who want to build things from scratches. \n",
    "\n",
    "## Training a Multi-layer Perception. \n",
    "\n",
    "We will use a two-layer perception as the example to show the idea. Note that the codes apply to other objective functions such as deep convolutional neural networks as well. We first define the network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"102pt\" height=\"442pt\"\n",
       " viewBox=\"0.00 0.00 102.00 442.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 438)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-438 98,-438 98,4 -4,4\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\"><title>data</title>\n",
       "<polygon fill=\"#8dd3c7\" stroke=\"black\" points=\"94,-58 -7.10543e-15,-58 -7.10543e-15,-3.55271e-15 94,-3.55271e-15 94,-58\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">data</text>\n",
       "</g>\n",
       "<!-- fc1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>fc1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-152 -7.10543e-15,-152 -7.10543e-15,-94 94,-94 94,-152\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-126.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-111.8\" font-family=\"Times,serif\" font-size=\"14.00\">128</text>\n",
       "</g>\n",
       "<!-- fc1&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>fc1&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-83.7443C47,-75.2043 47,-66.2977 47,-58.2479\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-93.8971 42.5001,-83.897 47,-88.8971 47.0001,-83.8971 47.0001,-83.8971 47.0001,-83.8971 47,-88.8971 51.5001,-83.8971 47,-93.8971 47,-93.8971\"/>\n",
       "</g>\n",
       "<!-- relu1 -->\n",
       "<g id=\"node3\" class=\"node\"><title>relu1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-246 -7.10543e-15,-246 -7.10543e-15,-188 94,-188 94,-246\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-220.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-205.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu1&#45;&gt;fc1 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>relu1&#45;&gt;fc1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-177.744C47,-169.204 47,-160.298 47,-152.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-187.897 42.5001,-177.897 47,-182.897 47.0001,-177.897 47.0001,-177.897 47.0001,-177.897 47,-182.897 51.5001,-177.897 47,-187.897 47,-187.897\"/>\n",
       "</g>\n",
       "<!-- fc2 -->\n",
       "<g id=\"node4\" class=\"node\"><title>fc2</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-340 -7.10543e-15,-340 -7.10543e-15,-282 94,-282 94,-340\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-314.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-299.8\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "<!-- fc2&#45;&gt;relu1 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>fc2&#45;&gt;relu1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-271.744C47,-263.204 47,-254.298 47,-246.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-281.897 42.5001,-271.897 47,-276.897 47.0001,-271.897 47.0001,-271.897 47.0001,-271.897 47,-276.897 51.5001,-271.897 47,-281.897 47,-281.897\"/>\n",
       "</g>\n",
       "<!-- out -->\n",
       "<g id=\"node5\" class=\"node\"><title>out</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"94,-434 -7.10543e-15,-434 -7.10543e-15,-376 94,-376 94,-434\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-401.3\" font-family=\"Times,serif\" font-size=\"14.00\">SoftmaxOutput</text>\n",
       "</g>\n",
       "<!-- out&#45;&gt;fc2 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>out&#45;&gt;fc2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-365.744C47,-357.204 47,-348.298 47,-340.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-375.897 42.5001,-365.897 47,-370.897 47.0001,-365.897 47.0001,-365.897 47.0001,-365.897 47,-370.897 51.5001,-365.897 47,-375.897 47,-375.897\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7fdaec88c190>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import mxnet as mx\n",
    "num_classes = 10\n",
    "net = mx.sym.Variable('data')\n",
    "net = mx.sym.FullyConnected(data=net, name='fc1', num_hidden=128)\n",
    "net = mx.sym.Activation(data=net, name='relu1', act_type=\"relu\")\n",
    "net = mx.sym.FullyConnected(data=net, name='fc2', num_hidden=num_classes)\n",
    "net = mx.sym.SoftmaxOutput(data=net, name='out')\n",
    "mx.viz.plot_network(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The free variables include the weight and the bias from each fully connected layer (`fc1` and `fc2`), the example for variable `data`, and the label for the softmax output `out`. We list all these variables' name by `list_argument`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['data', 'fc1_weight', 'fc1_bias', 'fc2_weight', 'fc2_bias', 'out_label']\n"
     ]
    }
   ],
   "source": [
    "print(net.list_arguments())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run forward and backward, we need to bind data to all free variables first. We can create all `NDArray`s and then bind them as we did on the [Symbol tutorial](./symbol.ipynb). There is also function named `simple_bind` that simplifies this procedure. This function first inferences the shapes of all free variables by using the provided data shape, and then allocate and bind data, which can be accessed by the attribute `arg_arrays` of the returned executor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('fc2_weight', (10L, 128L))\n",
      "('fc1_weight', (128L, 100L))\n",
      "('out_label', (100L,))\n",
      "('fc2_bias', (10L,))\n",
      "('data', (100L, 100L))\n",
      "('fc1_bias', (128L,))\n"
     ]
    }
   ],
   "source": [
    "num_features = 100\n",
    "batch_size = 100\n",
    "ex = net.simple_bind(ctx=mx.cpu(), data=(batch_size, num_features))\n",
    "args = dict(zip(net.list_arguments(), ex.arg_arrays))\n",
    "for name in args:\n",
    "    print(name, args[name].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change `ctx` to GPU we can let the arrays be allocated on GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('fc2_weight', (10L, 128L), gpu(0))\n",
      "('fc1_weight', (128L, 100L), gpu(0))\n",
      "('out_label', (100L,), gpu(0))\n",
      "('fc2_bias', (10L,), gpu(0))\n",
      "('data', (100L, 100L), gpu(0))\n",
      "('fc1_bias', (128L,), gpu(0))\n"
     ]
    }
   ],
   "source": [
    "ex = net.simple_bind(ctx=mx.gpu(), data=(batch_size, num_features))\n",
    "args = dict(zip(net.list_arguments(), ex.arg_arrays))\n",
    "for name in args:\n",
    "    print(name, args[name].shape, args[name].context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we initialize the weights by random values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for name in args:\n",
    "    data = args[name]\n",
    "    if 'weight' in name:\n",
    "        data[:] = mx.random.uniform(-0.1, 0.1, data.shape)\n",
    "    if 'bias' in name:\n",
    "        data[:] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Before training, we generate a synthetic dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEACAYAAABVtcpZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnX9wXFeV5783trWDBE5idYhmzNjEYbADS4gp8oOEHx0Q\n4PRSCkOqNj8gE3pqIbNgKjveaSVmoJ60m4S4VWhmiedXIOm1kuXH7szCeBJl7JioM3HWYciMoiSY\ndmyLSI6QTLcdU6MWSI44+8fr13r9+v2471e/193nU9Uldfd9955+Up9z7znnniuICAzDMEx7ck7U\nAjAMwzDRwUaAYRimjWEjwDAM08awEWAYhmlj2AgwDMO0MWwEGIZh2phAjIAQ4kEhxEkhxAsO7S4X\nQpwVQnwqiHEZhmEYfwS1EsgB+LhdAyHEOQDuA7AvoDEZhmEYnwRiBIjoIIDXHJp9CcDfAvhFEGMy\nDMMw/mlITEAI8TsAPklEfwVANGJMhmEYxplGBYb/HMCduudsCBiGYWLA6gaN814A3xVCCAAJANcJ\nIc4S0V5jQyEEFzNiGIZxCRF5mlwHuRIQsJjhE9GmyuMiqHGBL5gZAF37pnwoihK5DCx/9HKw/M35\naGb5/RDISkAI8W0ASQDdQohpAAqADgBERA8YmvNMn2EYJiYEYgSI6BYXbf8wiDEZhmEY//CO4QBJ\nJpNRi+ALlj9aWP5oaXb5vSL8+pOCRghBcZOJYRgmzgghQDEIDDMMwzBNBhsBhmGYNoaNAMMwTBvD\nRoBhGKaNYSPAMAzTxrARYBiGaWPYCDAMw7QxbAQYhmHaGDYCDMMwbQwbAYZhmDaGjQDDMEwbw0aA\nYRimjWEjwDAM08awEQiSUgkYGlJ/MgzDNAGBGAEhxINCiJNCiBcs3r9FCDFReRwUQrwriHFjRy4H\n9PerP+MEGyeGYSwI6qD5HID7AYxYvD8J4INE9EshxDYA3wRwVUBjx4d0uvZnXNCMEwBkMtHKwjBM\nrAjsUBkhxEYA/0BElzq0Ow/Ai0T0uxbv86EyQVMqqYYgnQYSiailYRgmYPwcKhOFEfgTAG8nos9b\nvM9GgGEYxgV+jEBQ7iAphBDXAkgDeL9du4GBgervyWSybc/+ZBiGMSOfzyOfzwfSV8NWAkKISwH8\nHYBtRHTcph9eCTAMw7ggLmcMi8qj/g0hNkA1ALfaGYC2IW7ZOnGTh2GYhhFUiui3Afw/AG8XQkwL\nIdJCiNuFEJrf/6sA1gH4SyHEuBDin4MYt2lpVCqprHKPa2orwzChE0hMgIhucXj/cwA+F8RYLUGj\nUkllU0Pt5HGbWcSZSAzTVDQ0MMxUSCQak68va2zs5HG7x4D3JDBMUxFYYDgoODAcM3glwDCxJxb7\nBIKCjQDDMIw74pIdxLQrnF3EME0LG4F2IixlzdlFDNO0sBEIk5jMkEsLJQw9M4TSQ7trlbWVfG7l\nTqeBbNZ9tlOpBAwMqA9eRTBMJHB2UJjEJFMmN55D/4F+gHqRUZQVZW0ln1u5vWQ7lUrAbbcBo6Pq\n864uziZimAhgIxAmMSktnd6aBp7KI33PKDDwsZWsHSv5GiF3LqcagN5e4JprIr9HDNOucHZQ2DQg\nZbK0UEJuPIf01jQSnRZjaHL09QF790afwnnkCLBjBzA8DGzeHJ0cDNMCcHZQnGlA0FRz9+TGbcbQ\nXDZ799bIU40XLDTYJ793r7oS2Lt35bWYxFAYpp1gd1DYNMC1kt6aBsplpJ8pA5tL9jN8gzzVeEG5\njMxEl/kKIYzVjNl9iUkMhWHaCiKK1UMViXFNNksEqD9dUCwXKXswS8VdivX1Vn0Xi+prxaJJxzbv\nWQrj4RqGYaiiNz3pXF4JtAoeVxyJzgQy12TUFYToqr1eH0fQ+tavCuxm7sb3ZFYTXmsqcakKhvGO\nV+sR1gPtvBIIYSZcnemXXczWtdcVpX4FoF8VuFkJaNelUtafr1hUx1QUd/fA4yqIYVoF+FgJRK70\n6wRqZyMQgjLLHswSBkDZg7V9FstFyt6TomLnynh1riFFqVfyXg1VsagaALvPp31+t/eA3UhMm+PH\nCHCKaJyQdWu4cH9YpY8O7R9A/6FBZKkXme3fARIJDD0zhP4D/ci+T7EOEvvBTG79awCwe7f6c/t2\ndu0wjCSRHzQvhHgQwCcAnCTrM4a/AeA6AGUAnyWi54MYu6WQ9YnL+NtLJWD3biQWFpDp7AQ2A+hc\n6SL9PID9QPr6a6rXpLemV35+LAQFbPb5jJ9lYMBb3xwXYBhveF1C6B8A3g/gMgAvWLx/HYDHKr9f\nCeBZm76CXik1D7JuDSt/u5nv3sq94taF4hQ/cOGKqYlT6K/349bhuADTxiDq7CAiOiiE2GjT5HoA\nI5W2PxJCnCuEuJCITgYxfssgmydvnFGbZQal1b0DWFgAOjvrs4bcZuLs3g0MDqp96mfrVq/rMLqk\nqnsTAGQ2V+Q6dUrdQazVEnKbJWR2D3h1wDCONCpFdD2AE7rnM5XX2Ajo8ZjmWeoEclcD6U6gquoS\niTqlrCnjvs192Htkb12cQKr8hBULC+puX03h6spC5Ep7V5T+NZlat9NfVAxfPq8agFTK28Y6GVcT\nwzB1xHKfwIBOeSWTSSSTychkaSge8+RrZtbXGK7XzYZzR9R2+VfyGD02Wte+rh/9THr7drXSp3Gm\nDQCKov7UK9wdO1DKjyKXnUTfrh/U7GhOJBIr42r9feAD6s/h4XB3JTNMC5DP55HP54PpzKsfyfgA\nsBHWMYG/BnCj7nkBwIUWbYN3mJFDvnyTY/xsNc91vnLt9UKxQMpjGVLu7qXidKHSSZGKuxTK7lNW\n7pGTn13f93RBTTnV+isUKPuHW1bSU7NZKnZCbWP2N2CfPsN4BnHYJwDgrQBetHgvhZXA8FWIIDBs\nlS/fNOiCpk4GreazWgRbs/ek1Db3pCovmGwCKxTsg8GV94vTBUo9kqq7v8YAcHVMsz0L+xR1f4LV\nmAzDWBK5EQDwbQA/B7AIYBpAGsDtAD6va7MbwDEAEwDeY9NXKDcpjisBVzLplLSTQZPpt27mrjcW\nLlYARCtGJ/WIxSzfQa6az8MrAoZxTeRGIMhHWEYgjrhS5i5WAisdOKRcukz7rI47XTtbd2XMTPq2\nTBllGEYKNgLNiJkP3oBvF5bLGT2RvULX5FHGFO+rKp7pM0zg+DECscwOagtyOSTuHEQmmzXfnVsq\nqdk071OqKZVWWKZ26rNjzHLmjdkzpRJyD9yG/rP1mUPAyo7i8lLZOhvJCc7YYZh44dV6hPVAG60E\nbN0eLmbMdisGqfMCtHb3pKjQrQaLC8WC5Ww/jvEVhmlnwO6g1qImW0bCN27nq69m7exTbI1O1ZDc\no5Z6rjMsAfnqC0U1k+jxlx+n1COqsWEYxh9sBJoE2Rm0UQHXZfJYXugta8dMtjpZJVYm+n0Ixs+p\nvdc70ksYAF2QvaAqG8Mw/vBjBDgm4BEvJRZsd/bqqCmrACD38A7VT/8wkPnyY9YDGPzt+n6cZKye\nMCbZtxna59t/fD8O/OwAyktlDFw7UPNe5n0ZdJzTgS9d8SXc/8/3Y/jjw7ZyMQwTMl6tR1gPNMlK\nwEvmjldfutNKwLFfD64cs1WE0ziaq+eLj36xmkVkKiOngTJMoIAPlWk8+pUAAO+F1zwNrmb6lG7q\nQ256L8pLZQz+0yCyvVnz2fzQENDfj9IuBblruqTkLC2UcNv3b8PosVEoH1LQtaYL5bNlDD5lMU6p\nhIG/uRmDrx9A5n0ZXNB1gfU4FXmQzXJhN4YJAD+HypwTtDDtguY+0ZdGzo3nGjN4pTpm7uEdqntJ\nANnerHUqaToNZLPIvX1BlfOgenpXaaGEoWeGUFoo1V2S6Exgz+/vQbY3CxDUcchmnFwOeOIAAKBz\nTWf13tjJw2miDBM9HBMIAKMPP/wBK+Pd1AdMJ51n9okESl9Mo/xnN0PJA32/tYChrqHqCgIwj1Fo\nhq60UEJXR+0Koi4mkk5jO5XRdRmQvnK7vfweq6UyDBM87A5qBgI4HKV6fvCaFPDey9F/aLDq5vHi\nxqr2Z+WCgo/zCfgwGIZxReRnDDMBYKf4AjgcxbhaKa9Zed1LHCO9NV1zRoCZspbKhjL73AEeBuPr\noByGaQM4JhAXNMWXM4krBOxDT3Qm0NXRhcGnBmviGHYxArM+MhNdSNw5aC4zVEOhjyGY9m/yuUs3\n9WHonhRKN/V5/IS67hsdr2GYZsNrWlFYDzRJimjghJU2Wek3u0+p3YBmku7pOu3VqSKoAdP+TfrQ\nt/NbooJLXDDtAHjHcHPSEAVV2elb3OVc+TMIeaTqGDnsO9C/3vSHATFMA2AjEDQN2swkpeD8yuJy\ntu4Xu9IRRmTOU1DGFFKetC63zTBMDIwAgG1Qzw1+GcCdJu93A3gcwPMAXgTwWZu+wrpP8jSo5r2U\nMg5BlqBm165dPy6udy0n70Jm2phIjQDU4PIxqAfNr6ko+i2GNgqAr1V+TwA4BWC1RX/h3SlZ4qRQ\nHGSROkpS0g3jtn87H39x2nkl4GfsemHMjSXHBJh2IGojcBWAx3XP7zKuBqCeN7y78vtFAF626S+k\n29SayMyWlScVtZbPk4plG6nzf2WuierkMAtjyTEFph3wYwSC2CewHsAJ3fNXAVxhaPNNAD8UQvwc\nwBsB3BjAuLGnmqO+oQ+J7+4NZfOT1G5lYfhpglVOv13/ppVHozo5zGIXcsN3czNMk9GozWI7AUwQ\n0bVCiIsBPCGEuJSI5s0aDwwMVH9PJpNIJpMNETJoqop1TR6ZP1WPbAy6XIJeEZttjCotlAAClA8p\n2H6FdTkHK2XpWGLaOK5OGZcWStj9o92AALZfsT2SzVoy8jNMs5HP55HP54PpzOsSQntAdQf9o+65\nmTtoFMA1uuc/BPBei/5CWS6Fgqy/3nDiV1iYuT6CcIeYuX1k0ji119kdwzDhgojdQT8G8DYhxEYA\nswBuAnCzoc1PAfQCeEYIcSGAtwOYDGDsaHEob1AzC21AwTSz2XwQ7hAzV5H+Nasx0lvTKC+VASE5\nfqmE0kO7kbsMSL8/mpUDw7QdXq0H1c7etwE4AuAogLtoJRj8eVrJCPoHABMAXgBws01f4ZnLoIlT\nFlGIOK0EAkNRKHu188qBM34YphbwoTLtSRTF0UIdc2AApaFB5DK9SPd/x7J/mQqmDNNOcBXRNkX2\nzGIzLJW5VTXTyuu5d5fRf8j6DAJfbN+ORFcXMuk0YGNgOOOncZSWlpCbm0O6pweJjo6oxWFCgI1A\nE+NHGVoaEKs4R+X19C4FsDvFzA+Sh81wxk/jyM3NoX9SDd9lNmyIWBomDNgd1KZ4XQnwQS/tBa8E\nmgM+Y7hVKZXUQ9lL1vX93ZwBIIU2GzcqeqvXmZYm0dGBzIYNbABaGDYCccbuoBmticdDU8I8bCVw\nw8S0PKWlJQxNT6O0tBS1KG0HxwTijEQJBq9xgTCDq34C1s0Ku038wbGH6OCYABM47Xiu79D0NPon\nJ5HdtImVmAfYiPrDT0yAjUCL0/IKOSYBa1ZiTJRwYJixpOUPWpeImzQCDqAyzQrHBFqclt9YFVXp\n6jaDVzqtC7uDGFta3p3ESMExj3jD7iDGF3YpnS3vTmpxgkq9TPf0ILtpE9I9PQFJxsQFNgI+aYWc\neDtFn96aRtZFmYjSQglD+wdQyg7YbnJjGoOWepmbm7NsI2MoOObRunBMwCetkBPv+ghJG3LjObXA\n3H4gI7oaco4CY026pwfl5WUUl5Yw8LOfYfv69VVFrvn5y8vLGJyaAsA5+u0IGwGfhBV4bWTmo6yi\nl5EpvTUNlMtIvwEcrI0BiY4OdK1aVVXyXatWVRW9tkpQNm6suno4ANyGeD2IQP+AeqhMAcDLMBwt\nqWuTBDAO4CUAYzZ9+T9hoQXIZokA9WdciKNM7UBxcZGyU1NUXFz0fL0yOUnK5GRNH2b9ZqemCGNj\nlJ2a8i030zjg41CZIAzAOQCOAdgIYA2A5wFsMbQ5F8BPAKyvPE/Y9BfajYoCr6dgxfHQsjjK1A4o\nk5OEsTFSJidN3zcqcz9Gw6/B8UpU47YKfoxAEO6gKwAcJaIpABBCfBfA9ZWVgcYtAP6OiGYqWr5t\nIoZeYwaSpfUbShxlaicWlpcxND1d56rR3DrFpSX8ZGEB7+zsxNCrrwKQ8/EbXUBRxAW81g5i95V/\ngjAC6wGc0D1/Faph0PN2AGuEEGMA3gjgG0T0cABjx5pSCSg/k4byvhberMVUCVIh6fvavn49ulat\nQnl52VRRammb+0+fxoEzZ7D0m99A2bgR5eVllJaWHGWJQ/E27TO4TUGNg+zNTqMCw6sBvAfAhwF0\nATgkhDhERMcaNH4k5HLA4J0JZLMZJD7mrY+YlMZhJAhSIRn7ymzYgNLSErpWrapTlNrsva+7GzuO\nH8fwxRdj76lT6J+crAkE69EbGa8KOEi8rkDiIHuzE4QRmAGg/+u9pfKanlcBlIjo1wB+LYT4JwDv\nhhpLqGNgYKD6ezKZRDKZDEDMxhNERQOr0x6Z+KFXSH5XBWbKTa8ozfrf3NWFxy69VL1uzZq66/Vo\nRqa8vFw1LGG7U8Jw3ZgZj3ZwEeXzeeTz+WA68xpMoJVA7iqsBIY7oAaGLzG02QLgiUrbTgAvAniH\nRX+hBE6aFQ7GNidhZNnog6da/6mJCalgqlXwOHP0qG3QOUgalXlkNU4rB58RZWCYiJaFENsB7Iea\nKfQgEf1UCHF7RbAHiKgghNgH4AUAywAeIKLDfsduBzgY25wYZ/JBzE71LqJ0Tw/yZ85g9PRp5Obm\nHF0pRveSNoMe+NnPPMnihUa5bqzG4fiBOVxAjmEagFkBNreGwdjezfVWbUtLS9g9o3pv9buJgyYO\nLpo4yBAWXEDOBRJntwfWj1kbq+uCkouJJ2YF2LSZ6W2FglSBN2P9nkRHB9I9PcjNzeFIuWxb/8eq\n9o9+R7FdfSG/yNQwChuuf2SBVz9SWA+EHBMIaterXT+aH19RatsUi0SplPl1Ue7G5bhDNBQXFyk1\nMeHLT675v9926JClb9/JF+73fRla2R8fBxDljuGgH2EbgaAUnl0/mkJXlNo22uupVP11Vv01QkFz\nOYjoCKIkhGZIrIyAWaDUzbjG64NS6Fo/hfl5NhA+YSMQM4JU6DIrjjANGhM/CvPzlJqYoML8PBFZ\n1waye88uU8eo5I3jBZXlo89w4npF/vBjBNqmimhDq3JaZPR4yfSx22sQ1B4CzkBqLnYcP47R06cB\nAI9deikSHR0YuOii6vtaAPQDa9cifeQICr/6FbKbNtX4ws0yaKxKS+89dao63p4tW6rlqe12JMsE\nYbWx+7q7kTzvPN7wFRFtExgO4zxy2WCun6CvpqDNDFc6DWSzKwai3YPLS0slTE8PYWmptW/A8MUX\nI7VuHYYvvrjmde1wmN0zM+ifnKwagC1veIOpsreqQQSgpuxEuqcHqXXrqumoMsFkmUCwFqjd3NXF\nAdso8bqECOuBkNxBYbg87Hz8Zu3C8Llrn6tQsA46twtTU1kaGwNNTbXnDdDcK8rkJGWnpujQa6/V\nuHGM7ew2UznFARoRTA6LOMvmFXBMIBrssn2M7TQDpFfaboySlRHTGyIZg+R1nGZgcbFIU1NZWlxs\nQuFtkFVadu307xl9/G76agUF2opnJrAR8IkfxWem4GVWBU7Gw9iXcTVhNCZujYqTfO26mmgksgrV\nbuauD/ra9afvQ59Wmjl61JVC96tA42BE4iBD0LAR8IkbxSeTGiqzKjAqbVmlb/V+UDTzSiBowlYW\nRveNW9eKdtiMUblrhkGfEaTvo7i4SFuefbbmWtlx/aZ1ejEixcVFyhw9Sr3j47arl3bGjxFom+wg\nO9Lqsbgol9Wgql32kF1GjkzVUH0mjv56Y7/GvowZPEFUKHWSL+4sLZUwN5dDT08aHR3Bp3zJ1Jox\nyiAjkxaY/cRaQs+6/ZhevgH9U/9mOY5VmeWF5WUAwAfXrkVfdze+84tfQNm4EQDqzhQ2ViD9yHnn\nYfE3v8GHzzvPMsvH7PNr/WhlMLT3ZEsyeKkflJubqx6Ss+P48WqlVCYY2AhAVXxdXaoS7uqyV4J2\nytePAnVS+sYUV7Ox2u3sgbm5HCYnVcu5YUPwlstKYekVvVGG4zPfxMmpL6O8/DouuWinab+acu1Z\ntx+/e/preO+b3oSOTZ92pRhLS0sYn58HAFx7/vnYe+oUBqemakpTLFikcebm5vAXs7MAgNmzZ/Hg\nyZOm5w6Ylcbu6+7G3lOn0NfdXdPGymAajYOXcwPSPT0oVj6vMSOKCQCvS4iwHogoMBylG8RNLMHO\n/dNu/vwgAsFeXD6HJ++lsTHQ4cl762QYnhynG8dup+HJcctxtN9n52dqrjWTxeozGktJuzk0Xh9L\nMLp1rO6H1pfmRtJnEdnFI1oxCBtHwDGB5kavvM0MQrGolqBQFHtDEWbpiVaNFcgoKaOCs1L0Zm3d\njKP5+HvHx6vXW6W9yihgmTZ298NoLHrHx2vk0+RVJict/0FaMQgbR/wYAXYHRYTedaN3BZnFHHI5\nYHAQUBR7d4+VOyqIncXNesKZk49exkdtdHXcuv4deH3Vl3GryTVW7g6ncUpLS3jml78EABw4c6Z6\nRkBPj/rPof00jlNaWsJthUJ1R69VmWrNh+90kpheztzcXE1s4Zpzz8WBM2eQfOMyijNfx7++dhWA\nSvXiZv0HYXglEBVWG83MJlTaRrBMhmwzhqxo55VAEBvIvM5mZd1V+iJwvePjpjWArLA6Ycxqs5c2\neze6c6zkMktB1dxhN47dvjKuxT8Iu4MaA6J2BwHYBqAA4GUAd9q0uxzAWQCfsmkTzl0ywYti86oM\njdcVi0S9vVStNmqHU1XSIGIAmqEpFNxfG2cD4aSIw3RXyBog7YjHD/7Lv7hy3RC539Slf13vznEz\n5uJikZ47ejfdMJ53TNlkd1BjiNQIQK0/pJ0xvAbqGcNbLNr9EMCjURgBM0XlRYl6Vbxm12nnDdgZ\nAbt4QJDKV7/j2C1xDUjLzMTDnKkax7dSiHpfu1GmMJWo3gjoA78yufhuYymtups7LkRtBK4C8Lju\n+V1mqwEAdwD4zwAeisIImCkqs9m5k1INciUgE+xtlII1Wwk00t0UBjIzcSclG6TyslKcZqWhrWr4\nyMgsg5mrR8v8SU1MSF3v5ErSl4gO4m/ht30rE7URuAHqwfHa888A+Iahze8AGKv8novLSsCIV4Ub\n9DkBfvt2g98d0HEmCAVupbzc+PvdZuhYXa8RxCEvZsbFqqaQ0WDIyGmMVYSxKgvCQLaKIWkGI/C/\nAVxBK0bgBpv+QrpNznhVuF6UZVxmz404tKZRaIpmfr4Q2OzdSnnJ+vvDcDdZKVy78g92ZSSc0Pq3\nK3ERhGEKQoEHYUiaET9GQKjXe0cIcRWAASLaVnl+V0WgXbo2k9qvABIAygA+T0R7TfojRVGqz5PJ\nJJLJpC8ZrfC6w9Z4XTPv1G1m2Y1MTw9hcrIf69alcPr0KDZtyga6k1ifbgpAqmSFVTkFs9RV2dIL\nRo6Uy9hx/DiGL74Ym7u66t7Xl3jIbtrkasduaWkJu2dmsLC8jM5VqwCoJSmUjRtrUk29yO718wbZ\nZxgyNIJ8Po98Pl99Pjg4CCISnjrzaj20B4BVWAkMd0ANDF9i0z4Sd5AZQQR53cyW4zCzjoMMYbG4\nWKTJSYWOHs3Q5KQSeBDSa7qp2WrCrC/ZgHAQKwErnPo2SzX1SqvMwuMAYpIiegTAUQB3VV67Heps\n39g2ksCwGUEEed0YErdGJ6ggtB8Zmo0gD5Yx3kevMQYzmcz6cgoIa4TperHaMewlBdWrDIx7IjcC\nQT7CNAJulWrQ2UJuxw8yHdWrDM2Gk6I2e9+yPk9ABtOL8bBTsrKzejvs6grJGCK3/TLhwkZAEqcv\ntdua/V5STN0Qxkqg3TGblVvW52nQfXQzjtUOYddjSszCSzMzNLZzJx195ZWVtg41ggrz84EYKcYd\nbAQkcfqyuT29y9g+iJmjVQE5VurBILMSsJu5h5H14ub/pri4SDeM5y0L2K00tJ6hSMcbKoKN7dy5\nMru3EFa/AqgpLCcJu4b8wUYgIKyUrb7Oj13tniCUtdedxUFi9zniYpDClMMurmDm7tAC0lbBaDsX\nyeJikQ4fztLwcJGKRRt3j+4Da7V7fjy+rSYd1kyBm81Q7FxB+g1e2pilmZnqyV5HpqdNdzlalaOQ\nhd1I/vBjBLiKqA6rKpzayWMLC8Dll1sf/BLEqVxhnRjmBruCkHEpFhmmHFaVOwGgr7sb+TNn0Nfd\nXU3zXF4uY2pqEACwalVXXVrqrd2r0XNmPz7avb2uv7m5HE6e7McNNwCJRAZD03N1p4IBqPnAF9/x\nOZz9t4M4fXoUx4/vwOnTowCA7+HGlWqnxn8k3c/02rXqr4aKprm5OYyePo3UunXo6+7G0KlTSN9x\nBxIdHfjJCy/gwJkz+ONzzsFjJicw6aunbl+/vpo+KouXE8eYgPBqPcJ6IKZVRIPMqgkjQB0kcV8J\nGGfPXq53GzzW0M9YtRXD0aMZGh/vpaNHM6bZPtrM3WxlYVZfSL8SqL4/W+uLNNsY59elYhcULszP\n0/WHDtErZn5Rk+tlxmCCA+wOCp8glV+YaZpOckatxP2WcigWiUZG5NJArT6rU8qmnTvIrCja5KRi\n2j47NUVrx35Aj4zfIb1vQevzxGsv0sjETnrp6Fdq+m5UITZTZe3wjyvj0mG3TziwEWgywlTETgYm\nSgNE5D+PP5slWru2SIqSpdlZ+xs4PFykG29UVwx6nDZvuVW0Vu2Li4s0MrHTuV/djdPk+P6z19LY\nGOiR8TtqrvFy/wKrp+OQteBmJTAzP8tVRQOEjUCMiLLgWxDv+0HGwPidyc7OqgZg7dqioyE7fFhV\nmIcPOyvMsGbYUisM3Y0zrgRm52d8l2QOup5OcXaWsiMjVJyd9TyrCHJTH8NGIHK87iD2Qpx3/IZh\nYIx9aspjZCRrOY6XQnJulauXz2om1+JikaYOK7Q4rFh25seFsrioZhMNT467XgmUZmZMP2R2ZESV\nZ2TE8x9DC6moAAAcM0lEQVSdzxcIFjYCEeO1lpAdVv0Ui3LnEDQDMvfKaPRklIdmKA4fzkr/LbRr\nJiZStn1r4w8PFz2vevSzYOOM2Myl4mcl4GvGbTHjqFkJeICDw8HDRiAgwtyhG2TJiDivBmQpFuWO\n1/Qz49YU9fCw88pgcbFIExMpxyCsplR//OMUDQ8XaXbWeo+AzFkExjGcZv3GPv1kOjkSku+Qg8PB\nw0YgIMJUrm77jnuapl+0+xHGJjhN8c3OqnEDLTZgVPJW19kFYY3GQmtj1s5uA5nV2IdPzFBqZIoK\nsyaz5GKRFocV1X1kFyguFGhx2zb65pNPej7MxhOS/5h18rTCP3TEsBEIiDD/F2X7jsv3QS9HWL5+\no1srqHGsZstuD5uxKidh9OmbKXu3J5Jp7R95RDHNaCIi05mEaX+Vre3/cOWVpExO1u4C9vC5a7D7\nI5nNdLz4/BjXsBEImCgVcVy+D3o5wpLJeJ+DGieIE8bsXEFOcQOr6/V96I3D7PwMPTJ+B7109Cv0\n0kvqnoPnnjOJZ8j+Y+pWAloJB7uCc8aZuW0cwe6PZCaf1l5RWntpGzFsBAImSkUcl+9D2CsBovqa\nSEGP4zUoarUhzSxuYMX8fIEmJlI0P1+oed1ooObnC9U9ASMTO6vv3/31AuHqLCm73GU2Ge+hjCvI\n6KO3zViyyUwwHUsTSPtjRz27aVH8GAGuHWRCHOr3RI2xDlKUdYK8YlcDyI5cDrj77jR27wauvnrl\n2o6OBLZs2VNzxKQVWk2fX/1qElu3Pl09QrKjI4ENGzLVozDPnMnjvF+N4cwbrsW1G26p9n323+eA\nf+sHLgNQStueATo3l8PkpFpX6Hvfy6C/H1i9uoQbblD7Mj1O8sgRYMcOYHgY6YsuArBSt6ejI4EN\nj1bqA73eVV8gy6R2EKDWHqrWLjKOefPNavt2/lLFFa/WQ/+AerJYAcDLAO40ef8WABOVx0EA77Lp\nKyxj2RS4WYWEmc3UCMJyB1n17/S+MaDsJ1YxP1+gZ5/dYrlbWFspvPbaobrNZCNPpahQLFD2YJaK\nZefNJ2YrAS0YXl2xVN5YnFFXIK9vq6RmpVKWN0cfhK6Z5VvckOLiIt0/Pk7z994b3h9V91ki/weO\nEYjSHQTgHKycMbwG6hnDWwxtrgJwLq0YjGdt+gvtRjUDhYL6vSwUnNt6/X7FJe5gJOiMKLclNKzc\nR17vl91uYavMo5GnUrT2blD2oGRw1eK9uphE5UOc3qm6s2aezKz8o1n0UZVxJEX3j48TxsZoeHLc\nPs5ivFlhKOy4/gNHSNRG4CoAj+ue32W2GtC9fx6AEzbvh3KTmoV2Wgm4wcv33u5zapVI7767WHVv\nWwVz3Rhmff9mmUVWz6syl4srKwAZZAKvRNWbMf/KofpYhe7m1u1hGEnR4lrQ/L330v3j4zR31zY6\n+AObmEgj/rma8R84ZKI2AjcAeED3/DMAvmHT/k/07U3eD+UmNQth7zhuVsIKGo+MZOuMS90GLg8G\nyCkoLZ0yrB3bOLtI2SxRqVCks/cqdGK8kpIqG3ittDsxblLxVCdMndz6QHAmQwTQa3f0+sq8iopW\nLlXhxwg0NDAshLgWQBrA++3aDQwMVH9PJpNIJpOhyhUngjiYBojP4S9BYXdftMNdenrS1QCsVZuO\njjQefjiBW29NY9MmYMuWNLLZ2nilPtC6YUPGU6KAU1Ba9u+Tm5vDrvFxrPvOj7DreztweT6H5Ogg\nFqeAuS9XDrDJZIBSyT7wWhmw514FuE5Bz/8qA58rqTdWd3N7lgxy6wPBvb0AgPPOuwavHv0Ozv71\nIH7xR2W85bIB+RsDqLLaBLrDwvh3bWby+Tzy+XwwnXm1HtoDqjvoH3XPTd1BAC4FcBTAxQ79hWMq\nJWil2XMrfRYnZFJB7Wb+RhYXi3T4mELDTytULNeXdtC307/uxpWj//vYzVCLi4s0tnMnEUDHtqTo\n1KFC7UpAFr1Py2JpYyqHfiWgO0zm7L0KEaD+tLvejEb69HU3mlcC4bmDVmElMNwBNTB8iaHNhooB\nuEqiv7DukyMcb4o32vfZeLCVzJfbbeZP9mCWMKAGaK2MTLVQ3TGFsgezpIwp1Wvc4Li7eLZQf8C1\nBVafR6+0F2cKdHpnihZnaoMcejm0sbXr6sY1ZBvZlsomw9+okTOUNvlSR2oE1PGxDcCRiqK/q/La\n7QA+X/n9mwBOAfhXAOMA/tmmr/DulAOtPHuOw2fzWzpD+z5L6kPr/nUzdq1Prciclmap7CqSsi9L\nhekiDQ+rQWSrlcDw06ryV55UXAV1nXY21yhVmQyhQoHGUlnqRv1ZCyfGFTp2O+jEuGJrdLQSGNpp\naSfGlfpxLWIIVUU/U3/8pFU2lK3xdvNPa9U2Dv/4DSByIxDkI0ojEBdaNatOVgardlYrger7ku4Y\n/SzfLK9eP76dzNXVxZmC1Lg1f9eimnljpRhnZ2uNj63CNFjHsZT6uaxKUFv2VSzS6Z0pOvgD0OSk\n4jyeIZvI7H3j55Kpr2TXjyVx+AePEDYCLUar7a/RlHNhuiglg5msMvLrlbseGd+9vsyDfqxikSxX\nAlZKzHi99ruyq7hSCiKbpcW1ag6+vl9tBv7II4rtHgajQTGzjlo5iPvHx+1XEZrPv5JhdHqnQ20k\nkz+GF3ePY42nIFYCbQIbgRaj1f6frZSzDLIZkETWKwFTl4WDQi9MFyl1j2q43FYEtVpJKPvU+6Ds\nU/38UyMp1d+vQzMCL72k2MY9bCcKlZtWmpmh4clxOvo/t9HiWovGWkcyew1sPrfXOk3S17balyJg\n2AgwsUbWTWP2PZfdC2U3nl5pHT6mVIO5eoyKLXWPqrBT97g/eN5qJaCXy61hsRujDp2F0O/6dVoJ\n6IO8dpjtJfCUsVRB6jO3ubvHCTYCTEtg9j33MgG0W3kMP63Qjd8CDT+tqP1bGCj9SkDaiDm185Gu\nqO/b8Z7ox7FYcZi1f+W5DI2NqfEAO6xKUsgqaE8BYV4J2MJGgGkJAtstbaOMje/JuKqMbYplNXNI\n2VWskVVLEc3sz5iP76As7T6/Xoa6bopFOvunGXrtv/TWpX3WBI2tbmylzam7eqWMQJ3Axki9wx8y\n0IAwQ0RsBJgY0agJm9k4Xmr8mBkMo5I3zsI1VxGuztboKeVJ1Qj07uldyT7S968T2kx/2uk+q5VA\nsVyksS+oWUFaULfuRmlZQ19I1RsmzR2UydDZr2TMXTpajGHKkAVlkQV0emfKVokHmhrKEBEbASZG\nNGoSZzaOtn8glXJYDRQtz0WxVfLVcTuL1KvUrwSqWVC6MtBmK41iWXU1obNYs+fBi+5TxhTqzoCe\n+oOrzVcClQ819oUUdWdMVjwymy8qbca+kKr9LCYCT01l6eAPKsZI9oOw0vcNGwEmNsRlJWDn5tEn\nxBh1Xua+AuEzvXS1kqlT8mbjOsUBzN7XZNvyh1k6dMgme9PkWmO+/7ee7KW1d6sb1Wzvl4Wc1d3D\nL9oJYrES0PfjlO5ph9+ZAxsRNgJM6xBkXKDGpWPI2FEUorvvXsn/197/0N+os93eh1J1/Wl7HTL3\nFaj3oZWDXzAAUsYU+bpBupWAnd4zM2R6f7r+EBrp0tNVIdSMnpfyvbb+edkAtp8UUe3my2Yn1cEx\nhOapIsowTgRV/TTRmUDXRAb9dwJdQn1N3+/AADA9rVaVfK6Yx/Fn92DwzgQy9w2j623A8MeHa+Ua\nz6H/QD8emgYKv84Dbx/Fjn3Ant/fAwAonilj8Ll+lMvAwMfsBU90JrDn8xnk1thUJi2V8IV8GW94\nr4KbtqarVVC7u/sA1FYnvdqieqptsc5cDqu/PIiu24Dzzn8bXv9MEUs9pdp+SiWUh2/DiStHga32\nlTe9HuUJoFrFdK5y5CbgssonnwfrD6/WI6wHeCXQ1lglnNi1tWpjla+voT/JS9mXte9LN3v/4O+v\nrAQ0tN3AvX1Fmp21qQqqixsMP63Qc4cz1WqlNVRmt2fvVev4jI/bz9jNcNpQdvZehV67Qz1m8tjt\nK30bi8ed3plydvN4WMItLhbpxLiijuMhbZZZAewOYloNmRV+EF6AYrlIypMKKWMritjKf24MKBuN\njBZbHRmxdo1oLp7UIym68VugsTHQjd8yiV1UOtcOgRkbA01M6Mo5SChdKb1cMQavPJehyUmlphqo\nvnicdFqnXSqqgampLB27HW3vygkCNgJMyyGjwKTaOAVuddlAmiKWCSobS0Lo5QlkJVBhcbFIR49m\naHy8t+ZQ+qD94MbSGvoZuiaH40rAZXnXunHCDPC2ePCYjQDTVhSLVBOc1V4z+447bQbLZonQXaAL\n7kjRoZcrfRkMh/65fiexzK7iIFwcmoKemNBVHbVRalZv2cnidxew7cAW1BWd81sj3I4WDx77MQIc\nGGZigZsTB3M5YOiFHdXg7GOffswyoJzemq75aSSdBh4q7EXh/FH89/+TxGNfrg9IakFhjdGz/UhO\nG36fBvoP9CP/Sr4aLM6N5/CJnjJOnhgEYB/sLC2UkBvPIb01jURn7Q3o7u7DmTN5bNjwVZx3XlIN\nvnZYn7dpdS9mZnZjamoQVCxi/f5OzF0HvPkd29HRkUBHR6JWPi/B1kRCbS/5h6w57vF7AEZHgVQq\nnAAvB48tYSPAxAI3WUF9N5Xw2NI7QRcuYfjjwyiVgHIZUJT673iiM4HMNdYdJhLADwbS2PEwMHyr\nerFe6WeuydQZknIZKD+Txs03A+gF+jb34TsvfQe9F/Vi9NgocuM5AKpRWP0RBTdsylpmzWjKv3y2\njMGnBqtj6s9NPnVqL06fHsWb3nQ5Vq3qsvwsWl99N6VRLidQLqvGNQHVwp7z4QWs+SVwwX/7PlaP\nHas9p9jsxnhJz5L4Q5pmOmm3x8p4+D2XOKjDu1sRr0sI/QPqyWIFAC/D5HzhSptvQD157HkAl9n0\nFdKCiYkzbjwJRhdPUCt9sx2/puMbxtPvFdCCzE59GD+L8VQy7ZAbbR/D1FS2WmZ6ZCRbe58qN+/+\nRxXz+kK6TCOtpMPr23ptq356dmNJ/CE97SnQaokrijt52gRE6Q4SQpwDYDeAjwD4OYAfCyH+nogK\nujbXQT1g/veEEFcC+GuoB9QzDAB3EzXjzFyb/ff1AUND3ieLxhWAGaWFEsrvziFzXx/K796L0kK6\nRh6tj641XbZ9aK4f/bWJzgRKJWDoL4ClpTRefBF41xV96LhSbbt2NfDk08COh8v449kSBvorH7Iy\n+07freBXvVm1z83Q3Rt1jNXpNM4HgPNzWJVO4y02N2luLocT4/1404/y6NixR/6GSvwhe3rSWF4u\nY3m5jKWlkukeB0ueeaayvPHwB2bM8Wo9tAdUZf647vldMKwGoCr9G3XPfwrgQov+QrGUTOtjtiLQ\nz2i9lHgwoh0Mo+0sVvbVzmZlVhMy2UeKop5o9q0n1H0MWlv9wTQrgwaf+WIsBBd0Dr/r1UDYgeMm\nBxEHhtcDOKF7/iqAKxzazFReOxnA+AwDwDz2pw8+PlQABg+t7Oo9UjqCHft2YPjjw9ic2Fy95tTC\nKcsgLZ5PA/sBurAPOJnEwm/1YahrqNo20ZlAemsat33/NoweGwVQv6pIb02jvFRG+WwZpYVSzRj6\nz7CwkMPk5Ch2fyCF6yorhj+6sg+/9+/247LfLq7Mog2zb308QZtlm71mR0dHQl0BnK/64bX7uLxc\nxqpVXdUYh5s+9bjeYZxIAHv2rMQFmMCIZWB4YGCg+nsymUQymYxMFqZ5MPNE1Cib7wLYD+ANaeBj\nwI59O6qK+rFPP1Z15eRfyVsq8JtvBn78OvDVT3Tj6X0ZLL17AP9SGMRqKuOP3z8AANh9MIfRY6Po\n3ZBC34Z0nYsq0ZlAV0eXqdtI/xmWllTZL+/uw6lSDks9aSz9ci/W0wEUf34Ab/qtC0yDujVZN5X3\ntdfOnMljy5Y9ckpbJ0xPRZbl5XK1bwDeyjwA9dlIMnBwt0o+n0c+nw+mM69LCO0B1R30j7rnMu6g\nAtgdFDmyJ2a1CkavSaFYoNQjur0GHlw5ZsdV6g+Ur7p3dlnvPXDCuJFrclKp7u6t+XyVPmfP1Jd4\nWFws1u4z0L3uxs1jrGLayDIPXFbCGkS5WQzAKgDHAGwE0AE1++cSQ5sUgMdoxWg8a9NfaDeKqcXP\nAfBxIwyDZnXgjNX5xdU2RdWff/hwlmZnVUOg+fKtykPY+fNllZ/T39PugPiakhQxxVel0hYnUiOg\njo9tAI5ATQG9q/La7QA+r2uzu2IsJgC8x6av0G4UU0srrQScFGDdLmCJWKpbI6lXskaFZXmvA9zJ\n6uXvabVCCAs/5w7wSsCayI1AkA82AowXnBSgl70FbpWq0W0jpbBsrNHiorqaGB6uPdzGrzI0Xh+2\ncq32XznA5uAP0FDD0w6wEWAYB7ysBMyus2P2TIFGnkrR7JnaIx4dS15bjKEZlRtvrBxzaags6lWB\nSrlVAkw71cbTUk6lSlMzrmAjwDAhkT2YpbV3q6d3efXJW606iuUiKWNKzcH0eupWArqdv2YKdH6+\nQBMTqoK1Q2rmb3GIvBfFrV8JtHIlzyhhI8AwPrE8Q6CsHjyjzZxtD7C3OYfArsKpdr6A42rD4axf\nzcUyMZGqu8a14jU58lE2MMu++8bDRoBhfGI1iy+WizT8tEKHj6kpmUFmVBXLRcrsy1Dvnl46NH1I\n2u1kJoP+3IGalYDPwLOXOAdn8TQeP0YglpvFGMYLduWYndDX8NEXrMwdyaH/h4N4vTeLzMUJx9LU\nbkh0JnBB1wU48LMD6FjVYblBzU5Wjbm5HF59dQibNmXR1bVZ1ziN118vY+6jZbzZbZ0e1G62k93g\n5eu8YabxeLUeYT3AKwHGI0HM0vVnCWezEieT+Uyzla1c6oTdLF1mZm51vcy+CCZ6wO4ghglm30P1\nDOB7slJu9GbYcCejuK0MhfHz+TEoTHj4MQLsDmJaBqcDZGSoLe3srn0Y+HFxaci4caxcOMbPJ+Pq\nMatdxMQXoRqR+CCEoLjJxLQ2fg+tCpOhZ4bQf6Af2d6sbwNnhtvqolH1ydgjhAARCS/XnhO0MAwT\nJ0oLJQw9M4TSQsmyjXYiYi7XODmWlkqYnh7C0pK1XIA6A89qB8WEgDZrn5sL7sNrKw82AM0BGwGm\npdHKQ2vn/pqRTgPZrLcy9aWSeppZyV6X18khq3w1F5dXV5ATPT1pbLI5AxmQN1hhEgcZWhWOCTAt\njYzP3k+Zeolz1QGoh9HnX8mjb7PucHVEn0YpEy+Ig48/DjK0KmwEmJYmiGCxHWanmZmx98hejB4b\nRfKtSWQSGW+HqkREHAxWHGRoVTgwzDANIIgsn1aEg8jB4CcwzEaAYZjImJ4ewuRkPzZtyjbNyiiO\n+DEC7A5iGCYy2M0TPb5WAkKI8wF8D+rRkq8A+I9E9EtDm7cAGAFwIYDfAPgmEX3Dpk9eCTAMw7gg\nyn0CdwE4QESbATwJYKdJm9cB7CCidwJ4H4AvCiG2+ByXYSJHNj2UYeKMXyNwPYA9ld/3APiksQER\nzRHR85Xf5wH8FMB6n+MyTOQ0YpMZw4SN35jAm4noJKAqeyHEm+0aCyHeCuAyAD/yOS7DRI5seijD\nxBlHIyCEeAKqP7/6EgAC8BWT5pbOfCHEGwH8LYA7KisCSwYGBqq/J5NJJJNJJzEZpuH42WTGMH7I\n5/PI5/OB9OU3MPxTAEkiOimE6AEwRkSXmLRbDeBRAI8T0f9w6JMDwwzDMC6IMjC8F8BnK7/fBuDv\nLdo9BOCwkwFgGEYOmcJ4DCODXyOwC8BHhRBHAHwEwH0AIIT4bSHEo5XfrwHwaQAfFkKMCyH+VQix\nzee4DNPWyBTGYxgZeMcwwzQhXIaC0cNlIxiGYdoYPlSGYRiG8QQbAYZhmDaGjQDDMEwbw0aAYRim\njWEjwDAM08awEWAYhmlj2AgwDMO0MWwEGIZh2hg2AgzDMG0MGwGGYZg2ho0AwzBMG8NGgGEYpo1h\nI8AwDNPGsBFgGIZpY3wZASHE+UKI/UKII0KIfUKIc23anlM5UGavnzEZhmGY4PC7ErgLwAEi2gzg\nSQA7bdreAeCwz/FiTVAHP0cFyx8tLH+0NLv8XvFrBK4HsKfy+x4AnzRrJIR4C4AUgG/5HC/WNPs/\nEcsfLSx/tDS7/F7xawTeTEQnAYCI5gC82aLdnwHIAOAjwxiGYWLEaqcGQognAFyofwmqMv+KSfM6\nJS+E+A8AThLR80KIZOV6hmEYJgb4OmNYCPFTAEkiOimE6AEwRkSXGNrcC+AzAF4H8AYAbwLwf4no\nDyz65NUCwzCMSyI5aF4IsQvAaSLaJYS4E8D5RHSXTfsPAfivRNTneVCGYRgmMPzGBHYB+KgQ4giA\njwC4DwCEEL8thHjUr3AMwzBMuPhaCTAMwzDNTSQ7hoUQ24QQBSHEyxU3klmbbwghjgohnhdCXNZo\nGe1wkl8IcYsQYqLyOCiEeFcUclohc/8r7S4XQpwVQnyqkfI5Ifn/kxRCjAshXhJCjDVaRjsk/n+6\nhRCPV/73XxRCfDYCMU0RQjwohDgphHjBpk2cv7u28jfBd9fx/lfayX93iaihD6iG5xiAjQDWAHge\nwBZDm+sAPFb5/UoAzzZaTp/yXwXg3Mrv25pNfl27HwJ4FMCnopbb5f0/F8BPAKyvPE9ELbdL+RUA\nX9NkB3AKwOqoZa/I834AlwF4weL92H53JeWP7XdXRn7d/5j0dzeKlcAVAI4S0RQRnQXwXaibzvRc\nD2AEAIjoRwDOFUJciHjgKD8RPUtEv6w8fRbA+gbLaIfM/QeALwH4WwC/aKRwEsjIfwuAvyOiGQAg\nolKDZbRDRv45qFl0qPw8RUSvN1BGS4joIIDXbJrE+bvrKH/Mv7sy9x9w+d2NwgisB3BC9/xV1N9o\nY5sZkzZRISO/nv8E4PFQJXKHo/xCiN8B8Eki+ivEb1+HzP1/O4B1QogxIcSPhRC3Nkw6Z2Tk/yaA\ndwohfg5gAmrJlWYhzt9dt8Ttu+uIl++u42YxxjtCiGsBpKEu4ZqJPweg91XHzRA4sRrAewB8GEAX\ngENCiENEdCxasaTZCWCCiK4VQlwM4AkhxKVENB+1YO1CO313ozACMwA26J6/pfKasc3vOrSJChn5\nIYS4FMADALYRkdPyrZHIyP9eAN8VQgioPunrhBBniSgOFWBl5H8VQImIfg3g10KIfwLwbqi++KiR\nkf8aAPcAABEdF0L8DMAWAM81REJ/xPm7K0WMv7syuP/uRhDYWIWVwFgH1MDYJYY2KawEl65CjIIz\nkvJvAHAUwFVRy+tFfkP7HOIVGJa5/1sAPFFp2wngRQDviFp2F/J/HYBS+f1CqO6VdVHLrpPvrQBe\ntHgvtt9dSflj+92Vkd/QTuq72/CVABEtCyG2A9gPNSbxIBH9VAhxu/o2PUBEo0KIlBDiGIAy1GVZ\nLJCRH8BXAawD8JcVi3yWiK6ITuoVJOWvuaThQtog+f9TEELsA/ACgGUADxBRLMqYS97/rwHICSEm\noC7n+4nodHRSryCE+DaAJIBuIcQ01EymDjTBdxdwlh8x/u4CUvLrkfru8mYxhmGYNoaPl2QYhmlj\n2AgwDMO0MWwEGIZh2hg2AgzDMG0MGwGGYZg2ho0AwzBMG8NGgGEYpo1hI8AwDNPG/H+L/Ctp3hr0\nxQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdaec7e7b90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "class ToyData:\n",
    "    def __init__(self, num_classes, num_features):\n",
    "        self.num_classes = num_classes\n",
    "        self.num_features = num_features\n",
    "        self.mu = np.random.rand(num_classes, num_features)\n",
    "        self.sigma = np.ones((num_classes, num_features)) * 0.1\n",
    "    def get(self, num_samples):\n",
    "        num_cls_samples = num_samples / self.num_classes\n",
    "        x = np.zeros((num_samples, self.num_features))\n",
    "        y = np.zeros((num_samples, ))\n",
    "        for i in range(self.num_classes):\n",
    "            cls_samples = np.random.normal(self.mu[i,:], self.sigma[i,:], (num_cls_samples, self.num_features))\n",
    "            x[i*num_cls_samples:(i+1)*num_cls_samples] = cls_samples\n",
    "            y[i*num_cls_samples:(i+1)*num_cls_samples] = i\n",
    "        return x, y\n",
    "    def plot(self, x, y):\n",
    "        colors = ['r', 'b', 'g', 'c', 'y']\n",
    "        for i in range(self.num_classes):\n",
    "            cls_x = x[y == i]\n",
    "            plt.scatter(cls_x[:,0], cls_x[:,1], color=colors[i%5], s=1)\n",
    "        plt.show()\n",
    "\n",
    "toy_data = ToyData(num_classes, num_features)\n",
    "x, y = toy_data.get(1000)\n",
    "toy_data.plot(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can start the training. Here we use the plain minibatch stochastic gradient descent with fixed learning rate. For every 10 iterations we plot the accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration 0, accuracy 0.210000\n",
      "iteration 10, accuracy 0.990000\n",
      "iteration 20, accuracy 1.000000\n",
      "iteration 30, accuracy 1.000000\n",
      "iteration 40, accuracy 1.000000\n",
      "iteration 50, accuracy 1.000000\n",
      "iteration 60, accuracy 1.000000\n",
      "iteration 70, accuracy 1.000000\n",
      "iteration 80, accuracy 1.000000\n",
      "iteration 90, accuracy 1.000000\n"
     ]
    }
   ],
   "source": [
    "# @@@ AUTOTEST_OUTPUT_IGNORED_CELL\n",
    "learning_rate = 0.1\n",
    "final_acc = 0\n",
    "for i in range(100):\n",
    "    x, y = toy_data.get(batch_size)\n",
    "    args['data'][:] = x\n",
    "    args['out_label'][:] = y\n",
    "    ex.forward(is_train=True)\n",
    "    ex.backward()\n",
    "    for weight, grad in zip(ex.arg_arrays, ex.grad_arrays):\n",
    "        weight[:] -= learning_rate * (grad / batch_size)\n",
    "    if i % 10 == 0:\n",
    "        acc = (mx.nd.argmax_channel(ex.outputs[0]).asnumpy() == y).sum()\n",
    "        final_acc = acc\n",
    "        print('iteration %d, accuracy %f' % (i, float(acc)/y.shape[0]))\n",
    "assert final_acc > 0.95, \"Low training accuracy.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On this section we show how to use the imperative NDArray and symbolic Symbol together to implement a complete training algorithm. The former can be often used for\n",
    "\n",
    "- data containers\n",
    "- programs that requires flexibility, such as implementing the updating rules and monitoring the progress in optimization method\n",
    "- implementing Symbol operators\n",
    "- debugging such as printing and step-by-step execution\n",
    "\n",
    "While the later can be used for defining the object function, which benefits from the heavy optimization placed on Symbol and auto differentation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Parallelism with Multi-devices\n",
    "\n",
    "On the [NDArray tutorial](./ndarray.ipynb) we mentioned that the backend system is able to automatically parallel the computations. This feature makes developing parallel programs as easy as writing serial programs in MXNet. \n",
    "\n",
    "Here we show how to develope a training program using mutliple devices, such as GPUs and CPUs, with data parallelism. In MXNet, a device means a computation resource with its own memory. It could be **a GPU chip** or **all CPUs chips**:\n",
    "\n",
    "- A GPU chip is a GPU unit that contains both computational units and memory. For Nvidia GPUs, we can use `nvidia-smi` to list all units. Usually a physical GPU card only contains a single GPU chip, but some cards may have more than one unit. For example, each Tesla K80 contains two GK210 chips. \n",
    "- All CPUs. Even though there can be more than one physical CPU chips, we still simply treat all CPUs as single device which can be refered as `mx.cpu()` in MXNet. The reason is that these CPUs share the same main memory. \n",
    "\n",
    "Here is a figure (from nvidia) shown the memory structure and how data are communicated between devices. \n",
    "\n",
    "![](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/notebooks/gpu_mem.png)\n",
    "\n",
    "Assume each iteration we will train a minibatch with size $n$. In data parallism, we divide this batch into all available devices according to their computational power. Each device will compute the gradient on a part of the batch, and these gradients are then merged. \n",
    "\n",
    "Now we extend the above training program into multiple devices, the new function accepts a network, a data iterator, a list of devices and their computation power.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def train(network, data_shape, data, devs, devs_power):    \n",
    "    # partition the batch into each device\n",
    "    batch_size = float(data_shape[0])\n",
    "    workloads = [int(round(batch_size/sum(devs_power)*p)) for p in devs_power]\n",
    "    print('workload partition: ', zip(devs, workloads))\n",
    "    # create an executor for each device\n",
    "    exs = [network.simple_bind(ctx=d, data=tuple([p]+data_shape[1:])) for d, p in zip(devs, workloads)]\n",
    "    args = [dict(zip(network.list_arguments(), ex.arg_arrays)) for ex in exs]    \n",
    "    # initialize weight on dev 0\n",
    "    for name in args[0]:\n",
    "        arr = args[0][name]\n",
    "        if 'weight' in name:\n",
    "            arr[:] = mx.random.uniform(-0.1, 0.1, arr.shape)\n",
    "        if 'bias' in name:\n",
    "            arr[:] = 0\n",
    "    # run 50 iterations\n",
    "    learning_rate = 0.1 \n",
    "    acc = 0\n",
    "    for i in range(50):\n",
    "        # broadcast weight from dev 0 to all devices\n",
    "        for j in range(1, len(devs)):\n",
    "            for name, src, dst in zip(network.list_arguments(), exs[0].arg_arrays, exs[j].arg_arrays):\n",
    "                if 'weight' in name or 'bias' in name:\n",
    "                    src.copyto(dst)\n",
    "        # get data                 \n",
    "        x, y = data() \n",
    "        for j in range(len(devs)):\n",
    "            # partition and assign data\n",
    "            idx = range(sum(workloads[:j]), sum(workloads[:j+1]))\n",
    "            args[j]['data'][:] = x[idx,:].reshape(args[j]['data'].shape)\n",
    "            args[j]['out_label'][:] = y[idx].reshape(args[j]['out_label'].shape)\n",
    "            # forward and backward\n",
    "            exs[j].forward(is_train=True)\n",
    "            exs[j].backward()\n",
    "            # sum over gradient on dev 0\n",
    "            if j > 0:\n",
    "                for name, src, dst in zip(network.list_arguments(), exs[j].grad_arrays, exs[0].grad_arrays):\n",
    "                    if 'weight' in name or 'bias' in name:\n",
    "                        dst += src.as_in_context(dst.context)\n",
    "        # update weight on dev 0        \n",
    "        for weight, grad in zip(exs[0].arg_arrays, exs[0].grad_arrays):            \n",
    "            weight[:] -= learning_rate * (grad / batch_size)\n",
    "        # monitor\n",
    "        if i % 10 == 0:\n",
    "            pred = np.concatenate([mx.nd.argmax_channel(ex.outputs[0]).asnumpy() for ex in exs])\n",
    "            acc = (pred == y).sum() / batch_size\n",
    "            print('iteration %d, accuracy %f' % (i, acc))\n",
    "    return acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can train the previous network using both cpu and gpu. It should give similar results as using cpu only. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('workload partition: ', [(cpu(0), 17), (gpu(0), 83)])\n",
      "iteration 0, accuracy 0.170000\n",
      "iteration 10, accuracy 1.000000\n",
      "iteration 20, accuracy 1.000000\n",
      "iteration 30, accuracy 1.000000\n",
      "iteration 40, accuracy 1.000000\n"
     ]
    }
   ],
   "source": [
    "# @@@ AUTOTEST_OUTPUT_IGNORED_CELL\n",
    "batch_size = 100\n",
    "acc = train(net, [batch_size, num_features], lambda : toy_data.get(batch_size), [mx.cpu(), mx.gpu()], [1, 5])\n",
    "assert acc > 0.95, \"Low training accuracy.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the previous network is too small to see any performance benefits moving to multiple devices on such a network. Now we consider use a slightly more complex network: LeNet-5 for hands digits recognition. We first define the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"110pt\" height=\"1276pt\"\n",
       " viewBox=\"0.00 0.00 110.00 1276.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 1272)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-1272 106,-1272 106,4 -4,4\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\"><title>data</title>\n",
       "<polygon fill=\"#8dd3c7\" stroke=\"black\" points=\"94,-58 -7.10543e-15,-58 -7.10543e-15,-3.55271e-15 94,-3.55271e-15 94,-58\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">data</text>\n",
       "</g>\n",
       "<!-- convolution58 -->\n",
       "<g id=\"node2\" class=\"node\"><title>convolution58</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-168 -7.10543e-15,-168 -7.10543e-15,-110 94,-110 94,-168\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-127.8\" font-family=\"Times,serif\" font-size=\"14.00\">5x5/1, 20</text>\n",
       "</g>\n",
       "<!-- convolution58&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>convolution58&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-99.8131C47,-86.1516 47,-71.0092 47,-58.3283\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-109.906 42.5001,-99.9062 47,-104.906 47.0001,-99.9062 47.0001,-99.9062 47.0001,-99.9062 47,-104.906 51.5001,-99.9062 47,-109.906 47,-109.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"71\" y=\"-80.3\" font-family=\"Times,serif\" font-size=\"14.00\">1x28x28</text>\n",
       "</g>\n",
       "<!-- activation87 -->\n",
       "<g id=\"node3\" class=\"node\"><title>activation87</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-278 -7.10543e-15,-278 -7.10543e-15,-220 94,-220 94,-278\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-252.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-237.8\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
       "</g>\n",
       "<!-- activation87&#45;&gt;convolution58 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>activation87&#45;&gt;convolution58</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-209.813C47,-196.152 47,-181.009 47,-168.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-219.906 42.5001,-209.906 47,-214.906 47.0001,-209.906 47.0001,-209.906 47.0001,-209.906 47,-214.906 51.5001,-209.906 47,-219.906 47,-219.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.5\" y=\"-190.3\" font-family=\"Times,serif\" font-size=\"14.00\">20x24x24</text>\n",
       "</g>\n",
       "<!-- pooling58 -->\n",
       "<g id=\"node4\" class=\"node\"><title>pooling58</title>\n",
       "<polygon fill=\"#80b1d3\" stroke=\"black\" points=\"94,-388 -7.10543e-15,-388 -7.10543e-15,-330 94,-330 94,-388\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-362.8\" font-family=\"Times,serif\" font-size=\"14.00\">Pooling</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-347.8\" font-family=\"Times,serif\" font-size=\"14.00\">max, 2x2/2</text>\n",
       "</g>\n",
       "<!-- pooling58&#45;&gt;activation87 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>pooling58&#45;&gt;activation87</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-319.813C47,-306.152 47,-291.009 47,-278.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-329.906 42.5001,-319.906 47,-324.906 47.0001,-319.906 47.0001,-319.906 47.0001,-319.906 47,-324.906 51.5001,-319.906 47,-329.906 47,-329.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.5\" y=\"-300.3\" font-family=\"Times,serif\" font-size=\"14.00\">20x24x24</text>\n",
       "</g>\n",
       "<!-- convolution59 -->\n",
       "<g id=\"node5\" class=\"node\"><title>convolution59</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-498 -7.10543e-15,-498 -7.10543e-15,-440 94,-440 94,-498\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-472.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-457.8\" font-family=\"Times,serif\" font-size=\"14.00\">5x5/1, 50</text>\n",
       "</g>\n",
       "<!-- convolution59&#45;&gt;pooling58 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>convolution59&#45;&gt;pooling58</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-429.813C47,-416.152 47,-401.009 47,-388.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-439.906 42.5001,-429.906 47,-434.906 47.0001,-429.906 47.0001,-429.906 47.0001,-429.906 47,-434.906 51.5001,-429.906 47,-439.906 47,-439.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.5\" y=\"-410.3\" font-family=\"Times,serif\" font-size=\"14.00\">20x12x12</text>\n",
       "</g>\n",
       "<!-- activation88 -->\n",
       "<g id=\"node6\" class=\"node\"><title>activation88</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-608 -7.10543e-15,-608 -7.10543e-15,-550 94,-550 94,-608\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-582.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-567.8\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
       "</g>\n",
       "<!-- activation88&#45;&gt;convolution59 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>activation88&#45;&gt;convolution59</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-539.813C47,-526.152 47,-511.009 47,-498.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-549.906 42.5001,-539.906 47,-544.906 47.0001,-539.906 47.0001,-539.906 47.0001,-539.906 47,-544.906 51.5001,-539.906 47,-549.906 47,-549.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"67.5\" y=\"-520.3\" font-family=\"Times,serif\" font-size=\"14.00\">50x8x8</text>\n",
       "</g>\n",
       "<!-- pooling59 -->\n",
       "<g id=\"node7\" class=\"node\"><title>pooling59</title>\n",
       "<polygon fill=\"#80b1d3\" stroke=\"black\" points=\"94,-718 -7.10543e-15,-718 -7.10543e-15,-660 94,-660 94,-718\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-692.8\" font-family=\"Times,serif\" font-size=\"14.00\">Pooling</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-677.8\" font-family=\"Times,serif\" font-size=\"14.00\">max, 2x2/2</text>\n",
       "</g>\n",
       "<!-- pooling59&#45;&gt;activation88 -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>pooling59&#45;&gt;activation88</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-649.813C47,-636.152 47,-621.009 47,-608.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-659.906 42.5001,-649.906 47,-654.906 47.0001,-649.906 47.0001,-649.906 47.0001,-649.906 47,-654.906 51.5001,-649.906 47,-659.906 47,-659.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"67.5\" y=\"-630.3\" font-family=\"Times,serif\" font-size=\"14.00\">50x8x8</text>\n",
       "</g>\n",
       "<!-- flatten29 -->\n",
       "<g id=\"node8\" class=\"node\"><title>flatten29</title>\n",
       "<polygon fill=\"#fdb462\" stroke=\"black\" points=\"94,-828 -7.10543e-15,-828 -7.10543e-15,-770 94,-770 94,-828\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-795.3\" font-family=\"Times,serif\" font-size=\"14.00\">Flatten</text>\n",
       "</g>\n",
       "<!-- flatten29&#45;&gt;pooling59 -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>flatten29&#45;&gt;pooling59</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-759.813C47,-746.152 47,-731.009 47,-718.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-769.906 42.5001,-759.906 47,-764.906 47.0001,-759.906 47.0001,-759.906 47.0001,-759.906 47,-764.906 51.5001,-759.906 47,-769.906 47,-769.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"67.5\" y=\"-740.3\" font-family=\"Times,serif\" font-size=\"14.00\">50x4x4</text>\n",
       "</g>\n",
       "<!-- fullyconnected58 -->\n",
       "<g id=\"node9\" class=\"node\"><title>fullyconnected58</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-938 -7.10543e-15,-938 -7.10543e-15,-880 94,-880 94,-938\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-912.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-897.8\" font-family=\"Times,serif\" font-size=\"14.00\">500</text>\n",
       "</g>\n",
       "<!-- fullyconnected58&#45;&gt;flatten29 -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>fullyconnected58&#45;&gt;flatten29</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-869.813C47,-856.152 47,-841.009 47,-828.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-879.906 42.5001,-869.906 47,-874.906 47.0001,-869.906 47.0001,-869.906 47.0001,-869.906 47,-874.906 51.5001,-869.906 47,-879.906 47,-879.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-850.3\" font-family=\"Times,serif\" font-size=\"14.00\">800</text>\n",
       "</g>\n",
       "<!-- activation89 -->\n",
       "<g id=\"node10\" class=\"node\"><title>activation89</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-1048 -7.10543e-15,-1048 -7.10543e-15,-990 94,-990 94,-1048\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-1022.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-1007.8\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
       "</g>\n",
       "<!-- activation89&#45;&gt;fullyconnected58 -->\n",
       "<g id=\"edge9\" class=\"edge\"><title>activation89&#45;&gt;fullyconnected58</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-979.813C47,-966.152 47,-951.009 47,-938.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-989.906 42.5001,-979.906 47,-984.906 47.0001,-979.906 47.0001,-979.906 47.0001,-979.906 47,-984.906 51.5001,-979.906 47,-989.906 47,-989.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-960.3\" font-family=\"Times,serif\" font-size=\"14.00\">500</text>\n",
       "</g>\n",
       "<!-- fullyconnected59 -->\n",
       "<g id=\"node11\" class=\"node\"><title>fullyconnected59</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-1158 -7.10543e-15,-1158 -7.10543e-15,-1100 94,-1100 94,-1158\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-1132.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-1117.8\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "<!-- fullyconnected59&#45;&gt;activation89 -->\n",
       "<g id=\"edge10\" class=\"edge\"><title>fullyconnected59&#45;&gt;activation89</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-1089.81C47,-1076.15 47,-1061.01 47,-1048.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-1099.91 42.5001,-1089.91 47,-1094.91 47.0001,-1089.91 47.0001,-1089.91 47.0001,-1089.91 47,-1094.91 51.5001,-1089.91 47,-1099.91 47,-1099.91\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-1070.3\" font-family=\"Times,serif\" font-size=\"14.00\">500</text>\n",
       "</g>\n",
       "<!-- out -->\n",
       "<g id=\"node12\" class=\"node\"><title>out</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"94,-1268 -7.10543e-15,-1268 -7.10543e-15,-1210 94,-1210 94,-1268\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-1235.3\" font-family=\"Times,serif\" font-size=\"14.00\">SoftmaxOutput</text>\n",
       "</g>\n",
       "<!-- out&#45;&gt;fullyconnected59 -->\n",
       "<g id=\"edge11\" class=\"edge\"><title>out&#45;&gt;fullyconnected59</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-1199.81C47,-1186.15 47,-1171.01 47,-1158.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-1209.91 42.5001,-1199.91 47,-1204.91 47.0001,-1199.91 47.0001,-1199.91 47.0001,-1199.91 47,-1204.91 51.5001,-1199.91 47,-1209.91 47,-1209.91\"/>\n",
       "<text text-anchor=\"middle\" x=\"54\" y=\"-1180.3\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7fdaec9f6e90>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def lenet():\n",
    "    data = mx.sym.Variable('data')\n",
    "    # first conv\n",
    "    conv1 = mx.sym.Convolution(data=data, kernel=(5,5), num_filter=20)\n",
    "    tanh1 = mx.sym.Activation(data=conv1, act_type=\"tanh\")\n",
    "    pool1 = mx.sym.Pooling(data=tanh1, pool_type=\"max\",\n",
    "                           kernel=(2,2), stride=(2,2))\n",
    "    # second conv\n",
    "    conv2 = mx.sym.Convolution(data=pool1, kernel=(5,5), num_filter=50)\n",
    "    tanh2 = mx.sym.Activation(data=conv2, act_type=\"tanh\")\n",
    "    pool2 = mx.sym.Pooling(data=tanh2, pool_type=\"max\",\n",
    "                           kernel=(2,2), stride=(2,2))\n",
    "    # first fullc\n",
    "    flatten = mx.sym.Flatten(data=pool2)\n",
    "    fc1 = mx.sym.FullyConnected(data=flatten, num_hidden=500)\n",
    "    tanh3 = mx.sym.Activation(data=fc1, act_type=\"tanh\")\n",
    "    # second fullc\n",
    "    fc2 = mx.sym.FullyConnected(data=tanh3, num_hidden=10)\n",
    "    # loss\n",
    "    lenet = mx.sym.SoftmaxOutput(data=fc2, name='out')\n",
    "    return lenet\n",
    "mx.viz.plot_network(lenet(), shape={'data':(128,1,28,28)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we prepare the mnist dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAA9CAYAAABbalkHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHhVJREFUeJztnXlQVFfaxp9e6I2loWmUtWl2BdlkExCICiGRmBDXxCzG\niZqJmcRkMjNxZlKVSk3FmqVSlck2RuPEbJqKCwomaiRuQFSUIGvYZBGRHbqBbrqbpt/vDz+6NJoE\noS/K5Pyq7h9038v73NPnPvc9555zLo+IwGAwGIzpCf9OC2AwGAzGxGEmzmAwGNMYZuIMBoMxjWEm\nzmAwGNMYZuIMBoMxjWEmzmAwGNMYZuIMBoMxjWEmzmAwGNMYZuIMBoMxjWEmzmAwGNMYIdcBeDze\nlM/rJyIe08F03I4WpoPpmA46bgXLxBkMBmMaw3kmzmBMBF9fXzz++OPIyMiAyWRCbW0t9uzZg9On\nT99paQzGXQUzccZdSUREBDIzM5GUlASLxYKoqCjExsaioaEB5eXlqKmpwVdffQWLxXKnpf4snp6e\nWLx4Mdzd3bF//35UV1ffaUlTyvPPP4+YmBjk5uYiPz8fAwMDnMRxdnZGTEwMoqKifnIfV1dXqFQq\nODo6YmRkBAUFBfjiiy/Q2dnJiabrSUxMxJIlS/Dhhx+isbHRpv97Sk1cJBIhOzsbcrkcNTU1KC0t\nxdDQ0FRKGDf29vbw8/PD7NmzER4eDp1Oh+LiYgCAVqtFQ0MDZxVSKpUiPT0dKSkpcHV1BZ9/c6+X\n2WxGWVkZdu/ejd7eXk503Emqqqrw7rvvYt++fYiOjsaKFSsQFxeHyMhIJCUloaOjA4GBgaitrcWp\nU6eg0+nutOSbEIlESE9Px1NPPYWKigr09PTcaUlTSkBAALKzs5GQkAClUomuri4UFhbaPI5CocB9\n992H9evXw8fHBzweD9cvsT32t0wmg7OzM8RiMSwWC8RiMc6dO8epiQsEAixfvhxr1qxBREQEDhw4\nML1N/Pnnn8eKFSvg6OiIS5cuYfPmzXddZuLh4YGkpCSEhYUhJCQEgYGBUKvVMBgMyMjIAAD09/fj\nwoULOHXqFM6ePWtzDcuWLcPTTz+NqKgoODo6or+/H2VlZSgvL4dAIEBSUhLCw8Mxb9486PV6/Pe/\n/7W5hjtNY2Mjrly5ApFIhNDQUOh0OqSkpMDFxQUeHh5QqVSYOXMmrl69ikWLFuHEiRM4d+4cent7\n74rsXCwW4/7778fSpUvR1taGL7/8El1dXTaP4+joiIyMDKSkpICIMDQ0hG3btqGtrQ138l0BfD4f\nGzZsQFhYGKRSKdzc3ODo6GjzOCKRCAkJCdi4cSPi4+MhEAjGddzIyAh6eno4vbHa2dkhNjYWL730\nEqKiosDj8cat73aYUhPPyspCVFQURCIRlEoloqOj0dvbC61WC6PReEcrXVBQEOLi4hAXF4fExESo\n1Wo4OTlBLBZb9/H29gZwrQLExcUhKCgI/f39qK2ttZkOhUKBBx98EImJiRAIBOjr60NXVxdOnTqF\nTz75BABw9uxZrF69GqmpqUhNTeXMxMViMXx8fBAZGYnh4WFrxuLg4ABvb28oFIpbHjcwMICTJ0+i\npaVlUvFNJhNMJhPKysrw9ttvo7CwEAqFAsHBwfD09ERycjKio6MRHR2N2NhY7Nu3Dzk5OWhtbZ1U\nXFswb948PP744xgZGcGHH37ISQbq7OyM++67D5s2bUJCQgIAQKfTob6+HgcPHuSspTgeRCIR7r33\nXiiVShiNRhQXF9s8Ax1DqVQiKCjotgyyp6cHlZWVaG5u5kQTcO36WbhwIaKjo2FnZwedTsdJgjGl\nJl5YWIjQ0FAoFAoIhUI88sgjCA8PR2trK7q6utDc3IyWlhb09/djZGRkynSFhYXhiSeewJIlS+Dv\n7w+RSPSz+9vZ2cHDwwMhISFwdna2qZaoqCio1WqYTCYUFBTgzJkzcHNzQ0tLCzo6OmA0GtHR0QGh\nUIi5c+dCJpPZNP4YqampCA8PR2hoKObNmwedToerV68CAJycnKBWq+Hm5nbLZmt/fz9UKhX+9re/\n2USL0WhEbW0tamtrIRQKMWPGDHh4eGDRokXw9fXFsmXLkJSUBGdnZzQ3N6OzsxMmk8kmsSeCi4sL\nHnroIbi6uuLjjz9GQUGBzeuzUChESEgIfve73yE+Ph6jo6MAAJlMhlWrVqG4uPiOmvhYq4nP56Os\nrAxHjx6d9E39VphMJvT29qK3txdKpRJmsxnd3d1oa2uz7sPjXRulN1ZXm5ubcfHiRRQUFMBsNttc\nE3CtG8Xb2xsZGRkQCAQwm83Izc1Fd3e3zWNNqYl/9tlnsLOzw+zZs+Hm5oaZM2ciIiICfD4fg4OD\nqKqqwsWLF1FfX4/6+no0NzdDo9FwmqFLpVIsX74cjz/+ODw8PMDj8WAwGKDX63HlyhU0NjZicHDw\nhmOcnZ3h4OCAnJwc1NTU2FRPZGQk3NzcUF1djXfeeQf5+fkIDg4Gn8+33sVHR0dhNBoxOjoKiUQC\nJycnm1+wS5cuxYoVK+Ds7AyBQACBQAAej4fR0VEQEUwmE3Q6HYaGhqDX68Hj8SCVSuHl5QWFQoHl\ny5fbzMSvx2w24+rVq7h69SpKSkowc+ZMmM1mrFq1CsHBwQgICIBMJrtjJs7j8bBw4ULMnTsX58+f\nx6lTp2AwGGwex9vbGytXrkRiYiKGhoZQUFAAuVyOmJgYpKWlYc6cOejp6YFWq+XMqH4KmUyGZ555\nBq6urrBYLDh06BBKSko4KQcA0Ov10Gg0AK61Anfv3m19fnUrSktL0dDQwImWMezt7XHPPfdg/vz5\n4PF40Ov1eO+99zhpJU6pidfV1eHPf/4zgoKC4O/vD7FYjDlz5mDWrFnw9/fHnDlzkJKSAoPBgPPn\nz+Prr79GQUEBWlpaOMvM/fz8kJSUBFdXV/B4PBiNRpSUlKC8vBxFRUU4fvw4Ojo6bjjG19cXAQEB\nOH78uM31KJVKSKVS1NXVobW1FWaz+abnBnK5HAEBAXB1dYVAIIBcLre5iVdVVcHZ2RkSiQQA4O7u\nDj6fj97eXhiNRuh0OnR1daGrqwu9vb2QSCSIjIzEqlWrIBaLrdkP13R2duKzzz6zPgSWSCQQCrmv\n1jweDzwe76bmcWBgINasWQOj0YgjR46gqanJ5rGFQiGio6Oxbt06mM1mNDY2YvPmzfD29sbrr7+O\niIgILFu2DDNmzMDhw4c57TL4MTweD97e3oiPj4dUKoXBYEBZWRlnDw/H+pnH6ltfXx927Nhh8+Tq\ndnFwcEBqaqp1UEJzczPa29s58bE7MsRwLNMGgNzcXEgkEkRFRSEkJASJiYlITExEeno6YmNjkZOT\ngy+//BJVVVU3ZcS2QKlUwsnJCQKBAEQEvV6P/Px8fP7552htbYXRaLzpmJaWFk6ahsC1loFQKIRC\noYCDg8NN34vFYoSGhiIiIgItLS0oKiqydnPYku3bt2P79u3Wv8PDwyEUClFfX3/DiCI+nw+5XI7w\n8HDEx8dDIpFgYGAAFRUVNtd0K1xdXa19r2azGSaTydq1wBUymQx+fn6wWCz44YcfrJ87ODjg6aef\nhkqlwltvvYUzZ85wEt/d3R3R0dGQSCTo7u7GsWPHUFlZia6uLpw7dw5BQUF45JFHkJGRgc7Ozik1\n8bGWtr29PXg8Hi5duoS+vj7OHjYLhUK4ubnB19cXAKzJhFwut+4z9gBzaGiI87oxhlgshp+fn/Xv\nb7/9lruReETE6QaAbmezt7enjIwM2rVrF3V2dpJOp6MTJ07Q0qVLSaFQjOt/3K6Ot99+m/r6+shi\nsZDFYqGLFy/SG2+8QWlpaeTl5UX29va3dQ4T1QGANmzYQDU1NVRUVEQrV64kDw8P8vLyIj8/P1Kp\nVDR//nz65JNPqL6+nl588UWys7PjRMd4NpFIRH5+frR+/XoqLCykkZER6unpoQMHDlBgYCDnOng8\nHj377LPU3t5Oo6OjVFtbS8uWLSORSHTbZXI7ccPCwigvL4+2b99+w+fp6el08eJFev/99ykkJGRC\ndWQ8x2RmZlJFRQVpNBr69NNPyd7enjw9Pekvf/kLtba20sjICFksFmpvb6fs7GzOdPx4EwgEFBwc\nTDt27KChoSEymUz06quvkoeHx4SvmV/aXyKR0KOPPkoNDQ00OjpKZrOZhoeHaWBgwLp98cUXtGbN\nGpo1axbJZDL6/yn0nJUHj8ejiIgI6u3ttXrKb3/7W3JxcbntMh2Xx95tJj62+fn50WuvvUZ1dXVk\nMBjohx9+oA0bNpBQKJzQif/c/m+++SZ1dnaS2Wym0dFRGh0dJb1eT/X19bRv3z5asWIFOTo6klgs\nHlcFmKgOAKRQKOijjz6iwcFBqq6upq1bt9L+/fuptLSU8vPzqbS0lPr7++ntt98mmUzGmY6fq6B8\nPp+kUinFx8fTtm3bqL+/n0ZGRqijo4M+/PBDCgoK4lyHUCgkX19funDhAhmNRtJqtfSvf/2LAgIC\nJlQm443L5/Np3rx5VFJSQrm5udbPpVIp/ec//6GmpiZat24dyeVyznQ8+uijNDQ0RC0tLfTiiy+S\nt7c3bd26lQwGAxmNRmsd3rVrF0VGRnJaHtdvjo6O9Nprr5HRaCSLxUKDg4OUnZ1NUql0wtfML+2f\nnJxMhw4dsp7zrTaLxUI6nY6++eYbeuCBB8aVlE2mPORyOa1Zs4b0er3VxLds2UJxcXG3VRa30jGt\nTBy4lpU/+OCDdOLECTIajXT48GEKDw+f0In/3P5r166l48ePk0ajuWUlqKqqojfffJOys7Np5syZ\nk/oBxnNceno6FRYW3nBTGcsyent76Z133iFPT0/OdQDXTEsoFJJAICCBQEAKhYICAwNpw4YNVFlZ\nadXW0dFB7777Lvn5+XGi4/pNKBTSnDlzaHh42HqRbtmyhXx8fCZcJuM9TqVS0ZYtW6ihoYGysrJu\nqENNTU308ccf06xZszjTIRQK6amnniKdTkdtbW307rvv0tatW0mn01F3dzddvHiR9Ho9mc1m+uCD\nD8atZbK/C4/HIx8fHyoqKrLW25MnT1JoaOht/Z/b1fHCCy/cdJ2MXSs//nxoaIhyc3MpLS3tF5Ox\nyZTHvHnzbmjZj20ajYbi4+MnVR7TzsSBa3f3l19+mXQ6HV2+fJleeeWVCZ34z+0vEAjIx8eHPvnk\nExoYGLipQoyMjJDBYKDOzk7asmXLpH6A8Rzn4OBA77//Pg0MDJDRaCS9Xk96vZ6+++47eumll2jh\nwoUUGxtL3t7enOoAQCkpKbR161bavHkzbdq0iU6ePEkajYaGh4dpZGSERkdHqa2tjV5//XUSiUQ/\ne3HYon6MNdkrKipuiD9//nwSCAQT/m3Gc8xY0/3y5cv073//2xpPIBBQcXExVVRUUFxcHPH5fM50\nODs70+9//3vrDWx4eJgaGxvpiSeeIAcHB3ruueeovb2dBgYGaPHixeNuOU72d5FIJJSZmWn9Tcxm\nM0VGRt5Wy3UiOlasWEE1NTXWa9VgMFBbWxtpNJpbJkEmk4l27dpFMTExnJVHeHg4FRcX32TiRqOR\nEhMTJ1Ue09LEeTwePfXUUzQwMECtra3017/+dUIn/kvH8Pl8cnZ2Jg8PD1q3bh3l5eVRU1OT9WIZ\nqwRarZbOnj1LzzzzDCc6gGvZRX19PXV0dNC2bdtoz549VuPs6+ujzs5Oam9vp9raWtq6dSv5+vra\nXMfChQvprbfeorKyMtJqtaTRaKi/v5+Gh4dvuDj6+/tp69at5OzszFl5XL+5uLjQyy+/TEajkcxm\nM7W1tVF6evqkm6njOSY5OZmOHj1KhYWF1j5voVBIb7zxBmk0Gjp06BBt2bKFsrKySKlUcqKDx+NR\nUlIS7d27l+rr6+nUqVP06KOPWs//wIEDNDQ0RDt27KDg4GBOy+P6zdPTkz799FOrYdXX14+7a2sy\nOu6//346e/YsjY6OUmVlJT3wwAMUHBxM4eHhFB8fTwkJCfTSSy9ReXm59Vquqqqi1atXc1YeIpGI\n0tLSSKPRWMvj4MGD9Oqrr95WC/qnrpkfb3f9AlgqlQpRUVGQSCTQarWcxbFYLNBoNNBoNPjyyy9x\n9OhRzJo1C+Hh4UhOTsaCBQsgl8vh4OCAqKgorF27Ft3d3di/f79NdcycORNpaWlQqVTYt28fKioq\nkJWVBSJCa2urdSw2EUGpVMLV1RUymQyvvPIK2tvbbaJh8eLF+MMf/oDIyEjY29tDp9NZx4pbLJYb\nhg+OTbYYG6fLJW5ubli9ejU2bdoEgUCA9vZ2bNiwAWfOnMHw8PBPHufk5ARfX19otVpcvnx5QrHV\najWWLFmCyMhItLa24vXXX0dQUBB4PB58fHwgFArh5OQEd3d3jIyM3HJUky0gInz//fd48cUXIZVK\nrdPHjUYj5s+fj6ioKIjFYuTk5HA2gupWSKVSREREjBkePvjggylZ06egoACrV6+2Dme8evUqjEYj\neDyedXhfbW0tBAIBnn32WajVavj6+iIiIgLffPMNJ9PuTSYTurq6bhifX1dXh927d3My1PKuNnGV\nSoUnn3wS2dnZsFgsqK6uxtGjRzmPOzAwgIGBAfT09KC0tBR5eXl47LHH8MILL0Aul8POzg5qtRqx\nsbE2N/Hs7GyEh4ejr68P+fn5OHPmDLRaLT7++GPU1taCz+dDqVRCLBbD09MTGzduRFRUFKKioiZt\n4v7+/li7di0WL14Mb29vlJWVQafTYfbs2fD29rZOyiotLYVIJEJMTAyIaEqGbcnlcqSnp+P555+H\nu7s7BgcHsXHjRpw+ffonF79SKBRITk7GkiVLoFAosHPnzgmbuJubGwICAqBQKKzm0N7ebp3XsHHj\nRlRWVmJoaAhdXV2cLshlMBhw5cqVGz4TiUR47rnnoFQqMTQ0hN7e3imb8DS2AJW/vz+Aa0P68vLy\nOBkS/GOGhoZ+cujeWL3UaDQ4cOAAEhIS4ObmBqFQCJ1Ox+mMVoPBgPr6eutyCFKpFIODg5xcK3et\nidvb2yM9PR2rVq2Cl5cXWlpacOTIEVRVVU2ZhuHhYQwPD6Onpwf5+fnW8evAtfGpXEx5DwkJgUKh\nwPnz51FdXY1Lly6hq6sLJpPJelEIhUIIBAKEhoYiKysLgYGBk15cSCgUYv369dby3rVrFwwGA+Li\n4qwVf3BwEHv37sXnn3+OzMxMxMTE2OKUfxEnJyekpKTgiSeegK+vL/R6Pfbv348TJ07c0iyVSiXU\najXS09OxdOlS+Pj4oLy8HN9///2ENXR3d+Po0aOoqalBVVUVuru74eHhgZSUFOzcuROHDx9GT0+P\nNROdSuzs7BAVFYW0tDRIJBLk5OSgs7OTcy0SiQS+vr5ISEjAggULIJVKQUSorKxEd3c35zd3uVwO\nPp+P/v7+X9y3p6cHdXV1SEhIgEQigUaj4fQmZzQa0dTUZDVxFxcXTha/Au5SE5fJZHjggQfwm9/8\nBoGBgRAIBNDr9eDz+ViwYAEcHBygVqvR3d2Njz76iFMtarUaixYtwoIFCxAQEGD9vL6+Hp9//rnN\n44nFYggEAvT390On01m7K67HbDZjdHQUcrkcXl5e6O/vn/Tkmj/96U9YuXIlvLy8IBQKERERAbFY\nDJVKBYFAgG+++Qb5+fk4efIkysvLres229nZwdHREUKhkLPp3ZGRkdi0aRNiYmKg0+lw+PBhvPXW\nW9DpdHBycoKTkxP4fD5SU1Ph6+sLlUoFtVoNlUoFkUiEb7/9Fjk5OZOaFNXe3o5Dhw5BKBSiu7sb\nzs7OePjhhzE0NISdO3eir6/vjhg4n8+Hl5cXNm/eDKVSiYGBAezZs+emWcZcYDabMTAwAB6Ph/j4\neADA4OAg/vGPf3C6xLSDgwOSk5Nx7733ws3NDVqtFoODgxgYGIDJZEJTUxMuX74MtVqNy5cvw9PT\nE7GxsdYu0ZqamptaMrbGaDSivr7e2v0YEREBqVTKSay7zsRnzpyJJUuWYO3atZg7d651MSoPDw+s\nXLkSJpMJYrEYrq6uOH/+PGcmnpaWhsjISISEhCAtLQ1+fn7WKej9/f04c+YMzp8/z0ls4JqZ29nZ\n/eT3AQEBePjhh+Hg4IDc3FxcunRpUvHGMtaxbGHMpOvq6nDkyBHk5uaivLz8prU4ZDIZwsPDkZSU\nxNlbdzw8PBATEwO5XI7BwUGIxWKkpKQgJSUFKpUK7u7uEAgECAsLg1KphEwmQ19fH86dO4eCggJc\nuHABdXV1k9IwtvAYcK3rIjg4GAsWLEBeXh7q6+unbCbgjxGJRAgJCUFmZiYEAgHq6upQXV39s88I\nbIXZbIbRaIRUKoVCoYDFYkF7ezsKCgo4zXIXLlyIdevWITExEU5OTjCbzRgeHoZer4fZbEZnZye6\nu7sxY8YM9PT0wMXFBWq1GkqlEkKhEEVFRbhw4QJn+oBr3TzHjh3DihUrEBwcDHd3d86Wg7jrTFyp\nVCIjIwMJCQk3vAzB1dUVcrkcfX19qKmpwfHjx22+Frm7uztmz56NsLAwZGVlITQ01JrpjdHd3Y2v\nvvoKe/bssWnsMbq6ujA8PAwPD48bpg5fj1AoRGRkJLKysnDlyhXk5eVN+qJpbGyEl5cXrly5Ai8v\nL+v65ZWVlSgqKkJLS8sNRjU2nVulUkEqlXK6VopWq0VbWxucnJwglUqRlJSEiIgIALhheYK6ujqU\nlZWhvr4eFRUVuHjxImpra6HX622qZ8aMGbjvvvvQ3t6ODz744I4ZOHCtmf7ggw9as7xDhw5NaavA\nw8MDqampEAgEGBwcxK5duzhfgTQyMhIxMTHWpZCFQiEkEglcXFwAwDoF/1YMDw+jpqaG8+WKjUYj\nKioqcPr0aQQGBkIqleLJJ5/Ee++9Z7MBCGPcdSbe39+PmpoalJeXQywWo7OzE42NjTCZTBgZGUFn\nZ6d1GUlbZxvz58/HY489huTkZLi6ut70vVarxbFjx/D++++jpKTEprHHOHnyJDIzM+Hv74958+ah\nqakJV69ehdlshpOTE1QqFcLCwpCZmQmtVou9e/faZLGfHTt2oKSkBM3NzfD19UVxcTHOnz//kw/o\nSkpKsG3bNsTFxeHChQucZja1tbX47LPP8NBDD8HDwwNubm7W5rBGo7HeUC5cuICGhgbU1tZy8gIG\n4FrmGxQUhMTEROzduxelpaWcxBkPjo6OSE1NxUMPPWT97Ny5cza/af0cM2bMQFJSEkQiEbRaLaqr\nqzl/KcfQ0NBtX/ujo6NobW3F6dOnUV5ezpGyGzEYDPjqq6+wdOlSKBQKPPvsszh48OD/vom3t7dj\nz549qKurg0wmQ1NTE0pKSjA8PAwisvYHc0FUVJR1RUPg2lAhvV6PkZERtLS04NSpUzh8+DBnBg4A\nRUVFOHnyJNRqNZYvXw6FQoGqqiprdh4dHY25c+fCaDRi27Zt2L17t036H48ePXpbI38aGhqwfft2\nfP3119BqtZyOxmhubsZHH32Euro6BAYGQqVS4bvvvrvh+/Ly8il5RZuLiwvCwsIwODiIvLw8zuP9\nkpbU1FR4enreMQ3XryIokUgQGBjI+QqWpaWlyM/PR3x8vLW1KpFIoFQqrWt3X9/n3dPTg9bWVhQX\nF+PAgQOcL0M7hslkwunTp3HgwAEsXboUDg4OnHSp3HUmPvZ0u7Kycspjt7e3o6WlBTKZDBqNBlVV\nVWhqasLg4CAKCwuRm5vLuQaz2Yy9e/fC3t4e8+fPx+rVqyGRSEBEsFgs6OnpQUNDA77++mvs3LmT\ns/HI46Gvrw99fX1TEqurqws5OTlTEuvn4PP50Gg0OHjw4B1/g5DBYLC+/ICIoNPpOBvG9lNcuXIF\nu3btwuzZs6HT6VBSUsJ5V05BQQFaW1sRGxtrfduWi4sLQkJCIBKJMDw8fMPzqrGWfUdHx5S+bIaI\nMDAwgL///e8wmUxwd3cf10ia24XHdYH//7TbKYWIbkoFxqMjMDAQ99xzDxYtWoSamhrk5ubetPTq\nVOgArjVTx56oh4WFQSaToa2tDceOHcORI0dua/TBZHTYkrtFx620TEcddnZ2SEhIwMGDB8Hj8XD6\n9GmsX79+Qm+P+V8oj1+DjlvBTJzp+NXpuJWW6aiDz+fD398ff/zjHyGRSPDPf/4TNTU1E8rE/xfK\n49eg41YwE2c6fnU6bqWF6WA6poOOW8H/pR0YDAaDcffCeSbOYDAYDO5gmTiDwWBMY5iJMxgMxjSG\nmTiDwWBMY5iJMxgMxjSGmTiDwWBMY5iJMxgMxjSGmTiDwWBMY5iJMxgMxjSGmTiDwWBMY5iJMxgM\nxjSGmTiDwWBMY5iJMxgMxjSGmTiDwWBMY5iJMxgMxjSGmTiDwWBMY5iJMxgMxjSGmTiDwWBMY5iJ\nMxgMxjSGmTiDwWBMY5iJMxgMxjTm/wCnZ4S+tsK86wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdaec9f64d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_mldata\n",
    "import numpy as np \n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "class MNIST:\n",
    "    def __init__(self):\n",
    "        mnist = fetch_mldata('MNIST original')\n",
    "        p = np.random.permutation(mnist.data.shape[0])\n",
    "        self.X = mnist.data[p]\n",
    "        self.Y = mnist.target[p]\n",
    "        self.pos = 0        \n",
    "    def get(self, batch_size):\n",
    "        p = self.pos\n",
    "        self.pos += batch_size\n",
    "        return self.X[p:p+batch_size,:], self.Y[p:p+batch_size]\n",
    "    def reset(self):\n",
    "        self.pos = 0        \n",
    "    def plot(self):\n",
    "        for i in range(10):\n",
    "            plt.subplot(1,10,i+1)\n",
    "            plt.imshow(self.X[i].reshape((28,28)), cmap='Greys_r')\n",
    "            plt.axis('off')\n",
    "        plt.show()\n",
    "        \n",
    "mnist = MNIST()\n",
    "mnist.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first train lenet on a single GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('workload partition: ', [(gpu(0), 1024)])\n",
      "iteration 0, accuracy 0.071289\n",
      "iteration 10, accuracy 0.815430\n",
      "iteration 20, accuracy 0.896484\n",
      "iteration 30, accuracy 0.912109\n",
      "iteration 40, accuracy 0.932617\n",
      "time for train lenent on cpu 2.708110 sec\n"
     ]
    }
   ],
   "source": [
    "# @@@ AUTOTEST_OUTPUT_IGNORED_CELL\n",
    "import time\n",
    "batch_size = 1024\n",
    "shape = [batch_size, 1, 28, 28]\n",
    "mnist.reset()\n",
    "tic = time.time()\n",
    "acc = train(lenet(), shape, lambda:mnist.get(batch_size), [mx.gpu(),], [1,])\n",
    "assert acc > 0.8, \"Low training accuracy.\"\n",
    "print('time for train lenent on cpu %f sec' % (time.time() - tic))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we try multiple GPUs. The following codes needs 4 GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('workload partition: ', [(gpu(0), 512), (gpu(1), 512)])\n",
      "iteration 0, accuracy 0.104492\n",
      "iteration 10, accuracy 0.741211\n",
      "iteration 20, accuracy 0.876953\n",
      "iteration 30, accuracy 0.914062\n",
      "iteration 40, accuracy 0.924805\n",
      "time for train lenent on 2 GPU 1.623732 sec\n",
      "('workload partition: ', [(gpu(0), 256), (gpu(1), 256), (gpu(2), 256), (gpu(3), 256)])\n",
      "iteration 0, accuracy 0.092773\n",
      "iteration 10, accuracy 0.777344\n",
      "iteration 20, accuracy 0.887695\n",
      "iteration 30, accuracy 0.908203\n",
      "iteration 40, accuracy 0.916992\n",
      "time for train lenent on 4 GPU 1.086430 sec\n"
     ]
    }
   ],
   "source": [
    "# @@@ AUTOTEST_OUTPUT_IGNORED_CELL\n",
    "for ndev in (2, 4):\n",
    "    mnist.reset()\n",
    "    tic = time.time()\n",
    "    acc = train(lenet(), shape, lambda:mnist.get(batch_size), \n",
    "          [mx.gpu(i) for i in range(ndev)], [1]*ndev)\n",
    "    assert acc > 0.9, \"Low training accuracy.\"\n",
    "    print('time for train lenent on %d GPU %f sec' % (\n",
    "            ndev, time.time() - tic))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "As can be seen, using more GPUs accelerates the speed. The speedup is not perfect because the network is still simple, we cannot fully hide the communication cost over multiple GPUs by pipelining the computation and communication. We observed better results by using the state-of-the-art networks. The following figure shows the speedup of three imagenet winners by using 8 Nvidia Tesla M40\n",
    "\n",
    "![](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/notebooks/m40.png)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}