{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook is focused on the basics of using [MXNet](http://mxnet.io/) in [Julia](http://julialang.org/) to create a simple Multilayer Perceptron (MLP). This basic neural network building block is described more fully in [Wikipedia](https://en.wikipedia.org/wiki/Multilayer_perceptron). \n", "\n", "We will be making predictions on the famous MNIST data set, which is a labeled set of individual handwritten digits from zero to nine. A brief description is available in [Wikipedia](https://en.wikipedia.org/wiki/MNIST_database). The data is available at http://yann.lecun.com/exdb/mnist/, where there are also descriptions of the effectiveness of the different approaches applied to this data and references to related papers. Instead of this source, however, I will be using the data from [Kaggle](http://kaggle.com/) as I have learned a lot from the competitions there and like to use it as an example.\n", "\n", "I also intend to use this as brief tutorial on looking at data using the [Julia](http://julialang.org/) language. It assumes some basic knowledge of Julia, although someone who knows Python or R could probably figure it all out." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Get data\n", "\n", "You can get the data from https://www.kaggle.com/c/digit-recognizer/data. The Kaggle website describes the files as follows:\n", "\n", ">The data files `train.csv` and `test.csv` contain gray-scale images of hand-drawn digits, from zero through nine.\n", "\n", ">Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n", "\n", ">The training data set, `train.csv`, has 785 columns. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.\n", "\n", ">Each pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).\n", "\n", ">For example, pixel31 indicates the pixel that is in the fourth column from the left, and the second row from the top, as in the ascii-diagram below.\n", "\n", ">Visually, if we omit the \"pixel\" prefix, the pixels make up the image like this:\n", "```\n", "000 001 002 003 ... 026 027\n", "028 029 030 031 ... 054 055\n", "056 057 058 059 ... 082 083\n", " | | | | ... | |\n", "728 729 730 731 ... 754 755\n", "756 757 758 759 ... 782 783 \n", "```\n", "The test data set, `test.csv`, is the same as the training set, except that it does not contain the \"label\" column.\n", "\n", ">Your submission file should be in the following format: For each of the 28000 images in the test set, output a single line with the digit you predict. For example, if you predict that the first image is of a 3, the second image is of a 7, and the third image is of a 8, then your submission file would look like:\n", "\n", ">3\n", "\n", ">7\n", "\n", ">8\n", "\n", ">(27997 more lines)\n", "\n", "\n", "Of course, we can learn a lot of this by just looking at the data. The code below assumes you downloaded the csv files and put them in a folder named `data`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "using DataFrames, CSV" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A few comments for those new to Julia. Most people start working with the REPL where the contents of any line you type is immediately evaluated and returned. Working with Jupyter notebooks using iJulia allows the possibility of depicting such results graphically, and in the case of `DataFrames`, they are rendered in a nice tabular format. In this case, by following the line with a semicolon, I will suppress the normal output as I don't want to potentially clutter the notebook with a table that has 42000 rows in it. I encourage you to download this notebook and experiment with it, removing the semicolon and seeing what the raw data looks like. Of course, you can use functions like `head()` and `tail()` as demonstrated in the [Stats](#Stats) section.\n", "\n", "I am also using the `@time` macro here to as I like to track how long different processes take and memory involved. You can delete the leading `@time` from any command line if you just want to execute the operation." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 6.867796 seconds (15.87 M allocations: 751.361 MiB, 6.01% gc time)\n" ] } ], "source": [ "@time train = CSV.read(\"data/train.csv\");" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

6 rows × 785 columns (omitted printing of 774 columns)

labelpixel0pixel1pixel2pixel3pixel4pixel5pixel6pixel7pixel8pixel9
Int64Int64Int64Int64Int64Int64Int64Int64Int64Int64Int64
110000000000
200000000000
310000000000
440000000000
500000000000
600000000000
" ], "text/latex": [ "\\begin{tabular}{r|cccccccccccc}\n", "\t& label & pixel0 & pixel1 & pixel2 & pixel3 & pixel4 & pixel5 & pixel6 & pixel7 & pixel8 & pixel9 & \\\\\n", "\t\\hline\n", "\t& Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & \\\\\n", "\t\\hline\n", "\t1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & $\\dots$ \\\\\n", "\t2 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & $\\dots$ \\\\\n", "\t3 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & $\\dots$ \\\\\n", "\t4 & 4 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & $\\dots$ \\\\\n", "\t5 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & $\\dots$ \\\\\n", "\t6 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & $\\dots$ \\\\\n", "\\end{tabular}\n" ], "text/plain": [ "6×785 DataFrame. Omitted printing of 777 columns\n", "│ Row │ label │ pixel0 │ pixel1 │ pixel2 │ pixel3 │ pixel4 │ pixel5 │ pixel6 │\n", "│ │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mInt64\u001b[39m │\n", "├─────┼───────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤\n", "│ 1 │ 1 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │\n", "│ 2 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │\n", "│ 3 │ 1 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │\n", "│ 4 │ 4 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │\n", "│ 5 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │\n", "│ 6 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "first(train, 6)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "(42000, 785)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# to get the size/shape of the DataFrame\n", "size(train)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "28×28 Array{Int64,2}:\n", " 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 15 94 89 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 89 220 253 251 214 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 240 253 253 253 218 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 … 253 253 253 250 95 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 195 80 94 131 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0\n", " ⋮ ⋮ ⋱ ⋮ ⋮ \n", " 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 80 207 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 123 247 253 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 191 248 253 235 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 188 250 253 208 77 … 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 255 253 167 13 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 94 93 10 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# first element is label, rest is 28x28=784 pixel values, so to look at first row\n", "reshape([train[1,col] for col in 2:785], 28, 28)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are several plotting libraries available for Julia. [Plots.jl](https://juliaplots.github.io/) is one of my favorites as it is developing an excellent ecosystem that allows multiple backends including [PyPlot](https://github.com/JuliaPy/PyPlot.jl) which is a wrapper around [Matplotlib](http://matplotlib.org/) and [Plotly](https://github.com/sglyon/PlotlyJS.jl) which is an interface to the [plotly.js](https://plot.ly/javascript) visualization library. Use StatPlots if you want to plot a DataFrame directly." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "┌ Info: Recompiling stale cache file /home/milton/.julia/compiled/v1.1/PyPlot/oatAj.ji for PyPlot [d330b81b-6aea-500a-939a-2ce795aea3ee]\n", "└ @ Base loading.jl:1184\n" ] }, { "data": { "text/plain": [ "Plots.PyPlotBackend()" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "using Plots\n", "# I like Plotly for interactivity. \n", "plotly(legend=false)\n", "# to use it on GitHub, you have to setup Plotly Online and inject their code into the notebook\n", "# so just switch to PyPlot\n", "pyplot()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 12.920942 seconds (27.09 M allocations: 1.324 GiB, 7.05% gc time)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFPxJREFUeJzt3X+M1/Wd4PHXF0ZADhvuBmXPwPjdiUxJC9uhcxrOQ5DEZimFDVk0JouENiXTpvRY7zzdXvZue79ktcfBSTuu9rYxFbNTrJiz3a4RGpNSco21/Vos0drpDshMt/wod2RdBGSYz/3hOlEZTL/fvuDzneHx+Au+8Mr7la9fw5P3l5lvpSiKIgAASDOh7AUAAMYbgQUAkExgjVNvvPFG1Gq1eOONN8peBQAuOy1lL8DF8bOf/Sy6uroiYmJEVMpeB4CLpCjOlr0Co3CDBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1hN6PTp07Fq1aro6OiIzs7OWLZsWRw8eDAiIm655ZZob2+Pzs7O6OzsjK1bt5a7LABwnpayF2B03d3d8fGPfzwqlUp85Stfie7u7ti1a1dERGzbti1WrFhR8oYAwIW4wWpCU6ZMieXLl0elUomIiIULF0Z/f3/JWwEAvymBNQZs27YtVq5cOfLze+65J+bPnx933HGH8AKAJiSwmtymTZuir68v7rvvvoiI2L59e7zyyivx0ksvxc033+ytQgBoQpWiKIqyl2B0mzdvjm984xvx3e9+N6ZPnz7q75kyZUr88pe/jNbW1nc9XqvVoqurKyImRkTl4i8LQCmK4mzZKzAKN1hNasuWLdHb2xu7d+8eiauhoaE4cuTIyO/ZuXNnzJw587y4AgDK5asIm9Dg4GDcfffd0d7eHkuXLo2IiMmTJ8dzzz0Xn/jEJ+LMmTMxYcKEmDFjRnzrW98qeVsA4L0EVhOaNWtWXOid2x/96EeXeBsAoF7eIgQASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgWUvZC0C9/vx3u+ueGY5KQ2f9p4En6545O3S8obMAGD/cYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJGspewGo1z39N9U9M3zuTENndX/2ubpn2h6b3NBZp978u4bmAGg+brAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIVimKoih7CfLVarXo6uqKiIkRUSl7nVSn/3xq3TPD/+a/NnRWS8u0umcm/O9/19BZf7xhbd0zPUceaugsYPwoirNlr8Ao3GABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACSrFEVRlL0E+Wq1WnR1dUXExIiolL1O6c5sntzQ3NnP/VndM5MmzWjorDcHn6175oHF1zR01v88urvumddP/21DZwEXV1GcLXsFRuEGCwAgmcBqQqdPn45Vq1ZFR0dHdHZ2xrJly+LgwYMREXH06NFYtmxZzJkzJ+bNmxd79+4td1kA4DwCq0l1d3fHq6++Gj/5yU9ixYoV0d3dHRERX/jCF2LhwoXR19cXjz76aKxZsyaGhoZK3hYAeCeB1YSmTJkSy5cvj0rlrX87tXDhwujv74+IiCeeeCI2bNgQERE33HBDzJw50y0WADQZgTUGbNu2LVauXBnHjx+P4eHhuPrqq0d+rVqtxqFDh0rcDgB4r5ayF+D9bdq0Kfr6+uLhhx+OU6dOjdxqvc0XgQJA83GD1cQ2b94cTz31VDzzzDMxderUaG1tjYiIY8eOjfye1157Ldra2spaEQAYhcBqUlu2bIne3t7YvXt3TJ8+feTx22+/PXp6eiIi4oUXXojDhw/HokWLyloTABiFtwib0ODgYNx9993R3t4eS5cujYiIyZMnx/PPPx8PPPBArF27NubMmROTJk2K7du3R0uL/4wA0Ez8ydyEZs2adcF/WzVz5szYtWvXJd4IAKiHtwgBAJIJLACAZD7seZzyYc85/rrr9rpnluyZ19BZkyf/TkNzjXj9X99f90zrXxy4CJsAvy0f9tyc3GABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACRrKXsBaGYrfvzNumfuuv7qhs7a/OB/r3tm+A//R0NnTfqP/6rumYH/94GGztq6p/6ztvzdQw2dBdAs3GABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQrFIURVH2EuSr1WrR1dUVERMjolL2OvwGZk1bXPfMwcf2NXTW8B98qaG5Rpw+/kLdMzNn/01DZ7157vWG5s41OAfNoCjOlr0Co3CDBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQrKXsBYC3DP7DnrpnPrG2u6Gz/vrvJzY014grZyyse+bvT9U/ExERf/nZhsamfO6qumfOnXu9obOAy4MbLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZC1lLwA07tmTX21o7kvXV+qe+ZOtTzR01vCK+xuaa0Rl/cMNzZ0a+mzdM3f9l881dNZDRx5qaA4YW9xgAQAkE1hNaOPGjVGtVqNSqcT+/ftHHq9WqzF37tzo7OyMzs7O2LFjR4lbAgAX4i3CJnTbbbfFvffeG4sWLTrv15588smYN29eCVsBAL8pgdWEFi9eXPYKAMBvwVuEY8yaNWti/vz5sX79+jh27FjZ6wAAoxBYY8iePXti3759UavVorW1NdatW1f2SgDAKLxFOIa0tbVFRMQVV1wRd911V3R0dJS8EQAwGjdYY8TJkyfjxIkTIz/v7e2NBQsWlLgRAHAhbrCa0IYNG+Lpp5+Ow4cPx6233hrTpk2LXbt2xerVq+PcuXNRFEW0t7fHY489VvaqAMAoBFYT6unpiZ6envMef/HFF0vYBgCol7cIAQCSCSwAgGSVoiiKspcgX61Wi66uroiYGBH1f7AvvNeMqZ0NzR1+/EDdM8N/8KWGzqpUJjY0VxTn6p55881fN3TWgx8+VPfMnx54pKGzuDwUxdmyV2AUbrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJJViqIoyl6CfLVaLbq6uiJiYkRUyl6Hy9i0yb9b98xVV/xOQ2cN/OW+huaGb9va0FwjTp88WPfM5P+1paGzZn/x9+qeOXryhYbOojxFcbbsFRiFGywAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACS+bDnccqHPXM5mn7lhxqa61v7Zt0z/3TRaw2dNfxHX2loriF/saHukXs3dTd01MwpQ3XP/Mt//quGzrrl/3y7obnxyoc9Nyc3WAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAySpFURRlL0G+Wq0WXV1dETExIiplrwPjzvQrP9TQ3OE/O1j3zIR7tjZ0VrMbGvqHhua+/KFX6575k/6vNnTWWFAUZ8tegVG4wQIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZD3sep3zYMzSnfzb19+qe+W+z/0VDZ925p/6/Q09pvaGhsy6pvQ/UPTLplv6LsEhz8GHPzckNFgBAMoEFAJBMYDWhjRs3RrVajUqlEvv37x95vK+vL2666abo6OiIG2+8MV5++eUStwQALkRgNaHbbrst9u7dG9ddd927Hv/MZz4T3d3d8fOf/zzuvffe+PSnP13ShgDA+xFYTWjx4sUxa9asdz129OjRqNVqceedd0ZExOrVq+PAgQNx8ODBEjYEAN6PwBojBgYG4tprr42WlpaIiKhUKtHW1haHDh0qeTMA4L0E1hhSqbz72y34DhsA0JwE1hgxe/bsGBwcjKGhoYh4K64GBgaira2t5M0AgPcSWGPENddcEwsWLIjHH388IiJ27twZ1Wo1qtVquYsBAOcRWE1ow4YNMWvWrBgcHIxbb701rr/++oiIeOSRR+KRRx6Jjo6OuP/+++NrX/tayZsCAKNpKXsBztfT0xM9PT3nPf7BD34wfvCDH5SwEQBQDzdYAADJBBYAQDJvEQJcQv/3jZfqnvncq/XPREQcueGzdc/86V9taeisWPhv6x4ZHj7d0FETT59qaA4uJTdYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJKsURVGUvQT5arVadHV1RcTEiKiUvQ4wRvyTydc1NPcfrv39umdOn2vs7/j/+dDDDc2NV0VxtuwVGIUbLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZC1lLwBA8zh55rWG5v79ga8mbwJjmxssAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJrDGoWq3G3Llzo7OzMzo7O2PHjh1lrwQAvENL2QvQmCeffDLmzZtX9hoAwCjcYAEAJBNYY9SaNWti/vz5sX79+jh27FjZ6wAA7yCwxqA9e/bEvn37olarRWtra6xbt67slQCAd6gURVGUvQSN+9WvfhUdHR3x+uuvv+vxWq0WXV1dETExIiql7AbAxVcUZ8tegVG4wRpjTp48GSdOnBj5eW9vbyxYsKDEjQCA9/JVhGPMkSNHYvXq1XHu3LkoiiLa29vjscceK3stAOAdBNYY097eHi+++GLZawAA78NbhAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJCspewFuDhOnTr1jz8qSt0DgIurVqvF3LlzY+rUqWWvwjsIrHHq4MGD//ij4TLXAOAi6+rqih//+Mfx0Y9+tOxVeIdKURSuOMahX//61/Hss89GtVqNK6+8sux1ALiI3GA1H4EFAJDMP3IHAEgmsAAAkgksxoVqtRpz586Nzs7O6OzsjB07dpS90iW1cePGqFarUalUYv/+/SOP9/X1xU033RQdHR1x4403xssvv1zilpfGhZ6Ly/U1cvr06Vi1alV0dHREZ2dnLFu2bOSLYI4ePRrLli2LOXPmxLx582Lv3r3lLnsJvN/zccstt0R7e/vIa2Tr1q3lLsvYVsA4cN111xU//elPy16jNN/73veKgYGB856HpUuXFo8++mhRFEXxzW9+s1i4cGFJG146F3ouLtfXyKlTp4rvfOc7xfDwcFEURfHlL3+5+NjHPlYURVF86lOfKr74xS8WRVEUP/zhD4u2trbi7NmzZa16Sbzf87FkyZLi29/+dpnrMY64wYJxYPHixTFr1qx3PXb06NGo1Wpx5513RkTE6tWr48CBA+/4Fh7j02jPxeVsypQpsXz58qhUKhERsXDhwujv74+IiCeeeCI2bNgQERE33HBDzJw5c9zfYr3f8wGZBBbjxpo1a2L+/Pmxfv36OHbsWNnrlG5gYCCuvfbaaGl569vdVSqVaGtri0OHDpW8WXm8RiK2bdsWK1eujOPHj8fw8HBcffXVI79WrVYvu9fH28/H2+65556YP39+3HHHHcKL34rAYlzYs2dP7Nu3L2q1WrS2tsa6devKXqkpvP239LcVl/F3ZfEaidi0aVP09fXFfffdFxFeH+99PrZv3x6vvPJKvPTSS3HzzTfHihUrSt6QsUxgMS60tbVFRMQVV1wRd911V3z/+98veaPyzZ49OwYHB2NoaCgi3vrDc2BgYOS5utxc7q+RzZs3x1NPPRXPPPNMTJ06NVpbWyMi3nWT99prr102r4/3Ph8Rb/0/E/FWeH7+85+P/v7+OH78eJlrMoYJLMa8kydPxokTJ0Z+3tvbGwsWLChxo+ZwzTXXxIIFC+Lxxx+PiIidO3dGtVqNarVa7mIluNxfI1u2bIne3t7YvXt3TJ8+feTx22+/PXp6eiIi4oUXXojDhw/HokWLylrzkhnt+RgaGoojR46M/J6dO3fGzJkzR0IU6uU7uTPm9ff3x+rVq+PcuXNRFEW0t7fHgw8+eFmFxIYNG+Lpp5+Ow4cPx4wZM2LatGnxi1/8Il599dX45Cc/GcePH48PfOAD8fWvfz0+/OEPl73uRTXac7Fr167L9jUyODgYs2fPjvb29rjqqqsiImLy5Mnx/PPPx5EjR2Lt2rVx4MCBmDRpUjz00EOxZMmSkje+uC70fDz33HOxZMmSOHPmTEyYMCFmzJgRW7ZsiY985CMlb8xYJbAAAJJ5ixAAIJnAAgBIJrAAAJL9f8nsY6ICWfsMAAAAAElFTkSuQmCC" }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# heatmap() plots a 2D array\n", "@time heatmap(reshape([train[1,col] for col in 2:785], 28, 28), aspect_ratio=:equal)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.071421 seconds (110.27 k allocations: 5.517 MiB)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFEJJREFUeJzt3X+M1PeZ2PFn8PLzcERvsbexYD23Cht0gWbI1hbnYjCSoyMETkjYsnTYIrmgTXQbIXSuiatr66pX06Sl0JDgi9OLrIDVDY6x6uRybiByFcI1cpxsgoPsOGsBhs0dP8KJqsWA2d1v//B5he0FZSYPfGeX1+sv78Cjz6PxWH7zmWWnUhRFEQAApJlQ9gIAAOONwAIASCawxqk33ngj+vr64o033ih7FQC47rSUvQBXxy9+8Yvo6uqKiBsiolL2OgBcJUVxsewVGIUbLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcBqQufPn49Vq1ZFZ2dn1Gq1WLZsWRw5ciQiIu66667o6OiIWq0WtVottm7dWu6yAMB7tJS9AKPr7u6Oj33sY1GpVOLLX/5ydHd3x549eyIiYtu2bbFixYqSNwQALscNVhOaMmVKLF++PCqVSkRELFy4MA4dOlTyVgDAb0pgjQHbtm2LlStXjnz90EMPxfz58+O+++4TXgDQhARWk9u0aVP09/fHo48+GhERO3fujFdeeSVeeumluPPOO71VCABNqFIURVH2Eoxu8+bN8Y1vfCO+973vxYwZM0b9PVOmTIlf/epX0dra+o7H+/r6oqurKyJuiIjK1V8WgFIUxcWyV2AUbrCa1JYtW6K3tzf27t07EleDg4Nx4sSJkd+ze/fuaGtre09cAQDl8rcIm9DAwEA8+OCD0dHREUuXLo2IiMmTJ8fzzz8fH//4x+PChQsxYcKEmDlzZnzrW98qeVsA4N0EVhOaNWtWXO6d2x//+MfXeBsAoF7eIgQASCawAACSCSwAgGS+BwuAEef/47SG5loefqzumf/c8b8bOutfHf5qQ3NwLbnBAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJkPewYYpy5snlz3zNBn/6Khs4aHLtQ/E5WGzoKxwA0WAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAECylrIXAODK/rrr3obmLv5pre6ZSS3TGzrr/3xmc90z/+7YPzR0FowFbrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBI5sOeAa6hDe//07pnluxra+isSZNm1j0z4X/8y4bOat9xY90zFwdPN3QWjAVusAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkrWUvQDAWDRr+uKG5jZ/cUfdM8OT/0tDZ7058N26Zz7X80BDZ51787GG5mC8coMFAJBMYDWh8+fPx6pVq6KzszNqtVosW7Ysjhw5EhERJ0+ejGXLlsWcOXNi3rx5sX///nKXBQDeQ2A1qe7u7nj11VfjZz/7WaxYsSK6u7sjIuLhhx+OhQsXRn9/fzzxxBOxZs2aGBwcLHlbAOBSAqsJTZkyJZYvXx6VSiUiIhYuXBiHDh2KiIinnnoqenp6IiLitttui7a2NrdYANBkBNYYsG3btli5cmWcPn06hoeH46abbhr5tWq1GkePHi1xOwDg3fwtwia3adOm6O/vj6985Stx7ty5kVuttxVFUdJmAMDluMFqYps3b45nnnkmnnvuuZg2bVq0trZGRMSpU6dGfs/rr78e7e3tZa0IAIxCYDWpLVu2RG9vb+zduzdmzJgx8vi9994b27dvj4iIF198MY4fPx6LFi0qa00AYBTeImxCAwMD8eCDD0ZHR0csXbo0IiImT54cL7zwQnzhC1+IBx54IObMmROTJk2KnTt3RkuLf40A0Ez8n7kJzZo167LfW9XW1hZ79uy5xhsBAPXwFiEAQDKBBQCQzFuEwHXvD3+nu+6Z7+zY1dBZw3/U2Ac3N+ILi2+ue2b7CR/aDBncYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJGspewGALI/+3qcbmtv42m11zwwXXQ2ddeHkD+qeefMv/rahs/7rSX+GhrL4rw8AIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkPuwZaDozp9Uamvvc1qcamiui/g97btSZP/t53TOzv3H4KmwCXE1usAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkrWUvQAwvk2f/Ht1zxx/8nBDZw2v+E8NzVUamDl/+sWGztq67180MHWgobOA8rjBAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBI1lL2AsD4duPEf1r3zPAfPXwVNsnVNvtvGpo79+bfJW8CNCM3WAAAyQRWE1q/fn1Uq9WoVCpx8ODBkcer1WrMnTs3arVa1Gq12LVrV4lbAgCX4y3CJnTPPffExo0bY9GiRe/5taeffjrmzZtXwlYAwG9KYDWhxYsXl70CAPBb8BbhGLNmzZqYP39+rFu3Lk6dOlX2OgDAKATWGLJv3744cOBA9PX1RWtra6xdu7bslQCAUXiLcAxpb2+PiIiJEyfGhg0borOzs+SNAIDRuMEaI86ePRtnzpwZ+bq3tzcWLFhQ4kYAwOW4wWpCPT098eyzz8bx48fj7rvvjunTp8eePXti9erVMTQ0FEVRREdHR+zYsaPsVQGAUVSKoijKXoJ8fX190dXVFRE3RESl7HW4jr1/+h/UPfP6mU/kL3IFlcoNdc+8b+q/begsP8mdbEVxsewVGIW3CAEAkgksAIBkvgcL+I3MmPr7Dc0d+6sDdc8UDbxl91v5q8/UPfLm0NSrsAgwXrjBAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBI1lL2AsDY0P/Amw3NDd+ztf6hYqihs+LxnobGpq6fWvfM0ND/begs4PrgBgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkPuwZrkMzpv5+3TP/ZFF/Q2cNNzDz5pu/buisjf++u6G5oaHHGpoDuBw3WAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyVrKXgBo3O9O+2cNzf3dv3mt7pnhP/5yQ2edP3uk7pkv1f6hobMeO/F4Q3MA2dxgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkMyHPcMY9h9m//OG5iY81JO8yeVN/m9b6p7588MXrsImANeOGywAgGQCCwAgmcBqQuvXr49qtRqVSiUOHjw48nh/f3/ccccd0dnZGbfffnu8/PLLJW4JAFyOwGpC99xzT+zfvz9uvfXWdzz+6U9/Orq7u+OXv/xlbNy4MT71qU+VtCEAcCUCqwktXrw4Zs2a9Y7HTp48GX19fXH//fdHRMTq1avj8OHDceTIkRI2BACuRGCNEceOHYtbbrklWlre+ouflUol2tvb4+jRoyVvBgC8m8AaQyqVyju+LoqipE0AgCsRWGPE7NmzY2BgIAYHByPirbg6duxYtLe3l7wZAPBuAmuMuPnmm2PBggXx5JNPRkTE7t27o1qtRrVaLXcxAOA9BFYT6unpiVmzZsXAwEDcfffd8YEPfCAiIh5//PF4/PHHo7OzMz7/+c/H1772tZI3BQBG46NymtD27dtj+/bt73n8gx/8YPzwhz8sYSMAoB5usAAAkgksAIBk3iKEJvFI+2fqnrl/3+BV2OQy/rKnobHZjyxoYOrFhs4CaBZusAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEjmw54h2e9MvrWhuT//7/+r/qHWP2vorEZs3NTd0NzJs48lbwLQ/NxgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkayl7ARhv/vUtf9jY4MKu3EWStU0ZLHsFgDHDDRYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJfNgzJDs/1NifW4aHz9c9M2HClIbOGhz8f3XP/MH7/76hs+JwY2MAY5kbLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZJWiKIqylyBfX19fdHV1RcQNEVEpex1+Axf/5/vrnimmTG3orK1/cnfdM5879NWGzgKurqK4WPYKjMINFgBAMoE1BlWr1Zg7d27UarWo1Wqxa9euslcCAC7RUvYCNObpp5+OefPmlb0GADAKN1gAAMkE1hi1Zs2amD9/fqxbty5OnTpV9joAwCUE1hi0b9++OHDgQPT19UVra2usXbu27JUAgEv4HqwxqL29PSIiJk6cGBs2bIjOzs6SNwIALuUGa4w5e/ZsnDlzZuTr3t7eWLBgQYkbAQDv5gZrjDlx4kSsXr06hoaGoiiK6OjoiB07dpS9FgBwCYE1xnR0dMRPf/rTstcAAK7AW4QAAMkEFgBAMoEFAJDM92BBk5i47O+v4WlfvYZnAVx/3GABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAECylrIX4Oo4d+7cP/5TUeoeAFxdfX19MXfu3Jg2bVrZq3AJgTVOHTly5B//abjMNQC4yrq6uuInP/lJfOQjHyl7FS5RKYrCFcc49Otf/zq++93vRrVajalTp5a9DgBXkRus5iOwAACS+SZ3AIBkAgsAIJnAYlyoVqsxd+7cqNVqUavVYteuXWWvdE2tX78+qtVqVCqVOHjw4Mjj/f39cccdd0RnZ2fcfvvt8fLLL5e45bVxuefien2NnD9/PlatWhWdnZ1Rq9Vi2bJlI38J5uTJk7Fs2bKYM2dOzJs3L/bv31/ustfAlZ6Pu+66Kzo6OkZeI1u3bi13Wca2AsaBW2+9tfj5z39e9hql+f73v18cO3bsPc/D0qVLiyeeeKIoiqL45je/WSxcuLCkDa+dyz0X1+tr5Ny5c8V3vvOdYnh4uCiKovjSl75UfPSjHy2Koig++clPFo888khRFEXxox/9qGhvby8uXrxY1qrXxJWejyVLlhTf/va3y1yPccQNFowDixcvjlmzZr3jsZMnT0ZfX1/cf//9ERGxevXqOHz48CU/wmN8Gu25uJ5NmTIlli9fHpVKJSIiFi5cGIcOHYqIiKeeeip6enoiIuK2226Ltra2cX+LdaXnAzIJLMaNNWvWxPz582PdunVx6tSpstcp3bFjx+KWW26Jlpa3ftxdpVKJ9vb2OHr0aMmblcdrJGLbtm2xcuXKOH36dAwPD8dNN9008mvVavW6e328/Xy87aGHHor58+fHfffdJ7z4rQgsxoV9+/bFgQMHoq+vL1pbW2Pt2rVlr9QU3v5T+tuK6/insniNRGzatCn6+/vj0UcfjQivj3c/Hzt37oxXXnklXnrppbjzzjtjxYoVJW/IWCawGBfa29sjImLixImxYcOG+MEPflDyRuWbPXt2DAwMxODgYES89T/PY8eOjTxX15vr/TWyefPmeOaZZ+K5556LadOmRWtra0TEO27yXn/99evm9fHu5yPirf9mIt4Kz89+9rNx6NChOH36dJlrMoYJLMa8s2fPxpkzZ0a+7u3tjQULFpS4UXO4+eabY8GCBfHkk09GRMTu3bujWq1GtVotd7ESXO+vkS1btkRvb2/s3bs3ZsyYMfL4vffeG9u3b4+IiBdffDGOHz8eixYtKmvNa2a052NwcDBOnDgx8nt2794dbW1tIyEK9fKT3BnzDh06FKtXr46hoaEoiiI6Ojrii1/84nUVEj09PfHss8/G8ePHY+bMmTF9+vR47bXX4tVXX41PfOITcfr06Xjf+94XX//61+NDH/pQ2eteVaM9F3v27LluXyMDAwMxe/bs6OjoiBtvvDEiIiZPnhwvvPBCnDhxIh544IE4fPhwTJo0KR577LFYsmRJyRtfXZd7Pp5//vlYsmRJXLhwISZMmBAzZ86MLVu2xIc//OGSN2asElgAAMm8RQgAkExgAQAkE1gAAMn+P8Rf9OpKjHrnAAAAAElFTkSuQmCC" }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# to see it upright, use the rotl90() function to rotate left 90 degrees\n", "# there are also rotr90() and rot180() functions\n", "@time heatmap(rotl90(reshape([train[1,col] for col in 2:785], 28, 28)), aspect_ratio=:equal)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XlclNX+B/DPsIOkFCpmihMqWoEOkkrmWtrVUvNmpkUupeFNy6yuZeW91r25dDPtYhi2WeqNTGmxbNEWM/u1aLjmEqkIaIhaVqwCc35/jJwzxsCwzPM8s3zer1evvjzzLEf7NB7PeZ7nmIQQAkRERETkMn5GN4CIiIjI27CDRURERORiHtXBKikpQVZWFkpKSoxuCnkR5oq0wFyRFpgrzxFgdAMa4sCBA0hMTATgD8BkdHM8jhAVRjfBLTFXTcNcOcZcNQ1z5Rhz1TR65sqjRrCIiIiIPAE7WEREREQuxg4WERERkYuxg0VERETkYuxgEREREbmYbh2ssrIyjBo1CrGxsbBYLBg6dChycnIAAAMHDkRMTAwsFgssFguWLFmiV7PIwzFXpAXmirTAXPkWXV/TkJKSgmHDhsFkMuG5555DSkoKNm7cCABITU3F8OHD9WwOeQnmirTAXJEWmCvfodsIVkhICK6//nqYTLb3diQlJeHw4cN6XZ68FHNFWmCuSAvMlW8x7B6s1NRUjBgxQv48a9YsxMfHY+zYsQwcNRpzRVpgrkgLzJV3M6SDNX/+fGRnZ2PevHkAgFWrVmH//v3YvXs3+vXrxyFSahTmirTAXJEWmCvvZxJCCD0vuGjRIrzxxhv45JNPEBER4XCfkJAQHDt2DJGRkedtz8rK4hIBTeDNS08wV8ZhrpgrLTBXzJUW9MyVrje5L168GBkZGeeFqrKyEqdPn0ZUVBQAIDMzE1FRUTVC5SvKFoTJOmD2Mlk/HfN/AIBHjryge5vcnSfmqnlIrKwjAtoCANYmBjvc13Lj57J+YslkWZ8otX25vnLqeblNQNe/L3k1T8yVI36mQFlvv6a/rKusagKj9+bNsrZ6ccfGHXhLrsg53TpY+fn5ePDBBxETE4NBgwYBAIKDg/HZZ5/hhhtuQHl5Ofz8/NCyZUusX79er2aRh2OuSAvMFWmBufItunWw2rVrh9pmI7dv365XM8jLMFekBeaKtMBc+RZdpwjJsfJFamqo6p5/y9paVa5qzrV7pNCgtrK+P+pGWc/J7iTrwMAWTs4yRlaPW56UdVnP8QCAh5N7y21/+ewCWR8p2tTg9pL38fMLl3X8xlsd7xP0naytVb9p3iYy1tmtXWRd/v4vso74T4msq6zFmlw7wP9CWU9tqfKYdmKZo909GpfKISIiInIxjmAZ5P1ENSpRMc0i66AA9bfN3/62SNaP56m/ZZB7Cw++VNY/318k65B5iU0+t2nAHFmHnvv3pW9Pkdv2/vSmrG/qkSLrrZUfy7q4/GiT20FEnss8tJWsD564TdZhz66U9R9lhzS5dqvQrrL+71Pqoa20SZpczlAcwSIiIiJyMXawiIiIiFxMtynCsrIyjBs3Dvv27UNYWBjatGmD9PR0mM1mFBYWYsKECTh06BCCg4ORnp6Ovn376tU03cy8eJqsB2yJknVQUEtZ+73zd1lHr1Q3LFdUnta4dZ7JHXM19oJhsg6Z10Pz69kL6nSLrN//XW1/6fJKWd998BU9m+SR3DFXWlrW+SZZpxxYYWBLvJu75Op40VZZi1J1i8quG9QN6DGZmlz6PGJ8uqxvmfm9rN8887yj3T2OriNYKSkpOHjwIHbu3Inhw4cjJcV2j8js2bORlJSE7OxsrFixAsnJyaisrHRyNiIb5oq0wFyRFpgr36FbB6uuVcTffPNNTJ8+HQDQs2dPREVFYevWrbWei6gac0VaYK5IC8yVbzHsKcLqVcRPnz4Nq9WKVq3UUw1msxm5ublGNc3l2oXblqdY9F/1hIY1+BlZn81XT3g9PH28rEvPet97QbRmVK5GN79b1stmv2z3SeOmCFfHfynrfb+p5ZPmP/6i2unOdNRX8tfNZX20m2rrwnzvGIrXmrd/X91+q5oPSplrYEN8jDvk6vcZW2QdPfawrP3fbiZrrd6JZc/f5H3vejSkg1W9inh6ejpKS0tlb76azutPk5dgrkgLzBVpgbnyfro/Rbho0SK89dZb+PDDDxEWFiYXszx58qTc5+jRo4iOjta7aeTBmCvSAnNFWmCufIOuI1iOVhEHgDFjxiAtLQ2PP/44tm3bhoKCAo9/KucvzdRLHjesXAMAsI58xuG+T/VvLWtvXC5Aa+6QqxcnbJC13/1pTvc3rfqbrA+uuarG58tz1MtKvylR03j/naay0vp+25Tzkf/+nzpwkuNpw9AWcbJ+6Ea13M5C5031We6QK/I+7parfT91lHXU6Bmybh7yrKx/LdnrsuuV2i3FVFZof49ZWM2dPZxuHazaVhH/9ttv8dRTT2H8+PHo3LkzgoKCsGrVKgQE8CXz5BxzRVpgrkgLzJVv0e2/Xl2riEdFRWHjxo16NYW8CHNFWmCuSAvMlW9h99iF5l06VdYP/dRT1lZhW4OuvFA9GXb231/J+tlCvlDfU5lguzHVL8DqdN8nL90u60NFagp59S+OpoW/dLANOFtRKOv8c/UXL4yS2/pNKJe1n1+ww3M0H6aeCJq2zvby22WcmibySe8evVjWg3S43pnSfbIOXvej3ScpNXf2cPyTnYiIiMjF2MEiIiIicjFOETZRyzC1jtPDS96UtUDPGvueeWCPrNu/cUTbhpEu+oRNAAA0e6af031f/kUNjecXbaljz4YZ/O07sp5lVsP983MTHe4vhi2Q9QNXvQAAWPaOw13JCwhxVtaV8+6XdcBjS4xoDrmZYjdZjeehxAOyzvjEwIa4EEewiIiIiFxMtxGsGTNmYP369Th69Cj27NmDuDjbe3nMZjNCQkIQEhICAHjkkUcwduxYvZrVKOHB6h1FBavVSJR1+H9kbf9O3rLT2wAAS7Zcbbd1l1bN8ylG56pni2Z1fl566htZlwvtl5t47Yy6kf4fpypkHdoySfNrexOjc+VK9sucLHxJvX9tzmNGtMa3uWOuTpZXybqqSvvvqNp0XVymfuhmWDNcSrcO1s0334yHHnrI4YvT1q1bJ4NG1BDMFWmBuSItMFe+RbcOVv/+/fW6FPkQ5oq0wFyRFpgr3+IWN7knJyfDarWid+/eWLBgwXkrirujCwLbyNo6crbT/aPafwAAKD17XLM2UU165Orn0roXZA188Q1Z/1Km/WrxhcXbZF32qFqSIvQFx1OE7ebY7nAN3qAyXV5RoFHrvIOnfV/Zvw9tzOX77D65Uv/GUK2MytWGouWyNmWpEbRt16mla7qsD5V1lbXUddd+ZbSsr73lAlmHBuXI2pP/3DT8JvctW7Zg165dyMrKQmRkJCZOnGh0k8gLMFekBeaKtMBceSfDR7CqVwsPDAzEzJkzERsba3CLyBswV6QF5oq0wFx5J0M7WMXFxaioqJArimdkZCAhIcHIJtUpIvRyAEDeS+oJQGHyd7yz3dM6Z6tCHe9DmtA6V2FB7WT9yracc5Xjd04FPKJWpL944UpZu/I9WLX553uDZb20ln38E2xL5QT5q3djlVfUsrOP87Tvq2p+Jrupng0TDGwJOeJOuZp8vbpH7KWCLrKOi9gq610la112vQNnWsh6ROs+sr7lgiBZv3bac5fx0q2DNX36dLz77rsoKCjA4MGDER4ejo0bN2L06NGoqqqCEAIxMTFYuXKl85MRncNckRaYK9ICc+VbdOtgpaWlIS0trcb2HTt26NUE8kLMFWmBuSItMFe+xfB7sDxJ9njbkhPWm+2WmBDqJW1YPl2WoTPsnrqo+kPztpF+AvzUlEtQzOg69jTWD0UlRjeBiDzI6l/UdFz6sRtl/fXDn8s6bK7rrrewQN0qMf33SNed2E0Y/hQhERERkbdhB4uIiIjIxThF6ET1k4MAcGHfbACA1e7zs2dPyfqhf6XIuqrKc598oLr9UX5I1qfuWAwAaLniAaOaQ0SkqZLjLe1++t1l5/2jTH2Xhr75tKyXTVMvHX19gZo6rKg87bJr64EjWEREREQuplsHa8aMGTCbzTCZTNi7d6/cnp2djT59+iA2Nha9evXCvn376jgL0fmYK9ICc0VaYK58i25ThLWtIj516lSkpKRg0qRJWLduHSZPnoyvv/5ar2Y5dFFYN1kf/8dPsrbe9hwAoKw4R25bavlF1stOqDWdSB9G5ErYPTn66pe26/69Hscd+e//yTr8b9qs/RcefKmsP1/9tqwF+jraHYdHvQIAKCo77LI2eANP+r4iz+FJuTrxoFoDsMPNqvZ70ba2pdVa7vC4qGa9ZT08VL2A+daOtnMMnL5B7RwWIkvxV/V0vnrNKLDp03dlPfD/3qtn692DbiNY/fv3R7t27c7bVlhYiKysLNx+++0AgNGjR+PIkSPIycnRq1nk4Zgr0gJzRVpgrnyLoTe55+XloW3btggIsDXDZDIhOjoaubm5MJvNhrXryfZqlXm/WdNrfB784mJZP3bEcS+ejKNnrp48/jEAYNba1XKbGJPqeOdJ6bIs9ldLKU2437Zczeu/Nu7BiAvD4mS9969qu7jxMYf7lxapG0uv+9z2vjYB0ahr+xJ3/b4iz+auuXroE/Xn4Btrp8p6U6ptFOnnouZy2+hxarS84t5rZB0UqkbqsWQFAOCev98lN+WVqNmAOanrZH3l5zfL+n/Z7RvVfndg+E3uJpPpvJ+F4Bc9NR1zRVpgrkgLzJV3MrSD1b59e+Tn56OyshKALVR5eXlyZXGixmCuSAvMFWmBufJehk4Rtm7dGgkJCVi9ejUmTZqEzMxMmM1mQ4ZF50arKZvbt1Q63ul523Rh+7n2K51v07BV1Bh65qq4/CgAYEyKys/a8EdkLYYtcHicGK+mC1/9zXZs3iOTHO77s1+hrMOt6v0wF8A2vXfezew3PuO0zZWPZsj6aFGO0/3Jxp2+rxoi+6+XOt+JDOOuufqo/AP1w+fqxvSk/1U/BPar3HbqsY6yfr/XCVln5p2U9Sclpecqx7dC5Hx/k6x3NqK97ki3Eazp06ejXbt2yM/Px+DBg9GpUycAwPLly7F8+XLExsZi4cKFePnll/VqEnkB5oq0wFyRFpgr36LbCFZtq4h36dLF8MdRyXMxV6QF5oq0wFz5Fp9eKqdZcAdZP/a6Wi0ckY6XPXlovm0pnMJiLoND53v7dzXl98+775b17F3qZYKhLeLgiOke27Gf3eP43Gdz3pF11YUX1zhfbe+4qs2i9cPsfnq+QceS52nTLVvWJj+f/sqnBqi+/QEAAgfbf5Lr5MhdjbpeXtVeu5+ub9Q53I3hTxESEREReRt2sIiIiIhczKfHi+e0/Yv6ISmx9h3PiQqp5elCIjvz89S0W9yA22U9ppGPxgSZRzXquLLCrbJe0CtU1osKMhvXEPJ4wsrvMCK9cASLiIiIyMXcpoNlNpvRtWtXWCwWWCwWrFmzxugmkRdgrkgLzBVpgbnyLm41Rbhu3TrExTl+0koLZVWqf2m1lsnaz0+t8F1ZWSTrqy7+2VYc0b5t5Dp658re5IOfyfrGx9XL+oIef1qT65X+pp7EmREXJutXT/PJV1czMlfkvXw1V8Vn82RtWjtD1j0jx8r6xZPwKG4zgkVERETkLdyqg5WcnIz4+HhMmTIFJ096WFeV3BZzRVpgrkgLzJX3MAk3WbY7NzcX0dHRqKiowJw5c7Bnzx588MEH5+2TlZWFxMREAP4ATA7P01gVH6kXOIoQ9cTVkjvVG9YePvyCS6+pNyEqjG6C7ozOlb0A/xayTmmZLOtpCbZpvS4bJsht9i+EtH/yy377gWGvAACu/kKtVVhpLZW1/YsCtcRcGZsrZ26JUC++/d9px09L33bR97Je+5t7vHyWuXLvXGmp9MlwWZ/aq9Y5bP9G415iak/PXLnNPVjVK4cHBgZi5syZiI2NNbhF5A2YK9ICc0VaYK68i1t0sIqLi1FRUYGIiAgAQEZGBhISEnRtQ+DQn2v5xLNHrXyZO+TKXmXVb7JedkLddL7so3OF/xadW0SN4W65cubNM2pE6k1/AxtCdfK0XLmav18zWQderoL69mtX2u3V9BEsPblFB+vEiRMYPXo0qqqqIIRATEwMVq5caXSzyMMxV6QF5oq0wFx5H7foYMXExGDHjh1GN4O8DHNFWmCuSAvMlfdxiw4WERER+a4qa7GsA26y/+Rl3dviKm71mgYiIiIib8AOFhEREZGLsYNFRERE5GLsYBERERG5GDtYRERERC7GDhYRERGRi3lUB6u0tHqdNcF/GvFPVlYWSkpKGvrb7vWYK+ZKC8wVc6UF5spzcuVR78HKyck5V1mNbIbHSkxMxPfff48ePXoY3RS3wlw1DXPlGHPVNMyVY8xV0+iZK5MQQmh+FRc5deoUPv74Y5jNZoSGhhrdHI/UtWtXhIWFGd0Mt8JcNR1zVRNz1XTMVU3MVdPplSuP6mAREREReQKPugeLiIiIyBOwg0VERETkYl7dwTKbzejatSssFgssFgvWrFnjcL8ZM2bAbDbDZDJh7969cnt2djb69OmD2NhY9OrVC/v27avXcc6uW1ZWhlGjRiE2NhYWiwVDhw6VNy4WFhZi6NCh6Ny5M+Li4rB169Z6HTdw4EDExMTIay5ZsqQpv3VUB+aKtMBckRaYKwMJL9ahQwexZ88ep/t98cUXIi8vr8b+gwYNEitWrBBCCLF27VqRlJRUr+OcXbe0tFRs2LBBWK1WIYQQS5cuFUOGDBFCCHHHHXeIuXPnCiGE+O6770R0dLSoqKhwetyAAQPEe++95/TXSk3HXJEWmCvSAnNlHHawatn/xIkTokWLFvI/qtVqFVFRUeLIkSNOr9PQ627btk107NhRCCFEs2bNRGFhofysZ8+e4vPPP3d6nLsFy5sxV6QF5oq0wFwZx6unCAEgOTkZ8fHxmDJlCk6ePFnv4/Ly8tC2bVsEBNheFWYymRAdHY3c3FyXXzc1NRUjRozA6dOnYbVa0apVK/mZ2Wyu9ZrVx1WbNWsW4uPjMXbsWBw+fLhe7aTGYa5IC8wVaYG5MoZXd7C2bNmCXbt2ISsrC5GRkZg4cWKDjjeZTOf9LOr5RouGXHf+/PnIzs7GvHnzGnTNPx+3atUq7N+/H7t370a/fv0wfPjwerWVGo65Ii0wV6QF5spAxg2e6ev48eMiPDy8zn3+PDTavHnzRg2N1ve6Tz/9tEhMTBS//vqr3BYWFuZ0aNTRcX8WHBwsTp06Vevn5BrMFWmBuSItMFf68toRrOLiYpw5c0b+nJGRgYSEhHof37p1ayQkJGD16tUAgMzMTJjNZpjNZpdcd/HixcjIyMCmTZsQEREht48ZMwZpaWkAgG3btqGgoAB9+/at87jKykqcOHFC7pOZmYmoqChERkbW+9dL9cNcMVdaYK6YKy0wVwbnyrCuncYOHTokLBaLiI+PF3FxcWLkyJEOe91CCDFt2jRxySWXCH9/fxEVFSVvmDtw4IBISkoSnTt3FomJiWLv3r1Oj6vPdfPy8gQAERMTI7p37y66d+8uevXqJYQQoqCgQAwZMkR06tRJXH755WLz5s1OjysqKhKJiYkiLi5OdOvWTVxzzTVi586drvvNJIm5Yq60wFwxV1pgrozNFZfKISIiInIxr50iJCIiIjIKO1hERERELsYOFhEREZGLsYNFRERE5GLsYBERERG5GDtYRERERC7mUR2skpISZGVloaSkxOimkBdhrkgLzBVpgbnyHAFGN6AhDhw4gMTERAD+AEzOdqc/EaLC6Ca4JeaqaZgrx5irpmGuHGOumkbPXHnUCBYRERGRJ2AHi4iIiMjF2MEiIiIicjF2sIiIiIhcjB0sIiIiIhfTrYNVVlaGUaNGITY2FhaLBUOHDkVOTg4AYODAgYiJiYHFYoHFYsGSJUv0ahZ5OOaKtMBckRaYK9+i62saUlJSMGzYMJhMJjz33HNISUnBxo0bAQCpqakYPny4ns0hL8FckRaYK9ICc+U7dBvBCgkJwfXXXw+TyfbejqSkJBw+fFivy5OXYq5IC8wVaYG58i2G3YOVmpqKESNGyJ9nzZqF+Ph4jB07loGjRmOuSAvMFWmBufJuhnSw5s+fj+zsbMybNw8AsGrVKuzfvx+7d+9Gv379OERKjcJckRaYK9ICc+X9TEIIoecFFy1ahDfeeAOffPIJIiIiHO4TEhKCY8eOITIy8rztWVlZXCKgCbx56QnmyjjMFXOlBeaKudKCnrnS9Sb3xYsXIyMj47xQVVZW4vTp04iKigIAZGZmIioqqkao9GSyC22rZlfK+s3ubQAAHdoek9var53h9HzHxz0LAOi7Qf2PlFf8payFqGp8Y8ljcuWMnylQ1tuv6S/rbuO/kLUYn67q1L8BAPo+eqvc9l3pKi2b6FO8JVfkXpgr36FbBys/Px8PPvggYmJiMGjQIABAcHAwPvvsM9xwww0oLy+Hn58fWrZsifXr1+vVLPJwzBVpgbkiLTBXvkW3Dla7du1Q22zk9u3b9WoGeRnmirTAXJEWmCvfousUoTsL8G8h69ROo2V9174+dR4nrJVOz33x6/cAAA7ZbVtraSfr5L0Z6nycLvQ5/n6hAIDjE6PltoteUtN+pY/tlnXRHYtl3Xy57ebYT+44Kbfd32marF8+tcz1jSUionrhUjlERERELubTI1hBga1lXbKmXNbixrpHrVxhzM6B6tpx6j/DlP2van5tci9fXj0QAHDRS2rk9PCoV2Td5b2TdnuruuyKxwEAQX9fKrct+XGvrD+JvlbWR4s+dVFricgXXBgWJ+u3usfIus9fbW+d93vwOYfHVVT8pupH1Ij7D9u7AQC2Hm/v8Lh/5W+SdUnFcVlbreWOdvcIHMEiIiIicjF2sIiIiIhcTLcpwrKyMowbNw779u1DWFgY2rRpg/T0dJjNZhQWFmLChAk4dOgQgoODkZ6ejr59+2repouD42Utbry1jj21NXHpZ7IuuXuyrGf+tFLWVi9+6V5TuGOu6uPvbdXN6N0+vgQAYHr3Qbnt8g1nnZ7jzH4zAKDZsY/lttBL/iLrhZeq6cRb9zS6qT7JXXJln5OpPb+X9YKvEmWdU+J4CmUHvgUAdMAVcttFCK/3te+MURm8ZbJ6EOfoZnXtaz8NlXVu0eZ6n9tXuUuuopr1lvU/LlFTgZMnrJF1wCMzHR5bXm47NnDbsw4/9wtSmQhb9ISse/7p3392P3rJ2mT3fsmMJ8fLeuK+NwF4zrShriNYKSkpOHjwIHbu3Inhw4cjJSUFADB79mwkJSUhOzsbK1asQHJyMiornT+dRwQwV6QN5oq0wFz5Dt06WHWtIv7mm29i+vTpAICePXsiKioKW7du1atp5MGYK9ICc0VaYK58i2FPEVavIn769GlYrVa0atVKfmY2m5Gbm6vJdVuGWWR96PUdshZwPkV49uxpAMBvKWrqLvJex7+FZ6+4SdYhIZfUfeJB/5Tl3QfU5hebq6cx9hSvc9o+Mi5X9WH/rrWF6W/IWgT/BwBw08RxcluVdbnT87V5dT8AIGZdB7lt73Y1lTNmgXp/1u0jm9mdu7ghzSYYl6v+bU7JukPmnbJ+XqiRDZNJfQcJu+3YYXtxZXmni+WmkOaXOdzX0TnO35Yk67YjP1TX6F5Uv18IOWRUro6t+EHWYvRddp9cJavS2XNk/fHGa2T9r59sU8d7itUTy/ZuCJ8q6zWFJ2T9qsXWkVyT7/hdj7e2V+M9k6eqbI7bNUDWHQfa/n/o86VdBt2YIR2s6lXE09PTUVpaKnvz1XRef5q8BHNFWmCuSAvMlffT/SnCRYsW4a233sKHH36IsLAwuZjlyZPqhtyjR48iOjq6tlMQ1cBckRaYK9ICc+UbdB3BcrSKOACMGTMGaWlpePzxx7Ft2zYUFBRo9vTEhp5RshY33ON0/4psNeXy0IAuAIC0E2p4FascHzcp8mpZv/D2agCA6eqHG9JUZL37nawXTLENu/4zx/nUka9xh1zVx5vxw2Qtbhgs6/yxtiVt3i9q3KN+pyoP2/2kXuInhi2QdYcwNcV8uOijRl3H17hDrvxMahTjhcvV98FL+Wpqbkq72p4MvPLcv/+w2/adox3Pc1PvbwAArV693+HnP04tkTWfHGw4d8jV/IeSZW1+YovannNG1geKC+yOeL3e524T4i/rZy9XU5xzcl6u87gvD6r6/ofbyPq3/mrJr14PHgEA+H+lnlSsspbWu216062DVdsq4t9++y2eeuopjB8/Hp07d0ZQUBBWrVqFgACffsk81RNzRVpgrkgLzJVv0e2/Xl2riEdFRWHjxo16NYW8CHNFWmCuSAvMlW/xme6xv59tSLHHY/ZPPoxxelzAp1/IOu1E/ddze/W0GtbsfZft6Z87MtT0nn/3qTWOqcHu6cJHXvoXAOCFUf3ltvyiLTUOIfcSGBAp65FPqvwIqCnCv3zUFgBgFVmNukbLALVOWGDnmxt1DnJPQ+97R9bpT06Q9c4S9ULIe3507TXvvNU2bSWEetrL7yP1RNmgr1rVOIY8i5a3mrx8Sv3Zh1O17wcAA8PUk7FP986TdZc31JPR/i3VE6wThti+I6usdtdwY1wqh4iIiMjF2MEiIiIicjGfmSJ8Pe6vtuLaa+reEUB5uXo52t+fSLH7pHHDkncffAUA8Hg/tf5T/tK/qR0mpjs/ybnpwsPrn5SbQgYHy9pT1mbyNasvV2sDimEqe7lj1BD9T8U7dW0Teaaro4+oH1w8LRgU2FrWgb3PvY/JpJ4Gm3+PenHymVI+yUz14+9/gazf6m57inpY6i65zXRVH1mX/qHesh3ymlrnMOKhTbIuLj+qSTu1whEsIiIiIhfTbQRrxowZWL9+PY4ePYo9e/YgLs62grfZbEZISAhCQkIAAI888gjGjh3r8uuP3nUdAEBYnS+eGbRirqzTC1232OaJ4m9l3e5eu9Es1H80yzRA3Wxqwr0ua5unMjpXzlwQdNbh9u+yY2VtFduadI0tI4873F5WqNYx+91a4HAfcsyYEQN3AAAgAElEQVToXFnCbOcsGa6WufnqyT9q273J+gaOkLV1SA8AQPF9/5Lb/p3P5ZVcwehcOWK/hFd67F9lHezveEmbagd/U+9f69JCvZftiij1fRT3DzXqerZXdwBA2Vz1cNb0kerBnvdKN8v6TKn9n7ueNWplT7cO1s0334yHHnrI4YvT1q1bJ4NG1BDMFWmBuSItMFe+RbcOVv/+/Z3vRNRAzBVpgbkiLTBXvsUtbnJPTk6G1WpF7969sWDBgvNWFDdC2pJJdj+9pMk17KcLL7m3p6yP9MuUdVDM6DrP0Tl0iKwPFK93Yeu8gzvk6roZ78paYKis5x75zWXXiOq3z/EHS9SK86dKTjjehxpMz1wF//S53U9X1rpfU3381geyFqZeAIDXN6rvl4rKupc5oaYz6vtqaku1bM74lzfL2q/3A406X9Uu9RDEy7cOlPWy/B0AgD3nLcHjGe+zaizDb3LfsmULdu3ahaysLERGRmLixIlGN4m8AHNFWmCuSAvMlXcyfASrerXwwMBAzJw5E7GxsU6OIHKOuSItMFekBebKOxnawSouLkZFRYVcUTwjIwMJCQlGNskQhcXqKTL/33rU+7jnu6l60NeubJFnMzpXLcMssi4douqgbxfLOqfcyRoSDWE3Dm3yU/9Lf/Gp/f0ea113PR+lZ66ql8KJGnC53Vbt/icXgx9XtXDdk9PknNHfV2kn1DTdC/3V0l7tQtY42t2hV+JDZd33CfUU4eR7dsh69cPJ8DW6dbCmT5+Od999FwUFBRg8eDDCw8OxceNGjB49GlVVVRBCICYmBitXrtSrSeQFmCvSAnNFWmCufItuHay0tDSkpaXV2L5jxw4HexPVD3NFWmCuSAvMlW8x/B4sOt/W6dGyHvB/BjaEXMNuuuXsO6dVXVHY5FMHBtiG88VlF6mNdi/S3XfmwiZfg4x1prSWJ0RdYEyLu2VtslsWp9pL+UU1tpF3q6hU31FHijbVsef57G9R8R/aTNYrLrtN1ht3/Gy7xvPt5LbIVHW9Kmtpg9rqCQx/ipCIiIjI27CDRURERORinCJ0Mxc2q/+w/JbjbTVsCTVWmF+ErP0ujJd1cDfXPsl3YXAMAOBsdHt1DbvPt52qOe1DVG1AlFonUwi17lzRfQsAADtLcvRuEmns6jDb+7X2WNWc3u9lP7r0GlVWtW7lhB/UzfrPJIwBAHzz3aVyW3kv9TT15SnDZf1j0fsubZNROIJFRERE5GK6dbBmzJgBs9kMk8mEvXv3yu3Z2dno06cPYmNj0atXL+zbp91NneR9mCvSAnNFWmCufItuU4S1rSI+depUpKSkYNKkSVi3bh0mT56Mr7/2rbdm3ttmmqy7buhQ7+P+lfeaFs3xKO6Yq3KhnoapKv1Zs+vc08r2Utrg9moty/Jytebgnspjml3b27ljrlxtdJJqt8l0layf2/CXc9VykGsZkauLwtQbqT/+4QwAoP8ViXJbFlw7RVibXSW2WyRuTpoqt737xgH1+daTsr7y6ptk/UPxWzq0Thu6jWD1798f7dq1O29bYWEhsrKycPvttwMARo8ejSNHjiAnJ0evZpGHY65IC8wVaYG58i2G3uSel5eHtm3bIiDA1gyTyYTo6Gjk5ubCbDYb1q7pL3wi67SR18v6p6IPHO3eKDHhQ2V9X9J2WQcF1b1UzmtxW2VttZa5rD3exOhcNTOp90+Ftohz6blvv0iNds7cZq3x+Y4h6uVpB4o3uPTavs7oXLlaq8HZsrba3eT+WWGFEc3xWVrnavbFSbI+/bBtWbasEuNebLqhSI2MxowbKOtDb6k8fj9nj6zD56jv08qqX7VtnIsZfpO7yWQ672chhEEtIW/CXJEWmCvSAnPlnQztYLVv3x75+fmorLS9fVoIgby8PLmyOFFjMFekBeaKtMBceS9Dpwhbt26NhIQErF69GpMmTUJmZibMZrM2w+1r77H9e/SzTnc1DZgj6zd7Z8i6x6eNu3R0+EAAwNu91G93l1fCZB3c/man58i7JR0AcNcBNbQrwL/lOKJrrhpAtFRL2vj7q/edVVX9UedxA8PulPWyH9XfiUIvtE0ni+f+JrfdtLN7k9tJjrlrrhrimrDJsrbe1kvWjpbKIX3omatffnGv5bNyizbLeu5ktXTTEznqJvxbn86S9apflunSLlfRbQRr+vTpaNeuHfLz8zF48GB06tQJALB8+XIsX74csbGxWLhwIV5++WW9mkRegLkiLTBXpAXmyrfoNoJV2yriXbp08djHnMl4zBVpgbkiLTBXvsVnlsqJGN8KAHDGOkNuE2NSnR53+YcjZV35vu1pv1cevam23aU7Hlol67KhtsdyQ6IG1qut1UzvzZL1lR/alkOxX9KC3NPRki9kLVLVE6mmGemyntZKDXsvLag57N0itKus58T/IuvQC/uocy+zTQ12e1Q96XqyeH1jm00+RohKo5tAOjjwm/pj/oHbtwAAmn2p3rdYXH5U9zY58tQxtazOE5vekfU/+qq2rvKwrzfDnyIkIiIi8jbsYBERERG5mM9MERaVHwEAzH5AvahxwRjnx/n7N5O1uPEZAMAdN9bnilfLKqReLbSxnxZsfVt7Wf9SsrsBZyEjVVnVUjkvpasnAO9Ss9NY9M4WWZ8dacvk3d33y22dXlRP+wS3V1OAlfvV8kj/XHgXAOBA8fMuaDX5GpMpwK7mU4TeatWZNbJeetU1AID7WneU2xbkvyBrI29BqbIWy9q6V01fx9xu95Q1pwiJiIiIfJvbdLDMZjO6du0Ki8UCi8WCNWvWOD+IyAnmirTAXJEWmCvv4lZThOvWrUNcnGvXbvuzZ46r6ZTmHVNk/cihnppet4ZN/5Dl/KlqzvFf+WpotKqK04KuoEeuarPw2GFZ3/6rerlsaM+Zsn7u5+rK8TqUVmu5rB8fqn4dTx/j1KCRjMyVK9g/RWja+E9Zf3WWTxcaydW5qqg8Letnh14KAHj0oFpw+tprhsv6um2fydrZC5BdbW3322R99jY19vPOdfbLCK2CJ3GbESwiIiIib+FWHazk5GTEx8djypQpOHnypNHNIS/BXJEWmCvSAnPlPUzCTZbtzs3NRXR0NCoqKjBnzhzs2bMHH3zwwXn7ZGVlITExEYA/AJPD8zSEye4cF4R0lvW7Ceolj/1m2b008txThPVxYrx6iWn2ETMAYMvxtnLbE3krZG0/BaQlISp0uY47MSJXtWkZZpH1tUFXyfrpv9je4HzxdXvltuI9bWQ9+ZURss783f2mBZkrY3PVENk39pZ1h0z1hOuj0btkvei4e6z3xlxpl6sFl6rbY+79Qb3IM+Sj/8j67qnJsl555i1Zl1cUNOqa1XqE3SrrVQPyZd1p/ThZZ137nqz7ffWtrCurfm3StQF9c+U292BVrxweGBiImTNnIjY21uAWkTdgrkgLzBVpgbnyLm7RwSouLkZFRQUiIiIAABkZGUhISND8ugJq8O73sh9lPehrVeO8VXGmNPJKvFndCEblqjanSnbKeo19Xf2g0HkPDOXb1e43auXL3C1XDdF2cStZ29/k7i6jVr5Mz1w9ckS9++qDlpNknTFKjWYt+9+7sv5v9BWyPjXXNro+73O1bNeZs+rPUsuF6l1aY7qr77n2j9pmaqq6XSm3BW7eLOtX4rfLeuqBj+r3C3FzbtHBOnHiBEaPHo2qqioIIRATE4OVK1c6P5CoDswVaYG5Ii0wV97HLTpYMTEx2LFjh9HNIC/DXJEWmCvSAnPlfdyig0VERNq4MEy9Uyk45mZZW6v0ebiG3NuXJa/Kut3ranvQ2tay/jAxXNZX//X/AADPLFTTiQGn81SdqNYEs6a+IuvvHrgOALD6x0Ny24unfpd1lVU9+OUt3Oo1DURERETegB0sIiIiIhfjFCERkRcTwipr+2lBv4/mGNEc8hBnKwplfe036olCfHPu3w+/C8cm1bL9g1q2ey+OYBERERG5GDtYRERERC7GKUIiIi92pnSfrIMCphrYEiLf4lEjWKWlpecqwX8a8U9WVhZKSkoa+tvu9Zgr5koLzBVzpQXmynNy5VEjWDk5Oecqa127US0SExPx/fffo0ePHkY3xa0wV03DXDnGXDUNc+UYc9U0eubKJIQQml/FRU6dOoWPP/4YZrMZoaGhRjfHI3Xt2hVhYWFGN8OtMFdNx1zVxFw1HXNVE3PVdHrlyqM6WERERESewKPuwSIiIiLyBOxgEREREbmYV3ewzGYzunbtCovFAovFgjVr1jjcb8aMGTCbzTCZTNi7d6/cnp2djT59+iA2Nha9evXCvn376nWcs+uWlZVh1KhRiI2NhcViwdChQ+WNi4WFhRg6dCg6d+6MuLg4bN26tV7HDRw4EDExMfKaS5YsacpvHdWBuSItMFekBebKQMKLdejQQezZs8fpfl988YXIy8ursf+gQYPEihUrhBBCrF27ViQlJdXrOGfXLS0tFRs2bBBWq1UIIcTSpUvFkCFDhBBC3HHHHWLu3LlCCCG+++47ER0dLSoqKpweN2DAAPHee+85/bVS0zFXpAXmirTAXBmHHaxa9j9x4oRo0aKF/I9qtVpFVFSUOHLkiNPrNPS627ZtEx07dhRCCNGsWTNRWFgoP+vZs6f4/PPPnR7nbsHyZswVaYG5Ii0wV8bx6ilCAEhOTkZ8fDymTJmCkydP1vu4vLw8tG3bFgEBtleFmUwmREdHIzc31+XXTU1NxYgRI3D69GlYrVa0atVKfmY2m2u9ZvVx1WbNmoX4+HiMHTsWhw8frlc7qXGYK9ICc0VaYK6M4dUdrC1btmDXrl3IyspCZGQkJk6c2KDjTSbTeT+Ler7RoiHXnT9/PrKzszFv3rwGXfPPx61atQr79+/H7t270a9fPwwfPrxebaWGY65IC8wVaYG5MpBxg2f6On78uAgPD69znz8PjTZv3rxRQ6P1ve7TTz8tEhMTxa+//iq3hYWFOR0adXTcnwUHB4tTp07V+jm5BnNFWmCuSAvMlb68dgSruLgYZ86ckT9nZGQgISGh3se3bt0aCQkJWL16NQAgMzMTZrMZZrPZJdddvHgxMjIysGnTJkRERMjtY8aMQVpaGgBg27ZtKCgoQN++fes8rrKyEidOnJD7ZGZmIioqCpGRkfX+9VL9MFfMlRaYK+ZKC8yVwbkyrGunsUOHDgmLxSLi4+NFXFycGDlypMNetxBCTJs2TVxyySXC399fREVFyRvmDhw4IJKSkkTnzp1FYmKi2Lt3r9Pj6nPdvLw8AUDExMSI7t27i+7du4tevXoJIYQoKCgQQ4YMEZ06dRKXX3652Lx5s9PjioqKRGJiooiLixPdunUT11xzjdi5c6frfjNJYq6YKy0wV8yVFpgrY3PFpXKIiIiIXMxrpwiJiIiIjMIOFhEREZGLsYNFRERE5GLsYBERERG5GDtYRERERC7GDhYRERGRi3lUB6ukpARZWVkoKSkxuinkRZgr0gJzRVpgrjxHgNENaIgDBw4gMTERgD8Ak7Pd6U+EqDC6CW6JuWoa5sox5qppmCvHmKum0TNXHjWCRUREROQJ2MEiIiIicjF2sIiIiIhcjB0sIiIiIhdjB4uIiIjIxXTrYJWVlWHUqFGIjY2FxWLB0KFDkZOTAwAYOHAgYmJiYLFYYLFYsGTJEr2aRR6OuSItMFekBebKt+j6moaUlBQMGzYMJpMJzz33HFJSUrBx40YAQGpqKoYPH65nc8hLMFekBeaKtMBc+Q7dRrBCQkJw/fXXw2SyvbcjKSkJhw8f1uvy5KWYK9ICc0VaYK58i2H3YKWmpmLEiBHy51mzZiE+Ph5jx45l4KjRmCvSAnNFWmCuvJshHaz58+cjOzsb8+bNAwCsWrUK+/fvx+7du9GvXz8OkVKjMFekBeaKtMBceT+TEELoecFFixbhjTfewCeffIKIiAiH+4SEhODYsWOIjIw8b3tWVhaXCGgCb156grkyDnPFXGmBuWKutKBnrnS9yX3x4sXIyMg4L1SVlZU4ffo0oqKiAACZmZmIioqqESqi2rhbrvz9QmU9MnyirB+y5Djcv3mzYgBA7PtqX5Of+l9TvDRF1hnP3ibrew59AwAoqTgpt1VW/drIVtOfuVuuyDswV75Dtw5Wfn4+HnzwQcTExGDQoEEAgODgYHz22We44YYbUF5eDj8/P7Rs2RLr16/Xq1nk4Zgr0gJzRVpgrnyLbh2sdu3aobbZyO3bt+vVDPIyzBVpgbkiLTBXvkXXKUJvZAkbK+uMQfmyDgktlXXLTnm2bfMWOD7Ja3+T5XupN8v66R+DZf11yWtNbitpp3WznrL+arB6dqRDZoLdXvZ1TUJUqbpK1bgjTZbj7lCbx+FqAMAnSZvktuu3r6tvk4mISENcKoeIiIjIxTiC1QChQW0BANuv7Si3tV95oaxDLrzG4XEmkz+A80cozjNBjVAMn6A2Dy7OkfXvd3cDALR7fXeD2kz6eKdHS1m3n3tK1mVF2bIOCe8sa7/V02VdHYuzh5vLbX8ci5J15O3Fat+Bj9W49nX//EzWP76UJOvY9d/Uu/3kvfz9L5D1RSGxDvd5uE0PWc84eGW9z+3nr0bZyx77u6yHPWt7xcCXJa/W+1zkHZoFd5B1D/9Bsl569VEAQJcB2+S2invU7M3hMd/L+sav1djP0aJPNWmnHjiCRURERORi7GARERERuZhuU4RlZWUYN24c9u3bh7CwMLRp0wbp6ekwm80oLCzEhAkTcOjQIQQHByM9PR19+/bVq2l16hB+rawPbra9Y8jPcrvDfc/+U03fVJaEyNpksj01IkTDXgoXept6B0qLF0cDAPacVlOSlk3qqZMqq5pG8iXukqs+X34o6+j+A2X98YCvZb3s+xJZp51QWbDKF9/9YndGVUesuVzW7ye+K+tem2+0HX+9engiuu1ydQo+5d1o7pKrpogOHwgAODhbfU/4PXynw31NJrv3ronKel/DanfXQ+ATT8h647j/AQA69+kvt+UXban3eb2VN+TKkVmX3C3rBUtfl7V15FWyFqL6QSD1IJeaYAYu++BSWf+49H5ZX/SImnIsLj/qgtbqR9cRrJSUFBw8eBA7d+7E8OHDkZKSAgCYPXs2kpKSkJ2djRUrViA5ORmVlfX/n5x8G3NFWmCuSAvMle/QrYNV1yrib775JqZPt93027NnT0RFRWHr1q16NY08GHNFWmCuSAvMlW8x7CnC6lXET58+DavVilatWsnPzGYzcnNzjWraeU/d/PT0F+qDhHQA5z8N+Otdi2Xd5lW1TEmtTww2QHCa+tvLM5fafj/uTN4ht4Vubi3rovIjTb6eN3CHXOUWbZb1ZRvsP/mqUecL879I1kn3qHNbcWOjzkcN5w65qo8rmt0k6/f+YmuT38NPNfm81v3/k/Xsv1hk/Z+37ab9EtU0kd9lyQCA1+I2ym3X8qHWGjwlV/aqlwL7+Y72clvIQvXy1KqI+bJ2dFNM0YxFsi4vCpN15CvT1E73LpHl8ldUcG7fu6pRbTaKIR2s6lXE09PTUVpaKnvz1XRef5q8BHNFWmCuSAvMlffT/SnCRYsW4a233sKHH36IsLAwuZjlyZNqwdqjR48iOjpa76aRB2OuSAvMFWmBufINuo5gOVpFHADGjBmDtLQ0PP7449i2bRsKCgoMfXriwHD11Ja4Sz11U/33i1N3pMptXdeqv2W4YlrQXnlFgazv+fFl27/n2u/xu0uv56k8JVeNNTq8m6ytN483sCW+xVNyNbudmpr7xxd5sg6IvsPR7o0SmPWdrJ8/9ZOs5623qn0Sax73W1moy9rgLTwlV9VTgQAwKnySrF/bdRwAENjuOofHlf+hXq78+eDjsp633/bM4LYyNe0ZEXqZrI8+p5aaCw5rJ+vmQeUNbbrb0K2DVdsq4t9++y2eeuopjB8/Hp07d0ZQUBBWrVqFgAC+ZJ6cY65IC8wVaYG58i26/deraxXxqKgobNy40eFnRHVhrkgLzBVpgbnyLeweO3Dxfy+RdfU6ggDw823PAQAue6dIbvO0F5+RZ7Afno8JPyvrs3kfyDqo/fW6toncx5VhybJ+9PsKWQdGjpB1Q14Y6kzJqJmyfm/ZYVkHd1Jvs7Wipiva2z0Ft9NlzSEd3Bulppifyr3C7pM4AOdPBRbPVB3DW9ep9VA3l7xZ5zV+Lzsk61evLJT1yXL1RP6CYx/Xv9FuhkvlEBEREbkYO1hERERELsYpwnOmtlYvOfNr01HW9k8GXvaObdiyPtOC4cFqXaWAc9M9wm4Q/bfSA41vLHmVsCDbEzNf9Y+V2654/Jjawapedlvur17y6MjZ9j1k/W6PTrK+aZdaK7Gq6o9Gt5WMY/8C5G/eVFMy1sgnZe3nr1Z3szp5qLniyDuyDjipvtOuGtwdAJBVkiG3Pd9FPU09+Ye/yNrkP1LVVeppL7837wMAJH2qMkjub3lXNS2Y/KHddxDUFKH/hjkAgKdnjJXb/p27z25f+7om+z9rn5v3gqxLR6r3gF1zqfqOqqz6zWm73RVHsIiIiIhcTLcRrBkzZmD9+vU4evQo9uzZg7g4241yZrMZISEhCAkJAQA88sgjGDt2bF2nchn7vxHa96RFQJrD/R2NXLVu1lPWa7pdLOueGeq3NvjczchlZepvBDk3fSZryya12n2VtbhebScbd8xVQzULjAIAXPGR8/bZrz5/Nt9286epSt0EH9JB3eQ8TL26CEXz1QjWjU/dJeuNxS82tLk+wR1zFRnSVdZVrdV/c9jdzG4/auXoJne/T9Vo18Wj1Pff72U/2u1lG4HoFK4eopiyVN1obBXqhVfnXS9rmaxvTJkIAPi1ZLmjX4rPcsdc2Ztw2zpZ+7VbKOvqUSsAaHWb7SGwX0vS1b5+6psp0P9CWV8WOEDWX+04d+N6dAe5zRrwX1nbf7e18tvViNa7H906WDfffDMeeughhy9OW7dunQwaUUMwV6QF5oq0wFz5Ft06WP3799frUuRDmCvSAnNFWmCufItb3OSenJwMq9WK3r17Y8GCBeetKK6lC4LUauDijvsc7lP2iFqb5r+dpwAA7n5Z3XRc1qW7rEMi1XShI8HBbWTdZcNtsn7/qpayHrYt01mzqZ6MylVD/VpmW3rkyF9fk9suHfxdbbtLNz1quyH1hDgjt42/WE03zzh4paz9H1Ur2Gf8rIb+I59vRIN9nFG5KizeJutlt02R9eQdakmtkNB2qMvMiX+V9e9lakrP/naJ8RG25ZheWqKmj6sGPevwfMUPqCnHLi+r77fCYk4NNpQ7fF9VTJsoa/spO2uEmvbbPrT6HX3qz7uLux+Utf+j/5C1yaS6GI6mrCsq1HeX39NPyfpbq7khzXZbht/kvmXLFuzatQtZWVmIjIzExIkTnR9E5ARzRVpgrkgLzJV3MnwEq3q18MDAQMycOROxsbFOjiByjrkiLTBXpAXmyjsZ2sEqLi5GRUWFXFE8IyMDCQkJul3/j/Ij6oel01V9r3qKMGTBE7KettC2bI4Qamg0xO58ptfUOfI+qPm+olZL1VNAwa37yXrAa6dl3fJKddypEq4t0RhG56qhqt/z0uW9/1Mb36vPkTWfAPx3i1tc0yiqwZ1ydf9PL8l6VdStsv7m97qnCBduOynrK4aod1uNTFRTy61es73zqApqWtD01TPqener767J+/Pszm5fU325U67CtqySddWI+bIWVz8o63ZXu+56QSvULThBc+2nEHe77iIG0q2DNX36dLz77rsoKCjA4MGDER4ejo0bN2L06NGoqqqCEAIxMTFYuXKlXk0iL8BckRaYK9ICc+VbdOtgpaWlIS2t5vulduzYoVcTyAsxV6QF5oq0wFz5FsPvwTJSlbVU1rfOvVvWK/+mnmwICoqUdekvtmH0n+9Uw5dDPg+SdW6R/RDndvxZxdSf1Q/XDpRlYJfbZd3HXz0ptJ7Lz1M9DAtPkfWg9b8b2BIygv2SNj+NVNN0Hd+9tca+QRcPlPXkvWq7ydRb1hWHbE8y/+/GS+S2mYfVi01LzqqnXcm7XHibeqJ9QKD6c+7fvbNlvfGwbfmj62J+ktue/K6LrDuEqyVvFuaqp+yr+b/3qKxb/t3+XrO6l9jxRIY/RUhERETkbdjBIiIiInIxn54itJf5u3rj4g8t1QrxgVBTgL+ZTgEAcovsnvZqCKtVlkKoRbzO5n0g612iqHHnJp/1ULx60WRwmxsMbAkZbcAX6raHY6b6f737+avXSv4y9wQA4O7sr+W2qqo/XNA6cnd/lB2S9fv29Sf2e9nW0X34sNoSFKjmm09NVeM2fv69ZH324JsAgCfuUy/ZPlPq3W865ggWERERkYvp1sGaMWMGzGYzTCYT9u5Vvd3s7Gz06dMHsbGx6NWrF/bt874b3Ug7zBVpgbkiLTBXvkW3KcLaVhGfOnUqUlJSMGnSJKxbtw6TJ0/G119/XctZ9HGgeL1Lz2cJGwsAKE1U6znZv6DU71W1tuHRojOg+vOkXDWWn5+avgn0Vxl6I24AAODqW+zXr3Q8RWj65HFZx6/qaPfJkRr7kmfl6sqwZFnP7mx/G4LtqebyX+xeARAcocpml8raqu5YwEVPRAEAYt9Tn+8vfsdl7fVlnpQrZ4ID1dqTb3dXv57g+YNkXfK7WqPwtp62pw/f/8O7pwXt6TaC1b9/f7Rrd/5bhgsLC5GVlYXbb7e9pmD06NE4cuQIcnJy9GoWeTjmirTAXJEWmCvfYuhN7nl5eWjbti0CAmzNMJlMiI6ORm5uLsxms5FNc6ltz9lGGMSFNV8wBwCvZIy2++llHVrk3bwhV/7+F8j64IjLZB29brKDva9xeA6/j+bIOmbcFbI+XrS5ye3zRUbnqkWoWmrr4Fj1rqGwpTGyDglVf3hb//MAAGDkk2PlthvbqhHyyTsKHB4XEHMjAKBXyA9y2/7iJjWd6mB0rhrr8yS1nAmINrUAACAASURBVE/iZ4Mc7rMwTuXt/T+Wa94md2P4Te4mk+m8n4UQBrWEvAlzRVpgrkgLzJV3MrSD1b59e+Tn56Oy0navgBACeXl5cmVxosZgrkgLzBVpgbnyXoZOEbZu3RoJCQlYvXo1Jk2ahMzMTJjNZrceFq2v1+PV8jeYOND2b7t3X+Xd8pKs7/vJbs0KajIjchUTPlTW/+uhti//QU3frPpVLTFSZVVzLheHXwUAuDZIDbm/+IBa7NV/tqNpQcfs36l2xbhuss4t+rTe5yDHjP6+uiNCTQVf8Jh6OCEgRN1s7Pf6PbK+ecGdAIDNJWpqZrNa3QS9rx8m6ys/P/++IAC4qcMpWb92upGNJqeMzlVD3XfxNABAt48vcfh5Ze6Hsp6f974ubXJXuo1gTZ8+He3atUN+fj4GDx6MTp1sTxQsX74cy5cvR2xsLBYuXIiXX+Y9SFR/zBVpgbkiLTBXvkW3EazaVhHv0qWL2z+OSu6LuSItMFekBebKt3CpHBca0+JuVe9MlLXJz/bbXHpqu9w2aGMzWVdZ1fIW5FmimvUGAOz5Xr1/KLCjeir0Bbt9lz9bKeuqP9T7iPxvKAEAmHrYrzz/jNNrF82cBwC4YJTa9vbMAbI+WvSu03OQe7N/x9W/foyUdUCIeirUb+V9sg7720WyPltR91Nbeb+pd6pd6eDzt462bEhTyUfMPzfPHBCoMuj/P5XB5ncxN9UMf4qQiIiIyNuwg0VERETkYpwibITAADVUnxF3nayHvnNC1sLuicGzZbZHcFb3PSu35fJlj17hQtiWFQk6+r3cJjqOdrivaWaqrF3xP96IV2wv98teodYtKyxe5YIzk7tYP3qnrENC+8tapN4r62YP208LFtb73Ndfv0nWfv7qe8x+2RzyXRGhl8v66N9+k7VoP6DGvtteGSzr8oqPtG2YB+EIFhEREZGLuU0Hy2w2o2vXrrBYLLBYLFizZo3RTSIvwFyRFpgr0gJz5V3caopw3bp1iIuLM+Taw8JTZH1Zc/Xbsvj4Mlk/F2t74ePfFr0tt1mvH+j03Luu2wwAmHbwwzr3I21omavsUts0yxOTJ8lt/8hUmTH1mOb0HGVF2QCAExO/lNuiHytzuO/rkyyy/rrsDQCA1Vpe7/aS62iZK3+/UADABRerl30KoZ5C3bDyJlmfrXijxnEAkBRyS43zpiYdU+ebrbJkrVIZsr8O6c/IPwftLbm0p6xD/tOrxueF49Uz0n23/lTjc3KjESwiIiIib+FWHazk5GTEx8djypQpOHnypNHNIS/BXJEWmCvSAnPlPdxminDLli2Ijo5GRUUF5syZg4kTJ+KDDz5wfqCLxF6gfise3xsk63/9dqOsg6Nta8ZZRc3hUgAwbZ4n63ce7CvrMbs4NWgUrXNV/ZLYnGKT3PaXgeols+vvflzW+fs7yvr5LLVO4IpfbfkoKlfrywV/oNaXs1de8ZrD7aQvzb+vTLbvo4DmxQ4/HrYyV9YlbzeXtX8zu+ni+xx/TzlT/ssOAMDRUquTPcnVjP5z0H5N1eRl6pYFgZpZat1rv6y3newn675fHpR1ydl8VzfRo7hNB6t65fDAwEDMnDkTsbGxBreIvAFzRVpgrkgLzJV3cYsOVnFxMSoqKhARYVs+JCMjAwkJCbq2Ieu3EvWDfwtZBrW/vsa+fm89KOtPF6n3x0zap1YX/7nodRe3kBpKz1yt/mWZw+3Nz1vx5me7emud5yuvKGhym0gbeuSq+mb17C3qXUSdR/5P1n6XqSV0cJkqTSb1ld6Qm9Wf66qW8XrxZ9uN9ftL3qn38dR07vDn4Kvd7fJz9YN17An4t1F3GC3f00XWJWc/dX3DPJRbdLBOnDiB0aNHo6qqCkIIxMTEYOXKlUY3izwcc0VaYK5IC8yV93GLDlZMTAx27NhhdDPIyzBXpAXmirTAXHkft+hguYMvS16VdfMWte9X01uubgoR+bjqJW+6bVRL3/RLmiTrRT3Xy/qyzM6yDomIl3Xl4XcBAHtT1AMY9sZ9r25iP1K0yeE+RH+WPcL28tOen6klmkrPOr5Fwte51WsaiIiIiLwBO1hERERELsYpQiIiD2B/G0PvL+w+aPmx7m0h79T/q/fVD4Hv174j1QtHsIiIiIhcjB0sIiIiIhdjB4uIiIjIxTyqg1VaWnquEvynEf9kZWWhpMTujfUEgLlirrTBXDFXWmCuPCdXHnWTe05OzrmKi5A2RmJiIr7//nv06NHD6Ka4FeaqaZgrx5irpmGuHGOumkbPXJmEEELzq7jIqVOn8PHHH8NsNiM0NNTo5nikrl27IiwszOhmuBXmqumYq5qYq6ZjrmpirppOr1x5VAeLiIiIyBN41D1YRERERJ6AHSwiIiIiF/PqDpbZbEbXrl1hsVhgsViwZs0ah/vNmDEDZrMZJpMJe/fulduzs7PRp08fxMbGolevXti3b1+9jnN23bKyMowaNQqxsbGwWCwYOnSovHGxsLAQQ4cORefOnREXF4etW7fW67iBAwciJiZGXnPJkiVN+a2jOjBXpAXmirTAXBlIeLEOHTqIPXv2ON3viy++EHl5eTX2HzRokFixYoUQQoi1a9eKpKSkeh3n7LqlpaViw4YNwmq1CiGEWLp0qRgyZIgQQog77rhDzJ07VwghxHfffSeio6NFRUWF0+MGDBgg3nvvPae/Vmo65oq0wFyRFpgr47CDVcv+J06cEC1atJD/Ua1Wq4iKihJHjhxxep2GXnfbtm2iY8eOQgghmjVrJgoLC+VnPXv2FJ9//rnT49wtWN6MuSItMFekBebKOF49RQgAycnJiI+Px5QpU3Dy5Ml6H5eXl4e2bdsiIMD2qjCTyYTo6Gjk5ua6/LqpqakYMWIETp8+DavVilatWsnPzGZzrdesPq7arFmzEB8fj7Fjx+Lw4cP1aic1DnNFWmCuSAvMlTG8uoO1ZcsW7Nq1C1lZWYiMjMTEiRMbdLzJZDrvZ1HPN1o05Lrz589HdnY25s2b16Br/vm4VatWYf/+/di9ezf69euH4cOH16ut1HDMFWmBuSItMFcGMm7wTF/Hjx8X4eHhde7z56HR5s2bN2potL7Xffrpp0ViYqL49ddf5bawsDCnQ6OOjvuz4OBgcerUqVo/J9dgrkgLzBVpgbnSl9eOYBUXF+PMmTPy54yMDCQkJNT7+NatWyMhIQGrV68GAGRmZsJsNsNsNrvkuosXL0ZGRgY2bdqEiIgIuX3MmDFIS0sDAGzbtg0FBQXo27dvncdVVlbixIkTcp/MzExERUUhMjKy3r9eqh/mirnSAnPFXGmBuTI4V4Z17TR26NAhYbFYRHx8vIiLixMjR4502OsWQohp06aJSy65RPj7+4uoqCh5w9yBAwdEUlKS6Ny5s0hMTBR79+51elx9rpuXlycAiJiYGNG9e3fRvXt30atXLyGEEAUFBWLIkCGiU6dO4vLLLxebN292elxRUZFITEwUcXFxolu3buKaa64RO3fudN1vJknMFXOlBeaKudICc2VsrrhUDhEREZGLee0UIREREZFR2MEiIiIicjF2sIiIiIhcjB0sIiIiIhdjB4uIiIjIxdjBIiIiInIxj+pglZSUICsrCyUlJUY3hbwIc0VaYK5IC8yV5wgwugENceDAASQmJgLwB2Bytjv9iRAVRjfBLTFXTcNcOcZcNQ1z5Rhz1TR65sqjRrCIiIiIPAE7WEREREQuxg4WERERkYuxg0VERETkYuxgEREREbmYbh2ssrIyjBo1CrGxsbBYLBg6dChycnIAAAMHDkRMTAwsFgssFguWLFmiV7PIwzFXpAXmirTAXPkWXV/TkJKSgmHDhsFkMuG5555DSkoKNm7cCABITU3F8OHD9WwOeQnmirTAXJEWmCvfodsIVkhICK6//nqYTLb3diQlJeHw4cN6XZ68FHNFWmCuSAvMlW8x7B6s1NRUjBgxQv48a9YsxMfHY+zYsQwcNRpzRVpgrkgLzJV3M6SDNX/+fGRnZ2PevHkAgFWrVmH//v3YvXs3+vXrxyFSahTmirTAXJEWmCvvZxJCCD0vuGjRIrzxxhv45JNPEBER4XCfkJAQHDt2DJGRkedtz8rK4hIBTeDNS08wV8ZhrpgrLTBXzJUW9MyVrje5L168GBkZGeeFqrKyEqdPn0ZUVBQAIDMzE1FRUTVCRVQb5oq04Im5Su08WdZTV30LADBdea/DfUWq2r72lbGyPvh7MwDAv3PTtWiiz/PEXFHj6NbBys/Px4MPPoiYmBgMGjQIABAcHIzPPvsMN9xwA8rLy+Hn54eWLVti/fr1ejWLPBxzRVpgrkgLzJVv0a2D1a5dO9Q2G7l9+3a9mkFehrkiLTBXpAXmyrfoOkXo7U5NjZF1xLWFso69sy8A4HDRR7q3iVxvbMTdsr6ry88AgIHfjJLbqp6ZKusJT94p6zVnntehdeStAgPUdNGDbW6R9dz3s2Rt7RKrDgjoAQAQotLh+Uwzlsp6zL1qn/KyAtt5N7WQ27qPv1bWPxS/1dCmE+GKZjcBAL7Z7y+3PZDYStYvnlyme5u0xqVyiIiIiFyMI1guJITqr4rRz8r64X9+BQCYekD3JpGL2I9arZzziqz9H1xeY1/7bSuhRrPW/F2jxpHXCg1qK+t9I9vIum1GnKxNJousrUffU9s/cDJibvfXa7+OaiQh+No5AICqEfPltqxNabLuet0QWR8p2lT3NYjOmdimJQAguG2i3NahmRp9xUm9W6Q9jmARERERuRg7WEREREQuptsUYVlZGcaNG4d9+/YhLCwMbdq0QXp6OsxmMwoLCzFhwgQcOnQIwcHBSE9PR9++ffVqWpM83E5NHYX912xcQ3yUXrlyNi1YG/t9Kx90vI/9TfGOfLl2WJ2fD/72nXq3h+rHXb6vrg1Wy6i0zejmcJ9Td6TK+q63VTs2FDm+ud2RtuEdZf187BcAgKHfDlA79Jwuyx/feUrWkcPVTfW/l/1Y7+v5KnfJlV7CgtrJ+oEFrwMAxCdqGntuXpHubdKTriNYKSkpOHjwIHbu3Inhw4cjJSUFADB79mwkJSUhOzsbK1asQHJyMior6//lQL6NuSItMFekBebq/9u7+/CoqjsP4N9LQgIJCjYLEcQ4G0iINoGJWUKayos+0NKnSKkhsm1AYkGw0EZW1FUXH1xX3lbEFRoLRcQSMA0QW7SKgH1ASuti1vCSyMtGBRKEvIAgSwghmTn7x5Bzb2QmySRzX+bO9/M8PPxy58ycI3yf8XDPvfeEDsMmWG3tIr5582bMnev5F9Lw4cMRGxuLffv2GTU0CmLMFemBuSI9MFehxbS7CFt2ET9//jzcbjf69lXvYnE4HKisrDRraH7p0U19aFxEBLc1MJteudIu043xsdTXWe0tObbXXzMmeT2+J0NdOuQyYtdY7fvqyvylsh5QUKV55bNOfd6Zy+r/yB85PtxzbIv6JHFX9suyFmP+TdbR4W/K+lKneg5tVstVoD074Meydj/geS7byiHqA1VdrrWGj8lIpkywWnYRX716NRoaGuRsvoXB+0+TTTBXpAfmivTAXNmf4XcRLl++HG+//Ta2b9+OqKgouZllXZ36EIxTp04hLi7O6KFREGOuSA/MFemBuQoNhp7B8raLOABkZ2cjPz8fzz//PEpKSlBdXR30d0+IlY/K+l++vLWNltRVRuSq1RKbotYfjrhxeW5k9nZZ+7oDUNvGn7sS/aHtA/t16cLWrPZ9pSjq13X0igXqCyvbvgvVX7X1JQCApc+qn/vUg2rf3cIiA9pfqLFarvQ0MfG45ifPEuHmM1fNGYwJDJtg+dpFfP/+/Vi2bBmmTZuGhIQEREREoKCgAOHhfMg8tY+5Ij0wV6QH5iq0GPa319Yu4rGxsdi5c6dRQyEbYa5ID8wV6YG5Ci2cHuuk+VKUrK9cO23iSEhPXu/Oa7Uc5+PuPW2bJ3Lb7KNxubok489yYuvlSd5FGIxOuNVrctwH1b97ZdgMWe9If0DWP/zk7S73eV+U57PzSnvKY0Koz2Nyu7rcBYWIQasizB6CqbhVDhEREVGAcYJFREREFGBcIuyEsLCbZP3c4vWyFvgnM4ZDNqG9I3HMf3t/eGh7tPsajt3f2OUxkbk+q1eX/CaOfkTWf176mKzP1v+8y/20LAsCwKTbPL9H3pTgta1Yofb9dWMfr20odP0wepasIwany9r96hwAwP80Kje8x654BouIiIgowAw7g5WXl4d33nkHp06dQllZGZKTkwF4tgPo0aMHevToAQB45plnMGXKFKOG1Sku1//J+j+eVZ93teBnZowmtNkhVy1nrjp71qr1ljg8axUIVszVznp1W5GIX2tf2eC1fcuZ9v49U72+/ug/3Clr7QXtvs5ctVAef1XWdXULZT35NfUMm3aspLJirgLt6aHVsr527bysC9d6zrS63W8aPSTTGDbBmjx5Mp566imvD07bunWrDBqRP5gr0gNzRXpgrkKLYROsUaNGGdUVhRDmivTAXJEemKvQYomL3HNycuB2uzFixAgsWbKk1Y7iVtQzYoCsn/uvQllrL3J31ffUvIP7zJshWHK19nh/AMCYTr7f67O4SDdWzpX2mVjO734GAOjzO+9LTdqtd7TPufJH5OLnZP3nOPXi99ZLmNQRVs6VPzJf/lz94YBazzx6xITRmMv0i9z37t2LQ4cOobS0FDExMZg+fbrZQyIbYK5ID8wV6YG5sifTz2C17BbevXt3zJs3D4mJiSaPiOyAuSI9MFekB+bKnkydYNXX16OpqUnuKF5YWIjUVO93vFjBndGeu7zKCnbLY+InL3tt+6t1WZqfXtNzWPQtwZaroou/BQA8knFWHvPnjsJm8aastc/BeujFX9zQB3VeMORq1O++knVY8uNttu0Wpm7BJHa9IOvDS+/01lxKnrBX1kreKvWFX+bLsvy9TZ62H6htybtgyBV1jmETrLlz52Lbtm2orq7G2LFj0atXL+zcuRNZWVlwuVwQQiA+Ph4bNni/5ZjIG+aK9MBckR6Yq9Bi2AQrPz8f+fn5Nxw/cOCAUUMgG2KuSA/MFemBuQotpl+DFUzmx3lO4fpaFtT6bd5GWf9+YRsNia5rdTegotbaJcD2hM1fI+tN89XjRQqXCENB81snZd1tkefOwMZvjspjjU/vkPUf96nPYnriZLSsL139sM0+Iv96q6wv/kD9nus25J9lnfDCN57igw4OnILar2+dI+uw76nb46A4tG8nNf0uQiIiIiK74QSLiIiIKMC4ROiHWcc8d8ZMXbZFHgv711e9ti3dpd0K4X09h0U2F67kyrpl30IAGJm9HUDrZUFftMuM2s8jexm46hZZZ752CABwxd0kj310pULTWlt3XGOTuteccu32Tn0G2cuwW67I2u1S90P9w8KfalqF3oX7PINFREREFGCGTbDy8vLgcDigKArKy8vl8YqKCmRmZiIxMRHp6ek4ciT0HqdPncdckR6YK9IDcxVaDFsi9LWL+OzZszFr1izk5uZi69atmDFjBj7++GOjhuUXt/CcahfNPby+XvXgalmP+tshQ8YU6uyQK3+0utNwv+e3KS/ul4c2LHhD1r6WDhuXqw+YjHyi0WubUBesufqm4Zist+NYGy390yvyH2X9dP8fyPqaQ91zNVLTfl+e43rF70GtYM1Ve6Y+tFnz0/dktetsL+MHYyGGncEaNWoUBg4c2OpYbW0tSktLMXXqVABAVlYWTpw4gZMnTxo1LApyzBXpgbkiPTBXocXUi9yrqqowYMAAhId7hqEoCuLi4lBZWQmHw2Hm0Dql76okWSt/Oq6+4Gry0pr0YtVcTenzS1k/MuTsDa+3Ojvlh1bb4Lyo9qF9DpaW9sxWy9kvbqXTPqvmSi/as1YX156Wtfvnd8taiGZZK/tXyvpXZW1vt0MqO+eq4OvQ3ibO9IvcFUVp9bMQwqSRkJ0wV6QH5or0wFzZk6kTrNtvvx2nT59Gc7PnX0FCCFRVVcmdxYk6g7kiPTBXpAfmyr5MXSLs168fUlNTsXHjRuTm5qK4uBgOhyNoT4v2iB0jawXvmjeQEGfVXPm6AH1PRueWBslYVs1VoMVGjwAAfLX+M3nM/cBvZN0tTL2cXex6QdaDH1AvkThVz0x3VLDmqnt4jKzD+38j68aFz5gxHEsy7AzW3LlzMXDgQJw+fRpjx47F4MGDAQBr1qzBmjVrkJiYiKVLl2LdunVGDYlsgLkiPTBXpAfmKrQYdgbL1y7iQ4YMCarbUclamCvSA3NFemCuQgu3ygmga5+rzwIRcJs4ErIK7Z2DYfNHeG2z9nj/gPWnXYYEvPdH1rQyYYasZxUdBgA0FpyRx3q/8lWnPle7lDP/1gdlPS1FfdDloEVXZe2K7u35PX6Z+iGauwUvzH5J1iPeSpH1qcsfdGp8FJzSIn4sa/fU78u64kd/0LTabeCIrMf0uwiJiIiI7IYTLCIiIqIA4xJhFynd1D/CJWMHydrl2mnGcMhitA/w3ORjyU4u62keEtqRB3+2LD92ZHscrZxb1K11+IBR6/hO5DVZKym5AIDo7z8tj014fbZfn/fGg57lme+MPiGPuX6WrPahOGWtfWCo+1yJ5/XVj8lj76zLlvXkg59retHWFEpeH3VS85O6RLjtf4dojnOJkIiIiIgCyDITLIfDgaSkJDidTjidThQVFZk9JLIB5or0wFyRHpgre7HUEuHWrVuRnJzcfkOTXT3XR9bh7uY2WpIVWCVX2qW5TRfU5cKWZT3t3oG+lhO9a7+t62Xt8tIv/Phs8iXQuVpyol7Wk05fv8TgJ0vlseKvvb9PUdSvce1SHzAUAODy0Z/78OuyvrZJ3Wtw9hs/BQAUXtB+VqHvgVNAWeX7ypewsJsAAEMe/0Iec/3n47Jeera34WOyKsucwSIiIiKyC0tNsHJycpCSkoKZM2eirq7O7OGQTTBXpAfmivTAXNmHIiyybXdlZSXi4uLQ1NSEBQsWoKysDO+//36rNqWlpUhLSwMQBkDx+jlGCA+7RdYN19SH7v2741NZv1hlvbuzhGgyewiGC4ZcNYs3dflc7bJg5BONuvQBMFd65OrVhJkAgEd//3f14PC5Xttqlwj3ZKoP+6y50qvNPqaVF/g1JqMxV9b8vsqIeggAsPeSeudgbt9Dsn7rwmuGj8kfRubKMtdgtewc3r17d8ybNw+JiYkmj4jsgLkiPTBXpAfmyl4sMcGqr69HU1MT+vTxXDxeWFiI1NRUk0dFwS5YchWu5AJova2O9iJ4Le1Zqb9u+dENr2u33Sm6qN9Zq1BmRK4eq/BcgP5Ypvao9zNYZA/B8n21bYrnZp0PM67IY5sv/cWs4ViaJSZYNTU1yMrKgsvlghAC8fHx2LBhg9nDoiDHXJEemCvSA3NlP5aYYMXHx+PAgQNmD4NshrkiPTBXpAfmyn4sMcEKNs2uC7LuHjbTxJGQnWi3rSlSOnKTxJ/0GwwRkRex649fr4632Y4s9pgGIiIiIjvgBIuIiIgowDjBIiIiIgowTrCIiIiIAowTLCIiIqIA4wSLiIiIKMCCaoLV0NBwvRL81YlfpaWluHJFffoueTBXzJUemCvmSg/MVfDkKqieg3Xy5MnrldvMYQSttLQ0fPrpp7j77rvNHoqlMFddw1x5x1x1DXPlHXPVNUbmShFCCN17CZBz585hx44dcDgc6Nmzp9nDCUpJSUmIiooyexiWwlx1HXN1I+aq65irGzFXXWdUroJqgkVEREQUDILqGiwiIiKiYMAJFhEREVGA2XqC5XA4kJSUBKfTCafTiaKiIq/t8vLy4HA4oCgKysvL5fGKigpkZmYiMTER6enpOHLkSIfe116/V69exaRJk5CYmAin04nx48fLCxdra2sxfvx4JCQkIDk5Gfv27evQ+8aMGYP4+HjZ5yuvvNKVPzpqA3NFemCuSA/MlYmEjd1xxx2irKys3XYfffSRqKqquqH9vffeK9avXy+EEGLLli0iIyOjQ+9rr9+Ghgbx3nvvCbfbLYQQYtWqVWLcuHFCCCEefvhhsXDhQiGEEJ988omIi4sTTU1N7b5v9OjR4t133233v5W6jrkiPTBXpAfmyjycYPloX1NTI3r37i3/Ut1ut4iNjRUnTpxotx9/+y0pKRGDBg0SQggRHR0tamtr5WvDhw8Xu3fvbvd9VguWnTFXpAfmivTAXJnH1kuEAJCTk4OUlBTMnDkTdXV1HX5fVVUVBgwYgPBwz6PCFEVBXFwcKisrA97vypUrcf/99+P8+fNwu93o27evfM3hcPjss+V9LZ588kmkpKRgypQp+PLLLzs0Tuoc5or0wFyRHpgrc9h6grV3714cOnQIpaWliImJwfTp0/16v6IorX4WHXyihT/9Ll68GBUVFVi0aJFffX77fQUFBTh69CgOHz6MkSNHYsKECR0aK/mPuSI9MFekB+bKROadPDPWmTNnRK9evdps8+1TozfffHOnTo12tN+XXnpJpKWliQsXLshjUVFR7Z4a9fa+b4uMjBTnzp3z+ToFBnNFemCuSA/MlbFsewarvr4eFy9elD8XFhYiNTW1w+/v168fUlNTsXHjRgBAcXExHA4HHA5HQPpdsWIFCgsLsWvXLvTp00cez87ORn5+PgCgpKQE1dXVuOeee9p8X3NzM2pqamSb4uJixMbGIiYmpsP/vdQxzBVzpQfmirnSA3Nlcq5Mm9rp7IsvvhBOp1OkpKSI5ORkMXHiRK+zbiGEmDNnjrjttttEWFiYiI2NlRfMHTt2TGRkZIiEhASRlpYmysvL231fR/qtqqoSAER8fLwYNmyYGDZsmEhPTxdCCFFdXS3GjRsnBg8eLO666y6xZ8+edt93+fJlkZaWJpKTk8XQoUPFfffdJw4ePBi4P0ySmCvmSg/MFXOlB+bK3FxxqxwiIiKiALPtEiERERGRWTjBIiIiIgowF0ieGQAAAA5JREFUTrCIiIiIAuz/AVoycvvKF+hUAAAAAElFTkSuQmCC" }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# we can use an array comprehension to plot the first sixteen rows\n", "# the Plots library automatically resizes everything for you and adds the axes\n", "plot([heatmap(rotl90(reshape([train[i,col] for col in 2:785], 28, 28)), aspect_ratio=:equal) for i=1:16]...)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Stats" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have confirmed that the data really are characters that are arranged as described in the data description, we will take a brief eyeball on the data just to make sure there is nothing strange." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "\"42000×785 DataFrame\"" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# note that summary in Julia just gives basic information as it is supposed to work on ANY datatype\n", "summary(train)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "

785 rows × 8 columns

variablemeanminmedianmaxnuniquenmissingeltype
SymbolFloat64Int64Float64Int64NothingNothingDataType
1label4.4566404.09Int64
2pixel00.000.00Int64
3pixel10.000.00Int64
4pixel20.000.00Int64
5pixel30.000.00Int64
6pixel40.000.00Int64
7pixel50.000.00Int64
8pixel60.000.00Int64
9pixel70.000.00Int64
10pixel80.000.00Int64
11pixel90.000.00Int64
12pixel100.000.00Int64
13pixel110.000.00Int64
14pixel120.00300.0116Int64
15pixel130.011190500.0254Int64
16pixel140.0051428600.0216Int64
17pixel150.00021428600.09Int64
18pixel160.000.00Int64
19pixel170.000.00Int64
20pixel180.000.00Int64
21pixel190.000.00Int64
22pixel200.000.00Int64
23pixel210.000.00Int64
24pixel220.000.00Int64
25pixel230.000.00Int64
26pixel240.000.00Int64
27pixel250.000.00Int64
28pixel260.000.00Int64
29pixel270.000.00Int64
30pixel280.000.00Int64
" ], "text/latex": [ "\\begin{tabular}{r|cccccccc}\n", "\t& variable & mean & min & median & max & nunique & nmissing & eltype\\\\\n", "\t\\hline\n", "\t& Symbol & Float64 & Int64 & Float64 & Int64 & Nothing & Nothing & DataType\\\\\n", "\t\\hline\n", "\t1 & label & 4.45664 & 0 & 4.0 & 9 & & & Int64 \\\\\n", "\t2 & pixel0 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t3 & pixel1 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t4 & pixel2 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t5 & pixel3 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t6 & pixel4 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t7 & pixel5 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t8 & pixel6 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t9 & pixel7 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t10 & pixel8 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t11 & pixel9 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t12 & pixel10 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t13 & pixel11 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t14 & pixel12 & 0.003 & 0 & 0.0 & 116 & & & Int64 \\\\\n", "\t15 & pixel13 & 0.0111905 & 0 & 0.0 & 254 & & & Int64 \\\\\n", "\t16 & pixel14 & 0.00514286 & 0 & 0.0 & 216 & & & Int64 \\\\\n", "\t17 & pixel15 & 0.000214286 & 0 & 0.0 & 9 & & & Int64 \\\\\n", "\t18 & pixel16 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t19 & pixel17 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t20 & pixel18 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t21 & pixel19 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t22 & pixel20 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t23 & pixel21 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t24 & pixel22 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t25 & pixel23 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t26 & pixel24 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t27 & pixel25 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t28 & pixel26 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t29 & pixel27 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t30 & pixel28 & 0.0 & 0 & 0.0 & 0 & & & Int64 \\\\\n", "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n", "\\end{tabular}\n" ], "text/plain": [ "785×8 DataFrame. Omitted printing of 1 columns\n", "│ Row │ variable │ mean │ min │ median │ max │ nunique │ nmissing │\n", "│ │ \u001b[90mSymbol\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mFloat64\u001b[39m │ \u001b[90mInt64\u001b[39m │ \u001b[90mNothing\u001b[39m │ \u001b[90mNothing\u001b[39m │\n", "├─────┼──────────┼────────────┼───────┼─────────┼───────┼─────────┼──────────┤\n", "│ 1 │ label │ 4.45664 │ 0 │ 4.0 │ 9 │ │ │\n", "│ 2 │ pixel0 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 3 │ pixel1 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 4 │ pixel2 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 5 │ pixel3 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 6 │ pixel4 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 7 │ pixel5 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 8 │ pixel6 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 9 │ pixel7 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 10 │ pixel8 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "⋮\n", "│ 775 │ pixel773 │ 0.340214 │ 0 │ 0.0 │ 255 │ │ │\n", "│ 776 │ pixel774 │ 0.219286 │ 0 │ 0.0 │ 254 │ │ │\n", "│ 777 │ pixel775 │ 0.117095 │ 0 │ 0.0 │ 254 │ │ │\n", "│ 778 │ pixel776 │ 0.0590238 │ 0 │ 0.0 │ 253 │ │ │\n", "│ 779 │ pixel777 │ 0.0201905 │ 0 │ 0.0 │ 253 │ │ │\n", "│ 780 │ pixel778 │ 0.0172381 │ 0 │ 0.0 │ 254 │ │ │\n", "│ 781 │ pixel779 │ 0.00285714 │ 0 │ 0.0 │ 62 │ │ │\n", "│ 782 │ pixel780 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 783 │ pixel781 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 784 │ pixel782 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │\n", "│ 785 │ pixel783 │ 0.0 │ 0 │ 0.0 │ 0 │ │ │" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# note that summary in Julia just gives basic information as it is supposed to work on ANY datatype\n", "# those who are used to R may be looking for the describe() function which is for DataFrames\n", "describe(train)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Summary Stats:\n", "Length: 42000\n", "Missing Count: 0\n", "Mean: 97.163476\n", "Minimum: 0.000000\n", "1st Quartile: 0.000000\n", "Median: 35.000000\n", "3rd Quartile: 231.000000\n", "Maximum: 255.000000\n", "Type: Int64\n" ] } ], "source": [ "# the describe() function gives basic stats\n", "# you could describe(train) to do the entire DataFrame, but that would take a lot of space here\n", "describe(train[:,406])" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Summary Stats:\n", "Length: 42000\n", "Missing Count: 0\n", "Mean: 97.163476\n", "Minimum: 0.000000\n", "1st Quartile: 0.000000\n", "Median: 35.000000\n", "3rd Quartile: 231.000000\n", "Maximum: 255.000000\n", "Type: Int64\n" ] } ], "source": [ "# can also specify a column in a DataFrame with a Symbol\n", "# Symbols in Julia have a colon before them and are reserved when created (:pixel404)\n", "# I just just picked 404 because it is near the center for 28x28cv\n", "describe(train[!, :pixel404])" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X901fV9+PFXLBhEIAyCHiCGDJDGKW0Ay9B2Aoo141AODn/gsa6siFXocU6OsNPvdJ1HPGfntE47648zZbCqOazqrLOjjmDBunX+StExgcI0BjhBQWjotUKh3O8ffLlfYyAk8M69l/h4nMORTz7387nvvHIhTz75JJZks9lsAACQzCmFXgAAQHcjsAAAEutWgfWb3/wmGhoa4je/+U2hlwIAfIp1q8DasGFDjBs3LjZs2JD3525pacn7c37amXn+mXn+mXn+mXlhdLe5d6vAKqTf/e53hV7Cp46Z55+Z55+Z55+ZF0Z3m7vAAgBITGABACQmsAAAEmsVWHv37o0ZM2bEqFGjoqamJmpra6OxsfGIBz766KNx9tlnx4gRI+KGG26IAwcOdOqJr7jiihgyZEiUlJREJpNpte/ll1+OmpqaGDVqVFxyySXR3Nyc2/fGG2/E2LFjY/Xq1Z16PgCAfGlzBeuGG26IjRs3xtq1a2PatGlxww03tDnonXfeidtvvz1eeuml2Lx5c2zfvj0effTR3P633367TXBt3ry51Q1sN954Y6xdu7bNubPZbFx77bVx7733xi9/+cv44z/+47j11lsjImLdunWxatWqmD59+vG/xwAAXaxVYPXq1SumTp0aJSUlERExYcKEePvtt9sc9OSTT8bll18eZ555ZpSUlMSNN94YdXV1uf2PPvpoXHHFFfHb3/42Ig5dkZoyZUps2rQp95gpU6bEGWec0ebcr732WpSWlsakSZMiIuIb3/hGPPPMM7F///5YsmRJNDU1xU9+8pN48MEHT/y9BwDoAu3eg/W9730vvvKVr7R5e1NTUwwbNiy3XVVVFU1NTbntxYsXx4gRI2LGjBmxcuXKmDVrVjz99NNRXV19zAV98tx9+/aNvn37RnNzc9xzzz1x7733Rm1tbdx0001HPUcmk4k9e/bkfu3bt++YzwsAkEqPo+24++67Y9OmTfHQQw8dcf/hq1wRh76s90nf/e5345prronLLrss/vM//zPGjh3b4UV9/NxHOv+3v/3tdo+fOHFiq+2FCxfGokWLOvz8x2P37t1den7aMvP8M/P8M/P8M/PCONnnPmDAgFbbRwys73znO/H0009HfX199O7du83+ysrKVje/v/vuu1FZWdnqMT/+8Y/jtddei1tuuSUWLFgQK1asiH79+h1zgZ88969//ev49a9/HYMHDz7msYetWbMmampqctulpaVRWlra4eOPx/bt24/6DQGFUl5e3ubj0t188gVN1zPz/DPz/DPzwuhOc28TWPfcc0/U1dVFfX199O/f/4gHzZw5M770pS/FHXfcEWeccUY89NBDMWvWrNz+f/mXf4m//Mu/jH//93+P3//934977rknpkyZEj/5yU+OObxx48bF3r17Y/Xq1TFp0qR4+OGHY8aMGdGzZ88Ov1N9+vTpUMyl0tzcHOeee27enq+jep3WOzZuWN/tIwsAik2rwNq6dWssWLAghg8fHpMnT46IQ1d/Xn755bj++utj+vTpMX369Bg+fHj8zd/8TXzxi1+MgwcPxsUXXxxz5szJnee3v/1t1NfXx1lnnRUREbfeemsMGjSo1b1Q06dPj4aGhoiI+OxnPxtnn312rF69Ok455ZR47LHH4sYbb4yPPvoohg4dGo899liXD+JE5H6MxHUPRVSOKexiDtu+PvY+Ojt27twpsAAgz1oFVkVFxRHvp4qIeOSRR1ptz507N+bOnXvEx1599dVt3nbddde12n722WePuqgLLrgg3njjjaPuL1qVYyKGFUlgAQAF4ye5AwAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAk1iqwbr755qiqqoqSkpJYt27dEQ9YunRp9O/fP2pqaqKmpiYmT57cqSfMZDJx2WWXRXl5eZSXl7fZ/9xzz0V1dXWMHDkyZs6cGZlMJrdv1apVMWrUqGhsbOzUcwIA5FOrwLriiivipZdeimHDhrV70JQpU2Lt2rWxdu3a+OlPf9pq38aNG9s8fsOGDbnf9+zZMxYuXBj19fVtHpfJZGLOnDnxzDPPxObNm2Pw4MGxePHiiIhYvXp1bNu2LS688MKOv3cAAAXQKrAuuuiiqKioOKET3nnnnTF//vzIZrMREfHss8/GtGnTYseOHRERUVpaGpdcckn079+/zbErVqyI888/P6qrqyMiYt68eVFXVxcRh66cNTQ0xCuvvBIPPvjgCa0RAKArHdc9WGvWrImampr44he/GE8++WSrfcuWLYudO3fG3LlzY/ny5XHbbbfFypUrY9CgQcc8b1NTU6urZ1VVVbFt27Y4ePBgLF26NO69994YP3583HTTTe2eJ5PJxJ49e3K/9u3bdzzvJgDAcenR2QOmTZsWV111VfTu3TvWr18fX/7yl6OioiImTJhw6IQ9esQTTzwREydOjMcffzzWr18fVVVVHT5/SUlJu/uXLl16zHNMnDix1fbChQtj0aJFHV5DZ7W0tHTZuU9US0tL7Nq1q9DL6BK7d+8u9BI+dcw8/8w8/8y8ME72uQ8YMKDVdqcD6+M3pp9zzjkxderU+I//+I9cYEVELFmyJDKZTFxzzTWxYMGCqKuri1NPPfWY566srIwXXnght93Y2BhDhw6NU07p3IW2w1fYDistLY3S0tJOnaMzysrKuuzcJ6qsrKzNB7076c7vW7Ey8/wz8/wz88LoTnPv9JcIt23blvv9e++9Fy+88EKMGTMm97b7778/Hn744Vi1alUsWbIkKioq4vLLL4+9e/ce89y1tbXx6quv5m6Kf+CBB2LWrFmdXWL06dMn+vXrl/vVlXEFAPBJrQJr/vz5UVFREVu3bo0pU6bEyJEjIyJi6tSp8dprr0VExPe///0499xzo6amJi699NL4i7/4i7j44otz5+jVq1fU19fHwIEDIyLivvvui2nTpsX+/ftzjxk7dmxccMEFsXv37qioqIjrrrsuIiL69u0bjzzySMyYMSNGjhwZ27Zti29961tdOwEAgMRKsoe/3a8baGhoiHHjxsXrr78eY8eOzfvzxv95OWLYmGMfkA/v/iJi8R/mfRb5tGvXrm51OflkYOb5Z+b5Z+aF0d3m7ie5AwAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAk1iqwbr755qiqqoqSkpJYt27dUQ+66667YsSIETFixIi4/fbbO/WEmUwmLrvssigvL4/y8vI2+5977rmorq6OkSNHxsyZMyOTyeT2rVq1KkaNGhWNjY2dek4AgHxqFVhXXHFFvPTSSzFs2LCjHvDiiy9GXV1dvPnmm/HWW2/FihUr4vnnn8/t37hxY5tjNmzYkPt9z549Y+HChVFfX9/mcZlMJubMmRPPPPNMbN68OQYPHhyLFy+OiIjVq1fHtm3b4sILL+z8ewkAkEetAuuiiy6KioqKdg9Yvnx5zJ49O04//fQoLS2Nr3/961FXV5fbf+edd8b8+fMjm81GRMSzzz4b06ZNix07dkRERGlpaVxyySXRv3//NudesWJFnH/++VFdXR0REfPmzcude+nSpdHQ0BCvvPJKPPjggyfwLgMAdK1O34PV1NTU6gpXVVVVNDU15baXLVsWO3fujLlz58by5cvjtttui5UrV8agQYOO69zbtm2LgwcPxtKlS+Pee++N8ePHx0033dTueTKZTOzZsyf3a9++fZ19NwEAjluP4zmopKQk9/vDV6pyJ+zRI5544omYOHFiPP7447F+/fqoqqo6rnMfydKlS495jokTJ7baXrhwYSxatKjDa+islpaWLjv3iWppaYldu3YVehldYvfu3YVewqeOmeefmeefmRfGyT73AQMGtNrudGBVVla2usn83XffjcrKylaPWbJkSWQymbjmmmtiwYIFUVdXF6eeemqHzv3CCy/kthsbG2Po0KFxyimdu9C2Zs2aqKmpyW2XlpZGaWlpp87RGWVlZV127hNVVlbW5oPenXTn961YmXn+mXn+mXlhdKe5dzqwrrzyyvjmN78Z8+bNix49esSSJUvirrvuyu2///77Y+nSpbFq1aoYOHBg/Pmf/3lcfvnl8dRTT0WvXr3aPXdtbW3Mnz8/NmzYENXV1fHAAw/ErFmzOv1O9enTJ/r169fp4wCAjmtubo7m5uYk52ppaUlywaK8vLzNhZ9CaBVY8+fPjx/96Eexffv2mDJlSvTp0yc2b94cU6dOjTvvvDPOP//8mDRpUlx11VUxevToiIiYNWtW1NbW5s7Rq1evqK+vz93Eft9998WDDz4Y+/fvzwXW2LFjo7m5OXbv3h0VFRUxefLk+MEPfhB9+/aNRx55JGbMmBEHDhyI0aNHx7Jly/I1CwCgg5qbm2PIkCGFXkYbvU7rHRs3rC94ZJVkP3kT1UmsoaEhxo0bF6+//nqMHTs2788b/+fliGFj8va87Xr3FxGL/zDvs8inXbt2davLyScDM88/M88/M++Y3Oe+6x6KqCySz33b10c8OrsoPvcd103uAAARcSiuiuXiQhHxv8oBAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQWJvAqqqqiurq6qipqYmamppYvnz5EQ+86667YsSIETFixIi4/fbbO/WkmUwmLrvssigvL4/y8vI2+5977rmorq6OkSNHxsyZMyOTyURERH19fdx8883xJ3/yJ/HP//zPnXpOAIB8OeIVrCeffDLWrl0ba9eujauvvrrN/hdffDHq6urizTffjLfeeitWrFgRzz//fG7/xo0b2xyzYcOG3O979uwZCxcujPr6+jaPy2QyMWfOnHjmmWdi8+bNMXjw4Fi8eHFEREyZMiWqqqpi69at0atXr86/twAAeXBcXyJcvnx5zJ49O04//fQoLS2Nr3/961FXV5fbf+edd8b8+fMjm81GRMSzzz4b06ZNix07dkRERGlpaVxyySXRv3//NudesWJFnH/++VFdXR0REfPmzWt17ltvvTUeeOCBWLFixfEsHQCgyx0xsK699toYPXp0XH/99bko+rimpqYYNmxYbruqqiqamppy28uWLYudO3fG3LlzY/ny5XHbbbfFypUrY9CgQcdc0JHOvW3btjh48GD8wz/8Q8ybNy8WLVoUEyZMOOo5MplM7NmzJ/dr3759x3xeAIBUenzyDS+++GJUVlbG/v3746/+6q/ia1/7Wvzbv/1bmwNLSkpyvz98pSp30h494oknnoiJEyfG448/HuvXr4+qqqoOL+rj5/64uXPnduj4iRMnttpeuHBhLFq0qMPP31ktLS1ddu4T1dLSErt27Sr0MrrE7t27C72ETx0zzz8zzz8z7xif+1obMGBAq+02gVVZWRkRh+6TuuWWW2LUqFFtTlJZWRmNjY257XfffTd33GFLliyJTCYT11xzTSxYsCDq6uri1FNPPeYCKysr44UXXshtNzY2xtChQ+OUUzr+1cw1a9ZETU1Nbru0tDRKS0s7fHxnlZWVddm5T1RZWVmbD3p30p3ft2Jl5vln5vln5sfmc1/7WlXLhx9+GL/61a9y23V1dTFmzJg2B1155ZWxbNmy+PDDD2Pfvn2xZMmSmDVrVm7//fffHw8//HCsWrUqlixZEhUVFXH55ZfH3r17j7mg2traePXVV3M3xT/wwAOtzt0Rffr0iX79+uV+dWVcAQB8UqvAeu+992Ly5Mnxuc99LkaPHh1r1qyJf/qnf4qIiKlTp8Zrr70WERGTJk2Kq666KkaPHh3nnHNOfPnLX47a2trceXr16hX19fUxcODAiIi47777Ytq0abF///7cY8aOHRsXXHBB7N69OyoqKuK6666LiIi+ffvGI488EjNmzIiRI0fGtm3b4lvf+lbXTgEAIKFWXyIcPnx4/OIXvzjiAz95H9Ydd9wRd9xxxxEfe/3117d520033dRqu6Gh4aiLmj59ekyfPv2o+wEAipmf5A4AkJjAAgBITGABACQmsAAAEhNYAACJCSwAgMQEFgBAYgILACAxgQUAkJjAAgBITGABACQmsAAAEhNYAACJCSwAgMQEFgBAYgILACAxgQUAkJjAAgBITGABACQmsAAAEhNYAACJCSwAgMQEFgBAYgILACAxgQUAkJjAAgBITGABACQmsAAAEhNYAACJCSwAgMQEFgBAYgILACAxgQUAkJjAAgBITGABACQmsAAAEhNYAACJCSwAgMQEFgBAYgILACAxgQUAkJjAAgBITGABACQmsAAAEhNYAACJCSwAgMQEFgBAYgILACAxgQUAkJjAAgBITGABACQmsAAAEhNYAACJCSwAgMQEFgBAYgILACAxgQUAkJjAAgBIrEehFwAAHFtzc3M0NzcXehk569evL/QSiprA6uaK8Q9AeXl5VFZWFnoZACeN5ubmGDJkSKGXQScIrO7q1zsjIuKrX/1qgRfSVq/TesfGDetFFkAH5a5cXfdQROWYwi7msHUrIn707UKvomgJrO4qcyiwiuoPY0TE9vWx99HZ8bOf/SzOOeecEzpVS0tLlJWVJVmWq2rASaFyTMSwIvk7vXlDoVdQ1ARWd1dMfxgjivbKmqtqwGHbt2+PxsbGQi+jlWK83YP2CSzyqxivrP2/q2o7d+4UWPAp19zcHOeee26hl0E3ILAojGK7shbF9y9EX7bk06BovzOumP4RGOF+p5OQwAJftjzpFdsn6QiB3BFF/Z1xxfaPQPc7nXQEFhTxly1TfDNASrt3747f+73fK/QyWtmxY0fU1tYWehltlPY6LZ568ocxePDgEzpPym/m+N3vfhef+cxnkpwrhaK8WuRKEYkILDismP7FWqRX1YpaMX2Sfvvnsa/ulpg2bVqhV3JyKKY/e64UkYjAgmJUjFfVDv/LvpjWFPH/11WMn6SLaVbF+PFztYhuTGBBMSvGaCimNUUU9xWHYppVMX78ivljByfI/+wZACAxgQUAkJjAAgBITGABACQmsAAAEhNYAACJCSwAgMQEFgBAYgILACAxgQUAkNhJE1ibNm2KCy+8MEaNGhXjx4+Pt956q9BLAgA4opMmsL7xjW/EDTfcEL/85S9j4cKFMWfOnEIvCQDgiE6KwHr//fejoaEhvvrVr0ZExMyZM+Odd96JxsbGwi4MAOAIehR6AR2xZcuWGDJkSPTocWi5JSUlUVlZGU1NTVFVVdXm8ZlMJvbs2ZPbLi0tjdLS0q5f6Pb1Xf8cHfXBO4f+W0xriijOdVlTxxTjmiKKc13W1DHW1HHFuK5iXFMRraUkm81mC72IY3n99dfjT//0T+N//ud/cm/7whe+EN/97nfjoosuyr2toaEhxo0b1+b4hQsXxqJFi7psfVu3bo3xfzgh9u39qMue47iUnBKRPVjoVbRVjOuypo4pxjVFFOe6rKljrKnjinFdRbim0tN6xyv/9fOoqKjI6/MOGDCg1fZJcQXrrLPOiq1bt8aBAweiR48ekc1mY8uWLVFZWXnEx69ZsyZqampy2119BWvAgAHxysv/FQcOHOiy5zge+/bty8+Vu05Kta6WlpYoKytLsKLinFUxrmnHjh0xaNCgQi+jjWKcldd5xxTjmrzOOy7lmlK91svLy4/aB/l0UgTWGWecEWPGjInHHnssZs+eHU899VRUVVUd8cuDERF9+vSJfv365XWNFRUVbeqVrrVr1y4zzzMzzz8zzz8zL4zuNveTIrAiIh5++OGYPXt23H333dGvX79YtmxZoZcEAHBEJ01gffazn42f//znhV4GAMAxnRQ/pgEA4GQisBLYt29f/O3f/m3s27ev0Ev51DDz/DPz/DPz/DPzwuiOcz8pfkxDRx3+MQ2vv/56jB07Nm/Pu2fPnigrK4uWlpa831z/aWXm+Wfm+Wfm+WfmhdEd5+4KFgBAYgILACCxk+a7CDvio48O/ST19evz+6PyM5lMRESsXbs2+vTpk9fn/rQy8/wz8/wz8/wz88LoLnOvrq6O3r17R0Q3uwfr8ccfz/0PoQEA8unj94B3q8DauXNnPP/881FVVRWnnXZaoZcDAHyKdNsrWAAAxcBN7gAAiQksAIDEBNYJ2rRpU1x44YUxatSoGD9+fLz11luFXlK3VFVVFdXV1VFTUxM1NTWxfPnyiDD/lG6++eaoqqqKkpKSWLduXe7t7c3Y/E/M0WZ+tNd7hJmfqL1798aMGTNi1KhRUVNTE7W1tdHY2BgREe+//37U1tbG2WefHeedd1689NJLuePa20f72pv5pEmTYvjw4bnX+t/93d/ljjvpZ57lhEyePDn7j//4j9lsNpv94Q9/mJ0wYUJhF9RNDRs2LPvf//3fbd5u/umsWbMmu2XLljazbm/G5n9ijjbzo73es1kzP1EfffRR9sc//nH24MGD2Ww2m/37v//77KWXXprNZrPZP/uzP8v+9V//dTabzWZfeeWVbGVlZXb//v3H3Ef72pv5xIkTs//6r/96xONO9pkLrBPw3nvvZcvKynIf8IMHD2bPPPPM7DvvvFPYhXVDR/qEY/5d4+Ozbm/G5p9ORwPLzNN79dVXsyNGjMhms9ns6aefnn3//fdz+77whS9kf/rTnx5zH53z8Zm3F1gn+8x9ifAEbNmyJYYMGRI9ehz6ea0lJSVRWVkZTU1NBV5Z93TttdfG6NGj4/rrr48dO3aYfx60N2Pz71qffL1H+DunK3zve9+Lr3zlK/HBBx/EwYMHY9CgQbl9VVVV0dTU1O4+Ou/wzA+77bbbYvTo0XH11VfH22+/HRHRLWYusE5QSUlJq+2sn3rRJV588cV44403oqGhIQYOHBhf+9rXIsL886G9GZt/1zja6z3CzFO6++67Y9OmTbF48eKI8FrPh0/O/Ac/+EGsX78+3nzzzfijP/qjmDZtWu6xJ/vMBdYJOOuss2Lr1q1x4MCBiDj0wd+yZUtUVlYWeGXdz+GZ9uzZM2655Zb42c9+Zv550N6Mzb/rHOn1HuHvnJS+853vxNNPPx0rVqyI3r17x8CBAyMiclcLIyLefffdqKysbHcfHffJmUccek1HHIqpb37zm/H222/HBx980C1mLrBOwBlnnBFjxoyJxx57LCIinnrqqaiqqoqqqqrCLqyb+fDDD+NXv/pVbruuri7GjBlj/nnQ3ozNv2sc7fUe4e+cVO65556oq6uLlStXRv/+/XNvv/LKK+P73/9+RES8+uqrsX379vjSl750zH0c25FmfuDAgXjvvfdyj3nqqafizDPPzMXVST/zgtz51Y1s2LAhO2HChOzZZ5+dHTduXHbdunWFXlK387//+7/Zmpqa7OjRo7PnnXdedvr06bmbes0/nXnz5mWHDh2a/cxnPpM988wzczehtjdj8z8xR5p5e6/3bNbMT9SWLVuyEZEdPnx49vOf/3z285//fHb8+PHZbDab3b59e/bSSy/Njhw5MvsHf/AH2dWrV+eOa28f7TvazDOZTHbcuHHZ8847L/u5z30ue/HFF2fXrl2bO+5kn7n/VQ4AQGK+RAgAkJgM00mpAAAAEElEQVTAAgBITGABACT2fwG3shzbsUo6+wAAAABJRU5ErkJggg==" }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# see the distribution of values in that column\n", "histogram(train[!, :pixel404], bins=20)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGZxJREFUeJzt3X1wVeXZ6OE7kp4oUoIgekpjiIpRqUhoqnU6KChVqIMf05a2KhUURxGt06kFHEdbtVPGdqi+Y03VP/ymg8FKx1ZbqB4krTM9L1ak4gcVKyFQAUUwnJSAIOv8wek+xoSY6JOsEK9rhpG1P9Z69r32wI+9d7ZFWZZlAQBAMgfkvQAAgN5GYAEAJLZfBNb27dtj+fLlsX379ryXAgDwkfaLwFq1alVUV1fHqlWruv3YjY2N3X5M9jL7/Jh9fsw+P2afn944+/0isPL0/vvv572ETy2zz4/Z58fs82P2+emNsxdYAACJCSwAgMQEFgBAYgILACAxgQUAkJjAAgBITGABACQmsAAAEivOewE93caNG6O+vj7vZbRy6KGHRnl5ed7LAADaILDasWHDhvjCF76Q9zLadOBBfeMfq14VWQDQAwmsdmzYsGHvb757d0T5qHwX80EbX40d906NzZs3CywA6IEEVkeUj4oY2oMCCwDo0XzIHQAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgsVaBddZZZ8WJJ54YVVVVceqpp8aKFSs6vLP77rsvRowYEcXFxXHnnXe2uG779u1xwQUXxLBhw6KysjIWLlxYuO69996LK6+8MqZOnfrxHwkAQA/RKrAWLFgQL774YqxYsSKuvfbauPTSSyNibwStWbOmxW137NgRa9euLWxXV1fHggUL4sILL2x1oLlz50ZJSUm8/vrrsXjx4pgxY0Zs3bo1du3aFTfffHNcdtllqR8bAEAuWgXWgAEDCr9vbGyMAw7Ye5OXX345xo0bFytXroyIva9ITZw4MebPn1+4/ciRI+P4448v3OeDamtr46qrroqIiCOPPDJOO+20ePzxx2PJkiWxfv36qKmpiWXLlsXzzz+f9hECAHSz4rYuvPjii+OZZ56JiIhFixZFRMSoUaPioYceivPOOy/uv//+uPHGG+OMM86I6667rkMHamhoiKFDhxa2KyoqoqGhIaZOnRrjx4+P+vr6uOmmm6K6unqf+2hqaopt27YVtktKSqKkpKRDxwcA6C5tBtZDDz0UEREPPvhgzJw5M/7whz9ERMTo0aPjrrvuirFjx8b06dPjpptu6tTBioqKCr/PsqzFdRUVFfHAAw+0e/8xY8a02J41a1bMnj27U2vojMbGxi7bdwqNjY2xZcuWvJfRZbZu3Zr3Ej61zD4/Zp8fs89Pb5j9wIEDW2y3GVj/MWXKlJg+fXq88847MWjQoNi8eXNcf/31MXv27FiwYEEsXbo0xo4d26EDl5eXR319fQwePDgiItauXRtnn312pxZfV1cXVVVVhe2ufgWrtLS0y/adQmlpaasT2tv09sfXk5l9fsw+P2afn942+xYfltq2bVu8+eabhe3f/va3MWjQoBg4cGBs2rQpxo0bFzNmzIhbb701nnjiibj00ktj8eLFHTrQpEmToqamJiIi1qxZE3V1dXHuued2arH9+vWL/v37F355exAA6IlaBFZjY2Ocf/75MWLEiBg5cmTU1NTEE088EUVFRbFjx4644YYbYtq0aRERMXz48Fi0aFE0NzcX7j9v3rwoKyuLRx99NG688cYoKyuLF154ISIiZs6cGc3NzTFs2LAYP3581NTU9LpaBQCI+NBbhEcccUQsW7aszRsOHTq0xYfUIyIqKyujsrKysD158uSYPHlym/c/+OCDo7a29pOuFwCgx/NN7gAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgsRaBtWPHjjj//POjsrIyqqqqYsKECVFfX9/hnT355JPxpS99KUpKSuKHP/xhi+v27NkT3/ve9+Loo4+OYcOGxa9+9avCdVmWxU9+8pMYO3bsJ3owAAA9QatXsC6//PL4xz/+EStWrIiJEyfG5ZdfHhF7I+i1115rcds9e/bE6tWrC9vHHHNM3HvvvTFz5sxWB5o3b1688sor8dprr8WyZcvi5z//eaxatSoiIubMmRMXXXRR0gcGAJCXFoF14IEHxtlnnx1FRUUREXHKKafEG2+8ERER//rXv+JrX/taLFmyJCIi3n///bj44ovjF7/4ReH+lZWVMXLkyCguLm51oNra2pg+fXr06dMnBg4cGN/61rfikUceiRdffDFWrlwZd9xxR7z++uvx1FNPddmDBQDoDq1L6APuuOOOOOeccyIioqysLJ588smYOHFi3H777fHwww/HZz/72RZv9bWnoaEhhg4dWtiuqKiIv/3tb3HiiSfGI488EhERK1asiDPPPHOf+2hqaopt27YVtktKSqKkpKRDxwcA6C77DKw5c+bE6tWr4+677y5cdtxxx8Xjjz8eI0eOjLPOOitqa2sLr3Z1xAdvm2VZq+uXLl3a7v3HjBnTYnvWrFkxe/bsDh+/sxobG7ts3yk0NjbGli1b8l5Gl9m6dWveS/jUMvv8mH1+zD4/vWH2AwcObLHdZmDNnTs3Fi5cGE8//XT07du3cHlzc3P84Ac/iCuuuCKWLFkStbW18Z3vfKdDBy4vL4/6+vo46aSTIiJi7dq1UV5e3qnF19XVRVVVVWG7q1/BKi0t7bJ9p1BaWtrqhPY2vf3x9WRmnx+zz4/Z56e3zb5VYN12220xf/78ePrpp2PAgAGFy5uamuKcc86JU089NW655ZZ4880346yzzorm5ua45JJLPvJAkyZNinvuuSe+/vWvR2NjY9TW1saiRYs6tdh+/fpF//79O3UfAIDu1uJD7uvXr49rr7023n333Tj99NOjqqoqvvzlL0dExK5du2Ly5Mlxyy23RETEkCFDYsmSJXHAAf9/F0uXLo2ysrK47bbb4p577omysrL43e9+FxER3/3ud+PYY4+NysrKOOmkk2LmzJlx/PHHd9fjBADoNi1ewSorK2vzs1EREYccckhMmzatxWWHHXZYTJkypbA9duzYWL9+fZv379OnT9TU1HzS9QIA9Hi+yR0AIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEhMYAEAJCawAAASE1gAAIkJLACAxAQWAEBiAgsAIDGBBQCQmMACAEisRWBdc801UVFREUVFRfHSSy91akf33XdfjBgxIoqLi+POO+9scd327dvjggsuiGHDhkVlZWUsXLiwcN17770XV155ZUydOvXjPwoAgB6kRWB985vfjGeffTaGDh3a4kbvvfderFmzpsVlO3bsiLVr1xa2q6urY8GCBXHhhRe2OsjcuXOjpKQkXn/99Vi8eHHMmDEjtm7dGrt27Yqbb745LrvsspSPCQAgVy0C67TTTouysrJWN3r55Zdj3LhxsXLlyojY+4rUxIkTY/78+YXbjBw5Mo4//vg44IDW7zrW1tbGVVddFRERRx55ZJx22mnx+OOPx5IlS2L9+vVRU1MTy5Yti+effz7pgwMAyEOHPoM1atSoeOihh+K8886Lurq6mDBhQowePTquu+66Dh2koaGhxatiFRUV0dDQEOPHj48HH3wwfvSjH8XJJ58c1dXV7e6nqakptm3bVvi1c+fODh0fAKA7FXf0hqNHj4677rorxo4dG9OnT4+bbrqpUwcqKioq/D7LshbXVVRUxAMPPPCR+xgzZkyL7VmzZsXs2bM7tY7OaGxs7LJ9p9DY2BhbtmzJexldZuvWrXkv4VPL7PNj9vkx+/z0htkPHDiwxXaHA2vz5s1x/fXXx+zZs2PBggWxdOnSGDt2bIfuW15eHvX19TF48OCIiFi7dm2cffbZHV/1/1NXVxdVVVWF7ZKSkigpKen0fjqqtLS0y/adQmlpaasT2tv09sfXk5l9fsw+P2afn942+w69Rbhp06YYN25czJgxI2699dZ44okn4tJLL43Fixd36CCTJk2KmpqaiIhYs2ZN1NXVxbnnntvpxfbr1y/69+9f+NWVcQUA8HG1CKyrrroqysrKYv369fHVr341hg0bFhF7f2LwhhtuiGnTpkVExPDhw2PRokXR3NxcuO+8efOirKwsHn300bjxxhujrKwsXnjhhYiImDlzZjQ3N8ewYcNi/PjxUVNT0+tKFQDgP1q8RVhTU1N4pemDhg4d2uqrGyorK6OysrKwPXny5Jg8eXKbBzn44IOjtrY2xXoBAHo83+QOAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiQksAIDEBBYAQGICCwAgMYEFAJCYwAIASExgAQAkJrAAABITWAAAiRXnvQAAYP+zYcOG2LBhQ5J9NTY2RmlpaZJ9HXrooVFeXp5kX5+EwAIAOmXDhg0xZMiQvJfRpgMP6hv/WPVq7pElsACATim8cvXduyPKR+W7mA/a+GrsuHdqbN68WWABAPup8lERQ3tQYPUgPuQOAJCYwAIASCz3twhXr14dU6ZMic2bN8eAAQPigQceiOHDh+e9rP3Cq6++mvcSWukpP70BAHnKPbCuuOKKuPzyy2Pq1Knxm9/8JqZNmxZ//etf815Wz/Z/NkdExOTJk3NeSGslBx4Uj/3m0fjc5z73ifeV8sd233///ejTp0+SfaXUU9e1devWOOSQQ/JeRis9dV7+YQF8WK6B9dZbb8Xy5cvjT3/6U0REfOMb34irr7466uvro6KiIs+l9WxNewOrx/30xht/jZ3zvx8TJ07MeyXQrfzDonN66rr8w6LjeuI7KD1NroG1bt26GDJkSBQX711GUVFRlJeXR0NDQ5uB1dTUFNu2bStsl5SURElJSdcvdGMPeyK9s2bvf//Hgfmu48O2v7v3v2d+P+J/HpfvWj5ozX9HPHu/dXWUdXXOmy/Hzv/1S/+w4NOpp/392IPWk/tbhEVFRS22syzb523HjBnTYnvWrFkxe/bsLllXRERxcXGUHHhQ7Lx3apcd42MrOiCip67rqf/KexWtWVfnWBfwUXro30MlB/WN4uLi2LJlS7ced+DAgS22cw2sI444ItavXx+7d++O4uLiyLIs1q1bt8/PMtTV1UVVVVVhu6tfwRo4cGAs++//Hbt37+6yY3xcO3fu7J5X7zop5bpSvlXyaZhXSm+//XYMHjw472W00lPn5XnfOT11XZ73ndNTn/c95TORuQbWYYcdFqNGjYp58+bF1KlT47HHHouKiop9fv6qX79+0b9//25dY1lZWasqpXts2bLF7HNi9vkx+/yYfX564+xzf4vwnnvuialTp8acOXOif//+8eCDD+a9JACATyT3wDr22GN9LQMA0Kv4JncAgMQEVjt27twZP/vZz2Lnzp15L+VTx+zzY/b5Mfv8mH1+euvsi7L2vhehh1i+fHlUV1fH888/H1/84he77bjbtm2L0tLSaGxs7PYP13/amX1+zD4/Zp8fs89Pb529V7AAABITWAAAieX+U4Qd0dzcHBHd//8+ampqioiIFStWRL9+/br12J92Zp8fs8+P2efH7PPTm2Z/3HHHRd++fSNiP/kM1q9//euYPHly3ssAANinD35WfL8IrM2bN8fixYujoqIiDjrooLyXAwDQyn73ChYAwP7Eh9wBABITWAAAiQmsfVi9enV85StficrKyjj55JPjlVdeyXtJvVpFRUUcd9xxUVVVFVVVVVFbWxsRzkNXuOaaa6KioiKKioripZdeKlze3qydh09uX3Pf13M/wtxT2bFjR5x//vlRWVkZVVVVMWHChKivr4+IiLfeeismTJgQxxxzTJxwwgnx7LPPFu7X3nV0THuzHzt2bBx11FGF5/7tt99euF+vmH1Gm04//fTs/vvvz7Isyx599NHslFNOyXdBvdzQoUOzlStXtrrceUivrq4uW7duXauZtzdr5+GT29fc9/XczzJzT6W5uTl78sknsz179mRZlmW//OUvszPPPDPLsiy75JJLsh//+MdZlmXZsmXLsvLy8mzXrl0feR0d097sx4wZk/3+979v8369YfYCqw2bNm3KSktLCydzz5492eGHH56tWbMm34X1Ym39JeM8dK0Pzry9WTsPaXU0sMy96zz33HPZ0UcfnWVZlh188MHZW2+9VbjupJNOyp555pmPvI6P54Ozby+wesPsvUXYhnXr1sWQIUOiuHjv97AWFRVFeXl5NDQ05Lyy3u2iiy6KESNGxGWXXRZvv/2289CN2pu189D1Pvzcj/DnUFe644474pxzzol33nkn9uzZE4MHDy5cV1FREQ0NDe1ex8f3n9n/x8yZM2PEiBHx7W9/O954442IiF4ze4G1D0VFRS22M99m0aX+/Oc/x9///vdYvnx5DBo0KKZMmRIRzkN3am/WzkPX2ddzP8Lcu8KcOXNi9erV8dOf/jQiPO+704dn//DDD8err74aL774Ypx66qkxceLEwm17w+wFVhuOOOKIWL9+fezevTsi9p7YdevWRXl5ec4r673+M9vPfOYz8f3vfz/+8pe/OA/dqL1ZOw9dq63nfoQ/h7rC3LlzY+HChfHHP/4x+vbtG4MGDYqIKLxqGBGxdu3aKC8vb/c6Ou/Ds4/Y+xyP2BtTV199dbzxxhvxzjvv9JrZC6w2HHbYYTFq1KiYN29eREQ89thjUVFRERUVFfkurJf697//He+++25he/78+TFq1CjnoRu1N2vnoevs67kf4c+h1G677baYP39+PPXUUzFgwIDC5ZMmTYqampqIiHjuuedi48aNMXr06I+8jo5ra/a7d++OTZs2FW7z2GOPxeGHH16Iq14x+1w++bUfWLVqVXbKKadkxxxzTFZdXZ299NJLeS+p1/rnP/+ZVVVVZSNGjMhOOOGE7Nxzzy18kNd5SG/GjBnZ5z//+axPnz7Z4YcfXvjAaXuzdh4+ubbm3t5zP8vMPZV169ZlEZEdddRR2ciRI7ORI0dmJ598cpZlWbZx48bszDPPzIYNG5YNHz48W7p0aeF+7V1Hx+xr9k1NTVl1dXV2wgknZCeeeGJ2xhlnZCtWrCjcrzfM3v8qBwAgMW8RAgAkJrAAABITWAAAif1faw6QV82ehA0AAAAASUVORK5CYII=" }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# a point a little more to the edge has a lot more zeros\n", "histogram(train[!, :pixel397], bins=20)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEpJJREFUeJzt3W9snXXZwPGr7cbGMgdZKwQcbSFSB5XR0mkmEQx/AwQSdb4wEYE4YoyoEQ1/XoCJcQ4N0RdACJBnLhDIgoQsJEBYFJ3GBA3QMBE2MsY6R3YKq4Mtg23P1t7Piz00VEDmep1z99z9fBKS08J2Xxdjh++5z29tS1EURQAAkKa17AEAAKpGYAEAJDuiwHr33XdjcHAw3n333ex5AACa3hEF1saNG2NgYCA2btyYPc9h27VrV2nXrqeq7hVht2ZU1b0iqrtbVfeKqO5uVd0rotq7fZymfYtwdHS07BHqoqp7RditGVV1r4jq7lbVvSKqu1tV94qo9m4fp2kDCwBgqhJYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkGxG2QMw0fDwcAwNDTXseh0dHdHZ2dmw6wHAdCCwppBarRa9vb0Nvebso+fEKxs3iCwASCSwppBarXbowTfviejsr/8FhzfEvpXXxMjIiMACgEQCayrq7I/oakBgAQB14ZA7AEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAshllDwAAh2N4eDiGhoYacq2Ojo7o7OxsyLWoJoEFwJRXq9Wit7e3YdebffSceGXjBpHFERNYAEx5tVrt0INv3hPR2V/fiw1viH0rr4mRkRGBxRETWAA0j87+iK46BxYkEFiQwNkQAN5PYMEkORsCwL8TWDBJzoYA8O8EFmRxNgSA/+cLjQIAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAk85XcaRjfEBlgepnOz/sCi4bwDZEBppfp/rwvsGgI3xAZYHqZ7s/7AovG8g2Roa6m81syTFHT9Hm/aQPLkwjARNP9LRmYSpoysDyJAHzQdH9LBqaSpg2siPAkAvBhpulbMjCVNGVgjfMkAgBMQc0dWEBdOesIcGQEFvChnHWExmjkC5kIL2YaRWABH8pZR6i/Rr+QifBiplEEFvCfOesIddPQFzIRXsw0kMACgLJ5IVM5rWUPAABQNQILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQzyh4AoAzDw8MxNDTUkGt1dHREZ2dnQ64FTA0CC5h2arVa9Pb2Nux6s4+eE69s3CCyYBoRWMC0U6vVDj345j0Rnf31vdjwhti38poYGRkRWDCNCCxg+ursj+iqc2AB05JD7gAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAsgmBtWzZsmhpafnAXzt37ixrPgCApjMhsC677LJDn2xtjbPPPjtaWw/97ZkzZzZ+MgCAJjUhsNra2sYf33rrrVEURURE/P73v2/sVAAATWzG+z94763AsbGxuPTSS8c//9JLL8VXvvKVD/zgBx98MNatWzf+cV9fX5x//vl1GvVDDG9o2DXWrFkTg4ODdb3U1q1bJ1yz7qq6WwP3iqjublXdK6K6u1V1r4jq7uZ5P0mj/v39F1qK925TxaG7VsuXL4+2traYOXNm7Nu3LyIifvSjH8WvfvWr8R80ODgYAwMDH/oT3nTTTXUeOWL37t1x3/+sjNED/1v3azVcS2tEMVb2FPVht+ZT1b0iqrtbVfeKqO5uVd0roqG7tc08Kr597bKYN29eQ673737xi19M+HjCHazNmzdHRMTo6GiMjo6Of/7FF1/80J/s+uuvjwULFox/3Mg7WN/5znfi4MGDDblWrVaLWq3WkGvNnTs3enp6GnKtiOru1si9Iqq7W1X3iqjublXdK6K6u3nez9HR0RGdnZ0NudbhmHAH6+9//3uceeaZERFxxhlnxEsvvRRjY2Px3HPPTbhj9d4drOeffz7OOuusxk8dh97OnD9/finXrqeq7hVht2ZU1b0iqrtbVfeKqO5uVd0rotq7fZwJd7BuvPHG8cfv3bWaM2fOR74dCADAB00IrKeeeqqsOQAAKsNXcgcASNaUgbV///745S9/Gfv37y97lFRV3SvCbs2oqntFVHe3qu4VUd3dqrpXRLV3OxwTDrkfrrIPue/evTuOOeaY2LVrV2l/HLMeqrpXhN2aUVX3iqjublXdK6K6u1V1r4hq73Y4mvIOFgDAVCawAACSzfj4f+SD9u7dGxERGzaU86Xp9+zZExERL7zwQsydO7eUGeqhqntF2K0ZVXWviOruVtW9Iqq7W1X3iqj2bh9l4cKFMWfOnIg4wjNYDz30UFx55ZXpgwEANKv3n00/osAaGRmJtWvXRnd3dxx99NHpAwIANJtJ38ECAOCjOeQOAJBMYAEAJGu6wNq0aVOcffbZ0dPTE5///Ofj5ZdfLnukFD/4wQ+iu7s7Wlpa4h//+EfZ46TZt29ffPnLX46enp7o6+uLSy65JIaGhsoeK83FF18cixYtir6+vjjnnHPihRdeKHukVD/96U8r999kd3d3LFy4MPr6+qKvry8efvjhskdKsX///vje974Xp556avT29lbmDyK9/fbb479WfX190dPTEzNmzIidO3eWPdqkrV27NgYGBqK/vz8++9nPxv3331/2SGmeeuqpWLx4cSxatCiWLFkS69evL3ukxiuazHnnnVesWrWqKIqieOSRR4olS5aUO1CSP/3pT8W2bduKrq6u4sUXXyx7nDR79+4tnnjiiWJsbKwoiqK48847i4suuqjkqfK89dZb44/XrFlT9Pf3lzhNrueff7645JJLis7Ozkr9N1m132Pv+eEPf1h8//vfH/+9tn379pInqo/bb7+9uPzyy8seY9LGxsaK+fPnF+vXry+Koii2bNlSzJo1q9i9e3fJk03ezp07i/b29uLll18uiqIo1q1bV/T29pY8VeM11R2sN998MwYHB8dfmS1dujS2bNlSiTsi5557bixYsKDsMdLNnj07LrvssmhpaYmIiCVLlsRrr71W8lR5jj322PHHu3btitbWpvot9ZH2798f1113Xdx9993jv3ZMXe+8806sWrUqVqxYMf7rdcIJJ5Q8VX2sWrUqli1bVvYYad5+++2IOPRtZdrb22PWrFklTzR5mzdvjuOOOy5OO+20iIj40pe+FFu3bo3BwcGSJ2uspvq/wbZt2+LEE0+MGTMOfX3UlpaW6OzsjH/+858lT8bhuuOOO+KKK64oe4xUV111VZx00klxyy23VOYW/09+8pO48sor4+STTy57lLr4xje+EWeccUZce+21sWPHjrLHmbTNmzdHe3t7LF++PBYvXhznnHNOPP3002WPle6ZZ56Jf/3rX3H55ZeXPcqktbS0xG9/+9v46le/Gl1dXfHFL34x7r///jjqqKPKHm3STj311NixY0f89a9/jYiINWvWxJ49eypxM+S/0VSBFREfeDVd+CoTTWPFihWxadOm+PnPf172KKkeeOCB2LZtWyxfvjxuuOGGsseZtGeeeSaeffbZ+O53v1v2KHXx5z//OdavXx+Dg4PR3t4eV199ddkjTdqBAwfitddei9NPPz2ee+65uOuuu+LrX/96JeLx/X7zm9/EVVddNf4iu5kdPHgwbrvttnjsscdi69at8fTTT8fVV19dibNlxxxzTDz66KNx8803x8DAQKxbty5OP/30mDlzZtmjNVbZ71H+N954441i3rx5xYEDB4qiOPQe9vHHH19s2bKl3MESVfV8yO23314MDAxMOLNURbNnzy5GRkbKHmNSbrvttuKEE04ourq6iq6urqKtra048cQTiyeffLLs0dJt3769mDt3btljTNqOHTuK1tbW4uDBg+Of+9znPlf88Y9/LG+oZHv27Ck+8YlPFBs2bCh7lBTPPvtscdppp0343OLFi4s//OEPJU1UP/v27SuOPfbYYtOmTWWP0lBNdQfruOOOi/7+/njwwQcjIuLRRx+N7u7u6O7uLncw/qNf//rXsXr16vjd73434cxSs9u9e3ds3759/OM1a9ZEe3t7zJ8/v8SpJu/mm2+O7du3x9DQUAwNDcWCBQti7dq1cemll5Y92qS9884742deIiJWr14d/f39JU6Uo6OjIy644IJYu3ZtRERs3bo1tmzZEp/5zGdKnizPI488EosWLYqFCxeWPUqKk046KV5//fV45ZVXIiLi1Vdfjc2bN0dPT0/Jk+Wo1Wrjj3/2s5/F+eefH5/+9KdLnKjxmu4+67333hvXXHNNrFixIubNm1eZMy/XXXddPPbYYzE8PBwXXnhhzJ07N1599dWyx5q0119/PX784x/HKaecEuedd15ERMyaNSv+9re/lTzZ5O3atSuWLl0ae/fujdbW1vjkJz8Zjz/+uEPhU9gbb7wRS5cujdHR0SiKIk455ZR44IEHyh4rxT333BPf+ta34qabboq2tra47777KnXQfeXKlZU63H788cfHvffeG1/72teitbU1iqKIu+++Oz71qU+VPVqKW2+9Nf7yl7/EwYMH4wtf+EKsXLmy7JEazrfKAQBI1lRvEQIANAOBBQCQTGABACT7P8NKEf9tCA34AAAAAElFTkSuQmCC" }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# distribution of labels - seems fairly even\n", "histogram(train[!, :label], ticks=collect(0:9))" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8lNW9x/HvJCEbk4WwL0LAioIsEVIKVhTX4lJto7jcCIJefbVV8VrRUl5cVG6vVuVVrusFixVEWlTU8nqhRUVMqyBUhbCDGBJA1qyTTGafee4fvMglZdWcZyYz+bz/SjKTc34nM3nyzfOc5xyHZVmWAAAAYExSrAsAAABINAQsAAAAw75XwPJ4PFq3bp08Ho/pegAAAOLe9wpY27dv1/Dhw7V9+3bT9Zwxl8sVs77tlKjjkhhbPErUcUmJO7ZEHZeUuGNL1HFJiT2204nbS4ThcDjWJdgiUcclMbZ4lKjjkhJ3bIk6Lilxx5ao45ISe2ynE7cBCwAAoLUiYAEAABiWEusCAESX3+/Xjh07FAqFTvtcl8ulnJycKFQVfa1xbCkpKTr33HOVlpYW61IAtBABC2hDysrKVFBQILfbHetScBJZWVkqLS1Vv379Yl0KgBYgYAFtRCQS0V133aVOnTrpb3/7mzIzM2NdEv6Fx+PR7bffrjvvvFMrV65UUhKzOIB4RcAC2ogDBw7o73//u/785z/roosuinU5OIknn3xS//Zv/6aDBw+qR48esS4HwPfEv0dAG1FZWSlJOvvss2NcCU7l6Otz+PDhGFcCoCU4gwW0EZFIRNKRidTHcrvd8vl8xvtLT0+X0+k03m6iO/r6HH29AMQnAlYr4/F4onpg5Y9g2+Z2u/XywjdU7Tn9HYXfVcfMFN0z/pbTvr+WLl2qqVOnKi0tTQsXLtT48eP1+eefKyMj44z7KikpUSAQ0FVXXXXCx2tqanT99dfL7XZr3Lhx8vv9Ov/883XLLbcoEAioqKhIe/bs0ejRozVu3LhmbVVUVKiwsFBVVVVnPngAbR4BqxXxeDx6c+ECpTuiF7BSnTm6deKdhKw2yufzqdoTUsb5lyozO9dYu576OlVv+UQ+n++07605c+Zo5syZGjdunCSptLT0hM8LhULHnX07qqSkRG63+6QB66OPPlJOTo4+++yz4x5bv369ysvLtWXLFknSY489dsq2AOBMELBaEY/HI/ka9aPz+qhjTrbt/bncjVq168AZ/RFEYsvMzpWzQyejbXrP4DmTJ0/Wp59+qh07dmj27NlavXq1HA6HGhoa5HQ6lZ+fr7vvvlsrVqxQjx499Nhjj2nixIlyu92KRCK64YYbdNNNN2nOnDmKRCJasWKFioqKNGPGjKY+VqxYoYcfflj19fUqKCjQrFmz9Prrr6uwsFCXXXaZiouLtX//fhUUFKioqOi4tiZMmCBJmjFjht577z25XC4999xzuuaaa4z+vAAkFgJWK9QxJ1udclvXAoiAHZ577jlt3LhRU6ZM0XXXXXfC5+zZs0crV66Uw+HQAw88oGuvvVbTpk2TdOTSX15enn7xi1/I7XZr1qxZx33/FVdcoZkzZ2rZsmVasmSJJOn111+XJA0cOFDz5s3TlClT9OWXX0o6Mvfp2LYqKipUXV2t4cOHa+bMmVq+fLkeeOABAhaAUyJgAWjVJk2aJIfDIUm6+OKL9fDDD6uxsVGXXHKJrrjiiqjU0L59e91www2SpFGjRqmsrCwq/QKIXyzTAKBVO/by9Y033qhVq1bp3HPP1QsvvHDSs16mpaenN32cnJyscDgclX4BxC/OYAGIGzt37lS/fv00YcIEjRgxQhdeeKEkKTs7W/v27TPSh8m2ALRdBCwA8tTXter2jnrrrbe0aNEipaamyrIszZkzR5L085//XAsXLmyaqH7sJPfv6l/bOjrJHQC+CwIW0Ialp6erY2aKqrd8ckZ3/X0XHTNTml1aO5mSkpJmn1uW1fRxRUVFs8emTZvWNMH9WH379tX69etP2sfEiRM1ceLEps/nz5/f9PGYMWOaJrifrK1j18ByOp3NagSAEyFgAW2Y0+nUPeNvYSV3ADCMgAW0cU6nkyAEAIYRsFqZQDCoGld9VPqqcdXL5/dHpS8AANoSAlYr4vF4tH7TOvlqdikjPc32/rw+v3ZVeVXkdqtTJ7OreAMA0JYRsFoRr9erJEdA5/8gQz27dLC9v4NVLn1zuEZ+zmIBAGAUAasVys3KUIfsTNv7afSan9gMAAAIWECb53a7uYvwGPPnz2+2b+Gx1qxZo7vvvlspKSn6/e9/r2effVbPP/+8zj77bJWVlWncuHGyLEuTJ0+Ww+HQhRdeqP79+5+2XQCJh4AFtGFut1uL5/9JAbfLeNupzhzdOvHOuAxZJ7NgwQJNmDBBDz/8sCTpJz/5SdNjS5Ys0ahRo/Tiiy9KOrK+VqdOnZoCFoC2hYAFtGE+n08Bt0s/7tddOc72xtp1uRu1atcB+Xy+UwYsr9eriRMnatOmTWrXrp26du2qDz/8UJK0cOFCvfDCCwoGg8rKytKLL76oQYMGSZKeeuopvf7660pKSlJGRoZWrlypzMxMPf3001qwYIGSkpI0ZMgQvfTSS8rJydFjjz2mr7/+Wg0NDSorK1O3bt20ZMkS5eXlKRAI6P7779cnn3yinj176rzzzjthrb///e/1xhtvKDMzU4sWLVJJSYkKCgq0bNkyrVu3TrNnz1YkEtGqVat055136ssvv9TkyZM1ffp0PfHEE5KkhoYG3Xbbbdq8ebPS0tL05ptvql+/fsZ+7gBaDwIWAOU426tTbk7U+12+fLlqa2u1detWSVJNTY0kadWqVVq8eLH+8Y9/KC0tTZ9++qmKi4u1YcMGLViwQH/961+1atUqZWdnq7a2Vmlpafrb3/6mV199VZ9//rlyc3N1zz33aNq0aU1nlNauXasvvvhCeXl5uvXWWzV37lz99re/1dy5c1VeXq4tW7YoGAzq4osvVn5+/nG1Tp06Vdu3b1dhYaHuu+++Zo9NmDBBu3btktvt1qxZsyRJ77zzjqZMmdK0IfX8+fO1du1abdiwQX369NHUqVP11FNPae7cuXb9eAHEUFKsCwDQdg0dOlTbt2/Xr371K73xxhtq166dJGnp0qXasGGDfvSjH6mgoED333+/KisrFQgEtGzZMv3yl79Udna2JKlDhw5KTk7WihUrVFxcrNzcXEnSL3/5S61YsaKpr6uvvlp5eXmSpFGjRqmsrEyS9Mknn+iOO+5Qu3btlJmZqdtvv9228V500UXq06fPcTUASDycwWplwpGIAv6A/D77l04I+AMKhyO29wOcTL9+/bR161atXLlSK1as0COPPKLS0lJZlqU777xTM2fOPOO2LMuSw+Fo9rVjPz92X8Tk5GSFQqGm74uWk9UAIPEQsFoRr9erg5XVKtvbXvXuBtv7q6x161BllRobG23vCziRb7/9Vh06dND111+vsWPH6q9//av27t2rn/70p5owYYLuvvtunXXWWYpEIlq3bp0KCwt1/fXX66WXXtLPfvYzZWdnq66uTllZWbryyiv1yCOPaPLkycrKytLLL7+sK6644rQ1XH755Vq4cKFuueUWBYNB/fnPf1bv3r1bPLbs7Gy5XOZvHgAQHwhYrYjP51PEkpKz8pTWuavt/SWHDisU+VaBQMD2vtC6udxmQ/aZtrdp0yZNnTpVlmUpEolo/PjxGjJkiCTpiSee0A033KBwOKxgMKhrr71WhYWFGj9+vPbv369Ro0Y1XdZbsWKFrr76am3atEmjRo2Sw+FomuR+Ovfcc482btyogQMHqlevXho9erR2797dovEfbfehhx7SM8880zTJHUDb4bC+x/nxdevWafjw4frqq680bNgwO+o6rZqamqb5FIlizZo1+vV9E3TLdReoW7cutvd38OBhvfX+Bs16br5Gjhxpe3+J+JodFQ9jO9HvLcs0tD4tPb7Gw3vx+0rUsSXquKTEHtvpcAarFfF6vWp0e7Vv7yF5Gjy291db16BGt5dLhG2Y0+nUrRPvZKFRADCMgNWK+P1+JTuS1SOnp87q1t32/vY5DilJexQMBm3vC62X0+kkCAGAYQSsVig9NU0ZaRm295Oa0s72PgAAaIsIWK2NJQXDYQWC9k88D4bDsiyWaTDB4/EoEonOz5JLbwDQ+hGwWhGfzyd/0K/q+molp9i/Nk91fY38wYA8HvvneyUyj8ej+X9eoEgUXjNJysnI1l3jmTwOAK0ZAasV8fv9kkNKbZ+mzDz7/3imBhslh5iD1UIej0eekFe9huQru0O2rX01NjTq0MZ9p93jDwAQWwSsVigpJVkpqfa/NElJybb30ZZkd8hWdl709/NrKbfbzV2EAGAYAQtow9xut15Z+Ce5vPXG2z7TS5mPPfaYpk2bptTUVEnSxIkTT7ihsiTNmTNHXq9XDz744BnVMGbMmGYbLh/r0Ucf1Ztvvqns7Gy9+OKLmj17thYtWiRJmjt3rmbPnq309HR9/PHHeuWVV/TII4+cUbuwD3MdEU8IWEAb5vP55PLWq+uQnmqf1d5Yu9/lUubjjz+uKVOmNAWsU/nFL35x0sfC4bCSk8/8rOzTTz+tPXv2qHPnzpLUFK4k6X/+53+0cOFC/fCHP1RFRYWefvrpZgEL0efxeLR44RtKdaRFpb80Z5qKJ95GyML3RsACoPZZ7WNyefNoYLrwwguVlJSkDz/8UJK0detWXXHFFdqzZ48GDRqkxYsXKzU1VY899pjcbrdmzZql+fPna/HixerSpYu2bt2q559/Xjk5OZo0aZKCwaAGDBhw0kufF154oXw+ny6//HKNGTNGRUVFmjJlir788kvddNNNKisr0/jx4zVo0CC53W7V1dWpoKBAKSkp+vLLLyVJn376qZ566int379fV155pebMmROdH1ob5fF4FPZHdMG5hcrL7WhrX/Vul74oW8NcR7QIAQtAzMyZM0dz587V6tWrm/0hKy0t1ccff6zU1FRdfPHFevvtt3Xbbbcd9/2fffaZ1q9fr3POOUeSNHz4cE2ePFl33HGH1qxZox//+Mcn7Hf16tVyOBxN/ZaUlDQ9tmTJEuXn52vJkiUaNGiQKioqVFhYqNLS0mZtlJWVqaSkRIFAQAMHDtTnn3+uUaNGGfip4FTycjuqo80BCzCBgAWg1SkqKlJGxpHFdkeMGKGysrITPu+iiy5qClf19fXavHmzxo8fL0kaOXKkBg8ebFuNt956q5KTk5WRkaGCggKVlZURsGwWCAZU66qxvZ9aV82Ru7qBFiBgIWoSeYJqMBBUQ535ieL/qqGuvk0c+NPT05s+Tk5OVigUOuHz/vU1djgcttZ1rDOtEWZ4PB5t2Pi5fDUHbN/pwuv3qqLygNzuG9WpUydb+0LiImAhKjwej+YtekvBlMyo9NcxM0X3jL8lKiHL4/FoW+lXqnHtUlqGvRNw/V6/avc3yu12J8yBPysrSy6Xq8WvVXZ2tgYNGqRFixZp/Pjx+uc//6lNmza1uL7s7Gx5PB6FQiGlpHDIjBWv16skR0Dnn91e3bt0trWvw9XVqji0q038MwP7cLRAVHg8HtWHHModNFrODvYGA099naq3fBK1Caper1fJjoAG/CBDXbp1sLWv6kqXPvu22viBv7GhMWbtPfTQQ7rsssuUkZHRNMn9+3rttdc0adIkzZ49W8OGDdOPfvSjFrUnSXl5eSouLtbgwYPVvn37pknuiL5IJKyMtBRlptm7j2pqSpLCkbCtfSDxEbAQVc4OnWwPWJLktb2H5iKRiNLSk5SanmRrPyntpLDBy6zp6enKycjWoY37jLV5VE5GdrPLaCfz6KOP6tFHH236fP78+c0enzVrVtPHjz32WNPHEydO1MSJE5s9d+DAgVq7du0Z1WdZ/7+10ZgxY5oFp4qKimbP/eMf/9js82MnxUtHJsbDXl6vV4cOH9I3u8tU56q2ta+qujodPHhQjY1m//FA20LAAlrI6/Wq6vBh7d+TIq+7yta+amvcqjp8yNiB3+l06q7xd7KSO1q92tpaNbgb5A165Iucfs20lvAGPap3u1RbW2trP0hsBCyghXw+nxQJq0Nmmnp0sDdQOHwhRcIhBQIBY206nU6CEFo9l8ulYDCgJEWUmmzvzQzJshTwB+RyuWztB4mNgAUY0i4lRWnt7P3POiWF/SPRNvn9fsmSklPT1S4zy9a+klIbZUkKBoO29oPERsACAMQNhyNJSUn2znV0OOxtH20D7yIAAADDCFgAAACGxe0lwkReFRyIJrfbzV2EAGzRlv9Wx2XA8ng8enn+H+WPmLuT6lQ6ZuXpV3f/slW9cIAJbrdbi+b/RX63+RWr05xpKp5422l/b5YuXaqpU6cqLS1NCxcuNLZ/4NGNmK+66ipJatq0uarq9EtpBAIBFRUVac+ePRo9erQGDx4sr9erBx98UJJ01113ac2aNerfv78effRRff3117r55pubvt/hcKihoYFjBto0j8ejxQvfUKrD3h0ujjrTY060xGXAqqqq0rovVym3e4bSzmAhw5YIBgLauSWom38+rtW8aIApPp9PfrdfPzx7pLKdOcbarXe79EXZmjNaTX/OnDmaOXOmxo0bZ6z/UCikkpISud3upoD1Xaxfv17l5eXasmXLcY8dOnRIb731lurq6pSUlKT58+dr2bJlzQIWgCMBK+yP6IJzC5WX29HWvr7LMSda4jJgHd2apGBQj+hsTfLxHvakMiAYCMhdZ+8KzJLkrqtWwG/+ktepRCKWAv6gfF573ycBf9CW0+3Zzhx1tPkAeCKTJ0/Wp59+qh07dmj27NlavXq1li9frmnTpikUCqlDhw763//9Xw0cOFAlJSWaMmVK04rrmzdv1nXXXaeKioqms1OTJ0/WRx99pKKiIs2ZM0eRSEQrVqxQUVGRJkyYIEmaMWOG3nvvPblcLj333HO65pprmtW0detWFRcXa//+/SooKNCvf/1r7dq1S263W9OnT9ell14qj8ejYcOGqaioSPPmzVN9fb0KCgo0cuRIzZkzR5L04osv6p133tHhw4c1Y8YMTZo0Kbo/XMSNaF5Gk6J/KS0vt2NMji+xFpcB66is7Axl59i7ebDXE90/1InK4/Fo86Y1au+uU7v0DFv7Cvq8Cu4rl9t9Q1Q2RPZ6vfI0+HRwV6381fYGrFqXR556X8Js4fHcc89p48aNmjJliq677jodPnxYt99+uz755BMNHjxYixYt0s0336zNmzeftq3q6mr94Ac/0IwZMyQdWZjS7XY3bbVTUVGh6upqDR8+XDNnztTy5cv1wAMPHBewBg4cqHnz5jULc0e36MnNzdX777+vwsJClZaWSpJ69+6tZcuWHbddTnp6utauXatt27ZpxIgRGj9+PJtF4zjRvowmtb5LaYmK33ZEhdfrVdgRVu55XZXTrZutfTVUHta+b3dE7ayjy+VSJGwpN62jumbn2duZv1aR4L6EXWF67dq1KigoaJqHVVxcrHvvvVcHDhw47femp6frtttuO+Vz2rdvrxtuuEGSNGrUKJWVlbW86JMoLi6WJA0YMEApKSk6ePCgevXqZVt/iE/RvIwmRf9SWiAYUK2rxvZ+al01re5KEwELURMOhWTvBhdHWOGIQsFQFHo64ugK06mZTmXYHLDauQMJvcK0ZVlyOI5/lzgcDqWkpCgcDjd97V/vfGzfvv0Jv/dYx24+nZyc3Kw90/61r1AoOu/JtnzXVjxLxMtoHo9H60s/U8OhPUpPs3e+tM/v057qQxrnvjEqVy7OBAELUeH1elW/r0qHP9usemeFrX353G7V76uK+mU0VphuuVGjRumuu+7Stm3bNGDAAC1evFi9evVSt27dFA6HVV5erurqanXs2FELFy48ZVvZ2dnat2+f7TVnZ2e3mjOKHo9H//vqInms6Bzau2Sl6/677yBk4YRqampUWXVAQ/t2UF6uvdt81dU36uDWvaqpqVF+fr6tfZ2puA1YkUhEgYBfPpsnMwcCfoWjOPkwUfl8PqU70jX4rGHq1L2nrX3VHDqgv2//xuiGyImu3m02IHzf9jp37qyFCxequLhY4XBYubm5evPNNyVJPXv21JQpU1RYWKj8/HxdfPHFp2zr5z//uRYuXKiCgoJmk9xNu/zyyzVr1iwNHTpUo0aNaprkHgtVVVUq/WKtsrucpdR0e+enhvw+7a47qKqfX0vAwgn5fD6FLSmrS091tPm4Hzy4X6HI1lZ13I/LgOX1enWwskq79rZXtafB1r5c1W4dOhz9syGJqn37bDmzcm3tw1tv73sikaSnpyvNmaYvytYYbzvNmdbsMtnJlJSUNPt87NixGjt27AmfO336dE2fPr3p88cff1ySlJ+ff9z6Vn379tX69eubfe3Y5zidTlmWdcJ+xowZ0zTBXfr/Se4n6isnJ0erV69u9v3/2u6ZrL1lgtfrVXLIp6E9OiqvU1db+3LVVunzg2Wtbt4LWhcrWicoLEuRyIl/n2MlLgOWz+dTOOJQclZHpdl8EEkOHlYo8m2rSsWAKU6nU8UTb2Ml9wTh9Xp16OBufb0zWe332/uzb2x06+DBPfzziZPyer3yeNyqrdqrpIjH1r5qa6rl8dS3qvdjXAaso5JS2iklzd5bW5OS4/pHBJyW0+kkCCUIn8+ndilhDTw7U9272Tth+tBhS9+UB/nnEyfl9/tlKaKUjHZK62DvJet2vnpFIpFWdQNQ3KYHy4oo5A8o4PHa2k/IH1A4zBwsAPEjq326srPs/YPmbojeuk2ITz6fT8FgQCGfW8EGe5dqCHrdCgcD8njsPVP2XcRlwPJ6vfI3eOQuP6CkWretfbnrGhRo8LSq044AgMQRCgbV2NiotBT7Q2tjY6MCweicdfT7/UqyLGWmJik7095lGhpTkyVZnMFqKb/frxSlqHtOL3Xtbu+ilYd1SMna06peNAA4mUgkIp/fJ4/X3rP7Pr8vqtu7JCqPx6OtmzapcyBH2ZlZtvdX72nQ1ooNcrvdUVsvyuFIVkqyvcs0tMYlbOIyYB2VmpauNJtvRW6XymlwJDa3280k9wTh9Xrldnu0t+Jb1dfauzaXy9Ugd30jZ/dbyOv1SqGA+uRmqWfnLrb3d6DaoVVf+7n7MwriOmABaBm3263XX/ujvB7z8yMyMvN0+4S7TxuyHA6HGhoavlcYKy0t1ddff62bb775hI8HAgEVFRVpz549Gj16tAYPHiyv16sHH3xQknTXXXdpzZo16t+/vx599NHj2mpJbbHgcrkUDoaVmuSUM8Xe5VD8yVIwFG41i6zGu8yMNDkz7d2nVZLS3am294EjCFiIGsuKKBz0KxSw9z+ncNAfvbVX4pzP55PXU6Oh5+UYnRRd3+DRhu01tu93VlpaqmXLlp00YK1fv17l5eXasmXLcY8dOnRIb731lurq6pSUlKT58+efsq144Pf7JYfkzHWqQzd7t23yOoKSI3G3bQJaioCFqPB6vfI1utV46Fu1C9k7N6Sxulq+xgYuXXwH2VmZyutgev7Hdz+z8fDDD6ukpETBYFA5OTmaN2+ezjnnHFVWVqq4uFgHDhyQw+HQ8OHD9dRTT2nGjBmqr69XQUGBRo4c2WwV9a1bt6q4uFj79+9XQUGBfv3rX2vXrl1yu92aPn26Lr30Unk8Hg0bNkxFRUWaN2/eCdt68cUX9c477+jw4cOaMWOGJk2aZOwnZJeklGSlpNp7eE9KsndODRDvCFiIiiN3k0TUIbOduubaO5HT4XXLYbWuu0lwZn7zm9/omWeekSQtXrxYDz74oJYtW6bXX39d+fn5+vDDDyUd2eMsLy9PM2fO1LJly7RkyZLj2ho4cKDmzZunKVOmNK3KfnRF9tzcXL3//vsqLCxUaWmpJKl3794nbCs9PV1r167Vtm3bNGLECI0fP14pKRw6YU4oHFKdu17tM+y9006S6tz1CoY4NkYDRwlETSQSkc8XkN/m/SOPrPQftrUP2OPDDz/U888/r4aGBkUiEdXX10uSRo4cqdmzZ+uhhx7SJZdcop/85CdRq6m4uFiSNGDAAKWkpOjgwYPq1atX1PpHYvN6vdp7cJc+WveenO3b296fu7FRew9UcIY/CghYiAqfzyevr1pfl63SgUp7z2A11DfI56s5cncO4saePXs0efJk/fOf/1S/fv20ceNGXXbZZZKkUaNGqbS0VCtWrNDbb7+t6dOnH7fPoF2O3U8xOTlZoVAoKv2ibaitrZUvUK/cPK86d7D/rvVql1/enXWqra21va+2joCFqPD7/UpPtXRevwz17G7v5NuDh8LatjPCFh5xxuVyKTU1Vd26dZNlWXrhhReaHisvL1fPnj118803a+zYserSpYvcbreys7ON3cVmsi3gTPn9fkUUUW4np7p072B7f8HkkMJMoYgKAhaiypmZpiynvbciu1zchvxd1TeY3V7i+7Q3ePBgjRs3Tueff7569+6tK6+8sumxkpIS/eEPf1BycrLC4bCeeeYZ5eTk6PLLL9esWbM0dOhQjRo1qtkk9+/KZFvAmfL5fAqGQvIHg/L67F+byh8MKhgMtqotZRIVAQtow9LT05WRmacN22v0fe76O5WMzLxml9dOxrKspo+fffZZPfvss02fT58+XZI0adKkE969l5OTo9WrV5+07TFjxjRNcJf+f5K7JOXn56uqquqUbR1bm6RmzwdM8Pv9kiUpJVVJ6fbPwVJKgyyxvEY0ELCANszpdOr2CXezkjsQYw5HkpKS7N/upTVuKZOoCFhAG+d0OglCAGAYURYAAMAwAhbQRhy9/MAyA63b0dcnGpeLANiH32CgjejcubMkqaysLMaV4FSOvj5dunSJcSUAWoI5WEAb0b17d11yySWaNm2azjrrLGVmmtvcGWZ4PB799re/1ZgxY9StW7dYlwOgBeI4YFmKhIOK2Hy5IxIOHnerNhCPkpKS9Kc//UlDhw7V6NGjY10OTiIrK0srV67kEiEQ5+IyYPl8PgWDAQUa6uWtsXcIgYZ6hYMBFmVDQujXr5+qqqr09ddfn3YdnA8//FAvvzhL1152gbp27mhrXYcqq/XeylLdc+9Duuqqq2ztSzpea0C9AAASFklEQVSy/9ufFy5WciQ6ISbdma7ri3562rOG7dq1U//+/ZWWZv+WKQDsFZcBy+/3K8mylJmapOxMe3cfb0xNlsS2AkgcaWlpGjx48Gmft3PnTmWktVPPrh3Uq6e984FSkiLKSG+nvn37atiwYbb2JR3Z97Buf5WGnTVUWZn27o3p8Xm0ce9m9erVS/n5+bb2BaD1iMuAdZTDkayU5GSb++A0PZBovF6vFApoUI8e6tnZ3vB4oLpS6/d8dWTFbgBtRlwHLABoicyMNDkz7d0bM93N3phAW0TAAtAmBUMhueob1D7V3mkGrvoGBQIBW/sA0PoQsAC0ObW1tdpRUaGSrE3Kam/vHKyGxgbtqChXZWWlzj33XFv7AtB6ELAAtDl1dXWykqTMXp3UqVsPW/uKHD6o8FbJ7Xbb2g+A1oWABaBNikQiCkUiCitsaz/BUEjhiL19AGh9CFgA2hyfzyevr1pfl63SgUqbLxHWN8jnqzly5yKANoOABaDN8fv9Sk+1dF6/DPXsnmdrXwcPhbVtZ4SJ7kAbQ8AC0GY5M9OU5bR3mQaXi2UagLaIVTQBAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhBCwAAADDCFgAAACGEbAAAAAMI2ABAAAYRsACAAAwjIAFAABgGAELAADAMAIWAACAYQQsAAAAwwhYAAAAhhGwAAAADCNgAQAAGEbAAgAAMIyABQAAYBgBCwAAwDACFgAAgGEELAAAAMMIWAAAAIYRsAAAAAwjYAEAABhGwAIAADCMgAUAAGAYAQsAAMAwAhYAAIBhxwWs5cuXq7CwUEOGDNHIkSO1YcOGWNQFAAAQt1KO/aSiokLXXnutJMmyLFmWpYKCAlVXVysvLy8mBQIAAMSbZmewKisrFYlEjjyQ9P8PbdmyJbpVAQAAxLFmZ7Cef/55SVIkEpHD4Wj6+oEDB074zW63W/X19U2fp6WlKS0tzY46AQAA4kazgPXjH/9YCxculHTkEuFRGRkZJ/zmSy65pNnn9913nx5//HHTNR6noaFBliTLiihyTJ12sKyIZFlqaGhQTU2NrX1Fc1xS4o4tmuOSEndsiTouKXHHlqjjkhJ3bBz3zYj2+/FE/nUqVbOA1adPn6aPHQ5HU8iqq6s7o8bfeOMNTZw4sYUlnl5lZWXTHLFIOGxrX5ZlKWId6bO8vNzWvqI5LilxxxbNcUmJO7ZEHZeUuGNL1HFJiTs2jvtmRPv9eCKnDFgrVqyQJPXs2VN1dXVqbGyUJK1du1bjx48/beNH529FS02NKyp9BAJ+LV26VKWlpbb2tXv3boXCoaiMS0rcsUVzXFLiji1RxyUl7tgSdVxS4o6N474Z0fr5fRfNAlZtba0kad++fc2eFAgEzqixYyfG26lbt25q9If13t+jM/n+QGWDDtdt0fbt223tJxQKKcURjNq4pMQdW7TGJSXu2BJ1XFLiji1RxyUl7tg47pvj8UfUrVu3qPR1JhzWMZOtNm3apOHDhysYDDZ70q5du9S3b9+mz9etW6fhw4dryZIlGjhwYNPXO3XqpM6dO0eh7CPrdblc0UmsO3bs0OHDh6PSV3Z2toYOHRqVvqTEHVs0xyUl7tgSdVxS4o4tUcclJe7YOO6b0adPH40cOTIqfZ2JZmewXC6XQqGQJCk1NVXBYFC9e/duFq6O1bdvXw0YMMD+Kk9gxIgRCbk2V01NTUKOS2Js8ShRxyUl7tgSdVxS4o4tUcclJfbYTqfZNb0//vGPTRPbj14WzM7Ojn5VAAAAcaxZwFqwYEHTjH/LshSJRLRx48ZY1QYAABCX2OwZAADAsLgMWH6/X0899ZT8fn+sSzEqUcclMbZ4lKjjkhJ3bIk6Lilxx5ao45ISe2xnotldhGfq6F2EX331lYYNG2ZHXadUX1+vnJwcuVyuhJojlqjjkhhbPErUcUmJO7ZEHZeUuGNL1HFJiT22MxGXZ7AAAABaMwIWAACAYSmnf8rxvF6vJGnbtm1GizlTbrdbklRaWiqn0xmTGuyQqOOSGFs8StRxSYk7tkQdl5S4Y0vUcUmJPbaTOe+885SZmSnpe87BWrRokW6//XbjhQEAAMSrY+emf6+AVVVVpQ8++ED5+fnKyMgwXiAAAEC8afEZLAAAAJwck9wBAAAMI2ABAAAYFncBa+fOnbrwwgvVv39/jRgxQlu3bo11SUZMnjxZ+fn5cjgc2rx5c6zLMcbn8+lnP/uZ+vfvr4KCAo0dO1YVFRWxLsuYq666SkOGDFFBQYFGjx6t0tLSWJdk1OOPP55w78n8/Hydd955KigoUEFBgd54441Yl2SE3+/Xfffdp3POOUfnn39+wtyIVFdX1/RaFRQUqH///kpJSVFNTU2sS2uxDz74QMOHD9cFF1ygQYMGacGCBbEuyZjly5ersLBQQ4YM0ciRI7Vhw4ZYlxR9Vpy59NJLrVdffdWyLMt66623rJEjR8a2IEP+/ve/W3v37rX69Oljbdq0KdblGOP1eq333nvPikQilmVZ1vPPP29deeWVMa7KnNra2qaP3333XeuCCy6IYTVmffXVV9bYsWOt3r17J9R7MtF+x476j//4D+v+++9v+l3bv39/jCuyxzPPPGNdd911sS6jxSKRiJWXl2dt2LDBsizLKi8vt9LS0qz6+voYV9ZyNTU1VseOHa2tW7dalmVZJSUl1vnnnx/jqqIvrs5gHT58WOvWrWv6z+zGG29UeXl5QpwRufjii9WrV69Yl2Fcenq6rrnmGjkcDknSyJEjtWvXrhhXZU5ubm7Txy6XS0lJcfUrdVJ+v1/33nuvXnrppabXDq1XY2OjXn31VT3xxBNNr1f37t1jXJU9Xn31Vd11112xLsOYuro6SUe2lenYsaPS0tJiXFHLlZWVqUuXLhowYIAk6ZJLLtHu3bu1bt26GFcWXXH112Dv3r3q0aOHUlKOrI/qcDjUu3dv7dmzJ8aV4Uw999xz+ulPfxrrMoyaMGGCzjrrLE2fPj1hTvHPmDFDt99+u/r27RvrUmxRXFyswYMH69///d9VWVkZ63JarKysTB07dtTvfvc7FRYWavTo0fr4449jXZZxn3/+uaqrq3XdddfFupQWczgcevPNN1VUVKQ+ffrooosu0oIFC5Samhrr0lrsnHPOUWVlpdasWSNJevfdd+V2uxPiZMh3EVcBS9Jx/01brDIRN5544gnt3LlT//3f/x3rUox67bXXtHfvXv3ud7/Tww8/HOtyWuzzzz/XF198oV/96lexLsUW//jHP7RhwwatW7dOHTt21B133BHrklosGAxq165dGjhwoL788ku98MILuvXWWxMiPB7rT3/6kyZMmND0T3Y8C4VCevLJJ7V06VLt3r1bH3/8se64446EmFuWk5Ojt99+W1OnTtXw4cNVUlKigQMHql27drEuLbpifY3yuzh06JCVnZ1tBYNBy7KOXMPu2rWrVV5eHtvCDErU+SHPPPOMNXz48GZzlhJRenq6VVVVFesyWuTJJ5+0unfvbvXp08fq06ePlZycbPXo0cN6//33Y12acfv377ecTmesy2ixyspKKykpyQqFQk1f++EPf2h98sknsSvKMLfbbWVlZVnbtm2LdSlGfPHFF9aAAQOafa2wsNBauXJljCqyj8/ns3Jzc62dO3fGupSoiqszWF26dNEFF1yg119/XZL09ttvKz8/X/n5+bEtDKf0hz/8QX/5y1/00UcfNZuzFO/q6+u1f//+ps/fffdddezYUXl5eTGsquWmTp2q/fv3q6KiQhUVFerVq5c++OADXX311bEurcUaGxub5rxI0l/+8hddcMEFMazIjE6dOunyyy/XBx98IEnavXu3ysvLde6558a4MnPeeustDRkyROedd16sSzHirLPO0rfffqsdO3ZIkr755huVlZWpf//+Ma7MjAMHDjR9/F//9V+67LLL9IMf/CCGFUVf3J1nnTt3riZOnKgnnnhC2dnZCTPn5d5779XSpUt18OBBXXHFFXI6nfrmm29iXVaLffvtt3rooYfUr18/XXrppZKktLQ0rV27NsaVtZzL5dKNN94or9erpKQkde7cWcuWLWNSeCt26NAh3XjjjQqHw7IsS/369dNrr70W67KMmDNnju6880795je/UXJysl5++eWEmuj+yiuvJNTk9q5du2ru3Lm66aablJSUJMuy9NJLL6lnz56xLs2I//zP/9Rnn32mUCikUaNG6ZVXXol1SVHHVjkAAACGxdUlQgAAgHhAwAIAADCMgAUAAGDY/wGvSwPte0uSZQAAAABJRU5ErkJggg==" }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# we can plot different subsets to eyeball variation\n", "histogram([train[1:8400,:label] train[8401:16800,:label] train[16801:25200,:label] train[25201:33600,:label] train[33601:end,:label]], ticks=collect(0:9), legend=true, \n", "label=[\"first fifth\" \"second fifth\" \"third fifth\" \"fourth fifth\" \"last fifth\"], alpha=0.4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you really want to check stats, you might want to do a chi-square or Kolmogorov–Smirnov test to really see if the distribution is uniform. You can do these with the [HypothesisTests](https://github.com/JuliaStats/HypothesisTests.jl) package in Julia. Documentation is at https://juliastats.org/HypothesisTests.jl/stable/. The real focus of this notebook is MLP, so let us move on and start setting it up." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Preprocess data" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "search: \u001b[0m\u001b[1mp\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1mm\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mt\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1md\u001b[22m\u001b[0m\u001b[1mi\u001b[22m\u001b[0m\u001b[1mm\u001b[22m\u001b[0m\u001b[1ms\u001b[22m \u001b[0m\u001b[1mp\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1mm\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mt\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1md\u001b[22m\u001b[0m\u001b[1mi\u001b[22m\u001b[0m\u001b[1mm\u001b[22m\u001b[0m\u001b[1ms\u001b[22m! \u001b[0m\u001b[1mP\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1mm\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mt\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1md\u001b[22mD\u001b[0m\u001b[1mi\u001b[22m\u001b[0m\u001b[1mm\u001b[22m\u001b[0m\u001b[1ms\u001b[22mArray\n", "\n" ] }, { "data": { "text/latex": [ "\\begin{verbatim}\n", "permutedims(A::AbstractArray, perm)\n", "\\end{verbatim}\n", "Permute the dimensions of array \\texttt{A}. \\texttt{perm} is a vector specifying a permutation of length \\texttt{ndims(A)}.\n", "\n", "See also: \\href{@ref}{\\texttt{PermutedDimsArray}}.\n", "\n", "\\section{Examples}\n", "\\begin{verbatim}\n", "julia> A = reshape(Vector(1:8), (2,2,2))\n", "2×2×2 Array{Int64,3}:\n", "[:, :, 1] =\n", " 1 3\n", " 2 4\n", "\n", "[:, :, 2] =\n", " 5 7\n", " 6 8\n", "\n", "julia> permutedims(A, [3, 2, 1])\n", "2×2×2 Array{Int64,3}:\n", "[:, :, 1] =\n", " 1 3\n", " 5 7\n", "\n", "[:, :, 2] =\n", " 2 4\n", " 6 8\n", "\\end{verbatim}\n", "\\rule{\\textwidth}{1pt}\n", "\\begin{verbatim}\n", "permutedims(m::AbstractMatrix)\n", "\\end{verbatim}\n", "Permute the dimensions of the matrix \\texttt{m}, by flipping the elements across the diagonal of the matrix. Differs from \\texttt{LinearAlgebra}'s \\href{@ref}{\\texttt{transpose}} in that the operation is not recursive.\n", "\n", "\\section{Examples}\n", "\\begin{verbatim}\n", "julia> a = [1 2; 3 4];\n", "\n", "julia> b = [5 6; 7 8];\n", "\n", "julia> c = [9 10; 11 12];\n", "\n", "julia> d = [13 14; 15 16];\n", "\n", "julia> X = [[a] [b]; [c] [d]]\n", "2×2 Array{Array{Int64,2},2}:\n", " [1 2; 3 4] [5 6; 7 8]\n", " [9 10; 11 12] [13 14; 15 16]\n", "\n", "julia> permutedims(X)\n", "2×2 Array{Array{Int64,2},2}:\n", " [1 2; 3 4] [9 10; 11 12]\n", " [5 6; 7 8] [13 14; 15 16]\n", "\n", "julia> transpose(X)\n", "2×2 Transpose{Transpose{Int64,Array{Int64,2}},Array{Array{Int64,2},2}}:\n", " [1 3; 2 4] [9 11; 10 12]\n", " [5 7; 6 8] [13 15; 14 16]\n", "\\end{verbatim}\n", "\\rule{\\textwidth}{1pt}\n", "\\begin{verbatim}\n", "permutedims(v::AbstractVector)\n", "\\end{verbatim}\n", "Reshape vector \\texttt{v} into a \\texttt{1 × length(v)} row matrix. Differs from \\texttt{LinearAlgebra}'s \\href{@ref}{\\texttt{transpose}} in that the operation is not recursive.\n", "\n", "\\section{Examples}\n", "\\begin{verbatim}\n", "julia> permutedims([1, 2, 3, 4])\n", "1×4 Array{Int64,2}:\n", " 1 2 3 4\n", "\n", "julia> V = [[[1 2; 3 4]]; [[5 6; 7 8]]]\n", "2-element Array{Array{Int64,2},1}:\n", " [1 2; 3 4]\n", " [5 6; 7 8]\n", "\n", "julia> permutedims(V)\n", "1×2 Array{Array{Int64,2},2}:\n", " [1 2; 3 4] [5 6; 7 8]\n", "\n", "julia> transpose(V)\n", "1×2 Transpose{Transpose{Int64,Array{Int64,2}},Array{Array{Int64,2},1}}:\n", " [1 3; 2 4] [5 7; 6 8]\n", "\\end{verbatim}\n" ], "text/markdown": [ "```\n", "permutedims(A::AbstractArray, perm)\n", "```\n", "\n", "Permute the dimensions of array `A`. `perm` is a vector specifying a permutation of length `ndims(A)`.\n", "\n", "See also: [`PermutedDimsArray`](@ref).\n", "\n", "# Examples\n", "\n", "```jldoctest\n", "julia> A = reshape(Vector(1:8), (2,2,2))\n", "2×2×2 Array{Int64,3}:\n", "[:, :, 1] =\n", " 1 3\n", " 2 4\n", "\n", "[:, :, 2] =\n", " 5 7\n", " 6 8\n", "\n", "julia> permutedims(A, [3, 2, 1])\n", "2×2×2 Array{Int64,3}:\n", "[:, :, 1] =\n", " 1 3\n", " 5 7\n", "\n", "[:, :, 2] =\n", " 2 4\n", " 6 8\n", "```\n", "\n", "---\n", "\n", "```\n", "permutedims(m::AbstractMatrix)\n", "```\n", "\n", "Permute the dimensions of the matrix `m`, by flipping the elements across the diagonal of the matrix. Differs from `LinearAlgebra`'s [`transpose`](@ref) in that the operation is not recursive.\n", "\n", "# Examples\n", "\n", "```jldoctest; setup = :(using LinearAlgebra)\n", "julia> a = [1 2; 3 4];\n", "\n", "julia> b = [5 6; 7 8];\n", "\n", "julia> c = [9 10; 11 12];\n", "\n", "julia> d = [13 14; 15 16];\n", "\n", "julia> X = [[a] [b]; [c] [d]]\n", "2×2 Array{Array{Int64,2},2}:\n", " [1 2; 3 4] [5 6; 7 8]\n", " [9 10; 11 12] [13 14; 15 16]\n", "\n", "julia> permutedims(X)\n", "2×2 Array{Array{Int64,2},2}:\n", " [1 2; 3 4] [9 10; 11 12]\n", " [5 6; 7 8] [13 14; 15 16]\n", "\n", "julia> transpose(X)\n", "2×2 Transpose{Transpose{Int64,Array{Int64,2}},Array{Array{Int64,2},2}}:\n", " [1 3; 2 4] [9 11; 10 12]\n", " [5 7; 6 8] [13 15; 14 16]\n", "```\n", "\n", "---\n", "\n", "```\n", "permutedims(v::AbstractVector)\n", "```\n", "\n", "Reshape vector `v` into a `1 × length(v)` row matrix. Differs from `LinearAlgebra`'s [`transpose`](@ref) in that the operation is not recursive.\n", "\n", "# Examples\n", "\n", "```jldoctest; setup = :(using LinearAlgebra)\n", "julia> permutedims([1, 2, 3, 4])\n", "1×4 Array{Int64,2}:\n", " 1 2 3 4\n", "\n", "julia> V = [[[1 2; 3 4]]; [[5 6; 7 8]]]\n", "2-element Array{Array{Int64,2},1}:\n", " [1 2; 3 4]\n", " [5 6; 7 8]\n", "\n", "julia> permutedims(V)\n", "1×2 Array{Array{Int64,2},2}:\n", " [1 2; 3 4] [5 6; 7 8]\n", "\n", "julia> transpose(V)\n", "1×2 Transpose{Transpose{Int64,Array{Int64,2}},Array{Array{Int64,2},1}}:\n", " [1 3; 2 4] [5 7; 6 8]\n", "```\n" ], "text/plain": [ "\u001b[36m permutedims(A::AbstractArray, perm)\u001b[39m\n", "\n", " Permute the dimensions of array \u001b[36mA\u001b[39m. \u001b[36mperm\u001b[39m is a vector specifying a permutation\n", " of length \u001b[36mndims(A)\u001b[39m.\n", "\n", " See also: \u001b[36mPermutedDimsArray\u001b[39m.\n", "\n", "\u001b[1m Examples\u001b[22m\n", "\u001b[1m ≡≡≡≡≡≡≡≡≡≡\u001b[22m\n", "\n", "\u001b[36m julia> A = reshape(Vector(1:8), (2,2,2))\u001b[39m\n", "\u001b[36m 2×2×2 Array{Int64,3}:\u001b[39m\n", "\u001b[36m [:, :, 1] =\u001b[39m\n", "\u001b[36m 1 3\u001b[39m\n", "\u001b[36m 2 4\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m [:, :, 2] =\u001b[39m\n", "\u001b[36m 5 7\u001b[39m\n", "\u001b[36m 6 8\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> permutedims(A, [3, 2, 1])\u001b[39m\n", "\u001b[36m 2×2×2 Array{Int64,3}:\u001b[39m\n", "\u001b[36m [:, :, 1] =\u001b[39m\n", "\u001b[36m 1 3\u001b[39m\n", "\u001b[36m 5 7\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m [:, :, 2] =\u001b[39m\n", "\u001b[36m 2 4\u001b[39m\n", "\u001b[36m 6 8\u001b[39m\n", "\n", " ────────────────────────────────────────────────────────────────────────────\n", "\n", "\u001b[36m permutedims(m::AbstractMatrix)\u001b[39m\n", "\n", " Permute the dimensions of the matrix \u001b[36mm\u001b[39m, by flipping the elements across the\n", " diagonal of the matrix. Differs from \u001b[36mLinearAlgebra\u001b[39m's \u001b[36mtranspose\u001b[39m in that the\n", " operation is not recursive.\n", "\n", "\u001b[1m Examples\u001b[22m\n", "\u001b[1m ≡≡≡≡≡≡≡≡≡≡\u001b[22m\n", "\n", "\u001b[36m julia> a = [1 2; 3 4];\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> b = [5 6; 7 8];\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> c = [9 10; 11 12];\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> d = [13 14; 15 16];\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> X = [[a] [b]; [c] [d]]\u001b[39m\n", "\u001b[36m 2×2 Array{Array{Int64,2},2}:\u001b[39m\n", "\u001b[36m [1 2; 3 4] [5 6; 7 8]\u001b[39m\n", "\u001b[36m [9 10; 11 12] [13 14; 15 16]\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> permutedims(X)\u001b[39m\n", "\u001b[36m 2×2 Array{Array{Int64,2},2}:\u001b[39m\n", "\u001b[36m [1 2; 3 4] [9 10; 11 12]\u001b[39m\n", "\u001b[36m [5 6; 7 8] [13 14; 15 16]\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> transpose(X)\u001b[39m\n", "\u001b[36m 2×2 Transpose{Transpose{Int64,Array{Int64,2}},Array{Array{Int64,2},2}}:\u001b[39m\n", "\u001b[36m [1 3; 2 4] [9 11; 10 12]\u001b[39m\n", "\u001b[36m [5 7; 6 8] [13 15; 14 16]\u001b[39m\n", "\n", " ────────────────────────────────────────────────────────────────────────────\n", "\n", "\u001b[36m permutedims(v::AbstractVector)\u001b[39m\n", "\n", " Reshape vector \u001b[36mv\u001b[39m into a \u001b[36m1 × length(v)\u001b[39m row matrix. Differs from\n", " \u001b[36mLinearAlgebra\u001b[39m's \u001b[36mtranspose\u001b[39m in that the operation is not recursive.\n", "\n", "\u001b[1m Examples\u001b[22m\n", "\u001b[1m ≡≡≡≡≡≡≡≡≡≡\u001b[22m\n", "\n", "\u001b[36m julia> permutedims([1, 2, 3, 4])\u001b[39m\n", "\u001b[36m 1×4 Array{Int64,2}:\u001b[39m\n", "\u001b[36m 1 2 3 4\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> V = [[[1 2; 3 4]]; [[5 6; 7 8]]]\u001b[39m\n", "\u001b[36m 2-element Array{Array{Int64,2},1}:\u001b[39m\n", "\u001b[36m [1 2; 3 4]\u001b[39m\n", "\u001b[36m [5 6; 7 8]\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> permutedims(V)\u001b[39m\n", "\u001b[36m 1×2 Array{Array{Int64,2},2}:\u001b[39m\n", "\u001b[36m [1 2; 3 4] [5 6; 7 8]\u001b[39m\n", "\u001b[36m \u001b[39m\n", "\u001b[36m julia> transpose(V)\u001b[39m\n", "\u001b[36m 1×2 Transpose{Transpose{Int64,Array{Int64,2}},Array{Array{Int64,2},1}}:\u001b[39m\n", "\u001b[36m [1 3; 2 4] [5 7; 6 8]\u001b[39m" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "? permutedims" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 1.337898 seconds (6.89 k allocations: 502.799 MiB, 54.34% gc time)\n" ] }, { "data": { "text/plain": [ "42000×784 Array{Int64,2}:\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " ⋮ ⋮ ⋮ ⋱ ⋮ ⋮ \n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time a = convert(Matrix, train[:,2:end])" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.134896 seconds (7.85 k allocations: 453.027 KiB)\n" ] }, { "data": { "text/plain": [ "784×42000 LinearAlgebra.Transpose{Int64,Array{Int64,2}}:\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " ⋮ ⋮ ⋮ ⋱ ⋮ ⋮ \n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time transpose(a)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "transpose(a) == permutedims(a)" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.419683 seconds (6 allocations: 251.221 MiB, 1.94% gc time)\n" ] }, { "data": { "text/plain": [ "784×42000 Array{Int64,2}:\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " ⋮ ⋮ ⋮ ⋱ ⋮ ⋮ \n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time permutedims(a)" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "ename": "MethodError", "evalue": "MethodError: no method matching Array(::DataFrame)\nClosest candidates are:\n Array(!Matched::LinearAlgebra.SymTridiagonal) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/tridiag.jl:111\n Array(!Matched::LinearAlgebra.Tridiagonal) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/tridiag.jl:489\n Array(!Matched::LinearAlgebra.AbstractTriangular) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/triangular.jl:106\n ...", "output_type": "error", "traceback": [ "MethodError: no method matching Array(::DataFrame)\nClosest candidates are:\n Array(!Matched::LinearAlgebra.SymTridiagonal) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/tridiag.jl:111\n Array(!Matched::LinearAlgebra.Tridiagonal) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/tridiag.jl:489\n Array(!Matched::LinearAlgebra.AbstractTriangular) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/triangular.jl:106\n ...", "", "Stacktrace:", " [1] top-level scope at In[41]:1" ] } ], "source": [ "# need to split train into training and eval sets\n", "# convert DataFrame to an Array, remembering Julia is column-major like R and Matlab (unlike Python)\n", "X = transpose(Array(train[:,2:end]))\n", "y = Array(train[:,1])" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "(784,42000)" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "size(X)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "42000" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "N = size(X)[2]" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "42000" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "length(y)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "(0,255)" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "extrema(X)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "33.408911169825075" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mean(X)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "# scale X to range 0-1\n", "X = X./255;" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Split data" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "784×8400 Array{Float64,2}:\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " ⋮ ⋮ ⋱ ⋮ \n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n", " 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# we need to split the train data into a training set (cv_X) and an eval set (eval_X)\n", "split = 0.8\n", "cv_X = X[:,1:floor(Int,split*N)]\n", "eval_X = X[:,floor(Int,split*N)+1:N]" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "8400-element Array{Int64,1}:\n", " 0\n", " 7\n", " 7\n", " 2\n", " 2\n", " 6\n", " 5\n", " 7\n", " 8\n", " 5\n", " 3\n", " 0\n", " 2\n", " ⋮\n", " 0\n", " 5\n", " 3\n", " 1\n", " 9\n", " 6\n", " 4\n", " 0\n", " 1\n", " 7\n", " 6\n", " 9" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cv_y = y[1:floor(Int,split*N)]\n", "eval_y = y[floor(Int,split*N)+1:N]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Setup providers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we get to the heart of this notebook: setting up the problem in MXNet. Full documentation for the Julia `MXNet.jl` library is available at http://dmlc.ml/MXNet.jl/latest/. Here is the big picture:\n", "\n", "Neural networks in MXNet are set up as Models. Presently there are only Feedforward models that you define with the [Symbolic API](http://dmlc.ml/MXNet.jl/latest/api/symbolic-node/). The model is fed data through a Data Provider. The `fit()` or `train()` function trains a Model with a Data Provider with a chosen Optimizer, EvalMetric, and Initializer. \n", "\n", "KVStore which is a system to allow synchronization of data across different devices such as different CPUs or GPUs on the same or different machines. It automatically attempts to parallelize any operations that can be split for faster performance. This happens under the hood so you usually don't have to worry about it unless you have a complicated setup.\n", "\n", "Let's see how this all works in a basic MLP." ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "ename": "ArgumentError", "evalue": "ArgumentError: Package MXNet not found in current path:\n- Run `import Pkg; Pkg.add(\"MXNet\")` to install the MXNet package.\n", "output_type": "error", "traceback": [ "ArgumentError: Package MXNet not found in current path:\n- Run `import Pkg; Pkg.add(\"MXNet\")` to install the MXNet package.\n", "", "Stacktrace:", " [1] require(::Module, ::Symbol) at ./loading.jl:823", " [2] top-level scope at In[39]:1" ] } ], "source": [ "using MXNet" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "1000" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "batch_size = 1000" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "MXNet sets up a `AbstractDataProvider` class that helps abstract the data retrieval for the Executor from the KVStore. (I hope that is not too abstract). There are a few predefined concrete classes including one that specifically provides the MNIST dataset, appropriately named [`MNISTIter()`](http://dmlc.ml/MXNet.jl/latest/api/io/#MXNet.mx.MNISTIter-Tuple{}) or `MNISTProvider`. I want to demonstrate a more general case, so I will be using the ArrayDataProvider with an Array filled from the Kaggle file above. Documentation is at http://dmlc.ml/MXNet.jl/latest/api/io/#MXNet.mx.ArrayDataProvider, but briefly, parameters are:\n", "\n", "`ArrayDataProvider(data[, label]; batch_size, shuffle)`\n", "\n", "Shuffle is a boolean to shuffle the data each different epoch of training.\n", "\n", "Note that although the Executor can process in parallel, access to a DataProvider can **NOT** do so. _Don't_ call a DataProvider from more than one place. " ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "MXNet.mx.ArrayDataProvider(Array{Float32,N}[\n", "Float32[0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]],Symbol[:data],Array{Float32,N}[\n", "Float32[1.0 0.0 … 2.0 2.0]],Symbol[:softmax_label],1000,33600,true,0.0f0,0.0f0,MXNet.mx.NDArray[mx.NDArray{Float32}(784,1000)],MXNet.mx.NDArray[mx.NDArray{Float32}(1000,)])" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# since this is a training set, we specify shuffle to help randomize the training\n", "train_provider = mx.ArrayDataProvider(cv_X, cv_y, batch_size=batch_size, shuffle=true)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "MXNet.mx.ArrayDataProvider(Array{Float32,N}[\n", "Float32[0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]],Symbol[:data],Array{Float32,N}[\n", "Float32[0.0 7.0 … 6.0 9.0]],Symbol[:softmax_label],1000,8400,false,0.0f0,0.0f0,MXNet.mx.NDArray[mx.NDArray{Float32}(784,1000)],MXNet.mx.NDArray[mx.NDArray{Float32}(1000,)])" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# we don't want to randomize the eval set during training, so shuffle is false\n", "eval_provider = mx.ArrayDataProvider(eval_X, eval_y, batch_size=batch_size, shuffle=false)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# MLP" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At this point, we want to define the model by specifying the number of nodes in each layer and how they are connected to each other. Looking at the paper LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324. The article is available on ResearchGate at https://www.researchgate.net/profile/Yann_Lecun/publication/2985446_Gradient-based_learning_applied_to_document_recognition/links/0deec519dfa1983fc2000000/Gradient-based-learning-applied-to-document-recognition.pdf. They compared a one layer nets (28x28-300-10 and 28x28-1000-10) which had error rates of 4.7% and 4.5% on MNIST to two layer net (28x28-300-100-10) with error rate of 3.05% and their preference, a convolutional net with rate of 0.95%. The advantage of the ConvNet is that it captures more spatial relations without having to make the network 5 layers deep. I look at the [ConvNet](mnistLenet.ipynb) in a separate notebook. For now, let's try creating the first two fully-connected nets.\n", "\n", "For this network, we need to create a `FullyConnected` SymbolicNode for our hidden layer/s which will set up all the weights and a layer bias if desired. We will also need an `Activation` SymbolicNode so each node can do the job of separation. The `Activation()` function can specify an act_type = ’relu’, ’sigmoid’, ’softrelu’, or ’tanh’. There are also specific `SoftmaxActivation()` and `LeakyReLU()` functions the later of which has options for act_type = ’elu’, ’leaky’, ’prelu’, or ’rrelu’. These are all described in the Wikipedia article on [Activation Functions](https://en.wikipedia.org/wiki/Activation_function). \n", "\n", "Besides the hidden layer/s, we need input and output. The input is labeled with the `Variable()` function and will be taken from the `DataProvider` specified in the `fit()` instruction. We also need to create a `SymbolicNode` for the output layer. The MXNet API provides several choices including `LinearRegressionOutput`, `LogisticRegressionOutput`, `MAERegressionOutput` (mean absolute error), and `SoftmaxOutput`. The layers are connected by specifying the appropriate input data for each function." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "MXNet.mx.SymbolicNode(MXNet.mx.MX_SymbolHandle(Ptr{Void} @0x000000001962b220))" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = mx.Variable(:data)\n", "fc1 = mx.FullyConnected(data, name=:fc1, num_hidden=300)\n", "act1 = mx.Activation(fc1, name=:tanh1, act_type=:tanh)\n", "fc2 = mx.FullyConnected(act1, name=:fc2, num_hidden=10)\n", "mlp1 = mx.SoftmaxOutput(fc2, name=:softmax)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Display Model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`MXNet.jl` has a `to_graphviz()` function to produce a `.dot` file from a model. Unfortunately, I don't have a way to directly display `dot` output in Jupyter, so we will save as a `dot` file, convert to `png` and insert that into the notebook. Of course if you are playing with this yourself, you can use `dot` file viewer directly." ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "# save file - note that the `do` block automatically closes the filestream\n", "open(\"mlp1graph.dot\", \"w\") do fs\n", " print(fs, mx.to_graphviz(mlp1))\n", "end" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true } }, "outputs": [], "source": [ "# we can run the `dot` program to convert to png if it is installed on your computer\n", "run(pipeline(`dot -Tpng mlp1graph.dot`, stdout=\"mlp1graph.png\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can display this file:\n", "![mlp1graph.png](mlp1graph.png)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "MXNet.mx.FeedForward(MXNet.mx.SymbolicNode(MXNet.mx.MX_SymbolHandle(Ptr{Void} @0x000000001962b220)),MXNet.mx.Context[CPU0],#undef,#undef,#undef)" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# change context to gpu(number) if you have a gpu and want to use that for processing\n", "model = mx.FeedForward(mlp1, context=mx.cpu())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have our model and data, we need to choose an Optimizer to do backprop. For more background on how [backpropagation](https://en.wikipedia.org/wiki/Backpropagation) works to use an error function to adjust network weights, check out the Wikipedia article. To learn about optimization, read the Wikipedia article on [Stochastic Gradient Descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent). It describes several variants, including AdaGrad, RMSProp,and Adam.\n", "\n", "A lot of different optimizer options are available in MXNet. These include `SGD` (Stochastic Gradient Descent), `AdaGrad` (Adaptive Gradient), `RMSProp` (Root Mean Square Propagation) and its variants `ADAM` (Adaptive Moment Estimation), `AdaMax`, `Nadam` (Nesterov Adam). \n", "\n", "Documentation is available at http://dmlc.ml/MXNet.jl/latest/api/optimizer/#built-in-optimizers, and it includes references to the source papers." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "MXNet.mx.SGD(MXNet.mx.SGDOptions(0.1,0.9,0,1.0e-5,MXNet.mx.LearningRate.Fixed(0.1),MXNet.mx.Momentum.Fixed(0.9)),#undef)" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# we are going to use the basic Stochastic Gradient Descent optimizer with a fixed learning rate and momentum\n", "optimizer = mx.SGD(lr=0.1, momentum=0.9, weight_decay=0.00001)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Examine initial state" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At this point, we start training ith the data providers, model and optimizer set up above using the `fit()` function. We can specify the number of epochs to run. At this point, we will start with just one epoch, and see the output that is automatically generated. The first `fit()` will set up all the memory structures for the model, initialize the weighs, create the KVStore, and start training. This first `fit()` will also take longer than subsequent ones as Julia does Just In Time compilation of code using LLVM. Subsequent calls will not have to be compiled.\n", "\n", "Note that `fit()` also allows the choice of other evaluation metrics, other initializers, and setting up callback routines between epochs. Full documentation is available at http://dmlc.ml/MXNet.jl/latest/api/model/#MXNet.mx.fit-Tuple{MXNet.mx.FeedForward,MXNet.mx.AbstractOptimizer,MXNet.mx.AbstractDataProvider}. Here, I am just using the default metric (`Accuracy()`) and defualt initializer (`UniformInitializer(0.01)`). The later will intialize the weights of the network to uniformly distributed random numbers less that .01. " ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: Start training on MXNet.mx.Context[CPU0]\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Initializing parameters...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Creating KVStore...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: TempSpace: Total 4 MB allocated on CPU0\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Start training...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 001/001 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.6336\n", "\u001b[0m" ] }, { "name": "stdout", "output_type": "stream", "text": [ " 32.429736 seconds (7.59 M allocations: 540.899 MB, 2.80% gc time)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: time = 11.8315 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.7843\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Finish training on MXNet.mx.Context[CPU0]\n", "\u001b[0m" ] } ], "source": [ "@time mx.fit(model, optimizer, train_provider, eval_data=eval_provider, n_epoch=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I am taking a break in training here so we can take a quick peek at the initial internals of the network. Note all the parameters of the model are stored as MXNet.NDArrays which need to be `copy` to memory in an Array to play with them. Although the model is defined earlier, no memory is assigned until the `fit()` function is executed, so if you try to access the `arg_params` of the model before that, you will get `UndefRefError: access to undefined reference`." ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "Dict{Symbol,MXNet.mx.NDArray} with 4 entries:\n", " :fc1_weight => mx.NDArray{Float32}(784,300)\n", " :fc1_bias => mx.NDArray{Float32}(300,)\n", " :fc2_weight => mx.NDArray{Float32}(300,10)\n", " :fc2_bias => mx.NDArray{Float32}(10,)" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# note the input layer has 784*300 = 235,200 trainable weights stored in a NDArray. The output layer has 3000 weights.\n", "model.arg_params" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "300×10 Array{Float32,2}:\n", " -0.127887 0.0771416 0.0249448 … 0.145933 0.010249 \n", " -0.0196595 -0.0296244 0.0209594 -0.0811309 -0.0449037\n", " 0.0956391 -0.128065 -0.103133 0.0358045 -0.0394614\n", " -0.107717 0.0242749 0.0551586 0.0597582 -0.0298745\n", " -0.0984035 0.0960803 0.115726 -0.0061346 0.0479453\n", " -0.0183465 -0.207955 -0.000346427 … 0.0321886 0.0209672\n", " 0.120342 -0.122031 -0.024811 0.0252119 -0.0228585\n", " -0.0212993 0.0206527 0.0558286 -0.0572732 -0.0384446\n", " 0.0798484 -0.0635753 0.0840668 0.0380003 -0.0590102\n", " 0.0820934 0.0308938 0.12895 0.0494668 -0.120226 \n", " 0.145466 -0.128332 -0.110208 … 0.0314653 0.0496928\n", " 0.0384468 0.168292 -0.0586571 -0.0639272 -0.044879 \n", " 0.0877501 -0.0616897 -0.141263 0.0441554 0.0759257\n", " ⋮ ⋱ \n", " 0.0791338 -0.226709 -0.0443711 -0.0835878 0.0361731\n", " 0.0163245 0.045172 0.0819094 0.0264293 -0.0304361\n", " -0.0624429 -0.0678274 -0.0740325 … -0.0847413 0.0844292\n", " 0.0406324 0.0564422 -0.106171 0.0146266 0.0337845\n", " -0.0579353 0.0977567 -0.0983308 -0.112232 0.0105776\n", " -0.0233625 -0.0358019 -0.0986435 -0.0249763 0.0135266\n", " 0.0388482 0.0118654 -0.16399 0.0670357 0.120794 \n", " -0.0115069 0.0122509 -0.0674744 … 0.0703299 0.119037 \n", " 0.191457 -0.0999065 0.00194451 -0.0827886 -0.032837 \n", " 0.0187783 -0.0221313 -0.0729263 0.008342 0.0182222\n", " 0.00468942 -0.175613 0.0360089 0.0626512 0.0410399\n", " 0.124774 -0.122346 0.0201343 0.0144864 -0.0635198" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# let's examine the weights in the hidden layer\n", "w2 = copy(model.arg_params[:fc2_weight])" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "10×3 Array{Any,2}:\n", " (-0.280105,0.208179) 0.0080839 0.0957205\n", " (-0.267891,0.284985) 0.000389835 0.103438 \n", " (-0.202828,0.246249) -0.00487307 0.0891263\n", " (-0.212816,0.190532) 0.003161 0.0817025\n", " (-0.22252,0.225273) -0.00620983 0.0842009\n", " (-0.202098,0.172084) 0.00473183 0.0636334\n", " (-0.238122,0.210844) -0.00532095 0.0923733\n", " (-0.254803,0.263656) 0.00346573 0.0984543\n", " (-0.180616,0.145933) -0.00335081 0.0618578\n", " (-0.194464,0.198503) -0.000115971 0.0770083" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# let's get the extrema mean and standard deviation for each of the 10 groups\n", "[ [extrema(w2[:,i]) for i in 1:10] [mean(w2[:,i]) for i in 1:10] [std(w2[:,i]) for i in 1:10] ]" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# plot mean and std\n", "plot([ [mean(w2[:,i]) for i in 1:10] [std(w2[:,i]) for i in 1:10] ], legend=true, label=[\"mean\" \"std\"])" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# plot them as 30x10 arrays\n", "heatmap(reshape(w2[:,1], 10,30), aspect_ratio=:equal)" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "Plots.PyPlotBackend()" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# switch to PyPlot because Plotly doesn't support clims attribute and I will want to compare later\n", "pyplot()" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "plot([heatmap(reshape(w2[:,i], 10,30), aspect_ratio=:equal, clims=(-0.6,0.6)) for i=1:10]...)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train model 10 epochs " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok. Let's run it for a few epochs and watch what happens. Note that every time we call `fit()`, the default is to continue training the same model without resetting the weights. " ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: Start training on MXNet.mx.Context[CPU0]\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Initializing parameters...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Creating KVStore...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: TempSpace: Total 4 MB allocated on CPU0\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Start training...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 001/009 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.8643\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 9.1088 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.8391\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 002/009 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9066\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.3779 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9173\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 003/009 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9159\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.7162 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9248\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 004/009 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9223\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.3265 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9298\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 005/009 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9274\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.3956 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9319\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 006/009 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9330\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.5606 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9370\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 007/009 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9363\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 9.2693 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9400\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 008/009 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9426\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 9.4478 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9428\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 009/009 ==========\n", "\u001b[0m" ] } ], "source": [ "@time mx.fit(model, optimizer, train_provider, eval_data=eval_provider, n_epoch=9)" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "300×10 Array{Float32,2}:\n", " -0.18999 0.0523741 0.0521278 … 0.256539 0.0830147 \n", " -0.0566305 -0.0292824 0.000472053 -0.176414 -0.137913 \n", " -0.0510069 -0.15316 -0.15406 -0.00278085 -0.175738 \n", " -0.160078 0.0158039 0.0645762 0.101266 0.0279298 \n", " -0.121867 0.126162 0.203941 -0.0970455 0.1576 \n", " -0.27373 -0.234331 0.03053 … -0.02137 0.0339543 \n", " 0.165604 -0.171726 -0.0604989 0.0526133 -0.0206003 \n", " -0.0281766 0.0293302 0.0430013 -0.0955367 -0.137775 \n", " 0.126382 -0.076785 0.10942 0.0555131 -0.0920628 \n", " 0.128483 0.0501477 0.136502 0.0789185 -0.133262 \n", " 0.211869 -0.153832 -0.119128 … 0.0677559 0.0706252 \n", " 0.0836029 0.188536 -0.144794 -0.05817 -0.049355 \n", " 0.156777 -0.0915638 -0.21475 0.0954725 0.209062 \n", " ⋮ ⋱ \n", " 0.085546 -0.250519 -0.0668047 -0.137696 -0.00329412\n", " 0.00653774 0.0642993 0.191455 -0.0250502 0.0283506 \n", " -0.0594797 -0.109547 -0.072032 … -0.111813 0.00390982\n", " 0.0777404 0.0444848 -0.131087 0.0178672 0.0707252 \n", " -0.118424 0.150786 -0.147908 -0.172449 -0.0152548 \n", " -0.0545296 -0.045317 -0.174648 -0.0304535 -0.048231 \n", " 0.066315 0.0347018 -0.228092 0.080211 0.342091 \n", " 0.0115476 0.0107955 -0.0769882 … 0.116553 0.283723 \n", " 0.257533 -0.133712 0.104216 -0.208887 -0.0636919 \n", " 0.0174476 -0.00290674 -0.132057 0.01859 -0.0688223 \n", " -0.0502207 -0.193162 0.0843327 0.113111 0.0176484 \n", " 0.11381 -0.142134 0.0347138 0.00476562 -0.157754 " ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# let's examine how the weights in the hidden layer have changed after 10 more epochs of training\n", "w211 = copy(model.arg_params[:fc2_weight])" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "10×3 Array{Any,2}:\n", " (-0.357168,0.36403) 0.00906962 0.144172\n", " (-0.291686,0.346591) 0.000505016 0.124114\n", " (-0.35688,0.567968) -0.00167273 0.145727\n", " (-0.354625,0.29979) 0.0025032 0.126793\n", " (-0.335417,0.335986) -0.0121043 0.133443\n", " (-0.61578,0.406012) 0.0105811 0.144681\n", " (-0.34189,0.273747) -0.00809373 0.127449\n", " (-0.443309,0.390159) 0.00113069 0.145592\n", " (-0.423108,0.589517) -0.00721869 0.125767\n", " (-0.339918,0.404306) 0.00526154 0.132497" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# extrema mean and standard deviation for each of the 10 groups\n", "[ [extrema(w211[:,i]) for i in 1:10] [mean(w211[:,i]) for i in 1:10] [std(w211[:,i]) for i in 1:10] ]" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# plot comparison to previous \n", "plot([ [mean(w2[:,i]) for i in 1:10] [std(w2[:,i]) for i in 1:10] [mean(w211[:,i]) for i in 1:10] [std(w211[:,i]) for i in 1:10] ], \n", "label=[\"mean 1\" \"std 1\" \"mean 10\" \"std 10\"], legend=true)" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "plot([heatmap(reshape(w211[:,i], 10,30), aspect_ratio=:equal, clims=(-0.6,0.6)) for i=1:10]...)" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: TempSpace: Total 1 MB allocated on CPU0\n", "\u001b[0m" ] } ], "source": [ "# test on eval set\n", "preds = mx.predict(model, eval_provider)\n", "correct = 0\n", "for i = 1:size(preds)[2]\n", " if indmax(preds[:,i]) == eval_y[i]+1\n", " correct += 1\n", " end\n", "end\n", "correct/size(preds)[2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The accuracy already seems to be stabilizing around 95%, but we will run a few more epochs to see what happens" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train model 20 epochs " ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: Start training on MXNet.mx.Context[CPU0]\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Initializing parameters...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Creating KVStore...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: TempSpace: Total 4 MB allocated on CPU0\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Start training...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 001/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9494\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.3934 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9477\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 002/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9522\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.7644 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9524\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 003/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9553\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.9129 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9527\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 004/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9586\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.4194 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9558\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 005/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9613\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.2062 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9571\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 006/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9637\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 7.6110 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9586\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 007/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9666\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.5968 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9579\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 008/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9676\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.1803 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9600\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 009/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9694\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.5670 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9596\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 010/010 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9716\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 8.7106 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m" ] }, { "name": "stdout", "output_type": "stream", "text": [ " 93.436206 seconds (3.92 M allocations: 2.322 GB, 6.35% gc time)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: accuracy = 0.9629\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Finish training on MXNet.mx.Context[CPU0]\n", "\u001b[0m" ] } ], "source": [ "@time mx.fit(model, optimizer, train_provider, eval_data=eval_provider, n_epoch=10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It looks like it might have improved to 97%, but that is on the eval set. Let's go ahead and make predictions with the model we have trained using the Kaggle test set and create a file that can be submitted on the website. Note that running a prediction is a lot faster than training." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## run on test set" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 26" ] } ], "source": [ "@time test = readtable(\"data/test.csv\");" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "(28000,784)" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "size(test)" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "784×28000 Array{Int64,2}:\n", " 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " ⋮ ⋮ ⋮ ⋱ ⋮ ⋮ \n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_X = transpose(Array(test))" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "plot([heatmap(rotl90(reshape(Array(test_X[1:end,i]), 28, 28)), aspect_ratio=:equal) for i=1:16]...)" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "(0,255)" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "extrema(test_X)" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "33.3515450983965" ] }, "execution_count": 55, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mean(test_X)" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "# scale test_X to range 0-1\n", "test_X = test_X./255;" ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "MXNet.mx.ArrayDataProvider(Array{Float32,N}[\n", "Float32[0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]],Symbol[:data],Array{Float32,N}[],Symbol[],1000,28000,false,0.0f0,0.0f0,MXNet.mx.NDArray[mx.NDArray{Float32}(784,1000)],MXNet.mx.NDArray[])" ] }, "execution_count": 57, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_provider = mx.ArrayDataProvider(test_X, batch_size=batch_size, shuffle=false)" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: TempSpace: Total 1 MB allocated on CPU0\n", "\u001b[0m" ] }, { "name": "stdout", "output_type": "stream", "text": [ " 4.253657 seconds (20.44 k allocations: 86.709 MB, 10.64% gc time)\n" ] }, { "data": { "text/plain": [ "10×28000 Array{Float32,2}:\n", " 3.32992f-5 0.998368 1.43607f-5 … 1.062f-6 2.4091f-5 \n", " 1.16891f-10 3.90927f-8 0.000117007 1.37874f-8 2.84631f-9 \n", " 0.999818 0.000398155 7.91866f-5 5.73282f-7 0.999624 \n", " 0.00013095 1.55586f-5 0.000858953 0.00192545 0.000157291\n", " 7.27301f-7 1.95835f-9 0.0574737 0.0149685 6.15969f-5 \n", " 2.27206f-7 0.00120688 0.00746393 … 0.000177767 1.21758f-5 \n", " 2.64876f-7 7.05455f-6 1.17546f-5 1.13237f-7 5.25664f-6 \n", " 8.56755f-7 3.20001f-6 0.00172127 0.000133627 7.38207f-8 \n", " 1.45219f-5 1.30237f-6 0.0118794 7.05102f-5 0.000103447\n", " 1.1988f-6 2.87172f-7 0.92038 0.982722 1.20239f-5 " ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# this uses the previously trained model and the DataProvider specified above\n", "@time tpreds = mx.predict(model, test_provider)" ] }, { "cell_type": "code", "execution_count": 59, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "# create submission\n", "open(\"MLP1submission.csv\", \"w\") do f\n", " write(f, \"ImageId,Label\\n\")\n", " for i = 1:size(tpreds)[2]\n", " write(f, string(i),\",\",string(indmax(tpreds[:,i])-1),\"\\n\")\n", " end\n", "end" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When I submit the resulting file to https://kaggle.com/c/digit-recognizer/submit, I get a score of 0.95200. which is pretty close to the error rate of 4.7% reported above for this network. The validation accuracy was just beginning to plateau so it isn't clear if we could get a little more out of training, but I'm more interested here in demonstrating creating the two layer network. We will duplicate the 300-100 net that got an error rate of 4.5%." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# MLP - two layer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we define a new, two layer network with two FullyConnected Layers and tanh activations. We could use code similar to last time:\n", "```\n", "data = mx.Variable(:data)\n", "fc1 = mx.FullyConnected(data, name=:fc1, num_hidden=300)\n", "act1 = mx.Activation(fc1, name=:relu1, act_type=:relu)\n", "fc2 = mx.FullyConnected(act1, name=:fc2, num_hidden=100)\n", "act2 = mx.Activation(fc2, name=:relu2, act_type=:relu)\n", "fc3 = mx.FullyConnected(act2, name=:fc3, num_hidden=10)\n", "mlp2 = mx.SoftmaxOutput(fc3, name=:softmax)\n", "```\n", "but, to demonstrate a shortcut, we will use the MXNet chain macro to create this net using `=>` connections." ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "MXNet.mx.SymbolicNode(MXNet.mx.MX_SymbolHandle(Ptr{Void} @0x000000001cfa0a10))" ] }, "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# define new network\n", "mlp2 = @mx.chain mx.Variable(:data) => \n", " mx.FullyConnected(name=:fc1, num_hidden=300) =>\n", "mx.Activation(name=:tanh1, act_type=:tanh) =>\n", " mx.FullyConnected(name=:fc2, num_hidden=100) =>\n", "mx.Activation(name=:tanh2, act_type=:tanh) =>\n", " mx.FullyConnected(name=:fc3, num_hidden=10) =>\n", " mx.SoftmaxOutput(name=:softmax)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Display model" ] }, { "cell_type": "code", "execution_count": 61, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "# save file - note that the `do` block automatically closes the filestream\n", "open(\"mlp2graph.dot\", \"w\") do fs\n", " print(fs, mx.to_graphviz(mlp2))\n", "end" ] }, { "cell_type": "code", "execution_count": 62, "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true } }, "outputs": [], "source": [ "# we can run the `dot` program to convert to png if it is installed on your computer\n", "run(pipeline(`dot -Tpng mlp2graph.dot`, stdout=\"mlp2graph.png\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can display this file:\n", "![mlp2graph.png](mlp2graph.png)" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "MXNet.mx.FeedForward(MXNet.mx.SymbolicNode(MXNet.mx.MX_SymbolHandle(Ptr{Void} @0x000000001cfa0a10)),MXNet.mx.Context[CPU0],#undef,#undef,#undef)" ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# change context to gpu(number) if you have a gpu\n", "model = mx.FeedForward(mlp2, context=mx.cpu())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train model" ] }, { "cell_type": "code", "execution_count": 64, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: Start training on MXNet.mx.Context[CPU0]\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Initializing parameters...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Creating KVStore...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: TempSpace: Total 4 MB allocated on CPU0\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Start training...\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 001/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.1096\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 11.8936 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.1056\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 002/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.3144\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.0328 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.5613\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 003/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.7609\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.0750 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.8576\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 004/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.8721\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 11.7286 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.8989\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 005/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.8987\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.1276 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9143\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 006/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9137\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.9629 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9227\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 007/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9231\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.6956 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9293\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 008/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9311\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 13.9586 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9364\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 009/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9381\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 14.6798 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9411\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 010/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9443\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.7957 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9454\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 011/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9491\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.6748 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9521\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 012/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9544\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 13.5925 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9556\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 013/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9591\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 13.5508 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9570\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 014/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9620\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.6269 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9554\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 015/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9640\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 12.5352 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9579\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 016/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9683\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 11.9030 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9607\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 017/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9711\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 13.4604 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9604\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 018/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9729\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 11.2488 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9611\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 019/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9736\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 13.9827 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9648\n", "\u001b[0m\u001b[1m\u001b[34mINFO: == Epoch 020/020 ==========\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Training summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9768\n", "\u001b[0m\u001b[1m\u001b[34mINFO: time = 13.0553 seconds\n", "\u001b[0m\u001b[1m\u001b[34mINFO: ## Validation summary\n", "\u001b[0m\u001b[1m\u001b[34mINFO: accuracy = 0.9644\n", "\u001b[0m\u001b[1m\u001b[34mINFO: Finish training on MXNet.mx.Context[CPU0]\n", "\u001b[0m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "282.506040 seconds (8.59 M allocations: 4.680 GB, 4.63% gc time)\n" ] } ], "source": [ "@time mx.fit(model, optimizer, train_provider, eval_data=eval_provider, n_epoch=20)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After 20 epochs of training, we seem to be hitting a ceiling on validation accuracy. Let's use this newly trained model to predict on the test set and submit to Kaggle." ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[34mINFO: TempSpace: Total 1 MB allocated on CPU0\n", "\u001b[0m" ] }, { "name": "stdout", "output_type": "stream", "text": [ " 3.578575 seconds (5.62 k allocations: 86.121 MB, 0.41% gc time)\n" ] }, { "data": { "text/plain": [ "10×28000 Array{Float32,2}:\n", " 3.17956f-5 0.999099 1.12871f-6 … 1.32057f-6 6.29556f-6 \n", " 5.82953f-8 1.96553f-8 0.000100165 3.53726f-8 1.18404f-7 \n", " 0.99966 0.000367172 1.80263f-6 1.26685f-6 0.999141 \n", " 0.000230099 1.24621f-6 8.82856f-5 0.0032008 0.000576255\n", " 5.89889f-6 1.49464f-7 0.0208702 0.0349224 5.21555f-5 \n", " 1.43514f-7 0.000451465 0.000577244 … 0.000289193 7.39706f-7 \n", " 9.90732f-7 7.13181f-5 6.17309f-7 2.8054f-7 1.98969f-6 \n", " 1.8722f-5 4.91723f-6 0.000499786 0.00010417 4.51312f-6 \n", " 5.00769f-5 1.48886f-6 0.00483644 0.000480367 0.00021646 \n", " 1.78278f-6 3.23831f-6 0.973024 0.961 1.00423f-6 " ] }, "execution_count": 65, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# this uses the recently trained model and the same DataProvider on the test set \n", "@time tpreds = mx.predict(model, test_provider)" ] }, { "cell_type": "code", "execution_count": 66, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "# create submission\n", "open(\"MLP2submission.csv\", \"w\") do f\n", " write(f, \"ImageId,Label\\n\")\n", " for i = 1:size(tpreds)[2]\n", " write(f, string(i),\",\",string(indmax(tpreds[:,i])-1),\"\\n\")\n", " end\n", "end" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When I submit this new file to Kaggle, I get a score of 0.95657, which is similar to the error of 4.5% from the LeCun paper. Creating MLPs is very easy with MXNet and there is even an `MLP()` constructor. The equivalent code for this is:\n", "```\n", "mlp2 = @mx.chain mx.Variable(:data) =>\n", " mx.MLP([300, 100, 10]) =>\n", " mx.SoftmaxOutput(name=:softmax)\n", "```\n", "Running this model for 20 epochs and submitting to Kaggle got me a better score of 0.96543. I'm not sure why yet, although the defaults for this constructor are different. For example, `MLP()` defaults to using `:relu` activations which have become more standard as they are faster than the original `:tanh` activations I used. ReLU do have problems at zero due to the discontinuity there, so other variants like LeakyReLU have been developed as mentioned above.\n", "\n", "I hope this notebook has been helpful. Please leave a comment if you have suggestions for improvement." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[](http://creativecommons.org/licenses/by-sa/4.0/) \n", "\n", "Licensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-sa/4.0/)." ] } ], "metadata": { "kernelspec": { "display_name": "Julia 1.1.1", "language": "julia", "name": "julia-1.1" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "1.1.1" }, "toc": { "nav_menu": { "height": "150px", "width": "252px" }, "navigate_menu": true, "number_sections": true, "sideBar": true, "threshold": 4, "toc_cell": false, "toc_section_display": "block", "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }