{
 "metadata": {
  "name": "stacked_autoencoders"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Stacked Autoencoders\n",
      "by [Mehdi Mirza](http://www-etud.iro.umontreal.ca/~mirzamom/)\n",
      "\n",
      "## Introduction\n",
      "This notebook will show you how to perform layer-wise pre-training using denoising autoencoders (DAEs), and subsequently stack the layers to form a multilayer perceptron (MLP) which can be fine-tuned using supervised training. You can also look at this [more detailed tutorial of training DAEs using Theano](http://deeplearning.net/tutorial/dA.html#daa) as well as [this tutorial](http://deeplearning.net/tutorial/SdA.html#sda) which covers the stacked version.\n",
      "\n",
      "The methods used here can easily be adapted to other models such as contractive auto-encoders (CAEs) or restricted Boltzmann machines (RBMs) with only small modifications."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## First layer\n",
      "\n",
      "The first layer and its training algorithm are defined in the file `dae_l1.yaml`. Here we load the model and set some of its hypyerparameters."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "layer1_yaml = open('dae_l1.yaml', 'r').read()\n",
      "hyper_params_l1 = {'train_stop' : 50000,\n",
      "                   'batch_size' : 100,\n",
      "                   'monitoring_batches' : 5,\n",
      "                   'nhid' : 500,\n",
      "                   'max_epochs' : 10,\n",
      "                   'save_path' : '.'}\n",
      "layer1_yaml = layer1_yaml % (hyper_params_l1)\n",
      "print layer1_yaml"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "!obj:pylearn2.train.Train {\n",
        "    dataset: &train !obj:pylearn2.datasets.mnist.MNIST {\n",
        "        which_set: 'train',\n",
        "        start: 0,\n",
        "        stop: 50000\n",
        "    },\n",
        "    model: !obj:pylearn2.models.autoencoder.DenoisingAutoencoder {\n",
        "        nvis : 784,\n",
        "        nhid : 500,\n",
        "        irange : 0.05,\n",
        "        corruptor: !obj:pylearn2.corruption.BinomialCorruptor {\n",
        "            corruption_level: .2,\n",
        "        },\n",
        "        act_enc: \"tanh\",\n",
        "        act_dec: null,    # Linear activation on the decoder side.\n",
        "    },\n",
        "    algorithm: !obj:pylearn2.training_algorithms.sgd.SGD {\n",
        "        learning_rate : 1e-3,\n",
        "        batch_size : 100,\n",
        "        monitoring_batches : 5,\n",
        "        monitoring_dataset : *train,\n",
        "        cost : !obj:pylearn2.costs.autoencoder.MeanSquaredReconstructionError {},\n",
        "        termination_criterion : !obj:pylearn2.termination_criteria.EpochCounter {\n",
        "            max_epochs: 10,\n",
        "        },\n",
        "    },\n",
        "    save_path: \"./dae_l1.pkl\",\n",
        "    save_freq: 1\n",
        "}\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we can train the model using the YAML string in the same way as the previous tutorials:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pylearn2.config import yaml_parse\n",
      "train = yaml_parse.load(layer1_yaml)\n",
      "train.main_loop()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Parameter and initial learning rate summary:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvb: 0.0010000000475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\thb: 0.0010000000475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tW: 0.0010000000475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tWprime: 0.0010000000475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling sgd_update...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling sgd_update done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "compiling begin_record_entry...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "compiling begin_record_entry done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitored channels: \n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling accum...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "graph size: 23\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling accum done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 85.4375915527\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 1\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 50000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 29.1613636017\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 2\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 1000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 100000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 21.9736881256\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 3\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 1500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 150000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 18.4479560852\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 4\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 2000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 200000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 16.2897148132\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 5\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 2500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 250000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 14.8111886978\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 6\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 3000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 300000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 13.6504278183\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 7\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 3500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 350000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 12.9274587631\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 8\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 4000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 400000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 12.2765922546\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 9\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 4500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 450000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 11.7446937561\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 10\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 5000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 500000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 11.4141273499\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l1.pkl done. Time elapsed: 1.000000 seconds\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Second layer\n",
      "The second layer takes the output of the first layer as its input. Hence we must first apply the first layer's transformations to the raw data using `datasets.transformer_dataset.TransformerDataset`. This class takes two arguments:\n",
      "\n",
      "   - `raw`: the raw data\n",
      "   - `transformer`: a Pylearn2 block that transforms the raw data, which in our case is the `dae_l1.pkl` file from the previous step\n",
      "\n",
      "To train the second layer, we load the YAML file as before and set the hyperparameters before starting the training loop."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "layer2_yaml = open('dae_l2.yaml', 'r').read()\n",
      "hyper_params_l2 = {'train_stop' : 50000,\n",
      "                   'batch_size' : 100,\n",
      "                   'monitoring_batches' : 5,\n",
      "                   'nvis' : hyper_params_l1['nhid'],\n",
      "                   'nhid' : 500,\n",
      "                   'max_epochs' : 10,\n",
      "                   'save_path' : '.'}\n",
      "layer2_yaml = layer2_yaml % (hyper_params_l2)\n",
      "print layer2_yaml"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "!obj:pylearn2.train.Train {\n",
        "    dataset: &train !obj:pylearn2.datasets.transformer_dataset.TransformerDataset {\n",
        "        raw: !obj:pylearn2.datasets.mnist.MNIST {\n",
        "            which_set: 'train',\n",
        "            start: 0,\n",
        "            stop: 50000\n",
        "        },\n",
        "        transformer: !pkl: \"./dae_l1.pkl\"\n",
        "    },\n",
        "    model: !obj:pylearn2.models.autoencoder.DenoisingAutoencoder {\n",
        "        nvis : 500,\n",
        "        nhid : 500,\n",
        "        irange : 0.05,\n",
        "        corruptor: !obj:pylearn2.corruption.BinomialCorruptor {\n",
        "            corruption_level: .3,\n",
        "        },\n",
        "        act_enc: \"tanh\",\n",
        "        act_dec: null,    # Linear activation on the decoder side.\n",
        "    },\n",
        "    algorithm: !obj:pylearn2.training_algorithms.sgd.SGD {\n",
        "        learning_rate : 1e-3,\n",
        "        batch_size : 100,\n",
        "        monitoring_batches : 5,\n",
        "        monitoring_dataset : *train,\n",
        "        cost : !obj:pylearn2.costs.autoencoder.MeanSquaredReconstructionError {},\n",
        "        termination_criterion : !obj:pylearn2.termination_criteria.EpochCounter {\n",
        "            max_epochs: 10,\n",
        "        },\n",
        "    },\n",
        "    save_path: \"./dae_l2.pkl\",\n",
        "    save_freq: 1\n",
        "}\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "train = yaml_parse.load(layer2_yaml)\n",
      "train.main_loop()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Parameter and initial learning rate summary:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvb: 0.0010000000475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\thb: 0.0010000000475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tW: 0.0010000000475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tWprime: 0.0010000000475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling sgd_update...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling sgd_update done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "compiling begin_record_entry...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "compiling begin_record_entry done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitored channels: \n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling accum...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "graph size: 23\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling accum done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 51.0506210327\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 1\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 50000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 20.0142116547\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 2\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 1000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 100000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 12.8833475113\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 3\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 1500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 150000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 9.65194129944\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 4\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 2000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 200000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 7.71482992172\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 5\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 2500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 250000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 6.5238275528\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 6\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 3000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 300000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 5.69179153442\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 7\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 3500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 350000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 5.15888118744\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 8\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 4000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 400000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 4.75159025192\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 9\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 4500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 450000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 4.38682460785\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 10\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 5000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 500000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00100000016391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tobjective: 4.21171569824\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Saving to ./dae_l2.pkl done. Time elapsed: 0.000000 seconds\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Supervised fine-tuning\n",
      "Now that we have two pre-trained layers, we can stack them to form an MLP which can be trained in a supervised fashion. We use the MLP class as usual for this, except that we now use `models.mlp.PretrainedLayer` for the different layers so that we can pass our pre-trained layers (as pickle files) using the `layer_content` argument."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mlp_yaml = open('dae_mlp.yaml', 'r').read()\n",
      "hyper_params_mlp = {'train_stop' : 50000,\n",
      "                    'valid_stop' : 60000,\n",
      "                    'batch_size' : 100,\n",
      "                    'max_epochs' : 50,\n",
      "                    'save_path' : '.'}\n",
      "mlp_yaml = mlp_yaml % (hyper_params_mlp)\n",
      "print mlp_yaml"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "!obj:pylearn2.train.Train {\n",
        "    dataset: &train !obj:pylearn2.datasets.mnist.MNIST {\n",
        "        which_set: 'train',\n",
        "        start: 0,\n",
        "        stop: 50000\n",
        "    },\n",
        "    model: !obj:pylearn2.models.mlp.MLP {\n",
        "        batch_size: 100,\n",
        "        layers: [\n",
        "                 !obj:pylearn2.models.mlp.PretrainedLayer {\n",
        "                     layer_name: 'h1',\n",
        "                     layer_content: !pkl: \"./dae_l1.pkl\"\n",
        "                 },\n",
        "                 !obj:pylearn2.models.mlp.PretrainedLayer {\n",
        "                     layer_name: 'h2',\n",
        "                     layer_content: !pkl: \"./dae_l2.pkl\"\n",
        "                 },\n",
        "                 !obj:pylearn2.models.mlp.Softmax {\n",
        "                     max_col_norm: 1.9365,\n",
        "                     layer_name: 'y',\n",
        "                     n_classes: 10,\n",
        "                     irange: .005\n",
        "                 }\n",
        "                ],\n",
        "        nvis: 784\n",
        "    },\n",
        "    algorithm: !obj:pylearn2.training_algorithms.sgd.SGD {\n",
        "        learning_rate: .05,\n",
        "        learning_rule: !obj:pylearn2.training_algorithms.learning_rule.Momentum {\n",
        "            init_momentum: .5,\n",
        "        },\n",
        "        monitoring_dataset:\n",
        "            {\n",
        "                'valid' : !obj:pylearn2.datasets.mnist.MNIST {\n",
        "                              which_set: 'train',\n",
        "                              start: 50000,\n",
        "                              stop: 60000\n",
        "                          },\n",
        "            },\n",
        "        cost: !obj:pylearn2.costs.mlp.Default {},\n",
        "        termination_criterion: !obj:pylearn2.termination_criteria.And {\n",
        "            criteria: [\n",
        "                !obj:pylearn2.termination_criteria.MonitorBased {\n",
        "                    channel_name: \"valid_y_misclass\",\n",
        "                    prop_decrease: 0.,\n",
        "                    N: 100\n",
        "                },\n",
        "                !obj:pylearn2.termination_criteria.EpochCounter {\n",
        "                    max_epochs: 50\n",
        "                }\n",
        "            ]\n",
        "        },\n",
        "        update_callbacks: !obj:pylearn2.training_algorithms.sgd.ExponentialDecay {\n",
        "            decay_factor: 1.00004,\n",
        "            min_lr: .000001\n",
        "        }\n",
        "    },\n",
        "    extensions: [\n",
        "        !obj:pylearn2.training_algorithms.learning_rule.MomentumAdjustor {\n",
        "            start: 1,\n",
        "            saturate: 250,\n",
        "            final_momentum: .7\n",
        "        }\n",
        "    ]\n",
        "}\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "train = yaml_parse.load(mlp_yaml)\n",
      "train.main_loop()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Parameter and initial learning rate summary:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "/data/lisa/exp/mirzamom/pylearn2/pylearn2/models/mlp.py:41: UserWarning: MLP changing the recursion limit.\n",
        "  warnings.warn(\"MLP changing the recursion limit.\")\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvb: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\thb: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tW: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tWprime: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvb: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\thb: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tW: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tWprime: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tsoftmax_b: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tsoftmax_W: 0.0500000007451\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling sgd_update...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling sgd_update done. Time elapsed: 51.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "compiling begin_record_entry...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "compiling begin_record_entry done. Time elapsed: 0.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitored channels: \n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling accum...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "graph size: 75\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling accum done. Time elapsed: 31.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0500000119209\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.499999672174\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 2.30245757103\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 0.0650026649237\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 0.0641745403409\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 0.0624679774046\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.10553213954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.102753870189\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.101059176028\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.903100371361\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 2.30245757103\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.0125483665615\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.00897720176727\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.00411556242034\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 1\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 50000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0490099266171\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.499999672174\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.285481214523\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.37920033932\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.25995886326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.10580408573\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999643802643\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.891385912895\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.366638094187\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0814000219107\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.285481214523\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.306006103754\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.173898175359\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0752066597342\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 2\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 1000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 100000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0480394884944\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.500803589821\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.247136443853\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.53969144821\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.40233445168\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.25563120842\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999809861183\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.914137363434\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.396682620049\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.069399997592\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.247136443853\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.348902791739\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.193130522966\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0754316821694\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 3\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 1500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 150000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0470883138478\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.501606047153\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.209606900811\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.67392218113\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.51739025116\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.41721081734\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999855041504\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.925868034363\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.405808866024\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.06040000543\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.209606900811\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.398027926683\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.20821505785\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0778625309467\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 4\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 2000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 200000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0461559444666\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.502409934998\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.181997314095\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.88737154007\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.62824416161\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.44828641415\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999894917011\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.934701681137\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.424763649702\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0520000010729\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.181997314095\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.444758623838\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.222617387772\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0790278464556\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 5\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 2500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 250000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0452419146895\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.50321239233\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.159930184484\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.72115278244\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.47439146042\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.99988681078\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.940866410732\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.426196664572\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0440000146627\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.159930184484\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.464266389608\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.234414324164\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0797937735915\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 6\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 3000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 300000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0443461276591\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.504016280174\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.143035233021\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93126213551\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.79720795155\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.52031481266\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999934792519\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.948293268681\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.448669195175\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0376999974251\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.143035233021\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.501182496548\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.244007915258\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0815980285406\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 7\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 3500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 350000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0434680506587\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.504818737507\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.128972783685\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93631577492\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.84140181541\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.56303739548\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.99993532896\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.952728152275\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.457730174065\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0372999943793\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.128972783685\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.52207928896\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.249332204461\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0810364559293\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 8\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 4000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 400000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0426072925329\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.505622982979\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.123533077538\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.87326931953\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.61571848392\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999963104725\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.954613864422\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.463554471731\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0348999910057\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.123533077538\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.525155007839\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.253258258104\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0812314674258\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 9\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 4500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 450000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0417636223137\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.506425499916\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.119187682867\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.89299559593\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.65916585922\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999965846539\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.95620149374\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.464787423611\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0323999859393\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.119187682867\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.534337043762\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.255589127541\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0810972675681\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 10\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 5000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 500000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0409367084503\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.50722938776\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.107577241957\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.90345025063\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.70279407501\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999951183796\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.960036695004\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.468458265066\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0300999823958\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.107577241957\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.542799532413\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.256767898798\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0823005959392\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 11\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 5500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 550000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0401261113584\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.508031845093\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.107919149101\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.9149273634\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.76190459728\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999973893166\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.959668278694\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.47409799695\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0300999861211\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.107919149101\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.550510644913\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.258117824793\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0835975408554\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 12\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 6000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 600000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0393316075206\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.508835673332\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0998769327998\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.92226481438\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.80892860889\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999977052212\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.964593172073\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.500402808189\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0274999812245\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0998769327998\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.559845209122\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.259052544832\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0850235819817\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 13\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 6500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 650000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0385527797043\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.509638190269\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0978430137038\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.92790329456\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.8607878685\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999977111816\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.964517354965\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.493009746075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0281999818981\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0978430137038\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.565926074982\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.259754091501\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0865102484822\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 14\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 7000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 700000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0377893745899\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.510442078114\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0951417461038\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93095195293\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.90848994255\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999983549118\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.965570628643\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.493750423193\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0279999841005\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0951417461038\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.57372456789\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.26015779376\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0874916240573\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 15\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 7500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 750000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0370411500335\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.511244595051\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0946910232306\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93529140949\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93278777599\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999984383583\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.966287732124\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.497158616781\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0266999825835\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0946910232306\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.576683402061\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.260768920183\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0881127864122\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 16\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 8000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 800000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0363076739013\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.512048363686\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.089107722044\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.9356637001\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93223702908\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999985218048\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.96820807457\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.502092540264\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0256999880075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.089107722044\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.57947987318\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.260900110006\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0890503451228\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 17\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 8500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 850000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0355887822807\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.512850999832\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0881613865495\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93482923508\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93224895\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999977946281\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.968540728092\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.502689242363\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0259999874979\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0881613865495\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.581995129585\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.260852187872\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0897700637579\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 18\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 9000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 900000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.034884031862\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.513654768467\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0850231051445\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93542182446\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93202567101\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999984383583\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.969747781754\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.50995349884\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0240999888629\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0850231051445\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.582888245583\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.261032491922\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.091475315392\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 19\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 9500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 950000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0341933257878\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.514457404613\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0860132724047\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93475472927\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93239700794\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999982178211\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.968550920486\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.500067353249\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.024499990046\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0860132724047\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.585309565067\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.261046379805\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0925423651934\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 20\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 10000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1000000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0335162654519\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.515261173248\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.082815758884\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93560194969\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93234658241\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999988675117\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.970959126949\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.511843323708\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0254999864846\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.082815758884\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.587334752083\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.261245340109\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0929176732898\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 21\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 10500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1050000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0328526012599\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.516063690186\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0818511173129\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93555438519\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93388450146\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999989688396\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.972750782967\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.53289026022\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0240999888629\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0818511173129\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.584912240505\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.261357337236\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0945193096995\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 22\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 11000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1100000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0322021208704\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.51686757803\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0818284451962\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93543183804\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93241846561\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999989748001\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.971523821354\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.512000918388\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0234999898821\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0818284451962\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.585798323154\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.261481463909\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0941896960139\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 23\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 11500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1150000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0315644294024\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.517670154572\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0783765390515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.9352645874\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93146395683\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999990105629\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.972935736179\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.526244282722\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0227999929339\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0783765390515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.584616363049\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.261561661959\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0957764536142\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 24\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 12000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1200000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0309394672513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.518473863602\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0788094773889\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93574666977\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93321406841\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999989807606\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.973162353039\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.517908155918\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0223999936134\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0788094773889\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.585705161095\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.261755138636\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0961646363139\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 2.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 25\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 12500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1250000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0303268413991\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.519276380539\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 1.9999986887\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0773832127452\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93531489372\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93346488476\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999991297722\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.973752617836\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.529482901096\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0232999920845\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0773832127452\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.58470761776\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.26182243228\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0976147502661\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 26\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 13000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1300000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0297263283283\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.520080327988\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0760994702578\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93572044373\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93307471275\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999990880489\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.974325656891\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.535016596317\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0222999919206\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0760994702578\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.584625601768\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262017458677\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0985018312931\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 27\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 13500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1350000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0291377287358\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.520884275436\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0745258107781\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93582701683\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93377733231\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999992668629\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.974617183208\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.528786301613\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0223999880254\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0745258107781\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.582556009293\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262156039476\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.098942771554\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 28\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 14000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1400000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0285607334226\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.521686851978\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0740825012326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93625664711\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93552911282\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999993383884\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.975292444229\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.532864153385\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0215999912471\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0740825012326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.582411289215\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262312680483\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.0992849618196\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 29\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 14500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1450000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0279952250421\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.522490501404\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0735178291798\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93584036827\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93419444561\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999993622303\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.975481748581\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.530311584473\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0224999897182\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0735178291798\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.581294953823\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262381464243\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.099995970726\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 30\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 15000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1500000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.027440899983\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.523293077946\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0742838978767\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93606865406\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93509995937\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999993681908\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.975233256817\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.529448211193\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.021299989894\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0742838978767\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.579390466213\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262531936169\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.101199530065\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 31\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 15500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1550000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0268975384533\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.52409696579\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0728998035192\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93537724018\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.9334436655\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999993503094\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.975039601326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.530442178249\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0211999937892\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0728998035192\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.577543079853\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262552529573\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.101914271712\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 32\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 16000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1600000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0263649839908\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.524899542332\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0729000940919\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93582475185\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93382787704\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999994158745\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.976245224476\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.523617684841\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0215999912471\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0729000940919\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.575929939747\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262723714113\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102134265006\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 33\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 16500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1650000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0258428994566\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.525703251362\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0711924284697\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93601918221\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93474292755\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999995052814\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.976920008659\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.53466886282\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0214999932796\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0711924284697\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.575305998325\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262870043516\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102859780192\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 34\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 17000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1700000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.025331215933\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.526505768299\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0699434652925\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93592369556\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93427860737\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999995589256\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.976588606834\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.526415586472\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0207999944687\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0699434652925\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.573732554913\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.262948900461\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.103134132922\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 35\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 17500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1750000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0248296167701\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.527309715748\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0703471377492\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93556249142\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93350601196\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999995827675\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.977201640606\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.54014390707\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0216999910772\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0703471377492\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.569660007954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263015538454\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.103172667325\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 36\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 18000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1800000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.024337939918\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.528112351894\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0702705159783\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93602788448\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.9353749752\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.99999576807\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.977866590023\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.538337528706\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0210999920964\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0702705159783\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.570420324802\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263186216354\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.103417083621\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 37\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 18500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1850000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0238560270518\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.52891600132\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0700398087502\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93587827682\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93338406086\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999996423721\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.977614223957\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.5420165658\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0208999942988\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0700398087502\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.5683183074\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263258725405\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102311193943\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 38\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 19000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1900000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.023383660242\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.529718577862\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0708458870649\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93592023849\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93505263329\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999996066093\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.978010952473\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.540901720524\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0209999922663\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0708458870649\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.56648504734\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263367444277\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102654665709\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 39\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 19500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1950000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0229206457734\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.530522465706\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0704958662391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93585407734\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.9334679842\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999996304512\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.978026509285\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.547220349312\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0213999915868\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0704958662391\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.563822031021\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263476461172\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102645337582\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 40\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 20000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2000000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0224668364972\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.531325042248\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.069045573473\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93611621857\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93514800072\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999996364117\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.978076577187\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.548255085945\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0201999917626\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.069045573473\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.561926782131\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263601183891\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102276921272\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 41\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 20500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2050000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0220219288021\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.532128691673\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0694609582424\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93617403507\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93555283546\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999996304512\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.97806340456\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.5496789217\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0206999927759\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0694609582424\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.559349894524\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263698577881\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102876082063\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 42\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 21000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2100000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0215858761221\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.532931268215\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0682552531362\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93583071232\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93494808674\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999996244907\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.978460967541\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.536799430847\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0206999927759\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0682552531362\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.558497548103\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263731598854\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102174289525\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 43\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 21500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2150000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0211584754288\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.533735215664\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.068164549768\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93625628948\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93569409847\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999996840954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.978974223137\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.545736849308\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.020799992606\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.068164549768\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.55761551857\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263863831758\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102318763733\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 44\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 22000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2200000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0207395013422\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.534537792206\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0678072869778\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93612587452\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93490695953\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999996721745\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.978856146336\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.54448735714\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0202999915928\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0678072869778\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.557243168354\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263932317495\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102473787963\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 45\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 22500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2250000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0203288514167\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.535341382027\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0676843225956\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93607878685\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93488621712\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999997019768\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.979203939438\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.541955649853\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0211999919266\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0676843225956\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.554672718048\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.263984143734\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102155432105\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 46\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 23000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2300000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0199263226241\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.536144316196\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0666035562754\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93638789654\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93599748611\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999997377396\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.979231536388\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.552667915821\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0199999958277\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0666035562754\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.552026212215\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.264121174812\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.102099023759\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 47\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 23500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2350000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0195317566395\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.536948263645\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0681700259447\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93598556519\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93464744091\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999997317791\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.979587137699\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.54142510891\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0199999958277\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0681700259447\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.550442278385\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.264142274857\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.101656988263\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 48\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 24000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2400000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0191450119019\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.537750899792\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.069911248982\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93574678898\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.9335873127\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999997794628\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.979529380798\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.541589438915\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0213999953121\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.069911248982\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.548533499241\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.264166146517\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.101313956082\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 49\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 24500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2450000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0187659449875\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.538554370403\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0663670599461\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93616163731\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93532729149\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999997615814\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.979907333851\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.55751311779\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0203999932855\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0663670599461\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.54831713438\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.26429900527\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.101693704724\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 1.000000 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 50\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 25000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 2500000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.0183943510056\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.53935700655\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmonitor_seconds_per_epoch: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0667693391442\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_mean: 1.93614006042\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_col_norms_min: 1.93519842625\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999997913837\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.980073690414\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.548149049282\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.019999993965\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_nll: 0.0667693391442\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_max: 0.546491324902\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_mean: 0.26435393095\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_row_norms_min: 0.10142172128\n"
       ]
      }
     ],
     "prompt_number": 9
    }
   ],
   "metadata": {}
  }
 ]
}