{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# DeepLearning\n", "\n", "### Introduction\n", "In this notebook, we introduce H2O Deep Learning via fully-connected artificial neural networks. We also show many useful features of H2O such as hyper-parameter optimization, Flow, and checkpointing. There are other notebooks that use more complex convolutional neural networks ranging from LeNet all the way to Inception Resnet V2.\n", "\n", "### MNIST Dataset\n", "The MNIST database is a well-known academic dataset used to benchmark\n", "classification performance. The data consists of 60,000 training images and\n", "10,000 test images. Each image is a standardized $28^2$ pixel greyscale image of\n", "a single handwritten digit. A sample of the scanned handwritten digits is\n", "shown\n", "![Example MNIST digit images](images/mnist.png)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Checking whether there is an H2O instance running at http://localhost:54321. connected.\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
H2O cluster uptime:1 min 33 secs
H2O cluster version:3.11.0.99999
H2O cluster version age:1 hour and 55 minutes
H2O cluster name:arno
H2O cluster total nodes:1
H2O cluster free memory:13.96 Gb
H2O cluster total cores:12
H2O cluster allowed cores:12
H2O cluster status:locked, healthy
H2O connection url:http://localhost:54321
H2O connection proxy:None
Python version:2.7.12 final
" ], "text/plain": [ "-------------------------- ----------------------\n", "H2O cluster uptime: 1 min 33 secs\n", "H2O cluster version: 3.11.0.99999\n", "H2O cluster version age: 1 hour and 55 minutes\n", "H2O cluster name: arno\n", "H2O cluster total nodes: 1\n", "H2O cluster free memory: 13.96 Gb\n", "H2O cluster total cores: 12\n", "H2O cluster allowed cores: 12\n", "H2O cluster status: locked, healthy\n", "H2O connection url: http://localhost:54321\n", "H2O connection proxy:\n", "Python version: 2.7.12 final\n", "-------------------------- ----------------------" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import h2o\n", "h2o.init(nthreads=-1)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import os.path\n", "PATH = os.path.expanduser(\"~/h2o-3/\")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Parse progress: |█████████████████████████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "test_df = h2o.import_file(PATH + \"bigdata/laptop/mnist/test.csv.gz\")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Parse progress: |█████████████████████████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "train_df = h2o.import_file(PATH + \"bigdata/laptop/mnist/train.csv.gz\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Specify the response and predictor columns" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [], "source": [ "y = \"C785\"\n", "x = train_df.names[0:784]" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": true }, "outputs": [], "source": [ "train_df[y] = train_df[y].asfactor()\n", "test_df[y] = test_df[y].asfactor()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Train Deep Learning model and validate on test set" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from h2o.estimators.deepwater import H2ODeepWaterEstimator" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model = H2ODeepWaterEstimator(\n", " distribution=\"multinomial\",\n", " activation=\"rectifier\",\n", " mini_batch_size=128,\n", " hidden=[1024,1024],\n", " hidden_dropout_ratios=[0.5,0.5], ## for better generalization\n", " input_dropout_ratio=0.1,\n", " sparse=True, ## can result in speedup for sparse data\n", " epochs=10) ## need more epochs for a better model" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "deepwater Model Build progress: |█████████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "model.train(\n", " x=x, \n", " y=y,\n", " training_frame=train_df,\n", " validation_frame=test_df\n", ")" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampdurationtraining_speedepochsiterationssamplestraining_rmsetraining_loglosstraining_classification_errorvalidation_rmsevalidation_loglossvalidation_classification_error
02016-10-23 01:30:370.000 secNone0.00000000.0NaNNaNNaNNaNNaNNaN
12016-10-23 01:30:383.250 sec6585 obs/sec0.06826714096.00.4473946.9066140.2001610.4381786.6246860.1920
22016-10-23 01:30:5216.392 sec25255 obs/sec5.52960081331776.00.1185470.4794500.0140530.2054511.4555220.0422
32016-10-23 01:30:5822.772 sec26887 obs/sec8.465067124507904.00.0795240.2184240.0063240.1969751.3360980.0388
42016-10-23 01:31:0226.488 sec27373 obs/sec10.035200147602112.00.0625690.1352150.0039150.1972311.3420810.0389
52016-10-23 01:31:0227.130 sec27330 obs/sec10.035200147602112.00.0795240.2184240.0063240.1969751.3360980.0388
\n", "
" ], "text/plain": [ " timestamp duration training_speed epochs iterations \\\n", "0 2016-10-23 01:30:37 0.000 sec None 0.000000 0 \n", "1 2016-10-23 01:30:38 3.250 sec 6585 obs/sec 0.068267 1 \n", "2 2016-10-23 01:30:52 16.392 sec 25255 obs/sec 5.529600 81 \n", "3 2016-10-23 01:30:58 22.772 sec 26887 obs/sec 8.465067 124 \n", "4 2016-10-23 01:31:02 26.488 sec 27373 obs/sec 10.035200 147 \n", "5 2016-10-23 01:31:02 27.130 sec 27330 obs/sec 10.035200 147 \n", "\n", " samples training_rmse training_logloss training_classification_error \\\n", "0 0.0 NaN NaN NaN \n", "1 4096.0 0.447394 6.906614 0.200161 \n", "2 331776.0 0.118547 0.479450 0.014053 \n", "3 507904.0 0.079524 0.218424 0.006324 \n", "4 602112.0 0.062569 0.135215 0.003915 \n", "5 602112.0 0.079524 0.218424 0.006324 \n", "\n", " validation_rmse validation_logloss validation_classification_error \n", "0 NaN NaN NaN \n", "1 0.438178 6.624686 0.1920 \n", "2 0.205451 1.455522 0.0422 \n", "3 0.196975 1.336098 0.0388 \n", "4 0.197231 1.342081 0.0389 \n", "5 0.196975 1.336098 0.0388 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.scoring_history()" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "ModelMetricsMultinomial: deepwater\n", "** Reported on train data. **\n", "\n", "MSE: 0.00632403131935\n", "RMSE: 0.0795237783267\n", "LogLoss: 0.218424309446\n", "Mean Per-Class Error: 0.00637963860238\n", "Confusion Matrix: vertical: actual; across: predicted\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
0123456789ErrorRate
960.00.01.00.00.00.00.01.00.00.00.00207902 / 962
0.01115.00.00.00.00.00.00.00.00.00.00 / 1,115
0.00.01016.00.00.00.00.03.00.00.00.00294413 / 1,019
0.03.00.0976.00.00.00.05.00.00.00.00813018 / 984
0.01.00.00.01017.00.00.00.01.01.00.00294123 / 1,020
0.00.01.03.00.0912.01.00.06.01.00.012987012 / 924
0.00.00.00.01.00.0923.00.00.00.00.00108231 / 924
0.01.01.00.00.00.00.01043.00.00.00.00191392 / 1,045
0.00.01.00.00.00.00.03.0943.01.00.00527435 / 948
0.01.00.00.04.00.00.021.01.0994.00.026444727 / 1,021
960.01121.01020.0979.01022.0912.0924.01076.0951.0997.00.006324063 / 9,962
" ], "text/plain": [ "0 1 2 3 4 5 6 7 8 9 Error Rate\n", "--- ---- ---- --- ---- --- --- ---- --- --- ---------- ----------\n", "960 0 1 0 0 0 0 1 0 0 0.002079 2 / 962\n", "0 1115 0 0 0 0 0 0 0 0 0 0 / 1,115\n", "0 0 1016 0 0 0 0 3 0 0 0.00294406 3 / 1,019\n", "0 3 0 976 0 0 0 5 0 0 0.00813008 8 / 984\n", "0 1 0 0 1017 0 0 0 1 1 0.00294118 3 / 1,020\n", "0 0 1 3 0 912 1 0 6 1 0.012987 12 / 924\n", "0 0 0 0 1 0 923 0 0 0 0.00108225 1 / 924\n", "0 1 1 0 0 0 0 1043 0 0 0.00191388 2 / 1,045\n", "0 0 1 0 0 0 0 3 943 1 0.00527426 5 / 948\n", "0 1 0 0 4 0 0 21 1 994 0.0264447 27 / 1,021\n", "960 1121 1020 979 1022 912 924 1076 951 997 0.00632403 63 / 9,962" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Top-10 Hit Ratios: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
khit_ratio
10.9936759
20.9938767
30.9938767
40.9938767
50.9938767
60.9938767
70.9938767
80.9938767
90.9938767
100.9999999
" ], "text/plain": [ "k hit_ratio\n", "--- -----------\n", "1 0.993676\n", "2 0.993877\n", "3 0.993877\n", "4 0.993877\n", "5 0.993877\n", "6 0.993877\n", "7 0.993877\n", "8 0.993877\n", "9 0.993877\n", "10 1" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.model_performance(train=True) # training metrics" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "ModelMetricsMultinomial: deepwater\n", "** Reported on validation data. **\n", "\n", "MSE: 0.0387991228965\n", "RMSE: 0.196974929614\n", "LogLoss: 1.33609811491\n", "Mean Per-Class Error: 0.0394969339854\n", "Confusion Matrix: vertical: actual; across: predicted\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
0123456789ErrorRate
959.01.02.00.02.04.06.03.02.01.00.021428621 / 980
0.01122.03.00.00.00.03.04.03.00.00.011453713 / 1,135
3.02.0989.02.06.01.05.010.011.03.00.041666743 / 1,032
0.01.07.0969.01.04.01.010.010.07.00.040594141 / 1,010
2.01.04.01.0949.00.04.07.03.011.00.033604933 / 982
4.01.01.020.05.0824.013.07.014.03.00.076233268 / 892
5.02.02.00.08.06.0929.01.04.01.00.030271429 / 958
1.06.07.02.01.01.01.01001.03.05.00.026264627 / 1,028
1.00.06.07.03.05.03.08.0933.08.00.042094541 / 974
4.05.02.04.011.05.01.034.06.0937.00.071357872 / 1,009
979.01141.01023.01005.0986.0850.0966.01085.0989.0976.00.0388388 / 10,000
" ], "text/plain": [ "0 1 2 3 4 5 6 7 8 9 Error Rate\n", "--- ---- ---- ---- --- --- --- ---- --- --- --------- ------------\n", "959 1 2 0 2 4 6 3 2 1 0.0214286 21 / 980\n", "0 1122 3 0 0 0 3 4 3 0 0.0114537 13 / 1,135\n", "3 2 989 2 6 1 5 10 11 3 0.0416667 43 / 1,032\n", "0 1 7 969 1 4 1 10 10 7 0.0405941 41 / 1,010\n", "2 1 4 1 949 0 4 7 3 11 0.0336049 33 / 982\n", "4 1 1 20 5 824 13 7 14 3 0.0762332 68 / 892\n", "5 2 2 0 8 6 929 1 4 1 0.0302714 29 / 958\n", "1 6 7 2 1 1 1 1001 3 5 0.0262646 27 / 1,028\n", "1 0 6 7 3 5 3 8 933 8 0.0420945 41 / 974\n", "4 5 2 4 11 5 1 34 6 937 0.0713578 72 / 1,009\n", "979 1141 1023 1005 986 850 966 1085 989 976 0.0388 388 / 10,000" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Top-10 Hit Ratios: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
khit_ratio
10.9612
20.9651
30.9651
40.9651
50.9651
60.9651
70.9651
80.9651
90.9651
101.0
" ], "text/plain": [ "k hit_ratio\n", "--- -----------\n", "1 0.9612\n", "2 0.9651\n", "3 0.9651\n", "4 0.9651\n", "5 0.9651\n", "6 0.9651\n", "7 0.9651\n", "8 0.9651\n", "9 0.9651\n", "10 1" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.model_performance(valid=True) # validation metrics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Inspect the model in Flow\n", "It is highly recommended to use [Flow](http://localhost:54321/) to visualize the model training process and to inspect the model before using it for further steps." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using Crossvalidation\n", "\n", "If the value specified for nfolds is a positive integer, N-fold cross-validation is\n", "performed on the training frame and the cross-validation metrics are computed\n", "and stored as model output. \n", "\n", "To disable cross-validation, use `nfolds=0`, which is the default value." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Advanced users can also specify a fold column that defines the holdout\n", "fold associated with each row. By default, the holdout fold assignment is\n", "random. H2O supports other schemes such as round-robin assignment using the modulo\n", "operator." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perform 3-fold cross-validation on training_frame" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model_crossvalidated = H2ODeepWaterEstimator(\n", " distribution=\"multinomial\",\n", " activation=\"rectifier\",\n", " mini_batch_size=128,\n", " hidden=[1024,1024],\n", " hidden_dropout_ratios=[0.5,0.5],\n", " input_dropout_ratio=0.1,\n", " sparse=True,\n", " epochs=10,\n", " nfolds=3\n", ")" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "deepwater Model Build progress: |█████████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "model_crossvalidated.train(\n", " x=x,\n", " y=y,\n", " training_frame=train_df\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Extracting and Handling the Results\n", "\n", "We can now extract the parameters of our model, examine the scoring process,\n", "and make predictions on new data." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# View specified parameters of the Deep Learning model\n", "model_crossvalidated.params;" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model Details\n", "=============\n", "H2ODeepWaterEstimator : Deep Water\n", "Model Key: DeepWater_model_python_1477211337936_2\n", "Status of Deep Learning Model: MLP: [1024, 1024], 6.9 MB, predicting C785, 10-class classification, 606,208 training samples, mini-batch size 128\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "
input_neuronsratemomentum
7170.00311290.99
" ], "text/plain": [ " input_neurons rate momentum\n", "-- --------------- ---------- ----------\n", " 717 0.00311292 0.99" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "ModelMetricsMultinomial: deepwater\n", "** Reported on train data. **\n", "\n", "MSE: 0.00457317073525\n", "RMSE: 0.0676252226262\n", "LogLoss: 0.156785286247\n", "Mean Per-Class Error: 0.00458892837481\n", "Confusion Matrix: vertical: actual; across: predicted\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
0123456789ErrorRate
984.00.00.00.00.02.00.01.00.00.00.00303953 / 987
0.01090.00.00.00.00.00.00.00.00.00.00 / 1,090
0.00.0925.03.00.00.00.00.05.00.00.00857458 / 933
0.01.00.01032.00.05.00.01.02.00.00.00864559 / 1,041
0.00.00.00.0955.01.00.00.00.03.00.00417104 / 959
0.00.00.00.00.0899.00.00.01.00.00.00111111 / 900
0.00.00.00.00.06.0982.00.03.00.00.00908179 / 991
0.01.00.02.00.00.00.01014.00.00.00.00294993 / 1,017
0.00.00.01.00.00.00.00.0957.01.00.00208552 / 959
0.00.00.01.01.01.00.02.01.0957.00.00623056 / 963
984.01092.0925.01039.0956.0914.0982.01018.0969.0961.00.004573245 / 9,840
" ], "text/plain": [ "0 1 2 3 4 5 6 7 8 9 Error Rate\n", "--- ---- --- ---- --- --- --- ---- --- --- ---------- ----------\n", "984 0 0 0 0 2 0 1 0 0 0.00303951 3 / 987\n", "0 1090 0 0 0 0 0 0 0 0 0 0 / 1,090\n", "0 0 925 3 0 0 0 0 5 0 0.00857449 8 / 933\n", "0 1 0 1032 0 5 0 1 2 0 0.00864553 9 / 1,041\n", "0 0 0 0 955 1 0 0 0 3 0.00417101 4 / 959\n", "0 0 0 0 0 899 0 0 1 0 0.00111111 1 / 900\n", "0 0 0 0 0 6 982 0 3 0 0.00908174 9 / 991\n", "0 1 0 2 0 0 0 1014 0 0 0.00294985 3 / 1,017\n", "0 0 0 1 0 0 0 0 957 1 0.00208551 2 / 959\n", "0 0 0 1 1 1 0 2 1 957 0.00623053 6 / 963\n", "984 1092 925 1039 956 914 982 1018 969 961 0.00457317 45 / 9,840" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Top-10 Hit Ratios: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
khit_ratio
10.9954268
20.9962398
30.9962398
40.9962398
50.9962398
60.9962398
70.9962398
80.9962398
90.9962398
101.0
" ], "text/plain": [ "k hit_ratio\n", "--- -----------\n", "1 0.995427\n", "2 0.99624\n", "3 0.99624\n", "4 0.99624\n", "5 0.99624\n", "6 0.99624\n", "7 0.99624\n", "8 0.99624\n", "9 0.99624\n", "10 1" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "ModelMetricsMultinomial: deepwater\n", "** Reported on cross-validation data. **\n", "\n", "MSE: 0.0469650888528\n", "RMSE: 0.216714302372\n", "LogLoss: 1.61860098993\n", "Mean Per-Class Error: 0.0473418169932\n", "Confusion Matrix: vertical: actual; across: predicted\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
0123456789ErrorRate
5781.02.016.09.08.024.045.013.017.08.00.0239743142 / 5,923
1.06621.040.014.014.05.05.014.024.04.00.0179472121 / 6,742
27.033.05642.055.032.014.033.040.064.018.00.0530379316 / 5,958
15.019.081.05728.09.0120.011.035.074.039.00.0657315403 / 6,131
14.026.031.06.05535.014.040.035.023.0118.00.0525505307 / 5,842
35.017.011.062.09.05153.056.013.041.024.00.0494374268 / 5,421
39.012.018.02.013.049.05759.02.023.01.00.0268672159 / 5,918
14.031.047.018.035.015.03.05978.018.0106.00.0458101287 / 6,265
26.051.047.072.013.0102.033.011.05473.023.00.0646043378 / 5,851
18.019.018.046.099.049.04.0121.063.05512.00.0734577437 / 5,949
5970.06831.05951.06012.05767.05545.05989.06262.05820.05853.00.04696672,818 / 60,000
" ], "text/plain": [ "0 1 2 3 4 5 6 7 8 9 Error Rate\n", "---- ---- ---- ---- ---- ---- ---- ---- ---- ---- --------- --------------\n", "5781 2 16 9 8 24 45 13 17 8 0.0239743 142 / 5,923\n", "1 6621 40 14 14 5 5 14 24 4 0.0179472 121 / 6,742\n", "27 33 5642 55 32 14 33 40 64 18 0.0530379 316 / 5,958\n", "15 19 81 5728 9 120 11 35 74 39 0.0657315 403 / 6,131\n", "14 26 31 6 5535 14 40 35 23 118 0.0525505 307 / 5,842\n", "35 17 11 62 9 5153 56 13 41 24 0.0494374 268 / 5,421\n", "39 12 18 2 13 49 5759 2 23 1 0.0268672 159 / 5,918\n", "14 31 47 18 35 15 3 5978 18 106 0.0458101 287 / 6,265\n", "26 51 47 72 13 102 33 11 5473 23 0.0646043 378 / 5,851\n", "18 19 18 46 99 49 4 121 63 5512 0.0734577 437 / 5,949\n", "5970 6831 5951 6012 5767 5545 5989 6262 5820 5853 0.0469667 2,818 / 60,000" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Top-10 Hit Ratios: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
khit_ratio
10.9530333
20.9582833
30.9582833
40.9582833
50.9582833
60.9582833
70.9582833
80.9582833
90.9582833
101.0
" ], "text/plain": [ "k hit_ratio\n", "--- -----------\n", "1 0.953033\n", "2 0.958283\n", "3 0.958283\n", "4 0.958283\n", "5 0.958283\n", "6 0.958283\n", "7 0.958283\n", "8 0.958283\n", "9 0.958283\n", "10 1" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Cross-Validation Metrics Summary: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
meansdcv_1_validcv_2_validcv_3_valid
accuracy0.95303090.00090080.95309760.95455680.9514383
err0.04696910.00090080.04690230.04544320.0485617
err_count939.333316.756426938.0911.0969.0
logloss1.6186840.03093171.61799921.56545451.6725984
max_per_class_error0.08088480.00010570.08109500.08076130.0807980
mean_per_class_accuracy0.95261110.00095490.95260680.95426710.9509594
mean_per_class_error0.04738890.00095490.04739320.04573290.0490406
mse0.04696750.00090300.04690380.04543630.0485624
r20.99437440.00007850.99441340.99448660.9942232
rmse0.21669990.00208250.21657290.21315800.2203688
" ], "text/plain": [ " mean sd cv_1_valid cv_2_valid cv_3_valid\n", "----------------------- --------- ----------- ------------ ------------ ------------\n", "accuracy 0.953031 0.000900847 0.953098 0.954557 0.951438\n", "err 0.0469691 0.000900847 0.0469023 0.0454432 0.0485617\n", "err_count 939.333 16.7564 938 911 969\n", "logloss 1.61868 0.0309317 1.618 1.56545 1.6726\n", "max_per_class_error 0.0808848 0.000105659 0.081095 0.0807613 0.080798\n", "mean_per_class_accuracy 0.952611 0.000954862 0.952607 0.954267 0.950959\n", "mean_per_class_error 0.0473889 0.000954862 0.0473932 0.0457329 0.0490406\n", "mse 0.0469675 0.00090298 0.0469038 0.0454363 0.0485624\n", "r2 0.994374 7.84954e-05 0.994413 0.994487 0.994223\n", "rmse 0.2167 0.00208254 0.216573 0.213158 0.220369" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Scoring History: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
timestampdurationtraining_speedepochsiterationssamplestraining_rmsetraining_loglosstraining_classification_error
2016-10-23 01:32:02 0.000 secNone0.000.0nannannan
2016-10-23 01:32:0359.557 sec15226 obs/sec0.068266714096.00.42372326.18957570.1795732
2016-10-23 01:32:08 1 min 4.816 sec21045 obs/sec1.843227110592.00.24922592.13968200.0620935
2016-10-23 01:32:13 1 min 9.947 sec24762 obs/sec4.164266761249856.00.15093660.78501670.0227642
2016-10-23 01:32:18 1 min 14.976 sec27588 obs/sec6.8266667100409600.00.09773870.32994360.0095528
2016-10-23 01:32:23 1 min 20.077 sec28755 obs/sec9.4208138565248.00.08828940.26681990.0078252
2016-10-23 01:32:25 1 min 21.663 sec28923 obs/sec10.1034667148606208.00.06762520.15678530.0045732
" ], "text/plain": [ " timestamp duration training_speed epochs iterations samples training_rmse training_logloss training_classification_error\n", "-- ------------------- ---------------- ---------------- --------- ------------ --------- --------------- ------------------ -------------------------------\n", " 2016-10-23 01:32:02 0.000 sec 0 0 0 nan nan nan\n", " 2016-10-23 01:32:03 59.557 sec 15226 obs/sec 0.0682667 1 4096 0.423723 6.18958 0.179573\n", " 2016-10-23 01:32:08 1 min 4.816 sec 21045 obs/sec 1.8432 27 110592 0.249226 2.13968 0.0620935\n", " 2016-10-23 01:32:13 1 min 9.947 sec 24762 obs/sec 4.16427 61 249856 0.150937 0.785017 0.0227642\n", " 2016-10-23 01:32:18 1 min 14.976 sec 27588 obs/sec 6.82667 100 409600 0.0977387 0.329944 0.00955285\n", " 2016-10-23 01:32:23 1 min 20.077 sec 28755 obs/sec 9.4208 138 565248 0.0882894 0.26682 0.0078252\n", " 2016-10-23 01:32:25 1 min 21.663 sec 28923 obs/sec 10.1035 148 606208 0.0676252 0.156785 0.00457317" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Examine the trained model\n", "model_crossvalidated" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note: The validation error is based on the\n", "parameter `score_validation_samples`, which can be used to sample the validation set (by default, the entire validation set is used)." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.03949693398541933" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## Validation error of the original model (using a train/valid split)\n", "model.mean_per_class_error(valid=True)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.004588928374813425" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## Training error of the model trained on 100% of the data\n", "model_crossvalidated.mean_per_class_error(train=True)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.04734181699323046" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## Estimated generalization error of the cross-validated model\n", "model_crossvalidated.mean_per_class_error(xval=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Clearly, the model parameters aren't tuned perfectly yet, as 4-5% test set error is rather large." ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false }, "outputs": [], "source": [ "#ls ../../h2o-docs/src/booklets/v2_2015/source/images/" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## Predicting\n", "\n", "Once we have a satisfactory model (as determined by the validation or crossvalidation\n", "metrics), use the `h2o.predict()` command to compute and store\n", "predictions on new data for additional refinements in the interactive data science\n", "process." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "deepwater prediction progress: |██████████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "predictions = model_crossvalidated.predict(test_df)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Rows:10000\n", "Cols:11\n", "\n", "\n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
predict p0 p1 p2 p3 p4 p5 p6 p7 p8 p9
type enum int real int real real real real int real real
mins 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
mean NaN 0.0975 0.1154 0.1009 0.1010135065110.09459999965430.09438649284490.09449999998850.1015 0.0994000006540.100800000349
maxs 9.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
sigma NaN 0.2966522379070.3195200295830.301211325860.3013422760630.292676280751 0.292360290599 0.292537703227 0.3020047527560.2992132896940.301078768385
zeros 975 9025 8845 8991 8984 9053 9053 9053 8985 9004 8987
missing0 0 0 0 0 0 0 0 0 0 0
0 8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0
1 3 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
2 8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0
3 0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 1 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 5 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0
6 0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
7 1 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
8 5 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0
9 8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "predictions.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Variable Importance\n", "\n", "Variable importance allows us to view the absolute and relative predictive strength of\n", "each feature in the prediction task.\n", "Each H2O algorithm class has its own methodology for computing variable importance.\n", "\n", "You can enable the variable importance, by setting the `variable_importances` parameter to `True`.\n", "\n", "H2O’s Deep Learning uses the Gedeon method [Gedeon, 1997](http://users.cecs.anu.edu.au/~Tom.Gedeon/pdfs/ContribDataMinv2.pdf), which is disabled\n", "by default since it can be slow for large networks. \n", "\n", "If variable importance is a top priority in your analysis, consider training a Distributed Random Forest (DRF) model and compare the generated variable importances." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Train Deep Learning model and validate on test set and save the variable importances\n", "from h2o.estimators.deeplearning import H2ODeepLearningEstimator ## H2ODeepWaterEstimator doesn't yet have variable importances\n", "\n", "model_variable_importances = H2ODeepLearningEstimator(\n", " distribution=\"multinomial\",\n", " activation=\"RectifierWithDropout\", ## shortcut for hidden_dropout_ratios=[0.5,0.5,0.5]\n", " hidden=[32,32,32], ## smaller number of neurons to be fast enough on the CPU\n", " input_dropout_ratio=0.1,\n", " sparse=True,\n", " epochs=1, ## not interested in a good model here\n", " variable_importances=True) ## this is not yet implemented for DeepWaterEstimator" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "deeplearning Model Build progress: |██████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "model_variable_importances.train(\n", " x=x,\n", " y=y,\n", " training_frame=train_df,\n", " validation_frame=test_df)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0123
0C3481.0000001.0000000.002358
1C3760.9989980.9989980.002355
2C3490.9310290.9310290.002195
3C3770.9208400.9208400.002171
4C4030.9186170.9186170.002166
5C4350.8964820.8964820.002113
6C4340.8933350.8933350.002106
7C2960.8814380.8814380.002078
8C5700.8780900.8780900.002070
9C3800.8779600.8779600.002070
10C3240.8623760.8623760.002033
11C4620.8608620.8608620.002029
12C3500.8590920.8590920.002025
13C4610.8563300.8563300.002019
14C4070.8552720.8552720.002016
15C4880.8450430.8450430.001992
16C4910.8436130.8436130.001989
17C6570.8354340.8354340.001970
18C5130.8240580.8240580.001943
19C6580.8236820.8236820.001942
20C3750.8215780.8215780.001937
21C4630.8214070.8214070.001936
22C4020.8197040.8197040.001932
23C3510.8186270.8186270.001930
24C4580.8037310.8037310.001895
25C2690.7986740.7986740.001883
26C5670.7955900.7955900.001876
27C5170.7940200.7940200.001872
28C6280.7935770.7935770.001871
29C3780.7931400.7931400.001870
...............
687C5410.4582320.4582320.001080
688C750.4577600.4577600.001079
689C620.4569770.4569770.001077
690C2010.4556600.4556600.001074
691C6070.4553030.4553030.001073
692C6020.4541750.4541750.001071
693C7080.4525020.4525020.001067
694C1970.4516190.4516190.001065
695C6430.4506580.4506580.001062
696C2270.4497760.4497760.001060
697C1750.4484220.4484220.001057
698C480.4481590.4481590.001057
699C2370.4453730.4453730.001050
700C6000.4442640.4442640.001047
701C7330.4436740.4436740.001046
702C3370.4432980.4432980.001045
703C1700.4416260.4416260.001041
704C2540.4336530.4336530.001022
705C6910.4328080.4328080.001020
706C1350.4320970.4320970.001019
707C2230.4289170.4289170.001011
708C3440.4255980.4255980.001003
709C3080.4193130.4193130.000989
710C4260.4192000.4192000.000988
711C7070.4154910.4154910.000980
712C3960.4153220.4153220.000979
713C7760.4128110.4128110.000973
714C6650.4091490.4091490.000965
715C3360.4066790.4066790.000959
716C6900.3954910.3954910.000932
\n", "

717 rows × 4 columns

\n", "
" ], "text/plain": [ " 0 1 2 3\n", "0 C348 1.000000 1.000000 0.002358\n", "1 C376 0.998998 0.998998 0.002355\n", "2 C349 0.931029 0.931029 0.002195\n", "3 C377 0.920840 0.920840 0.002171\n", "4 C403 0.918617 0.918617 0.002166\n", "5 C435 0.896482 0.896482 0.002113\n", "6 C434 0.893335 0.893335 0.002106\n", "7 C296 0.881438 0.881438 0.002078\n", "8 C570 0.878090 0.878090 0.002070\n", "9 C380 0.877960 0.877960 0.002070\n", "10 C324 0.862376 0.862376 0.002033\n", "11 C462 0.860862 0.860862 0.002029\n", "12 C350 0.859092 0.859092 0.002025\n", "13 C461 0.856330 0.856330 0.002019\n", "14 C407 0.855272 0.855272 0.002016\n", "15 C488 0.845043 0.845043 0.001992\n", "16 C491 0.843613 0.843613 0.001989\n", "17 C657 0.835434 0.835434 0.001970\n", "18 C513 0.824058 0.824058 0.001943\n", "19 C658 0.823682 0.823682 0.001942\n", "20 C375 0.821578 0.821578 0.001937\n", "21 C463 0.821407 0.821407 0.001936\n", "22 C402 0.819704 0.819704 0.001932\n", "23 C351 0.818627 0.818627 0.001930\n", "24 C458 0.803731 0.803731 0.001895\n", "25 C269 0.798674 0.798674 0.001883\n", "26 C567 0.795590 0.795590 0.001876\n", "27 C517 0.794020 0.794020 0.001872\n", "28 C628 0.793577 0.793577 0.001871\n", "29 C378 0.793140 0.793140 0.001870\n", ".. ... ... ... ...\n", "687 C541 0.458232 0.458232 0.001080\n", "688 C75 0.457760 0.457760 0.001079\n", "689 C62 0.456977 0.456977 0.001077\n", "690 C201 0.455660 0.455660 0.001074\n", "691 C607 0.455303 0.455303 0.001073\n", "692 C602 0.454175 0.454175 0.001071\n", "693 C708 0.452502 0.452502 0.001067\n", "694 C197 0.451619 0.451619 0.001065\n", "695 C643 0.450658 0.450658 0.001062\n", "696 C227 0.449776 0.449776 0.001060\n", "697 C175 0.448422 0.448422 0.001057\n", "698 C48 0.448159 0.448159 0.001057\n", "699 C237 0.445373 0.445373 0.001050\n", "700 C600 0.444264 0.444264 0.001047\n", "701 C733 0.443674 0.443674 0.001046\n", "702 C337 0.443298 0.443298 0.001045\n", "703 C170 0.441626 0.441626 0.001041\n", "704 C254 0.433653 0.433653 0.001022\n", "705 C691 0.432808 0.432808 0.001020\n", "706 C135 0.432097 0.432097 0.001019\n", "707 C223 0.428917 0.428917 0.001011\n", "708 C344 0.425598 0.425598 0.001003\n", "709 C308 0.419313 0.419313 0.000989\n", "710 C426 0.419200 0.419200 0.000988\n", "711 C707 0.415491 0.415491 0.000980\n", "712 C396 0.415322 0.415322 0.000979\n", "713 C776 0.412811 0.412811 0.000973\n", "714 C665 0.409149 0.409149 0.000965\n", "715 C336 0.406679 0.406679 0.000959\n", "716 C690 0.395491 0.395491 0.000932\n", "\n", "[717 rows x 4 columns]" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Retrieve the variable importance\n", "import pandas as pd\n", "pd.DataFrame(model_variable_importances.varimp())" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABIUAAANLCAYAAADfCS3nAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xm8JFV9///XmxEUZxQhRmCQgEoEGQSCQCKKE0FBlqBi\nRNyiovm6EEdFoibqV436Ff2hUSC4BBI33EgMICIYokYUBVFRGRBhgAFkdxCQYZ05vz9ONben6eV2\nzx1mmHo9H4963NtVp845VV1d3fWpc06llIIkSZIkSZLaZZ3VXQFJkiRJkiQ98AwKSZIkSZIktZBB\nIUmSJEmSpBYyKCRJkiRJktRCBoUkSZIkSZJayKCQJEmSJElSCxkUkiRJkiRJaiGDQpIkSZIkSS1k\nUEiSJEmSJKmFDApJ0homyXVJlic5dhXlv3eT//Iku65EPh9q8lg6k/WTJGl1W9XfxZK0pjAoJGmt\nkeRTXcGOvxxz3Wd3rfvPq6iK01Wa6YEo50EvyY8NTunBLMlru84/B01znaHHfZJ1k+yb5ONJzk5y\nY5K7kyxJ8pMmqLvFBHXdL8m/JrmwyeuOJIuTnJnksCSPGTfPAeV0B6+7p3uS/C7JZUm+l+TIJM9L\n8pCZKHdtM8mxpfs8UN/FkrRaGRSStDb5fPO3AC8bc92Xd637uRmr0crxx+j0uJ9Wkq2+1hjjHMsD\n0yaZC1wPnAosAP4c2AiYBWwA7AS8HbgoyeumU1iSHZKcA3wDeDWwdZPXesBjgWcCRwKXJvn7MbZj\nlNIzrQM8CtgC2B04DPg6cFWStyfxt21/nicn576TtFbzroqktUYp5ewki4AnAC9Icmgp5a5R6yV5\nOPB86g+/haWU81dxVYcqpWy6OsuX9KC3PjVwUoCfAqcA5wA3NvP3Aw4FHgb8S5I/lFK+OCizJHsA\n/wU8osnzHOALwAXA7dSg0D7A3wCzgQ8n2RY4pJQyExfUHweO73o9hxrk2gHYs5keA3wI+Ksk+5dS\nfj8D5arF/C6W1BYGhSStbb4AvBd4JPBc4GvTWOdA6oVMYaq1kSQ9WBXgNODdpZSf91n+vSQnAWdS\nW/p8LMmJ/YLoSR5HbYkzB1gOLCil9I6x8jPglCRHAicD21IDRFcA75uB7bm+lHJhn/mnUwNQ2wFf\nBJ4MPBU4McnepZTlM1C2JElrNZvYSlrbfIGppt7T7ULW6Tq2HDhhxmskSQ+gUsplpZT9BwSEOml+\nCBwHBPgjavevfo6nBtkBDu8TEOrOcxHwLOCGJt93Jtl+gk0YSynlAmA34MKm3D2Av13V5UqStDYw\nKCRprVJKuRz4IfXCYO8kjx6WPsmm1K4HBfhOKeXaPmm2T/LuJN9OcnWSu5LcluTiJMcnecqIMlYY\nryXJo5K8N8n5SW7uHQB01BNPksxN8ndJ/jPJJUlubwZ7vSrJ15McOGo/9eS3TpLXJ/lRM3DsbUl+\nluTwJOuNk9eA/B+W5M1JvtNs211Jrk9yepK/WdVjgHQNyHta83qbZqDcy5MsbQas/VSSzXrW2yHJ\nF5rlncF0j07yR0PK+kpT1oXN68cm+UTzPi1ttvuUJHtOs+47NnXtvM+3JPlVM7juZkPW27p3cNkk\nBzX7/Jok9yb5VmcQWur4MgAPS//BfR/Tlfc6SZ6V5KO5/wDGP01yxLC6NXn0viebN/vp0mZf35jk\nm2Psp42TvK85hm/oqs+Pms/ftkPWXa3H52r23a7/n9C7MMkuwF/SdEMrpXx8VIallOuAtzUvZwEz\nOb7QsHKXUlsndbwtSQalb86jH2qO2e4Bs7+c5BnTKXPSPAZ8Pl+c5LvN8Xt7koVJ3p/kEdPdB6vS\nyuyvJBsleXWSE1IHKL+t+Zxdm+S0JK/KkIHCp3k+O60rfe95eKMkH2zKvr2p/3czYuDtDPkuTp+n\neCZ5Serg5zemnu8vbMrdYPjehSRbJvlM6vfSHam/Nf4jydOb5Y77JmnVKaU4OTk5rVUT8Bpqq59l\nwN+NSHt4V9qX9Fm+d7O8k6Z36iz7v0PK+FCTZimwDXBln/wO6kp/bTPv2D55rd9V7rD6fAN42ID6\n7N21/nzgfwbktxz4ObDRqO0asu1PAa4aUd8fDCpjmu/3j4bVo1m+jNqdZh/gDwPqcTXwuGadVwJ3\nDUh3MfDoAWV9uUlzIfAXwE1D3qMPjNiu9wL3Dnlvbu8+bnrW3bprvYOBr/bJ5zTg//Tk2e89uhd4\nTFfeRwxJ28nnVmDfEe9Zpw7zgSVD9tOhI/bTq5r3dNgxduGqOD679vNy4LSVOIZf21WHvu/puMf9\nNPN4cVe5r+uz/F+6lr90jHwfQh3oejlwJ7D+BHXrPk+9bYz1vte13o5Djpnbh3y2ljfbnhHH3UR5\nsOLn80XUrm+D8rkCePwDeWzN9P4CrhvxGVsO/Bj4oxGfs6Hns6703efhedTv3EHlfmTIdg/7Lu4+\nPp9G7ao+aP8sZMh3HFPfS/3Wv5c6mPrI71snJyenSae1+e6XpPb6GvVCBKa6hg3S6WL2B+pAqr0e\nAtwGfIl6AT2f+uSefah3w6+i3kV/z6i7jtTWmV8HHg18lNpCaeemDotGrNudxzLgDGpAa2/qhe0z\nqcGwc5v67Av88zTy+wi1JcCp1DGYdgb+mtqCoADbU8cIGVuSJzX5zAVuBt4PPK8p4znAp6g/eHcD\nvj7srv4M2ZJ6sXA98HrqE5meQb2YKcCmwKeTPI3areYianBoF+p79ZUmn62o+22YRwInUgfy/QD1\nKUl/AbylKR/gH5L07eKS5DDg/1JbvF0LvKlZ/xlNfkupAcITkgzq9tPxDup7+j/UC6qdgb2oF1Zf\npY7D8m9N2ruB7Zp5nWl7anCrYxb1uD8aeCn1/duZOlj7x6ifpTnAV5I8fkTdtqB+JpZSj+enAbtS\nP1u3NGmOTHK/ViwAzf47nrovbqce8/tQP6PzqU/eOpP6melddyaPzwfr04nmd/1/UZ/l3S1ATp1u\npqWUe4FvNS/XpR67D5Qzu/7fvXdhkpdTj5mHUQO8b2LqPPpC6rm1AK+jHhP3MxN5dDkMeAk1+Pii\nJo/9qZ+LAmwOnJ7kYSPyWSVmaFtD3b53Ugc53wV4OvX7+b+b9Xehdv8eZdj5rNcjqDdI5lCD7POb\n9K8DrmnSvHW6LcOG+DDwAupvj+cx9R6e0SzfBvj/+q2YZGvgP6jnsLup57A9qPvjNcAlzbqjzvOS\nNLnVHZVycnJyWhUT9QK+c9ftTwekeXJXmn8bkObRwCOGlLMe9cJyOXDRgDSdO3zLqa1Pnj6i7sPu\nTs4Cthix/v9ryrobeGyf5d13OJcBHxuQzxe60r1yyHYNaqFzXrPu2cAGA9Ic0FWPabdE6MljOi2F\nOvv/l/3qAhzVta3X0wzA2yfdyZ2ygEf2Wf7lrrKWArv2SbM59YJkOTUY8cie5ZsCdzR1uRzYuE8e\nu1KDIMuAS+m5Q8+Kd9aXAZ8csQ+nfReaGshZZ8jyP+k6hj89jffk18Af90mzR9c2fGjAflzaLL+K\nAZ/zJu1mq+L4ZNW0FDqM2rph1HT+dN+zIe9Tp3XCVcCsnuUP6dr2RRPkv6Brm948wfqTthTat2u9\no3uWbUJtxbaMGtQc1LLlSKbO13+yCvLo/nwuB04ckMf7u9K9dwaOrbFaCs3EtjbLnzBGHZ/aZ/m4\n57Pu8/AN/cqnBmrubPL78oB8pttSqO8xTr2J0/l9MOg741vN8nuAvfosfzh1IPf7vlcmOQ6cnJyc\nhk22FJK0tup+itig1kJ/0/V/3zuUpZSbSim3DSqklHI3U+OxPLFpfTAwOfUi+QdD0gxVSllWSlk8\nItl7qK0sZgF/NSLt1UzVv9eh1KAFwBumXUkgybOorTUK8PJSyi390pVSTqHeyQ21Vc6qVKhdkfrV\npTNmRKiPun518972+mTz96HUwMywso4qpZx7vwWlXMXUPn8k9x8Q/TVN/gBvLKVc37OcJt+PNvV9\nHPVCeJAbqS2UZkQpZXEZ8lSnUsqV1LvdobY+G5od8IZSyo198vkONfAR+rT4oG5Tp/XEq0oplwyp\n02+7X8/w8Vm6pplwJPCraUwTD+DctHr6DPWCswDvKaX0tqbaiLrdULv/jKv7uB04Dtcq8Luu/zfs\nWfZGaquRy6kX8YPes3+gfm4ewv2/P2Yij45Qg7uvG7D8vdSWIgH+zwPQmrLXjGxrqQOQD1RK+TRT\nLdWeN6JO45zPCvCOfuWXUn5N7b466PwyjrNLn/G2mvNkp8Xu/b4zkmxJbeVUgBNKKd/uk8dSatBM\nklYZg0KS1lZnUC9KQu3isoLmx/WLm5dXl1K+O51Mkzw0dVDcJyWZl2QeUxdOADuMyOJL0ylnulIH\n/Z3bDMTZqc/WTDWNH1WfL5dS7um3oJRyK7ULQ4CdpjNYZpcDmr+/HHVBAHy/+buqu5hcX0o5q9+C\n5gLhLuqP858MCbz9ouv/UV2jPjtk2deoF4NQn9bUrfP6hlLKsC47/9pnnX5OKqXcOWT5SknyyCSP\nS7Jt1zHY2bY/TrLJkNVvaII/g/y0+dtvX+/X/L2olHJmn+XDzMjxWUq5uJQyq5n2610+oTLGNKl/\nYupi9FullH/rk6Z7gOM/TFBG9zqPHJhq5nWX2ztI819Rt/mUPkGw+zTnxHOo576nroI87ksGfLOU\n8ru+C2v+nRscG1Nbtz6QZnJb75M6MPwTO+eL5pzx22b9Ud9Z45zPltG/W1lH5/yyaVbuoQrDnlr6\n067/e89jezD1++GLgzIopfyE2nVPklaJgSP9S9KDWSllWZIvUe8obpnkaaU+grljT+pYIoUhP8YA\nkswB3gwcBDyJ2gJnkKFPO6N2X1opTUDrFdSWC7sy1VqiV5lGfX4yYvm5wKupP1y3oz7ZbTp2bv7u\nmPp0q+l4eJI5pZRJLkCn4zcjlt9K3V/D0v2+6/9hTwX6QxNo6quUcleSX1IvoHov9LajvnfnDats\nKeWqJNdRLxa3G5J0pY+5XkkeR32q1H7UblzDPJrBLU1GXegsaf6usK+TPBz4U+p+6hvoG2FNPD47\nDi6lnDgqUZIfUcfFGkuSQ6hjuxTq/v+bAUm7W0jOGbecnnVunWD9SXUfK/eV24zJM695+eYkb55m\nfvcFNWcijz6mcw7ueDKr4PPcz0xva5LnUcflezrDj6eZ/A69ppRy+5DlS7r+fwQrtjIbx8BzfZ8y\nunWft3/KcOdRb/hI0owzKCRpbfZ5ppqZv5wVAxrdTdwHDm6Z5E+p48tsztSd+X536Dt3+9YfUp97\nm6bgE0uyPrU7yx4j6tMxrD5Qx1sYprsLyEYj0nZ7DOO3ZCjU7iyr4qK7UMd0GKYTHBiWrjuAMCw4\neNOQZR2dfXvffm0Cfo9qXo56b6AGWzZh+Htz85BlY0vyXOqYHQ9jeIuVUZ+Jcd6T3n3dfeF47Yg8\n+lnTjs9uq6yLUJIDgU9Tt+Uq6hgmSwYkX8LUPhoW1Bhk467/J73YnkT3sbGkZ34Yv5VV9/E7E3n0\nWlXn4JU1I9uaZB3gc0y12B11zhj1nTXO+Wy65xcYfj5fmXKGldHp3ri8lDJqu+7XxVaSZopBIUlr\nrVLKL5L8inp39YVJ3lhKuadpZXAg9YfpT4e16KB299qc+sPuM9QnSv0auKnT7aoJ1HTuRg67oBvY\n/H4M/8RUQOhM6hOSfg5c192kPsk51NYQoy4wZ2oclF6dH78/AQ4ZY73pBFMeDGZiv87UezMTxx1Q\nu31Qg6gPpY5b9RHqcbgIuLXUp06RZB/gm53VZqr8GdS64zPJXtTz2SzqBeZepZSrB6Uvpdyb5CJg\nW2pryw0Gjb00wE5d//9iYKqZ92dd/3e3ROu+ID+WqfHBRrlrhvPotarOwStrprb19dSAUKF+3o5q\n/v62+yZJkq9Sn2Y26nwxY+czSVJlUEjS2u7z1Me5Poo6PsLXqQGh2dQfqZ8btGKSHaiPli3Au0sp\nHxqQ9AG5e9vccX1VU5//LqU8Z0jy6dZp4zGWD2pR0M/vqE+pengp5cIx1ltb/PE00nT27X37tZRS\nkvyeeryOem+gtuAojPferIyDqV0/CrBvKeVHA9Kt6s9Ed3Bm0wnWb9XxmeTp1HPfetQukHuVUkZ1\np4Q6ntK2zf9/xYiutl3lPQTonJ/uAX48VoVXzrO7/u8e1L+7tVKZ8H2fiTx6rapz8MqaqW19DfV8\ncSH1yZv3Dkj3QLaCWlN0Wgetk2TDEa2FpvOdIkkTcaBpSWu7E5i6s9h5ylOn69g91EfXDzKv6/+v\nDUm385BlM6m7m9DAMUeSbMjoQZA7dhlj+QXTzBNq6yWAbZI8amjKtdOcJNsMWtgMaro99WKpd79e\nQL1bPvS4SvJYprr1jPPe9DPd1gqdz8Q1QwJCsIo/E00Lg99Q99MzJsiiNcdnkqcApzLV9W3fUsp0\nW+58tuv/vxuj2Bcz1UXva6WUO8ZYd2LNtj69KXdRKeW+8WeasaA6g4o/bZL8ZyKPPlbVOXilzOC2\ndoKKJw0KCDU3PHZkzW01taos7Pr/KSPSPlC/MyS1kEEhSWu1Usp11O4tAfZtnnLS6X51+qCnvjS6\nW1POHpLu9Std0emZbn1ey/S77BycZN1+C5I8AngBdV/9bMyuI6c0f2dRH2vcRq8YsuxFTL2HvU/O\n6rx+TJL9h+Txt33WmVSn62HfY6FL5xh8+KAEzcDsL1nJ+kzHN5q/2zRdo8bRiuMzybbA6dSnf90J\nHFBKmXarnVLKucD/Us8nuyRZMI0yN6F2K4QakD9y3HpPoukW3N3y88N9kp1C3ZYdm9ZTk5iJPDo6\n30t9W8kkmcXUQOA3AL9ayfLGtVLb2oyR1umGNuw76yDgj8av3oPed5gKhL18UKIku+Ig05JWIYNC\nktqg80jfdaktg2b1zB/kkq7/X9kvQZK3UB/t/EC4hqmxi17aXDD01udpwLuZ/h3XzYFB3eKOYWog\nzGPHqCellG9QxxEJ8M4kBwxLn2THJMO6wz3YBFiQ5H53d5sWPp19fhv375JzHHVcjgCfSPKYPnns\nDLy1eXkZcNpK1rczWPM6SbYckq7zmXhUv/e0OSY/ywPT1eEoaqAjwPFJnjgoYZLNul/P1PGZZOsk\ny5tpZd+DGZXk8cC3qRfbdwN/XUr53gRZvZp6nAb4aJLXDSnzCdQA5cbUc9AHulvrrCpJtqd2Udu2\nKffbwL/3Sfox6qDAAb447Jhp8j2gT4u/mcijo1CDJZ9qAii93gM8sUn3mVLKdJ+UN1NWaltLKYXa\n2ijA85sbDb3rbA18nLqNa+L4Y6tMKeVypm5avSTJ3r1pksymjh3YtlZUkh5AjikkqQ3+i3pRM4ep\npuw3M9XSYJBzqF1Unkh9JO8fU5+6dC11PJJXAM+ljluxsneMR2oGfv0K9SJtF+CsJB+n/uh+FHAA\n9ZG/S6jBo62mke15wGHNj/3PAL+lbtsbgWdSf4j+uJTS7wJrlIOBHwEbAP+V5CRqt7dLqQN3b0wd\njPa51KbxH6C2algbXEPdxu8l+ShwBrW74m7AO5i6aH57bwusUsq1Sf4R+CjwOOBnSY6gHo/rAntT\nA0IPp7bE+Nvm4mtlnN31/9FNedczdSFyWVPGV4D3NvX4UpJ/Bv6H2i1pe+pxsz0PwGeilHJVkjdS\nj9vNqPvp09R9fT318c9PBp4HPJYVu4PCzB6fa9QFWxNIPJM63lIBjgAWNy0lB1lSSrnfk9xKKZcl\neQHwn9Rz6LFJXkENqv+KGjTYDNiP2tphfabGa3v/DG3Sxj11n03tSrsD8Cxgz051qeMgvahfAKWU\n8tskr6EGYv+Eesz8O/V9/S11zKXNgb8A/pp6LnwWXY8cn4k8epzXpPt+kk9Qz+ebUAdAf0GT5jLq\ne7iy5jcPRhjl66WU22ZoWz9PPQ62AH6c5CPU8YXWp47/tIB6k/p8VhwkvC3eBPyM+jTHU5IcQ22h\ndRv1+H479bv8J8CurGHnGklriVKKk5OT01o/AcdTL/Q6079Mc72dqUGWZT3rL2/mnUu9IOq8fluf\nPD7ULF86zTKvbfI6ts+yDYFfDqjP8mbdP6de7C4HTuuTx95d9Z1PvagftH0/Bx49oJ4jtwt4EvUC\nYFB9O+UsAw6f8L390bB6DNsX093vXWkeOuK9/nKz/ELqhdJNQ7b3/42oz3uogaRB780fgBcOWHfr\nrnQHTXM/njTkfXpMV7rXAvcOqde/Aft0vd51Jd6T6Rxjr6YGQQfVfRmwcFUcn137eeS2jNjO107w\nfg087pn6jI8zDTzumzx3oJ7vOvtj0L66pd++mmCfTHcbOvX5LfDWaeb9PAaf17vzvQv485nOgxU/\nny+iBl0GfZ6uAJ4wA8fWdKdlwBNncFvXY+o7pt/6t1AHMb/v3Lmy57NheQ353D2mz/Jh38Xd36P3\nO8d1pRv6ndGk2Zd6Pu+3f+6lBoY+3Lz+3cp+tpycnJx6J7uPSWqLz1HvsBXqD6svTGelUsp51LuX\n/wospnbDuIl6QfZm6gCcnSchDbuDV0YsH7ROb31upgYb3kcddPQO4FbqgJVHADuWUs6ZRpmdZXcw\ndbf2HOqTiW6n3rV9O/XH7rDHcA/drlLKRcB21HExvg5c2ZR5F/Ui7jvAPzX1XpmxR0bt33H2/3TS\njUxT6tgtO1G74S2idnW6ifqo9meXUv5xxPrvowYl/61Zfyn17vEF1FZEW5dSBg44zvjH3AuBf6De\nkb6F+jnpfF666/VpajDxFOqjze+mtoz6JnBgKeWQrnVn4j0ZdYwdD/wp9aLpZ9RWgPdQn570Q2or\nhb7dw2bo+CxM9vkelM9MrVMmmAYXVMovSim7UltO/Tu1Ncjvqcf11dR99ffAViv5WR61Dcuo7/Fi\n6nhHHwOeD/xJKeWj08q0lJOALannuO9SW5bdTT33LQJOprbg2KLrfDrjeUxlVV5GPQb/l3qOuAO4\nCPgg8ORSyqIh60/HSh0HK7OtpZS7qV2sD6N+Pm9vpt8ARwN/Vmp3zu56DtuGcbd5JtLNRJ2GncNO\no7ZqPI56XN9FDUidBDyrlPJh6rhgUM/NkjSjUoqtECVJmglJvky98//rUsq2o9JLap9mHJ2LqIGC\nF5dShj3dUiLJWdTux2eWUu439pAkrQxbCkmSJEnSGqgZ/P+pzctpPz1QkqbLoJAkSZIkrQbNU/sG\nLXs49YmO61Bblk2r67skjcOnj0mSJEnS6vGFJAD/wdS4aI+kPm3sDdSnUBbqAzIuXV2VlLT2Migk\nSZIkSatHqAGgv+izrDOY9VeAwx/ISklqj7UqKPSud73r4cA2wK8/8IEPLF3d9ZEktdJMPIlK0trN\n84Q6/g54HvBMYDPgj5v5NwBnA58tpZy5muomqQXWqqAQNSD00ze84Q2rux6SpBbqeqLntnjBJ6mP\nnif/frWZ1FJDngT9uGZ66QNXG0kPEpnJzBxoWpIkSZIkqYUMCkmSJEmSJLWQQSFJkiRJkqQWMigk\nSZIkSZLUQgaFJEmSJEmSWsigkCRJkiRJUgsZFJIkSZIkSWohg0KSJEmSJEktZFBIkiRJkiSphQwK\nSZIkSZIktZBBIUmSJEmSpBYyKCRJkiRJktRCBoUkSZIkSZJayKCQJEmSJElSCxkUkiRJkiRJaiGD\nQpIkSZIkSS1kUEiSJEmSJKmFDApJkiRJkiS1kEEhSZIkSZKkFjIoJEmSJEmS1EIGhSRJkiRJklrI\noJAkSZIkSVILGRSSJEmSJElqIYNCkiRJkiRJLWRQSJIkSZIkqYUMCkmSJEmSJLWQQSFJkiRJkqQW\nMigkSZIkSZLUQgaFJEmSJEmSWsigkCRJkiRJUgsZFJIkSZIkSWohg0KSJEmSJEktZFBIkiRJkiSp\nhQwKSZIkSZIktZBBIUmSJEmSpBYyKCRJkiRJktRCBoUkSZIkSZJayKCQJEmSJElSCxkUkiRJkiRJ\naiGDQpIkSZIkSS1kUEiSJEmSJKmFDApJkiRJkiS1kEEhSZIkSZKkFjIoJEmSJEmS1EIGhSRJkiRJ\nklrIoJAkSZIkSVILGRSSJEmSJElqIYNCkiRJkiRJLWRQSJIkSZIkqYUMCkmSJEmSJLWQQSFJkiRJ\nkqQWMigkSZIkSZLUQgaFJEmSJEmSWsigkCRJkiRJUgsZFJIkSZIkSWohg0KSJEmSJEktZFBIkiRJ\nkiSphQwKSZIkSZIktZBBIUmSJEmSpBYyKCRJkiRJktRCBoUkSZIkSZJayKCQJEmSJElSCxkUkiRJ\nkiRJaiGDQpIkSZIkSS1kUEiSJEmSJKmFDApJkiRJkiS1kEEhSZIkSZKkFjIoJEmSJEmS1EIGhSRJ\nkiRJklrIoJAkSZIkSVILGRSSJEmSJElqIYNCkiRJkiRJLWRQSJIkSZIkqYUMCkmSJEmSJLWQQSFJ\nkiRJkqQWMigkSZIkSZLUQgaFJEmSJEmSWsigkCRJkiRJUgsZFJIkSZIkSWohg0KSJEmSJEktZFBI\nkiRJkiSphQwKSZIkSZIktZBBIUmSJEmSpBYyKCRJkiRJktRCBoUkSZIkSZJayKCQJEmSJElSCxkU\nkiRJkiRJaiGDQpIkSZIkSS1kUEiSJEmSJKmFDApJkiRJkiS1kEEhSZIkSZKkFjIoJEmSJEmS1EIG\nhSRJkiRJklrIoJAkSZIkSVILGRSSJEmSJElqIYNCkiRJkiRJLWRQSJIkSZIkqYUMCkmSJEmSJLWQ\nQSFJkiRJkqQWMigkSZIkSZLUQgaFJEmSJEmSWsigkCRJkiRJUgsZFJIkSZIkSWohg0KSJEmSJEkt\nZFBIkiRJkiSphQwKSZIkSZIktZBBIUmSJEmSpBYyKCRJkiRJktRCBoUkSZIkSZJayKCQJEmSJElS\nCxkUkiRJkiRJaiGDQpIkSZIkSS1kUEiSJEmSJKmFDApJkiRJkiS1kEEhSZIkSZKkFjIoJEmSJEmS\n1EIGhSRJkiRJklrIoJAkSZIkSVILGRSSJEmSJElqIYNCkiRJkiRJLWRQSJIkSZIkqYUMCkmSJEmS\nJLWQQSFJkiRJkqQWMigkSZIkSZLUQgaFJEmSJEmSWsigkCRJkiRJUgsZFJIkSZIkSWohg0KSJEmS\nJEktZFAQw279AAAgAElEQVRIkiRJkiSphQwKSZIkSZIktZBBIUmSJEmSpBYyKCRJkiRJktRCBoUk\nSZIkSZJayKCQJEmSJElSCxkUkiRJkiRJaiGDQpIkSZIkSS1kUEiSJEmSJKmFDApJkiRJkiS1kEEh\nSZIkSZKkFjIoJEmSJEmS1EIGhSRJkiRJklrIoJAkSZIkSVILGRSSJEmSJElqIYNCkiRJkiRJLWRQ\nSJIkSZIkqYUMCkmSJEmSJLWQQSFJkiRJkqQWMigkSZIkSZLUQgaFJEmSJEmSWsigkCRJkiRJUgsZ\nFJIkSZIkSWohg0KSJEmSJEktZFBIkiRJkiSphQwKSZIkSZIktZBBIUmSJEmSpBYyKCRJkiRJktRC\nBoUkSZIkSZJayKCQJEmSJElSCxkUkiRJkiRJaiGDQpIkSZIkSS1kUEiSJEmSJKmFDApJkiRJkiS1\nkEEhSZIkSZKkFjIoJEmSJEmS1EIGhSRJkiRJklrIoJAkSZIkSVILGRSSJEmSJElqIYNCkiRJkiRJ\nLWRQSJIkSZIkqYUMCkmSJEmSJLWQQSFJkiRJkqQWMigkSZIkSZLUQgaFJEmSJEmSWsigkCRJkiRJ\nUgsZFJIkSZIkSWohg0KSJEmSJEktZFBIkiRJkiSphQwKSZIkSZIktZBBIUmSJEmSpBYyKCRJkiRJ\nktRCBoUkSZIkSZJayKCQJEmSJElSCxkUkiRJkiRJaiGDQpIkSZIkSS1kUEiSJEmSJKmFDApJkiRJ\nkiS1kEEhSZIkSZKkFjIoJEmSJEmS1EIGhSRJkiRJklrIoJAkSZIkSVILGRSSJEmSJElqIYNCkiRJ\nkiRJLWRQSJIkSZIkqYUesrorsCrsf9RZLGHO6q6GJEmSJEnSjLniiP1mND9bCkmSJEmSJLWQQSFJ\nkiRJkqQWmigolGTjJEcnWZTkziSLk5ySZI9m+aeSXJpkaZIbkpyUZOsBea2X5Pwky5Ns37NslyRn\nJrk5yZIkp/emkSRJkiRJ0vjGDgol2QL4GfCXwFuB7YDnAN8FjmmSnQe8EtgG2AsIcEaS9MnyI8DV\nQOkpZzbwLeAKYFfgacBtwOlJZo1bb0mSJEmSJE2ZZKDpTwLLgF1KKXd2zb8oyfEApZTjuuZfmeRd\nwPnAlsDlnQVJ9gGeDbwA2LennG2ADYH3lFJ+26R/H/ALYAvgsgnqLkmSJEmSJMZsKZRkQ2Bv4Jie\ngBAApZRb+6wzGziEGsS5qmv+xsBngJcBd/Qp7mLgd8Crk6ybZH3gNcCF1NZDkiRJkiRJmtC43ce2\nonYFu3hUwiSvT3IbtcvX3sBepZR7u5L8O3BsKeXn/dYvpfwBeCbwcmrQ6DZqV7R9SynLx6y3JEmS\nJEmSuozbfazfmECDfBH4NrApcDhwYpLdSil3J1kAzAE+PCjfJA8Djgd+ALyoqevhwGlJdi6l3NWv\n0IULF7LogtO5hxWHHZr9pPnM3nb+GNWXJEmSJElae40bFLqEOiD0NsDJwxKWUjqthBYlOQe4GXg+\n8FVqC6CnAnf1jD19XpITSimvAl4KbFFK+YvOwiQvbfJ5LvC1fuXOmzePxVsfzBLmjLlpkiRJkiRJ\n7TFW97FSys3AGcChzRg/K0iywZByAjy0ef1GYIeuaR9qsOkg4J1NmvWB3m5ipZnGfmqaJEmSJEmS\npkwSXDkUmAWcm+TAJFsl2abpEnZ2ki2TvCPJTkk2T7IbcCKwFDgNoJRydSnlws5EbYEU4LJSyjVN\nOf8NbJjkX5r851HHIboH+O5KbbUkSZIkSVLLjR0UKqVcDuxEDcwcCfyKOnbQXsBhwF3A7sA3qcGe\nLwO3ALuVUm4alnVPORcDfwU8GTgb+F9gE2DvUsr149ZbkiRJkiRJU8YdUwiAJiizoJn62W/M/BZD\nz8jQdf7/AP8zdgUlSZIkSZI0lGPzSJIkSZIktZBBIUmSJEmSpBaaqPvYmu7UBbszd+7c1V0NSZIk\nSZKkNZYthSRJkiRJklrIoJAkSZIkSVILGRSSJEmSJElqobVyTKH9jzqLJcxZ3dWQJEmSJEmaEVcc\nsd+M52lLIUmSJEmSpBaaKCiUZOMkRydZlOTOJIuTnJJkj2b5p5JcmmRpkhuSnJRk66715ydZnmRZ\n87d7ekpPWa9M8oskdyS5LsnRK7fJkiRJkiRJGrv7WJItgLOBJcBbgQuAdYHnAMcA2wLnAV8ErgQ2\nAt4HnJHkcaWUAvwQ2KQn6w8Ae5RSftpV1mHAW4DDgXOB2cCW49ZZkiRJkiRJK5pkTKFPAsuAXUop\nd3bNvyjJ8QCllOO65l+Z5F3A+dSAzuWllHuBGzoJkjwEeC7wia55jwLeD+xXSvleV34XTFBnSZIk\nSZIkdRmr+1iSDYG9gWN6AkIAlFJu7bPObOAQ4DLgqgFZP5faouizXfOeDQTYPMmFSa5K8tUkjx2n\nzpIkSZIkSbq/cccU2ooaqLl4VMIkr09yG3AbNZC0V9NCqJ9DgDNKKdd0zXs8MAv4B2AB8AJq4Oi/\nm5ZFkiRJkiRJmtC4QaGMkfaLwI7AM4DfACcmWe9+GSabUYNGx/UsWofave2NpZQzSynnAi8G/hR4\n5pj1liRJkiRJUpdxW9xcAhRgG+DkYQlLKZ1WQouSnAPcDDwf+GpP0kOAm4Bv9My/tvl7UVeeNyW5\nCfiTQeUuXLiQRReczj3MWmH+7CfNZ/a284dVWZIkSZIkqTXGCgqVUm5OcgZwaJKjSil3dC9PskEp\n5ZY+q65DbWX00D7LXgl8rpSyrGf+D5u/WwPXNPlvBDwaWDyojvPmzWPx1gezhDnT2SRJkiRJkqRW\nGrf7GMCh1LF+zk1yYJKtkmyTZAFwdpItk7wjyU5JNk+yG3AisBQ4rTujJHtSn0h2fG8hpZRLgFOA\nTyR5apLtgM8BFwLfnaDekiRJkiRJaowdFCqlXA7sRA3MHAn8Cvg2sBdwGHAXsDvwTWp3sy8DtwC7\nlVJu6snuEOCHpZTfDCju5cA5wKlNeXcC+/RpVSRJkiRJkqQxTPQUr1LK9dQngi0YkGS/aebz0hHL\n/wD8bTNJkiRJkiRphkzSfUySJEmSJEkPcgaFJEmSJEmSWmii7mNrulMX7M7cuXNXdzUkSZIkSZLW\nWLYUkiRJkiRJaiGDQpIkSZIkSS1kUEiSJEmSJKmF1soxhfY/6iyWMGd1V0OSJEmSJI1wxRH7re4q\ntJYthSRJkiRJklrIoJAkSZIkSVILTRQUSrJxkqOTLEpyZ5LFSU5Jskez/FNJLk2yNMkNSU5KsvWA\nvNZLcn6S5Um271m2Z5IfJrk1yTVJjkhiIEuSJEmSJGkljR1gSbIF8DPgL4G3AtsBzwG+CxzTJDsP\neCWwDbAXEOCMJOmT5UeAq4HSU84OwDeB04AdgRcBBwBHjFtnSZIkSZIkrWiSgaY/CSwDdiml3Nk1\n/6IkxwOUUo7rmn9lkncB5wNbApd3FiTZB3g28AJg355yDgJ+UUr5YPP6siRvA76a5H2llNsnqLsk\nSZIkSZIYs6VQkg2BvYFjegJCAJRSbu2zzmzgEOAy4Kqu+RsDnwFeBtzRp7iHAr1l3Ak8DHjKOPWW\nJEmSJEnSisbtPrYVtSvYxaMSJnl9ktuA26iBpL1KKfd2Jfl34NhSys8HZHEGsFuSg5Osk2Qz4N3N\nsk3HrLckSZIkSZK6jNt9rN+YQIN8Efg2NYBzOHBikt1KKXcnWQDMAT48KN9Syn8n+Xtqd7UvUFsJ\nvR/YHVg+qNCFCxey6ILTuYdZK8yf/aT5zN52/hjVlyRJkiRJWnuNGxS6hDog9DbAycMSllI6rYQW\nJTkHuBl4PvBV4JnAU4G7esaePi/JCaWUVzV5fBz4eJJNmvUfRx1o+rJB5c6bN4/FWx/MEuaMuWmS\nJEmSJEntMVb3sVLKzdRuXYcmWb93eZINhpQT6jhBAG8Eduia9qEGmw4C3tmn3OtKKXcBLwGupD79\nTJIkSZIkSROa5OljhwI/AM5N8h7gl00+ewGvTbIfcDC169iNwObAO4Cl1MfLU0q5ujvDJLdTg0aX\nlVKu6Zp/OHA6tbvYC4C3AS8spazw+HpJkiRJkiSNZ+ygUCnl8iQ7UVv0HEkdM+hGanDoMOAu6rg/\nbwI2BK4Hvg/sVkq5aVjWfebtA/wjtYXRL4ADSinfHrfOkiRJkiRJWtEkLYUopVwPLGimfvYbM7/F\n0DMydJ2/5/i1kyRJkiRJ0ijjPpJekiRJkiRJawGDQpIkSZIkSS00UfexNd2pC3Zn7ty5q7sakiRJ\nkiRJayxbCkmSJEmSJLWQQSFJkiRJkqQWMigkSZIkSZLUQmvlmEL7H3UWS5izuqshSZIkSVIrXXHE\nfqu7CpoGWwpJkiRJkiS10ERBoSQbJzk6yaIkdyZZnOSUJHs0yz+V5NIkS5PckOSkJFt3rT8/yfIk\ny5q/3dNTmjSvGJBmWZJHz8zmS5IkSZIktdPY3ceSbAGcDSwB3gpcAKwLPAc4BtgWOA/4InAlsBHw\nPuCMJI8rpRTgh8AmPVl/ANijlPLT5vVXgG/1pPkcsF4p5aZx6y1JkiRJkqQpk4wp9ElgGbBLKeXO\nrvkXJTkeoJRyXNf8K5O8Czgf2BK4vJRyL3BDJ0GShwDPBT7RmVdKuasnzaOBPYBXTVBnSZIkSZIk\ndRmr+1iSDYG9gWN6AkIAlFJu7bPObOAQ4DLgqgFZP5faouizQ4p/BXA78J/j1FmSJEmSJEn3N+6Y\nQlsBAS4elTDJ65PcBtxGDSTt1bQQ6ucQ4IxSyjVDsjwEOKFpQSRJkiRJkqSVMG5QKGOk/SKwI/AM\n4DfAiUnWu1+GyWbUoNFxvcu60jwV2AY4fqzaSpIkSZIkqa9xxxS6BCjUAM3JwxKWUjqthBYlOQe4\nGXg+8NWepIcANwHfGJLda4DzSynnj6rgwoULWXTB6dzDrBXmz37SfGZvO3/U6pIkSZIkSa0wVlCo\nlHJzkjOAQ5McVUq5o3t5kg1KKbf0WXUdaiujh/ZZ9krgc6WUZf3KbMYkeiHw9unUcd68eSze+mCW\nMGc6ySVJkiRJklpp3O5jAIcCs4BzkxyYZKsk2yRZAJydZMsk70iyU5LNk+wGnAgsBU7rzijJntQn\nkg3rFnZwU94JE9RVkiRJkiRJfYz9SPpSyuVJdgLeCRwJbArcCPwSOAy4C9gdeBOwIXA98H1gt1LK\nTT3ZHQL8sJTymyFFHgL8Z78nm0mSJEmSJGkyYweFAEop1wMLmqmf/aaZz0unkeZpY1RNkiRJkiRJ\n0zBJ9zFJkiRJkiQ9yBkUkiRJkiRJaiGDQpIkSZIkSS000ZhCa7pTF+zO3LlzV3c1JEmSJEmS1li2\nFJIkSZIkSWohg0KSJEmSJEkttFZ2H9v/qLNYwpzVXQ1JkiRJklrliiP2W91V0BhsKSRJkiRJktRC\nBoUkSZIkSZJaaKKgUJKNkxydZFGSO5MsTnJKkj36pP1WkuVJDuiZv2GSE5LckuTmJMclmd21fKNm\n3d82ZVzZlPmISeosSZIkSZKkKWOPKZRkC+BsYAnwVuACYF3gOcAxwLZdad8CLANKn6y+BGwM7Ams\nB3wW+DTwsmb5cuAk4J3AjcBWwLHAhl1pJEmSJEmSNIFJBpr+JDXQs0sp5c6u+RclOb7zIsmOwFuA\nnYHrujNIsg2wN/CUUsrPm3lvBL6Z5PBSynWllN9Tg0QdVyU5Fjh8gjpLkiRJkiSpy1jdx5JsSA3m\nHNMTEAKglHJrk2594ATgDaWUG/pk9VTg5k5AqHEmtUXRnw8oey5wIPC9ceosSZIkSZKk+xt3TKGt\ngAAXj0j3z8APSimnDli+CbBCsKiUsozaJW2T7vlJvpTkduBq4Bbgb8essyRJkiRJknqM230sIxPU\nAaX3AHacqEb392bgvcATgQ9RA06HDkq8cOFCFl1wOvcwa4X5s580n9nbzp+hKkmSJEmSJD24jRsU\nuoTaxWsb4OQBaZ4JPB64JVkhhvT1JN8vpexBHWPoMd0Lk8wCNqJn/KGm+9kNwG+S3AycleSfSinX\n9yt83rx5LN76YJYwZ8xNkyRJkiRJao+xuo+VUm4GzgAObcYNWkGSDaitebYHduiaAN4EvKr5/0fA\no5L8Wdfqe1JbIp0zpAqzqEGph45Tb0mSJEmSJK1okqePHQr8ADg3yXuAXzb57AW8tpQyj57xgpoW\nQ1eVUhYDlFJ+neQM4F+TvJ76SPqjgS+XUq5r1tmH+sj6nwB/ALYDPkIdq+jKCeotSZIkSZKkxthB\noVLK5Ul2At4JHAlsCtxIDQ4dNmi1PvNeAhxDferYcuA/qK2JOu6gDir9MWrLoKuA/wQ+PG6dJUmS\nJEmStKJJWgrRjOezoJmmk35Wn3m/B142ZJ3vAU+bpH6SJEmSJEkabtxH0kuSJEmSJGktYFBIkiRJ\nkiSphSbqPramO3XB7sydO3d1V0OSJEmSJGmNZUshSZIkSZKkFjIoJEmSJEmS1EIGhSRJkiRJklpo\nrRxTaP+jzmIJc1Z3NSRJkiRJWqNcccR+q7sKWoPYUkiSJEmSJKmFJgoKJdk4ydFJFiW5M8niJKck\n2aNP2m8lWZ7kgJ75Jzfr3ZHkmiSfT7JpT5rlPdOyJAdNUmdJkiRJkiRNGbv7WJItgLOBJcBbgQuA\ndYHnAMcA23alfQuwDCh9svoO8EHgWmAz4KPAicDTe9K9AjgdSPP69+PWWZIkSZIkSSuaZEyhT1ID\nPbuUUu7smn9RkuM7L5LsCLwF2Bm4rjeTUsonul5eleQI4L+SzCqlLOtadksp5cYJ6ilJkiRJkqQB\nxuo+lmRDYG/gmJ6AEACllFubdOsDJwBvKKXcMI18NwJeCvywJyAE8C9JbkxyTpJXjVNfSZIkSZIk\n9TfumEJbUbtxXTwi3T8DPyilnDosUZIjkvwBuAnYHHheT5J3AwcBzwL+Azg2yd+NWWdJkiRJkiT1\nGLf7WEYmqANK7wHsOI38PgIcB2wBvAf4ArB/Z2Ep5YNdaX+RZDbw99SxiyRJkiRJkjShcYNCl1AH\njd4GOHlAmmcCjwduSVaIIX09yfdLKfc9oayUsoQ6YPWlSX5NHVvoz0sp5wzI+1zg3UnWLaXc0y/B\nwoULWXTB6dzDrBXmz37SfGZvO3/0FkqSJEmSJLXAWEGhUsrNSc4ADk1yVCnlju7lSTYAPgT8a8+q\nFwBvAoZ1J+tEcR46JM2fATcPCggBzJs3j8VbH8wS5gzJRpIkSZIkqd0mefrYocAPgHOTvAf4ZZPP\nXsBrSynzgBUGl25aDF1VSlncvN4V2KXJ52bqWEX/RG2J9KMmzf7AxsCPgTub/P+B2uVMkiRJkiRJ\nK2HsoFAp5fIkOwHvBI4ENgVupAaHDhu0Ws/rpcCBwHuB2cC1wLeAD3a1ArqHGoD6GHUso0uBN5dS\njhu3zpIkSZIkSVrRJC2FKKVcDyxopumkn9Xz+gJgzxHrnAGcMUn9JEmSJEmSNNy4j6SXJEmSJEnS\nWsCgkCRJkiRJUgsZFJIkSZIkSWqhicYUWtOdumB35s6du7qrIUmSJEmStMaypZAkSZIkSVILGRSS\nJEmSJElqIYNCkiRJkiRJLbRWjim0/1FnsYQ5q7sakiRJkiStVlccsd/qroLWYLYUkiRJkiRJaqGJ\ngkJJNk5ydJJFSe5MsjjJKUn26JP2W0mWJzmgZ/7JzXp3JLkmyeeTbDqgvI2SXJ1kWZJHTlJnSZIk\nSZIkTRk7KJRkC+BnwF8CbwW2A54DfBc4piftW4BlQOmT1XeAFwJPBA4EngCcOKDY44Hzx62rJEmS\nJEmS+ptkTKFPUgM9u5RS7uyaf1GS4zsvkuwIvAXYGbiuN5NSyie6Xl6V5Ajgv5LMKqUs68rn9cAG\nwPuBfSaoryRJkiRJknqM1VIoyYbA3sAxPQEh4P9n7/6j7Srre9+/P6aUkyY9GKoCu9fyo2kTsjmI\nqVjEcgKxBdrk2Fs83sG9tKVw7LCaNlRgeGmxB9G2UqSiIUewwq0/irX11iHcCARFqCAeaUsVsvkh\nhiTQKiGYNPyMxOR7/5hzy9rLvZPMFXsSs96vMdbYe83nO5/5Xf9+xvM8E6rqybZuOnAt8NaqenwX\n5j0QOAP4Ul8gNA94B/AbwPYuvUqSJEmSJGlqXbePzQYCPLiTusuBO6pqxY6KklyS5GngCeDlwP/e\nM/ajwCeA86vqXzv2KUmSJEmSpB3oun0sOy1oDpReCByzC/NdClwNHApcBHwcWNyOXQLcV1V/3ffs\nHfYwNjbG6lU3sZVpE67POHIBM+Yt2IWWJEmSJEmS9n1dQ6GHaA6NngtcN0XNScARwOZkQn7z6SRf\nrKrvvaGsqjYCG4FvJHmA5myhn6+qr7TzHJXkjW152s+GJH9SVRdP9vDR0VHWzTmdjczs+NMkSZIk\nSZKGR6dQqKo2JVkJLEmyrKqe6x1PcgDwHuDDfbeuAs4BdrSdbHxpz/7t39OA6T3jr6Z5C9kvAA93\n6VuSJEmSJEkTDfL2sSXAHcBdSS4C7mnnORl4c1WNAhMOl25XDD1aVeva768Gjm3n2URzVtG7aFYi\nfRmgqtb0zfFSmpVCD4wfaC1JkiRJkqTBdD1oejysmQ/cClwG3AvcTBMKnTvVbX3fn6VZCfR54AGa\nlUVfBU6sqq07enzXfiVJkiRJkvT9BlkpRFWtB5a2n12pn9b3fRXwuo7P/HvoOz1akiRJkiRJA+m8\nUkiSJEmSJEk//AyFJEmSJEmShtBA28f2diuWnsDIyMiebkOSJEmSJGmv5UohSZIkSZKkIWQoJEmS\nJEmSNIQMhSRJkiRJkobQPnmm0OJlt7ORmXu6DUmSJEmSfmDWXrJoT7egfYwrhSRJkiRJkoaQoZAk\nSZIkSdIQGigUSnJQkiuSrE6yJcm6JNcnWZhkVjv2QJJn27EPJPmPfXPMT3Jzkk1JNiT5UJIZkzzr\nt5J8LclzSR5LcsWgP1aSJEmSJEmNzqFQkkOBu4ETgfOAo4BTgVuB5cAhwMHAucAocGY7fnXPHIcA\nnwO+Dry6HR8FPtL3rHOBdwN/CswDfhFY2bVnSZIkSZIkTTTIQdNXAtuAY6tqS8/1+5NcU1VPAm/s\nub4myYXAx5O8qKq2A4uB56vqd8eLkvwOcE+SI6rq4SQvpgmEFlXVbT3zrRqgZ0mSJEmSJPXotFIo\nySzgFGB5XyAEQBsITebFwJNtIASwP/B8X834fL/Q/j0ZCPDyJPcleTTJ3yT537r0LEmSJEmSpO/X\ndfvYbJqg5sFdvSHJS4B3AB/qufwF4OAk5yfZrw2b3gMUzfYzgMOBacAfAEuBNwAHAp9LMsgKJ0mS\nJEmSJLW6hivpVJz8OPBZmi1fF49fr6r7kpwJvI8mDPousAx4HBhfTfSitr/fq6pb2vn+T+Ax4CSa\nM4m+z9jYGKtX3cRWpk24PuPIBcyYt6BL+5IkSZIkSfusrqHQQzSreeYC1+2oMMlMmkOh/w04raq2\n9Y5X1SeBTyZ5KfBMe/k8YHX7/7fav/f33PNEkieAn5rquaOjo6ybczobmbnLP0qSJEmSJGnYdNo+\nVlWbaIKeJUmm948nOaD9++PAzcBzwOurqv/8oN45N1TVs8Dpbf3n26EvtX/n9Mx/IPASYF2XviVJ\nkiRJkjRR51fSA0tozvq5K8lpSWYnmZtkKXBnGwh9Dvgx4E3Ai5Mc1H6+97wkS5K8MsnPJFkCXAFc\nMH5YdVU9BFwPfCDJa5IcBXwUuA+4dTd+syRJkiRJ0tDrfGBzVa1JMh+4ELiM5mDoDcA9wLnAfODY\ntvwb7d/QbDs7HHikvfZq4J3ATOAB4Ler6hN9j/sN4HJgBc1ZQ7cBv9y/FU2SJEmSJEndDPQWr6pa\nT/NGsKVTlEyb4nrvHGfuQs3TwG+3H0mSJEmSJP2ADLJ9TJIkSZIkST/kDIUkSZIkSZKG0EDbx/Z2\nK5aewMjIyJ5uQ5IkSZIkaa/lSiFJkiRJkqQhZCgkSZIkSZI0hAyFJEmSJEmShtA+eabQ4mW3s5GZ\ne7oNSZIkSZJ2y9pLFu3pFrQPc6WQJEmSJEnSEBooFEpyUJIrkqxOsiXJuiTXJ1nYjt+WZHvPZ1uS\nD/bcv6Dn+va+z8/11L08yWeTPJPksSSXJjHIkiRJkiRJ2k2dt48lORS4E9gInAesAvYDTgWWA/OA\nAv4C+CMg7a3P9kzzJeDgvqn/GFhYVf/UPudFwA3AN4HjgBHg48DzwDu69i1JkiRJkqQXDHKm0JXA\nNuDYqtrSc/3+JNf0fH+2qjZMNkFVfRd4fPx7kh8BfhX4QE/ZKcBc4KSqegK4N8kfAZckeWc7hyRJ\nkiRJkgbQaStWklk0Yc3yvkAIgKp6sufrGUk2JLk3yZ8mmb6DqX8VOBD4SM+144B720Bo3ErgAGC0\nS9+SJEmSJEmaqOtKodk028Ee3EndtcA6mq1fRwOXAj8L/Ncp6s8GVlbVN3uuHQys76tb3zP2tV1v\nW5IkSZIkSb26hkLZeQlU1dU9X8eSfAu4JcnhVbVmwoTJT9KsPpoqMJIkSZIkSdIPWNdQ6CGaQ6Tn\nAtd1uO8umkBpNrCmb+xs4Ang/+u7/hhwbN+1g3rGJjU2NsbqVTexlWkTrs84cgEz5i3o0LIkSZIk\nSdK+q1MoVFWbkqwEliRZVlXP9Y4nOaCqNk9y6ytpwqRvTTL2W8BHq2pb3/UvA3+Y5CU95wqdDGwG\n7puqx9HRUdbNOZ2NzNy1HyVJkiRJkjSEOh003VoCTAPuSnJaktlJ5iZZCtyZ5PAk70gyP8mhSV4P\nfBT4+6pa1TtRktcBhwHX9D8EuJkm/Pl4kqOTnAK8m+aQ660D9C1JkiRJkqRW51fSV9WaJPOBC4HL\ngEOADcA9wLnAVuAXgXOAGcCjwKeAP5lkurOBL1XV1yd5zvYki4ErgTuBZ2jeTnZR154lSZIkSZI0\nUedQCKCq1gNL289kTtzFec7YyfijwOJOzUmSJEmSJGmnBtk+JkmSJEmSpB9yhkKSJEmSJElDaKDt\nY+BFoz0AACAASURBVHu7FUtPYGRkZE+3IUmSJEmStNdypZAkSZIkSdIQMhSSJEmSJEkaQoZCkiRJ\nkiRJQ2ifPFNo8bLb2cjMPd2GJEmSJEm7Ze0li/Z0C9qHuVJIkiRJkiRpCBkKSZIkSZIkDaGBQqEk\nByW5IsnqJFuSrEtyfZKF7fhVSb6R5Nkkjyf5TJI5fXP8THt9Q5LNSW5PcmJfzcuTfDbJM0keS3Jp\nEoMsSZIkSZKk3dQ5YElyKHA3cCJwHnAUcCpwK7C8LftH4LeAucDJQICVSdIz1WeBae0884GvASuS\nvKx9zouAG2jOPToOOLOd811de5YkSZIkSdJEgxw0fSWwDTi2qrb0XL8/yTUAVXV1z/VHkrwD+Cpw\nGLAmyU8As4GzqmoMIMkFwFtpQqYvAKfQhEonVdUTwL1J/gi4JMk7q+q7A/QuSZIkSZIkOq4USjKL\nJqxZ3hcIAVBVT05yzwzgbOBh4NG27tvAA8BvJvmxJD8CvAVYD/xTe+txwL1tIDRuJXAAMNqlb0mS\nJEmSJE3UdfvYbJqtYA/urDDJW5I8BTxFEySd3Le655doto09BTwHnAOcWlWb2/GDaUKiXut7xiRJ\nkiRJkjSgrtvHsvOS7/kr4GbgEOB84FNJjq+q59vxD9KEPK8FtgBvojlT6FVV1R8G7bKxsTFWr7qJ\nrUybcH3GkQuYMW/BoNNKkiRJkiTtU7qGQg8BRXPWz3U7Kqyq8VVCq5N8BdgE/BrwN0leB/wK8OKq\neqa95XeTnExzoPSlwGPAsX3THtT+fWyq546OjrJuzulsZGanHyZJkiRJkjRMOm0fq6pNNOf6LEky\nvX88yQE7eE6A/dvv02nCpe19ddt7evoy8J+SvKRn/GRgM3Bfl74lSZIkSZI0UedX0gNLaF4lf1eS\n05LMTjI3yVLgziSHJbkgyfwkL09yPPAp4FmaV8xDE/j8G/CxJEcn+Zkk76V5O9ln25qbacKfj7c1\npwDvpjnkeuugP1iSJEmSJEkDhEJVtYbmgOhbgcuAe2kCnJOBc4HvACfQhDsPAX9Ns7rn+PE3ibVv\nHzsVmAncAvwDcDzw+qq6t63ZDiwGtgF3Ah8DPgJcNNAvlSRJkiRJ0vd0PVMIgPYg6KXtZzKLdmGO\nu4Ff3knNozTBkCRJkiRJkn6ABtk+JkmSJEmSpB9yhkKSJEmSJElDaKDtY3u7FUtPYGRkZE+3IUmS\nJEmStNdypZAkSZIkSdIQMhSSJEmSJEkaQoZCkiRJkiRJQ2ifPFNo8bLb2cjMPd2GJEmSJGmIrb1k\n0Z5uQdohVwpJkiRJkiQNoYFCoSQHJbkiyeokW5KsS3J9koXt+FVJvpHk2SSPJ/lMkjk99x+a5Ook\nD7c1DyV5Z5L9pnjegUn+Jcm2JP9xsJ8qSZIkSZKkcZ23jyU5FLgT2AicB6wC9gNOBZYD84B/BP4K\neAQ4ELgYWJnk8KoqYC4Q4LeB1cBRwNXAjwFvn+Sx1wBfBQ7p2q8kSZIkSZK+3yBnCl0JbAOOraot\nPdfvT3INQFVd3XP9kSTvoAl1DgPWVNVKYGVPzdoklwG/Q18olOQtwAHAu4FfHqBfSZIkSZIk9em0\nfSzJLOAUYHlfIARAVT05yT0zgLOBh4FHdzD9i2lWH/XeOw94B/AbwPYuvUqSJEmSJGlqXc8Umk2z\n7evBnRUmeUuSp4CnaIKkk6vqu1PUzgZ+F7iq59qPAp8Azq+qf+3YpyRJkiRJknagayiUDrV/BRwD\n/Gfg68Cn2qBn4oTJTwI3An9TVf9Pz9AlwH1V9dd9z+7SgyRJkiRJkibR9Uyhh4Dxg6Kv21FhVY2v\nElqd5CvAJuDXgL8Zr0kyAnwBuKOq3tw3xUnAUUneOF7efjYk+ZOquniy546NjbF61U1sZdqE6zOO\nXMCMeQt27VdKkiRJkiTt4zqFQlW1KclKYEmSZVX1XO94kgOqavMkt76IJtDZv6f2J2kCoX+gOXOo\n32nA9J7vr6Z5C9kv0JxPNKnR0VHWzTmdjczcxV8lSZIkSZI0fAZ5+9gS4A7griQXAfe085wMvDnJ\nIuB04GZgA/By4ALgWeAG+N4KoduANTRvG3tZ0uwKq6r17d81vQ9N8lKaYOmByQ60liRJkiRJ0q7r\nHApV1Zok84ELgcuAQ2jCn3uAc4HvACcA5wCzgPXAF4Hjq+qJdppfAo5oP+NvJAvN1rSJ+776Ht+1\nX0mSJEmSJH2/QVYKja/mWdp+JrNoJ/d/FPhox2f+PTsOjCRJkiRJkrSLur59TJIkSZIkSfsAQyFJ\nkiRJkqQhZCgkSZIkSZI0hAY6U2hvt2LpCYyMjOzpNiRJkiRJkvZarhSSJEmSJEkaQoZCkiRJkiRJ\nQ2if3D62eNntbGTmnm5DkiRJkjSE1l6yaE+3IO0SVwpJkiRJkiQNIUMhSZIkSZKkITRQKJTkoCRX\nJFmdZEuSdUmuT7Jwktobk2xP8vpJxhYl+Z9Jnk2yMcmne8aOTvKJJI+042NJlg7SryRJkiRJkibq\nfKZQkkOBO4GNwHnAKmA/4FRgOTCvp/ZtwDagJpnnDcBfABcAX2jnOKqn5OeA9cAZwKPA8cCHk3y3\nqj7YtW9JkiRJkiS9YJCDpq+kCXqOraotPdfvT3LN+JckxwBvA14FPNY7QZJpwPuB86rqIz1DD4z/\nU1V/2ffctUmOB04DDIUkSZIkSZJ2Q6ftY0lmAacAy/sCIQCq6sm2bjpwLfDWqnp8kqnmAyNt7d1J\nvpnkhiSjO2nhAJoVSpIkSZIkSdoNXc8Umg0EeHAndZcDd1TViinGj2jnuQh4F7AI2ATcluTFk93Q\nrhL6P4APdexZkiRJkiRJfbpuH8tOC5oDpRcCx+ygbDyM+uOq+kx731nAvwBvBD7cN+dRwGeAd1bV\nLTt6/tjYGKtX3cRWpk24PuPIBcyYt2Bn7UuSJEmSJA2FrqHQQzSHRs8Frpui5iSalUCbkwkZ0qeT\nfLGqFgLfaq/dPz5YVc8neRj4qd6bkswDPg9cVVXv2VmDo6OjrJtzOhuZuYs/SZIkSZIkafh02j5W\nVZuAlcCS9tygCZIcALwHOBp4Rc8H4BzgrPb/fwK+A8zpuXc/4DBgXc+1UZo3k/1lVf33Lr1KkiRJ\nkiRpaoO8fWwJcAdwV5KLgHvaeU4G3lxVo8CEw6XbFUOPVtU6gKp6KslVwMVJ/oUmCHo7zSqkT7X3\nHEUTCN0IvD/JQe1026rqiQH6liRJkiRJUqtzKFRVa5LMBy4ELgMOATbQhEPnTnXbJNfOB7YCHwOm\nA18BFlbV5nb8DcBPAL/efsato9meJkmSJEmSpAENslKIqloPLG0/u1I/bZJr22hWB719insuBi4e\npD9JkiRJkiTtWNdX0kuSJEmSJGkfYCgkSZIkSZI0hAbaPra3W7H0BEZGRvZ0G5IkSZIkSXstVwpJ\nkiRJkiQNIUMhSZIkSZKkIWQoJEmSJEmSNIT2yTOFFi+7nY3M3NNtSJIkSZKGyNpLFu3pFqROXCkk\nSZIkSZI0hAYKhZIclOSKJKuTbEmyLsn1SRa241cl+UaSZ5M8nuQzSeb0zbE2yfaez7Ykb++reXmS\nzyZ5JsljSS5NYpAlSZIkSZK0mzpvH0tyKHAnsBE4D1gF7AecCiwH5gH/CPwV8AhwIHAxsDLJ4VVV\n7VQFvAP4MJD22lM9z3kRcAPwTeA4YAT4OPB8e58kSZIkSZIGNMiZQlcC24Bjq2pLz/X7k1wDUFVX\n91x/JMk7gK8ChwFresaerqoNUzznFGAucFJVPQHcm+SPgEuSvLOqvjtA75IkSZIkSaLj9rEks2jC\nmuV9gRAAVfXkJPfMAM4GHgYe7Ru+IMkTSe5Ocn6SaT1jxwH3toHQuJXAAcBol74lSZIkSZI0Udfz\neWbTbPV6cGeFSd6S5CmaLWGnACf3re75AHA6cCJwFfCHwJ/1jB8MrO+bdn3PmCRJkiRJkgbUdftY\ndl7yPX8F3AwcApwPfCrJ8VX1PEBVvb+ndlWS54EPJfmDqtrasS9JkiRJkiR10DUUeojmgOi5wHU7\nKqyq8VVCq5N8BdgE/BrwN1Pcclfbz2Htcx4Dju2rOaj9+9hUzx0bG2P1qpvYyrQJ12ccuYAZ8xbs\nqGVJkiRJkqSh0SkUqqpNSVYCS5Isq6rneseTHFBVmye59UU0q4z238H0rwS2A4+3378M/GGSl/Sc\nK3QysBm4b6pJRkdHWTfndDYyc9d+lCRJkiRJ0hDqeqYQwBJgGnBXktOSzE4yN8lS4M4khyW5IMn8\nJC9PcjzwKeBZmlfMk+S4JOckOTrJ4UnOAN4HfLwnVLqZJvz5eFt3CvBumkOu3V4mSZIkSZK0Gzq/\nkr6q1iSZD1wIXEZzZtAG4B7gXOA7wAnAOcAsmsOhvwgc37Pi5zs0h0xfRLN6aA3w58DlPc/ZnmQx\ncCVwJ/AM8JH2HkmSJEmSJO2GzqEQQFWtB5a2n8ks2sn9/wy8Zhee8yiwuHODkiRJkiRJ2qFBto9J\nkiRJkiTph5yhkCRJkiRJ0hAyFJIkSZIkSRpCA50ptLdbsfQERkZG9nQbkiRJkiRJey1XCkmSJEmS\nJA0hQyFJkiRJkqQhZCgkSZIkSZI0hPbJM4UWL7udjczc021IkiRJkvZxay9ZtKdbkAbmSiFJkiRJ\nkqQhNFAolOSgJFckWZ1kS5J1Sa5PsnCS2huTbE/y+knGFiX5n0meTbIxyaf7xj+Q5B/bZ9w9SK+S\nJEmSJEn6fp23jyU5FLgT2AicB6wC9gNOBZYD83pq3wZsA2qSed4A/AVwAfCFdo6j+soKuAb4eeDo\nrr1KkiRJkiRpcoOcKXQlTdBzbFVt6bl+f5Jrxr8kOQZ4G/Aq4LHeCZJMA94PnFdVH+kZeqC3rqp+\nv61/GYZCkiRJkiRJPzCdto8lmQWcAizvC4QAqKon27rpwLXAW6vq8Ummmg+MtLV3J/lmkhuSjHb9\nAZIkSZIkSequ65lCs4EAD+6k7nLgjqpaMcX4Ee08FwHvAhYBm4Dbkry4Y0+SJEmSJEnqqOv2sey0\noDlQeiFwzA7KxsOoP66qz7T3nQX8C/BG4MMd+/qesbExVq+6ia1Mm3B9xpELmDFvwaDTSpIkSZIk\n7VO6hkIP0Rz+PBe4boqak2hWAm1OJmRIn07yxapaCHyrvXb/+GBVPZ/kYeCnOvY0wejoKOvmnM5G\nZu7ONJIkSZIkSfu0TtvHqmoTsBJY0p4bNEGSA4D30BwK/YqeD8A5wFnt//8EfAeY03PvfsBhwLpO\nv0CSJEmSJEmdDfL2sSXAHcBdSS4C7mnnORl4c1WNAhMOl25XDD1aVesAquqpJFcBFyf5F5og6O00\nq5A+1XPfTwM/DhwCTE8yHjCNVdV3B+hdkiRJkiRJDBAKVdWaJPOBC4HLaAKbDTTh0LlT3TbJtfOB\nrcDHgOnAV4CFVbW5p+Zq4D/3fL+7/Xs48EjX3iVJkiRJktQYZKUQVbUeWNp+dqV+2iTXttGsDnr7\nDu47aZD+JEmSJEmStGNdX0kvSZIkSZKkfYChkCRJkiRJ0hAaaPvY3m7F0hMYGRnZ021IkiRJkiTt\ntVwpJEmSJEmSNIQMhSRJkiRJkoaQoZAkSZIkSdIQ2ifPFFq87HY2MnNPtyFJkiRJ2oetvWTRnm5B\n2i2uFJIkSZIkSRpChkKSJEmSJElDaKBQKMlBSa5IsjrJliTrklyfZOEktTcm2Z7k9X3XZyW5Nsnm\nJJuSXJ1kRs/4me1929q/23u+v2SQviVJkiRJktTofKZQkkOBO4GNwHnAKmA/4FRgOTCvp/ZtwDag\nJpnqE8BBwOuAHwU+AnwI+PV2/JPAjX33fBT40ap6omvfkiRJkiRJesEgB01fSRP0HFtVW3qu35/k\nmvEvSY4B3ga8Cnisd4Ikc4FTgJ+rqn9ur/0e8Nkk51fVY1X1HeDxnnteAiwEzhqgZ0mSJEmSJPXo\ntH0sySyaMGd5XyAEQFU92dZNB64F3lpVj/fXAa8BNo0HQq3P06wo+vkpHn8m8Azwd116liRJkiRJ\n0vfreqbQbCDAgzupuxy4o6pWTDF+MD2rgACqahvNlrSDp7jnbODadgWRJEmSJEmSdkPX7WPZaUFz\noPRC4JiBOpp8ztcAc4EzdlY7NjbG6lU3sZVpE67POHIBM+Yt+EG1JEmSJEmS9EOtayj0EM0Wr7nA\ndVPUnAQcAWxOJmRIn07yxapaSHPG0Mt6B5NMAw6k7/yh1puAr1bVV3fW4OjoKOvmnM5GZu6sVJIk\nSZIkaWh12j5WVZuAlcCS9tygCZIcALwHOBp4Rc8H4BxeOCT6y8CLk7yy5/bX0axE+krfnDOANwJX\nd+lVkiRJkiRJUxvk7WNLgDuAu5JcBNzTznMy8OaqGqXvvKB2xdCjVbUOoKoeSLIS+HCSt9C8kv4K\n4K+rqn+l0OnANJqDqyVJkiRJkvQD0DkUqqo1SeYDFwKXAYcAG2jCoXOnum2Sa/8XsJzmrWPbgf+X\nZjVRv7OBvxt/s5kkSZIkSZJ23yArhaiq9cDS9rMr9dMmufZvwK/vwr2v7dygJEmSJEmSdqjrK+kl\nSZIkSZK0DzAUkiRJkiRJGkIDbR/b261YegIjIyN7ug1JkiRJkqS9liuFJEmSJEmShpChkCRJkiRJ\n0hAyFJIkSZIkSRpC++SZQouX3c5GZu7pNiRJkiRJP2TWXrJoT7cg/S/jSiFJkiRJkqQhNFAolOSg\nJFckWZ1kS5J1Sa5PsnCS2huTbE/y+r7rP5PkM0k2JNmc5PYkJ/bVHJvk80k2JdmY5KYkRw/SsyRJ\nkiRJkl7QORRKcihwN3AicB5wFHAqcCuwvK/2bcA2oCaZ6rPAtHae+cDXgBVJXtbeOwO4EVgLvBp4\nLfAUcFOSaV37liRJkiRJ0gsGOVPoSpqg59iq2tJz/f4k14x/SXIM8DbgVcBjvRMk+QlgNnBWVY21\n1y4A3koTMn0BmAvMAi6qqn9tay6mCY8OBR4eoHdJkiRJkiTRcaVQklnAKcDyvkAIgKp6sq2bDlwL\nvLWqHp+k7tvAA8BvJvmxJD8CvAVYD/xTW/Yg8G3gvyXZr53zTcB9NKuHJEmSJEmSNKCuK4VmA6EJ\nbHbkcuCOqlqxg5pfAj5DsyVsO00gdGpVbQaoqqeTnNTW/Pf2nq8Dp1TV9o59S5IkSZIkqUfXM4Wy\n04LmQOmFNFvHduSDNEHQa4FjacKfFUkOauf5D8A1wB00ZwodD6wCbkiyf8e+JUmSJEmS1KPrSqGH\naA6NngtcN0XNScARwOZkQob06SRfrKqFSV4H/Arw4qp6ph3/3SQnA2cClwJnAIdW1XHjEyQ5A9gE\n/Crwt5M9fGxsjNWrbmIrE8+innHkAmbMW9Dpx0qSJEmSJO2rOoVCVbUpyUpgSZJlVfVc73iSA4D3\nAB/uu3UVcA4wvp1sOk241L8NbDsvrF6aPsl4tZ8pVziNjo6ybs7pbGTmrv0oSZIkSZKkIdT5lfTA\nEppXyd+V5LQks5PMTbIUuLOqHq+q+3o/7X2PVtW69v8vA/8GfCzJ0Ul+Jsl7gcNoXlUP8DlgVpL/\n0c4/CvwlsBW4dbCfK0mSJEmSJBggFKqqNcB8mmDmMuBe4GbgZODcqW7rm+PbwKnATOAW4B9ozgx6\nfVXd29Y8CPwX4D8BdwJ/DxxMc9D0+q59S5IkSZIk6QVdzxQCoA1llrafXamfNsm1u4Ff3sl9t9CE\nRpIkSZIkSfoBGmT7mCRJkiRJkn7IGQpJkiRJkiQNoYG2j+3tViw9gZGRkT3dhiRJkiRJ0l7LlUKS\nJEmSJElDyFBIkiRJkiRpCBkKSZIkSZIkDaF98kyhxctuZyMz93QbkiRJkqQfEmsvWbSnW5D+l3Ol\nkCRJkiRJ0hAyFJIkSZIkSRpCA4VCSQ5KckWS1Um2JFmX5PokCyepvTHJ9iSv77s+P8nNSTYl2ZDk\nQ0lm9NV8IMk/ts+4e5BeJUmSJEmS9P06h0JJDgXuBk4EzgOOAk4FbgWW99W+DdgGVN/1Q4DPAV8H\nXt3ePwp8pO9xBVwDfLJrn5IkSZIkSZraIAdNX0kT9BxbVVt6rt+f5JrxL0mOAd4GvAp4rG+OxcDz\nVfW7PfW/A9yT5Iiqehigqn6/HXsZcPQAvUqSJEmSJGkSnVYKJZkFnAIs7wuEAKiqJ9u66cC1wFur\n6vFJptofeL7v2vh8v9ClJ0mSJEmSJHXXdfvYbCDAgzupuxy4o6pWTDH+BeDgJOcn2a8Nm95Ds13s\nkI49SZIkSZIkqaOu28ey04LmQOmFwDFT1VTVfUnOBN5HEwZ9F1gGPA5s79jTBGNjY6xedRNbmTbh\n+owjFzBj3oLdmVqSJEmSJGmf0TUUeohmNc9c4Lopak4CjgA2JxMypE8n+WJVLQSoqk8Cn0zyUuCZ\ntuY84OGOPU0wOjrKujmns5GZuzONJEmSJEnSPq3T9rGq2gSsBJa05wZNkOQAmpU/RwOv6PkAnAOc\nNcmcG6rqWeB04Dmat5JJkiRJkiTp39Egbx9bAtwB3JXkIuCedp6TgTdX1SjNNrDvaVcMPVpV63qu\nLQHuBJ5u770UePv4YdVtzU8DP05zztD0JOMB01hVfXeA3iVJkiRJksQAoVBVrUkyH7gQuIwmsNlA\nEw6dO9Vtk1x7NfBOYCbwAPDbVfWJvpqrgf/c8/3u9u/hwCNde5ckSZIkSVJjkJVCVNV6YGn72ZX6\naZNcO3MX7jupe3eSJEmSJEnama6vpJckSZIkSdI+wFBIkiRJkiRpCA20fWxvt2LpCYyMjOzpNiRJ\nkiRJkvZarhSSJEmSJEkaQoZCkiRJkiRJQ8hQSJIkSZIkaQjtk2cKLV52OxuZuafbkCRJkiTtBdZe\nsmhPtyDtlVwpJEmSJEmSNIQGCoWSHJTkiiSrk2xJsi7J9UkW9tS8JsktSZ5OsjnJbUn27xlfm2R7\nz2dbkrf3jJ/Zc72/7iW797MlSZIkSZKGW+ftY0kOBe4ENgLnAauA/YBTgeXAvCSvAW4E/gRYAmwD\nXgFs75mqgHcAHwbSXnuqZ/yT7Ry9Pgr8aFU90bVvSZIkSZIkvWCQM4WupAl5jq2qLT3X709yTfv/\n+4D3V9V7e8YfmmSup6tqw2QPqarvAI+Pf29XBy0EzhqgZ0mSJEmSJPXotH0sySzgFGB5XyAEQFU9\nmeSlwM8DTyT5UpLH2q1jr51kyguSPJHk7iTnJ5m2g8efCTwD/F2XniVJkiRJkvT9uq4Umk2z1evB\nHdQc0f69iGZ72ddoAp1bkoxW1ep2/APA3TTb0I4HLgEOBs6fYt6zgWvbFUSSJEmSJEnaDV1Doey8\n5Hurj66qqo+1/5+b5HU0wc6FAFX1/p57ViV5HvhQkj+oqq0THtqcUTQXOKNjv5IkSZIkSZpE11Do\nIZoDoucC101R86327/191+8HfmoHc9/V9nMY33/+0JuAr1bVV3fW4NjYGKtX3cRWJu5Em3HkAmbM\nW7Cz2yVJkiRJkoZCp1CoqjYlWQksSbKsqp7rHU9yQFWtTfJNYE7f7T8L3LCD6V9J83ayx3svJpkB\nvBH4v3elx9HRUdbNOZ2NzNyVckmSJEmSpKE0yNvHlgB3AHcluQi4p53nZODNwCjwXuCdSe4Bvgr8\nFk1I9AaAJMfRHEZ9K81r6I+neWPZx6tqc9/zTgemAdcO0KskSZIkSZIm0TkUqqo1SebTnA10GXAI\nsIEmHDq3rflAkv1pgp4DaQ6b/sWqWtNO8x2asOciYH9gDfDnwOWTPPJs4O+q6smuvUqSJEmSJGly\ng6wUoqrWA0vbz1Q1lwKXTjH2z8BrdvFZk73KXpIkSZIkSbvhRTsvkSRJkiRJ0r7GUEiSJEmSJGkI\nGQpJkiRJkiQNoYHOFNrbrVh6AiMjI3u6DUmSJEmSpL2WK4UkSZIkSZKGkKGQJEmSJEnSENont48t\nXnY7G5m5p9uQJEmSJP07WHvJoj3dgrRPcKWQJEmSJEnSEDIUkiRJkiRJGkIDhUJJDkpyRZLVSbYk\nWZfk+iQL2/Hbkmzv+WxL8sG+Of4wyZeSPJNk4yTPODDJjUn+tX3GI+0zf3ywnypJkiRJkqRxnc8U\nSnIocCewETgPWAXsB5wKLAfmAQX8BfBHQNpbn+2baj/gb4EvA2dP8qjtwGeAC4ENwGzgg8As4Ne7\n9i1JkiRJkqQXDHLQ9JXANuDYqtrSc/3+JNf0fH+2qjZMNUlVXQyQ5Mwpxv8N+FDPpUfb1UbnD9Cz\nJEmSJEmSenTaPpZkFnAKsLwvEAKgqp7s+XpGkg1J7k3yp0mm706jSUaA04DbdmceSZIkSZIkdV8p\nNJtmO9iDO6m7FlgHfBM4GrgU+Fngv3ZtMMkngF8FpgPXA7/ddQ5JkiRJkiRN1DUUys5LoKqu7vk6\nluRbwC1JDq+qNR2f+fvAO2lCpfcAlwNLpioeGxtj9aqb2Mq0CddnHLmAGfMWdHy0JEmSJEnSvqlr\nKPQQzSHSc4HrOtx3F02gNBvoFApV1ePA48DXk2wCbk/yrqpaP1n96Ogo6+aczkZmdnmMJEmSJEnS\nUOl0plBVbQJWAksmOyMoyQFT3PpKmjDpW507nGhaO8/+uzmPJEmSJEnSUBvk7WNLgDuAu5JcBNzT\nznMy8OYki4EzgBuAbwOvAN4H/H1VrRqfJMnLgQOBQ4FpSV7RDn2jqp5J8svAQcA/AE8DR9GcTXRH\nVT0yQN+SJEmSJElqdQ6FqmpNkvnAhcBlwCHABppw6FxgK/CLwDnADOBR4FPAn/RN9S7gN3u+393+\nPQn4IvAczaHS76NZGfQo8HfAn3XtWZIkSZIkSRMNslKI9jyfpe1nMifuwhxnAWftYPw24LUDtCdJ\nkiRJkqSd6HSmkCRJkiRJkvYNhkKSJEmSJElDaKDtY3u7FUtPYGRkZE+3IUmSJEmStNdypZAkSZIk\nSdIQMhSSJEmSJEkaQoZCkiRJkiRJQ2ifPFNo8bLb2cjMPd2GJEmSJOnfwdpLFu3pFqR9giuFIcEF\naAAAD/ZJREFUJEmSJEmShtBAoVCSg5JckWR1ki1J1iW5PsnCnprXJLklydNJNie5Lcn+PeNrk2zv\n+WxL8va+5xyb5PNJNiXZmOSmJEcP/nMlSZIkSZIEA4RCSQ4F7gZOBM4DjgJOBW4Flrc1rwFuBG4C\nXtV+lgPbe6Yq4B3AQcDBwCHAFT3PmdHOsRZ4NfBa4CngpiTTuvYtSZIkSZKkFwxyptCVwDbg2Kra\n0nP9/iTXtP+/D3h/Vb23Z/yhSeZ6uqo2TPGcucAs4KKq+leAJBcDXwMOBR4eoHdJkiRJkiTRcaVQ\nklnAKcDyvkAIgKp6MslLgZ8HnkjypSSPtVvHXjvJlBckeSLJ3UnO71sB9CDwbeC/JdkvyXTgTcB9\nNKuHJEmSJEmSNKCu28dmA6EJbKZyRPv3IuBDNCHS3cAtSX66p+4DwOk029CuAv4Q+LPxwap6GjgJ\n+A3gOZqtYycDv1JVvdvQJEmSJEmS1FHXUCgd5ryqqj5WVV+rqnNpgqSzx4uq6v1V9cWqWlVVfwGc\nC/xekv0AkvwH4BrgDpozhY4HVgE39B5YLUmSJEmSpO66nin0EM0B0XOB66ao+Vb79/6+6/cDP7WD\nue9q+zmsfc4ZwKFVddx4QZIzgE3ArwJ/O9kkY2NjrF51E1uZeBb1jCMXMGPegh08XpIkSZIkaXh0\nCoWqalOSlcCSJMuq6rne8SQHVNXaJN8E5vTd/rPADTuY/pU0byd7vP0+nYlvK4MmkCp2sMJpdHSU\ndXNOZyMzd/6DJEmSJEmShlTnV9IDS4BpwF1JTksyO8ncJEuBO9ua9wJLk7whyU8neTdNSHQNQJLj\nkpyT5Ogkh7crgN4HfLyqNrdzfA6YleR/tPOPAn8JbAVuHfgXS5IkSZIkqfsr6atqTZL5wIXAZcAh\nwAbgHppzgaiqD7Tn/rwPOJDmNfK/WFVr2mm+Q3PI9EXA/sAa4M+By3ue82CS/9LW3EmzauifgVOq\nan33nypJkiRJkqRxnUMhgDaUWdp+pqq5FLh0irF/Bl6zC8+5BbhlkB4lSZIkSZI0tUG2j0mSJEmS\nJOmHnKGQJEmSJEnSEDIUkiRJkiRJGkIDnSm0t1ux9ARGRkb2dBuSJEmSJEl7LVcKSZIkSZIkDSFD\nIUmSJEmSpCFkKCRJkiRJkjSEDIUkSZIkSZKGkKGQJEmSJEnSEDIUkiRJkiRJGkKGQpIkSZIkSUPI\nUEiSJEmSJGkIGQpJkiRJkiQNIUMhSZIkSZKkIWQoJEmSJEmSNIQMhSRJkiRJkoaQoZAkSZIkSdIQ\nMhSSJEmSJEkaQoZCkiRJkiRJQ8hQSJIkSZIkaQgZCkmSJEmSJA0hQyFJkiRJkqQhZCgkSZIkSZI0\nhAyFJEmSJEmShpChkCRJkiRJ0hAyFJIkSZIkSRpChkKSJEmSJElDyFBIkiRJkiRpCBkKSZIkSZIk\nDSFDIUmSJEmSpCFkKCRJkiRJkjSEDIUkSZIkSZKGkKGQJEmSJEnSEDIUkiRJkiRJGkKGQpIkSZIk\nSUPIUEiSJEmSJGkIGQpJkiRJkiQNIUMhSZIkSZKkIWQoJEmSJEmSNIQMhSRJkiRJkoaQoZAkSZIk\nSdIQMhSSJEmSJEkaQoZCkiRJkiRJQ8hQSJIkSZIkaQgZCkmSJEmSJA0hQyFJkiRJkqQhZCgkSZIk\nSZI0hAyFJEmSJEmShpChkCRJkiRJ0hAyFJIkSZIkSRpChkKSJEmSJElDyFBIkiRJkiRpCBkKSZIk\nSZIkDSFDIUmSJEmSpCFkKCRJkiRJkjSEDIUkSZIkSZKGkKGQJEmSJEnSEDIUkiRJkiRJGkKGQpIk\nSZIkSUPIUEiSJEmSJGkIGQpJkiRJkiQNIUMhSZIkSZKkIWQoJEmSJEmSNIQMhSRJkiRJkoaQoZCk\n/7+9ew+VtK7jOP752k3dSihBiYwUummkXSkiKSI3i8rIKDMyiyBSEpJKaKMLVnQzKpJuUvtHmfpf\nF0gxDCIzKjHQvIAlVJppZVG53fbXHzMLp81Vzzhzjs/5vl6w4Hl8Zvz6x5eZfZ9nngEAAKAhUQgA\nAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAA\nAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAA\noCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACg\nIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAh\nUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFR\nCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEI\nAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgA\nAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAA\nAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAA\noCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACg\nIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAh\nUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFR\nCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEI\nAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgA\nAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAAAKAhUQgAAACgIVEIAAAAoCFRCAAA\nAKAhUQgAAACgIVEIAAAAoKEtF4WuueaazR4BtrTzzz9/s0eALc2OwWrZMVgtOwarVVUnLfP5RCFg\nXbzQw2rZMVgtOwarZcdg5UQhAAAAAO4bUQgAAACgIVEIAAAAoKEHbvYAS7Z/ktx2222bPQdsWbt2\n7crNN9+82WPAlmXHYLXsGKyWHYPVOuCAAw7asWPHgWefffbfl/F8Wy0KPfaoo47KRRddtNlzwJa1\nbdu2nHvuuZs9BmxZdgxWy47BatkxWK3t27cfm+SJSa5cxvPVGGMZz3O/sGPHjkcm2Z7kpiS7Nnca\nAAAAgKW7bllXCm2pKAQAAADAveNG0wAAAAANiUIAAAAADYlCAAAAAA2JQgAAAAANiUIAAAAADU0u\nClXVaVX1q6q6s6quqKpn3sP5z6+qn1XVrqq6oapO2ahZYYrWs2NV9cqquqSqfl9Vf66qy6vquI2c\nF6Zmva9jax733Kr6V1VdueoZYcoWeK/44Kr6UFXdNH+/+MuqeuMGjQuTs8COnVxVV1XV36rq5qo6\nr6oesVHzwlRU1fOq6ptV9duq2l1VL78Xj7nPvWNSUaiqXpPkk0nel+SpSX6e5OKqOngf5z82ybeT\nfC/J0Uk+neTLVfWijZgXpma9O5bk2CSXJDk+ydOSXJbkW1V19AaMC5OzwI7tedxBSXYmuXTlQ8KE\nLbhjFyV5QZJTkzw+yUlJrl/xqDBJC/x97LmZvX59KcmRSU5M8qwkX9yQgWFatiW5Ksnbkox7OnlZ\nvaPGuMf/1v1GVV2R5MdjjDPmP1eSXyf5zBjjY3dx/keTHD/GeMqaY+cnOWiM8ZINGhsmY707to/n\nuDrJN8YYZ69uUpimRXds/tp1Q5LdSV4xxnjaRswLU7PAe8UXJ/l6kiPGGHds6LAwQQvs2JlJ3jrG\neNyaY6cnedcY4zEbNDZMTlXtTnLCGOObd3POUnrHZK4UqqoHJXl6ZhUsSTJmRevSJM/Zx8Oenf//\nrerFd3M+tLXgju39HJXkYUn+uIoZYcoW3bGqOjXJ4Uk+sOoZYcoW3LGXJflpkndX1W+q6vqq+nhV\n7b/ygWFiFtyxHyU5rKqOnz/HIUleneQ7q50WWlhK75hMFEpycJIHJLl1r+O3Jjl0H485dB/nP7yq\nHrLc8WDyFtmxvb0zs8seL1ziXLBVrHvHqupxST6c5OQxxu7VjgeTt8jr2BFJnpfkqCQnJDkjs4+3\nfG5FM8KUrXvHxhiXJ3l9kguq6p9JbknypySnr3BO6GIpvWNKUQi4H6uq1yV5b5JXjzFu3+x5YOqq\nar8kX0vyvjHGjXsOb+JIsBXtl9nHMl83xvjpGOO7Sd6R5BS/QIT7rqqOzOw+J+/P7P6T2zO7+vUL\nmzgWsMYDN3uAdbg9yX+SHLLX8UOS/G4fj/ndPs7/yxjjH8sdDyZvkR1LklTVazO7YeCJY4zLVjMe\nTN56d+xhSZ6R5Jiq2nPVwn6ZfVLzn0mOG2N8f0WzwhQt8jp2S5LfjjH+uubYtZkF2EcnufEuHwU9\nLbJjZyX54RjjnPnPV1fV25L8oKreM8bY+yoH4N5bSu+YzJVCY4x/JflZkhfuOTa/f8kLk1y+j4f9\naO35c8fNjwNrLLhjqaqTkpyX5LXz37ACd2GBHftLkicnOSazb5Q4Osnnk1w3/+cfr3hkmJQFX8d+\nmORRVXXgmmNPyOzqod+saFSYpAV37MAk/97r2O7MvlnJ1a9w3yyld0wmCs2dk+QtVfWGqnpiZm+O\nD0zy1SSpqo9U1c41538+yRFV9dGqesK8Sp84fx7g/61rx+YfGduZ5MwkP6mqQ+Z/Hr7xo8Mk3Osd\nGzO/WPsnye+T7BpjXDvGuHOT/h/g/my97xW/nuQPSb5SVU+qqmOTfCzJea4qh7u03h37VpJXVdVb\nq+rw+VfUfzqzbzC72yvRoZuq2lZVR1fVMfNDR8x/Pmz+71fSO6b08bGMMS6sqoOTfDCzy6KuSrJ9\njHHb/JRDkxy25vybquqlST6V5O2Z/cbnzWOMve/QDWT9O5bkLZndcPBz+d+bcu5M8qbVTwzTssCO\nAeuwwHvFv1XVi5J8NslPMgtEF2R2jzxgLwvs2M6qemiS05J8IskdmX172VkbOjhMwzOSXJbZlXQj\nySfnx/f83WolvaNm3yIIAAAAQCdT+/gYAAAAAEsgCgEAAAA0JAoBAAAANCQKAQAAADQkCgEAAAA0\nJAoBAAAANCQKAQAAADQkCgEAAAA0JAoBAAAANCQKAQAAADQkCgEAAAA09F8U6mT9UyrVZAAAAABJ\nRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "model_variable_importances.varimp_plot(num_of_features=20)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model Comparison with Grid Search\n", "\n", "Grid search provides more subtle insights into the model tuning and selection\n", "process by inspecting and comparing our trained models after the grid search process is complete. \n", "\n", "To learn when and how to select different parameter\n", "configurations in a grid search, refer to Parameters for parameter descriptions\n", "and configurable values.\n", "\n", "There are different strategies to explore the hyperparameter combinatorial space:\n", "\n", "- Cartesian Search: test *every* single combination\n", "- Random Search: sample combinations\n", "\n", "## Cartesian Search\n", "In this example, two different network topologies and two different learning rates are specified. This grid search model trains all 4 different models (all possible combinations of these parameters); other parameter combinations can\n", "be specified for a larger space of models. Note that the models will most likely\n", "converge before the default value of epochs, since early stopping is enabled." ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from h2o.grid.grid_search import H2OGridSearch" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": true }, "outputs": [], "source": [ "hyper_parameters = {\n", " \"hidden\":[[200,200,200],[300,300]], \n", " \"learning_rate\":[1e-3,5e-3],\n", "}\n", "\n", "model_grid = H2OGridSearch(H2ODeepWaterEstimator, hyper_params=hyper_parameters)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "deepwater Grid Build progress: |██████████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "model_grid.train(\n", " x=x, \n", " y=y,\n", " distribution=\"multinomial\", \n", " epochs=50, ## might stop earlier since we enable early stopping below\n", " training_frame=train_df, \n", " validation_frame=test_df,\n", " score_interval=2, ## score no more than every 2 seconds\n", " score_duty_cycle=0.5, ## score up to 50% of the time - to enable early stopping\n", " score_training_samples=1000, ## use a subset of the training frame for faster scoring\n", " score_validation_samples=1000, ## use a subset of the validation frame for faster scoring\n", " stopping_rounds=3,\n", " stopping_tolerance=0.05,\n", " stopping_metric=\"misclassification\",\n", " sparse = True,\n", " mini_batch_size=256\n", ")" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " hidden learning_rate \\\n", "0 [300, 300] 0.005 \n", "1 [200, 200, 200] 0.005 \n", "2 [200, 200, 200] 0.001 \n", "3 [300, 300] 0.001 \n", "\n", " model_ids \\\n", "0 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_3 \n", "1 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_2 \n", "2 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_0 \n", "3 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_1 \n", "\n", " logloss \n", "0 1.7487250978191204 \n", "1 3.379028978326725 \n", "2 3.4048331134204655 \n", "3 3.8290355168863095 \n" ] }, { "data": { "text/plain": [] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# print model grid search results\n", "model_grid" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_3 mean per class error: 0.023297786771\n", "Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_2 mean per class error: 0.0541836154055\n", "Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_0 mean per class error: 0.0828773953361\n", "Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_1 mean per class error: 0.0851016690147\n" ] } ], "source": [ "for gmodel in model_grid:\n", " print gmodel.model_id + \" mean per class error: \" + str(gmodel.mean_per_class_error())" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01
0Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_30.052769
1Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_20.099326
2Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_00.096929
3Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_10.112104
\n", "
" ], "text/plain": [ " 0 1\n", "0 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_3 0.052769\n", "1 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_2 0.099326\n", "2 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_0 0.096929\n", "3 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_4_model_1 0.112104" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "grid_results = pd.DataFrame([[m.model_id, m.mean_per_class_error(valid=True)] for m in model_grid])\n", "grid_results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Random Grid Search\n", "\n", "If the search space is too large you can let the GridSearch algorithm select the parameter, by sampling from the parameter space. \n", "\n", "Just specify how many models (and/or how much training time) you want, and provide a seed to make the random selection deterministic." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": false }, "outputs": [], "source": [ "hyper_parameters = {\n", " \"hidden\":[[1000,1000],[2000]],\n", " \"learning_rate\":[s*1e-3 for s in range(30,100)],\n", " \"momentum_start\":[s*1e-3 for s in range(0,900)],\n", " \"momentum_stable\":[s*1e-3 for s in range(900,1000)],\n", "}" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": true }, "outputs": [], "source": [ "search_criteria = {\"strategy\":\"RandomDiscrete\", \"max_models\":10, \"max_runtime_secs\":100, \"seed\":123456}\n", "\n", "model_grid_random_search = H2OGridSearch(H2ODeepWaterEstimator,\n", " hyper_params=hyper_parameters,\n", " search_criteria=search_criteria)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "deepwater Grid Build progress: |██████████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "model_grid_random_search.train(\n", " x=x, y=y,\n", " distribution=\"multinomial\", \n", " epochs=50, ## might stop earlier since we enable early stopping below\n", " training_frame=train_df, \n", " validation_frame=test_df,\n", " score_interval=2, ## score no more than every 2 seconds\n", " score_duty_cycle=0.5, ## score up to 50% of the wall clock time - scoring is needed for early stopping\n", " score_training_samples=1000, ## use a subset of the training frame for faster scoring\n", " score_validation_samples=1000, ## use a subset of the validation frame for faster scoring\n", " stopping_rounds=3,\n", " stopping_tolerance=0.05,\n", " stopping_metric=\"misclassification\",\n", " sparse = True,\n", " mini_batch_size=256)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": true }, "outputs": [], "source": [ "grid_results = pd.DataFrame([[m.model_id, m.mean_per_class_error(valid=True)] for m in model_grid_random_search])" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01
0Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_5_model_00.026025
1Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_5_model_30.028335
2Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_5_model_20.038755
3Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_5_model_10.041428
\n", "
" ], "text/plain": [ " 0 1\n", "0 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_5_model_0 0.026025\n", "1 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_5_model_3 0.028335\n", "2 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_5_model_2 0.038755\n", "3 Grid_DeepWater_py_2_sid_9b8f_model_python_1477211337936_5_model_1 0.041428" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "grid_results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model Checkpoints \n", "\n", "\n", "\n", "H2O supporst model checkpoints. You can store the `state` of training and resume it later.\n", "Checkpointing can be used to reload existing models that were saved to\n", "disk in a previous session. \n", "\n", "To resume model training, use checkpoint model keys (model id) to incrementally\n", "train a specific model using more iterations, more data, different data, and\n", "so forth. To further train the initial model, use it (or its key) as a checkpoint\n", "argument for a new model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To improve this initial model, start from the previous model and add iterations by\n", "building another model, specifying checkpoint=previous model id, and\n", "changing train samples per iteration, target ratio comm to comp,\n", "or other parameters. Many parameters can be changed between checkpoints,\n", "especially those that affect regularization or performance tuning." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can use GridSearch with checkpoint restarts to scan a broader range of hyperparameter combinations." ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Re-start the training process on a saved DL model using the ‘checkpoint‘ argument\n", "model_checkpoint = H2ODeepWaterEstimator(\n", " checkpoint=model.model_id,\n", " activation=\"rectifier\",\n", " distribution=\"multinomial\",\n", " mini_batch_size=128,\n", " hidden=[1024,1024],\n", " hidden_dropout_ratios=[0.5,0.5],\n", " input_dropout_ratio=0.1,\n", " sparse=True,\n", " epochs=20) ## previous model had 10 epochs, so we need to only train for 10 more to get to 20 epochs" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "deepwater Model Build progress: |█████████████████████████████████████████████████████████████| 100%\n" ] } ], "source": [ "model_checkpoint.train(\n", " x=x,\n", " y=y,\n", " training_frame=train_df,\n", " validation_frame=test_df)" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampdurationtraining_speedepochsiterationssamplestraining_rmsetraining_loglosstraining_classification_errorvalidation_rmsevalidation_loglossvalidation_classification_error
02016-10-23 01:30:370.000 secNone0.00000000.0NaNNaNNaNNaNNaNNaN
12016-10-23 01:30:383.250 sec6585 obs/sec0.06826714096.00.4473946.9066140.2001610.4381786.6246860.1920
22016-10-23 01:30:5216.392 sec25255 obs/sec5.52960081331776.00.1185470.4794500.0140530.2054511.4555220.0422
32016-10-23 01:30:5822.772 sec26887 obs/sec8.465067124507904.00.0795240.2184240.0063240.1969751.3360980.0388
42016-10-23 01:31:0226.488 sec27373 obs/sec10.035200147602112.00.0625690.1352150.0039150.1972311.3420810.0389
52016-10-23 01:31:0227.130 sec27330 obs/sec10.035200147602112.00.0795240.2184240.0063240.1969751.3360980.0388
62016-10-23 01:36:1528.640 sec27158 obs/sec10.103467148606208.00.1202000.4990160.0144480.2140091.5766830.0458
72016-10-23 01:36:2639.508 sec26899 obs/sec14.609067214876544.00.1418720.6951810.0201280.1923321.2714730.0370
82016-10-23 01:36:3245.717 sec27510 obs/sec17.4762672561048576.00.0978040.3303830.0095660.1772071.0801580.0314
92016-10-23 01:36:3851.301 sec27836 obs/sec20.0021332931200128.00.0773210.2051490.0059780.1690900.9807420.0286
\n", "
" ], "text/plain": [ " timestamp duration training_speed epochs iterations \\\n", "0 2016-10-23 01:30:37 0.000 sec None 0.000000 0 \n", "1 2016-10-23 01:30:38 3.250 sec 6585 obs/sec 0.068267 1 \n", "2 2016-10-23 01:30:52 16.392 sec 25255 obs/sec 5.529600 81 \n", "3 2016-10-23 01:30:58 22.772 sec 26887 obs/sec 8.465067 124 \n", "4 2016-10-23 01:31:02 26.488 sec 27373 obs/sec 10.035200 147 \n", "5 2016-10-23 01:31:02 27.130 sec 27330 obs/sec 10.035200 147 \n", "6 2016-10-23 01:36:15 28.640 sec 27158 obs/sec 10.103467 148 \n", "7 2016-10-23 01:36:26 39.508 sec 26899 obs/sec 14.609067 214 \n", "8 2016-10-23 01:36:32 45.717 sec 27510 obs/sec 17.476267 256 \n", "9 2016-10-23 01:36:38 51.301 sec 27836 obs/sec 20.002133 293 \n", "\n", " samples training_rmse training_logloss training_classification_error \\\n", "0 0.0 NaN NaN NaN \n", "1 4096.0 0.447394 6.906614 0.200161 \n", "2 331776.0 0.118547 0.479450 0.014053 \n", "3 507904.0 0.079524 0.218424 0.006324 \n", "4 602112.0 0.062569 0.135215 0.003915 \n", "5 602112.0 0.079524 0.218424 0.006324 \n", "6 606208.0 0.120200 0.499016 0.014448 \n", "7 876544.0 0.141872 0.695181 0.020128 \n", "8 1048576.0 0.097804 0.330383 0.009566 \n", "9 1200128.0 0.077321 0.205149 0.005978 \n", "\n", " validation_rmse validation_logloss validation_classification_error \n", "0 NaN NaN NaN \n", "1 0.438178 6.624686 0.1920 \n", "2 0.205451 1.455522 0.0422 \n", "3 0.196975 1.336098 0.0388 \n", "4 0.197231 1.342081 0.0389 \n", "5 0.196975 1.336098 0.0388 \n", "6 0.214009 1.576683 0.0458 \n", "7 0.192332 1.271473 0.0370 \n", "8 0.177207 1.080158 0.0314 \n", "9 0.169090 0.980742 0.0286 " ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_checkpoint.scoring_history()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Specify a model and a file path. The default path is the current working directory." ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/arno/h2o-3/examples/deeplearning/notebooks/DeepWater_model_python_1477211337936_1\n" ] } ], "source": [ "model_path = h2o.save_model(\n", " model = model,\n", " #path = \"/tmp/mymodel\",\n", " force = True)\n", "\n", "print model_path" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "-rw-rw-r-- 1 arno arno 7.0M Oct 23 01:36 /home/arno/h2o-3/examples/deeplearning/notebooks/DeepWater_model_python_1477211337936_1\r\n" ] } ], "source": [ "!ls -lah $model_path" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After restarting H2O, you can load the saved model by specifying the host and model file path. \n", "\n", "Note: The saved model must be the same version used to save the model." ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Load model from disk\n", "saved_model = h2o.load_model(model_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also use the following commands to retrieve a model from its H2O key.\n", "This is useful if you have created an H2O model using the web interface and\n", "want to continue the modeling process in another language, for example **R**." ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model Details\n", "=============\n", "H2ODeepWaterEstimator : Deep Water\n", "Model Key: DeepWater_model_python_1477211337936_6\n", "Status of Deep Learning Model: MLP: [1024, 1024], 6.9 MB, predicting C785, 10-class classification, 1,200,128 training samples, mini-batch size 128\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "
input_neuronsratemomentum
7170.00227260.99
" ], "text/plain": [ " input_neurons rate momentum\n", "-- --------------- --------- ----------\n", " 717 0.0022726 0.99" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "ModelMetricsMultinomial: deepwater\n", "** Reported on train data. **\n", "\n", "MSE: 0.00597847748094\n", "RMSE: 0.077320614851\n", "LogLoss: 0.205148734501\n", "Mean Per-Class Error: 0.0060167753978\n", "Confusion Matrix: vertical: actual; across: predicted\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
0123456789ErrorRate
995.00.01.00.01.00.03.00.00.02.00.00698607 / 1,002
0.01094.00.00.01.00.01.04.01.00.00.00635797 / 1,101
0.00.01021.00.01.00.00.02.01.00.00.00390244 / 1,025
0.00.00.0995.00.05.01.00.00.02.00.00797618 / 1,003
0.00.00.00.0978.00.00.00.00.08.00.00811368 / 986
1.00.00.00.00.0912.00.00.01.00.00.00218822 / 914
1.00.01.00.00.00.0967.00.00.00.00.00206402 / 969
0.02.00.00.00.01.01.01059.00.01.00.00469925 / 1,064
4.00.00.00.00.03.03.00.0926.05.00.015940515 / 941
0.00.00.00.00.01.00.01.00.01029.00.00193992 / 1,031
1001.01096.01023.0995.0981.0922.0976.01066.0929.01047.00.005978560 / 10,036
" ], "text/plain": [ "0 1 2 3 4 5 6 7 8 9 Error Rate\n", "---- ---- ---- --- --- --- --- ---- --- ---- ---------- -----------\n", "995 0 1 0 1 0 3 0 0 2 0.00698603 7 / 1,002\n", "0 1094 0 0 1 0 1 4 1 0 0.00635786 7 / 1,101\n", "0 0 1021 0 1 0 0 2 1 0 0.00390244 4 / 1,025\n", "0 0 0 995 0 5 1 0 0 2 0.00797607 8 / 1,003\n", "0 0 0 0 978 0 0 0 0 8 0.00811359 8 / 986\n", "1 0 0 0 0 912 0 0 1 0 0.00218818 2 / 914\n", "1 0 1 0 0 0 967 0 0 0 0.00206398 2 / 969\n", "0 2 0 0 0 1 1 1059 0 1 0.00469925 5 / 1,064\n", "4 0 0 0 0 3 3 0 926 5 0.0159405 15 / 941\n", "0 0 0 0 0 1 0 1 0 1029 0.00193986 2 / 1,031\n", "1001 1096 1023 995 981 922 976 1066 929 1047 0.00597848 60 / 10,036" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Top-10 Hit Ratios: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
khit_ratio
10.9940215
20.9949183
30.9949183
40.9949183
50.9949183
60.9949183
70.9949183
80.9949183
90.9949183
101.0
" ], "text/plain": [ "k hit_ratio\n", "--- -----------\n", "1 0.994022\n", "2 0.994918\n", "3 0.994918\n", "4 0.994918\n", "5 0.994918\n", "6 0.994918\n", "7 0.994918\n", "8 0.994918\n", "9 0.994918\n", "10 1" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "ModelMetricsMultinomial: deepwater\n", "** Reported on validation data. **\n", "\n", "MSE: 0.0285914362701\n", "RMSE: 0.169090024159\n", "LogLoss: 0.980741728993\n", "Mean Per-Class Error: 0.0286482984194\n", "Confusion Matrix: vertical: actual; across: predicted\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
0123456789ErrorRate
972.00.00.01.00.00.05.01.00.01.00.00816338 / 980
2.01116.03.00.01.01.03.00.08.01.00.016740119 / 1,135
10.00.0992.06.01.00.04.08.010.01.00.038759740 / 1,032
0.00.01.0976.00.018.01.04.010.00.00.033663434 / 1,010
0.01.06.01.0942.00.05.03.02.022.00.040733240 / 982
6.00.01.04.01.0867.05.01.04.03.00.028026925 / 892
4.02.01.01.04.03.0942.00.01.00.00.016701516 / 958
1.04.013.04.00.00.00.0992.06.08.00.035019536 / 1,028
6.00.03.07.00.07.06.03.0938.04.00.036961036 / 974
1.03.00.02.06.06.02.09.03.0977.00.031714632 / 1,009
1002.01126.01020.01002.0955.0902.0973.01021.0982.01017.00.0286286 / 10,000
" ], "text/plain": [ "0 1 2 3 4 5 6 7 8 9 Error Rate\n", "---- ---- ---- ---- --- --- --- ---- --- ---- ---------- ------------\n", "972 0 0 1 0 0 5 1 0 1 0.00816327 8 / 980\n", "2 1116 3 0 1 1 3 0 8 1 0.0167401 19 / 1,135\n", "10 0 992 6 1 0 4 8 10 1 0.0387597 40 / 1,032\n", "0 0 1 976 0 18 1 4 10 0 0.0336634 34 / 1,010\n", "0 1 6 1 942 0 5 3 2 22 0.0407332 40 / 982\n", "6 0 1 4 1 867 5 1 4 3 0.0280269 25 / 892\n", "4 2 1 1 4 3 942 0 1 0 0.0167015 16 / 958\n", "1 4 13 4 0 0 0 992 6 8 0.0350195 36 / 1,028\n", "6 0 3 7 0 7 6 3 938 4 0.036961 36 / 974\n", "1 3 0 2 6 6 2 9 3 977 0.0317146 32 / 1,009\n", "1002 1126 1020 1002 955 902 973 1021 982 1017 0.0286 286 / 10,000" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Top-10 Hit Ratios: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
khit_ratio
10.9714
20.9740000
30.9740000
40.9740000
50.9740000
60.9740000
70.9740000
80.9740000
90.9740000
101.0
" ], "text/plain": [ "k hit_ratio\n", "--- -----------\n", "1 0.9714\n", "2 0.974\n", "3 0.974\n", "4 0.974\n", "5 0.974\n", "6 0.974\n", "7 0.974\n", "8 0.974\n", "9 0.974\n", "10 1" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Scoring History: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
timestampdurationtraining_speedepochsiterationssamplestraining_rmsetraining_loglosstraining_classification_errorvalidation_rmsevalidation_loglossvalidation_classification_error
2016-10-23 01:30:37 0.000 secNone0.000.0nannannannannannan
2016-10-23 01:30:38 3.250 sec6585 obs/sec0.068266714096.00.44739356.90661420.20016060.43817856.62468550.192
2016-10-23 01:30:5216.392 sec25255 obs/sec5.529681331776.00.11854680.47944970.01405340.20545081.45552180.0422
2016-10-23 01:30:5822.772 sec26887 obs/sec8.4650667124507904.00.07952380.21842430.00632400.19697491.33609810.0388
2016-10-23 01:31:0226.488 sec27373 obs/sec10.0352147602112.00.06256900.13521500.00391490.19723091.34208050.0389
2016-10-23 01:31:0227.130 sec27330 obs/sec10.0352147602112.00.07952380.21842430.00632400.19697491.33609810.0388
2016-10-23 01:36:1528.640 sec27158 obs/sec10.1034667148606208.00.12019980.49901600.01444800.21400931.57668260.0458
2016-10-23 01:36:2639.508 sec26899 obs/sec14.6090667214876544.00.14187160.69518060.02012750.19233231.27147330.037
2016-10-23 01:36:3245.717 sec27510 obs/sec17.47626672561048576.00.09780370.33038300.00956560.17720721.08015830.0314
2016-10-23 01:36:3851.301 sec27836 obs/sec20.00213332931200128.00.07732060.20514870.00597850.16909000.98074170.0286
" ], "text/plain": [ " timestamp duration training_speed epochs iterations samples training_rmse training_logloss training_classification_error validation_rmse validation_logloss validation_classification_error\n", "-- ------------------- ---------- ---------------- --------- ------------ ----------- --------------- ------------------ ------------------------------- ----------------- -------------------- ---------------------------------\n", " 2016-10-23 01:30:37 0.000 sec 0 0 0 nan nan nan nan nan nan\n", " 2016-10-23 01:30:38 3.250 sec 6585 obs/sec 0.0682667 1 4096 0.447394 6.90661 0.200161 0.438178 6.62469 0.192\n", " 2016-10-23 01:30:52 16.392 sec 25255 obs/sec 5.5296 81 331776 0.118547 0.47945 0.0140534 0.205451 1.45552 0.0422\n", " 2016-10-23 01:30:58 22.772 sec 26887 obs/sec 8.46507 124 507904 0.0795238 0.218424 0.00632403 0.196975 1.3361 0.0388\n", " 2016-10-23 01:31:02 26.488 sec 27373 obs/sec 10.0352 147 602112 0.062569 0.135215 0.00391488 0.197231 1.34208 0.0389\n", " 2016-10-23 01:31:02 27.130 sec 27330 obs/sec 10.0352 147 602112 0.0795238 0.218424 0.00632403 0.196975 1.3361 0.0388\n", " 2016-10-23 01:36:15 28.640 sec 27158 obs/sec 10.1035 148 606208 0.1202 0.499016 0.014448 0.214009 1.57668 0.0458\n", " 2016-10-23 01:36:26 39.508 sec 26899 obs/sec 14.6091 214 876544 0.141872 0.695181 0.0201275 0.192332 1.27147 0.037\n", " 2016-10-23 01:36:32 45.717 sec 27510 obs/sec 17.4763 256 1.04858e+06 0.0978037 0.330383 0.00956556 0.177207 1.08016 0.0314\n", " 2016-10-23 01:36:38 51.301 sec 27836 obs/sec 20.0021 293 1.20013e+06 0.0773206 0.205149 0.00597848 0.16909 0.980742 0.0286" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Retrieve model by H2O key\n", "model = h2o.get_model(model_id=model_checkpoint._id)\n", "model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusions\n", "\n", "In this Notebook you learned to:\n", "- use a H2O Deep Learning model (both CPU and GPU)\n", "- use GridSearch\n", "- use Checkpointing\n", "- use Early Stopping" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python [default]", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.12" } }, "nbformat": 4, "nbformat_minor": 1 }