{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How to recover best model from experiment log?\n", "Due to system error or other reason where scan_object is no longer available, it's still possible to get best model/s using nothing but the experiment log. In the below notebook you will learn exactly how.\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] } ], "source": [ "import sys\n", "sys.path.insert(0, '/Users/mikko/Documents/GitHub/talos/')\n", "\n", "import sys\n", "sys.path.insert(0, '/Users/mikko/Documents/GitHub/wrangle/')\n", "\n", "import talos\n", "import wrangle\n", "from keras.models import Sequential\n", "from keras.layers import Dense" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First we'll have to perform the `Scan()` experiment to produce the experiment log. Because the experiment log is stored on local machine, interrupted `Scan()` or other reason will not affect its availability. The experiment log is updated after each permutation; it contains an up-to-date record of the experiment." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 10/10 [00:34<00:00, 3.28s/it]\n" ] } ], "source": [ "# load the data\n", "x, y = talos.templates.datasets.iris()\n", "x_train, y_train, x_val, y_val = wrangle.array_split(x, y, .3)\n", "\n", "# set the parameter space boundary\n", "p = {'activation':['relu', 'elu'],\n", " 'optimizer': ['Nadam', 'Adam'],\n", " 'losses': ['logcosh'],\n", " 'shapes': ['brick'],\n", " 'first_neuron': [16, 32, 64, 128],\n", " 'hidden_layers':[0, 1, 2, 3],\n", " 'dropout': [.2, .3, .4],\n", " 'batch_size': [20, 30, 40, 50],\n", " 'epochs': [10]}\n", "\n", "# define the input model\n", "def iris_model(x_train, y_train, x_val, y_val, params):\n", "\n", " model = Sequential()\n", " model.add(Dense(params['first_neuron'], input_dim=4, activation=params['activation']))\n", "\n", " talos.utils.hidden_layers(model, params, 3)\n", "\n", " model.add(Dense(3, activation='softmax'))\n", " model.compile(optimizer=params['optimizer'], loss=params['losses'], metrics=['acc'])\n", "\n", " out = model.fit(x_train, y_train, callbacks=[talos.utils.ExperimentLogCallback('minimal_iris', params)],\n", " batch_size=params['batch_size'],\n", " epochs=params['epochs'],\n", " validation_data=[x_val, y_val],\n", " verbose=0)\n", "\n", " return out, model\n", "\n", "# start the experiment\n", "scan_object = talos.Scan(x=x_train,\n", " y=y_train,\n", " x_val=x_val,\n", " y_val=y_val,\n", " model=iris_model,\n", " experiment_name='reactivate',\n", " params=p,\n", " round_limit=10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can assume the case where we no longer have access to the `scan_object`. In this `Scan(...experiment_name...)` was set to \"reactivate\" so we'll find a folder with that name in the present working directory. Next we have to find out what is the name of the experiment log." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 96\r\n", "-rw-r--r-- 1 mikko staff 16K Sep 26 20:17 092619200831.csv\r\n", "-rw-r--r-- 1 mikko staff 1.3K Sep 26 20:18 092619201824.csv\r\n", "-rw-r--r-- 1 mikko staff 1.3K Sep 26 22:13 092619221236.csv\r\n", "-rw-r--r-- 1 mikko staff 1.3K Sep 26 22:18 092619221803.csv\r\n", "-rw-r--r-- 1 mikko staff 1.3K Sep 26 22:31 092619223042.csv\r\n", "-rw-r--r-- 1 mikko staff 1.3K Sep 26 22:35 092619223459.csv\r\n", "-rw-r--r-- 1 mikko staff 1.3K Sep 26 22:35 092619223524.csv\r\n", "-rw-r--r-- 1 mikko staff 1.3K Sep 26 22:56 092619225556.csv\r\n", "-rw-r--r-- 1 mikko staff 1.3K Sep 26 23:04 092619230425.csv\r\n" ] } ], "source": [ "# get the name of the experiment log\n", "!ls -lhtr reactivate" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this case it will be the most recent one `092619223042.csv` so let's go ahead and recover the best models." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from talos.utils.recover_best_model import recover_best_model\n", "results, models = recover_best_model(x_train=x_train,\n", " y_train=y_train,\n", " x_val=x_val,\n", " y_val=y_val,\n", " experiment_log='reactivate/092619221803.csv',\n", " input_model=iris_model,\n", " n_models=5,\n", " task='multi_label')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can access the cross-validation results:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | round_epochs | \n", "val_loss | \n", "val_acc | \n", "loss | \n", "acc | \n", "activation | \n", "batch_size | \n", "dropout | \n", "epochs | \n", "first_neuron | \n", "hidden_layers | \n", "losses | \n", "optimizer | \n", "shapes | \n", "crossval_mean_f1score | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "10 | \n", "0.028760 | \n", "0.955556 | \n", "0.031489 | \n", "0.885714 | \n", "relu | \n", "20 | \n", "0.3 | \n", "10 | \n", "128 | \n", "2 | \n", "logcosh | \n", "Adam | \n", "brick | \n", "0.930236 | \n", "
9 | \n", "10 | \n", "0.028902 | \n", "0.933333 | \n", "0.029437 | \n", "0.866667 | \n", "elu | \n", "50 | \n", "0.3 | \n", "10 | \n", "128 | \n", "2 | \n", "logcosh | \n", "Adam | \n", "brick | \n", "0.900427 | \n", "
2 | \n", "10 | \n", "0.023247 | \n", "0.888889 | \n", "0.020156 | \n", "0.942857 | \n", "elu | \n", "30 | \n", "0.3 | \n", "10 | \n", "128 | \n", "2 | \n", "logcosh | \n", "Adam | \n", "brick | \n", "0.980606 | \n", "
4 | \n", "10 | \n", "0.044717 | \n", "0.866667 | \n", "0.048760 | \n", "0.780952 | \n", "relu | \n", "30 | \n", "0.4 | \n", "10 | \n", "128 | \n", "2 | \n", "logcosh | \n", "Adam | \n", "brick | \n", "0.529231 | \n", "
8 | \n", "10 | \n", "0.047718 | \n", "0.866667 | \n", "0.052877 | \n", "0.761905 | \n", "elu | \n", "30 | \n", "0.3 | \n", "10 | \n", "64 | \n", "1 | \n", "logcosh | \n", "Adam | \n", "brick | \n", "0.860539 | \n", "