{ "cells": [ { "cell_type": "markdown", "id": "impossible-johnston", "metadata": {}, "source": [ "\"Drawing\"" ] }, { "cell_type": "markdown", "id": "reserved-museum", "metadata": {}, "source": [ "\n", "\n", "# Optimization Strategies for Deep Learning with Hyperactive (v3)\n", "\n", "
\n", "\n", " \n", "In this tutorial you will learn how to automate the process of selecting the best deep learning model for a given dataset. The structure and hyperparameters of a neural network have a big impact on the performance of your predictive model. Unfortunately, it is often a tedious endeavor to search for good model-parameters yourself. A solution to this is to automate this search process. Hyperactive is a python package designed to solve this problem. Hyperactive has some unique properties that enable you to automatically explore new models or improve existing ones. Some of those are:\n", " \n", " - Very easy to use. Only a few lines of new code to learn.\n", " - Flexible search space that can contain python objects\n", " - Sequential model based optimization techniques\n", " - Hyperactive-memory \"remembers\" parameters to save time\n", " - Results are processed for easy further use\n", "\n", " \n", "### [You can learn more about Hyperactive on Github](https://github.com/SimonBlanke/Hyperactive)\n", "\n", "## Table of contents:\n", "* [An introduction to Hyperactive](#first-optimization-run)\n", "* [First Deep Learning Optimization run](#deep1)\n", "* [Hyperparameter Optimization](#deep2)\n", "* [Continuing the Search](#continuing-search)\n", "* [Neural Architecture Search](#nas)\n", "* [Pretrained Neural Architecture Search](#pretrained-nas)\n", "\n" ] }, { "cell_type": "markdown", "id": "23b1447d", "metadata": {}, "source": [ "\n", "\n", "This tutorial is made for version 3 of Hyperactive" ] }, { "cell_type": "code", "execution_count": 1, "id": "international-domestic", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Device mapping:\n", "/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device\n", "/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device\n", "\n" ] } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "import plotly.express as px\n", "import plotly.graph_objects as go\n", "from plotly.subplots import make_subplots\n", "from scipy import stats\n", "\n", "from tensorflow.keras.datasets import mnist\n", "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Dense, Flatten\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.optimizers import SGD\n", "from keras.utils import np_utils\n", "from tensorflow import keras\n", "\n", "import tensorflow as tf\n", "\n", "config = tf.compat.v1.ConfigProto()\n", "config.gpu_options.allow_growth = True\n", "config.log_device_placement = True\n", "\n", "sess = tf.compat.v1.Session(config=config)\n", "tf.compat.v1.keras.backend.set_session(sess)\n", "\n", "verbose = 0\n", "\n", "from hyperactive import Hyperactive\n", "from hyperactive import BayesianOptimizer, EvolutionStrategyOptimizer\n", "\n", "color_scale = px.colors.sequential.Jet" ] }, { "cell_type": "markdown", "id": "olympic-german", "metadata": {}, "source": [ "\n", " \n", "Some helper functions we might want to use later. You can skip these. Lets move on to the introduction to Hyperactive." ] }, { "cell_type": "code", "execution_count": 2, "id": "foster-spain", "metadata": {}, "outputs": [], "source": [ "def warn(*args, **kwargs):\n", " pass\n", "import warnings\n", "warnings.warn = warn\n", "\n", "\n", "def func2str(row):\n", " return row.__name__\n", "\n", "\n", "def add_categorical_score(search_data):\n", " score = search_data[\"score\"]\n", " score_min = np.amin(score)\n", " score_max = np.amax(score)\n", " score_diff = score_max - score_min\n", " \n", " score_best_ = score_max - score_diff*0.25\n", " score_worst_ = score_min + score_diff*0.25\n", "\n", " def categorize_column(row):\n", " if row['score'] > score_best_:\n", " score_cat = \"best 25%\"\n", " elif row['score'] < score_worst_:\n", " score_cat = \"worst 25%\"\n", " else:\n", " score_cat = \"average\"\n", " return score_cat\n", "\n", " search_data[\"score_categorical\"] = search_data.apply(categorize_column, axis=1)\n", " return search_data" ] }, { "cell_type": "markdown", "id": "blind-wrong", "metadata": {}, "source": [ "\n", "\n", "
\n", " \n", "If you are already familiar with Hyperactive you can skip this section and start with [First Deep Learning Optimization run](#deep1).\n", "\n", "## An introduction to Hyperactive \n", "\n", "
\n", "\n", "\n", "This is where the interesting stuff begins. The following code is a very simple example to show how an optimization run looks like:\n", " " ] }, { "cell_type": "code", "execution_count": 3, "id": "parallel-thriller", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Results: 'inverted_quadratic_function' \n", " Best score: -0.02250000000006299 \n", " Best parameter:\n", " 'x' : -0.15000000000020997 \n", " \n", " Evaluation time : 0.0027713775634765625 sec [43.42 %]\n", " Optimization time : 0.0036106109619140625 sec [56.58 %]\n", " Iteration time : 0.006381988525390625 sec [15669.1 iter/sec]\n", " \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r" ] } ], "source": [ "# the objective function defines the \"model\" we want to optimize\n", "def inverted_quadratic_function(parameters):\n", " # access the parameters from the search space\n", " score = (parameters[\"x\"]*parameters[\"x\"])\n", " # Hyperactive always maximizes the value returned by the objective function\n", " return -score\n", "\n", "# the search space defines where to search for the optimal parameters\n", "search_space = {\n", " \"x\" : list(np.arange(-10, 10, 0.01))\n", "}\n", "\n", "# Hyperactive will run the objective function n_iter-times and search for the best parameters\n", "hyper = Hyperactive()\n", "hyper.add_search(inverted_quadratic_function, search_space, n_iter=100)\n", "hyper.run()" ] }, { "cell_type": "markdown", "id": "stopped-reservation", "metadata": {}, "source": [ "\n", "After performing an optimization run Hyperactive can return the collected search data. From the \"results\"-method you get the search data in form of a pandas dataframe. Each row contains the parameter-set, score (and other information) used in the iteration. You can use this dataframe to plot the search data or do your own data exploration with it." ] }, { "cell_type": "code", "execution_count": 4, "id": "raising-remainder", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
xeval_timeiter_timescore
03.590.0002020.000647-12.8881
16.610.0000400.000056-43.6921
2-6.010.0001280.000140-36.1201
3-2.020.0000200.000311-4.0804
41.970.0000410.000461-3.8809
...............
951.300.0000170.000028-1.6900
96-2.930.0000170.000028-8.5849
971.060.0000170.000028-1.1236
981.350.0000170.000029-1.8225
99-5.010.0000170.000028-25.1001
\n", "

100 rows × 4 columns

\n", "
" ], "text/plain": [ " x eval_time iter_time score\n", "0 3.59 0.000202 0.000647 -12.8881\n", "1 6.61 0.000040 0.000056 -43.6921\n", "2 -6.01 0.000128 0.000140 -36.1201\n", "3 -2.02 0.000020 0.000311 -4.0804\n", "4 1.97 0.000041 0.000461 -3.8809\n", ".. ... ... ... ...\n", "95 1.30 0.000017 0.000028 -1.6900\n", "96 -2.93 0.000017 0.000028 -8.5849\n", "97 1.06 0.000017 0.000028 -1.1236\n", "98 1.35 0.000017 0.000029 -1.8225\n", "99 -5.01 0.000017 0.000028 -25.1001\n", "\n", "[100 rows x 4 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# access the results from a pandas dataframe\n", "search_data = hyper.results(inverted_quadratic_function)\n", "search_data" ] }, { "cell_type": "markdown", "id": "cooperative-direction", "metadata": {}, "source": [ "\n", "\n", "The example already shows a lot of the features of Hyperactive:\n", "\n", " - objective functions can be anything you want:\n", " - a mathematical function\n", " - a machine-/deep-learning model (sklearn, keras, pytorch, ...)\n", " - a simulation environment\n", " - you could even access code from another language\n", " - the search space dictionary can have as many dimensions as you want\n", " - via \"add_search\" you can run multiple different optimizations in parallel\n", " " ] }, { "cell_type": "code", "execution_count": 5, "id": "intellectual-setting", "metadata": {}, "outputs": [ { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hovertemplate": "x=%{x}
score=%{y}", "legendgroup": "", "marker": { "color": "#636efa", "symbol": "circle" }, "mode": "markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ 3.5899999999997103, 6.609999999999644, -6.010000000000085, -2.02000000000017, 1.9699999999997448, 5.95999999999966, -10, 9.989999999999576, -1.3600000000001842, 6.87999999999964, -9.630000000000008, 9.559999999999583, -7.730000000000048, 3.049999999999722, 4.199999999999697, 4.959999999999681, -1.9100000000001724, -4.670000000000114, -1.5700000000001797, 1.7499999999997495, -4.4900000000001175, -1.9500000000001716, 5.339999999999673, 0.4199999999997779, 0.15999999999978343, -1.1700000000001882, 6.369999999999649, -1.2600000000001863, 6.32999999999965, 2.609999999999731, -0.8800000000001944, 6.139999999999656, -6.000000000000085, -8.790000000000026, 4.669999999999687, 7.699999999999623, -0.6900000000001985, 7.539999999999626, 0.649999999999773, -7.790000000000047, -5.5200000000000955, 3.9599999999997024, -8.040000000000042, -0.9200000000001936, 8.7599999999996, -8.710000000000027, 5.219999999999676, -1.4500000000001823, -5.110000000000104, 5.169999999999677, 5.709999999999665, 3.4299999999997137, -7.420000000000055, -6.110000000000083, -9.590000000000009, -0.9300000000001933, 6.339999999999652, 3.4799999999997127, -6.390000000000077, -8.180000000000039, -6.70000000000007, -0.15000000000020997, -8.280000000000037, -9.98, 0.5399999999997753, 8.679999999999602, 3.509999999999712, 9.839999999999577, -4.5800000000001155, -5.640000000000093, 7.989999999999618, -5.090000000000105, 8.6499999999996, 9.359999999999587, -0.32000000000020634, -8.10000000000004, 4.609999999999689, -0.9200000000001936, 4.859999999999683, 8.359999999999609, 0.5299999999997755, -1.3000000000001855, 2.719999999999729, -5.530000000000095, 2.2799999999997382, 8.659999999999602, 2.2699999999997384, -4.420000000000119, 0.8399999999997689, 5.789999999999663, -4.040000000000127, -7.840000000000046, -8.920000000000023, 9.469999999999587, -9.240000000000016, 1.2999999999997591, -2.9300000000001507, 1.0599999999997642, 1.349999999999758, -5.010000000000106 ], "xaxis": "x", "y": [ -12.88809999999792, -43.69209999999529, -36.120100000001024, -4.0804000000006875, -3.8808999999989946, -35.52159999999594, -100, -99.80009999999152, -1.8496000000005008, -47.33439999999505, -92.73690000000015, -91.39359999999203, -59.75290000000075, -9.302499999998304, -17.639999999997457, -24.601599999996836, -3.648100000000659, -21.80890000000106, -2.464900000000564, -3.0624999999991234, -20.160100000001055, -3.802500000000669, -28.515599999996507, -0.17639999999981343, -0.025599999999930696, -1.3689000000004405, -40.57689999999553, -1.5876000000004695, -40.06889999999557, -6.812099999998597, -0.7744000000003421, -37.699599999995776, -36.00000000000102, -77.26410000000045, -21.80889999999708, -59.28999999999419, -0.47610000000027386, -56.85159999999436, -0.4224999999997049, -60.68410000000073, -30.470400000001053, -15.681599999997644, -64.64160000000066, -0.8464000000003561, -76.737599999993, -75.86410000000048, -27.248399999996614, -2.1025000000005285, -26.112100000001064, -26.728899999996656, -32.60409999999617, -11.764899999998036, -55.056400000000814, -37.33210000000101, -91.96810000000016, -0.8649000000003596, -40.195599999995586, -12.110399999998, -40.832100000000985, -66.91240000000063, -44.890000000000946, -0.02250000000006299, -68.5584000000006, -99.60040000000001, -0.29159999999975733, -75.34239999999309, -12.320099999997979, -96.82559999999168, -20.976400000001057, -31.809600000001048, -63.8400999999939, -25.908100000001067, -74.8224999999931, -87.60959999999227, -0.10240000000013205, -65.61000000000065, -21.252099999997128, -0.8464000000003561, -23.61959999999692, -69.88959999999345, -0.28089999999976206, -1.6900000000004822, -7.398399999998525, -30.580900000001055, -5.198399999998807, -74.9955999999931, -5.152899999998812, -19.536400000001052, -0.7055999999996118, -33.5240999999961, -16.321600000001027, -61.46560000000072, -79.56640000000041, -89.68089999999218, -85.3776000000003, -1.6899999999993738, -8.584900000000884, -1.1235999999995, -1.8224999999993468, -25.100100000001067 ], "yaxis": "y" } ], "layout": { "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "x" } }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "title": { "text": "score" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = px.scatter(search_data, x=\"x\", y=\"score\")\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "casual-northern", "metadata": {}, "source": [ "\n", "\n", "The 1D scatter plot above shows the score dependend on the parameter \"x\". The default optimization algorithm is a random search, which is good for exploring the search space of an objective function. This makes the inverted quadratic function clearly visible in our scatter plot.\n", "\n", "For objective functions that are evaluated fast it makes sense to make some extensive exploration of the search space with a random search in the beginning. This is not recommendable for expensive objective functions like deep learning hyperparameter optimization or even neural architecture search!
\n", "Each function evaluation costs a lot of time. Therefore it should be carfully calculated which positions in the search space to evaluate next. We will utilize more sophisticated optimization strategies for our approach to deep learning optimization" ] }, { "cell_type": "markdown", "id": "affiliated-floor", "metadata": {}, "source": [ "\n", "Lets try out Hyperactive on a deep learning model!\n", "For this we choose keras as a deep learning framework and the mnist dataset.\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "strategic-waters", "metadata": {}, "outputs": [], "source": [ "# load dataset\n", "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n", "\n", "img_width = 28\n", "img_height = 28\n", "\n", "x_train = x_train.astype(\"float32\")\n", "x_train /= 255.0\n", "x_test = x_test.astype(\"float32\")\n", "x_test /= 255.0\n", "\n", "# reshape input data\n", "x_train = x_train.reshape(x_train.shape[0], img_width, img_height, 1)\n", "x_test = x_test.reshape(x_test.shape[0], img_width, img_height, 1)\n", "\n", "# one hot encode outputs\n", "y_train = np_utils.to_categorical(y_train)\n", "y_test = np_utils.to_categorical(y_test)\n", "num_classes = y_test.shape[1]" ] }, { "cell_type": "markdown", "id": "senior-sphere", "metadata": {}, "source": [ "\n", " \n", "If you have limited computing ressources you can restore the following lines of code to make the training quicker:" ] }, { "cell_type": "code", "execution_count": 7, "id": "controlling-calvin", "metadata": {}, "outputs": [], "source": [ "# x_train = x_train[0:1000]\n", "# y_train = y_train[0:1000]\n", "\n", "# x_test = x_test[0:1000]\n", "# y_test = y_test[0:1000]" ] }, { "cell_type": "markdown", "id": "liked-display", "metadata": {}, "source": [ "\n", " \n", "## First deep learning optimization run \n", "\n", "
\n", "\n", "In the first deep learning optimization run we will tune the number of epochs and the learning rate. Since we will keep using the same data set and roughly the same neural network size we can keep the best results of those two parameters for the next searches. This way we have less parameters to optimize later. We will use Evolution strategy for this search:\n", "\n", " \n", "
\n", " \n", " \n", "### Evolution strategy\n", " \n", "Evolution strategy is a meta-heuristic population based optimization technique. It works by initializing a population of individual optimizers in the search space. Each optimizer can move on its own or new ones can be created by mixing the properties of two optimizers. Evolution strategy is a reliable algorithm even for very non-linear search spaces.\n", " \n", " \n", "The following plots show an example of the path the evolution strategy algorithm would take in different objective functions:\n", " \n", "\n", "\n", "\n", "
\n", " \n" ] }, { "cell_type": "code", "execution_count": 8, "id": "handmade-mechanics", "metadata": { "scrolled": true }, "outputs": [], "source": [ "def deep_learning_model(params):\n", " model = Sequential()\n", " model.add(Conv2D(10, (4, 4), input_shape=(img_width, img_height, 1), activation=\"relu\"))\n", " \n", " model.add(MaxPooling2D(pool_size=(2, 2)))\n", " model.add(Flatten())\n", " model.add(Dense(100, activation=\"relu\"))\n", " model.add(Dense(num_classes, activation=\"softmax\"))\n", "\n", " opt = keras.optimizers.Adam(learning_rate=params[\"learning_rate\"])\n", " model.compile(loss=\"categorical_crossentropy\", optimizer=opt, metrics=[\"accuracy\"])\n", " model.fit(\n", " x_train,\n", " y_train,\n", " validation_data=(x_test, y_test),\n", " epochs=params[\"epochs\"],\n", " batch_size=500,\n", " verbose=verbose,\n", " )\n", " _, score = model.evaluate(x=x_test, y=y_test, verbose=verbose)\n", "\n", " return score\n", " \n", "search_space_0 = {\n", " \"learning_rate\": [1/(10**x) for x in range(0, 9)],\n", " \"epochs\": list(range(3, 15)),\n", "}" ] }, { "cell_type": "code", "execution_count": 9, "id": "burning-innocent", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Results: 'deep_learning_model' \n", " Best score: 0.987500011920929 \n", " Best parameter:\n", " 'learning_rate' : 0.01 \n", " 'epochs' : 6.0 \n", " \n", " Evaluation time : 202.410710811615 sec [99.99 %]\n", " Optimization time : 0.013862848281860352 sec [0.01 %]\n", " Iteration time : 202.42457365989685 sec [4.05 sec/iter]\n", " \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r" ] } ], "source": [ "optimizer = EvolutionStrategyOptimizer()\n", "\n", "hyper_0 = Hyperactive()\n", "hyper_0.add_search(deep_learning_model, search_space_0, n_iter=50, optimizer=optimizer)\n", "hyper_0.run()" ] }, { "cell_type": "code", "execution_count": 10, "id": "derived-seating", "metadata": {}, "outputs": [], "source": [ "search_data_0 = hyper_0.results(deep_learning_model)" ] }, { "cell_type": "code", "execution_count": 11, "id": "olive-prayer", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hovertemplate": "learning_rate=%{x}
epochs=%{y}
score=%{marker.color}", "legendgroup": "", "marker": { "color": [ 0.984000027179718, 0.9607999920845032, 0.987500011920929, 0.9854999780654907, 0.9370999932289124, 0.9466999769210815, 0.1525000035762787, 0.10100000351667404, 0.1251000016927719, 0.08919999748468399, 0.9835000038146973, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.9370999932289124, 0.9833999872207642, 0.14419999718666077, 0.10100000351667404, 0.1251000016927719, 0.08919999748468399, 0.984000027179718, 0.8647000193595886, 0.987500011920929, 0.14419999718666077, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.9839000105857849, 0.9836999773979187, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.9370999932289124, 0.987500011920929, 0.987500011920929, 0.9370999932289124, 0.987500011920929, 0.987500011920929, 0.0957999974489212, 0.987500011920929, 0.9370999932289124, 0.987500011920929, 0.14419999718666077, 0.9370999932289124, 0.987500011920929, 0.987500011920929, 0.0957999974489212, 0.987500011920929 ], "coloraxis": "coloraxis", "symbol": "circle" }, "mode": "markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ 0.01, 0.0001, 0.01, 0.01, 0.0001, 0.0001, 1e-08, 1, 1e-08, 1, 0.01, 0.01, 0.01, 0.01, 0.0001, 0.001, 1e-08, 1, 1e-08, 1, 0.01, 1e-05, 0.01, 1e-08, 0.01, 0.01, 0.01, 0.01, 0.001, 0.01, 0.01, 0.01, 0.01, 0.01, 0.0001, 0.01, 0.01, 0.0001, 0.01, 0.01, 1, 0.01, 0.0001, 0.01, 1e-08, 0.0001, 0.01, 0.01, 1, 0.01 ], "xaxis": "x", "y": [ 4, 12, 6, 9, 6, 9, 3, 3, 14, 14, 5, 6, 6, 6, 6, 9, 6, 3, 14, 14, 4, 13, 6, 6, 6, 6, 6, 6, 6, 7, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6 ], "yaxis": "y" } ], "layout": { "coloraxis": { "colorbar": { "title": { "text": "score" } }, "colorscale": [ [ 0, "rgb(0,0,131)" ], [ 0.2, "rgb(0,60,170)" ], [ 0.4, "rgb(5,255,255)" ], [ 0.6, "rgb(255,255,0)" ], [ 0.8, "rgb(250,0,0)" ], [ 1, "rgb(128,0,0)" ] ] }, "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "learning_rate" }, "type": "log" }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "title": { "text": "epochs" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = px.scatter(search_data_0, \n", " x=\"learning_rate\", \n", " y=\"epochs\", \n", " color=\"score\", \n", " color_continuous_scale=color_scale, \n", " log_x=True)\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "entertaining-drove", "metadata": {}, "source": [ "\n", " \n", "The 2D scatter plot in the cell above shows which values for the number of epochs and the learning rate that were explored during the optimization run. The scores the model achieved is represented as a color-dimension.
\n", " \n", "The low scores make the scaling of the better scores less visible. Lets remove the worst scores so that we can distinguish differences in the better scores:" ] }, { "cell_type": "code", "execution_count": 12, "id": "photographic-architecture", "metadata": {}, "outputs": [], "source": [ "score_max = np.amax(search_data_0[\"score\"])\n", "score_std = search_data_0[\"score\"].std()\n", "search_data_0_f = search_data_0[abs(search_data_0[\"score\"]-score_max) < score_std*0.3]" ] }, { "cell_type": "code", "execution_count": 13, "id": "dirty-helicopter", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hovertemplate": "learning_rate=%{x}
epochs=%{y}
score=%{marker.color}", "legendgroup": "", "marker": { "color": [ 0.984000027179718, 0.9607999920845032, 0.987500011920929, 0.9854999780654907, 0.9370999932289124, 0.9466999769210815, 0.9835000038146973, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.9370999932289124, 0.9833999872207642, 0.984000027179718, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.9839000105857849, 0.9836999773979187, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.9370999932289124, 0.987500011920929, 0.987500011920929, 0.9370999932289124, 0.987500011920929, 0.987500011920929, 0.987500011920929, 0.9370999932289124, 0.987500011920929, 0.9370999932289124, 0.987500011920929, 0.987500011920929, 0.987500011920929 ], "coloraxis": "coloraxis", "symbol": "circle" }, "mode": "markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ 0.01, 0.0001, 0.01, 0.01, 0.0001, 0.0001, 0.01, 0.01, 0.01, 0.01, 0.0001, 0.001, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.001, 0.01, 0.01, 0.01, 0.01, 0.01, 0.0001, 0.01, 0.01, 0.0001, 0.01, 0.01, 0.01, 0.0001, 0.01, 0.0001, 0.01, 0.01, 0.01 ], "xaxis": "x", "y": [ 4, 12, 6, 9, 6, 9, 5, 6, 6, 6, 6, 9, 4, 6, 6, 6, 6, 6, 6, 7, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6 ], "yaxis": "y" } ], "layout": { "coloraxis": { "colorbar": { "title": { "text": "score" } }, "colorscale": [ [ 0, "rgb(0,0,131)" ], [ 0.2, "rgb(0,60,170)" ], [ 0.4, "rgb(5,255,255)" ], [ 0.6, "rgb(255,255,0)" ], [ 0.8, "rgb(250,0,0)" ], [ 1, "rgb(128,0,0)" ] ] }, "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "learning_rate" }, "type": "log" }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "title": { "text": "epochs" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = px.scatter(search_data_0_f, \n", " x=\"learning_rate\", \n", " y=\"epochs\", \n", " color=\"score\", \n", " color_continuous_scale=color_scale, \n", " log_x=True)\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "strong-annotation", "metadata": {}, "source": [ "\n", " \n", "In the scatter plot with the filtered data it is easier to see how the number of epochs and the learning rate affects the best scores. A higher number of epochs also increases the training time of the model in each optimization iteration. As a comprimise between training time and score we set the parameters to the following:" ] }, { "cell_type": "code", "execution_count": 14, "id": "organized-saskatchewan", "metadata": {}, "outputs": [], "source": [ "learning_rate = 0.01\n", "epochs = 9" ] }, { "cell_type": "code", "execution_count": 15, "id": "sweet-accountability", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "name": "evaluation time", "nbinsx": 15, "type": "histogram", "x": [ 6.703615188598633, 18.08081293106079, 9.397381782531738, 13.648757457733154, 9.358824968338013, 13.660170555114746, 4.976018905639648, 4.954785108566284, 20.90669584274292, 20.91729497909546, 7.868107557296753, 3.790855407714844e-05, 2.4318695068359375e-05, 2.288818359375e-05, 2.4318695068359375e-05, 13.634851694107056, 9.324142932891846, 3.62396240234375e-05, 3.0517578125e-05, 2.0742416381835938e-05, 1.9073486328125e-05, 19.497190713882446, 3.314018249511719e-05, 2.384185791015625e-05, 0.00011610984802246094, 2.0742416381835938e-05, 2.002716064453125e-05, 1.9550323486328125e-05, 9.299234628677368, 10.761934518814087, 3.504753112792969e-05, 2.4080276489257812e-05, 2.3365020751953125e-05, 2.3126602172851562e-05, 2.2649765014648438e-05, 2.288818359375e-05, 2.3126602172851562e-05, 2.2172927856445312e-05, 2.2649765014648438e-05, 2.193450927734375e-05, 9.419921875, 3.361701965332031e-05, 3.170967102050781e-05, 2.3603439331054688e-05, 2.3365020751953125e-05, 2.1219253540039062e-05, 3.170967102050781e-05, 2.2411346435546875e-05, 7.081031799316406e-05, 2.0265579223632812e-05 ] }, { "name": "optimization time", "nbinsx": 15, "type": "histogram", "x": [ 0.0011382102966308594, 0.0005822181701660156, 0.0010085105895996094, 0.0006000995635986328, 0.0006082057952880859, 0.00063323974609375, 0.0006337165832519531, 0.0005648136138916016, 0.0005900859832763672, 0.0007152557373046875, 0.0006186962127685547, 8.58306884765625e-05, 4.100799560546875e-05, 3.814697265625e-05, 0.00010275840759277344, 0.0006074905395507812, 0.0006682872772216797, 0.00012731552124023438, 7.295608520507812e-05, 6.008148193359375e-05, 6.103515625e-05, 0.0005869865417480469, 0.0001480579376220703, 5.888938903808594e-05, 7.843971252441406e-05, 6.699562072753906e-05, 6.008148193359375e-05, 6.031990051269531e-05, 0.0005929470062255859, 0.0006701946258544922, 0.00014734268188476562, 5.698204040527344e-05, 7.152557373046875e-05, 6.818771362304688e-05, 4.7206878662109375e-05, 7.081031799316406e-05, 6.580352783203125e-05, 4.673004150390625e-05, 6.723403930664062e-05, 6.890296936035156e-05, 0.0006394386291503906, 0.00014853477478027344, 5.745887756347656e-05, 7.176399230957031e-05, 4.673004150390625e-05, 4.458427429199219e-05, 7.390975952148438e-05, 6.747245788574219e-05, 5.626678466796875e-05, 6.508827209472656e-05 ] } ], "layout": { "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# times in seconds\n", "eval_times = search_data_0[\"eval_time\"]\n", "opt_times = search_data_0[\"iter_time\"]-search_data_0[\"eval_time\"]\n", "\n", "fig = go.Figure()\n", "fig.add_trace(go.Histogram(x=eval_times, name=\"evaluation time\", nbinsx=15))\n", "fig.add_trace(go.Histogram(x=opt_times, name=\"optimization time\", nbinsx=15))\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "southern-profile", "metadata": {}, "source": [ "\n", "\n", "The histogram above shows multiple interesting things. The optimization time is much smaller than most of the function evaluation times. We want a powerful optimization algorithm but we do not want to waste much extra time.
\n", " \n", "Another interesting thing is that some of the function evaluation times are extremely short. This is the Hyperactive-memory-dictionary working its magic. Each time a position in the search space is selected and evaluated Hyperactive saves the position as a key and the score as a value in a dictionary. When the optimizer selects the next position Hyperactive will look up if it exists in the memory-dictionary (which is very fast). If the position is encountered again Hyperactive will just extract the score from the dictionary instead of reevaluating the objective funtion." ] }, { "cell_type": "markdown", "id": "immediate-negative", "metadata": {}, "source": [ "\n", " \n", "Lets extract the best parameters from Hyperactive. Since the size and type of neural networks will be similar we can use the best learning rate and epochs we just found. This way we can concentrate on optimizing other hyperparameters in the following runs.\n", " \n", "" ] }, { "cell_type": "markdown", "id": "anticipated-transaction", "metadata": {}, "source": [ "\n", "\n", "## Hyperparameter Optimization \n", "\n", "
\n", "\n", "In the next step we can start optimizing some model specific hyperparameters. In this case the number of filters and kernel size in the first convolutional layer. We also want to know the best number of neurons in the dense layer. We will continue with the Bayesian optimization algorithm for all further searches:\n" ] }, { "cell_type": "markdown", "id": "large-senate", "metadata": {}, "source": [ "\n", " \n", "### What is Bayesian Optimization?\n", "\n", "
\n", "\n", "Bayesian optimization is a global optimization technique that uses a machine learning model (surrogate model) to approximate the objective function. It relies on a gaussian process regressor fitting to known positions and scores in the search space and predicting where to search next. It follows the following steps:\n", " \n", " - fit the gaussian process regressor to the training data (positions in search space) and the target (score of each position).\n", " - the regressor makes a prediction of every position in the search space\n", " - from the predictions an acquisition function is calculated that determines which position to evaluate next\n", " - after the evaluation the algorithm adds the position and score to the training data\n", " \n", "\n", "Since the regressor is trained in every iteration the optimization step takes a long time compared to other algorithms. This is why it is often used for objective functions with a long evaluation time. The long optimization time does not matter if the evaluation time is even longer. In those cases it is much more important that each new position is carfully selected to avoid wasted time of an evaluation that has a low score. \n", "\n", "The following plots show an example of the path a bayesian optimization algorithm would take in different objective functions:\n", " \n", "\n", "\n", "\n", "
\n", " \n", " \n", "It is also interesting to see how the Bayesian optimization algorithm quickly starts to focus on the area of the search space with the best scores.\n", " " ] }, { "cell_type": "code", "execution_count": 16, "id": "latter-georgia", "metadata": { "scrolled": true }, "outputs": [], "source": [ "def deep_learning_model(params):\n", " model = Sequential()\n", " model.add(Conv2D(params[\"filters.0\"], (params[\"kernel_size.0\"], params[\"kernel_size.0\"]), input_shape=(img_width, img_height, 1), activation=\"relu\"))\n", " \n", " model.add(MaxPooling2D(pool_size=(2, 2)))\n", " model.add(Flatten())\n", " model.add(Dense(params[\"units.0\"], activation=\"relu\"))\n", " model.add(Dense(num_classes, activation=\"softmax\"))\n", "\n", " opt = keras.optimizers.Adam(learning_rate=learning_rate)\n", " model.compile(loss=\"categorical_crossentropy\", optimizer=opt, metrics=[\"accuracy\"])\n", " model.fit(\n", " x_train,\n", " y_train,\n", " validation_data=(x_test, y_test),\n", " epochs=int(epochs),\n", " batch_size=1024,\n", " verbose=verbose,\n", " )\n", " _, score = model.evaluate(x=x_test, y=y_test, verbose=verbose)\n", "\n", " return score\n", " \n", "search_space_1 = {\n", " \"filters.0\": list(range(5, 15)),\n", " \"kernel_size.0\": list(range(3, 6)),\n", " \"units.0\": list(range(10, 500, 10)),\n", "\n", "}" ] }, { "cell_type": "code", "execution_count": 17, "id": "acting-specialist", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Results: 'deep_learning_model' \n", " Best score: 0.9886000156402588 \n", " Best parameter:\n", " 'filters.0' : 12 \n", " 'kernel_size.0' : 4 \n", " 'units.0' : 200 \n", " \n", " Evaluation time : 430.05269265174866 sec [99.87 %]\n", " Optimization time : 0.5456101894378662 sec [0.13 %]\n", " Iteration time : 430.5983028411865 sec [14.35 sec/iter]\n", " \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r" ] } ], "source": [ "optimizer = BayesianOptimizer()\n", "\n", "hyper_1 = Hyperactive()\n", "hyper_1.add_search(deep_learning_model, \n", " search_space_1, \n", " n_iter=30, \n", " optimizer=optimizer, \n", " )\n", "hyper_1.run()" ] }, { "cell_type": "code", "execution_count": 18, "id": "outstanding-airline", "metadata": {}, "outputs": [], "source": [ "search_data_1 = hyper_1.results(deep_learning_model)\n", "best_para_1 = hyper_1.best_para(deep_learning_model)" ] }, { "cell_type": "code", "execution_count": 19, "id": "current-sleeve", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hovertemplate": "filters.0=%{x}
kernel_size.0=%{y}
units.0=%{z}
score=%{marker.color}", "legendgroup": "", "marker": { "color": [ 0.9886000156402588, 0.9825000166893005, 0.9871000051498413, 0.9840999841690063, 0.9878000020980835, 0.9868000149726868, 0.9635999798774719, 0.9868999719619751, 0.9866999983787537, 0.9811999797821045, 0.9869999885559082, 0.9878000020980835, 0.9866999983787537, 0.9861999750137329, 0.98580002784729, 0.9872999787330627, 0.9843000173568726, 0.9883999824523926, 0.9883999824523926, 0.9843000173568726, 0.9882000088691711, 0.9836000204086304, 0.9872999787330627, 0.9868000149726868, 0.9869999885559082, 0.9861999750137329, 0.9876999855041504, 0.9876000285148621, 0.9879999756813049, 0.9873999953269958 ], "coloraxis": "coloraxis", "symbol": "circle" }, "mode": "markers", "name": "", "scene": "scene", "showlegend": false, "type": "scatter3d", "x": [ 12, 5, 9, 5, 10, 10, 5, 5, 14, 14, 14, 14, 5, 14, 14, 14, 8, 10, 5, 5, 7, 5, 12, 10, 12, 14, 10, 14, 7, 11 ], "y": [ 4, 3, 4, 4, 4, 5, 3, 5, 3, 5, 5, 5, 5, 5, 3, 5, 3, 5, 5, 5, 5, 3, 5, 5, 5, 3, 5, 5, 5, 5 ], "z": [ 200, 120, 250, 100, 490, 270, 10, 490, 490, 10, 370, 230, 380, 170, 300, 430, 440, 210, 300, 240, 330, 320, 210, 330, 490, 210, 190, 260, 300, 230 ] } ], "layout": { "autosize": false, "coloraxis": { "colorbar": { "title": { "text": "score" } }, "colorscale": [ [ 0, "rgb(0,0,131)" ], [ 0.2, "rgb(0,60,170)" ], [ 0.4, "rgb(5,255,255)" ], [ 0.6, "rgb(255,255,0)" ], [ 0.8, "rgb(250,0,0)" ], [ 1, "rgb(128,0,0)" ] ] }, "height": 700, "legend": { "tracegroupgap": 0 }, "scene": { "domain": { "x": [ 0, 1 ], "y": [ 0, 1 ] }, "xaxis": { "title": { "text": "filters.0" } }, "yaxis": { "title": { "text": "kernel_size.0" } }, "zaxis": { "title": { "text": "units.0" } } }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "3D scatter plot" }, "width": 950 } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig1 = px.scatter_3d(search_data_1, \n", " x='filters.0', \n", " y='kernel_size.0', \n", " z='units.0', \n", " color='score', \n", " color_continuous_scale=color_scale,\n", " title=\"3D scatter plot\",\n", " )\n", "\n", "fig1.update_layout(\n", " autosize=False,\n", " width=950,\n", " height=700,)\n", "\n", "fig1.show()" ] }, { "cell_type": "markdown", "id": "synthetic-badge", "metadata": {}, "source": [ "\n", " \n", "The image above shows a 3D scatter plot where each spatial dimension corresponds to a parameter and one color dimension showing the score. Based on this plot you could change the search space for your next optimization run to focus more on areas with high performing paramter sets." ] }, { "cell_type": "markdown", "id": "accepted-knight", "metadata": {}, "source": [ "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "found-equivalent", "metadata": {}, "source": [ "\n", "\n", "" ] }, { "cell_type": "markdown", "id": "critical-pasta", "metadata": {}, "source": [ "\n", "\n", "## Continuing the Search \n", "\n", "
\n", "\n", "Hyperactive makes it very easy to continue a search. The search data you already used for data exploration can just be passed to Hyperactive. This is done in multiple ways:\n", " \n", " - You can extract the best parameters via the \"best_para\"-method. This can than be passed to \"initialize\" to start at this position in the search space\n", " - The search data from the \"results\"-method can be passed to \"memory_warm_start\". The search data is automaticaly added into the memory-dictionary.\n", " - You can also pass the search data to \"warm_start_smbo\". This has the effect that the Bayesian optimizer can do more precise approximations in the beginning of the optimization run.\n", " \n", " \n", " " ] }, { "cell_type": "code", "execution_count": 20, "id": "bright-hughes", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Results: 'deep_learning_model' \n", " Best score: 0.9896000027656555 \n", " Best parameter:\n", " 'filters.0' : 16 \n", " 'kernel_size.0' : 6 \n", " 'units.0' : 550 \n", " \n", " Evaluation time : 1292.1899983882904 sec [99.8 %]\n", " Optimization time : 2.5350046157836914 sec [0.2 %]\n", " Iteration time : 1294.725003004074 sec [18.5 sec/iter]\n", " \n" ] } ], "source": [ "initialize = {\"vertices\": 8, \"random\": 4, \"warm_start\": [best_para_1]}\n", "\n", "optimizer = BayesianOptimizer(warm_start_smbo=search_data_1)\n", "\n", "search_space_2 = {\n", " \"filters.0\": list(range(10, 17)),\n", " \"kernel_size.0\": list(range(3, 7)),\n", " \"units.0\": list(range(400, 800, 10)),\n", "\n", "}\n", "\n", "hyper_2 = Hyperactive()\n", "hyper_2.add_search(deep_learning_model, \n", " search_space_2, \n", " n_iter=70, \n", " memory_warm_start=search_data_1, \n", " initialize=initialize,\n", " optimizer=optimizer)\n", "hyper_2.run()" ] }, { "cell_type": "code", "execution_count": 21, "id": "international-terrorist", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filters.0kernel_size.0units.0eval_timeiter_timescore
011568017.38150517.3824580.9850
115470020.96248220.9633100.9869
210666017.44716117.4477480.9864
312379017.78145317.7820380.9856
416340019.68982919.6904150.9864
.....................
6512652018.20443718.2854530.9884
661665500.0000560.0829290.9896
6713654018.98947819.0348200.9885
681665500.0000510.0848280.9896
691665500.0000500.1071230.9896
\n", "

70 rows × 6 columns

\n", "
" ], "text/plain": [ " filters.0 kernel_size.0 units.0 eval_time iter_time score\n", "0 11 5 680 17.381505 17.382458 0.9850\n", "1 15 4 700 20.962482 20.963310 0.9869\n", "2 10 6 660 17.447161 17.447748 0.9864\n", "3 12 3 790 17.781453 17.782038 0.9856\n", "4 16 3 400 19.689829 19.690415 0.9864\n", ".. ... ... ... ... ... ...\n", "65 12 6 520 18.204437 18.285453 0.9884\n", "66 16 6 550 0.000056 0.082929 0.9896\n", "67 13 6 540 18.989478 19.034820 0.9885\n", "68 16 6 550 0.000051 0.084828 0.9896\n", "69 16 6 550 0.000050 0.107123 0.9896\n", "\n", "[70 rows x 6 columns]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "search_data_2 = hyper_2.results(deep_learning_model)\n", "search_data_2" ] }, { "cell_type": "code", "execution_count": 22, "id": "prime-debate", "metadata": {}, "outputs": [], "source": [ "search_data_2 = search_data_2.append(search_data_1, ignore_index=True)\n" ] }, { "cell_type": "code", "execution_count": 23, "id": "passing-fight", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hovertemplate": "filters.0=%{x}
kernel_size.0=%{y}
units.0=%{z}
score=%{marker.color}", "legendgroup": "", "marker": { "color": [ 0.9850000143051147, 0.9868999719619751, 0.9864000082015991, 0.9855999946594238, 0.9864000082015991, 0.9872999787330627, 0.9868999719619751, 0.98580002784729, 0.9872999787330627, 0.9824000000953674, 0.9879999756813049, 0.9853000044822693, 0.988099992275238, 0.9879999756813049, 0.9878000020980835, 0.9871000051498413, 0.987500011920929, 0.9887999892234802, 0.9890999794006348, 0.9872000217437744, 0.9889000058174133, 0.9872999787330627, 0.9882000088691711, 0.9858999848365784, 0.9886999726295471, 0.9861999750137329, 0.9886000156402588, 0.9872000217437744, 0.9846000075340271, 0.9896000027656555, 0.9878000020980835, 0.9878000020980835, 0.989300012588501, 0.9879999756813049, 0.9884999990463257, 0.9872999787330627, 0.9868999719619751, 0.9896000027656555, 0.988099992275238, 0.9884999990463257, 0.9860000014305115, 0.9889000058174133, 0.9872999787330627, 0.98580002784729, 0.9865000247955322, 0.9871000051498413, 0.9865999817848206, 0.9873999953269958, 0.9882000088691711, 0.9872999787330627, 0.9894999861717224, 0.9882000088691711, 0.9886999726295471, 0.9855999946594238, 0.9865999817848206, 0.9848999977111816, 0.9868999719619751, 0.9868999719619751, 0.9872000217437744, 0.9869999885559082, 0.9883000254631042, 0.9884999990463257, 0.9866999983787537, 0.9883000254631042, 0.9890000224113464, 0.9883999824523926, 0.9896000027656555, 0.9884999990463257, 0.9896000027656555, 0.9896000027656555, 0.9886000156402588, 0.9825000166893005, 0.9871000051498413, 0.9840999841690063, 0.9878000020980835, 0.9868000149726868, 0.9635999798774719, 0.9868999719619751, 0.9866999983787537, 0.9811999797821045, 0.9869999885559082, 0.9878000020980835, 0.9866999983787537, 0.9861999750137329, 0.98580002784729, 0.9872999787330627, 0.9843000173568726, 0.9883999824523926, 0.9883999824523926, 0.9843000173568726, 0.9882000088691711, 0.9836000204086304, 0.9872999787330627, 0.9868000149726868, 0.9869999885559082, 0.9861999750137329, 0.9876999855041504, 0.9876000285148621, 0.9879999756813049, 0.9873999953269958 ], "coloraxis": "coloraxis", "symbol": "circle" }, "mode": "markers", "name": "", "scene": "scene", "showlegend": false, "type": "scatter3d", "x": [ 11, 15, 10, 12, 16, 10, 10, 10, 16, 16, 10, 16, 12, 10, 12, 10, 10, 11, 10, 10, 16, 16, 16, 13, 16, 16, 16, 16, 16, 16, 15, 16, 16, 16, 16, 16, 15, 16, 16, 16, 10, 16, 16, 16, 10, 16, 10, 16, 11, 10, 16, 16, 16, 16, 16, 16, 10, 12, 10, 16, 10, 15, 16, 10, 12, 12, 16, 13, 16, 16, 12, 5, 9, 5, 10, 10, 5, 5, 14, 14, 14, 14, 5, 14, 14, 14, 8, 10, 5, 5, 7, 5, 12, 10, 12, 14, 10, 14, 7, 11 ], "y": [ 5, 4, 6, 3, 3, 6, 3, 3, 6, 6, 6, 3, 4, 6, 6, 6, 6, 6, 6, 6, 3, 6, 3, 3, 3, 3, 6, 3, 6, 6, 6, 6, 6, 6, 6, 3, 6, 5, 6, 4, 3, 4, 5, 5, 3, 3, 3, 5, 6, 6, 6, 6, 6, 5, 6, 6, 6, 6, 6, 5, 6, 6, 4, 6, 6, 6, 6, 6, 6, 6, 4, 3, 4, 4, 4, 5, 3, 5, 3, 5, 5, 5, 5, 5, 3, 5, 3, 5, 5, 5, 5, 3, 5, 5, 5, 3, 5, 5, 5, 5 ], "z": [ 680, 700, 660, 790, 400, 790, 790, 400, 400, 790, 400, 790, 400, 400, 400, 550, 450, 420, 420, 430, 590, 580, 610, 590, 550, 570, 530, 530, 510, 550, 550, 540, 630, 650, 620, 640, 630, 550, 560, 550, 520, 600, 610, 540, 460, 600, 740, 560, 410, 480, 680, 670, 690, 680, 710, 450, 600, 420, 750, 630, 410, 680, 590, 510, 530, 520, 550, 540, 550, 550, 200, 120, 250, 100, 490, 270, 10, 490, 490, 10, 370, 230, 380, 170, 300, 430, 440, 210, 300, 240, 330, 320, 210, 330, 490, 210, 190, 260, 300, 230 ] } ], "layout": { "autosize": false, "coloraxis": { "colorbar": { "title": { "text": "score" } }, "colorscale": [ [ 0, "rgb(0,0,131)" ], [ 0.2, "rgb(0,60,170)" ], [ 0.4, "rgb(5,255,255)" ], [ 0.6, "rgb(255,255,0)" ], [ 0.8, "rgb(250,0,0)" ], [ 1, "rgb(128,0,0)" ] ] }, "height": 700, "legend": { "tracegroupgap": 0 }, "scene": { "domain": { "x": [ 0, 1 ], "y": [ 0, 1 ] }, "xaxis": { "title": { "text": "filters.0" } }, "yaxis": { "title": { "text": "kernel_size.0" } }, "zaxis": { "title": { "text": "units.0" } } }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "3D scatter plot" }, "width": 950 } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig1 = px.scatter_3d(search_data_2, \n", " x='filters.0', \n", " y='kernel_size.0', \n", " z='units.0', \n", " color='score', \n", " color_continuous_scale=color_scale,\n", " title=\"3D scatter plot\",\n", " )\n", "\n", "fig1.update_layout(\n", " autosize=False,\n", " width=950,\n", " height=700,)\n", "\n", "fig1.show()" ] }, { "cell_type": "markdown", "id": "diagnostic-manchester", "metadata": {}, "source": [ "\n", "\n", "Based on the plot above the neural network should have even more units in its dense layer, more filters and maybe we should explore the search space region with a kernel size of 7 or 8 (which is quite unusal for a convolutional layer). However I would like to continue with a more intriguing approach to deep learning optimization:" ] }, { "cell_type": "markdown", "id": "athletic-membership", "metadata": {}, "source": [ "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "homeless-marijuana", "metadata": {}, "source": [ "\n", " \n", "## Neural Architecture Search \n", "\n", "
\n", " \n", "Neural architecture search (nas) describes the optimization of a neural network by altering its structure, instead of just its hyperparameters. This means, that the number of layers and type of layers can vary during the optimization process. This is often difficult to achieve, because you have to somehow create a search space that contains the layers of a neural network. Hyperactive solves this by allowing the user to create search space dimensions that contain functions. The following example will show you how: " ] }, { "cell_type": "code", "execution_count": 24, "id": "novel-morris", "metadata": { "scrolled": true }, "outputs": [], "source": [ "def deep_learning_model(params):\n", " filters_0 = params[\"filters.0\"]\n", " kernel_size_0 = params[\"kernel_size.0\"]\n", " \n", " model = Sequential()\n", " model.add(Conv2D(filters_0, (kernel_size_0, kernel_size_0), input_shape=(img_width, img_height, 1), activation=\"relu\"))\n", " \n", " model.add(MaxPooling2D(pool_size=(2, 2)))\n", " \n", " # the next two lines are layers that are put in during the optimization run\n", " model = params[\"layer.0\"](params, model)\n", " model = params[\"layer.1\"](params, model)\n", "\n", " model.add(Flatten())\n", " model.add(Dense(params[\"dense.0\"], activation=\"relu\"))\n", " model.add(Dense(num_classes, activation=\"softmax\"))\n", "\n", " model.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n", " model.fit(\n", " x_train,\n", " y_train,\n", " validation_data=(x_test, y_test),\n", " epochs=10,\n", " verbose=verbose,\n", " )\n", " _, score = model.evaluate(x=x_test, y=y_test, verbose=verbose)\n", "\n", " return score\n", "\n", "\n", "# the next 4 functions are layers or layer-compositions:\n", "def Conv2D_MaxPooling2D_layer(params, model):\n", " filters_1 = params[\"layer.0.filters\"]\n", " kernel_size_1 = params[\"layer.0.kernel_size\"]\n", " model.add(Conv2D(filters_1, (kernel_size_1, kernel_size_1), activation='relu'))\n", " model.add(MaxPooling2D(pool_size=(2, 2)))\n", " return model\n", "\n", "def Conv2D_layer(params, model):\n", " filters_1 = params[\"layer.0.filters\"]\n", " kernel_size_1 = params[\"layer.0.kernel_size\"]\n", " model.add(Conv2D(filters_1, (kernel_size_1, kernel_size_1), activation='relu'))\n", " return model\n", "\n", "def Dropout_layer(params, model):\n", " model.add(Dropout(params[\"layer.1.rate\"]))\n", " return model\n", "\n", "def no_layer(params, model):\n", " return model\n", "\n", "\n", "# you can put the layers into lists like any other variable\n", "search_space_3 = {\n", " \"filters.0\": list(range(5, 15)),\n", " \"kernel_size.0\": list(range(3, 6)),\n", " \n", " \"layer.0\": [Conv2D_MaxPooling2D_layer, Conv2D_layer, no_layer],\n", " \"layer.0.filters\": list(range(5, 15)),\n", " \"layer.0.kernel_size\": list(range(3, 6)),\n", " \n", " \"layer.1\": [Dropout_layer, no_layer],\n", " \"layer.1.rate\": list(np.arange(0.1, 0.9, 0.1)),\n", "\n", " \"dense.0\": list(range(100, 600, 20)),\n", "}" ] }, { "cell_type": "code", "execution_count": 25, "id": "dying-tennis", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Results: 'deep_learning_model' \n", " Best score: 0.992900013923645 \n", " Best parameter:\n", " 'filters.0' : 11 \n", " 'kernel_size.0' : 5 \n", " 'layer.0' : \n", " 'layer.0.filters' : 14 \n", " 'layer.0.kernel_size' : 5 \n", " 'layer.1' : \n", " 'layer.1.rate' : 0.5 \n", " 'dense.0' : 540 \n", " \n", " Evaluation time : 2511.8917701244354 sec [98.58 %]\n", " Optimization time : 36.27178335189819 sec [1.42 %]\n", " Iteration time : 2548.1635534763336 sec [50.96 sec/iter]\n", " \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r" ] } ], "source": [ "optimizer = BayesianOptimizer()\n", "\n", "hyper_3 = Hyperactive()\n", "hyper_3.add_search(deep_learning_model, search_space_3, n_iter=50, optimizer=optimizer)\n", "hyper_3.run()" ] }, { "cell_type": "code", "execution_count": 26, "id": "duplicate-cradle", "metadata": {}, "outputs": [], "source": [ "search_data_3 = hyper_3.results(deep_learning_model)" ] }, { "cell_type": "code", "execution_count": 27, "id": "equivalent-pharmaceutical", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filters.0kernel_size.0layer.0layer.0.filterslayer.0.kernel_sizelayer.1layer.1.ratedense.0eval_timeiter_timescore
083no_layer115no_layer0.734055.32111955.3220730.9876
1135Conv2D_layer84no_layer0.338044.90404144.9050520.9902
294Conv2D_layer94Dropout_layer0.434045.92813545.9291410.9911
354no_layer63no_layer0.730041.35968241.3604310.9879
4143no_layer95no_layer0.350075.15665675.1572990.9849
\n", "
" ], "text/plain": [ " filters.0 kernel_size.0 layer.0 layer.0.filters \\\n", "0 8 3 no_layer 11 \n", "1 13 5 Conv2D_layer 8 \n", "2 9 4 Conv2D_layer 9 \n", "3 5 4 no_layer 6 \n", "4 14 3 no_layer 9 \n", "\n", " layer.0.kernel_size layer.1 layer.1.rate dense.0 eval_time \\\n", "0 5 no_layer 0.7 340 55.321119 \n", "1 4 no_layer 0.3 380 44.904041 \n", "2 4 Dropout_layer 0.4 340 45.928135 \n", "3 3 no_layer 0.7 300 41.359682 \n", "4 5 no_layer 0.3 500 75.156656 \n", "\n", " iter_time score \n", "0 55.322073 0.9876 \n", "1 44.905052 0.9902 \n", "2 45.929141 0.9911 \n", "3 41.360431 0.9879 \n", "4 75.157299 0.9849 " ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "search_data_3_str = search_data_3.copy()\n", "\n", "search_data_3_str[\"layer.0\"] = search_data_3[\"layer.0\"].apply(func2str)\n", "search_data_3_str[\"layer.1\"] = search_data_3[\"layer.1\"].apply(func2str)\n", "\n", "search_data_3_str_f = search_data_3_str[(np.abs(stats.zscore(search_data_3_str[\"score\"])) < 3)]\n", "parameter_names = list(search_space_3.keys()) # + [\"score_categorical\"]\n", "\n", "search_data_3_str_f.head()" ] }, { "cell_type": "code", "execution_count": 28, "id": "israeli-responsibility", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "dimensions": [ { "label": "filters.0", "values": [ 8, 13, 9, 5, 14, 9, 14, 14, 5, 5, 5, 12, 5, 12, 5, 5, 14, 5, 9, 9, 5, 9, 9, 5, 14, 10, 10, 8, 9, 8, 5, 10, 9, 9, 10, 11, 9, 12, 11, 12, 11, 12, 11, 11, 11, 12, 14, 14, 12 ] }, { "label": "kernel_size.0", "values": [ 3, 5, 4, 4, 3, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 3, 3, 5, 3, 5, 5, 5, 5, 5, 3, 5, 3, 5, 5, 5, 3, 3, 5, 5, 5, 3, 3, 5, 4, 5, 5, 3, 5, 3 ] }, { "label": "layer.0", "values": [ "no_layer", "Conv2D_layer", "Conv2D_layer", "no_layer", "no_layer", "Conv2D_layer", "no_layer", "Conv2D_MaxPooling2D_layer", "no_layer", "no_layer", "no_layer", "no_layer", "no_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "no_layer", "no_layer", "Conv2D_MaxPooling2D_layer", "no_layer", "no_layer", "Conv2D_MaxPooling2D_layer", "no_layer", "Conv2D_MaxPooling2D_layer", "no_layer", "no_layer", "no_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "no_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer", "Conv2D_MaxPooling2D_layer" ] }, { "label": "layer.0.filters", "values": [ 11, 8, 9, 6, 9, 13, 14, 5, 5, 14, 14, 14, 8, 14, 14, 14, 14, 14, 9, 10, 7, 14, 10, 14, 14, 14, 14, 10, 14, 14, 9, 14, 14, 14, 14, 14, 12, 14, 14, 14, 14, 14, 14, 14, 12, 12, 14, 14, 14 ] }, { "label": "layer.0.kernel_size", "values": [ 5, 4, 4, 3, 5, 5, 3, 5, 5, 3, 3, 3, 3, 3, 5, 3, 3, 3, 3, 3, 3, 5, 5, 3, 5, 5, 5, 5, 5, 5, 5, 3, 3, 5, 3, 3, 3, 3, 5, 5, 5, 5, 4, 4, 5, 3, 3, 3, 3 ] }, { "label": "layer.1", "values": [ "no_layer", "no_layer", "Dropout_layer", "no_layer", "no_layer", "Dropout_layer", "no_layer", "no_layer", "Dropout_layer", "no_layer", "no_layer", "no_layer", "no_layer", "Dropout_layer", "no_layer", "no_layer", "no_layer", "Dropout_layer", "no_layer", "no_layer", "no_layer", "Dropout_layer", "Dropout_layer", "Dropout_layer", "no_layer", "no_layer", "no_layer", "no_layer", "no_layer", "no_layer", "no_layer", "no_layer", "no_layer", "no_layer", "Dropout_layer", "no_layer", "no_layer", "Dropout_layer", "Dropout_layer", "Dropout_layer", "Dropout_layer", "Dropout_layer", "no_layer", "Dropout_layer", "Dropout_layer", "Dropout_layer", "Dropout_layer", "Dropout_layer", "Dropout_layer" ] }, { "label": "layer.1.rate", "values": [ 0.7000000000000001, 0.30000000000000004, 0.4, 0.7000000000000001, 0.30000000000000004, 0.8, 0.8, 0.8, 0.1, 0.1, 0.8, 0.6, 0.6, 0.1, 0.1, 0.1, 0.8, 0.8, 0.1, 0.1, 0.8, 0.4, 0.6, 0.1, 0.1, 0.1, 0.1, 0.30000000000000004, 0.5, 0.8, 0.8, 0.1, 0.4, 0.1, 0.6, 0.5, 0.6, 0.4, 0.5, 0.5, 0.5, 0.6, 0.6, 0.4, 0.4, 0.5, 0.4, 0.1, 0.30000000000000004 ] }, { "label": "dense.0", "values": [ 340, 380, 340, 300, 500, 520, 100, 580, 580, 420, 580, 400, 440, 580, 580, 260, 580, 340, 400, 580, 580, 580, 580, 580, 580, 340, 480, 380, 580, 580, 580, 380, 540, 580, 580, 580, 580, 540, 540, 580, 540, 580, 540, 560, 520, 580, 580, 480, 520 ] } ], "domain": { "x": [ 0, 1 ], "y": [ 0, 1 ] }, "line": { "color": [ 0.9876000285148621, 0.9901999831199646, 0.991100013256073, 0.9879000186920166, 0.9848999977111816, 0.9897000193595886, 0.9866999983787537, 0.9869999885559082, 0.9876000285148621, 0.9873999953269958, 0.988099992275238, 0.9832000136375427, 0.9864000082015991, 0.9901999831199646, 0.9889000058174133, 0.9873999953269958, 0.9869999885559082, 0.9828000068664551, 0.9884999990463257, 0.9879999756813049, 0.9884999990463257, 0.9898999929428101, 0.9890000224113464, 0.9854000210762024, 0.9869999885559082, 0.9901000261306763, 0.9901999831199646, 0.9883999824523926, 0.9908000230789185, 0.9898999929428101, 0.9873999953269958, 0.9868999719619751, 0.991100013256073, 0.9872000217437744, 0.991599977016449, 0.9919000267982483, 0.9898999929428101, 0.9922000169754028, 0.992900013923645, 0.9926000237464905, 0.9926999807357788, 0.9911999702453613, 0.9904000163078308, 0.9919000267982483, 0.9907000064849854, 0.9919000267982483, 0.9914000034332275, 0.9919999837875366, 0.9923999905586243 ], "coloraxis": "coloraxis" }, "name": "", "type": "parcats" } ], "layout": { "autosize": false, "coloraxis": { "colorbar": { "title": { "text": "score" } }, "colorscale": [ [ 0, "rgb(0,0,131)" ], [ 0.2, "rgb(0,60,170)" ], [ 0.4, "rgb(5,255,255)" ], [ 0.6, "rgb(255,255,0)" ], [ 0.8, "rgb(250,0,0)" ], [ 1, "rgb(128,0,0)" ] ] }, "height": 700, "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "width": 950 } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = px.parallel_categories(search_data_3_str_f, \n", " color=\"score\", \n", " color_continuous_scale=color_scale, \n", " dimensions=parameter_names, \n", " )\n", "fig.update_layout(\n", " autosize=False,\n", " width=950,\n", " height=700,)\n", "\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "lyric-network", "metadata": {}, "source": [ "\n", "\n", "In the image above you can see a parallel coordinates plot. It enables us to visualize multidimensional data very easily. Each of the bars represents one dimension. Each line has a specific path through those bars that you can see when hovering over it. Each path is one row from the search-dataframe. The color is dependend on the score and helps showing the parameter sets performing the best. There is a lot of information in this plot, so it is important to take your time studying it. \n", " \n", " " ] }, { "cell_type": "markdown", "id": "apart-recipe", "metadata": {}, "source": [ "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "horizontal-stick", "metadata": {}, "source": [ "\n", "\n", "## Pretrained Neural Architecture Search \n", "\n", "
\n", " \n", "Pretrained neural architecture search is a technique to reduce the computation time by using a pretrained neural network during the optimization process. It works by performing the following steps:\n", " \n", " - creating and training an entire neural network on the dataset\n", " - removing the layers of the neural network you want to optimize\n", " - making the remaining neural network layers not trainable\n", " - adding the new layers and parameters during the optimization run\n", " \n" ] }, { "cell_type": "code", "execution_count": 29, "id": "promotional-monitoring", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_163\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "conv2d_195 (Conv2D) (None, 25, 25, 15) 255 \n", "_________________________________________________________________\n", "max_pooling2d_192 (MaxPoolin (None, 12, 12, 15) 0 \n", "_________________________________________________________________\n", "flatten_163 (Flatten) (None, 2160) 0 \n", "_________________________________________________________________\n", "dense_326 (Dense) (None, 200) 432200 \n", "_________________________________________________________________\n", "dense_327 (Dense) (None, 10) 2010 \n", "=================================================================\n", "Total params: 434,465\n", "Trainable params: 434,465\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "# create model and train it\n", "model = Sequential()\n", "model.add(Conv2D(15, (4, 4), input_shape=(img_width, img_height, 1), activation=\"relu\"))\n", "model.add(MaxPooling2D(pool_size=(2, 2)))\n", "\n", "model.add(Flatten())\n", "model.add(Dense(200, activation=\"relu\"))\n", "model.add(Dense(num_classes, activation=\"softmax\"))\n", "\n", "model.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n", "model.fit(\n", " x_train,\n", " y_train,\n", " validation_data=(x_test, y_test),\n", " epochs=10,\n", " batch_size=1024,\n", " verbose=verbose,\n", ")\n", "\n", "model.summary()" ] }, { "cell_type": "markdown", "id": "frank-ceremony", "metadata": {}, "source": [ "\n", "\n", "The neural network above is our pretrained model. Now lets remove some of the layers and make the remaining layers not trainable." ] }, { "cell_type": "code", "execution_count": 30, "id": "disciplinary-indiana", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_163\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "conv2d_195 (Conv2D) (None, 25, 25, 15) 255 \n", "_________________________________________________________________\n", "max_pooling2d_192 (MaxPoolin (None, 12, 12, 15) 0 \n", "=================================================================\n", "Total params: 255\n", "Trainable params: 0\n", "Non-trainable params: 255\n", "_________________________________________________________________\n" ] } ], "source": [ "model_pretrained = model\n", "n_layers = len(model_pretrained.layers)\n", "\n", "# delete the last 3 layers\n", "for i in range(3):\n", " model_pretrained.pop()\n", "\n", "# set remaining layers to not-trainable\n", "for layer in model_pretrained.layers:\n", " layer.trainable = False\n", "\n", "model_pretrained.summary()" ] }, { "cell_type": "markdown", "id": "tested-zoning", "metadata": {}, "source": [ "\n", "\n", "The summary shows the new structure of the pretrained neural network. Only the first few layers are left, but this should already save some time." ] }, { "cell_type": "code", "execution_count": 31, "id": "finnish-magic", "metadata": {}, "outputs": [], "source": [ "def deep_learning_model(params):\n", " # it is important to make a copy here\n", " model = keras.models.clone_model(model_pretrained)\n", " \n", " model = params[\"layer.0\"](params, model)\n", " model = params[\"layer.1\"](params, model)\n", "\n", " model.add(Flatten())\n", " model.add(Dense(params[\"dense.0\"], activation=\"relu\"))\n", " model.add(Dense(num_classes, activation=\"softmax\"))\n", "\n", " model.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n", " model.fit(\n", " x_train,\n", " y_train,\n", " validation_data=(x_test, y_test),\n", " epochs=10,\n", " verbose=verbose,\n", " )\n", " _, score = model.evaluate(x=x_test, y=y_test, verbose=verbose)\n", " return score\n", "\n", "\n", "search_space_4 = { \n", " \"layer.0\": [Conv2D_MaxPooling2D_layer, Conv2D_layer, no_layer],\n", " \"layer.0.filters\": list(range(5, 12)),\n", " \"layer.0.kernel_size\": list(range(3, 6)),\n", " \n", " \"layer.1\": [Dropout_layer, no_layer],\n", " \"layer.1.rate\": list(np.arange(0.1, 0.9, 0.1)),\n", "\n", " \"dense.0\": list(range(100, 600, 20)),\n", "}" ] }, { "cell_type": "code", "execution_count": 32, "id": "light-portrait", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Results: 'deep_learning_model' \n", " Best score: 0.9923999905586243 \n", " Best parameter:\n", " 'layer.0' : \n", " 'layer.0.filters' : 11 \n", " 'layer.0.kernel_size' : 3 \n", " 'layer.1' : \n", " 'layer.1.rate' : 0.2 \n", " 'dense.0' : 560 \n", " \n", " Evaluation time : 1466.3828752040863 sec [99.83 %]\n", " Optimization time : 2.4778082370758057 sec [0.17 %]\n", " Iteration time : 1468.860683441162 sec [29.38 sec/iter]\n", " \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r" ] } ], "source": [ "optimizer = BayesianOptimizer()\n", "\n", "hyper_4 = Hyperactive()\n", "hyper_4.add_search(deep_learning_model, search_space_4, n_iter=50, optimizer=optimizer)\n", "hyper_4.run()" ] }, { "cell_type": "code", "execution_count": 33, "id": "unexpected-guatemala", "metadata": {}, "outputs": [], "source": [ "search_data_4 = hyper_4.results(deep_learning_model)" ] }, { "cell_type": "code", "execution_count": 34, "id": "personalized-festival", "metadata": {}, "outputs": [], "source": [ "search_data_3[\"optimization technique\"] = \"neural architecture search\"\n", "search_data_4[\"optimization technique\"] = \"pretrained neural architecture search\"\n", "\n", "search_data_merged = pd.concat([search_data_3, search_data_4], axis=0)\n", "\n", "search_data_merged = search_data_merged[search_data_merged[\"eval_time\"] > 1]\n", "search_data_merged = search_data_merged[(np.abs(stats.zscore(search_data_merged[\"score\"])) < 3)]" ] }, { "cell_type": "code", "execution_count": 35, "id": "hispanic-fountain", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hovertemplate": "optimization technique=neural architecture search
eval_time=%{x}
score=%{y}", "legendgroup": "neural architecture search", "marker": { "color": "#636efa", "symbol": "circle" }, "mode": "markers", "name": "neural architecture search", "orientation": "v", "showlegend": true, "type": "scatter", "x": [ 55.32111930847168, 44.904040813446045, 45.92813467979431, 41.359681606292725, 75.15665626525879, 52.095128297805786, 35.39965319633484, 43.485817193984985, 46.01356911659241, 42.30825233459473, 45.61400294303894, 49.51336908340454, 42.7988018989563, 47.46293568611145, 45.423229694366455, 39.46376585960388, 68.88382124900818, 49.42523670196533, 47.946072578430176, 56.43350911140442, 60.45997858047485, 68.49068760871887, 45.3531334400177, 60.62251377105713, 69.45072507858276, 47.78406572341919, 46.14115118980408, 39.64819955825806, 45.66249990463257, 58.66202664375305, 44.68727493286133, 52.88329720497131, 46.361032485961914, 44.90488529205322, 48.045682191848755, 55.47366261482239, 59.518582344055176, 44.344648599624634, 47.0784113407135, 43.79492926597595, 57.87875294685364, 58.48376965522766, 46.56584334373474, 40.34215426445007, 46.61242723464966, 44.161619424819946, 57.88534450531006, 44.87786412239075, 55.96027183532715 ], "xaxis": "x", "y": [ 0.9876000285148621, 0.9901999831199646, 0.991100013256073, 0.9879000186920166, 0.9848999977111816, 0.9897000193595886, 0.9866999983787537, 0.9869999885559082, 0.9876000285148621, 0.9873999953269958, 0.988099992275238, 0.9832000136375427, 0.9864000082015991, 0.9901999831199646, 0.9889000058174133, 0.9873999953269958, 0.9869999885559082, 0.9828000068664551, 0.9884999990463257, 0.9879999756813049, 0.9884999990463257, 0.9898999929428101, 0.9890000224113464, 0.9854000210762024, 0.9869999885559082, 0.9901000261306763, 0.9901999831199646, 0.9883999824523926, 0.9908000230789185, 0.9898999929428101, 0.9873999953269958, 0.9868999719619751, 0.991100013256073, 0.9872000217437744, 0.991599977016449, 0.9919000267982483, 0.9898999929428101, 0.9922000169754028, 0.992900013923645, 0.9926000237464905, 0.9926999807357788, 0.9911999702453613, 0.9904000163078308, 0.9919000267982483, 0.9907000064849854, 0.9919000267982483, 0.9914000034332275, 0.9919999837875366, 0.9923999905586243 ], "yaxis": "y" }, { "hovertemplate": "optimization technique=pretrained neural architecture search
eval_time=%{x}
score=%{y}", "legendgroup": "pretrained neural architecture search", "marker": { "color": "#EF553B", "symbol": "circle" }, "mode": "markers", "name": "pretrained neural architecture search", "orientation": "v", "showlegend": true, "type": "scatter", "x": [ 29.992302417755127, 23.943808555603027, 31.23397469520569, 32.98116612434387, 21.742459058761597, 32.04669451713562, 21.11833953857422, 29.36512064933777, 19.52925753593445, 32.566452503204346, 29.639880418777466, 33.623088359832764, 33.10226631164551, 30.99123740196228, 27.468989610671997, 33.004640102386475, 31.54503607749939, 30.472487688064575, 33.5914249420166, 27.884968042373657, 38.12792444229126, 27.498496055603027, 27.76973271369934, 32.28403639793396, 30.22796368598938, 30.392902374267578, 29.519784927368164, 29.076900720596313, 29.913541316986084, 27.209765195846558, 29.5145366191864, 32.07697343826294, 33.34197926521301, 32.212623834609985, 32.1302227973938, 29.62118673324585, 30.665143489837646, 29.568500995635986, 29.95565128326416, 27.090330839157104, 30.79551100730896, 28.96682620048523, 30.063374996185303, 32.58991241455078, 30.404109001159668, 32.247243881225586, 27.751819372177124, 30.024922609329224, 29.497297525405884 ], "xaxis": "x", "y": [ 0.9879999756813049, 0.9829999804496765, 0.9909999966621399, 0.9904999732971191, 0.982200026512146, 0.9840999841690063, 0.9771000146865845, 0.9861999750137329, 0.9835000038146973, 0.9896000027656555, 0.9890000224113464, 0.9897000193595886, 0.9919000267982483, 0.989799976348877, 0.9889000058174133, 0.9900000095367432, 0.9889000058174133, 0.9886000156402588, 0.9682999849319458, 0.9876000285148621, 0.9864000082015991, 0.9889000058174133, 0.9876999855041504, 0.9908999800682068, 0.9894999861717224, 0.9921000003814697, 0.9854999780654907, 0.9902999997138977, 0.9896000027656555, 0.9869999885559082, 0.9904999732971191, 0.9923999905586243, 0.9908999800682068, 0.9908999800682068, 0.9914000034332275, 0.9886999726295471, 0.9907000064849854, 0.9894000291824341, 0.9901999831199646, 0.989799976348877, 0.9911999702453613, 0.989799976348877, 0.9894999861717224, 0.9908999800682068, 0.9884999990463257, 0.9905999898910522, 0.9908999800682068, 0.9879999756813049, 0.9896000027656555 ], "yaxis": "y" } ], "layout": { "legend": { "title": { "text": "optimization technique" }, "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "eval_time" } }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "title": { "text": "score" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = px.scatter(search_data_merged, x=\"eval_time\", y=\"score\", color=\"optimization technique\")\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "olive-toyota", "metadata": {}, "source": [ "\n", "\n", "The 2D scatter plot above shows that the architecture search with the pretrained model was significantly faster than the regular optimization. We have a time save of about 15-20 seconds which would enable us to search more parameters in the same time. This would increase our chances to find even better parameter sets for the neural network.
\n", "The pretrained model achieved a lower score on average. Since transfer learning is a common practice in deep learning the results of the pretrained model are still relevant. Also the lower score of the pretrained neural architecture search does not affect the final model. After the final network-design is chosen you are free to retrain your entire network to get a better score. " ] }, { "cell_type": "markdown", "id": "apparent-lodge", "metadata": {}, "source": [ "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "funded-trailer", "metadata": {}, "source": [ "\n", " \n", "This notebook showed only some of the main features of Hyperactive. If you want to learn more about this project, suggest new features or report bugs you can do that in the official Github repository:\n", "#### [Hyperactive](https://github.com/SimonBlanke/Hyperactive)\n" ] }, { "cell_type": "markdown", "id": "banned-state", "metadata": {}, "source": [ "\n", "\n", "" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 5 }