{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Benchmarking" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 0. Setup the logging\n", "\n", "This step sets up logging in our environment to increase our visibility over\n", "the steps that Draco performs." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import logging;\n", "\n", "logging.basicConfig(level=logging.INFO)\n", "logging.getLogger().setLevel(level=logging.ERROR)\n", "logging.getLogger('draco').setLevel(level=logging.INFO)\n", "\n", "import warnings\n", "warnings.simplefilter(\"ignore\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Running the Benchmarking\n", "\n", "The user API for the Draco Benchmarking is the `draco.benchmark.evaluate_templates` function.\n", "\n", "The `evaluate_templates` function accepts the following arguments:\n", "* `templates (list)`: List of templates to try.\n", "* `window_size_rule (list)`: List of tupples (int, str or Timedelta object).\n", "* `metric (function or str)`: Metric to use. If an ``str`` is give it must be one of the metrics defined in the `draco.metrics.METRICS` dictionary.\n", "* `tuning_iterations (int)`: Number of iterations to be used.\n", "* `init_params (dict)`: Initialization parameters for the pipelines.\n", "* `target_times (DataFrame)`: Contains the specefication problem that we are solving, which has three columns:\n", " * `turbine_id`: Unique identifier of the turbine which this label corresponds to.\n", " * `cutoff_time`: Time associated with this target.\n", " * `target`: The value that we want to predict. This can either be a numerical value\n", " or a categorical label. This column can also be skipped when preparing\n", " data that will be used only to make predictions and not to fit any\n", " pipeline.\n", "* `readings (DataFrame)`: Contains the signal data from different sensors, with the following columns:\n", " * `turbine_id`: Unique identifier of the turbine which this reading comes from.\n", " * `signal_id`: Unique identifier of the signal which this reading comes from.\n", " * `timestamp (datetime)`: Time where the reading took place, as a datetime.\n", " * `value (float)`: Numeric value of this reading.\n", "* `preprocessing (int, list or dict)`: Number of preprocessing steps to be used.\n", "* `cost (bool)`: Wheter the metric is a cost function (the lower the better) or not.\n", "* `test_size (float)`: Percentage of the data set to be used for the test.\n", "* `cv_splits (int)`: Amount of splits to create.\n", "* `random_state (int)`: Random number of train_test split.\n", "* `output_path (str)`: Path where to save the benchmark report.\n", "* `cache_path (str)`: If given, cache the generated cross validation splits in this folder. Defatuls to ``None``." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "templates = [\n", " 'lstm_prob_with_unstack',\n", " 'double_lstm_prob_with_unstack'\n", "]\n", "window_size_rule = [('1d', '1h'), ('2d', '2h')]\n", "init_params = {\n", " 'lstm_prob_with_unstack': {\n", " 'keras.Sequential.LSTMTimeSeriesClassifier#1': {\n", " 'epochs': 1,\n", " }\n", " },\n", " 'double_lstm_prob_with_unstack': {\n", " 'keras.Sequential.DoubleLSTMTimeSeriesClassifier#1': {\n", " 'epochs': 1,\n", " }\n", " }\n", "}\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:draco.benchmark:Evaluating template lstm_prob_with_unstack on problem None (1d, 1h)\n", "2023-04-07 14:33:33.017625: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n", "2023-04-07 14:33:33.043631: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc3e937a8e0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n", "2023-04-07 14:33:33.043643: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version\n", "INFO:draco.pipeline:New configuration found:\n", " Template: lstm_prob_with_unstack \n", " Hyperparameters: \n", " ('sklearn.impute.SimpleImputer#1', 'strategy'): mean\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'lstm_1_units'): 80\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'dropout_1_rate'): 0.3\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'dense_1_units'): 80\n", "INFO:draco.benchmark:Evaluating template lstm_prob_with_unstack on problem None (2d, 2h)\n", "INFO:draco.pipeline:New configuration found:\n", " Template: lstm_prob_with_unstack \n", " Hyperparameters: \n", " ('sklearn.impute.SimpleImputer#1', 'strategy'): mean\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'lstm_1_units'): 80\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'dropout_1_rate'): 0.3\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'dense_1_units'): 80\n", "INFO:draco.pipeline:New configuration found:\n", " Template: lstm_prob_with_unstack \n", " Hyperparameters: \n", " ('sklearn.impute.SimpleImputer#1', 'strategy'): median\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'lstm_1_units'): 137\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'dropout_1_rate'): 0.612475373625103\n", " ('keras.Sequential.LSTMTimeSeriesClassifier#1', 'dense_1_units'): 191\n", "INFO:draco.benchmark:Evaluating template double_lstm_prob_with_unstack on problem None (1d, 1h)\n", "INFO:draco.pipeline:New configuration found:\n", " Template: double_lstm_prob_with_unstack \n", " Hyperparameters: \n", " ('sklearn.impute.SimpleImputer#1', 'strategy'): mean\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'lstm_1_units'): 80\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'dropout_1_rate'): 0.3\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'lstm_2_units'): 80\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'dropout_2_rate'): 0.3\n", "INFO:draco.pipeline:New configuration found:\n", " Template: double_lstm_prob_with_unstack \n", " Hyperparameters: \n", " ('sklearn.impute.SimpleImputer#1', 'strategy'): constant\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'lstm_1_units'): 245\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'dropout_1_rate'): 0.4308586778212253\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'lstm_2_units'): 221\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'dropout_2_rate'): 0.5926391753395145\n", "INFO:draco.benchmark:Evaluating template double_lstm_prob_with_unstack on problem None (2d, 2h)\n", "INFO:draco.pipeline:New configuration found:\n", " Template: double_lstm_prob_with_unstack \n", " Hyperparameters: \n", " ('sklearn.impute.SimpleImputer#1', 'strategy'): mean\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'lstm_1_units'): 80\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'dropout_1_rate'): 0.3\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'lstm_2_units'): 80\n", " ('keras.Sequential.DoubleLSTMTimeSeriesClassifier#1', 'dropout_2_rate'): 0.3\n" ] } ], "source": [ "from draco.benchmark import evaluate_templates\n", "\n", "results = evaluate_templates(\n", " templates=templates,\n", " window_size_rule=window_size_rule,\n", " init_params=init_params,\n", " tuning_iterations=3,\n", " cv_splits=3,\n", ")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
problem_namewindow_sizeresample_ruletemplatedefault_testdefault_cvtuned_cvtuned_testtuning_metrictuning_metric_kwargsfit_predict_timedefault_cv_timeaverage_cv_timetotal_timestatusaccuracy_threshold/0.5f1_threshold/0.5fpr_threshold/0.5tpr_threshold/0.5
0None1d1hlstm_prob_with_unstack0.4945050.5899050.5899050.322650roc_auc_score{'threshold': 0.5}0 days 00:00:03.8731570 days 00:00:14.3695360 days 00:00:08.1784220 days 00:00:47.144655OK0.2808990.2558141.00.0
1None2d2hlstm_prob_with_unstack0.4465810.5430560.5615700.707875roc_auc_score{'threshold': 0.5}0 days 00:00:03.4604670 days 00:00:12.1219050 days 00:00:08.2759190 days 00:00:44.449291OK0.7303370.5862071.00.0
2None1d1hdouble_lstm_prob_with_unstack0.8131870.3079930.5926960.417582roc_auc_score{'threshold': 0.5}0 days 00:00:05.4609850 days 00:00:18.1036600 days 00:00:14.0118770 days 00:01:11.192546OK0.3033710.3673471.00.0
3None2d2hdouble_lstm_prob_with_unstack0.2457260.6639190.6639190.293346roc_auc_score{'threshold': 0.5}0 days 00:00:05.5688350 days 00:00:17.9483610 days 00:00:14.0038160 days 00:01:11.051792OK0.3033710.1842111.00.0
\n", "
" ], "text/plain": [ " problem_name window_size resample_rule template \\\n", "0 None 1d 1h lstm_prob_with_unstack \n", "1 None 2d 2h lstm_prob_with_unstack \n", "2 None 1d 1h double_lstm_prob_with_unstack \n", "3 None 2d 2h double_lstm_prob_with_unstack \n", "\n", " default_test default_cv tuned_cv tuned_test tuning_metric \\\n", "0 0.494505 0.589905 0.589905 0.322650 roc_auc_score \n", "1 0.446581 0.543056 0.561570 0.707875 roc_auc_score \n", "2 0.813187 0.307993 0.592696 0.417582 roc_auc_score \n", "3 0.245726 0.663919 0.663919 0.293346 roc_auc_score \n", "\n", " tuning_metric_kwargs fit_predict_time default_cv_time \\\n", "0 {'threshold': 0.5} 0 days 00:00:03.873157 0 days 00:00:14.369536 \n", "1 {'threshold': 0.5} 0 days 00:00:03.460467 0 days 00:00:12.121905 \n", "2 {'threshold': 0.5} 0 days 00:00:05.460985 0 days 00:00:18.103660 \n", "3 {'threshold': 0.5} 0 days 00:00:05.568835 0 days 00:00:17.948361 \n", "\n", " average_cv_time total_time status \\\n", "0 0 days 00:00:08.178422 0 days 00:00:47.144655 OK \n", "1 0 days 00:00:08.275919 0 days 00:00:44.449291 OK \n", "2 0 days 00:00:14.011877 0 days 00:01:11.192546 OK \n", "3 0 days 00:00:14.003816 0 days 00:01:11.051792 OK \n", "\n", " accuracy_threshold/0.5 f1_threshold/0.5 fpr_threshold/0.5 \\\n", "0 0.280899 0.255814 1.0 \n", "1 0.730337 0.586207 1.0 \n", "2 0.303371 0.367347 1.0 \n", "3 0.303371 0.184211 1.0 \n", "\n", " tpr_threshold/0.5 \n", "0 0.0 \n", "1 0.0 \n", "2 0.0 \n", "3 0.0 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.16" } }, "nbformat": 4, "nbformat_minor": 2 }