{ "cells": [ { "cell_type": "markdown", "id": "01afaac8", "metadata": {}, "source": [ "# Using eval_metric in XGBoost \n", "\n", "This notebook shows how to use XGBoost's Evaluation Metric (`eval_metric`) with H2O. \n", "\n" ] }, { "cell_type": "markdown", "id": "fcd3edb3", "metadata": {}, "source": [ "### What is eval_metric?\n", "\n", "Original (non-H2O) XGBoost library allows users to define one or more evaluation metrics that will be calculated on both training and validations datasets after each iteration. If no evaluation metric is defined by the user XGBoost will assign it based on the choice of the objective function. For example in the case of binary classification, XGBoost will use logloss to report performance on training and validation data. Logloss measures the extent to which predicted probabilities diverge from the actual class labels. You might want to consider evaluating the performance using a different metric, e.g. for imbalanced problems, where it is more important to correctly predict the positive minority class, you might want to use the Area under the PR curve and specify `eval_metric=\"aucpr\"`. Evaluation metric can easily be used for early stopping." ] }, { "cell_type": "markdown", "id": "8b0e4ed4", "metadata": {}, "source": [ "### Is eval_metric needed with H2O XGBoost?\n", "\n", "H2O's approach to calculating metrics is different. By default, H2O will calculate all appropritate metrics for the given problem. For example for binary classification, H2O will report logloss, AUC, AUCPR and also additional metrics. Users thus typically don't need to worry about selecting the appropriate metric before model training. When early stopping is used, users will need to chose from one built-in early stopping metrics. For consistency between different model types and/or algorithm implementations, these are always calculated by H2O itself and, in XGBoost's specific case independect of XGBoost's eval_metric implementation." ] }, { "cell_type": "markdown", "id": "9e6a1a1a", "metadata": {}, "source": [ "### When should eval_metric be considered?\n", "\n", "While you typically don't need to specify your custom `eval_metric`, there are cases doing so would be beneficial.\n", "\n", "Case 1: H2O doesn't provide a suitable built-in metric. Example: If you want to calculate classification error for a different threshold than the one automatically determined by H2O, you can do so only by specifying `eval_metric=\"error@\"` because H2O currently doesn't have this capability.\n", "\n", "Case 2: Frequent scoring. By default H2O uses timing-triggered scoring, it is trying to make sure that majority of the time is used on model training as opposed to just model scoring. You can override this behavior and specify manually at what iterations you want model to be scored (eg. each iteration, every 5th iteration, ...). Because H2O calculates all possible metrics as opposed just few in native XGBoost, and needs to extract XGBoost model from the native memory, it can have a significant overhead when user desires to score very frequently. This can slow down the model building." ] }, { "cell_type": "markdown", "id": "09ce2240", "metadata": {}, "source": [ "### Example\n", "\n", "We will create a synthetic classification dataset, show XGBoost model training with default parameters and compare it to the output of model training with eval_metric used for early stopping." ] }, { "cell_type": "code", "execution_count": 1, "id": "f18c59e7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "versionFromGradle='3.39.0',projectVersion='3.39.0.99999',branch='michalk_eval-metric-ntb',lastCommitHash='256ed83c89220493cca0574ebd517eb09e3611fd',gitDescribe='jenkins-master-5998-dirty',compiledOn='2022-10-28 14:15:26',compiledBy='kurkami'\n", "Checking whether there is an H2O instance running at http://localhost:54321 . connected.\n" ] }, { "data": { "text/html": [ "\n", " \n", "
\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
H2O_cluster_uptime:5 hours 14 mins
H2O_cluster_timezone:America/New_York
H2O_data_parsing_timezone:UTC
H2O_cluster_version:3.39.0.99999
H2O_cluster_version_age:1 day
H2O_cluster_name:kurkami
H2O_cluster_total_nodes:1
H2O_cluster_free_memory:3.051 Gb
H2O_cluster_total_cores:8
H2O_cluster_allowed_cores:8
H2O_cluster_status:locked, healthy
H2O_connection_url:http://localhost:54321
H2O_connection_proxy:{\"http\": null, \"https\": null}
H2O_internal_security:False
Python_version:3.10.6 final
\n", "
\n" ], "text/plain": [ "-------------------------- -----------------------------\n", "H2O_cluster_uptime: 5 hours 14 mins\n", "H2O_cluster_timezone: America/New_York\n", "H2O_data_parsing_timezone: UTC\n", "H2O_cluster_version: 3.39.0.99999\n", "H2O_cluster_version_age: 1 day\n", "H2O_cluster_name: kurkami\n", "H2O_cluster_total_nodes: 1\n", "H2O_cluster_free_memory: 3.051 Gb\n", "H2O_cluster_total_cores: 8\n", "H2O_cluster_allowed_cores: 8\n", "H2O_cluster_status: locked, healthy\n", "H2O_connection_url: http://localhost:54321\n", "H2O_connection_proxy: {\"http\": null, \"https\": null}\n", "H2O_internal_security: False\n", "Python_version: 3.10.6 final\n", "-------------------------- -----------------------------" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import h2o\n", "h2o.init(strict_version_check=False)" ] }, { "cell_type": "code", "execution_count": 2, "id": "9f3da16b", "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import make_classification\n", "from sklearn.model_selection import train_test_split\n", "\n", "X, y = make_classification(n_samples=100000, \n", " n_informative=5,\n", " n_classes=2,\n", " random_state=42)\n", "X_train, X_validation, y_train, y_validation = train_test_split(X, y, test_size=0.25)" ] }, { "cell_type": "code", "execution_count": 3, "id": "172e6b47", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n", "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n", "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n", "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n" ] } ], "source": [ "from h2o import H2OFrame\n", "train = H2OFrame(y_train, column_names=[\"y\"]).cbind(H2OFrame(X_train))\n", "valid = H2OFrame(y_validation, column_names=[\"y\"]).cbind(H2OFrame(X_validation))\n", "train[\"y\"] = train[\"y\"].asfactor()\n", "valid[\"y\"] = valid[\"y\"].asfactor()" ] }, { "cell_type": "code", "execution_count": 4, "id": "d08b576f", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
y C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20
1-0.460911 0.587072 1.27653 1.65893 0.82732 0.900837 1.27825 -1.15342 -1.10419 0.231584 1.42246 0.632382 1.6532 0.0807537 0.63636 -0.0407703-0.712485-0.987504 0.529023 0.813985
0-0.122517 0.926967-3.27666 1.65648 1.26381 0.866601 -0.112373 4.88467 -2.56306 1.37596 1.66414 0.750439-0.712589 0.011264 0.338901-0.91639 -1.37998 -1.37817 -0.702391-0.222837
1 1.07287 1.35581 0.504084-1.9799 0.860703-0.753541 1.17781 0.364174 2.49498 -0.309902 0.52471 -0.128492 0.133623 1.77568 -1.70823 0.263603 -1.69966 0.77875 -0.578858 1.53087
1-0.497834 0.427476-1.67148 -1.71013 1.11473 0.0850045-2.81089 3.2892 0.676264 1.04835 1.46596 0.818714-1.03108 -1.65528 0.301107 1.19965 0.120588 0.820176-0.930966-0.82086
1 0.923038 1.51161 1.66059 1.45418 -2.3419 -0.757573 0.308842 -3.57087 -0.939606 0.369506 0.990806 0.67289 1.53918 1.79014 0.726961 0.0339329-0.542889-1.01856 1.97659 1.50693
0 0.575923 2.20724 0.595904-0.416675 0.21922 0.787082 -0.0753548-0.413841-0.280823 0.756811 -0.115625-1.03631 0.617746 1.15633 -1.3063 1.83909 0.610228 0.62274 -0.157638-1.19873
0-0.931317 3.38117 -1.25809 0.323422 4.46615 0.366347 -0.178002 4.59046 -0.995683 0.248196 -3.7437 1.22383 -1.00021 -0.754449 0.573945 1.94618 -2.16485 -0.285859-3.99979 -1.39355
0 0.956366 1.02616 -0.55144 -0.66649 2.47247 1.14188 1.10357 2.61941 1.2818 0.528047 -1.10656 -0.664716-0.849829 1.49331 -0.183118 0.618475 -0.413351 0.410573-2.14432 2.35245
0-0.0523249 1.73885 -1.47901 0.254941 -1.37106 -0.0549625-1.19023 0.848204-0.424188-0.0600663-0.781962 1.1958 -0.456581-0.740832 2.26734 2.01902 1.89101 0.348762 0.260157 0.133197
0-0.818169 -0.746954-1.28428 -0.0786964 0.708029-0.252285 -0.19201 1.33074 0.229397-0.599344 -0.141543-0.721745 0.442618 0.953017 1.00111 -0.862009 0.790705 1.56841 0.101824 1.48008 
[75000 rows x 21 columns]
" ], "text/plain": [ " y C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20\n", "--- ---------- --------- --------- ---------- --------- ---------- ---------- --------- --------- ---------- --------- --------- --------- ---------- --------- ---------- --------- --------- --------- ---------\n", " 1 -0.460911 0.587072 1.27653 1.65893 0.82732 0.900837 1.27825 -1.15342 -1.10419 0.231584 1.42246 0.632382 1.6532 0.0807537 0.63636 -0.0407703 -0.712485 -0.987504 0.529023 0.813985\n", " 0 -0.122517 0.926967 -3.27666 1.65648 1.26381 0.866601 -0.112373 4.88467 -2.56306 1.37596 1.66414 0.750439 -0.712589 0.011264 0.338901 -0.91639 -1.37998 -1.37817 -0.702391 -0.222837\n", " 1 1.07287 1.35581 0.504084 -1.9799 0.860703 -0.753541 1.17781 0.364174 2.49498 -0.309902 0.52471 -0.128492 0.133623 1.77568 -1.70823 0.263603 -1.69966 0.77875 -0.578858 1.53087\n", " 1 -0.497834 0.427476 -1.67148 -1.71013 1.11473 0.0850045 -2.81089 3.2892 0.676264 1.04835 1.46596 0.818714 -1.03108 -1.65528 0.301107 1.19965 0.120588 0.820176 -0.930966 -0.82086\n", " 1 0.923038 1.51161 1.66059 1.45418 -2.3419 -0.757573 0.308842 -3.57087 -0.939606 0.369506 0.990806 0.67289 1.53918 1.79014 0.726961 0.0339329 -0.542889 -1.01856 1.97659 1.50693\n", " 0 0.575923 2.20724 0.595904 -0.416675 0.21922 0.787082 -0.0753548 -0.413841 -0.280823 0.756811 -0.115625 -1.03631 0.617746 1.15633 -1.3063 1.83909 0.610228 0.62274 -0.157638 -1.19873\n", " 0 -0.931317 3.38117 -1.25809 0.323422 4.46615 0.366347 -0.178002 4.59046 -0.995683 0.248196 -3.7437 1.22383 -1.00021 -0.754449 0.573945 1.94618 -2.16485 -0.285859 -3.99979 -1.39355\n", " 0 0.956366 1.02616 -0.55144 -0.66649 2.47247 1.14188 1.10357 2.61941 1.2818 0.528047 -1.10656 -0.664716 -0.849829 1.49331 -0.183118 0.618475 -0.413351 0.410573 -2.14432 2.35245\n", " 0 -0.0523249 1.73885 -1.47901 0.254941 -1.37106 -0.0549625 -1.19023 0.848204 -0.424188 -0.0600663 -0.781962 1.1958 -0.456581 -0.740832 2.26734 2.01902 1.89101 0.348762 0.260157 0.133197\n", " 0 -0.818169 -0.746954 -1.28428 -0.0786964 0.708029 -0.252285 -0.19201 1.33074 0.229397 -0.599344 -0.141543 -0.721745 0.442618 0.953017 1.00111 -0.862009 0.790705 1.56841 0.101824 1.48008\n", "[75000 rows x 21 columns]\n" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train" ] }, { "cell_type": "markdown", "id": "24ddbce8", "metadata": {}, "source": [ "##### Train XGBoost model with logloss as stopping metric" ] }, { "cell_type": "markdown", "id": "f29a8bbe", "metadata": {}, "source": [ "We are specifying 1000 trees to be built and providing stopping criteria to make the model stop reasonably early for the purpose of this example." ] }, { "cell_type": "code", "execution_count": 5, "id": "525ad2d1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "xgboost Model Build progress: |██████████████████████████████████████████████████| (done) 100%\n" ] }, { "data": { "text/html": [ "
Model Details\n",
       "=============\n",
       "H2OXGBoostEstimator : XGBoost\n",
       "Model Key: XGBoost_model_python_1666962310297_518\n",
       "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", " \n", "\n", "
Model Summary:
number_of_trees
22.0
\n", "
\n", "
\n", "
ModelMetricsBinomial: xgboost\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.03475774769265435\n",
       "RMSE: 0.18643429859511995\n",
       "LogLoss: 0.12901406890397824\n",
       "Mean Per-Class Error: 0.04387881533186782\n",
       "AUC: 0.9893340459497513\n",
       "AUCPR: 0.988704085388505\n",
       "Gini: 0.9786680918995025
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.5101676438305829
01ErrorRate
035155.02174.00.0582 (2174.0/37329.0)
11112.036559.00.0295 (1112.0/37671.0)
Total36267.038733.00.0438 (3286.0/75000.0)
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Maximum Metrics: Maximum metrics at their respective thresholds
metricthresholdvalueidx
max f10.51016760.9569918214.0
max f20.30828060.9715081263.0
max f0point50.75326070.9592084147.0
max accuracy0.55617120.9564201.0
max precision0.99745651.00.0
max recall0.00160281.0399.0
max specificity0.99745651.00.0
max absolute_mcc0.55617120.9129277201.0
max min_per_class_accuracy0.62398530.9548073184.0
max mean_per_class_accuracy0.55617120.9563597201.0
max tns0.997456537329.00.0
max fns0.997456537460.00.0
max fps0.001602837329.0399.0
max tps0.001602837671.0399.0
max tnr0.99745651.00.0
max fnr0.99745650.99439890.0
max fpr0.00160281.0399.0
max tpr0.00160281.0399.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Gains/Lift Table: Avg response rate: 50.23 %, avg score: 50.22 %
groupcumulative_data_fractionlower_thresholdliftcumulative_liftresponse_ratescorecumulative_response_ratecumulative_scorecapture_ratecumulative_capture_rategaincumulative_gainkolmogorov_smirnov
10.01017330.99610971.99092141.99092141.00.99675161.00.99675160.02025430.020254399.092139899.09213980.0202543
20.02002670.99571531.99092141.99092141.00.99588941.00.99632740.01961720.039871599.092139899.09213980.0398715
30.030680.99539641.99092141.99092141.00.99554951.00.99605730.02120990.061081599.092139899.09213980.0610815
40.04010670.99519441.98810541.99025950.99858560.99527080.99966760.99587240.01874120.079822798.810538599.02595230.0797959
50.05177330.99492971.98637071.98938320.99771430.99500840.99922740.99567770.02317430.102997098.637072198.93832160.1029166
60.10001330.99330971.98817001.98879800.99861800.99416570.99893350.99494840.09590930.198906398.816998798.87980320.1986920
70.150080.99049841.98402871.98720700.99653790.99205770.99813430.99398410.09933370.298240098.402872898.72069930.2976775
80.20001330.98496051.98241551.98601080.99572760.98791250.99753350.99246830.09898860.397228698.241545998.60107860.3962375
90.300040.95610461.96730211.97977370.98813650.97403870.99440070.98632430.19678270.594011396.730209697.97737250.5906359
100.40001330.87706131.91232541.96291670.96052280.92100760.98593380.97000000.19118150.785192991.232540896.29167040.7738880
110.50001330.63834761.67635581.90560610.8420.80511860.95714780.93702460.16763560.952828467.635581790.56060550.9097788
120.60.07812450.42611401.65905690.21402850.28761850.83331110.82880470.04260570.9954342-57.388600665.90569230.7944912
130.70001330.01282220.02388791.42543460.01199840.03390010.71596730.71523390.00238910.9978233-97.611212842.54346480.5983483
140.800080.00579300.01220281.24868060.00612920.00841650.62718730.62683170.00122110.9990444-98.779715124.86805790.3997516
150.90.00344280.00451641.11055070.00226850.00452990.55780740.55774240.00045130.9994956-99.548363211.05507040.1999028
161.00.00016720.00504371.00.00253330.00262680.502280.50223080.00050441.0-99.49563320.00.0
\n", "
\n", "
\n", "
ModelMetricsBinomial: xgboost\n",
       "** Reported on validation data. **\n",
       "\n",
       "MSE: 0.038600168581279404\n",
       "RMSE: 0.1964692560714765\n",
       "LogLoss: 0.14264211397729937\n",
       "Mean Per-Class Error: 0.04874265748103474\n",
       "AUC: 0.9860367984758349\n",
       "AUCPR: 0.9841777019906762\n",
       "Gini: 0.9720735969516698
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.4899955771186135
01ErrorRate
011844.0840.00.0662 (840.0/12684.0)
1385.011931.00.0313 (385.0/12316.0)
Total12229.012771.00.049 (1225.0/25000.0)
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Maximum Metrics: Maximum metrics at their respective thresholds
metricthresholdvalueidx
max f10.48999560.9511699217.0
max f20.23816390.9684011281.0
max f0point50.76376010.9521032144.0
max accuracy0.48999560.951217.0
max precision0.99746831.00.0
max recall0.00183541.0398.0
max specificity0.99746831.00.0
max absolute_mcc0.48999560.9026291217.0
max min_per_class_accuracy0.63036380.9494966182.0
max mean_per_class_accuracy0.48999560.9512573217.0
max tns0.997468312684.00.0
max fns0.997468312244.00.0
max fps0.001343512684.0399.0
max tps0.001835412316.0398.0
max tnr0.99746831.00.0
max fnr0.99746830.99415390.0
max fpr0.00134351.0399.0
max tpr0.00183541.0398.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Gains/Lift Table: Avg response rate: 49.26 %, avg score: 49.47 %
groupcumulative_data_fractionlower_thresholdliftcumulative_liftresponse_ratescorecumulative_response_ratecumulative_scorecapture_ratecumulative_capture_rategaincumulative_gainkolmogorov_smirnov
10.010280.99612442.02987982.02987981.00.99673131.00.99673130.02086720.0208672102.9879831102.98798310.0208672
20.020320.99580122.02987982.02987981.00.99591751.00.99632920.02038000.0412472102.9879831102.98798310.0412472
30.030160.99546152.02162832.02718770.99593500.99561020.99867370.99609460.01989280.0611400102.1628287102.71876830.0610611
40.041080.99519442.02244442.02592680.99633700.99528870.99805260.99588040.02208510.0832251102.2444374102.59268030.0830674
50.052560.99492972.01573432.02370060.99303140.99500850.99695590.99569000.02314060.1063657101.5734327102.37005930.1060503
60.10.99328592.01961062.02176030.99494100.99415520.9960.99496190.09581030.2021760101.9610625102.17603120.2013876
70.150.99026942.02338422.02230160.99680.99194900.99626670.99395760.10116920.3033452102.3384216102.23016130.3022415
80.20.98406602.01364082.02013640.9920.98739450.99520.99231680.10068200.4040273101.3640792102.01364080.4021351
90.30.95238971.98522252.00849840.9780.97170510.98946670.98544620.19852220.602549598.5222475100.84984300.5963212
100.40.86710571.92351411.98725240.94760.91425670.9790.96764880.19235140.794900992.351412898.72523550.7783447
110.50.58223531.61740821.91328350.79680.78250590.942560.93062020.16174080.956641861.740824991.32835340.9000350
120.60.06677650.37755761.65732920.1860.24801800.81646670.81685320.03775580.9943975-62.244235165.73292190.7773524
130.70.01257710.01867491.42323570.00920.03044820.70114290.70450960.00186750.9962650-98.132510642.32357440.5839345
140.800480.00574120.01858571.24691750.00915610.00815740.61428140.61710020.00186750.9981325-98.141431724.69174880.3895705
150.90140.00345920.01045921.10848460.00515260.00452110.54608390.54851640.00105550.9991880-98.954084910.84846330.1927390
161.00.00026090.00823481.00.00405680.00264460.492640.49469340.00081201.0-99.17651930.00.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Scoring History:
timestampdurationnumber_of_treestraining_rmsetraining_loglosstraining_auctraining_pr_auctraining_lifttraining_classification_errorvalidation_rmsevalidation_loglossvalidation_aucvalidation_pr_aucvalidation_liftvalidation_classification_error
2022-10-28 14:19:27 0.031 sec0.00.50.69314720.50.50228001.00.497720.50.69314720.50.49264001.00.50736
2022-10-28 14:19:27 0.150 sec1.00.39960100.50845560.95996050.95288241.96994430.085080.40029390.50957660.95847540.95012122.00732560.08816
2022-10-28 14:19:27 0.213 sec2.00.33707780.40320480.96933580.96384321.97497810.07869330.33787260.40431480.96896420.96274522.01807820.08072
2022-10-28 14:19:27 0.272 sec3.00.29571350.33356510.97357640.96929711.98235830.071240.29728040.33556900.97275850.96726902.01648130.07328
2022-10-28 14:19:27 0.337 sec4.00.26905650.28670500.97693860.97366611.98244040.06877330.27119300.28943030.97536800.97091772.01661260.07028
2022-10-28 14:19:28 0.461 sec5.00.25297700.25489940.97823490.97539811.98300130.069360.25504370.25752650.97695490.97241972.02045310.06956
2022-10-28 14:19:28 0.553 sec6.00.24305500.23288590.97925360.97673331.98269450.068640.24525710.23572950.97780850.97375372.01988040.06872
2022-10-28 14:19:28 0.646 sec7.00.23667030.21730430.98010050.97795181.98401250.06553330.23889240.22020640.97879450.97510872.01890750.06696
2022-10-28 14:19:28 0.742 sec8.00.22742820.20046420.98168940.97958491.98834250.061720.23028640.20405440.98033300.97727142.02195060.06516
2022-10-28 14:19:28 0.831 sec9.00.22350480.19088780.98234440.98052191.98839480.06080.22675650.19510480.98081280.97790932.02227730.06432
------------------------------------------------
2022-10-28 14:19:28 1.227 sec13.00.20932990.16188170.98492080.98357531.98853420.05493330.21402170.16829750.98290890.98037032.02275740.0584
2022-10-28 14:19:28 1.321 sec14.00.20801160.15872260.98524520.98395381.98830860.05440.21296180.16569250.98313260.98069342.02227730.05852
2022-10-28 14:19:29 1.417 sec15.00.20656260.15561140.98565950.98469331.99092140.05385330.21188150.16339150.98333770.98101622.02278230.05792
2022-10-28 14:19:29 1.542 sec16.00.20360960.15179390.98634740.98538081.99092140.05202670.20991210.16077760.98370300.98148282.02176030.0566
2022-10-28 14:19:29 1.676 sec17.00.20012800.14708220.98688680.98591041.99092140.05037330.20687410.15655080.98420400.98215552.02987980.05524
2022-10-28 14:19:29 1.797 sec18.00.19523640.14096880.98776670.98683231.99092140.04790670.20297700.15166650.98486960.98297462.02987980.05272
2022-10-28 14:19:29 1.925 sec19.00.19436900.13917510.98800680.98712831.99092140.04766670.20250510.15058350.98497030.98292112.02987980.05232
2022-10-28 14:19:29 2.069 sec20.00.19194890.13612130.98840070.98757891.99092140.04669330.20091700.14841520.98529860.98348892.02987980.05108
2022-10-28 14:19:29 2.195 sec21.00.18862570.13193560.98896110.98821931.99092140.04445330.19831050.14489470.98575460.98383082.02176030.0496
2022-10-28 14:19:29 2.322 sec22.00.18643430.12901410.98933400.98870411.99092140.04381330.19646930.14264210.98603680.98417772.02987980.049
\n", "
\n", "
[23 rows x 16 columns]
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Variable Importances:
variablerelative_importancescaled_importancepercentage
C1170178.751.00.4261358
C235732.66406250.50916640.2169741
C1321608.48632810.30790640.1312100
C815654.54980470.22306680.0950568
C312402.26074220.17672390.0753084
C195739.76416020.08178780.0348527
C52835.91918950.04040990.0172201
C688.93524930.00126730.0005400
C1251.98764800.00074080.0003157
C1749.17110440.00070070.0002986
C748.05213170.00068470.0002918
C1446.54721830.00066330.0002826
C1043.87886050.00062520.0002664
C442.43465420.00060470.0002577
C1839.39529800.00056140.0002392
C132.55399320.00046390.0001977
C1525.95281600.00036980.0001576
C2023.11923220.00032940.0001404
C1622.99838070.00032770.0001396
C918.91131590.00026950.0001148
\n", "
\n", "
\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section.
" ], "text/plain": [ "Model Details\n", "=============\n", "H2OXGBoostEstimator : XGBoost\n", "Model Key: XGBoost_model_python_1666962310297_518\n", "\n", "\n", "Model Summary: \n", " number_of_trees\n", "-- -----------------\n", " 22\n", "\n", "ModelMetricsBinomial: xgboost\n", "** Reported on train data. **\n", "\n", "MSE: 0.03475774769265435\n", "RMSE: 0.18643429859511995\n", "LogLoss: 0.12901406890397824\n", "Mean Per-Class Error: 0.04387881533186782\n", "AUC: 0.9893340459497513\n", "AUCPR: 0.988704085388505\n", "Gini: 0.9786680918995025\n", "\n", "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.5101676438305829\n", " 0 1 Error Rate\n", "----- ----- ----- ------- ----------------\n", "0 35155 2174 0.0582 (2174.0/37329.0)\n", "1 1112 36559 0.0295 (1112.0/37671.0)\n", "Total 36267 38733 0.0438 (3286.0/75000.0)\n", "\n", "Maximum Metrics: Maximum metrics at their respective thresholds\n", "metric threshold value idx\n", "--------------------------- ----------- -------- -----\n", "max f1 0.510168 0.956992 214\n", "max f2 0.308281 0.971508 263\n", "max f0point5 0.753261 0.959208 147\n", "max accuracy 0.556171 0.9564 201\n", "max precision 0.997456 1 0\n", "max recall 0.00160282 1 399\n", "max specificity 0.997456 1 0\n", "max absolute_mcc 0.556171 0.912928 201\n", "max min_per_class_accuracy 0.623985 0.954807 184\n", "max mean_per_class_accuracy 0.556171 0.95636 201\n", "max tns 0.997456 37329 0\n", "max fns 0.997456 37460 0\n", "max fps 0.00160282 37329 399\n", "max tps 0.00160282 37671 399\n", "max tnr 0.997456 1 0\n", "max fnr 0.997456 0.994399 0\n", "max fpr 0.00160282 1 399\n", "max tpr 0.00160282 1 399\n", "\n", "Gains/Lift Table: Avg response rate: 50.23 %, avg score: 50.22 %\n", "group cumulative_data_fraction lower_threshold lift cumulative_lift response_rate score cumulative_response_rate cumulative_score capture_rate cumulative_capture_rate gain cumulative_gain kolmogorov_smirnov\n", "------- -------------------------- ----------------- ---------- ----------------- --------------- ---------- -------------------------- ------------------ -------------- ------------------------- -------- ----------------- --------------------\n", "1 0.0101733 0.99611 1.99092 1.99092 1 0.996752 1 0.996752 0.0202543 0.0202543 99.0921 99.0921 0.0202543\n", "2 0.0200267 0.995715 1.99092 1.99092 1 0.995889 1 0.996327 0.0196172 0.0398715 99.0921 99.0921 0.0398715\n", "3 0.03068 0.995396 1.99092 1.99092 1 0.995549 1 0.996057 0.0212099 0.0610815 99.0921 99.0921 0.0610815\n", "4 0.0401067 0.995194 1.98811 1.99026 0.998586 0.995271 0.999668 0.995872 0.0187412 0.0798227 98.8105 99.026 0.0797959\n", "5 0.0517733 0.99493 1.98637 1.98938 0.997714 0.995008 0.999227 0.995678 0.0231743 0.102997 98.6371 98.9383 0.102917\n", "6 0.100013 0.99331 1.98817 1.9888 0.998618 0.994166 0.998933 0.994948 0.0959093 0.198906 98.817 98.8798 0.198692\n", "7 0.15008 0.990498 1.98403 1.98721 0.996538 0.992058 0.998134 0.993984 0.0993337 0.29824 98.4029 98.7207 0.297677\n", "8 0.200013 0.98496 1.98242 1.98601 0.995728 0.987913 0.997533 0.992468 0.0989886 0.397229 98.2415 98.6011 0.396237\n", "9 0.30004 0.956105 1.9673 1.97977 0.988136 0.974039 0.994401 0.986324 0.196783 0.594011 96.7302 97.9774 0.590636\n", "10 0.400013 0.877061 1.91233 1.96292 0.960523 0.921008 0.985934 0.97 0.191182 0.785193 91.2325 96.2917 0.773888\n", "11 0.500013 0.638348 1.67636 1.90561 0.842 0.805119 0.957148 0.937025 0.167636 0.952828 67.6356 90.5606 0.909779\n", "12 0.6 0.0781245 0.426114 1.65906 0.214029 0.287618 0.833311 0.828805 0.0426057 0.995434 -57.3886 65.9057 0.794491\n", "13 0.700013 0.0128222 0.0238879 1.42543 0.0119984 0.0339001 0.715967 0.715234 0.00238911 0.997823 -97.6112 42.5435 0.598348\n", "14 0.80008 0.00579305 0.0122028 1.24868 0.00612925 0.00841651 0.627187 0.626832 0.0012211 0.999044 -98.7797 24.8681 0.399752\n", "15 0.9 0.00344276 0.00451637 1.11055 0.00226848 0.00452987 0.557807 0.557742 0.000451276 0.999496 -99.5484 11.0551 0.199903\n", "16 1 0.000167174 0.00504367 1 0.00253333 0.00262682 0.50228 0.502231 0.000504367 1 -99.4956 0 0\n", "\n", "ModelMetricsBinomial: xgboost\n", "** Reported on validation data. **\n", "\n", "MSE: 0.038600168581279404\n", "RMSE: 0.1964692560714765\n", "LogLoss: 0.14264211397729937\n", "Mean Per-Class Error: 0.04874265748103474\n", "AUC: 0.9860367984758349\n", "AUCPR: 0.9841777019906762\n", "Gini: 0.9720735969516698\n", "\n", "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.4899955771186135\n", " 0 1 Error Rate\n", "----- ----- ----- ------- ----------------\n", "0 11844 840 0.0662 (840.0/12684.0)\n", "1 385 11931 0.0313 (385.0/12316.0)\n", "Total 12229 12771 0.049 (1225.0/25000.0)\n", "\n", "Maximum Metrics: Maximum metrics at their respective thresholds\n", "metric threshold value idx\n", "--------------------------- ----------- -------- -----\n", "max f1 0.489996 0.95117 217\n", "max f2 0.238164 0.968401 281\n", "max f0point5 0.76376 0.952103 144\n", "max accuracy 0.489996 0.951 217\n", "max precision 0.997468 1 0\n", "max recall 0.00183535 1 398\n", "max specificity 0.997468 1 0\n", "max absolute_mcc 0.489996 0.902629 217\n", "max min_per_class_accuracy 0.630364 0.949497 182\n", "max mean_per_class_accuracy 0.489996 0.951257 217\n", "max tns 0.997468 12684 0\n", "max fns 0.997468 12244 0\n", "max fps 0.00134346 12684 399\n", "max tps 0.00183535 12316 398\n", "max tnr 0.997468 1 0\n", "max fnr 0.997468 0.994154 0\n", "max fpr 0.00134346 1 399\n", "max tpr 0.00183535 1 398\n", "\n", "Gains/Lift Table: Avg response rate: 49.26 %, avg score: 49.47 %\n", "group cumulative_data_fraction lower_threshold lift cumulative_lift response_rate score cumulative_response_rate cumulative_score capture_rate cumulative_capture_rate gain cumulative_gain kolmogorov_smirnov\n", "------- -------------------------- ----------------- ---------- ----------------- --------------- ---------- -------------------------- ------------------ -------------- ------------------------- -------- ----------------- --------------------\n", "1 0.01028 0.996124 2.02988 2.02988 1 0.996731 1 0.996731 0.0208672 0.0208672 102.988 102.988 0.0208672\n", "2 0.02032 0.995801 2.02988 2.02988 1 0.995918 1 0.996329 0.02038 0.0412472 102.988 102.988 0.0412472\n", "3 0.03016 0.995462 2.02163 2.02719 0.995935 0.99561 0.998674 0.996095 0.0198928 0.06114 102.163 102.719 0.0610611\n", "4 0.04108 0.995194 2.02244 2.02593 0.996337 0.995289 0.998053 0.99588 0.0220851 0.0832251 102.244 102.593 0.0830674\n", "5 0.05256 0.99493 2.01573 2.0237 0.993031 0.995009 0.996956 0.99569 0.0231406 0.106366 101.573 102.37 0.10605\n", "6 0.1 0.993286 2.01961 2.02176 0.994941 0.994155 0.996 0.994962 0.0958103 0.202176 101.961 102.176 0.201388\n", "7 0.15 0.990269 2.02338 2.0223 0.9968 0.991949 0.996267 0.993958 0.101169 0.303345 102.338 102.23 0.302241\n", "8 0.2 0.984066 2.01364 2.02014 0.992 0.987394 0.9952 0.992317 0.100682 0.404027 101.364 102.014 0.402135\n", "9 0.3 0.95239 1.98522 2.0085 0.978 0.971705 0.989467 0.985446 0.198522 0.60255 98.5222 100.85 0.596321\n", "10 0.4 0.867106 1.92351 1.98725 0.9476 0.914257 0.979 0.967649 0.192351 0.794901 92.3514 98.7252 0.778345\n", "11 0.5 0.582235 1.61741 1.91328 0.7968 0.782506 0.94256 0.93062 0.161741 0.956642 61.7408 91.3284 0.900035\n", "12 0.6 0.0667765 0.377558 1.65733 0.186 0.248018 0.816467 0.816853 0.0377558 0.994398 -62.2442 65.7329 0.777352\n", "13 0.7 0.0125771 0.0186749 1.42324 0.0092 0.0304482 0.701143 0.70451 0.00186749 0.996265 -98.1325 42.3236 0.583935\n", "14 0.80048 0.00574124 0.0185857 1.24692 0.00915605 0.00815736 0.614281 0.6171 0.00186749 0.998133 -98.1414 24.6917 0.389571\n", "15 0.9014 0.00345923 0.0104592 1.10848 0.0051526 0.0045211 0.546084 0.548516 0.00105554 0.999188 -98.9541 10.8485 0.192739\n", "16 1 0.000260908 0.00823481 1 0.0040568 0.00264459 0.49264 0.494693 0.000811952 1 -99.1765 0 0\n", "\n", "Scoring History: \n", " timestamp duration number_of_trees training_rmse training_logloss training_auc training_pr_auc training_lift training_classification_error validation_rmse validation_logloss validation_auc validation_pr_auc validation_lift validation_classification_error\n", "--- ------------------- ---------- ----------------- ------------------- ------------------- ------------------ ------------------ ------------------ ------------------------------- ------------------- -------------------- ------------------ ------------------- ------------------ ---------------------------------\n", " 2022-10-28 14:19:27 0.031 sec 0.0 0.5 0.6931471805600504 0.5 0.5022800000000001 1.0 0.49772 0.5 0.6931471805599932 0.5 0.49263999999999997 1.0 0.50736\n", " 2022-10-28 14:19:27 0.150 sec 1.0 0.39960097366259284 0.5084555759713797 0.9599604712562773 0.952882361784991 1.969944257252065 0.08508 0.4002938936840949 0.5095766191321826 0.9584754441256724 0.9501212341159861 2.007325610768287 0.08816\n", " 2022-10-28 14:19:27 0.213 sec 2.0 0.3370778235169717 0.40320483936633433 0.9693357563355386 0.9638432211755918 1.974978083921303 0.07869333333333334 0.3378725612813438 0.4043147576521532 0.9689642384208382 0.9627452021209266 2.0180782041889165 0.08072\n", " 2022-10-28 14:19:27 0.272 sec 3.0 0.2957135469964664 0.33356511573213676 0.9735764390034864 0.9692971186564519 1.9823582956342736 0.07124 0.2972804020126436 0.33556896107331324 0.972758545365196 0.9672690232547145 2.0164812843739717 0.07328\n", " 2022-10-28 14:19:27 0.337 sec 4.0 0.2690565369849074 0.2867049786815229 0.9769385896258127 0.973666085141308 1.9824403701018989 0.06877333333333334 0.2711929535273588 0.2894303318159203 0.9753679683707979 0.9709177118281687 2.0166126426753443 0.07028\n", " 2022-10-28 14:19:28 0.461 sec 5.0 0.2529770080480673 0.25489941558805895 0.9782348903583494 0.9753981493946517 1.9830012735164846 0.06936 0.2550436637594876 0.25752647933993555 0.976954942633842 0.9724197019498086 2.0204531445917966 0.06956\n", " 2022-10-28 14:19:28 0.553 sec 6.0 0.24305497030792145 0.23288588902305773 0.9792535639846858 0.9767332522440302 1.982694450495822 0.06864 0.24525713259217433 0.23572949157271123 0.9778084651737403 0.9737536794534183 2.0198804230789538 0.06872\n", " 2022-10-28 14:19:28 0.646 sec 7.0 0.23667027877522748 0.21730431422414767 0.9801004932398384 0.9779517651384936 1.9840124751203916 0.06553333333333333 0.2388923863271564 0.22020638093402833 0.978794451615705 0.9751086853604877 2.018907507702571 0.06696\n", " 2022-10-28 14:19:28 0.742 sec 8.0 0.22742820294228833 0.2004641682608729 0.981689446457674 0.979584885842671 1.988342484694663 0.06172 0.23028641120487947 0.20405435411113004 0.9803330089878547 0.9772714003156485 2.021950613023709 0.06516\n", " 2022-10-28 14:19:28 0.831 sec 9.0 0.22350478838486162 0.1908877726426607 0.9823443926985863 0.9805219447464963 1.9883948484251914 0.0608 0.22675651465031937 0.19510476775550073 0.9808127993480623 0.9779093493480279 2.0222772849300505 0.06432\n", "--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---\n", " 2022-10-28 14:19:28 1.227 sec 13.0 0.20932986426153485 0.16188173031435207 0.9849208281386209 0.9835752931336927 1.9885342025018198 0.054933333333333334 0.2140216611245386 0.16829754841446237 0.9829088919260484 0.9803702989731918 2.022757445741668 0.0584\n", " 2022-10-28 14:19:28 1.321 sec 14.0 0.20801159542673212 0.15872260659114573 0.9852451840387032 0.9839537573047635 1.9883086406824775 0.0544 0.2129618210780405 0.16569253851178234 0.9831326172024832 0.9806933766171955 2.0222772849300505 0.05852\n", " 2022-10-28 14:19:29 1.417 sec 15.0 0.20656261820821578 0.15561137441472306 0.9856594600307703 0.9846933313933812 1.9909213984231904 0.053853333333333336 0.2118814635733488 0.16339145578701472 0.9833376856363834 0.9810162497973599 2.022782349187026 0.05792\n", " 2022-10-28 14:19:29 1.542 sec 16.0 0.2036095613587592 0.1517938621462097 0.9863473648236762 0.9853807508666774 1.9909213984231904 0.052026666666666666 0.20991212994517833 0.1607776039482004 0.9837030416011293 0.9814827764043785 2.0217603117895417 0.0566\n", " 2022-10-28 14:19:29 1.676 sec 17.0 0.20012801070573083 0.1470822013139249 0.986886830974453 0.985910377892329 1.9909213984231904 0.05037333333333333 0.2068740603495647 0.1565507907729844 0.9842040109503664 0.9821554851954529 2.029879831113998 0.05524\n", " 2022-10-28 14:19:29 1.797 sec 18.0 0.1952363704640891 0.14096879922102418 0.9877666946033187 0.9868323340675447 1.9909213984231904 0.04790666666666667 0.20297695900076984 0.15166647071587894 0.9848695695625415 0.9829746186958338 2.029879831113998 0.05272\n", " 2022-10-28 14:19:29 1.925 sec 19.0 0.19436898343164097 0.1391751330056711 0.9880068361300588 0.9871283090191609 1.9909213984231904 0.04766666666666667 0.2025051413752741 0.15058346705393647 0.9849703081904262 0.9829211071563434 2.029879831113998 0.05232\n", " 2022-10-28 14:19:29 2.069 sec 20.0 0.19194891175623852 0.13612128709347052 0.988400723075942 0.9875788628040455 1.9909213984231904 0.04669333333333334 0.2009170028210971 0.14841515569437927 0.9852986065255842 0.9834889458246532 2.029879831113998 0.05108\n", " 2022-10-28 14:19:29 2.195 sec 21.0 0.18862571988282295 0.1319356174549523 0.9889610721498387 0.9882193402029281 1.9909213984231904 0.04445333333333333 0.1983104566213673 0.1448947109368754 0.9857545901273815 0.9838307814948767 2.0217603117895417 0.0496\n", " 2022-10-28 14:19:29 2.322 sec 22.0 0.18643429859511995 0.12901406890397824 0.9893340459497513 0.988704085388505 1.9909213984231904 0.043813333333333336 0.1964692560714765 0.14264211397729937 0.9860367984758349 0.9841777019906762 2.029879831113998 0.049\n", "[23 rows x 16 columns]\n", "\n", "\n", "Variable Importances: \n", "variable relative_importance scaled_importance percentage\n", "---------- --------------------- ------------------- ------------\n", "C11 70178.8 1 0.426136\n", "C2 35732.7 0.509166 0.216974\n", "C13 21608.5 0.307906 0.13121\n", "C8 15654.5 0.223067 0.0950568\n", "C3 12402.3 0.176724 0.0753084\n", "C19 5739.76 0.0817878 0.0348527\n", "C5 2835.92 0.0404099 0.0172201\n", "C6 88.9352 0.00126727 0.000540028\n", "C12 51.9876 0.000740789 0.000315677\n", "C17 49.1711 0.000700655 0.000298574\n", "C7 48.0521 0.000684711 0.00029178\n", "C14 46.5472 0.000663267 0.000282642\n", "C10 43.8789 0.000625244 0.000266439\n", "C4 42.4347 0.000604665 0.00025767\n", "C18 39.3953 0.000561357 0.000239214\n", "C1 32.554 0.000463873 0.000197673\n", "C15 25.9528 0.00036981 0.000157589\n", "C20 23.1192 0.000329434 0.000140383\n", "C16 22.9984 0.000327711 0.00013965\n", "C9 18.9113 0.000269474 0.000114832\n", "\n", "[tips]\n", "Use `model.explain()` to inspect the model.\n", "--\n", "Use `h2o.display.toggle_user_tips()` to switch on/off this section." ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from h2o.estimators.xgboost import H2OXGBoostEstimator\n", "model_def = H2OXGBoostEstimator(ntrees=1000, max_depth=6, score_each_iteration=True, \n", " stopping_rounds=3, stopping_tolerance=0.05, stopping_metric=\"logloss\")\n", "model_def.train(y=\"y\", training_frame=train, validation_frame=valid)" ] }, { "cell_type": "markdown", "id": "7e5c5e83", "metadata": {}, "source": [ "Scoring history will show us some of the metrics that were calculated on training and validation datasets. In our case, `validation_logloss` was used as the metric for early stopping." ] }, { "cell_type": "code", "execution_count": 6, "id": "798f28fd", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampdurationnumber_of_treestraining_rmsetraining_loglosstraining_auctraining_pr_auctraining_lifttraining_classification_errorvalidation_rmsevalidation_loglossvalidation_aucvalidation_pr_aucvalidation_liftvalidation_classification_error
02022-10-28 14:19:270.031 sec0.00.5000000.6931470.5000000.5022801.0000000.4977200.5000000.6931470.5000000.4926401.0000000.50736
12022-10-28 14:19:270.150 sec1.00.3996010.5084560.9599600.9528821.9699440.0850800.4002940.5095770.9584750.9501212.0073260.08816
22022-10-28 14:19:270.213 sec2.00.3370780.4032050.9693360.9638431.9749780.0786930.3378730.4043150.9689640.9627452.0180780.08072
32022-10-28 14:19:270.272 sec3.00.2957140.3335650.9735760.9692971.9823580.0712400.2972800.3355690.9727590.9672692.0164810.07328
42022-10-28 14:19:270.337 sec4.00.2690570.2867050.9769390.9736661.9824400.0687730.2711930.2894300.9753680.9709182.0166130.07028
52022-10-28 14:19:280.461 sec5.00.2529770.2548990.9782350.9753981.9830010.0693600.2550440.2575260.9769550.9724202.0204530.06956
62022-10-28 14:19:280.553 sec6.00.2430550.2328860.9792540.9767331.9826940.0686400.2452570.2357290.9778080.9737542.0198800.06872
72022-10-28 14:19:280.646 sec7.00.2366700.2173040.9801000.9779521.9840120.0655330.2388920.2202060.9787940.9751092.0189080.06696
82022-10-28 14:19:280.742 sec8.00.2274280.2004640.9816890.9795851.9883420.0617200.2302860.2040540.9803330.9772712.0219510.06516
92022-10-28 14:19:280.831 sec9.00.2235050.1908880.9823440.9805221.9883950.0608000.2267570.1951050.9808130.9779092.0222770.06432
102022-10-28 14:19:280.940 sec10.00.2200400.1829670.9830000.9812101.9882980.0592270.2236560.1877050.9813560.9786272.0218880.06240
112022-10-28 14:19:281.039 sec11.00.2149260.1734630.9838780.9822241.9888190.0570270.2188990.1787480.9821480.9794342.0236150.06084
122022-10-28 14:19:281.135 sec12.00.2120740.1672610.9843480.9827911.9886170.0559870.2162370.1729570.9824910.9797112.0229750.05932
132022-10-28 14:19:281.227 sec13.00.2093300.1618820.9849210.9835751.9885340.0549330.2140220.1682980.9829090.9803702.0227570.05840
142022-10-28 14:19:281.321 sec14.00.2080120.1587230.9852450.9839541.9883090.0544000.2129620.1656930.9831330.9806932.0222770.05852
152022-10-28 14:19:291.417 sec15.00.2065630.1556110.9856590.9846931.9909210.0538530.2118810.1633910.9833380.9810162.0227820.05792
162022-10-28 14:19:291.542 sec16.00.2036100.1517940.9863470.9853811.9909210.0520270.2099120.1607780.9837030.9814832.0217600.05660
172022-10-28 14:19:291.676 sec17.00.2001280.1470820.9868870.9859101.9909210.0503730.2068740.1565510.9842040.9821552.0298800.05524
182022-10-28 14:19:291.797 sec18.00.1952360.1409690.9877670.9868321.9909210.0479070.2029770.1516660.9848700.9829752.0298800.05272
192022-10-28 14:19:291.925 sec19.00.1943690.1391750.9880070.9871281.9909210.0476670.2025050.1505830.9849700.9829212.0298800.05232
202022-10-28 14:19:292.069 sec20.00.1919490.1361210.9884010.9875791.9909210.0466930.2009170.1484150.9852990.9834892.0298800.05108
212022-10-28 14:19:292.195 sec21.00.1886260.1319360.9889610.9882191.9909210.0444530.1983100.1448950.9857550.9838312.0217600.04960
222022-10-28 14:19:292.322 sec22.00.1864340.1290140.9893340.9887041.9909210.0438130.1964690.1426420.9860370.9841782.0298800.04900
\n", "
" ], "text/plain": [ " timestamp duration number_of_trees training_rmse \\\n", "0 2022-10-28 14:19:27 0.031 sec 0.0 0.500000 \n", "1 2022-10-28 14:19:27 0.150 sec 1.0 0.399601 \n", "2 2022-10-28 14:19:27 0.213 sec 2.0 0.337078 \n", "3 2022-10-28 14:19:27 0.272 sec 3.0 0.295714 \n", "4 2022-10-28 14:19:27 0.337 sec 4.0 0.269057 \n", "5 2022-10-28 14:19:28 0.461 sec 5.0 0.252977 \n", "6 2022-10-28 14:19:28 0.553 sec 6.0 0.243055 \n", "7 2022-10-28 14:19:28 0.646 sec 7.0 0.236670 \n", "8 2022-10-28 14:19:28 0.742 sec 8.0 0.227428 \n", "9 2022-10-28 14:19:28 0.831 sec 9.0 0.223505 \n", "10 2022-10-28 14:19:28 0.940 sec 10.0 0.220040 \n", "11 2022-10-28 14:19:28 1.039 sec 11.0 0.214926 \n", "12 2022-10-28 14:19:28 1.135 sec 12.0 0.212074 \n", "13 2022-10-28 14:19:28 1.227 sec 13.0 0.209330 \n", "14 2022-10-28 14:19:28 1.321 sec 14.0 0.208012 \n", "15 2022-10-28 14:19:29 1.417 sec 15.0 0.206563 \n", "16 2022-10-28 14:19:29 1.542 sec 16.0 0.203610 \n", "17 2022-10-28 14:19:29 1.676 sec 17.0 0.200128 \n", "18 2022-10-28 14:19:29 1.797 sec 18.0 0.195236 \n", "19 2022-10-28 14:19:29 1.925 sec 19.0 0.194369 \n", "20 2022-10-28 14:19:29 2.069 sec 20.0 0.191949 \n", "21 2022-10-28 14:19:29 2.195 sec 21.0 0.188626 \n", "22 2022-10-28 14:19:29 2.322 sec 22.0 0.186434 \n", "\n", " training_logloss training_auc training_pr_auc training_lift \\\n", "0 0.693147 0.500000 0.502280 1.000000 \n", "1 0.508456 0.959960 0.952882 1.969944 \n", "2 0.403205 0.969336 0.963843 1.974978 \n", "3 0.333565 0.973576 0.969297 1.982358 \n", "4 0.286705 0.976939 0.973666 1.982440 \n", "5 0.254899 0.978235 0.975398 1.983001 \n", "6 0.232886 0.979254 0.976733 1.982694 \n", "7 0.217304 0.980100 0.977952 1.984012 \n", "8 0.200464 0.981689 0.979585 1.988342 \n", "9 0.190888 0.982344 0.980522 1.988395 \n", "10 0.182967 0.983000 0.981210 1.988298 \n", "11 0.173463 0.983878 0.982224 1.988819 \n", "12 0.167261 0.984348 0.982791 1.988617 \n", "13 0.161882 0.984921 0.983575 1.988534 \n", "14 0.158723 0.985245 0.983954 1.988309 \n", "15 0.155611 0.985659 0.984693 1.990921 \n", "16 0.151794 0.986347 0.985381 1.990921 \n", "17 0.147082 0.986887 0.985910 1.990921 \n", "18 0.140969 0.987767 0.986832 1.990921 \n", "19 0.139175 0.988007 0.987128 1.990921 \n", "20 0.136121 0.988401 0.987579 1.990921 \n", "21 0.131936 0.988961 0.988219 1.990921 \n", "22 0.129014 0.989334 0.988704 1.990921 \n", "\n", " training_classification_error validation_rmse validation_logloss \\\n", "0 0.497720 0.500000 0.693147 \n", "1 0.085080 0.400294 0.509577 \n", "2 0.078693 0.337873 0.404315 \n", "3 0.071240 0.297280 0.335569 \n", "4 0.068773 0.271193 0.289430 \n", "5 0.069360 0.255044 0.257526 \n", "6 0.068640 0.245257 0.235729 \n", "7 0.065533 0.238892 0.220206 \n", "8 0.061720 0.230286 0.204054 \n", "9 0.060800 0.226757 0.195105 \n", "10 0.059227 0.223656 0.187705 \n", "11 0.057027 0.218899 0.178748 \n", "12 0.055987 0.216237 0.172957 \n", "13 0.054933 0.214022 0.168298 \n", "14 0.054400 0.212962 0.165693 \n", "15 0.053853 0.211881 0.163391 \n", "16 0.052027 0.209912 0.160778 \n", "17 0.050373 0.206874 0.156551 \n", "18 0.047907 0.202977 0.151666 \n", "19 0.047667 0.202505 0.150583 \n", "20 0.046693 0.200917 0.148415 \n", "21 0.044453 0.198310 0.144895 \n", "22 0.043813 0.196469 0.142642 \n", "\n", " validation_auc validation_pr_auc validation_lift \\\n", "0 0.500000 0.492640 1.000000 \n", "1 0.958475 0.950121 2.007326 \n", "2 0.968964 0.962745 2.018078 \n", "3 0.972759 0.967269 2.016481 \n", "4 0.975368 0.970918 2.016613 \n", "5 0.976955 0.972420 2.020453 \n", "6 0.977808 0.973754 2.019880 \n", "7 0.978794 0.975109 2.018908 \n", "8 0.980333 0.977271 2.021951 \n", "9 0.980813 0.977909 2.022277 \n", "10 0.981356 0.978627 2.021888 \n", "11 0.982148 0.979434 2.023615 \n", "12 0.982491 0.979711 2.022975 \n", "13 0.982909 0.980370 2.022757 \n", "14 0.983133 0.980693 2.022277 \n", "15 0.983338 0.981016 2.022782 \n", "16 0.983703 0.981483 2.021760 \n", "17 0.984204 0.982155 2.029880 \n", "18 0.984870 0.982975 2.029880 \n", "19 0.984970 0.982921 2.029880 \n", "20 0.985299 0.983489 2.029880 \n", "21 0.985755 0.983831 2.021760 \n", "22 0.986037 0.984178 2.029880 \n", "\n", " validation_classification_error \n", "0 0.50736 \n", "1 0.08816 \n", "2 0.08072 \n", "3 0.07328 \n", "4 0.07028 \n", "5 0.06956 \n", "6 0.06872 \n", "7 0.06696 \n", "8 0.06516 \n", "9 0.06432 \n", "10 0.06240 \n", "11 0.06084 \n", "12 0.05932 \n", "13 0.05840 \n", "14 0.05852 \n", "15 0.05792 \n", "16 0.05660 \n", "17 0.05524 \n", "18 0.05272 \n", "19 0.05232 \n", "20 0.05108 \n", "21 0.04960 \n", "22 0.04900 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_def.scoring_history()" ] }, { "cell_type": "markdown", "id": "45b72bf5", "metadata": {}, "source": [ "##### Train XGBoost model and use eval_metric=\"logloss\" for early stopping" ] }, { "cell_type": "markdown", "id": "59bcee0a", "metadata": {}, "source": [ "We will use the same parameters as in the first case and add `eval_metric=\"logloss\"`. To use actually use the value of the `eval_metric` for early stopping, we also need to specify `stopping_metric=\"custom\"`." ] }, { "cell_type": "code", "execution_count": 7, "id": "2b4650c3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "xgboost Model Build progress: |██████████████████████████████████████████████████| (done) 100%\n" ] }, { "data": { "text/html": [ "
Model Details\n",
       "=============\n",
       "H2OXGBoostEstimator : XGBoost\n",
       "Model Key: XGBoost_model_python_1666962310297_567\n",
       "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", " \n", "\n", "
Model Summary:
number_of_trees
22.0
\n", "
\n", "
\n", "
ModelMetricsBinomial: xgboost\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.03475774769265435\n",
       "RMSE: 0.18643429859511995\n",
       "LogLoss: 0.12901406890397824\n",
       "Mean Per-Class Error: 0.04387881533186782\n",
       "AUC: 0.9893340459497513\n",
       "AUCPR: 0.988704085388505\n",
       "Gini: 0.9786680918995025
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.5101676438305829
01ErrorRate
035155.02174.00.0582 (2174.0/37329.0)
11112.036559.00.0295 (1112.0/37671.0)
Total36267.038733.00.0438 (3286.0/75000.0)
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Maximum Metrics: Maximum metrics at their respective thresholds
metricthresholdvalueidx
max f10.51016760.9569918214.0
max f20.30828060.9715081263.0
max f0point50.75326070.9592084147.0
max accuracy0.55617120.9564201.0
max precision0.99745651.00.0
max recall0.00160281.0399.0
max specificity0.99745651.00.0
max absolute_mcc0.55617120.9129277201.0
max min_per_class_accuracy0.62398530.9548073184.0
max mean_per_class_accuracy0.55617120.9563597201.0
max tns0.997456537329.00.0
max fns0.997456537460.00.0
max fps0.001602837329.0399.0
max tps0.001602837671.0399.0
max tnr0.99745651.00.0
max fnr0.99745650.99439890.0
max fpr0.00160281.0399.0
max tpr0.00160281.0399.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Gains/Lift Table: Avg response rate: 50.23 %, avg score: 50.22 %
groupcumulative_data_fractionlower_thresholdliftcumulative_liftresponse_ratescorecumulative_response_ratecumulative_scorecapture_ratecumulative_capture_rategaincumulative_gainkolmogorov_smirnov
10.01017330.99610971.99092141.99092141.00.99675161.00.99675160.02025430.020254399.092139899.09213980.0202543
20.02002670.99571531.99092141.99092141.00.99588941.00.99632740.01961720.039871599.092139899.09213980.0398715
30.030680.99539641.99092141.99092141.00.99554951.00.99605730.02120990.061081599.092139899.09213980.0610815
40.04010670.99519441.98810541.99025950.99858560.99527080.99966760.99587240.01874120.079822798.810538599.02595230.0797959
50.05177330.99492971.98637071.98938320.99771430.99500840.99922740.99567770.02317430.102997098.637072198.93832160.1029166
60.10001330.99330971.98817001.98879800.99861800.99416570.99893350.99494840.09590930.198906398.816998798.87980320.1986920
70.150080.99049841.98402871.98720700.99653790.99205770.99813430.99398410.09933370.298240098.402872898.72069930.2976775
80.20001330.98496051.98241551.98601080.99572760.98791250.99753350.99246830.09898860.397228698.241545998.60107860.3962375
90.300040.95610461.96730211.97977370.98813650.97403870.99440070.98632430.19678270.594011396.730209697.97737250.5906359
100.40001330.87706131.91232541.96291670.96052280.92100760.98593380.97000000.19118150.785192991.232540896.29167040.7738880
110.50001330.63834761.67635581.90560610.8420.80511860.95714780.93702460.16763560.952828467.635581790.56060550.9097788
120.60.07812450.42611401.65905690.21402850.28761850.83331110.82880470.04260570.9954342-57.388600665.90569230.7944912
130.70001330.01282220.02388791.42543460.01199840.03390010.71596730.71523390.00238910.9978233-97.611212842.54346480.5983483
140.800080.00579300.01220281.24868060.00612920.00841650.62718730.62683170.00122110.9990444-98.779715124.86805790.3997516
150.90.00344280.00451641.11055070.00226850.00452990.55780740.55774240.00045130.9994956-99.548363211.05507040.1999028
161.00.00016720.00504371.00.00253330.00262680.502280.50223080.00050441.0-99.49563320.00.0
\n", "
\n", "
\n", "
ModelMetricsBinomial: xgboost\n",
       "** Reported on validation data. **\n",
       "\n",
       "MSE: 0.038600168581279404\n",
       "RMSE: 0.1964692560714765\n",
       "LogLoss: 0.14264211397729937\n",
       "Mean Per-Class Error: 0.04874265748103474\n",
       "AUC: 0.9860367984758349\n",
       "AUCPR: 0.9841777019906762\n",
       "Gini: 0.9720735969516698
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.4899955771186135
01ErrorRate
011844.0840.00.0662 (840.0/12684.0)
1385.011931.00.0313 (385.0/12316.0)
Total12229.012771.00.049 (1225.0/25000.0)
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Maximum Metrics: Maximum metrics at their respective thresholds
metricthresholdvalueidx
max f10.48999560.9511699217.0
max f20.23816390.9684011281.0
max f0point50.76376010.9521032144.0
max accuracy0.48999560.951217.0
max precision0.99746831.00.0
max recall0.00183541.0398.0
max specificity0.99746831.00.0
max absolute_mcc0.48999560.9026291217.0
max min_per_class_accuracy0.63036380.9494966182.0
max mean_per_class_accuracy0.48999560.9512573217.0
max tns0.997468312684.00.0
max fns0.997468312244.00.0
max fps0.001343512684.0399.0
max tps0.001835412316.0398.0
max tnr0.99746831.00.0
max fnr0.99746830.99415390.0
max fpr0.00134351.0399.0
max tpr0.00183541.0398.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Gains/Lift Table: Avg response rate: 49.26 %, avg score: 49.47 %
groupcumulative_data_fractionlower_thresholdliftcumulative_liftresponse_ratescorecumulative_response_ratecumulative_scorecapture_ratecumulative_capture_rategaincumulative_gainkolmogorov_smirnov
10.010280.99612442.02987982.02987981.00.99673131.00.99673130.02086720.0208672102.9879831102.98798310.0208672
20.020320.99580122.02987982.02987981.00.99591751.00.99632920.02038000.0412472102.9879831102.98798310.0412472
30.030160.99546152.02162832.02718770.99593500.99561020.99867370.99609460.01989280.0611400102.1628287102.71876830.0610611
40.041080.99519442.02244442.02592680.99633700.99528870.99805260.99588040.02208510.0832251102.2444374102.59268030.0830674
50.052560.99492972.01573432.02370060.99303140.99500850.99695590.99569000.02314060.1063657101.5734327102.37005930.1060503
60.10.99328592.01961062.02176030.99494100.99415520.9960.99496190.09581030.2021760101.9610625102.17603120.2013876
70.150.99026942.02338422.02230160.99680.99194900.99626670.99395760.10116920.3033452102.3384216102.23016130.3022415
80.20.98406602.01364082.02013640.9920.98739450.99520.99231680.10068200.4040273101.3640792102.01364080.4021351
90.30.95238971.98522252.00849840.9780.97170510.98946670.98544620.19852220.602549598.5222475100.84984300.5963212
100.40.86710571.92351411.98725240.94760.91425670.9790.96764880.19235140.794900992.351412898.72523550.7783447
110.50.58223531.61740821.91328350.79680.78250590.942560.93062020.16174080.956641861.740824991.32835340.9000350
120.60.06677650.37755761.65732920.1860.24801800.81646670.81685320.03775580.9943975-62.244235165.73292190.7773524
130.70.01257710.01867491.42323570.00920.03044820.70114290.70450960.00186750.9962650-98.132510642.32357440.5839345
140.800480.00574120.01858571.24691750.00915610.00815740.61428140.61710020.00186750.9981325-98.141431724.69174880.3895705
150.90140.00345920.01045921.10848460.00515260.00452110.54608390.54851640.00105550.9991880-98.954084910.84846330.1927390
161.00.00026090.00823481.00.00405680.00264460.492640.49469340.00081201.0-99.17651930.00.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Scoring History:
timestampdurationnumber_of_treestraining_rmsetraining_loglosstraining_auctraining_pr_auctraining_lifttraining_classification_errortraining_customvalidation_rmsevalidation_loglossvalidation_aucvalidation_pr_aucvalidation_liftvalidation_classification_errorvalidation_custom
2022-10-28 14:19:30 0.025 sec0.00.50.69314720.50.50228001.00.497720.6931470.50.69314720.50.49264001.00.507360.693147
2022-10-28 14:19:30 0.139 sec1.00.39960100.50845560.95996050.95288241.96994430.085080.5084560.40029390.50957660.95847540.95012122.00732560.088160.509577
2022-10-28 14:19:30 0.179 sec2.00.33707780.40320480.96933580.96384321.97497810.07869330.4032050.33787260.40431480.96896420.96274522.01807820.080720.404315
2022-10-28 14:19:30 0.228 sec3.00.29571350.33356510.97357640.96929711.98235830.071240.3335650.29728040.33556900.97275850.96726902.01648130.073280.335569
2022-10-28 14:19:30 0.280 sec4.00.26905650.28670500.97693860.97366611.98244040.06877330.2867050.27119300.28943030.97536800.97091772.01661260.070280.28943
2022-10-28 14:19:30 0.356 sec5.00.25297700.25489940.97823490.97539811.98300130.069360.2548990.25504370.25752650.97695490.97241972.02045310.069560.257526
2022-10-28 14:19:30 0.434 sec6.00.24305500.23288590.97925360.97673331.98269450.068640.2328860.24525710.23572950.97780850.97375372.01988040.068720.235729
2022-10-28 14:19:30 0.511 sec7.00.23667030.21730430.98010050.97795181.98401250.06553330.2173040.23889240.22020640.97879450.97510872.01890750.066960.220206
2022-10-28 14:19:30 0.587 sec8.00.22742820.20046420.98168940.97958491.98834250.061720.2004640.23028640.20405440.98033300.97727142.02195060.065160.204054
2022-10-28 14:19:31 0.676 sec9.00.22350480.19088780.98234440.98052191.98839480.06080.1908880.22675650.19510480.98081280.97790932.02227730.064320.195105
------------------------------------------------------
2022-10-28 14:19:31 1.058 sec13.00.20932990.16188170.98492080.98357531.98853420.05493330.1618820.21402170.16829750.98290890.98037032.02275740.05840.168298
2022-10-28 14:19:31 1.160 sec14.00.20801160.15872260.98524520.98395381.98830860.05440.1587230.21296180.16569250.98313260.98069342.02227730.058520.165693
2022-10-28 14:19:31 1.246 sec15.00.20656260.15561140.98565950.98469331.99092140.05385330.1556110.21188150.16339150.98333770.98101622.02278230.057920.163391
2022-10-28 14:19:31 1.345 sec16.00.20360960.15179390.98634740.98538081.99092140.05202670.1517940.20991210.16077760.98370300.98148282.02176030.05660.160778
2022-10-28 14:19:31 1.462 sec17.00.20012800.14708220.98688680.98591041.99092140.05037330.1470820.20687410.15655080.98420400.98215552.02987980.055240.156551
2022-10-28 14:19:31 1.566 sec18.00.19523640.14096880.98776670.98683231.99092140.04790670.1409690.20297700.15166650.98486960.98297462.02987980.052720.151666
2022-10-28 14:19:32 1.684 sec19.00.19436900.13917510.98800680.98712831.99092140.04766670.1391750.20250510.15058350.98497030.98292112.02987980.052320.150583
2022-10-28 14:19:32 1.802 sec20.00.19194890.13612130.98840070.98757891.99092140.04669330.1361210.20091700.14841520.98529860.98348892.02987980.051080.148415
2022-10-28 14:19:32 1.916 sec21.00.18862570.13193560.98896110.98821931.99092140.04445330.1319360.19831050.14489470.98575460.98383082.02176030.04960.144895
2022-10-28 14:19:32 2.057 sec22.00.18643430.12901410.98933400.98870411.99092140.04381330.1290140.19646930.14264210.98603680.98417772.02987980.0490.142642
\n", "
\n", "
[23 rows x 18 columns]
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Variable Importances:
variablerelative_importancescaled_importancepercentage
C1170178.751.00.4261358
C235732.66406250.50916640.2169741
C1321608.48632810.30790640.1312100
C815654.54980470.22306680.0950568
C312402.26074220.17672390.0753084
C195739.76416020.08178780.0348527
C52835.91918950.04040990.0172201
C688.93524930.00126730.0005400
C1251.98764800.00074080.0003157
C1749.17110440.00070070.0002986
C748.05213170.00068470.0002918
C1446.54721830.00066330.0002826
C1043.87886050.00062520.0002664
C442.43465420.00060470.0002577
C1839.39529800.00056140.0002392
C132.55399320.00046390.0001977
C1525.95281600.00036980.0001576
C2023.11923220.00032940.0001404
C1622.99838070.00032770.0001396
C918.91131590.00026950.0001148
\n", "
\n", "
\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section.
" ], "text/plain": [ "Model Details\n", "=============\n", "H2OXGBoostEstimator : XGBoost\n", "Model Key: XGBoost_model_python_1666962310297_567\n", "\n", "\n", "Model Summary: \n", " number_of_trees\n", "-- -----------------\n", " 22\n", "\n", "ModelMetricsBinomial: xgboost\n", "** Reported on train data. **\n", "\n", "MSE: 0.03475774769265435\n", "RMSE: 0.18643429859511995\n", "LogLoss: 0.12901406890397824\n", "Mean Per-Class Error: 0.04387881533186782\n", "AUC: 0.9893340459497513\n", "AUCPR: 0.988704085388505\n", "Gini: 0.9786680918995025\n", "\n", "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.5101676438305829\n", " 0 1 Error Rate\n", "----- ----- ----- ------- ----------------\n", "0 35155 2174 0.0582 (2174.0/37329.0)\n", "1 1112 36559 0.0295 (1112.0/37671.0)\n", "Total 36267 38733 0.0438 (3286.0/75000.0)\n", "\n", "Maximum Metrics: Maximum metrics at their respective thresholds\n", "metric threshold value idx\n", "--------------------------- ----------- -------- -----\n", "max f1 0.510168 0.956992 214\n", "max f2 0.308281 0.971508 263\n", "max f0point5 0.753261 0.959208 147\n", "max accuracy 0.556171 0.9564 201\n", "max precision 0.997456 1 0\n", "max recall 0.00160282 1 399\n", "max specificity 0.997456 1 0\n", "max absolute_mcc 0.556171 0.912928 201\n", "max min_per_class_accuracy 0.623985 0.954807 184\n", "max mean_per_class_accuracy 0.556171 0.95636 201\n", "max tns 0.997456 37329 0\n", "max fns 0.997456 37460 0\n", "max fps 0.00160282 37329 399\n", "max tps 0.00160282 37671 399\n", "max tnr 0.997456 1 0\n", "max fnr 0.997456 0.994399 0\n", "max fpr 0.00160282 1 399\n", "max tpr 0.00160282 1 399\n", "\n", "Gains/Lift Table: Avg response rate: 50.23 %, avg score: 50.22 %\n", "group cumulative_data_fraction lower_threshold lift cumulative_lift response_rate score cumulative_response_rate cumulative_score capture_rate cumulative_capture_rate gain cumulative_gain kolmogorov_smirnov\n", "------- -------------------------- ----------------- ---------- ----------------- --------------- ---------- -------------------------- ------------------ -------------- ------------------------- -------- ----------------- --------------------\n", "1 0.0101733 0.99611 1.99092 1.99092 1 0.996752 1 0.996752 0.0202543 0.0202543 99.0921 99.0921 0.0202543\n", "2 0.0200267 0.995715 1.99092 1.99092 1 0.995889 1 0.996327 0.0196172 0.0398715 99.0921 99.0921 0.0398715\n", "3 0.03068 0.995396 1.99092 1.99092 1 0.995549 1 0.996057 0.0212099 0.0610815 99.0921 99.0921 0.0610815\n", "4 0.0401067 0.995194 1.98811 1.99026 0.998586 0.995271 0.999668 0.995872 0.0187412 0.0798227 98.8105 99.026 0.0797959\n", "5 0.0517733 0.99493 1.98637 1.98938 0.997714 0.995008 0.999227 0.995678 0.0231743 0.102997 98.6371 98.9383 0.102917\n", "6 0.100013 0.99331 1.98817 1.9888 0.998618 0.994166 0.998933 0.994948 0.0959093 0.198906 98.817 98.8798 0.198692\n", "7 0.15008 0.990498 1.98403 1.98721 0.996538 0.992058 0.998134 0.993984 0.0993337 0.29824 98.4029 98.7207 0.297677\n", "8 0.200013 0.98496 1.98242 1.98601 0.995728 0.987913 0.997533 0.992468 0.0989886 0.397229 98.2415 98.6011 0.396237\n", "9 0.30004 0.956105 1.9673 1.97977 0.988136 0.974039 0.994401 0.986324 0.196783 0.594011 96.7302 97.9774 0.590636\n", "10 0.400013 0.877061 1.91233 1.96292 0.960523 0.921008 0.985934 0.97 0.191182 0.785193 91.2325 96.2917 0.773888\n", "11 0.500013 0.638348 1.67636 1.90561 0.842 0.805119 0.957148 0.937025 0.167636 0.952828 67.6356 90.5606 0.909779\n", "12 0.6 0.0781245 0.426114 1.65906 0.214029 0.287618 0.833311 0.828805 0.0426057 0.995434 -57.3886 65.9057 0.794491\n", "13 0.700013 0.0128222 0.0238879 1.42543 0.0119984 0.0339001 0.715967 0.715234 0.00238911 0.997823 -97.6112 42.5435 0.598348\n", "14 0.80008 0.00579305 0.0122028 1.24868 0.00612925 0.00841651 0.627187 0.626832 0.0012211 0.999044 -98.7797 24.8681 0.399752\n", "15 0.9 0.00344276 0.00451637 1.11055 0.00226848 0.00452987 0.557807 0.557742 0.000451276 0.999496 -99.5484 11.0551 0.199903\n", "16 1 0.000167174 0.00504367 1 0.00253333 0.00262682 0.50228 0.502231 0.000504367 1 -99.4956 0 0\n", "\n", "ModelMetricsBinomial: xgboost\n", "** Reported on validation data. **\n", "\n", "MSE: 0.038600168581279404\n", "RMSE: 0.1964692560714765\n", "LogLoss: 0.14264211397729937\n", "Mean Per-Class Error: 0.04874265748103474\n", "AUC: 0.9860367984758349\n", "AUCPR: 0.9841777019906762\n", "Gini: 0.9720735969516698\n", "\n", "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.4899955771186135\n", " 0 1 Error Rate\n", "----- ----- ----- ------- ----------------\n", "0 11844 840 0.0662 (840.0/12684.0)\n", "1 385 11931 0.0313 (385.0/12316.0)\n", "Total 12229 12771 0.049 (1225.0/25000.0)\n", "\n", "Maximum Metrics: Maximum metrics at their respective thresholds\n", "metric threshold value idx\n", "--------------------------- ----------- -------- -----\n", "max f1 0.489996 0.95117 217\n", "max f2 0.238164 0.968401 281\n", "max f0point5 0.76376 0.952103 144\n", "max accuracy 0.489996 0.951 217\n", "max precision 0.997468 1 0\n", "max recall 0.00183535 1 398\n", "max specificity 0.997468 1 0\n", "max absolute_mcc 0.489996 0.902629 217\n", "max min_per_class_accuracy 0.630364 0.949497 182\n", "max mean_per_class_accuracy 0.489996 0.951257 217\n", "max tns 0.997468 12684 0\n", "max fns 0.997468 12244 0\n", "max fps 0.00134346 12684 399\n", "max tps 0.00183535 12316 398\n", "max tnr 0.997468 1 0\n", "max fnr 0.997468 0.994154 0\n", "max fpr 0.00134346 1 399\n", "max tpr 0.00183535 1 398\n", "\n", "Gains/Lift Table: Avg response rate: 49.26 %, avg score: 49.47 %\n", "group cumulative_data_fraction lower_threshold lift cumulative_lift response_rate score cumulative_response_rate cumulative_score capture_rate cumulative_capture_rate gain cumulative_gain kolmogorov_smirnov\n", "------- -------------------------- ----------------- ---------- ----------------- --------------- ---------- -------------------------- ------------------ -------------- ------------------------- -------- ----------------- --------------------\n", "1 0.01028 0.996124 2.02988 2.02988 1 0.996731 1 0.996731 0.0208672 0.0208672 102.988 102.988 0.0208672\n", "2 0.02032 0.995801 2.02988 2.02988 1 0.995918 1 0.996329 0.02038 0.0412472 102.988 102.988 0.0412472\n", "3 0.03016 0.995462 2.02163 2.02719 0.995935 0.99561 0.998674 0.996095 0.0198928 0.06114 102.163 102.719 0.0610611\n", "4 0.04108 0.995194 2.02244 2.02593 0.996337 0.995289 0.998053 0.99588 0.0220851 0.0832251 102.244 102.593 0.0830674\n", "5 0.05256 0.99493 2.01573 2.0237 0.993031 0.995009 0.996956 0.99569 0.0231406 0.106366 101.573 102.37 0.10605\n", "6 0.1 0.993286 2.01961 2.02176 0.994941 0.994155 0.996 0.994962 0.0958103 0.202176 101.961 102.176 0.201388\n", "7 0.15 0.990269 2.02338 2.0223 0.9968 0.991949 0.996267 0.993958 0.101169 0.303345 102.338 102.23 0.302241\n", "8 0.2 0.984066 2.01364 2.02014 0.992 0.987394 0.9952 0.992317 0.100682 0.404027 101.364 102.014 0.402135\n", "9 0.3 0.95239 1.98522 2.0085 0.978 0.971705 0.989467 0.985446 0.198522 0.60255 98.5222 100.85 0.596321\n", "10 0.4 0.867106 1.92351 1.98725 0.9476 0.914257 0.979 0.967649 0.192351 0.794901 92.3514 98.7252 0.778345\n", "11 0.5 0.582235 1.61741 1.91328 0.7968 0.782506 0.94256 0.93062 0.161741 0.956642 61.7408 91.3284 0.900035\n", "12 0.6 0.0667765 0.377558 1.65733 0.186 0.248018 0.816467 0.816853 0.0377558 0.994398 -62.2442 65.7329 0.777352\n", "13 0.7 0.0125771 0.0186749 1.42324 0.0092 0.0304482 0.701143 0.70451 0.00186749 0.996265 -98.1325 42.3236 0.583935\n", "14 0.80048 0.00574124 0.0185857 1.24692 0.00915605 0.00815736 0.614281 0.6171 0.00186749 0.998133 -98.1414 24.6917 0.389571\n", "15 0.9014 0.00345923 0.0104592 1.10848 0.0051526 0.0045211 0.546084 0.548516 0.00105554 0.999188 -98.9541 10.8485 0.192739\n", "16 1 0.000260908 0.00823481 1 0.0040568 0.00264459 0.49264 0.494693 0.000811952 1 -99.1765 0 0\n", "\n", "Scoring History: \n", " timestamp duration number_of_trees training_rmse training_logloss training_auc training_pr_auc training_lift training_classification_error training_custom validation_rmse validation_logloss validation_auc validation_pr_auc validation_lift validation_classification_error validation_custom\n", "--- ------------------- ---------- ----------------- ------------------- ------------------- ------------------ ------------------ ------------------ ------------------------------- ----------------- ------------------- -------------------- ------------------ ------------------- ------------------ --------------------------------- -------------------\n", " 2022-10-28 14:19:30 0.025 sec 0.0 0.5 0.6931471805600504 0.5 0.5022800000000001 1.0 0.49772 0.693147 0.5 0.6931471805599932 0.5 0.49263999999999997 1.0 0.50736 0.693147\n", " 2022-10-28 14:19:30 0.139 sec 1.0 0.39960097366259284 0.5084555759713797 0.9599604712562773 0.952882361784991 1.969944257252065 0.08508 0.508456 0.4002938936840949 0.5095766191321826 0.9584754441256724 0.9501212341159861 2.007325610768287 0.08816 0.509577\n", " 2022-10-28 14:19:30 0.179 sec 2.0 0.3370778235169717 0.40320483936633433 0.9693357563355386 0.9638432211755918 1.974978083921303 0.07869333333333334 0.403205 0.3378725612813438 0.4043147576521532 0.9689642384208382 0.9627452021209266 2.0180782041889165 0.08072 0.404315\n", " 2022-10-28 14:19:30 0.228 sec 3.0 0.2957135469964664 0.33356511573213676 0.9735764390034864 0.9692971186564519 1.9823582956342736 0.07124 0.333565 0.2972804020126436 0.33556896107331324 0.972758545365196 0.9672690232547145 2.0164812843739717 0.07328 0.335569\n", " 2022-10-28 14:19:30 0.280 sec 4.0 0.2690565369849074 0.2867049786815229 0.9769385896258127 0.973666085141308 1.9824403701018989 0.06877333333333334 0.286705 0.2711929535273588 0.2894303318159203 0.9753679683707979 0.9709177118281687 2.0166126426753443 0.07028 0.28943\n", " 2022-10-28 14:19:30 0.356 sec 5.0 0.2529770080480673 0.25489941558805895 0.9782348903583494 0.9753981493946517 1.9830012735164846 0.06936 0.254899 0.2550436637594876 0.25752647933993555 0.976954942633842 0.9724197019498086 2.0204531445917966 0.06956 0.257526\n", " 2022-10-28 14:19:30 0.434 sec 6.0 0.24305497030792145 0.23288588902305773 0.9792535639846858 0.9767332522440302 1.982694450495822 0.06864 0.232886 0.24525713259217433 0.23572949157271123 0.9778084651737403 0.9737536794534183 2.0198804230789538 0.06872 0.235729\n", " 2022-10-28 14:19:30 0.511 sec 7.0 0.23667027877522748 0.21730431422414767 0.9801004932398384 0.9779517651384936 1.9840124751203916 0.06553333333333333 0.217304 0.2388923863271564 0.22020638093402833 0.978794451615705 0.9751086853604877 2.018907507702571 0.06696 0.220206\n", " 2022-10-28 14:19:30 0.587 sec 8.0 0.22742820294228833 0.2004641682608729 0.981689446457674 0.979584885842671 1.988342484694663 0.06172 0.200464 0.23028641120487947 0.20405435411113004 0.9803330089878547 0.9772714003156485 2.021950613023709 0.06516 0.204054\n", " 2022-10-28 14:19:31 0.676 sec 9.0 0.22350478838486162 0.1908877726426607 0.9823443926985863 0.9805219447464963 1.9883948484251914 0.0608 0.190888 0.22675651465031937 0.19510476775550073 0.9808127993480623 0.9779093493480279 2.0222772849300505 0.06432 0.195105\n", "--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---\n", " 2022-10-28 14:19:31 1.058 sec 13.0 0.20932986426153485 0.16188173031435207 0.9849208281386209 0.9835752931336927 1.9885342025018198 0.054933333333333334 0.161882 0.2140216611245386 0.16829754841446237 0.9829088919260484 0.9803702989731918 2.022757445741668 0.0584 0.168298\n", " 2022-10-28 14:19:31 1.160 sec 14.0 0.20801159542673212 0.15872260659114573 0.9852451840387032 0.9839537573047635 1.9883086406824775 0.0544 0.158723 0.2129618210780405 0.16569253851178234 0.9831326172024832 0.9806933766171955 2.0222772849300505 0.05852 0.165693\n", " 2022-10-28 14:19:31 1.246 sec 15.0 0.20656261820821578 0.15561137441472306 0.9856594600307703 0.9846933313933812 1.9909213984231904 0.053853333333333336 0.155611 0.2118814635733488 0.16339145578701472 0.9833376856363834 0.9810162497973599 2.022782349187026 0.05792 0.163391\n", " 2022-10-28 14:19:31 1.345 sec 16.0 0.2036095613587592 0.1517938621462097 0.9863473648236762 0.9853807508666774 1.9909213984231904 0.052026666666666666 0.151794 0.20991212994517833 0.1607776039482004 0.9837030416011293 0.9814827764043785 2.0217603117895417 0.0566 0.160778\n", " 2022-10-28 14:19:31 1.462 sec 17.0 0.20012801070573083 0.1470822013139249 0.986886830974453 0.985910377892329 1.9909213984231904 0.05037333333333333 0.147082 0.2068740603495647 0.1565507907729844 0.9842040109503664 0.9821554851954529 2.029879831113998 0.05524 0.156551\n", " 2022-10-28 14:19:31 1.566 sec 18.0 0.1952363704640891 0.14096879922102418 0.9877666946033187 0.9868323340675447 1.9909213984231904 0.04790666666666667 0.140969 0.20297695900076984 0.15166647071587894 0.9848695695625415 0.9829746186958338 2.029879831113998 0.05272 0.151666\n", " 2022-10-28 14:19:32 1.684 sec 19.0 0.19436898343164097 0.1391751330056711 0.9880068361300588 0.9871283090191609 1.9909213984231904 0.04766666666666667 0.139175 0.2025051413752741 0.15058346705393647 0.9849703081904262 0.9829211071563434 2.029879831113998 0.05232 0.150583\n", " 2022-10-28 14:19:32 1.802 sec 20.0 0.19194891175623852 0.13612128709347052 0.988400723075942 0.9875788628040455 1.9909213984231904 0.04669333333333334 0.136121 0.2009170028210971 0.14841515569437927 0.9852986065255842 0.9834889458246532 2.029879831113998 0.05108 0.148415\n", " 2022-10-28 14:19:32 1.916 sec 21.0 0.18862571988282295 0.1319356174549523 0.9889610721498387 0.9882193402029281 1.9909213984231904 0.04445333333333333 0.131936 0.1983104566213673 0.1448947109368754 0.9857545901273815 0.9838307814948767 2.0217603117895417 0.0496 0.144895\n", " 2022-10-28 14:19:32 2.057 sec 22.0 0.18643429859511995 0.12901406890397824 0.9893340459497513 0.988704085388505 1.9909213984231904 0.043813333333333336 0.129014 0.1964692560714765 0.14264211397729937 0.9860367984758349 0.9841777019906762 2.029879831113998 0.049 0.142642\n", "[23 rows x 18 columns]\n", "\n", "\n", "Variable Importances: \n", "variable relative_importance scaled_importance percentage\n", "---------- --------------------- ------------------- ------------\n", "C11 70178.8 1 0.426136\n", "C2 35732.7 0.509166 0.216974\n", "C13 21608.5 0.307906 0.13121\n", "C8 15654.5 0.223067 0.0950568\n", "C3 12402.3 0.176724 0.0753084\n", "C19 5739.76 0.0817878 0.0348527\n", "C5 2835.92 0.0404099 0.0172201\n", "C6 88.9352 0.00126727 0.000540028\n", "C12 51.9876 0.000740789 0.000315677\n", "C17 49.1711 0.000700655 0.000298574\n", "C7 48.0521 0.000684711 0.00029178\n", "C14 46.5472 0.000663267 0.000282642\n", "C10 43.8789 0.000625244 0.000266439\n", "C4 42.4347 0.000604665 0.00025767\n", "C18 39.3953 0.000561357 0.000239214\n", "C1 32.554 0.000463873 0.000197673\n", "C15 25.9528 0.00036981 0.000157589\n", "C20 23.1192 0.000329434 0.000140383\n", "C16 22.9984 0.000327711 0.00013965\n", "C9 18.9113 0.000269474 0.000114832\n", "\n", "[tips]\n", "Use `model.explain()` to inspect the model.\n", "--\n", "Use `h2o.display.toggle_user_tips()` to switch on/off this section." ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_eval = H2OXGBoostEstimator(ntrees=1000, max_depth=6, score_each_iteration=True, \n", " eval_metric=\"logloss\",\n", " stopping_rounds=3, stopping_tolerance=0.05, stopping_metric=\"custom\")\n", "model_eval.train(y=\"y\", training_frame=train, validation_frame=valid)" ] }, { "cell_type": "markdown", "id": "8582fbb0", "metadata": {}, "source": [ "The scoring history for model with `eval_metric=\"logloss\"` will look similar to the scoring history of the first model. This is expected - we didn't actually changed the training behavior, we only changed the source of the values that trigger early stopping. In this case, we are stopping on values of `validation_custom`. This value correspond to the value calculated and returned by XGBoost. It should be close to H2O's own `validation_logloss` value, there can be only a small difference caused by a different precision in XGBoost and H2O (the values should be within absolute tolerance of 1e-5). This is, however, something to keep in mind. There can be edge cases where H2O metric will differ slightly from the conceptually same XGBoost metric and this might cause the models to stop at a different iteration.\n", "\n", "The scoring history will also have value of `eval_metric` for the training frame - see column `training_custom`." ] }, { "cell_type": "code", "execution_count": 8, "id": "46135033", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampdurationnumber_of_treestraining_rmsetraining_loglosstraining_auctraining_pr_auctraining_lifttraining_classification_errortraining_customvalidation_rmsevalidation_loglossvalidation_aucvalidation_pr_aucvalidation_liftvalidation_classification_errorvalidation_custom
02022-10-28 14:19:300.025 sec0.00.5000000.6931470.5000000.5022801.0000000.4977200.6931470.5000000.6931470.5000000.4926401.0000000.507360.693147
12022-10-28 14:19:300.139 sec1.00.3996010.5084560.9599600.9528821.9699440.0850800.5084560.4002940.5095770.9584750.9501212.0073260.088160.509577
22022-10-28 14:19:300.179 sec2.00.3370780.4032050.9693360.9638431.9749780.0786930.4032050.3378730.4043150.9689640.9627452.0180780.080720.404315
32022-10-28 14:19:300.228 sec3.00.2957140.3335650.9735760.9692971.9823580.0712400.3335650.2972800.3355690.9727590.9672692.0164810.073280.335569
42022-10-28 14:19:300.280 sec4.00.2690570.2867050.9769390.9736661.9824400.0687730.2867050.2711930.2894300.9753680.9709182.0166130.070280.289430
52022-10-28 14:19:300.356 sec5.00.2529770.2548990.9782350.9753981.9830010.0693600.2548990.2550440.2575260.9769550.9724202.0204530.069560.257526
62022-10-28 14:19:300.434 sec6.00.2430550.2328860.9792540.9767331.9826940.0686400.2328860.2452570.2357290.9778080.9737542.0198800.068720.235729
72022-10-28 14:19:300.511 sec7.00.2366700.2173040.9801000.9779521.9840120.0655330.2173040.2388920.2202060.9787940.9751092.0189080.066960.220206
82022-10-28 14:19:300.587 sec8.00.2274280.2004640.9816890.9795851.9883420.0617200.2004640.2302860.2040540.9803330.9772712.0219510.065160.204054
92022-10-28 14:19:310.676 sec9.00.2235050.1908880.9823440.9805221.9883950.0608000.1908880.2267570.1951050.9808130.9779092.0222770.064320.195105
102022-10-28 14:19:310.773 sec10.00.2200400.1829670.9830000.9812101.9882980.0592270.1829670.2236560.1877050.9813560.9786272.0218880.062400.187705
112022-10-28 14:19:310.854 sec11.00.2149260.1734630.9838780.9822241.9888190.0570270.1734630.2188990.1787480.9821480.9794342.0236150.060840.178748
122022-10-28 14:19:310.943 sec12.00.2120740.1672610.9843480.9827911.9886170.0559870.1672610.2162370.1729570.9824910.9797112.0229750.059320.172957
132022-10-28 14:19:311.058 sec13.00.2093300.1618820.9849210.9835751.9885340.0549330.1618820.2140220.1682980.9829090.9803702.0227570.058400.168298
142022-10-28 14:19:311.160 sec14.00.2080120.1587230.9852450.9839541.9883090.0544000.1587230.2129620.1656930.9831330.9806932.0222770.058520.165693
152022-10-28 14:19:311.246 sec15.00.2065630.1556110.9856590.9846931.9909210.0538530.1556110.2118810.1633910.9833380.9810162.0227820.057920.163391
162022-10-28 14:19:311.345 sec16.00.2036100.1517940.9863470.9853811.9909210.0520270.1517940.2099120.1607780.9837030.9814832.0217600.056600.160778
172022-10-28 14:19:311.462 sec17.00.2001280.1470820.9868870.9859101.9909210.0503730.1470820.2068740.1565510.9842040.9821552.0298800.055240.156551
182022-10-28 14:19:311.566 sec18.00.1952360.1409690.9877670.9868321.9909210.0479070.1409690.2029770.1516660.9848700.9829752.0298800.052720.151666
192022-10-28 14:19:321.684 sec19.00.1943690.1391750.9880070.9871281.9909210.0476670.1391750.2025050.1505830.9849700.9829212.0298800.052320.150583
202022-10-28 14:19:321.802 sec20.00.1919490.1361210.9884010.9875791.9909210.0466930.1361210.2009170.1484150.9852990.9834892.0298800.051080.148415
212022-10-28 14:19:321.916 sec21.00.1886260.1319360.9889610.9882191.9909210.0444530.1319360.1983100.1448950.9857550.9838312.0217600.049600.144895
222022-10-28 14:19:322.057 sec22.00.1864340.1290140.9893340.9887041.9909210.0438130.1290140.1964690.1426420.9860370.9841782.0298800.049000.142642
\n", "
" ], "text/plain": [ " timestamp duration number_of_trees training_rmse \\\n", "0 2022-10-28 14:19:30 0.025 sec 0.0 0.500000 \n", "1 2022-10-28 14:19:30 0.139 sec 1.0 0.399601 \n", "2 2022-10-28 14:19:30 0.179 sec 2.0 0.337078 \n", "3 2022-10-28 14:19:30 0.228 sec 3.0 0.295714 \n", "4 2022-10-28 14:19:30 0.280 sec 4.0 0.269057 \n", "5 2022-10-28 14:19:30 0.356 sec 5.0 0.252977 \n", "6 2022-10-28 14:19:30 0.434 sec 6.0 0.243055 \n", "7 2022-10-28 14:19:30 0.511 sec 7.0 0.236670 \n", "8 2022-10-28 14:19:30 0.587 sec 8.0 0.227428 \n", "9 2022-10-28 14:19:31 0.676 sec 9.0 0.223505 \n", "10 2022-10-28 14:19:31 0.773 sec 10.0 0.220040 \n", "11 2022-10-28 14:19:31 0.854 sec 11.0 0.214926 \n", "12 2022-10-28 14:19:31 0.943 sec 12.0 0.212074 \n", "13 2022-10-28 14:19:31 1.058 sec 13.0 0.209330 \n", "14 2022-10-28 14:19:31 1.160 sec 14.0 0.208012 \n", "15 2022-10-28 14:19:31 1.246 sec 15.0 0.206563 \n", "16 2022-10-28 14:19:31 1.345 sec 16.0 0.203610 \n", "17 2022-10-28 14:19:31 1.462 sec 17.0 0.200128 \n", "18 2022-10-28 14:19:31 1.566 sec 18.0 0.195236 \n", "19 2022-10-28 14:19:32 1.684 sec 19.0 0.194369 \n", "20 2022-10-28 14:19:32 1.802 sec 20.0 0.191949 \n", "21 2022-10-28 14:19:32 1.916 sec 21.0 0.188626 \n", "22 2022-10-28 14:19:32 2.057 sec 22.0 0.186434 \n", "\n", " training_logloss training_auc training_pr_auc training_lift \\\n", "0 0.693147 0.500000 0.502280 1.000000 \n", "1 0.508456 0.959960 0.952882 1.969944 \n", "2 0.403205 0.969336 0.963843 1.974978 \n", "3 0.333565 0.973576 0.969297 1.982358 \n", "4 0.286705 0.976939 0.973666 1.982440 \n", "5 0.254899 0.978235 0.975398 1.983001 \n", "6 0.232886 0.979254 0.976733 1.982694 \n", "7 0.217304 0.980100 0.977952 1.984012 \n", "8 0.200464 0.981689 0.979585 1.988342 \n", "9 0.190888 0.982344 0.980522 1.988395 \n", "10 0.182967 0.983000 0.981210 1.988298 \n", "11 0.173463 0.983878 0.982224 1.988819 \n", "12 0.167261 0.984348 0.982791 1.988617 \n", "13 0.161882 0.984921 0.983575 1.988534 \n", "14 0.158723 0.985245 0.983954 1.988309 \n", "15 0.155611 0.985659 0.984693 1.990921 \n", "16 0.151794 0.986347 0.985381 1.990921 \n", "17 0.147082 0.986887 0.985910 1.990921 \n", "18 0.140969 0.987767 0.986832 1.990921 \n", "19 0.139175 0.988007 0.987128 1.990921 \n", "20 0.136121 0.988401 0.987579 1.990921 \n", "21 0.131936 0.988961 0.988219 1.990921 \n", "22 0.129014 0.989334 0.988704 1.990921 \n", "\n", " training_classification_error training_custom validation_rmse \\\n", "0 0.497720 0.693147 0.500000 \n", "1 0.085080 0.508456 0.400294 \n", "2 0.078693 0.403205 0.337873 \n", "3 0.071240 0.333565 0.297280 \n", "4 0.068773 0.286705 0.271193 \n", "5 0.069360 0.254899 0.255044 \n", "6 0.068640 0.232886 0.245257 \n", "7 0.065533 0.217304 0.238892 \n", "8 0.061720 0.200464 0.230286 \n", "9 0.060800 0.190888 0.226757 \n", "10 0.059227 0.182967 0.223656 \n", "11 0.057027 0.173463 0.218899 \n", "12 0.055987 0.167261 0.216237 \n", "13 0.054933 0.161882 0.214022 \n", "14 0.054400 0.158723 0.212962 \n", "15 0.053853 0.155611 0.211881 \n", "16 0.052027 0.151794 0.209912 \n", "17 0.050373 0.147082 0.206874 \n", "18 0.047907 0.140969 0.202977 \n", "19 0.047667 0.139175 0.202505 \n", "20 0.046693 0.136121 0.200917 \n", "21 0.044453 0.131936 0.198310 \n", "22 0.043813 0.129014 0.196469 \n", "\n", " validation_logloss validation_auc validation_pr_auc validation_lift \\\n", "0 0.693147 0.500000 0.492640 1.000000 \n", "1 0.509577 0.958475 0.950121 2.007326 \n", "2 0.404315 0.968964 0.962745 2.018078 \n", "3 0.335569 0.972759 0.967269 2.016481 \n", "4 0.289430 0.975368 0.970918 2.016613 \n", "5 0.257526 0.976955 0.972420 2.020453 \n", "6 0.235729 0.977808 0.973754 2.019880 \n", "7 0.220206 0.978794 0.975109 2.018908 \n", "8 0.204054 0.980333 0.977271 2.021951 \n", "9 0.195105 0.980813 0.977909 2.022277 \n", "10 0.187705 0.981356 0.978627 2.021888 \n", "11 0.178748 0.982148 0.979434 2.023615 \n", "12 0.172957 0.982491 0.979711 2.022975 \n", "13 0.168298 0.982909 0.980370 2.022757 \n", "14 0.165693 0.983133 0.980693 2.022277 \n", "15 0.163391 0.983338 0.981016 2.022782 \n", "16 0.160778 0.983703 0.981483 2.021760 \n", "17 0.156551 0.984204 0.982155 2.029880 \n", "18 0.151666 0.984870 0.982975 2.029880 \n", "19 0.150583 0.984970 0.982921 2.029880 \n", "20 0.148415 0.985299 0.983489 2.029880 \n", "21 0.144895 0.985755 0.983831 2.021760 \n", "22 0.142642 0.986037 0.984178 2.029880 \n", "\n", " validation_classification_error validation_custom \n", "0 0.50736 0.693147 \n", "1 0.08816 0.509577 \n", "2 0.08072 0.404315 \n", "3 0.07328 0.335569 \n", "4 0.07028 0.289430 \n", "5 0.06956 0.257526 \n", "6 0.06872 0.235729 \n", "7 0.06696 0.220206 \n", "8 0.06516 0.204054 \n", "9 0.06432 0.195105 \n", "10 0.06240 0.187705 \n", "11 0.06084 0.178748 \n", "12 0.05932 0.172957 \n", "13 0.05840 0.168298 \n", "14 0.05852 0.165693 \n", "15 0.05792 0.163391 \n", "16 0.05660 0.160778 \n", "17 0.05524 0.156551 \n", "18 0.05272 0.151666 \n", "19 0.05232 0.150583 \n", "20 0.05108 0.148415 \n", "21 0.04960 0.144895 \n", "22 0.04900 0.142642 " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_eval.scoring_history()" ] }, { "cell_type": "markdown", "id": "78f7a080", "metadata": {}, "source": [ "##### Train XGBoost model, use eval_metric=\"logloss\" for early stopping and disable H2O metrics to speed-up model training\n", "\n", "In this example we will keep the same parameters as in the previous case and add flag `score_eval_metric_only=True`. This flag will instruct H2O to disable its own scoring and solely rely on `eval_metric` for early stopping and recording the scoring history." ] }, { "cell_type": "code", "execution_count": 9, "id": "2966b1f4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "xgboost Model Build progress: |██████████████████████████████████████████████████| (done) 100%\n" ] }, { "data": { "text/html": [ "
Model Details\n",
       "=============\n",
       "H2OXGBoostEstimator : XGBoost\n",
       "Model Key: XGBoost_model_python_1666962310297_616\n",
       "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", " \n", "\n", "
Model Summary:
number_of_trees
22.0
\n", "
\n", "
\n", "
ModelMetricsBinomial: xgboost\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.03475774769265435\n",
       "RMSE: 0.18643429859511995\n",
       "LogLoss: 0.12901406890397824\n",
       "Mean Per-Class Error: 0.04387881533186782\n",
       "AUC: 0.9893340459497513\n",
       "AUCPR: 0.988704085388505\n",
       "Gini: 0.9786680918995025
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.5101676438305829
01ErrorRate
035155.02174.00.0582 (2174.0/37329.0)
11112.036559.00.0295 (1112.0/37671.0)
Total36267.038733.00.0438 (3286.0/75000.0)
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Maximum Metrics: Maximum metrics at their respective thresholds
metricthresholdvalueidx
max f10.51016760.9569918214.0
max f20.30828060.9715081263.0
max f0point50.75326070.9592084147.0
max accuracy0.55617120.9564201.0
max precision0.99745651.00.0
max recall0.00160281.0399.0
max specificity0.99745651.00.0
max absolute_mcc0.55617120.9129277201.0
max min_per_class_accuracy0.62398530.9548073184.0
max mean_per_class_accuracy0.55617120.9563597201.0
max tns0.997456537329.00.0
max fns0.997456537460.00.0
max fps0.001602837329.0399.0
max tps0.001602837671.0399.0
max tnr0.99745651.00.0
max fnr0.99745650.99439890.0
max fpr0.00160281.0399.0
max tpr0.00160281.0399.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Gains/Lift Table: Avg response rate: 50.23 %, avg score: 50.22 %
groupcumulative_data_fractionlower_thresholdliftcumulative_liftresponse_ratescorecumulative_response_ratecumulative_scorecapture_ratecumulative_capture_rategaincumulative_gainkolmogorov_smirnov
10.01017330.99610971.99092141.99092141.00.99675161.00.99675160.02025430.020254399.092139899.09213980.0202543
20.02002670.99571531.99092141.99092141.00.99588941.00.99632740.01961720.039871599.092139899.09213980.0398715
30.030680.99539641.99092141.99092141.00.99554951.00.99605730.02120990.061081599.092139899.09213980.0610815
40.04010670.99519441.98810541.99025950.99858560.99527080.99966760.99587240.01874120.079822798.810538599.02595230.0797959
50.05177330.99492971.98637071.98938320.99771430.99500840.99922740.99567770.02317430.102997098.637072198.93832160.1029166
60.10001330.99330971.98817001.98879800.99861800.99416570.99893350.99494840.09590930.198906398.816998798.87980320.1986920
70.150080.99049841.98402871.98720700.99653790.99205770.99813430.99398410.09933370.298240098.402872898.72069930.2976775
80.20001330.98496051.98241551.98601080.99572760.98791250.99753350.99246830.09898860.397228698.241545998.60107860.3962375
90.300040.95610461.96730211.97977370.98813650.97403870.99440070.98632430.19678270.594011396.730209697.97737250.5906359
100.40001330.87706131.91232541.96291670.96052280.92100760.98593380.97000000.19118150.785192991.232540896.29167040.7738880
110.50001330.63834761.67635581.90560610.8420.80511860.95714780.93702460.16763560.952828467.635581790.56060550.9097788
120.60.07812450.42611401.65905690.21402850.28761850.83331110.82880470.04260570.9954342-57.388600665.90569230.7944912
130.70001330.01282220.02388791.42543460.01199840.03390010.71596730.71523390.00238910.9978233-97.611212842.54346480.5983483
140.800080.00579300.01220281.24868060.00612920.00841650.62718730.62683170.00122110.9990444-98.779715124.86805790.3997516
150.90.00344280.00451641.11055070.00226850.00452990.55780740.55774240.00045130.9994956-99.548363211.05507040.1999028
161.00.00016720.00504371.00.00253330.00262680.502280.50223080.00050441.0-99.49563320.00.0
\n", "
\n", "
\n", "
ModelMetricsBinomial: xgboost\n",
       "** Reported on validation data. **\n",
       "\n",
       "MSE: 0.038600168581279404\n",
       "RMSE: 0.1964692560714765\n",
       "LogLoss: 0.14264211397729937\n",
       "Mean Per-Class Error: 0.04874265748103474\n",
       "AUC: 0.9860367984758349\n",
       "AUCPR: 0.9841777019906762\n",
       "Gini: 0.9720735969516698
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.4899955771186135
01ErrorRate
011844.0840.00.0662 (840.0/12684.0)
1385.011931.00.0313 (385.0/12316.0)
Total12229.012771.00.049 (1225.0/25000.0)
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Maximum Metrics: Maximum metrics at their respective thresholds
metricthresholdvalueidx
max f10.48999560.9511699217.0
max f20.23816390.9684011281.0
max f0point50.76376010.9521032144.0
max accuracy0.48999560.951217.0
max precision0.99746831.00.0
max recall0.00183541.0398.0
max specificity0.99746831.00.0
max absolute_mcc0.48999560.9026291217.0
max min_per_class_accuracy0.63036380.9494966182.0
max mean_per_class_accuracy0.48999560.9512573217.0
max tns0.997468312684.00.0
max fns0.997468312244.00.0
max fps0.001343512684.0399.0
max tps0.001835412316.0398.0
max tnr0.99746831.00.0
max fnr0.99746830.99415390.0
max fpr0.00134351.0399.0
max tpr0.00183541.0398.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Gains/Lift Table: Avg response rate: 49.26 %, avg score: 49.47 %
groupcumulative_data_fractionlower_thresholdliftcumulative_liftresponse_ratescorecumulative_response_ratecumulative_scorecapture_ratecumulative_capture_rategaincumulative_gainkolmogorov_smirnov
10.010280.99612442.02987982.02987981.00.99673131.00.99673130.02086720.0208672102.9879831102.98798310.0208672
20.020320.99580122.02987982.02987981.00.99591751.00.99632920.02038000.0412472102.9879831102.98798310.0412472
30.030160.99546152.02162832.02718770.99593500.99561020.99867370.99609460.01989280.0611400102.1628287102.71876830.0610611
40.041080.99519442.02244442.02592680.99633700.99528870.99805260.99588040.02208510.0832251102.2444374102.59268030.0830674
50.052560.99492972.01573432.02370060.99303140.99500850.99695590.99569000.02314060.1063657101.5734327102.37005930.1060503
60.10.99328592.01961062.02176030.99494100.99415520.9960.99496190.09581030.2021760101.9610625102.17603120.2013876
70.150.99026942.02338422.02230160.99680.99194900.99626670.99395760.10116920.3033452102.3384216102.23016130.3022415
80.20.98406602.01364082.02013640.9920.98739450.99520.99231680.10068200.4040273101.3640792102.01364080.4021351
90.30.95238971.98522252.00849840.9780.97170510.98946670.98544620.19852220.602549598.5222475100.84984300.5963212
100.40.86710571.92351411.98725240.94760.91425670.9790.96764880.19235140.794900992.351412898.72523550.7783447
110.50.58223531.61740821.91328350.79680.78250590.942560.93062020.16174080.956641861.740824991.32835340.9000350
120.60.06677650.37755761.65732920.1860.24801800.81646670.81685320.03775580.9943975-62.244235165.73292190.7773524
130.70.01257710.01867491.42323570.00920.03044820.70114290.70450960.00186750.9962650-98.132510642.32357440.5839345
140.800480.00574120.01858571.24691750.00915610.00815740.61428140.61710020.00186750.9981325-98.141431724.69174880.3895705
150.90140.00345920.01045921.10848460.00515260.00452110.54608390.54851640.00105550.9991880-98.954084910.84846330.1927390
161.00.00026090.00823481.00.00405680.00264460.492640.49469340.00081201.0-99.17651930.00.0
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Scoring History:
timestampdurationnumber_of_treestraining_rmsetraining_loglosstraining_auctraining_pr_auctraining_lifttraining_classification_errortraining_customvalidation_rmsevalidation_loglossvalidation_aucvalidation_pr_aucvalidation_liftvalidation_classification_errorvalidation_custom
2022-10-28 14:19:33 0.030 sec0.0nannannannannannan0.693147nannannannannannan0.693147
2022-10-28 14:19:33 0.118 sec1.0nannannannannannan0.508456nannannannannannan0.509577
2022-10-28 14:19:33 0.127 sec2.0nannannannannannan0.403205nannannannannannan0.404315
2022-10-28 14:19:33 0.135 sec3.0nannannannannannan0.333565nannannannannannan0.335569
2022-10-28 14:19:33 0.144 sec4.0nannannannannannan0.286705nannannannannannan0.28943
2022-10-28 14:19:33 0.151 sec5.0nannannannannannan0.254899nannannannannannan0.257526
2022-10-28 14:19:33 0.161 sec6.0nannannannannannan0.232886nannannannannannan0.235729
2022-10-28 14:19:33 0.169 sec7.0nannannannannannan0.217304nannannannannannan0.220206
2022-10-28 14:19:33 0.177 sec8.0nannannannannannan0.200464nannannannannannan0.204054
2022-10-28 14:19:33 0.185 sec9.0nannannannannannan0.190888nannannannannannan0.195105
------------------------------------------------------
2022-10-28 14:19:33 0.224 sec13.0nannannannannannan0.161882nannannannannannan0.168298
2022-10-28 14:19:33 0.234 sec14.0nannannannannannan0.158723nannannannannannan0.165693
2022-10-28 14:19:33 0.243 sec15.0nannannannannannan0.155611nannannannannannan0.163391
2022-10-28 14:19:33 0.251 sec16.0nannannannannannan0.151794nannannannannannan0.160778
2022-10-28 14:19:33 0.261 sec17.0nannannannannannan0.147082nannannannannannan0.156551
2022-10-28 14:19:33 0.270 sec18.0nannannannannannan0.140969nannannannannannan0.151666
2022-10-28 14:19:33 0.279 sec19.0nannannannannannan0.139175nannannannannannan0.150583
2022-10-28 14:19:33 0.291 sec20.0nannannannannannan0.136121nannannannannannan0.148415
2022-10-28 14:19:33 0.301 sec21.0nannannannannannan0.131936nannannannannannan0.144895
2022-10-28 14:19:33 0.310 sec22.00.18643430.12901410.98933400.98870411.99092140.04381330.1290140.19646930.14264210.98603680.98417772.02987980.0490.142642
\n", "
\n", "
[23 rows x 18 columns]
\n", "
\n", " \n", "
\n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Variable Importances:
variablerelative_importancescaled_importancepercentage
C1170178.751.00.4261358
C235732.66406250.50916640.2169741
C1321608.48632810.30790640.1312100
C815654.54980470.22306680.0950568
C312402.26074220.17672390.0753084
C195739.76416020.08178780.0348527
C52835.91918950.04040990.0172201
C688.93524930.00126730.0005400
C1251.98764800.00074080.0003157
C1749.17110440.00070070.0002986
C748.05213170.00068470.0002918
C1446.54721830.00066330.0002826
C1043.87886050.00062520.0002664
C442.43465420.00060470.0002577
C1839.39529800.00056140.0002392
C132.55399320.00046390.0001977
C1525.95281600.00036980.0001576
C2023.11923220.00032940.0001404
C1622.99838070.00032770.0001396
C918.91131590.00026950.0001148
\n", "
\n", "
\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section.
" ], "text/plain": [ "Model Details\n", "=============\n", "H2OXGBoostEstimator : XGBoost\n", "Model Key: XGBoost_model_python_1666962310297_616\n", "\n", "\n", "Model Summary: \n", " number_of_trees\n", "-- -----------------\n", " 22\n", "\n", "ModelMetricsBinomial: xgboost\n", "** Reported on train data. **\n", "\n", "MSE: 0.03475774769265435\n", "RMSE: 0.18643429859511995\n", "LogLoss: 0.12901406890397824\n", "Mean Per-Class Error: 0.04387881533186782\n", "AUC: 0.9893340459497513\n", "AUCPR: 0.988704085388505\n", "Gini: 0.9786680918995025\n", "\n", "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.5101676438305829\n", " 0 1 Error Rate\n", "----- ----- ----- ------- ----------------\n", "0 35155 2174 0.0582 (2174.0/37329.0)\n", "1 1112 36559 0.0295 (1112.0/37671.0)\n", "Total 36267 38733 0.0438 (3286.0/75000.0)\n", "\n", "Maximum Metrics: Maximum metrics at their respective thresholds\n", "metric threshold value idx\n", "--------------------------- ----------- -------- -----\n", "max f1 0.510168 0.956992 214\n", "max f2 0.308281 0.971508 263\n", "max f0point5 0.753261 0.959208 147\n", "max accuracy 0.556171 0.9564 201\n", "max precision 0.997456 1 0\n", "max recall 0.00160282 1 399\n", "max specificity 0.997456 1 0\n", "max absolute_mcc 0.556171 0.912928 201\n", "max min_per_class_accuracy 0.623985 0.954807 184\n", "max mean_per_class_accuracy 0.556171 0.95636 201\n", "max tns 0.997456 37329 0\n", "max fns 0.997456 37460 0\n", "max fps 0.00160282 37329 399\n", "max tps 0.00160282 37671 399\n", "max tnr 0.997456 1 0\n", "max fnr 0.997456 0.994399 0\n", "max fpr 0.00160282 1 399\n", "max tpr 0.00160282 1 399\n", "\n", "Gains/Lift Table: Avg response rate: 50.23 %, avg score: 50.22 %\n", "group cumulative_data_fraction lower_threshold lift cumulative_lift response_rate score cumulative_response_rate cumulative_score capture_rate cumulative_capture_rate gain cumulative_gain kolmogorov_smirnov\n", "------- -------------------------- ----------------- ---------- ----------------- --------------- ---------- -------------------------- ------------------ -------------- ------------------------- -------- ----------------- --------------------\n", "1 0.0101733 0.99611 1.99092 1.99092 1 0.996752 1 0.996752 0.0202543 0.0202543 99.0921 99.0921 0.0202543\n", "2 0.0200267 0.995715 1.99092 1.99092 1 0.995889 1 0.996327 0.0196172 0.0398715 99.0921 99.0921 0.0398715\n", "3 0.03068 0.995396 1.99092 1.99092 1 0.995549 1 0.996057 0.0212099 0.0610815 99.0921 99.0921 0.0610815\n", "4 0.0401067 0.995194 1.98811 1.99026 0.998586 0.995271 0.999668 0.995872 0.0187412 0.0798227 98.8105 99.026 0.0797959\n", "5 0.0517733 0.99493 1.98637 1.98938 0.997714 0.995008 0.999227 0.995678 0.0231743 0.102997 98.6371 98.9383 0.102917\n", "6 0.100013 0.99331 1.98817 1.9888 0.998618 0.994166 0.998933 0.994948 0.0959093 0.198906 98.817 98.8798 0.198692\n", "7 0.15008 0.990498 1.98403 1.98721 0.996538 0.992058 0.998134 0.993984 0.0993337 0.29824 98.4029 98.7207 0.297677\n", "8 0.200013 0.98496 1.98242 1.98601 0.995728 0.987913 0.997533 0.992468 0.0989886 0.397229 98.2415 98.6011 0.396237\n", "9 0.30004 0.956105 1.9673 1.97977 0.988136 0.974039 0.994401 0.986324 0.196783 0.594011 96.7302 97.9774 0.590636\n", "10 0.400013 0.877061 1.91233 1.96292 0.960523 0.921008 0.985934 0.97 0.191182 0.785193 91.2325 96.2917 0.773888\n", "11 0.500013 0.638348 1.67636 1.90561 0.842 0.805119 0.957148 0.937025 0.167636 0.952828 67.6356 90.5606 0.909779\n", "12 0.6 0.0781245 0.426114 1.65906 0.214029 0.287618 0.833311 0.828805 0.0426057 0.995434 -57.3886 65.9057 0.794491\n", "13 0.700013 0.0128222 0.0238879 1.42543 0.0119984 0.0339001 0.715967 0.715234 0.00238911 0.997823 -97.6112 42.5435 0.598348\n", "14 0.80008 0.00579305 0.0122028 1.24868 0.00612925 0.00841651 0.627187 0.626832 0.0012211 0.999044 -98.7797 24.8681 0.399752\n", "15 0.9 0.00344276 0.00451637 1.11055 0.00226848 0.00452987 0.557807 0.557742 0.000451276 0.999496 -99.5484 11.0551 0.199903\n", "16 1 0.000167174 0.00504367 1 0.00253333 0.00262682 0.50228 0.502231 0.000504367 1 -99.4956 0 0\n", "\n", "ModelMetricsBinomial: xgboost\n", "** Reported on validation data. **\n", "\n", "MSE: 0.038600168581279404\n", "RMSE: 0.1964692560714765\n", "LogLoss: 0.14264211397729937\n", "Mean Per-Class Error: 0.04874265748103474\n", "AUC: 0.9860367984758349\n", "AUCPR: 0.9841777019906762\n", "Gini: 0.9720735969516698\n", "\n", "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.4899955771186135\n", " 0 1 Error Rate\n", "----- ----- ----- ------- ----------------\n", "0 11844 840 0.0662 (840.0/12684.0)\n", "1 385 11931 0.0313 (385.0/12316.0)\n", "Total 12229 12771 0.049 (1225.0/25000.0)\n", "\n", "Maximum Metrics: Maximum metrics at their respective thresholds\n", "metric threshold value idx\n", "--------------------------- ----------- -------- -----\n", "max f1 0.489996 0.95117 217\n", "max f2 0.238164 0.968401 281\n", "max f0point5 0.76376 0.952103 144\n", "max accuracy 0.489996 0.951 217\n", "max precision 0.997468 1 0\n", "max recall 0.00183535 1 398\n", "max specificity 0.997468 1 0\n", "max absolute_mcc 0.489996 0.902629 217\n", "max min_per_class_accuracy 0.630364 0.949497 182\n", "max mean_per_class_accuracy 0.489996 0.951257 217\n", "max tns 0.997468 12684 0\n", "max fns 0.997468 12244 0\n", "max fps 0.00134346 12684 399\n", "max tps 0.00183535 12316 398\n", "max tnr 0.997468 1 0\n", "max fnr 0.997468 0.994154 0\n", "max fpr 0.00134346 1 399\n", "max tpr 0.00183535 1 398\n", "\n", "Gains/Lift Table: Avg response rate: 49.26 %, avg score: 49.47 %\n", "group cumulative_data_fraction lower_threshold lift cumulative_lift response_rate score cumulative_response_rate cumulative_score capture_rate cumulative_capture_rate gain cumulative_gain kolmogorov_smirnov\n", "------- -------------------------- ----------------- ---------- ----------------- --------------- ---------- -------------------------- ------------------ -------------- ------------------------- -------- ----------------- --------------------\n", "1 0.01028 0.996124 2.02988 2.02988 1 0.996731 1 0.996731 0.0208672 0.0208672 102.988 102.988 0.0208672\n", "2 0.02032 0.995801 2.02988 2.02988 1 0.995918 1 0.996329 0.02038 0.0412472 102.988 102.988 0.0412472\n", "3 0.03016 0.995462 2.02163 2.02719 0.995935 0.99561 0.998674 0.996095 0.0198928 0.06114 102.163 102.719 0.0610611\n", "4 0.04108 0.995194 2.02244 2.02593 0.996337 0.995289 0.998053 0.99588 0.0220851 0.0832251 102.244 102.593 0.0830674\n", "5 0.05256 0.99493 2.01573 2.0237 0.993031 0.995009 0.996956 0.99569 0.0231406 0.106366 101.573 102.37 0.10605\n", "6 0.1 0.993286 2.01961 2.02176 0.994941 0.994155 0.996 0.994962 0.0958103 0.202176 101.961 102.176 0.201388\n", "7 0.15 0.990269 2.02338 2.0223 0.9968 0.991949 0.996267 0.993958 0.101169 0.303345 102.338 102.23 0.302241\n", "8 0.2 0.984066 2.01364 2.02014 0.992 0.987394 0.9952 0.992317 0.100682 0.404027 101.364 102.014 0.402135\n", "9 0.3 0.95239 1.98522 2.0085 0.978 0.971705 0.989467 0.985446 0.198522 0.60255 98.5222 100.85 0.596321\n", "10 0.4 0.867106 1.92351 1.98725 0.9476 0.914257 0.979 0.967649 0.192351 0.794901 92.3514 98.7252 0.778345\n", "11 0.5 0.582235 1.61741 1.91328 0.7968 0.782506 0.94256 0.93062 0.161741 0.956642 61.7408 91.3284 0.900035\n", "12 0.6 0.0667765 0.377558 1.65733 0.186 0.248018 0.816467 0.816853 0.0377558 0.994398 -62.2442 65.7329 0.777352\n", "13 0.7 0.0125771 0.0186749 1.42324 0.0092 0.0304482 0.701143 0.70451 0.00186749 0.996265 -98.1325 42.3236 0.583935\n", "14 0.80048 0.00574124 0.0185857 1.24692 0.00915605 0.00815736 0.614281 0.6171 0.00186749 0.998133 -98.1414 24.6917 0.389571\n", "15 0.9014 0.00345923 0.0104592 1.10848 0.0051526 0.0045211 0.546084 0.548516 0.00105554 0.999188 -98.9541 10.8485 0.192739\n", "16 1 0.000260908 0.00823481 1 0.0040568 0.00264459 0.49264 0.494693 0.000811952 1 -99.1765 0 0\n", "\n", "Scoring History: \n", " timestamp duration number_of_trees training_rmse training_logloss training_auc training_pr_auc training_lift training_classification_error training_custom validation_rmse validation_logloss validation_auc validation_pr_auc validation_lift validation_classification_error validation_custom\n", "--- ------------------- ---------- ----------------- ------------------- ------------------- ------------------ ----------------- ------------------ ------------------------------- ----------------- ------------------ -------------------- ------------------ ------------------- ----------------- --------------------------------- -------------------\n", " 2022-10-28 14:19:33 0.030 sec 0.0 nan nan nan nan nan nan 0.693147 nan nan nan nan nan nan 0.693147\n", " 2022-10-28 14:19:33 0.118 sec 1.0 nan nan nan nan nan nan 0.508456 nan nan nan nan nan nan 0.509577\n", " 2022-10-28 14:19:33 0.127 sec 2.0 nan nan nan nan nan nan 0.403205 nan nan nan nan nan nan 0.404315\n", " 2022-10-28 14:19:33 0.135 sec 3.0 nan nan nan nan nan nan 0.333565 nan nan nan nan nan nan 0.335569\n", " 2022-10-28 14:19:33 0.144 sec 4.0 nan nan nan nan nan nan 0.286705 nan nan nan nan nan nan 0.28943\n", " 2022-10-28 14:19:33 0.151 sec 5.0 nan nan nan nan nan nan 0.254899 nan nan nan nan nan nan 0.257526\n", " 2022-10-28 14:19:33 0.161 sec 6.0 nan nan nan nan nan nan 0.232886 nan nan nan nan nan nan 0.235729\n", " 2022-10-28 14:19:33 0.169 sec 7.0 nan nan nan nan nan nan 0.217304 nan nan nan nan nan nan 0.220206\n", " 2022-10-28 14:19:33 0.177 sec 8.0 nan nan nan nan nan nan 0.200464 nan nan nan nan nan nan 0.204054\n", " 2022-10-28 14:19:33 0.185 sec 9.0 nan nan nan nan nan nan 0.190888 nan nan nan nan nan nan 0.195105\n", "--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---\n", " 2022-10-28 14:19:33 0.224 sec 13.0 nan nan nan nan nan nan 0.161882 nan nan nan nan nan nan 0.168298\n", " 2022-10-28 14:19:33 0.234 sec 14.0 nan nan nan nan nan nan 0.158723 nan nan nan nan nan nan 0.165693\n", " 2022-10-28 14:19:33 0.243 sec 15.0 nan nan nan nan nan nan 0.155611 nan nan nan nan nan nan 0.163391\n", " 2022-10-28 14:19:33 0.251 sec 16.0 nan nan nan nan nan nan 0.151794 nan nan nan nan nan nan 0.160778\n", " 2022-10-28 14:19:33 0.261 sec 17.0 nan nan nan nan nan nan 0.147082 nan nan nan nan nan nan 0.156551\n", " 2022-10-28 14:19:33 0.270 sec 18.0 nan nan nan nan nan nan 0.140969 nan nan nan nan nan nan 0.151666\n", " 2022-10-28 14:19:33 0.279 sec 19.0 nan nan nan nan nan nan 0.139175 nan nan nan nan nan nan 0.150583\n", " 2022-10-28 14:19:33 0.291 sec 20.0 nan nan nan nan nan nan 0.136121 nan nan nan nan nan nan 0.148415\n", " 2022-10-28 14:19:33 0.301 sec 21.0 nan nan nan nan nan nan 0.131936 nan nan nan nan nan nan 0.144895\n", " 2022-10-28 14:19:33 0.310 sec 22.0 0.18643429859511995 0.12901406890397824 0.9893340459497513 0.988704085388505 1.9909213984231904 0.043813333333333336 0.129014 0.1964692560714765 0.14264211397729937 0.9860367984758349 0.9841777019906762 2.029879831113998 0.049 0.142642\n", "[23 rows x 18 columns]\n", "\n", "\n", "Variable Importances: \n", "variable relative_importance scaled_importance percentage\n", "---------- --------------------- ------------------- ------------\n", "C11 70178.8 1 0.426136\n", "C2 35732.7 0.509166 0.216974\n", "C13 21608.5 0.307906 0.13121\n", "C8 15654.5 0.223067 0.0950568\n", "C3 12402.3 0.176724 0.0753084\n", "C19 5739.76 0.0817878 0.0348527\n", "C5 2835.92 0.0404099 0.0172201\n", "C6 88.9352 0.00126727 0.000540028\n", "C12 51.9876 0.000740789 0.000315677\n", "C17 49.1711 0.000700655 0.000298574\n", "C7 48.0521 0.000684711 0.00029178\n", "C14 46.5472 0.000663267 0.000282642\n", "C10 43.8789 0.000625244 0.000266439\n", "C4 42.4347 0.000604665 0.00025767\n", "C18 39.3953 0.000561357 0.000239214\n", "C1 32.554 0.000463873 0.000197673\n", "C15 25.9528 0.00036981 0.000157589\n", "C20 23.1192 0.000329434 0.000140383\n", "C16 22.9984 0.000327711 0.00013965\n", "C9 18.9113 0.000269474 0.000114832\n", "\n", "[tips]\n", "Use `model.explain()` to inspect the model.\n", "--\n", "Use `h2o.display.toggle_user_tips()` to switch on/off this section." ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_eval_only = H2OXGBoostEstimator(ntrees=1000, max_depth=6, score_each_iteration=True, \n", " eval_metric=\"logloss\",\n", " stopping_rounds=3, stopping_tolerance=0.05, stopping_metric=\"custom\",\n", " score_eval_metric_only=True)\n", "model_eval_only.train(y=\"y\", training_frame=train, validation_frame=valid)" ] }, { "cell_type": "markdown", "id": "5967ea3d", "metadata": {}, "source": [ "Scoring history will show undefined values for H2O metrics for all scoring iteration except for the final one. Values of columns `training_custom` and `validation_custom` will be the only ones populated for all of the iterations.\n", "\n", "In the final iteration H2O performs full scoring, that is why we see all values defined in the last row of the scoring history." ] }, { "cell_type": "code", "execution_count": 10, "id": "7d99e876", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampdurationnumber_of_treestraining_rmsetraining_loglosstraining_auctraining_pr_auctraining_lifttraining_classification_errortraining_customvalidation_rmsevalidation_loglossvalidation_aucvalidation_pr_aucvalidation_liftvalidation_classification_errorvalidation_custom
02022-10-28 14:19:330.030 sec0.0NaNNaNNaNNaNNaNNaN0.693147NaNNaNNaNNaNNaNNaN0.693147
12022-10-28 14:19:330.118 sec1.0NaNNaNNaNNaNNaNNaN0.508456NaNNaNNaNNaNNaNNaN0.509577
22022-10-28 14:19:330.127 sec2.0NaNNaNNaNNaNNaNNaN0.403205NaNNaNNaNNaNNaNNaN0.404315
32022-10-28 14:19:330.135 sec3.0NaNNaNNaNNaNNaNNaN0.333565NaNNaNNaNNaNNaNNaN0.335569
42022-10-28 14:19:330.144 sec4.0NaNNaNNaNNaNNaNNaN0.286705NaNNaNNaNNaNNaNNaN0.289430
52022-10-28 14:19:330.151 sec5.0NaNNaNNaNNaNNaNNaN0.254899NaNNaNNaNNaNNaNNaN0.257526
62022-10-28 14:19:330.161 sec6.0NaNNaNNaNNaNNaNNaN0.232886NaNNaNNaNNaNNaNNaN0.235729
72022-10-28 14:19:330.169 sec7.0NaNNaNNaNNaNNaNNaN0.217304NaNNaNNaNNaNNaNNaN0.220206
82022-10-28 14:19:330.177 sec8.0NaNNaNNaNNaNNaNNaN0.200464NaNNaNNaNNaNNaNNaN0.204054
92022-10-28 14:19:330.185 sec9.0NaNNaNNaNNaNNaNNaN0.190888NaNNaNNaNNaNNaNNaN0.195105
102022-10-28 14:19:330.194 sec10.0NaNNaNNaNNaNNaNNaN0.182967NaNNaNNaNNaNNaNNaN0.187705
112022-10-28 14:19:330.202 sec11.0NaNNaNNaNNaNNaNNaN0.173463NaNNaNNaNNaNNaNNaN0.178748
122022-10-28 14:19:330.213 sec12.0NaNNaNNaNNaNNaNNaN0.167261NaNNaNNaNNaNNaNNaN0.172957
132022-10-28 14:19:330.224 sec13.0NaNNaNNaNNaNNaNNaN0.161882NaNNaNNaNNaNNaNNaN0.168298
142022-10-28 14:19:330.234 sec14.0NaNNaNNaNNaNNaNNaN0.158723NaNNaNNaNNaNNaNNaN0.165693
152022-10-28 14:19:330.243 sec15.0NaNNaNNaNNaNNaNNaN0.155611NaNNaNNaNNaNNaNNaN0.163391
162022-10-28 14:19:330.251 sec16.0NaNNaNNaNNaNNaNNaN0.151794NaNNaNNaNNaNNaNNaN0.160778
172022-10-28 14:19:330.261 sec17.0NaNNaNNaNNaNNaNNaN0.147082NaNNaNNaNNaNNaNNaN0.156551
182022-10-28 14:19:330.270 sec18.0NaNNaNNaNNaNNaNNaN0.140969NaNNaNNaNNaNNaNNaN0.151666
192022-10-28 14:19:330.279 sec19.0NaNNaNNaNNaNNaNNaN0.139175NaNNaNNaNNaNNaNNaN0.150583
202022-10-28 14:19:330.291 sec20.0NaNNaNNaNNaNNaNNaN0.136121NaNNaNNaNNaNNaNNaN0.148415
212022-10-28 14:19:330.301 sec21.0NaNNaNNaNNaNNaNNaN0.131936NaNNaNNaNNaNNaNNaN0.144895
222022-10-28 14:19:330.310 sec22.00.1864340.1290140.9893340.9887041.9909210.0438130.1290140.1964690.1426420.9860370.9841782.029880.0490.142642
\n", "
" ], "text/plain": [ " timestamp duration number_of_trees training_rmse \\\n", "0 2022-10-28 14:19:33 0.030 sec 0.0 NaN \n", "1 2022-10-28 14:19:33 0.118 sec 1.0 NaN \n", "2 2022-10-28 14:19:33 0.127 sec 2.0 NaN \n", "3 2022-10-28 14:19:33 0.135 sec 3.0 NaN \n", "4 2022-10-28 14:19:33 0.144 sec 4.0 NaN \n", "5 2022-10-28 14:19:33 0.151 sec 5.0 NaN \n", "6 2022-10-28 14:19:33 0.161 sec 6.0 NaN \n", "7 2022-10-28 14:19:33 0.169 sec 7.0 NaN \n", "8 2022-10-28 14:19:33 0.177 sec 8.0 NaN \n", "9 2022-10-28 14:19:33 0.185 sec 9.0 NaN \n", "10 2022-10-28 14:19:33 0.194 sec 10.0 NaN \n", "11 2022-10-28 14:19:33 0.202 sec 11.0 NaN \n", "12 2022-10-28 14:19:33 0.213 sec 12.0 NaN \n", "13 2022-10-28 14:19:33 0.224 sec 13.0 NaN \n", "14 2022-10-28 14:19:33 0.234 sec 14.0 NaN \n", "15 2022-10-28 14:19:33 0.243 sec 15.0 NaN \n", "16 2022-10-28 14:19:33 0.251 sec 16.0 NaN \n", "17 2022-10-28 14:19:33 0.261 sec 17.0 NaN \n", "18 2022-10-28 14:19:33 0.270 sec 18.0 NaN \n", "19 2022-10-28 14:19:33 0.279 sec 19.0 NaN \n", "20 2022-10-28 14:19:33 0.291 sec 20.0 NaN \n", "21 2022-10-28 14:19:33 0.301 sec 21.0 NaN \n", "22 2022-10-28 14:19:33 0.310 sec 22.0 0.186434 \n", "\n", " training_logloss training_auc training_pr_auc training_lift \\\n", "0 NaN NaN NaN NaN \n", "1 NaN NaN NaN NaN \n", "2 NaN NaN NaN NaN \n", "3 NaN NaN NaN NaN \n", "4 NaN NaN NaN NaN \n", "5 NaN NaN NaN NaN \n", "6 NaN NaN NaN NaN \n", "7 NaN NaN NaN NaN \n", "8 NaN NaN NaN NaN \n", "9 NaN NaN NaN NaN \n", "10 NaN NaN NaN NaN \n", "11 NaN NaN NaN NaN \n", "12 NaN NaN NaN NaN \n", "13 NaN NaN NaN NaN \n", "14 NaN NaN NaN NaN \n", "15 NaN NaN NaN NaN \n", "16 NaN NaN NaN NaN \n", "17 NaN NaN NaN NaN \n", "18 NaN NaN NaN NaN \n", "19 NaN NaN NaN NaN \n", "20 NaN NaN NaN NaN \n", "21 NaN NaN NaN NaN \n", "22 0.129014 0.989334 0.988704 1.990921 \n", "\n", " training_classification_error training_custom validation_rmse \\\n", "0 NaN 0.693147 NaN \n", "1 NaN 0.508456 NaN \n", "2 NaN 0.403205 NaN \n", "3 NaN 0.333565 NaN \n", "4 NaN 0.286705 NaN \n", "5 NaN 0.254899 NaN \n", "6 NaN 0.232886 NaN \n", "7 NaN 0.217304 NaN \n", "8 NaN 0.200464 NaN \n", "9 NaN 0.190888 NaN \n", "10 NaN 0.182967 NaN \n", "11 NaN 0.173463 NaN \n", "12 NaN 0.167261 NaN \n", "13 NaN 0.161882 NaN \n", "14 NaN 0.158723 NaN \n", "15 NaN 0.155611 NaN \n", "16 NaN 0.151794 NaN \n", "17 NaN 0.147082 NaN \n", "18 NaN 0.140969 NaN \n", "19 NaN 0.139175 NaN \n", "20 NaN 0.136121 NaN \n", "21 NaN 0.131936 NaN \n", "22 0.043813 0.129014 0.196469 \n", "\n", " validation_logloss validation_auc validation_pr_auc validation_lift \\\n", "0 NaN NaN NaN NaN \n", "1 NaN NaN NaN NaN \n", "2 NaN NaN NaN NaN \n", "3 NaN NaN NaN NaN \n", "4 NaN NaN NaN NaN \n", "5 NaN NaN NaN NaN \n", "6 NaN NaN NaN NaN \n", "7 NaN NaN NaN NaN \n", "8 NaN NaN NaN NaN \n", "9 NaN NaN NaN NaN \n", "10 NaN NaN NaN NaN \n", "11 NaN NaN NaN NaN \n", "12 NaN NaN NaN NaN \n", "13 NaN NaN NaN NaN \n", "14 NaN NaN NaN NaN \n", "15 NaN NaN NaN NaN \n", "16 NaN NaN NaN NaN \n", "17 NaN NaN NaN NaN \n", "18 NaN NaN NaN NaN \n", "19 NaN NaN NaN NaN \n", "20 NaN NaN NaN NaN \n", "21 NaN NaN NaN NaN \n", "22 0.142642 0.986037 0.984178 2.02988 \n", "\n", " validation_classification_error validation_custom \n", "0 NaN 0.693147 \n", "1 NaN 0.509577 \n", "2 NaN 0.404315 \n", "3 NaN 0.335569 \n", "4 NaN 0.289430 \n", "5 NaN 0.257526 \n", "6 NaN 0.235729 \n", "7 NaN 0.220206 \n", "8 NaN 0.204054 \n", "9 NaN 0.195105 \n", "10 NaN 0.187705 \n", "11 NaN 0.178748 \n", "12 NaN 0.172957 \n", "13 NaN 0.168298 \n", "14 NaN 0.165693 \n", "15 NaN 0.163391 \n", "16 NaN 0.160778 \n", "17 NaN 0.156551 \n", "18 NaN 0.151666 \n", "19 NaN 0.150583 \n", "20 NaN 0.148415 \n", "21 NaN 0.144895 \n", "22 0.049 0.142642 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_eval_only.scoring_history()" ] }, { "cell_type": "markdown", "id": "037ab44c", "metadata": {}, "source": [ "Models `model_eval` and `model_eval_only` are guaranteed to be identical in behavior (same trees, same thresholds,..). The only technical difference between them is that the first one doesn't have full scoring history.\n", "\n", "We can also see that flag `score_eval_metric_only=True` saved us some training time. Model `model_eval_only` was built faster:" ] }, { "cell_type": "code", "execution_count": 11, "id": "b74d7bad", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Duration (s) with H2O scoring: 21.281\n", "Duration (s) with only eval_metric scored: 4.67\n" ] } ], "source": [ "def total_duration(scoring_history):\n", " return(sum(map(lambda x: float(x.strip().split(' ')[0]), scoring_history[\"duration\"].tolist())))\n", "\n", "\n", "print(\"Duration (s) with H2O scoring: %s\" % total_duration(model_eval.scoring_history()))\n", "print(\"Duration (s) with only eval_metric scored: %s\" % total_duration(model_eval_only.scoring_history()))\n" ] }, { "cell_type": "code", "execution_count": 12, "id": "5935b7cb", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 0.025 sec\n", "1 0.139 sec\n", "2 0.179 sec\n", "3 0.228 sec\n", "4 0.280 sec\n", "5 0.356 sec\n", "6 0.434 sec\n", "7 0.511 sec\n", "8 0.587 sec\n", "9 0.676 sec\n", "10 0.773 sec\n", "11 0.854 sec\n", "12 0.943 sec\n", "13 1.058 sec\n", "14 1.160 sec\n", "15 1.246 sec\n", "16 1.345 sec\n", "17 1.462 sec\n", "18 1.566 sec\n", "19 1.684 sec\n", "20 1.802 sec\n", "21 1.916 sec\n", "22 2.057 sec\n", "Name: duration, dtype: object" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_eval.scoring_history()[\"duration\"]" ] }, { "cell_type": "code", "execution_count": 13, "id": "b7b5211d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 0.030 sec\n", "1 0.118 sec\n", "2 0.127 sec\n", "3 0.135 sec\n", "4 0.144 sec\n", "5 0.151 sec\n", "6 0.161 sec\n", "7 0.169 sec\n", "8 0.177 sec\n", "9 0.185 sec\n", "10 0.194 sec\n", "11 0.202 sec\n", "12 0.213 sec\n", "13 0.224 sec\n", "14 0.234 sec\n", "15 0.243 sec\n", "16 0.251 sec\n", "17 0.261 sec\n", "18 0.270 sec\n", "19 0.279 sec\n", "20 0.291 sec\n", "21 0.301 sec\n", "22 0.310 sec\n", "Name: duration, dtype: object" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_eval_only.scoring_history()[\"duration\"]" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" } }, "nbformat": 4, "nbformat_minor": 5 }