{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Rulefit demo - Titanic Dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## H2O Rulefit algorithm\n", "\n", "Rulefit algorithm combines tree ensembles and linear models to take advantage of both methods: a tree ensemble accuracy and a linear model interpretability. The general algorithm fits a tree ensebmle to the data, builds a rule ensemble by traversing each tree, evaluates the rules on the data to build a rule feature set and fits a sparse linear model (LASSO) to the rule feature set joined with the original feature set.\n", "\n", "For more information, refer to: http://statweb.stanford.edu/~jhf/ftp/RuleFit.pdf by Jerome H. Friedman and Bogden E. Popescu.\n", "\n", "## Demo example\n", "\n", "We will train a rulefit model to predict the rules defining whether or not someone will survive:\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Checking whether there is an H2O instance running at http://localhost:54321 . connected.\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
H2O_cluster_uptime:4 mins 19 secs
H2O_cluster_timezone:Europe/Prague
H2O_data_parsing_timezone:UTC
H2O_cluster_version:3.34.0.99999
H2O_cluster_version_age:17 minutes
H2O_cluster_name:zuzanaolajcova
H2O_cluster_total_nodes:1
H2O_cluster_free_memory:3.546 Gb
H2O_cluster_total_cores:12
H2O_cluster_allowed_cores:12
H2O_cluster_status:locked, healthy
H2O_connection_url:http://localhost:54321
H2O_connection_proxy:{\"http\": null, \"https\": null}
H2O_internal_security:False
H2O_API_Extensions:Algos, AutoML, Core V3, TargetEncoder, Core V4
Python_version:3.8.1 final
" ], "text/plain": [ "-------------------------- ----------------------------------------------\n", "H2O_cluster_uptime: 4 mins 19 secs\n", "H2O_cluster_timezone: Europe/Prague\n", "H2O_data_parsing_timezone: UTC\n", "H2O_cluster_version: 3.34.0.99999\n", "H2O_cluster_version_age: 17 minutes\n", "H2O_cluster_name: zuzanaolajcova\n", "H2O_cluster_total_nodes: 1\n", "H2O_cluster_free_memory: 3.546 Gb\n", "H2O_cluster_total_cores: 12\n", "H2O_cluster_allowed_cores: 12\n", "H2O_cluster_status: locked, healthy\n", "H2O_connection_url: http://localhost:54321\n", "H2O_connection_proxy: {\"http\": null, \"https\": null}\n", "H2O_internal_security: False\n", "H2O_API_Extensions: Algos, AutoML, Core V3, TargetEncoder, Core V4\n", "Python_version: 3.8.1 final\n", "-------------------------- ----------------------------------------------" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import h2o\n", "from h2o.estimators import H2ORuleFitEstimator, H2ORandomForestEstimator\n", "\n", "# init h2o cluster\n", "h2o.init()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n" ] } ], "source": [ "df = h2o.import_file(\"https://s3.amazonaws.com/h2o-public-test-data/smalldata/gbm_test/titanic.csv\",\n", " col_types={'pclass': \"enum\", 'survived': \"enum\"})\n", "x = [\"age\", \"sibsp\", \"parch\", \"sex\", \"pclass\"]\n", "\n", "# Split the dataset into train and test\n", "train, test = df.split_frame(ratios=[.8], seed=1234)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the `algorithm` parameter, a user can set whether algorithm will use DRF or GBM to fit a tree enseble. \n", "\n", "Using the `min_rule_length` and `max_rule_length` parameters, a user can set interval of tree enseble depths to be fitted. The bigger this interval is, the more tree ensembles will be fitted (1 per each depth) and the bigger the rule feature set will be.\n", "\n", "Using the `max_num_rules` parameter, the maximum number of rules to return can be set.\n", "\n", "Using the `model_type` parameter, the type of base learners in the enseble can be set.\n", "\n", "Using the `rule_generation_ntrees` parameter, the number of trees for tree enseble can be set." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rulefit Model Build progress: |██████████████████████████████████████████████████| (done) 100%\n", "Model Details\n", "=============\n", "H2ORuleFitEstimator : RuleFit\n", "Model Key: RuleFit_model_python_1636562504000_1\n", "\n", "\n", "Rulefit Model Summary: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
familylinkregularizationnumber_of_predictors_totalnumber_of_active_predictorsnumber_of_iterationsrule_ensemble_sizenumber_of_treesnumber_of_internal_treesmin_depthmax_depthmean_depthmin_leavesmax_leavesmean_leaves
0binomiallogitLasso (lambda = 0.01292 )207848320776.0500.0500.00.010.05.50.0135.041.552
\n", "
" ], "text/plain": [ " family link regularization number_of_predictors_total \\\n", "0 binomial logit Lasso (lambda = 0.01292 ) 20784 \n", "\n", " number_of_active_predictors number_of_iterations rule_ensemble_size \\\n", "0 8 3 20776.0 \n", "\n", " number_of_trees number_of_internal_trees min_depth max_depth \\\n", "0 500.0 500.0 0.0 10.0 \n", "\n", " mean_depth min_leaves max_leaves mean_leaves \n", "0 5.5 0.0 135.0 41.552 " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "ModelMetricsBinomialGLM: rulefit\n", "** Reported on train data. **\n", "\n", "MSE: 0.14668202166384883\n", "RMSE: 0.3829908897922362\n", "LogLoss: 0.4616331658988569\n", "Null degrees of freedom: 1053\n", "Residual degrees of freedom: 1045\n", "Null deviance: 1405.0919048764067\n", "Residual deviance: 973.1227137147903\n", "AIC: 991.1227137147903\n", "AUC: 0.8361042692939246\n", "AUCPR: 0.7904193564939762\n", "Gini: 0.6722085385878491\n", "\n", "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.44132286664639514: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01ErrorRate
00526.0122.00.1883(122.0/648.0)
11106.0300.00.2611(106.0/406.0)
2Total632.0422.00.2163(228.0/1054.0)
\n", "
" ], "text/plain": [ " 0 1 Error Rate\n", "0 0 526.0 122.0 0.1883 (122.0/648.0)\n", "1 1 106.0 300.0 0.2611 (106.0/406.0)\n", "2 Total 632.0 422.0 0.2163 (228.0/1054.0)" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Maximum Metrics: Maximum metrics at their respective thresholds\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
metricthresholdvalueidx
0max f10.4413230.7246383.0
1max f20.1600330.7838327.0
2max f0point50.8090130.7744781.0
3max accuracy0.5238050.7903232.0
4max precision0.8090130.9190481.0
5max recall0.1563081.0000008.0
6max specificity0.8550410.9737650.0
7max absolute_mcc0.5238050.5509682.0
8max min_per_class_accuracy0.4413230.7389163.0
9max mean_per_class_accuracy0.4413230.7753223.0
10max tns0.855041631.0000000.0
11max fns0.855041217.0000000.0
12max fps0.156308648.0000008.0
13max tps0.156308406.0000008.0
14max tnr0.8550410.9737650.0
15max fnr0.8550410.5344830.0
16max fpr0.1563081.0000008.0
17max tpr0.1563081.0000008.0
\n", "
" ], "text/plain": [ " metric threshold value idx\n", "0 max f1 0.441323 0.724638 3.0\n", "1 max f2 0.160033 0.783832 7.0\n", "2 max f0point5 0.809013 0.774478 1.0\n", "3 max accuracy 0.523805 0.790323 2.0\n", "4 max precision 0.809013 0.919048 1.0\n", "5 max recall 0.156308 1.000000 8.0\n", "6 max specificity 0.855041 0.973765 0.0\n", "7 max absolute_mcc 0.523805 0.550968 2.0\n", "8 max min_per_class_accuracy 0.441323 0.738916 3.0\n", "9 max mean_per_class_accuracy 0.441323 0.775322 3.0\n", "10 max tns 0.855041 631.000000 0.0\n", "11 max fns 0.855041 217.000000 0.0\n", "12 max fps 0.156308 648.000000 8.0\n", "13 max tps 0.156308 406.000000 8.0\n", "14 max tnr 0.855041 0.973765 0.0\n", "15 max fnr 0.855041 0.534483 0.0\n", "16 max fpr 0.156308 1.000000 8.0\n", "17 max tpr 0.156308 1.000000 8.0" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Gains/Lift Table: Avg response rate: 38.52 %, avg score: 38.52 %\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
groupcumulative_data_fractionlower_thresholdliftcumulative_liftresponse_ratescorecumulative_response_ratecumulative_scorecapture_ratecumulative_capture_rategaincumulative_gainkolmogorov_smirnov
010.1954460.8550412.3818212.3818210.9174760.8550410.9174760.8550410.4655170.465517138.182123138.1821230.439283
120.3481970.5238051.4028391.9523500.5403730.5308910.7520440.7128390.2142860.67980340.28394095.2349630.539371
230.4003800.4145251.1328261.8455400.4363640.4413230.7109000.6774520.0591130.73891613.28257984.5539650.550645
340.5284630.3073350.7884331.5893290.3037040.3073350.6122080.5877460.1009850.839901-21.15672358.9328830.506568
451.0000000.1563080.3395251.0000000.1307850.1582080.3851990.3852030.1600991.000000-66.0475170.0000000.000000
\n", "
" ], "text/plain": [ " group cumulative_data_fraction lower_threshold lift \\\n", "0 1 0.195446 0.855041 2.381821 \n", "1 2 0.348197 0.523805 1.402839 \n", "2 3 0.400380 0.414525 1.132826 \n", "3 4 0.528463 0.307335 0.788433 \n", "4 5 1.000000 0.156308 0.339525 \n", "\n", " cumulative_lift response_rate score cumulative_response_rate \\\n", "0 2.381821 0.917476 0.855041 0.917476 \n", "1 1.952350 0.540373 0.530891 0.752044 \n", "2 1.845540 0.436364 0.441323 0.710900 \n", "3 1.589329 0.303704 0.307335 0.612208 \n", "4 1.000000 0.130785 0.158208 0.385199 \n", "\n", " cumulative_score capture_rate cumulative_capture_rate gain \\\n", "0 0.855041 0.465517 0.465517 138.182123 \n", "1 0.712839 0.214286 0.679803 40.283940 \n", "2 0.677452 0.059113 0.738916 13.282579 \n", "3 0.587746 0.100985 0.839901 -21.156723 \n", "4 0.385203 0.160099 1.000000 -66.047517 \n", "\n", " cumulative_gain kolmogorov_smirnov \n", "0 138.182123 0.439283 \n", "1 95.234963 0.539371 \n", "2 84.553965 0.550645 \n", "3 58.932883 0.506568 \n", "4 0.000000 0.000000 " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rfit = H2ORuleFitEstimator(algorithm=\"drf\", \n", " min_rule_length=1, \n", " max_rule_length=10, \n", " max_num_rules=100, \n", " model_type=\"rules_and_linear\",\n", " rule_generation_ntrees=50,\n", " seed=1234)\n", "rfit.train(training_frame=train, x=x, y=\"survived\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output for the Rulefit model includes:\n", " - model parameters\n", " - rule importences in tabular form\n", " - training and validation metrics of the underlying linear model" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Rule Importance: \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
variablecoefficientrule
0M2T21N131.298409e+00(sex in {female}) & (sibsp < 3.5 or sibsp is NA) & (pclass in {1, ...
1M2T23N21-8.453746e-01(sex in {male} or sex is NA) & (pclass in {2, 3} or pclass is NA) ...
2M1T0N73.809983e-01(pclass in {1, 2}) & (sex in {female})
3M1T28N10-3.448192e-01(sex in {male} or sex is NA) & (age >= 13.496771812438965 or age i...
4M1T23N73.310857e-01(sex in {female}) & (sibsp < 2.5 or sibsp is NA)
5M1T37N10-2.319945e-01(sex in {male} or sex is NA) & (age >= 14.977890968322754 or age i...
6M4T3N45-2.797404e-02(sex in {male} or sex is NA) & (pclass in {2, 3} or pclass is NA) ...
7M1T1N72.887806e-14(pclass in {1, 2}) & (sex in {female})
\n", "
" ], "text/plain": [ " variable coefficient \\\n", "0 M2T21N13 1.298409e+00 \n", "1 M2T23N21 -8.453746e-01 \n", "2 M1T0N7 3.809983e-01 \n", "3 M1T28N10 -3.448192e-01 \n", "4 M1T23N7 3.310857e-01 \n", "5 M1T37N10 -2.319945e-01 \n", "6 M4T3N45 -2.797404e-02 \n", "7 M1T1N7 2.887806e-14 \n", "\n", " rule \n", "0 (sex in {female}) & (sibsp < 3.5 or sibsp is NA) & (pclass in {1, ... \n", "1 (sex in {male} or sex is NA) & (pclass in {2, 3} or pclass is NA) ... \n", "2 (pclass in {1, 2}) & (sex in {female}) \n", "3 (sex in {male} or sex is NA) & (age >= 13.496771812438965 or age i... \n", "4 (sex in {female}) & (sibsp < 2.5 or sibsp is NA) \n", "5 (sex in {male} or sex is NA) & (age >= 14.977890968322754 or age i... \n", "6 (sex in {male} or sex is NA) & (pclass in {2, 3} or pclass is NA) ... \n", "7 (pclass in {1, 2}) & (sex in {female}) " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from IPython.display import display\n", "display(rfit.rule_importance())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are several rules that can be recapped as:\n", "\n", "### Higgest Likelihood of Survival:\n", "1. women in class 1 or 2 with 3 siblings/spouses aboard or less\n", "2. women in class 1 or 2\n", "3. women with 2 siblings/spouses aboard or less\n", "\n", "### Lowest Likelihood of Survival:\n", "1. male in class 2 or 3 of age >= 9.4\n", "2. male of age >= 13.4\n", "3. male of age >= 14.8\n", "4. male in class 2 or 3 with no parents/children aboard of age between 14 to 61\n", "\n", "Note: The rules are additive. That means that if a passenger is described by multiple rules, their probability is added together from those rules." ] } ], "metadata": { "kernelspec": { "display_name": "h2o3pyenv", "language": "python", "name": "h2o3pyenv" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.1" } }, "nbformat": 4, "nbformat_minor": 2 }