{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# H2O Uplift Distributed Random Forest \n",
"\n",
"### Author: Veronika Maurerova veronika.maurerova@h2o.ai\n",
"\n",
"## Modeling Uplift \n",
"\n",
"Distributed Uplift Random Forest (Uplift DRF) is a classification tool for modeling uplift - the incremental impact of a treatment. This tool is very useful for example in marketing or in medicine. This machine learning approach is inspired by the A/B testing method. \n",
"\n",
"To model uplift, the analyst needs to collect data specifically - before the experiment, the objects are divided usually into two groups: \n",
"\n",
"- **treatment group**: receive some kind of treatment (for example customer get some type of discount) \n",
"- **control group**: is separated from the treatment (customers in this group get no discount). \n",
"\n",
"Then the data are prepared and an analyst can gather information about the response - for example, whether customers bought a product, patients recovered from the disease, or similar. \n",
"\n",
"## Uplift approaches \n",
"\n",
"There are several approaches to model uplift: \n",
"\n",
"- Meta-learner algorithms\n",
"- Instrumental variables algorithms\n",
"- Neural-networks-based algorithms\n",
"- Tree-based algorithms \n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tree Based Uplift Algorithm\n",
"\n",
"Tree-based algorithm means in every tree, it takes information about treatment/control group assignment and information about response directly into a decision about splitting a node. The uplift score is the criterion to make a decision similar to the Gini coefficient in the standard decision tree. \n",
"\n",
"**Uplift metric to decide best split**\n",
"\n",
"The goal is to maximize the differences between the class distributions in the treatment and control sets, so the splitting criteria are based on distribution divergences. The distribution divergence is calculated based on the ``uplift_metric`` parameter. In H2O-3, three ``uplift_metric`` types are supported:\n",
"\n",
"- **Kullback-Leibler divergence** (``uplift_metric=\"KL\"``) - uses logarithms to calculate divergence, asymmetric, widely used, tends to infinity values (if treatment or control group distributions contain zero values). \n",
"\n",
"$ KL(P, Q) = \\sum_{i=0}^{N} p_i \\log{\\frac{p_i}{q_i}}$\n",
"\n",
"- **Squared Euclidean distance** (``uplift_metric=\"euclidean\"``) - symmetric and stable distribution, does not tend to infinity values. \n",
"\n",
"$ E(P, Q) = \\sum_{i=0}^{N} (p_i-q_i)^2$\n",
"\n",
"\n",
"- **Chi-squared divergence** (``uplift_metric=\"chi_squared\"``) - Euclidean divergence normalized by control group distribution. Asymmetric and also tends to infinity values (if control group distribution contains zero values). \n",
"\n",
"$X^2(P, Q) = \\sum_{i=0}^{N} \\frac{(p_i-q_i)^2}{q_i}$\n",
"\n",
"where:\n",
"\n",
"- $P$ is treatment group distribution\n",
"\n",
"- $Q$ is control group distribution\n",
"\n",
"In a tree node the result value for a split is sum: $metric(P, Q) + metric(1-P, 1-Q)$. \n",
"\n",
"For the split gain value, the result within the node is normalized using a Gini coefficient (Eclidean or ChiSquared) or entropy (KL) for each distribution before and after the split.\n",
"\n",
"\n",
"**Uplift score in each leaf is calculated as:**\n",
"\n",
"- $TP = (TY1 + 1) / (T + 2)$\n",
"- $CP = (CY1 + 1) / (C + 2)$\n",
"- $uplift\\_score = TP - CP $\n",
"\n",
"where:\n",
"- $T$ how many observations in a leaf are from the treatment group (how many data rows in a leaf have ``treatment_column`` label == 1) \n",
"- $C$ how many observations in a leaf are from the control group (how many data rows in the leaf have ``treatment_column`` label == 0)\n",
"- $TY1$ how many observations in a leaf are from the treatment group and respond to the offer (how many data rows in the leaf have ``treatment_column`` label == 1 and ``response_column`` label == 1)\n",
"- $CY1$ how many observations in a leaf are from the control group and respond to the offer (how many data rows in the leaf have ``treatment_column`` label == 0 and ``response_column`` label == 1)\n",
"\n",
"**Note**: A higher uplift score means more observations from treatment group respond to the offer than from control group. Which means offered treatment has positive effect. The uplift score can be negative, if more observations from control group respond to the offer without treatment.\n",
"\n",
" \n",
" \n",
"\n",
"![Difference between SDT and Uplift DT](https://blog.h2o.ai/wp-content/uploads/2022/01/tree.png)\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## H2O Implementation (Major release 3.36)\n",
"\n",
"The H2O-3 implementation of Uplift DRF is based on DRF because the principle of training is similar to DRF. It is tree based uplift algorithm. Uplift DRF generates a forest of classification uplift trees, rather than a single classification tree. Each of these trees is a weak learner built on a subset of rows and columns. More trees will reduce the variance. Classification takes the average prediction over all of their trees to make a final prediction. \n",
"\n",
"Currently, in H2O-3 only binomial trees are supported, as well as the uplift curve metric and Area Under Uplift curve (AUUC) metric, normalized AUUC, and the Qini value. We are working on adding also regression trees and more metrics, for example, Qini coefficient, and more. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Start H2O-3"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"versionFromGradle='3.37.0',projectVersion='3.37.0.99999',branch='master',lastCommitHash='a1c95a407aec53a6cbc551484bd02d7d80b3bcb6',gitDescribe='jenkins-master-5950-dirty',compiledOn='2022-09-13 10:48:53',compiledBy='kurkami'\n"
]
}
],
"source": [
"import h2o\n",
"from h2o.estimators.uplift_random_forest import H2OUpliftRandomForestEstimator\n",
"\n",
"import matplotlib as mpl\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.style as style\n",
"\n",
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"To demonstrate how Uplift DRF works, Criteo dataset is used. \n",
"\n",
"**Source:**\n",
"\n",
"Diemert Eustache, Betlei Artem} and Renaudin, Christophe and Massih-Reza, Amini, \"A Large Scale Benchmark for Uplift Modeling\", ACM, Proceedings of the AdKDD and TargetAd Workshop, KDD, London,United Kingdom, August, 20, 2018, https://ailab.criteo.com/criteo-uplift-prediction-dataset/.\n",
"\n",
"\n",
"\n",
"**Description:**\n",
"\n",
"- The dataset was created by The Criteo AI Lab\n",
"- Consists of 13M rows, each one representing a user with 12 features, a treatment indicator and 2 binary labels (visits and conversions).\n",
"- Positive labels mean the user visited/converted on the advertiser website during the test period (2 weeks).\n",
"- The global treatment ratio is 84.6%.\n",
"\n",
"**Detailed description of the columns:**\n",
"\n",
"- **f0, f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11**: feature values (dense, float)\n",
"- **treatment**: treatment group (1 = treated, 0 = control)\n",
"- **conversion**: whether a conversion occured for this user (binary, label)\n",
"- **visit**: whether a visit occured for this user (binary, label)\n",
"- **exposure**: treatment effect, whether the user has been effectively exposed (binary)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
f0
\n",
"
f1
\n",
"
f2
\n",
"
f3
\n",
"
f4
\n",
"
f5
\n",
"
f6
\n",
"
f7
\n",
"
f8
\n",
"
f9
\n",
"
f10
\n",
"
f11
\n",
"
treatment
\n",
"
conversion
\n",
"
visit
\n",
"
exposure
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
12.616365
\n",
"
10.059654
\n",
"
8.976429
\n",
"
4.679882
\n",
"
10.280525
\n",
"
4.115453
\n",
"
0.294443
\n",
"
4.833815
\n",
"
3.955396
\n",
"
13.190056
\n",
"
5.300375
\n",
"
-0.168679
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
0
\n",
"
\n",
"
\n",
"
1
\n",
"
12.616365
\n",
"
10.059654
\n",
"
9.002689
\n",
"
4.679882
\n",
"
10.280525
\n",
"
4.115453
\n",
"
0.294443
\n",
"
4.833815
\n",
"
3.955396
\n",
"
13.190056
\n",
"
5.300375
\n",
"
-0.168679
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
0
\n",
"
\n",
"
\n",
"
2
\n",
"
12.616365
\n",
"
10.059654
\n",
"
8.964775
\n",
"
4.679882
\n",
"
10.280525
\n",
"
4.115453
\n",
"
0.294443
\n",
"
4.833815
\n",
"
3.955396
\n",
"
13.190056
\n",
"
5.300375
\n",
"
-0.168679
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
0
\n",
"
\n",
"
\n",
"
3
\n",
"
12.616365
\n",
"
10.059654
\n",
"
9.002801
\n",
"
4.679882
\n",
"
10.280525
\n",
"
4.115453
\n",
"
0.294443
\n",
"
4.833815
\n",
"
3.955396
\n",
"
13.190056
\n",
"
5.300375
\n",
"
-0.168679
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
0
\n",
"
\n",
"
\n",
"
4
\n",
"
12.616365
\n",
"
10.059654
\n",
"
9.037999
\n",
"
4.679882
\n",
"
10.280525
\n",
"
4.115453
\n",
"
0.294443
\n",
"
4.833815
\n",
"
3.955396
\n",
"
13.190056
\n",
"
5.300375
\n",
"
-0.168679
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
0
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" f0 f1 f2 f3 f4 f5 f6 \\\n",
"0 12.616365 10.059654 8.976429 4.679882 10.280525 4.115453 0.294443 \n",
"1 12.616365 10.059654 9.002689 4.679882 10.280525 4.115453 0.294443 \n",
"2 12.616365 10.059654 8.964775 4.679882 10.280525 4.115453 0.294443 \n",
"3 12.616365 10.059654 9.002801 4.679882 10.280525 4.115453 0.294443 \n",
"4 12.616365 10.059654 9.037999 4.679882 10.280525 4.115453 0.294443 \n",
"\n",
" f7 f8 f9 f10 f11 treatment conversion \\\n",
"0 4.833815 3.955396 13.190056 5.300375 -0.168679 1 0 \n",
"1 4.833815 3.955396 13.190056 5.300375 -0.168679 1 0 \n",
"2 4.833815 3.955396 13.190056 5.300375 -0.168679 1 0 \n",
"3 4.833815 3.955396 13.190056 5.300375 -0.168679 1 0 \n",
"4 4.833815 3.955396 13.190056 5.300375 -0.168679 1 0 \n",
"\n",
" visit exposure \n",
"0 0 0 \n",
"1 0 0 \n",
"2 0 0 \n",
"3 0 0 \n",
"4 0 0 "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"control_name = \"control\"\n",
"treatment_column = \"treatment\"\n",
"response_column = \"visit\"\n",
"feature_cols = [\"f\"+str(x) for x in range(0,12)]\n",
"\n",
"df = pd.read_csv(\"/home/0xdiag/bigdata/server/criteo/criteo-uplift-v2.1.csv\")\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prepare data\n",
"\n",
"Inspiration from: https://www.kaggle.com/code/hughhuyton/criteo-uplift-modelling/notebook\n",
"\n",
"To modeling uplift the treatment and control group data have to have similar distribution. In real world usually the control group is smaller than the treatment group. It is also a case of Crieteo dataset and we have to rebalanced the data to have a similar size."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of samples: 13979592\n",
"The dataset is largely imbalanced: \n",
"1 0.85\n",
"0 0.15\n",
"Name: treatment, dtype: float64\n",
"Percentage of users that visit: 4.7%\n",
"Percentage of users that convert: 0.29%\n",
"Percentage of visitors that convert: 6.21%\n"
]
}
],
"source": [
"print('Total number of samples: {}'.format(len(df)))\n",
"print('The dataset is largely imbalanced: ')\n",
"print(df['treatment'].value_counts(normalize = True))\n",
"print('Percentage of users that visit: {}%'.format(100*round(df['visit'].mean(),4)))\n",
"print('Percentage of users that convert: {}%'.format(100*round(df['conversion'].mean(),4)))\n",
"print('Percentage of visitors that convert: {}%'.format(100*round(df[df[\"visit\"]==1][\"conversion\"].mean(),4)))"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Class 0: 2096937\n",
"Class 1: 11882655\n",
"Proportion: 6 : 1\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAF6CAYAAAAu3Z1/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAco0lEQVR4nO3de5SlVX3m8e/DRQGFtiONiDK0YpRBxzhjuxRHBy8otpdg8IJBswYHV4+zRp14v8S1bPES0BEykWSAiDgqygAiBhXl4oB3kyZeErGJGBGEiI1Wc7ERkf7NH+9b4eRQ3fVWd++uOtXfz1pnnTp7v/t9f+cUq3h6v7t2paqQJElSGzvNdwGSJEmLmWFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZpoSZYnqSS7zHct05K8NMlF2/B830/ylP7r1Uk+vg3P/bYkH9pW55N0T4YtaYFKctvIY2OS20dev3QbXucjSd69rc43x2sPCkpJHp7knCQ3Jbk5yfeSvC7Jztur1pFaPpLkN0lu7R//kORPkyyZPqaqzqyqZw4816yffVU9sqou28rSSfKUJD8dO/d7q+oVW3tuSZtm2JIWqKq67/QDuBZ43kjbmdPHLaQZnRaSHAh8C7gO+HdVtQR4EbAC2HOeynpfVe0JLANeDjwB+FqS+2zLiyz27620ozBsSRNmenYiyZuT/Aw4I8lOSd6S5EdJfpHk7CS/MzLmnCQ/62eFvpzkkX37KuClwJv6GbML+vZrkryxn0H6VZLTkzwgyYX9bM4lSZaOnP8JSb6eZH2S707f8ur7LkvyriRf68delGTvvvvL/fP6/vqHzPCW3wl8vapeV1X/DFBVV1XV0VW1fobP5+VJftBf65+S/NeRvr2TfLav85dJvpJkp77vzUmu78ddleTps30vqurXVfW3wO8D96cLXiQ5JslX+6+T5KQkP09yS5K/T/KoWT77Nyf5HvCrJLv0bYeNXHq3JP+3r/XvkvzeyHusJA8bef2RJO/ug+CFwH4jM6T7Zey2ZJLfT3fbcn3/vfu3I33XJHlD/9/FzX0Nu832OUk7OsOWNJn2BX4HOABYBbwaeD5wKLAfMAX8xcjxFwK/C+wD/B1wJkBVndZ//b5+xux5I2NeADwDeDjwvP4cb6ObzdkJeA1AkgcBnwPe3df0BuBTSZaNnOtouiCyD3Cv/hiA/9Q/36+//jdmeK+HAecO+1gA+DnwXGCv/ponJfkPfd/rgZ/27+EB/fupJI8AXgU8rp+xOhy4ZugFq+pW4GLgyTN0P5PufT4cWAK8GPjFLJ/9HwLPoftcfjvDOY8AzqH7vD8BnJ9k11lq/BWwErhhZIb0htFjkjwc+CTwx3Sf0eeBC5Lca+SwFwPPAh4CPBo4ZnPXlbTAw1aSD/f/GvyHAceelOQ7/eMfk6zfDiVK82Uj8I6quqOqbgdeCfxJVf20qu4AVgMvnL4NVVUfrqpbR/p+LyNrjDbhg1V1Y1VdD3wF+FZVfbuqfg18Gvj3/XEvAz5fVZ+vqo1VdTGwBnj2yLnOqKp/7Gs9G3jMHN7r/YF/HnpwVX2uqn5UncuBi7g7BN0JPBA4oKrurKqvVPcHYu8C7g0cnGTXqrqmqn40hxoBbqALP+PupLvdeRCQqvrB9AzdZvx5VV3Xf14zuaKqzq2qO4ETgd3obmVuraOAz1XVxf25/yewO/DEsdpuqKpfAhcwt++ltENa0GEL+Ajdv6BmVVWvrarHVNVjgA8C5zWsS5pv6/rQM+0A4NP9rZ/1wA/oAsQDkuyc5Pj+FuMt3D1jszebd+PI17fP8Pq+I9d+0fS1++s/iS7UTPvZyNcbRsYO8Yuxc21WkpVJvtnfJlxPF/qm3+v7gauBi/pbjG8BqKqr6WZzVgM/T3JWkv3mUCPAg4BfjjdW1ZeAk+lmGn+e5LQke81yruuG9lfVRrrZurnWO5P9gJ+Mnfs6uvc2bWu+l9IOaUGHrar6MmM/vJIcmOQLSa7o11scNMPQP6SbCpcWqxp7fR2wsqruN/LYrZ+VOpruttNhdLexlvdjsolzzdV1wMfGrn2fqjp+C97HTC6hu6U5qyT3Bj5FNyPzgKq6H92tsEB3u6+qXl9VD6VbZ/W66bVZVfWJqnoSXXgs4IQh1+yve1+6z/crM/VX1Z9X1WOBg+luJ75xumsTp5ztc9l/5No7AQ+mm1mDLgDtMXLsvnM47w1073/63Omvdf0s4yRtxoIOW5twGvDq/gfXG4C/HO1McgDdWoIvzUNt0nw5BXhP/98/SZYlOaLv2xO4g26GaA/gvWNjbwQeuhXX/jjwvCSH97Nou6VbxP/gAWPX0d0S3dz13wE8Mcn7k+wLkORhST6e5H5jx96L7nbgOuC3SVbSrZmiH/fcfmyAm+lm/zYmeUSSp/Vh7dd0M3cbZys+yb2TPBY4n26d3BkzHPO4JI/v11T9qj//9Lm39LN/bJIj+9vEf0z3/f1m3/cd4Oj+e/EsunV8024E7r+ZW8hnA89J8vS+3tf35/76FtQoqTdRYav/1+MTgXOSfAc4lXveXngJcG5V3bWdy5Pm0/8C/pru9titdP/jfXzf91G6W0PXA1dy9/+Up51Ot1ZpfZLz53rhqrqObubsbXQh5zq6mZtZf75U1QbgPXTbJqxPco91R/3aqUPoZuS+n+RmutmrNcCtY8feSrdw/2y68HM03ecy7XfpZspuA74B/GVV/T+6gHY8cBPdbbJ9gLdupvQ39Z/zL+g+3yuAJ/aL0MftBfxVX89P+jHv7/u29LP/DN36qingj4Aj+zVWAP+D7hca1tP9tuO/nLeq1tLN+v9Tf81/deuxqq6iW4P3QbrP4nl0W478Zg61SRqTbm3owpVkOfDZqnpUv87hqqra5PqNJN8G/ntV+S8xSZI07yZqZquqbgF+nORF8C/714zuL3MQsJTuX6ySJEnzbkGHrSSfpAtOj0i3ieOxdNPixyb5LvB9utsX014CnFULfbpOkiTtMBb8bURJkqRJtqBntiRJkiadYUuSJKmhBfsX5ffee+9avnz5fJchSZI0qyuuuOKmqlo2U9+CDVvLly9nzZo1812GJEnSrJL8ZFN93kaUJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgaFrSQPS3Jqku8luSvJZQPGPC7JGUmuTrIhyVVJ3pFkt62uWpIkaUIM3frhkcCzgW8Cuw4ccxRwIHAC8EPg0cC7+ucXzK1MSZKkyTQ0bF1QVZ8BSHIusPeAMcdX1U0jry9L8mvg1CQHVNUm96OQJElaLAbdRqyqjXM98VjQmvbt/nm/uZ5PkiRpEm3vBfKHABuBH23n60qSJM2L7Ra2kuwLvB34WFX9fBPHrEqyJsmadevWba/SJEmSmtkuYSvJvYCzgduA127quKo6rapWVNWKZctm/FuOkiRJE6X5H6JOEuCjdL/R+B+raqr1NSVJkhaK5mEL+DPgCOAZVbV2O1xvUXvIWz433yVoQvz4+OfMdwmSJBqHrSRvBV4FvLiqvtryWpIkSQvRoLCVZA+6TU0BHgTsleSF/evPV9WGJFcDl1fVsf2Yo4H3Ah8Brk/yhJFT/qiqXAEvSZIWvaEzW/sA54y1Tb9+CHBNf66dR/qf2T8f0z9GvZwuhEmSJC1qg8JWVV0DZJZjlo+9PoZ7hixJkqQdyvbe1FSSJGmHYtiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0NCltJHpbk1CTfS3JXkssGjluS5IwkU0luTnJmkvtvVcWSJEkTZJeBxz0SeDbwTWDXOZz/bODhwCuAjcAJwPnAk+dwDkmSpIk1NGxdUFWfAUhyLrD3bAOSHAI8Ezi0qr7ct10PfCvJYVV1yRbWLEmSNDEG3Uasqo1bcO6VwI3TQas/z98AP+77JEmSFr2WC+QPAtbO0P6Dvk+SJGnRaxm2lgLrZ2if6vvuIcmqJGuSrFm3bl3D0iRJkraPBbX1Q1WdVlUrqmrFsmXL5rscSZKkrdYybE0BS2ZoX9r3SZIkLXotw9ZaZl6btam1XJIkSYtOy7B1IbBvkidNNyRZATy075MkSVr0Bu2zlWQPuk1NAR4E7JXkhf3rz1fVhiRXA5dX1bEAVfWNJBcBH03yBu7e1PSr7rElSZJ2FEM3Nd0HOGesbfr1Q4Br+nPtPHbMUcBJwIfpZtE+C7xmSwqVJEmaRIPCVlVdA2SWY5bP0LYeeHn/kCRJ2uEsqK0fJEmSFhvDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqaFDYSnJwkkuTbEhyQ5Ljkuw8YNyKJBcl+WX/uCTJ47e+bEmSpMkwa9hKshS4BCjgCOA44PXAO2cZt38/bhfgj/rHLsDFSQ7YurIlSZImwy4DjnklsDtwZFXdQheW9gJWJ3lf3zaT5wB7An9QVTcDJPk6cBPwbOB/b3X1kiRJC9yQ24grgS+Ohaqz6ALYoZsZtyvwW+BXI2239W2ZY52SJEkTaUjYOghYO9pQVdcCG/q+TflUf8wHkuyTZB/gJGAKOGfLypUkSZosQ8LWUmD9DO1Tfd+MquoG4KnAC4Ab+8eRwOFVtW7OlUqSJE2gZls/JHkg3QzWFXS3Ilf2X38uyb/ZxJhVSdYkWbNunXlMkiRNviFhawpYMkP70r5vU95It27rhVX1har6At0s113AG2YaUFWnVdWKqlqxbNmyAaVJkiQtbEPC1lrG1mb12zrswdharjEHAd+vqjunG6rqN8D3gQPnXqokSdLkGRK2LgQOT7LnSNtRwO3A5ZsZ9xPgUUnuNd2Q5N7Ao4Br5l6qJEnS5BkStk4B7gDOS3JYklXAauDE0e0gklyd5PSRcR8C9gM+neQ5SZ4LnA88EDhtG9UvSZK0oM0atqpqCng6sDNwAd3O8ScB7xg7dJf+mOlxVwDPotvY9GPAR+luPT6jqr67LYqXJEla6IbsIE9VXQk8bZZjls/Qdilw6RZVJkmStAg02/pBkiRJhi1JkqSmDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqaFBYSvJwUkuTbIhyQ1Jjkuy88CxRyb52yS3J/lFki8kuc/WlS1JkjQZZg1bSZYClwAFHAEcB7weeOeAsa8APgFcCKwEXgH8ENhly0uWJEmaHENCzyuB3YEjq+oW4OIkewGrk7yvb7uHJHsDJwGvrqq/Gun69NYWLUmSNCmG3EZcCXxxLFSdRRfADt3MuBf3z/9nC2uTJEmaeEPC1kHA2tGGqroW2ND3bcrjgauAY5P8NMmdSb6V5IlbXK0kSdKEGRK2lgLrZ2if6vs2ZV/gEcDbgTcDzwN+BXwhyQNmGpBkVZI1SdasW7duQGmSJEkLW8utHwLcFzi2qs6sqi8AzwfuAl4104CqOq2qVlTVimXLljUsTZIkafsYEramgCUztC/t+zY3roDLphv6dV9XAAcPL1GSJGlyDQlbaxlbm5Vkf2APxtZyjfkB3exWxtoDbJxDjZIkSRNrSNi6EDg8yZ4jbUcBtwOXb2bcZ/vnp043JFkCPBb47hzrlCRJmkhDwtYpwB3AeUkOS7IKWA2cOLodRJKrk5w+/bqq1gCfAU5P8p+TPAf4a+BO4C+24XuQJElasGYNW1U1BTwd2Bm4gG7n+JOAd4wdukt/zKiXAecDJwLn0gWtp/XnlCRJWvQG/dmcqroSeNosxyyfoe024L/1D0mSpB1Oy60fJEmSdniGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLU0KCwleTgJJcm2ZDkhiTHJdl56EWS7JRkTZJK8twtL1eSJGmy7DLbAUmWApcAVwJHAAcCH6ALam8feJ1XAA/ewholSZIm1pCZrVcCuwNHVtXFVXUK8E7gdUn2mm1wH9beA/zJVlUqSZI0gYaErZXAF6vqlpG2s+gC2KEDxr8L+Bpw6dzLkyRJmmxDwtZBwNrRhqq6FtjQ921SkkcD/wV4w5YWKEmSNMmGhK2lwPoZ2qf6vs35IHByVV09pJgkq/qF9GvWrVs3ZIgkSdKC1mzrhyQvAR4BvHvomKo6rapWVNWKZcuWtSpNkiRpuxkStqaAJTO0L+377iHJrsD7gROAnZLcD5heTH+fJHvOvVRJkqTJMyRsrWVsbVaS/YE9GFvLNeI+dFs9nEgXyKaA7/Z9ZwHf3pJiJUmSJs2s+2wBFwJvTLJnVd3atx0F3A5cvokxtwFPHWvbF/gk8DbgS1tQqyRJ0sQZErZOAV4DnJfkBOChwGrgxNHtIJJcDVxeVcdW1W+By0ZPkmR5/+XfV9W3tr50SZKkhW/WsFVVU0meDpwMXED3m4kn0QWu8XMN/hM+kiRJO4IhM1tU1ZXA02Y5Zvks/dcAGVqYJEnSYtBs6wdJkiQZtiRJkpoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4PCVpKDk1yaZEOSG5Icl2TnWcY8LskZSa7ux12V5B1Jdts2pUuSJC18u8x2QJKlwCXAlcARwIHAB+iC2ts3M/So/tgTgB8Cjwbe1T+/YKuqliRJmhCzhi3glcDuwJFVdQtwcZK9gNVJ3te3zeT4qrpp5PVlSX4NnJrkgKr6ydaVLkmStPANCVsrgS+Ohaqz6GasDgUumGnQWNCa9u3+eT/AsCVJC8XqJfNdgSbF6pvnu4KJM2TN1kHA2tGGqroW2ND3zcUhwEbgR3McJ0mSNJGGhK2lwPoZ2qf6vkGS7Eu3xutjVfXzoeMkSZIm2XbZ+iHJvYCzgduA127muFVJ1iRZs27duu1RmiRJUlNDwtYUMNPN/KV932YlCfBR4JHAs6tqk2Oq6rSqWlFVK5YtWzagNEmSpIVtyAL5tYytzUqyP7AHY2u5NuHP6LaMeEZVDTlekiRp0Rgys3UhcHiSPUfajgJuBy7f3MAkbwVeBbysqr66xVVKkiRNqCFh6xTgDuC8JIclWQWsBk4c3Q6i3yn+9JHXRwPvpbuFeH2SJ4w8vEcoSZJ2CLPeRqyqqSRPB06m21NrPXASXeAaP9fon/B5Zv98TP8Y9XLgI3OsVZIkaeIMWbNFVV0JPG2WY5aPvT6Ge4YsSZKkHcp22fpBkiRpR2XYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNDQpbSQ5OcmmSDUluSHJckp0HjFuS5IwkU0luTnJmkvtvfdmSJEmTYZfZDkiyFLgEuBI4AjgQ+ABdUHv7LMPPBh4OvALYCJwAnA88eYsrliRJmiCzhi3glcDuwJFVdQtwcZK9gNVJ3te33UOSQ4BnAodW1Zf7tuuBbyU5rKou2TZvQZIkaeEachtxJfDFsVB1Fl0AO3SWcTdOBy2Aqvob4Md9nyRJ0qI3JGwdBKwdbaiqa4ENfd/gcb0fzDJOkiRp0RgStpYC62don+r7tvU4SZKkRWPImq3tJskqYFX/8rYkV81nPZooewM3zXcRC0lOmO8KpEXBny3j3pn5rmChOmBTHUPC1hSwZIb2pX3f5sYtm8u4qjoNOG1ATdK/kmRNVa2Y7zokLS7+bNG2MOQ24lrG1lgl2R/Yg5nXZG1yXG9Ta7kkSZIWnSFh60Lg8CR7jrQdBdwOXD7LuH2TPGm6IckK4KF9nyRJ0qI3JGydAtwBnJfksH5d1WrgxNHtIJJcneT06ddV9Q3gIuCjSY5M8nzgTOCr7rGlBrz9LKkFf7Zoq6WqZj8oORg4GTiE7jcMPwSsrqq7Ro65Brisqo4ZabsfcBLwB3TB7rPAa6rKxYaSJGmHMChsSZIkacsM+kPU0kKU5GFJTk3yvSR3JblsvmuSNPmSHJzk0iQbktyQ5LgkO893XZpcC2qfLWmOHgk8G/gmsOs81yJpEUiyFLgEuBI4AjgQ+ADd5MTb57E0TTBvI2piJdmpqjb2X58L7F1VT5nfqiRNsiRvBd4EHDD9S2BJ3kT3i2H7jv2dYGkQbyNqYk0HLUnahlYCXxwLVWcBuwOHzk9JmnSGLUmS7naPjber6lpgAzNv1C3NyrAlSdLdltJtcTRuqu+T5sywJUmS1JBhS5Kku00BS2ZoX9r3SXNm2JIk6W5rGVublWR/YA/G1nJJQxm2JEm624XA4Un2HGk7CrgduHx+StKkc1NTTawke9BtagrwIGCvJC/sX3++qjbMT2WSJtgpwGuA85KcADyUbo+tE91jS1vKTU01sZIsB368ie6HVNU1268aSYtFkoOBk4FD6H4z8UPA6qq6az7r0uQybEmSJDXkmi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhv4/7K3m0HMYDecAAAAASUVORK5CYII=\n",
"text/plain": [
"
H2OUpliftRandomForestEstimator : Uplift Distributed Random Forest\n",
"Model Key: UpliftDRF_model_python_1665008325837_1\n",
"
\n",
"
\n",
" \n",
"
\n",
"
\n",
"
Model Summary:
\n",
"
\n",
"
number_of_trees
\n",
"
number_of_internal_trees
\n",
"
model_size_in_bytes
\n",
"
min_depth
\n",
"
max_depth
\n",
"
mean_depth
\n",
"
min_leaves
\n",
"
max_leaves
\n",
"
mean_leaves
\n",
"
\n",
"
20.0
\n",
"
40.0
\n",
"
175049.0
\n",
"
15.0
\n",
"
15.0
\n",
"
15.0
\n",
"
293.0
\n",
"
394.0
\n",
"
344.5
\n",
"
\n",
"
\n",
"
\n",
"\n",
"[tips]\n",
"Use `model.show()` for more details.\n",
"Use `model.explain()` to inspect the model.\n",
"--\n",
"Use `h2o.display.toggle_user_tips()` to switch on/off this section.
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plot_uplift_score(h2o_uplift_pred['uplift_predict'].as_data_frame().uplift_predict)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluate the model"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"perf_h2o = h2o_uplift_model.model_performance(h2o_test_df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"### Area Under Uplift Curve (AUUC) calculation\n",
"\n",
"To calculate AUUC for big data, the predictions are binned to histograms. Due to this feature the results should be different compared to exact computation.\n",
"\n",
"To define AUUC, binned predictions are sorted from largest to smallest value. For every group the cumulative sum of observations statistic is calculated. The uplift is defined based on these statistics.\n",
"\n",
"\n",
"#### Types of AUUC\n",
"\n",
"\n",
"| AUUC type | Formula |\n",
"|:----------:|:-------------------------------------------:|\n",
"| **Qini** | $TY1 - CY1 * \\frac{T}{C}$ |\n",
"| **Lift** | $\\frac{TY1}{T} - \\frac{CY1}{C}$ |\n",
"| **Gain** | $(\\frac{TY1}{T} - \\frac{CY1}{C}) * (T + C)$ |\n",
"\n",
"\n",
"Where:\n",
"\n",
"- **T** how many observations are in the treatment group (how many data rows in the bin have treatment_column label == 1)\n",
"- **C** how many observations are in the control group (how many data rows in the bin have treatment_column label == 0)\n",
"- **TY1** how many observations are in the treatment group and respond to the offer (how many data rows in the bin have treatment_column label == 1 and response_column label == 1)\n",
"- **CY1** how many observations are in the control group and respond to the offer (how many data rows in the bin have treatment_column label == 0 and response_column label == 1)\n",
"\n",
"\n",
"The resulting AUUC value is:\n",
"\n",
"- Not normalized.\n",
"- The result could be a positive or negative number.\n",
"- Higher number means better model.\n",
"\n",
"More information about normalization is in **Normalized AUUC** section.\n",
"\n",
"\n",
"For some observation groups the results should be NaN. In this case, the results from NaN groups are linearly interpolated to calculate AUUC and plot uplift curve.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
"
\n",
"
\n",
"
AUUC table (number of bins: 168): All types of AUUC value
\n",
"
uplift_type
\n",
"
qini
\n",
"
lift
\n",
"
gain
\n",
"
AUUC value
\n",
"
21585.4882157
\n",
"
0.0253586
\n",
"
25328.4067094
\n",
"
AUUC normalized
\n",
"
0.8805330
\n",
"
0.0253586
\n",
"
0.8782349
\n",
"
AUUC random value
\n",
"
15585.3752152
\n",
"
0.0065580
\n",
"
18335.7289893
\n",
"
\n",
"
\n"
],
"text/plain": [
"AUUC table (number of bins: 168): All types of AUUC value\n",
"uplift_type qini lift gain\n",
"----------------- -------- ---------- --------\n",
"AUUC value 21585.5 0.0253586 25328.4\n",
"AUUC normalized 0.880533 0.0253586 0.878235\n",
"AUUC random value 15585.4 0.00655803 18335.7"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"perf_h2o.auuc_table()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Cumulative Uplift curve plot\n",
"\n",
"To plot the uplift curve, the ``plot_uplift``method can be used. There is specific parameter ``metric`` which can be ``\"qini\", \"gain\", or \"lift\"``. The most popular is the Qini uplift curve which is similar to the ROC curve. The Gain and Lift curves are also known from traditional binomial models. \n",
"\n",
"Depending on these curves, you can decide how many observations (for example customers) from the test dataset you send an offer to get optimal gain."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABIQklEQVR4nO3deZzN1f/A8dd77EWyJeFrVJYo+1YqilTaS5uKaZO0IUlaCJWK9n6JUlIiLaJCKiPZsu+yy9j3fRvz/v1xPqNrzHJn/dy59/18PD6Pe+/5bO8zl3nP53zO5xxRVYwxxphQE+V3AMYYY0xyLEEZY4wJSZagjDHGhCRLUMYYY0KSJShjjDEhyRKUMcaYkGQJykQsEekpIl/6HUcwRCRGRP4K+LxfRM713hcSkTEiskdERvoXZWgQkbEi0jartzU5zxKUyXEi0lpEZnm/ZDd5vyQu9Tuu1IjI5yLSJxP7x4rIQ0nKmopIXEaOp6qFVXW197EVUBoooaq3BxOrOE+KyCIROSAicSIyUkQuykg8oURVr1XVIVm9rcl5lqBMjhKRzsA7wKu4X6r/A/4PuMnHsHK7CsByVY1Pxz7vAk8BTwLFgcrAKOC69J5cRPKmdx9jgqKqttiSIwtQFNgP3J7KNp8DfQI+NwXiAj6vBZ4BFgAHgE9xiW4ssA/4DSiW3L4B+zf33vcEvgxYNxLYDOwB/gSqe+XtgGPAUS/+MV75OcB3wDZgDfBkKvWKBR5KUpa0bopLGKuB7cCbQJS3Lgb4K8m25wMve3Ed82J7JLlYk5y3EnAcaBBsvCmc/zFghVf3j4B+SY7xI9A5vT+rIP4d5QH6eT+j1V4cCuRNGnti3N72u7xzX5va92JL6Cx2BWVy0sVAQeCHTB7nNuAq3F/9N+CSU3egFK5V4MkMHncs7pf3WcAc4CsAVR3ovX9DXdPaDSISBYwB5gNlgWZARxG5OqOV8twC1APq4K4qH0htY1XtgbsaHeHF9nHSWJPZrRkuMf6dyVhvBhoC1YCvgTtFRABEpBjQAhieDT+rh4Hrgdq4n1WrNLZvCPwDlATeAD5NjNOENktQJieVALZr+pqikvO+qm5R1Q3AZGCGqs5V1cO45Fc7IwdV1cGquk9Vj+CurmqKSNEUNq8PlFLVXqp6VN39oEHAXRk5d4DXVXWnqv6Lawq9O5PHS04JYFMWHOc1L9ZDuO9Bgcu8da2Aaaq6kaz/Wd0BvKOq61V1J/BaGtuvU9VBqnocGAKUwV11mxBnbccmJ+0ASopI3kwmqS0B7w8l87lweg8oInmAV4DbcVdiCd6qkrgmv6QqAOeIyO6Asjy4X9TJiQfyJSnLh2uOC7Q+4P06XNNYVtuB+yWdWSdiVVUVkeG4hPon0BpI7CEZ9M9KRP4HLAk4bnLf5Tmc+nNKzeaA4x30Lp7S/W/E5Dy7gjI5aRpwBNc0lJIDwGkBn8/OxPlOOpaXhEqlsG1rXJNac9y9sujE3bzXpMP+rwfWqOqZAUsRVW2ZwvH/DThmooqc+su1fMD7/wEbUzheatKaouB3oJyI1Etlm2C+h6Tn+RpoJSIVcM1q33nlQf+sVPVfr2mycArJCdzVX9KfkwlDlqBMjlHVPcBLwIcicrOInCYi+UTkWhF5w9tsHtBSRIqLyNlAx0yccjlQUESuE5F8wAtAgRS2LYJLnjtwv5hfTbJ+C3BuwOe/gX0i8qz3HFIeEblQROqncPwRwP0i0sDr4l0Z6AQMT7LdMyJSTETK43rZjQiyrqnFehJVXYHrOfm119U9v4gUFJG7RKSbt9k84FbvOzofeDCtk6rqXFzHhU+A8aq621uV3p9VWr4BnhSRct69rm5p7WByJ0tQJkepan+gMy5ZbMP9df04roszwFDczfS1wK9k7Bd04rn2AB1wvzA34K4KUnru6Avc1cwGXBPT9CTrPwWqichuERnl3c+4HqiF6xmW+Is52XtWqjoe94v0M1yT4S+4+yEDk2z6IzAblyB+9s6bXifFmsI2TwIfAB8Cu4FVuA4aY7z1b+N6Am7x4vwqyHMPw12FDkssSO/PKgiDgPG4fydzgO8zeBwT4kTVJiw0JhSIiAKVVHWl37HkJiISjUt8+bKgA44JIXYFZYwxJiRZgjLGGBOSrInPGGNMSLIrKGOMMSHJEpQxxpiQZAnKhDVvmotdIlIgSVmKU1+ISLSIaNJRupNOYyEiZUTkU2/KkH0iskxEXhaR09MR31ki8rWIbPTmc5oiIg2TnGO0t169HmtJYzrqTV2SuOQJWH+HiCz14lsiIjcHrIsRkeNJ9m0aRMwvebE0DygrLiIjRGSHiGwXka9E5IyA9bVEZLJXxzgReTFgXSMRmSAiO0Vkm7hpP7JipAuTy1mCMmHL+2V+GW7Egxuz+NjFcSNjFAIuVtUiuAFszwTOS8ehCgMzgbq4aS+GAD+LSOIoCgnAONwAuSl5I3D0Be+5I0SkLG64oc7AGbhR4IeJyFkB+05Lsm9sGvU+DzccVNKx/PoAxXCjY5yHG+uuZ8D6YbghkIoDTYAOIpL4nRTDPQ8WjRsWaR/ueTET4SxBmXDWBvfA7edAVs+a2hn3i/ReVV0L4A1e+pSqLgj2IKq6WlXfUtVNqnrcGzk9P1DFW79FVf8Pl8TSqxywW1XHqvMz7mHl9CTQpD4EnsU9xBuoIjBKVfd6D0j/AFQPWB8NfOXVcRVuCozqAF58I719D+IeIG6ciRhNmLAEZcJZG9wICF8BV4tIVo5g3Rz4XlUTUtpARBZ4ozkkt/xfCvvUwiWo9Dys28FrHpstIoFXWrOApSJyoze80M244ZwCE2htr0luuYi8mLRZM0lstwNHVPWXZFZ/CFzvDdNUDHfFNzZg/TtAG3FDW1XBTb3yWwqnuhxYnHqVTSSw0cxNWBI3hXwF4BtV3S4iq3ADwr6dRadIc8oKVa2RngN692yGAi97VyHBeA94Gjd8UgtghIhsVtUpqnpcRL7ANa8VxF313K6qB7x9/wQuxA3xVB03rFQ8yUxfISJFcOMTXpVCHHNwiXWH9/l33Hh/iX7CDSfVBTeSeS9VPeWqUERq4MZrtBmWjV1BmbDVFvhVVbd7n4fxXzNfWlNfxAeUpbRNVk1ZAYCIFMKNgzddVdOa3+gEVZ2jqjtUNd67svkKuNU7ZnPcBH1NccmjCfCJd5WW2Ly4RlUTVHUh0IuUJ//rCQxNbM5Mxje4wXmL4O53rcKbbsO7XzfOO35B3EjkV4tIhyQ/g/NxV11PqWpK05aYCGIJyoQd75f9HUATEdksIptxI4fXFJGapD31xSZcIkptm9+AW8TNFptSHIuT9JALXAYEbFcAN1huHG7K9sxQ/psipBbwp6rO8pLQTGAGrnkyrX2TaoYbQTzx51ke+EZEng0418eqekBV9wMDgMTpNM4FjqvqF14ijcON4n5iug1xU3T8BvRW1aHprrUJTxoC887bYktWLrhJ83bi5gk6O2D5E+gPXA1sBRrgfiFXBpYC7QOO8TVulOwSuCunu3Gjfpf21hfHjbg+FKjglZUF3gJqpCPWfLgrp1FA3hS2KQicjksgVYCCAeta4XoCRuGa+PYBTb11TXAjh9fyPtfGXfm18D5fG1CfqsAioEcKMZRI8rNcj+vNV9hbPxF4H9ersRCueW+qt+4M72fX2ovzbFwPyFcDfm6rgC5+/9uxJbQW3wOwxZasXnDNSf2TKb8DN7tqXuAB3I34vbgOCd2AqIBti/HfNB27gClA4yTHOwcY7B1zH7AM6AGclo5Ym3iJ5yCwP2C5LGAbTboErJuMu/+0Fzf9xF1Jjv+4V799wGrg6YB1/XDTaRzw1vXCjQieuH4xcE8Kca8Fmgd8rohLtDtwfxyMw43Mnrj+SlxPxD3ez2tQ4s/J+5lpkvrv9/vfkS3+LzYWnzHGmJBk96CMMcaEJEtQxhhjQpIlKGOMMSHJEpQxxpiQFHEjSZQsWVKjo6MzvP+BAwc4/fSgB6vOtaye4SVS6gmRU9dwqufs2bO3q2qppOURl6Cio6OZNWtWhvePjY2ladOmWRdQiLJ6hpdIqSdETl3DqZ4isi65cmviM8YYE5KyLUGJSHkRmehNkrZYRJ7yynuKyAYRmectgcOdPCciK0XkHxG5OqD8Gq9spYh0CyivKCIzvPIRIpI/u+pjjDEmZ2XnFVQ87qn1akAj4DERqeate1tVa3nLLwDeurtwoypfA/yfN0VAHtxQ/tcC1YC7A47zunes83FP+z+YjfUxxhiTg7ItQambgG2O934fbqyzsqnschMwXFWPqOoa3PAsDbxlpbqRl4/iBpm8SUQEN3zKt97+Q4Cbs6UyxhhjclyOdJLwpt6ujRtJuTHwuIi0wU2o9rSq7sIlr+kBu8XxX0Jbn6S8IW7wyt2qGp/M9knP3w5oB1C6dGliY2MzXJf9+/dnav/cwuoZXiKlnhA5dY2EemZ7ghKRwsB3QEdV3SsiHwG9cYND9saNLv1AdsagbhrtgQD16tXTzPR8CaeeM6mxeoaXSKknRE5dI6Ge2dqLT0Ty4ZLTV6r6PYCqblHV4+qmyh6Ea8IDN2p0+YDdy3llKZXvAM4MmKI6sdwYY0w2O3oUsnus8Wy7gvLuEX0KLFXVtwLKy6hq4lTZt+DmoAEYDQwTkbdw0xhUAv7GzddTSUQq4hLQXUBrVVURmYibD2c4brbUH7OrPsYYEy5U4dAh2L8f9u377zXwfWplixbBpk0AY2jQoBG//16KwoWzPs7sbOJrDNwHLBSReV5Zd1wvvFq4Jr61eDOIqupiEfkGWILrAfiYqh4HEJHHgfFAHmCwqi72jvcsMFxE+gBzcQnRGGPCSkKCSw6BSWPevDMzlFwSXxMSgjt3/vxQpAgULvzf64UXQsWKsGDBRv79dwCHD7+YuxKUqv5F8tNH/5LKPq8AryRT/kty+6nqav5rIjTGmJBw7Fj6kkZqiWTfPjh4MLmz1Er23KeddnJCKVIESpWCc889OckkrkurLH/A06UbN26ke/fuXHzxxTzyyCMcP/4QAHnyZPmPEIjAoY6MMSZQepq7gk0yR48Gd+6oqOQTRPnyaSeSVavmcemltU4qO/307EkWhw4don///rz22mvEx8dTtWpVAPJkV2byWIIyxuQqyTV3BSaLWbPKMGdO8Fcr+/fD8ePBnTt//uSTxtlnB39FEvhaqBBIcu1MQYiN3U29ehnbNz3GjRtHu3btWL9+PbfddhtvvPEG5557bvafGEtQxphsllpzV3qbvVJu7gpU5cS70047NVGUKuXun6QnoSTX3BXuVBURISoqihIlSjB06FCaNGmSozFYgjLGnKAKhw9n/v5JVjV3FS58anNXWolkwYKpXH31JdnW3BXu4uLieO655yhbtix9+/alRYsWNG/enKionB9b3BKUMblY0uaulJLGwoUV+OmntK9g0tPclS9f8okisbkrvTfkM9PcFWjTpqOccUbmjxNpDh48yJtvvsnrr79OQkIC3bqdGJfbl+QElqCMCXmqsHw5LFgAK1a494mv27cHe5SKJzV3Jb6WLAnR0env2VWkSGQ1d4W7P/74g7Zt2xIXF8cdd9zB66+/TmYmds0qlqCMCTHx8TB3LkyeDH/95ZZt2/5bf845ULky3HKLex9MQpk1K5ZmzZr6VicTmuLj48mbNy9nn3025cqVY9iwYVx22WV+h3WCJShjfHbgAMyY4RLS5MkwfborA3cz/9pr4bLLoG5dqFSJDD0QafdiTKD169fTrVs34uPjGTFiBNWqVWPq1KlIVrSxZiFLUMbksB073FVRYkKaM8ddNYlAjRoQE+MS0qWXQtnUJqgxJp0OHDjAG2+8wZtvvomq0qVLFxISEoiKigq55ASWoIzJVqqwbt3JCWnpUrcuf35o0ACeecYlo0sugTPP9DVcE8amT5/ObbfdxsaNG7nrrrvo27cvFSpU8DusVFmCMiYLJSTA4sUnJ6S4OLfujDOgcWO47z6XkOrXh4IF/Y3XhL9Dhw5RqFAhzjvvPKpXr87IkSO55JJL/A4rKJagjMmEo0dh1qz/OjRMmQK7drl1Zcq4prrE5cIL7V6QyTnr1q3j2WefZfXq1UyfPp1SpUrx66+/+h1WuliCMiYd9u6FadP+uzr6+2/3YCtAlSpw663/JaSKFbPmuR5j0mP//v307duXfv36ERUVRdeuXYmPjyd/LnwuwBKUManYvPnk5rr5810zXp48ULs2tG//X4eGs87yO1oT6RYtWsRVV13F5s2bueeee3jttdcoX7582juGKEtQxnhUYeXKkxPSypVuXaFC0KgRvPCCS0aNGrlnjIwJBbt27aJYsWJUqlSJJk2a0LFjRxo1auR3WJlmCcpErOPHYfnywixY8N89pM2b3brixV0ieuQR91qnjo2cYELPmjVrePbZZ5k+fTrLli3jtNNOY/jw4X6HlWUsQZmIceiQu2eUmIymToV9+9x8BRUqQLNm/90/qlrVDV5qTCjau3cvgwYN4rvvviNPnjx069YtJJ9jyixLUCZs7d59cnPdrFlu6gdwPeruuQdKlFjCI49UIxc305sI8++//9KgQQO2bNlCmzZtePXVVykbpk90W4IyYeP4cZeExo+HcePc8EEJCW7U7Xr1oFMn11zXuLFrwgOIjd1K+fLV/A3cmCBs2LCBsmXLUr58ee68806qVq3Ko48+6ndY2coSlMnVNm6EX391CWnCBNi503XtrlcPnn8errzSjdZw2ml+R2pMxqxatYquXbsyfvx4li9fzjnnnMO7775LbGys36FlO0tQJlc5csQ9DDtunLtSWrDAlZ99NtxwA1x9NVx1lZtGwpjcbO/evbzyyiu888475MuXj+eee45ixYr5HVaOsgRlQl58vLtKGjoUxoxxI33ny+ea615/3SWlGjXsoVgTPvbs2UOVKlXYsmULMTExvPLKK5xzzjl+h5XjLEGZkHX4MHz0Ebzxhuv+Xbw43HsvXHcdXHFFxqadMCaULVu2jKpVq1K0aFE6d+7MlVdeSb169fwOyzeWoExIWbMGfvvtv2XnTtf9+6OPoGVLexbJhKcVK1bwzDPPMHr0aObMmUOtWrXo2rWr32H5zhKU8VVCAkycCN984xLS6tWuvGxZuPFGaNPGXS0ZE452795Nnz59eO+99yhQoACvvvoqVatW9TuskGEJyvji339h8GD4/HM3X1KRIq7HXadO0Ly5G3jV7imZcHbs2DFq1arFv//+ywMPPECfPn04++yz/Q4rpFiCMjkmPh5+/hkGDoSxY11Z8+bQty/cfLPNjWQiw99//039+vXJly8fffr0oXr16tSuXdvvsEKSDeZist3Ro9CnD0RHu0Q0d657RmnNGtc77667LDmZ8Ld8+XJuvPFGGjZsyM8//wzAvffea8kpFXYFZbLNkSPw7LOuGW/PHmjRAj74AK6/HvLavzwTIXbt2kXv3r15//33KVSoEK+//jrNmzf3O6xcIduuoESkvIhMFJElIrJYRJ7yyouLyAQRWeG9FvPKRUTeE5GVIrJAROoEHKutt/0KEWkbUF5XRBZ6+7wn4ThaYi52113w7rtuJIeRI92DtTffbMnJRA5VpWnTprzzzjvcf//9rFixgq5du1LQmgyCkp1NfPHA06paDWgEPCYi1YBuwO+qWgn43fsMcC1QyVvaAR+BS2hAD6Ah0ADokZjUvG0eDtjvmmysjwmCKvTr555RGjUKevd2zXitWvkdmTE5548//uDo0aOICG+++SZz585l4MCBlC5d2u/QcpVsS1CquklV53jv9wFLgbLATcAQb7MhwM3e+5uAL9SZDpwpImWAq4EJqrpTVXcBE4BrvHVnqOp0VVXgi4BjGR+owtNPwzPPuNEeYmLce2MixbJly7juuuto1qwZn332GQAtWrSgZs2aPkeWO+VIY4uIRAO1gRlAaVXd5K3aDCT+SVEWWB+wW5xXllp5XDLlyZ2/He6qjNKlS2dqkMX9+/dHxCCN6a1nQgK8+24lRo8uy623xvHgg2s47bTjTJuWfTFmBfs+w48fdd2zZw9Dhgzhxx9/pFChQjz66KNUrFgxW+OIiO9UVbN1AQoDs4Fbvc+7k6zf5b3+BFwaUP47UA/oArwQUP6iV1YP+C2g/DLgp7TiqVu3rmbGxIkTM7V/bpGeesbHq95/vyqodu2qmpCQfXFlNfs+w48fdW3WrJlGRUVp+/btdcuWLTlyznD6ToFZmszv62y9ghKRfMB3wFeq+r1XvEVEyqjqJq+ZbqtXvgEInDaunFe2AWiapDzWKy+XzPYmB8XHQ9u2MGwY9OjhFuuqYiLBuHHjqF+/PiVKlOCNN94gX758XHTRRX6HFVaysxefAJ8CS1X1rYBVo4HEnnhtgR8Dytt4vfkaAXvUNQWOB1qISDGvc0QLYLy3bq+INPLO1SbgWCYHHD0Kd97pktNrr0HPnpacTPhbsmQJ1157Lddeey3vv/8+AHXq1LHklA2y8wqqMXAfsFBE5nll3YG+wDci8iCwDrjDW/cL0BJYCRwE7gdQ1Z0i0huY6W3XS1V3eu87AJ8DhYCx3mJywOHDrmfezz/D229Dx45+R2RM9tq+fTs9e/ZkwIABFClShLfffpsOHTr4HVZYy7YEpap/ASn9Pd0sme0VeCyFYw0GBidTPgu4MBNhmgw4eNA9zzRhghtlvH17vyMyJvt16tSJr7/+mvbt29OzZ09K2qyY2c6GOjLpsm+fm/bit9/gs88sOZnwpar89NNPLF++HIDevXszf/58PvjgA0tOOcQSlAnanj1u9tq//oKvvnLPORkTjhYtWsTVV1/NDTfcwNtvvw1AdHQ01atX9zmyyGIJygRlxw43ceCsWW7uprvv9jsiY7Letm3b6NChAzVr1mTWrFm8++67vPfee36HFbFsVDSTpq1b3bQYy5fDDz+4KdeNCUf9+/dn4MCBPPbYY/To0YMSJUr4HVJEswRlUrVxo7tyWrcOxoyBq67yOyJjso6qMnr0aEqWLEnjxo3p1q0bbdu25YILLvA7NIM18ZlU/PsvXH45xMW5CQYtOZlwsmDBApo3b87NN9/Mu+++C8CZZ55pySmEWIIyydq4sSCXXw7btrnRyJs08TsiY7LG1q1beeSRR6hduzbz5s3j/fff56uvvvI7LJMMa+Izp/jnH3jqqdokJMAff0Ddun5HZEzWGTlyJIMHD+aJJ57gpZdeonjx4n6HZFJgCcqcZNEi1yEiPl6YNAlq1PA7ImMyR1UZNWoUx48fp1WrVrRr146rrrqKypUr+x2aSYM18ZkT5s6Fpk0hKgreeWeeJSeT682bN48rr7ySW2+9lQEDBgCQL18+S065hCUoA8CMGXDllXDaafDnn1ChwkG/QzImw7Zs2cLDDz9MnTp1WLhwIf/3f//HuHHj/A7LpJMlKMNff7keesWLu+R0/vl+R2RM5syaNYshQ4bQqVMnVq5cyaOPPkrevHZHI7exbyzC/fEH3HADlCvn3pdNdk5iY0KbqvLdd9+xceNGatSoQcuWLVm1ahXly5dPe2cTsuwKKoKNG+dGhahYESZNsuRkcqc5c+bQtGlTbr/9dr788kuOHz+OiFhyCgOWoCLUjz/CTTdB1aoQGwtnn+13RMakz+bNm3nwwQepV68eS5cuZcCAAUydOpU8efL4HZrJItbEF4HWrYO77oJatdxVVLFifkdkTPpt27aNr7/+mi5duvD8889TtGhRv0MyWSzFBCUi76hqRxEZA2jS9ap6Y7ZGZrLNM8+4qdm//daSk8k9VJWRI0cya9Ys3njjDS666CLi4uLsQdswltoV1FDvtV9OBGJyRmwsjBwJL78M1kRvcotZs2bRsWNHpkyZQq1atTh48CCnnXaaJacwl2KCUtXZ3uuknAvHZKejR+HJJ+F//4MuXfyOxpi0bd26lWeeeYYvvviCs846i0GDBnH//ffbfaYIkeY9KBFpDPQEKnjbC6Cqem72hmaykio8+CAsXOiuoE47ze+IjEmbqjJ27FieffZZunfvzhlnnOF3SCYHBdNJ4lOgEzAbOJ694Zjs8v338OWX8OKL0KqV39EYkzxVZfjw4YwaNYrhw4dTunRp1qxZw+mnn+53aMYHwXQz36OqY1V1q6ruSFyyPTKTZZYuhQcegNq14fnn/Y7GmOTNmDGDxo0b07p1a1asWMG2bdsALDlFsGAS1EQReVNELhaROolLtkdmssTkydC4MRQq5J59KlDA74iMOdnOnTu57777aNSoEWvWrGHw4MHMnDmTs846y+/QjM+CaeJr6L3WCyhT4MqsD8dkpe3b4frr3ejkw4dbrz0TmgoVKsTMmTPp3r073bp1o0iRIn6HZEJEmglKVa/IiUBM1uvbF/bvdx0jqlXzOxpjnISEBIYNG8aAAQOYMGEChQoVYuHCheTLl8/v0EyISe1B3XtV9UsR6ZzcelV9K/vCMpm1YQN88AG0aWPJyYSOadOm0bFjR/7++2/q1q3Lli1biI6OtuRkkpXaPajEO5NFUlhMiJk4Edq3hwMH4JVXICEBevTwOypjYP/+/bRu3ZpLLrmE9evX8/nnn/P3338THR3td2gmhKX2oO7H3uvLOReOyahdu+Dmm2HvXti3zz3r9OCDYP//jZ8SEhKIiori9NNPZ8uWLbzwwgs8++yzFC5c2O/QTC4QzIO676W2XlWfzLpwTEb16OGunEqWhGHDIG9e6NXL76hMpEpISGDo0KH07duX2NhYSpcuzYQJE4iKsgkUTPCC+ddSEKgDrPCWWkB+3IO7s1PaSUQGi8hWEVkUUNZTRDaIyDxvaRmw7jkRWSki/4jI1QHl13hlK0WkW0B5RRGZ4ZWPEJH86ah3WJk7F95/H26/HUaNcmVt20KpUr6GZSLUlClTaNiwITExMZxxxhns2rULwJKTSbdgupnXAC5V1XgAERkATFbV9mns9znwAfBFkvK3VfWkAWhFpBpwF1AdOAf4TUQqe6s/BK4C4oCZIjJaVZcAr3vHGu7F9CDwURD1CSvLlsEVXj/Lzp2hfn1YsAAqV059P2OyWnx8PPfddx/Dhw+nbNmyDB06lNatW1tiMhkWzL+cYkDgAFiFvbJUqeqfwM4g47gJGK6qR1R1DbASaOAtK1V1taoeBYYDN4mI4J7D+tbbfwhwc5DnCivvv++6kk+f7pITwEUX2QO5JuccO3YMgLx581K4cGF69OjBP//8w7333mvJyWRKMFdQfYG5IjIRN1Ds5bjBYzPqcRFpA8wCnlbVXUBZYHrANnFeGcD6JOUNgRLA7sSruiTbn0JE2gHtAEqXLk1sbGyGg9+/f3+m9s8qCQkwZsw5fPrpeTRpsp1Dh5aSlWGFSj2zm9Uz4xISEhg/fjyfffYZffv25dxzz+Wee+4BYObMmVl6rvSw7zSMqGqaC3A27irnJuDsYPbx9osGFgV8Lg3kwV25vQIM9so/AO4N2O5ToJW3fBJQfp+3bUnclVViefnA86S21K1bVzNj4sSJmdo/q7z4oiqoVq2qumpV1h8/VOqZ3ayeGfPnn39qnTp1FNCGDRvq/Pnzs/T4mWHfae4DzNJkfl8HNeW7qm4GfsxcKgRV3ZL4XkQGAT95Hzd4SSZROa+MFMp3AGeKSF51V1GB24e9v/6CPn2gbFk3SkTeoL5FYzJPVWnbti1Dhw6lfPnyfPXVV9x99924VndjslaONhCLSJmAj7cAiT38RgN3iUgBEakIVAL+BmYClbwee/lxHSlGexl3Iu4KC6AtWZBAc4N9+9zoENHRbpRyS04mJxw4cABVRUSoWrUqvXr1YtmyZbRu3dqSk8k22fbrTUS+BpoCJUUkDugBNBWRWrjBZtcCjwCo6mIR+QZYAsQDj6nqce84jwPjcU2Dg1V1sXeKZ4HhItIHmItrFgx7zz8Pa9fCn3+Cjalpstvx48f5/PPPef755xk8eDAtW7ake/fufodlIkRQCUpELgUqqepnIlIKKKyut12KVPXuZIpTTCKq+gruvlTS8l+AX5IpX43r5RcxBg50vfZuvRUuvdTvaEy4i42NpVOnTsybN4/GjRtTpkyZtHcyJgul2cQnIj1wVyvPeUX5gC+zMyhzqq+/hkcegebN3XtjstOjjz7KFVdcwc6dOxk+fDiTJ0+mdu3afodlIkwwV1C3ALWBOQCqulFErHEph332mXsdNgzyR+yYGSY77d27l0KFCpEvXz4aN25MuXLl6Ny5M4UKFfI7NBOhgukkcdTrlKAAImLzL+ewHTvgjz/gueds+CKT9Y4fP86gQYOoVKkSH3/8MQD33nsvzz//vCUn46tgEtQ3IvIxrlv3w8BvwKDsDcsE6t0bjh+HVq3S3taY9Pjjjz+oU6cO7dq1o0qVKlxyySV+h2TMCcHMqNtPRK4C9gJVgJdUdUK2R2YAWLIE3n0XHn8c6tTxOxoTTrp27cqbb75JdHQ0I0eO5LbbbrMu4yakBDPdRmdghCUlf4wb5167dvU3DhMe9uzZg4hwxhln0LJlS4oVK0anTp0oWLCg36EZc4pgmviKAL+KyGQReVxESmd3UOY/v/wCVatC+fJpb2tMSo4fP86AAQM4//zz6dmzJwBNmzblueees+RkQlaaCUpVX1bV6sBjQBlgkoj8lu2RGVasgN9/h7uTe6LMmCBNmDCBhx56iEcffZTq1atz3333+R2SMUFJz1BHW4HNuHHwzsqecEygIUMgKgoeftjvSExu9corr9CiRQuOHDnCd999x8SJE+15JpNrBHMPqgNwB1AKGAk8rG7CQJONjh93Cerqq8Ee4DfpsWvXLg4dOsQ555xDq1atyJs3L7Vr16ZFixZ+h2ZMugRzBVUe6Kiq1VW1pyWnnPHppxAX56ZuNyYY8fHxfPjhh5x//vk88cQTAFSpUoVnn32W/PZ0t8mFUkxQIpI4i+6bwL8iUjxwyZnwItO//8Kjj7rpNG66ye9oTG4wfvx4atasyeOPP07NmjV56aWX/A7JmExLrYlvGHA9MBs3ikTgAxIKnJuNcUW0N990M+b++itYByuTlo8++ogOHTpw3nnnMWrUKG688UZ7nsmEhRQTlKpe771WzLlwzNGjMHQo3HMPVKvmdzQmVO3cuZOtW7dStWpV7rjjDg4fPkyHDh0oUKCA36EZk2WCGc3892DKTNb45RfYs8e6lpvkHTt2jPfff5/zzz+fNm3aoKqUKFGCTp06WXIyYSe1e1AFvXtNJUWkWMD9p2igbI5FGEFUoW9fqFABrMOVSWrs2LHUqFGDJ598krp16/Lpp59aU54Ja6ndg3oE6Aicg7sPlfg/YS/wQfaGFZm++QZmzIABAyBfPr+jMaHk22+/5fbbb6dy5cqMGTOG6667zpKTCXup3YN6F3hXRJ5Q1fdzMKaIdPgwdOsGNWtCTIzf0ZhQsH37dlauXEmjRo248cYbGThwIG3btrUu4yZiBDOa+fsiciFQDSgYUP5FdgYWaXr1grVr3eCwdishsh09epT/+7//4+WXX6ZIkSKsXr2a/Pnz87ANKWIiTLBTvr/vLVcAbwA3ZnNcEWX/fvjwQ7jzTjdyhIlMqspPP/3ERRddRKdOnWjQoAG//PILefMGM/G1MeEnmJEkWgHNgM2qej9QEyiarVFFmC++gL17oWNHvyMxfpo8eTI33HADIsLPP//MuHHjuPDCC/0OyxjfBJOgDqlqAhDvjS6xFTf8kckCqvDBB1CvHjRs6Hc0Jqdt27aNX375BYDLLruMESNGsHDhQlq2bGmdIEzEC6btYJaInImb5n02sB+Ylp1BRZLff4elS93AsPb7KHIcPXqU999/n969e6OqxMXFUaRIEe644w6/QzMmZAQzH1QHVd2tqgOAq4C2XlOfySRVuOoq9/7OO/2NxeQMVeXHH3+kevXqdOnShcaNGzN9+nSKFCnid2jGhJwUr6BEpE5q61R1TvaEFDlmz3avHTpYz71IsXz5cm655RaqVq3K2LFjueaaa/wOyZiQlVoTX/9U1ilwZRbHEnG+/Rby5oXevf2OxGSnrVu3MnbsWNq2bUuVKlWYMGECl19+OfnsaWxjUpXag7pX5GQgkeiPP6BRIyhuk5eEpSNHjvDee+/Rp08fDh8+TPPmzSlbtizNmjXzOzRjcoVgZtRtk1y5PaibOT/+CDNnurH3THhRVUaNGkWXLl1YvXo1119/Pf369aNsWRvC0pj0CKYXX/2A9wVxz0TNASxBZcJ330HRovDUU35HYrLa9u3bue+++4iOjmb8+PE21boxGRRML74nApaHgTpA4bT2E5HBIrJVRBYFlBUXkQkissJ7LeaVi4i8JyIrRWRBYAcNEWnrbb9CRNoGlNcVkYXePu9JLntoZO5cuPRSm5AwXGzevJnXX38dVaVUqVJMmjSJefPmWXIyJhOCeVA3qQNAMJMYfg4k7aLUDfhdVSsBv3ufAa4FKnlLO+AjcAkN6AE0BBoAPRKTmrfNwwH75ZruUIcOuWefatXyOxKTWYcPH6Zv375UqlSJF198kUWL3N9jdevWtSGKjMmkYMbiGyMio73lJ+Af4Ie09lPVP4GdSYpvAoZ474cANweUf6HOdOBMESkDXA1MUNWdqroLmABc4607Q1Wnq6rimhtvJpcYNAiOH4fGjf2OxGSUqvLtt99ywQUX8Nxzz9GsWTMWL17MRRdd5HdoxoSNYP7E6xfwPh5Yp6pxGTxfaVXd5L3fDJT23pcF1gdsF+eVpVYel0x5skSkHe7KjNKlSxMbG5vB8GH//v2Z2h/g009rce65eSlYcBaZPFS2yYp65gYZreeRI0fo0KEDhQsXpl+/ftStW5cNGzawYcOGrA8yC0TK9wmRU9dIqGcw021MAvDG4cvrvS+uqkmvjtJFVVVENDPHSMe5BgIDAerVq6dNmzbN8LFiY2PJzP5Ll8KCBW56jSuuyPhxsltm65lbpKeemzZt4u2336Z3794UKFCAKVOmULFixVzRlBcp3ydETl0joZ7BNPG1E5HNwAJgFm48vlkZPN8Wr3kO73WrV76BkwegLeeVpVZeLpnykDdkiJst95FH/I7EBOvQoUO88sorVKpUiXfeeYfp06cDUKlSpVyRnIzJrYLpJPEMcKGqRqvquapaUVXPzeD5RgOJPfHaAj8GlLfxevM1AvZ4TYHjgRYiUszrHNECGO+t2ysijbzee20CjhXSpkxxI5efdZbfkZi0qCojRoygatWqvPDCC7Ro0YIlS5bQpEkTv0MzJiIE8+ffKuBgeg8sIl8DTYGSIhKH643XF/hGRB4E1gGJQzf/ArQEVnrnuh9AVXeKSG9gprddr4CmxQ64noKFgLHeEtKOHnUP5z72mN+RmGCoKv3796dYsWIMGTIk7JtTjAk1wSSo54CpIjIDOJJYqKpPpraTqt6dwqpTxnnxeuIl+2tbVQcDg5MpnwXkqtncYmPhyBH3/JMJTRs2bKBXr1706dOHUqVKMXr0aEqVKkWePHn8Ds2YiBNME9/HwB/AdNz9p8TFpNOgQVCiBLRs6XckJqmDBw/Su3dvKleuzOeff87UqVMBOPvssy05GeOTYK6g8qlq52yPJMwNG+ZGL+/UyabWCDVff/01zz77LOvXr6dVq1a8/vrrnHtuRm+zGmOySjAJaqz3HNEYTm7iy1Q380iyezc8+KCbMTcmxu9oTFLff/89JUuW5Msvv+Tyyy/3OxxjjCeYBJV4L+m5gDIF7E/MIC1ZAocPw88/Q40afkdj4uLi6N69O926uZG2PvnkE4oUKUJUVEZG/jLGZJdgHtQNZtw9k4q5c91rlSr+xhHpDhw4wJtvvskbb7xBQkICV111FeXLl6do0aJ+h2aMSYbNB5UDRo6EatXAbmv4Z/jw4XTp0oUNGzZw55130rdvX6Kjo8N+qBhjcjObDyqbHT4M06bBk0+6e1DGH3PmzKFMmTIMHz6cS62fvzG5QjBNfE8EfhaRM4Hh2RVQuFmxwj2gW6+e35FEln///Zdu3boRExNDixYt6NWrF/nz57f7TMbkItk5H5QBVq50r+ef728ckWL//v28+OKLVKlShR9++IE1a9YAULBgQUtOxuQywdyDGoPrtQcuoVUDvsnOoMLJ+PFucNhKlfyOJPyNHDmSjh07snHjRlq3bs1rr73G//73P7/DMsZkUE7PBxVRtm6FgQPdyOVnnOF3NOFLVRERduzYQbly5fj222+5+OKL/Q7LGJNJKbZ5iMj5ItJYVScFLFOACiJyXg7GmGtNnQqq0CbZfpAms9auXcudd97JoEGDAHj44YeZNm2aJSdjwkRqjfLvAHuTKd/rrTNpmDoV8ueHOnX8jiS87Nu3j+eff56qVasyZswYDh50g+3nyZPH7jMZE0ZSa+IrraoLkxaq6kIRic6+kMLHr79CgwY29l5W+vHHH2nfvj2bN2/m3nvv5bXXXqNcuXJp72iMyXVSS1BnprKuUBbHEXbi4mD+fOjf3+9IwkNCQgJRUVEULFiQ6OhoRo0aRcOGDf0OyxiTjVJrD5klIg8nLRSRh7DpNtKUOLyR/Q7NnNWrV9OqVSu6d+8OwNVXX83UqVMtORkTAVJLUB2B+0UkVkT6e8sk4EHgqRyJLheLjXX3n2rV8juS3Gnv3r1069aNCy64gLFjx1KsWLET68SG5DAmIqTYxKeqW4BLROQK/pu59mdV/SNHIsvlxo6FJk3g9NP9jiT3GT9+PG3btmXLli20adOGV199lbJly/odljEmhwUz1NFEYGIOxBI2Vq6EpUuhXTu/I8ldjh49Sv78+SlXrhxVqlRhzJgx1K9fP+0djTFhKZgHdU06ffede73lFn/jyC1WrVrFM888Q758+RgxYgTVq1dn0qRJfodljPGZPTSSDb77zg0OW6GC35GEtj179tC1a1eqVavGr7/+Ss2aNVHVtHc0xkQEu4LKYmvXwsyZ8NprfkcS2iZPnsxtt93G9u3biYmJ4ZVXXqFMmTJ+h2WMCSF2BZXFRoxwr3fe6W8coWr//v0AVK1alQYNGjBz5kwGDx5syckYcwq7gspCqvDZZ3DJJVDRJiQ5yYoVK3jmmWfYtGkT06ZNo1SpUvz0009+h2WMCWF2BZWFpk2Df/6Bhx7yO5LQsXv3brp06UL16tX5/fffueWWWzh+/LjfYRljcgG7gspCkye71xtv9DeOUDF37lxatGjBjh07eOCBB+jTpw9nn32232EZY3IJu4LKQrNmuaa9EiX8jsRf27dvB6BatWpcc801zJ49m08++cSSkzEmXSxBZZH9++GHH1z38kj1zz//cMMNN1CnTh0OHjxIgQIFGDp0KLVr1/Y7NGNMLmQJKot8+SUcPw433eR3JDlv165ddOrUiQsvvJBJkybxxBNPkCdPHr/DMsbkcr7cgxKRtcA+4DgQr6r1RKQ4MAKIBtYCd6jqLnEjg74LtAQOAjGqOsc7TlvgBe+wfVR1SE7WI9DUqVCmDLRu7VcE/li9ejX169dn9+7dPPTQQ/Tq1YvSpUv7HZYxJgz4eQV1harWUtXERrFuwO+qWgn43fsMcC1QyVvaAR8BeAmtB9AQaAD0EJFi+GTFCrjgAoiUgbbXrVsHQMWKFYmJiWHOnDl8/PHHlpyMMVkmlJr4bgISr4CGADcHlH+hznTgTBEpA1wNTFDVnaq6C5gAXJPDMQOuae+ff+D88/04e85aunQpLVu2pHr16mzcuBERoX///tSsWdPv0IwxYcavbuYK/CoiCnysqgNxU8xv8tZvBhL/FC8LrA/YN84rS6n8FCLSDnf1RenSpYmNjc1w4Pv37z9l/4ULz2DXrjqULr2Y2NhtGT52KElazz179jBkyBB+/PFHChUqRJs2bVi0aBHLly/3L8gskNz3GY4ipZ4QOXWNhHr6laAuVdUNInIWMEFElgWuVFX1kleW8BLgQIB69epp06ZNM3ys2NhYku6/aJF7bd++Oueck+FDh5TAeu7cuZNKlSqxe/du2rVrR69evShVqpS/AWaR5L7PcBQp9YTIqWsk1NOXJj5V3eC9bgV+wN1D2uI13eG9bvU23wCUD9i9nFeWUnmOW7wYzjzTdZIIF6rKwoULAShevDjdu3dn3rx5fPTRR2GTnIwxoS3HE5SInC4iRRLfAy2ARcBooK23WVvgR+/9aKCNOI2APV5T4HighYgU8zpHtPDKctyiRXDhheHTQWLx4sV07dqVmjVrsmDBAgCefvppLrroIp8jM8ZEEj+a+EoDP7je4+QFhqnqOBGZCXwjIg8C64A7vO1/wXUxX4nrZn4/gKruFJHewExvu16qujPnquEkJLgEFQ6jl2/fvp0ePXrw8ccfU6hQId566y2qVq3qd1jGmAiV4wlKVVcDp3T5UtUdQLNkyhV4LIVjDQYGZ3WM6bFyJezeDbl9ZvIjR45Qs2ZNtmzZQvv27bnqqqu4KRKfOjbGhIxQ6maeK82Y4V4bNvQ3joxQVSZPnoyqUqBAAd544w3mz5/PBx98QNGiRf0OzxgT4SxBZdKMGVC4sHtINzdZuHAhLVq04PLLL2f8eHfr7p577qF69eo+R2aMMY4lqEyaMcM17+WWoee2bdvGo48+Sq1atZg9ezbvvfcezZqd0rJqjDG+s/mgMuHIEZg/Hzp39juS4KgqTZo0Yfny5Tz++OP06NGD4sWL+x2WMbnesWPHiIuL4/Dhwzl2zqJFi7J06dIcO19WKFiwIOXKlSNfvnxBbW8JKhPmz4djx0K7g4SqMm7cOJo1a0b+/Pl57733KFu2LBfktjZJY0JYXFwcRYoUITo6Gsmh50327dtHkSJFcuRcWUFV2bFjB3FxcVSsWDGofayJLxNmeh3cQzVBLViwgObNm9OyZUuGDh0KQPPmzS05GZPFDh8+TIkSJXIsOeVGIkKJEiXSdZVpCSoTZs6Es86C8uXT3jYnbd26lUceeYTatWszb948PvjgA9q2bZv2jsaYDLPklLb0/oysiS8TZs50V0+h9u/yjjvuYMqUKTz55JO89NJLFCvm2ywkxhiTYXYFlUH79sHSpaHRvKeq/PDDD+zc6QbSeOedd1i0aBFvv/22JSdjTLI+//xzHn/8cQAGDBjAF198AcCyZcuoVasWtWvXZtWqVQwbNsy3GC1BZdDMmaAKDRr4G8fcuXO54ooruPXWW/noo48AqFWrFlWqVPE3MGNMrtG+fXvatGkDwKhRo2jVqhVz585l/fr1viYoa+LLoEmTICoKGjf25/ybN2/mhRdeYPDgwZQoUYKPPvqIhx56yJ9gjDEndOwI8+Zl7TFr1YJ33kl9m7Vr13L99dezyJv/p1+/fifmjKpZsyaTJk0iPj6ewYMH0yDJX9Y9e/akcOHCVKtWjXfeeYc8efLw+++/c+jQIZYuXUqtWrVo27YtnTp1ytqKpcESVAZNmgS1a8MZZ/hz/s6dO/Ptt9/SuXNnXnjhBc4880x/AjHGhLyDBw8yb948/vzzTx544IETSSypli1b0r59ewoXLkyXLl2IjY2lX79+/PTTTzkcsWMJKgMOH4bp0+GxZIewzR6qynfffUeNGjWoXLkyr732Gi+//DKVKlXKuSCMMWlK60rHD3fffTcAl19+OXv37mX37t3+BhQkuweVAX//7UaRaNIkZ843e/ZsmjRpwu23386HH34IQIUKFSw5GWNOyJs3LwkJCSc+Bz5vlLR7d27pEm8JKgP++su9Xnpp9p5n48aN3H///dSvX59ly5bx8ccf89Zbb2XvSY0xuVLp0qXZunUrO3bs4MiRIyc1y40YMQKAv/76i6JFiwY9W0GRIkXYt29ftsQbDGviy4B//oFy5SC7h7F7++23GTZsGM888wzdu3e3KTCMMSnKly8fL730Eg0aNKBs2bInTTZasGBBateuzbFjxxg8OPgp9GrUqEGePHmoWbMmMTExOd5JAlWNqKVu3bqaGRMnTtRLLlFt2jRTh0lWQkKCDh8+XCdPnqyqqrt27dKVK1dm/YmCMHHiRF/Om9OsnuHHj7ouWbIkx8+5d+/eoLZr0qSJzpw5M5ujCV5yPytglibz+9qa+DJg1So477ysPebMmTO59NJLueuuuxgwYAAAZ555Judl9YmMMSaXsASVTgcP5mHLFjj//Kw53oYNG2jbti0NGjRg1apVfPLJJwwZMiRrDm6MiXixsbHUq1fP7zAyxO5BpdPatacDkFUDNXz//fcMHz6cbt268dxzz3GGXw9WGWNMiLEElU4LF7qOCpdckrH9VZWvv/6afPnycfvtt9O+fXuuv/76oOdHMcaYSGFNfOm0YEFRKlWC0qXTv+/06dO55JJLuOeee/j8888B1/PGkpMxxpzKElQ6JCTAokVF0/380/r167n33nu5+OKLWbt2LZ999hljxozJniCNMSZMWIJKh9WrYe/efOlu3ps/fz7ffvstzz//PCtWrCAmJoaoKPvRG2NCV3R0NNu3b/c1BrsHlQ5r1rjXtEYYSkhIYNiwYezYsYOnnnqK6667jjVr1lCmTJnsD9IYE/FOPEeUy/8QtgSVDuvWudcKFVLeZtq0aXTs2JG///6byy67jCeeeIKoqChLTsZEkKZNm55Sdscdd9ChQwcOHjxIy5YtT1kfExNDTEwM27dvp1WrVieti42NTfOca9eu5eqrr6Zhw4bMnj2bBg0asHDhQg4dOkSrVq14+eWXAXdl1LZtW8aMGcOxY8cYOXIkVatWZceOHdx9991s2LCBiy++GPf8rPPWW2+dGIHioYceomPHjqxdu5ZrrrmGRo0aMXXqVOrXr8/9999Pjx492Lp1K1999dUp03qkV+5Orzls3TqIilLKlTt1XVxcHK1bt+aSSy4hLi6OIUOGEBsbm+v/gjHG5B4rVqygQ4cOLF68mP79+zNr1iwWLFjApEmTWLBgwYntSpYsyZw5c3j00Ufp168fAC+//DKXXnopixcv5pZbbuHff/8F3GDVn332GTNmzGD69OkMGjSIuXPnArBy5Uqefvppli1bxrJlyxg2bBh//fUX/fr149VXX810fewKKh3WroVSpY6QN2/BU9bt3LmT0aNH8+KLL9K1a1cKFy6c8wEaY0JCalc8p512WqrrS5YsGdQVU3IqVKhAo0aNAPjmm28YOHAg8fHxbNq0iSVLllCjRg0Abr31VgDq1q3L999/D8Cff/554v11111HsWLFADfA7C233MLpp59+Yt/Jkydz4403UrFiRS666CIAqlevTrNmzRARLrroItauXZuhOgTK9QlKRK4B3gXyAJ+oat/sOte6dVC69GGgIAkJCQwdOpQFCxbQv39/atSowYYNG2xAV2OMbxKTyJo1a+jXrx8zZ86kWLFixMTEnDT9RoECBQDIkycP8fHxGT5f4nEAoqKiTnyOiorK1HFPHDPTR/CRiOQBPgSuBaoBd4tItew63333QcuWm/jrr79o2LAhMTExTJky5cQXb8nJGBMK9u7dy+mnn07RokXZsmULY8eOTXOfyy+/nGHDhgEwduxYdu3aBcBll13GqFGjOHjwIAcOHOCHH37gsssuy9b4E+XqBAU0AFaq6mpVPQoMB27KrpNdf/0mpk/vwGWXXcamTZv48ssvmTp1KgULntrkZ4wxfqlZsya1a9ematWqtG7dmsaNG6e5T48ePfjzzz+pXr0633//Pf/73/8AqFOnDjExMTRo0ICGDRvy0EMPUbt27eyuApD7m/jKAusDPscBDbPrZCLC/Pnz6dmzJ126dDlxOW2MMX6Ljo5m0aJFJz4njlaTVOC9oXr16p2431WiRAl+/fXXZPfp3LkznTt3Dvp8SddllAR2JcxtRKQVcI2qPuR9vg9oqKqPJ9muHdAOoHTp0nWHDx+e4XPu3LmT4tk9U2EI2L9/f0R09LB6hh8/6lq0aFHOz6opDoJ0/Phx8uTJk6PnzAorV65kz549J5VdccUVs1X1lCHXc/sV1AagfMDncl7ZSVR1IDAQoF69eprcMwrBio2NTfYZh3Bj9QwvkVJP8KeuS5cupUiRIjl6zn379uX4ObNC4uy+wcjt96BmApVEpKKI5AfuAkb7HJMxJgLl5taonJLen1GuTlCqGg88DowHlgLfqOpif6MyxkSaggULsmPHDktSqVBVduzYka5OZbm9iQ9V/QX4xe84jDGRq1y5csTFxbFt27YcO+fhw4dzXQ/iggULUi65oXhSkOsTlDHG+M2Ped1iY2NzrLu3X3J1E58xxpjwZQnKGGNMSLIEZYwxJiTl6gd1M0JEtgHrMnGIkoC/00zmDKtneImUekLk1DWc6llBVUslLYy4BJVZIjIruSeew43VM7xESj0hcuoaCfW0Jj5jjDEhyRKUMcaYkGQJKv0G+h1ADrF6hpdIqSdETl3Dvp52D8oYY0xIsisoY4wxIckSlDHGmJBkCSoZInKNiPwjIitFpFsy6wuIyAhv/QwRifYhzCwRRF1jRGSbiMzzlof8iDMzRGSwiGwVkWSn+BTnPe9nsEBE6uR0jFkhiHo2FZE9Ad/lSzkdY1YQkfIiMlFElojIYhF5KpltwuU7DaauYfG9JktVbQlYgDzAKuBcID8wH6iWZJsOwADv/V3ACL/jzsa6xgAf+B1rJut5OVAHWJTC+pbAWECARsAMv2POpno2BX7yO84sqGcZoI73vgiwPJl/t+HynQZT17D4XpNb7ArqVA2Alaq6WlWPAsOBm5JscxMwxHv/LdBMRCQHY8wqwdQ111PVP4GdqWxyE/CFOtOBM0WkTM5El3WCqGdYUNVNqjrHe78PNxdc2SSbhct3Gkxdw5YlqFOVBdYHfI7j1H8QJ7ZRN2niHqBEjkSXtYKpK8BtXjPJtyJSPmdCy1HB/hzCwcUiMl9ExopIdb+DySyveb02MCPJqrD7TlOpK4TZ95rIEpRJyxggWlVrABP478rR5D5zcGOe1QTeB0b5G07miEhh4Dugo6ru9Tue7JRGXcPqew1kCepUG4DAq4RyXlmy24hIXqAosCNHostaadZVVXeo6hHv4ydA3RyKLScF853neqq6V1X3e+9/AfKJSEmfw8oQEcmH+4X9lap+n8wmYfOdplXXcPpek7IEdaqZQCURqSgi+XGdIEYn2WY00NZ73wr4Q727lblMmnVN0m5/I64NPNyMBtp4Pb8aAXtUdZPfQWU1ETk78V6piDTA/f/PdX9YeXX4FFiqqm+lsFlYfKfB1DVcvtfk2JTvSahqvIg8DozH9XIbrKqLRaQXMEtVR+P+wQwVkZW4m9J3+RdxxgVZ1ydF5EYgHlfXGN8CziAR+RrX06mkiMQBPYB8AKo6APgF1+trJXAQuN+fSDMniHq2Ah4VkXjgEHBXLv3DqjFwH7BQROZ5Zd2B/0F4facEV9dw+V5PYUMdGWOMCUnWxGeMMSYkWYIyxhgTkixBGWOMCUmWoIwxxoQkS1DGGGMyJK0BipPZ/o6AgW+HpbW9JShjAoiIikj/gM9dRKRnFh37cxFplUXHmuGNXP2vnDzafHRWHD+Z83XPwD4xIvJBdsRjQsbnwDXBbCgilYDngMaqWh3omNY+lqCMOdkR4NZQexLfG7HkBFVtqKq1gJdwo+nX8pa16TlOOqQ7QZnwl9wAxSJynoiME5HZIjJZRKp6qx4GPlTVXd6+W9M6viUoY04WDwwEOiVdkfQKSET2e69NRWSSiPwoIqtFpK+I3CMif4vIQhE5L+AwzUVklogsF5Hrvf3ziMibIjLTG5T3kYDjThaR0cCStAIXkRu8K6u5IvKbiJT2ynuKyFARmYJ7wLyUiEzwmlk+EZF1iQlZRO714p4nIh97sfUFCnllX6W0nVd+v1e3v3EPmZrIMxB4QlXrAl2A//PKKwOVRWSKiEwXkTSvvCxBGXOqD4F7RKRoOvapCbQHLsA9+V9ZVRvgxi98ImC7aNw0J9cBA0SkIPAgbiie+kB94GERqehtXwd4SlUrBxHDX0AjVa2Nmzqla8C6akBzVb0bN8LEH14zy7d4oxKIyAXAnbgmmFrAceAeVe0GHPKu0O5JaTtxw2K9jEtMl3rnNBFE3KC2lwAjvZEvPsbNaQVu5KJKuNFO7gYGiciZqR3PhjoyJglV3SsiXwBP4oaOCcbMxLHeRGQV8KtXvhC4ImC7b1Q1AVghIquBqkALoEbA1VlR3H/ko8DfqromyBjKASO8RJEfCNxvtKom1uVS4BavruNEZJdX3gw3GPBMb2i3QkByzTApbdcQiFXVbd7PYQTur2YTOaKA3d4fLknF4SaOPAasEZHluH/nM1M7mDHmVO/grmxODyiLx/s/IyJRuCSQ6EjA+4SAzwmc/Idg0rHFFDfr6xMB95EqqmpigjuQjpjfx81+fBHwCFAwYF0wxxFgSEAcVVS1Zya2MxHGmwpkjYjcDm6wWxGp6a0ehbt6wmtSrgysTu14lqCMSYaq7gS+wSWpRGv5b7qRG/EGYk2n20UkyrsvdS7wD26w3kfFTauAiFQWkdNTO0gKivLflBJtU9luCnCHd64WQDGv/HeglYic5a0rLiIVvHXHEuNLZbsZQBMRKeFte3sG6mByEXEDFE8DqohInIg8CNwDPCgi84HF/DdL93hgh4gsASYCz6hqqqOuWxOfMSnrDzwe8HkQ8KP3H28c6bu6SfQv8DdwBtBeVQ+LyCe4e1NzxLWZbQNuzsCxe+La/ncBfwAVU9juZeBrEbkP98tlM7BPVbeLyAvAr94V4jHgMWAd7sb3AhGZ492HOmU7VZ0urkv+NGA3MC8DdTC5iHdPMzmndIDwRljv7C1BsdHMjYkwIlIAOO5Nt3Ix8FEK9wyM8ZVdQRkTef4HfONd/RzFPZ9iTMixKyhjjDEhyTpJGGOMCUmWoIwxxoQkS1DGGGNCkiUoY4wxIckSlDHGmJD0/5QVvnacn4RvAAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"perf_h2o.plot_uplift(metric=\"lift\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Qini value and Average Excess Cumulative Uplift (AECU)\n",
"\n",
"Qini value is calculated as the difference between the Qini AUUC and area under the random uplift curve (random AUUC). The random AUUC is computed as diagonal from zero to overall gain uplift. \n",
"\n",
"The Qini value can be generalized for all AUUC metric types. So AECU for Qini metric is the same as Qini value, but the AECU can be also calculated for Gain and Lift metric type. These values are stored in ``aecu_table``.\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
"
\n",
"
\n",
"
AECU values table: All types of AECU value
\n",
"
uplift_type
\n",
"
qini
\n",
"
lift
\n",
"
gain
\n",
"
AECU value
\n",
"
6000.1130005
\n",
"
0.0188005
\n",
"
6992.6777202
\n",
"
\n",
"
\n"
],
"text/plain": [
"AECU values table: All types of AECU value\n",
"uplift_type qini lift gain\n",
"------------- ------- --------- -------\n",
"AECU value 6000.11 0.0188005 6992.68"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"perf_h2o.aecu_table()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Normalized AUUC\n",
"\n",
"To get normalized AUUC, you have to call ``auuc_normalized`` method. The normalized AUUC is calculated from uplift values which are normalized by uplift value from maximal treated number of observations. So if you have for example uplift values [10, 20, 30] the normalized uplift is [1/3, 2/3, 1]. If the maximal value is negative, the normalization factor is the absolute value from this number. The normalized AUUC can be again negative and positive and can be outside of (0, 1) interval. The normalized AUUC for ``auuc_metric=\"lift\"`` is not defined, so the normalized AUUC = AUUC for this case. Also the ``plot_uplift`` with ``metric=\"lift\"`` is the same for ``normalize=False`` and ``normalize=True``.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABHh0lEQVR4nO3deZxN9f/A8dfbWMa+9pOSJUSkbJEilFJUWiRK0UJR+aJsbYpvK+3fFopEWVtVlJQpLWQJkV3KJEKWGeuMef/++JzRNWa5Y+bOuXPv+/l43Mfce865577Pvdz3/ZzP57w/oqoYY4wx4aaA3wEYY4wx6bEEZYwxJixZgjLGGBOWLEEZY4wJS5agjDHGhCVLUMYYY8KSJSgTcUTkURF5x+84giEiPUTku4DHiSJyune/qIh8IiJ7RGS6f1GGPxFpKSJr/I7D5C5LUCbXiMiNIrLI+5L9S0RmiUgLv+PKjIiMF5H/5uD5cSJyR5plrUUk/kT2p6olVHWj97ATUBEor6rXBxOrOH1FZIWI7BOReBGZLiL1TySe/EJV56lqbb/jMLnLEpTJFSIyAHgBeAL3pVoFeBXo6GNY+V1VYK2qJmfjOS8C/wH6AuWAM4CPgA7ZfXERKZjd5xiTq1TVbnbL0Q0oDSQC12eyzXjgvwGPWwPxAY83AQOB5cA+YCwu0c0CEoA5QNn0nhvw/Lbe/UeBdwLWTQe2AnuAb4F63vJeQBJw2Iv/E2/5KcD7wHbgN6BvJscVB9yRZlnaY1NcwtgI7ABGAgW8dT2A79JsWxN4zIsryYvtzvRiTfO6tYAjQNNg483g9e8G1nnH/howKs0+PgYGZPe9CuLfUQzwrPce/Qbc48VT0Ft/K7DK+/ewEbgzi39P93v/nvYAU4FYv/+v2C17N2tBmdzQHIgFPszhfq4DLsH96r8Sl5weAE7Ctfb7nuB+Z+G+vP8PWAK8C6CqY7z7z6g7tXaliBQAPgGWAacCFwP9RKTdiR6U5xqgCdAI16q8LbONVXUYrjU61YttdNpY03naxbgv6Z9yGOvVQDOgLjAZuEFEBEBEygKXAlNC8F71BC4HGuDep6vTrP8buAIohUtWz4tIo0z21xm4DKgOnI1LxiYfsQRlckN5YIdm71RUel5W1W2q+icwD1igqj+r6kFc8mt4IjtV1XGqmqCqh3Ctq3NEpHQGm58LnKSqw1X1sLr+oDeALify2gGeVtV/VPUP3KnQrjncX3rKA3/lwn6e9GI9gPscFGjpresE/KiqW8j996oz8KKqxqvqLuCpwJWq+pmqblDnG2B2QFzpeUlVt6jqP7hE2uAE4zI+sXPMJjfsBCqISMEcJqltAfcPpPO4RHZ3KCIxwOPA9biWWIq3qgLu1E9aVYFTRGR3wLIY3Bd1epKBQmmWFcKdjgu0OeD+77hTY7ltJ1ApF/ZzNFZVVRGZgkuo3wI3AqkjJIN+r0SkCvBrwH7T+yxP4dj3KfA+InI5MAzXwi4AFAN+yeQ4tgbc309o3nMTQtaCMrnhR+AQx5+SCbQP94WS6uQcvN4x+/KS0EkZbHsj7pRaW1xfWbXUp3l/05bz3wz8pqplAm4lVbV9Bvv/I2CfqarjklCg0wLuVwG2ZLC/zGQ19cBXQGURaZLJNsF8DmlfZzLQSUSq4k79ve8tD/q9UtU/vFOTJTJITuBaf5UDHh99z0SkiPe6o4CKqloGmMm/n6OJQJagTI6p6h7gEeAVEblaRIqJSCERuVxEnvE2Wwq0F5FyInIy0C8HL7kWiBWRDiJSCHgIKJLBtiVxyXMn7ov5iTTrtwGnBzz+CUgQkcHedUgxInKWiJybwf6nAreKSFNviPcZQH9gSprtBopIWRE5DTfKbmqQx5pZrMdQ1XW4kZOTvaHuhUUkVkS6iMgQb7OlwLXeZ1QTuD2rF1XVn3EDF94EvlDV3d6q7L5XWZkG/EdEThWRMsDggHWFcZ/xdiDZa01deoKvY/IJS1AmV6jqs8AAXLLYjvt1fQ9uiDPARFxn+iZc38GJfEGnvtYeoA/uC/NPXKsgo+uOJuBaM3/iTjHNT7N+LFBXRHaLyEeqegTXEd8AN5Is9Ys53T4rVf0CGAK8hTtlOBN4GxiTZtOPgcW4BPGZ97rZdUysGWzTF/gf8AqwG9iAG6Dxibf+edxIwG1enO8G+dqTcK3QSakLsvteBeEN3L+N5cDPuPcyGTiiqgnesU0DduFaxjNO8HVMPiGqNmGhMaEkIgrUUtX1fseSn3itpNdVtarfsRh/WAvKGBMWvNOE7UWkoIicihsQkdNLF0w+ZgnKGBMuBHeB8i7cKb5VuL5NE6XsFJ8xxpiwZC0oY4wxYckSlIl6gRXJReQmEZmdy/uvJiJqxVeNyR5LUCbkRGSTiPwtIsUDlt0hInE+hpUuVX1XVX25vsZLlLu8i1LTLs9wSo+MEmDa6TlEpJKIjPWmQkkQkdUi8ljg5xJknNVEZK6I7Pf20TaTbcuJyFQR2SkiO0TkXREp5a2rIm5qlsCbish9Ac+/V0R+E5G94qZyaRGwbqC4aUUSvG0GZuc4TPizBGXySgzuAtUc8S6Gjbh/tyJSDVdXToGrQrD/criKH0WB5qpaEleYtwxQI5u7m4wbxFAeeBB4T0QyquTxX6AsrrpGDVyF+kch3eoS9XGlqN73Ym6Gq8fXCXdt1VjgQ69yCLhBFbd4+78MuEdEcloz0YSRiPuPbsLWSOB+r0LAcUTkfBFZKG722IUicn7AujgReVxEvsfVVDvd+6XdR0TWeb+gR4hIDRH5wfu1PU1ECnvPLysin4rIdq+F8qmIVM4gjqMz3IrIoDS/7pNEZLy3rnRAa+RPEflv6henV1FhlNdi2EhwczHdgruIeDzQPah3NHsG4Kap6KaqmwBUdbOq/kdVlwe7E3GVMhoBw1T1gKq+j6uHd10GT6kOfKSqe70LrD8E6mWw7S3At6nx4UpIrVTVxepGc03A1VD8Py/+Z1R1iaomq+oa3MXQFwR7LCb8WYIyeWURbi6i+9Ou8H7dfwa8hPtV/hzwmYiUD9jsZtz8TSX5t85dO6AxcB4wCFe9oRuuhttZ/FsxvACu0kNVXB28A7hqC5nyvgBTf92fiauQkVoBYzyuykFNXJX1S4HU03A9cRUWGuKm2OiU1Wvhvpzf9W7tRKRiEM/JjrbAB6qaktEGIrJcXJWK9G6vepvVAzZ6lR1SLSPjpPMKcIX3I6EsLpHNSue1U1tDbwcsngXEiEgzL/nfhqvEsTWD57cEVmZ0fCb/sQRl8tIjwL3pnA7qAKxT1Yner+HJwGrcnFCpxqvqSm99aqXwZ7xf5iuBFcBsVd3o/VKfhTc9h6ruVNX3VXW/98X6ONAq2KBFpCiuZNOLqjrLSx7tgX6quk9V/8aVEEo9vdQZeMFrofwDPJnF/lvgkuc0VV2MK090Y7DxBSnLqThU9ew0hV8Db328zUpwfBX4PbgfDulZgqujt9O7HcHVC0yrBe7033sByxJwp/u+w9VTHAb00vSvjXmUf3+ImAhhCcrkGVVdAXyKq10X6BSOr/79O24SvFSbOV5Q03OIK4w6WkR+F5G9uGkjygT0ZWRlLLBGVZ/2HlfFTanxV2oLAxiNd+qJ46eNSHtsaXXHJdcd3uNJHHuaL6spPZIDlmW0TW5NxZGImzAwUClcMknPNFxx35Ledhv4d7qOQN2B91U1MWDZ7biJCevhklw34FMROWbaDBG5B9f66uDN+WUihCUok9eG4U6BBSafLbgv/UBVcAVeU+XkivL7gNpAM1UtBVzoLc9yqgZxVcDP4Niq35txv+grBLQwSqlq6mmuvzh+eo2M9l8U1+JqJSJbRWQrrhr6OSJyjrdZVlN6/IVLRJltMwe4JrMBJiKyMp1Rdam3173NVuL6AANbTOeQ8am1BsBor6WZCLyOa32mfQ+u59jTe6nP/VRV16pqiqp+7h1rYP/kbbgfPBerakYFg00+ZQnK5CmvYOpUjp2+fSZwhojcKK4O2w246cY/zaWXLYlrUe32+ruGBfMkccVK+wLXeLPLph7DX7iq28+KSCkRKeAN0Eg9bTgN6Csilb1+l7QtxkBX40571cV9ITfA9XfNw7UKIIspPbyq4u8Dj4tIeXFTnXT19pna3/McrgXztrh5nRA3rcVzInK2t596gaPq0tzu8rZZi+sHGiZuKo9rcNOpp84RldZC4A5xdfaK4voR0w7KuAZX3mhuOs/tICKne8d9Ce7Hwgov/ptw06dcom42XxNhLEEZPwwHjl57o6o7cYMK7sOdihoEXBFwyiunXsANr96BGyn3eZDPuwE3EeKqdFoSt+BOO/2K+3J9j39Pob0BfIEbPLAE+CCT1+gOvOUNud6aesMN4rhJ3CzFwUzp0Qf4B/fl/zduqpMOqroNwOsLOx/X0logIgm4CQ73ANmtst4FN/gjdVr2Tqq6HY5e6BzYmroN17KLx7WIT+f4UYrdgYnp9C1NwCXhOGAvbhDNnaq62lv/X1zf2sJ0Ph8TAawWnzHGmLBkLShjjDFhyRKUMcaYsGQJyhhjTFiyBGWMMSYs5bvy/xUqVNBq1arlaB/79u2jePFsFXDO16LteMGOOVrYMUeGxYsX71DV4woO57sEVa1aNRYtWpSjfcTFxdG6devcCSgfiLbjBTvmaGHHHBlEJN1qK3aKzxhjTFiyBGWMMSYsWYIyxhgTlixBGWOMCUuWoIwxxoSlkCUoERknIn+LyIoM1ouIvCQi672ZPBuFKhZjjDH5TyhbUOOByzJZfzlQy7v1Al4LYSzGGGNySVISJCdnvV1Ohew6KFX9VkSqZbJJR2CCV2J/voiUEZFK3lw7xhhjQkAV9u+HxERISHB/s3N/61ZYtgxgGVWrJjJ+/AWE6rIsPy/UPZVjp8WO95Ydl6BEpBeulUXFihWJi4vL0QsnJibmeB/5SbQdL9gxR4tIP2ZVSEoqwIEDMRw4EMP+/TH8809BFi1axv79MUeXB9727y94zPbpbaOa5WTSAMTEpFCs2BGKFTtCbOwRihY9QsGCv1OhwuPs2DGNLVua8d13Y3DTuOW+fFFJQlXH4E3O1qRJE83pVdSReCV2ZqLteMGOOVqE2zEnJ59YqyTt/cBlwZ5KK1AASpRwt5Il3d+yZaFKlWOXZed+4cIFcD1Bhdi9ezdPPvkkL774IgCDBg1kyJAhlC1bNmTvp58J6k/gtIDHlb1lxhgTcikpsG/fiSeO9O4fOhT86xcvfnxiqFABqlXLPHFs3LiMCy4457jlRYuCBNcwOiG7d+/mpZde4oYbbmDEiBFUqVIldC/m8TNBzQDuEZEpQDNgj/U/GWPSowoHDhyfGH76qRw7dmQviaTe378/+NcvUuT4hFGyJFSq9O/y7LRQihWDmJgTey/i4nZxwQUn9tzsSElJYcqUKcTFxTFmzBiqVavGpk2bqFixYuhf3BOyBCUik4HWQAURiQeGAYUAVPV1YCbQHlgP7AduDVUsxpi8dehQzloiae8nJroWz/HOPm5JTMy/CSQwMVStemySCDahFC8OhQqF/C0LK19//TUDBw5kyZIlNGjQgN27d1OmTJk8TU4Q2lF8XbNYr8DdoXp9Y0xwkpOPTQQnmkQC7yclBffaIuknhpNPhpo1s04oa9cuoVWrRmn6TUJ7qiuSxcfH06tXL2bNmkWVKlWYOHEiN954IwUK+FPTIV8MkjDGOCkp7tRUeklk/vyT2LAh+wnl4MHgX79Yscw74rPbQilWLGfJpFChvZx11ok/3zhHjhwhJiaGUqVKsWHDBkaOHMk999xDbGysr3FZgjImDCQnw6ZNsH79sbc//zw2oezbl9le6h3zqHDh9BNDxYon1m9SvPiJ95uY8LR7926eeuop5syZw/z58ylVqhS//vorMWHyQVuCMsYH+/bBggXw3Xcwbx78+OOxyad4cXeK67TT0u9PSS+J/PrrT1x0UdOj/SaFC/t3fCa8HTp0iNdee40RI0awa9cuunXrxr59+yhdunTYJCewBGVMnvj7b5eMUm9LlsCRI+701jnnwK23QqNGUKuWS0wVK2b/1Nf+/fvJg5G/Jp/buHEjbdu25bfffuOSSy7h6aefpmHDhn6HlS5LUMbkMlV3ei4wIa1d69bFxkKzZjBkCLRoAc2bQ+nS/sZrosO2bduoWLEiVapUoWnTprz++utceumlfoeVKUtQxuRQcjIsXXpsQtq2za0rV84lop493d9GjezUm8lbK1asYMiQISxYsIANGzZQqlQppkyZ4ndYQbEEZUw2uRFz/yaj+fP/7T+qXh3atXPJqEULqF3blaAxJq/9+eefPPLII4wfP56SJUvywAMPUDif/TqyBGVMFrZtO7Z19PPPrv+oQAHXf3TbbS4ZXXABnHqq39EaA7/99hv16tXjyJEj9OvXjwceeIDy5cv7HVa2WYIyJoAqrFv3bzKaN8/1J4GrddasGQwd+m//UalS/sZrTKrDhw+zYMECWrZsSfXq1Rk2bBidO3emevXqfod2wixBmaiWlOT6j+bN+zcpbd/u1pUv7xLRXXe5vw0bWv+RCT+qyvTp0xk6dCjx8fH89ttvnHLKKQwePNjv0HLMEpSJKgkJx/cfpRYNPf10aN/+2P4jK5ljwtk333zDwIEDWbhwIfXr1+fjjz+mUqVKfoeVayxBmYj211/w/ff/nq5butSVCypQABo0gDvu+Lf/6JRT/I7WmODFx8dz8cUXU6lSJcaPH0+3bt3C6iLb3GAJykQMVXe90bx58N57dbjjDtiwwa0rWhTOOw8efPDf/qOSJf2N15js2rJlCx9//DGtW7emcuXKfPbZZ1x44YUULVrU79BCwhKUybcOH3Yj6gJH2O3Y4daVLl2ONm2gd29o2dL1H0XblAkmcuzdu5eRI0fy7LPPkpycTP/+/alSpQrt2rXzO7SQsgRl8o3kZFe/bvZs10qaP99NYgdQowZcccW//UdbtvxAmzatfYzWmJw7fPgwY8aMYfjw4Wzfvp0uXbpw5ZVX5slstuHAEpQJa5s3wxdfwOefw5w5sGeP6z9q2BB69fq3/yhtv/BfNjeziQAJCQk89NBDNGzYkJEjR9KkSRPi4uL8DivPWIIyYeXAAdc6+vxzl5h+/dUtr1wZOnWCyy6Diy92cxAZE4m+/fZbJk6cyOjRoylfvjxLly6latWqSBQOKbUEZXy3Zg3MmuUSUlycm0CvSBG48EK4/XZXOqhuXRvybSLbqlWrGDJkCDNmzODUU0/ljz/+oFq1alSrVs3v0HxjCcr4Yvt2mDwZ3n7bTT0B7rqjXr1cK6lVKzfbqjGRbs+ePQwcOJCxY8dSokQJnnzySfr27Usx+w9gCcrkra1bYdQoeOUV11Jq1Aiefx6uvhqi+IeiiUKqiohQtGhR5s2bx7333stDDz1EhQoV/A4tbFiCMiF1+DD88IMbeTd7tmsticAtt8CAAVC/vt8RGpO3kpKSeOONNxg7dizz5s2jWLFiLFu2LN9VGs8LlqBMrktMhOnT4b334Jtv3FQUMTHu4tjHHoPOnd3pPGOiiarywQcfMHToUNatW0erVq3YuXMnxYoVs+SUAUtQJleouuuSxo6FqVNdkjr9dOjRAy69FFq3tsrfJnrt2rWLDh068OOPP1K3bl0++eQTOnToEJUj87LDEpTJkV274J13YMwYWLECiheHG25wcySdf76NvDPRbffu3ZQpU4YyZcpw2mmncdttt9GjRw8KFrSv3mDYu2ROyMqV8PTT7lTewYNw7rnwxhsuOVmNOxPttm7dyqOPPsrkyZNZvXo1lSpVYurUqX6Hle9YgjLZsnixm7Dvyy9da+nWW6FnT1fZwZhol5iYyKhRoxg1ahSHDh2id+/e1r+UA5agTNC++w7atnUDHu66CwYPtqHhxqTau3cvderU4a+//uL666/niSeeoGbNmn6Hla9ZgjJZ+uUXeOABVwvv5JNdwdaKFf2Oyhj/qSqLFy+mSZMmlCpVin79+tGqVSuaNWvmd2gRoUAody4il4nIGhFZLyJD0llfRUTmisjPIrJcRNqHMh6TffPnu5JDn34KZcrAxImWnIwB+OGHH2jRogVNmzZl+fLlAAwaNMiSUy4KWYISkRjgFeByoC7QVUTqptnsIWCaqjYEugCvhioek31ff+1O6ZUr5xLVmjVubiVjotmaNWu49tprueCCC9i4cSOjR4+mbt20X20mN4TyFF9TYL2qbgQQkSlAR+DXgG0USL06pjSwJYTxmGz45BO4/no3z9KXX9p06MYA7Nu3j2bNmnHkyBGGDx/OgAEDKF68uN9hRSxR1dDsWKQTcJmq3uE9vhlopqr3BGxTCZgNlAWKA21VdXE6++oF9AKoWLFi4ylTpuQotsTEREqUKJGjfeQn2T3er7/+P554og41auzjmWeWU7p0UgijC41o+4zBjjlUDhw4wNdff0379u0REX788Ufq1KlDWZ/mfInEz7lNmzaLVbXJcStUNSQ3oBPwZsDjm4H/pdlmAHCfd785rnVVILP9Nm7cWHNq7ty5Od5HfpKd4x0zRlVEtWVL1T17QhdTqEXbZ6xqx5zbkpKSdPTo0XryyScroPPnzw/Za2VHJH7OwCJN5/s+lIMk/gROC3hc2VsW6HZgGoCq/gjEAlbK1yfPPeemu2jXzk0YaKWJTDRSVT7++GPq16/PnXfeSY0aNfjhhx9s8IMPQpmgFgK1RKS6iBTGDYKYkWabP4CLAUTkTFyC2h7CmEw6VOHRR+G++9ystR9/bHMxmeh16NAh7r77blSVjz76iHnz5tG8eXO/w4pKIUtQqpoM3AN8AazCjdZbKSLDReQqb7P7gJ4isgyYDPTwmnsmj6i6xPTYY66w6+TJYBe+m2izdu1a7r77bg4fPkxsbCxfffUVK1asoGPHjlbQ1UchvVBXVWcCM9MseyTg/q/ABaGMwWTsyBFXEeLNN6FvXzdxYIGQXhlnTHjZtm0bw4cPZ8yYMRQpUoRbbrmFZs2aUdvmgwkL9nUUpQ4fhptucsnpoYfghRcsOZnocfjwYUaMGEHNmjUZPXo0vXr1YsOGDdbPFGas1FEUOnDAXeP02WeuIvmgQX5HZEzeKliwIDNmzODSSy/liSeesBZTmLLfzFEmIQHat4eZM+G11yw5meigqnzyySecf/757NixgwIFChAXF8f7779vySmMWYKKIv/840oXzZvnaurddZffERkTegsWLKB169ZcddVV7Ny5k/j4eACrAJEPWIKKElu3umnXly6F9993/U/GRLKkpCRuuOEGzjvvPFavXs1rr73GihUraNCggd+hmSBZgooC27YV4cILYcMG1+/UsaPfERkTOvv37wegUKFCxMbGMmzYMNavX89dd91FoUKFfI7OZIcNkohwa9dC374NOXTIFX09/3y/IzImNPbv38/zzz/Ps88+yw8//ECdOnV4++23/Q7L5IC1oCLY8uVueozDhwswd64lJxOZkpOTGTt2LLVq1eKhhx6iVatWNs16hLAWVIRasAAuuwxKlICRI5fSsGFTv0MyJtcdOXKEZs2asWTJEs477zymTp1KixYt/A7L5BJrQUWguXPh4ouhfHk3Yq9Klf1+h2RMrlq9ejUAMTEx3HTTTbz33ntHZ7g1kcMSVIT59FO4/HKoVs0lp2rV/I7ImNyzYcMGunTpwplnnsmcOXMAGDBgANddd53VzItAWSYoEblWRNaJyB4R2SsiCSKyNy+CM9kzdSpccw3Urw/ffAOVKvkdkTG5Y8eOHfTr148zzzyTTz75hIcffpimTe20daQLpg/qGeBKVV0V6mDMiXvzTTeXU8uWbrp2m8vJRIqUlBSaN2/Oxo0buf3223n00Uc55ZRT/A7L5IFgEtQ2S07h7Y03XHK67DJ3Ea7N5WTyuyNHjvDee+9x3XXXUbBgQV544QWqV69O3bp1/Q7N5KFg+qAWichUEenqne67VkSuDXlkJijx8dC/P1xyiU00aPI/VWXWrFk0aNCALl268NFHHwHQoUMHS05RKJgEVQrYD1wKXOndrghlUCZ4/fu7eZ1Gj7aJBk3+tnjxYtq2bUv79u05cOAA06ZN47rrrvM7LOOjLE/xqeqteRGIyb7Zs+G992DECKhe3e9ojDlxqkqPHj3YunUrL7/8Mr169bKLbU3GCUpEBqnqMyLyMnDcNOyq2jekkZlMbdni+p1q1oT77/c7GmOyb+fOnTz77LMMGTKEUqVKMXXqVCpXrkwpG+FjPJm1oFIHRizKi0BM8I4cgWuvhe3b4fPPITbW74iMCd6BAwd46aWXePLJJ0lISKBJkyZce+211sdkjpNhglLVT7y/Vm0xzAwf7koZvf22G1ZuTH6gqkyYMIGHHnqI+Ph4rrjiCp566inq1avnd2gmTGXZByUiJwGDgbrA0d/qqnpRCOMyGZg0ySWoW2+Fm2/2OxpjgiciTJ48mUqVKjFx4kRat27td0gmzAUziu9d3Om+6sBjwCZgYQhjMhkYMMBNNNiyJbz+OlhlFxPulixZQvv27dm0aRMAkydPZv78+ZacTFCCSVDlVXUskKSq36jqbYC1nvLYokXw/PNw9dXuYlwb4GTC2aZNm+jWrRuNGzfmp59+Ys2aNQCULVuWAgWsBKgJTjD/UpK8v3+JSAcRaQiUC2FMJh1Dh0KFCq7f6aST/I7GmPSpKoMHD6Z27dq8//77PPDAA2zYsIF27dr5HZrJh4IpdfRfESkN3Ae8jLtwt39IozLHmDPH3Z5/3mrsmfCUlJREoUKFEBH27dtHt27deOyxx6hcubLfoZl8LMsWlKp+qqp7VHWFqrZR1caqOiMvgotWqtCjB5x1FuzaBYMHQ9Wq0Lu335EZc6yUlBQmTpxIzZo1+eGHHwB4+eWXGTt2rCUnk2PBjOJ7KZ3Fe4BFqvpx7odkpk51p/IATj0VDhxwy4oU8TcuYwLNnj2bQYMGsWzZMho3bkzBgu7rxOZlMrklmD6oWKABsM67nQ1UBm4XkRdCFlmUSkqCnj2haVPo2NElp3PPhc6d/Y7MmH917tyZdu3asXfvXiZNmsRPP/1k8zOZXBdMH9TZwAWqegRARF4D5gEtgF8ye6KIXAa8CMQAb6rqU+ls0xl4FFdOaZmq3pidA4gkyclw3XWQmAj33QcNGsDvv8PIkX5HZgzEx8dzyimnUKBAAdq0acP5559P7969KWJNexMiwSSoskAJ3Gk9gOJAOVU9IiKHMnqSiMQArwCXAPHAQhGZoaq/BmxTCxiKS4C7ROT/TvA4IsL997vJBi++2CWqmBj4+We/ozLRbteuXTzxxBO8/PLLjBs3jhtvvJHe1iFq8kCwM+ouFZE4QIALgSdEpDgwJ5PnNQXWq+pGABGZAnQEfg3YpifwiqruAlDVv7N9BBEiMRHGjXOn9T74AOxSEeO3gwcP8sorr/D444+ze/duunfvTkurrWXykKgeV6j8+I1EKuESDsBCVd0SxHM6AZep6h3e45uBZqp6T8A2HwFrgQtwpwEfVdXP09lXL6AXQMWKFRtPmTIly5gzk5iYSIkSJXK0j9z0yy+lGTmyNps3F+N//1tCvXp7c3X/4Xa8ecGOOefuu+8+lixZQtOmTenVqxc1atTItX3nFvucI0ObNm0Wq2qT41aoakhuQCdcv1Pq45uB/6XZ5lPgQ6AQrpTSZqBMZvtt3Lix5tTcuXNzvI/csnGjarFiqqA6dWpoXiOcjjev2DGfmDlz5ui+fftUVXX27Nn65Zdf5nifoWSfc2TAjQo/7vs+lCeS/gROC3hc2VsWKB6YoapJqvobrjVVK4QxhZWkJOjaFQ4ehC++sJF6xj/Lli2jXbt2tG3bltGjRwNwySWX0LZtW58jM9EslAlqIVBLRKqLSGGgC5D2At+PgNYAIlIBOAPYGMKYwsqIEW7ajClT4NJL/Y7GRKM//viD7t2707BhQxYuXMhzzz1Hnz59/A7LGCC4QRKISAuglqq+5U2/UcJr8WRIVZNF5B7gC1z/0jhVXSkiw3HNuRneuktF5FfgCDBQVXfm5IDyiwUL4PHHXXXy66/3OxoTrXr27Mk333zDwIEDGTJkCGXLlvU7JGOOCqaSxDCgCVAbeAvXX/QObmBDplR1JjAzzbJHAu4rMMC7RY05c9xovZNOgsce8zsaE00OHTrEq6++yg033MApp5zCiy++SLFixahSpYrfoRlznGBaUNcADYElAKq6RURKhjSqCLZ3L1x5pet3+uUXOP10vyMy0SAlJYUpU6bw4IMPsmnTJkSEfv36UadOHb9DMyZDwfRBHfZaOgrgXf9kTtDHH7vkNHGiJSeTN77++mvOPfdcbrrpJsqUKcPs2bPp16+f32EZk6VgWlDTRGQ0UEZEegK3AW+ENqzINWUKVKni+p6MyQvjx49nx44dTJw4kRtvvNEmDDT5RjDTbYwC3gPex/VDPaKqL4c6sEg0aRLMnAm33GLTtZvQ2bx5M7feeis/e3WyXnjhBdasWUO3bt0sOZl8JZhBEgOAqar6ZR7EE9EeecRVJn/4Yb8jMZEoMTGRIUOG8OKLL5KSkkKLFi1o2LAh5crZBNgmfwrmFF9JYLaI/ANMBaar6rbQhhV51q2DDRugf38oXNjvaEykef311xk8eDAJCQl069aNESNGULVqVb/DMiZHgjnF95iq1gPuBioB34hIZkViTTqmT3d/L7/c3zhM5EhJSUktGcZff/3FGWecweLFi5kwYYIlJxMRsnNC+m9gK7ATiOppMbIrORn+9z9XLcJG7pncEBcXR7NmzZgxwxVneeSRRxg5ciQNGzb0OTJjck+WCUpE+nhTbXwFlAd6qurZoQ4sksyaBX/9BTaFjsmpFStWcMUVV9CmTRu2bdt2dHr1mJgYnyMzJvcF0wd1GtBPVZeGOJaINXYsVKwIHTr4HYnJzx588EGeeuopSpYsydNPP829995L0aJF/Q7LmJDJMEGJSClV3QuM9B4fMxRIVf8JcWwR4fPPYcYMGDgQChXyOxqT3+zZs4fY2FiKFClCnTp16NevHw888ADly5f3OzRjQi6zU3yTvL+LgUXe38UBj00W9u+Hbt2gVCkYEFXVBk1OHT58mJdeeokaNWrw2muvAXDzzTfz7LPPWnIyUSPDFpSqXuH9rZ534USWV1+FnTth3jx3is+YrKgq06dPZ+jQoWzcuJGLLrqICy+80O+wjPFFMIMkvgpmmTne229D8+bQooXfkZj8olevXtxwww0UL16cWbNmMWfOHBo1auR3WMb4IrM+qFigGFBBRMoCqcV5SgGn5kFs+dqCBbBihRtebkxmfv31VypWrEj58uXp0aMHLVq0oFu3bjYyz0S9zFpQd+L6m+pwbP/Tx4B97WZh2DAoX97V3TMmPVu2bKFnz57Ur1+fp556CoALLriA7t27W3Iyhsz7oF4EXhSRe604bPZMmwZffAHPPAMlbeYsk8bevXt55plneO6550hOTqZv374MHjzY77CMCTtZXgelqi+LyFlAXSA2YPmEUAaWXyUmwoMPQv360Lev39GYcPSf//yH8ePH06VLFx5//HFOt/IixqQr2CnfW+MS1EzgcuA7wBJUOh55xBWFnTMHihTxOxoTDlSV9957j7PPPpvatWvz8MMP06dPH84991y/QzMmrAVTi68TcDGwVVVvBc4BSoc0qnxqzx544w03GeFFF/kdjQkH3377Lc2bN6dz5868+uqrAJx++umWnIwJQjAJ6oCqpgDJIlIKVzT2tNCGlT+9+aY7xde/v9+RGL+tWrWKjh070qpVK+Lj4xk3bhzPPfec32EZk68EU4tvkYiUwU3zvhhIBH4MZVD5UXIyvPQStGoFdtmKGTt2LHFxcTzxxBP85z//oVixYn6HZEy+E8wgiT7e3ddF5HOglKouD21Y+c8HH8Aff7gkZaJPQkICo0aNolWrVlx00UU89NBDDBkyhAoVKvgdmjH5Voan+ESkUdobUA4o6N03nqQkuOEGd/+KK/yNxeStpKQkXn31VWrWrMnw4cOZO3cuAGXKlLHkZEwOZdaCejaTdQrYMADPV17hp2eeAbu+Mnp89tln9O/fn3Xr1tGqVSs++eQTmjZt6ndYxkSMzC7UbZOXgeRn06dD6dJ23VO0UFVEhE2bNlG4cGE+/fRT2rdvf3TyQGNM7gjmOqh0i/XYhbrOkSPw5Zdw8cV23VOkW716NUOHDqV9+/b07NmTO++8kzvvvJOCBYMZa2SMya5ghpmfG3BrCTwKXBXCmPKVkSNh82bo0sXvSEyobN26ld69e3PWWWfx1VdfkZSUBEDBggUtORkTQsGM4rs38LE35HxKMDsXkcuAF4EY4E1VfSqD7a4D3gPOVdV8NRniF1/AWWdBp05+R2JC4Y033qB///4cOnSIPn368PDDD3PSSSf5HZYxUSGYFlRa+4AsJzEUkRjgFVxppLpAVxGpm852JYH/AAtOIBZfpaTA4sXQsiVY90PkSEpK4sCBAwCccsoptG/fnlWrVvHSSy9ZcjImDwUzYeEnIjLDu30KrAE+DGLfTYH1qrpRVQ/jWl0d09luBPA0cDAbcYeFVasgIQEaN/Y7EpMbVJUPP/yQs846i//+978AdOjQgWnTplGzZk2fozMm+gRzAn1UwP1k4HdVjQ/ieacCmwMexwPNAjfwrqc6TVU/E5GBGe1IRHoBvQAqVqxIXFxcEC+fscTExBzvA+CJJ+pQqND/UazYT8TFhW9+za3jzU+ye8wrVqzg9ddfZ+XKlVStWpWSJUvmu/fMPufoEFXHrKpB3XAz6ZZLvQWxfSdcv1Pq45uB/wU8LgDEAdW8x3FAk6z227hxY82puXPn5ngfBw6oli2r2r17jncVcrlxvPlNdo75ySefVEArVaqkY8aM0aSkpNAFFkL2OUeHSDxmYJGm830fzDDzXsBw3Cm4FNzU7wpkNYnNnxxbVLaytyxVSeAsIM67fuRkYIaIXKX5YKDEhAmwa5fNmJtfbdu2jZSUFCpVqkSHDh1ITk6mf//+FC9e3O/QjDGeYE7xDQTOUtUd2dz3QqCWiFTHJaYuwI2pK1V1D3C0FoyIxAH354fkBPDOO25SwjZ2OXO+kpiYyHPPPcfIkSPp2LEj77zzDvXr16d+/fp+h2aMSSOYUXwbgP3Z3bGqJgP3AF8Aq4BpqrpSRIaLSL6+jurQIfjpJ7jkEhu9l18kJyczevRoatWqxbBhw2jXrh2PPPKI32EZYzIRTAtqKPCDiCwADqUuVNUsC/uo6kzcLLyBy9L9VlDV1kHEEhYWLXJJqmVLvyMxwRo+fDgjRoygRYsWfPDBBzRv3tzvkIwxWQgmQY0GvgZ+wfVBRb1Jk6BwYbjwQr8jMZmZP38+RYoUoWHDhvTp04dGjRrRsWNHq5lnTD4RTIIqpKoDQh5JPpGQ4AZIdOkC5cr5HY1Jz9q1axk2bBjffvstHTt25KOPPuLkk0/m6quv9js0Y0w2BNMHNUtEeolIJREpl3oLeWRhaN8+qFPHTet+551+R2PS2rZtG3fffTf16tVj4cKFPPbYY7zzzjt+h2WMOUHBtKC6en+HBiwLZph5xJk2DbZsccnJujDCz4QJExg9ejR33nknF198Mddee63fIRljciCYYrFZ1t2LFitWQGwsvPqqjd4LB8nJyYwfP54KFSpw9dVXc88993DVVVdRu3bt6LnS3pgIZvNBBUkV5syBBg2gwImU2DW5RlX59NNPGTx4MKtWraJLly5cffXVFC1alNq1a/sdnjEml9h8UEFatw6WL4cbb8x6WxM6S5YsoXXr1lx11VUcOXKEDz74gEmTJvkdljEmBEI6H1QkmTfP/b30Un/jiHbr169n9erVvPrqq9xxxx0UKlTI75CMMSFyItOBBjUfVKT59VcoWhRq1fI7kuiyfft2RowYQeXKlRk0aBDXX3897du3p0SJEn6HZowJsWD6oD7BjdoDd0qwLjAtlEGFo1WrXHKy/qe8sX//fp5//nmefvpp9u/fT//+/QEQEUtOxkSJUM4HFTGWL4fZs6FvlsWdTG6YOXMmPXv2ZMuWLVx99dU8+eST1KlTx++wjDF5LMMEJSI1gYqq+k2a5ReISBFV3RDy6MLExImu5fTgg35HErlUlUOHDhEbG0uFChWoWrUqU6dOpUWLFn6HZozxSWYnrF4A9qazfK+3LmrMmwfNmkH58n5HEpkWLlzIRRddxL33uvE4TZs25fvvv7fkZEyUyyxBVVTVX9Iu9JZVC1lEYSYxERYvtsrlobBx40a6du1K06ZNWblyJY0aNTq6zgq6GmMy64Mqk8m6orkcR9j68ENITobLL/c7ksgyefJkunfvTqFChXjooYcYOHAgpUqV8jssY0wYyawFtUhEeqZdKCJ3AItDF1J4+fZbV7Xczjbl3P79+/nzzz8BuOCCC7jttttYt24dI0aMsORkjDlOZi2ofsCHInIT/yakJkBh4JoQxxU25s+HJk2s9l5OHDlyhAkTJvDwww9z5pln8uWXX1KlShVef/11v0MzxoSxDFtQqrpNVc8HHgM2ebfHVLW5qm7Nm/D89fvvrkBsu3Z+R5I/qSozZ86kQYMG3HbbbZx66qk8/PDDfodljMkngil1NBeYmwexhJ0ZM9zfK6/0N478aty4cdxxxx3UrFmT6dOnc91119ngB2NM0E6k1FHUmDoV6te38kbZ8dtvv7F9+3aaNm1K586dOXz4MLfffjuFCxf2OzRjTD5jhXsysHkzfP893HCD35HkDzt37qR///7Url2b3r17o6qULFmS3r17W3IyxpwQS1AZmDzZ/bUElbkDBw7w1FNPUaNGDV566SVuueUWZsyYYafyjDE5ZgkqA2++6S7OrVnT70jC2/vvv8/QoUNp2bIly5cv58033+TUU0/1OyxjTASwPqh0LFvmJijs18/vSMKPqvLFF1+wa9cuunbtSteuXTn99NM5//zz/Q7NGBNhrAWVjjfegCJFoEsXvyMJL0uWLOGSSy7h8ssv54UXXkBViYmJseRkjAkJS1BpqML06XD11a6ChIHff/+dbt260bhxY5YuXcqLL77IvHnzrJ/JGBNSdoovjU2b4O+/oVUrvyMJHxs3bjza1zR48GBKly7td0jGmChgCSqNr792f5s18zcOPx08eJCXX36ZhIQEhg8fTps2bdi8eTMVKlTwOzRjTBQJ6Sk+EblMRNaIyHoRGZLO+gEi8quILBeRr0SkaijjyYoqPPEElC7tLtCNNqk188444wwGDRrEL7/8QkpKCoAlJ2NMngtZghKRGOAV4HKgLtBVROqm2exnoImqng28BzwTqniCsX49bNwIgwdDoUJ+RpL3Fi9eTOPGjenevTv/93//x9dff82HH35IgQLWTWmM8Ucov32aAutVdaOqHgamAB0DN1DVuaq633s4H6gcwniytGyZ+3vZZX5GkbeSk5MBKFWqFAcOHGDy5Mn89NNPtGnTxufIjDHRLpR9UKcCmwMexwOZ9ezcDswKYTxZWrPG/Y2G2nu///47Dz/8MAkJCXz44YfUqlWLVatWWYvJGBM2wmKQhIh0w801le7YORHpBfQCqFixInFxcTl6vcTExHT3MW1aA6pWLcSiRQtztP9wE3i8CQkJvPvuu3zwwQeICNdddx1fffUVMTEx/gaZyzL6jCOZHXN0iKpjVtWQ3IDmwBcBj4cCQ9PZri2wCvi/YPbbuHFjzam5c+cet+zQIdVChVQHDcrx7sNO6vHGxcVp2bJlVUS0R48e+scff/gbWAil9xlHOjvm6BCJxwws0nS+70N5PmchUEtEqotIYaALMCNwAxFpCIwGrlLVv0MYS5bWrIGkJGjQwM8ocl9KSgo7d+4E4Oyzz6Zt27YsXbqUt956i9NOO83n6IwxJmMhO8Wnqskicg/wBRADjFPVlSIyHJctZwAjgRLAdK8qwR+qelWoYsrM8uXu79ln+/HqofHll18yaNAgEhMTueaaayhbtizTpk3zOyxjjAlKSPugVHUmMDPNskcC7rcN5etnx7JlULgwnHGG35Hk3LJlyxg0aBCzZ8+mWrVq3HTTTX6HZIwx2WZDtjzz57vTe/n9+qcvv/yShg0bsnDhQp577jlWr15N27ZtbXSeMSbfsW8t4PBhWLgQWrTwO5ITs3v3bhYsWABA69atefzxx9mwYQP9+/enSJEiPkdnjDEnxhIUsGQJHDwIF1zgdyTZc+jQIZ5//nlq1KjBtddeS1JSEoUKFWLo0KGULVvW7/CMMSZHwuI6KL99/737m18SVEpKClOmTOHBBx9k06ZNXHrppTz99NMUyu/nJ43Jp5KSkoiPj+fgwYMhf63SpUuzatWqkL9OKMTGxlK5cuWgv6ssQeESVI0aULGi35EE55tvvuGmm26iQYMGzJ49m0suucTvkIyJavHx8ZQsWZJq1aqFfJ60hIQESpYsGdLXCAVVZefOncTHx1O9evWgnhP1p/hUXYIK99bTL7/8wsSJEwHXzzRz5kwWL15sycmYMHDw4EHKly9vk3hmQkQoX758tlqZUZ+gNm50ExSG66zlmzdv5tZbb+Wcc85hyJAhHDp0CBHh8ssvt5F5xoQRS05Zy+57FPXfcD/+6P42b+5vHGnt3r2boUOHcsYZZzBp0iTuu+8+fvnlFxuVZ4yJGpagfoSSJaFePb8jOdbmzZsZOXIk119/PWvXrmXkyJGUK1fO77CMMRFi/Pjx3HPPPQC8/vrrTJgwAYDVq1fToEEDGjZsyIYNG5g0aZJvMVqC+hGaNgW/i3mnpKQwdepUBg8eDED9+vX57bffmDBhAlWr+jrRsDEmwt11113ccsstAHz00Ud06tSJn3/+mc2bN/uaoKJ6FN/u3a7E0YMP+htHXFwcAwcOZNGiRTRo0IBHH32UokWLWjFXY/Khfv1g6dLc3WeDBvDCC5lvs2nTJq644gpWrFgBwKhRo45OzXHOOefwzTffkJyczLhx42jatOkxz3300UcpUaIEdevW5YUXXiAmJoavvvqKAwcOsGrVKho0aED37t3p379/7h5YFqI6QcXFQUoKtPWpIuDvv//O3XffzWeffcZpp53G22+/zU033RRxczMZY/y1f/9+li5dyrfffsttt912NIml1b59e+666y5KlCjB/fffT1xcHKNGjeLTTz/N44idqE5QX34JxYvDeefl7euqKiJCbGwsy5Yt4+mnn+bee++laNGieRuIMSbXZdXS8UPXrl0BuPDCC9m7dy+7d+/2N6AgRXWCmjMHWrVyVczzwp49e3j66af56aef+PLLL6lYsSIbN260ChDGmBwrWLAgKSkpRx8HXm+Udnh3fhkSH7WDJP74A9auzZvTe4cPH+all16iRo0aPPnkk1SsWJH9+/cDWHIyxuSKihUr8vfff7Nz504OHTp0zGm5qVOnAvDdd99RunRpSpcuHdQ+S5YsSUJCQkjiDUbUtqB++MH9bdMmtK+zatUqrrjiCjZu3MhFF13EM888Q+PGjUP7osaYqFOoUCEeeeQRmjZtyqmnnkqdOnWOrouNjaVhw4YkJSUxbty4oPd59tlnExMTwznnnEOPHj1skEReWb0aRCDgM8xV//zzD+XKlaNatWrUqVOHV155hXbt2uWbprUxJv/p27cvffv2PWZZ69at6datGy+k6Rzr0aMHPXr0ANwovlSB9wsVKsTXX38domizFrUJas0aqFoVYmNzd78rV65kyJAhrFixgtWrV1O0aFE+++yz3H0RY4yJAlGboNauhdq1c29/f/75J8OGDeOtt96iRIkSPPDAA7m3c2OMOUFxcXF+h3DCojJBpaS4BJVbFcxXrlzJueeeS3JyMn379uXBBx+kQoUKubNzY4yJUlE5im/jxuIkJkJOxiocPnyYRYsWAVC3bl3uu+8+Vq9ezfPPP2/JyRhjckFUJqiff3bToV90Ufafq6pMnz6devXqcdFFF/HPP/8gIowYMYLTTz89lyM1xpjoFZUJaunSMtSsCdktdTdv3jyaN29O586diY2NZerUqZQtWzY0QRpjTJSLugSVkgLLlpXJ9vVPa9eu5cILLyQ+Pp5x48axdOlSLr/8chs2boyJSNWqVWPHjh2+xhB1CerPP2HfvoJB9T9t2bLl6DTrZ5xxBu+//z5r167l1ltvtYKuxpiwparHlD3Kr6JuFN/Gje5vZt1FCQkJjBw5kmeffZaUlBQuu+wyTjrpJK699tq8CdIYk6+1bt36uGWdO3emT58+7N+/n/bt2x+3PvXC2R07dtCpU6dj1gUzVHzTpk20a9eOZs2asXjxYpo2bcovv/zCgQMH6NSpE4899hjgWkbdu3fnk08+ISkpienTp1OnTh127txJ165d+fPPP2nevDmqenTfzz333NEKFHfccQf9+vVj06ZNXHbZZZx33nn88MMPnHvuudx6660MGzaMv//+m3ffffe4aT2yK+paUJklqKSkJF555RVq1KjBiBEjuPLKK1mxYgUnnXRS3gZpjDEnYN26dfTp04eVK1fy7LPPsmjRIpYvX84333zD8uXLj25XoUIFlixZQu/evRk1ahQAjz32GC1atGDlypVcc801/PHHHwAsXryYt956iwULFjB//nzeeOMNfv75ZwDWr19/dATz6tWrmTRpEt999x2jRo3iiSeeyPHxRF0LasMGKFBAqVLl+L6jrVu3ct9993HeeecxcuRIzj33XB8iNMbkd5m1eIoVK5bp+goVKpzwxbVVq1blPG/+oGnTpjFmzBiSk5P566+/+PXXXzn77LMBjp4Naty4MR988AEA33777dH7HTp0ODoA7LvvvuOaa66hePHiR587b948rrrqKqpXr079+vUBqFevHhdffDEiQv369dm0adMJHUOgkLagROQyEVkjIutFZEg664uIyFRv/QIRqRbKeMC1oCpWPEhqEfHvvvuO++67D1XltNNOY9myZcydO9eSkzEm30lNIr/99hujRo3iq6++Yvny5XTo0OGY6TeKFCkCQExMDMnJySf8eqn7AShQoMDRxwUKFMjRfo/uM8d7yICIxACvAJcDdYGuIlI3zWa3A7tUtSbwPPB0qOJJddZZ0KLFDlavXs0111xDy5YtmTx5Mlu3bgWgdu3aNjLPGJOv7d27l+LFi1O6dGm2bdvGrFmzsnzOhRdeyKRJkwCYNWsWu3btAqBly5Z89NFH7N+/n3379vHhhx/SsmXLkMafKpQtqKbAelXdqKqHgSlAxzTbdATe9u6/B1wsIc4OvXvv4tChezjrrLP46quv+O9//8u6deuoVKlSKF/WGGPyzDnnnEPDhg2pU6cON954IxcEUddt2LBhfPvtt9SrV48PPviAKlWqANCoUSN69OhB06ZNadasGXfccQcNGzYM9SEAoe2DOhXYHPA4HmiW0Taqmiwie4DyQMgG3xcsWJAff/yRPn368PDDD9sACGNMRKhWrRorVqw4+nj8+PHpbhfYN9SkSZOj/V3ly5dn9uzZ6T5nwIABDBgwIOjXS7vuROWLQRIi0gvoBW7WyJxW533ttdcoX748K1euzIXowl9iYmK+rmh8IuyYo0O4HHPp0qXzbObZI0eO+DrLbU4dPHgw6M8slAnqTyCwmFBlb1l628SLSEGgNLAz7Y5UdQwwBqBJkyaa3jUG2REXF5fudQqRKtqOF+yYo0W4HPOqVasoWbJknrxWQkJCnr1WKKTO7huMUPZBLQRqiUh1ESkMdAFmpNlmBtDdu98J+FoDrw4zxph8wr66spbd9yhkCUpVk4F7gC+AVcA0VV0pIsNF5Cpvs7FAeRFZDwwAjhuKbowx4S42NpadO3daksqEqrJz505iszGNeUj7oFR1JjAzzbJHAu4fBK4PZQzGGBNqlStXJj4+nu3bt4f8tQ4ePJitL/lwEhsbS+XKlYPePl8MkjDGmHBWqFAhqlevnievFRcXl2fDvP0WdbX4jDHG5A+WoIwxxoQlS1DGGGPCkuS3UScish34PYe7qUAIq1WEoWg7XrBjjhZ2zJGhqqoeV9Yn3yWo3CAii1S1id9x5JVoO16wY44WdsyRzU7xGWOMCUuWoIwxxoSlaE1QY/wOII9F2/GCHXO0sGOOYFHZB2WMMSb8RWsLyhhjTJizBGWMMSYsRWyCEpHLRGSNiKwXkeOqpItIERGZ6q1fICLVfAgzVwVxzD1EZLuILPVud/gRZ24RkXEi8reIpDt1pzgvee/HchFplNcx5rYgjrm1iOwJ+IwfSW+7/EREThORuSLyq4isFJH/pLNNxHzWQR5vxH3O6VLViLsBMcAG4HSgMLAMqJtmmz7A6979LsBUv+POg2PuAfzP71hz8ZgvBBoBKzJY3x6YBQhwHrDA75jz4JhbA5/6HWcuH3MloJF3vySwNp1/2xHzWQd5vBH3Oad3i9QWVFNgvapuVNXDwBSgY5ptOgJve/ffAy4WEcnDGHNbMMccUVT1W+CfTDbpCExQZz5QRkQq5U10oRHEMUccVf1LVZd49xNw88udmmaziPmsgzzeqBCpCepUYHPA43iO/4CPbqNucsU9QPk8iS40gjlmgOu8UyDvichpeROab4J9TyJNcxFZJiKzRKSe38HkJu9UfENgQZpVEflZZ3K8EMGfc6pITVAmfZ8A1VT1bOBL/m1BmsixBFfX7BzgZeAjf8PJPSJSAngf6Keqe/2OJ9SyON6I/ZwDRWqC+hMIbB1U9palu42IFARKAzvzJLrQyPKYVXWnqh7yHr4JNM6j2PwSzL+DiKKqe1U10bs/EygkIhV8DivHRKQQ7sv6XVX9IJ1NIuqzzup4I/VzTitSE9RCoJaIVBeRwrhBEDPSbDMD6O7d7wR8rV7vYz6V5TGnOSd/Fe7cdiSbAdzijfA6D9ijqn/5HVQoicjJqX2pItIU9388P//wwjuescAqVX0ug80i5rMO5ngj8XNOT0RO+a6qySJyD/AFbnTbOFVdKSLDgUWqOgP3D2CiiKzHdTp38S/inAvymPuKyFVAMu6Ye/gWcC4Qkcm40UwVRCQeGAYUAlDV14GZuNFd64H9wK3+RJp7gjjmTkBvEUkGDgBd8vkPL4ALgJuBX0RkqbfsAaAKRORnHczxRuLnfBwrdWSMMSYsReopPmOMMfmcJShjjDFhyRKUMcaYsGQJyhhjTFiyBGWMMeaEZFW8OJ3tOwcUwZ2U1faWoEzUEREVkWcDHt8vIo/m0r7Hi0inXNrXAq9S9R9ybBX6armx/3Re74ETeE4PEflfKOIx+cJ44LJgNhSRWsBQ4AJVrQf0y+o5lqBMNDoEXBtuV957FU2OUtVmqtoAeARXbb+Bd9uUnf1kQ7YTlIlu6RUvFpEaIvK5iCwWkXkiUsdb1RN4RVV3ec/9O6v9W4Iy0SgZGAP0T7sibQtIRBK9v61F5BsR+VhENorIUyJyk4j8JCK/iEiNgN20FZFFIrJWRK7wnh8jIiNFZKFXrPfOgP3OE5EZwK9ZBS4iV3otq59FZI6IVPSWPyoiE0Xke9wF6CeJyJfeqZQ3ReT31IQsIt28uJeKyGgvtqeAot6ydzPazlt+q3dsP+EuKjUm0BjgXlVtDNwPvOotPwM4Q0S+F5H5IpJly8sSlIlWrwA3iUjpbDznHOAu4Ezclf5nqGpTXF3DewO2q4ab/qQD8LqIxAK348rvnAucC/QUkere9o2A/6jqGUHE8B1wnqo2xE2pMihgXV2grap2xVWY+No7lfIeXhUCETkTuAF3mqUBcAS4SVWHAAe8FtpNGW0nrlzWY7jE1MJ7TWOAowVuzweme1UwRuPmtwJXuagWrhJKV+ANESmT2f4istSRMVlR1b0iMgHoiysVE4yFqfXdRGQDMNtb/gvQJmC7aaqaAqwTkY1AHeBS4OyA1llp3H/Ww8BPqvpbkDFUBqZ6iaIwEPi8GaqaeiwtgGu8Y/1cRHZ5yy/GFQle6JVyKwqkd6olo+2aAXGqut17H6bifhkbA67Rs9v7UZNWPG4iySTgNxFZi/s/sDCznRkTrV7AtWyKByxLxvt/ISIFcEkg1aGA+ykBj1M49sde2vphipvp9d6AfqTqqpqa4PZlI+aXcbMi1wfuBGID1gWzHwHeDoijtqo+moPtjDnKmxbkNxG5HlzhWxE5x1v9Ea71hHe6+QxgY2b7swRlopaq/gNMwyWpVJv4dxqSq/AKsWbT9SJSwOuXOh1Ygyvi21vcNAqIyBkiUjyznWSgNP9OI9E9k+2+Bzp7r3UpUNZb/hXQSUT+z1tXTkSqeuuSUuPLZLsFQCsRKe9te/0JHIOJEOKKF/8I1BaReBG5HbgJuF1ElgEr+Xdm7y+AnSLyKzAXGKiqmVZgt1N8Jto9C9wT8PgN4GPvP9fnZK91k+oP4CegFHCXqh4UkTdxfVNLxJ0z2w5cfQL7fhR3fn8X8DVQPYPtHgMmi8jNuC+QrUCCqu4QkYeA2V4LMQm4G/gd17m9XESWeP1Qx22nqvPFDcn/EdgNLD2BYzARwuvvTM9xAyC8ausDvFtQrJq5MRFIRIoAR7xpWJoDr2XQL2BM2LIWlDGRqQowzWv9HMZdg2JMvmItKGOMMWHJBkkYY4wJS5agjDHGhCVLUMYYY8KSJShjjDFhyRKUMcaYsPT/+sSJcc7DJ4QAAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"perf_h2o.plot_uplift(metric=\"gain\", normalize=True)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8782349075050278"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"perf_h2o.auuc_normalized()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Scoring histrory and importance of number of trees\n",
"\n",
"To speed up the calculation of AUUC, the predictions are binned into quantile histograms. To calculate precision AUUC the more bins the better. The more trees usually produce more various predictions and then the algorithm creates histograms with more bins. So the algorithm needs more iterations to get meaningful AUUC results. \n",
"You can see in the scoring history table the number of bins as well as the result AUUC. There is also Qini value parameter, which reflects the number of bins and then is a better pointer of the model improvement. In the scoring history table below you can see the algorithm stabilized after building 6 trees. But it depends on data and model settings on how many trees are necessary."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"[tips]\n",
"Use `model.show()` for more details.\n",
"Use `model.explain()` to inspect the model.\n",
"--\n",
"Use `h2o.display.toggle_user_tips()` to switch on/off this section.