{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "dW9C26GZCIJc" }, "source": [ "# The overview of the basic approaches to solving the Uplift Modeling problem\n", "\n", "
\n", "
\n", " \n", " \n", " \n", "
\n", " SCIKIT-UPLIFT REPO | \n", " SCIKIT-UPLIFT DOCS | \n", " USER GUIDE\n", "
\n", " RUSSIAN VERSION\n", "
" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "9Mz7V_YaCIKC" }, "source": [ "## Content\n", "\n", "* [Introduction](#Introduction)\n", "* [1. Single model approaches](#1.-Single-model-approaches)\n", " * [1.1 Single model](#1.1-Single-model-with-treatment-as-feature)\n", " * [1.2 Class Transformation](#1.2-Class-Transformation)\n", "* [2. Approaches with two models](#2.-Approaches-with-two-models)\n", " * [2.1 Two independent models](#2.1-Two-independent-models)\n", " * [2.2 Two dependent models](#2.2-Two-dependent-models)\n", "* [Conclusion](#Conclusion)\n", "\n", "## Introduction\n", "\n", "Before proceeding to the discussion of uplift modeling, let's imagine some situation:\n", "\n", "A customer comes to you with a certain problem: it is necessary to advertise a popular product using the sms.\n", "You know that the product is quite popular, and it is often installed by the customers without communication, that the usual binary classification will find the same customers, and the cost of communication is critical for us...\n", "\n", "And then you begin to understand that the product is already popular, that the product is often installed by customers without communication, that the usual binary classification will find many such customers, and the cost of communication is critical for us...\n", "\n", "Historically, according to the impact of communication, marketers divide all customers into 4 categories:\n", "\n", "

\n", " \"Customer\n", "

\n", "\n", "- **`Do-Not-Disturbs`** *(a.k.a. Sleeping-dogs)* have a strong negative response to a marketing communication. They are going to purchase if *NOT* treated and will *NOT* purchase *IF* treated. It is not only a wasted marketing budget but also a negative impact. For instance, customers targeted could result in rejecting current products or services. In terms of math: $W_i = 1, Y_i = 0$ or $W_i = 0, Y_i = 1$.\n", "- **`Lost Causes`** will *NOT* purchase the product *NO MATTER* they are contacted or not. The marketing budget in this case is also wasted because it has no effect. In terms of math: $W_i = 1, Y_i = 0$ or $W_i = 0, Y_i = 0$.\n", "- **`Sure Things`** will purchase *ANYWAY* no matter they are contacted or not. There is no motivation to spend the budget because it also has no effect. In terms of math: $W_i = 1, Y_i = 1$ or $W_i = 0, Y_i = 1$.\n", "- **`Persuadables`** will always respond *POSITIVE* to the marketing communication. They is going to purchase *ONLY* if contacted (or sometimes they purchase *MORE* or *EARLIER* only if contacted). This customer's type should be the only target for the marketing campaign. In terms of math: $W_i = 0, Y_i = 0$ or $W_i = 1, Y_i = 1$.\n", "\n", "\n", "Because we can't communicate and not communicate with the customer at the same time, we will never be able to observe exactly which type a particular customer belongs to.\n", "\n", "Depends on the product characteristics and the customer base structure some types may be absent. In addition, a customer response depends heavily on various characteristics of the campaign, such as a communication channel or a type and a size of the marketing offer. To maximize profit, these parameters should be selected.\n", "\n", "Thus, when predicting uplift score and selecting a segment by the highest score, we are trying to find the only one type: **persuadables**.\n", "\n", "Thus, in this task, we don’t want to predict the probability of performing a target action, but to focus the advertising budget on the customers who will perform the target action only when we interact. In other words, we want to evaluate two conditional probabilities separately for each client:\n", "\n", "\n", "* Performing a targeted action when we influence the client. \n", " We will refer such clients to the **test group (aka treatment)**: $P^T = P(Y=1 | W = 1)$,\n", "* Performing a targeted action without affecting the client. \n", " We will refer such clients to the **control group (aka control)**: $P^C = P(Y=1 | W = 0)$,\n", "\n", "where $Y$ is the binary flag for executing the target action, and $W$ is the binary flag for communication (in English literature, _treatment_)\n", "\n", "The very same cause-and-effect effect is called **uplift** and is estimated as the difference between these two probabilities:\n", "\n", "$$ uplift = P^T - P^C = P(Y = 1 | W = 1) - P(Y = 1 | W = 0) $$\n", "\n", "Predicting uplift is a cause-and-effect inference task. The point is that you need to evaluate the difference between two events that are mutually exclusive for a particular client (either we interact with a person, or not; you can't perform two of these actions at the same time). This is why additional requirements for source data are required for building uplift models.\n", "\n", "To get a training sample for the uplift simulation, you need to conduct an experiment: \n", "1. Randomly split a representative part of the client base into a test and control group\n", "2. Communicate with the test group\n", "\n", "The data obtained as part of the design of such a pilot will allow us to build an uplift forecasting model in the future. It is also worth noting that the experiment should be as similar as possible to the campaign, which will be launched later on a larger scale. The only difference between the experiment and the campaign should be the fact that during the pilot, we choose random clients for interaction, and during the campaign - based on the predicted value of the Uplift. If the campaign that is eventually launched differs significantly from the experiment that is used to collect data about the performance of targeted actions by clients, then the model that is built may be less reliable and accurate.\n", "\n", "So, the approaches to predicting uplift are aimed at assessing the net effect of marketing campaigns on customers.\n", "\n", "All classical approaches to uplift modeling can be divided into two classes:\n", "1. Approaches with the same model\n", "2. Approaches using two models\n", "\n", "Let's download [RetailHero.ai contest data](https://retailhero.ai/c/uplift_modeling/overview):" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:36.009546Z", "start_time": "2020-05-30T22:29:36.005593Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 700 }, "colab_type": "code", "id": "DKvg0hDcCfvi", "outputId": "c1864df4-2d46-4817-f865-2f3d46eb0c21" }, "outputs": [], "source": [ "import urllib.request\n", "\n", "url = 'https://drive.google.com/u/0/uc?id=1fkxNmihuS15kk0PP0QcphL_Z3_z8LLeb&export=download'\n", "urllib.request.urlretrieve(url, '/content/retail_hero.zip')\n", "\n", "!unzip /content/retail_hero.zip\n", "!pip install scikit-uplift==0.1.2 catboost=0.22" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "76e99aut_1nH" }, "source": [ "Now let's preprocess it a bit:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:39.095040Z", "start_time": "2020-05-30T22:29:36.013799Z" }, "colab": {}, "colab_type": "code", "id": "wojgYP76CIKH" }, "outputs": [], "source": [ "%matplotlib inline\n", "\n", "import pandas as pd; pd.set_option('display.max_columns', None)\n", "from sklearn.model_selection import train_test_split\n", "\n", "\n", "# reading data\n", "df_clients = pd.read_csv('/content/uplift_data/clients.csv', index_col='client_id')\n", "df_train = pd.read_csv('/content/uplift_data/uplift_train.csv', index_col='client_id')\n", "df_test = pd.read_csv('/content/uplift_data/uplift_test.csv', index_col='client_id')\n", "\n", "# extracting features\n", "df_features = df_clients.copy()\n", "df_features['first_issue_time'] = \\\n", " (pd.to_datetime(df_features['first_issue_date'])\n", " - pd.Timestamp('1970-01-01')) // pd.Timedelta('1s')\n", "df_features['first_redeem_time'] = \\\n", " (pd.to_datetime(df_features['first_redeem_date'])\n", " - pd.Timestamp('1970-01-01')) // pd.Timedelta('1s')\n", "df_features['issue_redeem_delay'] = df_features['first_redeem_time'] \\\n", " - df_features['first_issue_time']\n", "df_features = df_features.drop(['first_issue_date', 'first_redeem_date'], axis=1)\n", "\n", "indices_train = df_train.index\n", "indices_test = df_test.index\n", "indices_learn, indices_valid = train_test_split(df_train.index, test_size=0.3, random_state=123)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "LB42pR6iCIK9" }, "source": [ "For convenience, we will declare some variables:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:40.203839Z", "start_time": "2020-05-30T22:29:39.097441Z" }, "colab": {}, "colab_type": "code", "id": "_HyKmGQ_CILw" }, "outputs": [], "source": [ "X_train = df_features.loc[indices_learn, :]\n", "y_train = df_train.loc[indices_learn, 'target']\n", "treat_train = df_train.loc[indices_learn, 'treatment_flg']\n", "\n", "X_val = df_features.loc[indices_valid, :]\n", "y_val = df_train.loc[indices_valid, 'target']\n", "treat_val = df_train.loc[indices_valid, 'treatment_flg']\n", "\n", "X_train_full = df_features.loc[indices_train, :]\n", "y_train_full = df_train.loc[:, 'target']\n", "treat_train_full = df_train.loc[:, 'treatment_flg']\n", "\n", "X_test = df_features.loc[indices_test, :]\n", "\n", "cat_features = ['gender']\n", "\n", "models_results = {\n", " 'approach': [],\n", " 'uplift@30%': []\n", "}" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "olRc5k9iCIMP" }, "source": [ "## 1. Single model approaches\n", "\n", "### 1.1 Single model with treatment as feature\n", "\n", "The most intuitive and simple uplift modeling technique. A training set consists of two groups: treatment samples and control samples. There is also a binary treatment flag added as a feature to the training set. After the model is trained, at the scoring time it is going to be applied twice:\n", "with the treatment flag equals `1` and with the treatment flag equals `0`. Subtracting these model's outcomes for each test sample, we will get an estimate of the uplift.\n", "\n", "

\n", " \"Solo\n", "

" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:42.146943Z", "start_time": "2020-05-30T22:29:40.207117Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 444 }, "colab_type": "code", "id": "Aq5-jjbeCIMa", "outputId": "83b9f995-a4ec-4c7b-933c-e60521956ab6" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABJgAAAGrCAYAAACMgi0UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdfbxVZZnw8d8Voli+ITIOgg6MYY02ioUvmRVpipmolRbak2gvjA2VpmVWznhKnalGszTTx5LUxlJTUzB7oZQx84XA0MSXRxRKCJUEX8hRQa/nj3Uf2BzPORzYZ599Dvy+n8/+7LXvda+1rrXRdZ99rfu+V2QmkiRJkiRJ0rp6TbMDkCRJkiRJUt9mgkmSJEmSJEl1McEkSZIkSZKkuphgkiRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUFxNM0nogIloi4r/L8g4RsSwi+q3Dfr4UEd/v/gglaf0WEfMj4t1NOvalEXFmWX57RDy0jvu5KCL+rXujk6S+q7Pra0S8ISJmR8RzEfGZ5kW5Mp4xEbGg5vOciBizDvtZ53ZEMsGkhiqJjtbXKxHxvzWfP9yNx1l58e9pETE8IjIiNmrG8dvKzD9n5maZ+XJn9do2QmXb/8jMjzc2QklaNxFxdETMLG3Iooj4eUTs2w37bVob0t0y87eZ+YY11YuIYyPitjbbHp+ZZzQuOknqWeVv9Ne3KVt5Y3ZttHN9PQW4JTM3z8zzmnmjoT2ZuUtmTl9TvbbfUVfbEak9JpjUUCXRsVlmbgb8GRhXU3ZFa73ekpzpDfwuJOnVIuIk4FvAfwDbAjsA3wUO64Fj99h12TZAkvqMfwDmNGrn6zIaQWo2E0xqitbeMxHxhYh4HPhBRLwmIk6NiEci4qmIuDoitq7Z5icR8XhEPBMRt0bELqV8IvBh4JRyV3tqKZ8fEZ+PiHsj4m8RcUlEbFvueD8XEb+OiIE1+987Im6PiKcj4p7aLqURMT0izoiI35VtfxUR25TVt5b3p8vx39rO+bZExDURcVXZ/u6I2K1m/fzyXdwL/C0iNlpDPCMi4n/KvqYB29SsW61HVURsHRE/iIi/RMTSiLg+Il4H/BzYrqZH2XZt7+hExKGle+3T5Tv4pzYxf658v8+UcxtQ1m0TETeW7ZZExG8jwuuNpHUSEVsCXwUmZeZ1mfm3zFyemVMz8/OlziYR8a1yrftLWd6krGttc06OiCdL76fjyrrO2pC21+UOr4lriP/SqIafTSvX7f+JiH+oWZ8RMSkiHgYeLmWHRDX04unSFuxaU3/30o48FxFXAQNq1rUdIrF9RFwXEYujalu/U+K+CHhrOeena+I8s2bbT0TE3HIdnxIR27WJ+fiIeLjEeEFERFn3+nKOz0TEX0uMktTr1LQPXyrXq/nRwSiL2utrRNwMvAv4TrmO/pjqxsfU8vmUtT1WuQZfGBE3RcTfgHeVv8+vLdfweVEzFC8iNi3bLI2I+4E92hxvZY+qiOhXjvtIaTtmlfah9XfMPSXuD7XTjvxTafOeLm3goW1iviAiflb2e1dE7FjWRUScW9rdZyPijxHxprX8J1If4w8+NdPfA1tTZf8nAp8GDgfeCWwHLAUuqKn/c2Ak8HfA3cAVAJl5cVn+RukZNa5mmw8ABwA7AePKPr4EDKb67/8zABExFPgZcGaJ6XPAtRExuGZfRwPHleNvXOoAvKO8b1WOf0cH53sY8JOy/x8B10dE/5r1RwHvBbaiujvfWTw/AmZRJZbOACZ0cEyAHwKvBXYpsZ+bmX8D3gP8paZH2V9qN4qInYAfAyeW7+smqkZz45pqHwQOAkYAuwLHlvKTgQVlu22pvvPsJEZJ6sxbqZIoP+2kzpeBvYFRwG7AnsBpNev/HtgSGAp8DLggIgauoQ2pvS7/I2u+Jnbmw1TX622A2eWYtQ4H9gJ2jojdgcnAvwCDgP8LTIkqibYxcD3VtX1rqnblA+0dMKq73zcCfwKGl3O/MjMfAI4H7ijnvFU72+4H/CfVdX5I2ceVbaodQvWDZtdSb2wpPwP4FTAQGAac3/lXI0lN9fdU1+ahVH9TXxwRnQ4Ry8z9gN8CnyrX0aNYfbTGN9bxWEcDZwGbA7cDU4F7Sv39gRMjovVaezqwY3mNpfPfAydRtWkHA1sAHwWez8zW3zG7lbhXuyFQfqtMpbqm/x3V77Ur2sQ8HvgK1TV/bokf4ECq30k7UbW/HwSe6iRGrQdMMKmZXgFOz8wXM/N/qf7Y/XJmLsjMF4EW4IgoPXEyc3JmPlezbreo7mp35vzMfCIzF1I1Andl5h8y8wWqHyq7l3r/B7gpM2/KzFcycxowk+oi3OoHmfn/SqxXU/2IWRuzMvOazFwOfJPqx9LeNevPy8zHyv47jCcidqD6g/7fynd3K9WF/1UiYghVIun4zFxa7vj/Txfj/RDws8ycVmI+G9gU2KdNzH/JzCUlhtbvZDnVD5J/KMf8bWaaYJK0rgYBf83MFZ3U+TDw1cx8MjMXU/2x+5Ga9cvL+uWZeROwDFjTHBO11+WuXBM787PMvLW0YV+m6j20fc36/8zMJeVYE4H/m5l3ZebLmXkZ8CJVm7E30B/4VjmXa4Dfd3DMPalu2Hy+9Pp6ITNv66BuWx8GJmfm3SXmL5aYh9fU+VpmPp2ZfwZuYfU24B+A7dbymJLULK1/V/8P1U3eDzbpWDdk5u8y8xXgn4HBmfnVzHwpMx8FvkeV0KFsd1ZpOx4DzuvkmB8HTsvMh7JyT2Z2JdmzN7AZ1fX+pcy8merGxVE1dX6amTNKG30Fq7cFmwNvBCIzH8jMRV04pvowE0xqpsUl0dPqH4Cflu6XTwMPAC8D25ZunV8r3TqfBeaXbbahc0/ULP9vO583qzn2ka3HLsfflypJ0urxmuXna7btqsdaF0qjsYDqD/9XrV9DPNsBS0svpFZ/6uCY2wNLMnPpWsZKOc7K/ZaYH6O6g9Kqo+/kv6juYPwqIh6NiFPX4fiS1OopYJvofH6i1a5ZZbn2GvtUmwRVV67jtdflrlwTu7SvzFwGLKHzNuDkNm3A9qX+dsDCNkn7ztqAP60hMdeRtue7jOrfoSttwClAADPKcIqPrsPxJak7vEyVlK/Vnyr50aq9v6u3ozHWdKy2bcF2bdqCL1GNDqBsV1u/o7YAqvbgkXWIdzvgsdLm1R5njW1BSUZ9h2pEypMRcXFEbLEOMagPMcGkZmrbo+Ux4D2ZuVXNa0DpfXQ01RCzd1N1sRxetokO9rW2HgN+2ObYr8vMr63DeXRk5Z3qqOYjGgbUDkur3U9n8SwCBkY1j1KrHTo45mPA1hHxquEPXYj7L1QNW2vMUc5h4Rq2o/Q0Ozkz/xE4FDgpIvZf03aS1IE7qHrwHN5JndWuWVTXxb90ULetjq6HteXrfE0satuAzaiGt3XWBpzVpg14bWb+mKoNGFqO36qzNmCHDhJza9sGvI6qJ1lX2oDHM/MTmbkd1TC/70abpzhJUg/5M6t+N7QawerJmPb+ru5q+1GrK78J1nSstm3BvDZtweaZ2TrCYhE1bQsdtwWt+9qxC/G19Rdg+1h9LtUd6GLbl5nnZeZbgJ2phsp9fh1iUB9igkm9yUXAWVEmPo2IwRHR+nSgzal+XDxFNZ/Qf7TZ9gmq+THW1X8D4yJibOktNaBMcDesC9suphrut6bjvyUi3l/+0D+R6nzuXNt4MvNPVMPlvhIRG0f1iO5x7e2kdEP9OdUf9wMjon9EtI61fgIY1Mkww6uB90bE/mX89ckl5tvXcJ6tk9O+vvwAeobq7tEra9hMktqVmc8A/041b9LhEfHacj17T0S0znPxY+C00nZsU+p39THUXWlD1vmaWBwcEfuWOZTOAO4sQxra8z3g+IjYq0yS+rqIeG9EbE6VbFsBfKZ8B++nGgrXnhlUP0C+VvYxICLeVnPOwzqZQ+rHwHERMSqqydL/g2qY+fw1nWhEHFnTfi6l+sFkGyCpGa6iahuGRfVAoXdT/d18TZt6rX9Xv51qfrmfrMOxuvp7pKvHmgE8F9UDJzYtvwneFBGtk3lfDXyx/I0/jGp+pI58HzgjIkaWdmXXiBjUhbjvouqVdEppc8ZQfX9t5+R7lYjYo7Rj/YG/AS9gW7DeM8Gk3uTbwBSqYVXPUSVf9irrLqe607AQuJ9XJ2YuoZoY9emIuH5tD1z+yD+MqtvpYqos/+fpwv8jmfk81WR2vyvH37uDqjdQzeGxlGpekPeXeTzWJZ6jqb6bJVQT/F3eSYgfoeoG/CDwJFVyi8x8kOoHxKMl7tW6AmfmQ1RzQZ0P/JWqMRmXmS91cqxWI4FfU81xcgfw3cy8pQvbSVK7MvMcqklKT2PVdfFTVBNeQ/VQhJnAvcAfqR4Gcear99SuNbYhdV4ToXo4w+lU1+23lH21KzNnAp+gGlqwlGrI8bFl3UvA+8vnJVTtynUd7OflEufrqe7iLyj1AW6merz24xHx13a2/TXwb8C1VEmqHVk178ea7AHcFRHLqNr1E8rcIZLU075KdSPgNqrr6TeAD2fmfTV1Hi/r/kI1h9Dx5e/ktfWfVMmspyPicx3U6fKxyjX8EKo5jeZRtT3fpxrNAdVcg38q635F9fCHjnyTKiH1K+BZqnZv07KuBbisxL3a3FOlzRlHNafrX4HvAsd08fvZguqGydIS51NU02hoPRbpvLtSw0VEC/D6zOzwB4Ukaf0UEZcCCzLztDXVlST1nNIj578zsyujFvrMsaRmsQeTJEmSJEmS6mKCSZIkSZIkSXVxiJwkSZIkSZLqYg8mSZIkSZIk1WWjZgfQCNtss00OHz682WFIUq80a9asv2bm4GbH0Uy2E5LUMdsJ2wlJ6khnbcR6mWAaPnw4M2fObHYYktQrRcSfmh1Ds9lOSFLHbCdsJySpI521EQ6RkyRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUl/VyDiZJG4bly5ezYMECXnjhhWaH0isNGDCAYcOG0b9//2aHIkld4nW9Z/WFdiIi5gPPAS8DKzJzdERsDVwFDAfmAx/MzKUREcC3gYOB54FjM/Pusp8JwGllt2dm5mU9eR6S+rYNsX1alzbCBJOkPmvBggVsvvnmDB8+nOpvSrXKTJ566ikWLFjAiBEjmh2OJHWJ1/We08faiXdl5l9rPp8K/CYzvxYRp5bPXwDeA4wsr72AC4G9SkLqdGA0kMCsiJiSmUt78iQk9V0bWvu0rm2EQ+Qk9VkvvPACgwYN2iAu8msrIhg0aNAGdZdFUt/ndb3n9PF24jCgtQfSZcDhNeWXZ+VOYKuIGAKMBaZl5pKSVJoGHNTTQUvquza09mld2wgTTJL6tA3lIr8u/G4k9UVeu3pOH/muE/hVRMyKiImlbNvMXFSWHwe2LctDgcdqtl1QyjoqX01ETIyImRExc/Hixd15DpLWA33kmtlt1uV8HSInSZIkqbfaNzMXRsTfAdMi4sHalZmZEZHdcaDMvBi4GGD06NHdsk9J2pCYYJK03pg6tXv3N25c5+ufeuop9t9/fwAef/xx+vXrx+DBgwGYMWMGG2+88Vofc/r06Wy88cbss88+a7Xd8OHDmTlzJttss81aH1OSeqt581q6dX8jRqx5fxHBSSedxDnnnAPA2WefzbJly2hp6Xjb66+/np122omdd955rdati3Xd32abbcayZcu6JYaelJkLy/uTEfFTYE/giYgYkpmLyhC4J0v1hcD2NZsPK2ULgTFtyqc3OHRJ67GW6S3du78xa97f/PnzOeSQQ7jvvvtWbdfSwmabbcbnPve5dre59NJLmTlzJt/5zne46KKLeO1rX8sxxxzDgw8+yPjx44kIrrnmGu666y6OPvrous/DIXKStI4GDRrE7NmzmT17Nscffzyf/exnV37eeOONWbFixVrvc/r06dx+++0NiFaS1BWbbLIJ1113HX/961/XXLm4/vrruf/++9d63bq0E53tb30TEa+LiM1bl4EDgfuAKcCEUm0CcENZngIcE5W9gWfKULpfAgdGxMCIGFj288sePBVJarrjjz+eY445BqjakiOOOII//OEPPPbYY/zoRz/qlmOYYJKkbnTsscdy/PHHs9dee3HKKafwyCOPcNBBB/GWt7yFt7/97Tz4YNWzf+rUqey1117svvvuvPvd7+aJJ55g/vz5XHTRRZx77rmMGjWK3/72tyxevJgPfOAD7LHHHuyxxx787ne/A6reUwceeCC77LILH//4x8m0J78kdYeNNtqIiRMncu65575q3fz589lvv/3Ydddd2X///fnzn//M7bffzpQpU/j85z/PqFGjeOSRR1bWb2/dmDFjOPHEExk9ejTf/va3mTVrFu985zt5y1vewtixY1m0qJpa6Hvf+x577LEHu+22Gx/4wAd4/vnn291fR+3MvHnzeOtb38o///M/c9ppp/XMl9f9tgVui4h7gBnAzzLzF8DXgAMi4mHg3eUzwE3Ao8Bc4HvAvwJk5hLgDOD35fXVUiZJ64UxY8ZwwgknMGrUKN70pjcxY8aMV9VpaWnh7LPP5qabbuJb3/oWF154Ie9617s49dRT+e1vf8uoUaPabfvWhkPkJKmbLViwgNtvv51+/fqx//77c9FFFzFy5Ejuuusu/vVf/5Wbb76ZfffdlzvvvJOI4Pvf/z7f+MY3OOecczj++ONX6+Z69NFH89nPfpZ9992XP//5z4wdO5YHHniAr3zlK+y77778+7//Oz/72c+45JJLmnzWkrT+mDRpErvuuiunnHLKauWf/vSnmTBhAhMmTGDy5Ml85jOf4frrr+fQQw/lkEMO4Ygjjlit/j777NPuupdeeomZM2eyfPly3vnOd3LDDTcwePBgrrrqKr785S8zefJk3v/+9/OJT3wCgNNOO41LLrmET3/606/aX0ftzAknnMAnP/lJjjnmGC644IIGf2ONkZmPAru1U/4UsH875QlM6mBfk4HJ3R2jJPUWzz//PLNnz+bWW2/lox/96GpD6WodfPDBq/3mmD59OmeffTY33nhj3TGYYJKkbnbkkUfSr18/li1bxu23386RRx65ct2LL74IVEmoD33oQyxatIiXXnqJESNGtLuvX//616sNhXj22WdZtmwZt956K9dddx0A733vexk4cGADz0iSNixbbLEFxxxzDOeddx6bbrrpyvI77rhj5bX3Ix/5yKsSUF31oQ99CICHHnqI++67jwMOOACAl19+mSFDhgBw3333cdppp/H000+zbNkyxo4d+6r9dNbO/O53v+Paa69dGesXvvCFdYpVktQ7dPRUt9byo446CoB3vOMdPPvsszz99NM9FlsrE0yS1M1e97rXAfDKK6+w1VZbMXv27FfV+fSnP81JJ53EoYceyvTp0zucPPaVV17hzjvvZMCAAY0MWZLUxoknnsib3/xmjjvuuG7fd2s7kZnssssu3HHHHa+qc+yxx3L99dez2267cemllzJ9+vRX1emsnYEN75HakrQ+GzRoEEuXLl2tbMmSJStvVLe95jejDXAOJklqkC222IIRI0bwk5/8BKh+SNxzzz0APPPMMwwdOhSAyy67bOU2m2++Oc8999zKzwceeCDnn3/+ys+tPyLe8Y53rJyM7+c///mrGhtJUn223nprPvjBD642BHmfffbhyiuvBOCKK67g7W9/O/Dqa3etzta94Q1vYPHixSsTTMuXL2fOnDkAPPfccwwZMoTly5dzxRVXtLu/ztqZt73tbavFKknq2zbbbDOGDBnCzTffDFTJpV/84hfsu+++AFx11VUA3HbbbWy55ZZsueWWXdpvZ+3U2rIHk6T1xrhxzY7g1a644go++clPcuaZZ7J8+XLGjx/PbrvtRktLC0ceeSQDBw5kv/32Y968eQCMGzeOI444ghtuuIHzzz+f8847b+VcICtWrOAd73gHF110EaeffjpHHXUUu+yyC/vssw877LBDk89UkrrfiBEtTT3+ySefzHe+852Vn88//3yOO+44/uu//ovBgwfzgx/8AIDx48fziU98gvPOO49rrrmGHXfcceU2bdfV2njjjbnmmmv4zGc+wzPPPMOKFSs48cQT2WWXXTjjjDPYa6+9GDx4MHvttdfKP/7b7q+jdubb3/42Rx99NF//+tc57LDDeuDbkqQNR8uYlqYc9/LLL2fSpEmcdNJJAJx++ukr25wBAwaw++67s3z5ciZP7vqUc7vuuiv9+vVjt91249hjj+Wzn/3sOscX6+OTh0aPHp0zZ85sdhjSOps6tXrvjQmT3uSBBx7gn/7pn5odRq/W3ncUEbMyc3STQuoVbCfUF8yb17JyudmJlp7idb3n2U60z3ZCPaVlesuq5SYlLbRmvb19GjNmDGeffTajR3fvpXtt2wiHyEmSJEmSJKkuDpGTJEmSJEnqo9p7EEQz2INJUp+2Pg7z7S5+N5L6Iq9dPcfvWpK6bkO7Zq7L+dqDSVKfNWDAAJ566ikGDRrko5jbyEyeeuopBgwY0OxQJK3BhjjXUke8rvcc2wlJ6roNrX1a1zbCBJOkPmvYsGEsWLCAxYsXNzuUXmnAgAEMGzas2WFIUpd5Xe9ZthOS1DUbYvu0Lm2ECSZJfVb//v0ZMWJEs8OQJHUTr+uSpN7I9qlrnINJkiRJkiRJdTHBJEmSJEmSpLqYYJIkSZIkSVJdTDBJkiRJkiSpLiaYJEmSJEmSVBcTTJIkSZIkSaqLCSZJkiRJkiTVxQSTJEmSJEmS6rJRswOQtPamTl21PG5c8+KQJEmSJAnswSRJkiRJkqQ62YNJ6iNqey1JkiRJktSb2INJktQUETEgImZExD0RMScivlLKL42IeRExu7xGlfKIiPMiYm5E3BsRb67Z14SIeLi8JjTrnCRJkqQNlT2YJEnN8iKwX2Yui4j+wG0R8fOy7vOZeU2b+u8BRpbXXsCFwF4RsTVwOjAaSGBWREzJzKU9chaSJEmS7MEkSWqOrCwrH/uXV3ayyWHA5WW7O4GtImIIMBaYlplLSlJpGnBQI2OXJEmStDoTTJKkpomIfhExG3iSKkl0V1l1VhkGd25EbFLKhgKP1Wy+oJR1VN72WBMjYmZEzFy8eHG3n4skSZK0ITPBJElqmsx8OTNHAcOAPSPiTcAXgTcCewBbA1/opmNdnJmjM3P04MGDu2OXkiRJkgoTTJKkpsvMp4FbgIMyc1EZBvci8ANgz1JtIbB9zWbDSllH5ZIkSZJ6iAkmSVJTRMTgiNiqLG8KHAA8WOZVIiICOBy4r2wyBTimPE1ub+CZzFwE/BI4MCIGRsRA4MBSJkmSJKmH+BQ5SVKzDAEui4h+VDc8rs7MGyPi5ogYDAQwGzi+1L8JOBiYCzwPHAeQmUsi4gzg96XeVzNzSQ+ehyRJkrTBM8EkSWqKzLwX2L2d8v06qJ/ApA7WTQYmd2uAkiRJkrrMIXKSJEmSJEmqiwkmSZIkSZIk1cUEkyRJkiRJkupigkmSJEmSJEl1McEkSZIkSZKkuphgkiRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUFxNMkiRJkiRJqosJJkmSJEmSJNXFBJMkSZIkSZLq0rAEU0RsHxG3RMT9ETEnIk4o5S0RsTAiZpfXwTXbfDEi5kbEQxExtqb8oFI2NyJObVTMkiRJkiRJWnsbNXDfK4CTM/PuiNgcmBUR08q6czPz7NrKEbEzMB7YBdgO+HVE7FRWXwAcACwAfh8RUzLz/gbGLkmSJEmSpC5qWIIpMxcBi8rycxHxADC0k00OA67MzBeBeRExF9izrJubmY8CRMSVpa4JJkmSJEmSpF6gR+ZgiojhwO7AXaXoUxFxb0RMjoiBpWwo8FjNZgtKWUflbY8xMSJmRsTMxYsXd/MZSJIkSZIkqSMNTzBFxGbAtcCJmfkscCGwIzCKqofTOd1xnMy8ODNHZ+bowYMHd8cuJUmSJEmS1AWNnIOJiOhPlVy6IjOvA8jMJ2rWfw+4sXxcCGxfs/mwUkYn5ZIkSZIkSWqyRj5FLoBLgAcy85s15UNqqr0PuK8sTwHGR8QmETECGAnMAH4PjIyIERGxMdVE4FMaFbckSZIkSZLWTiN7ML0N+Ajwx4iYXcq+BBwVEaOABOYD/wKQmXMi4mqqybtXAJMy82WAiPgU8EugHzA5M+c0MG6pT5k6ddXyuHHNi0OSJEmStOFq5FPkbgOinVU3dbLNWcBZ7ZTf1Nl2kiRJkiRJap4eeYqcJEmSJEmS1l8mmCRJkiRJklQXE0ySJEmSJEmqSyMn+ZYkSdJamDevpdkhSJIkrRN7MEmSJEmSJKkuJpgkSZIkSZJUFxNMkiRJkiRJqotzMEm92NSp615/3LjujUWSJEmSpI7Yg0mSJEmSJEl1sQeTJElSD/NpcZIkaX1jDyZJkiRJkiTVxR5MkiRJkiQ1UMv0llXLY1o6rCf1ZfZgkiRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUFxNMkiRJknqtiOgXEX+IiBvL5xERcVdEzI2IqyJi41K+Sfk8t6wfXrOPL5byhyJibHPORJLWb07yLfUSU6c2OwJJkqRe6QTgAWCL8vnrwLmZeWVEXAR8DLiwvC/NzNdHxPhS70MRsTMwHtgF2A74dUTslJkv9/SJSNL6zB5MUg+bOnXVS5IkSR2LiGHAe4Hvl88B7AdcU6pcBhxelg8rnynr9y/1DwOuzMwXM3MeMBfYs2fOQJI2HPZgkiRJktRbfQs4Bdi8fB4EPJ2ZK8rnBcDQsjwUeAwgM1dExDOl/lDgzpp91m6zUkRMBCYC7LDDDt17FlIXtExvWbU8pqXDelJvZYJJWk/V9pAaN655cUiSJK2LiDgEeDIzZ0XEmEYfLzMvBi4GGD16dDb6eJK0vnGInCSpKSJiQETMiIh7ImJORHyllDt5qyQJ4G3AoRExH7iSamjct4GtIqL1RvkwYGFZXghsD1DWbwk8VVvezjaSpG5igkmS1CwvAvtl5m7AKOCgiNibVZO3vh5YSjVpK9RM3gqcW+rRZvLWg4DvRkS/Hj0TSVK3y8wvZuawzBxOdZ2/OTM/DNwCHFGqTQBuKMtTymfK+pszM0v5+HKjYgQwEpjRQ6chSRsME0ySpKW8UjQAACAASURBVKbIyrLysX95JU7eKknq3BeAkyJiLtUcS5eU8kuAQaX8JOBUgMycA1wN3A/8ApjkE+Qkqfs5B5PURD5JThu60tNoFvB64ALgEZy8VZLURmZOB6aX5Udp50ZCZr4AHNnB9mcBZzUuQql7OeG3+iJ7MEmSmiYzX87MUVTzYewJvLGBx7o4M0dn5ujBgwc36jCSJEnSBskEkySp6TLzaao5Nd6Kk7dKkiRJfY4JJklSU0TE4IjYqixvChwAPICTt0qSJEl9jnMwSZKaZQhwWZmH6TXA1Zl5Y0TcD1wZEWcCf2D1yVt/WCZvXUL1RCEyc05EtE7eugInb5UkSZJ6nAkmSVJTZOa9wO7tlDt5qyRJWm/VTuAtrU8cIidJkiRJkqS62INJ2sBMnbpqedy45sUhSZIkSVp/2INJkiRJkiRJdTHBJEmSJEmSpLqYYJIkSZIkSVJdTDBJkiRJkiSpLiaYJEmSJEmSVBcTTJIkSZIkSaqLCSZJkiRJkiTVxQSTJEmSJEmS6rJRswOQ1HhTpzY7AknSvHktzQ5BkrQeaZnesmp5TEuH9aSeYoJJkiSpl6tNTo0Y0dJhPUmSpGZxiJwkSZIkSZLqYoJJkiRJkiRJdTHBJEmSJEmSpLqYYJIkSZIkSVJdTDBJkiRJkiSpLiaYJEmSJEmSVJeNmh2AJEmSJElas5bpLc0OQeqQPZgkSZIkSZJUFxNMkiRJkiRJqosJJkmSJEmSJNXFBJMkSZIkSZLqYoJJkiRJkiRJdTHBJEmSJEmSpLqYYJIkSZIkSVJdGpZgiojtI+KWiLg/IuZExAmlfOuImBYRD5f3gaU8IuK8iJgbEfdGxJtr9jWh1H84IiY0KmZJkiRJkiStvUb2YFoBnJyZOwN7A5MiYmfgVOA3mTkS+E35DPAeYGR5TQQuhCohBZwO7AXsCZzempSSJEmSJElS8zUswZSZizLz7rL8HPAAMBQ4DLisVLsMOLwsHwZcnpU7ga0iYggwFpiWmUsycykwDTioUXFLkiRJkiRp7fTIHEwRMRzYHbgL2DYzF5VVjwPbluWhwGM1my0oZR2Vtz3GxIiYGREzFy9e3K3xS5IkSZIkqWMNTzBFxGbAtcCJmfls7brMTCC74ziZeXFmjs7M0YMHD+6OXUqSJEmSJKkLGppgioj+VMmlKzLzulL8RBn6Rnl/spQvBLav2XxYKeuoXJIkSZIkSb3ARo3acUQEcAnwQGZ+s2bVFGAC8LXyfkNN+aci4kqqCb2fycxFEfFL4D9qJvY+EPhio+KWJEmSJKkvaZnesmp5TEuH9aRGaliCCXgb8BHgjxExu5R9iSqxdHVEfAz4E/DBsu4m4GBgLvA8cBxAZi6JiDOA35d6X83MJQ2MW5IkSZIkSWuhYQmmzLwNiA5W799O/QQmdbCvycDk7otOkiRJkiRJ3aWRPZgkSZLUzebNa1m5PGJES4f1JEmSepIJJkmSJEmSeqna+ZWk3qyhT5GTJEmSJEnS+s8EkyRJkiRJkupigkmSJEmSJEl1McEkSWqKiNg+Im6JiPsjYk5EnFDKWyJiYUTMLq+Da7b5YkTMjYiHImJsTflBpWxuRJzajPORJEmSNmRO8i1JapYVwMmZeXdEbA7MiohpZd25mXl2beWI2BkYD+wCbAf8OiJ2KqsvAA4AFgC/j4gpmXl/j5yFJEmSJBNMkqTmyMxFwKKy/FxEPAAM7WSTw4ArM/NFYF5EzAX2LOvmZuajABFxZalrgkmSJEnqIQ6RkyQ1XUQMB3YH7ipFn4qIeyNickQMLGVDgcdqNltQyjoqb3uMiRExMyJmLl68uJvPQJIkSdqwmWCSJDVVRGwGXAucmJnPAhcCOwKjqHo4ndMdx8nMizNzdGaOHjx4cHfsUpIkSVLhEDlJUtNERH+q5NIVmXkdQGY+UbP+e8CN5eNCYPuazYeVMjoplyRJktQD7MEkSWqKiAjgEuCBzPxmTfmQmmrvA+4ry1OA8RGxSUSMAEYCM4DfAyMjYkREbEw1EfiUnjgHSZIkSRV7MEkbsKlTVy2PG9e8OLTBehvwEeCPETG7lH0JOCoiRgEJzAf+BSAz50TE1VSTd68AJmXmywAR8Sngl0A/YHJmzunJE5EkSZI2dCaYJElNkZm3AdHOqps62eYs4Kx2ym/qbDtJkiRJjeUQOUmSJEmSJNXFBJMkSZIkSZLqYoJJkiRJkiRJdTHBJEmSJEmSpLqYYJIkSZIkSVJdfIqcJACmTl21PG5c8+KQJEmSJPU99mCSJEmSJElSXUwwSZIkSZIkqS4OkZMkSeqj5s1rWbk8YkRLh/UkSZIazR5MkiRJkiRJqosJJkmSJEmSJNXFBJMkSZKkXiciBkTEjIi4JyLmRMRXSvmIiLgrIuZGxFURsXEp36R8nlvWD6/Z1xdL+UMRMbY5ZyRJ67c1JpgiYpOulEmSJElSN3oR2C8zdwNGAQdFxN7A14FzM/P1wFLgY6X+x4ClpfzcUo+I2BkYD+wCHAR8NyL69eiZSNIGoCs9mO7oYpkkSZIkdYusLCsf+5dXAvsB15Tyy4DDy/Jh5TNl/f4REaX8ysx8MTPnAXOBPXvgFCRpg9LhU+Qi4u+BocCmEbE7EGXVFsBreyA2SZIkSRuw0tNoFvB64ALgEeDpzFxRqiyg+s1CeX8MIDNXRMQzwKBSfmfNbmu3qT3WRGAiwA477NDt5yJJ67sOE0zAWOBYYBhwDqsSTM8BX2psWJIkSZI2dJn5MjAqIrYCfgq8sYHHuhi4GGD06NHZqONow9EyvaXZIUg9qsMEU2ZeBlwWER/IzGt7MCZJkiRJWikzn46IW4C3AltFxEalF9MwYGGpthDYHlgQERsBWwJP1ZS3qt1GktRNujIH07CI2CIq34+IuyPiwIZHJkmSJGmDFRGDS88lImJT4ADgAeAW4IhSbQJwQ1meUj5T1t+cmVnKx5enzI0ARgIzeuYsJGnD0dkQuVYfzcxvl8d5DgI+AvwQ+FVDI5PUK0ydump53LjmxSFJkjY4Q6hGVPSjujF+dWbeGBH3A1dGxJnAH4BLSv1LgB9GxFxgCdWT48jMORFxNXA/sAKYVIbeSZK6UVcSTK1zLx0MXF4u0NHZBpIkSWqeefNaVi6PGNHSYT2pN8vMe4Hd2yl/lHaeApeZLwBHdrCvs4CzujtGSdIqXRkiNysifkWVYPplRGwOvNLYsCRJkiRJktRXdKUH08eAUcCjmfl8RAwCjmtsWJIkSZIkSeorOkwwRcQbM/NBquQSwD86Mk6SJEmSJEltddaD6SRgInBOO+sS2K8hEUmSJEmSJKlP6TDBlJkTy/u7ei4cSZIkSZIk9TVdmYOJiNgHGF5bPzMvb1BMkiRJWku1T46TJEnqaWtMMEXED4EdgdnAy6U4ARNMkiRJkiRJ6lIPptHAzpmZjQ5GkiRJPaO2x9OIES0d1pMkSeqKriSY7gP+HljU4FgkSZLUQA6jkyRJjdJhgikiplINhdscuD8iZgAvtq7PzEMbH54kSZLqYVJJkiT1hM56MJ3dY1FIkiRJkqS6tUxvWbU8pqXDelJ36zDBlJn/05OBSJIkSZIkqW96TbMDkCRJkiRJUt9mgkmSJEmSJEl1WWOCKSLGRYSJKEmSJEmSJLWrs0m+W30I+FZEXAtMzswHGxyTpCabOrXZEUiSJEmS+pI19kzKzP8D7A48AlwaEXdExMSI2Lzh0UmSJEmSJKnX69LQt8x8FrgGuBIYArwPuDsiPt3A2CRJkiRJktQHdGUOpsMi4qfAdKA/sGdmvgfYDTi5seFJkiRJkiSpt+vKHEzvB87NzFtrCzPz+Yj4WGPCkiRJkiRJUl/RlSFyj7dNLkXE1wEy8zcNiUqStN6LiO0j4paIuD8i5kTECaV864iYFhEPl/eBpTwi4ryImBsR90bEm2v2NaHUfzgiJjTrnKS+at68lpUvSZKkddGVBNMB7ZS9p7sDkSRtcFYAJ2fmzsDewKSI2Bk4FfhNZo4EflM+Q9X2jCyvicCFUCWkgNOBvYA9gdNbk1KSJEmSekaHCaaI+GRE/BF4Y7lT3PqaB9y7ph1HxOSIeDIi7qspa4mIhRExu7wOrln3xXJX+qGIGFtTflApmxsRp7Y9jiSpb8rMRZl5d1l+DngAGAocBlxWql0GHF6WDwMuz8qdwFYRMQQYC0zLzCWZuRSYBhzUg6ciSZIkbfA6m4PpR8DPgf9k1d1jgOcyc0kX9n0p8B3g8jbl52bm2bUF5Y71eGAXYDvg1xGxU1l9AVUvqgXA7yNiSmbe34XjS5L6iIgYDuwO3AVsm5mLyqrHgW3L8lDgsZrNFpSyjsrbHmMiVc8ndthhh+4LXpIkSVKnQ+QyM+cDk4Dnal6twxE6VeZt6koiCqq70ldm5ouZOQ+YSzXMYU9gbmY+mpkvAVeWupKk9UREbAZcC5yYmc/WrsvMBLI7jpOZF2fm6MwcPXjw4O7YpSRJkqSiswTTj8r7LGBmeZ9V83ldfaoMtZtcM0dGXXelobozHREzI2Lm4sWL6whPktRTIqI/VXLpisy8rhQ/UYa+Ud6fLOULge1rNh9WyjoqlyRJktRDOkwwZeYh5X1EZv5jeW99/eM6Hu9CYEdgFLAIOGcd9/Mq3pmWpL4lIgK4BHggM79Zs2oK0PokuAnADTXlx5Snye0NPFOG0v0SODAiBpYbFweWMkmSJEk9pMM5mGof/9ye1olZ10ZmPlGz/+8BN5aPnd199q60JK2f3gZ8BPhjRMwuZV8CvgZcHREfA/4EfLCsuwk4mGoY9fPAcQCZuSQizgB+X+p9tYtzBUqSJK3XWqa3rFoe09JhPak7dDbJd2e9ixLYb20PFhFDaiZufR/Q+oS5KcCPIuKbVJN8jwRmAAGMjIgRVIml8cDRa3tcSVLvk5m3UV3n27N/O/WTal7A9vY1GZjcfdFJ3WPevJZmhyBJktQjOkwwZea76tlxRPwYGANsExELgNOBMRExiipBNR/4l3KsORFxNXA/sAKYlJkvl/18imqoQz9gcmbOqScuSZIkSZIkda/Ohsjtl5k3R8T721tfMxlruzLzqHaKL+mk/lnAWe2U30Q1LELqU6ZOXbU8blzz4pAkSZIkqdE6GyL3TuBmoL2fxgl0mmCSJEmSJEnShqGzIXKnl/fjei4cSZIkSZIk9TWvWVOFiBgUEedFxN0RMSsivh0Rg3oiOEmSJEmSJPV+a0wwAVcCi4EPAEeU5asaGZQkSZIkSZL6js7mYGo1JDPPqPl8ZkR8qFEBSZIkSZIkqW/pSg+mX0XE+Ih4TXl9EPhlowOTJEmSJElS39BhD6aIeI7qaXEBnAj8d1n1GmAZ8LmGRyetJ6ZObXYEkiRJkiQ1TmdPkdu8JwORJEmSJElS39SVOZiIiIHASGBAa1lm3tqooCRJkiRJktR3rDHBFBEfB04AhgGzgb2BO4D9GhuapN6sdtjfuHHNi0OSJEmS1HxdmeT7BGAP4E+Z+S5gd+DphkYlSZIkSZKkPqMrCaYXMvMFgIjYJDMfBN7Q2LAkSZIkSZLUV3RlDqYFEbEVcD0wLSKWAn9qbFiSJEmSJEnqK9aYYMrM95XFloi4BdgS+EVDo5IkSZIkSVKf0dWnyL0Z2BdI4HeZ+VJDo5IkSVJTzJvXsnJ5xIiWDutJkiTVWuMcTBHx78BlwCBgG+AHEXFaowOTJEmSJElS39CVHkwfBnarmej7a8Bs4MxGBiZJkiRJkqS+oStPkfsLMKDm8ybAwsaEI0mSJEmSpL6mwx5MEXE+1ZxLzwBzImJa+XwAMKNnwpMkSZIkSVJv19kQuZnlfRbw05ry6Q2LRpIkSZIkSX1OhwmmzLysdTkiNgZ2Kh8fyszljQ5MkiRJkqS+pGV6S7NDkJpmjZN8R8QYqqfIzQcC2D4iJmTmrY0NTZIkSZIkSX1BV54idw5wYGY+BBAROwE/Bt7SyMAkSZIkSZLUN3TlKXL9W5NLAJn5/4D+jQtJkiRJkiRJfUlXejDNiojvA/9dPn+YVROAS9qATJ3a7AgkSZIkSb1RVxJMxwOTgM+Uz78FvtuwiCRJkiRJktSndJpgioh+wD2Z+Ubgmz0TkiRJkiRJkvqSTudgysyXgYciYoceikeSJEmSJEl9TFeGyA0E5kTEDOBvrYWZeWjDopIkSZIkSVKf0ZUE0781PApJkiRJkiT1WR0mmCJiANUE368H/ghckpkreiowSZIkSZIk9Q2dzcF0GTCaKrn0HuCcHolIkiRJkiRJfUpnQ+R2zsx/BoiIS4AZPROSJEmSJEmS+pLOEkzLWxcyc0VE9EA4kvqiqVNXLY8b17w4JEmSJEnN0VmCabeIeLYsB7Bp+RxAZuYWDY9OkiRJkiRJvV6HczBlZr/M3KK8Ns/MjWqWTS5JkiRJapiI2D4ibomI+yNiTkScUMq3johpEfFweR9YyiMizouIuRFxb0S8uWZfE0r9hyNiQrPOSZLWZ51N8i1JkiRJzbICODkzdwb2BiZFxM7AqcBvMnMk8JvyGaoHE40sr4nAhVAlpIDTgb2APYHTW5NSkqTuY4JJktQUETE5Ip6MiPtqyloiYmFEzC6vg2vWfbHclX4oIsbWlB9UyuZGxKltjyNJ6psyc1Fm3l2WnwMeAIYCh1E98ZryfnhZPgy4PCt3AltFxBBgLDAtM5dk5lJgGnBQD56K1Cu0TG9Z+ZIawQSTJKlZLqX9P/DPzcxR5XUTQLljPR7YpWzz3YjoFxH9gAuo7lrvDBxV6kqS1iMRMRzYHbgL2DYzF5VVjwPbluWhwGM1my0oZR2Vtz3GxIiYGREzFy9e3K3xS9KGwASTJKkpMvNWYEkXqx8GXJmZL2bmPGAu1TCHPYG5mfloZr4EXFnqSpLWExGxGXAtcGJmPlu7LjMTyO44TmZenJmjM3P04MGDu2OXkrRBMcEkSeptPlUmZ51cM0dGXXelwTvTktQXRUR/quTSFZl5XSl+ogx9o7w/WcoXAtvXbD6slHVULknqRiaYJEm9yYXAjsAoYBFwTnft2DvTktS3REQAlwAPZOY3a1ZNAVqfBDcBuKGm/JjyNLm9gWfKULpfAgdGxMBy4+LAUiZJ6kYbNTsASZJaZeYTrcsR8T3gxvKxs7vP3pWWpPXT24CPAH+MiNml7EvA14CrI+JjwJ+AD5Z1NwEHUw2jfh44DiAzl0TEGcDvS72vZmZXh2hLkrrIBJMkqdeIiCE1E7e+D2h9wtwU4EcR8U1gO6pHUM8AAhgZESOoEkvjgaN7NmpJUiNk5m1U1/n27N9O/QQmdbCvycDk7otOktSWCSZJUlNExI+BMcA2EbEAOB0YExGjqCZsnQ/8C0BmzomIq4H7gRXApMx8ueznU1RDHfoBkzNzTg+fiiRJkrTBM8EkSWqKzDyqneJLOql/FnBWO+U3UQ2LkCRJktQkTvItSZIkSZKkuphgkiRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUFyf5liRJUrvmzWtZuTxiREuH9SRJkuzBJEmSJEmSpLqYYJIkSZIkSVJdTDBJkiRJkiSpLiaYJEmSJEmSVJeGJZgiYnJEPBkR99WUbR0R0yLi4fI+sJRHRJwXEXMj4t6IeHPNNhNK/YcjYkKj4pUkSZIkSdK6aWQPpkuBg9qUnQr8JjNHAr8pnwHeA4z8/+3db4yl1X0f8O8vQY5VFxcI6y3lj9lU66ZulaRoC7yIYlxSjIkwNEYUSy4bRLpyStoXbSVTuRIj+0VJK7WKVYuGWisvlmzHtUTZTaidNTGijUwKrTFgN4G11y67AXZtHFqV/onb0xf3WXO9O7N7Z++f57l3Ph/p6j73zJm73z0zd86d35zzPN1tT5L7klFBKsk9Sa5KcmWSe04UpQAAAAAYhrkVmFprjyV55aTmm5Ls6473Jbl5rP2BNvJ4kvOq6qIk70pysLX2Smvte0kO5tSiFQAAAAA9WvQ5mLa31l7sjl9Ksr07vjjJC2P9jnRtG7Wfoqr2VNWTVfXk8ePHZ5saAAAAgA31dpLv1lpL0mb4fPe31na11nZt27ZtVk8LAAAAwBksusD0crf1Ld39sa79aJJLx/pd0rVt1A4AAADAQCy6wLQ/yYkrwe1O8tBY++3d1eSuTvJqt5XuC0muq6rzu5N7X9e1AQAAADAQ58zriavq00muSXJhVR3J6Gpw9yb5bFXdmeTbSW7tuj+c5IYkh5K8luSOJGmtvVJVH0nyRNfvw621k08cDgAAAECP5lZgaq29b4MPXbtO35bkrg2eZ2+SvTOMBgAAAMAM9XaSbwAAAABWgwITAAAAAFNRYAIAAABgKgpMAAAAAExlbif5BgAAgFW39uha3xFgEKxgAgAAAGAqCkwAAAAATMUWOWCmDhx4/fjGG/vLAQAAwOJYwQQAAADAVBSYAAAAAJiKAhMAAAAAU3EOJgAAzujw4bUfHO/YsbZhPwBga7KCCQAAAICpKDABAAAAMBVb5GCGDhzoOwEAAAAsnhVMAAAAAEzFCiZgIcZXd914Y385AAAAmD0rmAAAAACYihVMwNw4JxUAAMDWoMAEAAAAW8jao2uvH1+ztmE/2Axb5AAAAACYigITAAAAAFNRYAKgF1W1t6qOVdWzY20XVNXBqnq+uz+/a6+q+mhVHaqqp6vqirHP2d31f76qdvfxf4Gt5vDhtR/cAAASBSYA+vOJJNef1HZ3kkdaazuTPNI9TpJ3J9nZ3fYkuS8ZFaSS3JPkqiRXJrnnRFEKAABYHAUmAHrRWnssySsnNd+UZF93vC/JzWPtD7SRx5OcV1UXJXlXkoOttVdaa99LcjCnFq0AAIA5cxU5mNKBA30ngJWyvbX2Ynf8UpLt3fHFSV4Y63eka9uo/RRVtSej1U+57LLLZhgZAACwggmAQWqttSRths93f2ttV2tt17Zt22b1tAAAQBSYABiWl7utb+nuj3XtR5NcOtbvkq5to3YAAGCBFJgAGJL9SU5cCW53kofG2m/vriZ3dZJXu610X0hyXVWd353c+7quDXrlCmsAwFbjHEwA9KKqPp3kmiQXVtWRjK4Gd2+Sz1bVnUm+neTWrvvDSW5IcijJa0nuSJLW2itV9ZEkT3T9PtxaO/nE4QAAwJwpMAHQi9ba+zb40LXr9G1J7trgefYm2TvDaAAAp7X26FrfEWBwbJEDAAAAYCoKTAAAAABMRYEJAAAAgKkoMAEAAAAwFQUmAAAAAKaiwAQAAADAVBSYAAAAAJiKAhMAAAAAU1FgAgAAAGAqCkwAAAAATOWcvgMAAKyCw4fX+o4AANAbK5gAAAAAmIoVTAAAnLXxlVs7dqxt2A8AWG1WMAEAAAAwFQUmAABgcKpqb1Udq6pnx9ouqKqDVfV8d39+115V9dGqOlRVT1fVFWOfs7vr/3xV7e7j/wKwFSgwAQAAQ/SJJNef1HZ3kkdaazuTPNI9TpJ3J9nZ3fYkuS8ZFaSS3JPkqiRXJrnnRFEKgNlSYAIAAAantfZYkldOar4pyb7ueF+Sm8faH2gjjyc5r6ouSvKuJAdba6+01r6X5GBOLVoBMAMKTAAAwLLY3lp7sTt+Kcn27vjiJC+M9TvStW3Ufoqq2lNVT1bVk8ePH59taoAtQIEJAABYOq21lqTN8Pnub63taq3t2rZt26yeFmDLUGACAACWxcvd1rd098e69qNJLh3rd0nXtlE7ADOmwAQAACyL/UlOXAlud5KHxtpv764md3WSV7utdF9Icl1Vnd+d3Pu6rg2AGTun7wAAAAAnq6pPJ7kmyYVVdSSjq8Hdm+SzVXVnkm8nubXr/nCSG5IcSvJakjuSpLX2SlV9JMkTXb8Pt9ZOPnE4ADOgwAQAAAxOa+19G3zo2nX6tiR3bfA8e5PsnWE0ANZhixwAAAAAU+mlwFRV36qqZ6rqqap6smu7oKoOVtXz3f35XXtV1Uer6lBVPV1VV/SRGQAAAID19blF7p2tte+MPb47ySOttXur6u7u8QeTvDvJzu52VZL7unvozYEDfScAAACA4RjSFrmbkuzrjvcluXms/YE28niS805cmhQAAACA/vVVYGpJfqeq/lNV7enatneXEk2Sl5Js744vTvLC2Oce6dp+SFXtqaonq+rJ48ePzys3AAAAACfpa4vcz7bWjlbVW5IcrKo/GP9ga61VVdvME7bW7k9yf5Ls2rVrU58LAAAAwNnrZQVTa+1od38syYNJrkzy8omtb939sa770SSXjn36JV0bAAAAAAOw8AJTVb2pqs49cZzkuiTPJtmfZHfXbXeSh7rj/Ulu764md3WSV8e20gEAAADQsz62yG1P8mBVnfj3P9Va+3xVPZHks1V1Z5JvJ7m16/9wkhuSHEryWpI7Fh8ZAAAAgI0svMDUWvtmkp9ep/27Sa5dp70luWsB0QAAAAA4C31dRQ4AAACAFaHABAAAAMBUFJgAAAAAmIoCEwAAAABT6eMqcqyIAwdeP77xxv5yAAAAAP2yggkAAACAqVjBBADATBw+vPaD4x071jbsBwCsHiuYABicqvpWVT1TVU9V1ZNd2wVVdbCqnu/uz+/aq6o+WlWHqurpqrqi3/QAALD1WMHEupxfCRiAd7bWvjP2+O4kj7TW7q2qu7vHH0zy7iQ7u9tVSe7r7oEeWc0EAFuLAhMAy+KmJNd0x/uSPJpRgemmJA+01lqSx6vqvKq6qLX2Yi8pAYCVtPboWt8R5mL8/7V2zdqG/eBMFJgAGKKW5HeqqiX5jdba/Um2jxWNXkqyvTu+OMkLY597pGv7oQJTVe1JsidJLrvssjlGZysZX6UDALCVKTBtcbbC0Qffd0zgZ1trR6vqLUkOVtUfjH+wtda64tPEuiLV/Umya9euTX0uAABwegpMQK8Um1hPa+1od3+sqh5McmWSl09sfauqi5Ic67ofTXLp2Kdf0rUBAAAL4ipyAAxKVb2pqs49cZzkuiTP0+9nggAADRlJREFUJtmfZHfXbXeSh7rj/Ulu764md3WSV51/CQAAFssKJgCGZnuSB6sqGc1Tn2qtfb6qnkjy2aq6M8m3k9za9X84yQ1JDiV5Lckdi48MAKyqVT25N8yaAhMAg9Ja+2aSn16n/btJrl2nvSW5awHRBsPl3wEAGBoFpgFyThoAhkIxCwCASSgwrbB5FKrGn3NoFOYAAACgHwpMK6avApDiDgAAAGxdriIHAAAAwFSsYGJhlnGV0zJmBgAAgEVTYGKpTbIlUJEIAADYjLVH1/qOAEtHgWkFzOq8S/M4f9NGzzlJ0WeawtAi/y8AAACw1SkwsaVYzQQAAACzp8DEUlAYAra6w4fX+o4AACvNtjiYjgITvbP1DIBpjRfgduxY27AfAADzocC0RSjinGqjVVHGCtjqln21lGLTsPn6AMBqUmACBsNWSAAA5mF8+9vaNWsb9gPOngITS8cKIwD6tOwrvABgIwpxTEOBCQBWkG1Is6WoBABwegpMAMDKFlAU2oZhVb+/gOXnynEwOwpMA+ecNIth2x0AAMD6bJ1jEgpMALDirOIBYCuyOgkWS4EJAJaY4tFs2coFAHB2FJgAgC1HYQ4ATs8KMDZLgWmJOE8QALxuVquNrFrqj0IfAKwOBSbOaLOFLYUwAE5no6LCvIsNCkkAAPOjwAQAW9Q8VgApDAEwdLZ+wXwoMA2EVT8AAAAM3UYFurVr1m9n61BgAhiAjYrMN9642BwAAABnQ4EJAFaErWQAbEXjK2qsooH+KDABgze+useKHpjONEWoRV7xS7EMAGC5KDABg+S8ZAAAAMtDgQkAmItJVjxZqQTALLlCHPRHgalHVmjA1mO7HwAAsIoUmADmQCGJWVil1T2r9H+BeVrkuc6gb07OvVo2Wj3ma7t1KDAB9MQqRpaZghEAAOMUmAAGzEooYKuwcuf0jA8AQ6fABDBnVioBcDasFGQVOQk3rC4FJgBgZvxCzCxYrQNbj/MxwfJTYAKWypC3jFmpBMDJJimWKczCxpw4GpaHAhMAACyAlVkwGdvoYDkpMAEAMFjLWJSZZEWSVUtsJfMoGClCLQ/bH7cOBSZgJfS1dc62OAAAAAWmhRjyOWOAzeurqORnCUD/NlpRtcgVScu4qguA1afABCwtq4dYRbbNwPLwegXYHNvlVtvSFJiq6vokv57kR5N8vLV2b8+RgIGax0ofxazhW9Q8McnqhUnagc3bqKAzj9eW4tHq8fvE7ExyZTfnSIKtZykKTFX1o0k+luSvJzmS5Imq2t9a+3q/yYBVsFHxyDa05TG0eWKzv5j6RRams9lCrtfc1jO0eWLIJikebfZzoQ9WSy3eUhSYklyZ5FBr7ZtJUlWfSXJTkplPCLP6RdNqBxiGaV6LXsdLZWHzxDT8UguLtdVec1ZMntZSzBOTmKSIs9lfpid6TsUjZmxW31OTrJybpnC66TxzKGwtS7GsWmt9ZzijqrolyfWttV/uHv+tJFe11n51rM+eJHu6h38hyR9u8HQXJvnOHONOS77pDDnfkLMl8k1rmfK9tbW2rc8wszbDeWKZvo5DNPR8yfAzyjedoedLhp/xwiRvMk8s7TzRB2NyKmOyPuNyqmUckw1/l1iWFUxn1Fq7P8n9Z+pXVU+21nYtINJZkW86Q8435GyJfNOSb/gmmSeGPk7yTW/oGeWbztDzJcPP2OW7vO8cfViFeaIPxuRUxmR9xuVUqzYmP9J3gAkdTXLp2ONLujYASMwTAJyeeQJgzpalwPREkp1VtaOq3pDktiT7e84EwHCYJwA4HfMEwJwtxRa51tr3q+pXk3who8uK7m2tfe0sn+6M2+h6Jt90hpxvyNkS+aYlX49mOE8MfZzkm97QM8o3naHnS4afcej5zsoWmif6YExOZUzWZ1xOtVJjshQn+QYAAABguJZlixwAAAAAA6XABAAAAMBUVrbAVFXXV9UfVtWhqrp7nY///ar6elU9XVWPVNVbB5bvA1X1TFU9VVX/oarePpRsY/3eW1WtqhZ6WcUJxu6Xqup4N3ZPVdUvDylf1+fW7vvva1X1qSHlq6p/MTZ2z1XVHw8s32VV9aWq+kr3+r1hYPne2v1MebqqHq2qSxaYbW9VHauqZzf4eFXVR7vsT1fVFYvKNjTmiPnmG+tnnjjLjF0fc8XZ5zNXnD6f+WJCVXVBVR2sque7+/PX6fMzVfXl7rX6dFX9zT6yLsokY9L1+3xV/XFV/daiMy7KBK/1H6uq3+w+/vtVdfniUy7WBGPyc1X1n6vq+1V1Sx8Z+zD0954z01pbuVtGJ+77RpKfSPKGJF9N8vaT+rwzyZ/qjn8lyW8OLN+bx47fk+TzQ8nW9Ts3yWNJHk+ya2Bj90tJ/uWAv/d2JvlKkvO7x28ZUr6T+v/djE6COZh8GZ0I71e647cn+dbA8v2bJLu747+W5JMLzPdzSa5I8uwGH78hyb9LUkmuTvL7i8o2pJs5Yv75un7miekymiumGz9zxekzmi8mH6t/muTu7vjuJL+2Tp+3JdnZHf+5JC8mOa/v7H2OSfexa5PcmOS3+s48p3GY5LX+d5L8q+74tizw/cSAx+TyJD+V5IEkt/SdeUDj0tt7z1neVnUF05VJDrXWvtla+z9JPpPkpvEOrbUvtdZe6x4+nmSRfzmaJN9/G3v4piSLOhv7GbN1PpLk15L8rwXlOmHSfH2ZJN/fTvKx1tr3kqS1dmxg+ca9L8mnF5JsZJJ8Lcmbu+M/k+SPBpbv7Ul+tzv+0jofn5vW2mNJXjlNl5uSPNBGHk9yXlVdtJh0g2KOmHO+jnliY+aK6ZgrpmS+2JSbkuzrjvclufnkDq2151prz3fHf5TkWJJtC0u4eGcckyRprT2S5L8vKlQPJnmtj4/V55JcW1W1wIyLNsl7mG+11p5O8v/6CNiTob/3nJlVLTBdnOSFscdHuraN3JnRX2kWZaJ8VXVXVX0jo78S/L2hZOuWSV/aWvvtBWUaN+nX9r3d8sLPVdWli4mWZLJ8b0vytqr6vap6vKquX1i6Tbw2umWZO/L6G+BFmCTfWpL3V9WRJA9n9JfzRZkk31eT/GJ3/DeSnFtVP76AbJPY7M/GVWWOmI55YnrmiumYK+bPfPG67a21F7vjl5JsP13nqroyoxUK35h3sB5takxW2CSvkx/0aa19P8mrSYb0Wp81PzvWN/T3njOzqgWmiVXV+5PsSvLP+s5ystbax1prfz7JB5P8477zJElV/UiSf57kH/Sd5TQOJLm8tfZTSQ7m9b8aDMU5GW19uCajv/r+66o6r9dE67styedaa/+37yAneV+ST7TWLsloCf8nu+/LofiHSd5RVV9J8o4kR5MMbQyZkDli88wTM2OumI65golV1Rer6tl1bievMGg5zYrRbpXXJ5Pc0Vpb6tUZsxoTYHOG/N5zEuf0HWBOjiYZ/2vkJV3bD6mqn0/yoSTvaK397wVlSybMN+YzSe6ba6LXnSnbuUn+cpJHu9WdfzbJ/qp6T2vtyQHkS2vtu2MPP57RX/cXZZKv7ZGMzmXwJ0kOV9VzGf0S8cRA8p1wW5K75p7oh02S784k1ydJa+3LVfXGJBdmtBy993zd0vhfTJKq+tNJ3ttaW+jJb09jsz97VpU5YjrmiemZK6Zjrpi/LTVftNZ+fqOPVdXLVXVRa+3FroC07vdQVb05yW8n+VC3rXCpzWJMtoBJXicn+hypqnMy2rL73ayuLfWzYxOG/t5zZob0l5xZeiLJzqraUVVvyOjNz/7xDlX1V5L8RpL3LPi8BpPm2zn28BeSPD+EbK21V1trF7bWLm+tXZ7R/tBF/dJwxnzJD/56dMJ7kvyXBWWbKF+Sf5vRX6RTVRdmtA3imwPKl6r6ySTnJ/nygnJtJt9/zeikkamqv5jkjUmODyVfVV049lfyf5Rk74KyTWJ/kttr5Ookr44tcd9KzBFzzGeemIi5Yv75zBXTMV+8bn+S3d3x7iQPndyh+zo/mNF5qz63wGx9OeOYbBGT/CwaH6tbkvxut+prVU00f2xBQ3/vOTttAGcan8cto+XQz2W0//lDXduHM/qCJckXk7yc5Knutn9g+X49yde6bF9K8peGku2kvo9mgVcHmnDs/kk3dl/txu4nB5avMto+8vUkzyS5bUj5usdrSe5dZK5NjN/bk/xe9/V9Ksl1A8t3S0a/7D+X0cqIH1tgtk9ndOWaP8lo9cOdST6Q5ANj33sf67I/s+jX7pBu5oj55jupr3ni7DKaK6YbP3PF6fOZLyYfqx9P8kj39fpikgu69l1JPt4dv78by6fGbj/Td/Y+x6R7/O8zKuz+z+777F19Z5/DWJzptf7GjK4aeSjJf0zyE31nHsCY/NXu++F/ZLSa62t9Zx7IuPT63nNWt+r+MwAAAABwVlZ1ixwAAAAAC6LABAAAAMBUFJgAAAAAmIoCEwAAAABTUWACAAAAYCoKTAAAAABMRYEJAAAAgKn8f8mlgi0bCaoJAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# installation instructions: https://github.com/maks-sh/scikit-uplift\n", "# link to the documentation: https://scikit-uplift.readthedocs.io/en/latest/\n", "from sklift.metrics import uplift_at_k\n", "from sklift.viz import plot_uplift_preds\n", "from sklift.models import SoloModel\n", "\n", "# sklift supports all models, \n", "# that satisfy scikit-learn convention\n", "# for example, let's use catboost\n", "from catboost import CatBoostClassifier\n", "\n", "\n", "sm = SoloModel(CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True))\n", "sm = sm.fit(X_train, y_train, treat_train, estimator_fit_params={'cat_features': cat_features})\n", "\n", "uplift_sm = sm.predict(X_val)\n", "\n", "sm_score = uplift_at_k(y_true=y_val, uplift=uplift_sm, treatment=treat_val, strategy='by_group', k=0.3)\n", "\n", "models_results['approach'].append('SoloModel')\n", "models_results['uplift@30%'].append(sm_score)\n", "\n", "# get conditional probabilities (predictions) of performing the target action \n", "# during interaction for each object\n", "sm_trmnt_preds = sm.trmnt_preds_\n", "# And conditional probabilities (predictions) of performing the target action \n", "# without interaction for each object\n", "sm_ctrl_preds = sm.ctrl_preds_\n", "\n", "# draw the probability (predictions) distributions and their difference (uplift)\n", "plot_uplift_preds(trmnt_preds=sm_trmnt_preds, ctrl_preds=sm_ctrl_preds);" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:42.172080Z", "start_time": "2020-05-30T22:29:42.150138Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 235 }, "colab_type": "code", "id": "VH0YzW0JCIM0", "outputId": "f9e2ae2f-904d-4210-9271-fb74d97e2402" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
feature_namefeature_score
0first_redeem_time65.214393
1issue_redeem_delay12.564364
2age7.891613
3first_issue_time7.262806
4treatment4.362077
5gender2.704747
\n", "
" ], "text/plain": [ " feature_name feature_score\n", "0 first_redeem_time 65.214393\n", "1 issue_redeem_delay 12.564364\n", "2 age 7.891613\n", "3 first_issue_time 7.262806\n", "4 treatment 4.362077\n", "5 gender 2.704747" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# You can also access the trained model with the same ease.\n", "# For example, to build the importance of features:\n", "sm_fi = pd.DataFrame({\n", " 'feature_name': sm.estimator.feature_names_,\n", " 'feature_score': sm.estimator.feature_importances_\n", "}).sort_values('feature_score', ascending=False).reset_index(drop=True)\n", "\n", "sm_fi" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Fi_ypKsrCINU" }, "source": [ "### 1.2 Class Transformation\n", "\n", "Simple yet powerful and mathematically proven uplift modeling method, presented in 2012.\n", "The main idea is to predict a slightly changed target $Z_i$:\n", "\n", "$$\n", "Z_i = Y_i \\cdot W_i + (1 - Y_i) \\cdot (1 - W_i),\n", "$$\n", "\n", "where \n", "\n", "* $Z_i$ - new target variable of the $i$ client; \n", "* $Y_i$ - target variable of the $i$ client;\n", "* $W_i$ - flag for communication of the $i$ client; \n", "\n", "\n", "In other words, the new target equals 1 if a response in the treatment group is as good as a response in the control group and equals 0 otherwise:\n", "\n", "$$\n", "Z_i = \\begin{cases}\n", " 1, & \\mbox{if } W_i = 1 \\mbox{ and } Y_i = 1 \\\\\n", " 1, & \\mbox{if } W_i = 0 \\mbox{ and } Y_i = 0 \\\\\n", " 0, & \\mbox{otherwise}\n", " \\end{cases}\n", "$$\n", "\n", "Let's go deeper and estimate the conditional probability of the target variable:\n", "\n", "$$ \n", "P(Z=1|X = x) = \\\\\n", "= P(Z=1|X = x, W = 1) \\cdot P(W = 1|X = x) + \\\\\n", "+ P(Z=1|X = x, W = 0) \\cdot P(W = 0|X = x) = \\\\\n", "= P(Y=1|X = x, W = 1) \\cdot P(W = 1|X = x) + \\\\\n", "+ P(Y=0|X = x, W = 0) \\cdot P(W = 0|X = x).\n", "$$\n", "\n", "We assume that $ W $ is independent of $X = x$ by design.\n", "Thus we have: $P(W | X = x) = P(W)$ and\n", "\n", "$$\n", "P(Z=1|X = x) = \\\\\n", "= P^T(Y=1|X = x) \\cdot P(W = 1) + \\\\\n", "+ P^C(Y=0|X = x) \\cdot P(W = 0)\n", "$$\n", "\n", "Also, we assume that $P(W = 1) = P(W = 0) = \\frac{1}{2}$, which means that during the experiment the control and the treatment groups were divided in equal proportions. Then we get the following:\n", "\n", "$$\n", "P(Z=1|X = x) = \\\\\n", "= P^T(Y=1|X = x) \\cdot \\frac{1}{2} + P^C(Y=0|X = x) \\cdot \\frac{1}{2} \\Rightarrow \\\\\n", "2 \\cdot P(Z=1|X = x) = \\\\\n", "= P^T(Y=1|X = x) + P^C(Y=0|X = x) = \\\\\n", "= P^T(Y=1|X = x) + 1 - P^C(Y=1|X = x) \\Rightarrow \\\\\n", "\\Rightarrow P^T(Y=1|X = x) - P^C(Y=1|X = x) = \\\\\n", " = uplift = 2 \\cdot P(Z=1|X = x) - 1\n", "$$\n", "\n", "Thus, by doubling the estimate of the new target $Z$ and subtracting one we will get an estimation of the uplift:\n", "\n", "$$\n", "uplift = 2 \\cdot P(Z=1) - 1\n", "$$\n", "\n", "This approach is based on the assumption: $P(W = 1) = P(W = 0) = \\frac{1}{2}$, That is the reason that it has to be used only in cases where the number of treated customers (communication) is equal to the number of control customers (no communication)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:43.218019Z", "start_time": "2020-05-30T22:29:42.175286Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 71 }, "colab_type": "code", "id": "j8ZobWF-CINc", "outputId": "16ccac52-b9f5-4d55-a05c-1d3270819f91", "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/Maksim/Library/Python/3.6/lib/python/site-packages/ipykernel_launcher.py:5: UserWarning: It is recommended to use this approach on treatment balanced data. Current sample size is unbalanced.\n", " \"\"\"\n" ] } ], "source": [ "from sklift.models import ClassTransformation\n", "\n", "\n", "ct = ClassTransformation(CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True))\n", "ct = ct.fit(X_train, y_train, treat_train, estimator_fit_params={'cat_features': cat_features})\n", "\n", "uplift_ct = ct.predict(X_val)\n", "\n", "ct_score = uplift_at_k(y_true=y_val, uplift=uplift_ct, treatment=treat_val, strategy='by_group', k=0.3)\n", "\n", "models_results['approach'].append('ClassTransformation')\n", "models_results['uplift@30%'].append(ct_score)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "jaCZSMxyCIOV" }, "source": [ "## 2. Approaches with two models\n", "\n", "The two-model approach can be found in almost any uplift modeling work and is often used as a baseline. However, using two models can lead to some unpleasant consequences: if you use fundamentally different models for training, or if the nature of the test and control group data is very different, then the scores returned by the models will not be comparable. As a result, the calculation of the uplift will not be completely correct. To avoid this effect, you need to calibrate the models so that their scores can be interpolated as probabilities. The calibration of model probabilities is described perfectly in [scikit-learn documentation](https://scikit-learn.org/stable/modules/calibration.html).\n", "\n", "### 2.1 Two independent models\n", "\n", "The main idea is to estimate the conditional probabilities of the treatment and control groups separately.\n", "\n", "1. Train the first model using the treatment set.\n", "2. Train the second model using the control set.\n", "3. Inference: subtract the control model scores from the treatment model scores.\n", "\n", "

\n", " \"Two\n", "

" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:44.967862Z", "start_time": "2020-05-30T22:29:43.220363Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 444 }, "colab_type": "code", "id": "ydBCIN_XCIOb", "outputId": "b6f2a15a-cdbc-42ba-c013-95cae220c946" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABJsAAAGrCAYAAABntZYXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdfbxVVZ348c83fKDyGfk5CDIwRTU6o5gYZlakpWaSPfiA9ks0i7HxMS3TcsZb6ow1muVD+rMgtTHR1FEwm9KUMfMRjUw0f2JoQqgkoJI/FfT7+2OvC4fLvZfL9Zx77uV+3q/Xfd191lp77+++6F7nfM9aa0dmIkmSJEmSJNXDm5odgCRJkiRJktYdJpskSZIkSZJUNyabJEmSJEmSVDcmmyRJkiRJklQ3JpskSZIkSZJUNyabJEmSJEmSVDcmm6R1QES0RMR/lu3hEbE0IgZ04zhfi4gf1j9CSVq3RcQTEfHhJp370og4o2y/PyIe7eZxLo6If6lvdJLUd3V2f42Id0bErIh4MSKObV6UK+IZFxHzal7Pjohx3ThOt/sRqZbJJjVUSXq0/rweEf+v5vVn6nieFR1BT4uIERGREbFeM87fVmb+KTM3yszXOmvXtkMq+/5bZn6+sRFKUvdExCERMbP0IQsi4ucRsVsdjtu0PqTeMvPXmfnONbWLiMMi4o42+x6Zmac3LjpJ6lnlPfrb25St+JJ2bbRzfz0JuC0zN87M85r5pUN7MnO7zJyxpnZt/0Zd7UekNTHZpIYqSY+NMnMj4E/A+JqyK1rb9ZZETW/g30KSVhcRJwDfBf4N2AoYDnwf2K8Hzt1j92X7AEnqM/4WmN2og3dnloLUm5hsUlO0jqqJiK9GxNPAjyLiTRFxckQ8HhHPRcTVEbFFzT4/jYinI+L5iLg9IrYr5ZOAzwAnlW+7p5fyJyLiKxHxYET8NSImR8RW5ZvwFyPilojYvOb4u0TEnRGxJCJ+VzvsNCJmRMTpEfGbsu8vI2LLUn17+b2knP+97VxvS0RcExFXlf0fiIgdauqfKH+LB4G/RsR6a4hnZET8TznWzcCWNXWrjLSKiC0i4kcR8eeIWBwR10fEW4GfA1vXjDTbuu03PRHx8TIEd0n5G/x9m5i/XP6+z5drG1jqtoyIG8t+iyLi1xHh/UZSt0TEpsA3gaMy87rM/GtmLsvM6Zn5ldJmw4j4brnX/blsb1jqWvucEyPi2TIq6vBS11kf0va+3OE9cQ3xXxrVFLWby337fyLib2vqMyKOiojHgMdK2b5RTc9YUvqC7Wva71j6kRcj4ipgYE1d22kU20TEdRGxMKq+9YIS98XAe8s1L6mJ84yafb8QEXPKfXxaRGzdJuYjI+KxEuOFERGl7u3lGp+PiL+UGCWp16npH75W7ldPRAezL2rvrxFxK/Ah4IJyH72S6kuQ6eX1SWt7rnIPvigiboqIvwIfKu/Pry338LlRM10vIt5c9lkcEQ8DO7c534qRVhExoJz38dJ33F/6h9bPMb8rcR/UTj/y96XPW1L6wI+3ifnCiPhZOe49EfG2UhcRcW7pd1+IiN9HxD+s5T+R+jA//KmZ/gbYgupbgUnAMcAngA8CWwOLgQtr2v8cGAX8L+AB4AqAzLykbH+7jJgaX7PPp4GPAO8AxpdjfA0YTPXf/7EAETEU+BlwRonpy8C1ETG45liHAIeX829Q2gB8oPzerJz/rg6udz/gp+X4PwGuj4j1a+oPBj4GbEb1rX1n8fwEuJ8qyXQ6MLGDcwL8GHgLsF2J/dzM/CvwUeDPNSPN/ly7U0S8A7gSOL78vW6i6kA3qGl2ILA3MBLYHjislJ8IzCv7bUX1N89OYpSkzryXKqHyX520+TqwCzAa2AF4D3BqTf3fAJsCQ4EjgAsjYvM19CG19+W/Y833xM58hup+vSUwq5yz1ieAscC2EbEjMAX4J2AQ8H+AaVEl1DYArqe6t29B1a98ur0TRvWt+I3Ak8CIcu1TM/MR4EjgrnLNm7Wz7+7Av1Pd54eUY0xt02xfqg8325d2e5Xy04FfApsDw4DzO//TSFJT/Q3VvXko1XvqSyKi02lkmbk78Gvg6HIfPZhVZ3F8u5vnOgQ4E9gYuBOYDvyutN8DOD4iWu+1pwFvKz970fnngROo+rR9gE2AzwEvZWbr55gdStyrfDlQPqtMp7qn/y+qz2tXtIl5AvANqnv+nBI/wJ5Un5PeQdX/Hgg810mMWseYbFIzvQ6clpmvZOb/o3rj+/XMnJeZrwAtwP5RRuhk5pTMfLGmboeovu3uzPmZ+UxmzqfqEO7JzN9m5stUH1p2LO3+N3BTZt6Uma9n5s3ATKobcqsfZeb/LbFeTfWBZm3cn5nXZOYy4DtUH5x2qak/LzOfKsfvMJ6IGE715v5fyt/udqpOYDURMYQqqXRkZi4uIwH+p4vxHgT8LDNvLjGfDbwZ2LVNzH/OzEUlhta/yTKqDyd/W87568w02SSpuwYBf8nM5Z20+Qzwzcx8NjMXUr3x/WxN/bJSvywzbwKWAmtak6L2vtyVe2JnfpaZt5c+7OtUo4q2qan/98xcVM41Cfg/mXlPZr6WmZcBr1D1GbsA6wPfLddyDXBfB+d8D9WXN18po8Fezsw7Omjb1meAKZn5QIn5lBLziJo2Z2Xmksz8E3Abq/YBfwtsvZbnlKRmaX1f/T9UX/ge2KRz3ZCZv8nM14F/BAZn5jcz89XM/CPwA6rkDmW/M0vf8RRwXifn/DxwamY+mpXfZWZXEj+7ABtR3e9fzcxbqb7EOLimzX9l5r2lj76CVfuCjYF3AZGZj2Tmgi6cU+sIk01qpoUl6dPqb4H/KkM0lwCPAK8BW5Whn2eVoZ8vAE+Ufbakc8/UbP+/dl5vVHPuA1rPXc6/G1XCpNXTNdsv1ezbVU+1bpQOZB7Vh4DV6tcQz9bA4jI6qdWTHZxzG2BRZi5ey1gp51lx3BLzU1TfrLTq6G/yH1TfbPwyIv4YESd34/yS1Oo5YMvofD2jVe5ZZbv2Hvtcm2RVV+7jtfflrtwTu3SszFwKLKLzPuDENn3ANqX91sD8Ngn8zvqAJ9eQpOtI2+tdSvXv0JU+4CQggHvLlIvPdeP8klQPr1El6GutT5UIadXe++qtaYw1nattX7B1m77ga1SzBij71bbvqC+Aqj94vBvxbg08Vfq82vOssS8oiakLqGaqPBsRl0TEJt2IQX2UySY1U9uRLk8BH83MzWp+BpZRSYdQTUP7MNUwzBFln+jgWGvrKeDHbc791sw8qxvX0ZEV32BHtX7RMKB26lrtcTqLZwGweVTrLrUa3sE5nwK2iIjVpkh0Ie4/U3VyrTFHuYb5a9iPMgLtxMz8O+DjwAkRscea9pOkDtxFNbLnE520WeWeRXVf/HMHbdvq6H5YW97te2JR2wdsRDUFrrM+4Mw2fcBbMvNKqj5gaDl/q876gOEdJOnWtg94K9UIs670AU9n5hcyc2uqqYDfjzZPg5KkHvInVn5uaDWSVRMz7b2v7mr/UasrnwnWdK62fcHcNn3BxpnZOvNiATV9Cx33Ba3HelsX4mvrz8A2seraq8PpYt+Xmedl5k7AtlTT6b7SjRjUR5lsUm9yMXBmlEVTI2JwRLQ+ZWhjqg8az1GtP/RvbfZ9hmo9je76T2B8ROxVRlENLIvjDevCvguppgSu6fw7RcSnypv+46mu5+61jSczn6SaUveNiNggqsd+j2/vIGWo6s+p3uhvHhHrR0Tr3OxngEGdTEW8GvhYROxR5mufWGK+cw3X2bqw7dvLh6Hnqb5Ven0Nu0lSuzLzeeBfqdZZ+kREvKXczz4aEa3rYlwJnFr6ji1L+64+2rorfUi374nFPhGxW1lz6XTg7jLtoT0/AI6MiLFlgdW3RsTHImJjqsTbcuDY8jf4FNV0ufbcS/Vh5KxyjIER8b6aax7WyZpTVwKHR8ToqBZa/zeqqehPrOlCI+KAmv5zMdWHJ/sASc1wFVXfMCyqhxF9mOp98zVt2rW+r34/1Xp0P+3Gubr6eaSr57oXeDGqh1W8uXwm+IeIaF0I/GrglPIefxjVekod+SFwekSMKv3K9hExqAtx30M1Wumk0ueMo/r7tV3DbzURsXPpx9YH/gq8jH1Bv2KySb3J94BpVFOvXqRKxIwtdZdTfQMxH3iY1ZM0k6kWVV0SEdev7YnLG/79qIamLqTK/n+FLvw/kpkvUS2E95ty/l06aHoD1Zofi6nWEflUWfejO/EcQvW3WUS1OODlnYT4Waqhwn8AnqVKdJGZf6D6MPHHEvcqw4Uz81GqtaPOB/5C1bGMz8xXOzlXq1HALVRrotwFfD8zb+vCfpLUrsw8h2qB01NZeV88mmqxbKgeqDATeBD4PdWDJM5Y/UjtWmMf8gbviVA92OE0qvv2TuVY7crMmcAXqKYfLKaalnxYqXsV+FR5vYiqX7mug+O8VuJ8O9W3+/NKe4BbqR7Z/XRE/KWdfW8B/gW4liph9TZWrhOyJjsD90TEUqp+/biy1ogk9bRvUn0pcAfV/fTbwGcy86GaNk+Xuj9TrTl0ZHmfvLb+nSqxtSQivtxBmy6fq9zD96VaA2kuVd/zQ6pZHlCtTfhkqfsl1YMjOvIdquTUL4EXqPq9N5e6FuCyEvcqa1WVPmc81RqwfwG+Dxzaxb/PJlRfniwucT5HtdSG+olI1+yVGi4iWoC3Z2aHHy4kSeumiLgUmJeZp66prSSp55SROv+ZmV2ZzdBnziX1Bo5skiRJkiRJUt2YbJIkSZIkSVLdOI1OkiRJkiRJdePIJkmSJEmSJNXNes0OoBG23HLLHDFiRLPDkKRe6f777/9LZg5udhzNZD8hSR2zn7CfkKSOdLWPWCeTTSNGjGDmzJnNDkOSeqWIeLLZMTSb/YQkdcx+wn5CkjrS1T7CaXSSJEmSJEmqG5NNkiRJkiRJqhuTTZIkSZIkSaqbdXLNJkn9w7Jly5g3bx4vv/xys0PplQYOHMiwYcNYf/31mx2KJHWJ9/WeZT8hSV3TH/unN9pHmGyS1GfNmzePjTfemBEjRhARzQ6nV8lMnnvuOebNm8fIkSObHY4kdYn39Z5jPyFJXdff+qd69BFOo5PUZ7388ssMGjSoX9zw11ZEMGjQoH717Yukvs/7es+xn5Ckrutv/VM9+giTTZL6tP5yw+8O/zaS+iLvXT3Hv7UkdV1/u2e+0es12SRJkiRJkqS6cc0mSeuM6dPre7zx4zuvf+6559hjjz0AePrppxkwYACDBw8G4N5772WDDTZY63POmDGDDTbYgF133XWt9hsxYgQzZ85kyy23XOtzSlJvNXduS12PN3Lkmo8XEZxwwgmcc845AJx99tksXbqUlpaO973++ut5xzvewbbbbrtWdd3R3eNttNFGLF26tC4xSFJ/1zKjpb7HG7fm4z3xxBPsu+++PPTQQyv3a2lho4024stf/nK7+1x66aXMnDmTCy64gIsvvpi3vOUtHHroofzhD39gwoQJRATXXHMN99xzD4cccki9LgdwZJMkddugQYOYNWsWs2bN4sgjj+RLX/rSitcbbLABy5cvX+tjzpgxgzvvvLMB0UqSumLDDTfkuuuu4y9/+UuX97n++ut5+OGH17quO/1EZ8eTJKkjRx55JIceeihQ9SX7778/v/3tb3nqqaf4yU9+UvfzmWySpDo67LDDOPLIIxk7diwnnXQSjz/+OHvvvTc77bQT73//+/nDH/4AwPTp0xk7diw77rgjH/7wh3nmmWd44oknuPjiizn33HMZPXo0v/71r1m4cCGf/vSn2Xnnndl55535zW9+A1Sjqvbcc0+22247Pv/5z5OZzbxsSVpnrLfeekyaNIlzzz13tbonnniC3Xffne2335499tiDP/3pT9x5551MmzaNr3zlK4wePZrHH398Rfv26saNG8fxxx/PmDFj+N73vsf999/PBz/4QXbaaSf22msvFixYAMAPfvADdt55Z3bYYQc+/elP89JLL7V7vI76mblz5/Le976Xf/zHf+TUU0/tmT+eJKkpxo0bx3HHHcfo0aP5h3/4B+69997V2rS0tHD22Wdz00038d3vfpeLLrqID33oQ5x88sn8+te/ZvTo0e32fd3lNDpJqrN58+Zx5513MmDAAPbYYw8uvvhiRo0axT333MM///M/c+utt7Lbbrtx9913ExH88Ic/5Nvf/jbnnHMORx555CpDYQ855BC+9KUvsdtuu/GnP/2Jvfbai0ceeYRvfOMb7Lbbbvzrv/4rP/vZz5g8eXKTr1qS1h1HHXUU22+/PSeddNIq5ccccwwTJ05k4sSJTJkyhWOPPZbrr7+ej3/84+y7777sv//+q7Tfdddd26179dVXmTlzJsuWLeODH/wgN9xwA4MHD+aqq67i61//OlOmTOFTn/oUX/jCFwA49dRTmTx5Msccc8xqx+uonznuuOP44he/yKGHHsqFF17Y4L+YJKnZXnrpJWbNmsXtt9/O5z73uVWm29XaZ599VvnMMWPGDM4++2xuvPHGusZjskmS6uyAAw5gwIABLF26lDvvvJMDDjhgRd0rr7wCVAmpgw46iAULFvDqq68ycuTIdo91yy23rDJd4oUXXmDp0qXcfvvtXHfddQB87GMfY/PNN2/gFUlS/7LJJptw6KGHct555/HmN795Rfldd9214t772c9+drVkVFcddNBBADz66KM89NBDfOQjHwHgtddeY8iQIQA89NBDnHrqqSxZsoSlS5ey1157rXaczvqZ3/zmN1x77bUrYv3qV7/arVglSb1DR0+Hay0/+OCDAfjABz7ACy+8wJIlS3ostvaYbJKkOnvrW98KwOuvv85mm23GrFmzVmtzzDHHcMIJJ/Dxj3+cGTNmdLjw7Ouvv87dd9/NwIEDGxmyJKmN448/nne/+90cfvjhdT92az+RmWy33Xbcddddq7U57LDDuP7669lhhx249NJLmTFjxmptOutnoP89pluS1mWDBg1i8eLFq5QtWrRoxZfWbe/5ze4DXLNJkhpkk002YeTIkfz0pz8Fqg8Vv/vd7wB4/vnnGTp0KACXXXbZin023nhjXnzxxRWv99xzT84///wVr1s/UHzgAx9YsZDfz3/+89U6HknSG7PFFltw4IEHrjJNedddd2Xq1KkAXHHFFbz//e8HVr931+qs7p3vfCcLFy5ckWxatmwZs2fPBuDFF19kyJAhLFu2jCuuuKLd43XWz7zvfe9bJVZJUt+20UYbMWTIEG699VagSjT993//N7vtthsAV111FQB33HEHm266KZtuummXjttZP/VGOLJJ0jpj/PhmR7C6K664gi9+8YucccYZLFu2jAkTJrDDDjvQ0tLCAQccwOabb87uu+/O3LlzARg/fjz7778/N9xwA+effz7nnXfeirVDli9fzgc+8AEuvvhiTjvtNA4++GC22247dt11V4YPH97kK5Wk+hs5sqWp5z/xxBO54IILVrw+//zzOfzww/mP//gPBg8ezI9+9CMAJkyYwBe+8AXOO+88rrnmGt72tret2KdtXa0NNtiAa665hmOPPZbnn3+e5cuXc/zxx7Pddttx+umnM3bsWAYPHszYsWNXfBBoe7yO+pnvfe97HHLIIXzrW99iv/3264G/liT1Hy3jWppy3ssvv5yjjjqKE044AYDTTjttRZ8zcOBAdtxxR5YtW8aUKVO6fMztt9+eAQMGsMMOO3DYYYfxpS99qS6xxrr4BKMxY8bkzJkzmx2G1FTTp6/c7o1JmHp45JFH+Pu///tmh9Grtfc3ioj7M3NMk0LqFewntK6bO7dlxXazEzZrw/t6z7OfaJ/9hHqrlhktK7eblPDoj3p7/zRu3DjOPvtsxoyp7637jfQRTqOTJEmSJElS3TiNTpIkSZIkqY9q7yESzWaySVqH1E6d6y8ys+lPWuit1sVp0pLWfd7Xe479hCR1XX/rn95oH+E0Okl91sCBA3nuued8s9yOzOS5555j4MCBzQ5FkrrM+3rPsZ+QpK7rb/1TPfoIRzZJ6rOGDRvGvHnzWLhwYbND6ZUGDhzIsGHDmh2GJHWZ9/WeZT8hSV3TH/unN9pHmGyS1Getv/76jBw5stlhSJLqxPu6JKk3sn9aeyabJEmS1gFz57Y0OwRJkiTANZskSZIkSZJURyabJEmSJEmSVDcmmyRJkiRJklQ3JpskSZIkSZJUNyabJEmSJEmSVDcmmyRJkiRJklQ3JpskSZIkSZJUN+s1OwBJjTd9+srt8eObF4ckSZIkad3nyCZJkiRJkiTVjSObpD6udtSSJEmSJEnN5sgmSZIkSZIk1Y3JJkmSJEmSJNWNySZJkiRJkiTVjckmSZIkSZIk1Y3JJkmSJEmSJNWNySZJkiRJkiTVjckmSZIkSZIk1Y3JJkmSJEmSJNWNySZJkiRJkiTVjckmSZIkSZIk1Y3JJklSU0TEwIi4NyJ+FxGzI+IbpXxkRNwTEXMi4qqI2KCUb1hezyn1I2qOdUopfzQi9mrOFUmSJEkCWK/ZAUhae9OnNzsCqS5eAXbPzKURsT5wR0T8HDgBODczp0bExcARwEXl9+LMfHtETAC+BRwUEdsCE4DtgK2BWyLiHZn5WjMuSpIkServTDZJkpoiMxNYWl6uX34S2B04pJRfBrRQJZv2K9sA1wAXRESU8qmZ+QowNyLmAO8B7mr8VUiSJDVHy4yWldvjWjpsJzWD0+gkSU0TEQMiYhbwLHAz8DiwJDOXlybzgKFleyjwFECpfx4YVFvezj6155oUETMjYubChQsbcTmSJEmSMNkkSWqizHwtM0cDw6hGI72rgee6JDPHZOaYwYMHN+o0kiRJUr9nskmS1HSZuQS4DXgvsFlEtE7zHgbML9vzgW0ASv2mwHO15e3sI0mSJKmHmWySJDVFRAyOiM3K9puBjwCPUCWd9i/NJgI3lO1p5TWl/tay7tM0YEJ5Wt1IYBRwb89chSRJkqS2XCBc6sVqnzo3fnzz4pAaZAhwWUQMoPry4+rMvDEiHgamRsQZwG+ByaX9ZODHZQHwRVRPoCMzZ0fE1cDDwHLgKJ9EJ0mSJDWPySZJUlNk5oPAju2U/5Fq/aa25S8DB3RwrDOBM+sdoyRJkqS15zQ6SZIkSZIk1Y3JJkmSJEmSJNVNw5JNEbFNRNwWEQ9HxOyIOK6Ut0TE/IiYVX72qdnnlIiYExGPRsReNeV7l7I5EXFyo2KWJEmSJEnSG9PINZuWAydm5gMRsTFwf0TcXOrOzcyzaxtHxLZUi71uB2wN3BIR7yjVF1I9pWgecF9ETMvMhxsYu9QvuAC5JEmSJKneGpZsyswFwIKy/WJEPAIM7WSX/YCpmfkKMLc8bah1gdg5ZcFYImJqaWuySZIkSZIkqZfpkTWbImIE1ROH7ilFR0fEgxExJSI2L2VDgadqdptXyjoqb3uOSRExMyJmLly4sM5XIEmSJEmSpK5oeLIpIjYCrgWOz8wXgIuAtwGjqUY+nVOP82TmJZk5JjPHDB48uB6HlCRJkiRJ0lpq5JpNRMT6VImmKzLzOoDMfKam/gfAjeXlfGCbmt2HlTI6KZckSVI3zJ3bsmJ75MiWDttJkiStrUY+jS6AycAjmfmdmvIhNc0+CTxUtqcBEyJiw4gYCYwC7gXuA0ZFxMiI2IBqEfFpjYpbkiRJkiRJ3dfIkU3vAz4L/D4iZpWyrwEHR8RoIIEngH8CyMzZEXE11cLfy4GjMvM1gIg4GvgFMACYkpmzGxi3JEmSJEmSuqmRT6O7A4h2qm7qZJ8zgTPbKb+ps/0kSZIkSZLUO/TI0+gkSZIkSZLUPzR0gXBJ9TN9erMjkCRJ6nkRMQCYCczPzH3L+q5TgUHA/cBnM/PViNgQuBzYCXgOOCgznyjHOAU4AngNODYzf9HzVyJJ/YcjmyRJkiT1ZscBj9S8/hZwbma+HVhMlUSi/F5cys8t7YiIbakeMrQdsDfw/ZLAkiQ1iMkmSZIkSb1SRAwDPgb8sLwOYHfgmtLkMuATZXu/8ppSv0dpvx8wNTNfycy5wBzgPT1zBZLUPzmNTpIkSVJv9V3gJGDj8noQsCQzl5fX84ChZXso8BRAZi6PiOdL+6HA3TXHrN1nhYiYBEwCGD58eH2vQnoDWma0NDsEaa05skmSJElSrxMR+wLPZub9PXG+zLwkM8dk5pjBgwf3xCklaZ3lyCZJkiRJvdH7gI9HxD7AQGAT4HvAZhGxXhndNAyYX9rPB7YB5kXEesCmVAuFt5a3qt1HktQAjmySJEmS1Otk5imZOSwzR1At8H1rZn4GuA3YvzSbCNxQtqeV15T6WzMzS/mEiNiwPMluFHBvD12GJPVLjmySJEmS1Jd8FZgaEWcAvwUml/LJwI8jYg6wiCpBRWbOjoirgYeB5cBRmflaz4ctSf2HySZJkiRJvVpmzgBmlO0/0s7T5DLzZeCADvY/EzizcRFKkmo5jU6SJEmSJEl1Y7JJkiRJkiRJdWOySZIkSZIkSXXjmk2SJEmSJPVhLTNaVm6Pa+mwndRTTDZJ/cz06c2OQJLU28yd27Jie+TIlg7bSZIkdYXT6CRJkiRJklQ3JpskSZIkSZJUNyabJEmSJEmSVDcmmyRJkiRJklQ3JpskSZIkSZJUNz6NTtJqap9YN3588+KQJEmSJPU9JpskSZL6iblzW5odgiRJ6gecRidJkiRJkqS6cWST1AvVTmOTJKkn1Y5+GjmypcN2kiRJHXFkkyRJkiRJkurGZJMkSZIkSZLqxmSTJEmSJEmS6sY1myRJkiRJ6kVaZrQ0OwTpDXFkkyRJkiRJkurGZJMkSZIkSZLqxmSTJEmSJEmS6sZkkyRJkiRJkurGZJMkSZIkSZLqxmSTJEmSJEmS6ma9ZgcgSZIkSZLqo2VGy8rtcS0dtpMayZFNkqSmiIhtIuK2iHg4ImZHxHGlvCUi5kfErPKzT80+p0TEnIh4NCL2qinfu5TNiYiTm3E9kiRJkiqObJIkNcty4MTMfCAiNgbuj4ibS925mXl2beOI2BaYAGwHbA3cEhHvKNUXAh8B5gH3RcS0zHy4R65CkiRJ0ipMNkmSmiIzFwALyvaLEfEIMLSTXfYDpmbmK8DciPpo3HUAACAASURBVJgDvKfUzcnMPwJExNTS1mSTJEmS1AROo5MkNV1EjAB2BO4pRUdHxIMRMSUiNi9lQ4GnanabV8o6Km97jkkRMTMiZi5cuLDOVyBJkiSplSObJElNFREbAdcCx2fmCxFxEXA6kOX3OcDn3uh5MvMS4BKAMWPG5Bs9ntRXzJ3b0uwQJElSP2OySZLUNBGxPlWi6YrMvA4gM5+pqf8BcGN5OR/Ypmb3YaWMTsolSZIk9TCn0UmSmiIiApgMPJKZ36kpH1LT7JPAQ2V7GjAhIjaMiJHAKOBe4D5gVESMjIgNqBYRn9YT1yBJkiRpdY5skiQ1y/uAzwK/j4hZpexrwMERMZpqGt0TwD8BZObsiLiaauHv5cBRmfkaQEQcDfwCGABMyczZPXkhkiRJklYy2SRJaorMvAOIdqpu6mSfM4Ez2ym/qbP9JEmSJPUck02SAJg+vdkRSJIkSZLWBSabJEmS1K7aJ9mNHNnSYTtJkqRaJpskSZK0VkxCSZKkzvg0OkmSJEmSJNWNySZJkiRJkiTVjckmSZIkSZIk1Y1rNkmSJGmNatdpkiT1DS0zWlZuj2vpsJ1Ub45skiRJkiRJUt04sklSp6ZPX7k9fnzz4pAkSZIk9Q2ObJIkSZIkSVLdOLJJkiSpj3IdJUmS1Bs1bGRTRGwTEbdFxMMRMTsijivlW0TEzRHxWPm9eSmPiDgvIuZExIMR8e6aY00s7R+LiImNilmSJEmSJElvTCOn0S0HTszMbYFdgKMiYlvgZOBXmTkK+FV5DfBRYFT5mQRcBFVyCjgNGAu8BzitNUElSZIkSZKk3qVhyabMXJCZD5TtF4FHgKHAfsBlpdllwCfK9n7A5Vm5G9gsIoYAewE3Z+aizFwM3Azs3ai4JUmSJEmS1H09skB4RIwAdgTuAbbKzAWl6mlgq7I9FHiqZrd5payj8rbnmBQRMyNi5sKFC+savyRJkiRJkrqm4cmmiNgIuBY4PjNfqK3LzASyHufJzEsyc0xmjhk8eHA9DilJkiRJkqS11NBkU0SsT5VouiIzryvFz5TpcZTfz5by+cA2NbsPK2UdlUuSJEmSJKmXWa9RB46IACYDj2Tmd2qqpgETgbPK7xtqyo+OiKlUi4E/n5kLIuIXwL/VLAq+J3BKo+KWJElS182d27Jie+TIlg7bSZI61zKjpdkhSHXTsGQT8D7gs8DvI2JWKfsaVZLp6og4AngSOLDU3QTsA8wBXgIOB8jMRRFxOnBfaffNzFzUwLglSZIkSZLUTQ1LNmXmHUB0UL1HO+0TOKqDY00BptQvOkmSJEmSJDVCjzyNTpIkSZIkSf2DySZJkiRJkiTVjckmSZIkSZIk1Y3JJkmSJEmSJNWNySZJkiRJkiTVjckmSZIkSZIk1Y3JJkmSJEmSJNWNySZJkiRJkiTVjckmSZIkSZIk1Y3JJkmSJEmSJNWNySZJkiRJkiTVjckmSZIkSZIk1Y3JJkmSJEmSJNWNySZJkiRJkiTVjckmSZIkSZIk1Y3JJkmSJEm9TkQMjIh7I+J3ETE7Ir5RykdGxD0RMSciroqIDUr5huX1nFI/ouZYp5TyRyNir+ZckST1HyabJEmSJPVGrwC7Z+YOwGhg74jYBfgWcG5mvh1YDBxR2h8BLC7l55Z2RMS2wARgO2Bv4PsRMaBHr0SS+hmTTZK6bPr0lT+SJEmNlJWl5eX65SeB3YFrSvllwCfK9n7lNaV+j4iIUj41M1/JzLnAHOA9PXAJktRvmWySJEmS1CtFxICImAU8C9wMPA4syczlpck8YGjZHgo8BVDqnwcG1Za3s0/tuSZFxMyImLlw4cJGXI4k9RsmmyRJkiT1Spn5WmaOBoZRjUZ6VwPPdUlmjsnMMYMHD27UaSSpXzDZJEmSJKlXy8wlwG3Ae4HNImK9UjUMmF+25wPbAJT6TYHnasvb2UeS1AAmmyRJkiT1OhExOCI2K9tvBj4CPEKVdNq/NJsI3FC2p5XXlPpbMzNL+YTytLqRwCjg3p65Cknqn9ZbcxNJkiRJ6nFDgMvKk+PeBFydmTdGxMPA1Ig4A/gtMLm0nwz8OCLmAIuonkBHZs6OiKuBh4HlwFGZ+VoPX4sk9SsmmyRJkiT1Opn5ILBjO+V/pJ2nyWXmy8ABHRzrTODMescoSWqf0+gkSZIkSZJUN2sc2RQRG2bmK2sqk9S/TJ++cnv8+ObFIUmSJEnqXboysumuLpZJktRlEbFNRNwWEQ9HxOyIOK6UbxERN0fEY+X35qU8IuK8iJgTEQ9GxLtrjjWxtH8sIiZ2dE5JkiRJjdfhyKaI+BtgKPDmiNgRiFK1CfCWHohNkrRuWw6cmJkPRMTGwP0RcTNwGPCrzDwrIk4GTga+CnyU6glCo4CxwEXA2IjYAjgNGANkOc60zFzc41ckSZIkqdNpdHtRveEfBpzDymTTi8DXGhuWJGldl5kLgAVl+8WIeITqS479gHGl2WXADKpk037A5eUx1ndHxGYRMaS0vTkzFwGUhNXewJU9djGSJEmSVugw2ZSZl1E9avTTmXltD8YkSepnImIE1ROH7gG2KokogKeBrcr2UOCpmt3mlbKOytueYxIwCWD48OH1C16SJEnSKrqyZtOwiNikrJXxw4h4ICL2bHhkkqR+ISI2Aq4Fjs/MF2rryiimrMd5MvOSzByTmWMGDx5cj0NKkiRJakdXkk2fK2/+9wQGAZ8FzmpoVJKkfiEi1qdKNF2RmdeV4mfK9DjK72dL+Xxgm5rdh5Wyjsol9bC5c1tW/EiSpP6rK8mm1rWa9qFaK2N2TZkkSd0SEQFMBh7JzO/UVE0DWp8oNxG4oab80DLSdhfg+TLd7hfAnhGxeXly3Z6lTJIkSVITdLZAeKv7I+KXwEjglPLEoNcbG5YkqR94H9Vo2d9HxKxS9jWq0bNXR8QRwJPAgaXuJqovPuYALwGHA2Tmoog4HbivtPtm62LhkiRJknpeV5JNRwCjgT9m5ksRMYjyBl+SpO7KzDvoeKTsHu20T+CoDo41BZhSv+gkSZIkdVeHyaaIeFdm/oEq0QTwd9WMB0mSJEmSJKl9nY1sOoHqEdHntFOXwO4NiUiSJEmSJEl9VofJpsycVH5/qOfCkSRJkiRJUl/WlTWbiIhdgRG17TPz8gbFJEmSJEmSpD5qjcmmiPgx8DZgFvBaKU7AZJMkSZIkSZJW0ZWRTWOAbctTgCRJkiRJkqQOvakLbR4C/qbRgUiSJEmSJKnv63BkU0RMp5outzHwcETcC7zSWp+ZH298eFL/MX16syOQJEmSJOmN62wa3dk9FoUkSZIkSZLWCR0mmzLzf3oyEEmSJEmSJPV9XVmzSZIkSZIkSeqSrjyNTpIkSZIk1VnLjJZmhyA1xBpHNkXE+IhwBJQkSZIkSZLWqCtJpIOAxyLi2xHxrkYHJEmSJEmSpL5rjcmmzPzfwI7A48ClEXFXREyKiI0bHp0kSZIkSZL6lC5Nj8vMF4BrgKnAEOCTwAMRcUwDY5MkSZIkSVIf05U1m/aLiP8CZgDrA+/JzI8COwAnNjY8SZIkSZIk9SVdeRrdp4BzM/P22sLMfCkijmhMWJIkSZIkSeqLujKN7um2iaaI+BZAZv6qIVFJkiRJkiSpT+pKsukj7ZR9tN6BSJIkSZIkqe/rMNkUEV+MiN8D74qIB2t+5gIPrunAETElIp6NiIdqyloiYn5EzCo/+9TUnRIRcyLi0YjYq6Z871I2JyJO7v6lSpIkSZIkqdE6W7PpJ8DPgX8HapM8L2bmoi4c+1LgAuDyNuXnZubZtQURsS0wAdgO2Bq4JSLeUaovpBpdNQ+4LyKmZebDXTi/JEmSJEmSelhnyabMzCci4qi2FRGxxZoSTpl5e0SM6GIc+wFTM/MVYG5EzAHeU+rmZOYfy3mnlrYmmyRJkiRJknqhNY1s2he4H0ggauoS+LtunvPoiDgUmAmcmJmLgaHA3TVt5pUygKfalI9t76ARMQmYBDB8+PBuhiapO6ZPX7k9fnzz4pAkSZIkNV+HazZl5r7l98jM/Lvyu/Wnu4mmi4C3AaOBBcA53TzOajLzkswck5ljBg8eXK/DSpIkSZIkaS10OLIpIt7d2Y6Z+cDaniwzn6k5/g+AG8vL+cA2NU2HlTI6KZckSZIkSVIv09k0us5GHSWw+9qeLCKGZOaC8vKTQOuT6qYBP4mI71AtED4KuJdq6t6oiBhJlWSaAByytueVJEmSJElSz+gw2ZSZH3ojB46IK4FxwJYRMQ84DRgXEaOpklVPAP9UzjU7Iq6mWvh7OXBUZr5WjnM08AtgADAlM2e/kbgkSZIkSZLUOJ1No9s9M2+NiE+1V5+Z13V24Mw8uJ3iyZ20PxM4s53ym4CbOjuXJEmSJEmSeofOptF9ELgVaO/ZUgl0mmySJEmSJElS/9PZNLrTyu/Dey4cSZIkSZIk9WVvWlODiBgUEedFxAMRcX9EfC8iBvVEcJIkSZIkSepb1phsAqYCC4FPA/uX7asaGZQkSZIkSZL6ps7WbGo1JDNPr3l9RkQc1KiAJEmSJElaV7XMaGn6eVvGNScG9R9dGdn0y4iYEBFvKj8HAr9odGCSJEmSJEnqezoc2RQRL1I9dS6A44H/LFVvApYCX254dJIkSZIkSepTOnsa3cY9GYgkSZIkSZL6vq6s2UREbA6MAga2lmXm7Y0KSpIkSZIkSX3TGpNNEfF54DhgGDAL2AW4C9i9saFJkiRJkiSpr+nKAuHHATsDT2bmh4AdgSUNjUqSJEmSJEl9UleSTS9n5ssAEbFhZv4BeGdjw5IkSZIkSVJf1JU1m+ZFxGbA9cDNEbEYeLKxYUmSJEmSJKkvWmOyKTM/WTZbIuI2YFPgvxsalSRJkiRJkvqkrj6N7t3AbkACv8nMVxsalSRJkiRJkvqkNa7ZFBH/ClwGDAK2BH4UEac2OjBJkiRJkiT1PV0Z2fQZYIeaRcLPAmYBZzQyMEmSJEmSJPU9XXka3Z+BgTWvNwTmNyYcSZIkSZIk9WUdJpsi4vyIOA94HpgdEZdGxI+Ah4AlPRWgJGndFBFTIuLZiHiopqwlIuZHxKzys09N3SkRMSciHo2IvWrK9y5lcyLi5J6+DkmSJEmr6mwa3czy+37gv2rKZzQsGklSf3IpcAFweZvyczPz7NqCiNgWmABsB2wN3BIR7yjVFwIfAeYB90XEtMx8uJGBS+q+uXNbVmyPHNnSYTtJktR3dZhsyszLWrcjYgOg9U39o5m5rNGBSZLWbZl5e0SM6GLz/YCpmfkKMDci5gDvKXVzMvOPABExtbQ12SRJkiQ1yRoXCI+IcVRPo3sCCGCbiJiYmbc3NjRp3TR9+srt8eObF4fUix0dEYdSjbA9MTMXA0OBu2vazCtlAE+1KR/b3kEjYhIwCWD48OH1jlmSJElS0ZWn0Z0D7JmZjwKUaQtXAjs1MjCpP6hNPEkC4CLgdCDL73OAz9XjwJl5CXAJwJgxY7Iex5QkSeqKlhktzQ5hFbXxtIxr6bCd1F1dSTat35poAsjM/xsR6zcwJklSP5WZz7RuR8QPgBvLy/nANjVNh7HyyagdlUuSJElqgq4km+6PiB8C/1lef4aVi4dLklQ3ETEkMxeUl5+kegIqwDTgJxHxHaoFwkcB91JN7x4VESOpkkwTgEN6NmpJ3eVi4ZIkrZu6kmw6EjgKOLa8/jXw/YZFJEnqFyLiSmAcsGVEzANOA8ZFxGiqaXRPAP8EkJmzI+JqqoW/lwNHZeZr5ThHA78ABgBTMnN2D1+KJEmSpBqdJpsiYgDwu8x8F/CdnglJktQfZObB7RRP7qT9mcCZ7ZTfBNxUx9Ak1YGjliRJ6r/e1Fll+db40YjwsT2SJEmSJElao06TTcXmwOyI+FVETGv9aXRgkiRJkvqviNgmIm6LiIcjYnZEHFfKt4iImyPisfJ781IeEXFeRMyJiAcj4t01x5pY2j8WERObdU2S1F90Zc2mf2l4FJIkSZK0quXAiZn5QERsTPXgopuBw4BfZeZZEXEycDLwVeCjVA+QGAWMBS4CxkbEFlTrAo6hWhPw/oiYlpmLe/yKJKmf6DDZFBEDqRYHfzvwe2ByZi7vqcAkSZIk9V/l6aQLyvaLEfEIMBTYj+oBEwCXATOokk37AZdnZgJ3R8RmETGktL05MxcBlITV3sCVPXYxktTPdDay6TJgGdXT5z4KbAsc1xNBSZIkad1Ru1i41B0RMQLYEbgH2KokogCeBrYq20OBp2p2m1fKOipve45JwCSA4cNdslaS3ojOkk3bZuY/AkTEZODenglJkiRJkioRsRFwLXB8Zr4QESvqMjMjIutxnsy8BLgEYMyYMXU5piT1V50lm5a1bmTm8tqbuiR1ZPr0ldvjxzcvDkmS1PdFxPpUiaYrMvO6UvxMRAzJzAVlmtyzpXw+sE3N7sNK2XxWTrtrLZ/RyLglqb/r7Gl0O0TEC+XnRWD71u2IeKGnApQkSZLU/0T1bfdk4JHM/E5N1TSg9YlyE4EbasoPLU+l2wV4vky3+wWwZ0RsXp5ct2cpkyQ1SIcjmzJzQE8GIkmSpDVz/SP1I+8DPgv8PiJmlbKvAWcBV0fEEcCTwIGl7iZgH2AO8BJwOEBmLoqI04H7Srtvti4WLklqjM6m0UmSJElSU2TmHUBHa3ns0U77BI7q4FhTgCn1i06S1JnOptFJkiRJkiRJa8VkkyRJkiRJkurGZJMkSZIkSZLqxmSTJEmSJEmS6sZkkyRJkiRJkurGZJMkSZIkSZLqxmSTJEmSJEmS6sZkkyRJkiRJkurGZJMkSZIkSZLqxmSTJEmSJEmS6sZkkyRJkiRJkurGZJMkSZIkSZLqxmSTJEmSJEmS6sZkkyRJkiRJkurGZJMkSZIkSZLqxmSTJEmSJEmS6sZkkyRJkiRJkurGZJMkSZIkSZLqpmHJpoiYEhHPRsRDNWVbRMTNEfFY+b15KY+IOC8i5kTEgxHx7pp9Jpb2j0XExEbFK0mSJEmSpDeukSObLgX2blN2MvCrzBwF/Kq8BvgoMKr8TAIugio5BZwGjAXeA5zWmqCSJEmSJElS79OwZFNm3g4salO8H3BZ2b4M+ERN+eVZuRvYLCKGAHsBN2fmosxcDNzM6gksSZIkSZIk9RI9vWbTVpm5oGw/DWxVtocCT9W0m1fKOipfTURMioiZETFz4cKF9Y1akiRJkiRJXbJes06cmRkRWcfjXQJcAjBmzJi6HVdS902fvnJ7/PjmxSFJkiRJ6jk9nWx6JiKGZOaCMk3u2VI+H9impt2wUjYfGNemfEYPxClJkiRJ0jqvZUbLyu1xLR22k9ZGT0+jmwa0PlFuInBDTfmh5al0uwDPl+l2vwD2jIjNy8Lge5YySZIkSZIk9UING9kUEVdSjUraMiLmUT1V7izg6og4AngSOLA0vwnYB5gDvAQcDpCZiyLidOC+0u6bmdl20XFJkiRJkiT1Eg1LNmXmwR1U7dFO2wSO6uA4U4ApdQxNkiRJkiRJDdLT0+gkSZIkSZK0DjPZJEmSJEmSpLox2SRJkiRJkqS6adiaTZIkSZIk9VctM1qaHYLUNI5skiRJkiRJUt38//buP1bSq7wP+PdxLIJKobZjs3VtgzfVopRWCUW3hj+isNSpMa4W04BcLCEWy+2K1LR/tJXiikrcwB91WqkVqIjGRRY2EhiKRNlNKMRssGgjnHoDxhia2AabejfGu8XEpXV/BHr6x7zrHdY7s3P3zsz7ztzPRxrNO2fO3Pu88+u888xzzqhsAgAYiMce23xue/fuzYn9AACGTLIJAIDeSbQBwPowjQ4AAACAuZFsAgAAAGBuJJsA6EVV3VFVx6vqobG2i6rqnqp6pDu/sGuvqvpgVT1aVQ9W1avHbrO/6/9IVe3vY18AAIBTJJsA6MtHk1x7WtutSQ631vYkOdxdTpI3JtnTnQ4k+XAySk4leW+S1yS5Ksl7TyaoAACAfkg2AUtx6NCpEyRJa+3LSZ4+rfn6JHd223cmefNY+11t5L4kF1TVpUnekOSe1trTrbUfJLknz09gAQAASyTZBMCQ7GqtPdltfy/Jrm77siRPjPU72rVNan+eqjpQVUeq6siJEyfmGzUAAPAcySYABqm11pK0Of6921trG621jUsuuWRefxYAADjN+X0HAABjnqqqS1trT3bT5I537ceSXDHW7/Ku7ViSvae137uEOAEA1s7mvZuntvduTuwHZ6OyCYAhOZjk5C/K7U/y2bH2d3S/SvfaJM900+2+kOSaqrqwWxj8mq4NAADoicomAHpRVZ/IqCrp4qo6mtGvyt2W5FNVdXOS7ya5oev+uSTXJXk0ybNJbkqS1trTVfX+JPd3/d7XWjt90XEAAGCJJJsA6EVr7cYJV119hr4tyS0T/s4dSe6YY2gAAMA2mEYHAAAAwNyobIIlOHSo7wgAAABgOSSbAAAYlMce23xue/fuzYn9AIBhkmwCAGCwJJ4AYPVYswkAAACAuVHZBCzd+BpW+/b1FwcAAADzp7IJAAAAgLlR2QQAMEDWKgIAVpXKJgAAAADmRrIJAAAAgLmRbAIAAABgbiSbAAAAAJgbySYAAAAA5kayCQAAAIC5Ob/vAAAAAIBh2bx389T23s2J/eBMVDYBAAAAMDeSTQAAAADMjWl0QK8OHTq1vW9ff3EAAAAwHyqbAAAAAJgbySYAAAAA5sY0OgAAAJiD8V9wg51MZRMAAAAAcyPZBAAAAMDcSDYBAAAAMDeSTQAAwOBU1R1VdbyqHhpru6iq7qmqR7rzC7v2qqoPVtWjVfVgVb167Db7u/6PVNX+PvYFYKeRbAIAAIboo0muPa3t1iSHW2t7khzuLifJG5Ps6U4Hknw4GSWnkrw3yWuSXJXkvScTVAAsjmQTAAAwOK21Lyd5+rTm65Pc2W3fmeTNY+13tZH7klxQVZcmeUOSe1prT7fWfpDknjw/gQXAnEk2AQAAq2JXa+3Jbvt7SXZ125cleWKs39GubVL781TVgao6UlVHTpw4Md+oAXYYySYAAGDltNZakjbHv3d7a22jtbZxySWXzOvPAuxIkk0AAMCqeKqbHpfu/HjXfizJFWP9Lu/aJrUDsECSTQAAwKo4mOTkL8rtT/LZsfZ3dL9K99okz3TT7b6Q5JqqurBbGPyarg2ABTq/7wAAAGAWjz22+dz27t2bE/uxHqrqE0n2Jrm4qo5m9KtytyX5VFXdnOS7SW7oun8uyXVJHk3ybJKbkqS19nRVvT/J/V2/97XWTl90HIA5k2wCAAAGp7V244Srrj5D35bklgl/544kd8wxNADOQrIJAGDgxit6AACGTrIJAAAAmGjz3s1T23s3J/aDkywQDgAAAMDcqGwCAOiRKXIAwLpR2QQAAADA3PRS2VRVjyf5YZIfJ/lRa22jqi5K8skkVyZ5PMkNrbUfVFUl+UBGP2X6bJJ3tta+2kfcsBWHDvUdAQAAsGjj6xkBI31WNr2+tfaq1tpGd/nWJIdba3uSHO4uJ8kbk+zpTgeSfHjpkQIAAAAwkyGt2XR9kr3d9p1J7k3ya137Xa21luS+qrqgqi5trT3ZS5QAAAzK+LpXu3dvTuwHACxHX8mmluR3qqol+c3W2u1Jdo0lkL6XZFe3fVmSJ8Zue7Rr+4lkU1UdyKjyKS972csWGDqwKONTD/ft6y8OAAAAzl1fyaZfbK0dq6qXJrmnqv5w/MrWWusSUTPrEla3J8nGxsaWbgsAAADAfPSyZlNr7Vh3fjzJZ5JcleSpqro0Sbrz4133Y0muGLv55V0bAAAAAAOz9MqmqnpRkvNaaz/stq9J8r4kB5PsT3Jbd/7Z7iYHk7y7qu5O8pokz1ivCQAAAJZv/Nf3NvduTuzHztbHNLpdST5TVSf//8dba5+vqvuTfKqqbk7y3SQ3dP0/l+S6JI8meTbJTcsPGQAAAIBZLD3Z1Fr7TpJfOEP795NcfYb2luSWJYQGwEBU1eNJfpjkx0l+1FrbqKqLknwyyZVJHk9yQ2vtBzX69uIDGX0x8WySd7bWvtpH3ADAzjBe3QM8Xy9rNgHADF7fWntVa22ju3xrksOttT1JDneXk+SNSfZ0pwNJPrz0SAEAgOdINgGwKq5Pcme3fWeSN4+139VG7ktywckfnAAAAJZPsgmAIWpJfqeq/qCqDnRtu8Z+IOJ7Ga0BmCSXJXli7LZHu7afUFUHqupIVR05ceLEouIGAIAdr48FwunBoUOntvft6y8OgBn9YmvtWFW9NMk9VfWH41e21lpVta38wdba7UluT5KNjY0t3RYAAJidyiYABqe1dqw7P57kM0muSvLUyelx3fnxrvuxJFeM3fzyrg0AAOiByiaYo/EKMuDcVNWLkpzXWvtht31NkvclOZhkf5LbuvPPdjc5mOTdVXV3ktckeWZsuh0AALBkkk0Mlql/sGPtSvKZqkpG49THW2ufr6r7k3yqqm5O8t0kN3T9P5fkuiSPJnk2yU3LDxkAADhJsoneSSoB41pr30nyC2do/36Sq8/Q3pLcsoTQAACAGVizCQAAAIC5Udm0xnba+kEqpAAAAKB/kk2sBIkkAGAWjz22+dz27t2bE/sBAIsj2cQZLTq5s9OqrgCA+RpPKgEAwyLZtANNSvSoGAJgVqpHAACYRLIJGCRTJwEAYNg27908tb13c2I/dh7JJlaOJAQAAAAMl2QTAEDH9EAAgO2TbGJQLBwOAAAAq02yibVn2h2w081SraOiBwCAeZFsAgAAgLMYXwwbmE6yiZW21Wl3qpwAmJVqLwCAcyPZBOdA0goAAADOTLKJs9oJiZXt7KNFzQF2rnOtfhq/HQCsg/Fphpt7Nyf2Y2eQbGLHkiQCAACA+ZNsgtPshEougJ3IGkw7j8ccmAcLg2+dKifO6zsAAAAAANaHyia2ZKdV/ey0/R0qjwMs16T1hCZVhuyE6pGdsI8AAPMiZkmetAAADHxJREFU2cRc7IRkgDWegHUgafJ8FusGAJgvySYAgCWQ1AIAdgrJJp4zr8qdnVDlBMDZ7bQqqp22v6vG4wMAyyPZxFoy5Q0AAAD6IdkEAMxknaaBbWdf1ul+AABYBMkmlka1EcCwSJosnvsYYDVt3rvZdwiw0iSbAAAAgIUYT9xt7t2c2I/1cl7fAQAAAACwPiSbAAAAAJgb0+iAlTK+9te+ff3FAQDAerFO0+KZUrdzSDZxzmZZ8Nui4ADsRBYGXx3jj9Xu3ZsT+wHrQ8IDFk+yCQAAgB1JNRMshjWbAAAAAJgblU0APZm0/pR1qVhlW50+ZgoTAMugggmWS7IJYACsb8a622pSSRKKRZIUhZ1BgmnYrJ213iSbgJU15AqgIcfGepn0IXjIC1QPOTYAALZPsgkA1oQkDgAAQyDZBAAAwFowdQ6GQbIJYMGsxwTLpcKLc2VtJgCYD8kmAAAAVooKpvUyy+NpEfHVItm0ZlRQAAAA60iCCVaHZNMakGACv/4G68zUJvqw1emYnqcAcIpkEwAwCNZaAgBYD5JNAACwICqeABZr0vRKazz1S7IJAFgZk6qffIinD6rxYDGszQSrT7IJWGurvpbTqsfPbFQ+wOqQYALox1aTkOP9VTktn2QTsHb6WjTfYv3QHwkAhmSW56MkM8DySDwtn2QTsGNMSgapGAKgT6aHArBuJJsAAGCAVD+xk1iniWVR5bQcK5Nsqqprk3wgyU8l+Uhr7baeQwLW0KQ1kqydNHzGCWDoTPfsl3FieCSYGKpZnpvjiSoJrOer1lrfMZxVVf1UkoeT/I0kR5Pcn+TG1tq3ztR/Y2OjHTlyZIkRnpt5fXi1Tgws16QkVF8xbFVV/UFrbWN+0fRv1ceJrX4AHa9w8OEVdp5Z3gO2UwllnBjeOLEqJI/gJ61j4mnWMWJVKpuuSvJoa+07SVJVdye5PskZB4chm/TBdJa1ZCSVYBi8FgdpbcYJgLORZD4nazNObCehM6kSY179gdlMej2tU3JqVZJNlyV5Yuzy0SSvGe9QVQeSHOgu/o+q+qMlxXZxkv+2pP81KzHNZogxJcOMS0yzWZWYXt5HIAu26HFiYI/tr5+tw8DiPSvxLpZ4F2ug8U58n5glXuPEcj9PzMtZH9tfP/v4sa3+CzbQ19pcrPO+JTt4/1b8NZds47PEqiSbzqq1dnuS25f9f6vqyNDKjMU0myHGlAwzLjHNRkzDtp1xYtXuR/EulngXS7yLtWrxLlNfnyfmZd0f23Xev3Xet8T+rbLt7Nt58w5mQY4luWLs8uVdGwAkxgkApjNOACzRqiSb7k+yp6p2V9ULkrwtycGeYwJgOIwTAExjnABYopWYRtda+1FVvTvJFzL6qdI7Wmvf7Dmsk4ZYaium2QwxpmSYcYlpNmLqyRLGiVW7H8W7WOJdLPEu1qrFOxcD/zwxL+v+2K7z/q3zviX2b5Wd875Va22egQAAAACwg63KNDoAAAAAVoBkEwAAAABzI9k0o6q6tqr+qKoerapbz3D9P6yqb1XVg1V1uKpePoCY3lVV36iqB6rqP1XVK/uOaazfW6qqVdXCfyJyhvvpnVV1orufHqiqv9N3TF2fG7rn1Der6uN9x1RV/2rsPnq4qv5k0THNGNfLqupLVfW17vV33QBienn3PvBgVd1bVZcvOJ47qup4VT004fqqqg928T5YVa9eZDyrbIjv9dMMcRyYZohjxDRDHD+mGeLYMs1Qx51JhjgeTTO0sYr5q6qLquqeqnqkO7/wDH1eVVVf6V7zD1bV3+4j1nMxy/51/T5fVX9SVb+17Bi3aobX5U9X1Se763+/qq5cfpTnbob9+6Wq+mpV/aiq3tpHjOdq1Y4Rt2ohx5StNaeznDJaRPDbSX42yQuSfD3JK0/r8/okf6bb/tUknxxATC8Z235Tks/3HVPX78VJvpzkviQbfceU5J1J/vXAnk97knwtyYXd5Zf2HdNp/f9+RgtrDuG+uj3Jr3bbr0zy+ABi+ndJ9nfbfz3JxxYc0y8leXWShyZcf12S/5Ckkrw2ye8v+rFbxdMQ3+vnEO9Sx4Htxtv1W9oYMYf7d6njxxziXerYMo/nw1j/pYw727x/lzoezSHepY5VTgt5nP95klu77VuT/MYZ+rwiyZ5u+y8keTLJBX3HPq/96667Osm+JL/Vd8xn2Z9ZXpd/L8m/6bbf1udxxoL278okP5/kriRv7TvmOe/bYI4RF7R/Wz6mVNk0m6uSPNpa+05r7f8muTvJ9eMdWmtfaq092128L8mivx2aJab/PnbxRUkWvRr8WWPqvD/JbyT53wuOZysxLdMsMf3dJB9qrf0gSVprxwcQ07gbk3xiwTHNGldL8pJu+88l+eMBxPTKJL/bbX/pDNfPVWvty0mentLl+iR3tZH7klxQVZcuMqYVNcT3+mmGOA5MM8QxYpohjh/TDHFsmWao484kQxyPphncWMVCXJ/kzm77ziRvPr1Da+3h1toj3fYfJzme5JKlRbg9Z92/JGmtHU7yw2UFtQ2zvC7H9/nTSa6uqlpijNsxy3HJ4621B5P8vz4C3IZVO0bcqoUcU0o2zeayJE+MXT7atU1yc0ZVBIs0U0xVdUtVfTujbwb+Qd8xddN3rmit/faCY5k5ps5bupLHT1fVFQOI6RVJXlFVv1dV91XVtQOIKcmo7D7J7pw6QO07rs0kb6+qo0k+l9G3333H9PUkv9Jt/60kL66qn1lwXNNs9T1spxrie/00QxwHphniGDHNEMePaYY4tkwz1HFnkiGOR9Os4ljF1u1qrT3ZbX8vya5pnavqqoyqFr696MDmZEv7twJmeV0+16e19qMkzyRZldflOh9vrtox4lYt5JhSsmnOqurtSTaS/Iu+Y0mS1tqHWmt/McmvJfmnfcZSVecl+ZdJ/lGfcZzBoSRXttZ+Psk9OfVtQp/Oz2i6w96Mvs39t1V1Qa8RnfK2JJ9urf2470A6Nyb5aGvt8oymi32se6716R8neV1VfS3J65IcSzKU+4s5GNp7/TRDGgemGfAYMc0Qx49phjy2TDO0cWeSIY5H0xirVkBVfbGqHjrD6fSqg5YplQZdRfPHktzUWhtMVcm89g+GYpWOEbdqq8eU5y8+pLVwLMn4t5WXd20/oap+Ocl7kryutfZ/hhDTmLuTfHihEZ09phcn+StJ7u2qQf98koNV9abW2pGeYkpr7ftjFz+SUaZ2kWZ57I5mtK7OnyZ5rKoezugDwv09xnTS25LcsqA4TjdLXDcnuTZJWmtfqaoXJrk4ozLxXmLqytR/JUmq6s8meUtrrc+Fbbf6frFTDfG9fpohjgPTDHGMmGaI48c0QxxbphnquDPJEMejaVZxrOIMWmu/POm6qnqqqi5trT3ZJZPO+Fyrqpck+e0k7+mm0w/GPPZvhczyPnKyz9GqOj+jKbnfz2pY5+PNVTtG3KqFHFMO+duWIbk/yZ6q2l1VL8jooOfgeIeq+qtJfjPJm5a0BsIsMe0Zu/g3kzzSZ0yttWdaaxe31q5srV2Z0VzWRX+ImOV+Gl+75k1J/ssC45kppiT/PqNvnlNVF2c09eE7PceUqvq5JBcm+coCY9lqXP81o4UhU1V/KckLk5zoM6aqunjs2+x/kuSOBcYzi4NJ3lEjr03yzFhZOqcM8b1+miGOA9MMcYyYZojjxzRDHFumGeq4M8kQx6NpVnGsYusOJtnfbe9P8tnTO3SP/2cyWrvx00uMbR7Oun8rZpb3kfF9fmuS3+2qulbBTO/rK2rVjhG3ajHHlG0Aq5+vwimjcuiHM5rj/J6u7X0ZPZmS5ItJnkryQHc6OICYPpDkm108X0ryl/uO6bS+92YJvzQ0w/30z7r76evd/fRzA4ipMppO8q0k30jytr5j6i5vJrlt0bFs8b56ZZLf6x6/B5JcM4CY3prRG/DDGVU7/PSC4/lERr8u86cZVS7cnORdSd419nz6UBfvN5bxulvV0xDf67cZ79LHge3Ee1rfpYwR27x/lz5+bDPepY8t230+9DHubOP+Xfp4tM14lzpWOS3kMf6ZJIe7x/GLSS7q2jeSfKTbfntGxwcPjJ1e1Xfs89q/7vJ/zCix+78yOg56Q9+xT9mns70uX5jRL0U+muQ/J/nZvmOe8/79te4x+p8ZVWx9s++Y57hvgzpGXMD+bfmYsrobAgAAAMC2mUYHAAAAwNxINgEAAAAwN5JNAAAAAMyNZBMAAAAAcyPZBAAAAMDcSDYBAAAAMDeSTQAAAADMzf8HhGhqbtZzVncAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from sklift.models import TwoModels\n", "\n", "\n", "tm = TwoModels(\n", " estimator_trmnt=CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True), \n", " estimator_ctrl=CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True), \n", " method='vanilla'\n", ")\n", "tm = tm.fit(\n", " X_train, y_train, treat_train,\n", " estimator_trmnt_fit_params={'cat_features': cat_features}, \n", " estimator_ctrl_fit_params={'cat_features': cat_features}\n", ")\n", "\n", "uplift_tm = tm.predict(X_val)\n", "\n", "tm_score = uplift_at_k(y_true=y_val, uplift=uplift_tm, treatment=treat_val, strategy='by_group', k=0.3)\n", "\n", "models_results['approach'].append('TwoModels')\n", "models_results['uplift@30%'].append(tm_score)\n", "\n", "plot_uplift_preds(trmnt_preds=tm.trmnt_preds_, ctrl_preds=tm.ctrl_preds_);" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "9a8BCVaYCIPH" }, "source": [ "### 2.2 Two dependent models\n", "\n", "The dependent data representation approach is based on the classifier chain method originally developed\n", "for multi-class classification problems. The idea is that if there are $L$ different classifiers, each of which solves the problem of binary classification and in the learning process,\n", "each subsequent classifier uses the predictions of the previous ones as additional features.\n", "The authors of this method proposed to use the same idea to solve the problem of uplift modeling in two stages.\n", "\n", "At the beginning we train the classifier based on the control data:\n", "\n", "$$\n", "P^C = P(Y=1| X, W = 0),\n", "$$\n", "\n", "Next, we estimate the $P_C$ predictions and use them as a feature for the second classifier.\n", "It effectively reflects a dependency between treatment and control datasets:\n", "\n", "$$\n", "P^T = P(Y=1| X, P_C(X), W = 1)\n", "$$\n", "\n", "To get the uplift for each observation, calculate the difference:\n", "\n", "$$\n", "uplift(x_i) = P^T(x_i, P_C(x_i)) - P^C(x_i)\n", "$$\n", "\n", "Intuitively, the second classifier learns the difference between the expected probability in the treatment and the control sets which is the uplift.\n", "\n", "

\n", " \"Two\n", "

" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:46.835046Z", "start_time": "2020-05-30T22:29:44.970479Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 444 }, "colab_type": "code", "id": "vBHTDpX-CIPR", "outputId": "32d97144-dd91-49d3-e020-026fdfb0460f" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABJgAAAGrCAYAAACMgi0UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdfbxVZZnw8d8VvlD5hsjjKGgwRc3ojGLhS2ZGWmIm0osa2pNoFmNDpWmZljOeUmeq0Sy1dCxJbCw0M4WyKcoYK1/RyETzkUQTQyUBlRwN9Hr+WPeBzfHsw4F99tnncH7fz2d/zlr3utda19rouve+9n3fKzITSZIkSZIkaUO9rNUBSJIkSZIkqX8zwSRJkiRJkqSGmGCSJEmSJElSQ0wwSZIkSZIkqSEmmCRJkiRJktQQE0ySJEmSJElqiAkmaSMQEW0R8V9leeeIWBERgzbgOJ+JiG/2fISStHGLiIci4m0tOvflEXF2WX5zRNy/gce5JCL+pWejk6T+q6v7a0S8LiLmRcQzEfHx1kW5Op5xEbGoZn1+RIzbgONscDsimWBSU5VER/vrxYj435r19/fgeVbf/HtbRIyMiIyITVpx/o4y84+ZuUVmvtBVvY6NUNn33zLzQ82NUJI2TEQcHRFzSxuyOCJ+HBH79cBxW9aG9LTM/GVmvm5d9SLi2Ij4VYd9T8jMs5oXnST1rvIZ/TUdylb/MLs+Orm/ngr8IjO3zMwLWvlDQ2cyc9fMnLOueh3fo+62I1JnTDCpqUqiY4vM3AL4IzChpuzK9np9JTnTF/heSNJLRcTJwFeAfwO2B3YGvg5M7IVz99p92TZAkvqNVwHzm3XwDRmNILWaCSa1RHvvmYj4dEQ8BnwrIl4WEadFxB8i4smIuDoitq3Z53sR8VhEPBURN0XErqV8CvB+4NTyq/asUv5QRHwqIu6OiL9ExGURsX35xfuZiPhZRAypOf4+EXFzRCyPiN/WdimNiDkRcVZE/Lrs+9OI2K5svqn8XV7O/8ZOrrctIq6JiKvK/ndFxO412x8q78XdwF8iYpN1xDMqIv6nHGs2sF3NtrV6VEXEthHxrYj4U0Qsi4jrIuKVwI+BHWt6lO3Y8RediDisdK9dXt6Dv+8Q8yfL+/tUubbBZdt2EfHDst/SiPhlRHi/kbRBImJr4PPA1My8NjP/kpkrM3NWZn6q1Nk8Ir5S7nV/Ksubl23tbc4pEfFE6f10XNnWVRvS8b5c9564jvgvj2r42exy3/6fiHhVzfaMiKkR8QDwQCk7NKqhF8tLW7BbTf09SjvyTERcBQyu2dZxiMROEXFtRCyJqm29qMR9CfDGcs3La+I8u2bfD0fEgnIfnxkRO3aI+YSIeKDE+LWIiLLtNeUan4qIP5cYJanPqWkfPlPuVw9FnVEWtffXiLgReCtwUbmPfpfqh49ZZf3U9T1XuQdfHBE3RMRfgLeWz+ffL/fwhVEzFC8iXl72WRYR9wJ7djjf6h5VETGonPcPpe24s7QP7d9jflvifl8n7cjflzZveWkDD+sQ89ci4kfluLdFxKvLtoiI80u7+3RE/C4i/mE9/4nUz/iFT630N8C2VNn/KcDHgHcBbwF2BJYBX6up/2NgNPB/gLuAKwEy89Ky/KXSM2pCzT7vBd4OvBaYUI7xGWAY1X//HweIiOHAj4CzS0yfBL4fEcNqjnU0cFw5/2alDsD+5e825fy31LneicD3yvG/A1wXEZvWbD8KeCewDdWv813F8x3gTqrE0lnA5DrnBPg28Apg1xL7+Zn5F+AdwJ9qepT9qXaniHgt8F3gpPJ+3UDVaG5WU+1I4GBgFLAbcGwpPwVYVPbbnuo9zy5ilKSuvJEqifKDLup8FtgHGAPsDuwFnFGz/W+ArYHhwPHA1yJiyDrakNr78t+y7ntiV95Pdb/eDphXzlnrXcDewC4RsQcwDfgnYCjwn8DMqJJomwHXUd3bt6VqV97b2Qmj+vX7h8DDwMhy7TMy8z7gBOCWcs3bdLLvAcC/U93ndyjHmNGh2qFUX2h2K/XGl/KzgJ8CQ4ARwIVdvzWS1FJ/Q3VvHk71mfrSiOhyiFhmHgD8EvhouY8exdqjNb60gec6GjgH2BK4GZgF/LbUPxA4KSLa77VnAq8ur/F0/X3gZKo27RBgK+CDwLOZ2f49ZvcS91o/CJTvKrOo7un/h+r72pUdYp4EfI7qnr+gxA9wENX3pNdStb9HAk92EaM2AiaY1EovAmdm5vOZ+b9UH3Y/m5mLMvN5oA04PEpPnMyclpnP1GzbPapftbtyYWY+npmPUjUCt2XmbzLzOaovKnuUev8XuCEzb8jMFzNzNjCX6ibc7luZ+f9KrFdTfYlZH3dm5jWZuRL4MtWXpX1qtl+QmY+U49eNJyJ2pvpA/y/lvbuJ6sb/EhGxA1Ui6YTMXFZ+8f+fbsb7PuBHmTm7xHwu8HJg3w4x/ykzl5YY2t+TlVRfSF5VzvnLzDTBJGlDDQX+nJmruqjzfuDzmflEZi6h+rD7gZrtK8v2lZl5A7ACWNccE7X35e7cE7vyo8y8qbRhn6XqPbRTzfZ/z8yl5VxTgP/MzNsy84XMnA48T9Vm7ANsCnylXMs1wB11zrkX1Q82nyq9vp7LzF/VqdvR+4FpmXlXifn0EvPImjpfyMzlmflH4Bes3Qa8CthxPc8pSa3S/rn6f6h+5D2yRee6PjN/nZkvAv8IDMvMz2fmXzPzQeAbVAkdyn7nlLbjEeCCLs75IeCMzLw/K7/NzO4ke/YBtqC63/81M2+k+uHiqJo6P8jM20sbfSVrtwVbAn8HRGbel5mLu3FO9WMmmNRKS0qip92rgB+U7pfLgfuAF4DtS7fOL5RunU8DD5V9tqNrj9cs/28n61vUnPuI9nOX8+9HlSRp91jN8rM1+3bXI+0LpdFYRPXB/yXb1xHPjsCy0gup3cN1zrkTsDQzl61nrJTzrD5uifkRql9Q2tV7T/6D6heMn0bEgxFx2gacX5LaPQlsF13PT7TWPass195jn+yQoOrOfbz2vtyde2K3jpWZK4CldN0GnNKhDdip1N8ReLRD0r6rNuDhdSTm6ul4vSuo/h260wacCgRwexlO8cENOL8k9YQXqJLytTalSn606+xz9Y40x7rO1bEt2LFDW/AZqtEBlP1q69drC6BqD/6wAfHuCDxS2rza86yzLSjJqIuoRqQ8ERGXRsRWGxCD+hETTGqljj1aHgHekZnb1LwGl95HR1MNMXsbVRfLkWWfqHOs9fUI8O0O535lZn5hA66jntW/VEc1H9EIoHZYWu1xuopnMTAkqnmU2u1c55yPANtGxEuGP3Qj7j9RNWztMUe5hkfXsR+lp9kpmfm3wGHAyRFx4Lr2k6Q6bqHqwfOuLuqsdc+iui/+qU7djurdD2vLN/ieWNS2AVtQDW/rqg04p0Mb8IrM/C5VGzC8nL9dV23AznUSc+vbBrySqidZd9qAxzLzw5m5I9Uwv69Hh6c4SVIv+SNrvje0G8XayZjOPld3t/2o1Z3vBOs6V8e2YGGHtmDLzGwfYbGYmraF+m1B+7Fe3Y34OvoTsFOsPZfqznSz7cvMCzLzDcAuVEPlPrUBMagfMcGkvuQS4JwoE59GxLCIaH860JZUXy6epJpP6N867Ps41fwYG+q/gAkRMb70lhpcJrgb0Y19l1AN91vX+d8QEe8pH/RPorqeW9c3nsx8mGq43OciYrOoHtE9obODlG6oP6b6cD8kIjaNiPax1o8DQ7sYZng18M6IOLCMvz6lxHzzOq6zfXLa15QvQE9R/Xr04jp2k6ROZeZTwL9SzZv0roh4RbmfvSMi2ue5+C5wRmk7tiv1u/sY6u60IRt8TywOiYj9yhxKZwG3liENnfkGcEJE7F0mSX1lRLwzIrakSratAj5e3oP3UA2F68ztVF9AvlCOMTgi3lRzzSO6mEPqu8BxETEmqsnS/41qmPlD67rQiDiipv1cRvWFyTZAUitcRdU2jIjqgUJvo/rcfE2Heu2fq99MNb/c9zbgXN39PtLdc90OPBPVAydeXr4T/ENEtE/mfTVwevmMP4JqfqR6vgmcFRGjS7uyW0QM7Ubct1H1Sjq1tDnjqN6/jnPyvURE7FnasU2BvwDPYVuw0TPBpL7kq8BMqmFVz1AlX/Yu266g+qXhUeBeXpqYuYxqYtTlEXHd+p64fMifSNXtdAlVlv9TdOP/kcx8lmoyu1+X8+9Tp+r1VHN4LKOaF+Q9ZR6PDYnnaKr3ZinVBH9XdBHiB6i6Af8eeIIquUVm/p7qC8SDJe61ugJn5v1Uc0FdCPyZqjGZkJl/7eJc7UYDP6Oa4+QW4OuZ+Ytu7CdJncrM86gmKT2DNffFj1JNeA3VQxHmAncDv6N6GMTZLz1Sp9bZhjR4T4Tq4QxnUt2331CO1anMnAt8mGpowTKqIcfHlm1/Bd5T1pdStSvX1jnOCyXO11D9ir+o1Ae4kerx2o9FxJ872fdnwL8A36dKUr2aNfN+rMuewG0RsYKqXT+xzB0iSb3t81Q/BPyK6n76JeD9mXlPTZ3HyrY/Uc0hdEL5nLy+/p0qmbU8Ij5Zp063z1Xu4YdSzWm0kKrt+SbVaA6o5hp8uGz7KdXDH+r5MlVC6qfA01Tt3svLtjZgeol7rbmnSpszgWpO1z8DXweO6eb7sxXVDybLSpxPUk2joY1YpPPuSk0XEW3AazKz7hcKSdLGKSIuBxZl5hnrqitJ6j2lR85/ZWZ3Ri30m3NJrWIPJkmSJEmSJDXEBJMkSZIkSZIa4hA5SZIkSZIkNcQeTJIkSZIkSWrIJq0OoBm22267HDlyZKvDkKQ+6c477/xzZg5rdRytZDshSfXZTthOSFI9XbURG2WCaeTIkcydO7fVYUhSnxQRD7c6hlaznZCk+mwnbCckqZ6u2giHyEmSJEmSJKkhJpgkSZIkSZLUEBNMkiRJkiRJashGOQeTpIFh5cqVLFq0iOeee67VofRJgwcPZsSIEWy66aatDkWSusX7eu+ynZCk7hmI7dOGtBEmmCT1W4sWLWLLLbdk5MiRRESrw+lTMpMnn3ySRYsWMWrUqFaHI0nd4n2999hOSFL3DbT2aUPbCIfISeq3nnvuOYYOHTogbvLrKyIYOnTogPqVRVL/532999hOSFL3DbT2aUPbCBNMkvq1gXKT3xC+N5L6I+9dvcf3WpK6b6DdMzfkek0wSZIkSZIkqSHOwSRpozFrVs8eb8KErrc/+eSTHHjggQA89thjDBo0iGHDhgFw++23s9lmm633OefMmcNmm23Gvvvuu177jRw5krlz57Lddtut9zklqa9auLCtR483atS6jxcRnHzyyZx33nkAnHvuuaxYsYK2tvr7Xnfddbz2ta9ll112Wa9tG2JDj7fFFluwYsWKHolBkga6tjltPXu8ces+3kMPPcShhx7KPffcs2a/tja22GILPvnJT3a6z+WXX87cuXO56KKLuOSSS3jFK17BMcccw+9//3smTZpERHDNNddw2223cfTRRzd8HfZgkqQNNHToUObNm8e8efM44YQT+MQnPrF6fbPNNmPVqlXrfcw5c+Zw8803NyFaSVJ3bL755lx77bX8+c9/7vY+1113Hffee+96b9uQdqKr40mSVM8JJ5zAMcccA1RtyeGHH85vfvMbHnnkEb7zne/0yDlMMElSDzr22GM54YQT2HvvvTn11FP5wx/+wMEHH8wb3vAG3vzmN/P73/8egFmzZrH33nuzxx578La3vY3HH3+chx56iEsuuYTzzz+fMWPG8Mtf/pIlS5bw3ve+lz333JM999yTX//610DVe+qggw5i11135UMf+hCZ2crLlqSNxiabbMKUKVM4//zzX7LtoYce4oADDmC33XbjwAMP5I9//CM333wzM2fO5FOf+hRjxozhD3/4w+r6nW0bN24cJ510EmPHjuWrX/0qd955J295y1t4wxvewPjx41m8eDEA3/jGN9hzzz3Zfffdee9738uzzz7b6fHqtTMLFy7kjW98I//4j//IGWec0TtvniSpJcaNG8eJJ57ImDFj+Id/+Aduv/32l9Rpa2vj3HPP5YYbbuArX/kKF198MW9961s57bTT+OUvf8mYMWM6bfvWh0PkJKmHLVq0iJtvvplBgwZx4IEHcskllzB69Ghuu+02/vmf/5kbb7yR/fbbj1tvvZWI4Jvf/CZf+tKXOO+88zjhhBPW6uZ69NFH84lPfIL99tuPP/7xj4wfP5777ruPz33uc+y3337867/+Kz/60Y+47LLLWnzVkrTxmDp1KrvtthunnnrqWuUf+9jHmDx5MpMnT2batGl8/OMf57rrruOwww7j0EMP5fDDD1+r/r777tvptr/+9a/MnTuXlStX8pa3vIXrr7+eYcOGcdVVV/HZz36WadOm8Z73vIcPf/jDAJxxxhlcdtllfOxjH3vJ8eq1MyeeeCIf+chHOOaYY/ja177W5HdMktRqzz77LPPmzeOmm27igx/84FpD6Wodcsgha33nmDNnDueeey4//OEPG47BBJMk9bAjjjiCQYMGsWLFCm6++WaOOOKI1duef/55oEpCve9972Px4sX89a9/ZdSoUZ0e62c/+9laQyGefvppVqxYwU033cS1114LwDvf+U6GDBnSxCuSpIFlq6224phjjuGCCy7g5S9/+eryW265ZfW99wMf+MBLElDd9b73vQ+A+++/n3vuuYe3v/3tALzwwgvssMMOANxzzz2cccYZLF++nBUrVjB+/PiXHKerdubXv/413//+91fH+ulPf3qDYpUk9Q31nurWXn7UUUcBsP/++/P000+zfPnyXoutnQkmSephr3zlKwF48cUX2WabbZg3b95L6nzsYx/j5JNP5rDDDmPOnDl1J4998cUXufXWWxk8eHAzQ5YkdXDSSSfx+te/nuOOO67Hj93eTmQmu+66K7fccstL6hx77LFcd9117L777lx++eXMmTPnJXW6amdg4D1SW5I2ZkOHDmXZsmVrlS1dunT1D9Ud7/mtaAOcg0mSmmSrrbZi1KhRfO973wOqLxK//e1vAXjqqacYPnw4ANOnT1+9z5Zbbskzzzyzev2ggw7iwgsvXL3e/iVi//33Xz0Z349//OOXNDaSpMZsu+22HHnkkWsNQd53332ZMWMGAFdeeSVvfvObgZfeu2t1te11r3sdS5YsWZ1gWrlyJfPnzwfgmWeeYYcddmDlypVceeWVnR6vq3bmTW9601qxSpL6ty222IIddtiBG2+8EaiSS//93//NfvvtB8BVV10FwK9+9Su23nprtt56624dt6t2an3Zg0nSRmPChFZH8FJXXnklH/nIRzj77LNZuXIlkyZNYvfdd6etrY0jjjiCIUOGcMABB7Bw4UIAJkyYwOGHH87111/PhRdeyAUXXLB6LpBVq1ax//77c8kll3DmmWdy1FFHseuuu7Lvvvuy8847t/hKJannjRrV1tLzn3LKKVx00UWr1y+88EKOO+44/uM//oNhw4bxrW99C4BJkybx4Q9/mAsuuIBrrrmGV7/61av36bit1mabbcY111zDxz/+cZ566ilWrVrFSSedxK677spZZ53F3nvvzbBhw9h7771Xf/jveLx67cxXv/pVjj76aL74xS8yceLEXni3JGngaBvX1pLzXnHFFUydOpWTTz4ZgDPPPHN1mzN48GD22GMPVq5cybRp07p9zN12241Bgwax++67c+yxx/KJT3xig+OLjfHJQ2PHjs25c+e2OgxpnWbNWrPcF5Mjfd19993H3//937c6jD6ts/coIu7MzLEtCqlPsJ3Qxm7hwrbVy61O0qwP7+u9z3aic7YT6g/a5rStWW5RwmOg6Ovt07hx4zj33HMZO7Znb93r20Y4RE6SJEmSJEkNcYicJEmSJElSP9XZgyBawR5Mkvq1jXGYb0/xvZHUH3nv6j2+15LUfQPtnrkh12uCSeqHZs1a8xrIBg8ezJNPPjngbvbdkZk8+eSTDB48uNWhSFK3eV/vPbYTktR9A6192tA2wiFykvqtESNGsGjRIpYsWdLqUPqkwYMHM2LEiFaHIUnd5n29d9lOSFL3DMT2aUPaCBNMkvqtTTfdlFGjRrU6DElSD/G+Lknqi2yfuqdpQ+QiYqeI+EVE3BsR8yPixFLeFhGPRsS88jqkZp/TI2JBRNwfEeNryg8uZQsi4rRmxSxJktRfLVzYtvolSZLU25rZg2kVcEpm3hURWwJ3RsTssu38zDy3tnJE7AJMAnYFdgR+FhGvLZu/BrwdWATcEREzM/PeJsYuSZIkSZKkbmpagikzFwOLy/IzEXEfMLyLXSYCMzLzeWBhRCwA9irbFmTmgwARMaPUNcEkSZIkSZLUB/TKU+QiYiSwB3BbKfpoRNwdEdMiYkgpGw48UrPbolJWr7zjOaZExNyImDuQJt6SJEmSJElqtaYnmCJiC+D7wEmZ+TRwMfBqYAxVD6fzeuI8mXlpZo7NzLHDhg3riUNKkiRJkiSpG5r6FLmI2JQquXRlZl4LkJmP12z/BvDDsvoosFPN7iNKGV2US5IkSZIkqcWa+RS5AC4D7svML9eU71BT7d3APWV5JjApIjaPiFHAaOB24A5gdESMiojNqCYCn9msuCVJkiRJkrR+mtmD6U3AB4DfRcS8UvYZ4KiIGAMk8BDwTwCZOT8irqaavHsVMDUzXwCIiI8CPwEGAdMyc34T45YkSZIkSdJ6aOZT5H4FRCebbuhin3OAczopv6Gr/SRJkiRJktQ6vfIUOUmSJEmSJG28TDBJkiRJkiSpISaYpD5o1qzqJUmSJElSf2CCSZIkSZIkSQ0xwSRJkiRJkqSGmGCSJEmSJElSQ0wwSZIkSZIkqSEmmCRJkiRJktQQE0ySJEmSJElqyCatDkBS882atWZ5woTWxSFJkiRJ2jjZg0mSJEmSJEkNMcEkSZIkqU+KiIci4ncRMS8i5paybSNidkQ8UP4OKeURERdExIKIuDsiXl9znMml/gMRMblV1yNJGzMTTJKkloiIwRFxe0T8NiLmR8TnSvmoiLitfEG4KiI2K+Wbl/UFZfvImmOdXsrvj4jxrbkiSVKTvDUzx2Tm2LJ+GvDzzBwN/LysA7wDGF1eU4CLoUpIAWcCewN7AWe2J6UkST3HOZgkSa3yPHBAZq6IiE2BX0XEj4GTgfMzc0ZEXAIcT/Ul4XhgWWa+JiImAV8E3hcRuwCTgF2BHYGfRcRrM/OFVlyUJKnpJgLjyvJ0YA7w6VJ+RWYmcGtEbBMRO5S6szNzKUBEzAYOBr7bu2FLjWub09bqEKS6TDBJklqifAFYUVY3La8EDgCOLuXTgTaqBNPEsgxwDXBRREQpn5GZzwMLI2IB1S/UtzT/KiRJTZbATyMigf/MzEuB7TNzcdn+GLB9WR4OPFKz76JSVq98LRExharnEzvvvHNPXoPUEJNK6i8cIidJapmIGBQR84AngNnAH4DlmbmqVKn9ErD6C0LZ/hQwlPX44hARcyNi7pIlS5pxOZKknrdfZr6eavjb1IjYv3Zj+bEie+JEmXlpZo7NzLHDhg3riUNK0oBigkmS1DKZ+UJmjgFGUPU6+rsmnssvDpLUz2Tmo+XvE8APqNqKx8vQN8rfJ0r1R4GdanYfUcrqlUuSepAJJklSy2XmcuAXwBuBbSKifQh37ZeA1V8QyvatgSfxi4MkbZQi4pURsWX7MnAQcA8wE2h/Etxk4PqyPBM4pjxNbh/gqTKU7ifAQRExpEzufVApkyT1IOdgkiS1REQMA1Zm5vKIeDnwdqqJu38BHA7M4KVfHCZTza10OHBjZmZEzAS+ExFfpprkezRwe69ejCSpGbYHflBNt8cmwHcy878j4g7g6og4HngYOLLUvwE4BFgAPAscB5CZSyPiLOCOUu/z7RN+SxuL2nma2sa11a0nNZMJJklSq+wATI+IQVQ9aq/OzB9GxL3AjIg4G/gNcFmpfxnw7TKJ91KqJ8eRmfMj4mrgXmAVMNUnyElS/5eZDwK7d1L+JHBgJ+UJTK1zrGnAtJ6OUZK0hgkmSVJLZObdwB6dlD9INcdGx/LngCPqHOsc4JyejlGSJElS9zgHkyRJkiRJkhpigkmSJEmSJEkNMcEkSZIkSZKkhphgkiRJkiRJUkNMMEmSJEmSJKkhPkVO6iNmzWp1BJIkSZIkbRh7MEmSJEmSJKkh9mCS+rnank8TJrQuDklS/7JwYdvq5VGj2urWkyRJ6g4TTNJGyiF3kiRJkqTe4hA5SZIkSZIkNcQeTFIfZi8kSZIkSVJ/YIJJ6gXOkyRJkiRJ2pg5RE6SJEmSJEkNMcEkSZIkSZKkhphgkiRJkiRJUkNMMEmSJEmSJKkhJpgkSZIkSZLUEBNMkiRJkiRJaogJJkmSJEmSJDXEBJMkSZIkSZIaYoJJkiRJkiRJDTHBJEmSJEmSpIaYYJIkSZIkSVJDTDBJkiRJkiSpISaYJEmSJEmS1JBNWh2AJEmSWmvhwrbVy6NGtdWtJ0mSVI89mCRJkiRJktQQE0ySJEmSJElqiAkmSZIkSZIkNcQEkyRJkiRJkhpigkmSJEmSJEkN8Sly0gAza9aa5QkTWheHJEmSJGnjYYJJ6mW1CR5JknrTwoVtrQ5BkiRtpBwiJ0mSJEmSpIbYg0mSJEmr1fZyGjWqrW49SZKkWiaYpI2Iw+8kSZIkSa3gEDlJkiRJkiQ1xASTJEmSJEmSGmKCSZIkSZIkSQ0xwSRJkiRJkqSGNC3BFBE7RcQvIuLeiJgfESeW8m0jYnZEPFD+DinlEREXRMSCiLg7Il5fc6zJpf4DETG5WTFLkiRJktSftc1pW/2SelMzezCtAk7JzF2AfYCpEbELcBrw88wcDfy8rAO8AxhdXlOAi6FKSAFnAnsDewFntielJEmSJEmS1HpNSzBl5uLMvKssPwPcBwwHJgLTS7XpwLvK8kTgiqzcCmwTETsA44HZmbk0M5cBs4GDmxW3JEmSJEmS1k+vzMEUESOBPYDbgO0zc3HZ9BiwfVkeDjxSs9uiUlavXJIkSZIkSX1A0xNMEbEF8H3gpMx8unZbZiaQPXSeKRExNyLmLlmypCcOKUmSJEmSpG5oaoIpIoDJs4wAACAASURBVDalSi5dmZnXluLHy9A3yt8nSvmjwE41u48oZfXK15KZl2bm2MwcO2zYsJ69EEmSJEmSJNXVzKfIBXAZcF9mfrlm00yg/Ulwk4Hra8qPKU+T2wd4qgyl+wlwUEQMKZN7H1TKJDVo1qw1L6m3dfG00baIeDQi5pXXITX7nF6eNnp/RIyvKT+4lC2IiNM6O58kSZKk5tmkicd+E/AB4HcRMa+UfQb4AnB1RBwPPAwcWbbdABwCLACeBY4DyMylEXEWcEep9/nMXNrEuCVJvaP9aaN3RcSWwJ0RMbtsOz8zz62tXJ5EOgnYFdgR+FlEvLZs/hrwdqp5+u6IiJmZeW+vXIUkSZKk5iWYMvNXQNTZfGAn9ROYWudY04BpPRedJKnVSi/VxWX5mYhof9poPROBGZn5PLAwIhYAe5VtCzLzQYCImFHqmmCSJEmSekmvPEVOkqSudHjaKMBHI+LuiJhWhkdDg08b9WEQkiRJUvM0c4icJEnr1PFpoxFxMXAW1VNGzwLOAz7Y6Hky81LgUoCxY8f2yBNMpf5g4cK2VocgSZIGABNMkqSW6expo5n5eM32bwA/LKtdPVV0nU8blSRJktQ8DpGTJLVEvaeNRsQONdXeDdxTlmcCkyJi84gYBYwGbqd6CMToiBgVEZtRTQQ+szeuQZIkSVLFHkySAJg1a83yhAmti0MDSr2njR4VEWOohsg9BPwTQGbOj4irqSbvXgVMzcwXACLio8BPgEHAtMyc35sXIkmSJA10JpgkSS3RxdNGb+hin3OAczopv6Gr/SRJkiQ1l0PkJEmSJEmS1BATTJIkSerUwoVtq19Sq0TEoIj4TUT8sKyPiojbImJBRFxV5t+jzNF3VSm/LSJG1hzj9FJ+f0SMb82VSNLGzQSTJEmS1ouJJ/WyE4H7ata/CJyfma8BlgHHl/LjgWWl/PxSj4jYheoBELsCBwNfj4hBvRS7JA0YJpgkSZIk9UkRMQJ4J/DNsh7AAcA1pcp04F1leWJZp2w/sNSfCMzIzOczcyGwANird65AkgYOJ/mWmqT2qWySJEnaIF8BTgW2LOtDgeWZuaqsLwKGl+XhwCMAmbkqIp4q9YcDt9Ycs3af1SJiCjAFYOedd+7Zq5CkAcAeTJIkSZL6nIg4FHgiM+/sjfNl5qWZOTYzxw4bNqw3TilJGxV7MEmSJGmdnG9JLfAm4LCIOAQYDGwFfBXYJiI2Kb2YRgCPlvqPAjsBiyJiE2Br4Mma8na1+0iSeog9mCRJkiT1OZl5emaOyMyRVJN035iZ7wd+ARxeqk0Gri/LM8s6ZfuNmZmlfFJ5ytwoYDRwey9dhiQNGPZgkiRJktSffBqYERFnA78BLivllwHfjogFwFKqpBSZOT8irgbuBVYBUzPzhd4PW5I2biaYJEmSJPVpmTkHmFOWH6STp8Bl5nPAEXX2Pwc4p3kRSpJMMEmSJPVTzoskSZL6CudgkiRJkiRJUkNMMEmSJEmSJKkhJpgkSZIkSZLUEBNMkiRJkiRJaogJJkmSJEmSJDXEBJMkSZIkSZIaskmrA5AkSVL/tXBh2+rlUaPa6taTJEkbN3swSZIkSZIkqSEmmCRJkiRJktQQE0ySJEmSJElqiHMwSZIkSZK0EWqb07ZmeVxb3XpST7AHkyRJkiRJkhpiDyZJLzFr1prlCRNaF4ckSZIkqX+wB5MkSZIkSZIaYg8mSZIkSZL6kNq5k6T+wh5MkiRJkiRJaogJJkmSJEmSJDXEBJMkSZIkSZIaYoJJkiRJkiRJDTHBJEmSJEmSpIaYYJIkSZIkSVJDTDBJkiRJkiSpISaYJEmSJEmS1JBNWh2AJEmSJElqrrY5bWuWx7XVrSdtKHswSZIkSZIkqSEmmCRJkiRJktQQE0ySJEmSJElqiAkmSZIkSZIkNcQEkyRJkiRJkhpigkmSJEmSJEkNMcEkSZIkSZKkhphgkiRJkiRJUkNMMEmSJEmSJKkhJpgkSZIkSZLUkHUmmCJi8+6USZIkSZIkaWDqTg+mW7pZJklSt0XEThHxi4i4NyLmR8SJpXzbiJgdEQ+Uv0NKeUTEBRGxICLujojX1xxrcqn/QERMbtU1SZIkSQPVJvU2RMTfAMOBl0fEHkCUTVsBr+iF2CRJG7dVwCmZeVdEbAncGRGzgWOBn2fmFyLiNOA04NPAO4DR5bU3cDGwd0RsC5wJjAWyHGdmZi7r9SuSJEmSBqi6CSZgPNWH/BHAeaxJMD0DfKa5YUmSNnaZuRhYXJafiYj7qH7YmAiMK9WmA3OoEkwTgSsyM4FbI2KbiNih1J2dmUsBSpLqYOC7vXYxkiRJ0gBXN8GUmdOB6RHx3sz8fi/GJEkaYCJiJLAHcBuwfUk+ATwGbF+WhwOP1Oy2qJTVK+94jinAFICdd96554KXJEmS1K05mEZExFZl7otvRsRdEXFQ0yOTJA0IEbEF8H3gpMx8unZb6a2UPXGezLw0M8dm5thhw4b1xCElSZIkFd1JMH2wfOA/CBgKfAD4QlOjkiQNCBGxKVVy6crMvLYUP16GvlH+PlHKHwV2qtl9RCmrVy6ply1c2Lb6JUmSBpbuJJja5146hGrui/k1ZZIkbZCICOAy4L7M/HLNpplA+5PgJgPX15QfU3rU7gM8VYbS/QQ4KCKGlCfOHVTKJEmSJPWSrib5bndnRPwUGAWcXp7082Jzw5IkDQBvouoV+7uImFfKPkPVS/bqiDgeeBg4smy7gerHjgXAs8BxAJm5NCLOAu4o9T7fPuG3JEmSpN7RnQTT8cAY4MHMfDYihlI+1EuStKEy81fU7xF7YCf1E5ha51jTgGk9F50kSZKk9VE3wRQRf5eZv6dKLgH8bTWaQZIkSZIkSVqjqzmYTi5/z+vkde66DhwR0yLiiYi4p6asLSIejYh55XVIzbbTI2JBRNwfEeNryg8uZQsi4rT1vD5JkiRJkiQ1Wd0eTJk5pfx96wYe+3LgIuCKDuXnZ+ZaCaqI2AWYBOwK7Aj8LCJeWzZ/DXg7sAi4IyJmZua9GxiTJEmSJEmSelh35mAiIvYFRtbWz8yOiaO1ZOZNETGym3FMBGZk5vPAwohYAOxVti3IzAdLHDNKXRNMkiRJkiRJfURXQ+QAiIhvUw2J2w/Ys7zGNnDOj0bE3WUI3ZBSNhx4pKbOolJWr7yzOKdExNyImLtkyZIGwpMkSZIkSdL66E4PprHALuXpPY26GDgLyPL3POCDPXBcMvNS4FKAsWPH9kSskiRJkiRJ6oZ19mAC7gH+pidOlpmPZ+YLmfki8A3WDIN7FNippuqIUlavXJIkSZIkSX1E3R5METGLqqfRlsC9EXE78Hz79sw8bH1PFhE7ZObisvpuquQVwEzgOxHxZapJvkcDtwMBjI6IUVSJpUnA0et7XkmSJEmSJDVPV0Pkzu1i2zpFxHeBccB2EbEIOBMYFxFjqBJXDwH/BJCZ8yPiaqrJu1cBUzPzhXKcjwI/AQYB0zJzfiNxSZIkSZIkqWfVTTBl5v80cuDMPKqT4su6qH8OcE4n5TcANzQSiyRJkiRJkpqnO3MwSZIkSZIkSXWZYJIkSZIkSVJD1plgiogJEWEiSpIkSZIkSZ3qTuLofcADEfGliPi7ZgckSZIkSRExOCJuj4jfRsT8iPhcKR8VEbdFxIKIuCoiNivlm5f1BWX7yJpjnV7K74+I8a25IknauK0zwZSZ/xfYA/gDcHlE3BIRUyJiy6ZHJ0mSJGmgeh44IDN3B8YAB0fEPsAXgfMz8zXAMuD4Uv94YFkpP7/UIyJ2ASYBuwIHA1+PiEG9eiWSNAB0a+hbZj4NXAPMAHYA3g3cFREfa2JskvqAWbPWvCRJknpLVlaU1U3LK4EDqL6bAEwH3lWWJ5Z1yvYDIyJK+YzMfD4zFwILgL164RIkaUDpzhxMEyPiB8Acqpv6Xpn5DmB34JTmhidJkiRpoIqIQRExD3gCmE01qmJ5Zq4qVRYBw8vycOARgLL9KWBobXkn+9Sea0pEzI2IuUuWLGnG5UjSRm2TbtR5D1UX1JtqCzPz2Yg4vs4+kiRJktSQzHwBGBMR2wA/AJo2J2xmXgpcCjB27Nhs1nkkaWPVnSFyj3VMLkXEFwEy8+dNiUqSJEmSisxcDvwCeCOwTUS0/1A+Ani0LD8K7ARQtm8NPFlb3sk+kqQe0p0E09s7KXtHTwciSZIkSe0iYljpuUREvJzqe8l9VImmw0u1ycD1ZXlmWadsvzEzs5RPKk+ZGwWMBm7vnauQpIGj7hC5iPgI8M/AqyPi7ppNWwK/bnZgkiRJkga0HYDp5YlvLwOuzswfRsS9wIyIOBv4DXBZqX8Z8O2IWAAspXpyHJk5PyKuBu4FVgFTy9A7SVIP6moOpu8APwb+HTitpvyZzFza1KgkSZIkDWiZeTewRyflD9LJU+Ay8zngiDrHOgc4p6djlCSt0VWCKTPzoYiY2nFDRGxrkkmSJEmSJEmw7h5MhwJ3AglEzbYE/raJcUmSJEmSJKmfqJtgysxDy99RvReOJEmSJEmS+puuJvl+fVc7ZuZdPR+OJEmSJEmS+puuhsid18W2BA7o4VgkSZIkSZLUD3U1RO6tvRmIJEmSJEmS+qeuhsgdkJk3RsR7Otuemdc2LyxJkiRJkiT1F10NkXsLcCMwoZNtCZhgkgaYWbPWLE/o7M4gSZIkSRqQuhoid2b5e1zvhSNJkiRJkqT+5mXrqhARQyPigoi4KyLujIivRsTQ3ghOkiRJkiRJfd86E0zADGAJ8F7g8LJ8VTODkiRJkiRJUv/R1RxM7XbIzLNq1s+OiPc1KyBJkiRJkiT1L91JMP00IiYBV5f1w4GfNC8kSZIkSZLULG1z2tYsj2urW09aH3UTTBHxDNXT4gI4CfivsullwArgk02PTpIkSZIkSX1eV0+R27I3A5EkSZIkSVL/1J0hckTEEGA0MLi9LDNvalZQkiRJkiRJ6j/WmWCKiA8BJwIjgHnAPsAtwAHNDU2SJEmSJEn9wcu6UedEYE/g4cx8K7AHsLypUUmSJEmSJKnf6E6C6bnMfA4gIjbPzN8Dr2tuWJIkSZIkSeovujMH06KI2Aa4DpgdEcuAh5sbliRJkiRJkvqLdSaYMvPdZbEtIn4BbA38d1OjkiRJkiRJUr/R3afIvR7YD0jg15n516ZGJUmSJEmSpH5jnXMwRcS/AtOBocB2wLci4oxmByZJkiRJkqT+oTs9mN4P7F4z0fcXgHnA2c0MTJIkSZIkSf1Dd54i9ydgcM365sCjzQlHkiRJkiRJ/U3dBFNEXBgRFwBPAfMj4vKI+BZwD7C8twKUJG2cImJaRDwREffUlLVFxKMRMa+8DqnZdnpELIiI+yNifE35waVsQUSc1tvXIUmSJKnrIXJzy987gR/UlM9pWjSSpIHkcuAi4IoO5edn5rm1BRGxCzAJ2BXYEfhZRLy2bP4a8HZgEXBHRMzMzHubGbikDbdwYdvq5VGj2urWkyRJ/UvdBFNmTm9fjojNgPYP8vdn5spmByZJ2rhl5k0RMbKb1ScCMzLzeWBhRCwA9irbFmTmgwARMaPUNcEkSZIk9aJ1TvIdEeOoniL3EBDAThExOTNvam5okqQB6qMRcQxVT9pTMnMZMBy4tabOolIG8EiH8r07O2hETAGmAOy88849HbMkSVJD2ua0tToEqSHdmeT7POCgzHxLZu4PjAfOb25YkqQB6mLg1cAYYDFVG9QjMvPSzBybmWOHDRvWU4eVJEmSRDd6MAGbZub97SuZ+f8iYtMmxiRJGqAy8/H25Yj4BvDDsvoosFNN1RGseaJpvXJJkiRJvaQ7CaY7I+KbwH+V9fezZgJwSZJ6TETskJmLy+q7qZ5cCjAT+E5EfJlqku/RwO1UQ7dHR8QoqsTSJODo3o1a0oZywm9JkjYe3UkwnQBMBT5e1n8JfL1pEUmSBoSI+C4wDtguIhYBZwLjImIMkFRz//0TQGbOj4irqSbvXgVMzcwXynE+CvwEGARMy8z5vXwpkiRJ0oDXZYIpIgYBv83MvwO+3DshSZIGgsw8qpPiy7qofw5wTiflNwA39GBoknqAvZMkSRpYupzku/w6fH9E+LgdSZIkSZIkdao7Q+SGAPMj4nbgL+2FmXlY06KS1OfNmrVmecKE1sUhSZIkSWq97iSY/qXpUUiSJEmSJKnfqptgiojBVBN8vwb4HXBZZq7qrcCk/qi2V48kSZIkSQNFVz2YpgMrqZ4a9w5gF+DE3ghKkiRJG4/aCb8lSdLGqasE0y6Z+Y8AEXEZcHvvhCRJkiRJkqT+pKunyK1sX3BonCRJkiRJkurpqgfT7hHxdFkO4OVlPYDMzK2aHp0kSZIkSZL6vLoJpswc1JuBSJIkad2cz0iSJPVFXQ2RkyRJkiRJktbJBJMkSZIkSZIa0tUcTJIkSZIkaSPWNqdtzfK4trr1pHWxB5MkSZIkSZIaYoJJkiRJkiRJDTHBJEmSJEmSpIY4B5Okhs2atWZ5woTWxSFJkiRJao2m9WCKiGkR8URE3FNTtm1EzI6IB8rfIaU8IuKCiFgQEXdHxOtr9plc6j8QEZObFa8kSZIkSZI2TDOHyF0OHNyh7DTg55k5Gvh5WQd4BzC6vKYAF0OVkALOBPYG9gLObE9KSZIkSZIkqW9oWoIpM28ClnYonghML8vTgXfVlF+RlVuBbSJiB2A8MDszl2bmMmA2L01aSZIkSZIkqYV6e5Lv7TNzcVl+DNi+LA8HHqmpt6iU1St/iYiYEhFzI2LukiVLejZqSZIkSZIk1dWyp8hlZgLZg8e7NDPHZubYYcOG9dRhJUmSJEmStA69nWB6vAx9o/x9opQ/CuxUU29EKatXLkmSJEmSpD6itxNMM4H2J8FNBq6vKT+mPE1uH+CpMpTuJ8BBETGkTO59UCmTJEmSJElSH7FJsw4cEd8FxgHbRcQiqqfBfQG4OiKOBx4GjizVbwAOARYAzwLHAWTm0og4C7ij1Pt8ZnacOFySJEmSJEkt1LQEU2YeVWfTgZ3UTWBqneNMA6b1YGiSmmjWrDXLEya0Lg5JktS/RcROwBVUDwZK4NLM/GpEbAtcBYwEHgKOzMxlERHAV6l+uH4WODYz7yrHmgycUQ59dmZOR5LUo1o2ybckSZIkdWEVcEpm7gLsA0yNiF2A04CfZ+Zo4OdlHeAdwOjymgJcDFASUmcCewN7AWeW6TckST3IBJMkSZKkPiczF7f3QMrMZ4D7gOHARKC9B9J04F1leSJwRVZuBbYpDxYaD8zOzKWZuQyYDRzci5ciSQOCCSZJkiRJfVpEjAT2AG4Dti8PBAJ4jGoIHVTJp0dqdltUyuqVdzzHlIiYGxFzlyxZ0qPxS9JAYIJJkiRJUp8VEVsA3wdOysyna7eVuVyzJ86TmZdm5tjMHDts2LCeOKQkDSgmmCRJkiT1SRGxKVVy6crMvLYUP16GvlH+PlHKHwV2qtl9RCmrVy5J6kFNe4qcJEmSJG2o8lS4y4D7MvPLNZtmApOBL5S/19eUfzQiZlBN6P1UZi6OiJ8A/1YzsfdBwOm9cQ1Sf9M2p23N8ri2uvWkzphgkiRJktQXvQn4APC7iJhXyj5DlVi6OiKOBx4GjizbbgAOARYAzwLHAWTm0og4C7ij1Pt8Zi7tnUuQpIHDBJMkSZKkPiczfwVEnc0HdlI/gal1jjUNmNZz0UmSOnIOJkmSJEmSpP/f3v3GSnbW9wH//hzL0BKo7Zq4jm3wpl2U0iqhaGvzgoKpE2NcLSYBuSAhFuR2Req0L9pKuKISN/CippVagYJoXWRhI/EvSC7rhECMw4o2wqnNP2NowQabehfDbrHj0rptYvr0xZy1x+sZ79w7f86ZO5+PNLpnzn3u3O+cO/c+5/7meZ7DXBSYAAAAAJiLAhMAAAAAc1FgAgAAAGAuCkwAAAAAzEWBCQAAAIC5KDABAAAAMBcFJgAAAADmosAEAAAAwFxO7zsAAAAj99+/9cT2nj1bU9sBAAyNAhMAAL1TXAOA9WaKHAAAAABzUWACAAAAYC4KTAD0oqpurKpjVXXP2L6zq+q2qrq3+3hWt7+q6v1VdV9V3V1VLx37mgNd+3ur6kAfzwUAADadAhMAfflwkitO2nddkttba3uT3N7dT5LXJNnb3Q4m+WAyKkgleVeSS5JcnORdJ4pSAADA6igwAdCL1toXkzx80u6rktzUbd+U5HVj+29uI3ckObOqzkvy6iS3tdYebq09kuS2PL1oBQAALJkCEwBDcm5r7aFu+4dJzu22z0/y4Fi7I92+afufpqoOVtVdVXXX8ePHF5saAAA23Ol9BwCASVprraraAh/vhiQ3JMm+ffsW9rgAALvR1uGtJ7cv3ZraDk4wggmAIflRN/Ut3cdj3f6jSS4ca3dBt2/afgAAYIUUmAAYkkNJTlwJ7kCST4/tf0t3NbmXJXm0m0r3uSSXV9VZ3eLel3f7AACAFTJFDoBeVNXHklya5JyqOpLR1eCuT/LJqromyfeTXN01/0ySK5Pcl+SxJG9Lktbaw1X1niR3du3e3Vo7eeFwAABgyRSYAOhFa+1NUz512YS2Lcm1Ux7nxiQ3LjAaAACwTabIAQAAADAXBSYAAAAA5mKKHAAAg3L//VtPbO/ZszW1HQAwHApMAAAMlmITAKwHBSZgaW699cnt/fv7ywEAAMByWYMJAAAAgLkoMAEAAAAwF1PkAAAGyNpDAMA6MYIJAAAAgLkoMAEAAAAwFwUmAAAAAOaiwAQAAADAXBSYAAAAAJiLAhMAAAAAc1FgAgAAAGAup/cdAAAAADbR1uGtviPAwhjBBAAAAMBcFJgAAAAAmIsCEwAAAABzUWACAAAAYC4KTAAAAADMRYEJAAAAgLkoMAEAAAAwFwUmAAAAAOaiwAQAAADAXBSYAAAAAJiLAhMAAAAAczm97wAAAACwKbYOb/UdYdvGM29dujW1HZtNgQlYiVtvfXJ7//7+cgAAALB4psgBAAAAMBcFJgAAAADm0ssUuap6IMlPkvw0yeOttX1VdXaSTyS5KMkDSa5urT1SVZXkfUmuTPJYkre21r7SR24AAPpz//1bT2zv2bM1tR0AsHp9jmB6VWvtJa21fd3965Lc3lrbm+T27n6SvCbJ3u52MMkHV54UAAAAgKmGtMj3VUku7bZvSnI4yTu6/Te31lqSO6rqzKo6r7X2UC8pAQBWbHzkDgDAEPU1gqkl+YOq+nJVHez2nTtWNPphknO77fOTPDj2tUe6fU9RVQer6q6quuv48ePLyg0AAADASfoqML28tfbSjKa/XVtVrxj/ZDdaqW3nAVtrN7TW9rXW9j3/+c9fYFQAAGDVqurGqjpWVfeM7Tu7qm6rqnu7j2d1+6uq3l9V91XV3VX10rGvOdC1v7eqDvTxXAA2QS9T5FprR7uPx6rqliQXJ/nRialvVXVekmNd86NJLhz78gu6fQAAa8/0N5jqw0l+O8nNY/tOrNt6fVVd191/R566buslGa3bekl3IaF3JdmX0RvYX66qQ621R1b2LAA2xMpHMFXVc6rquSe2k1ye5J4kh5KceEfhQJJPd9uHkryle1fiZUketf4SQ3LrrU/eAABYjNbaF5M8fNLuqzJarzXdx9eN7b+5jdyR5MzuTetXJ7mttfZwV1S6LckVy08PsHn6GMF0bpJbqurE9/9oa+2zVXVnkk9W1TVJvp/k6q79Z5JcmeS+JI8ledvqI7PbjReH9u/vLwcAAM9ou+u2zrSeazJa0zWjq1bnBS94wQIjA2yGlReYWmvfS/LLE/b/OMllE/a3JNeuIBpLoHDDJF4XAMC8Wmutqra1buspHu+GJDckyb59+xb2uACbopc1mAAAYFHG17Has2drajt2he2u23o0yaUn7T+8gpwAG6evq8gBAABs13bXbf1cksur6qzuinOXd/sAWDAjmAAAgMGpqo9lNPronKo6ktHV4K7PNtZtba09XFXvSXJn1+7drbWTFw4HYAEUmAAAgMFprb1pyqe2tW5ra+3GJDcuMBoAE5giBwAAAMBcFJgAAAAAmIspcgAMTlU9kOQnSX6a5PHW2r6qOjvJJ5JclOSBJFe31h6pqkryvozW3ngsyVtba1/pIzcAwG63dXjrye1Lt6a2Y/MYwQTAUL2qtfaS1tq+7v51SW5vre1Ncnt3P0lek2RvdzuY5IMrTwoAABvOCCZW5tZbn9zev7+/HAyL1wXbcFVGVxNKkpuSHE7yjm7/zd0Cr3dU1ZlVdV53eWoAAGAFjGACYIhakj+oqi9X1cFu37ljRaMfJjm32z4/yYNjX3uk2/cUVXWwqu6qqruOHz++rNwAALCRjGACYIhe3lo7WlU/l+S2qvqv459srbWqatt5wNbaDUluSJJ9+/Zt62sBAIBnZgQTAIPTWjvafTyW5JYkFyf5UVWdlyTdx2Nd86NJLhz78gu6fQAAwIoYwcSut4w1fsYfE1isqnpOktNaaz/pti9P8u4kh5IcSHJ99/HT3ZccSvKbVfXxJJckedT6SwDAkIxfeQ12KwUmAIbm3CS3VFUy6qc+2lr7bFXdmeSTVXVNku8nubpr/5kkVya5L8ljSd62+sgAALDZFJhgRkYtwWq01r6X5Jcn7P9xkssm7G9Jrl1BNACAmRm1xKZRYBqgTbhs+yY8RwAAANgUCkxsLCOSAAAAYDEUmJjICKMRRSgAWC/337/1xPaePVtT2wEAi6XABADA2hkvJAEA/VNgAgC2zSgRAADGKTCxUUx5AwAAgMVTYNoFphVNxtdOGvKaSkPOBgAAAJyaAhODMkuxDACWxdQ/AICdUWACBsNoNgAAgPWkwAQnsU4TQ6LoxiLMMirHyB0A2Jmtw1t9R4BBUGBirfnnGwAAAPqnwAQAMIFRXQAAs1Ng6pHRNwAAAKyr8emBW5duTW3HZlBgAgDYoZ2Ochr/OgCA3UCBiVPa7kirlcXRFwAADRBJREFUZSySbbQXAAAwFBb2hqdTYGIhhnbltaHlAaB/1lTaPH7mALA6CkzsGopKAADAohmtBLNRYAIAnmLa+kDTRoBswiiRTXiOAADzUGACgA2iUPJ0FtwGAJifAtMutlunjO3W5wXA7qaQBQDsZgpMAzft6mmLKrK4OhsAy7Jpo6U27fmuGz8fAFguBSZgkBQ/AQAA1ocC0xpZxqglAJjVbpriNc9z2U3HAQBgURSYAGBDKZQsn2MMAGwKBSZ2zEgoVsV0OQAAgGFTYOIJCkYAAADsxNbhrSe3L92a2o7dS4EJYGAUewEA+jVeLAFmc1rfAQAAAABYb0YwAQAsmMW918f4z2rPnq2p7QCAZ6bANBCmxAAAALAbWI9pM5kiBwAAAMBcjGACABZmu1PDTE8CYCgs7A3zUWACAJZuu4UkhSeWSSEUABZPgWnFrLUE8xn/Hdq/v78cMBTT/vEd8iLTQ84GAMDOKDCxLQpkAMOlcAMA22NaHCyOAhNATxRsAQDY7VxRbnMoMAEAu4qRXOyUtZYAYOcUmFbAKAUAAABgNzut7wAAAAAArDcjmABmsKir1xnRCNtn2hJ92O5US69TADadAhOwthZV9FkXm/Z82TzWTgJgFVw5rj8W/N7dFJgAAGBJjGwCmEyxafdRYAKYg1FFsFrTRjn5x50+GHUHsFyKUOtFgQlgimnrJc2yX7GJ7TDCAdaHohKsJ9Pihm3ZPx+FqtVQYAJ2haEVd4aWB3Y7//QzJLO8HhWWAXZuWkFK8ahfCkwAY1zlDYBVM/UTlsOopd1llp/nLCOVjGZaHgUmYNeZZfTQKgtJilYA7IRRTrB9ikqcsN2C1FP2KzztyNoUmKrqiiTvS/IzST7UWru+50jAGlDc2Rz6CWDoTOXsl35idzEKhWWapfDkNfh0a1FgqqqfSfKBJL+a5EiSO6vqUGvtW6vKYD0VgOEaQj+xKLP8A2okA2yeaaOZTK+bzW7qJzbZ1H/6jVpiRWZ5DW5ysWktCkxJLk5yX2vte0lSVR9PclWShXcICkkAa2ll/QRA34yE2hH9RI+m/fOtMMRutMnFpnUpMJ2f5MGx+0eSXDLeoKoOJjnY3f2fVfXtGR/7nCT/fe6EyyHbzgw121BzJbLt1Lpme+Eqg6zIMvuJZHA/6986VYOB5T0leZdL3uUaaN6pfydmyauf2H4/MauBvl5mspDsv3XqPmwZNv6492Tjs/fwel/FMZ/aR6xLgemUWms3JLlhu19XVXe11vYtIdLcZNuZoWYbaq5Etp2Sbb3stJ9I1u94yrtc8i6XvMu1bnlXaZ5+YlbrfPxl74fs/VjX7H3nPq2vb7xNR5NcOHb/gm4fACT6CQCemX4CYMnWpcB0Z5K9VbWnqs5I8sYkh3rOBMBw6CcAeCb6CYAlW4spcq21x6vqN5N8LqPLit7YWvvmgh5+qcNg5yTbzgw121BzJbLtlGwDseR+Ilm/4ynvcsm7XPIu17rlXYgV9BOzWufjL3s/ZO/HumbvNXe11vr8/gAAAACsuXWZIgcAAADAQCkwAQAAADCXjSkwVdUVVfXtqrqvqq6b8Pl/XFXfqqq7q+r2qnrhgLK9vaq+UVVfq6r/VFUvHkKusXavr6pWVSu7HOIMx+ytVXW8O2Zfq6q/N5RsXZuru9fbN6vqo0PJVlX/ZuyYfaeq/mRA2V5QVV+oqq92v6dXDijbC7u/G3dX1eGqumBFuW6sqmNVdc+Uz1dVvb/LfXdVvXQVudbZkPuKSYbaf0wz5H5lSo7B9jWTDLn/mZJlsH3SJEPupyYZat+1aarq7Kq6raru7T6eNaHNC6vqK91r/ZtV9fY+sp5sxuwvqaovdbnvrqq/20fWk82SvWv32ar6k6r63VVnPCnHqX5fn1VVn+g+/8dVddHqU042Q/ZXdK/vx6vqDX1knGbdzvvGDfYcsLW2628ZLeT33SS/kOSMJF9P8uKT2rwqyZ/vtn8jyScGlO15Y9uvTfLZIeTq2j03yReT3JFk34CO2VuT/PZAX2t7k3w1yVnd/Z8bSraT2v/DjBbAHES2jBas+41u+8VJHhhQtt9JcqDb/ttJPrKibK9I8tIk90z5/JVJfj9JJXlZkj9eRa51vQ25r5gj78r7j3nydu1W3q/McXx76WvmyNtL/zPP62Gs/cr6pDmOby/91Bx5e+m7Nu2W5F8mua7bvi7Jeye0OSPJs7rtn03yQJKfX5PsL0qyt9v++SQPJTlzHbJ3n7ssyf4kv9tj1ll+X/9Bkn/bbb8xPZ5/7CD7RUl+KcnNSd7Qd+ZtZh/Med8OsvdyDrgpI5guTnJfa+17rbU/TfLxJFeNN2itfaG19lh3944kq3oXZ5Zs/2Ps7nOSrGJl9lPm6rwnyXuT/J8VZNputj7Mku3vJ/lAa+2RJGmtHRtQtnFvSvKxlSSbLVtL8rxu+y8k+cGAsr04yR9221+Y8PmlaK19McnDz9DkqiQ3t5E7kpxZVeetItuaGnJfMclQ+49phtyvTDLkvmaSIfc/kwy5T5pkyP3UJIPtuzbQVUlu6rZvSvK6kxu01v60tfZ/u7vPynBmmcyS/TuttXu77R8kOZbk+StLON0psydJa+32JD9ZVagpZvl9HX8+n0pyWVXVCjNOM8u5yAOttbuT/L8+Aj6DdTvvGzfYc8Ch/PFatvOTPDh2/0i3b5prMnrXfxVmylZV11bVdzOqxv+jIeSq0XSbC1trv7eCPONm/Xm+vhvO+KmqunA10WbK9qIkL6qqP6qqO6rqigFlSzIaqp1kT5488Vy2WbJtJXlzVR1J8pmM3s1ehVmyfT3Jr3fbv5bkuVX1F1eQ7VS2+7dv0w25r5hkqP3HNEPuVyYZcl8zyZD7n0mG3CdNMuR+apJ17rt2m3Nbaw912z9Mcu6kRlV1YVXdndHP7b1dsaZvM2U/oaouzmg0xXeXHWwG28res1l+X59o01p7PMmjSYbw+7rO55rrdt43brDngJtSYJpZVb05yb4k/6rvLONaax9orf3lJO9I8s/7zlNVpyX510n+Sd9Zprg1yUWttV9KcluerPgPwekZTVO4NKN3ZP99VZ3Za6Kne2OST7XWftp3kDFvSvLh1toFGU39+kj3OhyCf5rklVX11SSvTHI0yZCOHQs21L5ikqH1H9OsQb8yyZD7mknWof+ZZIh90iRD7qcm0XctSFV9vqrumXA7eURBy5RRBK21B7u/JX8lyYGqWklBZBHZu8c5L8lHkryttbaSkSqLyg6nsk7nfeP6OAc8fRXfZACOJhl/V/GCbt9TVNWvJHlnkleODVMdRLYxH0/ywaUmGjlVrucm+etJDnejM/9SkkNV9drW2l09Z0tr7cdjdz+UUdV2FWb5eR7JaC2cP0tyf1V9J6MT/jsHkO2ENya5dsl5xs2S7ZokVyRJa+1LVfXsJOdkNBS712zdu4y/niRV9bNJXt9a63Ux2s52/75suiH3FZMMtf+YZsj9yiRD7msmGXL/M8mQ+6RJhtxPTbLOfdfaaa39yrTPVdWPquq81tpDXRHmGV8PrbUf1OjiHX8ro6lQS7WI7FX1vCS/l+Sd3ZT8lVjkce/ZLH9fTrQ5UlWnZzQN98fp3zqfa67bed+4wZ4DDvldlUW6M8neqtpTVWdkdKJyaLxBVf2NJP8uyWtXvCbBLNn2jt39O0nu7TtXa+3R1to5rbWLWmsXZTQndVX/BMxyzMbXmXltkv+yglwzZUvyHzJ69zhVdU5GUxa+N5BsqapfTHJWki+tINN2sv23jBZiTFX91STPTnJ8CNmq6pyxd6n/WZIbV5BrFoeSvKVGXpbk0bHh4jzdkPuKSYbaf0wz5H5lkiH3NZMMuf+ZZMh90iRD7qcmWee+a7c5lORAt30gyadPblBVF1TVn+u2z0ry8iTfXlnC6WbJfkaSWzJa83HpBbFtOGX2AZnl78v483lDkj/sRmb1baa/5QO1bud944Z7DtgGsAr6Km4ZDVX+TkZzgt/Z7Xt3Ri+WJPl8kh8l+Vp3OzSgbO9L8s0u1xeS/LUh5Dqp7eGs8Go/Mxyzf9Eds693x+wXB5StMpoG8q0k30jyxqFk6+5vJbl+VZm2cdxenOSPup/p15JcPqBsb8joj/Z3MhrF8KwV5fpYRldr+bOMRiZck+TtSd4+9lr7QJf7G6v8HV3X25D7ih3m7aX/2Gnek9qutF/Z4fHtra/ZYd7e+p+dvh766pN2eHx766d2mLeXvmvTbhmtk3N7d6w/n+Tsbv++JB/qtn81yd3da+fuJAf7zr2N7G/O6Dzka2O3l6xD9u7+f8yoEPy/MzqXenVPeU/1+/rsjK78eF+S/5zkF/o+xtvI/je7Y/u/Mhp19c2+M28j+6DO+7aZvZdzwOq+OQAAAADsyKZMkQMAAABgSRSYAAAAAJiLAhMAAAAAc1FgAgAAAGAuCkwAAAAAzEWBCQAAAIC5KDABAAAAMJf/DygP47UdXfJpAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "tm_ctrl = TwoModels(\n", " estimator_trmnt=CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True), \n", " estimator_ctrl=CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True), \n", " method='ddr_control'\n", ")\n", "tm_ctrl = tm_ctrl.fit(\n", " X_train, y_train, treat_train,\n", " estimator_trmnt_fit_params={'cat_features': cat_features}, \n", " estimator_ctrl_fit_params={'cat_features': cat_features}\n", ")\n", "\n", "uplift_tm_ctrl = tm_ctrl.predict(X_val)\n", "\n", "tm_ctrl_score = uplift_at_k(y_true=y_val, uplift=uplift_tm_ctrl, treatment=treat_val, strategy='by_group', k=0.3)\n", "\n", "models_results['approach'].append('TwoModels_ddr_control')\n", "models_results['uplift@30%'].append(tm_ctrl_score)\n", "\n", "plot_uplift_preds(trmnt_preds=tm_ctrl.trmnt_preds_, ctrl_preds=tm_ctrl.ctrl_preds_);" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "gsHLrueoCIP2" }, "source": [ "Similarly, you can first train the $P^T$ classifier, and then use its predictions as a feature for the $P^C$ classifier." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:48.818365Z", "start_time": "2020-05-30T22:29:46.838201Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 444 }, "colab_type": "code", "id": "gwcyMZogCIP4", "outputId": "91583f7c-c48d-40f2-c375-0a8d7cf69d9c" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABJgAAAGrCAYAAACMgi0UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzde5wcVZnw8d9juATlFkKWhQQ2UaMuuCRIEETUCHIRCbgKGvCVgGgWF7kIiqDsMgrsqgsiF4VFiQQXDYgICeIFhSwiNxONyPU1kCiJAWIuQOQFEnjeP+pM0gwzk8n09PRM5vf9fPozVadOVT3VgTrdT59zKjITSZIkSZIkqbte1ewAJEmSJEmS1L+ZYJIkSZIkSVJdTDBJkiRJkiSpLiaYJEmSJEmSVBcTTJIkSZIkSaqLCSZJkiRJkiTVxQSTtB6IiJaI+J+yvENErIiIQd04zucj4ts9H6Ekrd8iYn5EvKdJ574iIs4uy++IiIe7eZxLI+LfejY6Seq/Oru/RsQbI2JORDwTESc0L8rV8YyPiAU16/dHxPhuHKfb7YhkgkkNVRIdra+XIuL/1ax/pAfPs/rm39siYmREZERs0Izzt5WZf87MTTPzxc7qtW2Eyr7/kZkfb2yEktQ9EXFERMwqbciiiPhJROzVA8dtWhvS0zLzV5n5xrXVi4ijIuL2Nvsem5lnNS46Sepd5TP669uUrf5hdl20c389Fbg1MzfLzAub+UNDezJzp8ycubZ6bd+jrrYjUntMMKmhSqJj08zcFPgzMKGm7KrWen0lOdMX+F5I0itFxMnA14H/ALYBdgC+CRzSC+futfuybYAk9Rv/ANzfqIN3ZzSC1GwmmNQUrb1nIuJzEfE48J2IeFVEnBYRj0TEkoi4JiK2qtnnBxHxeEQ8FRG3RcROpXwy8BHg1PKr9oxSPj8iPhsR90bE3yLi8ojYpvzi/UxE/CIihtQcf4+IuCMilkfE72u7lEbEzIg4KyJ+Xfb9eURsXTbfVv4uL+d/WzvX2xIR10bE1WX/30bEmJrt88t7cS/wt4jYYC3xjIqI/y3HuhnYumbby3pURcRWEfGdiPhLRCyLiOsj4jXAT4DtanqUbdf2F52IOLh0r11e3oN/bBPzZ8r7+1S5tsFl29YRcWPZb2lE/CoivN9I6paI2AL4EnBcZl6XmX/LzJWZOSMzP1vqbBwRXy/3ur+U5Y3LttY255SIeLL0fjq6bOusDWl7X+7wnriW+K+IavjZzeW+/b8R8Q812zMijouIPwJ/LGUHRTX0YnlpC3auqb9LaUeeiYirgcE129oOkdg+Iq6LiMVRta0Xl7gvBd5Wrnl5TZxn1+z7iYiYW+7j0yNiuzYxHxsRfywxfiMiomx7fbnGpyLiryVGSepzatqHz5f71fzoYJRF7f01Im4B3g1cXO6j36f64WNGWT91Xc9V7sGXRMRNEfE34N3l8/kPyz18XtQMxYuITco+yyLiAWC3Nudb3aMqIgaV8z5S2o7ZpX1o/R7z+xL3h9tpR/6xtHnLSxt4cJuYvxERPy7HvTsiXle2RUScX9rdpyPiDxHx5nX8J1I/4xc+NdPfA1tRZf8nA8cD7wfeBWwHLAO+UVP/J8Bo4O+A3wJXAWTmZWX5q6Vn1ISafT4I7Au8AZhQjvF5YBjVf/8nAETEcODHwNklps8AP4yIYTXHOgI4upx/o1IH4J3l75bl/Hd2cL2HAD8ox/8ecH1EbFiz/XDgfcCWVL/OdxbP94DZVImls4BJHZwT4LvAq4GdSuznZ+bfgPcCf6npUfaX2p0i4g3A94GTyvt1E1WjuVFNtQ8BBwCjgJ2Bo0r5KcCCst82VO95dhKjJHXmbVRJlB91UucLwB7AWGAM8FbgjJrtfw9sAQwHjgG+ERFD1tKG1N6XX8va74md+QjV/XprYE45Z633A7sDO0bELsAU4F+AocB/A9OjSqJtBFxPdW/fiqpd+WB7J4zq1+8bgT8BI8u1T8vMB4FjgTvLNW/Zzr57A/9JdZ/fthxjWptqB1F9odm51Nu/lJ8F/BwYAowALur8rZGkpvp7qnvzcKrP1JdFRKdDxDJzb+BXwKfKffRwXj5a46vdPNcRwDnAZsAdwAzg96X+PsBJEdF6rz0TeF157U/n3wdOpmrTDgQ2Bz4GPJuZrd9jxpS4X/aDQPmuMoPqnv53VN/XrmoT80Tgi1T3/LklfoD9qL4nvYGq/f0QsKSTGLUeMMGkZnoJODMzn8/M/0f1YfcLmbkgM58HWoBDo/TEycwpmflMzbYxUf2q3ZmLMvOJzFxI1QjcnZm/y8znqL6o7FLq/R/gpsy8KTNfysybgVlUN+FW38nM/1tivYbqS8y6mJ2Z12bmSuBrVF+W9qjZfmFmPlaO32E8EbED1Qf6fyvv3W1UN/5XiIhtqRJJx2bmsvKL//92Md4PAz/OzJtLzOcCmwB7ton5L5m5tMTQ+p6spPpC8g/lnL/KTBNMkrprKPDXzFzVSZ2PAF/KzCczczHVh92P1mxfWbavzMybgBXA2uaYqL0vd+We2JkfZ+ZtpQ37AlXvoe1rtv9nZi4t55oM/Hdm3p2ZL2bmVOB5qjZjD2BD4OvlWq4FftPBOd9K9YPNZ0uvr+cy8/YO6rb1EWBKZv62xHx6iXlkTZ0vZ+byzPwzcCsvbwP+AdhuHc8pSc3S+rn6f6l+5P1Qk851Q2b+OjNfAv4JGJaZX8rMFzLzUeBbVAkdyn7nlLbjMeDCTs75ceCMzHw4K7/PzK4ke/YANqW637+QmbdQ/XBxeE2dH2XmPaWNvoqXtwWbAW8CIjMfzMxFXTin+jETTGqmxSXR0+ofgB+V7pfLgQeBF4FtSrfOL5dunU8D88s+W9O5J2qW/18765vWnPuw1nOX8+9FlSRp9XjN8rM1+3bVY60LpdFYQPXB/xXb1xLPdsCy0gup1Z86OOf2wNLMXLaOsVLOs/q4JebHqH5BadXRe/JfVL9g/DwiHo2I07pxfklqtQTYOjqfn+hl96yyXHuPXdImQdWV+3jtfbkr98QuHSszVwBL6bwNOKVNG7B9qb8dsLBN0r6zNuBPa0nMdaTt9a6g+nfoShtwKhDAPWU4xce6cX5J6gkvUiXla21Ilfxo1d7n6u1ojLWdq21bsF2btuDzVKMDKPvV1u+oLYCqPXikG/FuBzxW2rza86y1LSjJqIupRqQ8GRGXRcTm3YhB/YgJJjVT2x4tjwHvzcwta16DS++jI6iGmL2HqovlyLJPdHCsdfUY8N02535NZn65G9fRkdW/VEc1H9EIoHZYWu1xOotnETAkqnmUWu3QwTkfA7aKiFcMf+hC3H+hathaY45yDQvXsh+lp9kpmfla4GDg5IjYZ237SVIH7qTqwfP+Tuq87J5FdV/8Swd12+roflhb3u17YlHbBmxKNbytszbgnDZtwKsz8/tUbcDwcv5WnbUBO3SQmFvXNuA1VD3JutIGPJ6Zn8jM7aiG+X0z2jzFSZJ6yZ9Z872h1Shenoxp73N1V9uPWl35TrC2c7VtC+a1aQs2y8zWERaLqGlb6LgtaD3W67oQX1t/AbaPl8+lugNdbPsy88LM3BXYkWqo3Ge7EYP6ERNM6ksuBc6JMvFpRAyLiNanA21G9eViCdV8Qv/RZt8nqObH6K7/ASZExP6lt9TgMsHdiC7su5hquN/azr9rRHygfNA/iep67lrXeDLzT1TD5b4YERtF9YjuCe0dpHRD/QnVh/shEbFhRLSOtX4CGNrJMMNrgPdFxD5l/PUpJeY71nKdrZPTvr58AXqK6tejl9aymyS1KzOfAv6dat6k90fEq8v97L0R0TrPxfeBM0rbsXWp39XHUHelDen2PbE4MCL2KnMonQXcVYY0tOdbwLERsXuZJPU1EfG+iNiMKtm2CjihvAcfoBoK1557qL6AfLkcY3BEvL3mmkd0MofU94GjI2JsVJOl/wfVMPP5a7vQiDispv1cRvWFyTZAUjNcTdU2jIjqgULvofrcfG2beq2fq99BNb/cD7pxrq5+H+nque4BnonqgROblO8Eb46I1sm8rwFOL5/xR1DNj9SRbwNnRcTo0q7sHBFDuxD33VS9kk4tbc54qvev7Zx8rxARu5V2bEPgb8Bz2Bas90wwqS+5AJhONazqGarky+5l25VUvzQsBB7glYmZy6kmRl0eEdev64nLh/xDqLqdLqbK8n+WLvw/kpnPUk1m9+ty/j06qHoD1Rwey6jmBflAmcejO/EcQfXeLKWa4O/KTkL8KFU34IeAJ6mSW2TmQ1RfIB4tcb+sK3BmPkw1F9RFwF+pGpMJmflCJ+dqNRr4BdUcJ3cC38zMW7uwnyS1KzPPo5qk9AzW3Bc/RTXhNVQPRZgF3Av8gephEGe/8kjtWmsbUuc9EaqHM5xJdd/etRyrXZk5C/gE1dCCZVRDjo8q214APlDWl1K1K9d1cJwXS5yvp/oVf0GpD3AL1eO1H4+Iv7az7y+AfwN+SJWkeh1r5v1Ym92AuyNiBVW7fmKZO0SSetuXqH4IuJ3qfvpV4COZeV9NncfLtr9QzSF0bPmcvK7+kyqZtTwiPtNBnS6fq9zDD6Ka02geVdvzbarRHFDNNfinsu3nVA9/6MjXqBJSPweepmr3NinbWoCpJe6XzT1V2pwJVHO6/hX4JnBkF9+fzal+MFlW4lxCNY2G1mORzrsrNVxEtACvz8wOv1BIktZPEXEFsCAzz1hbXUlS7yk9cv4nM7syaqHfnEtqFnswSZIkSZIkqS4mmCRJkiRJklQXh8hJkiRJ6pPKk3C/DbyZasL4jwEPU03ePBKYD3woM5eVh4tcABxINTHxUZn523KcSVRzuAGcnZlTe/EyJGlAsAeTJEmSpL7qAuCnmfkmYAzwIHAa8MvMHA38sqxDNRHx6PKaDFwCEBFbUU2yvzvVUxfPjIghvXkRkjQQrJc9mLbeeuscOXJks8OQpD5p9uzZf83MYc2Oo5lsJySpY32lnYiILYA5wGuz5ktLRDwMjM/MRRGxLTAzM98YEf9dlr9fW6/1lZn/UspfVq89thOS1L7O2ogNejuY3jBy5EhmzZrV7DAkqU+KiD81O4Zms52QpI71oXZiFLAY+E5EjAFmAycC22TmolLncWCbsjwceKxm/wWlrKPyl4mIyVQ9n9hhhx1sJySpHZ21EQ6RkyRJktQXbQC8BbgkM3cB/saa4XAAlJ5NPTIkIzMvy8xxmTlu2LCmd+CSpH7HBJMkSZKkvmgBsCAz7y7r11IlnJ4oQ+Mof58s2xcC29fsP6KUdVQuSepBJpgkSZIk9TmZ+TjwWES8sRTtAzwATAcmlbJJwA1leTpwZFT2AJ4qQ+l+BuwXEUPK5N77lTJJUg9aL+dgkjQwrFy5kgULFvDcc881O5Q+afDgwYwYMYINN9yw2aFIUpd4X+9d/aSdOB64KiI2Ah4Fjqb6kfyaiDgG+BPwoVL3JuBAYC7wbKlLZi6NiLOA35R6X8rMpb13CZL6u4HYPnWnjTDBJKnfWrBgAZttthkjR44kIpodTp+SmSxZsoQFCxYwatSoZocjSV3ifb339Jd2IjPnAOPa2bRPO3UTOK6D40wBpvRsdJIGioHWPnW3jXCInKR+67nnnmPo0KED4ia/riKCoUOHDqhfWST1f97Xe4/thCR13UBrn7rbRphgktSvDZSbfHf43kjqj7x39R7fa0nquoF2z+zO9ZpgkiRJkiRJUl2cg0nSemPGjJ493oQJnW9fsmQJ++xTTQHx+OOPM2jQIIYNGwbAPffcw0YbbbTO55w5cyYbbbQRe+655zrtN3LkSGbNmsXWW2+9zueUpL5q3ryWHj3eqFFrP15EcPLJJ3PeeecBcO6557JixQpaWjre9/rrr+cNb3gDO+644zpt647uHm/TTTdlxYoVPRKDJA10LTNbevZ449d+vPnz53PQQQdx3333rdmvpYVNN92Uz3zmM+3uc8UVVzBr1iwuvvhiLr30Ul796ldz5JFH8tBDDzFx4kQigmuvvZa7776bI444ou7rsAeTJHXT0KFDmTNnDnPmzOHYY4/l05/+9Or1jTbaiFWrVq3zMWfOnMkdd9zRgGglSV2x8cYbc9111/HXv/61y/tcf/31PPDAA+u8rTvtRGfHkySpI8ceeyxHHnkkULUlhx56KL/73e947LHH+N73vtcj5zDBJEk96KijjuLYY49l991359RTT+WRRx7hgAMOYNddd+Ud73gHDz30EAAzZsxg9913Z5ddduE973kPTzzxBPPnz+fSSy/l/PPPZ+zYsfzqV79i8eLFfPCDH2S33XZjt91249e//jVQ9Z7ab7/92Gmnnfj4xz9O9eAcSVK9NthgAyZPnsz555//im3z589n7733Zuedd2afffbhz3/+M3fccQfTp0/ns5/9LGPHjuWRRx5ZXb+9bePHj+ekk05i3LhxXHDBBcyePZt3vetd7Lrrruy///4sWrQIgG9961vstttujBkzhg9+8IM8++yz7R6vo3Zm3rx5vO1tb+Of/umfOOOMM3rnzZMkNcX48eM58cQTGTt2LG9+85u55557XlGnpaWFc889l5tuuomvf/3rXHLJJbz73e/mtNNO41e/+hVjx45tt+1bFw6Rk6QetmDBAu644w4GDRrEPvvsw6WXXsro0aO5++67+dd//VduueUW9tprL+666y4igm9/+9t89atf5bzzzuPYY499WTfXI444gk9/+tPstdde/PnPf2b//ffnwQcf5Itf/CJ77bUX//7v/86Pf/xjLr/88iZftSStP4477jh23nlnTj311JeVH3/88UyaNIlJkyYxZcoUTjjhBK6//noOPvhgDjroIA499NCX1d9zzz3b3fbCCy8wa9YsVq5cybve9S5uuOEGhg0bxtVXX80XvvAFpkyZwgc+8AE+8YlPAHDGGWdw+eWXc/zxx7/ieB21MyeeeCKf/OQnOfLII/nGN77R4HdMktRszz77LHPmzOG2227jYx/72MuG0tU68MADX/adY+bMmZx77rnceOONdcdggkmSethhhx3GoEGDWLFiBXfccQeHHXbY6m3PP/88UCWhPvzhD7No0SJeeOEFRo0a1e6xfvGLX7xsKMTTTz/NihUruO2227juuusAeN/73seQIUMaeEWSNLBsvvnmHHnkkVx44YVssskmq8vvvPPO1ffej370o69IQHXVhz/8YQAefvhh7rvvPvbdd18AXnzxRbbddlsA7rvvPs444wyWL1/OihUr2H///V9xnM7amV//+tf88Ic/XB3r5z73uW7FKknqGzp6qltr+eGHHw7AO9/5Tp5++mmWL1/ea7G1MsEkST3sNa95DQAvvfQSW265JXPmzHlFneOPP56TTz6Zgw8+mJkzZ3Y4eexLL73EXXfdxeDBgxsZsiSpjZNOOom3vOUtHH300T1+7NZ2IjPZaaeduPPOO19R56ijjuL6669nzJgxXHHFFcycOfMVdTprZ2DgPVJbktZnQ4cOZdmyZS8rW7p06eofqtve85vRBjgHkyQ1yOabb86oUaP4wQ9+AFRfJH7/+98D8NRTTzF8+HAApk6dunqfzTbbjGeeeWb1+n777cdFF120er31S8Q73/nO1ZPx/eQnP3lFYyNJqs9WW23Fhz70oZcNQd5zzz2ZNm0aAFdddRXveMc7gFfeu2t1tu2Nb3wjixcvXp1gWrlyJffffz8AzzzzDNtuuy0rV67kqquuavd4nbUzb3/7218WqySpf9t0003ZdtttueWWW4AqufTTn/6UvfbaC4Crr74agNtvv50tttiCLbbYokvH7aydWlf2YJK03pgwodkRvNJVV13FJz/5Sc4++2xWrlzJxIkTGTNmDC0tLRx22GEMGTKEvffem3nz5gEwYcIEDj30UG644QYuuugiLrzwwtVzgaxatYp3vvOdXHrppZx55pkcfvjh7LTTTuy5557ssMMOTb5SSep5o0a1NPX8p5xyChdffPHq9Ysuuoijjz6a//qv/2LYsGF85zvfAWDixIl84hOf4MILL+Taa6/lda973ep92m6rtdFGG3Httddywgkn8NRTT7Fq1SpOOukkdtppJ8466yx23313hg0bxu677776w3/b43XUzlxwwQUcccQRfOUrX+GQQw7phXdLkgaOlvEtTTnvlVdeyXHHHcfJJ58MwJlnnrm6zRk8eDC77LILK1euZMqUKV0+5s4778ygQYMYM2YMRx11FJ/+9Ke7HV+sj08eGjduXM6aNavZYUhNNWPGmuW+mHjpCQ8++CD/+I//2Oww+rT23qOImJ2Z45oUUp9gO6H+Zt68ltXLzU66NJL39d5nO9E+2wn1By0zW9YsNynhMVD09fZp/PjxnHvuuYwb17O37nVtIxwiJ0lqqogYFBG/i4gby/qoiLg7IuZGxNURsVEp37iszy3bR9Yc4/RS/nBEvHImXEmSJEkNZYJJktRsJwIP1qx/BTg/M18PLAOOKeXHAMtK+fmlHhGxIzAR2Ak4APhmRAzqpdglSZKkppo5c2aP917qDhNM0npkxow1r4FifRzm21P6w3sTESOA9wHfLusB7A20TlQyFXh/WT6krFO271PqHwJMy8znM3MeMBd4a+9cgaSe1h/uXesL32tJ6rqBds/szvU6ybekfmvw4MEsWbKEoUOH+ijmNjKTJUuWMHjw4GaHsjZfB04FNivrQ4HlmbmqrC8Ahpfl4cBjAJm5KiKeKvWHA3fVHLN2n9UiYjIwGXBSdKmP8r7ee/pROyGpRu28S+o9A6196m4bYYJJUr81YsQIFixYwOLFi5sdSp80ePBgRowY0ewwOhQRBwFPZubsiBjf6PNl5mXAZVBN3tro80lad97Xe1dfbyckqa8YiO1Td9oIE0yS+q0NN9yQUaNGNTsMdd/bgYMj4kBgMLA5cAGwZURsUHoxjQAWlvoLge2BBRGxAbAFsKSmvFXtPpL6Ee/rkqS+yPapa5yDSZLUFJl5emaOyMyRVJN035KZHwFuBQ4t1SYBN5Tl6WWdsv2WrAaHTwcmlqfMjQJGA/f00mVIkiRJwh5MkqS+53PAtIg4G/gdcHkpvxz4bkTMBZZSJaXIzPsj4hrgAWAVcFxmvtj7YUuSJEkDlwkmSVLTZeZMYGZZfpR2ngKXmc8Bh3Ww/znAOY2LUJIkSVJnHCInSZIkSZKkuphgkiRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUFxNMkiRJkiRJqotPkZMGgBkz1ixPmNC8OCRJkiRJ6ycTTJIkSf3IvHktq5dHjWrpsJ4kSVJvMsEk9XO1vZMkSZIkSWoG52CSJEmSJElSXUwwSZIkSZIkqS4mmCRJkiRJklQXE0ySJEmSJEmqiwkmSZIkSZIk1cUEkyRJkiRJkupigkmSJEmSJEl1McEkSZIkSZKkumzQ7AAkSZIkSVL3tcxsWbM8vqXDelIj2YNJkiRJkiRJdTHBJEmSJEmSpLo4RE7qh2bMaHYEkiRJkiStYQ8mSZIkSZIk1cUEkyRJkiRJkupigkmSJEmSJEl1McEkSZIkSZKkuphgkiRJkiRJUl18ipzUh9U+LW7ChObFIUmSJKk5Wma2NDsEqUvswSRJkiRJkqS6mGCSJEmSJElSXUwwSZIkSZIkqS4NSzBFxPYRcWtEPBAR90fEiaW8JSIWRsSc8jqwZp/TI2JuRDwcEfvXlB9QyuZGxGmNilmSJEmSJEnrrpGTfK8CTsnM30bEZsDsiLi5bDs/M8+trRwROwITgZ2A7YBfRMQbyuZvAPsCC4DfRMT0zHyggbFLA4KTiEuSJEmSekLDEkyZuQhYVJafiYgHgeGd7HIIMC0znwfmRcRc4K1l29zMfBQgIqaVuiaYJEmSJEmS+oBemYMpIkYCuwB3l6JPRcS9ETElIoaUsuHAYzW7LShlHZW3PcfkiJgVEbMWL17cw1cgSZIkSZKkjjQ8wRQRmwI/BE7KzKeBS4DXAWOpejid1xPnyczLMnNcZo4bNmxYTxxSkiRJUhNFxPyI+EOZu3VWKdsqIm6OiD+Wv0NKeUTEhWXe1nsj4i01x5lU6v8xIiY163okaX3W0ARTRGxIlVy6KjOvA8jMJzLzxcx8CfgWa4bBLQS2r9l9RCnrqFySJEnS+u/dmTk2M8eV9dOAX2bmaOCXZR3gvcDo8ppM9cM2EbEVcCawO9V3jzNrRlFIknpII58iF8DlwIOZ+bWa8m1rqv0zcF9Zng5MjIiNI2IUVcNwD/AbYHREjIqIjagmAp/eqLglSZIk9WmHAFPL8lTg/TXlV2blLmDL8t1jf+DmzFyamcuAm4EDejtoSVrfNbIH09uBjwJ7ly6tcyLiQOCrpZvrvcC7gU8DZOb9wDVUk3f/FDiu9HRaBXwK+BnwIHBNqStJ6sciYnBE3BMRv4+I+yPii6X8ioiYV9N2jC3lDn2QpIEngZ9HxOyImFzKtikPFAJ4HNimLDunqyQ1USOfInc7EO1suqmTfc4Bzmmn/KbO9pMk9UvPA3tn5ooypPr2iPhJ2fbZzLy2Tf3aoQ+7Uw192L1m6MM4qi8isyNievmVWpLUv+2VmQsj4u+AmyPiodqNmZkRkT1xosy8DLgMYNy4cT1yTEkaSHrlKXKSJLVVhjCsKKsblldnH+gd+iBJA0xmLix/nwR+RDWH0hOt026Uv0+W6s7pKklNZIJJ6idmzFjzktYXETEoIuZQfTm4OTPvLpvOKcPgzo+IjUuZQx8kaQCJiNdExGaty8B+VPO3Tgdah0NPAm4oy9OBI8uQ6j2Ap8pQup8B+0XEkDK5936lTJLUg0wwSZKapsy1N5bq1+S3RsSbgdOBNwG7AVsBn+uhc12WmeMyc9ywYcN64pCSpMbahmr49O+pHv7z48z8KfBlYN+I+CPwnrIO1ZQajwJzqZ5W/a8AmbkUOIvq4UG/Ab5UyiRJPahhczBJktRVmbk8Im4FDsjMc0vx8xHxHeAzZb2zoQ/j25TPbGjAkqSGy8xHgTHtlC8B9mmnPIHjOjjWFGBKT8coSVrDHkySpKaIiGERsWVZ3gTYF3ioZl6NoHr09H1lF4c+SJIkSX2UPZgkSc2yLTA1IgZR/eBxTWbeGBG3RC5VX1UAACAASURBVMQwqieRzgGOLfVvAg6kGvrwLHA0VEMfIqJ16AM49EGSJEnqdSaYJElNkZn3Aru0U753B/Ud+iBJkiT1UQ6RkyRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUFxNMkiRJkiRJqosJJkmSJEmSJNXFBJMkSZIkSZLqYoJJkiRJkiRJddmg2QFI6l0zZjQ7AkmSJEnS+sYeTJIkSZIkSaqLCSZJkiRJkiTVxQSTJEmSJEmS6mKCSZIkSZIkSXUxwSRJkiRJkqS6+BQ5Sa9Q+6S5CROaF4ckSZIkqX+wB5MkSZIkSZLqYoJJkiRJkiRJdXGInNQH1Q5RkyRJkiSpr7MHkyRJkiRJkupigkmSJEmSJEl1McEkSZIkSZKkuphgkiRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUlw2aHYAkSZIkSVqjZWZLs0OQ1pk9mCRJkiRJklQXE0ySJEmSJEmqiwkmSZIkSZIk1cUEkyRJkiRJkupigkmSJEmSJEl1McEkSZIkSZKkuphgkiRJkiRJUl1MMEmSJEmSJKkuJpgkSZIkSZJUFxNMkiRJkiRJqosJJkmSJEmSJNVlg2YHIEkamCJiMHAbsDFVe3RtZp4ZEaOAacBQYDbw0cx8ISI2Bq4EdgWWAB/OzPnlWKcDxwAvAidk5s96+3qknjZvXkuzQ5AkSeoyezBJkprleWDvzBwDjAUOiIg9gK8A52fm64FlVIkjyt9lpfz8Uo+I2BGYCOwEHAB8MyIG9eqVSJIkSQOcPZgkATBjRrMj0ECTmQmsKKsbllcCewNHlPKpQAtwCXBIWQa4Frg4IqKUT8vM54F5ETEXeCtwZ+OvQpIkSRLYg0mS1EQRMSgi5gBPAjcDjwDLM3NVqbIAGF6WhwOPAZTtT1ENo1td3s4+kiRJknqBCSZJUtNk5ouZORYYQdXr6E2NOldETI6IWRExa/HixY06jSRJkjQgmWCSJDVdZi4HbgXeBmwZEa1DuEcAC8vyQmB7gLJ9C6rJvleXt7NP7Tkuy8xxmTlu2LBhDbkOSZIkaaAywSRJaoqIGBYRW5blTYB9gQepEk2HlmqTgBvK8vSyTtl+S5nHaTowMSI2Lk+gGw3c0ztXIUmSJAmc5FuS1DzbAlPLE99eBVyTmTdGxAPAtIg4G/gdcHmpfznw3TKJ91KqJ8eRmfdHxDXAA8Aq4LjMfLGXr0WSJEka0EwwSZKaIjPvBXZpp/xRqvmY2pY/BxzWwbHOAc7p6RglSZIkdY1D5CRJkiRJklQXezBJ6tSMGWuWJ0xoXhySJEmSpL7LHkySJEmSJEmqiwkmSZIkSZIk1aVhCaaI2D4ibo2IByLi/og4sZRvFRE3R8Qfy98hpTwi4sKImBsR90bEW2qONanU/2NETOronJIkSZIkSep9jezBtAo4JTN3BPYAjouIHYHTgF9m5mjgl2Ud4L3A6PKaDFwCVUIKOBPYneqpQme2JqUkSZIkSZLUfA1LMGXmosz8bVl+BngQGA4cAkwt1aYC7y/LhwBXZuUuYMuI2BbYH7g5M5dm5jLgZuCARsUtSZIkSZKkddMrczBFxEhgF+BuYJvMXFQ2PQ5sU5aHA4/V7LaglHVU3vYckyNiVkTMWrx4cY/GL0mSJKk5ImJQRPwuIm4s66Mi4u4ytcbVEbFRKd+4rM8t20fWHOP0Uv5wROzfnCuRpPVbwxNMEbEp8EPgpMx8unZbZiaQPXGezLwsM8dl5rhhw4b1xCElSZIkNd+JVKMhWn0FOD8zXw8sA44p5ccAy0r5+aUeZZqOicBOVCMhvhkRg3opdkkaMBqaYIqIDamSS1dl5nWl+Iky9I3y98lSvhDYvmb3EaWso3JJkiRJ67GIGAG8D/h2WQ9gb+DaUqXtlButU3FcC+xT6h8CTMvM5zNzHjCXam5XSVIPauRT5AK4HHgwM79Ws2k60PokuEnADTXlR5anye0BPFWG0v0M2C8ihpTJvfcrZZIkSWrHvHktq19SP/d14FTgpbI+FFiemavKeu30Gaun1ijbnyr1uzTlhiSpPhs08NhvBz4K/CEi5pSyzwNfBq6JiGOAPwEfKttuAg6k+kXhWeBogMxcGhFnAb8p9b6UmUsbGLckSZKkJouIg4AnM3N2RIzvhfNNpnqaNTvssEOjTydJ652GJZgy83YgOti8Tzv1Eziug2NNAab0XHSSJEmS+ri3AwdHxIHAYGBz4AKqp01vUHop1U6f0Tq1xoKI2ADYAlhCF6fcyMzLgMsAxo0b1yPzxErSQNIrT5GTJEmSpHWRmadn5ojMHEk1SfctmfkR4Fbg0FKt7ZQbrVNxHFrqZymfWJ4yNwoYDdzTS5chSQNGI4fISZIkaR04Z5LUJZ8DpkXE2cDvqOZ9pfz9bkTMBZZSJaXIzPsj4hrgAWAVcFxmvtj7YUvS+s0EkyRJkqQ+LTNnAjPL8qO08xS4zHwOOKyD/c8BzmlchJIkh8hJkiRJkiSpLiaYJEmSJEmSVBcTTJIkSZIkSaqLCSZJkiRJkiTVxQSTJEmSJEmS6mKCSZIkSZIkSXUxwSRJkiRJkqS6mGCSJEmSJElSXTZodgCSJEkD2bx5Lc0OQZIkqW72YJIkSZIkSVJdTDBJkiRJkiSpLiaYJEmSJEmSVBcTTJIkSZIkSaqLCSZJkiRJkiTVxafISeqyGTPWLE+Y0Lw4JEmSJLWvZWbLmuXxLR3Wk3qaPZgkSZIkSZJUF3swSZIk9VPz5rWsXh41qqXDepIkSY1mDyZJkiRJkiTVxQSTJEmSJEmS6mKCSZIkSZIkSXUxwSRJaoqI2D4ibo2IByLi/og4sZS3RMTCiJhTXgfW7HN6RMyNiIcjYv+a8gNK2dyIOK0Z1yNJkiQNZE7yLUlqllXAKZn524jYDJgdETeXbedn5rm1lSNiR2AisBOwHfCLiHhD2fwNYF9gAfCbiJiemQ/0ylVIkiRJWnuCKSI2zszn11YmaWCZMWPN8oQJzYtD/VdmLgIWleVnIuJBYHgnuxwCTCvtz7yImAu8tWybm5mPAkTEtFLXBJMkSZLUS7oyRO7OLpZJktQtETES2AW4uxR9KiLujYgpETGklA0HHqvZbUEp66i87TkmR8SsiJi1ePHiHr4CSZIkaWDrsAdTRPw91Qf0TSJiFyDKps2BV/dCbJKkASAiNgV+CJyUmU9HxCXAWUCWv+cBH6v3PJl5GXAZwLhx47Le40l9zbx5Lc0OQZIkDWCdDZHbHzgKGEH14b41wfQM8PnGhiVJGggiYkOq5NJVmXkdQGY+UbP9W8CNZXUhsH3N7iNKGZ2US5IkSeoFHSaYMnMqMDUiPpiZP+zFmCRJA0BEBHA58GBmfq2mfNsyPxPAPwP3leXpwPci4mtUk3yPBu6h+gFkdESMokosTQSO6J2rkCRJkgRde4rciIjYnKrn0reAtwCnZebPGxqZJGl993bgo8AfImJOKfs8cHhEjKUaIjcf+BeAzLw/Iq6hmrx7FXBcZr4IEBGfAn4GDAKmZOb9vXkhkiRJ0kDXlQTTxzLzgojYHxhK9WXgu4AJJklSt2Xm7awZfl3rpk72OQc4p53ymzrbT5IkSVJjdSXB1Prh/0DgyvILcntfCCRJktQP1U4QPmpUS4f1JEmSOtKVBNPsiPg5MAo4PSI2A15qbFiSJEmSJA0cLTNbmh2CVJeuJJiOAcYCj2bmsxExFDi6sWFJkiRJkiSpv+gwwRQRb8rMh6iSSwCvdWScJEmSJEmS2uqsB9PJwGTgvHa2JbB3QyKSJEmSJElSv9JhgikzJ5e/7+69cCRJkiRJktTfdGUOJiJiT2Bkbf3MvLJBMUmSJEmSJKkfWWuCKSK+C7wOmAO8WIoTMMEkSZIkSZKkLvVgGgfsmJnZ6GAkSZIkSZLU/7yqC3XuA/6+0YFIkiRJkiSpf+qwB1NEzKAaCrcZ8EBE3AM837o9Mw9ufHjSwDFjRrMjkCRJkiSpezobIndur0UhSZIkSZKkfqvDBFNm/m9vBiJJkiRJkqT+qSuTfEuSJKmfmjevZfXyqFEtHdaTJEmqR1cm+ZYkSZIkSZI6tNYEU0RMiAgTUZIkSZIkSWpXVxJHHwb+GBFfjYg3NTogSZIkSZIk9S9rTTBl5v8BdgEeAa6IiDsjYnJEbNbw6CRJkiRJktTndWnoW2Y+DVwLTAO2Bf4Z+G1EHN/A2CRJkiRJktQPdGUOpkMi4kfATGBD4K2Z+V5gDHBKY8OTJEmSJElSX7dBF+p8ADg/M2+rLczMZyPimMaEJUmSJEmSpP6iK0PkHm+bXIqIrwBk5i8bEpUkSZIkSZL6ja4kmPZtp+y9PR2IJEmSJEmS+qcOE0wR8cmI+APwpoi4t+Y1D7h3bQeOiCkR8WRE3FdT1hIRCyNiTnkdWLPt9IiYGxEPR8T+NeUHlLK5EXFa9y9VkiRJkiRJjdDZHEzfA34C/CdQm9h5JjOXduHYVwAXA1e2KT8/M8+tLYiIHYGJwE7AdsAvIuINZfM3qHpRLQB+ExHTM/OBLpxfkiRJkiRJvaCzBFNm5vyIOK7thojYam1Jpsy8LSJGdjGOQ4Bpmfk8MC8i5gJvLdvmZuaj5bzTSl0TTJIkSZIkSX3E2nowHQTMBhKImm0JvLab5/xURBwJzAJOycxlwHDgrpo6C0oZwGNtyndv76ARMRmYDLDDDjt0MzRJ3TFjxprlCROaF4ckSVp/RMRg4DZgY6rvLddm5pkRMQqYBgyl+q7y0cx8ISI2pho9sSuwBPhwZs4vxzodOAZ4ETghM3/W29cjNUPLzJY1y+NbOqwn9YQO52DKzIPK31GZ+dryt/XV3eTSJcDrgLHAIuC8bh7nFTLzsswcl5njhg0b1lOHlSRJktQczwN7Z+YYqu8PB0TEHsBXqKbdeD2wjCpxRPm7rJSfX+q1nY7jAOCbETGoV69EkgaADnswRcRbOtsxM3+7rifLzCdqjv8t4MayuhDYvqbqiFJGJ+WSJEmS1lOZmcCKsrpheSWwN3BEKZ8KtFD9kH1IWQa4Frg4IoKOp+O4s/FXIUkDR2dD5DrrXdR6Y18nEbFtZi4qq/8MtD5hbjrwvYj4GtUk36OBe6iG5Y0u3WAXUv3ycASSJEmqy7x5Lc0OQVqr0tNoNvB6qof/PAIsz8xVpUrt1BrDKdNrZOaqiHiKahhdZ9Nx1J7LKTckqQ4dJpgy8931HDgivg+MB7aOiAXAmcD4iBhLlaCaD/xLOdf9EXEN1eTdq4DjMvPFcpxPAT8DBgFTMvP+euKSJEmS1D+U7wRjI2JL4EfAmxp4rsuAywDGjRuXjTqPJK2vOhsit3dm3hIRH2hve2Ze19mBM/Pwdoov76T+OcA57ZTfBNzU2bkkSZIkrb8yc3lE3Aq8DdgyIjYovZhqp9BonXZjQURsAGxBNdl3Z9NxSJJ6SIeTfAPvKn8ntPM6qMFxSZIkSRrAImJY6blERGwC7As8CNwKHFqqTQJuKMvTyzpl+y1lHqfpwMSI2LhMvdE6HYckqQd1NkTuzPL36N4LR5IkSZIA2BaYWuZhehVwTWbeGBEPANMi4mzgd6wZJXE58N0yifdSqvlbO52OQ5LUczqb5BuAiBhKNX/SXlRzJ90OfCkzlzQ4NkmSJEkDVGbeC+zSTvmjVE+Ba1v+HHBYB8dqdzoOSVLP6WyIXKtpwGLgg1RdTRcDVzcyKEmSJEmSJPUfXUkwbZuZZ2XmvPI6G9im0YFJktZvEbF9RNwaEQ9ExP0RcWIp3yoibo6IP5a/Q0p5RMSFETE3Iu6NiLfUHGtSqf/HiJjU0TklSZIkNUZXEkw/j4iJEfGq8voQ8LNGByZJWu+tAk7JzB2BPYDjImJH4DTgl5k5GvhlWQd4L9XErKOBycAlUCWkqIZy7041ZOLM1qSUJEmSpN7RYYIpIp6JiKeBTwDfA14or2lUH+wlSeq2zFyUmb8ty89QPRloOHAIMLVUmwq8vywfAlyZlbuoHlO9LbA/cHNmLs3MZcDNwAG9eCmSJEnSgNfZU+Q2681AJEkDV0SMpJrI9W5gm8xcVDY9zpph2cOBx2p2W1DKOipve47JlB9Idthhh54LXpIkqZtaZrY0OwSpx6z1KXIAZajBaGBwa1lm3taooCRJA0dEbAr8EDgpM5+OiNXbMjMjInviPJl5GXAZwLhx43rkmJIkSZIqa52DKSI+DtxGNe/SF8vflsaGJUkaCCJiQ6rk0lWZeV0pfqIMfaP8fbKULwS2r9l9RCnrqFySJElSL+nKJN8nArsBf8rMd1MNYVje0KgkSeu9qLoqXQ48mJlfq9k0HWh9Etwk4Iaa8iPL0+T2AJ4qQ+l+BuwXEUNKj9v98GEUkiRJUq/qyhC55zLzuYggIjbOzIci4o0Nj0yStL57O/BR4A8RMaeUfR74MnBNRBwD/An4UNl2E3AgMBd4FjgaIDOXRsRZwG9KvS9l5tLeuQRJkqT+oXa+p5bxLR3Wk7qrKwmmBRGxJXA9cHNELKP6wC9JUrdl5u1AdLB5n3bqJ3BcB8eaAkzpueikxpo3r6XZIUiSJPWotSaYMvOfy2JLRNwKbAH8tKFRSZIkSZIkqd/o6lPk3gLsBSTw68x8oaFRSZIkSZIkqd/oylPk/h2YCgwFtga+ExFnNDowSZIkSZIk9Q9d6cH0EWBMZj4HEBFfBuYAZzcyMEmSJPUs536SJEmN0pUE01+AwcBzZX1jYGHDIpIkSVLT1CahRo1q6bCeJElSrQ4TTBFxEdWcS08B90fEzWV9X+Ce3glPkiRJkiRJfV1nPZhmlb+zgR/VlM9sWDSSJEmSJEnqdzpMMGXm1NbliNgIeENZfTgzVzY6MEmSJEmSJPUPa52DKSLGUz1Fbj4QwPYRMSkzb2tsaNL6acaMNcsTJjQvDkmSJEmSekpXJvk+D9gvMx8GiIg3AN8Hdm1kYNJAUJtskiRJkiSpv3pVF+ps2JpcAsjM/wts2LiQJEmSJEmS1J90pQfT7Ij4NvA/Zf0jrJkAXJIkSZIkSQNcVxJMxwLHASeU9V8B32xYRJIkSZIkSepXOk0wRcQg4PeZ+Sbga70TkiRJkiRJkvqTTudgyswXgYcjYodeikeSJEmSJEn9TFeGyA0B7o+Ie4C/tRZm5sENi0qSJEmSJEn9RlcSTP/W8CgkSZIkSZLUb3WYYIqIwVQTfL8e+ANweWau6q3AJEmSJEmS1D901oNpKrCS6qlx7wV2BE7sjaAkSZLWN/PmtTQ7BEmSpIbpLMG0Y2b+E0BEXA7c0zshSZIkSZIkqT/pLMG0snUhM1dFRC+EI6m/mzFjzfKECc2LQ5IkSZLUezpLMI2JiKfLcgCblPUAMjM3b3h0kiRJkiRJ6vM6TDBl5qDeDESSJEmSJEn906uaHYAkSZIkSZL6NxNMkiRJkiRJqosJJkmSJEmSJNXFBJMkSZIkSZLqYoJJkiRJkiRJdTHBJEmSJEmSpLqYYJIkSZIkSVJdTDBJkiRJkiSpLiaYJEmSJEmSVBcTTJIkSZIkSarLBs0OQJIkSX3TvHkt7ZaPGtV+uSRJGrjswSRJkiRJkqS6mGCSJEmSJElSXUwwSZKaIiKmRMSTEXFfTVlLRCyMiDnldWDNttMjYm5EPBwR+9eUH1DK5kbEab19HZIkSZJMMEmSmucK4IB2ys/PzLHldRNAROwITAR2Kvt8MyIGRcQg4BvAe4EdgcNLXUmSJEm9yEm+JUlNkZm3RcTILlY/BJiWmc8D8yJiLvDWsm1uZj4KEBHTSt0HejhcSZIkSZ2wB5Mkqa/5VETcW4bQDSllw4HHauosKGUdlb9CREyOiFkRMWvx4sWNiFuSJEkasEwwSZL6kkuA1wFjgUXAeT114My8LDPHZea4YcOG9dRhJUmSJOEQOUlSH5KZT7QuR8S3gBvL6kLg/7d3/8Fylfd9x99fQsGtfxQRFFWxJKOkclLaSRxGBWbq2nJJZExGlhMzFGYcywypxqlop206Y1xnhh3cpHI6kw6eeFyrjsbCU4OpW4qUUIisWOMmYxx5bCzArpGMcJEsUGIR4iltE9xv/9jnosPVXens3d1zzu6+XzM79+zZc+/9nL13z7Pnu8/znLWVTdeUdZxjvSRJkqSGTKwH04CrA10aEfsj4kj5uqKsj4j4aLkC0OGIuLLyPdvK9kciYtuk8kqS2hcRqyt3fwFYaEP2AjdFxMURsR7YAPwxcAjYEBHrI+Ii+hOB720ysyRJkqTJDpH7FGdfHeh24EBmbgAOlPvQv/rPhnLbTn+IBBFxKXAHcDX9yVzvqMzHIUmaYhFxD/Al4Cci4nhE3Ar8ZkQ8FhGHgbcB/xwgM58A7qM/efdDwI7M/EFmvgTcBjwMfBO4r2wrSZIkqUETGyI34OpAW4FNZXkPcBD4QFl/d2Ym8EhEXFI+xd4E7M/M0wARsZ9+0eqeSeWWJDUjM29eYvXvnGP7Xwd+fYn1DwIPjjGaJKkDImItcDewCkhgV2beVT6E/ixwOfA0cGNmPh8RAdwFXA+8CLwvM79aftY24NfKj/7XmbmnyX2RpHnQ9CTfqzLzZFl+ln5jAV4dSJIkSdIrvQT8amZeAVwD7IiIK3BUhCR1UmuTfGdmRkSO8eftAnYBbNy4cWw/V9Ly7dt3ZnnLlvZySFJbjh3rtR1Bmlrlg+mTZfn7EfFN+h82OypCkjqo6R5Mzy1M4Fq+nirrB10d6FxXDZIkSZI0B8rUGz8DfJkJjYpwRIQkjabpAtNeYOFKcNuAByrr31uuJncN8EJpNB4GNkfEitKNdXNZJ0mSJGkORMRrgP8M/LPM/PPqY6W30lhGL2TmrszcmJkbV65cOY4fKUlzZWJD5MrVgTYBl0XEcfrjnncC95UrBX0HuLFs/iD9yfiO0p+Q7xaAzDwdER+mfxlqgDsXurZKkiRJmm0R8VfoF5f+Y2b+l7L6uYhYnZknhxgVsWnR+oOTzC11Xe9g78zypt7A7aRhTPIqcktdHQjg2iW2TWDHgJ+zG9g9xmiSJEmSOq5cFe53gG9m5m9VHloYFbGTs0dF3BYR99Kf0PuFUoR6GPiNysTem4EPNrEPkjRPWpvkW5IkSZLO4e8BvwQ8FhGPlnX/CkdFSFInWWCSJEmS1DmZ+YdADHjYURGS1DFNT/ItSZIkSZKkGWOBSZIkSZIkSSOxwCRJkiRJkqSRWGCSJEmSJEnSSCwwSZIkSZIkaSQWmCRJkiRJkjQSC0ySJEmSJEkaiQUmSZIkSZIkjcQCkyRJkiRJkkZyYdsBJM2HffvOLG/Z0l4OSZIkSdL4WWCSJEmSJKkhvYO9tiNIE+EQOUmSJEmSJI3EApMkSZIkSZJGYoFJkiRJkiRJI7HAJEmSJEmSpJFYYJIkSZIkSdJILDBJkiRJkiRpJBe2HUCaB/v2tZ1AkiRJks7WO9g7s7ypN3A76XzswSRJkiRJkqSRWGCSJEmSJEnSSCwwSZIkSZIkaSTOwSSpcdU5qbZsaS+HJEmSJGk87MEkSZIkSZKkkdiDSZIkSUM5dqz38vL69b2B20mSpPlhDyZJkiRJkiSNxAKTJEmSJEmSRmKBSZIkSZIkSSOxwCRJkiRJkqSRWGCSJLUiInZHxKmIeLyy7tKI2B8RR8rXFWV9RMRHI+JoRByOiCsr37OtbH8kIra1sS+SJEnSvLPAJElqy6eA6xatux04kJkbgAPlPsA7gA3lth34OPQLUsAdwNXAVcAdC0UpSZIkSc2xwCRJakVmfhE4vWj1VmBPWd4DvKuy/u7sewS4JCJWA28H9mfm6cx8HtjP2UUrSZIkSRN2YdsBJEmqWJWZJ8vys8Cqsvx64JnKdsfLukHrzxIR2+n3fmLdunVjjCyd7dixXtsRJEmSGmUPJklSJ2VmAjnGn7crMzdm5saVK1eO68dKkiRJwh5Mklq2b9+Z5S1b2suhznguIlZn5skyBO5UWX8CWFvZbk1ZdwLYtGj9wQZySpIkSaqwwCRJ6pK9wDZgZ/n6QGX9bRFxL/0JvV8oRaiHgd+oTOy9Gfhgw5klSZJmQu9g78zypt7A7aSlWGCSJLUiIu6h3/vosog4Tv9qcDuB+yLiVuA7wI1l8weB64GjwIvALQCZeToiPgwcKtvdmZmLJw6XJEmSNGEWmCRJrcjMmwc8dO0S2yawY8DP2Q3sHmM0SZIkSUNykm9JkiRJkiSNxAKTJEmSJEmSRmKBSZIkSZIkSSOxwCRJkiRJkqSRWGCSJEmSJEnSSCwwSZIkSZIkaSQWmCRJkiRJkjSSC9sOIEmSJEnSLOsd7LUdQZo4ezBJkiRJkiRpJBaYJEmSJEmSNBILTJIkSZIkSRqJBSZJkiRJkiSNxEm+JUmSNHbHjvVeXl6/vjdwO0mSNBvswSRJkiRJkqSRWGCSJEmSJEnSSCwwSZIkSZIkaSTOwSRJkiRJkl6hd7B3ZnlTb+B20oJWejBFxNMR8VhEPBoRXynrLo2I/RFxpHxdUdZHRHw0Io5GxOGIuLKNzNKw9u07c5MkSdJwImJ3RJyKiMcr64Y+Z4iIbWX7IxGxrY19kaR50OYQubdl5psyc2O5fztwIDM3AAfKfYB3ABvKbTvw8caTSpIkSWrap4DrFq0b6pwhIi4F7gCuBq4C7lgoSkmSxqtLczBtBfaU5T3Auyrr786+R4BLImJ1GwElSZIkNSMzvwicXrR62HOGtwP7M/N0Zj4P7OfsopUkaQzamoMpgd+PiAQ+kZm7gFWZebI8/iywqiy/Hnim8r3Hy7qTlXVExHb6n1awbt26CUaXNCnV4YRbtrSXQ5Ikddaw5wyD1p/F8wlJGk1bPZjenJlX0u/KuiMi3lJ9MDOTfhGqtszclZkbM3PjypUrxxhVkiRJUtcs55zhPD/P8wlJGkErBabMPFG+ngLupz8e+rmFoW/l66my+QlgbeXb15R1kiRJkubLsOcMnktIUkMaLzBFxKsj4rULy8BmB8QjmQAAEIBJREFU4HFgL7BwVYdtwANleS/w3nJliGuAFyrdYiVJkiTNj2HPGR4GNkfEijK59+ayTpI0Zm3MwbQKuD8iFn7/ZzLzoYg4BNwXEbcC3wFuLNs/CFwPHAVeBG5pPrIkSZLO59ixXtsRNEMi4h5gE3BZRBynfzW4nQxxzpCZpyPiw8Chst2dmbl44nBJ0hg0XmDKzKeAn15i/feAa5dYn8COBqJJkiRpAqqFp/XrewO3k6oy8+YBDw11zpCZu4HdY4wmSVpCW1eRkyRJkiRpZvUO9tqOIDWqravISZIkSZIkaUbYg0mSJGkMnH9IkiTNMwtMc2LfvjPLW7a0l0OSJEmSJM0eC0ySJE0xJ09W2+y5JUmSwAKTNFbVnmKSJEmSJM0LC0zqLIf1SZIkSZI0HSwwqXUWkiRJkiSpu3oHe2eWN/UGbqf5dkHbASRJkiRJkjTd7ME0w+ZtPiB7QkmSJEmS1A4LTJoKFo+k+RIRTwPfB34AvJSZGyPiUuCzwOXA08CNmfl8RARwF3A98CLwvsz8ahu5JUmSpHllgUlLmnRBZ956V0lalrdl5p9W7t8OHMjMnRFxe7n/AeAdwIZyuxr4ePkqSZIkqSEWmObQoOKOPYMkddxWYFNZ3gMcpF9g2grcnZkJPBIRl0TE6sw82UpKSed07Fjv5eX163sDt5OkaeEE2FKfBSZJneSwyLmXwO9HRAKfyMxdwKpK0ehZYFVZfj3wTOV7j5d1rygwRcR2YDvAunXrJhhdkiRJmj8WmDR1LDxIc+HNmXkiIn4E2B8R/6P6YGZmKT7VVopUuwA2btw41PdKkiRJOjcLTJKkzsnME+XrqYi4H7gKeG5h6FtErAZOlc1PAGsr376mrJMcjiVJktQQC0zqFCf/lhQRrwYuyMzvl+XNwJ3AXmAbsLN8faB8y17gtoi4l/7k3i84/5IkSZLULAtMmnkOqZOmzirg/oiAfjv1mcx8KCIOAfdFxK3Ad4Aby/YPAtcDR4EXgVuajyxJkiTNNwtMkqROycyngJ9eYv33gGuXWJ/Ajgaiaco5XE6SNGnVK8pJ88YCk6basEPq7M0kSZIkSdL4WWCSlsFClSR1V7WnkiRJkpphgUnnNQ/FlFH20YnJJUmSJM2L6jDA3qbewO00fywwaW5ZGJIkjcreUpIkSX0WmKRF5qHHliQt5gTYkiRJGoUFJkmSJEmShuDV4qSzWWDSUOatd8+87W9X+XeQJEmSpG6zwKSxmIcCgHM2SZIkSdIZTvitKgtMkiRp6g072bZzTkmSJI2XBSa9bFw9dOahN5MkSZIkSTrDApNmksPZJOn8hu31I0mSJA1igUmSJOk8LMZJ0vxamGfIOYakc7PApMbYq0iSNE4WfSRJkrrDApMkSXqFQRNgz9vE2BawJElV1SumSTqbBSZJkqTCopIkSdLyWGCSJGkGTbq30bz1ZpIkSedW7eHlfFXzyQKTpKlSnctry5b2ckiaHfZakiQt5nA4aXgWmLRsdSbtdmJvSdJi9n6S6vG1IkmaJhe0HUCSJEmSJEnTzR5MkiRpJOPqZWFvjfnj31ySpNlhgUmSWjJoPinnmZIkSWqe8y5Jo7HAJEkd4HxlGsagSaknMVm1E2BLkiSpDgtMkqZWl3v6dDmbJGn6OJxQmgx7LUnjY4FJkqQZN0snpvaokiRJ6iYLTJIkaWxmqZglSZKWp9ozrLepN3A7zRYLTJI0Yc6vJA1mjyRJUtMcFtcsi03zwwKTJEmaCHszSZIkzQ8LTDPGnhKSpC6yp5JUj68VaTLstSRNngWmGWBRSfKqbdIs84Rbs87/cUnzyKFzs8cCkyRJc8QTWWn6OfxU0rSyJ9lss8AkSZKk1lk0kTRuFjOkZllgkiRJnWNPK0mSpOligUnSTJv2uZmmPb/qseeG9Eq+JiRpfjk30/SywCRp5rQ18b0T7kuSJEnDczjjbLDAJGluDCoA2TNIkqbDtPdscuinNHkWKmaLvZmmiwUmSZIkddagosy0F5skSZo1U1NgiojrgLuAHwI+mZk7W44kaQYNmvPIuZC6r6l2wpNaaTq19dqddK8lj0n1eT4xu+y1NB/szdR9U1FgiogfAj4G/BxwHDgUEXsz8xvtJhuNJ6xSNwwaOldnTqUm513ymDFYW+1Ekyd2Dq2Rhtfk68ZCT7fN6vnEvLGQpAWD/hcsPLVrKgpMwFXA0cx8CiAi7gW2AlPXIAx7IjuoB4Wk9vha7KROtROjnGh6kioNb9hC0iiFp+rrss7wvSZ5/DinTrUTTZrGXh8WkrRcdQpP0/iamBaRmW1nOK+IuAG4LjN/udz/JeDqzLytss12YHu5+xPAtxqKdxnwpw39rrrMVE8XM0E3c5mpnmnJ9IbMXNlGmElpuJ3o4t95gdmG19VcYLblMtvwFueynRiunejq33UYs7APMBv74T50g/sw2MA2Ylp6MJ1XZu4CdjX9eyPiK5m5senfey5mqqeLmaCbucxUj5m6bVztRJefU7MNr6u5wGzLZbbhdTVX05bbTszC8zcL+wCzsR/uQze4D8tzQZO/bAQngLWV+2vKOkmSwHZCknRuthOSNGHTUmA6BGyIiPURcRFwE7C35UySpO6wnZAknYvthCRN2FQMkcvMlyLiNuBh+pcV3Z2ZT7Qca0Hjw/JqMFM9XcwE3cxlpnrM1JKG24kuP6dmG15Xc4HZlstsw+tqrrGZcDsxC8/fLOwDzMZ+uA/d4D4sw1RM8i1JkiRJkqTumpYhcpIkSZIkSeooC0ySJEmSJEkaiQWmmiLiuoj4VkQcjYjbl3j8X0TENyLicEQciIg3dCDT+yPisYh4NCL+MCKuaDtTZbt3R0RGxMQvm1jjeXpfRPxJeZ4ejYhfbjtT2ebG8j/1RER8pu1MEfHvKs/RkxHxZ5POVDPXuoj4QkR8rbz+ru9ApjeU48DhiDgYEWsmnGd3RJyKiMcHPB4R8dGS93BEXDnJPLOii8f9IbI1fvyvk6uyXWNtQN1sbbQFdbOVbRptE+pma6ttqJmt8fahZq5G24hFv9v2Ygwi4tKI2B8RR8rXFUts84aI+Gp5bTwREe9vI+sgNffhTRHxpZL/cET8wzaynkud/SjbPRQRfxYRv9t0xkFqHCsujojPlse/HBGXN5/y3Grsw1vK6+CliLihjYzn0+X3e3V16n1hZno7z43+RIDfBn4MuAj4OnDFom3eBvy1svwrwGc7kOl1leV3Ag+1nals91rgi8AjwMa2MwHvA367Y/9PG4CvASvK/R9pO9Oi7f8J/ckxu/Bc7QJ+pSxfATzdgUz/CdhWlv8B8OkJZ3oLcCXw+IDHrwf+GxDANcCXJ/23m/ZbF4/7Q2Zr9PhfN1fZrrE2YMjnrNG2YMhsjbYJw/5NK9s30jYM8bw12j4MkavRNmLR77a9GM/z+JvA7WX5duAjS2xzEXBxWX4N8DTwo21nH3If3ghsKMs/CpwELmk7+7D7UR67FtgC/G7bmUueOseKfwz8+7J8Ew29zxjzPlwO/BRwN3BD25mXuQ+tvN8b8z409r7QHkz1XAUczcynMvMvgHuBrdUNMvMLmfliufsIMOlPo+pk+vPK3VcDk57R/byZig8DHwH+z4TzDJOpSXUy/SPgY5n5PEBmnupApqqbgXsmnKlurgReV5b/OvDdDmS6AviDsvyFJR4fq8z8InD6HJtsBe7OvkeASyJi9SQzzYAuHveHydb08b9WrqLJNmDYbG3oYpswTLaqptoG6Gb7UDdXo21Ele3F2GwF9pTlPcC7Fm+QmX+Rmf+33L2Y7o0cqbMPT2bmkbL8XeAUsLKxhPWcdz8AMvMA8P2mQtVQ51hR3bfPAddGRDSY8XzqvB95OjMPA/+vjYA1dPn9Xl2del/YtQNdV70eeKZy/3hZN8it9D/9maRamSJiR0R8m351/5+2nal0tV6bmb834Sy1MxXvLt0ePxcRazuQ6Y3AGyPijyLikYi4rgOZgH6Xb2A9Z94ct52rB7wnIo4DD9L/BL3tTF8HfrEs/wLw2oj44QnnOpdhj2Hq5nF/QReP/7VytdAGLOhiW7Cgi23CMNmAxtsG6Gb7UDdX19qIKtuLelZl5smy/CywaqmNImJtRBym/5x+pBRpuqLWPiyIiKvo94749qSDDWmo/eiQOq+1l7fJzJeAF4CuHCtgNo4XXX6/V1en3hdaYBqziHgPsBH4t21nAcjMj2XmjwMfAH6tzSwRcQHwW8CvtpljCfuAyzPzp4D9nPmkoE0X0h8SsYn+J8L/ISIuaTXRGTcBn8vMH7QdpLgZ+FRmrqHftf/T5X+tTf8SeGtEfA14K3AC6MrzpTHr2nF/QZeO/9DpNmBBF9uCBV1uExZ0rW2AbrYPYBsxFSLi8xHx+BK3xT0DkgG9ATLzmXJM+ZvAtohotPgxjn0oP2c18GnglsxsvCfKuPZDGkVX3+/V1dT7wgsn9YNnzAmg+knmmrLuFSLiZ4EPAW+tdIltNVPFvcDHJ5ro/JleC/wd4GDp3fk3gL0R8c7M/EpLmcjM71XufpJ+VXeS6vztjtOf9+AvgWMR8ST9k4tDLWZacBOwY0I5FquT61bgOoDM/FJEvAq4jH437lYylU8ofxEgIl4DvDszG5v4dgnDHi/UzeP+UNkqmjj+QzfbgLrZ2mgLFnSxTRgm24Im2wboZvtQK1cH24gq24siM3920GMR8VxErM7Mk6X4cs7/qcz8bvQnVv/79Ic6NWIc+xARrwN+D/hQGTbZuHH+LTqkzmttYZvjEXEh/aG+36M7ZuF40eX3e3V16n1hFz7FmQaHgA0RsT4iLqL/JmpvdYOI+BngE8A7G5oboU6mDZW7Pw8caTNTZr6QmZdl5uWZeTn9MayTPrGo8zxV5xZ4J/DNCeaplQn4r/Q/qSYiLqM/POKpljMRET8JrAC+NMEsw+b6n/QnbiQi/hbwKuBP2swUEZdVPiX/ILB7gnnq2Au8N/quAV6odCfX0rp43B8mW9PH//PmaqkNqJUNWmkLamej+TZhmGxttA11szXdPtTK1cE2osr2op69wLayvA14YPEGEbEmIv5qWV4BvBn4VmMJz6/OPlwE3E9/Xq7GCmNDOu9+dFSdY1h1324A/qD00uqKWm1Ex3X5/V5d3XpfmB2Y+XwabvS7Vj9Jf9zxh8q6O+n/owF8HngOeLTc9nYg013AEyXPF4C/3XamRdsepIErCNV4nv5NeZ6+Xp6nn+xApqA/lOQbwGPATW1nKvd7wM5JZxnyuboC+KPy93sU2NyBTDfQP3A/Sb8nxMUTznMP/Su7/CX9ng63Au8H3l/5f/pYyftYE6+7Wbh18bg/RLbGj/91ci3atpE2YIjnrPG2YIhsjbcJw/xN22gbaj5vjbcPNXM12kYsymZ7MZ7n8YeBA+Xv+Hng0rJ+I/DJsvxzwOHy/3cY2N527mXsw3vK/8qjldub2s4+7H6U+/+dfoH5f5f//bd3IPv5jhWvon/VyaPAHwM/1nbmZezD3y3P9/+i3/vqibYzL2MfWnu/N8Z9aOx9YZRfKEmSJEmSJC2LQ+QkSZIkSZI0EgtMkiRJkiRJGokFJkmSJEmSJI3EApMkSZIkSZJGYoFJkiRJkiRJI7HAJEmSJEmSpJFYYJIkSZIkSdJI/j8ZE6TTA68flwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "tm_trmnt = TwoModels(\n", " estimator_trmnt=CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True), \n", " estimator_ctrl=CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True), \n", " method='ddr_treatment'\n", ")\n", "tm_trmnt = tm_trmnt.fit(\n", " X_train, y_train, treat_train,\n", " estimator_trmnt_fit_params={'cat_features': cat_features}, \n", " estimator_ctrl_fit_params={'cat_features': cat_features}\n", ")\n", "\n", "uplift_tm_trmnt = tm_trmnt.predict(X_val)\n", "\n", "tm_trmnt_score = uplift_at_k(y_true=y_val, uplift=uplift_tm_trmnt, treatment=treat_val, strategy='by_group', k=0.3)\n", "\n", "models_results['approach'].append('TwoModels_ddr_treatment')\n", "models_results['uplift@30%'].append(tm_trmnt_score)\n", "\n", "plot_uplift_preds(trmnt_preds=tm_trmnt.trmnt_preds_, ctrl_preds=tm_trmnt.ctrl_preds_);" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "BLd9GH1WCIQT" }, "source": [ "## Conclusion\n", "\n", "Let's consider which method performed best in this task and use it to speed up the test sample:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:48.837333Z", "start_time": "2020-05-30T22:29:48.821645Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "colab_type": "code", "id": "x6pYalimCIQW", "outputId": "7910ceb8-e15e-4d4b-ae2d-4e0ab0ac7737" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
approachuplift@30%
1ClassTransformation0.061775
2TwoModels0.051637
3TwoModels_ddr_control0.047793
0SoloModel0.041614
4TwoModels_ddr_treatment0.033752
\n", "
" ], "text/plain": [ " approach uplift@30%\n", "1 ClassTransformation 0.061775\n", "2 TwoModels 0.051637\n", "3 TwoModels_ddr_control 0.047793\n", "0 SoloModel 0.041614\n", "4 TwoModels_ddr_treatment 0.033752" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(data=models_results).sort_values('uplift@30%', ascending=False)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "L55TyGbWCIQ0" }, "source": [ "From the table above you can see that the current task suits best for the approach to the transformation of the target line. Let's train the model on the entire sample and predict the test." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:51.343823Z", "start_time": "2020-05-30T22:29:48.840611Z" }, "colab": { "base_uri": "https://localhost:8080/", "height": 156 }, "colab_type": "code", "id": "WnVKI1qZCIQ3", "outputId": "b1efb1d4-1f83-4c53-d3c8-84060d7a2d11" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/Maksim/Library/Python/3.6/lib/python/site-packages/ipykernel_launcher.py:6: UserWarning: It is recommended to use this approach on treatment balanced data. Current sample size is unbalanced.\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "client_id,uplift\r\n", "000048b7a6,0.03777380619745441\r\n", "000073194a,0.0402001184660159\r\n", "00007c7133,-0.001255842638942739\r\n", "00007f9014,0.03165865533189738\r\n" ] } ], "source": [ "ct_full = ClassTransformation(CatBoostClassifier(iterations=20, thread_count=2, random_state=42, silent=True))\n", "ct_full = ct_full.fit(\n", " X_train_full, \n", " y_train_full, \n", " treat_train_full, \n", " estimator_fit_params={'cat_features': cat_features}\n", ")\n", "\n", "X_test.loc[:, 'uplift'] = ct_full.predict(X_test.values)\n", "\n", "sub = X_test[['uplift']].to_csv('sub1.csv')\n", "\n", "!head -n 5 sub1.csv" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "ExecuteTime": { "end_time": "2020-05-30T22:29:51.401905Z", "start_time": "2020-05-30T22:29:51.346944Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
feature_namefeature_score
0first_redeem_time79.642055
1age8.808502
2issue_redeem_delay5.113192
3first_issue_time3.558522
4gender2.877728
\n", "
" ], "text/plain": [ " feature_name feature_score\n", "0 first_redeem_time 79.642055\n", "1 age 8.808502\n", "2 issue_redeem_delay 5.113192\n", "3 first_issue_time 3.558522\n", "4 gender 2.877728" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ct_full = pd.DataFrame({\n", " 'feature_name': ct_full.estimator.feature_names_,\n", " 'feature_score': ct_full.estimator.feature_importances_\n", "}).sort_values('feature_score', ascending=False).reset_index(drop=True)\n", "\n", "ct_full" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "5wHQNSHdCIRj" }, "source": [ "This way we got acquainted with uplift modeling and considered the main basic approaches to its construction. What's next? Then you can plunge them into the intelligence analysis of data, generate some new features, select the models and their hyperparameters and learn new approaches and libraries.\n", "\n", "**Thank you for reading to the end.**\n", "\n", "**I will be pleased if you support the project with an star on [github](https://github.com/maks-sh/scikit-uplift/) or tell your friends about it.**" ] } ], "metadata": { "colab": { "name": "RetailHero_EN.ipynb", "provenance": [] }, "kernelspec": { "display_name": "python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 1 }