{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": false,
"sideBar": true,
"skip_h1_title": true,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"colab": {
"name": "2021-08-18-Causal_inference_for_decision_making_in_growth_hacking_and_upselling_in_Python.ipynb",
"provenance": [],
"collapsed_sections": []
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "iOQxSDQrumWb"
},
"source": [
"# Causal inference for decision-making in growth hacking and upselling in Python\n",
"\n",
"> \"In this article we discuss differences between experimental and observational data and pitfalls in using the latter for data-driven decision-making.\"\n",
"- toc: true\n",
"- branch: master\n",
"- badges: true\n",
"- comments: true\n",
"- categories: [python, dowhy, causal inference]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Oo6ETETld4t"
},
"source": [
"\n",
"## Introduction\n",
"\n",
"Wow, growth hacking *and* upselling all in the same article? Also Python.\n",
"\n",
"Okay, let's start at the beginning. Imagine the following scenario: You're responsible for increasing the amount of money users spend on your e-commerce platform.\n",
"\n",
"You and your team come up with different measures you could implement to achieve your goal. Two of these measures could be:\n",
"\n",
"- Provide a discount on your best-selling items,\n",
"- Implement a rewards program that incentivices repeat purchases.\n",
"\n",
"Both of these measures are fairly complex with each incurring a certain, probably known, amount of cost and an unknown effect on your customers' spending behaviour.\n",
"\n",
"To decide which of these two possible measures is worth both the effort and incurred cost you need to estimate their effect on customer spend.\n",
"\n",
"A natural way of estimating this effect is computing the following:\n",
"\n",
"$\\textrm{avg}(\\textrm{spend} | \\textrm{treatment} = 1) - \\textrm{avg}(\\textrm{spend} | \\textrm{treatment} = 0) = \\textrm{ATE}$.\n",
"\n",
"Essentially you would compute the average spend of users who received the treatment (received a discount or signed up for rewards) and subtract from that the average spend of users who didn't receive the treatment.\n",
"\n",
"Without discussing the details of the underlying potential outcomes framework, the above expression is called the average treatment effect (ATE).\n",
"\n",
"## Let's estimate the average treatment effect and make a decision!\n",
"\n",
"So now we'll just analyze our e-commerce data of treated and untreated customers and compute the average treatment effect (ATE) for each proposed measure, right? Right?\n",
"\n",
"Before you rush ahead with your ATE computations - now is a good time to take a step back and contemplate how your data was generated in the first place(data-generating process).\n",
"\n",
"## References and further material ...\n",
"\n",
"Before we continue: My example here is based on a tutorial by the authors of the excellent DoWhy library. You can find the original tutorial here:\n",
"\n",
"https://github.com/microsoft/dowhy/blob/master/docs/source/example_notebooks/dowhy_example_effect_of_memberrewards_program.ipynb\n",
"\n",
"And more on DoWhy here: https://microsoft.github.io/dowhy/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "axpOMHkmSMxB"
},
"source": [
"## Install and load libraries"
]
},
{
"cell_type": "code",
"metadata": {
"id": "dbEmWDDng9Xm"
},
"source": [
"!pip install dowhy --quiet"
],
"execution_count": 1,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "NtKCnld4-mAE"
},
"source": [
"import random\n",
"\n",
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"np.random.seed(42)\n",
"random.seed(42)"
],
"execution_count": 2,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "zdJwuUU8it_G"
},
"source": [
"## Randomized controlled trial / experimental data\n",
"\n",
"So where were we ... ah right! Where does our e-commerece data come from?\n",
"\n",
"Since we don't actually run an e-commerce operation here we will have to simulate our data (remember: these ideas are based on the above DoWhy tutorial).\n",
"\n",
"Imagine we observe the monthly spend of each of our 10,000 users over the course of a year. Each user will spend with a certain distribution (here, a Poisson distribution) and there are both high and low spenders with different mean spends.\n",
"\n",
"Over the course of the year, each user can sign up to our rewards program in any month and once they have signed up their spend goes up by 50% relative to what they would've spent without signing up.\n",
"\n",
"So far so mundane: Different customers show different spending behaviour and signing up to our rewards program increases their spend.\n",
"\n",
"Now the big question is: How are treatment assignment (rewards program signup) and outcome (spending behaviour) related? \n",
"\n",
"If treatment and outcome, interpreted as random variables, are independent of one another then according to the potential outcome framework we can compute the ATE as easily as shown above:\n",
"\n",
"$\\textrm{ATE} = \\textrm{avg}(\\textrm{spend} | \\textrm{treatment} = 1) - \\textrm{avg}(\\textrm{spend} | \\textrm{treatment} = 0)$\n",
"\n",
"When are treatment and outcome independent? The gold standard for achieving their independence in a data set is the randomized controlled trial (RCT).\n",
"\n",
"In our scenario what an RCT would look like is randomly signing up our users to our rewards program - indepndent of their spending behaviour or any other characteristic.\n",
"\n",
"So we would go through our list of 10,000 users and flip a coin for each of them, sign them up to our program in a random month of the year based on our coin, and send them on their merry way to continue buying stuff in our online shop.\n",
"\n",
"Let's put all of this into a bit of code that simulates the spending behaviour of our users according to our thought experiment:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "mlw4vvkie8pz"
},
"source": [
"# Creating some simulated data for our example\n",
"num_users = 10000\n",
"num_months = 12\n",
"\n",
"df = pd.DataFrame({\n",
" 'user_id': np.repeat(np.arange(num_users), num_months),\n",
" 'month': np.tile(np.arange(1, num_months+1), num_users), # months are from 1 to 12\n",
" 'high_spender': np.repeat(np.random.randint(0, 2, size=num_users), num_months),\n",
"})\n",
"\n",
"df['spend'] = None\n",
"df.loc[df['high_spender'] == 0, 'spend'] = np.random.poisson(250, df.loc[df['high_spender'] == 0].shape[0])\n",
"df.loc[df['high_spender'] == 1, 'spend'] = np.random.poisson(750, df.loc[df['high_spender'] == 1].shape[0])\n",
"df[\"spend\"] = df[\"spend\"] - df[\"month\"] * 10\n",
"\n",
"signup_months = np.random.choice(\n",
" np.arange(1, num_months),\n",
" num_users\n",
") * np.random.randint(0, 2, size=num_users) # signup_months == 0 means customer did not sign up\n",
"\n",
"df['signup_month'] = np.repeat(signup_months, num_months)\n",
"\n",
"# A customer is in the treatment group if and only if they signed up\n",
"df[\"treatment\"] = df[\"signup_month\"] > 0\n",
"\n",
"# Simulating a simple treatment effect of 50%\n",
"after_signup = (df[\"signup_month\"] < df[\"month\"]) & (df[\"treatment\"])\n",
"df.loc[after_signup, \"spend\"] = df[after_signup][\"spend\"] * 1.5"
],
"execution_count": 3,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "XrY6roZSZjCj"
},
"source": [
"Let's look at user `0` and their treatment assignment as well as spend (since we're sampling random variables here you'll see something different from me):"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 421
},
"id": "hlwbC0lLZiGw",
"outputId": "240c8090-2994-46a0-945a-a4277b3fcf39"
},
"source": [
"df.loc[df['user_id'] == 0]"
],
"execution_count": 4,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
user_id
\n",
"
month
\n",
"
high_spender
\n",
"
spend
\n",
"
signup_month
\n",
"
treatment
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
0
\n",
"
1
\n",
"
0
\n",
"
235
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
1
\n",
"
0
\n",
"
2
\n",
"
0
\n",
"
249
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
2
\n",
"
0
\n",
"
3
\n",
"
0
\n",
"
240
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
3
\n",
"
0
\n",
"
4
\n",
"
0
\n",
"
224
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
4
\n",
"
0
\n",
"
5
\n",
"
0
\n",
"
184
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
5
\n",
"
0
\n",
"
6
\n",
"
0
\n",
"
172
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
6
\n",
"
0
\n",
"
7
\n",
"
0
\n",
"
182
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
7
\n",
"
0
\n",
"
8
\n",
"
0
\n",
"
155
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
8
\n",
"
0
\n",
"
9
\n",
"
0
\n",
"
120
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
9
\n",
"
0
\n",
"
10
\n",
"
0
\n",
"
153
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
10
\n",
"
0
\n",
"
11
\n",
"
0
\n",
"
148
\n",
"
0
\n",
"
False
\n",
"
\n",
"
\n",
"
11
\n",
"
0
\n",
"
12
\n",
"
0
\n",
"
159
\n",
"
0
\n",
"
False
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user_id month high_spender spend signup_month treatment\n",
"0 0 1 0 235 0 False\n",
"1 0 2 0 249 0 False\n",
"2 0 3 0 240 0 False\n",
"3 0 4 0 224 0 False\n",
"4 0 5 0 184 0 False\n",
"5 0 6 0 172 0 False\n",
"6 0 7 0 182 0 False\n",
"7 0 8 0 155 0 False\n",
"8 0 9 0 120 0 False\n",
"9 0 10 0 153 0 False\n",
"10 0 11 0 148 0 False\n",
"11 0 12 0 159 0 False"
]
},
"metadata": {
"tags": []
},
"execution_count": 4
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q7SPSlRTe8p1"
},
"source": [
"## Average treatment effect on post-signup spend for experimental data\n",
"\n",
"The effect we're interested in is the impact of rewards signup on spending behaviour - i.e. the effect on post-signup spend.\n",
"\n",
"Since customers can sign up any month of the year, we'll choose one month at random and compute the effect with respect to that one month.\n",
"\n",
"So let's create a new table from our time series where we collect post-signup spend for those customers that signed up in `month = 6` alongside the spend of customers who never signed up."
]
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "_HxHA4GXe8p3",
"outputId": "39dc76c8-dbcb-4193-a370-b81c1ad85dd2"
},
"source": [
"month = 6\n",
"\n",
"post_signup_spend = (\n",
" df[df.signup_month.isin([0, month])]\n",
" .groupby([\"user_id\", \"signup_month\", \"treatment\"])\n",
" .apply(\n",
" lambda x: pd.Series(\n",
" {\n",
" \"post_spend\": x.loc[x.month > month, \"spend\"].mean(),\n",
" }\n",
" )\n",
" )\n",
" .reset_index()\n",
")\n",
"print(post_signup_spend)"
],
"execution_count": 5,
"outputs": [
{
"output_type": "stream",
"text": [
" user_id signup_month treatment post_spend\n",
"0 0 0 False 152.833333\n",
"1 3 0 False 162.166667\n",
"2 4 0 False 146.333333\n",
"3 6 0 False 153.666667\n",
"4 7 6 True 240.750000\n",
"... ... ... ... ...\n",
"5451 9990 0 False 629.833333\n",
"5452 9993 0 False 674.500000\n",
"5453 9994 0 False 681.000000\n",
"5454 9995 0 False 641.333333\n",
"5455 9998 0 False 658.833333\n",
"\n",
"[5456 rows x 4 columns]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "meQD6PzFq-ML"
},
"source": [
"To get the average treatment effect (ATE) of our rewards signup treatment we now compute the average post-signup spend of the customers who signed up and subtract from that the average spend of users who didn't sign up:"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "mispkMh35rVu",
"outputId": "ebe193cf-a539-4275-b75f-f0b0be235626"
},
"source": [
"post_spend = post_signup_spend\\\n",
" .groupby('treatment')\\\n",
" .agg({'post_spend': 'mean'})\n",
"\n",
"post_spend"
],
"execution_count": 6,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
post_spend
\n",
"
\n",
"
\n",
"
treatment
\n",
"
\n",
"
\n",
" \n",
" \n",
"
\n",
"
False
\n",
"
403.512239
\n",
"
\n",
"
\n",
"
True
\n",
"
610.140371
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" post_spend\n",
"treatment \n",
"False 403.512239\n",
"True 610.140371"
]
},
"metadata": {
"tags": []
},
"execution_count": 6
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wOJ2rZ25rvKE"
},
"source": [
"So the ATE of rewards signup on post-signup spend is:"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5vAxLCIoUTVT",
"outputId": "17a2496e-a140-46df-99b2-2dd2cc04d0bc"
},
"source": [
"post_spend.loc[True, 'post_spend'] - post_spend.loc[False, 'post_spend']"
],
"execution_count": 7,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"206.62813242372852"
]
},
"metadata": {
"tags": []
},
"execution_count": 7
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OKEJdagJr1IO"
},
"source": [
"Since we simulated the treatment effect ourselves (50% post-signup spend increase) let's see if we can recover this effect from our data:"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Ia_DKBM6rmS5",
"outputId": "3af5a20b-51e0-4c71-8e85-16530609d89a"
},
"source": [
"post_spend.loc[True, 'post_spend'] / post_spend.loc[False, 'post_spend']"
],
"execution_count": 8,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.5120740154875112"
]
},
"metadata": {
"tags": []
},
"execution_count": 8
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "urSbFWzysIHV"
},
"source": [
"The post-signup spend for treated customers is roughly 50% greater than the spend for untreated customers - exactly the treatment effect we simulated!\n",
"\n",
"Remember, however, that we are dealing with clean experimental data from a randomized controlled trial (RCT) here! The potential outcome framework tells us that for data from an RCT the simple ATE formula we used here yields the correct treatment effect due to independence of treatment assignment and outcome.\n",
"\n",
"So the fact that we recovered the actual (simulated) treatment effect is nice to see but not surprising."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Nfg1xDo7zABH"
},
"source": [
"## The issue with randomized controlled trials and observational data\n",
"\n",
"Our above thought experiment where we randomly assigned our customers to our rewards program isn't very realistic.\n",
"\n",
"Randomly signing up paying customers to rewards programs without their consent may upset some and may not even be permissible. The same issue with randomized treatment assignment pops up everywhere - clean randomized controlled trials are oftentimes too expensive, infeasible to implement, unethical, or not permitted.\n",
"\n",
"But since we still need to experiment with our shop to drive spending behaviour we'll still go ahead and implement our rewards program. Only that this time we'll place a regular signup page in our shop where our customers can decide for themselves if they want to sign up or not.\n",
"\n",
"Activating our signup page and simply observing how users and their spend behaves gives us **observational data**.\n",
"\n",
"We usually call \"observational data\" just \"data\" without giving much thought to where they came from. I mean we've all dealt with lots of different kinds of data (marketing data, R&D measurements, HR data, etc.) and all these data were simply \"observed\" and didn't come out of a carefully set up experiment.\n",
"\n",
"Simulating our observational data we've got the same 10,000 customers over a span of a year. We still have the same high and low spenders.\n",
"\n",
"Only that now our high spenders are far more likely to sign up to our rewards program than our low spenders. My reasoning for this is that customers who spend more are also more likely to show greater brand loyalty towards us and our rewards program. Further, they visit our shop more frequently hence are more likely to notice our new rewards program and the signup page. We could also add this behaviour as random variables to our simulation below but just take a shortcut and give low spenders a 5% chance of signing up and high spenders a 95% chance."
]
},
{
"cell_type": "code",
"metadata": {
"id": "vFQ5Bgf2zzKQ",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 419
},
"outputId": "2eef91a7-247a-45c1-fbbf-780900964a51"
},
"source": [
"num_users = 10000\n",
"num_months = 12\n",
"\n",
"df = pd.DataFrame({\n",
" 'user_id': np.repeat(np.arange(num_users), num_months),\n",
" 'month': np.tile(np.arange(1, num_months+1), num_users), # months are from 1 to 12\n",
" 'high_spender': np.repeat(np.random.randint(0, 2, size=num_users), num_months),\n",
"})\n",
"\n",
"df['spend'] = None\n",
"df.loc[df['high_spender'] == 0, 'spend'] = np.random.poisson(250, df.loc[df['high_spender'] == 0].shape[0])\n",
"df.loc[df['high_spender'] == 1, 'spend'] = np.random.poisson(750, df.loc[df['high_spender'] == 1].shape[0])\n",
"\n",
"signup_months = df[['user_id', 'high_spender']].drop_duplicates().copy()\n",
"signup_months['signup_month'] = None\n",
"\n",
"signup_months.loc[signup_months['high_spender'] == 0, 'signup_month'] = np.random.choice(\n",
" np.arange(1, num_months),\n",
" (signup_months['high_spender'] == 0).sum()\n",
") * np.random.binomial(1, .05, size=(signup_months['high_spender'] == 0).sum())\n",
"\n",
"signup_months.loc[signup_months['high_spender'] == 1, 'signup_month'] = np.random.choice(\n",
" np.arange(1, num_months),\n",
" (signup_months['high_spender'] == 1).sum()\n",
") * np.random.binomial(1, .95, size=(signup_months['high_spender'] == 1).sum())\n",
"\n",
"df = df.merge(signup_months)\n",
"\n",
"df[\"treatment\"] = df[\"signup_month\"] > 0\n",
"\n",
"after_signup = (df[\"signup_month\"] < df[\"month\"]) & (df[\"treatment\"])\n",
"df.loc[after_signup, \"spend\"] = df[after_signup][\"spend\"] * 1.5\n",
"\n",
"df"
],
"execution_count": 9,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"
"
],
"text/plain": [
" post_spend\n",
"treatment \n",
"False 275.891760\n",
"True 1075.543699"
]
},
"metadata": {
"tags": []
},
"execution_count": 11
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7D74LF4kUAkB",
"outputId": "6d8d05f5-b555-4343-e7d0-111122159cef"
},
"source": [
"post_spend.loc[True, 'post_spend'] - post_spend.loc[False, 'post_spend']"
],
"execution_count": 12,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"799.6519394104552"
]
},
"metadata": {
"tags": []
},
"execution_count": 12
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "NbeU9ZmlZGIZ",
"outputId": "217c9554-bbae-467e-8405-20dafc6c3098"
},
"source": [
"post_spend.loc[True, 'post_spend'] / post_spend.loc[False, 'post_spend']"
],
"execution_count": 13,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"3.8984263250854134"
]
},
"metadata": {
"tags": []
},
"execution_count": 13
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "W308IJJ4ZKBv"
},
"source": [
"Performing the exact same computation as above, now we're estimating an average treatment effect of almost 400% instead of the actual 50%!\n",
"\n",
"So what went wrong here?\n",
"\n",
"Observational data got us!\n",
"\n",
"Realize that in our observational data the outcome (spend) is not indepndent of treatment assignmnet (rewards program signup): High spenders are far more likely to sign up hence are overrepresented in our treatment group while low spenders are overrepresented in our control group (users that didn't sign up).\n",
"\n",
"So when we compute the above difference or ratio we don't just see the average treatment effect of rewards signup we also see the inherent difference in spending between high and low spenders.\n",
"\n",
"So if we ignore how our observational data are generated we'll overestimate the effect our rewards program has and likely make decisions that seem to be supported by data but in reality aren't.\n",
"\n",
"Also notice that we often make this same mistake when training machine learning algorithms on observational data. Chances are someone will ask you to train a regression model to predict the effectiveness of the rewards program and your model will end up with the same inflated estimate as above."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-wv-YTihe8p4"
},
"source": [
"So how do we fix this? And how can we estimate the true treatment effect from our observational data?\n",
"\n",
"Generally, we know from experience in e-commerece that people who tend to spend more are more likely to sign up to our rewards program. So we could segment our users into spend buckets and compute the treatment effect within each bucket to try and breeak this confounding link in our observational data.\n",
"\n",
"Notice that in practice we won't have a `high spender` flag for our customers so we'll have to go by our customers' observed spending behaviour.\n",
"\n",
"The causal inference framework offers an established approach here: Relying on our domain knowledge, we define a causal model that describes how we believe our observational data were generated.\n",
"\n",
"Let's draw this as a graph with nodes and edges:"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 248
},
"id": "Y51lPko52klm",
"outputId": "f02cc228-c836-41e1-a008-6bbb485ec8d4"
},
"source": [
"import os, sys\n",
"sys.path.append(os.path.abspath(\"../../../\"))\n",
"import dowhy\n",
"\n",
"causal_graph = \"\"\"digraph {\n",
"treatment[label=\"Program Signup in month i\"];\n",
"pre_spend;\n",
"post_spend;\n",
"U[label=\"Unobserved Confounders\"]; \n",
"pre_spend -> treatment;\n",
"pre_spend -> post_spend;\n",
"treatment->post_spend;\n",
"U->treatment; U->pre_spend; U->post_spend;\n",
"}\"\"\"\n",
"\n",
"model = dowhy.CausalModel(\n",
" data=post_signup_spend,\n",
" graph=causal_graph.replace(\"\\n\", \" \"),\n",
" treatment=\"treatment\",\n",
" outcome=\"post_spend\"\n",
")\n",
"model.view_model()"
],
"execution_count": 14,
"outputs": [
{
"output_type": "display_data",
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5ycZbn/8c83m0oJoQmEFggoJQmEHoRDEUGaWAIiEgGpAvLzcDyK0qVERUSagvSiIpGjB0QOSEcgIB0SKYEEEgwiJUQMhJTv74/rCUxiNrvZnZ1nduZ6v177yu7MM89cs9m99pr7ue/rlm1SSinVRo+yA0gppWaSSTellGook25KKdVQJt2UUqqhTLoppVRDmXRTSqmGMummlFINZdJNKaUayqSbUko1lEk3pZRqKJNuSinVUCbdlFKqoUy6KaVUQ5l0U0qphjLppm5EQ0EXgsaDZoJc/Du+uH1o2RGm1BZlP91U/7Q2cA2wMdAHaFnIQbOBD4DHga+CX6pdfCm1XybdVOe0N3AlrSfbBc0BZgIHgsd0XVwpdUwm3VTHtDdwFdCvAw9+DzggE2+qN5l0U53S2sDTwBKdOMkMYAh4YnViSqnz8kJaqlfXEEMKndGnOE9KdSOTbqpDGkZcNJtvDFeCCRPmP/KUU2D//Vs9UQswPGc1pHqSSTfVo8PpfJU7T+/ifCnVhUy6qR7tQPtmKrRHz+J8KdWFTLqpHg2u8vnWqfL5UuqwTLqpHvWu8vl6Vfl8KXVYJt1Ujz5Y2I0tLTBr1vy3zZoFvdpOqbPaPCKlGsmkm+rRiwu7cY01YNKk+W+bOBHWXLPN801o84iUaiSTbqpHdxG9FObzpS/B6afDlCkwdy7cfjvcdBOMHLnIc80uzpdSXcgVaakOadjcuTzUowd9K2997z046SQYMwbefhsGD455up/97CJPNgPYCvx0FwacUrtl0k11RVJP4NixYzlj883p0aNHx9+NzZmD587loV69PKKKIabUKTm8kOqGpI2Ah4Cd776bnXr04P3OnG/2bOZssAFrSNqtOhGm1HlZ6abSSeoDnECsHDsOuMK2q9FlTOIt4BLgAeCbtt+oUtgpdUhWuqlUkrYmGo8PATa2fbk/rAQ8BjiAGJed085TzimOPwA8xvYdwFDgdeBpSV+SpOq+ipTaLyvdVApJSwFnAPsAxwC/das/jFobuBoYTiyc6LmQgyp3jhi1sHaOkrYCLiOmkB1p+9VOv5CUFlNWuqnmJH2a6JU7ABhie0zrCRdi6x1vA2wFXAyMnzkTikd8AIwvbt8qjlt4/1zbY4FNgCeAJyQdmlVvqrWsdFPNSFoWOBv4FHCE7Vs6ca6/AZvZ/lsHHz+MqHr/CRxqe6ELMlKqtqx0U01I+jzwDMVuDp1JuNVg+ylgBPBH4CFJx0qqVmezlFqVlW7qUpJWBs4HhgGH2L6vSuftVKW7wLnWAS4lZkkcbPuZzp4zpdZkpZu6hMIBwJPEhauNqpVwq832BGBHYrjhLkknS6p2p7OUgKx0UxeQtCZxYWslonJ8rAueo2qV7gLnXQ24CFiTiP3hap4/pax0U9VI6iHpaOBR4F5gi65IuF3J9hRgT+BM4EZJP5bUmR2JU5pPJt1UFZI+QSTaLwPb2D7TdrfsY+vwa2JRxUDgKUm55U+qiky6qVMk9ZL0XeB+4DpgW9vPlhxWVdj+h+39gP8ErpZ0saRlyo4rdW+ZdFOHSRoOPAxsT4yvXmB7brlRVZ/tm4hlygaekbRnySGlbiyTblpskvpKGg3cCvwU+IztSeVG1bVsv2P7COCrwDmSfi1pxbLjSt1PJt20WCRtQ0wDWxcYZvuqRS/hbSy27yLmHE8hGujsl0uJ0+LIKWOpXSQtDYwGvgB8w/YNJcfTJVPGFjOGzYm5va8AX7c9uaxYUveRlW5qk6RdiAY1SwAblp1w64XtvwCbEePaj0k6XFL+TqVFyko3tUrS8sBPgO2Aw2zfVnJIH6qHSreSpA2Jqvd9ooHOCyWHlOpU/lVO/6ZYwjuSqG6nEQ1q6ibh1iPb44BPAv8LPCjpv4v93lKaT1a6aT6SVgEuBNYnGtTcX3JIC1VvlW4lSWsTWwT1J5YSP1VySKmOZKWbgA+r24OImQnjgeH1mnDrne2XgJ2IHg53SPp+sQ9cSpl0E0haC7gNOBrY2fYJtju1E2+zK5YSXwZsREwxe6zYLig1uUy6TUxSi6RjgL8AtwNb2n6i5LAaSjH88XngVOB3ks6RtGTJYaUSZdJtUpI2AO4DRgJb2/6h7dklh9WQiqr3emIp8QrEooqdSg4rlSSTbpMpGtScQHQEuxbY3vbzJYfVFGy/aXsUMYxzuaTLJA0oO65UW5l0m4ikTYFHgK2BTWz/rBEb1NQ7238kqt73gXGSPldySKmGMuk2AUn9JP2Q2ITxLGB326+UHFZTsz3d9lFE/+EfSbpe0kplx5W6XibdBifpP4hpYIOAobavbaYGNfXO9r3EDIeXiGbpo7KBTmPLxRENSlJ/4AfAZ4Gjbf++5JCqqp4XR3RUMfxzGTAVODzfjTSmrHQbkKTdgGeAXsQS3oZKuI3K9qPA5sCfiXm9R2UDncaTlW4DkbQC0VR8BNGg5o6SQ+oyjVjpVpK0PlH1ziGWYz9XckipSvKvaAMolvB+iWhQ8zrRXLxhE24zsP1XYFvgeuB+ScdlA53GkJVuNydpIPBzYB2iucrYkkOqiUavdCtJGgT8Alie+D/OVYPdWFa63VRR3R5CzEx4gph32xQJt9kU+8/tApwP3CbpDEl9y40qdVQm3W5I0mDgDuBw4FO2T7Y9s+SwUhcqlhJfSUwvWw94QtIny40qdUQm3W6kaFBzLPAQcDMwInu1NhfbU21/ETgeGCPpPElLlR1Xar9Mut2EpCHAA8S8261sn50NappXsU/dEKJR+jOSdi45pNROmXTrnKTekk4G7iKmEO1oe0LJYaU6YPst2wcSw0y/kHSFpOVKDiu1IZNuHSu2+J43YX647V9kg5q0INu3AkOBd4mq94slh5QWIaeM1SFJSwDfB/YH/hO4LvslzK+ZpowtjuLi2mXEisSjbb9WckhpAVnp1hlJOwBPAasQDWp+nQk3tVexr93GwHNEA50Ds4FOfclKt05IWgb4EbAbcKTtm0oOqa5lpds2SRsDlwNvEMvCJ5UbUYKsdOuCpD2Jt4MmGtRkwk2dVqxc2xK4E3hE0jeygU75stItkaQVgXOBLYBDbd9VckjdRla6i0fSJ4BLiULrkKK3QypB/tUrQbGEdz+iQc2rRIOaTLipyxRdyrYDfgXcJ+l7knqVHFZTykq3xiStRjSoGUQ0L3m43Ii6p6x0O07SmsDFwErEz+BjJYfUVLLSrRFJPSQdDjwO/AXYNBNuKoPtl4FdgXOAWySNltSv5LCaRibdGpC0LnEx4yBiy/Pv2/6g5LBSEysa6FwNDAMGEw10ti05rKaQSbcLSeop6VvAg8D/Ap+0Pa7ksFL6kO2/294HOA64TtKFkpYuO65Glkm3i0gaRiTbXYEtbJ9je07JYaW0ULZ/RzTQ6UssJd615JAaVibdKpPUR9KpRL/bi4GdbL9Uclgptcn227YPBg4GfibpaknLlx1Xo8mkW0WStgIeI5Zhbmz70lzCm7ob27cTDXTeIqrevXMpcfXklLEqkLQkcBrwZeCbwPWZbLtWThmrDUkjiAY6zxHL06eWHFK3l5VuJ0n6FLHIYUWiQc1vMuGmRmH7QWA48TP+pKSvZdXbOVnpdpCkAcCPgZ2BI2z/seSQmkpWurUnaSOi6p1GNNDJaxUdkJVuB0jai2hQ8wHRoCYTbmp4tp8EtgJuBR6W9E1JLSWH1e1kpbsYJK0EnEe83TrE9r0lh9S0stItl6SPA5cAvYmlxONLDqnbyEq3HYoGNfsTzcUnARtlwk3NzPbzwA7AVcA9kk6U1LvksLqFrHTbIGkN4CJgVeIv+iMlh5TISreeSFqdmJOevyPtkJVuK4oGNUcSG0PeT/yC5w9TSguwPRnYndj55GZJPyr2+UsLkUl3IYrxqruJjSH/w/YZtmeVG1VK9atooPNLYlHF6sT0su1KDqsuZdKtUDSo+Q7wAPBbYNvssJ9S+9l+3faXgf8Cfinp55L6lx1XPcmkWyjmID4E7ARsbvu8bFCTUsfYvpFooNNCLCXeveSQ6kbTJ11JfSWdDvwJuADY2fbEksNKqduzPc32YcCBwHmSrpW0Qslhla6pk66krYmdHDYgpoFdkUt4U6ou23cSzdL/TlS9+zbzUuKmnDImaSngDGBv4Bjghky23UtOGeueJG0JXA5MIBrovFpySDXXdJWupJ2J5h3LEA1qfpsJN6XasP0QsAnxDvMJSYc2W9XbNJWupGWBnxCraA63fWvJIaVOyEq3+5M0lGig8y5wqO0XSw6pJpqi0pX0BaJBzbtEdZsJN6WS2X4aGAHcDDwk6dhmaKDT0JWupJWJGQlDiAY1fy45pFQlWek2FknrEA10liCWEj9TckhdpiEr3aJBzQFEg5rnia1zMuGmVKdsTwA+RQw33CXp5EZtoNNwSVfSmsAtxLY5n7H9PdvvlxxWSqkNtufa/gXROnUz4FFJW5QcVtV126QraRlJQyq+7iHpaKJBzT3EtuePlRZgSqlDbE8BPgucCdwo6ceVDXQkrSppUEnhdVodJF0NBV0IGg+aCXLx7/ji9qGtPPBSYKykj0laD7gX2BfYxvbobFCTUvdVNND5NdFAZxXgKUk7SOoJ3AHcXny+EB3OKTVR4oU0rQ1cQ2xX3odYo72g2cSWOI8DX4XYk6lYSfYnoBfwIrEp5CnAz2zP7fLQU+nyQlpzkbQH8HNgKrAhYOA42xdUHNXhnFJLJSVd7Q1cSevfmAXNAWYCB0r8DzAO+ETFfUfYvrQLAk11KpNu85G0CfAwH+WMd4FBtt/sTE4Bj6l+tK0rYXhBexNbfCxB+745FMctAVx11llcDKy9wH0/krRMVcNMKdWNYtXaL5g/Zy0BXNDZnFI8vmZqXOlqbWIJboe7yv/rXzB0KJ44kbnAe0QTjYnAAVn1NI+sdJtLsWhi3hjvykB/wGutRcsLL/B+Swt9O3H6GcAQqE13wUVWupImSdqpis93DVH+c8opsP/+i3+Cfv2Y8/TTPAYsZXtp2+vY/vTi/vJJcjEhO6VU52zPsb0P0fP6QqAvMHjsWJ7t0YNenTx9HyI31USHhxdav3LY6iOGEQPcnVrm16MHLUsuyfo263bmPCml7sv2LJtlPvYx1pA6l1OInDS8ZrMabLf6AR++hX8X+DZxxfBg4BXg3uKYrwF/Bd4GbgXWrHj8ucBkYPqgQbx+zz3MtvEtt+BevXDPnnjJJfGwYfGU222Hjz8ejxgRt++xB37jDbzffnjppfFmm+GJE7HNLJsLgPWIWQxvAc8B+1Q895XEX8SbgX8SfyEHF/fdW7yWfxWv7UuL+j7kR/19AH8DBpYdR360+f80CfguML7IEVcAfYv7DiVaPL4F3Djv/xMQcA7wOjCduHD+IPAzYBYx++DdTTdloh05ZWEfP/gBHjgQL7UU/vjH8e23x+0nn4y/+EW8zz5x3/Dh+PHHI6cUzz8QuAH4BzF0eUzF6zkFuB64usgr44hhrnn3DwceW+T3pB3fsJ2KzwcViepqYEmgH7BX8U1bH+gJnAA8UPH4/YHlgZ6nnsprK62E33vvoxf+la/M/5TbbYcHD8YTJuBp0/D66+N118V/+hOeNQuPGoUPPDCOfecdxhcJ/aDiuYcDbwAbFM99JfAmsEVx/y+B6ypiM7BO2T+U+dGxj0y63eOjyCHPEJtVLkfsrH06sGPx+7oJ8fb+fD4q5HYhFjkNKBLwTkWi/WeRhK8HZDO+tad+9lm82mr41Vfj64kTI6/YkXt69sRjxuAPPsBnnYUHDcIzZzKOePf/KHAS0Ju4aP8SsEsR2ynA+8BuRIU8Ghhb3NcbeBn4z0V9TxZziACAU2z/C0DSEcBoF5s3SjoT+J6kNW2/bPvajx6mZc85B557DjbaqPWTH3QQDB4cn++6K4wfDzsVo8p77w0nnhif/+EPfIL4K7QkcETx8GeBsyTdQlTBzxFJdwvgHeALxaq1efaX9EYHvgepfEsAB0l6p+xA0iItDYwlCjSAJ4HDiaT7OLB18TEROFLSScAKRJI+gUhiyxHvupcqzvEF4PU5cxjQ0srAQksLzJwZ+WPFFWHQoPnv33RTGDkyPj/2WDj7bHjwQdYFNgdWtP394tCXJF1CLLya153wz7b/CCDpGqLlAMBWxNqBny7qG9KRpDu54vM1gXMlnV1xm4BVgZclfYsYjhi4zDL0nj4d3mgjxa200kef9+v371+/+24RxGR6AGsBCz73c0TCXYYYOlivuG8V4j9tvYrjBxP/wan76UX8/80oO5C0SL2Iwmje790AYFkiR0xi/t/H94ltfV4jCqivEkl7MvNffxIwoKWl9fy1zjrw05/GBftx42CXXeAnP4GBA+P+1Vf/6NgePWC11WDqVHoSOW2gpGkVp2sB7qv4+rWKz2cAfYtrXAOBV12Uva1pK+ku7MGVt00GznDsdz8fSdsS48CfAsZNm8Z7yy5L73nhdLZX/GqrMRu42/anF3a/pCuBKbZPKL7entjl9+ji66OAUx3djVI3U/RIPsE5ZayuFSvJbrZ9UfH1rsCWwN3Am7a/Xdy+JFEBf8v2pIrHf4wY712buL7Uixii2JMYnmi1E9l++8XH9Olw+OHwne/ANcUchckVpePcuTBlCqyyCrOJnDbRdkcu1E8FVpWkRSXetmYv/J35FyIs6CLgu5I2hA+b0MybaLw0seTuH0DP73yHd6ZP/+iBK60EkybFC+6I3XdnAvBxSaMk9So+Npe0fjtP0dZrSylVx1GSVpO0HHA88Btizu1BkjaW1IdobvOQ7UnF7/GWknoRF7v/RVSUZxJjv3+z/U+iBcBCPfcc3HlnDDH07RvvkntUZLtHH4X/+R+YPTsq4j59YMQIXiBWvP1T0nck9ZPUImmIpM3b8TofJHLeMYs6qK2kOxo4oSi1Ry54p+3fAT8ErpM0nRgw37W4+1bg/4h+ti+/8w4TKkv6vYvUvPzysMkmbb6YBc0eMIA7gJ2JsZa/ESX/DynmAbfDKcBVkqZJ2mexI0gptdevgNuIC1IvAqfbvh04kZglMJUYKtq3OL4/0dD8bWJM9zXiounpRP+FDSRN22IL+hJJ7t/MnAnHHQcrrAArrwyvvw6jR390/157wW9+A8suG9XvDTcwu3dv7rI9B9iDmN46kaimLyWGKxfJ9gfEePOBizquhivSNIz4S9Dh1Wjz2MyQ2Ar8dOfjSt1RrkjrHiRNInZtub3a5541S8N69mSsRL/Fedwpp8CECXDttfPdPANqk1Nq2HvBTwFPEI0mOmz2bHjgAZaQuFPSk5JulPSj4q1LSqkBFf2yj5c0RtIjkqb27s1jDzxAv7lzO5dTiJz0eK2KuI7MXuiMUXSy98LcucweNQoRsw5WIK527sz8sxhSSo3FwGHAGpU3brstc5dY4t9XpI0fD2usseCtrZpJ5KaaKKG144cdgRbrLUHhvenTOWSZZTgH+Fhx2yzg27YXOTcuNZYcXmg+kj4J3AUf9lp4D9jQZjM6kVOAA2rZ3rHWlS7x4gQd7H3Zv7/HgGYA1xLf5OnA/5M0zvafuibmlFKZJG0A/Ji4sLUsUfmea3siMLEzOaUJ+ulC8SKHEitVZtDKFcji9hnFcUMqvjn/S6xsmUOsZvk6cKmkyyQt25WRp5RqR1JvSScS+x5eRWxeMJ2YRnbaR0d2OqfUTInb9XwYwlBiUvQOwDrEW4dZRE+Hu4CLFzbAXez6u2HFcryliSlunweOLqazpQaVwwuNT9JmxJbsU4jdYSYXtw8H+tge28ojO5RTaqUOkm51FSvhLgWeAr5h+7U2HpK6oUy6javY+fcU4ADgWOBXbS2t7U7qYDfg6rJ9H7AR8VftSUlfLbb6SCnVOUnbEUOHqwNDbf+ykRIuNGClW6nYyO4yYsnv4bZfLjmkVCVZ6TYWSf2JFaV7AkfavrHkkLpMw1W6lWw/RrR1vBd4VNJRkhr6NafU3UjanWgh0AIMaeSECw1e6VaStB5R9c4lliU+V3JIqROy0u3+JK1A9J4dARxq+86SQ6qJpqn6bD8LbEt0OPqzpOOKLkYppRpS2Jeobv9OjN02RcKFJqp0K0kaBFwMrAgcbPvxUgNKiy0r3e5J0qrEXmeDid+9h0oOqeaaptKtVDRJ/gyxceatks6U1LfcqFJqXEV1eyjR9OpxYJNmTLjQpEkXwOEqomHOx4EnirXdKaUqkjQYuIPY/XdH26cUvWebUtMm3Xlsv2Z7JNHRfoyk84vVbSmlTih2XTgWeAj4AzDCzh7YTZ9057F9A7AhsYne05J2KTmklLotSUOAB4hdGLa0/ZNiV4aml0m3gu23bX+N6Nt5kaQrszl6Su1XNKg5mehxcCnwKdut7mXWjDLpLoTt24iORe8Az0j6YskhpVT3JG0BPApsCmxs+5JGW8JbDU05ZWxxFBfXLgXGE93LppYcUiKnjNWTokHNacBXgG8Cv8lk27qsdNtg+35gOPBXooHOQdlAJ6UgaQdiC66ViCW812XCXbSsdBeDpI2JpcRvAocV831TCbLSLZekZYCziPnuX7d9c8khdRtZ6S4G208AWxJzDh+RdIyk9mwNklLDkLQnsYR3DlHdZsJdDFnpdpCkjxNjvS1EA52/lhxSU8lKt/YkrQicB2xG/MzfU3JI3VJWuh1k+3lge+CXwL2Sjs8GOqkRFUt49yPGbicDG2XC7bisdKtA0hrARcBAoonHoyWH1PCy0q0NSasDPwfWAL5m+5GSQ+r2stKtAtuvALsTW0T/UdIPJPUrOayUOkxSD0lHAI8RO+dulgm3OjLpVknRQOdaYlHFWsT0sv8oOayUFpukdYE7iY0ht7N9ejM3qKm2TLpVZvt1218Cvg38StKFxf5PKdU1ST0l/TfwIPA7YBvb40sOq+Fk0u0itn8PDAH6EA10di05pJRaJWkYkWx3AbawfW42qOkamXS7kO1ptg8BvgZcKOlqScuXHVdK80jqI+n7wO3EBbNP236p5LAaWibdGrB9BzHW+wbRQGefXEqcyiZpK+JC2VCiQc3luYS36+WUsRorftAvA14AjswpTx2TU8Y6TtKSwOnAvsAxwG8z2dZOVro1ZnsssAnwJLFF0MFZ9aZakbQTschhOWIJ75hMuLWVlW6JiosXlwHTgUNzLK39stJdPJIGAGcDOwFH2L6l5JCaVla6JbL9FDACuAV4WNI3s4FOqjZJnwPGAe8R1W0m3BJlpVsnJK1DNNDpSywlHldySHUtK922SVoJOB/YiGhQc1/JISWy0q0bticAOwKXA3dLOlFS75LDSt1Q0aBmFPAU8CLRoCYTbp3ISrcOSVqNaKCzBlH1/qXkkOpOVroLVzRfuhhYmfjZeazkkNICstKtQ7anAHsCo4GbJJ1V7EOV0kIVDWqOJDaGvI9YVZYJtw5l0q1TRQOdXxMT11clGuhsX25UqR5J+gRwD7Ex5La2z7Q9q+SwUisy6dY52/+wvR9wLHCNpIuK/alSkysa1BwH3A9cTyTcZ0sOK7Uhk243YfsmooEORAOd3cuMJ5Wr2CT1IWAHYmz7fNtzSw4rtUMm3W7E9ju2jyD6nJ4r6ZfFvlWpSUjqK+kM4FZiv7LP5K7U3Usm3W7I9l3AMOBvRNX75VxK3PgkfRJ4AvgEMQ3sqlzC2/3klLFuTtIWxFLiScDXi5kPDa+ZpoxJWgo4ExgJfMP2DSWHlDohK91uzvbDwKbAX4DHJR0mKf9fG4SknYFngKWJJbyZcLu5rHQbiKQhRNU7g2igM6HkkLpMo1e6kpYjGtTsABxm+7aSQ0pVkhVRA7H9DLA1cCMwVtJ/SepZclhpMUn6IlHdTieq20y4DSQr3QYlaW3gEuJt6cG2ny45pKpqxEpX0irABcAGRIOa+0sOKXWBrHQbVNGbdyfgF8Cdkk6V1KfksNJCFA1qDiQa2/8VGJ4Jt3Fl0m1gxVLiS4GNi49HJW1ZclipgqRBxJzbY4CdbZ9g+/1Sg0pdKpNuE7D9KvA54DTg95J+UuyTlUpSNKj5BjHr5A6iQc0TJYeVaiCTbpMoqt7fEA10VgSekrRjyWE1JUnrE53A9gG2sf1D27NLDivVSCbdJmP7DdujiLezV0q6pNg/K3UxSb0kHU8k3F8C29l+ruSwUo1l0m1Stm8mGujMAp6RtFfJITU0SZsQQwnbAJvY/lk2qGlOmXSbmO3pto8E9gPOknSdpI+VHVcjkdRP0g+IzUfPBnaz/UrJYaUSZdJN2L6X2LzwZaKBzv7ZQKfzJG1LNKhZCxhm+5psUJNycUSaj6TNiKXEU4AjbE8uOaSFqufFEZL6E1stfQ44yvbvSw4p1ZGsdNN8bD8CbAY8CDwm6evZQKf9JO0KPA30IZbwZsJN88lKN7VK0gZE1TuLWJb6fMkhfajeKl1JywPnEBfKDrN9e8khpTqVFUxqle3xRBK5AXhA0rezgc78iiW8+xANat4EhmbCTYuSlW5qF0lrEX0clgO+ZvvJkuMpvdKVNBD4GbAu8U7gwbJiSd1HVrqpXWxPBHYmumD9SdLpkvqWHFYpiur2YGJmwlPEvNtMuKldMummdiuWEl9BTC/bgNipYuuSw6qpomXm7cDXgZ1sn2R7ZslhpW4kk25abLanAl8ETgR+K+ncYh+vhiWpRdI3gYeB/wO2sv1UyWGlbiiTbuqQour9LdFAZwCxqOLTJYfVJSRtCNwPfB4YYfusbFCTOiqTbuoU22/aPoB4u32ppMslLVt2XNUgqbekk4C7gSuBHWy/UGpQqdvLpJuqwvb/EQ10ZgDjJH2h5JA6RdLmwCPAlsSFsouyQU2qhpwylqqu6DlwKbEy62jbr1XpvAIGE8XCfcS48uvARNuzqvQcSwCnAqOAY4FfZ7+EVE1Z6aaqs30fMcPheaJZ+gFVaqCzMfAC8CjRiP0WYk+xr1bh3EjantinbDVikcOvMuGmastKN3WpomGRh88AAAYTSURBVI/sZcDfgcNtv9yJc4lY+bVBxc3vAqvant6J8y4D/BDYAzjS9o0dPVdKbclKN3Up248BWwD3EBtjHt3RBjpF1fktItECvA/8uJMJdw8ikfcANsyEm7paVrqpZiStR1S9JpbNPtuBc1RWu/8CBnYk6UpaETiXuFB2qO07F/ccKXVEVrqpZookuy1wHfBnSd+T1GsxzzGv2gW4aHETbrGE98vERb6pxNhtJtxUM1npplJIGgRcDHyMaKDzeDseNRQ4wmaH2bNZr1cvBHwAvAjcBVwEfnoRz7ka8HNiJ4eDbT/UyZeR0mLLpJtKUwwVfBU4ixh2ONX2+ws5cm3gGmL2Qh+gZSGnm00k4MfjnH6p4nl6AIcAZxANe0bb/qCaryWl9sqkm0onaWXgfGAYUYH+ueLevYnVYK0l2wXNAWYCB4LHSFoHuARYojj3M9WMPaXFlWO6qXS2X7O9N/Bd4HpJF0hauki4VxEJsz0Jl+K4JWyuGj1aVwNjgZuArTPhpnqQlW6qK0XfhrPXXZdd/vpXlm9poU9Hz/X++8w9+2x2PP5431Ol2CYRsy5yZ4jUYZl0U1164w2NW3ZZ1mtp6fi7MZs5EmPB21Qjpky6qRpyeCHVTPv3V9OwFVZgUGcSbjwfLcDwYtZDSnUhk27qNEmTJH1X0nhJb0u6QlJfSdtLmiLpO5JeA66Q1EPScZJelPSmpOslLbfAKQ+Hj4YV3n8f9t8fll8eBgyAzTeHv/897tt+e/jud2GLLaB/f9hrL3jrrY9O9OCD9Fl3XW6TNE3Sk0V/hXlx3y3pNEn3S/qnpNskrVBx/yhJLxdxHt8F37rUhDLppmr5CrAL0QXs48AJxe0rE5tZrgkcBnwD+BywHTAQeBu4cIFz7UDFhbOrroJ33oHJk+HNN+Gii6Bfv48OvvpquPxymDoVevaEY46J2199FfbYg5bRo/mgiOFbwA3FarR59gMOIuYL9y6Ombf9/M+JbmMDgeWJRjgpdUom3VQtF9iebPstYj7sl4vb5wIn255p+z3gCOB421OKvcVOAUYuMPQwuPLEvXpFsp0wAVpaYNNNo6qdZ9QoGDIEllwSTjsNrr8e5syBa6+F3XaDkSNZ2fZc238ieuTuVnH6K2w/X8R2PTEXGGAk8Afb9xZxnli8lpQ6JZNuqpbJFZ+/TFSHAP9YYMHDmsDvirf704jWjHOAlSqO6V154lGjYJddYN99YeBA+Pa3YVZF99zVV684+Zpx3xtvwMsvw5gxMGAAvSuebxtglYrTV/b6nQHM2+ttYOVrsv0v4M02vwsptSGTbqqWitTHGsDfis8XnB4zGdjV9oCKj762X604Zr7VYr16wcknw/jx8MAD8Ic/xJDChyesSPevvBLHr7BCJONRo2DaND6oeK4lbf+gHa9nauVrKpqbL9+Ox6W0SJl0U7UcJWm14qLY8cBvWjnuIuAMSWtCdPuStNcCx7xY+cVdd8HTT8eQQf/+kVR7VPzkXnttJOQZM+Ckk2DkyBiG2H9/uOkm+N3vmFrs5jvv4l57xmZ/C+whaRtJvYHvk78vqQryhyhVy6+A24CXiKR5eivHnQvcCNwm6Z/EirEtFzjmLqKXAgCvvRaJtH9/WH992G67qGDnGTUKDjwQVl45Zjqcd17cvvrq8PvfM+d736MX8A+iyv5v2vFzb3sccFTxuqYSF/ymtPW4lNqSiyNSp1V/0YCGAQ8Sy38Xafvto6I95JBWD5kBbLWo7mMp1VJWuqkO+SngCeICW2fMAR7PhJvqSSbdVBckfUXSu/M+evRg46WWomXDDTt12pnEPNuU6kYOL6Q69mGXsX5tHbkQ7wEHgMdUN6aUOqeda+FTKoPHgKAT/XS7LLSUOigr3dQNaG3gamA4sXBiYcVC5c4Ro8ATaxdfSu2XSTd1IxpKNMPZAVgH6AXMAiYQ08wuzotmqd5l0k0ppRrK2QsppVRDmXRTSqmGMummlFINZdJNKaUayqSbUko1lEk3pZRqKJNuSinVUCbdlFKqoUy6KaVUQ5l0U0qphjLpppRSDWXSTSmlGsqkm1JKNZRJN6WUauj/A/N2LzrFWJsFAAAAAElFTkSuQmCC\n",
"text/plain": [
"