{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": false, "sideBar": true, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false }, "colab": { "name": "2021-08-18-Causal_inference_for_decision_making_in_growth_hacking_and_upselling_in_Python.ipynb", "provenance": [], "collapsed_sections": [] } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "iOQxSDQrumWb" }, "source": [ "# Causal inference for decision-making in growth hacking and upselling in Python\n", "\n", "> \"In this article we discuss differences between experimental and observational data and pitfalls in using the latter for data-driven decision-making.\"\n", "- toc: true\n", "- branch: master\n", "- badges: true\n", "- comments: true\n", "- categories: [python, dowhy, causal inference]" ] }, { "cell_type": "markdown", "metadata": { "id": "6Oo6ETETld4t" }, "source": [ "\n", "## Introduction\n", "\n", "Wow, growth hacking *and* upselling all in the same article? Also Python.\n", "\n", "Okay, let's start at the beginning. Imagine the following scenario: You're responsible for increasing the amount of money users spend on your e-commerce platform.\n", "\n", "You and your team come up with different measures you could implement to achieve your goal. Two of these measures could be:\n", "\n", "- Provide a discount on your best-selling items,\n", "- Implement a rewards program that incentivices repeat purchases.\n", "\n", "Both of these measures are fairly complex with each incurring a certain, probably known, amount of cost and an unknown effect on your customers' spending behaviour.\n", "\n", "To decide which of these two possible measures is worth both the effort and incurred cost you need to estimate their effect on customer spend.\n", "\n", "A natural way of estimating this effect is computing the following:\n", "\n", "$\\textrm{avg}(\\textrm{spend} | \\textrm{treatment} = 1) - \\textrm{avg}(\\textrm{spend} | \\textrm{treatment} = 0) = \\textrm{ATE}$.\n", "\n", "Essentially you would compute the average spend of users who received the treatment (received a discount or signed up for rewards) and subtract from that the average spend of users who didn't receive the treatment.\n", "\n", "Without discussing the details of the underlying potential outcomes framework, the above expression is called the average treatment effect (ATE).\n", "\n", "## Let's estimate the average treatment effect and make a decision!\n", "\n", "So now we'll just analyze our e-commerce data of treated and untreated customers and compute the average treatment effect (ATE) for each proposed measure, right? Right?\n", "\n", "Before you rush ahead with your ATE computations - now is a good time to take a step back and contemplate how your data was generated in the first place(data-generating process).\n", "\n", "## References and further material ...\n", "\n", "Before we continue: My example here is based on a tutorial by the authors of the excellent DoWhy library. You can find the original tutorial here:\n", "\n", "https://github.com/microsoft/dowhy/blob/master/docs/source/example_notebooks/dowhy_example_effect_of_memberrewards_program.ipynb\n", "\n", "And more on DoWhy here: https://microsoft.github.io/dowhy/" ] }, { "cell_type": "markdown", "metadata": { "id": "axpOMHkmSMxB" }, "source": [ "## Install and load libraries" ] }, { "cell_type": "code", "metadata": { "id": "dbEmWDDng9Xm" }, "source": [ "!pip install dowhy --quiet" ], "execution_count": 1, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "NtKCnld4-mAE" }, "source": [ "import random\n", "\n", "import pandas as pd\n", "import numpy as np\n", "\n", "np.random.seed(42)\n", "random.seed(42)" ], "execution_count": 2, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "zdJwuUU8it_G" }, "source": [ "## Randomized controlled trial / experimental data\n", "\n", "So where were we ... ah right! Where does our e-commerece data come from?\n", "\n", "Since we don't actually run an e-commerce operation here we will have to simulate our data (remember: these ideas are based on the above DoWhy tutorial).\n", "\n", "Imagine we observe the monthly spend of each of our 10,000 users over the course of a year. Each user will spend with a certain distribution (here, a Poisson distribution) and there are both high and low spenders with different mean spends.\n", "\n", "Over the course of the year, each user can sign up to our rewards program in any month and once they have signed up their spend goes up by 50% relative to what they would've spent without signing up.\n", "\n", "So far so mundane: Different customers show different spending behaviour and signing up to our rewards program increases their spend.\n", "\n", "Now the big question is: How are treatment assignment (rewards program signup) and outcome (spending behaviour) related? \n", "\n", "If treatment and outcome, interpreted as random variables, are independent of one another then according to the potential outcome framework we can compute the ATE as easily as shown above:\n", "\n", "$\\textrm{ATE} = \\textrm{avg}(\\textrm{spend} | \\textrm{treatment} = 1) - \\textrm{avg}(\\textrm{spend} | \\textrm{treatment} = 0)$\n", "\n", "When are treatment and outcome independent? The gold standard for achieving their independence in a data set is the randomized controlled trial (RCT).\n", "\n", "In our scenario what an RCT would look like is randomly signing up our users to our rewards program - indepndent of their spending behaviour or any other characteristic.\n", "\n", "So we would go through our list of 10,000 users and flip a coin for each of them, sign them up to our program in a random month of the year based on our coin, and send them on their merry way to continue buying stuff in our online shop.\n", "\n", "Let's put all of this into a bit of code that simulates the spending behaviour of our users according to our thought experiment:" ] }, { "cell_type": "code", "metadata": { "id": "mlw4vvkie8pz" }, "source": [ "# Creating some simulated data for our example\n", "num_users = 10000\n", "num_months = 12\n", "\n", "df = pd.DataFrame({\n", " 'user_id': np.repeat(np.arange(num_users), num_months),\n", " 'month': np.tile(np.arange(1, num_months+1), num_users), # months are from 1 to 12\n", " 'high_spender': np.repeat(np.random.randint(0, 2, size=num_users), num_months),\n", "})\n", "\n", "df['spend'] = None\n", "df.loc[df['high_spender'] == 0, 'spend'] = np.random.poisson(250, df.loc[df['high_spender'] == 0].shape[0])\n", "df.loc[df['high_spender'] == 1, 'spend'] = np.random.poisson(750, df.loc[df['high_spender'] == 1].shape[0])\n", "df[\"spend\"] = df[\"spend\"] - df[\"month\"] * 10\n", "\n", "signup_months = np.random.choice(\n", " np.arange(1, num_months),\n", " num_users\n", ") * np.random.randint(0, 2, size=num_users) # signup_months == 0 means customer did not sign up\n", "\n", "df['signup_month'] = np.repeat(signup_months, num_months)\n", "\n", "# A customer is in the treatment group if and only if they signed up\n", "df[\"treatment\"] = df[\"signup_month\"] > 0\n", "\n", "# Simulating a simple treatment effect of 50%\n", "after_signup = (df[\"signup_month\"] < df[\"month\"]) & (df[\"treatment\"])\n", "df.loc[after_signup, \"spend\"] = df[after_signup][\"spend\"] * 1.5" ], "execution_count": 3, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "XrY6roZSZjCj" }, "source": [ "Let's look at user `0` and their treatment assignment as well as spend (since we're sampling random variables here you'll see something different from me):" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 421 }, "id": "hlwbC0lLZiGw", "outputId": "240c8090-2994-46a0-945a-a4277b3fcf39" }, "source": [ "df.loc[df['user_id'] == 0]" ], "execution_count": 4, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_idmonthhigh_spenderspendsignup_monthtreatment
00102350False
10202490False
20302400False
30402240False
40501840False
50601720False
60701820False
70801550False
80901200False
901001530False
1001101480False
1101201590False
\n", "
" ], "text/plain": [ " user_id month high_spender spend signup_month treatment\n", "0 0 1 0 235 0 False\n", "1 0 2 0 249 0 False\n", "2 0 3 0 240 0 False\n", "3 0 4 0 224 0 False\n", "4 0 5 0 184 0 False\n", "5 0 6 0 172 0 False\n", "6 0 7 0 182 0 False\n", "7 0 8 0 155 0 False\n", "8 0 9 0 120 0 False\n", "9 0 10 0 153 0 False\n", "10 0 11 0 148 0 False\n", "11 0 12 0 159 0 False" ] }, "metadata": { "tags": [] }, "execution_count": 4 } ] }, { "cell_type": "markdown", "metadata": { "id": "Q7SPSlRTe8p1" }, "source": [ "## Average treatment effect on post-signup spend for experimental data\n", "\n", "The effect we're interested in is the impact of rewards signup on spending behaviour - i.e. the effect on post-signup spend.\n", "\n", "Since customers can sign up any month of the year, we'll choose one month at random and compute the effect with respect to that one month.\n", "\n", "So let's create a new table from our time series where we collect post-signup spend for those customers that signed up in `month = 6` alongside the spend of customers who never signed up." ] }, { "cell_type": "code", "metadata": { "scrolled": true, "colab": { "base_uri": "https://localhost:8080/" }, "id": "_HxHA4GXe8p3", "outputId": "39dc76c8-dbcb-4193-a370-b81c1ad85dd2" }, "source": [ "month = 6\n", "\n", "post_signup_spend = (\n", " df[df.signup_month.isin([0, month])]\n", " .groupby([\"user_id\", \"signup_month\", \"treatment\"])\n", " .apply(\n", " lambda x: pd.Series(\n", " {\n", " \"post_spend\": x.loc[x.month > month, \"spend\"].mean(),\n", " }\n", " )\n", " )\n", " .reset_index()\n", ")\n", "print(post_signup_spend)" ], "execution_count": 5, "outputs": [ { "output_type": "stream", "text": [ " user_id signup_month treatment post_spend\n", "0 0 0 False 152.833333\n", "1 3 0 False 162.166667\n", "2 4 0 False 146.333333\n", "3 6 0 False 153.666667\n", "4 7 6 True 240.750000\n", "... ... ... ... ...\n", "5451 9990 0 False 629.833333\n", "5452 9993 0 False 674.500000\n", "5453 9994 0 False 681.000000\n", "5454 9995 0 False 641.333333\n", "5455 9998 0 False 658.833333\n", "\n", "[5456 rows x 4 columns]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "meQD6PzFq-ML" }, "source": [ "To get the average treatment effect (ATE) of our rewards signup treatment we now compute the average post-signup spend of the customers who signed up and subtract from that the average spend of users who didn't sign up:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 142 }, "id": "mispkMh35rVu", "outputId": "ebe193cf-a539-4275-b75f-f0b0be235626" }, "source": [ "post_spend = post_signup_spend\\\n", " .groupby('treatment')\\\n", " .agg({'post_spend': 'mean'})\n", "\n", "post_spend" ], "execution_count": 6, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
post_spend
treatment
False403.512239
True610.140371
\n", "
" ], "text/plain": [ " post_spend\n", "treatment \n", "False 403.512239\n", "True 610.140371" ] }, "metadata": { "tags": [] }, "execution_count": 6 } ] }, { "cell_type": "markdown", "metadata": { "id": "wOJ2rZ25rvKE" }, "source": [ "So the ATE of rewards signup on post-signup spend is:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5vAxLCIoUTVT", "outputId": "17a2496e-a140-46df-99b2-2dd2cc04d0bc" }, "source": [ "post_spend.loc[True, 'post_spend'] - post_spend.loc[False, 'post_spend']" ], "execution_count": 7, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "206.62813242372852" ] }, "metadata": { "tags": [] }, "execution_count": 7 } ] }, { "cell_type": "markdown", "metadata": { "id": "OKEJdagJr1IO" }, "source": [ "Since we simulated the treatment effect ourselves (50% post-signup spend increase) let's see if we can recover this effect from our data:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Ia_DKBM6rmS5", "outputId": "3af5a20b-51e0-4c71-8e85-16530609d89a" }, "source": [ "post_spend.loc[True, 'post_spend'] / post_spend.loc[False, 'post_spend']" ], "execution_count": 8, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "1.5120740154875112" ] }, "metadata": { "tags": [] }, "execution_count": 8 } ] }, { "cell_type": "markdown", "metadata": { "id": "urSbFWzysIHV" }, "source": [ "The post-signup spend for treated customers is roughly 50% greater than the spend for untreated customers - exactly the treatment effect we simulated!\n", "\n", "Remember, however, that we are dealing with clean experimental data from a randomized controlled trial (RCT) here! The potential outcome framework tells us that for data from an RCT the simple ATE formula we used here yields the correct treatment effect due to independence of treatment assignment and outcome.\n", "\n", "So the fact that we recovered the actual (simulated) treatment effect is nice to see but not surprising." ] }, { "cell_type": "markdown", "metadata": { "id": "Nfg1xDo7zABH" }, "source": [ "## The issue with randomized controlled trials and observational data\n", "\n", "Our above thought experiment where we randomly assigned our customers to our rewards program isn't very realistic.\n", "\n", "Randomly signing up paying customers to rewards programs without their consent may upset some and may not even be permissible. The same issue with randomized treatment assignment pops up everywhere - clean randomized controlled trials are oftentimes too expensive, infeasible to implement, unethical, or not permitted.\n", "\n", "But since we still need to experiment with our shop to drive spending behaviour we'll still go ahead and implement our rewards program. Only that this time we'll place a regular signup page in our shop where our customers can decide for themselves if they want to sign up or not.\n", "\n", "Activating our signup page and simply observing how users and their spend behaves gives us **observational data**.\n", "\n", "We usually call \"observational data\" just \"data\" without giving much thought to where they came from. I mean we've all dealt with lots of different kinds of data (marketing data, R&D measurements, HR data, etc.) and all these data were simply \"observed\" and didn't come out of a carefully set up experiment.\n", "\n", "Simulating our observational data we've got the same 10,000 customers over a span of a year. We still have the same high and low spenders.\n", "\n", "Only that now our high spenders are far more likely to sign up to our rewards program than our low spenders. My reasoning for this is that customers who spend more are also more likely to show greater brand loyalty towards us and our rewards program. Further, they visit our shop more frequently hence are more likely to notice our new rewards program and the signup page. We could also add this behaviour as random variables to our simulation below but just take a shortcut and give low spenders a 5% chance of signing up and high spenders a 95% chance." ] }, { "cell_type": "code", "metadata": { "id": "vFQ5Bgf2zzKQ", "colab": { "base_uri": "https://localhost:8080/", "height": 419 }, "outputId": "2eef91a7-247a-45c1-fbbf-780900964a51" }, "source": [ "num_users = 10000\n", "num_months = 12\n", "\n", "df = pd.DataFrame({\n", " 'user_id': np.repeat(np.arange(num_users), num_months),\n", " 'month': np.tile(np.arange(1, num_months+1), num_users), # months are from 1 to 12\n", " 'high_spender': np.repeat(np.random.randint(0, 2, size=num_users), num_months),\n", "})\n", "\n", "df['spend'] = None\n", "df.loc[df['high_spender'] == 0, 'spend'] = np.random.poisson(250, df.loc[df['high_spender'] == 0].shape[0])\n", "df.loc[df['high_spender'] == 1, 'spend'] = np.random.poisson(750, df.loc[df['high_spender'] == 1].shape[0])\n", "\n", "signup_months = df[['user_id', 'high_spender']].drop_duplicates().copy()\n", "signup_months['signup_month'] = None\n", "\n", "signup_months.loc[signup_months['high_spender'] == 0, 'signup_month'] = np.random.choice(\n", " np.arange(1, num_months),\n", " (signup_months['high_spender'] == 0).sum()\n", ") * np.random.binomial(1, .05, size=(signup_months['high_spender'] == 0).sum())\n", "\n", "signup_months.loc[signup_months['high_spender'] == 1, 'signup_month'] = np.random.choice(\n", " np.arange(1, num_months),\n", " (signup_months['high_spender'] == 1).sum()\n", ") * np.random.binomial(1, .95, size=(signup_months['high_spender'] == 1).sum())\n", "\n", "df = df.merge(signup_months)\n", "\n", "df[\"treatment\"] = df[\"signup_month\"] > 0\n", "\n", "after_signup = (df[\"signup_month\"] < df[\"month\"]) & (df[\"treatment\"])\n", "df.loc[after_signup, \"spend\"] = df[after_signup][\"spend\"] * 1.5\n", "\n", "df" ], "execution_count": 9, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_idmonthhigh_spenderspendsignup_monthtreatment
001177811True
102172611True
203170411True
304172311True
405171811True
.....................
1199959999817450False
1199969999917770False
11999799991017760False
11999899991117440False
11999999991217680False
\n", "

120000 rows × 6 columns

\n", "
" ], "text/plain": [ " user_id month high_spender spend signup_month treatment\n", "0 0 1 1 778 11 True\n", "1 0 2 1 726 11 True\n", "2 0 3 1 704 11 True\n", "3 0 4 1 723 11 True\n", "4 0 5 1 718 11 True\n", "... ... ... ... ... ... ...\n", "119995 9999 8 1 745 0 False\n", "119996 9999 9 1 777 0 False\n", "119997 9999 10 1 776 0 False\n", "119998 9999 11 1 744 0 False\n", "119999 9999 12 1 768 0 False\n", "\n", "[120000 rows x 6 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 9 } ] }, { "cell_type": "markdown", "metadata": { "id": "rpP-8x-VYFAa" }, "source": [ "Now imagine you weren't aware of causality, confounders, high / low spenders, and all that. You simply published your rewards signup page and observed your customers' spending behaviour over a span of a year. Chances are you'll compute the average treatment effect the exact same way we did above for our randomized controlled trial:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "MuQ0udCR9qYK", "outputId": "ec2a770f-3736-4345-87b7-f20fc34854ca" }, "source": [ "month = 6\n", "\n", "post_signup_spend = (\n", " df[df.signup_month.isin([0, month])]\n", " .groupby([\"user_id\", \"signup_month\", \"treatment\"])\n", " .apply(\n", " lambda x: pd.Series(\n", " {\n", " \"post_spend\": x.loc[x.month > month, \"spend\"].mean(),\n", " \"pre_spend\": x.loc[x.month < month, \"spend\"].mean(),\n", " }\n", " )\n", " )\n", " .reset_index()\n", ")\n", "print(post_signup_spend)" ], "execution_count": 10, "outputs": [ { "output_type": "stream", "text": [ " user_id signup_month treatment post_spend pre_spend\n", "0 1 0 False 251.666667 239.0\n", "1 3 0 False 246.166667 252.8\n", "2 4 0 False 740.833333 737.2\n", "3 6 0 False 254.333333 247.0\n", "4 7 0 False 249.166667 253.2\n", "... ... ... ... ... ...\n", "5499 9992 0 False 246.000000 240.6\n", "5500 9995 0 False 254.833333 256.4\n", "5501 9996 0 False 248.833333 239.8\n", "5502 9997 0 False 249.500000 247.8\n", "5503 9999 0 False 761.666667 752.6\n", "\n", "[5504 rows x 5 columns]\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 142 }, "id": "yoJqX6g4Tjtw", "outputId": "5e17a814-b50e-4978-ffbd-8ce065e141a6" }, "source": [ "post_spend = post_signup_spend\\\n", " .groupby('treatment')\\\n", " .agg({'post_spend': 'mean'})\n", "\n", "post_spend" ], "execution_count": 11, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
post_spend
treatment
False275.891760
True1075.543699
\n", "
" ], "text/plain": [ " post_spend\n", "treatment \n", "False 275.891760\n", "True 1075.543699" ] }, "metadata": { "tags": [] }, "execution_count": 11 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "7D74LF4kUAkB", "outputId": "6d8d05f5-b555-4343-e7d0-111122159cef" }, "source": [ "post_spend.loc[True, 'post_spend'] - post_spend.loc[False, 'post_spend']" ], "execution_count": 12, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "799.6519394104552" ] }, "metadata": { "tags": [] }, "execution_count": 12 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "NbeU9ZmlZGIZ", "outputId": "217c9554-bbae-467e-8405-20dafc6c3098" }, "source": [ "post_spend.loc[True, 'post_spend'] / post_spend.loc[False, 'post_spend']" ], "execution_count": 13, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "3.8984263250854134" ] }, "metadata": { "tags": [] }, "execution_count": 13 } ] }, { "cell_type": "markdown", "metadata": { "id": "W308IJJ4ZKBv" }, "source": [ "Performing the exact same computation as above, now we're estimating an average treatment effect of almost 400% instead of the actual 50%!\n", "\n", "So what went wrong here?\n", "\n", "Observational data got us!\n", "\n", "Realize that in our observational data the outcome (spend) is not indepndent of treatment assignmnet (rewards program signup): High spenders are far more likely to sign up hence are overrepresented in our treatment group while low spenders are overrepresented in our control group (users that didn't sign up).\n", "\n", "So when we compute the above difference or ratio we don't just see the average treatment effect of rewards signup we also see the inherent difference in spending between high and low spenders.\n", "\n", "So if we ignore how our observational data are generated we'll overestimate the effect our rewards program has and likely make decisions that seem to be supported by data but in reality aren't.\n", "\n", "Also notice that we often make this same mistake when training machine learning algorithms on observational data. Chances are someone will ask you to train a regression model to predict the effectiveness of the rewards program and your model will end up with the same inflated estimate as above." ] }, { "cell_type": "markdown", "metadata": { "id": "-wv-YTihe8p4" }, "source": [ "So how do we fix this? And how can we estimate the true treatment effect from our observational data?\n", "\n", "Generally, we know from experience in e-commerece that people who tend to spend more are more likely to sign up to our rewards program. So we could segment our users into spend buckets and compute the treatment effect within each bucket to try and breeak this confounding link in our observational data.\n", "\n", "Notice that in practice we won't have a `high spender` flag for our customers so we'll have to go by our customers' observed spending behaviour.\n", "\n", "The causal inference framework offers an established approach here: Relying on our domain knowledge, we define a causal model that describes how we believe our observational data were generated.\n", "\n", "Let's draw this as a graph with nodes and edges:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 248 }, "id": "Y51lPko52klm", "outputId": "f02cc228-c836-41e1-a008-6bbb485ec8d4" }, "source": [ "import os, sys\n", "sys.path.append(os.path.abspath(\"../../../\"))\n", "import dowhy\n", "\n", "causal_graph = \"\"\"digraph {\n", "treatment[label=\"Program Signup in month i\"];\n", "pre_spend;\n", "post_spend;\n", "U[label=\"Unobserved Confounders\"]; \n", "pre_spend -> treatment;\n", "pre_spend -> post_spend;\n", "treatment->post_spend;\n", "U->treatment; U->pre_spend; U->post_spend;\n", "}\"\"\"\n", "\n", "model = dowhy.CausalModel(\n", " data=post_signup_spend,\n", " graph=causal_graph.replace(\"\\n\", \" \"),\n", " treatment=\"treatment\",\n", " outcome=\"post_spend\"\n", ")\n", "model.view_model()" ], "execution_count": 14, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5ycZbn/8c83m0oJoQmEFggoJQmEHoRDEUGaWAIiEgGpAvLzcDyK0qVERUSagvSiIpGjB0QOSEcgIB0SKYEEEgwiJUQMhJTv74/rCUxiNrvZnZ1nduZ6v177yu7MM89cs9m99pr7ue/rlm1SSinVRo+yA0gppWaSSTellGook25KKdVQJt2UUqqhTLoppVRDmXRTSqmGMummlFINZdJNKaUayqSbUko1lEk3pZRqKJNuSinVUCbdlFKqoUy6KaVUQ5l0U0qphjLppm5EQ0EXgsaDZoJc/Du+uH1o2RGm1BZlP91U/7Q2cA2wMdAHaFnIQbOBD4DHga+CX6pdfCm1XybdVOe0N3AlrSfbBc0BZgIHgsd0XVwpdUwm3VTHtDdwFdCvAw9+DzggE2+qN5l0U53S2sDTwBKdOMkMYAh4YnViSqnz8kJaqlfXEEMKndGnOE9KdSOTbqpDGkZcNJtvDFeCCRPmP/KUU2D//Vs9UQswPGc1pHqSSTfVo8PpfJU7T+/ifCnVhUy6qR7tQPtmKrRHz+J8KdWFTLqpHg2u8vnWqfL5UuqwTLqpHvWu8vl6Vfl8KXVYJt1Ujz5Y2I0tLTBr1vy3zZoFvdpOqbPaPCKlGsmkm+rRiwu7cY01YNKk+W+bOBHWXLPN801o84iUaiSTbqpHdxG9FObzpS/B6afDlCkwdy7cfjvcdBOMHLnIc80uzpdSXcgVaakOadjcuTzUowd9K2997z046SQYMwbefhsGD455up/97CJPNgPYCvx0FwacUrtl0k11RVJP4NixYzlj883p0aNHx9+NzZmD587loV69PKKKIabUKTm8kOqGpI2Ah4Cd776bnXr04P3OnG/2bOZssAFrSNqtOhGm1HlZ6abSSeoDnECsHDsOuMK2q9FlTOIt4BLgAeCbtt+oUtgpdUhWuqlUkrYmGo8PATa2fbk/rAQ8BjiAGJed085TzimOPwA8xvYdwFDgdeBpSV+SpOq+ipTaLyvdVApJSwFnAPsAxwC/das/jFobuBoYTiyc6LmQgyp3jhi1sHaOkrYCLiOmkB1p+9VOv5CUFlNWuqnmJH2a6JU7ABhie0zrCRdi6x1vA2wFXAyMnzkTikd8AIwvbt8qjlt4/1zbY4FNgCeAJyQdmlVvqrWsdFPNSFoWOBv4FHCE7Vs6ca6/AZvZ/lsHHz+MqHr/CRxqe6ELMlKqtqx0U01I+jzwDMVuDp1JuNVg+ylgBPBH4CFJx0qqVmezlFqVlW7qUpJWBs4HhgGH2L6vSuftVKW7wLnWAS4lZkkcbPuZzp4zpdZkpZu6hMIBwJPEhauNqpVwq832BGBHYrjhLkknS6p2p7OUgKx0UxeQtCZxYWslonJ8rAueo2qV7gLnXQ24CFiTiP3hap4/pax0U9VI6iHpaOBR4F5gi65IuF3J9hRgT+BM4EZJP5bUmR2JU5pPJt1UFZI+QSTaLwPb2D7TdrfsY+vwa2JRxUDgKUm55U+qiky6qVMk9ZL0XeB+4DpgW9vPlhxWVdj+h+39gP8ErpZ0saRlyo4rdW+ZdFOHSRoOPAxsT4yvXmB7brlRVZ/tm4hlygaekbRnySGlbiyTblpskvpKGg3cCvwU+IztSeVG1bVsv2P7COCrwDmSfi1pxbLjSt1PJt20WCRtQ0wDWxcYZvuqRS/hbSy27yLmHE8hGujsl0uJ0+LIKWOpXSQtDYwGvgB8w/YNJcfTJVPGFjOGzYm5va8AX7c9uaxYUveRlW5qk6RdiAY1SwAblp1w64XtvwCbEePaj0k6XFL+TqVFyko3tUrS8sBPgO2Aw2zfVnJIH6qHSreSpA2Jqvd9ooHOCyWHlOpU/lVO/6ZYwjuSqG6nEQ1q6ibh1iPb44BPAv8LPCjpv4v93lKaT1a6aT6SVgEuBNYnGtTcX3JIC1VvlW4lSWsTWwT1J5YSP1VySKmOZKWbgA+r24OImQnjgeH1mnDrne2XgJ2IHg53SPp+sQ9cSpl0E0haC7gNOBrY2fYJtju1E2+zK5YSXwZsREwxe6zYLig1uUy6TUxSi6RjgL8AtwNb2n6i5LAaSjH88XngVOB3ks6RtGTJYaUSZdJtUpI2AO4DRgJb2/6h7dklh9WQiqr3emIp8QrEooqdSg4rlSSTbpMpGtScQHQEuxbY3vbzJYfVFGy/aXsUMYxzuaTLJA0oO65UW5l0m4ikTYFHgK2BTWz/rBEb1NQ7238kqt73gXGSPldySKmGMuk2AUn9JP2Q2ITxLGB326+UHFZTsz3d9lFE/+EfSbpe0kplx5W6XibdBifpP4hpYIOAobavbaYGNfXO9r3EDIeXiGbpo7KBTmPLxRENSlJ/4AfAZ4Gjbf++5JCqqp4XR3RUMfxzGTAVODzfjTSmrHQbkKTdgGeAXsQS3oZKuI3K9qPA5sCfiXm9R2UDncaTlW4DkbQC0VR8BNGg5o6SQ+oyjVjpVpK0PlH1ziGWYz9XckipSvKvaAMolvB+iWhQ8zrRXLxhE24zsP1XYFvgeuB+ScdlA53GkJVuNydpIPBzYB2iucrYkkOqiUavdCtJGgT8Alie+D/OVYPdWFa63VRR3R5CzEx4gph32xQJt9kU+8/tApwP3CbpDEl9y40qdVQm3W5I0mDgDuBw4FO2T7Y9s+SwUhcqlhJfSUwvWw94QtIny40qdUQm3W6kaFBzLPAQcDMwInu1NhfbU21/ETgeGCPpPElLlR1Xar9Mut2EpCHAA8S8261sn50NappXsU/dEKJR+jOSdi45pNROmXTrnKTekk4G7iKmEO1oe0LJYaU6YPst2wcSw0y/kHSFpOVKDiu1IZNuHSu2+J43YX647V9kg5q0INu3AkOBd4mq94slh5QWIaeM1SFJSwDfB/YH/hO4LvslzK+ZpowtjuLi2mXEisSjbb9WckhpAVnp1hlJOwBPAasQDWp+nQk3tVexr93GwHNEA50Ds4FOfclKt05IWgb4EbAbcKTtm0oOqa5lpds2SRsDlwNvEMvCJ5UbUYKsdOuCpD2Jt4MmGtRkwk2dVqxc2xK4E3hE0jeygU75stItkaQVgXOBLYBDbd9VckjdRla6i0fSJ4BLiULrkKK3QypB/tUrQbGEdz+iQc2rRIOaTLipyxRdyrYDfgXcJ+l7knqVHFZTykq3xiStRjSoGUQ0L3m43Ii6p6x0O07SmsDFwErEz+BjJYfUVLLSrRFJPSQdDjwO/AXYNBNuKoPtl4FdgXOAWySNltSv5LCaRibdGpC0LnEx4yBiy/Pv2/6g5LBSEysa6FwNDAMGEw10ti05rKaQSbcLSeop6VvAg8D/Ap+0Pa7ksFL6kO2/294HOA64TtKFkpYuO65Glkm3i0gaRiTbXYEtbJ9je07JYaW0ULZ/RzTQ6UssJd615JAaVibdKpPUR9KpRL/bi4GdbL9Uclgptcn227YPBg4GfibpaknLlx1Xo8mkW0WStgIeI5Zhbmz70lzCm7ob27cTDXTeIqrevXMpcfXklLEqkLQkcBrwZeCbwPWZbLtWThmrDUkjiAY6zxHL06eWHFK3l5VuJ0n6FLHIYUWiQc1vMuGmRmH7QWA48TP+pKSvZdXbOVnpdpCkAcCPgZ2BI2z/seSQmkpWurUnaSOi6p1GNNDJaxUdkJVuB0jai2hQ8wHRoCYTbmp4tp8EtgJuBR6W9E1JLSWH1e1kpbsYJK0EnEe83TrE9r0lh9S0stItl6SPA5cAvYmlxONLDqnbyEq3HYoGNfsTzcUnARtlwk3NzPbzwA7AVcA9kk6U1LvksLqFrHTbIGkN4CJgVeIv+iMlh5TISreeSFqdmJOevyPtkJVuK4oGNUcSG0PeT/yC5w9TSguwPRnYndj55GZJPyr2+UsLkUl3IYrxqruJjSH/w/YZtmeVG1VK9atooPNLYlHF6sT0su1KDqsuZdKtUDSo+Q7wAPBbYNvssJ9S+9l+3faXgf8Cfinp55L6lx1XPcmkWyjmID4E7ARsbvu8bFCTUsfYvpFooNNCLCXeveSQ6kbTJ11JfSWdDvwJuADY2fbEksNKqduzPc32YcCBwHmSrpW0Qslhla6pk66krYmdHDYgpoFdkUt4U6ou23cSzdL/TlS9+zbzUuKmnDImaSngDGBv4Bjghky23UtOGeueJG0JXA5MIBrovFpySDXXdJWupJ2J5h3LEA1qfpsJN6XasP0QsAnxDvMJSYc2W9XbNJWupGWBnxCraA63fWvJIaVOyEq3+5M0lGig8y5wqO0XSw6pJpqi0pX0BaJBzbtEdZsJN6WS2X4aGAHcDDwk6dhmaKDT0JWupJWJGQlDiAY1fy45pFQlWek2FknrEA10liCWEj9TckhdpiEr3aJBzQFEg5rnia1zMuGmVKdsTwA+RQw33CXp5EZtoNNwSVfSmsAtxLY5n7H9PdvvlxxWSqkNtufa/gXROnUz4FFJW5QcVtV126QraRlJQyq+7iHpaKJBzT3EtuePlRZgSqlDbE8BPgucCdwo6ceVDXQkrSppUEnhdVodJF0NBV0IGg+aCXLx7/ji9qGtPPBSYKykj0laD7gX2BfYxvbobFCTUvdVNND5NdFAZxXgKUk7SOoJ3AHcXny+EB3OKTVR4oU0rQ1cQ2xX3odYo72g2cSWOI8DX4XYk6lYSfYnoBfwIrEp5CnAz2zP7fLQU+nyQlpzkbQH8HNgKrAhYOA42xdUHNXhnFJLJSVd7Q1cSevfmAXNAWYCB0r8DzAO+ETFfUfYvrQLAk11KpNu85G0CfAwH+WMd4FBtt/sTE4Bj6l+tK0rYXhBexNbfCxB+745FMctAVx11llcDKy9wH0/krRMVcNMKdWNYtXaL5g/Zy0BXNDZnFI8vmZqXOlqbWIJboe7yv/rXzB0KJ44kbnAe0QTjYnAAVn1NI+sdJtLsWhi3hjvykB/wGutRcsLL/B+Swt9O3H6GcAQqE13wUVWupImSdqpis93DVH+c8opsP/+i3+Cfv2Y8/TTPAYsZXtp2+vY/vTi/vJJcjEhO6VU52zPsb0P0fP6QqAvMHjsWJ7t0YNenTx9HyI31USHhxdav3LY6iOGEQPcnVrm16MHLUsuyfo263bmPCml7sv2LJtlPvYx1pA6l1OInDS8ZrMabLf6AR++hX8X+DZxxfBg4BXg3uKYrwF/Bd4GbgXWrHj8ucBkYPqgQbx+zz3MtvEtt+BevXDPnnjJJfGwYfGU222Hjz8ejxgRt++xB37jDbzffnjppfFmm+GJE7HNLJsLgPWIWQxvAc8B+1Q895XEX8SbgX8SfyEHF/fdW7yWfxWv7UuL+j7kR/19AH8DBpYdR360+f80CfguML7IEVcAfYv7DiVaPL4F3Djv/xMQcA7wOjCduHD+IPAzYBYx++DdTTdloh05ZWEfP/gBHjgQL7UU/vjH8e23x+0nn4y/+EW8zz5x3/Dh+PHHI6cUzz8QuAH4BzF0eUzF6zkFuB64usgr44hhrnn3DwceW+T3pB3fsJ2KzwcViepqYEmgH7BX8U1bH+gJnAA8UPH4/YHlgZ6nnsprK62E33vvoxf+la/M/5TbbYcHD8YTJuBp0/D66+N118V/+hOeNQuPGoUPPDCOfecdxhcJ/aDiuYcDbwAbFM99JfAmsEVx/y+B6ypiM7BO2T+U+dGxj0y63eOjyCHPEJtVLkfsrH06sGPx+7oJ8fb+fD4q5HYhFjkNKBLwTkWi/WeRhK8HZDO+tad+9lm82mr41Vfj64kTI6/YkXt69sRjxuAPPsBnnYUHDcIzZzKOePf/KHAS0Ju4aP8SsEsR2ynA+8BuRIU8Ghhb3NcbeBn4z0V9TxZziACAU2z/C0DSEcBoF5s3SjoT+J6kNW2/bPvajx6mZc85B557DjbaqPWTH3QQDB4cn++6K4wfDzsVo8p77w0nnhif/+EPfIL4K7QkcETx8GeBsyTdQlTBzxFJdwvgHeALxaq1efaX9EYHvgepfEsAB0l6p+xA0iItDYwlCjSAJ4HDiaT7OLB18TEROFLSScAKRJI+gUhiyxHvupcqzvEF4PU5cxjQ0srAQksLzJwZ+WPFFWHQoPnv33RTGDkyPj/2WDj7bHjwQdYFNgdWtP394tCXJF1CLLya153wz7b/CCDpGqLlAMBWxNqBny7qG9KRpDu54vM1gXMlnV1xm4BVgZclfYsYjhi4zDL0nj4d3mgjxa200kef9+v371+/+24RxGR6AGsBCz73c0TCXYYYOlivuG8V4j9tvYrjBxP/wan76UX8/80oO5C0SL2Iwmje790AYFkiR0xi/t/H94ltfV4jCqivEkl7MvNffxIwoKWl9fy1zjrw05/GBftx42CXXeAnP4GBA+P+1Vf/6NgePWC11WDqVHoSOW2gpGkVp2sB7qv4+rWKz2cAfYtrXAOBV12Uva1pK+ku7MGVt00GznDsdz8fSdsS48CfAsZNm8Z7yy5L73nhdLZX/GqrMRu42/anF3a/pCuBKbZPKL7entjl9+ji66OAUx3djVI3U/RIPsE5ZayuFSvJbrZ9UfH1rsCWwN3Am7a/Xdy+JFEBf8v2pIrHf4wY712buL7Uixii2JMYnmi1E9l++8XH9Olw+OHwne/ANcUchckVpePcuTBlCqyyCrOJnDbRdkcu1E8FVpWkRSXetmYv/J35FyIs6CLgu5I2hA+b0MybaLw0seTuH0DP73yHd6ZP/+iBK60EkybFC+6I3XdnAvBxSaMk9So+Npe0fjtP0dZrSylVx1GSVpO0HHA88Btizu1BkjaW1IdobvOQ7UnF7/GWknoRF7v/RVSUZxJjv3+z/U+iBcBCPfcc3HlnDDH07RvvkntUZLtHH4X/+R+YPTsq4j59YMQIXiBWvP1T0nck9ZPUImmIpM3b8TofJHLeMYs6qK2kOxo4oSi1Ry54p+3fAT8ErpM0nRgw37W4+1bg/4h+ti+/8w4TKkv6vYvUvPzysMkmbb6YBc0eMIA7gJ2JsZa/ESX/DynmAbfDKcBVkqZJ2mexI0gptdevgNuIC1IvAqfbvh04kZglMJUYKtq3OL4/0dD8bWJM9zXiounpRP+FDSRN22IL+hJJ7t/MnAnHHQcrrAArrwyvvw6jR390/157wW9+A8suG9XvDTcwu3dv7rI9B9iDmN46kaimLyWGKxfJ9gfEePOBizquhivSNIz4S9Dh1Wjz2MyQ2Ar8dOfjSt1RrkjrHiRNInZtub3a5541S8N69mSsRL/Fedwpp8CECXDttfPdPANqk1Nq2HvBTwFPEI0mOmz2bHjgAZaQuFPSk5JulPSj4q1LSqkBFf2yj5c0RtIjkqb27s1jDzxAv7lzO5dTiJz0eK2KuI7MXuiMUXSy98LcucweNQoRsw5WIK527sz8sxhSSo3FwGHAGpU3brstc5dY4t9XpI0fD2usseCtrZpJ5KaaKKG144cdgRbrLUHhvenTOWSZZTgH+Fhx2yzg27YXOTcuNZYcXmg+kj4J3AUf9lp4D9jQZjM6kVOAA2rZ3rHWlS7x4gQd7H3Zv7/HgGYA1xLf5OnA/5M0zvafuibmlFKZJG0A/Ji4sLUsUfmea3siMLEzOaUJ+ulC8SKHEitVZtDKFcji9hnFcUMqvjn/S6xsmUOsZvk6cKmkyyQt25WRp5RqR1JvSScS+x5eRWxeMJ2YRnbaR0d2OqfUTInb9XwYwlBiUvQOwDrEW4dZRE+Hu4CLFzbAXez6u2HFcryliSlunweOLqazpQaVwwuNT9JmxJbsU4jdYSYXtw8H+tge28ojO5RTaqUOkm51FSvhLgWeAr5h+7U2HpK6oUy6javY+fcU4ADgWOBXbS2t7U7qYDfg6rJ9H7AR8VftSUlfLbb6SCnVOUnbEUOHqwNDbf+ykRIuNGClW6nYyO4yYsnv4bZfLjmkVCVZ6TYWSf2JFaV7AkfavrHkkLpMw1W6lWw/RrR1vBd4VNJRkhr6NafU3UjanWgh0AIMaeSECw1e6VaStB5R9c4lliU+V3JIqROy0u3+JK1A9J4dARxq+86SQ6qJpqn6bD8LbEt0OPqzpOOKLkYppRpS2Jeobv9OjN02RcKFJqp0K0kaBFwMrAgcbPvxUgNKiy0r3e5J0qrEXmeDid+9h0oOqeaaptKtVDRJ/gyxceatks6U1LfcqFJqXEV1eyjR9OpxYJNmTLjQpEkXwOEqomHOx4EnirXdKaUqkjQYuIPY/XdH26cUvWebUtMm3Xlsv2Z7JNHRfoyk84vVbSmlTih2XTgWeAj4AzDCzh7YTZ9057F9A7AhsYne05J2KTmklLotSUOAB4hdGLa0/ZNiV4aml0m3gu23bX+N6Nt5kaQrszl6Su1XNKg5mehxcCnwKdut7mXWjDLpLoTt24iORe8Az0j6YskhpVT3JG0BPApsCmxs+5JGW8JbDU05ZWxxFBfXLgXGE93LppYcUiKnjNWTokHNacBXgG8Cv8lk27qsdNtg+35gOPBXooHOQdlAJ6UgaQdiC66ViCW812XCXbSsdBeDpI2JpcRvAocV831TCbLSLZekZYCziPnuX7d9c8khdRtZ6S4G208AWxJzDh+RdIyk9mwNklLDkLQnsYR3DlHdZsJdDFnpdpCkjxNjvS1EA52/lhxSU8lKt/YkrQicB2xG/MzfU3JI3VJWuh1k+3lge+CXwL2Sjs8GOqkRFUt49yPGbicDG2XC7bisdKtA0hrARcBAoonHoyWH1PCy0q0NSasDPwfWAL5m+5GSQ+r2stKtAtuvALsTW0T/UdIPJPUrOayUOkxSD0lHAI8RO+dulgm3OjLpVknRQOdaYlHFWsT0sv8oOayUFpukdYE7iY0ht7N9ejM3qKm2TLpVZvt1218Cvg38StKFxf5PKdU1ST0l/TfwIPA7YBvb40sOq+Fk0u0itn8PDAH6EA10di05pJRaJWkYkWx3AbawfW42qOkamXS7kO1ptg8BvgZcKOlqScuXHVdK80jqI+n7wO3EBbNP236p5LAaWibdGrB9BzHW+wbRQGefXEqcyiZpK+JC2VCiQc3luYS36+WUsRorftAvA14AjswpTx2TU8Y6TtKSwOnAvsAxwG8z2dZOVro1ZnsssAnwJLFF0MFZ9aZakbQTschhOWIJ75hMuLWVlW6JiosXlwHTgUNzLK39stJdPJIGAGcDOwFH2L6l5JCaVla6JbL9FDACuAV4WNI3s4FOqjZJnwPGAe8R1W0m3BJlpVsnJK1DNNDpSywlHldySHUtK922SVoJOB/YiGhQc1/JISWy0q0bticAOwKXA3dLOlFS75LDSt1Q0aBmFPAU8CLRoCYTbp3ISrcOSVqNaKCzBlH1/qXkkOpOVroLVzRfuhhYmfjZeazkkNICstKtQ7anAHsCo4GbJJ1V7EOV0kIVDWqOJDaGvI9YVZYJtw5l0q1TRQOdXxMT11clGuhsX25UqR5J+gRwD7Ex5La2z7Q9q+SwUisy6dY52/+wvR9wLHCNpIuK/alSkysa1BwH3A9cTyTcZ0sOK7Uhk243YfsmooEORAOd3cuMJ5Wr2CT1IWAHYmz7fNtzSw4rtUMm3W7E9ju2jyD6nJ4r6ZfFvlWpSUjqK+kM4FZiv7LP5K7U3Usm3W7I9l3AMOBvRNX75VxK3PgkfRJ4AvgEMQ3sqlzC2/3klLFuTtIWxFLiScDXi5kPDa+ZpoxJWgo4ExgJfMP2DSWHlDohK91uzvbDwKbAX4DHJR0mKf9fG4SknYFngKWJJbyZcLu5rHQbiKQhRNU7g2igM6HkkLpMo1e6kpYjGtTsABxm+7aSQ0pVkhVRA7H9DLA1cCMwVtJ/SepZclhpMUn6IlHdTieq20y4DSQr3QYlaW3gEuJt6cG2ny45pKpqxEpX0irABcAGRIOa+0sOKXWBrHQbVNGbdyfgF8Cdkk6V1KfksNJCFA1qDiQa2/8VGJ4Jt3Fl0m1gxVLiS4GNi49HJW1ZclipgqRBxJzbY4CdbZ9g+/1Sg0pdKpNuE7D9KvA54DTg95J+UuyTlUpSNKj5BjHr5A6iQc0TJYeVaiCTbpMoqt7fEA10VgSekrRjyWE1JUnrE53A9gG2sf1D27NLDivVSCbdJmP7DdujiLezV0q6pNg/K3UxSb0kHU8k3F8C29l+ruSwUo1l0m1Stm8mGujMAp6RtFfJITU0SZsQQwnbAJvY/lk2qGlOmXSbmO3pto8E9gPOknSdpI+VHVcjkdRP0g+IzUfPBnaz/UrJYaUSZdJN2L6X2LzwZaKBzv7ZQKfzJG1LNKhZCxhm+5psUJNycUSaj6TNiKXEU4AjbE8uOaSFqufFEZL6E1stfQ44yvbvSw4p1ZGsdNN8bD8CbAY8CDwm6evZQKf9JO0KPA30IZbwZsJN88lKN7VK0gZE1TuLWJb6fMkhfajeKl1JywPnEBfKDrN9e8khpTqVFUxqle3xRBK5AXhA0rezgc78iiW8+xANat4EhmbCTYuSlW5qF0lrEX0clgO+ZvvJkuMpvdKVNBD4GbAu8U7gwbJiSd1HVrqpXWxPBHYmumD9SdLpkvqWHFYpiur2YGJmwlPEvNtMuKldMummdiuWEl9BTC/bgNipYuuSw6qpomXm7cDXgZ1sn2R7ZslhpW4kk25abLanAl8ETgR+K+ncYh+vhiWpRdI3gYeB/wO2sv1UyWGlbiiTbuqQour9LdFAZwCxqOLTJYfVJSRtCNwPfB4YYfusbFCTOiqTbuoU22/aPoB4u32ppMslLVt2XNUgqbekk4C7gSuBHWy/UGpQqdvLpJuqwvb/EQ10ZgDjJH2h5JA6RdLmwCPAlsSFsouyQU2qhpwylqqu6DlwKbEy62jbr1XpvAIGE8XCfcS48uvARNuzqvQcSwCnAqOAY4FfZ7+EVE1Z6aaqs30fMcPheaJZ+gFVaqCzMfAC8CjRiP0WYk+xr1bh3EjantinbDVikcOvMuGmastKN3WpomGRh88AAAYTSURBVI/sZcDfgcNtv9yJc4lY+bVBxc3vAqvant6J8y4D/BDYAzjS9o0dPVdKbclKN3Up248BWwD3EBtjHt3RBjpF1fktItECvA/8uJMJdw8ikfcANsyEm7paVrqpZiStR1S9JpbNPtuBc1RWu/8CBnYk6UpaETiXuFB2qO07F/ccKXVEVrqpZookuy1wHfBnSd+T1GsxzzGv2gW4aHETbrGE98vERb6pxNhtJtxUM1npplJIGgRcDHyMaKDzeDseNRQ4wmaH2bNZr1cvBHwAvAjcBVwEfnoRz7ka8HNiJ4eDbT/UyZeR0mLLpJtKUwwVfBU4ixh2ONX2+ws5cm3gGmL2Qh+gZSGnm00k4MfjnH6p4nl6AIcAZxANe0bb/qCaryWl9sqkm0onaWXgfGAYUYH+ueLevYnVYK0l2wXNAWYCB4LHSFoHuARYojj3M9WMPaXFlWO6qXS2X7O9N/Bd4HpJF0hauki4VxEJsz0Jl+K4JWyuGj1aVwNjgZuArTPhpnqQlW6qK0XfhrPXXZdd/vpXlm9poU9Hz/X++8w9+2x2PP5431Ol2CYRsy5yZ4jUYZl0U1164w2NW3ZZ1mtp6fi7MZs5EmPB21Qjpky6qRpyeCHVTPv3V9OwFVZgUGcSbjwfLcDwYtZDSnUhk27qNEmTJH1X0nhJb0u6QlJfSdtLmiLpO5JeA66Q1EPScZJelPSmpOslLbfAKQ+Hj4YV3n8f9t8fll8eBgyAzTeHv/897tt+e/jud2GLLaB/f9hrL3jrrY9O9OCD9Fl3XW6TNE3Sk0V/hXlx3y3pNEn3S/qnpNskrVBx/yhJLxdxHt8F37rUhDLppmr5CrAL0QXs48AJxe0rE5tZrgkcBnwD+BywHTAQeBu4cIFz7UDFhbOrroJ33oHJk+HNN+Gii6Bfv48OvvpquPxymDoVevaEY46J2199FfbYg5bRo/mgiOFbwA3FarR59gMOIuYL9y6Ombf9/M+JbmMDgeWJRjgpdUom3VQtF9iebPstYj7sl4vb5wIn255p+z3gCOB421OKvcVOAUYuMPQwuPLEvXpFsp0wAVpaYNNNo6qdZ9QoGDIEllwSTjsNrr8e5syBa6+F3XaDkSNZ2fZc238ieuTuVnH6K2w/X8R2PTEXGGAk8Afb9xZxnli8lpQ6JZNuqpbJFZ+/TFSHAP9YYMHDmsDvirf704jWjHOAlSqO6V154lGjYJddYN99YeBA+Pa3YVZF99zVV684+Zpx3xtvwMsvw5gxMGAAvSuebxtglYrTV/b6nQHM2+ttYOVrsv0v4M02vwsptSGTbqqWitTHGsDfis8XnB4zGdjV9oCKj762X604Zr7VYr16wcknw/jx8MAD8Ic/xJDChyesSPevvBLHr7BCJONRo2DaND6oeK4lbf+gHa9nauVrKpqbL9+Ox6W0SJl0U7UcJWm14qLY8cBvWjnuIuAMSWtCdPuStNcCx7xY+cVdd8HTT8eQQf/+kVR7VPzkXnttJOQZM+Ckk2DkyBiG2H9/uOkm+N3vmFrs5jvv4l57xmZ/C+whaRtJvYHvk78vqQryhyhVy6+A24CXiKR5eivHnQvcCNwm6Z/EirEtFzjmLqKXAgCvvRaJtH9/WH992G67qGDnGTUKDjwQVl45Zjqcd17cvvrq8PvfM+d736MX8A+iyv5v2vFzb3sccFTxuqYSF/ymtPW4lNqSiyNSp1V/0YCGAQ8Sy38Xafvto6I95JBWD5kBbLWo7mMp1VJWuqkO+SngCeICW2fMAR7PhJvqSSbdVBckfUXSu/M+evRg46WWomXDDTt12pnEPNuU6kYOL6Q69mGXsX5tHbkQ7wEHgMdUN6aUOqeda+FTKoPHgKAT/XS7LLSUOigr3dQNaG3gamA4sXBiYcVC5c4Ro8ATaxdfSu2XSTd1IxpKNMPZAVgH6AXMAiYQ08wuzotmqd5l0k0ppRrK2QsppVRDmXRTSqmGMummlFINZdJNKaUayqSbUko1lEk3pZRqKJNuSinVUCbdlFKqoUy6KaVUQ5l0U0qphjLpppRSDWXSTSmlGsqkm1JKNZRJN6WUauj/A/N2LzrFWJsFAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "markdown", "metadata": { "id": "QxJ6MxYXmvhD" }, "source": [ "Our causal model states what we described above: Pre-signup spend influences both rewards signup (treatment assignment) and post-signup spend. This is the story about our high and low spenders.\n", "\n", "Treatment (rewards signup) influences post-signup spending behaviour - this is the effect we're actually interested in.\n", "\n", "We also added a node `U` to signify possible other confounding factors that may exist in reality but weren't observed as part of our data." ] }, { "cell_type": "markdown", "metadata": { "id": "nkek0w7Re8p5" }, "source": [ "## Identification / identifying the causal effect\n", "\n", "We will now apply do-calculus to our causal model from above to figure out ways to cleanly estimate the treatment effect we're after:" ] }, { "cell_type": "code", "metadata": { "scrolled": true, "colab": { "base_uri": "https://localhost:8080/" }, "id": "j1smWlfve8p5", "outputId": "7d06e053-e073-4354-ac65-50c63ef8fa7f" }, "source": [ "identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)\n", "print(identified_estimand)" ], "execution_count": 15, "outputs": [ { "output_type": "stream", "text": [ "Estimand type: nonparametric-ate\n", "\n", "### Estimand : 1\n", "Estimand name: backdoor\n", "Estimand expression:\n", " d \n", "────────────(Expectation(post_spend|pre_spend))\n", "d[treatment] \n", "Estimand assumption 1, Unconfoundedness: If U→{treatment} and U→post_spend then P(post_spend|treatment,pre_spend,U) = P(post_spend|treatment,pre_spend)\n", "\n", "### Estimand : 2\n", "Estimand name: iv\n", "No such variable found!\n", "\n", "### Estimand : 3\n", "Estimand name: frontdoor\n", "No such variable found!\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "RZKrj9Qne8p6" }, "source": [ "Very broadly and sloppily stated there a three ways to segment (or slice and dice) our observational data to get to subsets of our data within which we can cleanly compute the average treatment effect:\n", "\n", "- Backdoor adjustment,\n", "- Frontdoor adjustment, and\n", "- Instrumental variables.\n", "\n", "The above printout tells us that based on both the causal model we constructed and our observational data there is a backdoor-adjusted estimator for our desired treatment effect.\n", "\n", "This backdoor adjustment actually follows closely what we already said above: We'll compute the post-spend given pre-spend (segment our customers based on their spending behaviour)." ] }, { "cell_type": "markdown", "metadata": { "id": "KwNeZccJe8p6" }, "source": [ "## Estimating the treatment effect\n", "\n", "There are various ways to perform the backdoor adjustment that DoWhy identified for us above. One of them is called propensity score matching:" ] }, { "cell_type": "code", "metadata": { "scrolled": true, "colab": { "base_uri": "https://localhost:8080/" }, "id": "QXRu-N1ie8p7", "outputId": "ce197642-0ac6-4d68-c0b3-a1df51fd1f1e" }, "source": [ "estimate = model.estimate_effect(\n", " identified_estimand,\n", " method_name=\"backdoor1.propensity_score_matching\",\n", " target_units=\"ate\"\n", ")\n", "\n", "print(estimate)" ], "execution_count": 16, "outputs": [ { "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:760: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n", " y = column_or_1d(y, warn=True)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "*** Causal Estimate ***\n", "\n", "## Identified estimand\n", "Estimand type: nonparametric-ate\n", "\n", "## Realized estimand\n", "b: post_spend~treatment+pre_spend\n", "Target units: ate\n", "\n", "## Estimate\n", "Mean value: 159.15597747093042\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "RhH_KpINxzih" }, "source": [ "DoWhy provides us with an estimated ATE for our observational data that is pretty close to the ATE we computed for our experimental data from our randomized controlled trial.\n", "\n", "Even if the ATE estimate DoWhy provides doesn't match exactly our experimental ATE we're now in a much better position to take a decision regarding our rewards program based on our observational data.\n", "\n", "So next time we want to base decisions on observational data it'll be worthwhile defining a causal model of the underlying data-generating process and using a library such as DoWhy that helps us identify and apply adjustment strategies." ] } ] }