{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Project: Analyzing A/B test result to decide whether to launch new homepage design\n",
" \n",
"--By Lu Tang"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this project, I will do data analysis to help our client decide whether to launch two new features on their website. \n",
"\n",
"Here's the customer funnel for typical new users on their site:\n",
"\n",
"**View home page > Explore courses > View course overview page > Enroll in course > Complete course**\n",
"\n",
"Our client loses users as they go down the stages of this funnel, with only a few making it to the end. To increase student engagement, Our client is performing **A/B tests** to try out changes that will hopefully increase conversion rates from one stage to the next.\n",
"\n",
"A/B tests are used to test changes on a web page by running an experiment where a **control group** sees the old version, while the **experiment group** sees the new version. A **metric** is then chosen to measure the level of engagement from users in each group. These results are then used to judge whether one version is more effective than the other. A/B testing is very much like hypothesis testing with the following hypotheses:\n",
">- **Null Hypothesis**: The new version is no better, or even worse, than the old version\n",
">- **Alternative Hypothesis**: The new version is better than the old version\n",
"\n",
"If we fail to reject the null hypothesis, the results would suggest keeping the old version. If we reject the null hypothesis, the results would suggest launching the change. These tests can be used for a wide variety of changes, from large feature additions to small adjustments in color, to see what change maximizes your metric the most."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Feature Changes: Change homepage design. \n",
"Our client hopes that this new, more engaging design will increase the number of users that explore their courses, that is, move on to the second stage of the funnel.\n",
"\n",
"**Metric: Click through rate (CTR)**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Click through rate (CTR) is often defined as the the number of clicks divided by the number of views. Since our client uses cookies, we can identify unique users and make sure we don't count the same one multiple times. For this experiment, we'll define our click through rate as:\n",
"\n",
"CTR: # clicks by *unique* users / # views by *unique* users\n",
"\n",
"H0: CTR_{new} - CTR_{old} <= 0\n",
"\n",
"H1: CTR_{new} - CTR_{old} > 0 "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**The company has collected the data in homepage_actions.csv, we will use Python to analyze the data**"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# load library\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"# to make sure we get the same results everytime we run the code\n",
"np.random.seed(42)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>timestamp</th>\n",
" <th>id</th>\n",
" <th>group</th>\n",
" <th>action</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2016-09-24 17:42:27.839496</td>\n",
" <td>804196</td>\n",
" <td>experiment</td>\n",
" <td>view</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2016-09-24 19:19:03.542569</td>\n",
" <td>434745</td>\n",
" <td>experiment</td>\n",
" <td>view</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2016-09-24 19:36:00.944135</td>\n",
" <td>507599</td>\n",
" <td>experiment</td>\n",
" <td>view</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2016-09-24 19:59:02.646620</td>\n",
" <td>671993</td>\n",
" <td>control</td>\n",
" <td>view</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2016-09-24 20:26:14.466886</td>\n",
" <td>536734</td>\n",
" <td>experiment</td>\n",
" <td>view</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" timestamp id group action\n",
"0 2016-09-24 17:42:27.839496 804196 experiment view\n",
"1 2016-09-24 19:19:03.542569 434745 experiment view\n",
"2 2016-09-24 19:36:00.944135 507599 experiment view\n",
"3 2016-09-24 19:59:02.646620 671993 control view\n",
"4 2016-09-24 20:26:14.466886 536734 experiment view"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# load the data and display first 5 rows\n",
"df = pd.read_csv('homepage_actions.csv')\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1. Match the following characteristics of this dataset: \n",
"- total number of actions\n",
"- number of unique users\n",
"- sizes of the control and experiment groups (i.e., the number of unique users in each group)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(8188, 4)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# total number of actions\n",
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- The total number of actions is 8188"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"timestamp 8188\n",
"id 6328\n",
"group 2\n",
"action 2\n",
"dtype: int64"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# number of unique users\n",
"df.nunique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- The number of unique users is 6328"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"control 4264\n",
"experiment 3924\n",
"Name: group, dtype: int64"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# number of control and experiement group in this dataset (total number, not unique unser)\n",
"df['group'].value_counts()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"group\n",
"control 3332\n",
"experiment 2996\n",
"Name: id, dtype: int64"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# sizes of the control and experiment groups (i.e., the number of unique users in each group)\n",
"df.groupby('group')['id'].nunique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- The number of unique users in control group is 3332\n",
"- The number of unique users in experiment group is 2996"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. How long was the experiment run for? "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('2016-09-24 17:42:27.839496', '2017-01-18 10:24:08.629327')"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# duration of this experiment\n",
"df.timestamp.min(), df.timestamp.max()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- This experiment has run for 4 months"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. What action types are recorded in this dataset?"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['view', 'click'], dtype=object)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# action types in this experiment\n",
"df['action'].unique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- The action types are view and click"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4. Why would we use click through rate instead of number of clicks to compare the performances of control and experiment pages?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- The control and experiement group would have different number of total visitors\n",
"- More total click would occur in one version, even if there is a greater percentage of clicks in other version\n",
"- Getting the proportion of the users who click is more effective than getting the number of users who click when comparing groups of different sizes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The steps I took to analyze the results of this A/B test"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our Hypothesis for this test:\n",
"\n",
"H0: CTR_{experiment} - CTR_{control} <= 0\n",
"\n",
"H1: CTR_{experiment} - CTR_{control} > 0 "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step_1. Compute the observed difference between the metric, CTR (click through rate), for the control and experiment group."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The CTR for experiment group is 0.3097463284379172\n"
]
}
],
"source": [
"# experiment group data\n",
"experiment_df = df.query('group == \"experiment\"')\n",
"\n",
"# CTR for experiment group\n",
"experiment_ctr = experiment_df.query('action==\"click\"')['id'].nunique()/experiment_df.query('action==\"view\"')['id'].nunique()\n",
"\n",
"print(f'The CTR for experiment group is {experiment_ctr}')"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The CTR for control group is 0.2797118847539016\n"
]
}
],
"source": [
"# control group data\n",
"control_df = df.query('group == \"control\"')\n",
"\n",
"# CTR for control group\n",
"control_ctr = control_df.query('action==\"click\"')['id'].nunique()/control_df.query('action==\"view\"')['id'].nunique()\n",
"\n",
"print(f'The CTR for control group is {control_ctr}')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The observed difference in CTR betweent experiment group and control group is 0.030034443684015644\n"
]
}
],
"source": [
"# observed difference\n",
"obs_diff = experiment_ctr - control_ctr\n",
"print(f'The observed difference in CTR betweent experiment group and control group is {obs_diff}')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- In this sample, the experiment group click through rate is higher than control group's click through rate by about 0.03. We then need to test if this difference is statistically significance and not due by random chance. Let's bootstrap the sample to simulate the sampling distribution for the difference in proportions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step_2. Use Bootstrapping technique to simulate the sampling distribution for the difference in proportions"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"# create an empty list for the differences in click through rates between experiment and control group \n",
"diffs = []\n",
"# set sample size as the data size\n",
"size = df.shape[0]\n",
"for i in range(10000):\n",
" # use bootstrap technique to take sample from a group with replacement\n",
" # this is to simulate the creation of sampling distribution\n",
" boots_sample = df.sample(size, replace = True)\n",
" # the experiment group\n",
" experiement_df = boots_sample.query('group == \"experiment\"')\n",
" # get the control group \n",
" control_df = boots_sample.query('group == \"control\"')\n",
" # calculate the ctr in experiment group\n",
" experiment_ctr = experiment_df.query('action==\"click\"')['id'].nunique()/experiment_df.query('action==\"view\"')['id'].nunique()\n",
" # calculate the ctr in control group\n",
" control_ctr = control_df.query('action==\"click\"')['id'].nunique()/control_df.query('action==\"view\"')['id'].nunique()\n",
" # calculatet the differences nd append all the number to diffs list\n",
" diffs.append(experiment_ctr - control_ctr)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- By bootstrapping and then calculating repeated values of our statistics (experiment_ctr - control_ctr), we can gain an understanding of the sampling distribution of our statistics"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"# convert diffs list to numpy arrays\n",
"diffs = np.array(diffs)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Text(0, 0.5, 'frequency')"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEXCAYAAAC6baP3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X28HGV5//HPl4RnkKcECiFwAgYVbAVNA1bUKPJoFagPYAWBoimKWi22jaA/QHzAaqGVqoiYEhAFUZAICMRAoKBAAgZCDEgKRxISIYAEwpMErt8f971ksuzumTnn7Nk9yff9eu1rZ++Zuee6d2fn2rlndkYRgZmZWVnrdDoAMzMbXpw4zMysEicOMzOrxInDzMwqceIwM7NKnDjMzKwSJw4bMEmnSPphG+s/W9IXB6muHSStkDQiv54l6aODUXeu75eSjhqs+gr1bijpF5KWS7pksOsfTuo/w4rzDtq6tDZz4hjGJO0t6dd5Y/K4pJsl/XWn46pCUq+kZyU9JemJ3J7jJL28bkbEcRFxWsm63tVqmoh4MCI2iYgXByH2VyTMiDgwIqYNtO4G3g9sA2wVER9oQ/1DQlKPpJA0ssI8q32uZT9DSUdLuqlYVnZdstacOIYpSa8CrgDOArYExgCnAs93Mq5+ek9EbArsCJwO/Bvwg8FeSJWNVRfaEfh9RKysOmO72z3M31frj4jwYxg+gAnAEy3G7wxcBzwGPApcCGxeGN8L/AtwF/A0aUO9DfBL4CngV8AWedoeIIDJwBJgKXBCoa5TgB8WXu8F/Bp4ArgTmNQizl7gXXVlE4GXgNfn1+cBX87Do0gJ8wngceB/ST+ALsjzPAusAP61EPexwIPAjYWykbm+WcDXgNuA5cDlwJZ53CRgcaN4gQOAPwMv5OXdWajvo3l4HeALwB+AR4Dzgc3q3tOjcmyPAic1eY9OrVvWsSXrfrndTer9GLAwv4/Tge3q5h9ZmLbYrqOBm4Ez87xfblD3RGAO8CTwMHBGLn8w170iP95Mi3W1j891ZCGe+0nr7QPAh4HXAc8BL+b5nqhfl/Lrg4G5Oc7/Aw5oVmenv/Pd9Oh4AH7084ODV+Uv2jTgQPJGvjD+1cC+wPrAaNJG8z8L43uBW0jJYkze+NwB7JHnuQ44OU9b+6L+GNgY+EtgGXmDTyFx5LoeAw4ibdz2za9HN2lHL3WJI5c/CHw8D7/8ZSdt5M8G1s2PtwJqVFch7vNz3Bs22OjMAh4CXp+n+VmhLZNokjjq210YP4tVG9h/IG2YdwI2AS4FLqiL7fs5rjeQ9hZf1+R9Wm1ZJet+ud0N6nsnaSP9xvx5n0VOMPXvUYN2HQ2sBD4FjGxS/2+AI/PwJsBeLeous642+lxH5vY9Cbwmj9sW2K0Q5011cZ3HqnVpIunHwr6kdXUM8NpWdfqRHu6qGqYi4klgb1ZtfJZJmi5pmzx+YUTMiIjnI2IZcAbw9rpqzoqIhyPiIdIv91sj4rcR8TxwGSmJFJ0aEU9HxDzgf4APNQjtCOCqiLgqIl6KiBmkX54HVWziElIXXL0XSF/kHSPihYj438jf7hZOyXE/22T8BRFxd0Q8DXwR+GB/Drw28GHSL+37I2IF8Hng8LqunVMj4tmIuJO0d/aGQay7Vbs/DEyNiDvy5/154M2Sekouf0lEnBURK5vU/wLwakmjImJFRNzSrKKS62orLwGvl7RhRCyNiPkl5zuW9B7MyOvqQxFxzwDrXCs4cQxjEbEgIo6OiO1Jv5i3A/4TQNLWki6S9JCkJ4Efkrp5ih4uDD/b4PUmddMvKgz/IS+v3o7AB/KB7ickPUFKcNtWbN4YUjdIvW+QfmlfK+l+SVNK1LWowvg/kPZk6t+r/tgu11eseyRpL6/mj4XhZ3jlez6Qulu1e7X5c/J5jPS+l9HXe3ossAtwj6TZkv622YQl19WGcrI/DDgOWCrpSkmvLdcExpK6pwazzrWCE8caIv9SOo+UQCB16QTwVxHxKtKegAa4mLGF4R1IewX1FpF+wW9eeGwcEaeXXUg+M2wMcFP9uIh4KiJOiIidgPcA/yxpn9roJlX2tUdS364XSN04TwMbFeIaQepKKVvvElIiLda9ktUTdH+VqbtVfKvNL2ljYCtSt93TuXijwvR/UTd/y7ZHxH0R8SFga+DrwE/zMhrN19e62teyromIfUk/Tu4h7YH3OR9pXd25Yp2GE8ewJem1kk6QtH1+PZbUdVTrEtiUfFBQ0hjSgfCB+qKkjSTtBhwDXNxgmh8C75G0v6QRkjaQNKkWZx9telX+ZXoRqT9/XoNp/lbSqyWJ1A/9Yn5A2mju1I92HSFpV0kbAV8CfhrpVM/fAxtIerekdUkHo9cvzPcw0FM8dbjOj4HPShonaRPgq8DF0Y8zo9pQ94+AYyTtLmn9PP+tEdGbu4seIr0vIyT9A002sM1IOkLS6Ih4iXQiA6TPaRmpG6j4OfW1rjb9XCVtI+m9OSk9n+sprg/bS1qvSZg/IL0H+0haR9KY/L1qVafhxDGcPQXsCdwq6WlSwrgbOCGPP5V04HM5cCXp4OlA3UDqJpoJfDMirq2fICIWkc5UOZG0kVhE2hC0Wtd+IempPO1JpD7uY5pMO550xtcK0gHY70TErDzua8AXchfZ5yq06wLS3tofgQ2AT+e2LAc+AZzLql/iiwvz1f6I95ikOxrUOzXXfSPpzJznSAeUB8OA6o6ImaTjOT8jnSW3M3B4YZKPkT63x4DdSGfJVXEAMF/SCuC/gMMj4rmIeAb4CnBz/pz2ou91tdXnug5pnV9C6tp8O+kzg3SCx3zgj5IebfAe3EZaz87My76BtBfWqk5j1dkoZk3lA6YPAOsO0q9lMxvGvMdhZmaVOHGYmVkl7qoyM7NKvMdhZmaVrJEXJxs1alT09PR0Ogwzs2Hl9ttvfzQiRvc13RqZOHp6epgzZ06nwzAzG1Yk/aHvqdxVZWZmFTlxmJlZJU4cZmZWiROHmZlV4sRhZmaVOHGYmVklThxmZlaJE4eZmVXixGFmZpWskf8cN+tmPVOu7Mhye09/d0eWa2se73GYmVklThxmZlaJE4eZmVXixGFmZpU4cZiZWSVOHGZmVokTh5mZVeLEYWZmlThxmJlZJU4cZmZWiROHmZlV4sRhZmaVOHGYmVklThxmZlZJ2xKHpLGSrpe0QNJ8Sf+Uy0+R9JCkuflxUGGez0taKOleSfsXyg/IZQslTWlXzGZm1rd23o9jJXBCRNwhaVPgdkkz8rgzI+KbxYkl7QocDuwGbAf8StIuefS3gX2BxcBsSdMj4ndtjN3MzJpoW+KIiKXA0jz8lKQFwJgWsxwMXBQRzwMPSFoITMzjFkbE/QCSLsrTOnGYmXXAkBzjkNQD7AHcmos+KekuSVMlbZHLxgCLCrMtzmXNyuuXMVnSHElzli1bNsgtMDOzmrYnDkmbAD8DPhMRTwLfBXYGdiftkfxHbdIGs0eL8tULIs6JiAkRMWH06NGDEruZmb1SW+85LmldUtK4MCIuBYiIhwvjvw9ckV8uBsYWZt8eWJKHm5WbmdkQa+dZVQJ+ACyIiDMK5dsWJjsUuDsPTwcOl7S+pHHAeOA2YDYwXtI4SeuRDqBPb1fcZmbWWjv3ON4CHAnMkzQ3l50IfEjS7qTupl7gHwEiYr6kn5AOeq8Ejo+IFwEkfRK4BhgBTI2I+W2M28zMWmjnWVU30fj4xFUt5vkK8JUG5Ve1ms/MzIaO/zluZmaVOHGYmVklThxmZlaJE4eZmVXS1v9xmHWrnilXdjoEs2HLexxmZlaJE4eZmVXixGFmZpU4cZiZWSVOHGZmVokTh5mZVeLEYWZmlThxmJlZJU4cZmZWiROHmZlV4sRhZmaVOHGYmVklThxmZlaJE4eZmVXixGFmZpU4cZiZWSVOHGZmVokTh5mZVeLEYWZmlfie42ZriU7eZ7339Hd3bNk2+LzHYWZmlThxmJlZJU4cZmZWiROHmZlV0rbEIWmspOslLZA0X9I/5fItJc2QdF9+3iKXS9K3JC2UdJekNxbqOipPf5+ko9oVs5mZ9a2dexwrgRMi4nXAXsDxknYFpgAzI2I8MDO/BjgQGJ8fk4HvQko0wMnAnsBE4ORasjEzs6HXtsQREUsj4o48/BSwABgDHAxMy5NNAw7JwwcD50dyC7C5pG2B/YEZEfF4RPwJmAEc0K64zcystSE5xiGpB9gDuBXYJiKWQkouwNZ5sjHAosJsi3NZs/L6ZUyWNEfSnGXLlg12E8zMLGt74pC0CfAz4DMR8WSrSRuURYvy1QsizomICRExYfTo0f0L1szM+tTWxCFpXVLSuDAiLs3FD+cuKPLzI7l8MTC2MPv2wJIW5WZm1gHtPKtKwA+ABRFxRmHUdKB2ZtRRwOWF8o/ks6v2ApbnrqxrgP0kbZEPiu+Xy8zMrAPaea2qtwBHAvMkzc1lJwKnAz+RdCzwIPCBPO4q4CBgIfAMcAxARDwu6TRgdp7uSxHxeBvjNjOzFtqWOCLiJhofnwDYp8H0ARzfpK6pwNTBi87MzPrL/xw3M7NKnDjMzKwSJw4zM6vEicPMzCpx4jAzs0qcOMzMrBInDjMzq8SJw8zMKnHiMDOzSpw4zMysEicOMzOrxInDzMwqceIwM7NK+kwckrYcikDMzGx4KLPHcaukSyQdlG/OZGZma7EyiWMX4BzSTZkWSvqqpF3aG5aZmXWrPhNHJDMi4kPAR0m3e71N0g2S3tz2CM3MrKv0eQdASVsBR5D2OB4GPkW6P/juwCXAuHYGaGZm3aXMrWN/A1wAHBIRiwvlcySd3Z6wzMysW5VJHK/J9wN/hYj4+iDHY2ZmXa7MwfFrJW1eeyFpC0nXtDEmMzPrYmUSx+iIeKL2IiL+BGzdvpDMzKyblUkcL0raofZC0o5Aw64rMzNb85U5xnEScJOkG/LrtwGT2xeSmZl1sz4TR0RcLemNwF6AgM9GxKNtj8zMzLpSmT0OgPWBx/P0u0oiIm5sX1hmZtatyvwB8OvAYcB84KVcHIATh5nZWqjMHschpP9yPN/uYMzMrPuVOavqfmDdqhVLmirpEUl3F8pOkfSQpLn5cVBh3OclLZR0r6T9C+UH5LKFkqZUjcPMzAZXmT2OZ4C5kmYCL+91RMSn+5jvPOC/gfPrys+MiG8WCyTtChwO7AZsB/yqcAXebwP7AouB2ZKmR8TvSsRtZmZtUCZxTM+PSiLiRkk9JSc/GLgod4c9IGkhMDGPWxgR9wNIuihP68RhZtYhZU7HnSZpQ2CHiLh3EJb5SUkfAeYAJ+R/oo8BbilMsziXASyqK99zEGIwM7N+KnPr2PcAc4Gr8+vdJVXeA8m+C+xMuiT7UuA/aotpMG20KG8U52RJcyTNWbZsWT/DMzOzvpQ5OH4KqdvoCYCImEs/78EREQ9HxIsR8RLwfVZ1Ry0GxhYm3R5Y0qK8Ud3nRMSEiJgwevTo/oRnZmYllEkcKyNieV1Zv65VJWnbwstDgdoZV9OBwyWtL2kcMB64DZgNjJc0TtJ6pAPo/d3bMTOzQVDm4Pjdkv4eGCFpPPBp4Nd9zSTpx8AkYJSkxcDJwCRJu5MSTy/wjwARMV/ST0gHvVcCx0fEi7meTwLXACOAqRExv1ILzcxsUJVJHJ8iXejweeDHpI34aX3NlO9RXu8HLab/CvCVBuVXAVeViNPMzIZAmbOqniEljpPaH46ZmXW7Mtequp4GxzQi4p1ticjMzLpama6qzxWGNwDeRzoOYWZma6EyXVW31xXdXLipk5mZrWXKdFVtWXi5DvAm4C/aFpGZmXW1Ml1Vt7PqX9wrgQeAY9sZlJmZda8yXVX9+pe4mZmtmcp0Vf1dq/ERcenghWNmZt2uTFfVscDfANfl1+8AZgHLSV1YThxmZmuRMokjgF0jYim8fL2pb0fEMW2NzMzMulKZixz21JJG9jCwS7OJzcxszVZmj2OWpGtI16kK0hVqr29rVGZm1rXKnFX1SUmHAm/LRedExGXtDcvMzLpVmT0OgDuApyLiV5I2krRpRDzVzsDMzKw7lbl17MeAnwLfy0VjgJ+3MygzM+teZQ6OHw+8BXgSICLuA7ZuZ1BmZta9yiSO5yPiz7UXkkbSz1vHmpnZ8Fcmcdwg6URgQ0n7ApcAv2hvWGZm1q3KJI4pwDJgHuke4VcBX2hnUGZm1r1anlUlaQQwLSKOAL4/NCHZ2qRnypWdDsHMKmq5xxERLwKjJa03RPGYmVmXK/M/jl7SXf+mA0/XCiPijHYFZWZm3avpHoekC/LgYcAVedpNCw8zM1sLtdrjeJOkHYEHgbOGKB4zM+tyrRLH2cDVwDhgTqFcpP9x7NTGuMzMrEs17aqKiG9FxOuA/4mInQqPcRHhpGFmtpbq838cEfHxoQjEzMyGhzJ/ADQzM3uZE4eZmVXStsQhaaqkRyTdXSjbUtIMSffl5y1yuSR9S9JCSXdJemNhnqPy9PdJOqpd8ZqZWTnt3OM4DzigrmwKMDMixgMz82uAA4Hx+TEZ+C6kRAOcDOwJTAROriUbMzPrjLYljoi4EXi8rvhgYFoengYcUig/P5JbgM0lbQvsD8yIiMcj4k/ADF6ZjMzMbAgN9TGObSJiKUB+rt0QagywqDDd4lzWrPwVJE2WNEfSnGXLlg164GZmlnTLwXE1KIsW5a8sjDgnIiZExITRo0cPanBmZrbKUCeOh3MXFPn5kVy+GBhbmG57YEmLcjMz65ChThzTgdqZUUcBlxfKP5LPrtoLWJ67sq4B9pO0RT4ovl8uMzOzDilzWfV+kfRjYBIwStJi0tlRpwM/kXQs6eKJH8iTXwUcBCwEngGOAYiIxyWdBszO030pIuoPuJuZ2RBqW+KIiA81GbVPg2kDOL5JPVOBqYMYmpmZDUC3HBw3M7NhwonDzMwqaVtXlZlZTc+UKzuy3N7T392R5a7pvMdhZmaVOHGYmVklThxmZlaJE4eZmVXixGFmZpU4cZiZWSVOHGZmVokTh5mZVeLEYWZmlThxmJlZJU4cZmZWiROHmZlV4sRhZmaVOHGYmVklThxmZlaJE4eZmVXixGFmZpU4cZiZWSVOHGZmVokTh5mZVeLEYWZmlThxmJlZJU4cZmZWiROHmZlV4sRhZmaVOHGYmVklHUkcknolzZM0V9KcXLalpBmS7svPW+RySfqWpIWS7pL0xk7EbGZmSSf3ON4REbtHxIT8egowMyLGAzPza4ADgfH5MRn47pBHamZmL+umrqqDgWl5eBpwSKH8/EhuATaXtG0nAjQzs84ljgCulXS7pMm5bJuIWAqQn7fO5WOARYV5F+ey1UiaLGmOpDnLli1rY+hmZmu3kR1a7lsiYomkrYEZku5pMa0alMUrCiLOAc4BmDBhwivGm5nZ4OjIHkdELMnPjwCXAROBh2tdUPn5kTz5YmBsYfbtgSVDF62ZmRUNeeKQtLGkTWvDwH7A3cB04Kg82VHA5Xl4OvCRfHbVXsDyWpeWmZkNvU50VW0DXCaptvwfRcTVkmYDP5F0LPAg8IE8/VXAQcBC4BngmKEP2czMaoY8cUTE/cAbGpQ/BuzToDyA44cgNDMzK6GbTsc1M7NhwInDzMwqceIwM7NKnDjMzKwSJw4zM6ukU/8cty7TM+XKTodgZsOE9zjMzKwSJw4zM6vEicPMzCpx4jAzs0qcOMzMrBInDjMzq8SJw8zMKnHiMDOzSvwHQDNbY3Xyj629p7+7Y8tuN+9xmJlZJU4cZmZWiROHmZlV4sRhZmaVOHGYmVklThxmZlaJE4eZmVXixGFmZpU4cZiZWSVOHGZmVokTh5mZVeLEYWZmlfgih12kkxdkMzMry3scZmZWybBJHJIOkHSvpIWSpnQ6HjOztdWw6KqSNAL4NrAvsBiYLWl6RPyus5GZmTXWqa7nobgPyLBIHMBEYGFE3A8g6SLgYKAticPHGszMmhsuiWMMsKjwejGwZ3ECSZOByfnlCkn3DuLyRwGPDmJ93cBtGh7cpuGha9qkrw9o9h3LTDRcEocalMVqLyLOAc5py8KlORExoR11d4rbNDy4TcPDmtimVobLwfHFwNjC6+2BJR2KxcxsrTZcEsdsYLykcZLWAw4Hpnc4JjOztdKw6KqKiJWSPglcA4wApkbE/CEMoS1dYB3mNg0PbtPwsCa2qSlFRN9TmZmZZcOlq8rMzLqEE4eZmVWyVieOvi5jIml9SRfn8bdK6imM+3wuv1fS/kMZdyv9bZOkrSRdL2mFpP8e6rhbGUCb9pV0u6R5+fmdQx17KwNo10RJc/PjTkmHDnXszQzkO5XH75DXwc8NVcx9GcDn1CPp2cJndfZQx942EbFWPkgH2f8P2AlYD7gT2LVumk8AZ+fhw4GL8/Cuefr1gXG5nhHDvE0bA3sDxwH/3em2DFKb9gC2y8OvBx7qdHsGqV0bASPz8LbAI7XXw7VNhfE/Ay4BPtfp9gzC59QD3N3pNrTjsTbvcbx8GZOI+DNQu4xJ0cHAtDz8U2AfScrlF0XE8xHxALAw19dp/W5TRDwdETcBzw1duKUMpE2/jYja/33mAxtIWn9Iou7bQNr1TESszOUbUPdn2A4ayHcKSYcA95M+q24xoDatqdbmxNHoMiZjmk2Tv6jLga1KztsJA2lTtxqsNr0P+G1EPN+mOKsaULsk7SlpPjAPOK6QSDqp322StDHwb8CpQxBnFQNd/8ZJ+q2kGyS9td3BDpVh8T+ONunzMiYtpikzbycMpE3dasBtkrQb8HVgv0GMa6AG1K6IuBXYTdLrgGmSfhkRnd5bHEibTgXOjIgVXfZjfSBtWgrsEBGPSXoT8HNJu0XEk4Md5FBbm/c4ylzG5OVpJI0ENgMeLzlvJwykTd1qQG2StD1wGfCRiPi/tkdb3qB8VhGxAHiadAyn0wbSpj2Bf5fUC3wGODH/6bfT+t2m3JX9GEBE3E46VrJL2yMeAmtz4ihzGZPpwFF5+P3AdZGOek0HDs9nU4wDxgO3DVHcrQykTd2q322StDlwJfD5iLh5yCIuZyDtGpc3UEjaEXgN0Ds0YbfU7zZFxFsjoicieoD/BL4aEd1wdt9APqfRSvcSQtJOpO3E/UMUd3t1+uh8Jx/AQcDvSb8ETsplXwLem4c3IJ3hsZCUGHYqzHtSnu9e4MBOt2WQ2tRL+vW3gvQratehjn8w2wR8gfRrfG7hsXWn2zMI7TqSdAB5LnAHcEin2zIY61+hjlPokrOqBvg5vS9/Tnfmz+k9nW7LYD18yREzM6tkbe6qMjOzfnDiMDOzSpw4zMysEicOMzOrxInDzMwqceIwM7NKnDhKkrS5pE8UXk+SdMUQLv+Udl9qWtKnJS2QdGGDcXtIOredy29F0lX5D33tXMaJ/ZzvEEm7DnY8/SXpM5I26sd8syRNqDjPJEl/02L830pqeP2p/AfaX+VLjh9WNd7BJKlX0qhOxlBP0opOx9CME0d5m5Mun7wm+wRwUER8uMG4E4GzhjgelKwTEQdFxBNtXly/EgdwCOlS+69Q+4f3EPsM6dLrr1D7J/MgmgQ0TBy57VcC722SyPYA1o2I3SPi4jILa0P81h+d/gficHmQLqf8LOnfut8gfWFmkS6jfA9wIavu4f4m4AbgduAaYNu6ujYj/Ut7nfx6I9LVNdcFPka6zMGdpHsTbJSnOYX8b9q83Al5eBTQm4dH5NhmA3cB/9ikLf8M3J0fn8llZwN/Jl1t9bN1028K3Ft4vTEwNS/nt8DBhXqn5uG/zPVvlGO/ALgOuA/4WKGufynEe2ou6wEWAN/J9e+Y369Redw9wLm5/guBdwE357on9hHj0cClwNV5+n/P5acDL+bP98IW68FHcqx35jb9Denf9g/keXfOn89X8zpwQoV17NXAr1j1T+OdSRfQ+0Zu6zzgsDztJBqsf8CnC5/j9XnaFaR/Ot9KuufKPvk9mZffo/Xr16sm8R2Q47oTmJk/iz8CD+W2vxU4DzgDuB74jzzfmcAH6+ramvRP6+WF961ZXL3A/wNuAg6vq2c06XsyOz/ekssnAr/O9f0aeE3hO/LNvIy7gE8VlnFqbt884LUN2r8b6Z/hc/O843P5z0nf9fnA5ML0K0gX17w9f64T83t8P6v+dX40cDlpfbwXOLk4f6vvSUe3h50OYLg8qLspS/7iLidd9Gwd4Df5S7luXlFH5+kOI29M6+q7HHhHYZpz8/BWhWm+XFixT6HvxDEZ+EIeXh+YA4yrW+6b8hdjY2CTvLLvkcf1AqMaxPoO4GeF118FjsjDm5Mux7Bxfh9uBA7Ny35LIfY7gQ1zvIuA7UhXqz2HtMFbB7gCeFt+r18C9ioss5dViWMlKTGtk7+UU3MdBwM/7yPGo0lf3M1Il4r4AzA2T7eivu1178NupC/3qPx6y/x8HvD+wnSzgO/0Yx27FTg0D29ASrrvA2aQNnjbAA+Sbt40iQbrX6PPkXSl1g8W6l0E7JJfn8+qHw+zaJI4SBvoReT1qdD2UyhcHiS/F1dQuLEZ8GHgrAZ1TgKuKBFXL/CvTeL6UaHdOwAL8vCrWHWzq3eR11/g46REM7KuHb2s+q59gvx9rFvWWcCH8/B6wIZ1dWxISvBbFd73A/PwZcC1pO3DG4C5ufxo0lV0tyrMX/tur8jPDb8n7dze9fVYmy+rPhhui4jFAJLmkjZqT5CuVDojXx56BGnFqHcxKWFcT7pw2ndy+eslfZm0sduEtMdS1n7AX0l6f369GenCag8UptkbuCwins5xX0r6pfjbFvVuCyyrW857C8dcNiBdPnqBpKNJv4q+F6tfWPDyiHgWeFbS9aRfX3vnumrL3iTH+yDwh4i4pUk8D0TEvBz/fGBmRISkeaTPoGmMeXhmRCzP8/+OtEdTvOdCM+8EfhoRjwJERKurCpfqeqmRtCkwJiIuy3U/l8v3Bn4cES8CD0u6Afhr4Ekar383Naj+RdLGEtIFER+IiN/n19OA40kXFmxlL+DGSDcu66vtl+R4ax4h/VBopa+4mr2f7wJ2LVyK/VX5vdyMdLn58aQN+LqF6c+OfP+SunZcmp9vB/6uwbJ+A5yUr7h8aUTcl8s/rVW37x1LWocfI+35XZ3L5wHPR8QAi1DvAAAD4ElEQVQLdespwIzIV9HN38e9ST+8avaj8ffkxibvSds5cQxM8aZAL5LeTwHzI+LNfcw7HfiapC1JewHX5fLzSBetuzNvhCc1mHclq45PbVAoF+lXU6tk05+bHTzbYDnvi4h7G0w7nrSLXr+hqL8oWuR6vhYR31stwHTP5qdbxFN8318qvH6JVet0wxgl7Unjz60MUf7eJQ3jl3QNac9hTkR8tK7uZstspmw7nitsyPt7s4uBtH0D0jrUV/1V6qxZB3hz/lGyqjLpLFJX3aF5fZpVWE6zdtTez4bvZUT8SNKtwLuBayR9lLTOvSvH8IykWaz6rrwQeZeBwnoaES/VHftq9N1YrTk0+J50kg+Ol/cUqa+/L/cCoyW9GUDSuvlGQquJiBWk/tL/Iu2u177YmwJLJa1L2sVvpJeUbCBdxrnmGuDjeV4k7aJ0Z7WiG4FDJG2Uxx0K/G8fbVpA6n8vLudT0su3/NwjP2+W2/M20l3dirEdLGkDSVuRkuHsXM8/SNokzz9G0tZ9xFJWwxj78ELtvWtiJvDB3AZy0ofy6wYRsX+kg8EfrSt/ElisdPvU2hlHG5E+r8MkjZA0mvTe9nUJ/1bx3AP0SKp9nkeSjsX05TfA25VuI1C17buQumBa6W9c1wIv37dD0u55cDPSsRdI3UHF6Y8rXJZ+S0rKl0a/PyK+Rfrh91d5OX/KSeO1pD2zqvaVtKWkDUknWtTfAqCd35N+ceIoKe9K3izpbknfaDHdn0kb869LupN0IK3Z6YoXA0ew+m74F0l93TNIX6ZGvklKEL8m9fvXnAv8DrhD0t3A96j75RQRd5D2am7Lyzk3Ilp1UxER9wCb5S4AgNNIu/535eWclsvPJPXt/x44Fji9sILfRjrD5hbgtIhYEhHXkvqof5N3339KyQ1wCc1ibOWcPP0rTkcGiIj5wFeAG/Jne0YedRHwL0q3CN15ADEfSer2uIt0nOwvSH3jtYPx15H6+v9Yoh2/zF2C9W14DjgGuCS/5y+RToxoKSKWkY6hXZrbXltnfwEcmk+pbXZr1HeQPvtW9fcrLtLJABMk3ZW7HY/L5f9O2qO/mdRdXHMuqSv0rtyOvy+xjJrDgLtzt+BrScdhrgZG5s/sNNL6XdVNpBMt5pKOxRS7qWjz96RffFl1K0XSZ4GnIqLyfzkknUI60PfNQQ/MupqkbYAfRcQ+nY6lG+Xu6AkR0Q13OyzNexxW1ndZvU/drIwdgBM6HYQNLu9xmNXJxzBmNhi1T+3slzVZPgC8fl3xkbUz2cycOMzMrBJ3VZmZWSVOHGZmVokTh5mZVeLEYWZmlfx/X2RPe+GIZocAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# visualize the sample distribution for the differences in click through rates between experiment and control group¶\n",
"plt.hist(diffs)\n",
"plt.title('Sample Distribution for our statistics')\n",
"plt.xlabel('the value of (experiment_ctr - control_ctr) for each sample')\n",
"plt.ylabel('frequency')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- From the histogram of the sample distribution, we can see the majority of our statistics is centered around 0.03. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step_3. Use this sampling distribution to simulate the distribution under the null hypothesis, by creating a random normal distribution centered at 0 with the same spread and size."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"# simulate distribution under the null hypothesis\n",
"# nomal(loc=0.0, scale=diffs.std(), size=100000)\n",
"null_vals = np.random.normal(0, diffs.std(), diffs.size)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Text(0, 0.5, 'frequency')"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEXCAYAAAC6baP3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcHFW99/HPly2sly0BSQIkQvAa9F6WyOIaRANEFLiK4MMuEhcQ8UEfg+gFQRSU64J6QcBIQFZZrhHQEMImypZgCIksyYVIYmIIhi2AKPB7/jink5pO90xXz/T0JPN9v17zmupT2+9UVfevz6nqKkUEZmZmjVqj3QGYmdmqxYnDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4mgRSYdJuqVFy75E0jdbsewG1x+Stm/RsudJ+kAe/qqki3tw2cskvTkP9+g2lHSBpK/31PJapZX7rhlV+/t0Sb9od0yNaEWskrbJx+iaPbncVnDi6AZJ75b0B0nPS1oq6feS3gEQEZdHxJg+EOMdkj7Vrvm7IyK+FRFdrrvRGCNiw4h4ortxSTpa0t1Vy/5MRJzZ3WWvSvJ2CElfripfIGl0D69rtKQFNcpbfnzWW3dPi4in8jH6eqvX1V1OHE2S9C/AjcCPgM2AIcA3gFfbGZetTNJa7Y5hVdfJNlwKfCW/H6yfcOJo3g4AEXFlRLweEa9ExC0RMRNW/laav5l9TtIcSS9KOlPSdpLukfSCpGskrVNr3sL8K3UxSNpU0o2Slkh6Ng8PzePOAt4D/Dg3gX+cy/9V0pTcSnpM0sdrVbDe/NkHcl2elfQTSSrM90lJj+RxkyVtW28jSjpC0p8l/U3SqVXjlncHSFpX0i/ydM9JekDSlp3UMSQdL2kOMKfONhyYt8OLku6sxClpWJ52rUIsd0j6lKS3AhcAe+b1PZfHd+j6knScpLl5G0+SNLgwLiR9pt72q9oG1cvt8O1XqavnS5Jm5pbv1ZLWLYz/sqRFkhZK+mTVsgdIOlfSU5IWK3W3rVdcj6SvSPor8PM6u/AR4B7gi83E35MkzZL04cLrtSU9I2mnwj4dl7fFIkknF6YdIOkHedzCPDxA0gbAb4DBeX8vK+zLdSRdmo+f2ZJGFZY3WNJ1+X35pKQTC+N2kzRN6X2/WNL3cnmH407pc+CJvPwnJR3Wiu3WDCeO5j0OvC5poqT9JG3awDz7ArsCewD/D7gQOAzYGngb8Ikm4liD9KbeFtgGeAX4MUBEnAr8DjghN4FPyG+EKcAVwBZ5nf8tacfqBdeavzB6f+AdwL8DHwf2AZB0IPBV4D+AQXn+K2sFLmkkcD5wBDAY2BwYWqeeRwEbk7bV5sBngFe6iPFAYHdgZJ1lHgacCQwEZgCX15luuYh4JK/7nry+TWrU6/3At0nbZSvgz8BVVZPV3H5N+jjp2BoO/BtwdI5jX+BLwAeBEcAHquY7h/QFaCdge1Kr+T8L499Eak1vC4zrZP1fB74oabNu1KEnXAocXng9FlgUETMKZXuRtsUYYLzy+RXgVNL7cifSPtkN+FpEvATsByzM+3vDiFiY5/kIab9uAkwiv+8krQH8GniItE33Bk6SVNnHPwR+GBH/AmwHXFNdkfw+PQ/YLyI2At5JOkb7BCeOJkXEC8C7gQAuApbkb5ZbdjLbORHxQkTMBmYBt0TEExHxPOlbzc5NxPG3iLguIl6OiBeBs4D3dTLL/sC8iPh5RLwWEQ8C1wEfK7nqsyPiuYh4Crid9IYD+DTw7Yh4JCJeA74F7KTarY6PATdGxF0R8SrpA+iNOuv7JylhbJ9beNPzPujMtyNiaUS8Umf8TYV1n0pqRWzdxTIbcRgwISIezMs+JS97WGGaetuvGedFxMKIWEr6wKos6+PAzyNiVv4APL0yQ27hHAd8MW+jF0n76tDCct8ATouIVzvZhuQP5luAr3SjDo0YrNTaXP5Heg9W/AIYqxXdZkcAl1Ut4xsR8VJEPEz6wlX5snYYcEZEPB0RS0jdzkd0Ec/dEXFzPidxGSnhQPpCMCgizoiIf+TzahexYtv+E9he0sCIWBYR99ZZ/hvA2yStFxGL8udGn+DE0Q35w/HoiBhKajEMBn7QySyLC8Ov1Hi9YdkYJK0v6adK3T0vAHcBm6j+lRnbArtXvfkOI327LOOvheGXC7FvC/ywsOylgEjfvKoNBuZXXuQPt7/VWd9lwGTgqtyV8B1Ja3cR4/xGx0fEshzr4PqTN2wwqZVRXPbf6LgN6m2/ZtRbVoftW4yJ1BpcH5he2Fe/zeUVSyLi7w3G8J/AZyWVPY7KWBgRmxT/gOVdurkl8Hvgo5I2IbUUqluR1dujsr877LOqcfVUb/d1czfTtlQlOVIrvPKl8lhSS+9RpS7X/asXnN8Lh5Bat4sk3STpX7uIp9c4cfSQiHgUuISUQLrrJdKbGoAu3ownA28Bds9N3/dWZquEVjX9fODOqjfghhHx2TrLL3v75PnAp6uWv15E/KHGtItIXU8pYGl9Uqti5SAi/hkR34iIkaRm+/7AkV3E2FXsxXVvSOqWWUja/lDYB3RMrF0tdyHpw6Oy7A1I9fpLF/PV0uFYoFyC77B9SV2ZFc+QvqzsWNhPG0dEMYE1vO/z8X896QOyqDvxN2MiqbvqYFJ3YvU2r94elW6nDvusalwz74Enq94DG0XEWICImBMRnyB1FZ8DXJuPkQ4iYnJEfJDU3fkoqdXSJzhxNEnpBPPJWnEiemtSs7des7OMh4Ad80m9dSl0MdSwEekD4Lncx3xa1fjFwJsLr28EdlA6Kb12/nuH0knfWqrn78oFwCmVcyaSNpZ0cJ1prwX2V7qseR3gDOock5L2kvT23JJ6gdTcr1y2WDbGirGFdZ8J3BcR83NXxV+AwyWtqXRSebvCfIuBoXm+Wq4Ajsn7bwCpC+i+iJjXRIwzcpyb5S8QJ5WY9xrgaEkjc1JefmxExBukD6LvS9oCQNKQQj98M74BHEPq8++J+JvxP8AuwBdI5zyqfT230nfMsV6dy68EviZpkKSBpBZU5Xcai4HNJW3cYAz3Ay8oXViwXj6G3qZ8qb6kwyUNyvvguTxPh0twlS78+EhOKK8Cy6qnaScnjua9SDrxep+kl0gJYxapBdAtEfE46UP0VtIVQXd3MvkPgPVI3yDvJXU3FP0Q+JjS1Tvn5b7sMaT+1oWk5vY5wIA6y+8wfwOx35CXd1XuOptF6jKoNe1s4HjSB+0i4Fmg3hU3byIlmhdIV/LcyYo3dqkYC64gfZguJV20ULxq5Tjgy6Quph2BYovpNmA28FdJz9So11TS+Zrrcr22o+O5gzIuI32RmEc6j3B1p1N3jOM3pOPjNmBu/l/0lVx+b95Xt5Jar02JiCdzvMVvz03H32QMr5C2+3BSC6janaQ6TwXOjYjKj3S/CUwDZgIPAw/mskpr6krgidz11GkXVj7n8WHSuaYnSe/Ni0kXd0C6kGG2pGWkY/fQGl2Ca5A+SxaSjs/3AZ9rYBP0CoUf5GRmqxFJ/wnsEBGHF8qGkT7E184XbVg3+IdRZrbayN21x9L1FVHWDe6qMrPVgqTjSCemfxMRd7U7ntWZu6rMzKwUtzjMzKyU1fIcx8CBA2PYsGHtDsPMrD0eeyz9f0u5i+SmT5/+TEQM6mq61TJxDBs2jGnTprU7DDOz9hg9Ov2/445Ss0n6c9dTuavKzMxKcuIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KcOMzMrBQnDjMzK8WJw8zMSlktfzlu1pcNG39TW9Y77+wPtWW9tvpxi8PMzEpx4jAzs1KcOMzMrBQnDjMzK8WJw8zMSvFVVdYvtevKJrPVgVscZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZXixGFmZqW0LHFI2lrS7ZIekTRb0hdy+emS/iJpRv4bW5jnFElzJT0maZ9C+b65bK6k8a2K2czMutbKHwC+BpwcEQ9K2giYLmlKHvf9iDi3OLGkkcChwI7AYOBWSTvk0T8BPggsAB6QNCki/tTC2M3MrI6WJY6IWAQsysMvSnoEGNLJLAcAV0XEq8CTkuYCu+VxcyPiCQBJV+VpnTjMzNqgV85xSBoG7Azcl4tOkDRT0gRJm+ayIcD8wmwLclm98up1jJM0TdK0JUuW9HANzMysouWJQ9KGwHXASRHxAnA+sB2wE6lF8l+VSWvMHp2UdyyIuDAiRkXEqEGDBvVI7GZmtrKW3uRQ0tqkpHF5RFwPEBGLC+MvAm7MLxcAWxdmHwoszMP1ys3MrJe18qoqAT8DHomI7xXKtypMdhAwKw9PAg6VNEDScGAEcD/wADBC0nBJ65BOoE9qVdxmZta5VrY43gUcATwsaUYu+yrwCUk7kbqb5gGfBoiI2ZKuIZ30fg04PiJeB5B0AjAZWBOYEBGzWxi3mZl1opVXVd1N7fMTN3cyz1nAWTXKb+5sPjMz6z3+5biZmZXixGFmZqU4cZiZWSlOHGZmVooTh5mZleLEYWZmpThxmJlZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZXixGFmZqU4cZiZWSlOHGZmVooTh5mZleLEYWZmpThxmJlZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZXixGFmZqU4cZiZWSlOHGZmVooTh5mZldKyxCFpa0m3S3pE0mxJX8jlm0maImlO/r9pLpek8yTNlTRT0i6FZR2Vp58j6ahWxWxmZl1rZYvjNeDkiHgrsAdwvKSRwHhgakSMAKbm1wD7ASPy3zjgfEiJBjgN2B3YDTitkmzMzKz3tSxxRMSiiHgwD78IPAIMAQ4AJubJJgIH5uEDgEsjuRfYRNJWwD7AlIhYGhHPAlOAfVsVt5mZda5XznFIGgbsDNwHbBkRiyAlF2CLPNkQYH5htgW5rF559TrGSZomadqSJUt6ugpmZpa1PHFI2hC4DjgpIl7obNIaZdFJeceCiAsjYlREjBo0aFBzwZqZWZdamjgkrU1KGpdHxPW5eHHugiL/fzqXLwC2Lsw+FFjYSbmZmbVBK6+qEvAz4JGI+F5h1CSgcmXUUcCvCuVH5qur9gCez11Zk4ExkjbNJ8XH5DIzM2uDtVq47HcBRwAPS5qRy74KnA1cI+lY4Cng4DzuZmAsMBd4GTgGICKWSjoTeCBPd0ZELG1h3GZm1omWJY6IuJva5ycA9q4xfQDH11nWBGBCz0VnZmbN8i/HzcysFCcOMzMrxYnDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUpw4zMysFCcOMzMrpcvEIWmz3gjEzMxWDY20OO6T9EtJY/NT/czMrB9rJHHsAFxIeprfXEnfkrRDa8MyM7O+qsvEEcmUiPgE8CnSc8Lvl3SnpD1bHqGZmfUpXT46VtLmwOGkFsdi4PPAJGAn4JfA8FYGaGZmfUsjzxy/B7gMODAiFhTKp0m6oDVhmZlZX9VI4nhLREStERFxTg/HY2ZmfVwjieMWSQdHxHMAkjYFroqIfVobmpn1pGHjb2rbuued/aG2rdt6XiNXVQ2qJA2AiHgW2KJ1IZmZWV/WSOJ4XdI2lReStgVqdl2Zmdnqr5GuqlOBuyXdmV+/FxjXupCsP2ln94mZNafLxBERv5W0C7AHIOCLEfFMyyMzM7M+qZEWB8AAYGmefqQkIuKu1oVlZmZ9VSM/ADwHOASYDbyRiwNw4jAz64caaXEcSPotx6utDsbMzPq+Rq6qegJYu+yCJU2Q9LSkWYWy0yX9RdKM/De2MO4USXMlPSZpn0L5vrlsrqTxZeMwM7Oe1UiL42VghqSpwPJWR0Sc2MV8lwA/Bi6tKv9+RJxbLJA0EjgU2BEYDNxauAPvT4APAguAByRNiog/NRC3mZm1QCOJY1L+KyUi7pI0rMHJDyD9Gv1V4ElJc4Hd8ri5EfEEgKSr8rROHGZmbdLI5bgTJa0HbBMRj/XAOk+QdCQwDTg5/xJ9CHBvYZoFuQxgflX57rUWKmkc+fcl22yzTa1JzMysBzTy6NgPAzOA3+bXO0kq3QLJzge2I92SfRHwX5XV1Jg2OilfuTDiwogYFRGjBg0a1GR4ZmbWlUZOjp9O6jZ6DiAiZtDkMzgiYnFEvB4RbwAXsaI7agGwdWHSocDCTsrNzKxNGkkcr0XE81VlTd2rStJWhZcHAZUrriYBh0oaIGk4MAK4H3gAGCFpuKR1SCfQm23tmJlZD2jk5PgsSf8HWFPSCOBE4A9dzSTpSmA0MFDSAuA0YLSknUiJZx7waYCImC3pGtJJ79eA4yPi9bycE4DJwJrAhIiYXaqGZmbWoxpJHJ8n3ejwVeBK0of4mV3NlJ9RXu1nnUx/FnBWjfKbgZsbiNPMzHpBI1dVvUxKHKe2PhwzM+vrGrlX1e3UOKcREe9vSURmZtanNdJV9aXC8LrAR0nnIczMrB9qpKtqelXR7wsPdTIzs36mka6qzQov1wB2Bd7UsojMzKxPa6SrajorfsX9GvAkcGwrgzIzs76rka6qpn4lbmZmq6dGuqr+o7PxEXF9z4VjZmZ9XSNdVccC7wRuy6/3Au4Anid1YTlxmJn1I40kjgBGRsQiWH6/qZ9ExDEtjczMzPqkRm5yOKySNLLFwA71JjYzs9VbIy2OOyRNJt2nKkh3qL29pVGZmVmf1chVVSdIOgh4by66MCJuaG1YZmbWVzXS4gB4EHgxIm6VtL6kjSLixVYGZmZmfVMjj449DrgW+GkuGgL8TyuDMjOzvquRk+PHA+8CXgCIiDnAFq0MyszM+q5GEserEfGPygtJa9Hko2PNzGzV10jiuFPSV4H1JH0Q+CXw69aGZWZmfVUjiWM8sAR4mPSM8JuBr7UyKDMz67s6vapK0prAxIg4HLiod0IyM7O+rNMWR0S8DgyStE4vxWNmZn1cI7/jmEd66t8k4KVKYUR8r1VBmZlZ31W3xSHpsjx4CHBjnnajwp+ZmfVDnbU4dpW0LfAU8KNeisfMzPq4zhLHBcBvgeHAtEK5SL/jeHML4zIzsz6qbldVRJwXEW8Ffh4Rby78DY8IJw0zs36qy99xRMRneyMQMzNbNTTyA0AzM7PlnDjMzKyUliUOSRMkPS1pVqFsM0lTJM3J/zfN5ZJ0nqS5kmZK2qUwz1F5+jmSjmpVvGZm1phWtjguAfatKhsPTI2IEcDU/BpgP2BE/hsHnA8p0QCnAbsDuwGnVZKNmZm1R8sSR0TcBSytKj4AmJiHJwIHFsovjeReYBNJWwH7AFMiYmlEPAtMYeVkZGZmvai3z3FsGRGLAPL/ygOhhgDzC9MtyGX1ylciaZykaZKmLVmypMcDNzOzpK+cHFeNsuikfOXCiAsjYlREjBo0aFCPBmdmZiv0duJYnLugyP+fzuULgK0L0w0FFnZSbmZmbdLbiWMSULky6ijgV4XyI/PVVXsAz+eurMnAGEmb5pPiY3KZmZm1SSO3VW+KpCuB0cBASQtIV0edDVwj6VjSzRMPzpPfDIwF5gIvA8cARMRSSWcCD+TpzoiI6hPuZmbWi1qWOCLiE3VG7V1j2gCOr7OcCcCEHgzNzMy6oa+cHDczs1WEE4eZmZXixGFmZqU4cZiZWSlOHGZmVooTh5mZleLEYWZmpThxmJlZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZXixGFmZqU4cZiZWSlOHGZmVooTh5mZleLEYWZmpThxmJlZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZWyVrsDMLPV37DxN7VlvfPO/lBb1ru6c4vDzMxKcYvDgPZ9IzSzVU9bWhyS5kl6WNIMSdNy2WaSpkiak/9vmssl6TxJcyXNlLRLO2I2M7OknV1Ve0XEThExKr8eD0yNiBHA1PwaYD9gRP4bB5zf65GamdlyfekcxwHAxDw8ETiwUH5pJPcCm0jaqh0BmplZ+xJHALdImi5pXC7bMiIWAeT/W+TyIcD8wrwLclkHksZJmiZp2pIlS1oYuplZ/9auk+PvioiFkrYApkh6tJNpVaMsViqIuBC4EGDUqFErjTczs57RlhZHRCzM/58GbgB2AxZXuqDy/6fz5AuArQuzDwUW9l60ZmZW1OuJQ9IGkjaqDANjgFnAJOCoPNlRwK/y8CTgyHx11R7A85UuLTMz633t6KraErhBUmX9V0TEbyU9AFwj6VjgKeDgPP3NwFhgLvAycEzvh2xmZhW9njgi4gng32uU/w3Yu0Z5AMf3QmhmZtaAvnQ5rpmZrQKcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUpw4zMysFCcOMzMrxYnDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUpw4zMysFCcOMzMrxYnDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpZq90B2ArDxt/U7hDMzLrkxGFmq612fhmbd/aH2rbuVnNXlZmZleLEYWZmpThxmJlZKatM4pC0r6THJM2VNL7d8ZiZ9VerROKQtCbwE2A/YCTwCUkj2xuVmVn/tKpcVbUbMDcingCQdBVwAPCnVqzMl8WaWXe163OkN67mWlUSxxBgfuH1AmD34gSSxgHj8stlkh7rxvoGAs90Y/6+xHXpm1yXvmmVr4vOWT44EKlsXbZtZKJVJXGoRll0eBFxIXBhj6xMmhYRo3piWe3muvRNrkvf5Lo0ZpU4x0FqYWxdeD0UWNimWMzM+rVVJXE8AIyQNFzSOsChwKQ2x2Rm1i+tEl1VEfGapBOAycCawISImN3CVfZIl1cf4br0Ta5L3+S6NEAR0fVUZmZm2arSVWVmZn2EE4eZmZXSbxOHpM0kTZE0J//ftM50R+Vp5kg6qlD+W0kPSZot6YL86/a26E5dJK0v6SZJj+a6nN270a8UY3f3y1mS5kta1ntRd4ir01vjSBog6eo8/j5JwwrjTsnlj0napzfjrqXZukjaXNLtkpZJ+nFvx11LN+ryQUnTJT2c/7+/t2Ov1o267CZpRv57SNJBTQcREf3yD/gOMD4PjwfOqTHNZsAT+f+meXjTPO5f8n8B1wGHrop1AdYH9srTrAP8DthvVaxLHrcHsBWwrA2xrwn8L/DmvC0fAkZWTfM54II8fChwdR4emacfAAzPy1mzjfuhO3XZAHg38Bngx+2qQw/VZWdgcB5+G/CXVbgu6wNr5eGtgKcrr8v+9dsWB+mWJRPz8ETgwBrT7ANMiYilEfEsMAXYFyAiXsjTrEXage28yqDpukTEyxFxO0BE/AN4kPQ7mXbp7n65NyIW9UqkK1t+a5y8LSu3xikq1u9aYG9JyuVXRcSrEfEkMDcvr12arktEvBQRdwN/771wO9WduvwxIiq/GZsNrCtpQK9EXVt36vJyRLyWy9elG59Z/TlxbFn5gMn/t6gxTa1bnQypvJA0mZS1XyTtoHbpdl0AJG0CfBiY2qI4G9EjdWmTRuJaPk1+Ez8PbN7gvL2pO3Xpa3qqLh8F/hgRr7YozkZ0qy6Sdpc0G3gY+EwhkZSySvyOo1mSbgXeVGPUqY0uokbZ8iwdEftIWhe4HHg/6ZtvS7S6LpLWAq4Ezot8M8lWaXVd2qiRuOpN09fq1J269DXdroukHYFzgDE9GFczulWXiLgP2FHSW4GJkn4TEaVbhqt14oiID9QbJ2mxpK0iYpGkSn9ftQXA6MLrocAdVev4u6RJpOZhyxJHL9TlQmBORPygB8LtVG/slzZp5NY4lWkW5GS9MbC0wXl7U3fq0td0qy6ShgI3AEdGxP+2PtxO9ch+iYhHJL1EOm8zrWwQ/bmrahJQuRrnKOBXNaaZDIyRtGm+umcMMFnShvlDrfJNfSzwaC/EXE/TdQGQ9E3SwXVSL8TalW7Vpc0auTVOsX4fA26LdLZyEnBoviJmODACuL+X4q6lO3Xpa5quS+6+vQk4JSJ+32sR19edugzPn1dI2hZ4CzCvqSjaeYVAO/9IfX5TgTn5/2a5fBRwcWG6T5JOVM4FjsllW+YdOJN0wuxHNHl1Qh+oy1BSM/YRYEb++9SqWJdc/h3SN6438v/Tezn+scDjpCtfTs1lZwAfycPrAr/Mcd8PvLkw76l5vsdo45VtPVSXeaRvucvyfhjZ2/H3RF2ArwEvFd4bM4AtVtG6HJE/r2aQLoI5sNkYfMsRMzMrpT93VZmZWROcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUpw4zMyslH6VOCRtIulzhdejJd3Yi+s/XdKXWryOEyU9IunyGuN2lnRxK9ffGUk35x9UtXIdX21yvgMljezpeJol6SRJ6zcx3x2SRpWcZ7Skd3Yyfn9J3ygbS4PrHiZpVjeXMU/SdYXXH5N0SQPzLesshlrlrXgPVx97zezDOstt2futXyUOYBPSLYdXZ58DxkbEYTXGfZX0Y8VepWSNiBgbEc+1eHVNJQ7SXXhrJo7Kr2172Umk22CvRD3/7JfRQM3Eket+E/CRZhJZT+uk7qPy/aRWRXWPve5o5futvyWOs4Ht8oNMvpvLNpR0rdKDjC7Pt7hG0q6S7lR6eMvkyi1GKiRtnL/prJFfr6/0AKG1JR0n6QGlh6VcV+sNV/xWIWmgpHl5eE1J383zz5T06VoVkfR/Jc3KfyflsgtI9+mfJOmLVdNvBPxbRDyUX28gaUJezx8lHVBY7oQ8/Pa8/PXzN63LJN2m9PCk4wrL/nIh3m/ksmG55fPfpF+pbp2318A87lFJF+flXy7pA5J+n5e9WxcxHi3peqWHac2R9J1cfjawXt6/K7W4CvEemWN9KNfpncBHgO/mebfL++dbku4EvlBvWTWWvb2kW/OyH8zLUt6ns5QeCHRInnZ0Xk+H40/SicBg4HZJt+dpl0k6Q9J9wJ6S9s7b5OG8jRq61bfSQ4AezPFNVXrIz2eAL+a6v0fSJZK+l9d9TqRfCd8B7F9jeR2+gec6Divs/4uUHhB2i6T18jS75vXfAxxfmLfmsZ+30+2SriDd1bWWc6nhWNboAAAHb0lEQVTxpaFefI1sq87k/fpg4fUISdPz8DxJ50i6P/9tn8u3zdt8Zv6/Ta1jLy/y4Dzv45Le08X22UrSXXn+WYXpK++3DZQe1vZQHn9Id+vf1lsatOGn+sOAWYXXo0m3HB5KSqL3kB5AszbwB2BQnu4QYEKN5f2KFQ9BOoR8Swxg88I03wQ+n4dPB76Uh+8ARuXhgcC8PDwO+FoeHkC6AdnwqvXuSnoDbQBsSLqNwM553DxgYI1Y9wKuK7z+FnB4Ht6EdAuDDfJ2uAs4KK/7XYXYHwLWy/HOJ324jSHdIFF53huB9+Zt/QawR2Gd8/K8w4DXgLfneaYDE/IyDgD+p4sYjyY9vGlj0u0V/gxsnafr9AFOwI6kW3oMzK8rtzS5BPhYYbo7gP9u4hi7DzgoD69LajV8lHQDzDVJt6t5ivQgndHUOP5q7UfSbWE+XljufGCH/PpS4KTq46pGbIPyfMOr6n46+bgsbIsbKTxICjgM+FGNZVbPOyvv38o+3imXX1PYlzOB9+Xh75Lfk9Q59vN2eomq90HVcbUl6bY525Puz3RJZ/EVjxWqPheqPi9eoePtRv7Kivfw7YX6fYsV7/N5rLgVyJHAjXn418BRefiTrDjOL2HlY++/8vBY4NYuts/JhfWtCWxU9X77KHBRYfkblz2uq//6W4ujlvsjYkFEvEE6MIaRbv71NmCKpBmk+9XUerjR1aSEAflJW3n4bZJ+J+lh0huuTBN6DHBkXu99pHs3jaia5t3ADZEemLMMuB54TxfL3QpYUrWe8Xk9d5A+jLbJ2+Fo4DLgzuh4Y7dfRcQrEfEM6U2zW17OGOCPpJbFvxbi/XNE3Fsnnicj4uG8vtnA1EhH9cOkfVA3xjxuakQ8H+mW0H8Ctu2i/hXvB67NdSAiOrub69WdjFuJUqtuSETckJf994h4mbS/royI1yNiMXAn8I48W63jr5bXSU+ahHR8PhkRj+fXE0nJuit7AHdFelBUV3X/ZUS8Xnj9NOmLQhlPRsSMPDwdGCZpY2CTiLgzl19WmL6zY//+Stx1vE5KQqeUjLEr/xsRO1X+gAsK4y4GjlHqPjsEuKIw7srC/z3z8J6FaS4jHRf1XJ//T6fj+6HW9nkgx3E68PaIeLFqWQ8DH8itoPdExPNd1LlLq/Vt1RtUfCjL66RtImB2ROxZe5blJgHflrQZqRVwWy6/hHQDsYckHU3HW4BXvMaKrsJ1C+UifXPp7G6vte6335VXaqznoxHxWI1pR5BuTlf9QVF9Y7PKcyS+HRE/7RBg6g54qZN4itv9jcLrN1hxXNaMUdLu1N5vjRCNPzOiZvxKD/DaEpgWEZ+qWna9ddbTaD3+Xvggb2b/V+Zrtu7rko6hasXjuDJdRXXd1usihprHvqTRNeKp5TJS4pjdYHzddR1wGul9Pz0i/lYYF3WGaaAcVmy74jFR97NB0nuBDwGXSfpuRFy6fCURj0valdR6+bakWyLijE7W3aX+1uJ4EdiogekeAwZJ2hNA6bzFSq2G/G3/fuCHpOZo5Y29EbBI0tqkFkct80jJBlLTumIy8Nk8L5J2kLRB1bx3AQcqnXvYgNSt9Lsu6lRpxhfX83lp+TmdnfP/jXN93gtsLqkY2wGS1pW0OSkZPpCX80lJG+b5h0iq9dS+ZtSMsQv/rGy7OqYCH891ICd9aPzYICL2yd9AP1VV/gLpGQgH5mUPUDq/dRdwSO6jHkTatl3dMr2zeB4lfXuv7M8jSK2YrtwDvE/ptu1l674DqZun2jxgl7y8XUhdJ3VFOln7vKTKt+3i+6ORY7+zZf8T+D4dHw9QKr4ycmt3MnA+8POq0YcU/t+Th/9A6pmAVO+783Cjx17N7aN0i/SnI+Ii4Gfk+lZIGgy8HBG/IJ0L2oVu6leJI38j+H0+QfTdTqb7B+nD/BxJD5G6EOpdrng1cDgduzW+TmpKTqH+czrOJR0EfyD1Q1ZcTOp6eVDpUsCfUvUtNCIeJLVq7s/ruTgi/livPnmeR4GNc3cKwJmkczkz83rOzOXfJ/XtPw4cC5xdSAT3k66wuRc4MyIWRsQtpOb3Pblr7loa/ABuQL0YO3Nhnr7myfGImA2cBdyZ9+338qirgC8rnXDerta8DToCOFHSTNIHxZtIDwGaSTpHdBvw/yLirw3U4zfKJ8er6vB34Bjgl3mbv0HHLpSaImIJqZ/8+lz3yjH7a+CgfHK1XpfnXqR9X+06YLPcffJZ0nmorhwD/ETp5HixFdPlsd+An1XN00x8ZVxOajncUlU+QOlChi8AlQtVTiR1Kc0kHSeViy4aPfbqbZ/RwAxJfySdz/hh1XxvB+7P2+BU0nnXbvFt1fsRpSutXoyI0r/lyP2nyyLi3B4PzPo0SVsCV0TE3u2Opa9RumJr44j4eqFsHukChWfaFliL+RxH/3I+cHC7g7BVzjakK3esQNINwHakCy76Fbc4bLWUz2FMrTFq76qTmKul3E1S/duOIyKi3u8gzBrmxGFmZqX0q5PjZmbWfU4cZmZWihOHmZmV4sRhZmal/H/LrVMXpUNzZAAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# plot the null distribution\n",
"plt.hist(null_vals)\n",
"# plot the observed statistic with the null distribution\n",
"plt.axvline(obs_diff, c = 'red')\n",
"plt.title('Simulate the distribution under Null Hypothesis')\n",
"plt.xlabel('the value of (experiment_ctr - control_ctr) under Null Hypothesis')\n",
"plt.ylabel('frequency')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Our null hypothesis assume CTR_{experiment} - CTR_{control} <= 0, thus we simulate the null hypothesis distribution centered at zero, with the same standard deviation and same size as our sampling distribution. As we can see our observed difference (the red line) falls far to the right; it seems unlikely that our statistic is from this null. To prove our conclusio, we need to calculate the p-value, i.e. the probability of our observed statistic came from this null distribution"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step_4. Compute the p-value by finding the proportion of values in the null distribution that were greater than our observed difference."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The p-value is 0.0\n"
]
}
],
"source": [
"# compute the p-value\n",
"p_value = (null_vals > obs_diff).mean()\n",
"print(f'The p-value is {p_value}')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step_5. Use this p-value to determine the statistical significance of our observed difference."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Conclusion**\n",
"- We set p-value threshhold as 0.05, our calculated p-value is close to 0, which is less than our threshhold, thus we can conclude that our observed difference is statistically significant and we can reject null hypothesis.\n",
"- We can recommend to change the design of homepage to the new version."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Summary of my steps:**\n",
"\n",
"First I set hypothesis, and my null hypothesis is H0: CTR_{experiment} - CTR_{control} <= 0.\n",
"\n",
"In Step_1, I have calculated the observed difference in CTR between experiment and control group as **obs_diff**, i.e. CTR_{experiment} - CTR_{control} = 0.030034443684015644. \n",
"\n",
"I can not make conclusion to say the click through rate in experiment group is higher than the click through rate in control group by this one number (the observed difference) calculated by our collected data, because only judging from one sample, the result might be by random chance. I need to run statistical test to see if this number is statistically significant in order to reject our null hypothesis. \n",
"\n",
"With limited time and resources, I can not collect the same size of data 10000 times, thus I use bootstrapping technique to take sample from the a group (our sample data) with replacement 10000 times; this is to simulate the creation of sampling distribution for our statistics. \n",
"\n",
"From the histogram of the sample distribution, we can see the majority of the number is centered around 0.03. Since the null hypothesis is CTR_{experiment} - CTR_{control} <= 0, we need to create a normal distribution centered at 0, with the same standard deviation as our sampling distribution we simulated here; this is to simulate the null. \n",
"\n",
"I then calculated the p-value for our statistics, which is the observed difference in proportion by simulating the distribution under null hypothesis and then finding the probability that our statistic came from this distribution.\n",
"\n",
"With p-value under 0.05, the difference in click through rates for the control and expriment group does appear to be significant, i.e. it seems unlikely that our statistic is from this null. Thus I can reject null hypothesis, and based on these results, I can give recommendation that our company should launch the new homepage design."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}