{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"toc": true
},
"source": [
"<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
"<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#A/B-Testing-Case-Study\" data-toc-modified-id=\"A/B-Testing-Case-Study-1\"><span class=\"toc-item-num\">1 </span>A/B Testing Case Study</a></span><ul class=\"toc-item\"><li><span><a href=\"#Project-Summary\" data-toc-modified-id=\"Project-Summary-1.1\"><span class=\"toc-item-num\">1.1 </span>Project Summary</a></span></li><li><span><a href=\"#Data-and-Analysis\" data-toc-modified-id=\"Data-and-Analysis-1.2\"><span class=\"toc-item-num\">1.2 </span>Data and Analysis</a></span><ul class=\"toc-item\"><li><span><a href=\"#Data-Cleaning-and-EDA\" data-toc-modified-id=\"Data-Cleaning-and-EDA-1.2.1\"><span class=\"toc-item-num\">1.2.1 </span>Data Cleaning and EDA</a></span></li><li><span><a href=\"#Test-and-Control-Group-Assignment\" data-toc-modified-id=\"Test-and-Control-Group-Assignment-1.2.2\"><span class=\"toc-item-num\">1.2.2 </span>Test and Control Group Assignment</a></span></li><li><span><a href=\"#Conversion-Rates\" data-toc-modified-id=\"Conversion-Rates-1.2.3\"><span class=\"toc-item-num\">1.2.3 </span>Conversion Rates</a></span></li><li><span><a href=\"#Statistical-Testing\" data-toc-modified-id=\"Statistical-Testing-1.2.4\"><span class=\"toc-item-num\">1.2.4 </span>Statistical Testing</a></span><ul class=\"toc-item\"><li><span><a href=\"#Statistical-Framework\" data-toc-modified-id=\"Statistical-Framework-1.2.4.1\"><span class=\"toc-item-num\">1.2.4.1 </span>Statistical Framework</a></span></li><li><span><a href=\"#Power-Analysis\" data-toc-modified-id=\"Power-Analysis-1.2.4.2\"><span class=\"toc-item-num\">1.2.4.2 </span>Power Analysis</a></span></li><li><span><a href=\"#Hypothesis-Testing\" data-toc-modified-id=\"Hypothesis-Testing-1.2.4.3\"><span class=\"toc-item-num\">1.2.4.3 </span>Hypothesis Testing</a></span></li></ul></li></ul></li></ul></li></ul></div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## A/B Testing Case Study"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook contains an extract of an A/B test I conducted on web load data. I have generalized the description of the case study to keep it anonymized."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Project Summary\n",
"\n",
"The client is interested in improving their checkout flow. Currently, the checkout flow involves **three separate steps**. The A/B test aims to test a potential improvement to the first step that hopefully will improve conversion in this step.\n",
"\n",
"In the first step of the checkout flow, customers complete a 6 field form in order to proceed to the next checkout step. The client wants to test introducing a feature that reduces the number of fields in the form from 6 to 5. Rather than asking each applicant to fill in the 6th field, we can use a data vendor to look this up based on the other information entered by the customer for a cost of $2 per customer.\n",
"\n",
"The client conducts a split test that diverts on form loads. During the test, 50\\% of forms will load as the 6 field form and the other 50\\% will load as a 5 field form.\n",
"\n",
"What is the impact of this change?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Data and Analysis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Data Cleaning and EDA"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Import packages\n",
"import numpy as np\n",
"import pandas as pd\n",
"from matplotlib import pyplot as plt\n",
"import seaborn as sns\n",
"from scipy import stats"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Read data\n",
"loads = pd.read_csv('data/loads.csv', parse_dates=True) # DF of all web loads\n",
"testGroup = pd.read_csv('data/test_group.csv', parse_dates=True) # DF identifying test group observations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`loads` contains 190830 observations of loads of all versions of the form. Each observation is identified by a unique id (`load_id`) and contains data on the load's date (`load_date`), whether the load resulted in a completion of the first checkout step (`completed_form`) and second checkout step (`completed_second`), and whether the customer completed checkout (each checkout identified by a `checkout_id`)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 190830 entries, 0 to 190829\n",
"Data columns (total 6 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 Unnamed: 0 190830 non-null int64 \n",
" 1 load_id 190830 non-null object\n",
" 2 checkout_id 49323 non-null object\n",
" 3 load_date 190830 non-null object\n",
" 4 completed_form 190830 non-null int64 \n",
" 5 completed_second 190830 non-null int64 \n",
"dtypes: int64(3), object(3)\n",
"memory usage: 8.7+ MB\n"
]
}
],
"source": [
"loads.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`testGroup` identifies 8495 loads of the 5-field form. Each observation identifies the `load_id` associated with observations assigned to the test group and the date of the load (`assignment_date`). We notice that the only 8495 observations are from the test group, which probably means that our `loads` data contains many more observations than an assumed 50-50 split experiment."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 8495 entries, 0 to 8494\n",
"Data columns (total 3 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 Unnamed: 0 8495 non-null int64 \n",
" 1 load_id 8495 non-null object\n",
" 2 assignment_date 8495 non-null object\n",
"dtypes: int64(1), object(2)\n",
"memory usage: 199.2+ KB\n"
]
}
],
"source": [
"testGroup.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before exploring the number of observations in the `loads` and `testGroup` data, we merge them to identify the relevant control and test observations in our experiment. We transform `load_date` and `assignment_date` into datetime type, drop the column `Unnamed:0` as it was an index column from the original csv files, and then merge `loads` and `testGroup` dataframes together on the unique `load_id` identifier. We find that all 8495 observations from the test group successfully merge."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"8495 observations successfully merged\n"
]
}
],
"source": [
"# Transform date columns into datetime\n",
"loads['load_date'] = pd.to_datetime(loads['load_date'])\n",
"testGroup['assignment_date'] = pd.to_datetime(testGroup['assignment_date'])\n",
"\n",
"# Drop index columns\n",
"loads.drop('Unnamed: 0', axis=1, inplace=True)\n",
"testGroup.drop('Unnamed: 0', axis=1, inplace=True)\n",
"\n",
"# Merge prequals with intellicron_prequals\n",
"mergedDF = pd.merge(loads, testGroup, how='outer', on='load_id', indicator=True)\n",
"\n",
"print('{} observations successfully merged'.format(sum(mergedDF['_merge']=='both')))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After merging, we separate our data into two data frames. `test` includes all observations in the test group. `control` includes all observations not identified to be part of the test group."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# Get control and test data\n",
"test = mergedDF.loc[mergedDF['_merge'] == 'both', mergedDF.columns != '_merge']\n",
"control = mergedDF.loc[mergedDF['_merge'] == 'left_only', mergedDF.columns != '_merge']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Test and Control Group Assignment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The client did not give us any information on when they ran the experiment, so we investigate the date range of the experiment test group and compare it to our control group. We find that the experiment ranged from 2019-06-16 to 2019-06-30, while the control group contained data from 2019-01-01 to 2019-06-30. We have many observations of pre-experiment data of the 6-field form which can explain why the control group contains more observations."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test started 2019-06-16 00:00:00 and ended 2019-06-30 00:00:00\n",
"Control date range started 2019-01-01 00:00:00 and ended 2019-06-30 00:00:00\n"
]
}
],
"source": [
"# Check the test period\n",
"testStart = test['load_date'].min()\n",
"testEnd = test['load_date'].max()\n",
"print('Test started {} and ended {}'.format(testStart, testEnd))\n",
"\n",
"# Check control period\n",
"controlStart = control['load_date'].min()\n",
"controlEnd = control['load_date'].max()\n",
"print('Control date range started {} and ended {}'.format(controlStart, controlEnd))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From the dates analysis, we assume that the experiment occured between 2019-06-16 and ended 2019-06-30 and drop all observations from the control group outside of this date range."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# Get pre-experiment data\n",
"pre = control[control['load_date'] < testStart]\n",
"# Only get control observations during the test period\n",
"control = control[(control['load_date'] >= testStart) & (control['load_date'] <= testEnd)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After deleting the pre-experiment data, however, we still find a relatively large imbalance in the control and test sample sizes. The control group has 1,195 more observations than the test group."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"There are 18185 total trials\n",
"There are 9690 observations in the control group\n",
"There are 8495 observations in the test group\n",
"There are 1195 more observations in the control group than the test group\n"
]
}
],
"source": [
"# Check observations\n",
"nControl = len(control)\n",
"nTest = len(test)\n",
"\n",
"print('There are {} total trials'.format(nControl + nTest))\n",
"print('There are {} observations in the control group'.format(nControl))\n",
"print('There are {} observations in the test group'.format(nTest))\n",
"print('There are {} more observations in the control group than the test group'.format(nControl-nTest))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From plotting the observations across the test and control group, we can see that the client is consistently assigning more loads to the control group on every day of the experiment, so there is likely a non-random splitting process going on in the background."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABIQAAAJuCAYAAADB3dnoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdfdBddX33+8+XhJKiNDcPgYLx3IEWqyDhwRyg4jO1gFRBpjhUeRJa7k71qK0oiT1VYAabVodaxoc5DIrRwo05nmNhtPUgDw5k6oEG7oAgcoiKEEMhxsItKFTwd/5gFy8gkHUl++K6wu/1mnH23muvva5fvnucse+uvVa11gIAAABAP7aa7gUAAAAA8NwShAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQmdnTvYAk2WmnndqCBQumexkAAAAAzxs33HDDj1tr8zb03owIQgsWLMjKlSunexkAAAAAzxtV9cNnes9PxgAAAAA6IwgBAAAAdEYQAgAAAOjMjLiGEAAAADBz/eIXv8iaNWvy8MMPT/dS2IA5c+Zk/vz52XrrrQd/RhACAAAAntWaNWuy3XbbZcGCBamq6V4OE7TWsn79+qxZsya777774M/5yRgAAADwrB5++OHsuOOOYtAMVFXZcccdJ332liAEAAAAbJQYNHNtyncjCAEAAAB0xjWEAAAAgElZsPhrYz3enUuPHOvxnvHv3Hln/uVf/iVvf/vbJ/W5z3/+81m5cmU++clPJkm+8IUv5G//9m/TWktrLaecckpOP/30sa71ox/9aD70oQ+N9ZgTOUMIAAAA6MKdd96Ziy++eIPvPfroo4OO8c///M/5xCc+kcsvvzy33nprbrzxxsydO3ecy0zyeBCaSoIQAAAAsEX4whe+kIULF2bffffNCSeckB/+8Ic59NBDs3Dhwhx66KG56667kiQnn3xy3vOe9+SVr3xl9thjj3z5y19OkixevDjXXntt9ttvv/zd3/1dPv/5z+fYY4/Nm9/85vz+7/9+fvKTn+Too4/OwoULc/DBB+fmm29+2hr++q//Oh//+Mez2267JXn8lu9/8id/kiRZtWpVDj744CxcuDBvfetb8+///u9Jkte97nVZuXJlkuTHP/5xFixYkOTxM4+OOeaYHH744dlzzz3zwQ9+8Il1/vznP89+++2Xd7zjHXnooYdy5JFHZt99983LX/7yfOlLX9rsWQpCAAAAwIx366235pxzzslVV12Vm266KX//93+fd7/73TnxxBNz88035x3veEfe8573PLH/PffckxUrVuSrX/1qFi9enCRZunRpXv3qV2fVqlX58z//8yTJt771rSxbtixXXXVVPvKRj2T//ffPzTffnI9+9KM58cQTn7aOW265Ja94xSs2uMYTTzwxf/M3f5Obb745++yzT84666yN/rtWrVqVL33pS/n2t7+dL33pS7n77ruzdOnS/Pqv/3pWrVqViy66KF//+tez22675aabbsott9ySww8/fFNG+CSCEAAAADDjXXXVVfnDP/zD7LTTTkmSHXbYId/61reeuB7QCSeckBUrVjyx/9FHH52tttoqe+21V+69995nPO4b3/jG7LDDDkmSFStW5IQTTkiSvOENb8j69evzwAMPDFrfAw88kPvvvz+vfe1rkyQnnXRSrrnmmo1+7tBDD83cuXMzZ86c7LXXXvnhD3/4tH322WefXHHFFTnjjDNy7bXXjuUnaoIQAAAAMOO11jZ6e/WJ72+zzTZP+uwzecELXvCs+z31b+6999654YYbNrreiWbPnp1f/vKXSZKHH374Se9NXOesWbM2eC2jl7zkJbnhhhuyzz77ZMmSJTn77LMn9fc3RBACAAAAZrxDDz00y5cvz/r165MkP/nJT/LKV74yl1xySZLkoosuyqte9apnPcZ2222Xn/70p8/4/mte85pcdNFFSZJvfvOb2WmnnfIbv/EbT9pnyZIl+eAHP5h/+7d/S5I88sgjOe+88zJ37txsv/32ufbaa5MkX/ziF584W2jBggVPRKT/vJ7Rxmy99db5xS9+kSRZu3Zttt122xx//PE5/fTTc+ONNw46xrNx23kAAABgUp6r28RPtPfee+cv//Iv89rXvjazZs3K/vvvn/POOy+nnHJKPvaxj2XevHm58MILn/UYCxcuzOzZs7Pvvvvm5JNPzvbbb/+k988888y8853vzMKFC7Pttttm2bJlTzvGm970ptx77735vd/7vSfOWjrllFOSJMuWLcuf/umf5mc/+1n22GOPJ9Zz+umn521ve1u++MUv5g1veMOgf+9pp52WhQsX5oADDsiJJ56YD3zgA9lqq62y9dZb5zOf+cygYzyberbTpp4rixYtav95tW0AAABgZrntttvyspe9bLqXwbPY0HdUVTe01hZtaH8/GQMAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdmT3dCwAAAAC2MGfOHfPxHnjWt++///5cfPHF+bM/+7NJH/oTn/hETjvttGy77bZJkgcffDDvf//7c8UVV2TOnDnZcccd87GPfSwHHXTQJi19Q1atWpW1a9fmTW9609iOOW6CEAAAMGUWLP7a2I5159Ijx3YsYMty//3359Of/vQmB6Hjjz/+iSD0x3/8x9l9991zxx13ZKuttsr3v//93HbbbWNd76pVq7Jy5UpBCAAAAGBTLV68ON/73vey33775Y1vfGN23nnnLF++PI888kje+ta35qyzzspDDz2Ut73tbVmzZk0ee+yx/NVf/VXuvfferF27Nq9//euz00475YILLsh1112Xiy66KFtt9fhVdPbYY4/sscceSZJzzz03n/vc55I8Ho7e97735c4778wf/MEf5JZbbkmSfPzjH8+DDz6YM888M6973ety0EEH5eqrr87999+fz372sznooIPy4Q9/OD//+c+zYsWKLFmyJL/5m7+Z9773vUmSqso111yT7bbbbhom+SuCEAAAADCjLV26NLfccktWrVqVyy+/PF/+8pdz/fXXp7WWt7zlLbnmmmuybt267Lbbbvna1x4/M/GBBx7I3Llzc+655+bqq6/OTjvtlMsuuyz77bdfZs2a9bS/ccMNN+TCCy/Mddddl9ZaDjrooLz2ta/N9ttv/6xre/TRR3P99dfnn/7pn3LWWWfliiuuyNlnn52VK1fmk5/8ZJLkzW9+cz71qU/lkEMOyYMPPpg5c+aMf0iT5KLSAAAAwBbj8ssvz+WXX579998/BxxwQL773e/mjjvuyD777JMrrrgiZ5xxRq699trMnTu56xytWLEib33rW/OCF7wgL3zhC3PMMcfk2muv3ejnjjnmmCTJK17xitx5550b3OeQQw7JX/zFX+S8887L/fffn9mzp//8HEEIAAAA2GK01rJkyZKsWrUqq1atyurVq3PqqafmJS95SW644Ybss88+WbJkSc4+++ynfXbvvffOTTfdlF/+8pcbPO6GzJ49+0n7P/zww096f5tttkmSzJo1K48++ugGj7F48eJccMEF+fnPf56DDz443/3udwf/e6eKIAQAAADMaNttt11++tOfJkkOO+ywfO5zn8uDDz6YJPnRj36U++67L2vXrs22226b448/PqeffnpuvPHGp332t37rt7Jo0aJ85CMfeSIA3XHHHbn00kvzmte8Jv/4j/+Yn/3sZ3nooYfyla98Ja9+9auzyy675L777sv69evzyCOP5Ktf/eqk1psk3/ve97LPPvvkjDPOyKJFi2ZEEJr+c5QAAACALctGbhM/bjvuuGMOOeSQvPzlL88RRxyRt7/97fnd3/3dJMkLX/jC/MM//ENWr16dD3zgA9lqq62y9dZb5zOf+UyS5LTTTssRRxyRXXfdNVdffXUuuOCCvP/9789v//ZvZ9ttt33itvMHHHBATj755Bx44IFJHr+o9P77758k+fCHP5yDDjoou+++e1760pdudL2vf/3rs3Tp0uy3335ZsmRJVqxYkauvvjqzZs3KXnvtlSOOOGKKJjVcPdMpUc+lRYsWtZUrV073MgAAgDFz23l4frjtttvyspe9bLqXwbPY0HdUVTe01hZtaH8/GQMAAADojCAEAAAA0BlBCAAAANiomXDJGTZsU74bQQgAAAB4VnPmzMn69etFoRmotZb169dnzpw5k/qcu4wBAAAAz2r+/PlZs2ZN1q1bN91LYQPmzJmT+fPnT+ozghAAAADwrLbeeuvsvvvu070MxshPxgAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGUEIAAAAoDMbDUJV9TtVtWrCf/5nVb2vqnaoqm9U1R2jx+0nfGZJVa2uqtur6rCp/ScAAAAAMBkbDUKttdtba/u11vZL8ookP0vylSSLk1zZWtszyZWj16mqvZIcl2TvJIcn+XRVzZqi9QMAAAAwSZP9ydihSb7XWvthkqOSLBttX5bk6NHzo5Jc0lp7pLX2gySrkxw4jsUCAAAAsPkmG4SOS/LfR893aa3dkySjx51H21+U5O4Jn1kz2vYkVXVaVa2sqpXr1q2b5DIAAAAA2FSDg1BV/VqStyT5Pze26wa2tadtaO381tqi1tqiefPmDV0GAAAAAJtpMmcIHZHkxtbavaPX91bVrkkyerxvtH1NkhdP+Nz8JGs3d6EAAAAAjMfsSez7R/nVz8WS5LIkJyVZOnq8dML2i6vq3CS7JdkzyfWbv1QAAJg5Fiz+2liOc+fSI8dyHOiF/+7BeAwKQlW1bZI3JvlvEzYvTbK8qk5NcleSY5OktXZrVS1P8p0kjyZ5V2vtsbGuGgAAAIBNNigItdZ+lmTHp2xbn8fvOrah/c9Jcs5mrw4AAACAsZvsXcYAAAAA2MIJQgAAAACdEYQAAAAAOjOZu4wBAAAwBcZ156zE3bOAYZwhBAAAANAZZwgBAABbhjPnjuk4D4znOABbMEEIAAAAnmf8DJGN8ZMxAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADozOzpXgBAbxYs/tpYjnPn0iPHchwAAKA/zhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnXHbeQAAAKBbCxZ/bSzHuXPpkWM5znNFEAIA4Anj+h/FyZb3P4wBoCeCEABswfwf7wAAbArXEAIAAADojCAEAAAA0BlBCAAAAKAzriEEwIzk2jgAADB1nCEEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRm9nQvAAAAgDE6c+4Yj/XA+I4FzCjOEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwYFoar6L1X15ar6blXdVlW/W1U7VNU3quqO0eP2E/ZfUlWrq+r2qjps6pYPAAAAwGQNPUPo75N8vbX20iT7JrktyeIkV7bW9kxy5eh1qmqvJMcl2TvJ4Uk+XVWzxr1wAAAAADbNRm87X1W/keQ1SU5OktbafyT5j6o6KsnrRrstS/LNJGckOSrJJa21R5L8oKpWJzkwybfGvHZgBlmw+GtjO9adS48c27EAAAB4uiFnCO2RZF2SC6vqf1TVBVX1giS7tNbuSZLR486j/V+U5O4Jn18z2vYkVXVaVa2sqpXr1q3brH8EAAAAAMMNCUKzkxyQ5DOttf2TPJTRz8OeQW1gW3vahtbOb60taq0tmjdv3qDFAgAAALD5hgShNUnWtNauG73+ch4PRPdW1a5JMnq8b8L+L57w+flJ1o5nuQAAAABsro1eQ6i19m9VdXdV/U5r7fYkhyb5zug/JyVZOnq8dPSRy5JcXFXnJtktyZ5Jrp+KxcNzYVzXxnFdHAAAAGaKjQahkf8tyUVV9WtJvp/knXn87KLlVXVqkruSHJskrbVbq2p5Hg9GjyZ5V2vtsbGvHAAAAIBNMigItdZWJVm0gbcOfYb9z0lyzmasCwAAAIApMvQMIQAAAHj+OHPuGI/1wPiOBc+RIReVBgAAAOB5xBlCAAAwnZylAMA0EIQAgC64ayQAbKJxhWvRekbxkzEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGbed79S4br2buP0uAAAAbGmcIQQAAADQGUEIAAAAoDN+MgbMPGfOHdNxHhjPcWaqcc0pef7PCgAAeBJnCAEAAAB0RhACAAAA6IwgBAAAANCZ5901hNxOHQAAAODZOUMIAAAAoDOCEAAAAEBnnnc/GQMAAAB4zp05d4zHemB8x3oGzhACAAAA6IwzhACAx21h/18tAAA2nTOEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZF5UGAGBqjOtC5S5SDgBj5wwhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojLuMAQBMxrjunJW4exYAMG2cIQQAAADQGUEIAAAAoDOCEAAAAEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOjM7OlewIx25twxHeeB8RwHAAAAYAycIQQAAADQGUEIAAAAoDN+MgbA85+fAAMAwJM4QwgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANCZQUGoqu6sqm9X1aqqWjnatkNVfaOq7hg9bj9h/yVVtbqqbq+qw6Zq8QAAAABM3uxJ7Pv61tqPJ7xenOTK1trSqlo8en1GVe2V5LgkeyfZLckVVfWS1tpjY1s1bInOnDvGYz0wvmMBAADQnc35ydhRSZaNni9LcvSE7Ze01h5prf0gyeokB27G3wEAAABgjIYGoZbk8qq6oapOG23bpbV2T5KMHncebX9RkrsnfHbNaNuTVNVpVbWyqlauW7du01YPAAAAwKQN/cnYIa21tVW1c5JvVNV3n2Xf2sC29rQNrZ2f5PwkWbRo0dPeZwvip1AAAACwRRl0hlBrbe3o8b4kX8njPwG7t6p2TZLR432j3dckefGEj89PsnZcCwYAAABg82w0CFXVC6pqu/98nuT3k9yS5LIkJ412OynJpaPnlyU5rqq2qardk+yZ5PpxLxwAAACATTPkJ2O7JPlKVf3n/he31r5eVf+aZHlVnZrkriTHJklr7daqWp7kO0keTfIudxgDAAAAmDk2GoRaa99Psu8Gtq9PcugzfOacJOds9uoAAAAAGLvNue08AAAAAFsgQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADojCAEAAAB0RhACAAAA6IwgBAAAANAZQQgAAACgM4IQAAAAQGcEIQAAAIDOCEIAAAAAnRGEAAAAADozOAhV1ayq+h9V9dXR6x2q6htVdcfocfsJ+y6pqtVVdXtVHTYVCwcAAABg00zmDKH3JrltwuvFSa5sre2Z5MrR61TVXkmOS7J3ksOTfLqqZo1nuQAAAABsrkFBqKrmJzkyyQUTNh+VZNno+bIkR0/Yfklr7ZHW2g+SrE5y4HiWCwAAAMDmGnqG0CeSfDDJLyds26W1dk+SjB53Hm1/UZK7J+y3ZrTtSarqtKpaWVUr161bN+mFAwAAALBpNhqEquoPktzXWrth4DFrA9va0za0dn5rbVFrbdG8efMGHhoAAACAzTV7wD6HJHlLVb0pyZwkv1FV/5Dk3qratbV2T1XtmuS+0f5rkrx4wufnJ1k7zkUDAAAAsOk2eoZQa21Ja21+a21BHr9Y9FWtteOTXJbkpNFuJyW5dPT8siTHVdU2VbV7kj2TXD/2lQMAAACwSYacIfRMliZZXlWnJrkrybFJ0lq7taqWJ/lOkkeTvKu19thmrxQAAACAsZhUEGqtfTPJN0fP1yc59Bn2OyfJOZu5NgAAAACmwNC7jAEAAADwPCEIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzGw1CVTWnqq6vqpuq6taqOmu0fYeq+kZV3TF63H7CZ5ZU1eqqur2qDpvKfwAAAAAAkzPkDKFHkryhtbZvkv2SHF5VBydZnOTK1tqeSa4cvU5V7ZXkuCR7Jzk8yaeratZULB4AAACAydtoEGqPe3D0cuvRf1qSo5IsG21fluTo0fOjklzSWnuktfaDJKuTHDjWVQMAAACwyQZdQ6iqZlXVqiT3JflGa+26JLu01u5JktHjzqPdX5Tk7gkfXzPaBgAAAMAMMCgItdYea63tl2R+kgOr6uXPsntt6BBP26nqtKpaWVUr161bN2y1AAAAAGy2Sd1lrLV2f5Jv5vFrA91bVbsmyejxvtFua5K8eMLH5idZu4Fjnd9aW9RaWzRv3rxNWDoAAAAAm2LIXcbmVdV/GT3/9SS/l+S7SS5LctJot5OSXDp6flmS46pqm6raPcmeSa4f98IBAAAA2DSzB+yza5JlozuFbZVkeWvtq1X1rSTLq+rUJHclOTZJWmu3VtXyJN9J8miSd7XWHpua5QMAAAAwWRsNQq21m5Psv4Ht65Mc+gyfOSfJOZu9OgAAAADGblLXEAIAAABgyycIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZzYahKrqxVV1dVXdVlW3VtV7R9t3qKpvVNUdo8ftJ3xmSVWtrqrbq+qwqfwHAAAAADA5Q84QejTJ+1trL0tycJJ3VdVeSRYnubK1tmeSK0evM3rvuCR7Jzk8yaeratZULB4AAACAydtoEGqt3dNau3H0/KdJbkvyoiRHJVk22m1ZkqNHz49Kcklr7ZHW2g+SrE5y4LgXDgAAAMCmmdQ1hKpqQZL9k1yXZJfW2j3J49Eoyc6j3V6U5O4JH1sz2vbUY51WVSurauW6desmv3IAAAAANsngIFRVL0zyfyV5X2vtfz7brhvY1p62obXzW2uLWmuL5s2bN3QZAAAAAGymQUGoqrbO4zHootba/z3afG9V7Tp6f9ck9422r0ny4gkfn59k7XiWCwAAAMDmGnKXsUry2SS3tdbOnfDWZUlOGj0/KcmlE7YfV1XbVNXuSfZMcv34lgwAAADA5pg9YJ9DkpyQ5NtVtWq07UNJliZZXlWnJrkrybFJ0lq7taqWJ/lOHr9D2btaa4+NfeUAAAAAbJKNBqHW2ops+LpASXLoM3zmnCTnbMa6AAAAAJgik7rLGAAAAABbPkEIAAAAoDOCEAAAAEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGUEIAAAAoDOCEAAAAKQwRT4AABiDSURBVEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGUEIAAAAoDOCEAAAAEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGUEIAAAAoDOCEAAAAEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGUEIAAAAoDOCEAAAAEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGUEIAAAAoDOCEAAAAEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGUEIAAAAoDOCEAAAAEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6IwgBAAAAdEYQAgAAAOiMIAQAAADQGUEIAAAAoDOCEAAAAEBnBCEAAACAzghCAAAAAJ0RhAAAAAA6s9EgVFWfq6r7quqWCdt2qKpvVNUdo8ftJ7y3pKpWV9XtVXXYVC0cAAAAgE0z5Ayhzyc5/CnbFie5srW2Z5IrR69TVXslOS7J3qPPfLqqZo1ttQAAAABsto0GodbaNUl+8pTNRyVZNnq+LMnRE7Zf0lp7pLX2gySrkxw4prUCAAAAMAabeg2hXVpr9yTJ6HHn0fYXJbl7wn5rRtuepqpOq6qVVbVy3bp1m7gMAAAAACZr3BeVrg1saxvasbV2fmttUWtt0bx588a8DAAAAACeyaYGoXuratckGT3eN9q+JsmLJ+w3P8naTV8eAAAAAOO2qUHosiQnjZ6flOTSCduPq6ptqmr3JHsmuX7zlggAAADAOM3e2A5V9d+TvC7JTlW1JslHkixNsryqTk1yV5Jjk6S1dmtVLU/ynSSPJnlXa+2xKVo7AAAAAJtgo0GotfZHz/DWoc+w/zlJztmcRQEAAAAwdcZ9UWkAAAAAZjhBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOiMIAQAAAHRGEAIAAADojCAEAAAA0BlBCAAAAKAzghAAAABAZwQhAAAAgM4IQgAAAACdEYQAAAAAOjNlQaiqDq+q26tqdVUtnqq/AwAAAMDkTEkQqqpZST6V5IgkeyX5o6raayr+FgAAAACTM1VnCB2YZHVr7futtf9IckmSo6bobwEAAAAwCdVaG/9Bq/4wyeGttT8evT4hyUGttXdP2Oe0JKeNXv5OktvHvpDNt1OSH0/3IrYA5jSMOQ1nVsOY03BmNYw5DWNOw5nVMOY0nFkNY07DmNNwZjXMTJzTf22tzdvQG7On6A/WBrY9qTy11s5Pcv4U/f2xqKqVrbVF072Omc6chjGn4cxqGHMazqyGMadhzGk4sxrGnIYzq2HMaRhzGs6shtnS5jRVPxlbk+TFE17PT7J2iv4WAAAAAJMwVUHoX5PsWVW7V9WvJTkuyWVT9LcAAAAAmIQp+clYa+3Rqnp3kv8nyawkn2ut3ToVf2uKzeiftM0g5jSMOQ1nVsOY03BmNYw5DWNOw5nVMOY0nFkNY07DmNNwZjXMFjWnKbmoNAAAAAAz11T9ZAwAAACAGUoQAgAAAOiMIAQAAADQGUEIAAAAoDOCEAAAAEBnBCGALUBVvbCqzq6qW6vqgapaV1X/b1WdPN1rm0nMaTizGsachjGn4cxqGHMazqyGMadhzGm458OsBKGR58OX+Vwwp2HMaTizGuyiJN9PcliSs5Kcl+SEJK+vqo9O58JmGHMazqyGMadhzGk4sxrGnIYzq2HMaRhzGm6Ln1W11qZ7DTNCVV2a5CtJrkjytiQvSHJJkv89yY9aax+axuXNGOY0jDkNZ1bDVNVNrbV9J7z+19ba/1pVWyX5TmvtpdO4vBnDnIYzq2HMaRhzGs6shjGn4cxqGHMaxpyGez7MyhlCv7Kgtfb51tqa1tq5Sd7SWrsjyTuTHDPNa5tJzGkYcxrOrIZ5qKpelSRV9eYkP0mS1tovk9R0LmyGMafhzGoYcxrGnIYzq2HMaTizGsachjGn4bb4Wc2e7gXMIA9V1ataayue+mVW1RbxZT5HzGkYcxrOrIb50yQXVNVLktyS5JQkqap5ST41nQubYcxpOLMaxpyGMafhzGoYcxrOrIYxp2HMabgtflZ+MjZSVQuTXJDkiS+ztfb/jb7MP2qtnTetC5whzGkYcxrOrAAAAJ57ghDAFq6q3tlau3C61zHTmdNwZjWMOQ1jTsOZ1TDmNJxZDWNOw5jTcFvKrAShAbaUL3O6mdMw5jScWQ1TVXe11v6X6V7HTGdOw5nVMOY0jDkNZ1bDmNNwZjWMOQ1jTsNtKbMShAbYUr7M6WZOw5jTcGb1K1V18zO9leQlrbVtnsv1zFTmNJxZDWNOw5jTcGY1jDkNZ1bDmNMw5jTc82FWLio9spEvc5fnci0zmTkNY07DmdVguyQ5LMm/P2V7JfmX5345M5Y5DWdWw5jTMOY0nFkNY07DmdUw5jSMOQ23xc9KEPqVLf7LfI6Y0zDmNJxZDfPVJC9sra166htV9c3nfjkzljkNZ1bDmNMw5jScWQ1jTsOZ1TDmNIw5DbfFz8pPxkaq6rNJLmytrdjAexe31t4+DcuaccxpGHMazqwAAACee4IQwBakqnZJ8qIkLcna1tq907ykGcmchjOrYcxpGHMazqyGMafhzGoYcxrGnIbbkmclCD3FlvxlPpfMaRhzGs6snl1V7Z/kM0nmJvnRaPP8JPcn+bPW2o3TtbaZxJyGM6thzGkYcxrOrIYxp+HMahhzGsachns+zEoQGnk+fJnPBXMaxpyGM6thqmpVkv/WWrvuKdsPTvJ/tNb2nZ6VzSzmNJxZDWNOw5jTcGY1jDkNZ1bDmNMw5jTc82FWLir9Kxfmmb/MC5PM+C/zOWJOw5jTcGY1zP/f3r3HWlaXZxz/PgM4KgMiTAEFKoaLNr0wClJapVykwnhBYou3pFgIAy0VDcVCLxMp1hoslFAMILYFJVPwgtZpIzMGW6QiQQbpWFB0CCA3ocqtxCoS4O0fa83s3cOcmQf/2L+1z3k+yU4We0Py5rvWYe/zO2utvfXMRgBVdYOkrVsMNFDp5EsrTzp50smXVp508qWVJ5086eSb+lZZEBqZ+p05IenkSSdfWnlWSfoScBlwb//cbsAxwOpmUw1POvnSypNOnnTypZUnnXxp5UknTzr5pr5VLhnrSTof2ION78y7quq9rWYbknTypJMvrXySlgJvpbvXkoD7gH+pqquaDjYw6eRLK086edLJl1aedPKllSedPOnkm/ZWWRAaM+07c1LSyZNOvrSKiIiIiIiYrCwIRURMAUlbAMfT3XB7VVVdP/ba8qr6cLPhBiSdfGnlSSdPOvnSypNOvrTypJMnnXxzodWC1gMMhaQtJJ0o6a8k/eaM15a3mmto0smTTr60sl0MHAQ8DHxM0rljr72tzUiDlE6+tPKkkyedfGnlSSdfWnnSyZNOvqlvlQWhkanfmROSTp508qWVZ/+qendVnQf8OrBI0hckLaS7zC466eRLK086edLJl1aedPKllSedPOnkm/pWWRAamfqdOSHp5EknX1p5nrd+o6qeqqoTgLXAvwOLmk01POnkSytPOnnSyZdWnnTypZUnnTzp5Jv6VlkQGpn6nTkh6eRJJ19aeW6SdMT4E1X1IeBSYPcmEw1TOvnSypNOnnTypZUnnXxp5UknTzr5pr5Vbirdk7QCWFFVq2c8fzxwUVVt1WayYUknTzr50soj6SVV9UDrOYYunXxp5UknTzr50sqTTr608qSTJ518c6FVFoR6c2FnTkI6edLJl1YeSauAFwNfBVYD11XVU02HGqB08qWVJ5086eRLK086+dLKk06edPLNhVZZEOrNhZ05CenkSSdfWvkkPR84GFgKvBa4h67Z6qq6p+Fog5JOvrTypJMnnXxp5UknX1p50smTTr5pb5UFoTHTvjMnJZ086eRLq5+PpJfTNTsC2Lmq9m880iClky+tPOnkSSdfWnnSyZdWnnTypJNv2lplQWgTpm1ntpJOnnTypdXmSdoW2Au4s6oelfS8qnqy9VxDk06+tPKkkyedfGnlSSdfWnnSyZNOvmlslW8Zm0W/M7cHrqiqI4HXNR5pkNLJk06+tNo4SSskLe63Dwe+DXwUWCvp6KG/2UxKOvnSypNOnnTypZUnnXxp5UknTzr55kSrqsqjO0tqBbC43z4cuBf4CnA3cHTr+YbySKd0SqtmnW4Z274e2L3fXgx8q/V8Q3mkU1qlUzoN/ZFW6ZRW6TTkRzrNr1Y5Q2hkn6p6qN8+Aziwqg4D9gWWtxtrcNLJk06+tPIs6M+eAniG7j5L9O22bDbV8KSTL6086eRJJ19aedLJl1aedPKkk2/qW03FkBOyQNK2VfU4M3ampHQaSSdPOvnSynMmcI2kC4CvA5+TtBI4lO4G3NFJJ19aedLJk06+tPKkky+tPOnkSSff1LfKTaV7kt4OnA5cALwC2BNYvzMfrqpTG443GOnkSSdfWvkk7QksA/amW9C/D/hiVX256WADk06+tPKkkyedfGnlSSdfWnnSyZNOvmlvlQWhMdO+MyclnTzp5EuriIiIiIiIycqCUETElOi/veAoYBeggB8AK6tqKk5JnZR08qWVJ5086eRLK086+dLKk06edPJNe6ssCI2Z9p05KenkSSdfWm2epPPozqC6jO4MKoBdgWOA26vq/a1mG5J08qWVJ5086eRLK086+dLKk06edPLNhVZZEOrNhZ05CenkSSdfWnkkrauqvTfyvIB1VbVXg7EGJ518aeVJJ086+dLKk06+tPKkkyedfHOhVb7BZ+SNs+zMzwDrgPxS2kknTzr50srzhKT9q+rGGc+/BniixUADlU6+tPKkkyedfGnlSSdfWnnSyZNOvqlvlQWhkanfmROSTp508qWV51jgQknbMDqTajfgceD3Ww01QOnkSytPOnnSyZdWnnTypZUnnTzp5Jv6VrlkrCdpX+BCYGM786Sq+mar2YYknTzp5Esrj6Qtq+opSTvT3WtJwH1V9WDj0QYlnXxp5UknTzr50sqTTr608qSTJ518c6FVFoR6c2FnTkI6edLJl1YeSTfRLZitBlZX1ffbTjRM6eRLK086edLJl1aedPKllSedPOnkmwutsiDUmws7cxLSyZNOvrTySXoZsBQ4gm7x7DpgFXBtVf2s5WxDkk6+tPKkkyedfGnlSSdfWnnSyZNOvmlvlQWhMdO+MyclnTzp5Eur507SVsCBdM0OBn5UVW9qOtQApZMvrTzp5EknX1p50smXVp508qSTbxpbZUFoFtO4M1tIJ086+dLq5yNpl6q6v/UcQ5dOvrTypNPsJO1YVT/st9PJlFazk7RDVT3cb6eTKa086eRJJ980tMqC0CbkTee5SydPOvnSqiPpRcCfAUcBOwIF/BBYCZxVVY81HG8qSFpVVUtbzzEUkralO6Z2BVZV1eVjr11YVSc1G25A+vubnQE8A3wQOBl4G/Bd4P1V9UDD8QZD0vYznwK+CbyK7vPmI5OfapgkHVFVq/vt7YC/pftmzVuBU6rqv1vONxSSzgLOqaqHJO0HfJbu53Ar4JiqurbpgAMi6WbgC8AVVXVH63mGqj+Ozgbup3v/u4TuZ+92YFlVrW043mBIWgScBvwO3WeEJ4E7gIuq6lMtZxuaufD5fEHrAYZC0lmSFvfb+0m6E/iGpLslHZRfSDuSbpa0XNIeG3s9nTr9MXSNpBWSdpN0taTHJK2RtCSdRiQtkvQhSd+W9D+SfiTpBknvgRxTYz4LPAocXFXbV9UOwCHAY8Dnmk42IJJePctjX2BJ6/kG5lK6X9o/D7xT0uclLexfO6DdWIPzSeA7wL3ANcBPgTcDXwM+3m6swXmIbgFo/eMmukuAb+63Y+QjY9vnAA8AbwHWABc3mWiY3lRVD/XbZwPvqKo9gd+mW0SLkRcD2wHXSLpR0imSXtp6qAG6EPgb4EvA9cDFVbUd8KfARS0HG5h/Au4EDgfOBM4Hfg84VNJHNvUfzkNT//k8Zwj1JN1SVb/ab18DnFZVayTtDVxeVfu1nXAYJN1F98vD24EHgSuAz1TVD5oONjCSbqT7i/J2dG88p1TVlZJeD3y4qn6j6YADImkl8M/AV+iOq62BTwPLgfur6s8bjjcYkr5XVa94rq/NN5KeBq6lW+iY6YCqesGERxosSWurasnYP/8F8EbgSODqqnp1s+EGRNJ/VtWr+u17quoXx177fw3nM0kfAA4D/qSqbumfu6uqXt52suGRdPP6n6+N/BzmmOpJ+i7wK/03kd5QVQeMvbbhc3s865g6EHgX3ZmMt9GdNfSJlvMNxWb+f77htflO0reqap+xf15TVa+RtAD4TlW9suF4gzIXPp/nDKGRrSRt2W+/oKrWAFTVOmDh7P/ZvPNoVX2g/x/oqcBewM392TAnNJ5tSLaqqlVVdQVQVXUl3ca/Ac9vO9rg7F5Vn6yq+6rqXODIqrodOJbuw0x07pZ0mqSd1j8haSdJp9OduRCd24ATq+qQmQ+6MxhiZGH/4Q6Aqvpr4BPAfwA7NJtqeMY/K10247UtJjnIkFXVOcDxwAclnStpG7pT5+PZdpT0x5JOBbaVNL6Anc/mIxcAV0k6FFgt6TxJvyXpTCCX9syiqr7WX/K7C/BRIH+EHHlC0hskHQ2UpKMAJB0EPN12tEH5X0mvA5D0FuARgKp6ho3/wW0+m/rP53nTGcmbznOUN5xNyhuOL286nnfQ/ZJ+raRHJT0CfBXYnu7Mquj8JbO/t508wTmmwb8Ch44/0d8b4FS6+wVEZ2V/PwWqavn6JyXtCXyv2VQD1C/sH013ad3VwAsbjzRUfw9sAywCPgWsv2XBzuQz5wZV9TG6y+tOBN4KvJ7u0p77geMajjZE62Y+UVVPV9Xqqjq2xUAD9Qd073HH0V0OdYikx+guJXtfy8EG5g+Bc/s2p9O3kfQLdL8zx8jUfz7PJWNjJB1M9wOwN7Al3areF4FLquqphqMNhqRPV9U7W88xdJL2obtU7BngFLrj6j10H2KWVdX1DccbFEm/BvwD3c/drcBxVbWuf9N5V1Wd33TAAZH0Srqb+91QVT8ee37DDUpjQ6ddgG+k06ZtotXSqlrVbrJhyTHlGe9E98ePParq1nR6thxTnnTypZVH0i8BLyWdNqnvtAv5zLlZkvanuyJkjaRfpvum5Nuq6qrGo1myIGSQdGxVXdp6jqFLJ086+dJqRNL7gD+iuyRqCd23G63sX9tw74D5Lp18kk4G3ktabVI6efKz58sx5ckx5csx5emPqZPoviUynWaRTj5JZwBL6U4muRrYn+5elocBX+4vxx+0LTf/rwTd3dXzS+nmpZMnnXxpNbIM2Leqfixpd+BKSbtX1d+RS+vGpZPvBNLKkU6e/Oz5ckx5ckz5ckx5lgH7pdNmpZPvd+kWzRbSfeHSrlX1uKSz6c6WzYLQtJD0X7O9BOw0y2vzTjp50smXVrYt1p+yW1Xf7y9xvVLSy8ib87h08qWVJ5086eRLK086+dLKk06edPI9VVVPAz+RdEdVPQ5QVT+V9Ezj2SxZEBrZie7mYo/OeF5A7vcykk6edPKlledBSUuqai1A/1ebNwOXAPnq3ZF08qWVJ5086eRLK086+dLKk06edPI9KemFVfUTYN/1T0p6Ed29ZAcv9xDqSfpH4NKqum4jr11eVe9uMNbgpJMnnXxp5ZG0K91fIR7cyGuvraqvNxhrcNLJl1aedPKkky+tPOnkSytPOnnSySdpYVX9bCPPLwZeUlW3NBjrOcmCUERERERERETEPLOg9QARERERERERETFZWRCKiIiIiIiIiJhnsiAUERERERERETHPZEEoIiIiIiIiImKe+T+T/8xVmmY9MAAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 1440x720 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Get the observation counts by date\n",
"controlCountsByDate = control.groupby('load_date').size().reset_index(name='controlCounts').set_index('load_date')\n",
"testCountsByDate = test.groupby('load_date').size().reset_index(name='testCounts').set_index('load_date')\n",
"allCountsByDate = pd.concat([controlCountsByDate, testCountsByDate], axis=1)\n",
"\n",
"# Rename the index to d/m/YYYY format\n",
"allCountsByDate.index = [d.strftime(\"%d/%m/%Y\") for d in allCountsByDate.index]\n",
"\n",
"#Plot the counts\n",
"ax = allCountsByDate.plot.bar(figsize=(20, 10))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If assignment into the test and control groups was truly random with a 50% probability, we can assume that the experiment split, $X$, follow a binomial distribution:\n",
"$$\n",
"X \\sim Bin(n, p)\n",
"$$\n",
"\n",
"As our n is large, the binomial distribution can be approximated by the normal distribution where, \n",
"$$\n",
"X \\sim N(\\mu=np, \\sigma=\\sqrt{np(1-p)})\n",
"$$\n",
"\n",
"With the normal distribution, we can find the probability of assigning 8495 observations or fewer into the test group, given a 50% probability of assignment "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" n = 18185 \n",
" p = 0.5 \n",
" mu = 9092.5 \n",
" sigma = 67.4258852370512\n"
]
}
],
"source": [
"# Define distribution parameters\n",
"n = nControl + nTest\n",
"p = 0.5\n",
"mu = n*p\n",
"sigma = np.sqrt(n*p*(1-p))\n",
"\n",
"print(' n = ',n,'\\n',\n",
" 'p = ',p, '\\n',\n",
" 'mu = ', mu, '\\n',\n",
" 'sigma = ', sigma\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The chart below shows the pdf of the normal distribution with a mean of 9092.5 and a standard deviation of 67.4. We only show the distribution range for of +/- 3 standard deviations around the mean and observe that the 8495 is more than 3 standard deviations away from the mean. In fact 8495 is more than 8.8 standard deviations away from the mean and the probability of assigning 8495 or fewer transactions to the test group given a probability of assignment of 0.5 is almost 0. \n",
"\n",
"From these results, we can conclude that it is highly unlikely that our control and test group are the result of a 50/50 random assignment process. In particular, we would be concerned that the assignment process is non-random."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAz8AAAFlCAYAAADF31b7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdZ3SVxeL+/WvSCAFCCEkg9N4JLRR7V7AAIiAoFiyIioWjYjt6FCv2jti7IEVFRFHshRZaINTQO4GQRnr2PC/I+T38OQgbSDLZe38/a2Uts/eMuW7XMsmVe+4ZY60VAAAAAPi7INcBAAAAAKAiUH4AAAAABATKDwAAAICAQPkBAAAAEBAoPwAAAAACAuUHAAAAQEAIcR3gWMTExNgmTZq4jgEAAACgklq4cOEea23s4d7zqfLTpEkTJSUluY4BAAAAoJIyxmz6p/dY9gYAAAAgIFB+AAAAAAQEyg8AAACAgED5AQAAABAQKD8AAAAAAoJX5ccY09sYs9oYk2qMue8w7xtjzCul7ycbY7p6M9cYc1vpeynGmGdO/HIAAAAA4PCOutW1MSZY0uuSzpO0VdICY8x0a+2Kg4b1kdSy9KOnpPGSeh5prjHmLEn9JCVYawuMMXFleWEAAAAAcDBv7vz0kJRqrV1vrS2UNFEHSsvB+kn6yB4wV1KUMSb+KHNvlvS0tbZAkqy1u8vgegAAAADgsLwpP/UlbTno862lr3kz5khzW0k6zRgzzxjzmzGm++G+uDFmhDEmyRiTlJaW5kVcAAAAAPhf3pQfc5jXrJdjjjQ3RFItSb0k3SPpC2PM/4y31r5lrU201ibGxsZ6ERcAAAAA/tdRn/nRgbs1DQ/6vIGk7V6OCTvC3K2SpllrraT5xhiPpBhJ3N4BAAAAUOa8KT8LJLU0xjSVtE3SEElXHDJmuqRRxpiJOrDhQaa1docxJu0Ic7+SdLakX40xrXSgKO050QsCAPgnj8dqX26h0nIKtL+gWMUlViUeqxJrFREWosjwEEVWDVVURKiqhAS7jgsAqISOWn6stcXGmFGSZkkKlvSetTbFGDOy9P03Jc2UdKGkVEm5koYfaW7pv/o9Se8ZY5ZLKpR0TeldIABAACsq8WjZtkylbM9S6q5srdmVo/V7crQnp1AlnqP/mDBGqhsZrkbREWpcO0Jt4yPVuWGU2sZHKjyUUgQAgcz4Ut9ITEy0SUlJrmMAAMqQtVbLt2Xpl9W7NW/DXi3alKG8ohJJUvUqIWoRV10t4qqrbmS4YqqHKaZGFdUID1VokFFwkFFQkNH+gmJl5xcrK79Ie7ILtSl9vzbvzdXGvfu1J6dQkhQabNS+Xk2d0SpWZ7aOVUKDKAUHHe7RVACALzPGLLTWJh7uPW+WvQEAUKastUratE8zl+3QDym7tC0jT8ZIbepG6vLuDdWjabQ6NYxSvZrhOsxeOMdkR2aelm7J0JItmZq3Ya9e+XmtXv5praKrhem8tnV0adf66tEkWkEUIQDwe9z5AQBUmLTsAk1btFWTFmzR+j37FRYSpNNaxOiCDnV1bts6iq4WVu4Z0vcX6o+1afp51W79uGKXcgtLVD+qqvp3qacrejZW/aiq5Z4BAFB+jnTnh/IDACh3K3dkacJv6zQjeYeKPVaJjWvp8u4N1adjvKpXcbcIIbewWD+u2KUvF2/T72vSZIzRBe3raPgpTZXYuNYJ33UCAFQ8yg8AwIl56/fqjV/X6bc1aYoIC9bl3Rvqyp6N1CKuhuto/2NbRp4+mrNRn8/brKz8YnVpFKXR57bSaS1jKEEA4EMoPwCACrV8W6bGfb9Kf6zdo9rVwjT8lCa6qlcT1YwIdR3tqHILizV14VaN/3WdtmfmK7FxLf3rvFY6uUWM62gAAC9QfgAAFWJLeq6embVa3yzdrqiIUN16ZgtddVJjn9xiuqC4RF8kbdXrP6dqZ1a+zmodq39f3E7NY6u7jgYAOALKDwCgXOUXlWjCb+v1xq+pMka6/tSmuumM5ooMr/x3eo4mv6hEH83ZqFd/SlVeUYmuPbmJbjunpWpW9f1rAwB/RPkBAJSbX1fv1iPTU7Rxb64uSojXvy9qq/ia/rdj2p6cAj3/w2pNXLBFMdWr6LF+7dW7Q7zrWACAQ1B+AABlLjOvSGO/WaGpi7aqWUw1PdqvvU5rGes6Vrlbvi1T905NVsr2LPVuX1dj+7VXXGS461gAgFKUHwBAmfp19W7dN3WZ0nIKdMuZzTXq7BaqEuJ7z/Ucr+ISj97+Y4NenL1G4SFBeqx/B/XrXN91LACAjlx+gio6DADAd+UXlejfXy3Tte8vUI3wEH15y8m66/zWAVV8JCkkOEg3n9lc399xmlrWqaE7Ji7R6ElLlJ1f5DoaAOAI3J0sBwDwKevScnTrp4u0ame2RpzeTP86r5VP7uJWlprFVtekEb302i+peuWntUralK6XLu+ibo1ruY4GADgM7vwAAI7qq8XbdMmrf2pXVr7ev7a7HriwbcAXn/8KCQ7Snee20uSRJ0mSLp8wR+/9uUG+tKwcAAIF5QcA8I8Kikt0/7RlunPSErWvF6mZd5yms9rEuY5VKXVrHK1vbz/w32fsjBW6c9IS5RYWu44FADgIy94AAIeVll2gkZ8s1MJN+3Tzmc1113mtFBLM38yOJDI8VBOGddP439bpuR9Wa/XObL05rJuaxFRzHQ0AIO78AAAOY/m2TPV77U+lbM/U61d01b2921B8vBQUZHTrWS30wfAe2pmVr0te+1N/rt3jOhYAQJQfAMAhZi7boUFvzpGVNGXkyboogYM8j8cZrWL1zahTVa9mVV37/nx9sWCL60gAEPAoPwAASZK1VhN+W6dbPl2ktvE19PWoU9Shfk3XsXxaw+gITb75JJ3UvLbGTE3WM9+vksfDRggA4ArlBwAgj8fqsRkr9dR3q3RRQrw+u7GX4mqEu47lFyLDQ/Xetd01tEdDvfHrOt0+cbHyi0pcxwKAgMSGBwAQ4AqKS3TXF0s1I3mHhp/SRA9d1E5BQcZ1LL8SGhykJy/tqMa1q+np71ZpT06B3rmmu6pX4ccwAFQk7vwAQADLzi/SdR8s0IzkHbq/Txs9fDHFp7wYYzTyjOZ66fLOWrBxn654e67S9xe6jgUAAYXyAwABKjO3SMPemad569P1wuBOuumM5jKG4lPe+neprwnDumnVzmxdPmGOdmbmu44EAAGD8gMAAWhvToGGvj1XK3ccOIdmQNcGriMFlHPb1dGHw3toe0aeBr75tzbu2e86EgAEBMoPAASY3dn5Gvr2XK1Ly9Hb1yTq3HZ1XEcKSCc1r63PR/TS/oJiXf7WHG2gAAFAuaP8AEAA2ZmZryET5mpLep7eH95dZ7SKdR0poCU0iNLEESepqMRqCAUIAMod5QcAAsTurHwNeWuOdmcX6OPre+jk5jGuI0FS67o19NmNPVVUYjX0rbksgQOAckT5AYAAsDenQFe+M0+7swv04XU9lNgk2nUkHKRN3Uh9ekNPFRSXaAgFCADKDeUHAPxcZm6Rrnp3vjan5+q9a7urW+NariPhMNrGR+qzG3upoLhEQ9+eq637cl1HAgC/Q/kBAD+WU1Csa96fr9TdOXrr6kT1albbdSQcQdv4SH16w4FNEK56d7725BS4jgQAfoXyAwB+Kq+wRNd9sEDLtmXqtSu6sLmBj2hXL1LvD++uHZl5uvrd+crMK3IdCQD8BuUHAPxQUYlHIz9ZqKSN6Xrp8s46v31d15FwDLo1jtaEqxK1dne2bvhwgfIKS1xHAgC/QPkBAD9jrdW9U5P125o0PXlpR13SqZ7rSDgOZ7SK1UuXd1HSpn26+dOFKiz2uI4EAD6P8gMAfuaZWas1bdE23XVeKw3p0ch1HJyAixLi9eSlHfXr6jTdPXmpPB7rOhIA+LQQ1wEAAGXn/b82aPyv63Rlz0YadXYL13FQBob2aKSM3CKN+36V4muG6/4L27qOBAA+i/IDAH5iRvJ2jZ2xQue3q6Ox/TrIGOM6EsrIyDOaaXtGnib8vl4NalXVVSc1cR0JAHwS5QcA/MCcdXv1r0lL1a1RLb0ytIuCgyg+/sQYo/9c0k47MvP0n+kpiq9ZVee2q+M6FgD4HJ75AQAfty4tRzd9nKRGtSP0zjWJCg8Ndh0J5SAkOEivDO2iDvVr6rbPF2vplgzXkQDA51B+AMCH7dtfqOs/WKDQ4CC9f213RUWEuY6EchQRFqJ3r+mu2tXDdP2HC7QlPdd1JADwKZQfAPBRhcUHzvLZnpmvt67upobREa4joQLE1qiiD4Z3V1GJ1bXvcwgqABwLyg8A+CBrrR78cpnmbUjXswMT1K1xtOtIqEAt4mpowlXdtGlvrm7/fLFK2AIbALxC+QEAHzTh9/WavHCrbj+npfp1ru86Dhzo1ay2xvbroN/WpOmpmStdxwEAn+BV+THG9DbGrDbGpBpj7jvM+8YY80rp+8nGmK5Hm2uMecQYs80Ys6T048KyuSQA8G+zUnZq3PerdEmnehp9bkvXceDQFT0b6dqTm+idPzfoiwVbXMcBgErvqOXHGBMs6XVJfSS1kzTUGNPukGF9JLUs/RghabyXc1+01nYu/Zh5ohcDAP5uza5sjZ60RAkNovTswATO8oH+fVFbndoiRg9+tUxJG9NdxwGASs2bOz89JKVaa9dbawslTZTU75Ax/SR9ZA+YKynKGBPv5VwAgBcyc4s04qMkVasSoreu6saW1pB0YAvs16/oqga1InTTxwu1dR87wAHAP/Gm/NSXdPC99K2lr3kz5mhzR5Uuk3vPGFPL69QAEGBKPFZ3TFqsbRl5enNYV9WJDHcdCZVIzYhQvX11ogpLPLrxo4XKKyxxHQkAKiVvys/h1lQcuq3MP4050tzxkppL6ixph6TnD/vFjRlhjEkyxiSlpaV5ERcA/M8LP67Wr6vT9Ejf9uzshsNqEVddrwztolU7s/TAl8tkLTvAAcChvCk/WyU1POjzBpK2eznmH+daa3dZa0ustR5Jb+vAErn/Ya19y1qbaK1NjI2N9SIuAPiX75bt0Ou/rNPQHg11Zc/GruOgEjurdZxGn9tKXy7epo/mbHIdBwAqHW/KzwJJLY0xTY0xYZKGSJp+yJjpkq4u3fWtl6RMa+2OI80tfSbovy6VtPwErwUA/M7qndm6a/JSdW0UpUf6tncdBz5g1FktdE6bOD02YwUbIADAIY5afqy1xZJGSZolaaWkL6y1KcaYkcaYkaXDZkpaLylVB+7i3HKkuaVznjHGLDPGJEs6S9LosrssAPB9mblFGvHxgQ0Oxg/rpiohbHCAowsKMnrh8s6qX6uqbvl0kXZn57uOBACVhvGlNcGJiYk2KSnJdQwAKHfWWt340UL9tma3Jo7oxXM+OGYrd2Tp0jf+UkL9KH16Y0+FBnOuOYDAYIxZaK1NPNx7fCcEgErord/Xa/bKXXrgwrYUHxyXtvGRGndZguZvTNdTM1e5jgMAlQLlBwAqmfkb0vXMrNW6qGO8rj25ies48GH9OtfXtSc30Xt/bdD0pYfuVQQAgYfyAwCVyJ6cAt32+SI1io7Q05d1lDGHOzEA8N6DF7VVYuNaun9qstan5biOAwBOUX4AoJIo8VjdMXGxMnKL9MaVXVUjPNR1JPiB0OAgvXpFF4WFBOnWzxYrv4gDUAEELsoPAFQSL/+0Vn+l7tVj/TqobXyk6zjwI/E1q+qFyztr5Y4sjZ2xwnUcAHCG8gMAlcBva9L06s9rNbBbAw3u3vDoE4BjdFbrON18ZnN9Nm+zvl6yzXUcAHCC8gMAju3MzNfoSUvUuk4NPdavg+s48GN3nddKiY1r6YFpy3j+B0BAovwAgEMlHqs7Jx14DuP1K7uqahgHmaL8hAQH6ZWhPP8DIHBRfgDAofG/pmru+nQ92re9msdWdx0HAaBeVFW9MPjA8z+P8fwPgABD+QEARxZuSteLs9eqb6d6Gtitges4CCBntYnTyDOa69N5mzUjmfN/AAQOyg8AOJCZW6TbP1+ielHheuLSDpzngwp31/mt1KVRlO6ftkxb9+W6jgMAFYLyAwAVzFqr+6Yla1dWvl4dynk+cCM0OEgvX95F1kqjJy1Rice6jgQA5Y7yAwAV7PP5W/Td8p26+4LW6twwynUcBLBGtSP0eP8OWrBxn177OdV1HAAod5QfAKhAa3Zl69FvUnRayxiNOK2Z6ziA+nepr0u71NfLP61R0sZ013EAoFxRfgCgguQXlei2zxarRniInh/cSUFBPOeDymFsv/ZqUCtCd0xcoqz8ItdxAKDcUH4AoII8NXOlVu/K1nODOimuRrjrOMD/qREeqpeHdNbOrHw9+OVyWcvzPwD8E+UHACrAb2vS9OGcTbrulKY6s3Wc6zjA/+jSqJb+dV4rfbN0u6Yt2uY6DgCUC8oPAJSz9P2FunvyUrWqU11jerd2HQf4RyPPaK6eTaP18NfLtXHPftdxAKDMUX4AoBxZa/XAtGXKyC3US5d3UXhosOtIwD8KDjJ68fLOCgkO0h2Tlqi4xOM6EgCUKcoPAJSjKQu36vuUnbrr/NZqVy/SdRzgqOpFVdWTl3bU0i0Zev2Xda7jAECZovwAQDnZkp6rR79ZoR5No3Uj21rDh1yUEK/+nevplZ/XaumWDNdxAKDMUH4AoByUeKxGT1oiI+mFwZ0UzLbW8DGP9uuguBpVNPqLJcorLHEdBwDKBOUHAMrBm7+tU9KmfXq09PwUwNfUrBqq5wZ10vq0/Rr3/SrXcQCgTFB+AKCMLd+WqRd/XKOLEuJ1aZf6ruMAx+2UFjG67pSm+uDvjfp9TZrrOABwwig/AFCG8otKdOekJYqpXkVP9O8gY1juBt82pndrtYirrnumLFVGbqHrOABwQig/AFCGnv5ulVJ35+i5QZ0UFRHmOg5wwsJDg/XS5Z21N6dQD32d4joOAJwQyg8AlJE/1qbpg783avgpTXRqyxjXcYAy06F+Td15bkt9s3S7vl6yzXUcADhulB8AKANZ+UUaMyVZzWOr6d7ebVzHAcrcyDOaq2ujKD301XLtyMxzHQcAjgvlBwDKwGPfrNCurHw9P7izwkODXccBylxIcJBeGNxZxR6reyYny1rrOhIAHDPKDwCcoJ9W7tLkhVt185nN1blhlOs4QLlpElNND1zYVn+m7tGn8za7jgMAx4zyAwAnICO3UPdPW6Y2dWvo9nNauo4DlLsrezbSqS1i9OTMldqSnus6DgAcE8oPAJyAR6anKH1/oZ4b1ElVQljuBv9njNG4gQkKMkb3TFkqj4flbwB8B+UHAI7T98t36qsl2zXq7BbqUL+m6zhAhakfVVUPXdxWc9en6+O5m1zHAQCvUX4A4DjszSnQg18uU4f6kbr1rBau4wAVbnBiQ53RKlZPf7dKG/fsdx0HALxC+QGAY2St1UNfL1d2frGeH9RZocF8K0XgMcbo6cs6KiSY5W8AfAc/sQHgGM1I3qGZy3bqzvNaqnXdGq7jAM7E16yqRy5prwUb9+m9vza4jgMAR0X5AYBjsDs7Xw99vVydGkZpxGnNXMcBnBvQtb7ObRunZ2et1rq0HNdxAOCIKD8A4CVrrR6Ytlx5hSV6flAnhbDcDZAxRk9e2lHhocG6e/JSlbD8DUAlxk9uAPDStEXbNHvlLt1zQWu1iKvuOg5QacRFhmtsv/ZavDlD7/yx3nUcAPhHlB8A8MLOzHw98k2KujeppeGnNHUdB6h0+naqpwva19HzP67R2l3ZruMAwGFRfgDgKKy1un9asopLrJ4d2EnBQcZ1JKDSMcboiUs7qnqVEN09eamKSzyuIwHA/6D8AMBRfLVkm35ZnaYxvVurSUw113GASiumehU91q+Dlm7N1ITfWf4GoPLxqvwYY3obY1YbY1KNMfcd5n1jjHml9P1kY0zXY5h7tzHGGmNiTuxSAKDs7c7O1yPTV6hb41q65qQmruMAld5FCfG6sGNdvfzTWqXuZvkbgMrlqOXHGBMs6XVJfSS1kzTUGNPukGF9JLUs/Rghabw3c40xDSWdJ2nzCV8JAJSD/3ydoryiEo27LEFBLHcDvPJo3w6KCAvWmCnJ7P4GoFLx5s5PD0mp1tr11tpCSRMl9TtkTD9JH9kD5kqKMsbEezH3RUljJPGdEUClM3PZDn23fKdGn9uK3d2AYxBbo4oeuaS9Fm3O0Ad/b3QdBwD+jzflp76kLQd9vrX0NW/G/ONcY0xfSdustUuPMTMAlLv0/YV6+Ovl6li/pm48jd3dgGPVr3M9ndMmTs/OWqVNe/e7jgMAkrwrP4db53HonZp/GnPY140xEZIelPTwUb+4MSOMMUnGmKS0tLSjhgWAsjD2mxRl5hXp2UEJHGYKHIf/7v4WGhSke6cmy8PyNwCVgDc/0bdKanjQ5w0kbfdyzD+93lxSU0lLjTEbS19fZIype+gXt9a+Za1NtNYmxsbGehEXAE7M7BW79NWS7br1rBZqUzfSdRzAZ9WtGa5/X9xWc9en67P5PN4LwD1vys8CSS2NMU2NMWGShkiafsiY6ZKuLt31rZekTGvtjn+aa61dZq2Ns9Y2sdY20YGS1NVau7OsLgwAjkdmXpEe/GqZ2tStoVvObOE6DuDzBic21KktYvTUzJXaui/XdRwAAe6o5cdaWyxplKRZklZK+sJam2KMGWmMGVk6bKak9ZJSJb0t6ZYjzS3zqwCAMvLktyu1J6dQzw7spLAQlrsBJ8oYo6cGdJSVdP+0ZbKW5W8A3AnxZpC1dqYOFJyDX3vzoH+2km71du5hxjTxJgcAlKc/1qZpUtIWjTyjuTo2qOk6DuA3GkZH6L4+bfTw1ymavHCrBic2PPokACgH/FkTACTtLyjWfVOXqVlsNd15bkvXcQC/M6xnY/VoEq3HZqzQrqx813EABCjKDwBIeub7VdqemadnLktQeGiw6ziA3wkKMho3MEGFxR49+CXL3wC4QfkBEPDmb0jXh3M26dqTmyixSbTrOIDfahpTTXef31qzV+7W9KWHbhwLAOWP8gMgoOUVlmjMlKVqGF1V91zQ2nUcwO9dd2pTdWoYpUemp2hPToHrOAACDOUHQEB7cfYabdybq3EDEhQR5tUeMABOQHCQ0bMDE7S/oET/mc4GsAAqFuUHQMBavHmf3vljva7o2Ugnt4hxHQcIGK3q1NDt57TQt8k79P3yHa7jAAgglB8AAamguERjpiSrTmS47u/TxnUcIODcdEZztYuP1L+/StG+/YWu4wAIEJQfAAHptZ9TtXZ3jp4c0FE1wkNdxwECTmhwkJ4dlKCM3EI9NmOF6zgAAgTlB0DASdmeqTd+XacBXevrrNZxruMAAat9vZq6+czmmrZ4m35etct1HAABgPIDIKAUlXg0ZkqyakWE6eGL27mOAwS8UWe3UMu46npg2nJl5Re5jgPAz1F+AASUt35fr5TtWXq8fwdFRYS5jgMEvCohwXpmYIJ2Z+frqZmrXMcB4OcoPwACxtpd2Xp59lpdlBCv3h3quo4DoFSXRrV0w2nN9Pn8zfordY/rOAD8GOUHQEAo8VjdMyVZ1aoE69G+7V3HAXCI0ee2UpPaEbpvWrJyC4tdxwHgpyg/AALC+39t0JItGXqkb3vFVK/iOg6AQ1QNC9a4yxK0JT1Pz85a7ToOAD9F+QHg9zbu2a9nZ63WuW3j1LdTPddxAPyDns1q6+qTGuuDvzcqaWO66zgA/BDlB4Bf83is7p2arLCQID3ev6OMMa4jATiCMb3bqF7NqhozNVn5RSWu4wDwM5QfAH7t0/mbNW9Duv59UVvVrRnuOg6Ao6heJURPDeio9Wn79fJPa13HAeBnKD8A/Na2jDw9PXOlTm0Ro8GJDV3HAeCl01vFanBiA731+3olb81wHQeAH6H8APBL1lo9MG2ZrKSnBrDcDfA1D17UTrWrhWnMlGQVFntcxwHgJyg/APzS1EXb9NuaNI25oLUaRke4jgPgGNWsGqonLu2oVTuzNf7Xda7jAPATlB8Afmd3dr4em7FCiY1r6eqTmriOA+A4ndeujvp2qqfXflmrVTuzXMcB4AcoPwD8irVWD321XHlFJRo3MEFBQSx3A3zZI33bKzI8VGOmJKu4hOVvAE4M5QeAX5m5bKdmpezS6HNbqXlsdddxAJyg6GpheqRveyVvzdS7f25wHQeAj6P8APAb+/YX6j/Tl6tj/Zq68bSmruMAKCMXJ8Tr/HZ19PyPa7QuLcd1HAA+jPIDwG+MnbFCGblFemZggkKC+fYG+AtjjB7v30HhIUG6b2qyPB7rOhIAH8VvBwD8ws+rdunLxdt0y1kt1DY+0nUcAGUsLjJcD1/SXgs27tPHcze5jgPAR1F+APi8rPwiPTBtuVrVqa5RZ7VwHQdAObmsa32d0SpW475fpS3pua7jAPBBlB8APu+pmau0OztfzwzspLAQvq0B/soYoycHdJSRdP+0ZbKW5W8Ajg2/JQDwaX+n7tHn8zfrhtOaqXPDKNdxAJSz+lFVdf+FbfVn6h59kbTFdRwAPobyA8Bn5RYW695pyWpSO0Kjz23lOg6ACnJFj0bq1Sxaj89YqZ2Z+a7jAPAhlB8APuu5WWu0JT1P4y5LUNWwYNdxAFSQoCCjcZclqMjj0YNfsvwNgPcoPwB80sJN+/T+3xt0Va/G6tmstus4ACpY49rVdPf5rfXTqt2avnS76zgAfATlB4DPyS8q0ZgpS1WvZlXd26eN6zgAHBl+SlN1aRSl/0xPUVp2ges4AHwA5QeAz3n157Val7ZfTw7oqOpVQlzHAeBIcJDRswMTlFtQokemp7iOA8AHUH4A+JTl2zL15m/rNbBbA53RKtZ1HACOtYiroTvObalvl+3Q98t3uI4DoJKj/ADwGUUlHo2ZkqzoamF66KJ2ruMAqCRGnN5M7etF6t9fpSgjt9B1HACVGOUHgM+Y8Ns6rdiRpcf7d1DNiFDXcQBUEqHBQXpmYIIycgs1dsYK13EAVGKUHwA+YfXObL3801pdlBCvC9rXdR0HQCXTvl5N3Xxmc01btE2/rN7tOg6ASoryA6DSKyrx6O7JSxUZHqqxfdu7jgOgkhp1dgu1jKuuB6YtU3Z+kes4ACohyg+ASm/Cb0RrawgAACAASURBVOu0bFumHu/fQbWrV3EdB0AlVSUkWM8MTNCurHw99d0q13EAVEKUHwCV2qqdWXr5p7W6OCFefTrGu44DoJLr0qiWrj+1qT6bt1l/r9vjOg6ASobyA6DSOni526MsdwPgpX+d11pNakfovqnLlFtY7DoOgErEq/JjjOltjFltjEk1xtx3mPeNMeaV0veTjTFdjzbXGPNY6dglxpgfjDH1yuaSAPiLN39dp+XbsljuBuCYVA0L1rjLErQ5PVfPzVrjOg6ASuSo5ccYEyzpdUl9JLWTNNQYc+gBG30ktSz9GCFpvBdzn7XWJlhrO0uaIenhE78cAP5i5Y4svfIzy90AHJ+ezWrr6pMa6/2/NyhpY7rrOAAqCW/u/PSQlGqtXW+tLZQ0UVK/Q8b0k/SRPWCupChjTPyR5lprsw6aX02SPcFrAeAn/rvcrWbVUI3t18F1HAA+6t7ebVQ/qqrunryU5W8AJHlXfupL2nLQ51tLX/NmzBHnGmOeMMZskXSl/uHOjzFmhDEmyRiTlJaW5kVcAL5u/K/rlLL9wHK36GphruMA8FHVqoTo2YGdtHFvrp75frXrOAAqAW/KjznMa4fepfmnMUeca6190FrbUNKnkkYd7otba9+y1iZaaxNjY2O9iAvAl63ckaVXf16rSzrVU+8OLHcDcGJOal5b157cRB/8vZHd3wB4VX62Smp40OcNJG33cow3cyXpM0mXeZEFgB87eLkbu7sBKCv39m6jpjHVNGZKsnIKWP4GBDJvys8CSS2NMU2NMWGShkiafsiY6ZKuLt31rZekTGvtjiPNNca0PGh+X0mcRgYEuDd++e9yt44sdwNQZqqGBeu5QQnanpGnJ75d6ToOAIdCjjbAWltsjBklaZakYEnvWWtTjDEjS99/U9JMSRdKSpWUK2n4keaW/qufNsa0luSRtEnSyDK9MgA+ZcX2A8vd+naqp94d6rqOA8DPdGscrRtPa6YJv69X7w51dUYrltIDgchY6zubrCUmJtqkpCTXMQCUsaISj/q99pd2Zxfox9GnqxZ3fQCUg/yiEl3y6p/Kzi/WrNGnq2bVUNeRAJQDY8xCa23i4d7z6pBTAChPr/+SqhU7svTEpR0oPgDKTXhosJ4f3ElpOQUa+80K13EAOED5AeDUsq2Zeu3nVPXrXE8XtGe5G4DyldAgSrec2VxTF23Vjyt2uY4DoIJRfgA4k19UotFfLFHt6mEa25fDTAFUjNvObqm28ZG6f9oy7dtf6DoOgApE+QHgzHOzVit1d46eHdhJNSNYew+gYoSFBOn5QZ2UmVeoh6enHH0CAL9B+QHgxJx1e/XuXxt0Va/GOp1dlwBUsHb1InX72S31zdLtmrlsh+s4ACoI5QdAhcvOL9Ldk5eqcXSE7r+wjes4AALUzWc2V0KDmvr3V8u1J6fAdRwAFYDyA6DCPTZjhXZk5un5wZ0VEXbU48YAoFyEBB9Y/pZTUKwHv1wmXzr+A8DxofwAqFCzV+zSF0lbNfKM5urWuJbrOAACXMs6NXTXea00K2WXpi7a5joOgHJG+QFQYfbmFOi+aclqGx+pO89t5ToOAEiSbjitmXo0idYj01O0JT3XdRwA5YjyA6BCWGv14JfLlZVXrBcGd1JYCN9+AFQOwUFGzw/uJEm664ulKvGw/A3wV/z2AaBCfLVkm75P2al/nd9KbeMjXccBgP9Hw+gIPdq3veZvTNdbv693HQdAOaH8ACh32zPy9PDXKUpsXEs3ntbMdRwAOKwBXevrwo519cKPq7V8W6brOADKAeUHQLnyeKzumXJgGcnzgzspOMi4jgQAh2WM0RP9O6pWRJhGT1qi/KIS15EAlDHKD4By9eGcjforda8evKitGteu5joOABxRrWphenZQJ63dnaNx369yHQdAGaP8ACg3q3Zm6anvVumcNnG6okcj13EAwCtntIrVNSc11vt/bdQfa9NcxwFQhig/AMpFflGJ7vh8iSLDQzVuYIKMYbkbAN9xX5+2ah5bTXdPXqqM3ELXcQCUEcoPgHIx7vtVWr0rW88NSlBM9Squ4wDAMakaFqyXh3TR3pxCPfjVclnL9teAP6D8AChzv67erff/2qhrT26iM1vHuY4DAMelQ/2aGn1eK32bvENfL9nuOg6AMkD5AVCm9uYU6O7JyWpVp7ru69PGdRwAOCEjz2iubo1r6aGvl2tbRp7rOABOEOUHQJmx1ureqcnKyivSy0O6KDw02HUkADghwUFGLw7uLI/HavSkJSrxsPwN8GWUHwBl5rP5mzV75W7d26eN2sZHuo4DAGWiUe0Ije3XQfM3pOv1X1JdxwFwAig/AMpE6u4cPTZjhU5rGaPhJzdxHQcAytSArvXVv3M9vTR7jZI2pruOA+A4UX4AnLDCYo/umLhYVUOD9fygTgoKYltrAP7FGKPH+ndQg1oRumPiEmXmFbmOBOA4UH4AnLDnf1ytlO1ZGndZguIiw13HAYByUSM8VC8P6axdWfl6YNoytr8GfBDlB8AJ+XPtHr31+3oN7dFI57ev6zoOAJSrLo1q6a7zW+vbZTv0RdIW13EAHCPKD4DjlpZdoDsnLVGL2Op6+OJ2ruMAQIW46fRmOrVFjB6ZvkKpu7NdxwFwDCg/AI6Lx2P1ry+WKDu/SK9d0VVVw9jWGkBgCAoyemFwJ1UNC9Ztny9RflGJ60gAvET5AXBcxv+2Tn+s3aNH+rZX67o1XMcBgAoVFxmu5wYlaOWOLI37fpXrOAC8RPkBcMySNqbrhR/X6OKEeA3p3tB1HABw4uw2dTT8lCZ6/6+N+nnVLtdxAHiB8gPgmGTkFur2zxerflRVPTWgo4xhW2sAgeu+Pm3ULj5Sd09O1q6sfNdxABwF5QeA16y1GjMlWWk5BXrtii6qER7qOhIAOFUlJFivDO2ivMIS3TFxsYpLPK4jATgCyg8Ar33490b9sGKX7u3dRgkNolzHAYBKoUVcdT1xaQfNXZ+ul39a6zoOgCOg/ADwyvJtmXpy5iqd3SZO15/a1HUcAKhUBnRtoMsTG+q1X1L125o013EA/APKD4Cjyiko1m2fL1Z0tTA9N6gTz/kAwGE82q+9WtepodGTlmhHZp7rOAAOg/ID4Iistbp3arI27d2vl4Z0VnS1MNeRAKBSCg8N1utXdlVBUYlu+2yxinj+B6h0KD8AjujDvzfq2+QduueCNurVrLbrOABQqTWPra6nLktQ0qZ9eu6H1a7jADgE5QfAP1q0eZ+emLlS57aN002nN3MdBwB8Qt9O9TSsVyNN+G29Zq/g/B+gMqH8ADis9P2FuvXTRapbM1zPD+qsoCCe8wEAb/37onZqXy9Sd01eqq37cl3HAVCK8gPgf5R4rO6YuFh79xdq/JXdVDOC83wA4FiEhwbrjSu7yuOxuvWzxSos5vkfoDKg/AD4H6/+vFZ/rN2jRy5prw71a7qOAwA+qXHtanpmYIKWbsnQkzNXuo4DQJQfAIf4bU2aXv5prQZ0ra+hPRq6jgMAPq1Px3hdf2pTffD3Rn25eKvrOEDAo/wA+D/bM/J058TFahVXQ4/378B5PgBQBu7r00Y9m0br/mnLlLI903UcIKB5VX6MMb2NMauNManGmPsO874xxrxS+n6yMabr0eYaY541xqwqHf+lMSaqbC4JwPHILyrRzZ8uUmGxR28M66qIsBDXkQDAL4QGB+m1K7oqqmqYRn6yUBm5ha4jAQHrqOXHGBMs6XVJfSS1kzTUGNPukGF9JLUs/RghabwXc3+U1MFamyBpjaT7T/hqABwXa60e+mq5lm7J0PODO6t5bHXXkQDAr8TWqKLxw7pqV2aB7pi4RCUe6zoSEJC8ufPTQ1KqtXa9tbZQ0kRJ/Q4Z00/SR/aAuZKijDHxR5prrf3BWltcOn+upAZlcD0AjsMnczdp8sKtuu3sFurdoa7rOADgl7o0qqX/9G134NnK2WtcxwECkjflp76kLQd9vrX0NW/GeDNXkq6T9N3hvrgxZoQxJskYk5SWluZFXADHYv6GdD36zQqd3SZOo89t5ToOAPi1K3o00uDEBnrl51T9yAGoQIXzpvwc7onnQ+/V/tOYo841xjwoqVjSp4f74tbat6y1idbaxNjYWC/iAvDWjsw83fLpQjWMjtCLl3OQKQCUN2OMxvbroIQGNfWvSUu0Pi3HdSQgoHhTfrZKOni/2waStns55ohzjTHXSLpY0pXWWha/AhUov6hEIz9ZpLzCEr11VTfVrMpBpgBQEcJDgzV+WDeFhgTppo8XKju/yHUkIGB4U34WSGppjGlqjAmTNETS9EPGTJd0demub70kZVprdxxprjGmt6R7JfW11uaW0fUA8IK1Vg9//f9vcNCyTg3XkQAgoNSPqqrXhnbR+j37NXoSGyAAFeWo5ad0U4JRkmZJWinpC2ttijFmpDFmZOmwmZLWS0qV9LakW440t3TOa5JqSPrRGLPEGPNm2V0WgCP5ZN5mfZHEBgcA4NLJLWL08MXtNHvlbj33w2rXcYCA4NVBHtbamTpQcA5+7c2D/tlKutXbuaWvtzimpADKxN/r9ujR6SlscAAAlcDVJzXWqp3ZGv/rOrWuU0P9uxxuXygAZcWrQ04B+IcNe/br5k8WqUlMNb00hA0OAMA1Y4we7dtePZpGa8zUZC3ZkuE6EuDXKD9AgMjMK9L1Hy5QkJHevSZRkeFscAAAlUFYSJDeHNZNcTWqaMRHSdqZme86EuC3KD9AACgu8WjUZ4u0JT1Xbw7rpsa1q7mOBAA4SHS1ML1zTaL2FxRrxMdJyi8qcR0J8EuUHyAAjJ2xQn+s3aMn+ndUz2a1XccBABxGm7qRevHyzlq2LVP3TEkWp4AAZY/yA/i5j+Zs1EdzNmnE6c00uHvDo44HALhzfvu6uvv81vpm6Xa98lOq6ziA3/FqtzcAvun3NWl69JsVOrdtnO7t3cZ1HACAF245s7nWpeXoxdlr1Kh2VV3apYHrSIDfoPwAfip1d7Zu/WyRWsZV10tDuiiYnd0AwCcYY/T0gARtz8jTmCnJiq9ZVb1YsgyUCZa9AX5od3a+rnlvgaqEBOmdaxJVvQp/5wAAXxIWEqQJwxLVKDpCIz5KUuruHNeRAL9A+QH8zP6CYl33wQKl7y/Ue9d2V4NaEa4jAQCOQ82IUH0wvIfCQoI0/IP52pNT4DoS4PMoP4Af+e+W1iu2Z+m1K7oooUGU60gAgBPQMDpC71zTXWnZBbrhQ7bABk4U5QfwE9ZaPfR1in5ZnabH+nfQOW3ruI4EACgDnRtG6aXLu2jp1gyNnrREHg9bYAPHi/ID+Ik3fl2nz+dv1s1nNteVPRu7jgMAKEO9O9TVgxe21XfLd+rJmStdxwF8Fk9BA37gq8Xb9Oys1erXuZ7uOb+16zgAgHJw/alNtXVfnt75c4PiIqtoxOnNXUcCfA7lB/Bxf6xN0z1TlqpXs2g9MzBBQWxpDQB+yRijhy5up7ScAj05c5VqV6uiy7pxBhBwLCg/gA9bvHmfbvp4oZrHVteEqxJVJSTYdSQAQDkKDjJ6YXAnZeQWaszUZNWqFqqz2/CMJ+AtnvkBfNTaXdka/sECxVSvoo+u66GaVUNdRwIAVIAqIcGacFWi2sVH6pZPF2nhpn2uIwE+g/ID+KCt+3J11bvzFRocpE+u76m4yHDXkQAAFah6lRC9P7y76kaG67oPFmjtrmzXkQCfQPkBfMzenAJd/e587S8s1kfX9VCj2hxiCgCBKKZ6FX18fU+FhQTp6vfma1tGnutIQKVH+QF8SHZ+ka59f4G2ZeTpvWu7q218pOtIAACHGkZH6MPhPZRTUKxh78zT7ux815GASo3yA/iIvMIS3fhRklbsyNL4YV3VvUm060gAgEqgXb1IfTC8u3Zl5WvYO/OUvr/QdSSg0qL8AD4gv6hEIz5O0rwN6Xp+UCd29gEA/D+6NY7WO1cnauPeXF393jxl5Re5jgRUSpQfoJIrLPZo1GeL9MfaPRo3IEH9u9R3HQkAUAmd3CJGE4Z10+qd2Rr+/gLtLyh2HQmodCg/QCVWXOLRHRMXa/bK3XqsfwcN7t7QdSQAQCV2Vps4vTKkixZv3qcbP0pSflGJ60hApUL5ASqpEo/VXZOX6rvlO/Xvi9rqql6NXUcCAPiAPh3j9fzgTpqzfq9u/mShCoopQMB/UX6ASsjjsbpvarK+XrJdY3q31g2nNXMdCQDgQy7t0kBP9O+oX1an6ZZPFlGAgFKUH6CS8XisHvhymSYv3Ko7zmmpW85s4ToSAMAHXdGzkR7v30E/rdqtkR8vZAkcIMoPUKmUeKzumZKsiQu2aNRZLXTnuS1dRwIA+LBhvRrryUsP3AG6iQIEUH6AyqK4xKN/fbFEUxdt1ehzW+nuC1rLGOM6FgDAx13Rs5GeHtBRv61JYxMEBDzKD1AJFJV4dMekJfp6yXbdc0Fr3cEdHwBAGRrSo5GeuSxBf6buoQAhoFF+AMf+e47Pt8k79OCFbXXrWTzjAwAoe4O7N9S40gJ03QecA4TARPkBHMovKtEtny7UrJRdeuSSdrrxdHZ1AwCUn8GJDfX8oE6au36vhr07Txm5ha4jARWK8gM4kp1fpOHvL9Dslbv1eP8OuvaUpq4jAQACwICuDfTGld2Usi1LQ96aq93Z+a4jARWG8gM4sDenQFe8PU8LNqbrpcs7axgHmAIAKlDvDnX13rXdtWlvrga/OUdb9+W6jgRUCMoPUMG2ZeRp0IQ5WrMrW29d3U39u9R3HQkAEIBObRmjT27oqfT9hRr05hyl7s5xHQkod5QfoAKl7s7RwPF/Ky27QJ/c0FNnt6njOhIAIIB1a1xLE0ecpKISjwZPmKNlWzNdRwLKFeUHqCDJWzM06M2/VVRiNWnESereJNp1JAAA1K5epCaPPFlVQ4N1+Vtz9Ovq3a4jAeWG8gNUgJ9X7dKQt+aqWpUQTRl5ktrVi3QdCQCA/9M0ppqm3XKymtSupus/TNIXC7a4jgSUC8oPUM4+nrtJN3yYpGax1TTt5pPVJKaa60gAAPyPOpHhmnRTL53cvLbGTE3WS7PXyFrrOhZQpig/QDnxeKyemrlSD321XGe2jtOkEScpLjLcdSwAAP5RjfBQvXdtd13WtYFemr1W905NVlGJx3UsoMyEuA4A+KP8ohLdNXmpvk3eoat6NdZ/LmmnkGD+1gAAqPxCg4P03KAE1Y8K1ys/p2pXVoFeu6KLaoSHuo4GnDB+GwPK2L79hRr2zjx9m7xDD1zYRmP7taf4AAB8ijFG/zq/tZ4a0FF/pu7RwPFztCWds4Dg+7z6jcwY09sYs9oYk2qMue8w7xtjzCul7ycbY7oeba4xZpAxJsUY4zHGJJbN5QBurdqZpb6v/6nkbZl6/YquGnF6cxljXMcCAOC4DO3RSB8O76EdmXnq9/pfmr8h3XUk4IQctfwYY4IlvS6pj6R2koYaY9odMqyPpJalHyMkjfdi7nJJAyT9fuKXAbg3K2WnBrzxtwqKPJo0opcuSoh3HQkAgBN2assYfXXrKYqKCNWV78zVpAWbXUcCjps3d356SEq11q631hZKmiip3yFj+kn6yB4wV1KUMSb+SHOttSuttavL7EoARzweq5dnr9VNHy9Uyzo19M1tp6pLo1quYwEAUGaaxVbXl7ecol7Nauveqcv02IwVKvGwExx8jzflp76kgzd731r6mjdjvJkL+Kz9BcW69bNFenH2Gg3oUl+TRvRSHXZ0AwD4oZpVQ/X+td117clN9O6fGzT8gwXKyC10HQs4Jt6Un8M9sHBo1f+nMd7MPfIXN2aEMSbJGJOUlpZ2LFOBcrVp735dNv5vzUrZqQcvbKvnB3dSeGiw61gAAJSbkOAgPdK3vZ4a0FFz1+3Vxa/+qeXbMl3HArzmTfnZKqnhQZ83kLTdyzHezD0ia+1b1tpEa21ibGzssUwFys2slJ26+NU/tT0jT+9d2103nt6MjQ0AAAFjaI9G+mLkSfJ4rAaM/5vngOAzvCk/CyS1NMY0NcaESRoiafohY6ZLurp017dekjKttTu8nAv4jKISj56cuVI3fbxQTWpX07e3n6YzW8e5jgUAQIXr3DBKM24/TT2bRuveqct075Rk5ReVuI4FHNFRDzm11hYbY0ZJmiUpWNJ71toUY8zI0vfflDRT0oWSUiXlShp+pLmSZIy5VNKrkmIlfWuMWWKtvaCsLxAoK7uy8jXqs0VasHGfhvVqpIcubqcqISxzAwAEruhqYfpgeA+9NHuNXv05VSk7MjX+ym5qGB3hOhpwWMZa39mpIzEx0SYlJbmOgQD0V+oe3TFxsfYXlOjpyzqqX2f27QAA4GA/rdyl0ZOWyFrpyQEddUmneq4jIUAZYxZaaw97jijHzgNHUFjs0dPfrdKwd+epZtVQTR91CsUHAIDDOKdtHX17+2lqUae6bvt8scZMWarcwmLXsYD/x1GXvQGBan1aju6YuETLtmVqSPeGeujidqpWhf9lAAD4Jw2jI/TFTSfp5dlr9fqvqUratE+vDOmiDvVruo4GSOLOD/A/rLWatGCzLnrlT21Oz9X4K7vq6csSKD4AAHghNDhId1/QWp/e0FP7C4o14I2/9d6fG+ThUFRUApQf4CAZuYW65dNFunfqMnVuGKXv7zxNfTrGu44FAIDPObl5jL6743Sd3ipGY2es0FXvzdO2jDzXsRDgKD9AqR9Sduq8F3/Xjyt26b4+bfTJDT0VX7Oq61gAAPis6GphevvqRD1xaQct3pyhC178XV8s2CJf2nAL/oXyg4C3b3+h7pi4WCM+XqiY6lX01a2naOQZzRUcxKGlAACcKGOMruzZWLPuPF3t60VqzNRkXffBAu3KyncdDQGIra4R0Gal7NSDXy5XRm6hRp3dQrec2UJhIfxNAACA8uDxWH04Z6PGfb9KVUKC9Ujfdurfub6M4Q+OKDtH2uqa8oOAtCenQGO/WaHpS7erXXyknhvUSe3qRbqOBQBAQFiflqO7Jy/Vos0ZOr1VrB7v10GNanMwKsoG5Qco5fFYTVywReO+X6XcwmLddnZL3Xxmc4UGc7cHAICKVOKx+mTuJj07a7WKPR7dcU4r3XBaU34m44RRfgBJK7Zn6d9fLdOizRnq1Sxaj/fvoBZxNVzHAgAgoO3IzNMj01M0K2WX2tStoacGdFSXRrVcx4IPo/wgoO0vKNZLs9fovb82KqpqqB68qK0u7cL6YgAAKpMfUnbqP9NTtDMrX1f0aKS7zm+t6GphrmPBBx2p/HBqI/yWx2M1fel2Pf3dKu3MytfQHo10b+/WiorgGykAAJXN+e3r6uQWMXrhhzX6cM5GfbN0u/51Xitd2asxS+FQZrjzA7+0aPM+jf1mhZZsyVDH+jX1SN926tY42nUsAADghTW7sjX2mxX6M3WPWsZV138uaa9TW8a4jgUfwbI3BIztGXka9/0qfb1ku+JqVNGY3m00oEt9BXFmDwAAPsVaqx9W7NLj367QlvQ8nd+uju6/sK2axlRzHQ2VHOUHfi8zr0hv/b5O7/65QdZKI05vppFnNFe1KqzsBADAl+UXlejdPzfo9V9SVVDs0ZDuDXXHOS0VFxnuOhoqKcoP/FZeYYk++Huj3vxtnTLzinRJp3q6t3drNajFWQEAAPiTtOwCvfrzWn02b7NCg4N0/alNNeKMZooMD3UdDZUM5Qd+p7DYo0kLNuuVn1OVll2gs1rH6u4LWqt9vZquowEAgHK0cc9+Pf/jGn2zdLtqRYTq1rNa6MqejVU1LNh1NFQSlB/4jcJij75avE2v/rJWW9Lz1L1JLd1zQRv1aMpmBgAABJJlWzP1zKxV+mPtHsVUr6KbTm+mK3s1UkQYS94DHeUHPi+/qESTk7bozd/Wa1tGnjrUj9Rd57fWma1iOa8HAIAANm/9Xr3y81r9lbpXtauFacTpzTSsV2Oe+w1glB/4rNzCYn02b7Mm/L7+/2vvXqOrqs88jn+fXMiN3K/kRrgGBBQFFKHqKFNanI60TrV17FRr11xeONOZV7XLWZ01q6traWfetJ0Xrq52urRjte1ULVZneUWrInITuQghQAgkgdxJThJyCMl/XuwNHjEnZEMOJ8n5fdbaK/vs89/HfX48YJ699/kf2kNhVszO5+E75qvpERERkU/ZcayLH79Rzzv1HRRkzeChtTV8Y/Vsfb9fAlLzI1NOW+8gT73fyNMfNNI9MMSaeYU8fMd8bp5bqKZHREREotrZ2M1P36znrbp2MlKT+dqqKr79uTlUFWgypESh5kemjI9bevnFuw1s+qiZcyOOzy8u5e9vm6svKBUREZFADpzs5efveL9TDI84Niybxd/eMpflVXnxPjSJMTU/MqmdGx5hc107v3yvgS1HOslITebelZV8a+0cavRFZiIiInIFTvUM8sstDfx663FC4XNcX53H36yezZ3LZpGeqhnipiM1PzIpneoZ5Nntx/nN9hOc7BmkLCedB9fWcN+qanIzNWe/iIiITJzQ4BC/29HE/2xt5GhHP/mZqdy7qor7b5xNdaFuiZtO1PzIpDEy4vhTfTtPf3CcNw+2MTziuGVBEfffNJt1i0tITU6K9yGKiIjINOacY8uRTn71fiOvHWhlxDluW1jMX99Yze2L9LvIdKDmR+KuoaOf53c18dyHzTR1n6Ewawb3rqrivlXVOtsiIiIicXGy5wzPbDvBM9uO0x4KU5g1g43LK/jqikquKc+J9+HJZVLzI3HR3X+WP+5p4bkPm/nw+GmSDNbOL+Jrq6pYf00ZM1J0ZkVERETib2h4hLfr2vn9riZeP9DK0LBj8awcvrqiko3LyymamRbvQ5QA1PzIVdMfPsfmujY27W5hc10bQ8OORWXZ3H1DBRuXV1Cakx7vQxQRERGJqrv/LC/uaeF/dzaxp6mH5CRjzbxCvnTtLNZfU0Z+lr43aLJT8yMx1Ts4iaPntQAADR9JREFUxJsH2nh570nePtRO+NwIxdlpbLyunLtv0GVjERERmZoOtYZ4/sNmXtpzkuNdA6QkGWvnF/EX187iC9eUaYKmSUrNj0y49lCYzXVtvLLvFO/Ud3B2eITSnDQ2LJ3FhqVlrKwpIDlJX0YqIiIiU59zjv0tvfxxz0le2tvCia4zpCYbq+cWsm5RCesWl+pLVCcRNT9yxUZGHPtaenjzYBubD7bxUVMPABV5GWxYWsaGZWVcX5VPkhoeERERmcacc+xt7uGlPSd57UArR9v7AagtzWbdYq8RWl6Vp5PAcaTmRy5LeyjMliMdvFvfwea6djr6wpjB8qo87qgt4fZFJSwpz8FMf7lFREQkMTV09PPGgVZeP9DK9mPdDI84CrJmsGZeIbcsKGLt/CIq83VV6GpS8yPj0js4xAdHu3jvcAdbjnRwqLUPgJz0FG5dWMwdi0q4bWExhZrxREREROQzegaGeOtQG2/VtfPe4Q7aQmEA5hRlsXZ+IZ+bX8TNc4v0WaEYU/Mjo2oPhdnZ2M2u49180NDF3qbTjDhIT01iVU0Ba+YVsWZeIUsrcnXpVkRERCQA5xz1bX28W9/Be4c72Hq0k/6zw5h5t8itqilgZU0+q2oKKM/LiPfhTitqfoThEUd9W4idjd3sPNbNzuPdNHYOADAjOYlrK3NZM6+QNfOLuL46j7SU5DgfsYiIiMj0MTQ8wu4Tp9lyuJMdjV3sauym/+ww4H2GelVNPitqClhemUdtWba+D/EKjNX8pFztg5HYOzc8wtGOfvY29bC3uYd9zT18fLKXAf8vWNHMGayYnc/9N1WzYnY+Syty1eyIiIiIxFBqsndnzaqaAsD7fe3gqRDbj3Wx41g37x3p5IXdLYB3YnrRrGyWVeRyXWUeyypzWVAyk5RkNURXSld+prjewSHqW0PUneqj7lQve/1GZ3BoBICM1GSWlOewtCKXZRW5rKzJp7ogU5MUiIiIiEwizjmaus+wp6mHPc2nvZPYTT2EwucA72MJi2flsKgsm9rSbGrLcqgty6ZAX7r6GbrtbRoIDQ5xuK2P+tY+DrWGqGsNUd/ax6newQtjsmYks6Q8lyUVOSzzm525xTP1eR0RERGRKWhkxNHYNcCeptPsaephf0sPdadCdA8MXRhTnJ12oSFaWJrNnOIs5hRlUZg1I2FPdqv5mSJ6B4do7BjgWGc/xzr6OdbprTd29tPRd/bCuLSUJBaUzmRhSTYLSrOpLZvJgpJsKvIy9D07IiIiItOYc472UJiDp0LUnQp5P1t7qW/tI3xu5MK47PQU5hZlMbd4JnOKsi4sVQWZ5GZM79nm9JmfScA5R1f/WVpOD9J8eoDm04O0nD5zYWnqPkNn/9lP7VOWk87swkzWLSqlpiiLucVZ1JZmU1WQqas5IiIiIgnIzCjJSackJ51bFxZf2D484mjuPsPRjj4aOvovLNsaunhhdzOR1zuy01KoyM+gMj+TyvwMKvMzqMj75HFeZuq0vWqk5ucKOefoOTNEWyhMeyhMW2jQ+9kbpr3P23aqZ5Dm02c+1Y2D93mc8rx0yvMyWF+ew+zCLGoKM6kpyqK6IJPMGfrjEREREZFLS04yqgszqS7M5M9qP/3c4NAwjZ0DNHT00dR9xl8GaOoe4IOjnRc+V3ReWkoSpTnplOakUZKdTklO2qcel+akUZKTTnZaypRrkvTb9WXY1tDFD18+QIff8JwdHvnMmPTUJEqy0737MGdls25xCeV5GZTneZ11Rd707qpFREREZHJIT02mtiyb2rLsUZ/vOTPkN0NeY9TaO0hb7yCtvWEOnOrl7UNh+i5qkMBrkopmplE4cwYP3z6f9UvKYv1Wrti4mh8z+yLwYyAZ+Llz7rGLnjf/+TuBAeBB59yusfY1swLgN0ANcAy41znXfeVvKfYyUpPJSU9hXnHWhQanJDuNYn8pyU5j5hTshEVEREQk8eRmpJKbkcuS8tyoY/rD52gLhWntHfSbozAdfWE6+s7S2R8mLXVqfG3KJSc8MLNk4BDweaAJ2A7c55z7OGLMncA/4jU/NwE/ds7dNNa+ZvYjoMs595iZPQLkO+e+O9axTPcJD0RERERE5MqMNeHBeL4p6UbgsHPuqHPuLPAssPGiMRuBp5xnK5BnZrMuse9G4El//Ungy4HelYiIiIiISADjaX4qgBMRj5v8beMZM9a+pc65kwD+z5LR/uNm9ndmtsPMdrS3t4/jcEVERERERD5rPM3PaB9cufheuWhjxrPvmJxzP3POrXTOrSwuLr70DiIiIiIiIqMYT/PTBFRFPK4EWsY5Zqx9W/1b4/B/to3/sEVERERERIIZT/OzHVhgZnPMbAbwdWDTRWM2Ad80z2qgx7+Vbax9NwEP+OsPAH+4wvciIiIiIiIS1SWnunbOnTOzh4FX8Kar/m/n3H4z+wf/+SeAl/FmejuMN9X1t8ba13/px4Dfmtm3gePAPRP6zkRERERERCJccqrryURTXYuIiIiIyFiudKprERERERGRKU/Nj4iIiIiIJAQ1PyIiIiIikhDU/IiIiIiISEJQ8yMiIiIiIglBzY+IiIiIiCSEKTXVtZm1A43xPo4JVgR0xPsgpillG1vKN7aUb2wp39hRtrGlfGNL+cbW1cp3tnOueLQnplTzMx2Z2Y5o85DLlVG2saV8Y0v5xpbyjR1lG1vKN7aUb2xNhnx125uIiIiIiCQENT8iIiIiIpIQ1PzE38/ifQDTmLKNLeUbW8o3tpRv7Cjb2FK+saV8Yyvu+eozPyIiIiIikhB05UdERERERBKCmp8JZmb/Ymb7zWyfmT1jZulmdp2ZvW9me83sRTPLiRj/PTM7bGZ1ZvaFiO0r/PGHzewnZmbxeUeTS5B8zazGzM6Y2W5/eSLidZTvKMzsO362+83sn/1tBWb2mpnV+z/zI8arfgMIkq/qN7go+d7jPx4xs5UXjVf9jlOQbFW7wUXJ9z/M7KCZ7TGz580sL2K8ajeAIPmqfoOLku8P/Gx3m9mrZlYeMT6+9euc0zJBC1ABNAAZ/uPfAg8C24Hb/G0PAT/w168BPgLSgDnAESDZf24bcDNgwP8BG+L9/uK9XEa+NcC+KK+lfD+byVJgH5AJpACvAwuAHwGP+GMeAR7311W/sc1X9Tsx+S4GaoG3gJUR41W/sctWtTsx+a4HUvwxj+vf3quWr+p3YvLNiRjzT8AT/nrc61dXfiZeCpBhZil4hdCC9z+HP/nPvwb8lb++EXjWORd2zjUAh4EbzWwWXtG877xqeAr48tV8E5NYkHxHpXyjWgxsdc4NOOfOAW8DX8Gr0yf9MU/ySVaq32CC5jsq5RvVqPk65w445+pGGa/6Hb+g2Y5K2UYVLd9X/ccAW4FKf121G0zQfEelfKOKlm9vxJgs4PwkA3GvXzU/E8g51wz8J3AcOAn0OOdexeuI7/KH3QNU+esVwImIl2jyt1X46xdvT2iXkS/AHDP70MzeNrNb/G3Kd3T7gFvNrNDMMoE78bIsdc6dBPB/lvjjVb/BBM0XVL9BRMs3GtXv+AXNFlS7QYwn34fwzoSDajeooPmC6jeIqPma2Q/N7ARwP/B9f3zc6zclFi+aqMy7V38j3mW808DvzOwbeH+pfmJm3wc2AWfP7zLKy7gxtie0y8j3JFDtnOs0sxXAC2a2BOU7KufcATN7HO/qWR/eZelzY+yi+g3gMvJV/Qag+o0d1W5sXSpfM3vUf/z0+U2jvcwY2xPaZeSr+g1grHydc48Cj5rZ94CHgX9jEtSvrvxMrD8HGpxz7c65IeA5YI1z7qBzbr1zbgXwDN79jeB1tZFnHyrxbuNq4tOXX89vT3SB8vUvqXb66zv97QtRvlE5537hnLvBOXcr0AXUA63+5ejzl/3b/OGq34CC5Kv6DS5KvtGofgMIkq1qN7ho+ZrZA8CXgPv9W4FAtRtYkHxVv8GN49+HX/PJRxLiXr9qfibWcWC1mWX6M1SsAw6YWQmAmSUB/wqcnzlkE/B1M0szszl4HxDb5t/6EjKz1f7rfBP4w9V+M5NQoHzNrNjMkv31uXj5HlW+0UVkWQ3cjddMbgIe8Ic8wCdZqX4DCpKv6je4KPlGo/oNIEi2qt3gRsvXzL4IfBe4yzk3EDFctRtQkHxVv8FFyXdBxJC7gIP+evzr102CmSKm0wL8u/8HvA/4Fd5sFt8BDvnLY/hfLuuPfxTvrEIdEbNaACv91zgC/FfkPom8BMkX7yzDfrxLsLuAv1S+l8z3HeBjP7N1/rZC4A28MzlvAAUR41W/McpX9Tth+X4F74xiGGgFXokYr/qNQbaq3QnL9zDeZyN2+8sTEeNVuzHKV/U7Yfn+3s9qD/AiUBExPq71e/6XRBERERERkWlNt72JiIiIiEhCUPMjIiIiIiIJQc2PiIiIiIgkBDU/IiIiIiKSENT8iIiIiIhIQlDzIyIiIiIiCUHNj4iIiIiIJAQ1PyIiIiIikhD+H8XZ7g6/c2L6AAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 1008x432 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"def plotNormal(mu, sigma):\n",
" '''\n",
" Plots normal distribution given mu and sigma\n",
" '''\n",
" fig, ax = plt.subplots(figsize=(14, 6))\n",
" x = np.linspace(mu - 3*sigma, mu + 3*sigma, 1000)\n",
" plt.plot(x, stats.norm.pdf(x, mu, sigma))\n",
"\n",
"plotNormal(mu, sigma)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The z score is -8.861581837588803.\n",
"The probability of randomly assigning 8495 or fewer participants to the test group when the probability of assignment is 0.5 is 0.0\n"
]
}
],
"source": [
"zScore = (nTest-mu)/sigma\n",
"testCDF = stats.norm.cdf(zScore)\n",
"print('The z score is {}.'.format(zScore))\n",
"print('The probability of randomly assigning {} or fewer participants to the test group when the probability of assignment is 0.5 is {}'. format(nTest, round(testCDF, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Conversion Rates"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This section summarizes our observed conversion rates for both groups. Both the control and test groups had higher conversions than non-conversions, but the test group had a higher conversion rate on form loads."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEHCAYAAABfkmooAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAR50lEQVR4nO3df+xd9V3H8eeLgozpcDC+YNfiSkyzCMyxUJE5jTqM1F8DlzE7N6kbSQ1BRWNU8A9/18zfGyoYotiy6bBuY3RLcCN1sJjh2LeOrRRsaGBC00oL6AbTYMre/nE/za7fXr6fW+y932/7fT6Sk3vO+34+574v6/rqOefec1NVSJI0nxMWugFJ0uJnWEiSugwLSVKXYSFJ6jIsJEldJy50A5Nyxhln1KpVqxa6DUk6pmzfvv3JqpqZWz9uw2LVqlXMzs4udBuSdExJ8m+j6p6GkiR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXRMNiyRfTLIjyf1JZlvt9CR3JXm4PZ42NP76JLuT7Epy6VD9wraf3UluSJJJ9i1J+r+mcWTxfVV1QVWtadvXAduqajWwrW2T5FxgHXAesBa4McmyNucmYAOwui1rp9C3JKlZiG9wXwZ8b1vfDNwN/Eqr31ZVzwGPJtkNXJTki8CpVXUvQJJbgcuBO6fbtrR4PPZbr1noFrQIffOv7ZjYvid9ZFHAJ5JsT7Kh1c6qqn0A7fHMVl8BPD40d0+rrWjrc+uHSbIhyWyS2QMHDhzFtyFJS9ukjyzeUFV7k5wJ3JXkX+cZO+o6RM1TP7xYdTNwM8CaNWv8vVhJOkomemRRVXvb437gduAi4IkkywHa4/42fA9w9tD0lcDeVl85oi5JmpKJhUWSr0/yskPrwA8ADwBbgfVt2Hrgjra+FViX5OQk5zC4kH1fO1X1TJKL26egrhyaI0magkmehjoLuL19yvVE4G+r6h+SfBbYkuQq4DHgCoCq2plkC/AgcBC4pqqeb/u6GtgEnMLgwrYXtyVpiiYWFlX1CPDaEfWngEteYM5GYOOI+ixw/tHuUZI0Hr/BLUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVLXQtyi/Jhw4S/dutAtaBHa/gdXLnQL0oLwyEKS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdU08LJIsS/K5JB9r26cnuSvJw+3xtKGx1yfZnWRXkkuH6hcm2dGeuyFJJt23JOlrpnFkcS3w0ND2dcC2qloNbGvbJDkXWAecB6wFbkyyrM25CdgArG7L2in0LUlqJhoWSVYCPwz85VD5MmBzW98MXD5Uv62qnquqR4HdwEVJlgOnVtW9VVXArUNzJElTMOkji/cAvwx8dah2VlXtA2iPZ7b6CuDxoXF7Wm1FW59bP0ySDUlmk8weOHDg6LwDSdLkwiLJjwD7q2r7uFNG1Gqe+uHFqpurak1VrZmZmRnzZSVJPSdOcN9vAN6U5IeAlwCnJnk/8ESS5VW1r51i2t/G7wHOHpq/Etjb6itH1CVJUzKxI4uqur6qVlbVKgYXrv+xqt4BbAXWt2HrgTva+lZgXZKTk5zD4EL2fe1U1TNJLm6fgrpyaI4kaQomeWTxQt4NbElyFfAYcAVAVe1MsgV4EDgIXFNVz7c5VwObgFOAO9siSZqSqYRFVd0N3N3WnwIueYFxG4GNI+qzwPmT61CSNB+/wS1J6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lS18TCIslLktyX5PNJdib5zVY/PcldSR5uj6cNzbk+ye4ku5JcOlS/MMmO9twNSTKpviVJh5vkkcVzwBur6rXABcDaJBcD1wHbqmo1sK1tk+RcYB1wHrAWuDHJsravm4ANwOq2rJ1g35KkOSYWFjXwbNs8qS0FXAZsbvXNwOVt/TLgtqp6rqoeBXYDFyVZDpxaVfdWVQG3Ds2RJE3BRK9ZJFmW5H5gP3BXVX0GOKuq9gG0xzPb8BXA40PT97TairY+tz7q9TYkmU0ye+DAgaP7ZiRpCZtoWFTV81V1AbCSwVHC+fMMH3Udouapj3q9m6tqTVWtmZmZOfKGJUkjTeXTUFX1n8DdDK41PNFOLdEe97dhe4Czh6atBPa2+soRdUnSlEzy01AzSV7e1k8Bvh/4V2ArsL4NWw/c0da3AuuSnJzkHAYXsu9rp6qeSXJx+xTUlUNzJElTcOI4g5Jsq6pLerU5lgOb2yeaTgC2VNXHktwLbElyFfAYcAVAVe1MsgV4EDgIXFNVz7d9XQ1sAk4B7myLJGlK5g2LJC8BXgqc0b4Pcej6wanAK+ebW1VfAF43ov4UMDJkqmojsHFEfRaY73qHJGmCekcWPw38PINg2M7XwuLLwJ9PsC9J0iIyb1hU1XuB9yb52ar60yn1JElaZMa6ZlFVf5rkO4FVw3Oq6tYJ9SVJWkTGvcD9PuBbgPuBQxedD32bWpJ0nBsrLIA1wLntdhuSpCVm3O9ZPAB80yQbkSQtXuMeWZwBPJjkPgZ3kwWgqt40ka4kSYvKuGHxG5NsQpK0uI37aah7Jt2IJGnxGvfTUM/wtTu9fh2D36b4SlWdOqnGJEmLx7hHFi8b3k5yOXDRRDqSJC06L+qus1X1EeCNR7kXSdIiNe5pqDcPbZ7A4HsXfudCkpaIcT8N9aND6weBLzL4zWxJ0hIw7jWLd066EUnS4jXWNYskK5PcnmR/kieSfCjJyv5MSdLxYNwL3H/N4GdPXwmsAD7aapKkJWDcsJipqr+uqoNt2QTMTLAvSdIiMm5YPJnkHUmWteUdwFOTbEyStHiMGxbvAt4K/DuwD3gL4EVvSVoixv3o7G8D66vqPwCSnA78IYMQkSQd58Y9svi2Q0EBUFVPA6+bTEuSpMVm3LA4IclphzbakcW4RyWSpGPcuH/h/xHw6SQfZHCbj7cCGyfWlSRpURn3G9y3JpllcPPAAG+uqgcn2pkkadEY+1RSCwcDQpKWoBd1i3JJ0tJiWEiSugwLSVKXYSFJ6jIsJEldhoUkqWtiYZHk7CSfTPJQkp1Jrm3105PcleTh9jj8zfDrk+xOsivJpUP1C5PsaM/dkCST6luSdLhJHlkcBH6xqr4VuBi4Jsm5wHXAtqpaDWxr27Tn1gHnAWuBG5Msa/u6CdgArG7L2gn2LUmaY2JhUVX7qupf2vozwEMMfmXvMmBzG7YZuLytXwbcVlXPVdWjwG7goiTLgVOr6t6qKuDWoTmSpCmYyjWLJKsY3KX2M8BZVbUPBoECnNmGrQAeH5q2p9VWtPW59VGvsyHJbJLZAwcOHM23IElL2sTDIsk3AB8Cfr6qvjzf0BG1mqd+eLHq5qpaU1VrZmb81VdJOlomGhZJTmIQFH9TVR9u5SfaqSXa4/5W3wOcPTR9JbC31VeOqEuSpmSSn4YK8FfAQ1X1x0NPbQXWt/X1wB1D9XVJTk5yDoML2fe1U1XPJLm47fPKoTmSpCmY5A8YvQH4SWBHkvtb7VeBdwNbklwFPAZcAVBVO5NsYXBn24PANVX1fJt3NbAJOAW4sy2SpCmZWFhU1T8x+noDwCUvMGcjI35UqapmgfOPXneSpCPhN7glSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6ppYWCS5Jcn+JA8M1U5PcleSh9vjaUPPXZ9kd5JdSS4dql+YZEd77oYkmVTPkqTRJnlksQlYO6d2HbCtqlYD29o2Sc4F1gHntTk3JlnW5twEbABWt2XuPiVJEzaxsKiqTwFPzylfBmxu65uBy4fqt1XVc1X1KLAbuCjJcuDUqrq3qgq4dWiOJGlKpn3N4qyq2gfQHs9s9RXA40Pj9rTairY+ty5JmqLFcoF71HWImqc+eifJhiSzSWYPHDhw1JqTpKVu2mHxRDu1RHvc3+p7gLOHxq0E9rb6yhH1karq5qpaU1VrZmZmjmrjkrSUTTsstgLr2/p64I6h+rokJyc5h8GF7PvaqapnklzcPgV15dAcSdKUnDipHSf5APC9wBlJ9gC/Drwb2JLkKuAx4AqAqtqZZAvwIHAQuKaqnm+7uprBJ6tOAe5siyRpiiYWFlX1thd46pIXGL8R2DiiPgucfxRbkyQdocVygVuStIgZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1HTNhkWRtkl1Jdie5bqH7kaSl5JgIiyTLgD8HfhA4F3hbknMXtitJWjqOibAALgJ2V9UjVfU/wG3AZQvckyQtGScudANjWgE8PrS9B/iOuYOSbAA2tM1nk+yaQm9LwRnAkwvdxGKQP1y/0C3ocP75POTXczT28qpRxWMlLEb9F6jDClU3AzdPvp2lJclsVa1Z6D6kUfzzOR3HymmoPcDZQ9srgb0L1IskLTnHSlh8Flid5JwkXwesA7YucE+StGQcE6ehqupgkp8BPg4sA26pqp0L3NZS4qk9LWb++ZyCVB126l+SpP/jWDkNJUlaQIaFJKnLsNC8vM2KFqsktyTZn+SBhe5lKTAs9IK8zYoWuU3A2oVuYqkwLDQfb7OiRauqPgU8vdB9LBWGheYz6jYrKxaoF0kLyLDQfMa6zYqk459hofl4mxVJgGGh+XmbFUmAYaF5VNVB4NBtVh4CtnibFS0WST4A3Au8OsmeJFctdE/HM2/3IUnq8shCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkP4fkmxK8pbOmJ9K8soj3O+q3q23k3wgyReS/MKR7Ft6MY6J3+CWjnE/BTzAUbxVSpJvAr6zql51BHNObF+0lI6YRxY6LiW5sv2r+/NJ3pfkVUm2tdq2JN/cxm1KclOSTyZ5JMn3tB/VeSjJpqH9PZvkj5L8S5s/M+I1L0xyT5LtST6eZHk76lgD/E2S+5OcMmrc0PzPJ7kXuKbzFj8BnNn2+d1JLkjyz+393Z7ktLbPu5P8bpJ7gGvb9p8k+VR7j9+e5MNJHk7yO0flP76OT1Xl4nJcLcB5wC7gjLZ9OvBRYH3bfhfwkba+icHvdITBb3V8GXgNg39IbQcuaOMKeHtb/zXgz4bmvwU4Cfg0MNPqPw7c0tbvBta09fnGfQH4nrb+B8AD87zHVcPPz5n7W8B7hl77xqFxdwO/19avZXC0sxw4mcGNI1+x0P/7uSzOxdNQOh69EfhgVT0JUFVPJ3k98Ob2/PuA3x8a/9GqqiQ7gCeqagdAkp0M/lK+H/gq8Hdt/PuBD895zVcD5wN3JQFYBuwb0dvIcUm+EXh5Vd0z1OMPjvNmR8zdDPz90JC/mzPl0M0gdwA7q2pf288jDO4y/NQ4r6ulxbDQ8Sj0f3dj+Pnn2uNXh9YPbb/Q/0fm7j8M/uJ9/Ri9HTYuyctH7PNo+cqc7RfzfrXEec1Cx6NtwFuTvAIgyekMTv2sa8+/HfinI9znCQxONwH8xIj5u4CZdgRDkpOSnNeeewZ42Xzjquo/gS8l+a6hHsdSVV8C/iPJd7fSTwL3zDNFOmL+K0LHnaramWQjcE+S54HPAT8H3JLkl4ADwDuPcLdfAc5Lsh34EoNrDcOv+T/tYvYN7bTQicB7gJ0Mrmv8RZL/Bl7PIHRGjXtn6/G/GNwW/kisb6/xUuCRF/H+pHl5i3JpDEmerapvWOg+pIXiaShJUpdHFtIiluRS4PfmlB+tqh9biH60dBkWkqQuT0NJkroMC0lSl2EhSeoyLCRJXf8LfQqM9992EJUAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"ax = sns.countplot(x='completed_form', data=control)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEHCAYAAABfkmooAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAQfElEQVR4nO3df+xddX3H8eeLgohRJsgXxBYtWRozwA1Dx1BnXHQJdT8sM+LqVKqSdCFsw2Vxgf2hm1sXnbopTljIhi1qxM5foAlxpBOMkcm+VbQURmhgg4aOFvAHuAUDvvfH/XTefXv5fm5Z7/d72+/zkZzcc973fM59X6x99Zxz7+emqpAkaT5HLHYDkqTpZ1hIkroMC0lSl2EhSeoyLCRJXUcudgOTcsIJJ9TKlSsXuw1JOqRs27btoaqamVs/bMNi5cqVzM7OLnYbknRISfIfo+pehpIkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUdtt/glg5n9733JYvdgqbQC9+9fWLH9sxCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktQ18bBIsizJt5N8uW0fn+TGJHe3x+OG9r0syc4kdyU5d6h+VpLt7bnLk2TSfUuSfmohziwuAe4c2r4U2FpVq4CtbZskpwHrgNOBNcAVSZa1MVcCG4BVbVmzAH1LkpqJhkWSFcCvA38/VF4LbG7rm4HzhurXVtXjVXUvsBM4O8nJwLFVdUtVFXDN0BhJ0gKY9JnFh4E/Bn4yVDupqnYDtMcTW305cP/QfrtabXlbn1vfT5INSWaTzO7du/fgvANJ0uTCIslvAHuqatu4Q0bUap76/sWqq6pqdVWtnpmZGfNlJUk9R07w2K8AXpfk14BnAscm+STwYJKTq2p3u8S0p+2/CzhlaPwK4IFWXzGiLklaIBM7s6iqy6pqRVWtZHDj+p+r6i3A9cD6ttt64Lq2fj2wLsnRSU5lcCP71nap6tEk57RPQV0wNEaStAAmeWbxVN4HbElyIXAfcD5AVe1IsgW4A3gCuLiqnmxjLgI2AccAN7RFkrRAFiQsquom4Ka2/jDwmqfYbyOwcUR9Fjhjch1KkubjN7glSV2LcRnqkHDWu65Z7BY0hbZ94ILFbkFaFJ5ZSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSuiYWFkmemeTWJN9JsiPJn7X68UluTHJ3ezxuaMxlSXYmuSvJuUP1s5Jsb89dniST6luStL9Jnlk8Dry6qn4BOBNYk+Qc4FJga1WtAra2bZKcBqwDTgfWAFckWdaOdSWwAVjVljUT7FuSNMfEwqIGHmubR7WlgLXA5lbfDJzX1tcC11bV41V1L7ATODvJycCxVXVLVRVwzdAYSdICmOg9iyTLktwG7AFurKpvAidV1W6A9nhi2305cP/Q8F2ttrytz61LkhbIRMOiqp6sqjOBFQzOEs6YZ/dR9yFqnvr+B0g2JJlNMrt3794Db1iSNNKCfBqqqr4P3MTgXsOD7dIS7XFP220XcMrQsBXAA62+YkR91OtcVVWrq2r1zMzMQX0PkrSUTfLTUDNJntvWjwF+Ffg34HpgfdttPXBdW78eWJfk6CSnMriRfWu7VPVoknPap6AuGBojSVoAR07w2CcDm9snmo4AtlTVl5PcAmxJciFwH3A+QFXtSLIFuAN4Ari4qp5sx7oI2AQcA9zQFknSAplYWFTVd4GXjqg/DLzmKcZsBDaOqM8C893vkCRNkN/gliR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKlrrLBIsnWcmiTp8DTvN7iTPBN4FnBC+0W7fTPAHgu8YMK9SZKmRG+6j98F3skgGLbx07D4IfCxCfYlSZoi84ZFVX0E+EiS36+qjy5QT5KkKTPWRIJV9dEkLwdWDo+pqmsm1JckaYqMFRZJPgH8LHAbsG/a8H2/hy1JOsyNO0X5auC0qhr5c6aSpMPbuN+zuB14/iQbkSRNr3HPLE4A7khyK/D4vmJVvW4iXUmSpsq4YfGnk2xCkjTdxv001M2TbkSSNL3G/TTUoww+/QTwDOAo4EdVdeykGpMkTY9xzyyeM7yd5Dzg7Il0JEmaOk9r1tmq+iLw6oPciyRpSo17Ger1Q5tHMPjehd+5kKQlYtxPQ/3m0PoTwL8Daw96N5KkqTTuPYu3T7oRSdL0GvfHj1Yk+UKSPUkeTPK5JCsm3ZwkaTqMe4P748D1DH7XYjnwpVaTJC0B44bFTFV9vKqeaMsmYGaCfUmSpsi4YfFQkrckWdaWtwAPT7IxSdL0GDcs3gG8EfhPYDfwBsCb3pK0RIz70dk/B9ZX1fcAkhwPfJBBiEiSDnPjnln8/L6gAKiqR4CXTqYlSdK0GTcsjkhy3L6NdmYx7lmJJOkQN+5f+B8CvpHkswym+XgjsHFiXUmSpsq43+C+Jsksg8kDA7y+qu6YaGeSpKkx9qWkFg4GhCQtQU9rivJxJDklyVeT3JlkR5JLWv34JDcmubs9Dt8LuSzJziR3JTl3qH5Wku3tucuTZFJ9S5L2N7GwYDA77R9V1c8B5wAXJzkNuBTYWlWrgK1tm/bcOuB0YA1wRZJl7VhXAhuAVW1ZM8G+JUlzTCwsqmp3VX2rrT8K3MlgXqm1wOa222bgvLa+Fri2qh6vqnuBncDZSU4Gjq2qW6qqgGuGxkiSFsAkzyz+V5KVDL6X8U3gpKraDYNAAU5suy0H7h8atqvVlrf1ufVRr7MhyWyS2b179x7MtyBJS9rEwyLJs4HPAe+sqh/Ot+uIWs1T379YdVVVra6q1TMzznMoSQfLRMMiyVEMguJTVfX5Vn6wXVqiPe5p9V3AKUPDVwAPtPqKEXVJ0gKZ5KehAvwDcGdV/fXQU9cD69v6euC6ofq6JEcnOZXBjexb26WqR5Oc0455wdAYSdICmOSUHa8A3gpsT3Jbq/0J8D5gS5ILgfuA8wGqakeSLQy+y/EEcHFVPdnGXQRsAo4BbmiLJGmBTCwsqurrjL7fAPCapxizkRHTiFTVLHDGwetOknQgFuTTUJKkQ5thIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldEwuLJFcn2ZPk9qHa8UluTHJ3ezxu6LnLkuxMcleSc4fqZyXZ3p67PEkm1bMkabRJnllsAtbMqV0KbK2qVcDWtk2S04B1wOltzBVJlrUxVwIbgFVtmXtMSdKETSwsquprwCNzymuBzW19M3DeUP3aqnq8qu4FdgJnJzkZOLaqbqmqAq4ZGiNJWiALfc/ipKraDdAeT2z15cD9Q/vtarXlbX1ufaQkG5LMJpndu3fvQW1ckpayabnBPeo+RM1TH6mqrqqq1VW1emZm5qA1J0lL3UKHxYPt0hLtcU+r7wJOGdpvBfBAq68YUZckLaCFDovrgfVtfT1w3VB9XZKjk5zK4Eb2re1S1aNJzmmfgrpgaIwkaYEcOakDJ/k08CvACUl2Ae8B3gdsSXIhcB9wPkBV7UiyBbgDeAK4uKqebIe6iMEnq44BbmiLJGkBTSwsqupNT/HUa55i/43AxhH1WeCMg9iaJOkATcsNbknSFDMsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdR0yYZFkTZK7kuxMculi9yNJS8khERZJlgEfA14LnAa8Kclpi9uVJC0dh0RYAGcDO6vqnqr6MXAtsHaRe5KkJePIxW5gTMuB+4e2dwG/NHenJBuADW3zsSR3LUBvS8EJwEOL3cQ0yAfXL3YL2p9/Pvd5Tw7GUV40qniohMWo/wK1X6HqKuCqybeztCSZrarVi92HNIp/PhfGoXIZahdwytD2CuCBRepFkpacQyUs/hVYleTUJM8A1gHXL3JPkrRkHBKXoarqiSS/B3wFWAZcXVU7FrmtpcRLe5pm/vlcAKna79K/JEn/x6FyGUqStIgMC0lSl2GheTnNiqZVkquT7Ely+2L3shQYFnpKTrOiKbcJWLPYTSwVhoXm4zQrmlpV9TXgkcXuY6kwLDSfUdOsLF+kXiQtIsNC8xlrmhVJhz/DQvNxmhVJgGGh+TnNiiTAsNA8quoJYN80K3cCW5xmRdMiyaeBW4AXJ9mV5MLF7ulw5nQfkqQuzywkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwL6f8hyaYkb+js87YkLzjA467sTb2d5NNJvpvkDw/k2NLTcUj8Brd0iHsbcDsHcaqUJM8HXl5VLzqAMUe2L1pKB8wzCx2WklzQ/tX9nSSfSPKiJFtbbWuSF7b9NiW5MslXk9yT5FXtR3XuTLJp6HiPJflQkm+18TMjXvOsJDcn2ZbkK0lObmcdq4FPJbktyTGj9hsa/50ktwAXd97iPwEntmO+MsmZSf6lvb8vJDmuHfOmJH+Z5Gbgkrb9N0m+1t7jLyb5fJK7k/zFQfmPr8NTVbm4HFYLcDpwF3BC2z4e+BKwvm2/A/hiW9/E4Hc6wuC3On4IvITBP6S2AWe2/Qp4c1t/N/C3Q+PfABwFfAOYafXfBq5u6zcBq9v6fPt9F3hVW/8AcPs873Hl8PNzxr4X+PDQa18xtN9NwPvb+iUMznZOBo5mMHHk8xb7fz+X6Vy8DKXD0auBz1bVQwBV9UiSlwGvb89/Avirof2/VFWVZDvwYFVtB0iyg8FfyrcBPwE+0/b/JPD5Oa/5YuAM4MYkAMuA3SN6G7lfkp8BnltVNw/1+Npx3uyIsZuBfxza5TNzhuybDHI7sKOqdrfj3MNgluGHx3ldLS2GhQ5Hof+7G8PPP94efzK0vm/7qf4/Mvf4YfAX78vG6G2//ZI8d8QxD5Yfzdl+Ou9XS5z3LHQ42gq8McnzAJIcz+DSz7r2/JuBrx/gMY9gcLkJ4HdGjL8LmGlnMCQ5Ksnp7blHgefMt19VfR/4QZJfHupxLFX1A+B7SV7ZSm8Fbp5niHTA/FeEDjtVtSPJRuDmJE8C3wb+ALg6ybuAvcDbD/CwPwJOT7IN+AGDew3Dr/njdjP78nZZ6Ejgw8AOBvc1/i7JfwMvYxA6o/Z7e+vxvxhMC38g1rfXeBZwz9N4f9K8nKJcGkOSx6rq2Yvdh7RYvAwlSeryzEKaYknOBd4/p3xvVf3WYvSjpcuwkCR1eRlKktRlWEiSugwLSVKXYSFJ6vofkhH7L4G3S5MAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"ax = sns.countplot(x='completed_form', data=test)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Baseline conversion: 0.5253207448811145\n",
"Control conversion: 0.5186790505675954\n",
"Test conversion: 0.5483225426721601\n",
"Observed lift: 0.029643492104564628\n"
]
}
],
"source": [
"# Calculate conversion rates\n",
"conversionPre = pre['completed_form'].mean()\n",
"conversionControl = control['completed_form'].mean()\n",
"conversionTest = test['completed_form'].mean()\n",
"lift = conversionTest - conversionControl\n",
"print('Baseline conversion: {}'.format(conversionPre))\n",
"print('Control conversion: {}'.format(conversionControl))\n",
"print('Test conversion: {}'.format(conversionTest))\n",
"print('Observed lift: {}'.format(lift))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While the conversion rate in the first checkout stage of the test group seems to be higher than the control group, we are interested in the quality of conversions and if outcomes are improved across the the checkout funnel. Ideally, the new feature would ultimately lead to more sales conversions in the final stage of the funnel."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Control second stage conversion: 0.6707122960604854\n",
"Test second stage conversion: 0.6629454701588665\n"
]
}
],
"source": [
"conversionControlSecond = control['completed_second'].sum()/control['completed_form'].sum()\n",
"conversionTestSecond = test['completed_second'].sum()/test['completed_form'].sum()\n",
"print('Control second stage conversion: {}'.format(conversionControlSecond))\n",
"print('Test second stage conversion: {}'.format(conversionTestSecond))"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Control final stage conversion: 0.754375556214773\n",
"Test final stage conversion: 0.7470854922279793\n"
]
}
],
"source": [
"conversionControlFinal = control['checkout_id'].count()/control['completed_second'].sum()\n",
"conversionTestFinal = test['checkout_id'].count()/test['completed_second'].sum()\n",
"print('Control final stage conversion: {}'.format(conversionControlFinal))\n",
"print('Test final stage conversion: {}'.format(conversionTestFinal))"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Control Total conversion: 0.26243550051599585\n",
"Test Total conversion: 0.27157151265450263\n"
]
}
],
"source": [
"conversionControlTotal = control['checkout_id'].count()/nControl\n",
"conversionTestTotal = test['checkout_id'].count()/nTest\n",
"print('Control Total conversion: {}'.format(conversionControlTotal))\n",
"print('Test Total conversion: {}'.format(conversionTestTotal))"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.axes._subplots.AxesSubplot at 0x7f999a173a90>"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAJhCAYAAACaWIXAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZhmZX0m/vtLAyKIKHQbFVQwwQUJCHaM27gGBf2JmhDjviUyJnFLJhlxEo1mmWjMmMS4IBo0jhrCaKJEcTTuSzTSKDKgoohbi1EERTAq2/f3x/u2FE1BF9jU6Yf387muunjPc05V3d3w8NZd55znVHcHAABgJNtNHQAAAOCaUmQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABjO9lN947Vr1/bee+891bcHAAC2cSeffPJ3unvdcvsmKzJ77713NmzYMNW3BwAAtnFV9dWr2ufSMgAAYDiKDAAAMBxFBgAAGM5k98gAAMAiuvjii7Nx48b86Ec/mjrKNmOnnXbKXnvtlR122GHFn6PIAADAKtq4cWN23XXX7L333qmqqeNMrrtz7rnnZuPGjdlnn31W/HkuLQMAgFX0ox/9KHvssYcSM1dV2WOPPa7xGSpFBgAAVpkSc0XX5u9DkQEAAK6Rr3zlK3nzm998jT/v9a9/fZ7+9KdvlQzukQEAgAntfdQ7t+rX+8qLHrJVv96y32NeZB7zmMdcad8ll1yS7be/7muGMzIAALBg3vCGN+SAAw7IgQcemMc//vH56le/mgc84AE54IAD8oAHPCBf+9rXkiRPetKT8sxnPjP3uMc9ctvb3jZvectbkiRHHXVUPvKRj+TOd75z/uqv/iqvf/3r86u/+qt56EMfmgc+8IE577zz8vCHPzwHHHBA7na3u+XUU0/d6n8GZ2QAAGCBnH766fmzP/uzfOxjH8vatWtz3nnn5YlPfGKe8IQn5IlPfGKOPfbYPPOZz8zb3va2JMk3v/nNfPSjH83nP//5HH744TniiCPyohe9KH/5l3+Zd7zjHUlml4x9/OMfz6mnnprdd989z3jGM3LQQQflbW97W97//vfnCU94Qk455ZSt+udwRgYAABbI+9///hxxxBFZu3ZtkmT33XfPxz/+8Z9cJvb4xz8+H/3oR39y/MMf/vBst9122W+//fKtb33rKr/uIYcckt133z1J8tGPfjSPf/zjkyT3v//9c+655+b888/fqn8ORQYAABZId29xlbCl+29wgxtc4XOvyi677HK1x23tldoUGQAAWCAPeMADcvzxx+fcc89Nkpx33nm5xz3ukeOOOy5J8qY3vSn3ute9rvZr7Lrrrrnggguucv+9733vvOlNb0qSfPCDH8zatWtz4xvfeCv9CWbcIwMAAAvkTne6U/7gD/4g97nPfbJmzZocdNBBednLXpanPOUpeclLXpJ169blda973dV+jQMOOCDbb799DjzwwDzpSU/KTW960yvsf8ELXpAnP/nJOeCAA7Lzzjvn7//+77f6n6Ou7vTQdWn9+vW9YcOGSb43AABM5XOf+1zueMc7Th1jm7Pc30tVndzd65c73qVlAADAcBQZAABgOO6RAYCtaGs/oXu1rcYTwQG2BmdkAACA4SgyAADAcBQZAABgOO6RAQAu94Ldpk7w03vB+VMngG3a9773vbz5zW/Ob/3Wb13jz/3rv/7rHHnkkdl5552vg2TXjCIDAABT2tq/QNhCmf/e976XV77ylde6yDzucY9TZAAAgNV11FFH5Utf+lLufOc755BDDsnNbnazHH/88fnxj3+cRzziEXnhC1+YH/zgB3nkIx+ZjRs35tJLL83znve8fOtb38rZZ5+d+93vflm7dm0+8IEPTPrnUGQAAGCBvOhFL8ppp52WU045Je95z3vylre8JZ/85CfT3Tn88MPz4Q9/OOecc05uectb5p3vnC0pf/7552e33XbLS1/60nzgAx/I2rVrJ/5TuNkfAAAW1nve85685z3vyUEHHZSDDz44n//85/PFL34xP//zP5/3vve9ec5znpOPfOQj2W23be/+OWdkAABgQXV3nvvc5+a//tf/eqV9J598ck488cQ897nPzQMf+MA8//nPnyDhVXNGBgAAFsiuu+6aCy64IEnyoAc9KMcee2wuvPDCJMk3vvGNfPvb387ZZ5+dnXfeOY973OPye7/3e/nUpz51pc+dmjMyAACwQPbYY4/c8573zP7775/DDjssj3nMY3L3u989SXKjG90ob3zjG3PmmWfm93//97Pddttlhx12yKte9aokyZFHHpnDDjsst7jFLSa/2b+6e5JvvH79+t6wYcMk3xsArit7H/XOqSP8VL6y02OmjvDT8xwZtnGf+9zncsc73nHqGNuc5f5equrk7l6/3PEuLQMAAIajyAAAAMNRZAAAgOEoMgAAsMqmuk99W3Vt/j4UGQAAWEU77bRTzj33XGVmrrtz7rnnZqeddrpGn2f5ZQAAWEV77bVXNm7cmHPOOWfqKNuMnXbaKXvttdc1+hxFBgAAVtEOO+yQffbZZ+oYw3NpGQAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcFZUZKrq0Ko6o6rOrKqjltn/+1V1yvzjtKq6tKp23/pxAQAAVlBkqmpNklckOSzJfkkeXVX7LT2mu1/S3Xfu7jsneW6SD3X3eddFYAAAgJWckblrkjO7+6zuvijJcUkedjXHPzrJP2yNcAAAAMtZSZHZM8nXl2xvnI9dSVXtnOTQJG+9iv1HVtWGqtpwzjnnXNOsAAAASVZWZGqZsb6KYx+a5GNXdVlZdx/T3eu7e/26detWmhEAAOAKVlJkNia51ZLtvZKcfRXHPiouKwMAAK5jKykyJyXZt6r2qaodMysrJ2x+UFXtluQ+Sd6+dSMCAABc0fZbOqC7L6mqpyd5d5I1SY7t7tOr6mnz/UfPD31Ekvd09w+us7QAAABZQZFJku4+McmJm40dvdn265O8fmsFAwAAuCoreiAmAADAtkSRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGs/3UAYDx7X3UO6eO8FP7yoseMnUEAOAacEYGAAAYjiIDAAAMR5EBAACG4x4ZgCR5wW5TJ/jpveD8qRMAwKpxRgYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDgrKjJVdWhVnVFVZ1bVUVdxzH2r6pSqOr2qPrR1YwIAAFxu+y0dUFVrkrwiySFJNiY5qapO6O7PLjnmJklemeTQ7v5aVd3sugoMAACwkjMyd01yZnef1d0XJTkuycM2O+YxSf6pu7+WJN397a0bEwAA4HIrKTJ7Jvn6ku2N87GlbpfkplX1wao6uaqesNwXqqojq2pDVW0455xzrl1iAABg4a2kyNQyY73Z9vZJ7pLkIUkelOR5VXW7K31S9zHdvb67169bt+4ahwUAAEhWcI9MZmdgbrVke68kZy9zzHe6+wdJflBVH05yYJIvbJWUAAAAS6zkjMxJSfatqn2qasckj0pywmbHvD3Jf6mq7atq5yS/mORzWzcqAADAzBbPyHT3JVX19CTvTrImybHdfXpVPW2+/+ju/lxV/d8kpya5LMlru/u06zI4AACwuFZyaVm6+8QkJ242dvRm2y9J8pKtFw0AAGB5K3ogJgAAwLZEkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4208dgIm9YLepE/z0XnD+1AkAAFhlzsgAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhrP91AFGtvdR75w6wk/tKztNnQAAAK45Z2QAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxnRUWmqg6tqjOq6syqOmqZ/fetqvOr6pT5x/O3flQAAICZLT5HpqrWJHlFkkOSbExyUlWd0N2f3ezQj3T3/3cdZAQAALiClZyRuWuSM7v7rO6+KMlxSR523cYCAAC4aispMnsm+fqS7Y3zsc3dvao+U1Xvqqo7LfeFqurIqtpQVRvOOeecaxEXAABgZUWmlhnrzbY/leQ23X1gkr9N8rblvlB3H9Pd67t7/bp1665ZUgAAgLmVFJmNSW61ZHuvJGcvPaC7v9/dF85fn5hkh6pau9VSAgAALLGSInNSkn2rap+q2jHJo5KcsPSAqrp5VdX89V3nX/fcrR0WAAAgWcGqZd19SVU9Pcm7k6xJcmx3n15VT5vvPzrJEUl+s6ouSfLDJI/q7s0vPwMAANgqtlhkkp9cLnbiZmNHL3n98iQv37rRAAAAlreiB2ICAABsSxQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADGdFRaaqDq2qM6rqzKo66mqO+4WqurSqjth6EQEAAK5oi0WmqtYkeUWSw5Lsl+TRVbXfVRz34iTv3tohAQAAllrJGZm7Jjmzu8/q7ouSHJfkYcsc94wkb03y7a2YDwAA4EpWUmT2TPL1Jdsb52M/UVV7JnlEkqO3XjQAAIDlraTI1DJjvdn2Xyd5TndferVfqOrIqtpQVRvOOeeclWYEAAC4gu1XcMzGJLdasr1XkrM3O2Z9kuOqKknWJnlwVV3S3W9belB3H5PkmCRZv3795mUIAABgRVZSZE5Ksm9V7ZPkG0keleQxSw/o7n02va6q1yd5x+YlBgAAYGvZYpHp7kuq6umZrUa2Jsmx3X16VT1tvt99MQAAwKpayRmZdPeJSU7cbGzZAtPdT/rpYwEAAFy1FT0QEwAAYFuiyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADCcFRWZqjq0qs6oqjOr6qhl9j+sqk6tqlOqakNV3WvrRwUAAJjZfksHVNWaJK9IckiSjUlOqqoTuvuzSw57X5ITurur6oAkxye5w3URGAAAYCVnZO6a5MzuPqu7L0pyXJKHLT2guy/s7p5v7pKkAwAAcB1ZSZHZM8nXl2xvnI9dQVU9oqo+n+SdSZ6ydeIBAABc2UqKTC0zdqUzLt39z919hyQPT/Iny36hqiPn99BsOOecc65ZUgAAgLmVFJmNSW61ZHuvJGdf1cHd/eEkP1tVa5fZd0x3r+/u9evWrbvGYQEAAJKVFZmTkuxbVftU1Y5JHpXkhKUHVNXPVVXNXx+cZMck527tsAAAAMkKVi3r7kuq6ulJ3p1kTZJju/v0qnrafP/RSX4lyROq6uIkP0zya0tu/gcAANiqtlhkkqS7T0xy4mZjRy95/eIkL9660QAAAJa3ogdiAgAAbEsUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADD2X7qAAAAXP/sfdQ7p47wU/vKTo+ZOsJP7wXnT53gOuOMDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4ayoyFTVoVV1RlWdWVVHLbP/sVV16vzj36rqwK0fFQAAYGaLRaaq1iR5RZLDkuyX5NFVtd9mh305yX26+4Akf5LkmK0dFAAAYJOVnJG5a5Izu/us7r4oyXFJHrb0gO7+t+7+7nzzE0n22roxAQAALreSIrNnkq8v2d44H7sqv57kXT9NKAAAgKuz/QqOqWXGetkDq+6XWZG511XsPzLJkUly61vfeoURAQAArmglZ2Q2JrnVku29kpy9+UFVdUCS1yZ5WHefu9wX6u5junt9d69ft27dtckLAACwoiJzUpJ9q2qfqtoxyaOSnLD0gKq6dZJ/SvL47v7C1o8JAABwuS1eWtbdl1TV05O8O8maJMd29+lV9bT5/qOTPD/JHkleWVVJckl3r7/uYgMAAItsJffIpLtPTHLiZmNHL3n9G0l+Y+tGAwAAWN6KHogJAACwLVFkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGM6KikxVHVpVZ1TVmVV11DL771BVH6+qH1fV7239mAAAAJfbfksHVNWaJK9IckiSjUlOqqoTuvuzSw47L8kzkzz8OkkJAACwxErOyNw1yZndfVZ3X5TkuCQPW3pAd3+7u09KcvF1kBEAAOAKVlJk9kzy9SXbG+djAAAAk1hJkallxvrafLOqOrKqNlTVhnPOOefafAkAAIAVFZmNSW61ZHuvJGdfm2/W3cd09/ruXr9u3bpr8yUAAABWVGROSrJvVe1TVTsmeVSSE67bWAAAAFdti6uWdfclVfX0JO9OsibJsd19elU9bb7/6Kq6eZINSW6c5LKqenaS/br7+9dhdgAAYEFtscgkSXefmOTEzcaOXvL6PzK75AwAAOA6t6IHYgIAAGxLFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGo8gAAADDUWQAAIDhKDIAAMBwFBkAAGA4igwAADAcRQYAABiOIgMAAAxHkQEAAIajyAAAAMNRZAAAgOEoMgAAwHAUGQAAYDiKDAAAMBxFBgAAGI4iAwAADEeRAQAAhqPIAAAAw1FkAACA4SgyAADAcBQZAABgOIoMAAAwHEUGAAAYjiIDAAAMR5EBAACGs6IiU1WHVtUZVXVmVR21zP6qqpfN959aVQdv/agAAAAzWywyVbUmySuSHJZkvySPrqr9NjvssCT7zj+OTPKqrZwTAADgJ1ZyRuauSc7s7rO6+6IkxyV52GbHPCzJG3rmE0luUlW32MpZAQAAkiTbr+CYPZN8fcn2xiS/uIJj9kzyzaUHVdWRmZ2xSZILq+qMa5SWra6StUm+M3WOn8oLa+oEXA+YCzBjLsDlzIdtwm2uasdKisxyf/q+Fseku49JcswKvierpKo2dPf6qXPA1MwFmDEX4HLmw7ZtJZeWbUxyqyXbeyU5+1ocAwAAsFWspMiclGTfqtqnqnZM8qgkJ2x2zAlJnjBfvexuSc7v7m9u/oUAAAC2hi1eWtbdl1TV05O8O8maJMd29+lV9bT5/qOTnJjkwUnOTPKfSZ583UVmK3OpH8yYCzBjLsDlzIdtWHVf6VYWAACAbdqKHogJAACwLVFkAACA4SgyAADAcBSZBVRVt6mqX5q/vmFV7Tp1JlhtVfUzVfV3VfWu+fZ+VfXrU+eCKVTVvarqyfPX66pqn6kzwVTMh3EoMgumqp6a5C1JXj0f2ivJ26ZLBJN5fWarMd5yvv2FJM+eLA1MpKr+KMlzkjx3PrRDkjdOlwimYz6MRZFZPL+d5J5Jvp8k3f3FJDebNBFMY213H5/ksmS21HySS6eNBJN4RJLDk/wgSbr77CTO1LOozIeBKDKL58fdfdGmjaraPok1uFlEP6iqPTL/73/Tw3ynjQSTuKhnz2LYNBd2mTgPTMl8GMgWH4jJ9c6Hqup/JLlhVR2S5LeS/MvEmWAKv5vkhCQ/W1UfS7Iuya9OGwkmcXxVvTrJTeaXHz8lyWsnzgRTMR8G4oGYC6aqtkvy60kemKSSvLu7XzNtKlh9VXWDzC4lu31mc+GMJNt1948nDQYTmP9ia+n7wr9OHAkmYz6MQ5FZMFX1rO7+my2NwfVdVX2quw/e0hhc31XVi7v7OVsag0VgPozFPTKL54nLjD1ptUPAVKrq5lV1l8wurzyoqg6ef9w3yc4Tx4MpHLLM2GGrngK2DebDQNwjsyCq6tFJHpNkn6o6YcmuXZOcO00qmMSDMivveyV56ZLxC5L8jykCwRSq6jczu0/ytlV16pJduyb52DSpYBrmw5hcWrYgquo2SfZJ8udJjlqy64Ikp86XnoWFUVW/0t1vnToHTKWqdkty0yzzvtDd502TCqZhPoxJkQEWVlU9JMmdkuy0aay7/3i6RDCdqrpZrjgXvjZhHJiU+TAG98gsmKq6W1WdVFUXVtVFVXVpVX1/6lyw2qrq6CS/luQZma1M86tJbjNpKJhAVT20qr6Y5MtJPpTkK0neNWkomIj5MBZFZvG8PMmjk3wxyQ2T/EaSv500EUzjHt39hCTf7e4XJrl7kltNnAmm8KdJ7pbkC929T5IHxD0BLC7zYSCKzALq7jOTrOnuS7v7dUnuN3UmmMAP5//8z6q6ZZKLM7uPDBbNxd19bpLtqmq77v5AkjtPHQomYj4MxKpli+c/q2rHJKdU1V8k+WaSXSbOBFN4R1XdJMlLknwqScfTm1lM36uqGyX5cJI3VdW3k1gAhkVlPgzEzf4LZr562beS7Jjkd5LsluQV3f2lSYPBhKrqBpnd1HlJd/9g6jywmqpqlyQ/yuxescdm9r7wRis1sYjMh7EoMguuqm6a5Le6+8+mzgKrpar2THKLzJYev2i+Os2zkzypu285bTqYVlXdIcl/6+6nTp0FpmY+bNvcI7MgqupWVXVMVb2jqn6jqnauqv+V5IwkN5s6H6yWqnp2klMyW+TiE1X1xCSfy2zxi7tMmQ1WU1UdUFXvqarTqupPq+pnquqtSd6b5LNT54PVZD6MyT0yi+MNmS0j+NYkhyb5RJLTkxzQ3f8xZTBYZUcmuX13n1dVt05yZpJ7d/cnJs4Fq+01SV6V5OOZvS98Ksmbkzy2u380ZTCYgPkwIJeWLYiq+kx3H7hk+1tJbt3dP54wFqy6qvpUdx+8ZPu07t5/ykwwhao6pbvvvGT760n27u5LJ4wFkzAfxuSMzAKZ3w9T883/SLLz/Ka2uImNBbJXVb1syfbNlm539zMnyART2KmqDsrl7wsXJjmgqipJuvtTkyWD1Wc+DMgZmQVRVV9Jclkun6BLdXffdnUTwTTm98Rcpe7++9XKAlOqqg9cze7u7vuvWhiYmPkwJkUGAAAYjlXLAACA4SgyAADAcBQZAABgOFYtW0BVtSbJz2TJv//u/tp0iWD1VdW6JE9NsneuOBeeMlUmmEpV7ZnkNrniXPjwdIlgOubDOBSZBVNVz0jyR0m+ldkqZknSSQ6YLBRM4+1JPpLZU5s9J4CFVVUvTvJrmT29fNNc6CR+cGPhmA9jsWrZgqmqM5P8YnefO3UWmNLmDz+DRVVVZyQ5wAOSwXwYjXtkFs/Xk5w/dQjYBryjqh48dQjYBpyVZIepQ8A2wnwYiDMyC6aq/i7J7ZO8M8lPftvQ3S+dLBRMoKouSLJLkouSXDwf7u6+8XSpYPVV1VuTHJjkfbni+8IzJwsFEzEfxuIemcXztfnHjvMPWEjdvevUGWAbccL8AzAfhuKMzIKqql0z++3zhVNngalU1eFJ7j3f/GB3v2PKPDCVqtoxye3mm2d098VXdzxcn5kP41BkFkxV7Z/kfyfZfT70nSRP6O7Tp0sFq6+qXpTkF5K8aT706CQnd/dR06WC1VdV903y90m+kqSS3CrJEy03yyIyH8aiyCyYqvq3JH/Q3R+Yb983yf/s7ntMGgxWWVWdmuTO3X3ZfHtNkk93t6XIWShVdXKSx3T3GfPt2zvseSAAABciSURBVCX5h+6+y7TJYPWZD2Oxatni2WVTiUmS7v5gZjc8wyK6yZLXu02WAqa1w6Yf2pKku78QqzaxuMyHgbjZf/GcVVXPy+zysiR5XJIvT5gHpvLnST5dVR/I7PKBeyd57rSRYBIb5itabnpfeGySkyfMA1MyHwbi0rIFU1U3TfLCJPfK7Ie3Dyd5QXd/d9JgMIGqukVm98lUkn/v7v+YOBKsuqq6QZLfzhXfF17pgYAsIvNhLIoMsFCq6g7d/fmqOni5/d39qdXOBABcc4rMgqiqv+7uZ1fVvyS50r/07j58gliw6qrqmO4+cn5J2ea6u++/6qFgAlV1fHc/sqr+X5Z/X7DwBQvDfBiTIrMgquou3X1yVd1nuf3d/aHVzgTAdKrqFt39zaq6zXL7u/urq50JpmI+jEmRWWDz+2Vu1d2nTp0FVltV/WqS/9vdF1TVHyY5OMmfdPenJ44Gq6qqdknyw+6+bL7U7B2SvMtDAFlE5sNYLL+8YKrqg1V146raPclnkryuql46dS6YwPPmJeZeSR6U2QPQjp44E0zhw0l2qqo9k7wvyZOTvH7SRDAd82Egiszi2a27v5/kl5O8bv6Ap1+aOBNM4dL5Px+S5FXd/fYkO06YB6ZS3f2fmb0v/G13PyLJfhNngqmYDwNRZBbP9vMlZx+Z5B1Th4EJfaOqXp3ZXDhxvuSm/yeyiKqq7p7Z8zLeOR/znDkWlfkwEG/ai+eFSd6d5MzuPqmqbpvkixNngik8MrO5cGh3fy/J7kl+f9pIMIlnZfYw2H/u7tPn7wvLreoHi8B8GIib/RdIVa1J8szu/qups8CUqmq7JKd29/5TZ4Epzd8XXtTdSjwLz3wYjzMyC6S7L03ieTEsvO6+LMlnqurWU2eBKc3fF+4ydQ7YFpgP43HN3+L5t6p6eZJ/TPKDTYOeZs4CukWS06vqk7niXFD2WTSfrqoTkvyfXHEu/NN0kWAy5sNAXFq2YDzNHGY8HBZmqup1ywx3dz9l1cPAxMyHsSgywMKaP8F53+5+b1XtnGRNd18wdS4AYMvcI7Ngqupnqurvqupd8+39qurXp84Fq62qnprkLUlePR/aM8nbpksE06iq21XV+6rqtPn2AVX1h1PngimYD2NRZBbP6zNbcvaW8+0vJHn2ZGlgOr+d5J5Jvp8k3f3FJDebNBFM4zWZLTd7cZJ096lJHjVpIpiO+TAQRWbxrO3u45NcliTdfUkuf8I5LJIfd/dFmzaqavskrrVlEe3c3Z/cbOySSZLA9MyHgSgyi+cHVbVH5j+wVdXdkpw/bSSYxIeq6n8kuWFVHZLZCjX/MnEmmMJ3qupnc/n7whFJvjltJJiM+TAQN/svmKq6S5KXJdk/yWlJ1iU5Yn7qFBbG/KGYv57kgUkqs0suX9v+p8iCmT+5/Jgk90jy3SRfTvLY7v7qpMFgAubDWBSZBTS/hOb2mf3wdkZ3XzxxJFh1VfWIJCd294+nzgJTqqo13X1pVe2SZDsr97HIzIexuLRswVTVZ5L89yQ/6u7TlBgW2OFJvlBV/7uqHjIv+LCIvlxVxyS5W5ILpw4DEzMfBqLILJ7DM7tp7fiqOqmqfq+qbj11KFht3f3kJD+X2b0xj0nypap67bSpYBK3T/LezFby+3JVvbyq7jVxJpiK+TAQl5YtsKraN8nzMrv2c83UeWAKVbVDkkOTPDnJf+nudRNHgslU1U2T/E28L4D5MABnZBZQVe1dVf89yXFJ7pDZpWawUKrq0Kp6fZIzkxyR5LVJbjFpKJhIVd2nql6Z5FNJdkryyIkjwWTMh3E4I7Ngqurfk+yQ2eU0/9jdZ00cCSZRVcdlVubf5YZ/FllVfTnJKUmOT3JCd/9g4kgwGfNhLIrMgqmqO3T356fOAcC2oapu3N3fnzoHbAvMh7EoMgumqm6Q5FeS7J3kJ6s0dfcfT5UJplBVv5zkxUlultlS5JWku/vGkwaDVVZV65I8NVd+X3jKVJlgKubDWCw3unjenuT8JCcncTkNi+wvkjy0uz83dRCY2NuTfCSzlZounTgLTM18GIgzMgumqk7r7v2nzgFTq6qPdfc9p84BU6uqU7r7zlPngG2B+TAWq5Ytnn+rqp+fOgRsAzZU1T9W1aOr6pc3fUwdCibwjqp68NQhYBthPgzEGZkFU1WfzewhgF/O7NKyTfcFHDBpMFhlVfW6ZYbbddAsmqq6IMkuSS5KcvF82P1iLCTzYSyKzIKpqtssN97dX13tLAAAcG0pMguoqg5M8l/mmx/p7s9MmQemUFV7JfnbJPdM0kk+muRZ3b1x0mAwgao6PMm955sf7O53TJkHpmQ+jMM9Mgumqp6V5E2ZLTl7syRvrKpnTJsKJvG6JCckuWWSPZP8y3wMFkpVvSjJs5J8dv7xrPkYLBzzYSzOyCyYqjo1yd03Pam2qnZJ8nH3yLBolluZxmo1LKL5+8Kdu/uy+faaJJ/2vsAiMh/G4ozM4qlccV30S+djsGi+U1WPq6o184/HJTl36lAwkZsseb3bZClg22A+DMIDMRfP65L8e1X983z74Un+bsI8MJWnJHl5kr/K7B6Zf5uPwaL58ySfrqoPZPaLrXsnee60kWAy5sNAXFq2gKrq4CT3ymyCfri7Pz1xJAAmVFW3SPILmb0v/Ht3/8fEkWAy5sM4FJkFUVW/kGRtd79rs/HDk3yju0+eJhmsrqr6iyRndffRm43/TpKbd/dzpkkGq6uqHpRk1+5+y2bjj03y7e7+12mSweozH8akyCyIqvpgkid191c2G/+5JMd09/2nyAWrbf5Q2P033ci5ZHy7JKd29/7TJIPVVVWfSPLQ7j5ns/GbJ/nn7r77NMlg9ZkPY3Kz/+LYY/MSkyTdfWaSPVY/DkymNy8x88HLYuELFsvOm//QliTzy2h2mSAPTMl8GJAiszhueDX7TFAWyX9W1b6bD87HfjhBHpjKTlV1pUV/qmqHXP17BlwfmQ8DUmQWx3ur6s+q6gq/ca6qFyZ5/0SZYArPT/KuqnpSVf38/OPJSd453weL4p+SvGb+PLEkP3m22NHzfbBIzIcBuUdmQcwn42uT3DXJKfPhA5NsSPIb3X3hVNlgtVXV/kl+P8mm+2FOS/KX3f3/pksFq2v+2+c/TfIbSb46H751ZkvyP6+7L54qG6w282FMisyCqarbJrnTfPP07j5ryjwATKuqbpjk5+abZ3a3SyxZWObDWBQZAABgOO6RAQAAhqPIAAAAw7nSMnNc/1XVvZLs292vq6p1SW7U3V+eOhespqq6XZJXJfmZ7t6/qg5Icnh3/+nE0WBVzVezfGyS23b3H1fVrZPcvLs/OXE0WDVVdfDV7e/uT61WFlbOPTILpqr+KMn6JLfv7ttV1S2T/J/uvufE0WBVVdWHMlu57NXdfdB87LTu3v/qPxOuX6rqVUkuS3L/7r5jVd00yXu6+xcmjgarpqo+cDW7u7vvv2phWDFnZBbPI5IclORTSdLdZ1fVrtNGgkns3N2f3OzRSpdMFQYm9IvdfXBVfTpJuvu7VbXj1KFgNXX3/abOwDWnyCyei7q7q6qTnzxfBhbRd6rqZ5NsmgtHJPnmtJFgEhdX1ZpcPhfWZXaGBhbS/Flj+yXZadNYd79hukRcFUVm8RxfVa9OcpOqemqSpyR5zcSZYAq/neSYJHeoqm8k+XKSx00bCSbxsiT/nORmVfVnSY5I8ofTRoJpzC/Bv29mRebEJIcl+WgSRWYb5B6ZBVRVhyR5YJJK8u7u/teJI8Fk5mclt+vuC6bOAlOpqjskeUBm7wvv6+7PTRwJJlFV/y/JgUk+3d0HVtXPJHltdz904mgsQ5EBFlJV/e4yw+cnObm7T1ntPDCVqtp9meELuvviVQ8DE6uqT3b3Xavq5CT3S3JBktO6+04TR2MZLi1bMFV1QebXQS9xfpINSf5bd5+1+qlgEuvnH/8y335IkpOSPK2q/k93/8VkyWB1fSrJrZJ8N7MzMjdJ8s2q+naSp3b3yVOGg1W2oapuktll9ycnuTCJpci3Uc7ILJiqemGSs5O8ObM3rEcluXmSM5L8Znffd7p0sHqq6t1JfqW7L5xv3yjJWzJb2e/k7t5vynywWqrq6CT/3N3vnm8/MMmhSY5P8jfd/YtT5oOpVNXeSW7c3adOHIWrsN3UAVh1h3b3q7v7gu7+fncfk+TB3f2PSW46dThYRbdOctGS7YuT3Ka7f5jkx9NEgkms31RikqS735Pk3t39iSQ3mC4WrL6qet+m1939le4+dekY2xaXli2ey6rqkZn95jmZrU6zidNzLJI3J/lEVb19vv3QJP8wv/n/s9PFglV3XlU9J8lx8+1fS/Ld+ZLMlmFmIVTVTkl2TrJ2/lDYTQ8Zu3GSW04WjKvl0rIFU1W3TfI3Se6eWXH5RJLfSfKNJHfp7o9OGA9WVVWtT3LPzN6wPtrdGyaOBKuuqtYm+aMk98p8LiR5YWb3T966u8+cMB6siqp6VpJnZ1Zazl6y6/tJXtPdL58kGFdLkQEWWlXdLFd86NnXJowDwISq6hnd/bdT52BlFJkFMz91+utJ7pQr/vD2lMlCwQSq6vAk/yuz3759O7N7Zj5viU0WTVWtS/Lfc+X3hftPFgomUlU7JnlaknvPhz6Y5NWWI982udl/8fzvzFYpe1CSDyXZK7M10mHR/EmSuyX5Qnfvk+SXknxs2kgwiTcl+XySfTK7pOwrmS1FDovolUnuMv/nptevmjQRV8kZmQVTVZ/u7oOq6tTuPqCqdkjybr95Y9FU1YbuXl9Vn0lyUHdftulBaFNng9VUVSd39102vS/Mxz7U3feZOhuslqravrsvqarPdPeBm+270hjbBquWLZ5Np0a/V1X7J/mPJHtPFwcm8735s2M+nORN84f/XTJxJpjCpveFb1bVQzK70XmvCfPAFD6Z5OAkl1bVz3b3l5KfLJJ06aTJuEqKzOI5Zr6s4B8mOSHJjZI8b9pIMImHJflhZqv2PTbJbpldVgOL5k+rarck/y3J32a23Oyzp40Eq27Tcsu/l+QDVXXWfHvvJE+eJBFb5NKyBVNV+3T3l7c0Btd3VfXi7n7Olsbg+q6q7tndH9vSGFyfVdXGJC+db94wyZokP8hsAYwfdvdLr+pzmY6b/RfPW5cZe8syY3B9d8gyY4etegqY3nJLzVp+lkWzJrOrVHbN7Iqlmm9vPx9jG+TSsgVRVXfIbGnN3arql5fsunGWLLcJ13dV9ZtJfivJbavq1E3Dmb1h+Q00C6Oq7p7kHknWVdXvLtl148x+qINF8s3u/uOpQ3DNKDKL4/ZJ/r8kN0ny0CXjFyR56iSJYBpvTvKuJH+e5Kgl4xd093nTRIJJ7Jjlf+P8/SRHTJIIplNbPoRtjXtkFkxV3b27Pz51DphKVe2c5OJNDzerqtsneXCSr3b3P00aDiZQVbfp7q/OX980yffaDwcsmKra3S+zxuMemQVRVU+tqn27++M1c2xVnV9Vp1bVwVPng1X0fzNfcryqfi7Jx5PcNslvV9WLJswFq6qqnl9Vd+jur1bVDarq/Um+lORbVfVLU+eD1aTEjEmRWRzPyuxpzUny6CQHZvbD2+8m+ZuJMsEUbtrdX5y/fmKSf+juZ2R2o/9DposFq+7Xkpwxf/3EzH4mWJfkPkn+51ShAFZKkVkcl2y6lCaze2Xe0N3ndvd7k+wyYS5YbUsvmbl/kn9Nku6+KMllkySCaVy05BKyB2VW6i/t7s/FPbTAABSZxXFZVd2iqnZK8oAk712y74YTZYIpnFpVf1lVv5Pk55K8J0mq6ibTxoJV9+Oq2r+q1iW5X+ZzYW7niTIBrJgiszien2RDZpeXndDdpydJVd0nyVlX83lwffPUJN/J7D6ZB3b3f87H90vyl1OFggk8K7PniH0+yV9tejByVT04yaenDAawElYtWyBVtX2SXbv7u0vGdsnsv4MLp0sGAADXjCIDAAAMx6VlAADAcBQZAABgOJZXXDBV9b7ufsCWxuD6qqr+JVdcgvkKuvvwVYwDk6mqX766/d39T6uVBeDaUGQWxHzZ5Z2TrK2qmyap+a4bJ7nlZMFg9W1ameyXk9w8yRvn24/O5Q+NhUXw0KvZ10kUGWCb5mb/BVFVz0ry7MxKyzdyeZH5fpLXdPfLp8oGU6iq/7+9uwu1rKzjOP79OYNpjBGl2BiU00QvZk5pUsRkRC8EAwYleRGI0EVXQQbdRBcVRUV01UUpBmFEkI1DUpYS5ZjdzMWovb8RFYVRFuhhksb018VeZzpIdfbxYj9nt74fOKy1ns2B38U5e+//ev7Ps+5pe9V2Y5IkaXeykJmZJO9t+9nROaTRkvwMONL2N9P1AeCOti8dm0xavSRHgJcB52yOtf3ouESStD1by+bnT0nOa7uR5EPA5cDH2p4cHUxasRuAu5NsPhD2YuA94+JIYyT5PIvW4zcANwPXACeGhpKkJTgjMzNJftj2siSHgU+wWC/wwbavHhxNWrkkTwNeMl3+vO0/RuaRRtjyubB53Afc1vYto7NJ0v/i9svz8/h0PAJ8ru3XgbMH5pFGuoJFO80h4Nok1w3OI43w6HT8e5KLgMeAAwPzSNJSbC2bnz8muRF4E/Cp6Y60Ba1mJ8mXgIPA/fy7wC9wy7BQ0hjfSPJM4NPASRb/BzePjSRJ27O1bGaSPB14K/Cjtr9Ksh94edu7BkeTVmpa7H9JfROUzphubp3T9uHRWSRpO87IzEiSs4ATbS/dHGv7IPDguFTSMD9m8RwZ//41e0ley2LDi73TNW2dnZS0q1nIzEjbJ5I8kOR5bX8/Oo802PnAT5OcAM4s8m979bhI0urZZilpXVnIzM9+4CfTl7dTm4N+edMMfXh0AGmXeBW2WUpaQxYy8/OR0QGk3aDt8SQXAldOQyfa/nlkJmkQ2ywlrSUX+0uapSTvZLFL091AgNcBH2j7tZG5pFVL8j3gFSwegmmbpaS1YSEzE0nubXs4yQaL3uczLwFt+4xB0aQhkjwAvHlzFibJBcB32h4am0xarSSv/0/jbY+vOosk7YStZfPxLoC2540OIu0SZz2pleyv+EwlzZAFi6R15Yf2fBzbPElydGQQaZf4dpI7k1yf5Hrgm8C3BmeSVibJvdNxI8kjW342kjwyOp8kbcfWsplIcl/bVz75XJqzJG8HDrNosbyn7bFtfkX6v5Hk+W1/NzqHJD1VtpbNR//LuTRLSQ4Ad7S9bbo+N8nFbX87Npm0MseAy2ExU9/2HYPzSNKO2Fo2H4c2WwaAy2whkLgVeGLL9ePTmDQX2XL+gmEpJOkpckZmJtruGZ1B2mX2tj29edH2dJKzRwaSVsyZeklrzUJG0lz9JcnVbW8HSPI24KHBmaRVOjTNyAc4d8vsvNvyS1oLLvaXNEtJDgJfBp7L4m70H4Dr2v56aDBJkrQUCxlJs5ZkH4v3wo3RWSRJ0vJc7C9plpJcmOQLwK1tN5JckuTdo3NJkqTlWMhImqsvAncCF03XvwTeNyyNJEnaEQsZSXN1ftuvMm3B3PafLLZgliRJa8BCRtJcnUrybKZtZ5O8Bnh4bCRJkrQst1+WNFfvB24HDib5AXABcM3YSJIkaVnuWiZptpLsBV7M4rkZv2j72OBIkiRpSbaWSZqVJFcmeQ6cWRdzBfBx4DNJnjU0nCRJWpqFjKS5uRE4DZDkKuCTwC0s1sfcNDCXJEnaAdfISJqbPW3/Np1fC9zU9ihwNMn9A3NJkqQdcEZG0tzsmdbGALwR+O6W17y5I0nSmvBDW9LcfAU4nuQh4FHg+wBJXojbL0uStDbctUzS7EzPjNkP3NX21DT2ImBf25NDw0mSpKVYyEiSJElaO66RkSRJkrR2LGQkSZIkrR0LGUmSJElrx0JGkiRJ0tqxkJEkSZK0dv4FSEc7Le+ov9oAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 1008x576 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"chartDF = pd.DataFrame(data = [[conversionControl, conversionTest],\n",
" [conversionControlSecond, conversionTestSecond],\n",
" [conversionControlFinal, conversionTestFinal],\n",
" [conversionControlTotal, conversionControlTotal]\n",
" ],\n",
" index = ['First Stage Conversion Rate', \n",
" 'Second Stage Conversion Rate', \n",
" 'Final Stage Conversion Rate', \n",
" 'Total Conversion Rate'\n",
" ], \n",
" columns = ['control', 'test']\n",
" )\n",
"chartDF.plot.bar(figsize=(14, 8))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The analysis above shows that the quality of conversions could be a concern. While the test group had a higher conversion rate in the first checkout stage, leading to the test group having relatively more customers entering the checkout funnel than the control group, conversions at the second and final checkout stages were slightly lower, resulting in a muted improvement on sales. Though the client is mainly interested in increasing conversion at the first step of the checkout funnel, given the cost of $2 per look-up of the new feature, **we should be concerned about the cost-benefit of the feature as it does not seem to improve conversions on actual sales**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Statistical Testing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The section below will detail the statistical test used to analyze the experiment results."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Statistical Framework\n",
"Assuming a form load follow a Bernoulli distribution defined by a conversion rate (or conversion probability) $p$:\n",
"$$\n",
"X \\sim Bernoulli(p)\n",
"$$\n",
"Then, the conversion rate follows a normal distribution, where:\n",
"$$\n",
"\\hat{p} \\sim N \\left ( \\mu=p, \\sigma = \\sqrt{\\frac{p(1-p)}{n}} \\right )\n",
"$$\n",
"\n",
"The goal of the 5-field form is to generate a lift defined as the difference between the conversion rates between the test and control group:\n",
"$$\n",
"d = p_t - p_c\n",
"$$\n",
"where $d$ is lift of the test group over the control group, $p_t$ is the conversion rate of the test group (5-field form), and $p_c$ is the conversion rate of the control group (6-field form). \n",
"\n",
"The conversion rate, $p$, is defined as the ratio of completed forms, $X$, over total form loads, $n$:\n",
"\n",
"$$\n",
"p = \\frac{x}{n}\n",
"$$\n",
"\n",
"\n",
"To statistically analyze the lift, we conduct a hypothesis test, where the null hypothesis is that there is no lift and the alternative hypothesis is that the lift is greater than 0 to see if our results are significant. The formalized hypotheses are the following:\n",
"\n",
"$$ \n",
"H_0 : d = 0\n",
"$$\n",
"\n",
"$$\n",
"H_A : d > 0\n",
"$$\n",
"\n",
"As $p_t$ and $p_c$ are normally distributed, independent, and have the same variance, $d$ is normally distributed:\n",
"$$\n",
"\\hat{d} \\sim N \\left ( \\mu=d, \\sigma = \\sqrt{ p_{pool}(1-p_{pool}) \\left ( \\frac{1}{n_c}+\\frac{1}{n_t} \\right ) } \\right )\n",
"$$\n",
"where $p_{pool} = \\frac{x_c+x_t}{n_c+n_t}$.\n",
"\n",
"\n",
"Given the normal distribution of $d$, we can calculate a z test statistic for the hypothesis test:\n",
"$$\n",
"z = \\frac{\\hat{d}-0}{\\sqrt{ p_{pool}(1-p_{pool}) \\left ( \\frac{1}{n_c}+\\frac{1}{n_t} \\right ) }}\n",
"$$\n",
"For the hypothesis test, we assume a significance level ($\\alpha$) of 0.05. This means that we are willing to tolerate Type I errors, or false positive results, 5% of the time. If the p-value, or the probability of observing lift as or more extreme than the observed lift, is smaller than our sigificance level, we conclude that our results are statistically significant."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Power Analysis\n",
"Before we began the experiment, we should have first considered the power of our experiment.\n",
"\n",
"Power is the defined as the probability of rejecting the null hypothesis, given that it is false. Normally, power is calculated as the following:\n",
"\n",
"$$\n",
"Power = 1 - P(Type II error)\n",
"$$\n",
"\n",
"In the context of our desired experiment on form loads, power is the probability of correctly concluding that our new test feature creates a lift in the conversion rate compared to control.\n",
"\n",
"One of the harder to grasp concepts related to power is that **power is determined ex-ante to analyzing the actual results of the experiment and conditional on some counterfactual true effect and variance that is not observed, and the probability of Type I error ($\\alpha$)**. That is, we never know the 'true' power of our experiment as we will never know the true parameters of the population. If we can only determine ex-ante power, why is power analysis useful?\n",
"\n",
"Ex-ante power analysis is useful because before you even start the experiment and observe any results, power analysis tells you if your experiment design is 'good enough' for what you setting out to accomplish. The most typical application for power-analysis in A/B testing is in the context of sample size determination. Given that we believe that our new feature is going to result in some lift, for example 3 percentage points, is our planned sample size large enough to detect such a lift. The larger the sample size, the more likely would our experiment be able to detect this counterfactual lift. It is standard to assume $\\alpha=0.05$ and $power=0.80$ to be acceptable for an experiment, but these values can change depending on the preferences of the experimenter."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the example below, we calculate a power of 0.38 assuming that the baseline conversion rate of 52.5% is the baseline conversion rate, the true lift is 3 percentage points, 1000 observations per split in the experiment, and $\\alpha=0.05$. This means that even if the true lift is 3%, the null hypothesis will only be rejected the null hypothesis 38% of the time if we only collect 1000 observations per group. To achieve a power of 0.8 we must increase the sample size of our experiment.\n",
"\n",
"The chart also illustrates how power is affected by the effect of the alternative hypothesis and $\\alpha$. If the alternative hypothesis effect is larger and further away from the null hypothesis, the power of our test increases. If the $\\alpha$, or probability of type 1 error increases, our power increases as we are more likely to reject the null hypothesis."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzgAAAGDCAYAAADnHrUdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZyU1ZX/8c9lbRZZZJNNlijI3kILSLOoCEIjEJcoRqNZjDpq8ktmxonOZJzMZDIxM5kZZ5IYYxLNgkZUQKWbblDcUFyAkVUWWRpZRLDZEZCG+/vjVEnT9FJFV9XzVNX3/XrdVHfV89Rzursideqee67z3iMiIiIiIpIJ6gUdgIiIiIiISKIowRERERERkYyhBEdERERERDKGEhwREREREckYSnBERERERCRjKMEREREREZGMoQRHRCQgzrmvO+feTPSxIiIi2UwJjoiIxMw5933n3E7n3H7n3OPOucY1HJvrnFvqnPsscptb4bGvO+dOOOcOVRiXpeSHCDnn3Fedc1ucc4edc887586t4dhc59zCyN9jm3PuwUqPf8c5t9k5d8A5t8Q5NzL5P4GISLCU4IiISEycc1cB9wNjge5AT+Cfqzm2EfACMB1oDfwReCFyf9Tb3vvmFcZrSQw/aZxzDRL4XP2A3wBfAzoAnwGP1HDKU8AbwLnAGOCvnHNTIs81DHgIuB5oCfwemO2cq5+oeEVEwkgJjohIEjnn7nfObXTOHXTOfeCcu6aGY71z7rvOuU3OuU+dc//hnKtX6ZifO+f2Rj6Vn1jh/m8459ZErrPJOXdnEn6c24Dfe+9Xe+/3Aj8Gvl7NsZcBDYCHvffHvPf/CzjgiroG4Zy7LDJb8feR31Opc+7mCo+3dM79yTm3OzIT8sPo7zHy/ZDI17dEfud9I9/f7px7PvJ1vQp/uzLn3DPRmRTnXPfIed9yzn0EvFLXn6mCm4E53vs3vPeHgH8ErnXOnVPN8d2BJ733J7z3G4E3gX4VHlvtvV/qvffAn4C2QPsExisiEjpKcEREkmsjMAr7BP2fgenOuY41HH8NkAcMBqYC36zw2DBgHfYm9d+B3zvnXOSxXcDVQAvgG8B/O+cGV3UB59xI59y+GkZ1ZUz9gOUVvl8OdHDOtanm2BWRN9ZRKzj15hvg4kiCst45949xzoSch/0eOmOJ12POud6Rx36B/b57YrMat2K/E4DXseQLYDSwKXJM9PvXI19/F/hy5LFOwF7gV5ViGAP0Aa6qHJxz7vxafsdfrebnOu13HElaPgd6VXP8w8CtzrmGkZ//UuDlyGPFQH3n3LDIrM03gWXAzmqeS0QkIyjBERFJIu/9s977Hd77k977GcCHwNAaTvmZ936P9/4j7M3rTRUe2+K9/633/gRW8tURK2PCe1/kvd/ozevAfCyxqiqmN733rWoY1TUzaA7sr/B99OuqZhcqHxs9PnrsG0B/bDbhusjPeV81163OP0Zmh14HioAbIm/kbwQe8N4f9N6XAv+JlXyBJTDRhGYU8NMK34/hVIJzJ/AP3vtt3vtjwI+A6yslYT/y3h/23h+pHJj3/qNafsdPVfMz1fZ7q6wQK0E7AqzFZtgWRx47CMzEZnWOAf8E3FEp6RQRyThKcEREksg5d6tzbln0k3vsTX3bGk7ZWuHrLdjsQdQXn7x77z+LfNk8cp2Jzrl3nHN7ItcpqOU6Z+MQNkMUFf36YAzHRo8/COC93+S93xxJ/FYC/4K9UY/VXu/94QrfR39XbYFGke8rPtY58vXrwCjn3HlAfWAGkO+c647N+iyLHNcNW68S/butAU4QSSgjKv6tEqXG31tFkZK5Eux3lwN0Ba5yzt0dOeR2bNamH/Y7uQUodM51qvxcIiKZRAmOiEiSOOe6Ab8F7gXaeO9bAauwtSjV6Vrh6/OBHTFcpzH2Sf3PgQ6R68yt7jrOuVHu9O5llUeVMz/AamBQhe8HAZ9478uqOXZghRI6gIGR+6viq4u3Gq2dc80qfB/9XX0KHMcSlIqPbQfw3m/AFu5/F3jDe38QSxzvAN703p+MnLMVmFhp1iXHe7+9UsxVipSo1fQ7vrmaU0/7HTvnegKNgfVVHNsTOOG9/5P3vtx7vw14GktuiTzPHO/9+kgiWQJ8DIyoLm4RkUygBEdEJHmaYW+Cd4M1AsBmcGpyn3OutXOuK/D/sBmG2jTC3gTvBsojzQfGV3ew935hpe5llcfCak79E/At51xf51xr4IfAH6o59jVsxuO7zrnGzrl7I/e/Al/MOHWIfH0Rtpj+hejJzrk/OOeqe+6of3bONYokZFcDz0bK954BfuKcOyeSZP411s0t6nUs6YyWo71W6XuARyPP0S0STzvn3NRa4vlCpEStpt/xk9Wc+iQwOZKENsNmZ2ZFErHK1lto7quRpgjnYeV50TU8i4FJzrmezozD1vKsivXnEBFJR0pwRESSxHv/Abb+423gE2AA8FYtp70ALMVKpYqw1r61XecgNiPxDLYY/qvAi2cdePXXKcGaG7yKlX1twdZ1AOCcK3bO/X3k2M+xRfq3AvuwUqkvR+4HazW9wjl3GJttmgX8W4XLdaXm39VO7GfdgSUFd3nv10Ye+w5wGGsg8CbWSvnxCue+jq1peaOa7wH+B/sdznfOHQTewZo8JJX3fjVwF/Yz7YrEFS05wzn3qHPu0cixB4Brge9jv4tlWPLyk8jhf8JmdF4DDgD/C9xZ4fckIpKRnNYaioiEg3POAxdGyqiylrO9cpYDA733x6t4/DJguve+S6pjExGR8EvY5mQiIiKJEJnl6RN0HCIikp5UoiYiIiIiIhlDJWoiIiIiIpIxNIMjIiIiIiIZQwmOiIiIiIhkjFA2GWjbtq3v3r170GGISIyila4unm0aRfTCERGROli6dOmn3vt2le8PZYLTvXt3lixZEnQYIiIiIiISUs65LVXdrxI1EamzxYttSN1t376d7du3Bx1GauiFIyIiSaAER0TqbPVqG1J3u3fvZvfu3UGHkRp64YiISBIowRERERERkYwRyjU4IiIiIiLp6vjx42zbto2jR48GHUpGyMnJoUuXLjRs2DCm45XgiIiIiIgk0LZt2zjnnHPo3r07Tp0i68R7T1lZGdu2baNHjx4xnaMSNRERERGRBDp69Cht2rRRcpMAzjnatGkT12yYZnBEpM6+/vWgI8gcubm5QYeQOnrhiEgGU3KTOPH+LjWDIyIiIiIiGSOmBMc5N8E5t845t8E5d38Nx13inDvhnLs+3nNFJH0tWmRD6m7r1q1s3bo16DBSQy8cERFJgloTHOdcfeBXwESgL3CTc65vNcf9DJgX77kikt7Wr7chdVdWVkZZWVnQYaSGXjgiIkn1m9/8hrvvvvu0+/r168fatWtrPXfEiBExXWPfvn088sgjZxVfssQygzMU2OC93+S9/xx4GphaxXHfAWYCu87iXBERERERSaAVK1Zw8cUXf/H90aNH+eijj7jwwgtrPXdRjDPsNSU43ntOnjwZW7AJFEuC0xmoWC+xLXLfF5xznYFrgEfjPVdE0tfJk1BeHnQUkpaOHoVNm+D48aAjERHJWCtXrmTw4MGnfd+rVy/q169f67nNmzcH4PDhw0yaNIlBgwbRv39/ZsyYcdpx999/Pxs3biQ3N5f77ruP0tJS+vTpw913383gwYNZuHAh/fv3/+L4n//85/zoRz8CYPr06QwdOpTc3FzuvPNOTpw4kYCfOrYualW1LfCVvn8Y+IH3/kSlLgexnGsHOncHcAfA+eefH0NYIhKk4mL4q7+CnTuhQwfo0gU+/RQGDLDRsSOogYyc4bPP4LHH4D/+A3bsgJwcePNNuPFGuPJKaNQo6AhFRBLre9+DZcsS+5y5ufDww7Uetnr1aq699tovupAdOnSIq6++Oq5LlZSU0KlTJ4qKigDYv3//aY8/9NBDrFq1imWRn7G0tJR169bxxBNP8Mgjj1BaWlrl865Zs4YZM2bw1ltv0bBhQ+6++26efPJJbr311rjiq0osMzjbgK4Vvu8C7Kh0TB7wtHOuFLgeeMQ59+UYzwXAe/+Y9z7Pe5/Xrl27GMMXkVT79FO45RYoKICmTeHee6FlS/jgA7jvPpgwATp3hrZt4RvfsA/qJXb16tWjXr0MbHB58CA89BB07w7f/z6cd569WC64AGbNgkmToH17e9GUlGhmR0SkjrZu3Uq7du3YsmULpaWllJaWMm3aNAYOHMimTZv41re+xfXXX1/r8wwYMICXX36ZH/zgByxcuJCWLVvWek63bt0YPnx4jccsWLCApUuXcskll5Cbm8uCBQvYtGlTzD9fTWKZwVkMXOic6wFsB6YBX614gPf+i21FnXN/AAq998875xrUdq6IpAfv4S9/gf/3/2D/fvinf4IHHoDGjU8dU1YGK1faWLIE/vAHm+GZPds+qJfaDRw4MOgQEmvvXvjFL+yTxr17YehQ+Lu/g+HDoUkTO+boUXjjDXjlFXj2WXvhnHsuPPmkJUEiIukshpmWZFixYgX9+vU77b4PPviAqVOn0rNnT37/+9/HlOD06tWLpUuXMnfuXB544AHGjx/Pgw8+WOM5zZo1++LrBg0anLYOJ7php/ee2267jZ/+9Kfx/FgxqTXB8d6XO+fuxbqj1Qce996vds7dFXm88rqbWs9NTOgikioffWTlaHPnwrBh8LvfQYVy2i+0aQOXXWYDYNQouOMOmDIFXnjh1PtZyRLPPQff+hYcOAD5+fCVr0Be3pnZbk4OjB9v48gReP11e5Fdd519nZcXTPwiImls5cqV9O17evPi1atXx/1B2o4dOzj33HO55ZZbaN68OX/4wx9Oe/ycc87h4MGD1Z7foUMHdu3aRVlZGc2bN6ewsJAJEyYwduxYpk6dyve//33at2/Pnj17OHjwIN26dYsrvqrEMoOD934uMLfSfVUmNt77r9d2roikj9/+Fv76r62hwMMPW0la5bWJr79ut2PGnH7/7bfbsd/6FkyeDC++aGVtUr1orXL37t0DjaPOli+HW2+FHj3g7rvh4otPn+4DWLrUbocMOXVfkyY2azNkiJ1fUACLF0MC/sETEckmK1euPG29zZ49e/De06FDh7if57777qNevXo0bNiQX//616c93qZNG/Lz8+nfvz8TJ07knnvuOe3xhg0b8uCDDzJs2DB69OjBRRddBEDfvn3513/9V8aPH8/Jkydp2LAhv/rVrxKS4Djvq1zzH6i8vDy/ZMmSoMMQyXpFRXD11bb2+7e/teUTVYl+mPP1r1f9+J/+ZI+NGQOFhVBh5loqiS7SzM3NDTiSOti712ZdDh2yrLhXr6qPmzPHbidPrvrxDz+0LLljR3jvPWjdOjnxiogk2Jo1a+jTp0/QYVSrrKyMf/iHf+Cll17i9ttv54EHHgg6pFpV9Tt1zi313p8xzR/TDI6IZJ9du+Cb37SOaIWFZ374Ho9bb7WZnOgH8kVFEOk+KZnm5En7Q3/0Efz0pxDDXgvVuvBC67b23e/aC+e11+r2QhQREcBmXR59tNpVJmkvA1v1iEhdeQ/f/jbs22frvBPxnvLmm+253noLJk60plqSgf7t3ywjvv12W4RV117hQ4fCD38I77wDX/2qJVAiIiI1UIIjImf4/e9tvcxDD9kMTqJMm2ad2N5+25IcdQLOMPPmwYMPWk3jTTdBgwQVCVx9Ndx5p7WT/pu/ScxziohIxlKCIyKn2bDB9iQbO9ZaQseiSZPYO6R95Svwxz/aTE6ldYqCtdNskKjEIJVKS22G5UtfspKyWBZaNW4c+/Tg7bdbO76HH4b/+Z86hSoiIplNTQZE5Avl5TByJKxbZ3vZdOmSnOt4bx2Bly61deRt2iTnOpIiR4/aC2f9eviv/7KOaclQXm7Z93vv2RTjpEnJuY6ISB2FvclAOoqnyYBmcETkCz/5Cbz7Ljz6aPKSG7BlGf/936c2DJU0953vWLb6ve/BoEHJu06DBtZ0oHNnuOceS3hEREQqUYIjIoAlNj/+MdxyC9x4Y3znvvyyjXj07w933WXJ1Gpt//uFTZs2sWnTpqDDiN0TT9imnDfdZPvX1Ivjn5V337URjyZNrHZyyxb41a/iO1dERLKCEhwR4dAhS2w6d4Zf/jL+87dtsxGvf/kXaNECvv99K1sTOHDgAAcOHAg6jNgcPgw/+AHk5tpurvG229u1y0a8xoyBvn1tyvGzz+I/X0REMpoSHBHhr/8aNm6EP/8ZWrZM3XXbtIEf/QheeunUno+SRn79a9i9G772NWjVKnXXdc5mcXbvhp/9LHXXFRGRtKAERyTLvfMO/Pa3cN99MHp06q//V38FffpY999jx1J/fTlLhw/Dv/+77VMzbFjqrz9kCAwfbou59uxJ/fVFRCS0lOCIZLmf/hTOPRf+8R+DuX7DhtZ4a8MG+MUvgolBzkJ09ubGGyEnJ5gYvvMd2zH2Rz8K5voiIiH3m9/8hrvvvvu0+/r168fatWtrPXfEiBExXWPfvn088sgjZxVfsijBEcliq1dbt93vfheaNz/752nRwsbZmjABCgqsycHZLMnIJI0bN6ZxvGtZUi1RszfNmsW2X051eveGcePgscfgo4/O/nlERDLUihUruLhC6/6jR4/y0UcfceGFF9Z67qJFi2K6Rk0JjveekydPxhZsAinBEcli//7v0LQp3Htv3Z7n2mtt1MV//ZetF//hD+v2POmuT58+4d874ZFHEjN7c8UVNuoi2i76gQfq9jwiIhlo5cqVDB48+LTve/XqRf369Ws9t3nkk8/Dhw8zadIkBg0aRP/+/ZkxY8Zpx91///1s3LiR3Nxc7rvvPkpLS+nTpw933303gwcPZuHChfTv3/+L43/+85/zo8jM+/Tp0xk6dCi5ubnceeednDhxIgE/NaThdtkikghbtsBTT1lyE4aNNnv3toqjhx+2dTnJ2itS6ijotTeVdekC11wDTz8Nf//30K9f0BGJiJzme9+DZcsS+5y5ufbvZW1Wr17Ntddei3MOgEOHDnH11VfHda2SkhI6depEUVERAPv37z/t8YceeohVq1axLPJDlpaWsm7dOp544gkeeeQRSktLq3zeNWvWMGPGDN566y0aNmzI3XffzZNPPsmtt94aV3xV0QyOSJb6z/+0ZlR/8zd1f66SEht19eCDlmx973vZ2zZ6w4YNbNiwIegwqvfII/Dpp4lZe7NokY26+va3oVEj+Lu/q/tziYhkiK1bt9KuXTu2bNlCaWkppaWlTJs2jYEDB3L48GFuu+02vv3tb/Pkk0/W+DwDBgzg5Zdf5gc/+AELFy6kZQztVrt168bw4cNrPGbBggUsXbqUSy65hNzcXBYsWJCwfeA0gyOShXbvtr0Zb7nFPgCvq5076/4cYJ2Gf/xjm8GZPx+uuioxz5tODh06FHQI1Uv07E1ZWd2fAywrvukm23R00SKIcWGsiEgqxDLTkgwrVqygX6VZ7Q8++ICpU6cya9Ysrr/+eiZPnsyNN97IzTffXO3z9OrVi6VLlzJ37lweeOABxo8fz4MPPljjtZtVWF/ZoEGD09bhHD16FLD1Obfddhs//elPz+bHq5FmcESy0P/+Lxw9Gs4PvL/xDWjfXpvUh1IiZ28S7bbbrNPFffcFHYmISCisXLmSvn37nnbf6tWrGThwINu2baNr164Ata7H2bFjB02bNuWWW27hb//2b/m///u/0x4/55xzOHjwYLXnd+jQgV27dlFWVsaxY8coLCwEYOzYsTz33HPsinQX2rNnD1u2bIn756yKEhyRLHPgAPzyl7Zs4aKLgo7mTI0bW8VRYSFUU7YrQQjb2pvKmjeHb37TZnAi/3iKiGSzygnOnj178N7ToUMHunTpwrZt2wBq7XK2cuXKLxoB/OQnP+GHlboBtWnThvz8fPr37899VXzI1LBhQx588EGGDRvG1VdfzUWRNx99+/blX//1Xxk/fjwDBw5k3LhxfPzxx3X9sQFwPoSF7nl5eX7JkiVBhyGSkX7+c/uQ+9137b1qIvzhD3b79a8n5vm2boXu3S3Ohx5KzHOmi+gizdzc3IAjqeQ//sOm/P7zP2HMmMQ855w5djt5cmKe79gx+PKXoVs3WLw4Mc8pInIW1qxZE+qOmIcPH+bee+8lJyeHkSNH1liiFhZV/U6dc0u993mVj9UaHJEscuyYtWO+4orEJTeQ+C5sXbvC1Km2TuhHPwpfNVQyNWnSJOgQzpSs2ZsYFqrGpXFjuOEGq29cvhwGDUrs84uIZIhmzZrxxBNPBB1G0qhETSSL/OlP8PHHid8yZPLkxH0IH3XPPbYG/ZlnEvu8Yde7d2969+4ddBinS9bam9GjbSTS5MlQvz784heJfV4REUkbSnBEssSJE/Yh/JAhMHZs0NHU7oorbI2Qmg0ErLzcpv3y8sK59qaytm0taXr2Wds5VkREso4SHJEsMXMmbNhgszeR/b4SZs6cU8spEsU5uPtueO+97FpOsW7dOtatWxd0GKfMnWt9wAsKEl8r+MYbNhLtuuusm8b06Yl/bhERCT0lOCJZwHtbrN+rl63BTrSyssRtaVLRrbdCs2bZNYtz5MgRjhw5EnQYpzz+uM2KjBqV+Ofev99Gog0dCp06wWOPJf65RURiFMZGXukq3t+lEhyRLPDKK/D++9YEq5Z296HSsiV87Wvw9NO2BERSbOdOa7l8+eWJbwiQTPXqWR/0pUut2YCISIrl5ORQVlamJCcBvPeUlZWRE0cVgbqoiWSBxx+HVq0gDbpAnuGee+DRR+1nCOPGpBntz3+2xVvjxlnSkE4mT7YXzi9/Cb/9bdDRiEiWie4zs3v37qBDyQg5OTl06dIl5uOV4IhkuAMHYPZs2+g9Hdst9+9v2678+tfwN3+TXjNQac17yyoHDIBKO2GnhWizgWeegf/9Xwhj+20RyVgNGzakR48eQYeRtWL6SM45N8E5t845t8E5d38Vj091zq1wzi1zzi1xzo2s8Fipc25l9LFEBi8itXvuOThyxBKcZDnvPBvJcs89UFoKxcXJu0ZYNG/enObNmwcdBrz9Nqxday33kpUZt2mT+E2UKlKzARGRrORqqw10ztUH1gPjgG3AYuAm7/0HFY5pDhz23nvn3EDgGe/9RZHHSoE8733MFfR5eXl+yRLlQiKJMGaM7X2zbl3iu6elyvHj0L07DByYHUlOKNx+O/zlL/DUUxBHWUConDxpO8Z26gT6N0VEJOM455Z67/Mq3x/LDM5QYIP3fpP3/nPgaWBqxQO894f8qUypGaAVVSIhsHmzdeG97bb0TW4AGjaEO+6AkhJrdS1JdugQzJhhndOSOTWXbBWbDaxYEXQ0IiKSIrEkOJ2BrRW+3xa57zTOuWucc2uBIuCbFR7ywHzn3FLn3B11CVZE4vPnP9vt176W3OvMmmUjme64Axo0sLU4mWzNmjWsWbMm2CCefdaSnCuvtF96srzyio1kmjLFFm798pfJvY6IiIRGLAlOVZ/7njFD472fHSlL+zLw4woP5XvvBwMTgXucc6OrvIhzd0TW7yxRxwmRuvMe/vQn6/B7/vnJvdaBAzaSqWNHuPZaW/eeyRvUHzt2jGPHjgUbxOOPQ7dukHfGrH9iHT5sI5kqNhsI0/5CIiKSNLEkONuArhW+7wLsqO5g7/0bwJecc20j3++I3O4CZmMlb1Wd95j3Ps97n9euXbsYwxeR6ixaBBs3Jre5QKrdcw/s22fvVSVJ1q+HN9+EK66AMDQ7SITrrrMNRdVsQEQkK8SS4CwGLnTO9XDONQKmAS9WPMA5d4FzVuHvnBsMNALKnHPNnHPnRO5vBowHViXyBxCRqv3xj9C0qb23yxSjRkHPnrb2XZLk8cetpGvcuPReuFXR0KE2BfjYY0FHIiIiKVBrguO9LwfuBeYBa7AOaaudc3c55+6KHHYdsMo5twz4FXBjpOlAB+BN59xy4D2gyHtfkowfREROOXLE1ohfd13mfAgP9n572jRYsAB27Qo6mgxUXm6Z8dChkEn7N0SbDSxZomYDIiJZIKZ9cLz3c733vbz3X/Le/yRy36Pe+0cjX//Me9/Pe5/rvb/Ue/9m5P5N3vtBkdEveq6IJNeLL9qamFSVp3XpkrpOwtOmwYkTMHNmaq6Xai1atKBFixbBXLykBHbutOYCDRsm/3rt29tIBTUbEBHJGrXugxME7YMjUjcFBbBqlW2OWS+mjzHSh/fQrx+0awevvx50NBnmmmtg4ULb+yaZG3AG5W//FpYvh08+SU0CJyIiSVWXfXBEJI18/DHMmwe33JJ5yQ2cKlNbuBC2bw86mgzyySdQWGht91q3Djqa5JgwAfbu1W6xIiIZLgPf/ohkt6eesg3cU9k9bcYMG6ly4402k/Pss6m7ZqqsWrWKVasC6MUyfbqtwRk3LnWZ8fz5NlJl5EjIyVGXChGRDKcERySDeG9rxIcNg969U3fdI0dSu8VI795w8cXw9NOpu2aqlJeXU15entqLem/d0/r3t/q/VDl2zEaq5ORAfr6tNTp+PHXXFRGRlFKCI5JBli+HlSvh1luDjiT5pk2Dd9+FzZuDjiQDrFgBH3wAl11mSUAmGz/eNlNSmZqISMZSgiOSQf74R2jUyN78Z7obbrDbVJbGZaznnrOytFGjgo4k+fLzLYl76qmgIxERkSRRgiOSIY4fhyefhMmT4dxzg44m+bp3h0svzcwytZSbORMGDYKuXYOOJPlycmwtzrx58PnnQUcjIiJJoARHJEPMmwe7dwdTntajRzD7Qk6bZmV5a9ak/trJ0qpVK1q1apW6C37wgf0Chw+36b9U6tTJRqqNG6cyNRGRDKYERyRDPP20zdxMnJj6a48ZYyPVvvIVaxudSWVq3bt3p3v37qm74MyZ9kscOTJ114waMsRGqqlMTUQkoynBEckAn39uW5hMnZpd+xd27Gjr4p9+2hqByVmYOdO6p3XrFnQkqRMtU5s/X2VqIiIZSAmOSAZ49VXYv982og/C9Ok2gjBtGqxbZ6VqmWDFihWsWLEiNRfbsMF+cZdeGkz3tLlzbQQh2k0tqOuLiEjSKMERyQCzZ0OzZra0IAjl5TaCcO210KBB5jQbOHnyJCdPnkzNxVhuRWgAACAASURBVGbOtNuguqedOGEjCCNGaNNPEZEMpQRHJM2dOAHPPw8FBZm/hUlV2ra1xE5lamfhueegT59gOkQELSfHEjt1UxMRyThKcETS3DvvwCef2ExGtpo2DbZssY0/JUZbtsCSJdY9LRszY7DMeP9+lamJiGQYJTgiaW7WLOvuW1AQdCTBmToVGjfOnDK1lJg1y26zYXPP6kTL1NRNTUQkoyjBEUlj3tv6myuvhBYtgoujVy8bQWnZ0hK8Z54JbklHorRp04Y2bdok/0LPPQcXXmgjKOefbyMo0TI1dVMTEckoSnBE0tjy5bB5c/DlaSNG2AjStGnw8cfw5pvBxlFXXbt2pWvXrsm9yPbtsGiRlac1aZLca9Vk0CAbQYqWqRUVBRuHiIgkjBIckTQ2ezbUqwdTpgQdSfAmTYKmTW0WR2oxe7bdZnN5WpS6qYmIZBwlOCJpbNYse4/arl2wcfzhDzaC1KwZXHUVvPhiendTW7ZsGcuWLUvuRWbOtM5pF12U3OvUZs4cG0FSmZqISMZRgiOSpj78EFatCm5zzzCaMgW2bYP33w86khDbtQveeMM292zaNOhowmH8eJWpiYhkECU4ImkqWmWkBOeUSZPAOZvFkWo8/zycPAkjRwYdSXhceqmtRVKZmohIRlCCI5KmZs+GIUOCbUIVNu3a2ZIKJTg1mDkTunaFvn2DjiQ8tOmniEhGUYIjkoa2b7cNPoPunhZGU6daidrWrUFHEkJ79sArr1j3tGbNgo4mXK64Ag4cgJdeCjoSERGpIyU4ImnohRfsNizlaf362QiDaEe5oNeun6127drRLlldI158EcrLIT/favmC1rOnjTAYPhwaNDi1AaqIiKStBkEHICLxmzXLGmD16RN0JOaSS4KO4JTevW3T0RdfhLvvDjqa+HXu3Dl5T/7cc9CxY/B7z0SFJSsGaN4cBg+Gl1+2NnxhSABFROSsaAZHJM3s2QOvvRae2RuA48dthMWUKVaJdeBA0JHE78SJE5w4cSLxTxwtv7r0UnszHwbl5TbCYvRo+OgjWLEi6EhERKQOlOCIpJk5c+DEiXCtv3nySRthMWWKJVzz5wcdSfxWrlzJypUrE//EJSW2gP7SS8MzO1FcbCMsohufqkxNRCStKcERSTOzZlkTrCFDgo4kvC69FNq0ObVWSYDCQmjVCi6+OOhIwqtzZ9sAtbAw6EhERKQOlOCIpJFDh2xW4pprwvMhfBg1aGB74hQVhasCKjAnTsDcuZYVn3NO0NGE25gxsHw5fPJJ0JGIiMhZiinBcc5NcM6tc85tcM7dX8XjU51zK5xzy5xzS5xzI2M9V0RiV1ICR4+Ga/1NWE2ZAnv3wltvBR1JCLz7LpSVWTeI+vWDjibcRo2yhPD554OOREREzlKtCY5zrj7wK2Ai0Be4yTlXeYe4BcAg730u8E3gd3GcKyIxev55K73SJvS1Gz8eGjXSpp+ATWXVrw9DhwYdSfj17w8tW6q+UUQkjcUygzMU2OC93+S9/xx4Gpha8QDv/SHvvY982wzwsZ4rIrE5ccLWY0+aZCVYYZKbayNMzjkHxo6196lf/NcpDZx33nmcd955iX3SwkIYMAAS/bx11auXjTCpX98+QVi4EI4dCzoaERE5C7EkOJ2BinuCb4vcdxrn3DXOubVAETaLE/O5kfPviJS3Ldm9e3cssYtklXfesRbRkyYFHcmZwpjggJWpbdwIa9cGHUnsEp7gRNse5+XZlFaY9O5tI2xGj7YFby+9FHQkIiJyFmJJcKpaynzG56He+9ne+4uALwM/jufcyPmPee/zvPd5SdvFWySNRauMxo8POpIzffaZjbC5+mq7TacytePHj3M8kZsKFRXZ7bBhiXvORDl61EbYDBtm06SzZwcdiYiInIVYEpxtQNcK33cBdlR3sPf+DeBLzrm28Z4rItUrKrLKmVatgo7kTM88YyNsunSxxmHplOCsXr2a1atXJ+4JCwutr/iFFybuORPlpZfCOUvSvDkMHmyxpVN9o4iIALElOIuBC51zPZxzjYBpwGlvF5xzFzhnTWudc4OBRkBZLOeKSO22brUqozCWp4XdlCnw9tuwa1fQkQTg8GFYsMCyvGbNgo4mvYwebf/HW7486EhERCROtSY43vty4F5gHrAGeMZ7v9o5d5dz7q7IYdcBq5xzy7CuaTd6U+W5yfhBRDLZ3Ll2qwQnflOm2Ifw0UqtrPLKK7ZQfuhQbZwUr9Gj7XbWrGDjEBGRuMXUi8l7PxeYW+m+Ryt8/TPgZ7GeKyLxKSqC7t2hT5+gI0k/gwZZhdYLL8A3vhF0NClWWAhNm1q5lcSnUyfo2dP+z/cv/xJ0NCIiEoeYNvoUkeAcPWpVRpMm6UP4s+GczeLMnw9HjgQdTQpFp62GDAnnwq10MGYMLFsGO3cGHYmIiMRBCY5IyL32mnUoC3N5Wl6ejbCaMsWSmwULgo6kdp06daJTp051f6Lly2H7dvvDhG3jpKg+fcI9LTlqFJw8aTvsiohI2lCCIxJyRUXQpAlcdlnQkVSvf38bYXXZZbbxZzpsTt++fXvat29f9ycqLLTpqzC2h4664AIbYdWvH7RsmR4vHBER+YISHJEQi1YZjR1rSU5Y7d9vI6waNYKrrrJmDWHv+nv06FGOJmJvmMJCmx3p0qXuz5Ushw7ZCKv69a03+5tvWrMGERFJC0pwREJs7VrYvDnc5Wlg+yGGfU/EggLYscPabYfZ2rVrWbt2bd2e5JNP4L33rLlATk5iAkuGV1+1EWajR1sSFsb9ekREpEpKcERCLNrauKAg2DgywYQJdltcHGwcKVFcbFNVYS5PSxfDh0PDhmoXLSKSRpTgiIRYUREMGADnnx90JOmvY0e4+OIsSXAKC6Fdu3AvjEoXzZrZTNhLL4W/vlFERAAlOCKhtX+/lf6HvTwtnUycCG+9Bfv2BR1JEn3+OcybB5dcYm/Ope5Gj4Zt26xltIiIhJ4SHJGQmj8fysuV4CRSQQGcOAEvvxx0JEn0xhu2ZuSSS6Ce/hOfECNH2q26qYmIpAX96ycSUkVF0Lq1LQEIu0svtRF2w4bZnpdz5wYdSfW6dOlCl7p0PisqgsaNLcEJuwEDbIRd585WJzpvXtCRiIhIDEK6+5tIdjt50taKTJgQ3j0aK+rdO+gIYtOggbWLjq7Bdy7oiM7Utm3bsz/Ze5gzB3JzoS7PkyrduwcdQezy8+G556y+sVWroKMREZEaaAZHJISWLIFdu9KnPO3TT22kg4kTYefO8C6n+Oyzz/jss8/O7uT162HjRhgyJD0y43370mdBVH4+HD+eJV0qRETSmxIckRAqKrLlE9HWxmFXWGgjHYS9XfT69etZv3792Z0c7SueLu2hFy60kQ4uvtj2FEqXF7qISBZTgiMSQkVFtvamTZugI8k8HTrYBEeY1+GcteJi6NEDevYMOpLM07ixvXBee03tokVEQk4JjkjIfPwxLF2aPuVp6WjiRHj7bdi7N+hIEujQIeugFp1pkMTLz4cdO8Jb3ygiIoASHJHQiZZOKcFJnoICa+Qwf37QkSTQa6/ZHjhDhoSze0ImGDHCbl98Mdg4RESkRkpwREKmqAi6dIGBA4OOJHMNHQrnnhvedThnpbgYmjSxGRxJji5doGtXKCkJOhIREalBGrTZEckex4/bJpQ33pheH8KPHh10BPGpX/9Uu+iTJ8O1H2a3bt3iP8l7+2Fycy1zSxfpmIzl58OsWXDgALRoEXQ0IiJShRD9sy4iixbZ+6aJE4OOJD49e6bfuvaJE60V9/vvBx3J6Vq3bk3r1q3jO+nDD2HzZksY0qE9dFSXLjbSSX6+lQJmZJcKEZHMoARHJERKSuz96dixQUcSn507baSTq66yWbKwlakdOnSIQ4cOxXdS9IcYOjTxASVTOm2gFDV4sHVUU7toEZHQUoIjEiIlJfYBcbpVvpSUpN+yhPbtIS8vfB/Eb9iwgQ0bNsR3UnExdOuWftNob79tI52oXbSISOgpwREJiY8/tu6z6bK5ZyYoKIB334WysqAjqYMjR+D11239TZMmQUeTHfLzYft2WLEi6EhERKQKSnBEQmLePLtNt/U36WzixAxoF/3aa3D0qE1HpVNninSWn2+3L7wQbBwiIlIlJTgiIVFcDB07qj10KuXlQdu24VuHE5fiYtvYc8iQoCPJHtHmCOlWlykikiWU4IiEQHk5vPSSlafpQ/jUibaLLimxmZy0VFJiWXE6tYfOBPn5sHSptT0UEZFQUYIjEgLvvQd796bv+puxY9Ov81vUxImwe7e9Vw2DHj160KNHj9gO3rjRWkQPHpxe7aGjLrnERjqKtovWLI6ISOgowREJgZIS22xy3LigIzk7XbvaSEfRdtFh6abWsmVLWrZsGdvB6doeOuq882yko8GDoVEjmDMn6EhERKQSJTgiIVBSAsOHQ7z7O4bF1q020lHbtpYfhGUdzv79+9m/f39sBxcX21qQCy5IblDJko4bKEVF1z29+qraRYuIhExMCY5zboJzbp1zboNz7v4qHr/ZObciMhY55wZVeKzUObfSObfMObckkcGLZILdu2HJkvTunrZggY10VVBgZYJh2HNy8+bNbN68ufYDjx61N9cXX5y+7aEXL7aRrqLtoleuDDoSERGpoNYExzlXH/gVMBHoC9zknOtb6bDNwBjv/UDgx8BjlR6/3Huf673PS0DMIhll/nz7ADhd199kggkT7G/w0ktBRxKHN96wPXDUHjo4ahctIhJKsczgDAU2eO83ee8/B54GplY8wHu/yHu/N/LtO0CXxIYpkrmKi6FdOyvpl2AMGQJt2qTZevHiYlsDovbQwenaFTp3PrWJlYiIhEIsCU5noGJ1/bbIfdX5FlCxmt0D851zS51zd1R3knPuDufcEufckt27d8cQlkj6O3nS3htddZU1GZBgpGW76Gh76DZtgo4ku40caTWmBw8GHYmIiETE8paqqtqHKldUOucuxxKcH1S4O997PxgrcbvHOTe6qnO994957/O893nt2rWLISyR9Ld0qa37UHla8CZMgF27YNmyoCOJQWkprF1r034NGwYdTXYbMQKOHUuz6T8RkcwWy8YJ24CKDWC7ADsqH+ScGwj8DpjovS+L3u+93xG53eWcm42VvL1Rl6BFMkVJiS2fGD8+6EjqJhMStOjfoLg42HLBC2LpiBZt+Zaue8hEXXpp0BHU3ZAhp9pFf+UrQUcjIiLENoOzGLjQOdfDOdcImAa8WPEA59z5wCzga9779RXub+acOyf6NTAeWJWo4EXSXUmJrRFP90nLdN7OJKpDB3uvGvQH8c2bN6d58+Y1H1RSAp06wYUXpiaoZGnb1kY6y8mxjPi119QuWkQkJGpNcLz35cC9wDxgDfCM9361c+4u59xdkcMeBNoAj1RqB90BeNM5txx4Dyjy3mseXwTYswfeeSe920NHbdpkI91NmABvvw379gUXw969e9m7d2/1Bxw7Zj25Bw+GZs1SF1gybNtmI93l59tGUKtXBx2JiIgQW4ka3vu5wNxK9z1a4evbgdurOG8TMKjy/SICL79sC9ozobzrjUjRac+ewcZRVxMmwE9+Yn+b668PJoYtW7YA0Lq6XV/ffBMOH7YEJ93bQ7//vt12SfPGm9FSuzlzoH//YGMREZHYNvoUkcQrLobWrWHo0KAjkajhw6Fly+DL1GpUUmKNBdJ9/U0m6dYNOnY8tTZKREQCpQRHJADe2/vU8eOtRbGEQ4MGMG6c/W1Cu5yiuBgGDEj/tSuZxDnrprZ4sc2uiYhIoJTgiARg+XLYuTMzytMyzYQJsH07rApjO5ToOo+LL1Z76LDJz4ejR2H+/KAjERHJekpwRAIQLYG66qpg45AzRZPOUJapRYNSeVr45OVZ0llYGHQkIiJZL6YmAyKSWCUlkJtrZfuZ4Oqrg44gcTp3tgqw4mK4777UX79Xr17VP1hcbP2se/dOXUDJNGpU0BEkTtOmMGgQvPJK0JGIiGQ9zeCIpNiBA/DWW5lVnpYJ25lUNGGCNSs7eDD1127atClNmzY984HPP7f2boMHQ2375KSLVq1sZIr8fCgthXXrgo5ERCSrKcERSbGXX4by8szY/yZq3brMek83cSIcPw6vvpr6a3/66ad8+umnZz6waJFlXEOGpH976KjSUhuZYsQIu50zJ9g4RESynBIckRQrLoYWLU5tnZEJ3n7bRqbIz7c9NIPo+rtt2za2VbX5ZUmJtXnLy0t9UMmycqWNTNGzJ7RrB3Pn1n6siIgkjRIckRTy3t40jxunJlhh1qgRjB0bsnbRxcW2iWT79kFHItVxzrLjd9+FI0eCjkZEJGspwRFJoVWrrAVxJpWnZaqJE616av36oCPBXjQrVlh76EaNgo5GajJiBHz2mZoNiIgESAmOSApFS56U4IRftIV3KDannzfPbtUeOvyGDrXde7UOR0QkMEpwRFKouNg6yXbqFHQkUpsePawbcyj2wykutrUdffoEHYnUpnlz6zO+YEHQkYiIZC0lOCIpcuCAtR7OxNmba66xkWkmToTXXrOKo1S56KKLuOiii07dUV4OL71k7aGbNUtdIKlw+eU2Ms3IkbBhA2zcGHQkIiJZSQmOSIosWJB57aGjWra0kWkmTIBjx+D111N3zZycHHJyck7d8c47sH+/tYeul2H/yW7ePHP29Kko2iJRZWoiIoHIsH8tRcIrE9tDR61aZSPTjB4NOTmpLVPbtWsXu3btOnVHcbGt6cik9tBRGzbYyDS9esG554ZkAZeISPZRgiOSApneHnrJEhuZpkkTq6BK5fvUHTt2sGPHjlN3FBdD375w3nmpCyJV1qyxkWmcs25qixbZFKCIiKSUEhyRFFi9GrZty8zytEw3YQJ8+GFAyyl27oT337f1N2oPnV5GjIBDh2wRl4iIpJQSHJEUiM4ATJgQbBwSv4ICuw2k2ijaHjoTy9My3bBhtmZK63BERFJOCY5IChQXw8CB0Llz0JFIvC64AC68EIqKArh4cTG0bQv9+wdwcamTli2ttFDtokVEUk4JjkiSHTgACxeqPC2dFRTAq6+mtl005eUwfz5cfHHmtYfOFiNHwrp18NFHQUciIpJVlOCIJFkmt4eOuuEGG5mqoMDWir/6avKv1a9fP/r16weLF8Pevbb+JtPaQ0eNG2cjU116qXUYKSwMOhIRkaySof9qioRHtD30iBFBR5I8TZvayFSjR9vPN3du8q/VsGFDGjZsaC+cevXgkkuSf9Gg5OTYyFR9+kCrVkpwRERSrEHQAYhksmh76CuvzMz20FHLltltbm6wcSRLTo79DefOtb+pc8m71s6dOwE4r7jY3iB37Ji8iwVt3Tq77d072DiSpV49+2Rj4UL4/HN1whMRSRHN4IgkUba0h1627FSSk6kKCqC0FNauTe51du7cyc71621joYsvhsaNk3vBIK1fbyOTjRwJBw+mpr5RREQAJTgiSaX20JkjmqSmokyNVavsNpPL07LFpZfaTM4LLwQdiYhI1lCCI5JExcUwYAB06RJ0JFJX559v3ZpTkuCsXAmtW6s9dCY45xzrET9/ftCRiIhkDSU4Ikly8CC8+Wbml6dlk4ICW05x4EASL3LypCU4Q4ZA8+ZJvJCkzKhRsHEjbNgQdCQiIllBCY5IkixYAMePK8HJJAUF9jdN6t6NGzfC4cMwdGjmtofONvn5djt7drBxiIhkiZj+9XTOTXDOrXPObXDO3V/F4zc751ZExiLn3KBYzxXJVMXFVp0SfW+TyW6+2UamGzHCWn4XFSXvGgPef58Bq1fbDE6mmzgxOz4B+NKXoEOH5L5wRETkC7UmOM65+sCvgIlAX+Am51zfSodtBsZ47wcCPwYei+NckYyTLe2hoxo2zJ6fc/z4U+2ik6F+YSH18/OzozytQQMbmc4566b27rs2OyciIkkVywzOUGCD936T9/5z4GlgasUDvPeLvPd7I9++A3SJ9VyRTPTBB7B1a3Z8OA2weLGNbDBpEnz8MSxfnoQn/+gjtpeVsT1b2u6tXm0jG4waBUePnmqtKCIiSRNLgtMZ2Frh+22R+6rzLSD6X/CYz3XO3eGcW+KcW7J79+4YwhIJr+jG5QUFwcaRKtn0PjWaeySlm1pREbvbt2f3wIFJePIQ2rTJRjbIy7ONPl98MehIREQyXiwJTlV7dldZnOGcuxxLcH4Q77ne+8e893ne+7x27drFEJZIeBUWwuDB0LmmjwIkLZ13ni2PSVaCQ7t2dhHJLDk59sJ55ZXk1TeKiAgQW4KzDeha4fsuwI7KBznnBgK/A6Z678viOVckk5SVwaJFcPXVQUciyVJQAG+/DXv2JPBJP/vM2rMNGmRrNiTzjBoF27fDsmVBRyIiktFiSXAWAxc653o45xoB04DT5tidc+cDs4Cvee/Xx3OuSKYpLratTJTgZK6CAvsbJ3TvxldftTUagwbVfqykp5Ej7fb554ONQ0Qkw9Wa4Hjvy4F7gXnAGuAZ7/1q59xdzrm7Ioc9CLQBHnHOLXPOLanp3CT8HCKhUVh4qoxJMtMll0CbNgkuUysshGbNoHfvBD6phEqnTtC9e5LqG0VEJMr5ENYC5+Xl+SVLlgQdhkjcjh+3JRTXXw+/+13Q0Ugy3XILzJsHn3ySgP04vYdu3Swrjm4GOX8+nHuuytUyzcMPw9NPw86dliWLiMhZc84t9d7nVb5f22SLJNCbb8L+/TB5ctCRSLIVFMCnn0JCPotZudL6iquuMfONHAnl5TBnTtCRiIhkLCU4Igk0Zw40bgxjxwYdSWotWmQjm1x1lc3cJKTaKLrDfUEBW7duZevWrTUfnymWL0/ShkIhlptrpYhKcEREkkYJjkgCFRbC5Zdnxyb0Fa1fbyObtGkDw4efyk3qpLDQytM6dqSsrIyysrLaz8kEH31kI5s0aGAvnNdegxMngo5GRCQjKcERSZD16+HDD1Welk0KCqxE7ZNP6vAkn34K77wDkyYlLC4JuZEjrcf4W28FHYmISEZSgiOSINGKE71PzR7Rv3WdytRKStRXPNuMGGG3ahctIpIUSnBEEqSwEAYMsGZYkh0GDYKuXeGFF+rwJEVF0KGD+opnkzZtoE+fBG+kJCIiUUpwRBJg3z5YuDB7y9MaNLCRbZyDKVPsfepnn53FE5SX2wxOQcEXvabr1atHvTr3nU4T9evbyEajRsEHH1j3PBERSags+VdUJLlKSmy9cLZWGd1yi41sNHUqHDkCCxacxcmLFll2XKGuceDAgQwcODBxAYZZQYGNbJSfb/sf1Wn6T0REqqIERyQBCguhbVsYOjToSCTVxoyBFi3O8n1qYSE0bAjjxiU8Lgm5Pn2gdWu1ixYRSYIsLCoRSazycltkPmVK9lbbvP663Y4ZE2wcQWjUCCZOtPepJ07E+RooKoLRoy1DiigtLQWge0KjDKmlS+02G9cf1atn3dQWLLApwCZNgo5IRCRjaAZHpI7efhv27s3e8jSAzZttZKupU2HXLnjvvThO2rzZ1mBUeuHs27ePffv2JTbAsNqxw0a2uvxyW7yVkN1iRUQkSgmOSB1Fq4zGjw86EgnKxInWZCGuMrXoDqHqK569hg6FnBx47rmgIxERyShKcETqaM6cU+swJDu1amWvgbgTnF694MILkxaXhFxODgwfDi+9ZLWuIiKSEEpwROpg40ZYsya7y9PETJ0Ka9fC+vUxHHzgALzyil44YmVqZWXwxhtBRyIikjGU4IjUQWGh3Wb7+9QmTbRGesoUu33xxRgOLi6Gzz+Ha64546EGDRrQIFs2FWrc2EY2GznSGg6oTE1EJGGc9z7oGM6Ql5fnlyxZEnQYIrUaNw62b7e14iK5uXDOObbpa42mTYNXX7UF9tW1XZs/H84913YTlcx2553w6afWeEJ/bxGRmDnnlnrv8yrfrxkckbN04IC1R8722Rs5ZepU27tz9+4aDjp2zLpmTZ2avX3F5XSXXw5btsCyZUFHIiKSEZTgiJyl+fPh+HElOAAvv2wj202ZAidPnmqQVqUFC+DgwSrL0wA2bdrEpk2bkhNg2Lz7ro1sF91A6tlng41DRCRDKMEROUuzZkHbtjBiRNCRBG/bNhvZbvBg6NKllm5qs2dbHdsVV1T58IEDBzhw4EByAgybXbtsZLuOHa2jXnRRn4iI1IkSHJGzcOyYvReZOtX2PxEBWz4xZYrN7h05UsUBJ05Y9jNpkhbXy+kuvxxWrbJSNRERqRMlOCJn4eWXrcrouuuCjkTCZsoU25x+wYIqHnzrLVugU015mmSxyy4D71WmJiKSAEpwRM7CzJm2sWc1VUaSxS67zCrQqixTmz3bZm4mTkx1WBJ2F1xgpWpx7RYrIiJVUYIjEqfycnsPMnmyqoyiWrSwIafylzlzrOHAF7y3BOfKKy0Dqvb8xjTOlhdWs2Y2xOobL7/cmi7s2RN0NCIiaU0JjkicXn/d3n9ce23QkYTHtdfq91HRlCnwySfw3nsV7ly2zNZX1FKe1qdPH/r06ZPcAMPiiis0DVrRmDHWmvH554OOREQkrSnBEYnTrFnQpAlMmBB0JBJWBQW2xc1p1UazZ9uO9VOmBBaXhNygQTYVOnNm0JGIiKQ1JTgicTh50t6nTpwITZsGHU14lJTYENO6NYweDS++WOHO2bNh1Cho167Gczds2MCGDRuSG2BYLFpkQ0yDBvbCeeMNOHo06GhERNKWEhyROLzzDnz8sbqnVbZzpw05ZepU+OAD2LAB+59Vq2Lqnnbo0CEOHTqU/ADDoKzMhpxy2WVw6BAUFwcdiYhI2lKCIxKHmTOhYUPbxkSkJlOn2u3s2dH/Ab785cDikTQxfLh1qnjuuaAjERFJWzElOM65Cc65dc65Dc65+6t4/CLn3NvOuWPOub+t9Fipc26lc26Zc25JogIXSTXvbf3NuHHQsmXQeYA/PwAAIABJREFU0UjYde8OQ4ZEtjWZPRsGD4Zu3YIOS8IuJ8eSnJdeqtSGT0REYlVrguOcqw/8CpgI9AVucs71rXTYHuC7wM+reZrLvfe53vu8ugQrEqT334fSUpWnSexuuAEWL4bNb3+szT0ldpdfbhvCLlwYdCQiImkplhmcocAG7/0m7/3nwNPA1IoHeO93ee8XA8eTEKNIKMyaZZ2x1ATrTG3a2JDTfeUrdvssX4k5wWnSpAlNmjRJYlQh0rKlpkOrMnKkddx79tmgIxERSUsNYjimM7C1wvfbgGFxXMMD851zHviN9/6xqg5yzt0B3AFw/vnnx/H0Iqkxc6ZtU9G2bdCRhM/kyUFHEE49esAlLdbyzLGv8Xd9K098V6137972xZYtSYwsJEaPDjqCcGrVylpGz50bdCQiImkplhkcV8V9Po5r5HvvB2Mlbvc456r8F817/5j3Ps97n9euljaqIqm2Zg2sXavNLCVOe/dyw6HHWXpsABs3VfWfUpFqXHklbN4MS7R0VUQkXrEkONuArhW+7wLsiPUC3vsdkdtdwGys5E0krUT33dMyiqrNmWNDKikq4isnZwCxVxutW7eOdevWJTGoEHnjDRtyprFjrUztz38OOhIRkbQTS4KzGLjQOdfDOdcImAa8WMs5ADjnmjnnzol+DYwHVp1tsCJBmTULLr0UOnUKOpJw0nYm1Zg9m26dyhk+3PPMM7GdcuTIEY4cOZLcuMJi/34bcqa2beHii60Dn7qpiYjEpdYEx3tfDtwLzAPWAM9471c75+5yzt0F4Jw7zzm3Dfhr4IfOuW3OuRZAB+BN59xy4D2gyHuv/c4lrWzaZB3U1D1N4vLZZ1BSAl/+Mjfc4Hj/ffjww6CDkrQyYQJs3QqLFgUdiYhIWolpHxzv/VzvfS/v/Ze89z+J3Peo9/7RyNc7vfddvPctvPetIl8fiHReGxQZ/aLniqST6B6NWn8jcSkstCTn+uu5/nq7K9ZZHBHA2kXXr68yNRGROMWU4Ihks5kzrVKkR4+gI5G08tRTVtM4ejRdu8KIEUpwJE6tWsGwYfD88ypTExGJgxIckRrs2AFvv63Zm9qcd54Nidi711r8Tptmn8Bjm36uWGHd+GrSvHlzmjdvnoIgQ0AbKNXuqqtg1y5YsCDoSERE0oYSHJEaRMvTtP6mZhMm2JCImTPh+HH46le/uCtaplZbN7ULLriACy64IInBhciIETakemPGQMOGMH160JGIiKQNJTgiNfjLX6BvX+jTJ+hIJK385S/QqxcMHvzFXZ072wb1KlOTuDRvDvn5tqbr+PGgoxERSQtKcESqsWkTvPUWfO1rQUcSfrNm2RCsrvHVV232xp2+uecNN8CqVfDBB9WfvmbNGtasWZPkIEPilVdsSM0mTIA9e6wrn4iI1EoJjkg1nnrKbitUGUk1DhywIcCMGeA93HTTGQ9df73lPDWVqR07doxjx44lMcAQOXzYhtRs5EjIyVE3NRGRGCnBEamC91byPmYMnH9+0NFIWnnqKRgyxErUKunYEUaPVpmaxCknx144JSWQLcmviEgdKMERqcLSpbBuHdx8c9CRSFpZvx6WLKlx2u+GG6xEbfXqFMYl6W/CBDh4EF54IehIRERCTwmOSBWefBIaNTrV+UokJn/5i9Wg3XhjtYdcey3Uq6dZHInT8OHWcODJJ4OOREQk9JTgiFRSXm7vU6++Glq3Djqa9NCli42s5r29cC67zFqmVeO886z0MbpUp7IWLVrQokWL5MUZJu3b25DaNWoEl18OL7+sdUsiIrVQgiNSyYIF8MkncMstQUeSPq680kZWe/99q2uMoSvFDTfYoStXnvlYz5496dmzZxICDKFhw2xIbK66Cj77zPZZEhGRainBEalk+nRo1QoKCoKORNLKU0/Zhowx7AobLVObMSMFcUnmyMuz/zipTE1EpEZKcEQqOHTI9nO54QZo3DjoaNLHjBlZ/mb9xAkrT5s4Maa6xvbtYdw46/p74sTpj61atYpVq1YlKdCQmT/fhsSmQQObKn39ddi3L+hoRERCSwmOSAUvvGAVICpPi8+RIzay1sKFtsFnHJsmfeMbsHXrmftclpeXU15enuAAQ+rYMbU9jtdVV9nvLKs/URARqZkSHJEKpk+Hbt0gPz/oSCSt/OUv0KwZTJ4c8ylTp1q10RNPJDEuyTyDBkHbtvaaExGRKinBEYn45BOrlrn5ZlsfIRKTzz+HZ5+Fa66Bpk1jPi0nx15rs2er2kjiUK8ejB8Pb70FO3cGHY2ISCjpbZxIxNNPw8mT2txT4jRvHuzdG1d5WtQ3vgFHj9prTyRmV19t/ex/97ugIxERCSUlOCIR06fD4MHQt2/QkaSfHj1sZKWnnoI2bc6qT/bgwTBgADz++Kn7WrVqRatWrRIYYIh16mRD4tOrF/TuDX/8Y9WbKYmIZDklOCLYniRLlqi5wNkaM8ZG1jl0yDpT3HCDtYiOk3M2i7N4Maxebfd1796d7t27JzbOsBoyxIbE75prYMMGa3AhIiKnUYIjgm0rUa8eTJsWdCSSVp57ztrHnUV5WtQtt1j3XzUbkLhMmGC97H/966AjEREJHSU4kvW8t/K0K6+Ejh2DjiY9TZ9uI+v85jfQp0+d2u61a2dLKv78Zzh+HFasWMGKFSsSGGSIzZ1rQ+LX/P+3d99xUlX3/8dfn9lCW6lLXRBEQQQrrghKLFEjaARRsYI1Gms0scSSxPwSjTW2WIm9oSTRiCj2fK1gQMWCWAhFQMrSWZZtM+f3x9mVZVlklt3ZM+X9fDzO487cmdn56Fxm7ueecz4nDw45BCZOhLVrQ0cjIpJUlOBIxpsyBebO1fC0hqis9C2jfPYZTJ0K55zjx5o1wJlnwrJlMHkyxGIxYrFYIwWZ5KLRzVc6lfiNGuUX7nrssdCRiIgkFSU4kvGeeAJatICjjw4diaSUceP8EKFTT23wnxo+HDp31jA1qac994QePTatUiEiIkpwJLMVF/v5N8ceC9ttFzoaSRklJT4zHj0a2rdv8J/LzoaxY2HSJI02know8704M2bAJ5+EjkZEJGkowZGM9vTTsG4dnH9+6EgkpUyY4DORc85ptD95xhl+mN+UKY32JyUTHHkkZGXBvfeGjkREJGkowZGM5Zw/J9hjDxg8OHQ0qa1vX98yRnVxgaFDG+1P9u8PgwbB5MkdaN++Q6P93aS2/fa+ybbr0MEfh//8p181VkRElOBI5po6FT79FM47r8FzxDPefvv5lhEasbhAbWecAW+/3YNly3o06t9NWnvs4Zs0zKhRsHo1PPts6EhERJKCEhzJWPfd5+fdnHJK6EgkpTRicYHaTjwRmjdXsQGppyFDID8f/v730JGIiCSFuBIcMxtmZl+b2Wwzu7KOx/uZ2RQzKzOzy+rzWpEQli/3FztPPdUvJyEN8+ijvqW9Ri4uUFvbtnD66TOYMWMGpeUZcP3pxRd9k4bJyoKRI+GDD2D27NDRiIgEt9VfUDPLAu4BhgP9gZPMrH+tp60EfgXcug2vFWlyjzwC5eV+eJpI3J59ttGLC9Q2dKjPo174oGPC3kPS0IgRfmLhffeFjkREJLh4LhEOAmY75+Y458qBZ4CRNZ/gnFvmnJsGVNT3tSJNLRaD+++HAw6AAQNCRyMpZdy4Ri8uUFv//r5z6MFXuifsPSQNFRRAYaEvDanFU0Ukw8WT4BQAC2rcX1i1Lx4Nea1IQrz2GsyZo94bqacEFheoyQwOOgje+KQDH89qkbD3kTR0zDGwZAlMnBg6EhGRoOJJcOr6JXdx/v24X2tm55jZdDObXlRUFOefF6m/e++FTp38uYBI3BJYXKC2gw6C3OwYNz/RNeHvJWnkoIN85ZQHHggdiYhIUPEkOAuBmjVLuwPfx/n3436tc26cc67QOVfYsaPGnktifPcdvPQS/OIXkJsbOpr0MWBAmg/3S3BxgZo6duxI794dOWSvFTz/f21Zsjwroe8XVO/evknjyM31C3+++SZ8H+/PtIhI+oknwZkG9DGzHcwsFzgRiLf/uyGvFWl048b5ebgJnCOekfbZx7e01QTFBaoVFBRQUFDAqP2WUl4R4fanOyf8PYNJ+8w4gFGjoLIS7rwzdCQiIsFsNcFxzlUCFwKvArOACc65mWZ2rpmdC2BmXcxsIfAb4HdmttDMWm/ptYn6jxH5MeXlfpmIn/8cevYMHU16qajwLW01QXGBatFolGg0Ss/OpQwaUMxDL+RTVp6mK9FWVvomjWfHHWHfff2XXUlJ6GhERIKIa6EF59zLzrm+zrkdnXPXV+273zl3f9XtJc657s651s65tlW3127ptSIhPP88LFum4gKJ8NRTvqWlJiouUO3zzz/n888/B+CUYStYsSaHh1/okPD3DWLyZN+kcZ1+Oqxapbk4IpKxMmAlORHv3nthhx3g8MNDRyIp5Y47oEULGDu2yd96yG7r6dm1jDuf6YSLt7SLSGEh9Onjh6nFYqGjERFpckpwJCPMnAnvvAPnngsRHfUSrwUL4MknfVWKDk3fi2IGpwxbydfzW/DqlO2a/P0lRZnBaafB/PkwYULoaEREmpxO9SQj3HefLzB0xhmhI5GUcvvt/gr4pZcGC+GIoWto3SrKrU90CRaDpKBDD/X18G+9NXQkIiJNTgmOpL2VK+Gxx+D440EVyCVuK1b44gInnxy0KkXzXMcxP13FW9O3Y9bcZsHikBSTnQ2nnAIffQRvvx06GhGRJqUER9LeXXdBcTFccUXoSNLXnnv6llbuuQfWr2/yA6dLly506bJpb80Jh60iKwK3PJ5mJaP79vVNEuPoo6FVK7jxxtCRiIg0KSU4ktbWrvXzbI8+GnbbLXQ06SvtEpz1631mfNRRsOuuTfrWdSU4HdtVcuigtTzzWgdWrkmjr+2dd/ZNEqNVKzj2WHjtNZg1K3Q0IiJNJo1+KUU2d++9sHo1/O53oSNJbyUlabbkxkMP+SFqV17Z5G9dUVFBRR2LCp0yfCUbyiLc9UynJo8pYUpLfZPEOekkX1lFvTgikkGU4EjaWr8e/vpXGD4c9t47dDTpbcKENCrWVFHhJ2b/5Cew335N/vYzZ85k5szN10PeZYdS9uhTwgPPdUqfRVVff903SZyOHX1t/AkTYOnS0NGIiDQJJTiStsaNg+XL1Xsj9TR+vC8PHaD3ZmvGHLGSJStyePrV9qFDkVQydqzvKbv99tCRiIg0CSU4kpZKS+GWW+Dgg4NchJdUFYvBTTf5CVvDh4eOZjMHDFxH1/xybn+qsxb+lPjttBMMHuyv+qTVWFIRkbopwZG09MgjsHixem+kniZNgi+/9L03ZqGj2UxWBMYMX8mn37Zk0rttQocjqeS002DVKnjggdCRiIgknBIcSTsVFX4+7ZAhvgdHJC7OwQ03QK9eftGkJDXq4NV0al/BVXd3Uy+OxK+wEPr08WUlY7HQ0YiIJJQSHEk7Tz4J333ne2+S8CJ8Wios9C2lvfsuTJ0Kl1/uF0kMpFu3bnTr1m2Lj+fmOH55TBEz57Tk6VfaNWFkCbDLLr5J4pn5Xpz58+GZZ0JHIyKSUOaS8BJgYWGhmz59eugwJAVFo9CvH7RuDdOnK8GRejjiCL/q+7x50KJF6Gh47YbptC9oiUU2P4grozD6t73JznZ8+9xMsrICBCipp7LSr4vTvDl8/TU6cEQk1ZnZR865zS6xqgdH0sqECTB7tnpvmtqaNb6lrE8/hcmT4ZJLgic3paWllG5lbZjsLLhgdBFzFzXn7//Ob6LIEqC42DdpGtnZcNFF8L//wX33hY5GRCRh1IMjaSMWg91397c/+8yvbSdN49FH/fb000NG0QDHHANvvOHHNrZtGzSUGTNmALBscuUWe3DAH+9jfr8DxRsizHnhC5rlJt93+Va9+KLfHnVU2DgyiXNw6qm+hv7cudCqVeiIRES2mXpwJO298ALMnAnXXKPkRurh7bfh+efht78NntzURyQCF56wjO+LcrlrfMfQ4UiqMINf/xqKinxRDRGRNKTTQEkLzsF11/kiQUlcAEuSTSwGv/kN9OjhtylmyG7r2b1PCTc93pX1GzQmU+I0cCDsvz/ccQcsXRo6GhGRRqcER9LC+PHw8cd+7o3mzUrcnnjCHzg33hh87s22MINfnbCMFWuyufHRLqHDkVRy8cWwYYPv8hYRSTNKcCTlFRf7yr6FhTBmTOhoJGWsXw9XXw377gsnnRQ6mm22584bGLJbMXeO78zKNcruJU69e8ORR8Jjj8E334SORkSkUSnBkZR3/fXw/ffwt79p7k0oQ4b4llJuucUfOLfdllQl97p370737t3r9ZoLjl/GupIs/vxg1wRFlSC77eabhHHeeb7L+/LLQ0ciItKodDooKe3bb/356WmnweDBoaPJXDvv7FvKWLQIbr7ZT9jab7/Q0WwiPz+f/Pz6lX7u16uMn+6zlvuf68jionCLlNZbr16+SRidOsGJJ8LEifDBB6GjERFpNEpwJKX95jfQrJmfQiHhLF/uW8q4+mpfYOCmm0JHspmSkhJKSkrq/brzRxdRXm78/v5uCYgqQVav9k3COeMMvzLypZf6ai0iImlACY6krJdfhkmT4NproYvmVwc1aZJvKWH6dHj8cb+oZxL2HnzzzTd8sw1zInp1LeeIoWt4/KUOzJrbLAGRJcC77/om4eTlwdlnw9Spvta+iEgaUIIjKam83J+f7ryzX5hbJC7O+W6/jh19L06aOf+4InKyHWf+qZcuxkv8jjsOunaFK66AaDR0NCIiDaYER1LSHXf4+Td33AG5uaGjkZTx/PO+x+DPf/bDctJMp/aVXHTiMqZ+nse9/6jfPB7JYDk5cOGF/kv1gQdCRyMi0mBKcCTlfP+9Pz8dMQKGDQsdjaSMsjJfLWrAADjrrNDRJMyxB69m951KuOru7ny/LIUKDkhYhx0G/fv7ns3Fi0NHIyLSIEpwJOVceaUfonbbbaEjkZTyt7/BnDn+wMlO3xP/SAT+cM5iNpQZ51zfM3Q4kioiEfjjH/36UGeeqYIDIpLS4kpwzGyYmX1tZrPN7Mo6Hjczu6vq8c/MbGCNx+aZ2edmNsPMpjdm8JJ5PvjALz5/2WWw446ho5FqBxzgW9L69ltfjeKII+BnPwsdzY/q2bMnPXs2LDHp1bWcs0Yu56X32zLh9baNFFkC7LWXb5IceveGX/4SXnnFLwAqIpKizG3lKo2ZZQHfAIcBC4FpwEnOuS9rPOcI4CLgCGBf4E7n3L5Vj80DCp1zcReRLSwsdNOnKxeSTUWjftH5JUvgq6988R+RraqshKFD/Wrtn38OBQWhI4rLazdMp31BSyyybYuQVlTCyb/rTXFJhG+e+4K228UaOUJJS9EonH66Hws8a5YvPiAikqTM7CPnXGHt/fH04AwCZjvn5jjnyoFngJG1njMSeNx5U4G2ZqZvRWlUf/0rfPSRX4BeyU1yWbLEt6T0l7/Ahx/CffelRHJTXFxMcXFxg/9OTjZce/b3rFidzSW39miEyBIg5RZQygBZWX6SY0mJXyNHQ9VEJAXFk+AUAAtq3F9YtS/e5zjgNTP7yMzO2dZAJbP9979wzTVw7LF+4W1JLq+84lvSmTYN/vQnOPlkOOGE0NHEZfbs2cyePbtR/tauO5Zy/GEreeylfN78bxJeFZgyxTdJLr16+aFqr74Kjz4aOhoRkXqLJ8Gpa3xE7Us6P/ac/Z1zA4HhwAVmVudIfTM7x8ymm9n0oqKiOMKSTLF2LZx0EnTrBn//O9i2jdiRTFNSAmPG+CE2d98dOppgLhhdRJcO5fziz70oLdM/HonT2LG+qtoll/jhaiIiKSSeBGchUHN8Q3eg9rfdFp/jnKveLgOexw9524xzbpxzrtA5V9ixY8f4ope05xycdx7MmwdPPw3t2oWOSFLGFVf4eTePPZbRB06L5o6rz1jCvMXNuObebqHDkVSRleV7Pzds8HNyNFRNRFJIPAnONKCPme1gZrnAicDEWs+ZCJxaVU1tMLDGObfYzFqZ2XYAZtYK+BnwRSPGL2nuiSd8YvPHP8L++4eORlLGK6/APffAr38NP/1p6GiC22+P9Rw+ZA13ju/Mh1+0DB2OpIpeveDcc+H11+Hhh0NHIyISt60mOM65SuBC4FVgFjDBOTfTzM41s3OrnvYyMAeYDfwdOL9qf2fgPTP7FPgv8JJzLhlH6ksS+uYbOP98OPBAv/acSFxWrPCTo/v39wUGBIDLxy6lXetKRl22I8tWpu86QNLIxozxi+P++tewaFHoaERE4rLVMtEhqEy0lJXBkCEwfz58+il07x46IvkxC6pKjPQIXazLORg9GiZO9JUp9twzcED1t2bNGgA+vPfbBpWJrsvM/zXnF9f1pLB/CW+P+5qc0HlOdem9Ll3CxiE/bt48X6hjyBB4800/fE1EJAk0pEy0SJO7+mr45BM/KkLJTfLr0SMJkhuAJ5+Ef/3Ll7lNweQGoE2bNrRp0yYhf3vAjqVcdcYSpnyWx0U3b5+Q96iXLl2U3KSCXr18D87bb8PFF4eORkRkq0JfvxPZzOTJcNttcMEFMLL2ikuSlJKiB2fGDD+mcehQuOyygIE0THUPTqKMOGANX81rzgPPdWTvXdZz9qgVCX2/H6UenNQxejT8739+btsuu/gvaBGRJKUeHEkqixfDaafBbrv5BT0lNbz5pm/BfPcdHHEEtG0LzzyT0kNo5s6dy9y5cxP6Hr85ZSl77ryeC2/animfBSw6MG2ab5IaLrsMBg3yvTivvx46GhGRLVKCI0ljzRp/jrp+vT9HbdEidESSElatguHD/bo3kydDQe11iKW27Cy45VeLaNc6yqjLdmLJcnXmSxyys+Hmm/244eOOg6++Ch2RiEidlOBIUtiwAY46CmbOhOee8wWwRLaqrAxGjYJvv4Xnn4dddw0dUcpo1zrKXy9ZwOp1WYz4zU6UV2gRUIlDXh7cdZe/PWwYrFwZNh4RkToowZHgKirg+OPhvff8ujeHHx46IkkJsZhfgPDtt+HRR+Hgg0NHlHL67VDGNWctZtqXrTjvhmSoEiEpoaDAT5RctMhfmaqoCB2RiMgmlOBIULEYnHkmTJrk566ecELoiCRlXHmlH8t4442+hK1skyP2X8vJw1bw8MSO3Px4p9DhSKrYc0+45hr44AM46yxfol1EJElo4LUE45yvPPrkk76q73nnhY5IttWwYU38hnff7atQnH8+XHFFE795Yu20004ALKPp5jdcfNIyvlucy2/v6kG00rjqzKVN88ZDhjTN+0hiHHUUzJ0Ljz8O/fppRWYRSRpKcCSY667zQ7kvucRfCJTU1aRVfv/9b/jVr2DECH8AWXrNHcnLy2vy98yKwK2XLOTKuwu4+t7ulJZH+H/nLk78G+fnJ/49JLEuvNCvyHzNNf7f4lVXhY5IRERD1CSMe++FP/wBTj0V/vrXtDtHzThz5viWcBMnwkkn+VK148endDnoLVm1ahWrVq1q8vfNzoYbL1rEYfuu4U8PduOKuwoSP+po4ULfJHVFInDDDXDggb4H59JLNVxNRIJTD440uUce8Rf9jjoKHnzQ/z5KanvnHb/t3TuBb3LPPb7nZu+94cUXoWXA9VsSaP78+cHeOzsLrjv/e3JzHLc83oWycuOOSxcm7gLEJ5/4bffuCXoDaRK5ub589PXX++IDy5fDQw/5rFlEJAB9+0iTicX8Bb6bboJDD4Vnn4WcnNBRSdKLxXxBgVtu8Vnx+PHQqlXoqNJWVgSuPXsxOdmOu57pTGlZhPuu+k4XIuTHZWXB73/vF9t9/HFfPvof/4DmzUNHJiIZSAmONIl162DMGD/C6Nxz/dQJJTeyVaWlvhT0s8/6ggJ33ZWWw9KSTSQC15y5hNwcx7jnO1JWYTz8h/lKcuTHmfle1rZt/b/Vww6Dl16C1q1DRyYiGUYJjiTc/Pn+wvuXX8Lf/gYXXKA5NxKHlSvh6KPh3Xf98JfLLtOB04TM4PKxS2mWE+OxSfmsWpvF43+aR5u8WOjQJNmdeqpPcq67DoYOhTfegE4qQS4iTUfX4ySh3n8f9tkHvvsOJk/2c290jipbNXcu7LcffPihX+vm8st14ARgBhefVMQFo5cy6b227H5ifz78Ij3nPkkjGzHCDyv9+msYPBg+/TR0RCKSQZTgSMJULy7ftq0/Tz3ssNARSaL8/Oe+NYrXX/froyxb5q/8Ztjqr3379qVv376hw9jEGSNW8sBV8yktizD0rH7c9FhnYo3RkfOTn/gm6enAA323/erVUFjoq61Fo6GjEpEMoARHGl1xsR+GfcYZcMABMHUq7Lxz6KgkkfLzG2FJkzVr4Oyz4Wc/81nx++9n5Mlvy5YtaZmEFeL26reB8X+Zw767FnPl37oz7KKdKFrVwFHObdv6Julr771hwgTYd19fZeaAA2DevNBRiUiaU4IjjWriROjf31+0u/hiPyytffvQUUmiff21b9vs5ZdhwAB4+GG44gpfPniXXRotvlSyfPlyli9fHjqMOrXJi3HHpQu55OSl/N9H27HrCf15a1oDFiadN08nu5mgXTu44w6/COgnn8Cuu/oufq2XIyIJogRHGsXChXDMMTByJLRp4y++33GHKqVliilTfKu3VavgtNPgyCP9gTNliq8j3qJFo8eYKhYuXMjCJF780gzGDF/JQ7+fR06W49Dz+3L5nQWsW78NPyeff+6bpD8zOPZYePpp6NXLd/GPGuXXzBERaWRKcKRBolG4805/sf2VV+DGG+Hjj/38cJEf9cILvrvvqafgmmv8gTNoUOioJE79e5cx/i9z+Nm+a7n1iS7sMGJX7pnQkYrK0JFJUuvRw/fUnnOOLyHdv79fFLRSB46INB4lOLLNpk3zw6ovucRPlZg5E377W/XayFa8/z4MG+ZLQHfqBP/9ry8n26xZ6MiknlpoEN/nAAARP0lEQVQ2d1x/wfc8+Lt5dMuv4MKbt2eXYwfw3FttNfpItiwryyc4jzzih6/94hfQr5/v3VERAhFpBEpwpF6c84WtDj/cX2xftMjPH33pJdhhh9DRSdJyDt56y5fVGzrU99bcfLPPkgcODB2dNNCeO2/g4Wvnc9NFC6iIGsdesSODT9+Z92e0Ch2aJLNddvE9uNdfD7EYnHKKn5/z3HOanyMiDaIER+JSUeEvrg0c6Ms9f/YZ/OUv8NVXMHq0liiRLXDOV5rYf3845BBfieC22/w6N5dfDrm5oSOURmIGhwwqZsKNc7h0zBJmL2jO0F/048iLd+S1qds1TllpST9m/orZs8/Ctdf6MpzHHgt77eWvnCnREZFtYC4JvzwKCwvd9OnTQ4chwLp1fnj07bf7xTr79fMLyo8ZoxFFstGaNX7bpk3VjuJi+Pe/faWJjz7y4+6vvBLOPBOaNw8WZyooLS0F4J3bv6B9QUsskppXD0pKjYdeyOf5/7Rj7fostu9SxjnHLOfMEcvpml8136K42G/zGlCJTdJLZaX/7njkEVi6FPr08QUJxo6F7t1DRyciScbMPnLOFW62XwmO1FZaCq++Cv/6l58Hvnatn2Nz+eW+2FVE/X5Sl/JyX2ni6ad9vfANG2DHHX1p2LFj1VtTT6/dMD2lE5xqpWXG5A/a8OK7bfjs25ZkRRxHDl3DuccW8bPBa8nKCh2hJKWKCj9U7eWX/QRPM/jpTzdWX0vCtaJEpOkpwZEfVVLiRxL9858waZK/sNq+vS/7/Mtf+mICIpuJRuGdd/ji7v+DN95g17Uf+BU/R4+Gk0/25fSUEdfLsmXLAJjx0HdpkeDU9M13uTz3Vjte/7A1a4qzObjdJxy09zr2HN6Nn+6zjryWGscmdfjf//xFkzfe8L06eXlw/PH+e+YnP4FWmuslkqmU4MgmolG//MT778N//uOTm5IS6NjRXxw77jg46CBVRJNaolE/Aevtt317911YsYJHm/0SBg7k9N/3gEMP1YHTADNmzABg2eTKtEtwqpWVwytT2pAzeSLfLc3lwcozycmOsf8exRyx/1qG7beGXXcs1dw+2VQ0Ch984OfmvPeeH26QkwODB/vvnUMPhX320fePSAZRgpPh1q2DqVN9QvPBB/72unX+sYIC31Nz3HH+Ylh2dthYJYmsW+eHh7z3nk9o3nsPVq/2j/XuDQceCMOG8eiqkdCsGaefHjTatJAJCU61Du88T3klvNhmDFM/z+OjWa2Y+72f3Nc1v5zBu62ncJcSBvbzrVN7rZUiVYqL/Q/ZtGnw6ae+l8c537tz4IFwwAGwxx6w++7QpYsq4YikqS0lODqVTTMrV/rKZl99BbNmbbw9Z46vwmnmv+/HjvWjh/bfH3r21Hd/xtuwwR8wM2fCF19s3M6fv/E5ffr4LPjAA33r0WPjY482ecSSJnKz4aC913PQ3usBWLAkh3c+yePjr1ry3y9a8fx/2v3w3G4dyxnYr4S9+5XQr1cpvQvK6N29jA5tovoOyzR5eRt7bcAPXZsyxRc1+eQT38tTLT/fJzvVbcAAv65Bu3b68RNJU3ElOGY2DLgTyAIedM7dWOtxq3r8CKAEON0593E8r5X4OOcn+y9Z4teeqavNnQtFRRtf06wZ9O3rq22OGQNDhvie/Natw/13SBOLxXyPy4oVsGwZLFzoy+EtWLBpq5r3AfjhHf36+QPm7LP9ycCgQdCtW7j/DskYPbpUcMrwVZwyfBXOwYrVWcyc04Kv5zdj9oLmfPZtC15+rw0xt/HEdLuWUXYoKGPHqoSne6cKOrevoFP7Sjq3r6Bzh0o6tKnUdLB01rmzXzz46KP9D+bSpf7q3rff+h/HuXP9kNry8o2vycvzV/h69fKtZ0/Yfnu/AHHHjr516KBhDSIpaKv/as0sC7gHOAxYCEwzs4nOuS9rPG040Keq7QvcB+wb52vTQizmq1tWVvriLxUVUFa25bZ+ve9hr2u7erXviVm1ym9XrvT76lrguU0bP8SsoABGjPDrpvXr57c9e6IKRcnOuY0HTEXFxgOovNyPL6/Zysr8tqTEDx0rLt58u2aNT2ZWrIDly/1BVNcCJNtt53tgtt/eL27Uo4c/aAYMgJ120hh2SQpmkN8uyoF7F3Pg3r6ktHOwpjjCvO9zWbCsGYuLslmyIoclK3L46KuWvPR+G8orNs9kIhFHfttK8ttW0jYvSuu8KG1aRWlT43brvCgtmjlaNIvRolmMls39tkXzGC2aOXJzYuRmO3JzarYYOdmO7Cx1BiQNMz8srUsXP5m0WkmJH84we7a/qLNkib8qOGuWH4JbXba8tvbtfS9Qfj60beu/P1u33nS73Xa+sluLFr4Ufu1ts2b+ezU3129r3taBI9Lo4rksMQiY7ZybA2BmzwAjgZpJykjgcecn9Ew1s7Zm1hXoFcdrk9YFe0/lxS96EXVZxDCiLkKULGLOiJJF1EWodFlUuixiNDyTaGZltMoqpW3WOtpnr6V99lp2yF5L++x1tOvi73fKXkVBbtEPrVWWXzODdcDMqpbOfmzOWH0eq+t+9b66ttUtFtv0fs19sdjmt6PRTVvNfRUVdScf9dWihb8SmZfnf2w7dPDjEPPz/e3qlp/vE5kePWosWNM4jj++Uf9cRhswYAAAyyZ/SiwaI5LG6zGv3OdwAFysfnNB27SKskefDezRZ8Mm+2MOysqN1euyKFqdw/LV2axal8XqtdmsKs5izbos1q7PoqQ0QtHqZmwojVBSGmF9aYSy8ob/f45EHFkRn+z4rSNSdTti1Y/7bfX9SAQMf45r5qq2VY2N+2DTfdX3a263tA/AcJs9py7pf6691+a7WgI9nb/AVF4OlZVYNArRSqis+r5eXAkLoxCNgYv5bazq9o/aUNW2wgyo8cFvdrvmto7bNTab36+1v87HtvC8+kr740cA3i3ahezmyd2zGU90BcCCGvcX4ntptvacgjhfC4CZnQOcA7D99tvHEVbi9e+2mpIln5FlUbIsRgTntxYjixhZFiMnUkm2Rcm2KDlV2+rWPKucZlZBs4hvzSPlP9xulVX6Q8vLLqVlpJTsSH1PdnOrWobZ1l/nzX7x67hf++yg5jYSYdOzD9u4v7rVvG/mu9Bqtkhk4+3qq3g5OX4IRM37zZptvOrXvPmmrUULf7UwL8+XR02C4RNakqLx5FT1nrXp3JzVS0oDR9NUShr1rzUDurfwja51P6f2NY6KSmNDeYSyigjlFRHKKqu2FRHKKyNUVBqV0Y0tGjMqYxHfcx+LEI0azkE0Zv46hjNiMf8853zyFXNWdQ2k6rlV202upWCb3f8hXscPacoP4dcYqvfD637k/41ztsUnJGHNoaZl5r9z67WKtYNYFItGMReDmPNbF6u6H8N++JCrH8M/jt/W/MDthw+5jotuUOPxWM2DoFZIbgt5RhwHSINk+gGUOYpXltO2W/hzjx8TT3R1/TupfRRv6TnxvNbvdG4cMA58FbU44kq4C14cFjoEkZRQVfiLPfcMG0c6WLJkCQD7nrlr4EiagA4cERFJgHgSnIVAjXJJdAe+j/M5uXG8VkRSnM5TG091gtOlS5fAkTQBHTgiIpIA8Qw6ngb0MbMdzCwXOBGYWOs5E4FTzRsMrHHOLY7ztSIiIiIiIo1iqz04zrlKM7sQeBVf6vlh59xMMzu36vH7gZfxJaJn4wdTn/Fjr03If4mIiIiIiGS8uGYIOedexicxNffdX+O2Ay6I97UiIiIiIiKJkL71R0VEREREJOOYS8K6kIWFhW769OmhwxCROFVU+K3WB224aNWKvlmZsEqvDhwREWkAM/vIOVdYe39yF7EWkZSg89PGkxGJTTUdOCIikgAaoiYiDTZtmm/ScIsWLWLRokWhw2gaOnBERCQBlOCISIPNnOmbNFxRURFFRUWhw2gaOnBERCQBlOCIiIiIiEjaUIIjIiIiIiJpQwmOiIiIiIikDSU4IiIiIiKSNpJyHRwzKwLmh46jSj6wPHQQshl9LslHn0ny0WeSnPS5JB99JslJn0vySbbPpKdzrmPtnUmZ4CQTM5te1wJCEpY+l+SjzyT56DNJTvpcko8+k+SkzyX5pMpnoiFqIiIiIiKSNpTgiIiIiIhI2lCCs3XjQgcgddLnknz0mSQffSbJSZ9L8tFnkpz0uSSflPhMNAdHRERERETShnpwREREREQkbSjBqQczu8zMnJnlh44l05nZLWb2lZl9ZmbPm1nb0DFlKjMbZmZfm9lsM7sydDwCZtbDzP5jZrPMbKaZXRw6JvHMLMvMPjGzSaFjEc/M2prZP6t+U2aZ2ZDQMWU6M/t11XfXF2Y23syah44pE5nZw2a2zMy+qLGvvZm9bmbfVm3bhYxxS5TgxMnMegCHAd+FjkUAeB3Y1Tm3O/ANcFXgeDKSmWUB9wDDgf7ASWbWP2xUAlQClzrndgEGAxfoc0kaFwOzQgchm7gTeMU51w/YA30+QZlZAfAroNA5tyuQBZwYNqqM9SgwrNa+K4E3nXN9gDer7icdJTjxux24AtCkpSTgnHvNOVdZdXcq0D1kPBlsEDDbOTfHOVcOPAOMDBxTxnPOLXbOfVx1ex3+hK0gbFRiZt2BI4EHQ8cinpm1Bg4AHgJwzpU751aHjUqAbKCFmWUDLYHvA8eTkZxz7wAra+0eCTxWdfsx4OgmDSpOSnDiYGYjgEXOuU9DxyJ1OhOYHDqIDFUALKhxfyE6kU4qZtYL2Av4MGwkAtyBv1AWCx2I/KA3UAQ8UjV08EEzaxU6qEzmnFsE3IofMbMYWOOcey1sVFJDZ+fcYvAX04BOgeOpkxKcKmb2RtVYz9ptJHAN8IfQMWaarXwm1c+5Bj8c56lwkWY0q2OfejmThJnlAf8CLnHOrQ0dTyYzs58Dy5xzH4WORTaRDQwE7nPO7QWsJ0mH3GSKqjkdI4EdgG5AKzMbEzYqSTXZoQNIFs65Q+vab2a74f+RfWpm4IdCfWxmg5xzS5owxIyzpc+kmpmdBvwcOMSp3nkoC4EeNe53R0MJkoKZ5eCTm6ecc8+FjkfYHxhhZkcAzYHWZvakc04nbmEtBBY656p7OP+JEpzQDgXmOueKAMzsOWA/4MmgUUm1pWbW1Tm32My6AstCB1QX9eBshXPuc+dcJ+dcL+dcL/yX4UAlN2GZ2TDgt8AI51xJ6Hgy2DSgj5ntYGa5+ImgEwPHlPHMX415CJjlnLstdDwCzrmrnHPdq35HTgTeUnITXtVv+QIz27lq1yHAlwFDEj80bbCZtaz6LjsEFX5IJhOB06punwa8EDCWLVIPjqSqu4FmwOtVPWtTnXPnhg0p8zjnKs3sQuBVfKWbh51zMwOHJb63YCzwuZnNqNp3tXPu5YAxiSSri4Cnqi7SzAHOCBxPRnPOfWhm/wQ+xg9B/wQYFzaqzGRm44GDgHwzWwhcC9wITDCzs/DJ6OhwEW6ZaWSPiIiIiIikCw1RExERERGRtKEER0RERERE0oYSHBERERERSRtKcEREREREJG0owRERERERkbShBEdERJKSmY0ysxm1WszMhoeOTUREkpfKRIuISEows3OAU4CDnXOx0PGIiEhyUoIjIiJJz8z6Am8B+znnvgsdj4iIJC8NURMRkaRmZjnA08BlSm5ERGRr1IMjIiJJzcxuBLo6504LHYuIiCS/7NABiIiIbImZHQQcCwwMHIqIiKQI9eCIiEhSMrN2wMfAyc65KaHjERGR1KAeHBERSVbnAp2A+8ys5v4bnHPPhglJRESSnXpwREREREQkbaiKmoiIiIiIpA0lOCIiIiIikjaU4IiIiIiISNpQgiMiIiIiImlDCY6IiIiIiKQNJTgiIiIiIpI2lOCIiIiIiEjaUIIjIiIiIiJp4/8DdOx9Z9GjEkYAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 1008x432 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"def powerPlot(baseline, d, nC, nT, alpha):\n",
" '''\n",
" Plots null and alternative hypothesis with p-value and power shaded\n",
" \n",
" Parameters\n",
" ---\n",
" baseline: float\n",
" Baseline conversion rate\n",
" d: float\n",
" Counterfactual lift observed\n",
" nC: int\n",
" Number of observations in control group\n",
" nT: int\n",
" Number of observations in test group\n",
" alpha: float\n",
" Chosen alpha level\n",
" '''\n",
" \n",
" # Calculate Z stat\n",
" xC = nC*baseline\n",
" xT = nT*(baseline+d)\n",
" pPool = (xC+xT)/(nC + nT)\n",
" sePool = np.sqrt((pPool*(1-pPool))*(1/nC+1/nT))\n",
" zStat = d/sePool\n",
" \n",
" # define null and alternative distributions\n",
" h0 = stats.norm(loc = 0, scale = 1)\n",
" h1 = stats.norm(loc = zStat, scale = 1)\n",
" fig, ax = plt.subplots(figsize=(14, 6))\n",
" zCritical = h0.ppf(1 - alpha)\n",
" power = 1 - h1.cdf(zCritical)\n",
" x = np.linspace(-4, 10, 100)\n",
" mask = x > zCritical\n",
" hypotheses = [h1, h0]\n",
" labels = ['$H_1$ is true', '$H_0$ is true']\n",
" colors = ['red', 'blue']\n",
" for hypothesis, label, color in zip(hypotheses, labels, colors):\n",
" y = hypothesis.pdf(x)\n",
" line = plt.plot(x, y, label = label, color=color) \n",
" plt.fill_between(x = x[mask], y1 = 0.0, y2 = y[mask],\n",
" alpha = 0.2, color = color)\n",
" ax.axvline(x=0, c='blue', alpha=0.5, linestyle='--')\n",
" ax.axvline(x=zStat, c='red', alpha=0.5, linestyle='--')\n",
" ax.axvline(x=zCritical, c='grey', alpha=0.5, linestyle='--')\n",
" plt.legend()\n",
" plt.title('alpha = {}, power = {}'.format(alpha, round(power,2)))\n",
" plt.xlabel('Z')\n",
"\n",
"d = 0.03\n",
"nC = 4330\n",
"nT = 4330\n",
"alpha = 0.05\n",
"powerPlot(conversionPre, d, nC, nT, alpha)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Python does not have a built in tool like R to calculate minumum sample sizes, but there are plenty of tools, such as https://www.evanmiller.org/ab-testing/sample-size.html that help with this. Using this tool, we find that we need a minimum sample size of 4,351 given a baseline conversion rate of 0.525, a minimum detectable effect of 0.03, power of 0.80, and a significance level of 0.05. Our current sample size of 8,000+ observations in each of the control and test group well surpases the sample size requirement. \n",
"\n",
"If the magnitude of the minimum detectable effect is smaller, the minimum sample size also increases, e.g. a minimum detectable effect of 0.02 percentage points requires 9,790 observations per variation. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Hypothesis Testing\n",
"\n",
"In this section, I conduct a hypothesis test on the form load data and interpret the results."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"# Define function for A/B hypothesis test\n",
"\n",
"def twoProportionTest(xControl, xTest, nControl, nTest, alpha=0.05):\n",
" '''\n",
" Function to perform one-tailed A/B hypothesis test for two sample proportions.\n",
" \n",
" Parameters\n",
" ----------\n",
" xControl, xTest: int\n",
" Number of successful conversions in control and test groups\n",
" nControl, nTest: int\n",
" Total number of trials in control and test groups\n",
" alpha: float\n",
" Significance level, default is 0.05\n",
" \n",
" Returns\n",
" -------\n",
" lift: float\n",
" observed lift between test and control group\n",
" zStat: int\n",
" z statistic of hypothesis test\n",
" pVal: float\n",
" p-value or probability of type I error\n",
" statSig: bool\n",
" True if test is statistically at alpha\n",
" '''\n",
" pControl = xControl / nControl\n",
" pTest = xTest / nTest\n",
" pPool = (xControl + xTest) / (nControl + nTest)\n",
" lift = abs(pTest - pControl)\n",
" sePool = np.sqrt((pPool * (1-pPool)) * (1/nControl+1/nTest))\n",
" zStat = lift / sePool\n",
" pVal = stats.norm.sf(abs(zStat))\n",
" statSig = pVal < alpha\n",
" print('The observed lift between the control group and test group is {}'.format(round(lift, 4)))\n",
" print('The z test statistic is {} and the p-value is {}'.format(round(zStat, 3), round(pVal, 4)))\n",
" if statSig:\n",
" print('Test conclusion: p<alpha, the lift is statistically significant at a significance level of {}'.format(alpha))\n",
" else:\n",
" print('Test conclusion: p>alpha, the lift is not statistically significant at a significance level of {}'.format(alpha))\n",
" return lift, zStat, pVal, statSig\n",
"\n",
"# Plot\n",
"def twoProportionPlot(zStat, alpha=0.05):\n",
" '''\n",
" Function to plot the two proportions hypothesis test.\n",
" \n",
" Parameters\n",
" ---\n",
" zStat: float\n",
" Z test statistic of the two proportions test\n",
" '''\n",
" # Define distribution parameters\n",
" h0 = stats.norm(loc = 0, scale = 1)\n",
" zCritical = h0.ppf(1 - alpha)\n",
" x = np.linspace(-4, 4, 100)\n",
" mask = x > zCritical\n",
" y=h0.pdf(x)\n",
" \n",
" # Plot\n",
" fig, ax = plt.subplots(figsize=(10, 6))\n",
" line = plt.plot(x, y, color = 'blue')\n",
" plt.fill_between(x = x[mask], y1 = 0.0, y2 = y[mask],\n",
" alpha = 0.2, color = 'blue')\n",
" ax.axvline(x=0, c='blue', alpha=0.5, linestyle='--')\n",
" ax.axvline(x=zStat, c='red', alpha=0.5, linestyle='--')\n",
" ax.axvline(x=zCritical, c='grey', alpha=0.5, linestyle='--')\n",
" plt.text(zStat, 0.2, \"Z Test Stat: {}\".format(round(zStat,2)), \n",
" rotation=90, verticalalignment='center')\n",
" plt.text(zCritical, 0.2, \"Z Critical Value: {}\".format(round(zCritical,2)), \n",
" rotation=90, verticalalignment='center')"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The observed lift between the control group and test group is 0.0296\n",
"The z test statistic is 3.997 and the p-value is 0.0\n",
"Test conclusion: p<alpha, the lift is statistically significant at a significance level of 0.05\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAFlCAYAAADYqP0MAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZzO9f7/8cebsS+RJbIrQg7SRKVDiwoVbZL1WAqdVL6tOq2OX+V02ksLZQtJThySFK2nVIZkjSZLxpKxDlnHvH9/vIcmBtdwXdf7c13X8367fW4z87mWz7N4X17z+bw/r7ex1iIiIiIi4ZXPdwARERGReKQiS0RERCQCVGSJiIiIRICKLBEREZEIUJElIiIiEgEqskREREQiIMl3gNyULVvWVq9e3XcMEYkjB7vVGOM3h4hESRQH/dy5czdZa8sdvj+QRVb16tVJSUnxHUNERETkuIwxq3Pbr8uFIpIQ5sxxmwTT2rVrWbt2re8YEk8CMOhVZIlIQli82G0STOnp6aSnp/uOIfEkAINeRZaIiIhIBKjIEhEREYkAFVkiIiIiEaAiS0RERCQCjD3YRyJAkpOTrVo4iIiISCwwxsy11iYfvl9nskREREQiIKQiyxjTyhizzBiTaowZcIznnWeMOWCMuTGvrxURiaRvvnGbBNOaNWtYs2aN7xgSTwIw6I9bZBlj8gNDgNZAPaCjMabeUZ73L2BGXl8rIhJpy5e7TYJp8+bNbN682XcMiScBGPShnMlqAqRaa1dYa/cB44F2uTzvDuA/wMYTeK2IiIhIXAll7cJKQM5zuGlA05xPMMZUAq4DLgXOy8trc7xHb6A3QNWqVUOIJSJybD//DKmp7vsFC9zX6dOhYkVo2FCLRYtIZIVSZOX2MXT4LYkvAA9Yaw+YP39qhfJat9PaocBQcHcXhpBLROQIGRnw7rswcmTu0zGef959rV8funeHLl3gtNOimVBEEkUoRVYaUCXHz5WBdYc9JxkYn11glQXaGGMyQ3ytiMhJ+/ZbeOUVeP992L0b6taFp5+Giy6CfPncGSyA1q1h/nxXhN17LzzwALRpA337uq/iR758utldwiwplBInso7bJ8sYkwQsBy4D1gJzgE7W2lxXXTTGjAQ+sNZOzOtrD1KfLBEJVWYmDBwITzwBp5wCHTu6M1TnnXf8y4FLl8KoUTB6NKxfDz16wMsvQ7FiUYkuInHihPtkWWszgX64uwaXAhOstYuNMX2NMX1P5LUn8h8gInK4devgssvg//0/V1itWQOvvgpNmoQ236puXRg8GH79FR55xJ3datIEFutTSkTCQB3fRSQmzZjh5lPt2gWvvw5dux77+V984b62aHH058yc6d4zIwOGDHGFmybHR8eqVasAqF69utccEkdCGfRhoo7vIhIXrHVnnVq1ggoVYO7c4xdYACtXuu1YWrZ087UuuAB69nRF1r59YYktx7Ft2za2bdvmO4bEk1AGfYSpyBKRmPL44+7yYI8e8N13UKdOeN+/QgX4+GN3nNGjXaGVlRXeY4hIYvA/9V5EJESvvgr//Cf06gXDhkXuUl7+/PDYY1C4MAwYAOXLu9YPunQoInmhIktEYsKECdCvH7Rt6+ZgRaPguf9++O03V2BVqOAKLhGRUKnIEpHAmzXLTUhv1gzGjz+x9jdFiuT9NcbAM8+4QuvBB13T0h498v4+cnxJAehpJHHmRAZ9mOnuQhEJtHnz3M1B1avDl19C6dLRz7BvnzuDNnMmTJoE11wT/QwiEly6u1BEYs66da5D+6mnwkcf+SmwAAoWhIkToXFjuOkmV/iJiByPiiwRCSRrXRuFHTvckjiVKp3c+82c6bYTVbw4TJvmCr4uXdzSPRI+K1asYMWKFb5jSDw52UEfBiqyRCSQXn3VNRx95hmoV+/k3y8tzW0no1w51xV+6VJNgg+3jIwMMjIyfMeQeBKOQX+SVGSJSOD89JNbvLl1a7jtNt9p/uzyy+HOO+Gll+CTT3ynEZEgU5ElIoGyf7+7HFesGLz1VjB7Uw0e7NY97N4dtmzxnUZEgkpFlogEyj//6ZbKGToUKlb0nSZ3RYrAmDGwcaM70xbAm7RFJABUZIlIYMyeDU8+6c4QXX99eN+7ZEm3hUvjxq4gnDAB3nknfO+bqAoVKkShQoV8x5B4Eu5BfwLUJ0tEAmHnTmjUCA4cgB9/9P7ZGJIDB1wPr0WLYOFCqFLFdyIR8UF9skQk0J56Cn75BUaNio0CC9wah6NHu3lk99zjO42IBI2KLBHxbuVKePZZ6NwZmjePzDE++sht4VazJjzwALz3nutILycmNTWV1NRU3zEknkRq0OeBiiwR8e7++91ZocGDI3eMDRvcFgn33usuFfbv7y4hSt7t3LmTnTt3+o4h8SSSgz5EKrJExKsvvnBL1jzwAFSu7DvNiSlaFJ5+Gn74wTUrFREBFVki4tGBA+7sT5Uq7mxQLOvQAZo1g3/8A9S4XERARZaIeDRiBMyf784CFS3qO83JMQZeeMH1znriCd9pRCQIVGSJiBcZGfDQQ+7sT4cOkT9emTJui6TkZNfj64UX3J2SEroiRYpQpEgR3zEknkRj0B+H+mSJiBf33w///jfMmeOKk3ixbh3Uru3WOJw0yXcaEYkG9ckSkcD45Rd3tqd79/gqsABOP93Ny5o8GT791HcaEfFJRZaIRN3AgZCU5JbQiZapU90WDXffDVWrusuhAbxYEEjLli1j2bJlvmNIPInmoD8KFVkiElU//wxjx7qFlaO5APTmzW6LhsKF4cEH4dtv4ZNPonPMWLd79252797tO4bEk2gO+qNQkSUiUfXEE1CwINx3n+8kkdWjh+v7NXCgzmaJJCoVWSISNb/8AmPGQN++UKGC7zSRVaiQO5v1zTcwa5bvNCLig4osEYmaJ56AAgXcnYWJoFcvqFRJZ7NEElVIRZYxppUxZpkxJtUYMyCXx9sZYxYYY+YbY1KMMRfleGyVMWbhwcfCGV5EYseKFTB6NPTuHd25WAdVqBD9s2eFCsGAAfC//8Fnn0X32LGmePHiFC9e3HcMiSc+Bv1hjtsnyxiTH1gOXA6kAXOAjtbaJTmeUxz43VprjTENgAnW2jrZj60Ckq21m0INpT5ZIvHnllvcpcIVK1ybg0SxZw+ccQaceaZbp1FE4s/J9MlqAqRaa1dYa/cB44F2OZ9grd1p/6jWigE6MS4ih6xaBaNGwa23JlaBBe5OwwcegC+/hM8/951GRKIplCKrErAmx89p2fv+xBhznTHmJ2Aa0DPHQxb42Bgz1xjT+2TCikhsevJJyJfPFRu+vP++23y49VZ31WLgQD/HjwVLly5l6dKlvmNIPPE56LOFUmSZXPYdcabKWjsp+xLhtcCgHA81s9Y2BloDtxtjmud6EGN6Z8/nSklPTw8hlojEgtWr3ULQvXq5lga+ZGS4zYciRVyB+fnn7oyWHGnv3r3s3bvXdwyJJz4HfbZQiqw0oEqOnysD6472ZGvtl8AZxpiy2T+vy/66EZiEu/yY2+uGWmuTrbXJ5cqVCzG+iATds8+CMW4CeCLr0wdOOy26Xe5FxK9Qiqw5QC1jTA1jTEHgZmBKzicYY840xpjs7xsDBYHNxphixpgS2fuLAVcAi8L5HyAiwbV1KwwfDh07umVmElmRInDHHTBjBixe7DuNiETDcYssa20m0A+YASzF3Tm42BjT1xjTN/tpNwCLjDHzgSFAh+yJ8KcB/zPG/Ah8D0yz1n4Uif8QEQmeYcPg99/h//7Pd5Jg6NPHFVsvvOA7iYhEQ1IoT7LWfgh8eNi+13N8/y/gX7m8bgXQ8CQzikgM2r8fXn4ZLrkEGjXyncbvfLCDypaFbt1g5Eh32VAzI/5QsmRJ3xEk3gRg0Kvju4hExMSJkJYGd9/tO4nTsqXbfOvfH/buhdde850kWGrWrEnNmjV9x5B4EoBBryJLRMLOWnjuOahdG9q08Z0mWOrUcf9PhgxxjUpFJH6pyBKRsPv6a0hJcWdt8gXkU+bdd90WBHffDRs3wjvv+E4SHIsWLWLRIt0XJWEUgEEfkI8/EYknzz0Hp57q5h8Fxe7dbguCSy+FBg3g+ee1cPRBmZmZZGZm+o4h8SQAg15FloiE1S+/wOTJ7k66YsV8pwkmY9wdlwsXwqxZvtOISKSoyBKRsHrpJUhKgn79fCcJto4dXXPS557znUREIkVFloiEzbZtrvlohw6JtxB0XhUqBLffDtOng5bsE4lPKrJEJGzefBN27gxm89EaNdwWJH37QuHCak4KUKpUKUqVKuU7hsSTAAx6YwM46zI5OdmmpKT4jiEieZCVBbVqQaVKWgQ5L265xd1luG4dnHKK7zQiciKMMXOttcmH79eZLBEJi5kzYcUKuO0230liy223wa5d8PbbvpOISLipyBKRsHjtNbdMzPXX+06SuzFj3BY0554Lycnw+uuJ3c5hwYIFLFiwwHcMiScBGPQqskTkpKWlwdSp0LOnm9AdRJmZbgui226DxYtdE9dElZWVRVZWlu8YEk8CMOhVZInISXvrLTcnq3dv30liU4cObj6W1jMUiS8qskTkpGRmwrBhcOWVoPV9T0yxYq47/sSJkJ7uO42IhIuKLBE5KR98AGvXunYEcuL69oV9+2DECN9JRCRcVGSJyEl5/XWoXBmuusp3kmOrXdttQVWvHjRvDm+84S69JpoyZcpQpkwZ3zEkngRg0Cd5PbqIxLRffoEZM2DgQLeUTpBdeKHvBMfXty906uTaYVxxhe800VWlShXfESTeBGDQ60yWiJywoUMhf37o1ct3kvhw/fWuDYYmwIvEBxVZInJC9u516xS2beu6vAfdyJFuC7JChVwbjKlTXVuMRDJ//nzmz5/vO4bEkwAMehVZInJC/vMf2LRJHd7DrXdvOHDArQMpIrFNRZaInJA33oAzzoDLLvOdJL7UrOnaYbz5piu2RCR2qcgSkTz7+We3CPQtt0A+fYqE3a23urYYH3/sO4mInAx9PIpIno0c6Yqrbt18J4lP11wDZcu6OW8iErsCftO1iATNgQOuyGrdGk4/3Xea0J19tu8EoStYELp0gSFD3Ly3smV9J4q8cuXK+Y4g8SYAg15nskQkTz7+GNatc3fBxZLzznNbrOjZE/bvh7FjfSeJjkqVKlEpFm5TldgRgEGvIktE8mT4cHdm5eqrfSfJm/373RYr/vIXSE52i29b6ztN5B04cIADmukv4RSAQa8iS0RCtmkT/Pe/0LWru6QVS8aOjb2zQj17wsKFMG+e7ySRt3DhQhYuXOg7hsSTAAx6FVkiErKxY90vhrF2qTBWdewIhQtrArxIrAqpyDLGtDLGLDPGpBpjBuTyeDtjzAJjzHxjTIox5qJQXysiscFad+nqvPOgfn3faRJDqVJuqZ1x42DPHt9pRCSvjltkGWPyA0OA1kA9oKMxpt5hT5sFNLTWNgJ6Am/m4bUiEgPmzXOXrnQWK7p69oRt22DyZN9JRCSvQjmT1QRItdausNbuA8YD7XI+wVq709pDUzOLATbU14pIbBg+3F26uvlm30kSyyWXQLVqumQoEotC6ZNVCViT4+c0oOnhTzLGXAc8BZQHrsrLa0Uk2HbvdpesbrjBXcKKRY0a+U5wYvLlgx49YOBAWL3aFVzxqEKFCr4jSLwJwKAP5UyWyWXfETcUW2snWWvrANcCg/LyWgBjTO/s+Vwp6enpIcQSkWiZPNldsorlS4WNGgXiM/eEdO/uvo4a5TVGRFWoUEGFloRXAAZ9KEVWGlAlx8+VgXVHe7K19kvgDGNM2by81lo71FqbbK1NVudfkWAZPhyqV4eLL/ad5MTt2uW2WFStGrRsCSNGQFaW7zSRsX//fvbHUiMzCb4ADPpQiqw5QC1jTA1jTEHgZmBKzicYY840xpjs7xsDBYHNobxWRIJtzRqYNQv+9rfYXgx6wgS3xaru3WHVKvjqK99JImPx4sUsXrzYdwyJJwEY9Medk2WtzTTG9ANmAPmB4dbaxcaYvtmPvw7cAHQzxuwHdgMdsifC5/raCP23iEgEjB3r2jd07eo7SWK79looXhxGj4YWLXynEZFQhLRAtLX2Q+DDw/a9nuP7fwH/CvW1IhIbrHX/qDdrBmec4TtNYitaFNq3h/feg5dfdj+LSLDF8Ml/EYm0uXNh6VLo1s13EgH357Bjh1vaSESCT0WWiBzV6NFQqJA7gyL+NW8OVau6PxcRCb6QLheKSOLZtw/eeQfatoXSpX2nOXnJyb4TnLx8+dzcuKeegvXroWJF34nC5/TTT/cdQeJNAAa9zmSJSK4++gg2bYqfS4X168fHmotdu7o2DuPG+U4SXuXLl6d8+fK+Y0g8CcCgV5ElIrkaPRrKlYMrr/SdJDy2b3dbrDvrLGjaNP4uGe7Zs4c9WgVbwikAg15FlogcYetWmDoVOnWCAgV8pwmPSZPcFg+6dYMFC+DHH30nCZ+ffvqJn376yXcMiScBGPQqskTkCBMmuDlZ6o0VTB06uOL37bd9JxGRY1GRJSJHGD0a6tWDxo19J5HclCkDV13lGsVmZvpOIyJHoyJLRP4kNRW++cZdkjK5LfEugdCtG2zYADNn+k4iIkejIktE/uTtt11x1bmz7yRyLG3awKmnxt8EeJF4oj5ZInKIta7IuuwyqFzZd5rwuuAC3wnCq1AhuPlmGD4cMjKgZEnfiU5O5Xj7Cyf+BWDQ60yWiBzyzTewciV06eI7SfiddZbb4knXrrBnD7z/vu8kJ69s2bKULVvWdwyJJwEY9CqyROSQMWOgSBG4/nrfScJv0ya3xZOmTd3C3WPG+E5y8nbt2sWuXbt8x5B4EoBBryJLRADXsmHCBGjXDkqU8J0m/D74wG3x5ODcuU8/hXXrfKc5OcuXL2f58uW+Y0g8CcCgV5ElIgDMmAFbtmjCe6zp3NnNpRs/3ncSETmciiwRAdwlpzJl4mcZnURRu7ZbBzceLhmKxBsVWSJCRgZMmfJHJ3GJLV26wA8/wNKlvpOISE4qskSESZPcXWrxeFdhIujQAfLlcx3gRSQ41CdLRBgzBmrWhPPP950kcpo3950gcipUgMsvd0XWoEGx2am/WrVqviNIvAnAoNeZLJEEt26duzutc+fY/Mc5VDVrui1ede4Mq1a5XmexqHTp0pQuXfpP+2rXru0pjcSFAAx6nckSSXDjx0NWVvzfVbhhg/taoYLfHJFy7bWux9mYMdCsme80eVeiRAlMdpVvrQVc76yD+zMyMnzGk1gUgEGvM1kiCW7sWHd3Wrx1Qz/cRx+5LV6VKOEKrQkTXM+zWHPVVVfRvHlzfv75Z3bs2MGOHTuoWrUqO3bsUIElJyYAg15FlkgCW7oU5s2L/7NYiaJzZ9frbMYM30nybsCAAXTq1ImOHTvy0ksvkZWVdejMlkisUpElksDGjnV3pd18s+8kEg5XXAFly8Zuz6x69eoxc+ZMAFq0aMGePXs8JxI5OZqTJZKgrHVFVsuW8TtPKdEUKODaObz1lut9VrKk70R5ly9fPu68807at2/PDz/84DuOyEnRmSyRBDV7trsbTZcK40vnzq7n2fvv+06SN+vXr2fv3r2Am/j+0UcfMX36dF577TUyMzM9pxM5MSqyRBLUuHFQuDBcd53vJNFx2WVui3fnnw81asA77/hOkjf33HMPVatWBdz8rGnTptG0aVPmzJlD7969PaeTmBSAQa/LhSIJaP9+ePddaNvW3ZWWCKpU8Z0gOoyBTp3gqafcHeyxdCm4YsWKAMycOZM5c+aQL18+unTpQsOGDT0nk5gUgEGvM1kiCWjmTNi0yf1jnCjWrHFbIujc2fU+mzDBd5LQVaxYkSlTpgBQvXp11mT/YW3evNlnLIllARj0IRVZxphWxphlxphUY8yAXB7vbIxZkL19Y4xpmOOxVcaYhcaY+caYlHCGF5ETM24clC4NrVv7ThI9s2a5LRHUrQuNGrk/51hx3333MWjQIJo3b86+ffto1KgRl156KS1btuS5557zHU9iUQAG/XEvFxpj8gNDgMuBNGCOMWaKtXZJjqetBFpYa7caY1oDQ4GmOR6/xFq7KYy5ReQE/f67WxC6UycoWNB3GomUTp3g/vshNRXOPNN3muOrUKECw4YNo1ChQixfvpzu3btTuXJlzjvvPPLl00UXiU2hzMlqAqRaa1cAGGPGA+2AQ0WWtTbnalnfApXDGVJEwmfqVFdoJdKlwkTUsSM88ICbAP/II77ThK5u3brUrVvXdwyRsAjl14NKQM6LmmnZ+46mFzA9x88W+NgYM9cYc9RbRIwxvY0xKcaYlPT09BBiiciJGDcOKlUKxAL1EkGVK7s/47FjXU+0WKa7CyVWhXImK7d1DXIdssaYS3BF1kU5djez1q4zxpQHPjHG/GSt/fKIN7R2KO4yI8nJyTH+kSASTJs3w/Tp0L+/6/Qu8a1TJ+jTB374ARo39p3mxPXp08d3BIlR1lq+/+471q5dizGG008/nSZNmkRtyaZQiqw0IOd9kJWBdYc/yRjTAHgTaG2tPXQ7iLV2XfbXjcaYSbjLj0cUWSISeRMnQmZmYl4qbNXKd4Lou/FG6NfPnb0MepF15jEmjp177rlRTCLx4uPChfn7gw9Sq04dKlVyF+DS0tJITU3l1Vdf5Yorroh4BmOPcx7ZGJMELAcuA9YCc4BO1trFOZ5TFfgU6JZzfpYxphiQz1q7I/v7T4B/WmuPuSx2cnKyTUnRjYgi4daiBWzcCEuWuH5KEv/atYOUFPj1V8if33eaE9O7d2+GDh3qO4bEmLp16zJ9+nSqV6/+p/0rV66kTZs2LF26NGzHMsbMtdYmH77/uGeyrLWZxph+wAwgPzDcWrvYGNM3+/HXgUeBMsCr2afgMrMPdhowKXtfEjDueAWWiETGr7/Cl1/CP/+ZmAXWihXua82afnNEW6dOMGWK+7O/5BLfaY5uRfYfUKlSpf6031rLhx9+6COSxLjM3bupvG/fEfsrVarE/v37o5IhpI7v1toPgQ8P2/d6ju9vAW7J5XUrALXqFQmA8ePd10S8VAiuyIDEK7KuuQaKF3eXDINcZNWqVYuKFStSoECBQ/uMMVhr2bhxo8dkEqt6Jidz3uWXc/Pf/06V7O7va9asYfz48fTq1SsqGbSsjkiCGDcOmjaFM87wnUSiqWhRtz7lxInwyitQqJDvRLmrVKkSb7zxBq1z6ZBbJQDLo0jsefDqq7m2cWP+ay2zZ8/GWkvlypUZO3Ys9erVi0oGFVkiCWDxYvjxR3jxRd9JxIdOneDtt92dpdde6ztN7jp37kxGRkauj91///1RTiPxou7pp1O3e3dvx9dN3CIJYNw417Lhppt8JxEfWraEcuWCvcxOhw4dOOuss3J97I477ohyGol3jz/+eFSOoyJLJM5Z6/5xbdkSKlTwnUZ8SEqCDh1ct/+jnCwSSSjRaguiIkskzs2eDatWQefOvpP4dfXVbktUnTvDnj1u3cogql27NrVr1/YdQ+LJMQb9NddcE5UIKrJE4ty4cVC4cHDn4kRL2bJuS1RNm0KNGm6ZnSAqWrQoRYsW9R1D4kkug/7SSy+NagRNfBeJY/v3w7vvutv4S5b0ncavZcvc16NM+4l7xrgJ8E89BRs2BO/S8aZNmwAom8iVsIRVg4ODPfuWWmsty5cvp0GDBgAsWLAg4hl0Jkskjs2cCZs26VIhuMums2f7TuFX586QleUK76BJS0sjLS0NgMaHrQF0+M8ioahetCgNTj2VCRMmMHXqVKZOnUr58uUPfR8NKrJE4tjYsVCqVGKu2ydHqlsXGjUK9l2GAPPmzTvmzyKhmHLXXdyQnEzv3r358ccfqV69OgUKFKBatWpUq1YtKhlUZInEqd9/h8mToX374DaglOjr3Bm+/x5+/tl3kqNbvXo1M2fOBGD37t3s2LHDcyKJVdedey7Tp0/n888/p23btuzLZZmdSFKRJRKnpk51hVaiLqMjubv5Zjc/6513fCfJ3bBhw7jxxhvp06cP4C4jXpvod23ISSlWrBjPPfccgwYN4uGHH47qsVVkicSpsWOhUiVo3tx3EgmSypWhRQv398Na32mONGTIEL7++mtKZt+pUatWLa1dKGHRsGFD+vbtG9Vj6u5CkTi0eTN89BH07+86vYtbv0+cTp2gd2+YNw+i1JPxuOrUqQNAoUKFKFiw4KH9mZmZGGN8xZJYFoBBr49fkTj03nuQmam7CnM65RS3Cdx4IxQoEKyeWYULF6Zw4cK0aNGCJ598kt27d/PJJ5/Qvn37qDWOlDgTgEFvbADPFycnJ9uUlBTfMURiVvPmrnXD4sVu/o3AokXua/36fnMExbXXugnwa9ZA/vy+03DokmDZsmV56623+Pjjj7HWcuWVV3LLLbfobJbkXRQHvTFmrrU2+fD9OpMlEmdWr4avvnJnsfTv0h9SUtwmTufOsH49fP657yTOunXrWLduHfny5ePWW2/lvffeY+LEidx6660qsOTE5Bj0Vx+2vM7hP0eK5mSJxJmDd4117Og3hwTb1VdDiRLukuFll/lO84caNWrkWlStWLHCQxqJF8OGDTvmz5GiM1kiccRaGDMGLrwQatb0nUaCrEgRuOEG+M9/YPdu32n+kJKSwpw5c5gzZw5fffUVd955J126dPEdS2LYiy++SMWKFf+0b8KECVE5tooskTiyYIGbh6V/kyQUXbpARgZ88IHvJH8oU6bMoa1SpUr079+fTz/91HcsiWGjRo06Yt/IkSOjcmxdLhSJI2PGQFKS6/IucjwXXwwVK7q/N0H5O5NzCZ2srCxSUlLU8V1OyDvffsu4b79l5erVtG3b9tD+HTt2UKZMmahkUJElEicOHHDzsVq3hrJlfacJnptu8p0gePLndz2zXnrJ9VaL0r87uTr77LMBuOKKKw7tS0pKonr16lG7tCPx5cL+/am4ahWbBg7knnvuObS/RIkSNGjQICoZ1MJBJE58+qmbwPzuuyooJHTz58M558Brr0GUm2GLxI2jtXDQmSyRODFmjLtbTH0bczd/vvvaqJHfHEHTsCGcfbb7++OzyHr88ccBDi2nc7i77747imkkLmQP+m/37N7SJkEAACAASURBVOGOO+5g6dKl7Nu3jwMHDlCsWDEyMjIiHkFFlkgc2L0bJk50d4sVKeI7TTCpyMqdMa5n1j/+AStXQo0afnL89ttv2XnUE0vCJHvQ93vlFcaPH0/79u1JSUlh9OjRpKamRiWCiiyROPDBB7Bjh+4qlBPTqZMrssaNg4ce8pOhT58+ADRSFSwRcOaZZ3LgwAHy589Pjx49uPDCC6NyXBVZInFgzBg4/XR3t5hIXlWr5pZiGjPGFVs+Tybt2bOHt956i8WLF7Nnz55D+4cPH+4vlMS0okWLsm/fPho1asT9999PxYoV+f3336NybPXJEolxmzfDhx+6sxFBWINOYlOXLvDTT5Cjg4IXXbt2ZcOGDcyYMYMWLVqQlpZGiRIl/IaSmPb222+TlZXFK6+8QrFixVizZg3vv/9+VI6tIkskxr33HmRmunk1IifqxhuhYEG3zI5PqampDBo0iGLFivG3v/2NadOmsXDhQr+hJKZNnjyZwoULU7JkSR577DGee+45PohSB96QiixjTCtjzDJjTKoxZkAuj3c2xizI3r4xxjQM9bUicnLGjHF3hzVsePznJrLOnVWIHkvp0nDVVa7XWmZm9I//l7/8hb/85S8UKFAAgFKlSrFo0SK2b9/OqlWroh9IYl/2oA90x3djTH5gCHA5kAbMMcZMsdYuyfG0lUALa+1WY0xrYCjQNMTXisgJWrECvv4annrK7zyaWJD9b7ccQ5cuMGmS67mWoydoVOTPvtbdu3dvtm7dyqBBg2jbti07d+5k0KBB0Q0jceGdiRMZN24cK1eu/FPH94yMjEB1fG8CpFprVwAYY8YD7YBDhZK19pscz/8WqBzqa0XkxI0b57527Og3RyyYM8d9Pe88vzmCrE0bKFUK3n47+kVW7dq1ue666+jduzelS5emRYsWrFixIrohJK5cWKwYFa+6ik2bNnnr+B7K5cJKwJocP6dl7zuaXsD0vL7WGNPbGJNijElJT08PIZZIYrMWRo92dxRWq+Y7TfAtXuw2ObrChd1qAe+/Dzt3RvfYgwYNIj09nSuuuIKmTZvywgsvsH79+uiGkLhSbcsWLi5cmNmzZ9OiRQvq16/Pli1bsNaSlBSd5gqhFFm5XYTIdS0eY8wluCLrgby+1lo71FqbbK1NLleuXAixRBLbd9/Bzz9Dt26+k0g86dYNdu1yhVY0nXXWWdx555388ssvvPjii6xevZqmTZty6aWXMmzYsOiGkbhw9QsvsCgtDYD169dTv359hg8fTteuXXnhhReikiGUIisNqJLj58rAusOfZIxpALwJtLPWbs7La0Uk70aNct3db7jBdxKJJxdeCDVrurOkvpx//vk8//zzjB49mq1bt9KvXz9/YSRmrUxPp35lN3tpxIgRXH755UydOpXvvvsuan3XQimy5gC1jDE1jDEFgZuBKTmfYIypCrwPdLXWLs/La0Uk7/buhfHj4brr4ChLvYmcEGPc2axPP4U1a47//HCbM2cOd999N9WqVeOxxx6jd+/erF27NvpBJOYVyNE4cNasWbRp0wZwc7Ly5YtOB6vjXpS01mYaY/oBM4D8wHBr7WJjTN/sx18HHgXKAK9mrzuVmX3pL9fXRui/RSRhfPABbNumS4USGV27wuOPu55ZA6LUeOfll1/m448/5rTTTuPmm2/m66+/pnLlysd/ochRVDn1VF6eOZPKp5zCvHnzaNWqFQC7d+9m//79UclgrM11ipRXycnJNiUlxXcMkcBq187dLbdmjbq8S2T89a+waRMsWRKd9iADBw6kY8eO1K5dO/IHk4SwceNGHn30UdavX8/tt9/OFdm3zH722WfMnTuXe++9N2zHMsbMtdYmH7FfRZZIbElPd+sU/t//wdNP+04j8WrYMOjdG77/Xm0vRI7naEWWltURiTHjx7uO3LpUmDfffOM2CU379lCoUPQmwK9Zs4Y1PiaBSfwKwKBXkSUSY0aNgnPOgfr1fSeJLcuXu01CU6qUuyz9zjuwb1/kj7d582Y2b958/CeKhCoAgz463bhEJCwWL4a5c+H5530nkUTQrRtMmADTp7uCK5KWLl0KQFZWVq6PN27cOLIBJG59/fXXNGvW7Lj7IkFFlkgMefttN9Fdy+hINFx5JZQv7y4ZRrrIeu655wAoXrz4EY8ZY/j0008jG0Di1h133MG8efOOuy8SVGSJxIgDB2DMGGjVCk47zXcaSQRJSdC5M7zyCmzZAqeeGrljHezq3qhRo8gdRBLK7NRUvklNJT09/VARD26B6AMHDkQlg4oskRjx2Wewdq0uFZ6oKC1VFne6dXN/5959F267LXLHydkcctGiRSxZsoQ9e/bkyKE7PSRv9lnLzn37yMzMZMeOHYf2lyxZkokTJ0Ylg1o4iMSILl1cE9ING9xCviLRYC00bOiWcPruu8gfb+DAgXz++ecsWbKENm3aMH36dC666KKo/aMo8Wf16tVUq1YNcHP+du7cSckwL5WhFg4iMWz7dvjPf9xcLBVYEk3GQPfurl/W4iis1zFx4kRmzZpFhQoVGDFiBD/++CN79+6N/IElbj344INkZGTw+++/U69ePc466yz+/e9/R+XYKrJEYsD48bBnD/Ts6TtJ7PriC7dJ3nXp4i63jhgRuWOsWrWKVatWUaRIEfLly0dSUhIZGRmUL1+eFStWRO7AEr+yB/2SJUsoWbIkkydPpk2bNvz666+8/fbbUYmgIkskBowY4fpiJR9xMlpCtXKl2yTvypeHa65xd7dGasm3bdu2sW3bNpKTk9m2bRu33nor5557Lo0bN6ZJkyaROajEt+xBv3//fvbv38/kyZNp164dBQoUwERjrSg08V0k8JYscXNhnn02OmvIieSmRw+YNAk+/DCy7RxeffVVAPr27UurVq3IyMigQYMGkTugxL0+ffpQvXp1GjZsSPPmzVm9enXY52Qdjc5kiQTciBHuUk2XLr6TSCJr3RoqVIDhwyN7nEmTJrF9+3YAqlevTtWqVZk8eXJkDypx7c4772Tt2rV8+OGHGGOoVq0an332WVSOrSJLJMD273eNIK+5xl2yEfElKcm1c5g2zd3hGikDBw7klFNOOfRzqVKlGDhwYOQOKHHvt99+o1evXrRu3RqAJUuWMGrUqKgcW0WWSIBNnw4bN7pLNXJyihRxm5y4Hj3+aIobbklJSSQlJeW6rE5mZmb4DyjxL3vQd+/enSuvvJJ169YBULt2bV544YWoRFCfLJEAu/ZaNx9rzRo105RguPBC2LbNtXOIxBzBnj17UqpUKW6//XaMMbz88sts3bqVkSNHhv9gkhDOO+885syZwznnnMMPP/wAuJUF5s+fH7ZjqE+WSIz57TfXfLRrVxVYEhw9esDSpa5vViS8/PLLFCxYkA4dOtC+fXsKFy7MkCFDInMwiWsHz4AWK1aMzZs3H7qj8Ntvv/3TJelI0ke3SECNGeMuzehSYXjMnOm+tmzpN0es69AB7rrLTYBv2jR873uwF1bNmjUZPHhw+N5YElaTunWZ99prPPvss7Rt25ZffvmFZs2akZ6eHrUVBFRkiQSQte4fsQsugLp1faeJD2lpvhPEh5IloX17eOcdt6Zh0aLhed9HHnmE++67j2uuuSbXHkZTpkwJz4EkYdh9+yAtjXNbtuSLL75g2bJlWGs566yzKFCgQFQyqMgSCaA5c1x/rKFDfScROVKPHu6u1/ffD19rkauuugqAe++9NzxvKAkvfccOnpsxA7Zs+dP+jz/+GIC777474hlUZIkE0PDh7saYDh18JxE5UvPmULMmvPVW+IqsevXqATB//nzuuuuuPz324osv0qJFi/AcSBLGgawsdu7Zg92xw1sGFVkiAbNzJ4wb5y7JRKkpsUie5Mvn1tF8+GH4+WeoVSt87z1q1KgjiqyRI0cesU/keCqecgqPtmvnVjj3REWWSMCMHw87dkCfPr6TxBcVrOHVowc89hi8+Sb8618n/34zZ85k2rRprFy5krZt2x7av2PHDsqUKXPyB5CEY/Pn9z7w1SdLJGCaNIHdu2HBAq1VKMF2/fXwv/+5mwoKFjy591q9ejUrV67kwQcf/NPdhSVKlKBBgwYkqY+J5NGWLVs49dRTo3Kso/XJ0t9akQD54Qc36f2ll1RgSfD17u0WjZ48GW666eTeq1q1alSrVo3Zs2eHJ5wkvGgVWMeiZqQiATJ0KBQurMWgI+Gjj9wm4XP55VCtWnjugk1OTiY1NZUSJUpQsmTJQ9vBn0XyLACDXkWWSEDs3Aljx7o7CkuX9p0m/mzYENmFjRNR/vxw660waxakpp7ce7355pvs3LmTHTt2kJGRcWg7+LNIXs345JOjDvr33nsvKhlUZIkExMEJ7717+04iEroePVyxNWzYyb9XVlYW9evXP/k3EgHaPP88l/zrX6xdu/aIx5566qmoZAipyDLGtDLGLDPGpBpjBuTyeB1jzGxjzF5jzL2HPbbKGLPQGDPfGKPZ7CJHMXQonH226/IuEitOPx2uuQZGjIB9+07uvfLly0fDhg359ddfwxNOElqDKlXodP75nH/++UecuYrWTX/HLbKMMfmBIUBroB7Q0RhT77CnbQHuBJ45yttcYq1tlNvMexH5Y8J7796a8C6xp3dvSE93E+BP1vr16zn77LO57LLLaNu27aFNJK8McGuLFsyaNYunn36aHj16sGvXLvdYlD5oQ7m7sAmQaq1dAWCMGQ+0A5YcfIK1diOw0RhzVURSisS5gxPeu3b1nSR+qdVS5FxxxR8T4E/0LsMiRYoA8Nhjj4UxmSS0pCQoU4batWsze/ZsHn74Yc455xxGjx4dvQghPKcSsCbHz2lAXtZet8DHxhgLvGGtzfU+FGNMb6A3QNWqVfPw9iKx7eCE95tu0oT3SLrmGt8J4lf+/HDLLfDII24C/Jlnnsh75Oe33347YvmcL7/8kkqVKoUpqSQSW7LkoYGflJTE4MGDadWqFR07diQ9PT0qGUKZk5XbObW8XMxsZq1tjLvceLsxpnluT7LWDrXWJltrk8uVK5eHtxeJbZrwLvGgZ8+TmwDfv39/SpQoccT+okWL0r9//5NMJ4kot7OiF198MXPnzuWhhx6KSoZQiqw0oEqOnysD60I9gLV2XfbXjcAk3OVHEcn2xhtuwvuFF/pOEt+mTnWbREbOCfB79+b99cuXL6dQoUJH7E9OTmbVqlUnH1ASzrX58+c66EuXLs2AAUfcwxcRoRRZc4BaxpgaxpiCwM3AlFDe3BhTzBhT4uD3wBXAohMNKxJvvv8eUlKgb19NeI+0zZvdJpFz221uAvzEiXl/7Z49e9i9e3eujx1tv8gxBWDQH7fIstZmAv2AGcBSYIK1drExpq8xpi+AMaaCMSYNuBt42BiTZowpCZwG/M8Y8yPwPTDNWqueyyLZXnkFSpSAv/3NdxKRk9eyJdSu7f5e51W9evV4//33j9j/1ltvce6554YhnUj0hbR2obX2Q+DDw/a9nuP7DbjLiIfLABqeTECReLVxI7z7rpuLlctUFJGYky8f3H473HWXO0ObnIemPffddx933303X3755aGiKiUlhX379jFp0qQIJRaJLC0QLeLJsGGueePtt/tOIhI+3bvDQw+5s1kjR4b+ujJlyjBq1Ci2bt3KokVuVslVV13FpZdeGpGcItGgIkvEg8xMeO01t8BunTq+0ySGChV8J0gMJUtCt27w1lvw739DqDeLFy9eHIBGjRpxySWXRDChJIwADHqtXSjiweTJsHYt9OvnO0niaNXKbRJ5t9/u7jB8883QX3PmmWdy5ok02BI5mgAMehVZIh688gpUrw5XaY0EiUP16sFll7mztZmZvtOI+KMiSyTKFi6EL76Av//dNW+U6Hj/fbdJdPTrB2vWhN6bbNiwYSxdujTXxw5f3FckJAEY9CqyRKLslVfcOoU9e/pOklgyMtwm0XH11VC1Krz8cmjP79u3L926dWPt2rVHPPbUU0+FOZ0khAAMehVZIlG0dSuMGQOdO2vBYolvSUnubO1nn8Hixcd/fq1atWjdujXnn3/+EWeurM3LSm4iwaEiSySKRo6EXbvUtkESQ69eUKgQDBly/OcaY7j++uuZNWsWTz/9ND169GDXrl2HHhOJRSqyRKLkwAH3j02zZnDOOb7TiERe2bLQsSOMHg3btoX2mtq1azN79mxOO+00zjnnHL777rvIhhSJIBVZIlEyZQr88gvceafvJImpcmW3SXTddRf8/jsMHXrs5+XLl4+SJUsCkJSUxODBg3njjTfo2LEjP//8cxSSStwJwKA3QbzWnZycbFNSUnzHEAmriy6Cdetg+XI3X0UkUbRsCT/9BCtWQMGCuT9n8uTJXHvttUfs37p1K2+88QYDBgyIcEqRE2eMmWutPWIhKZ3JEomC2bPh66+hf38VWJJ47rnHNd99992jPye3AgugdOnSKrAkZqnIEomCZ5+FUqXUtsGnd9899j/yEjmtWrkGpc88A0e7eLJo0aJDaxaKhEUABr2KLJEI++UX1w/vttsge3k28WD3brdJ9BkD994LCxbAzJm5PyczM5NMtYeXcArAoFeRJRJhL7zgLhHecYfvJCL+dOrk1ut95hnfSUSiR0WWSARt3gzDh0OXLlCxou80Iv4UKuTurP34Y3dGSyQRqMgSiaDXX3fNR++5x3cSEf/69IFixdwcRZFEoCJLJEL27HHrtrVqBWef7TuN1KjhNvHn1FPdzR/vvOPuNsypVKlSlCpVyk8wiU8BGPQqskQiZNw4+O03N+FX/GvRwm3iV//+bvWDwxeOrl69OtWrV/eSSeJUAAa9iiyRCMjKcpdEGjWCSy/1nUYkOGrWhBtucJfSd+zwnUYkslRkiUTAlCmwZIk7i6W1bYNhzBi3iX/33gvbt8Nrr/2xb8GCBSzQjHgJpwAMehVZImFmLQwaBGecAR06+E4jB2Vmuk38a9IELr/cne3dtcvty8rKIisry28wiS8BGPQqskTCbPp0mDcP/vEPLaEjcjSPPAIbNx5/4WiRWKYiSySMDp7FqlYNunb1nUYkuP76Vzcn+emn3Z24IvFIRZZIGH36KXz7LQwYAAUK+E4jEmyPPALr18OIEb6TiESGLmaIhNGgQVCpEvTo4TuJHK52bd8J5HCXXgoXXACDB8Nnn5XRLyYSXgEY9CqyRMLkq6/giy/gxRfdEiISLBde6DuBHM4YdzarTRv47LMq9OrlO5HElQAMemOt9Z3hCMnJyTYlJcV3DJE8ueIK+PFHWLUKihTxnUYkNljr7jbcsgWWLdPNIhKbjDFzrbXJh+8PaU6WMaaVMWaZMSbVGDMgl8frGGNmG2P2GmPuzctrReLBd9/BJ5+4/j8qsIJp5Ei3SbAYAw8/DCVLzmfo0Pm+40g8CcCgP26RZYzJDwwBWgP1gI7GmHqHPW0LcCfwzAm8ViTmDRoEZcrAbbf5TiISe9q2hcqVYepUt+SOSLwI5UxWEyDVWrvCWrsPGA+0y/kEa+1Ga+0cYH9eXysS61JSYNo0+L//g+LFfacRiT3GuEJrwwa35qdIvAilyKoErMnxc1r2vlCczGtFYsI//uHOYt1xh+8kIrErORlOP92Np337fKcRCY9QiqzcVl4LdbZ8yK81xvQ2xqQYY1LS09NDfHsRv2bNcnOxHnoISpb0nUYkdhkD11wDaWnw8su+04iERyhFVhpQJcfPlYF1Ib5/yK+11g611iZba5PLlSsX4tuL+GMtPPggVKmiuVix4Oyz3SbBVK5cOS66qBz16sFTT8Hvv/tOJDEvAIM+lCJrDlDLGFPDGFMQuBmYEuL7n8xrRQJt0iSYMwf++U8oXNh3Gjme885zmwRTpUqVqFChEj17wubNrtASOSkBGPTHLbKstZlAP2AGsBSYYK1dbIzpa4zpC2CMqWCMSQPuBh42xqQZY0oe7bWR+o8RiZbMTDd3pF49rVEYK/bvd5sE04EDBzhw4ABnnw3Nmrmmvps3+04lMS0Agz6ktm/W2g+BDw/b93qO7zfgLgWG9FqRWDdqlGucOGkS5M/vO42EYuxY97V7d68x5CgWLlzIjh0AjbjjDujYER59FIYM8Z1MYlYABr0WiBbJo9274fHHoWlTaKeGJCJhd+aZcOWV8OabsHq17zQiJ05FlkgevfqquwNq8GB3R5SIhN9tt0FWlrssLxKrVGSJ5MH27fDkk+637Isv9p1GJH5VqgTXXQfjx8NizeSVGKUiSyQPnnjCLWT75JO+k4jEv1tugYIF3WoKIrFIRZZIiJYvhxdegB49oHFj32kkrxo1cpsEU4UKFShbtsKf9pUpAz17uoa/U6d6CiaxKwCD3lgbavP26ElOTrYpKSm+Y4j8yVVXwVdfuWKrQoXjP19E8mbTJrcWaM5+1Pv2wY03QqFC7o7eAgX85RM5GmPMXGtt8uH7dSZLJAQffui2xx5TgRWrdu1ymwTT/v372Z9LT6OCBeGee2DlSnjmGQ/BJHYFYNCryBI5jn37oH9/OOssLQIdyyZMcJsE0+LFi0lNzX2Ge/Pm0KSJmwv5229RDiaxKwCDXkWWyHG8+CL8/LObj1WwoO80IonHGLj/ftejTpPgJZaoyBI5hvXr3dqEV18NrVr5TiOSuKpXh5tuci0dZs/2nUYkNCqyRI7hwQdh71547jnfSUSkTx845RT4+98hgPdsiRxBRZbIUXz3nVuj8O67oVYt32lEpHhx6NcP5s93S+6IBJ2KLJFc7NsHt94Kp58ODz3kO42EQ3Ky2ySYTj/9dMqXP/24z2vbFurUgQcecC0fRI4qAINeRZZILp5+GhYuhNdegxIlfKeRcKhf320STOXLl6dMmfLHfV6+fK6VSkYG9O0bhWASuwIw6FVkiRxm6VIYNAg6dHC/NUt82L7dbRJMe/bsYe/ePSE9t1Yt+Nvf4D//gf/+N8LBJHYFYNCryBLJISvLrZdWvDi89JLvNBJOkya5TYLpp59+YsWKn0J+/i23QNWq7mxWRkYEg0nsCsCgV5ElksOrr8I337ieWOWPf+VCRDwpWNBdNvztN3dzikgQqcgSybZ6NQwYAFdeCV26+E4jIsfTsCHccAMMHw6ffeY7jciRVGSJ4Hru9Onjvn/jDddhWkSC78473YLSvXrBntCmdIlEjYosEeDtt2HGDHjqKahWzXcaEQlV0aKuzcrKla55sEiQJPkOIOLbihWuweFFF7lO0hKfLrjAdwI5lsqVK7N1K/zyS95f26wZtG7tbla55hq49NLw55MYFIBBryJLEtr+/dCpk+u9M2YM5M/vO5FEylln+U4gx1K2bNmTev2AAa4TfOfOsHgxnHpqmIJJ7ArAoNflQklojz/uls8ZNkyXCePdpk3qEB5ku3btYvfuXSf8+mLFYPBgSE+Hbt20tqEQiEGvIksS1mefuTlYvXpB+/a+00ikffCB2ySYli9fzqpVy0/qPc4+293AMm0aDBkSpmASuwIw6FVkSULavBm6doXateHFF32nEZFw6d4dGjeG++5zlw1FfFKRJQnHWnf2Kj0d3nnHXWYQkfiQLx888YRrVtq+Pezd6zuRJDIVWZJwXnvNrXc2eDCcc47vNCISbuXKuW7wS5fCXXf5TiOJTEWWJJSvv4b+/d3t3vrwFYlfLVq4M1lvvAEjR/pOI4kqpBYOxphWwItAfuBNa+3gwx432Y+3AXYB3a2187IfWwXsAA4Amdba5LClF8mDNWvcEhzVq8PYse6ygiSO5s19J5BjqVatGlu2wM8/h+89774bli1zk+Hr1IHzzw/fe0sMCMCgP26RZYzJDwwBLgfSgDnGmCnW2iU5ntYaqJW9NQVey/560CXWWt08Ld7s3g3XXQe7drm7CkuX9p1Ioq1mTd8J5FhKly7NgQPhfc8CBeDZZ13vrGuvhR9+gIoVw3sMCbAADPpQfpdvAqRaa1dYa/cB44F2hz2nHTDaOt8CpYwx+qssgWAt3HILzJvnzmDVres7kfiwYYPbJJh27tzJ77/vDPv7li4Nzz0H27a5bvCaCJ9AAjDoQymyKgFrcvyclr0v1OdY4GNjzFxjTO8TDSpyop55BsaNg//3/9yHrCSmjz5ymwRTamoqv/6aGpH3rlMHHn0U5s51v3CpUWmCCMCgD2VOlsll3+F/RY/1nGbW2nXGmPLAJ8aYn6y1Xx5xEFeA9QaoWrVqCLFEju+jj+CBB9wEWC0eK5K4rrzSzc8aPRoaNYJ77vGdSBJBKGey0oAqOX6uDKwL9TnW2oNfNwKTcJcfj2CtHWqtTbbWJpcrVy609CLHMHcu3HQTNGgAI0aAye1XARFJGP36uTWDH3gAJk70nUYSQShF1hygljGmhjGmIHAzMOWw50wBuhnnfGC7tXa9MaaYMaYEgDGmGHAFsCiM+UVytWwZtGrlFomdNk0NR0XE3VH8r3/BmWe6yfCzZvlOJPHuuEWWtTYT6AfMAJYCE6y1i40xfY0xfbOf9iGwAkgFhgF/z95/GvA/Y8yPwPfANGutZkVIRK1ZA5df7j5QP/kEKh0+g1BEElbRom5dwwoVoF07+P5734kknhkbwBmAycnJNiUlxXcMiUGbNsFf/wrr1sHnn6uju/xhTfatOVWqHPt54sf27dvZsgWWLTuFaMwY+e03t85hZiZ8842bHC9xJoqD3hgzN7c+oGrHKHFjxw7XyX3VKpg6VQWW/FmVKiqwguyUU06hRIlTona8005zS2xlZcGll8Kvv0bt0BItARj0KrIkLvz+uzv1/8MPMGFCIBr9SsCsWfPHL7YSPNu3b2fHju1RPWb16u7S4fbtrtBSH7U4E4BBryJLYt62be727C++gFGj1AtLcjdrliY6B9nKlStJS1sZ9ePWqeOala5d65bdWb066hEkUgIw6FVkSUxLT3e/gX7/Pbz7rrtjSEQkL5KT4ZVX3JzOCy5wdyeLDcsTggAADuhJREFUhIOKLIlZa9dCixawdCn8979w442+E4lIrGrUCN54w61v2qwZ/Pij70QSD1RkSUxascLdRZiWBjNmuAnvIiIno04dePNN17i4eXOYPdt3Iol1KrIk5qSkuAJr+3b49FNNcheR8KlRA956yzUwvuwyd5Zc5ESpyJKY8s47rsAqUAC+/NLNpRAJRatWbpNgOvPMM6la9UzfMQDXwHjECPf1uuvgiSe0qHRMCsCgV5ElMeHAAbfAc6dO0KQJzJkDZ5/tO5XEkgoV3CbBVLx4cYoVK+47xiFly8LIkW7e58MPQ4cOsHu371SSJwEY9CqyJPAyMlwPrMGDoU8ft1SO1hCXvFqxwm0STFu3bmX79q2+Y/xJ4cLw73/Drbe6BaUvvNDNA5UYEYBBryJLAm3RIte75qOP4NVX4fXXoWBB36kkFn35pdskmFavXs26dcFrUmWM++Vu8GDX2qFxY/jsM9+pJCQBGPQqsiSQsrLgxRfdnKtNm9zZq9tu851KRBLVZZfB8OFuPuhll8G998Levb5TSdCpyJLAWbfOzVXs3x8uvxwWLoRLLvGdSkQSXa1aMH68axnz7LNw3nmwZInvVBJkKrIkUN5/H/7yF/jf/9ylwSlT3EKuIiJBULQo/POf7vLh6tXu8uGLL+ruQ8mdiiwJhDVroH17uOEG16fmhx/cPAhjfCcTETlSy5burFaDBu6s+1//6uaQiuRkbADL7+TkZJuSkuI7hkTBvn3w/PPuN8OsLHjoIbj/fk1ul/DbtMl9LVvWbw7J3a5du9i8GRYvLhpTdw9b69ZNPbgkT79+7vOsZEnfySSag94YM9dae0TnRp3JEm9mznS/BQ4Y4OZeLV3q+tGowJJIKFtWBVaQFS1alCJFivqOkWfGwM03w3/+A1deCS+9BLVrw9ixuoToXQAGvYosibp58+Dqq11hlZkJ06bB5MlQvbrvZBLPli1zmwTTpk2b2Lp1k+8YJ+zUU90ZrGHD3FmsLl2gaVN3Z7SKLU8CMOhVZEnU/PijW6Li3HPhm2/gySfdHIY2bXwnk0Qwe7YW/A2ytLQ0NmyI/U6fjRrBmDFwzz2wciVccQVcdBF8/rnvZAkoAINeRZZE3Pz5blJ7o0ZuQeeBA92Hz4MPuo7KIiLxJH9+6NjRLS7drx/89JNrQ3Pxxa7Y0pmtxKEiSyIiM9MtQ9G8OZxzjuvY/vDDsGoVPPoonHKK74QiIpFVqBB07+6Krdtuc2fzL7nEtal58003UV7im4osCav0dNc/pmZNd/YqLQ2eeQZ+/RUGDYLSpX0nFBGJriJFoFcvmDrVXUbcvduth1i5srubetUq3wklUlRkyUn7/Xd45x246iqoWNFdBjzrLPfb288/uw8VFVcikuiKFHGXEd99192FePbZrnN8jRpu3tYbb8CWLb5TSjipT5ackN273fyq8eNh0iRXaFWpAp06QbduUK+e74Qif7Z9u/uqS9XBtGfPHjZvhoULC8dUn6yTlZbm7q6eNcs1ZS5QwC0r1rmz+6q/rychioP+aH2yVGRJyNLSXLuFDz5wHwi7d0OpUu6yYJcu7jexfDo3KiInaNMmSEkhoYqsg6x1NwlNn+4mx2/ZAklJ0KwZtG3rrhTUrq1VMIJKRZbk2YYN8NVX8OWXbtAfXDKiRg3X5+rqq6FFCze5UyToDv79rV/fbw7J3caNG9m6FVauLJ+QRVZOmZnw/ffu8/f7790aieDmul58sfvcbd4cqlVT0XVMURz0RyuykiJ+ZIkJe/bAggUwd677TfJ//4Ply91jxYrBBRe4y4BXXw116mhgS+w5+HubiqxgWrduHTt2AJT3HcW7pCS48EK3gWt58/nn7vP5vfdg+HC3v3Jlt2Zikyau/+A550Dx4t5iB08ABr2KrASTleV+K1q61G2LF7sO7IsXu9+ewHUubtbM3f1ysAVDgQJ+c4uIJKoaNdzWowfs3+8+u1NS3Imajz92Nx6B++W3dm1XcNWvD3Xruu2MM1zhJtGn/+1xaOdON39q5UpYscJ9Pfj9smVuLtVB5cu7IqpNGzcwzz1Xp6BFRIKqQAG35muDBu7nrCz3eb9okbv6kJrqlvIZN+7PrznjDKhV64+CrWZN97VKFTcvXJ/5kRFSkWWMaQW8COQH3rTWDj7scZP9eBtg1/9v795j7CjLOI5/f7vdZXfbSmlBCtvWlqQRAbGaUmr4x5RLWiVtNLGp8YaaEBNIMNGotYlGjIkJifqHREKUGCNKTJTYAIZLxPCPaKH2Qi2YhqDsltKltFVaetnt4x/vmZzTZds9ks68hzO/T/LmzMye6XmenjPnPPPOzDvALRGxtZ11bXonT8KhQ6kdOJDGomptr74Ke/emNjpKo8u9aWAg3Rfwsstg1arm3s3ll8O8eVlSMjOzc6CnBxYtSq24RdmpU3DwYBpC56WX0tGLkRHYuROeeOL0HW2AoaE0/M6ll8LwMMyfny4+KFpxn+U5c9JwPD4Pt33TFlmSeoG7gRuBEWCLpM0R8Y+Wp60BljbatcBPgWvbXPcdISJ9cCcm0mG18fFU/LQ+njjRbCdPwvHjqR071nw8dix9wI8ePb298UYqjlrb4cOpsDpy5MxxDQyk3qjh4dQ9fNNNaXp4uLnHcvHFvurPzKwuenrSDvS8ebByZXN5RPp9GhtLRdfoaLqi88CBZtuzJ/3uTC7EWg0MpILr/PNh9ux0Q+zZs1ObNSudxzs01GyDg2mdgYFUoBWP552Xetn6+5utry8d2mx97O1N0z0977wet3Z6slYAeyLiRQBJDwDrgNZCaR3wy0iXKj4taY6kS4DFbaxbuTVr0jlIp05N3SYmUmudnpgoJ5bBwdSKD2fxgR0eTh/gYs/hggvS9Ny5p+9hzJxZTlxm3Wb9+twR2NlceeWVvPZauvVMB1703jX6+5s74pMVnQnj42nn/vXXUzt4MO30HzmSWtEpUHQS7NuXTkl5883UTpxIHQplvI89Pang6u1N08XjVG2I9QwOwo5bzn0c7WqnyBoGXm6ZHyH1Vk33nOE21wVA0q3ArQCLFi1qI6y375prUtdo8UZIzenJb1xvb7OKbp2eXG3PmPHWqryvr1m9t1buRXU/MOAeJrOqDA3ljsDOpq+vj8HB9L3pUc87w9y5qbVjckEV0Tyic+xYmi6O8hRHfIoOjOLo0Pj4Wzs3imURp3d+FPNFYVg8p2hp2RAzZqTX6u8/9/8/7WinyJqqc25yfXqm57SzbloYcS9wL6RxstqI6227884y/3Uz60TbtqXHZcvyxmFT27dvHwDXXz8/cyTWNYqNvj/fRt9OkTUCLGyZXwDsbfM5/W2sa2ZWOhdZna0osubPd5Fl50gHbPTtHKzaAiyVtERSP7AB2DzpOZuBzylZCRyOiFfaXNfMzMys60zbkxUR45JuBx4lDcNwX0TskvTlxt/vAR4hDd+whzSEwxfOtm4pmZiZmZl1kLbGyYqIR0iFVOuye1qmA7it3XXNzMzMup2vbTMzMzMrgaIDByRZvnx5PFPc2NHM7Bw4eTI9+j6cnWmiMRhhb29v5kisa1S40Ut6NiKWT17uexeaWS24uOpsLq7snOuAjd6HC82sFrZsSc060+joKKOjo7nDsG7SARu9iywzq4Vdu1KzzjQ2NsbY2FjuMKybdMBG7yLLzMzMrAQusszMzMxK4CLLzMzMrAQusszMzMxK0JHjZEkaA/5V8stcCLxW8mt0qjrnDvXOv865Q73zd+71Vef8q8r9PRFx0eSFHVlkVUHSM1MNHFYHdc4d6p1/nXOHeufv3OuZO9Q7/9y5+3ChmZmZWQlcZJmZmZmVoM5F1r25A8iozrlDvfOvc+5Q7/yde33VOf+sudf2nCwzMzOzMtW5J8vMzMysNC6yAElfkxSSLswdS1UkfU/SDknbJD0m6dLcMVVF0l2Snm/k/6CkObljqpKkT0raJemUpFpccSRptaQXJO2R9M3c8VRJ0n2S9kt6LncsVZO0UNKTknY3PvN35I6pKpIGJP1N0vZG7t/NHVPVJPVK+rukh3LFUPsiS9JC4Ebg37ljqdhdEXF1RCwDHgK+nTugCj0OXBURVwP/BDZmjqdqzwGfAJ7KHUgVJPUCdwNrgCuAT0m6Im9UlfoFsDp3EJmMA1+NiPcBK4HbavTeHwdWRcQHgGXAakkrM8dUtTuA3TkDqH2RBfwI+DpQq5PTIuI/LbMzqVH+EfFYRIw3Zp8GFuSMp2oRsTsiXsgdR4VWAHsi4sWIOAE8AKzLHFNlIuIp4PXcceQQEa9ExNbG9H9JP7jDeaOqRiRvNGb7Gq023/OSFgAfA36WM45aF1mS1gKjEbE9dyw5SPq+pJeBT1OvnqxWXwT+mDsIK9Uw8HLL/Ag1+aG1JkmLgQ8Cf80bSXUah8u2AfuBxyOiNrkDPyZ1oJzKGcSMnC9eBUlPAPOn+NMm4FvATdVGVJ2z5R4Rf4iITcAmSRuB24HvVBpgiabLvfGcTaTDCfdXGVsV2sm/RjTFstrs0RtImgX8DvjKpF78rhYRE8CyxnmnD0q6KiK6/tw8STcD+yPiWUkfyRlL1xdZEXHDVMslvR9YAmyXBOmQ0VZJKyJiX4UhluZMuU/h18DDdFGRNV3ukj4P3AxcH104jsn/8d7XwQiwsGV+AbA3UyxWMUl9pALr/oj4fe54coiIQ5L+TDo3r+uLLOA6YK2kjwIDwLsk/SoiPlN1ILU9XBgROyPi3RGxOCIWk76IP9QtBdZ0JC1tmV0LPJ8rlqpJWg18A1gbEUdzx2Ol2wIslbREUj+wAdicOSargNIe9M+B3RHxw9zxVEnSRcWV05IGgRuoyfd8RGyMiAWN3/YNwJ9yFFhQ4yLL+IGk5yTtIB0yrc2lzcBPgNnA440hLO7JHVCVJH1c0gjwYeBhSY/mjqlMjYscbgceJZ34/NuI2JU3qupI+g3wF+C9kkYkfSl3TBW6DvgssKqxrW9r9G7UwSXAk43v+C2kc7KyDWVQVx7x3czMzKwE7skyMzMzK4GLLDMzM7MSuMgyMzMzK4GLLDMzM7MSuMgyMzMzK4GLLDMzM7MSuMgyMzMzK4GLLDMzM7MS/A86TnAB6y6G7gAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 720x432 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Conduct hypothesis test\n",
"xControl = control['completed_form'].sum()\n",
"xTest = test['completed_form'].sum()\n",
"lift, zStat, pVal, statSig = twoProportionTest(xControl, xTest, nControl, nTest)\n",
"\n",
"#Plot\n",
"twoProportionPlot(zStat)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our results show that the 5-field form in the test group resulted in a lift of 3 percentage points in the proportion of completed forms to form loads compared to the control group receiving the 6-field form. This result is statistically significant at the 5% level (blue shaded region in plot) and a even stricter than the 1% level, leading us to reject the null hypothesis that there was no lift between the two form variations.\n",
"\n",
"Following up this analysis, it is necessary to consider the total conversion rate through the entire funnel. Since the 5-field form feature will cost $2 for each applicant passing throught the first stage of the checkout funnel, a thorough cost-benefit analysis should be conducted. Preliminary analysis indicates that conversion in the first stage of the funnel does not translate into conversions into sales. If this is the case, implementing the 5-field form will result in a negative impact on profits."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": true,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": true,
"toc_position": {
"height": "calc(100% - 180px)",
"left": "10px",
"top": "150px",
"width": "165px"
},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
"nbformat_minor": 4
}