{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# DISPLAY OPTIONS (uncomment and customize to activate)\n",
"\n",
"# prevent annoying tendency of not displaying 'middle' columns if data is very wide\n",
"# change the max_columns value for even wider datasets\n",
"\n",
"# pd.options.display.max_columns = None\n",
"# pd.options.display.max_columns = 40\n",
"\n",
"# see \"all\" the rows if needed\n",
"# change max_rows to desired number\n",
"\n",
"# pd.options.display.max_rows = 105\n",
"\n",
"# pd.reset_option('^display.', silent=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Analysis of Font change on Website\n",
"## Project Goal : Determine if new font resulted in more food product sales by analyzing A/A/B testing via sales funnel."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Project Description\n",
"\n",
"We sell food products. We need to investigate user behavior for the company's app.\n",
"\n",
"We will study the sales funnel. Find out how users reach the purchase stage. How many users actually make it to this stage? How many get stuck at previous stages? Which stages in particular?\n",
"\n",
"Then look at the results of an A/A/B test. \n",
"\n",
"**PROBLEM:** The designers would like to change the fonts for the entire app, but the managers are afraid the users might find the new design intimidating. \n",
"\n",
"**SOLUTION:** They decide to make a decision based on the results of an A/A/B test.\n",
"\n",
"The users are split into three groups: two control groups get the old fonts and one test group gets the new ones. Find out which set of fonts produces better results.\n",
"\n",
"Creating two A groups has certain advantages. We can make it a principle that we will only be confident in the accuracy of our testing when the two control groups are similar. If there are significant differences between the A groups, this can help us uncover factors that may be distorting the results. Comparing control groups also tells us how much time and data we'll need when running further tests.\n",
"\n",
"We'll be using the same dataset for general analytics and for A/A/B analysis. In real projects, experiments are constantly being conducted. Analysts study the quality of an app using general data, without paying attention to whether users are participating in experiments.\n",
"\n",
"Description of the data\n",
"\n",
"Each log entry is a user action or an event.\n",
"\n",
"* ```EventName``` — event name (we will change to ```event_name```)\n",
"* ```DeviceIDHash``` — unique user identifier (we will change to ```user_id```)\n",
"* ```EventTimestamp``` — event time (we will change to ```date_time```)\n",
"* ```ExpId``` — experiment number: 246 and 247 are the control groups, 248 is the test group (we will change to ```group```)\n",
"\n",
"\n",
"### Step 1. Open the data file and read the general information\n",
"\n",
"File path: /datasets/logs_exp_us.csv Download dataset\n",
"\n",
"### Step 2. Prepare the data for analysis\n",
"\n",
"* Rename the columns in a way that's convenient for you\n",
"* Check for missing values and data types. Correct the data if needed\n",
"* Add a date and time column and a separate column for dates\n",
"\n",
"### Step 3. Study and check the data\n",
"\n",
"* How many events are in the logs?\n",
"* How many users are in the logs?\n",
"* What's the average number of events per user?\n",
"* What period of time does the data cover? Find the maximum and the minimum date. Plot a histogram by date and time. Can you be sure that you have equally complete data for the entire period? Older events could end up in some users' logs for technical reasons, and this could skew the overall picture. Find the moment at which the data starts to be complete and ignore the earlier section. What period does the data actually represent?\n",
"* Did you lose many events and users when excluding the older data?\n",
"* Make sure you have users from all three experimental groups.\n",
"\n",
"### Step 4. Study the event funnel\n",
"\n",
"* See what events are in the logs and their frequency of occurrence. Sort them by frequency.\n",
"* Find the number of users who performed each of these actions. Sort the events by the number of users. Calculate the proportion of users who performed the action at least once.\n",
"* In what order do you think the actions took place. Are all of them part of a single sequence? You don't need to take them into account when calculating the funnel.\n",
"* Use the event funnel to find the share of users that proceed from each stage to the next. (For instance, for the sequence of events A → B → C, calculate the ratio of users at stage B to the number of users at stage A and the ratio of users at stage C to the number at stage B.)\n",
"* At what stage do you lose the most users?\n",
"* What share of users make the entire journey from their first event to payment?\n",
"\n",
"### Step 5. Study the results of the experiment\n",
"\n",
"* How many users are there in each group?\n",
"* We have two control groups in the A/A test, where we check our mechanisms and calculations. See if there is a statistically significant difference between samples 246 and 247.\n",
"* Select the most popular event. In each of the control groups, find the number of users who performed this action. Find their share. Check whether the difference between the groups is statistically significant. Repeat the procedure for all other events (it will save time if you create a special function for this test). Can you confirm that the groups were split properly?\n",
"* Do the same thing for the group with altered fonts. Compare the results with those of each of the control groups for each event in isolation. Compare the results with the combined results for the control groups. What conclusions can you draw from the experiment?\n",
"* What significance level have you set to test the statistical hypotheses mentioned above? Calculate how many statistical hypothesis tests you carried out. With a statistical significance level of 0.1, one in 10 results could be false. What should the significance level be? If you want to change it, run through the previous steps again and check your conclusions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Table of Contents\n",
"\n",
"Opening Data\n",
" \n",
"[Data Preprocessing](#preprocessing_data)\n",
" \n",
"[Study and check the data](#data_analysis)\n",
"\n",
"[Study the event funnel](#funnel)\n",
"\n",
"[Study the results of the experimnent](#results)\n",
"\n",
"[Conclusion](#conclusion)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Opening the Data"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"from scipy import stats as st\n",
"import math as mth\n",
"%matplotlib inline\n",
"\n",
"sns.set_color_codes(\"pastel\")\n",
"\n",
"#practicum path\n",
"#path = \"/datasets/\"\n",
"\n",
"#local path\n",
"path = 'datasets/'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Note about path:\n",
"In the above cell, I have two path options. The local path works for me, and I have found I need to add the first forward slash for the reviewer's to work. I just toggle which one. I think in the last case, I hadn't put in the second forward slash. Hopefully, it works for you now."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(path + 'logs_exp_us.csv', sep = '\\t')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
" event_name user_id group \\\n",
"date_time \n",
"2019-07-30 08:19:44 MainScreenAppear 5613408041324010552 248 \n",
"2019-07-31 21:51:39 CartScreenAppear 1694940645335807244 248 \n",
"2019-08-01 02:59:37 MainScreenAppear 434103746454591587 248 \n",
"2019-08-01 03:47:46 MainScreenAppear 3761373764179762633 247 \n",
"2019-08-01 04:44:01 MainScreenAppear 2835328739789306622 248 \n",
"... ... ... ... \n",
"2019-08-07 19:26:44 MainScreenAppear 8870358373313968633 247 \n",
"2019-08-07 19:26:45 PaymentScreenSuccessful 4718002964983105693 247 \n",
"2019-08-07 19:27:29 PaymentScreenSuccessful 2382591782303281935 246 \n",
"2019-08-07 19:27:29 CartScreenAppear 2382591782303281935 246 \n",
"2019-08-07 19:36:58 MainScreenAppear 4097782667445790512 246 \n",
"\n",
" date_time event_date event_time \n",
"date_time \n",
"2019-07-30 08:19:44 2019-07-30 08:19:44 2019-07-30 08:19:44 \n",
"2019-07-31 21:51:39 2019-07-31 21:51:39 2019-07-31 21:51:39 \n",
"2019-08-01 02:59:37 2019-08-01 02:59:37 2019-08-01 02:59:37 \n",
"2019-08-01 03:47:46 2019-08-01 03:47:46 2019-08-01 03:47:46 \n",
"2019-08-01 04:44:01 2019-08-01 04:44:01 2019-08-01 04:44:01 \n",
"... ... ... ... \n",
"2019-08-07 19:26:44 2019-08-07 19:26:44 2019-08-07 19:26:44 \n",
"2019-08-07 19:26:45 2019-08-07 19:26:45 2019-08-07 19:26:45 \n",
"2019-08-07 19:27:29 2019-08-07 19:27:29 2019-08-07 19:27:29 \n",
"2019-08-07 19:27:29 2019-08-07 19:27:29 2019-08-07 19:27:29 \n",
"2019-08-07 19:36:58 2019-08-07 19:36:58 2019-08-07 19:36:58 \n",
"\n",
"[413 rows x 6 columns]"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df[df.duplicated()]"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"df = df.drop_duplicates()"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(243713, 6)"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data Preprocessing Conclusion\n",
"* We had no nulls, but did have 413 duplicate rows, which we dropped. 413 out of 244K doesn't seem like a lot, but in and of itself, that seems like a significant number of duplicates. **SUGGESTED ACTION**: follow up with data engineering to see why there are so many duplicates\n",
"* We converted the timestamp into datetime, as well as into separate date and time columns\n",
"* We have a cleaned up dataset of 243713 observations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Study and check the data\n",
" * How many events are in the logs?\n",
" * How many users are in the logs?\n",
" * What's the average number of events per user?\n",
" * What period of time does the data cover? \n",
" * Find the maximum and the minimum date\n",
" * Plot a histogram by date and time. \n",
" Can you be sure that you have equally complete data for the entire period? Older events could end up in some users' logs for technical reasons, and this could skew the overall picture. \n",
" * Find the moment at which the data starts to be complete and ignore the earlier section. What period does the data actually represent?\n",
" * Did you lose many events and users when excluding the older data?\n",
" * Make sure you have users from all three experimental groups.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
event_name
\n",
"
user_id
\n",
"
group
\n",
"
date_time
\n",
"
event_date
\n",
"
event_time
\n",
"
\n",
"
\n",
"
date_time
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
"
\n",
"
2019-07-25 04:43:36
\n",
"
MainScreenAppear
\n",
"
4575588528974610257
\n",
"
246
\n",
"
2019-07-25 04:43:36
\n",
"
2019-07-25
\n",
"
04:43:36
\n",
"
\n",
"
\n",
"
2019-07-25 11:11:42
\n",
"
MainScreenAppear
\n",
"
7416695313311560658
\n",
"
246
\n",
"
2019-07-25 11:11:42
\n",
"
2019-07-25
\n",
"
11:11:42
\n",
"
\n",
"
\n",
"
2019-07-25 11:28:47
\n",
"
PaymentScreenSuccessful
\n",
"
3518123091307005509
\n",
"
248
\n",
"
2019-07-25 11:28:47
\n",
"
2019-07-25
\n",
"
11:28:47
\n",
"
\n",
"
\n",
"
2019-07-25 11:28:47
\n",
"
CartScreenAppear
\n",
"
3518123091307005509
\n",
"
248
\n",
"
2019-07-25 11:28:47
\n",
"
2019-07-25
\n",
"
11:28:47
\n",
"
\n",
"
\n",
"
2019-07-25 11:48:42
\n",
"
PaymentScreenSuccessful
\n",
"
6217807653094995999
\n",
"
248
\n",
"
2019-07-25 11:48:42
\n",
"
2019-07-25
\n",
"
11:48:42
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" event_name user_id group \\\n",
"date_time \n",
"2019-07-25 04:43:36 MainScreenAppear 4575588528974610257 246 \n",
"2019-07-25 11:11:42 MainScreenAppear 7416695313311560658 246 \n",
"2019-07-25 11:28:47 PaymentScreenSuccessful 3518123091307005509 248 \n",
"2019-07-25 11:28:47 CartScreenAppear 3518123091307005509 248 \n",
"2019-07-25 11:48:42 PaymentScreenSuccessful 6217807653094995999 248 \n",
"\n",
" date_time event_date event_time \n",
"date_time \n",
"2019-07-25 04:43:36 2019-07-25 04:43:36 2019-07-25 04:43:36 \n",
"2019-07-25 11:11:42 2019-07-25 11:11:42 2019-07-25 11:11:42 \n",
"2019-07-25 11:28:47 2019-07-25 11:28:47 2019-07-25 11:28:47 \n",
"2019-07-25 11:28:47 2019-07-25 11:28:47 2019-07-25 11:28:47 \n",
"2019-07-25 11:48:42 2019-07-25 11:48:42 2019-07-25 11:48:42 "
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### How many events are in the logs?"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"There are 5 types of events, numbering in total 243713 total events.\n"
]
}
],
"source": [
"print(\"There are \" + str(df.event_name.nunique()) + \n",
" \" types of events, numbering in total \" +\n",
" str(df.shape[0]) +\n",
" \" total events.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### How many users are in the logs?"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"There are 7551 users.\n"
]
}
],
"source": [
"print(\"There are \" + str(df.user_id.nunique()) + \" users.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### What's the average number of events per user?"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"df_by_user = df.groupby('user_id').agg({'event_name':'count'}).reset_index()"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The average number of events per user is 32.27559263673685\n"
]
}
],
"source": [
"print(\"The average number of events per user is \" + str(df_by_user.event_name.mean()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### What period of time does the data cover?"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The time period covered by the data is from 2019-07-25 to 2019-08-07\n"
]
}
],
"source": [
"print(\"The time period covered by the data is from \" +\n",
" str(df.event_date.min()) +\n",
" \" to \" +\n",
" str(df.event_date.max()))"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"df_by_datetime = df.groupby('event_date').agg({'event_name':'count'}).reset_index()"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"sns.set(style = 'whitegrid')\n",
"\n",
"\n",
"f, ax = plt.subplots(figsize = (12, 9))\n",
"sns.barplot(x = 'event_date',\n",
" y = 'event_name',\n",
" data = df_by_datetime,\n",
" color = 'b',\n",
" alpha = 0.6).set_title(\"Number of events by date\", fontsize = 16)\n",
"ax.set(xlabel=\"Log Date\", ylabel = \"Number of Events\")\n",
"ax.set_xticklabels(ax.get_xticklabels(), rotation=45);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Observation\n",
"We can see that prior to August 1, 2019, there is barely any events logged. This can be due to a number of possible reasons:\n",
"\n",
"1. The website itself was not officially launched; perhaps those small early numbers were testers trying out the system\n",
"2. The sales funnel was not launched until a few days prior to August 1, and those small early numbers again were from testing\n",
"3. The website and sales funnel were installed, and a marketing campaign began on August 1 -- a combination of online ads and influencers. Perhaps those early numbers were also influencers invited to review the product (and got a coupon to get free product in return?)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"df_users_by_datetime = df.groupby('group').agg({'event_date':'count'}).reset_index()"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
" event_date group event_name\n",
"0 2019-07-25 246 4\n",
"1 2019-07-25 247 1\n",
"2 2019-07-25 248 4\n",
"3 2019-07-26 246 14\n",
"4 2019-07-26 247 8\n",
"5 2019-07-26 248 9\n",
"6 2019-07-27 246 24\n",
"7 2019-07-27 247 23\n",
"8 2019-07-27 248 8\n",
"9 2019-07-28 246 33\n",
"10 2019-07-28 247 36\n",
"11 2019-07-28 248 36\n",
"12 2019-07-29 246 55\n",
"13 2019-07-29 247 58\n",
"14 2019-07-29 248 71\n",
"15 2019-07-30 246 129\n",
"16 2019-07-30 247 138\n",
"17 2019-07-30 248 145\n",
"18 2019-07-31 246 620\n",
"19 2019-07-31 247 664\n",
"20 2019-07-31 248 746\n",
"21 2019-08-01 246 11561\n",
"22 2019-08-01 247 12306\n",
"23 2019-08-01 248 12274\n",
"24 2019-08-02 246 10946\n",
"25 2019-08-02 247 10990\n",
"26 2019-08-02 248 13618\n",
"27 2019-08-03 246 10575\n",
"28 2019-08-03 247 11024\n",
"29 2019-08-03 248 11683\n",
"30 2019-08-04 246 11514\n",
"31 2019-08-04 247 9942\n",
"32 2019-08-04 248 11512\n",
"33 2019-08-05 246 12368\n",
"34 2019-08-05 247 10949\n",
"35 2019-08-05 248 12741\n",
"36 2019-08-06 246 11726\n",
"37 2019-08-06 247 11720\n",
"38 2019-08-06 248 12342\n",
"39 2019-08-07 246 10612\n",
"40 2019-08-07 247 10091\n",
"41 2019-08-07 248 10393"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.groupby(['event_date', 'group']).agg({'event_name':'count'}).reset_index()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All three groups appear to be accounted for if we drop data prior to August 1st."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"DatetimeIndex: 243713 entries, 2019-07-25 04:43:36 to 2019-08-07 21:15:17\n",
"Data columns (total 6 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 event_name 243713 non-null object \n",
" 1 user_id 243713 non-null int64 \n",
" 2 group 243713 non-null int64 \n",
" 3 date_time 243713 non-null datetime64[ns]\n",
" 4 event_date 243713 non-null object \n",
" 5 event_time 243713 non-null object \n",
"dtypes: datetime64[ns](1), int64(2), object(3)\n",
"memory usage: 13.0+ MB\n"
]
}
],
"source": [
"df.info()"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [],
"source": [
"# drop data from before August 1\n",
"df = df[df['date_time'] >= '2019-08-01']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conlusion on data study and prep\n",
"* We eliminated all rows prior to August 1, 2019 -- with minimal impact, only some 2000 rows total out of 244K\n",
"* There are about 7500 users\n",
"* There are 5 event types that the user experiences\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Study the event funnel\n",
"\n",
"* See what events are in the logs and their frequency of occurrence. Sort them by frequency.\n",
"* Find the number of users who performed each of these actions. Sort the events by the number of users. Calculate the proportion of users who performed the action at least once.\n",
"* In what order do you think the actions took place. Are all of them part of a single sequence? You don't need to take them into account when calculating the funnel.\n",
"* Use the event funnel to find the share of users that proceed from each stage to the next. (For instance, for the sequence of events A → B → C, calculate the ratio of users at stage B to the number of users at stage A and the ratio of users at stage C to the number at stage B.)\n",
"* At what stage do you lose the most users?\n",
"* What share of users make the entire journey from their first event to payment?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### See what events are in the logs and their frequency of occurrence. Sort them by frequency."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"df_by_events = df.groupby(['event_name']).agg({'user_id':'count'}).sort_values('user_id', ascending = False).reset_index()"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
event_name
\n",
"
user_id
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
MainScreenAppear
\n",
"
117328
\n",
"
\n",
"
\n",
"
1
\n",
"
OffersScreenAppear
\n",
"
46333
\n",
"
\n",
"
\n",
"
2
\n",
"
CartScreenAppear
\n",
"
42303
\n",
"
\n",
"
\n",
"
3
\n",
"
PaymentScreenSuccessful
\n",
"
33918
\n",
"
\n",
"
\n",
"
4
\n",
"
Tutorial
\n",
"
1005
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" event_name user_id\n",
"0 MainScreenAppear 117328\n",
"1 OffersScreenAppear 46333\n",
"2 CartScreenAppear 42303\n",
"3 PaymentScreenSuccessful 33918\n",
"4 Tutorial 1005"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_by_events"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Observation\n",
"* Grouping by events shows that the MainScreenAppear is number one\n",
"* The Tutorial event has little to do with this study, and we will not include it when crunching the numbers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"### Find the number of users who performed each of these actions. \n",
"- Sort the events by the number of users. \n",
"- Calculate the proportion of users who performed the action at least once."
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"def create_df_events_by_unique_users(df):\n",
" new_df = df.groupby(['event_name']).agg({'user_id':'nunique'}).sort_values('user_id', ascending = False).reset_index()\n",
" new_df.columns = ['event_name', 'unique_users']\n",
" return new_df"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
"df_events_unique_users = create_df_events_by_unique_users(df)"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
event_name
\n",
"
unique_users
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
MainScreenAppear
\n",
"
7419
\n",
"
\n",
"
\n",
"
1
\n",
"
OffersScreenAppear
\n",
"
4593
\n",
"
\n",
"
\n",
"
2
\n",
"
CartScreenAppear
\n",
"
3734
\n",
"
\n",
"
\n",
"
3
\n",
"
PaymentScreenSuccessful
\n",
"
3539
\n",
"
\n",
"
\n",
"
4
\n",
"
Tutorial
\n",
"
840
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" event_name unique_users\n",
"0 MainScreenAppear 7419\n",
"1 OffersScreenAppear 4593\n",
"2 CartScreenAppear 3734\n",
"3 PaymentScreenSuccessful 3539\n",
"4 Tutorial 840"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_events_unique_users"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Order of events\n",
"When we sort by number of users in reverse order, we can see that, with the exception of tutorial, the order of events naturally follow the funnel shape.\n",
"\n",
"The MainScreenAppear has the most users, which makes sense. That's most likely the first page a visitor will see (by typing in the website, or clicking a link via search results, etc)\n",
"\n",
"Then, while the user is scrolling perhaps, or is about to navigate away, the OffersScreenAppear event occurs. This could also be a link that they click. Or it's something that pops up when they add an item to the shopping cart, etc.\n",
"\n",
"Now, the user finally goes to the cart to see what s/he has addeded, and then from there a certain percentage decides to finally clikc the \"pay now\" button, which brings up the final PaymentScreenSuccessful event."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Ratios of users\n",
"Use the event funnel to find the share of users that proceed from each stage to the next. (For instance, for the sequence of events A → B → C, calculate the ratio of users at stage B to the number of users at stage A and the ratio of users at stage C to the number at stage B.)\n",
"* At what stage do you lose the most users?\n",
"* What share of users make the entire journey from their first event to payment?"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"# We will drop the Tutorial event since it has nothing to do with the funnel.\n",
"df_events_unique_users = df_events_unique_users.drop(df_events_unique_users.index[4])\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"# This function will calculate the proportion of users that proceed from each stage to the next.\n",
"# Takes a dataframe grouped by event, and a group name so we know which group for which\n",
"# we're making the calculations\n",
"\n",
"def check_proportions(df_by_events, group):\n",
" print('For group ' + group + ':')\n",
" for i in range(1, len(df_by_events)):\n",
" print('The proportion of ' + df_by_events.loc[i, 'event_name'] +\n",
" ' users coming from ' +\n",
" df_by_events.loc[i-1, 'event_name'] +\n",
" ' is ' + \n",
" str(round((df_by_events.loc[i, 'unique_users'] /\n",
" df_by_events.loc[i-1, 'unique_users'] *\n",
" 100), 2)) + '%')"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"For group all:\n",
"The proportion of OffersScreenAppear users coming from MainScreenAppear is 61.91%\n",
"The proportion of CartScreenAppear users coming from OffersScreenAppear is 81.3%\n",
"The proportion of PaymentScreenSuccessful users coming from CartScreenAppear is 94.78%\n"
]
}
],
"source": [
"check_proportions(df_events_unique_users, 'all')"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"from plotly import graph_objects as go\n",
"\n",
"def show_funnel(df, group_name, color):\n",
" fig = go.Figure(go.Funnel(\n",
" name = group_name,\n",
" y = df['event_name'],\n",
" x = df['unique_users'],\n",
" textposition = \"inside\",\n",
" textinfo = \"value+percent previous\",\n",
" opacity = 0.65, marker = {\"color\": color,\n",
" \"line\": {\"width\": 1, \"color\": \"blue\"}},\n",
" connector = {\"line\": {\"color\": \"lightblue\", \"dash\": \"dot\", \"width\": 1}})\n",
" )\n",
"\n",
" fig.show()\n",
"\n",
"#fig = px.funnel(df_by_events, x='user_id', y='event_name')\n",
"#fig.show()"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"connector": {
"line": {
"color": "lightblue",
"dash": "dot",
"width": 1
}
},
"marker": {
"color": "deepskyblue",
"line": {
"color": "blue",
"width": 1
}
},
"name": "Group 246",
"opacity": 0.65,
"textinfo": "value+percent previous",
"textposition": "inside",
"type": "funnel",
"x": [
7419,
4593,
3734,
3539
],
"y": [
"MainScreenAppear",
"OffersScreenAppear",
"CartScreenAppear",
"PaymentScreenSuccessful"
]
}
],
"layout": {
"template": {
"data": {
"bar": [
{
"error_x": {
"color": "#2a3f5f"
},
"error_y": {
"color": "#2a3f5f"
},
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
}
},
"type": "bar"
}
],
"barpolar": [
{
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
}
},
"type": "barpolar"
}
],
"carpet": [
{
"aaxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"baxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"type": "carpet"
}
],
"choropleth": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "choropleth"
}
],
"contour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "contour"
}
],
"contourcarpet": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "contourcarpet"
}
],
"heatmap": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmap"
}
],
"heatmapgl": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmapgl"
}
],
"histogram": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "histogram"
}
],
"histogram2d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2d"
}
],
"histogram2dcontour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2dcontour"
}
],
"mesh3d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "mesh3d"
}
],
"parcoords": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "parcoords"
}
],
"pie": [
{
"automargin": true,
"type": "pie"
}
],
"scatter": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter"
}
],
"scatter3d": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter3d"
}
],
"scattercarpet": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattercarpet"
}
],
"scattergeo": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergeo"
}
],
"scattergl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergl"
}
],
"scattermapbox": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattermapbox"
}
],
"scatterpolar": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolar"
}
],
"scatterpolargl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolargl"
}
],
"scatterternary": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterternary"
}
],
"surface": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "surface"
}
],
"table": [
{
"cells": {
"fill": {
"color": "#EBF0F8"
},
"line": {
"color": "white"
}
},
"header": {
"fill": {
"color": "#C8D4E3"
},
"line": {
"color": "white"
}
},
"type": "table"
}
]
},
"layout": {
"annotationdefaults": {
"arrowcolor": "#2a3f5f",
"arrowhead": 0,
"arrowwidth": 1
},
"coloraxis": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"colorscale": {
"diverging": [
[
0,
"#8e0152"
],
[
0.1,
"#c51b7d"
],
[
0.2,
"#de77ae"
],
[
0.3,
"#f1b6da"
],
[
0.4,
"#fde0ef"
],
[
0.5,
"#f7f7f7"
],
[
0.6,
"#e6f5d0"
],
[
0.7,
"#b8e186"
],
[
0.8,
"#7fbc41"
],
[
0.9,
"#4d9221"
],
[
1,
"#276419"
]
],
"sequential": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"sequentialminus": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
]
},
"colorway": [
"#636efa",
"#EF553B",
"#00cc96",
"#ab63fa",
"#FFA15A",
"#19d3f3",
"#FF6692",
"#B6E880",
"#FF97FF",
"#FECB52"
],
"font": {
"color": "#2a3f5f"
},
"geo": {
"bgcolor": "white",
"lakecolor": "white",
"landcolor": "#E5ECF6",
"showlakes": true,
"showland": true,
"subunitcolor": "white"
},
"hoverlabel": {
"align": "left"
},
"hovermode": "closest",
"mapbox": {
"style": "light"
},
"paper_bgcolor": "white",
"plot_bgcolor": "#E5ECF6",
"polar": {
"angularaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"radialaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"scene": {
"xaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"yaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"zaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
}
},
"shapedefaults": {
"line": {
"color": "#2a3f5f"
}
},
"ternary": {
"aaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"baxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"caxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"title": {
"x": 0.05
},
"xaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"title": {
"standoff": 15
},
"zerolinecolor": "white",
"zerolinewidth": 2
},
"yaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"title": {
"standoff": 15
},
"zerolinecolor": "white",
"zerolinewidth": 2
}
}
}
}
},
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_funnel(df_events_unique_users, 'Group 246', 'deepskyblue')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Observation\n",
"We lose the most users from the MainScreenAppear to the OffersScreenAppear, i.e. the first event. So, the user comes onto the website, and basically right away decides the product isn't for him/her and leaves. Almost 40% of users who visit leave without making a purchase"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"def calculate_share_entire_journey(df_grouped):\n",
" return (str(round((df_grouped.loc[len(df_grouped) - 1, 'unique_users']) /\n",
" (df_grouped.loc[0, 'unique_users']) * 100, 2)) + '%')"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The share of users who make the entire journey from their first event to payment is: 47.7%\n"
]
}
],
"source": [
"print('The share of users who make the entire journey from their first event to payment is: ' +\n",
" (calculate_share_entire_journey(df_events_unique_users)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Study the results of the experiment\n",
"\n",
"* How many users are there in each group?\n",
"* We have two control groups in the A/A test, where we check our mechanisms and calculations. See if there is a statistically significant difference between samples 246 and 247.\n",
"* Select the most popular event. In each of the control groups, find the number of users who performed this action. Find their share. Check whether the difference between the groups is statistically significant. Repeat the procedure for all other events (it will save time if you create a special function for this test). Can you confirm that the groups were split properly?\n",
"* Do the same thing for the group with altered fonts. Compare the results with those of each of the control groups for each event in isolation. Compare the results with the combined results for the control groups. What conclusions can you draw from the experiment?\n",
"* What significance level have you set to test the statistical hypotheses mentioned above? Calculate how many statistical hypothesis tests you carried out. With a statistical significance level of 0.1, one in 10 results could be false. What should the significance level be? If you want to change it, run through the previous steps again and check your conclusions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Number of users in each group"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_funnel(df_248_events_users, 'Group 248', 'purple')"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The share of users who make the entire journey from their first event to payment is: 47.37%\n"
]
}
],
"source": [
"print('The share of users who make the entire journey from their first event to payment is: ' +\n",
" (calculate_share_entire_journey(df_248_events_users)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Observation\n",
"The percentage numbers are pretty close, so let's do some statistical tests\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [],
"source": [
"# I enclosed the code into a function that takes two dataframes and an alpha level\n",
"\n",
"def custom_ztest(df1, df2, alpha_level = 0.05):\n",
" \n",
" alpha = alpha_level\n",
" \n",
" print('Chosen alpha level: ', alpha, '\\n')\n",
" \n",
"\n",
" for i in range(1, len(df1)):\n",
" \n",
" successes = (df1.loc[i, 'unique_users'], df2.loc[i, 'unique_users'])\n",
" trials = (df1.loc[i-1, 'unique_users'], df2.loc[i-1, 'unique_users'])\n",
"\n",
" # success proportion in the first group:\n",
" p1 = successes[0]/trials[0]\n",
" \n",
" # success proportion in the second group:\n",
" p2 = successes[1]/trials[1]\n",
"\n",
" # success proportion in the combined dataset:\n",
" p_combined = (successes[0] + successes[1]) / (trials[0] + trials[1])\n",
" \n",
" # the difference between the datasets' proportions\n",
" difference = p1 - p2\n",
" \n",
" # print('sqrt operand: ', (p_combined * (1 - p_combined) * (1/trials[0] + 1/trials[1])))\n",
" \n",
" # calculating the statistic in standard deviations of the standard normal distribution\n",
" z_value = difference / mth.sqrt(abs(p_combined * (1 - p_combined) * (1/trials[0] + 1/trials[1])))\n",
" \n",
" # setting up the standard normal distribution (mean 0, standard deviation 1)\n",
" distr = st.norm(0, 1)\n",
"\n",
" p_value = (1 - distr.cdf(abs(z_value))) * 2\n",
"\n",
" print('p-value for proportion of ' + df1.loc[i, 'event_name'] +\n",
" ' users to that of ' +\n",
" df1.loc[i-1, 'event_name'] + ': ', p_value)\n",
"\n",
" if (p_value < alpha):\n",
" print(\"Rejecting the null hypothesis: there is a significant difference between the proportions\")\n",
" else:\n",
" print(\"Failed to reject the null hypothesis: there is no reason to consider the proportions different\")\n",
" print()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Null hypothesis #1:\n",
"That there is no significant differences between the two control groups from event to event when comparing proportions\n",
"### Alternate hyposthesis #1:\n",
"That there is significant difference between the two control groups from event to event when comparing proportions"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chosen alpha level: 0.05 \n",
"\n",
"p-value for proportion of OffersScreenAppear users to that of MainScreenAppear: 0.2622344959255778\n",
"Failed to reject the null hypothesis: there is no reason to consider the proportions different\n",
"\n",
"p-value for proportion of CartScreenAppear users to that of OffersScreenAppear: 0.6393754713145263\n",
"Failed to reject the null hypothesis: there is no reason to consider the proportions different\n",
"\n",
"p-value for proportion of PaymentScreenSuccessful users to that of CartScreenAppear: 0.18243439408067652\n",
"Failed to reject the null hypothesis: there is no reason to consider the proportions different\n",
"\n"
]
}
],
"source": [
"# test for statistical significance between groups 246 and 247\n",
"\n",
"custom_ztest(df_246_events_users, df_247_events_users)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Result: there is no significant difference in all event comparisons between 246 and 247\n",
"**Therefore,** we fail to reject the null hypothesis. So we can combine the two A/A groups together"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [],
"source": [
"# create a dataframe of both groups 246 and 247\n",
"\n",
"df_AA_group = df[df['group'] != 248]"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fig = go.Figure()\n",
"\n",
"fig.add_trace(go.Funnel(\n",
" name = 'AA Group',\n",
" y = df_AA_users['event_name'],\n",
" x = df_AA_users['unique_users'],\n",
" textinfo = \"value+percent previous\"))\n",
"\n",
"fig.add_trace(go.Funnel(\n",
" name = '248 users AKA B group',\n",
" orientation = \"h\",\n",
" y = df_248_events_users['event_name'],\n",
" x = df_248_events_users['unique_users'],\n",
" textinfo = \"value+percent previous\"))\n",
"\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The share of users who make the entire journey from their first event to payment is: 47.87%\n"
]
}
],
"source": [
"print('The share of users who make the entire journey from their first event to payment is: ' +\n",
" (calculate_share_entire_journey(df_AA_users)))"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The share of users who make the entire journey from their first event to payment is: 47.37%\n"
]
}
],
"source": [
"print('The share of users who make the entire journey from their first event to payment is: ' +\n",
" (calculate_share_entire_journey(df_248_events_users)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Null hypothesis #2:\n",
"That there is no significant differences between the AA group and the B group (aka group 248) from event to event\n",
"### Alternate hypothesis #2:\n",
"There is a significant difference between the AA group and the B group from event to event"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chosen alpha level: 0.05 \n",
"\n",
"p-value for proportion of OffersScreenAppear users to that of MainScreenAppear: 0.5308589432077624\n",
"Failed to reject the null hypothesis: there is no reason to consider the proportions different\n",
"\n",
"p-value for proportion of CartScreenAppear users to that of OffersScreenAppear: 0.23905954279947372\n",
"Failed to reject the null hypothesis: there is no reason to consider the proportions different\n",
"\n",
"p-value for proportion of PaymentScreenSuccessful users to that of CartScreenAppear: 0.017114434319547067\n",
"Rejecting the null hypothesis: there is a significant difference between the proportions\n",
"\n"
]
}
],
"source": [
"custom_ztest(df_AA_users, df_248_events_users)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Observation\n",
"The tests show that the only instance where there is significant difference is in the final part of the funnel, the PaymentScreenSuccessful event.\n",
"\n",
"Since this is (presumably) not a life/death situation, we choose the alpha level to be 0.05. Even with 0.1 as a suggestion, we would get the same results."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Conclusion"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### We analyzed a dataset of over 244k logs from about 7500 users to see if changing the fonts resulted in better sales.\n",
"\n",
"Our control group consisted of two A groups, and one test group (B group), each of which comprised ~2400 users.\n",
"\n",
"There were no significant differences between the two A groups, which indicated we can be confident in the accuracy of our testing.\n",
"\n",
"Intuitively, it would seem that the B group (group 248) did slightly better, but only a little bit. We can see that the PaymentScreenSuccessful proportions for the B group is almost a full 2% higher. However, in the A group the percentages for the first two parts of the funnels were around 1% higher in the A group.\n",
"\n",
"**UPDATE:** When we do a simple comparison of the amount of users from each group who make the entire journey, however, we get a difference of only 0.5%, with actually the control (A/A) group doing better. That difference turns out to be about 38 users.\n",
"\n",
"If anything, this test suggests that we should perhaps compare other metrics such as profits, to see if we can gain any more useful insights.\n",
"\n",
"Otherwise, from this testing, I would conclude that if there were any differences in switching fonts, that difference is very negligible. I would recommend we do something else that can show more significant difference.\n",
"\n",
"**RECOMMENDATION ON FONT CHANGE:** We suggest no change in font."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}