{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Event-based analytics\n", "\n", "I work at a startup that sells food products. I need to investigate user behavior for the company's app.\n", "First, I need to study the sales funnel and find out how users reach the purchase stage.\n", "- How many users actually make it to this stage?\n", "- How many get stuck at previous stages?\n", "- Which stages in particular?\n", "\n", "Then, I'll look at the results of an A/A/B test. The designers would like to change the fonts for the entire app, but the managers are afraid the users might find the new design intimidating. They decide to make a decision based on the results of the test.\n", "\n", "The users are split into three groups: two control groups get the old fonts and one test group gets the new ones. I need to find out which set of fonts produces better results.\n", "\n", "Creating two A groups has certain advantages. We can make it a principle that we will only be confident in the accuracy of our testing when the two control groups are similar. If there are significant differences between the A groups, this can help us uncover factors that may be distorting the results. Comparing control groups also tells us how much time and data we'll need when running further tests.\n", "\n", "## Description of the data\n", "\n", "*/datasets/logs_exp_us.csv*\n", "\n", "Each log entry is a user action or an event:\n", " - **EventName** — event name\n", " - **DeviceIDHash** — unique user identifier\n", " - **EventTimestamp** — event time\n", " - **ExpId** — experiment number (246 & 247 are the control groups, 248 is the test group)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 1: Download data & study general info" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Import libraries & data" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# import libraries\n", "\n", "import pandas as pd\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "import scipy.stats as st\n", "import numpy as np\n", "from plotly import graph_objects as go" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# try-except blocks handle errors that occur from changing file directories\n", "\n", "try:\n", " logs = pd.read_csv('logs_exp_us.csv', sep='\\t')\n", "except:\n", " logs = pd.read_csv('/datasets/logs_exp_us.csv', sep='\\t')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Study general info" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 244126 entries, 0 to 244125\n", "Data columns (total 4 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 EventName 244126 non-null object\n", " 1 DeviceIDHash 244126 non-null int64 \n", " 2 EventTimestamp 244126 non-null int64 \n", " 3 ExpId 244126 non-null int64 \n", "dtypes: int64(3), object(1)\n", "memory usage: 7.5+ MB\n" ] }, { "data": { "text/plain": [ "None" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EventNameDeviceIDHashEventTimestampExpId
0MainScreenAppear45755885289746102571564029816246
1MainScreenAppear74166953133115606581564053102246
2PaymentScreenSuccessful35181230913070055091564054127248
3CartScreenAppear35181230913070055091564054127248
4PaymentScreenSuccessful62178076530949959991564055322248
\n", "
" ], "text/plain": [ " EventName DeviceIDHash EventTimestamp ExpId\n", "0 MainScreenAppear 4575588528974610257 1564029816 246\n", "1 MainScreenAppear 7416695313311560658 1564053102 246\n", "2 PaymentScreenSuccessful 3518123091307005509 1564054127 248\n", "3 CartScreenAppear 3518123091307005509 1564054127 248\n", "4 PaymentScreenSuccessful 6217807653094995999 1564055322 248" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# study general info\n", "\n", "display(logs.info())\n", "display(logs.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Conclusion\n", "Immediately we can see there are a few issues we need to correct:\n", "- 'DeviceIDHash' & 'ExpId' should be changed to *object* dtype\n", "- 'EventTimestamps' should be changed to *datetime* dtype\n", "- We should rename columns to something more intuitive\n", "- We'll add a date & time column for easier analysis in the future\n", "- There are no missing values (no action needed)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 2: Data preprocessing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Changing data types" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# change 'DeviceIDHash' & 'ExpId' to object\n", "\n", "logs['DeviceIDHash'] = logs['DeviceIDHash'].astype(str)\n", "logs['ExpId'] = logs['ExpId'].astype(str)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# change 'EventTimestamp' to datetime\n", "\n", "logs['EventTimestamp'] = logs['EventTimestamp'].astype('datetime64[s]')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Change column names" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# change column names\n", "\n", "logs.columns = ['event', 'uid', 'datetime', 'group']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Add date & time columns\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# add columns & convert to datetime\n", "\n", "logs['time'] = pd.to_datetime(logs['datetime'], infer_datetime_format=True).dt.time\n", "logs['date'] = pd.to_datetime(logs['datetime'], infer_datetime_format=True).dt.date" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 244126 entries, 0 to 244125\n", "Data columns (total 6 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 event 244126 non-null object \n", " 1 uid 244126 non-null object \n", " 2 datetime 244126 non-null datetime64[ns]\n", " 3 group 244126 non-null object \n", " 4 time 244126 non-null object \n", " 5 date 244126 non-null object \n", "dtypes: datetime64[ns](1), object(5)\n", "memory usage: 11.2+ MB\n" ] }, { "data": { "text/plain": [ "None" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventuiddatetimegrouptimedate
0MainScreenAppear45755885289746102572019-07-25 04:43:3624604:43:362019-07-25
1MainScreenAppear74166953133115606582019-07-25 11:11:4224611:11:422019-07-25
2PaymentScreenSuccessful35181230913070055092019-07-25 11:28:4724811:28:472019-07-25
3CartScreenAppear35181230913070055092019-07-25 11:28:4724811:28:472019-07-25
4PaymentScreenSuccessful62178076530949959992019-07-25 11:48:4224811:48:422019-07-25
\n", "
" ], "text/plain": [ " event uid datetime group \\\n", "0 MainScreenAppear 4575588528974610257 2019-07-25 04:43:36 246 \n", "1 MainScreenAppear 7416695313311560658 2019-07-25 11:11:42 246 \n", "2 PaymentScreenSuccessful 3518123091307005509 2019-07-25 11:28:47 248 \n", "3 CartScreenAppear 3518123091307005509 2019-07-25 11:28:47 248 \n", "4 PaymentScreenSuccessful 6217807653094995999 2019-07-25 11:48:42 248 \n", "\n", " time date \n", "0 04:43:36 2019-07-25 \n", "1 11:11:42 2019-07-25 \n", "2 11:28:47 2019-07-25 \n", "3 11:28:47 2019-07-25 \n", "4 11:48:42 2019-07-25 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# verify results\n", "\n", "display(logs.info())\n", "display(logs.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 3: Study & analyze the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Number of events & users" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There were 244126 total events\n" ] } ], "source": [ "# count number of events\n", "\n", "total_events = logs['event'].count()\n", "print(f'There were {total_events} total events')" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There were 7551 unique users\n" ] } ], "source": [ "# count number of users\n", "\n", "total_users = logs['uid'].nunique()\n", "print(f'There were {total_users} unique users')" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Each user triggered an average of 32 events\n" ] } ], "source": [ "# average events per user\n", "\n", "avg_events_per_user = total_events / total_users\n", "print(f'Each user triggered an average of {int(avg_events_per_user)} events')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Data time period" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The earliest event was at 2019-07-25 04:43:36\n", "The latest event was at 2019-08-07 21:15:17\n" ] } ], "source": [ "min_datetime = logs['datetime'].min()\n", "max_datetime = logs['datetime'].max()\n", "\n", "print(f'The earliest event was at {min_datetime}')\n", "print(f'The latest event was at {max_datetime}')" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAFjCAYAAAA6vqL8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAl8ElEQVR4nO3de5gkdX3v8ffXBQFZLkvAlbOLLuqKclGUFRFD3BUEIigmSiQigjEhMajxhi7Rc4zPkSM5JkYRUDdiAPW4YmIiEQ3xEDbGBI+CtxXRwBNW5BLwAsgSY1z8nj/qN9KOM7s9M1Vd3VXv1/PUM9PV1f35/bpn6tu/quqqyEwkSVJ3PajtBkiSpGZZ7CVJ6jiLvSRJHWexlySp4yz2kiR1nMVekqSO267tBjRlzz33zBUrVsz5cffddx8777xz/Q0aw9w+9bWt3D71ta3cPvW1rdw+9bXN3IW69tprv5eZe814Z2Z2cjrkkENyPq666qp5PW6h2sjtU1/byu1TX9vK7VNf28rtU1/bzF0o4JqcpSa6GV+SpI6z2EuS1HEWe0mSOs5iL0lSx1nsJUnqOIu9JEkdZ7GXJKnjLPaSJHWcxV6SpI6z2EuS1HEWe0mSOs5iL0kamRVrL2fF2svbbkbvdPaqd5Kk8WGBb5cje0mSOs5iL0lSx1nsJUnqOIu9JEkdZ7GXJKnjLPaSJHWcxV6SpI6z2EuS1HEWe0mSOs5iL0lSx1nsJUnqOIu9JEkNGKeL/ljsJUmdNU4Ft00We0mSOs5iL0nqvL6P8C32kiR1nMVe0ljq+0hMqtN2bTdAkgZZ4KX6ObKXJKnjLPaSJHWcxV6SpI6z2EuS1HEWe0kaEb9hoLZY7CWpJRZ/jYrFXpKkjrPYS5LUcY0X+4hYFBFfjohPltt7RMRnIuKG8nPJwLJnRcSNEfGtiDhmYP4hEbGx3HduRETT7ZY0Gm7Klpo3ipH9HwDXD9xeC1yZmSuBK8ttImJ/4CTgAOBY4IKIWFQe8x7gdGBlmY4dQbsl9YwfPNRVjRb7iFgOHAe8f2D2CcDF5feLgecOzF+fmT/OzJuAG4FDI2JvYNfMvDozE7hk4DGSJGkbmh7ZvxN4PfDTgXlLM/N2gPLzoWX+MuA7A8vdUuYtK79Pny9JGoJbLBTVYLmBJ444HnhWZv5+RKwGXpeZx0fE3Zm5+8Byd2Xmkog4H7g6Mz9U5l8IfAq4GXhbZh5V5h8BvD4znz1D5ulUm/tZunTpIevXr59zuzdv3szixYvn/LiFaiO3T31tK7dPfZ1v7sZb7wHgoGW7/dztKVPz68wcti11587W12Hy6sydqzpe47m+r5OeO5g939d9rtasWXNtZq6a6b4mr3r3NOA5EfEsYEdg14j4EHBHROydmbeXTfR3luVvAfYZePxy4LYyf/kM839BZq4D1gGsWrUqV69ePedGb9iwgfk8bqHayO1TX9vK7VNf55t7Whlxbjp59c/dnjI1v87MYdtSd+5sfR0mr87cuarjNZ7+vrLxvqpN5xw30tym3tutZc/3da9TY5vxM/OszFyemSuoDrz7h8x8EXAZcGpZ7FTgE+X3y4CTImKHiNiX6kC8L5RN/fdGxGHlKPwXDzxGkiRtQxvXsz8HuDQiXkq1if5EgMy8LiIuBb4BbAHOyMz7y2NeBlwE7AR8ukySOsR9ylJzRlLsM3MDsKH8/n3gyFmWOxs4e4b51wAHNtdCSZK6yzPoSZLUcRZ7SZI6zmIvSVLHtXGAniSNFQ8OVNc5spckqeMs9pIkdZyb8SWpYe4mUNss9pKkxozbB52p9mztNL1d5GZ8SeoZr4LXPxZ7SZI6zs34ktRRjt41xZG9JEkdZ7GXJKnjLPaSxpoHk0kLZ7GXJKnjPEBPklQ7t8aMF0f2kiR1nCN7SZJqNI5bNRzZS5LUcY7sJUmdM46j6zZZ7CVpxKYXor5enEWj42Z8Seopz2HQHxZ7SZI6zmIvSVLHuc9eknpiFJvs3S0wnhzZS5LUcY7sJfXK4MjTo9/VF47sJUnqOEf2knphpn3J7l/WKIzDeRQc2UuS1HEWe0mSOs5iL0lSx7nPXlIrxnl/+TjsY5Xq5MhekqSOs9hLktRxFntJkjrOYi9JUsdZ7CVJ6jiLvSRp4q1Ye/lYf8OjbX71TpLUGRb8mTmylySp4xzZS1LHOLrVdI7sJUnqOEf2kkbKUac0ehZ7SZJqMM4fZN2ML0ljwq+PqSkWe0mSOs5iL0lSxzVW7CNix4j4QkR8NSKui4i3lPl7RMRnIuKG8nPJwGPOiogbI+JbEXHMwPxDImJjue/ciIim2i1JUtc0ObL/MfCMzHwCcDBwbEQcBqwFrszMlcCV5TYRsT9wEnAAcCxwQUQsKs/1HuB0YGWZjm2w3ZIkdUpjR+NnZgKby83ty5TACcDqMv9iYAPwhjJ/fWb+GLgpIm4EDo2ITcCumXk1QERcAjwX+HRTbZekOniwncZFVDW5oSevRubXAo8Gzs/MN0TE3Zm5+8Ayd2Xmkog4D/h8Zn6ozL+QqqBvAs7JzKPK/COAN2Tm8TPknU61BYClS5cesn79+jm3efPmzSxevHjOj1uoNnL71Ne2cvvU12FzN956z7ye+6Blu807c765s2UOm1t3X5vK3VresJlNZE9C7lzasK3XeaHWrFlzbWaumum+Rr9nn5n3AwdHxO7AX0fEgVtZfKb98LmV+TPlrQPWAaxatSpXr149p/YCbNiwgfk8bqHayO1TX9vK7VNfh809bZ6j3U0nz/y8w/Z1PrmzZQ6bW3dfm8rdWt6wmU1kT0LuXNqwrde5SSM5Gj8z76baXH8scEdE7A1Qft5ZFrsF2GfgYcuB28r85TPMlyRJQ2jyaPy9yoieiNgJOAr4JnAZcGpZ7FTgE+X3y4CTImKHiNiX6kC8L2Tm7cC9EXFYOQr/xQOPkSSpVZNwMqQmN+PvDVxc9ts/CLg0Mz8ZEVcDl0bES4GbgRMBMvO6iLgU+AawBTij7AYAeBlwEbAT1X58D86TJGlITR6N/zXgiTPM/z5w5CyPORs4e4b51wBb298vSZJm4YVwJEkTa9w3n48LT5crSVLHWewlaRaTcOCVNAw340uSNA+T9EHQkb0k9ZxbMLrPYi9JUse5GV9Spzli7Sbf17lxZC9J0gi0ubvEYi9JUsdZ7CVJ6jiLvSRJHWexlySp4yz2kiR1nMVekqSOs9hLklrj2ftGw2IvSVLHDVXsI+Jpw8yTJEnjZ9iR/buHnCdJksbMVs+NHxFPBQ4H9oqI1wzctSuwqMmGSZKkemzrQjgPBhaX5XYZmP9D4PlNNUqSpJl4MN/8bLXYZ+Y/Av8YERdl5rdH1CZJklSjYS9xu0NErANWDD4mM5/RRKMkabqpEd2mc45ruSXS5Bm22H8MeC/wfuD+5pojSZLqNmyx35KZ72m0JZKkieW+9PE27Ffv/jYifj8i9o6IPaamRlsmSZJqMezI/tTy88yBeQk8st7mSJLUvL4dAzJUsc/MfZtuiKRuczOv1J6hin1EvHim+Zl5Sb3NkSRJdRt2M/6TB37fETgS+BJgsZekHnOLzWQYdjP+KwZvR8RuwAcbaZEkSarVsCP76f4DWFlnQyRJC+MoW7MZdp/931IdfQ/VBXAeB1zaVKMkSVJ9hh3Z/8nA71uAb2fmLQ20R5Ik1Wyok+qUC+J8k+rKd0uA/2qyUZIkqT7Dbsb/DeDtwAYggHdHxJmZ+ZcNtk2SeqlvJ3yZNJN4bMSwm/HfCDw5M+8EiIi9gP8LWOwlSRpzw54b/0FThb74/hweK0mSWjTsyP7vIuIK4CPl9guATzXTJEmSVKetFvuIeDSwNDPPjIhfB36Zap/91cCHR9A+SZK0QNvaFP9O4F6AzPx4Zr4mM19NNap/Z7NNk6TxsGLt5RN5UJY0ZVvFfkVmfm36zMy8BljRSIskSVKttlXsd9zKfTvV2RBJktSMbR2g98WI+J3M/PPBmRHxUuDa5polSZPLTf4aN9sq9q8C/joiTuaB4r4KeDDwaw22S5Ik1WSrxT4z7wAOj4g1wIFl9uWZ+Q+Nt0ySJNVi2OvZXwVc1XBbJElSAzwLniRJHWexlySp4xor9hGxT0RcFRHXR8R1EfEHZf4eEfGZiLih/Fwy8JizIuLGiPhWRBwzMP+QiNhY7js3IqKpdkuS1DVNjuy3AK/NzMcBhwFnRMT+wFrgysxcCVxZblPuOwk4ADgWuCAiFpXneg9wOrCyTMc22G5JkjqlsWKfmbdn5pfK7/cC1wPLgBOAi8tiFwPPLb+fAKzPzB9n5k3AjcChEbE3sGtmXp2ZCVwy8BhJkrQNUdXPhkMiVgCfpfr63s2ZufvAfXdl5pKIOA/4fGZ+qMy/EPg0sAk4JzOPKvOPAN6QmcfPkHM61RYAli5desj69evn3NbNmzezePHiOT9uodrI7VNf28rtU1+3lbvx1ntqyTho2W5DZ9aZOz27jb42nTtTXtOZW8ueKbfO93MuuXW2YbbXeaHWrFlzbWaumum+YS9xO28RsRj4K+BVmfnDrexun+mO3Mr8X5yZuQ5YB7Bq1apcvXr1nNu7YcMG5vO4hWojt099bSu3T33dVu5pNZ1VbtPJP//82+prXbnTs9voa9O5M+U1nbm17Jly63w/55JbZxtme52b1OjR+BGxPVWh/3BmfrzMvqNsmqf8vLPMvwXYZ+Dhy4HbyvzlM8yXJElDaPJo/AAuBK7PzHcM3HUZcGr5/VTgEwPzT4qIHSJiX6oD8b6QmbcD90bEYeU5XzzwGEmStA1NbsZ/GnAKsDEivlLm/SFwDnBpuZjOzcCJAJl5XURcCnyD6kj+MzLz/vK4lwEXUV1p79NlkiRJQ2is2Gfm55h5fzvAkbM85mzg7BnmX8MD5+aXJGnkJvlqhp5BT5KkjrPYS5LUcRZ7SZI6zmIvSVLHWewlSeq4xs+gJ0maDINHm28657gWW6K6ObKXJKnjLPaSJHWcxV6SpI5zn72kTprks51JdXNkL0lSx1nsJUnqOIu9JEkdZ7GXJKnjPEBPUqM8UE6TrCt/v47sJUnqOIu9JEkd52Z8SZpwXdnUrOY4spckqeMs9pKk1q1Ye7lbKBpksZckqeMs9pIkdZzFXpKkjrPYS5I0Qm0cn2CxlySp4yz2kiR1nMVekqSOs9hLktRxFntJkjrOYi9JUsdZ7CVJ6jiLvSRJHWexl9QJXkhFmp3Xs5ckzZkfrCaLI3tJkjrOYi9JUsdZ7CVJ6jiLvSRJHWexlySp4yz2kibaxlvv8chwaRv86p2kTrHwS7/IYi9JNfGDhsaVm/ElSeo4i70kSR1nsZckqeMs9pIkdZzFXpKkjmus2EfEByLizoj4+sC8PSLiMxFxQ/m5ZOC+syLixoj4VkQcMzD/kIjYWO47NyKiqTZLksaTlzBemCZH9hcBx06btxa4MjNXAleW20TE/sBJwAHlMRdExKLymPcApwMryzT9OSVJ0lY0Vuwz87PAD6bNPgG4uPx+MfDcgfnrM/PHmXkTcCNwaETsDeyamVdnZgKXDDxGkiQNYdT77Jdm5u0A5edDy/xlwHcGlrulzFtWfp8+X5IkDSmqAXNDTx6xAvhkZh5Ybt+dmbsP3H9XZi6JiPOBqzPzQ2X+hcCngJuBt2XmUWX+EcDrM/PZs+SdTrXJn6VLlx6yfv36Obd58+bNLF68eM6PW6g2cvvU17Zy+9TX2XI33npPI1kHLdsNgDt/cA93/KiRiFkzYTR9HcwbZe707Dbe1+m5o8gbZe5s2QuxZs2aazNz1Uz3jfp0uXdExN6ZeXvZRH9nmX8LsM/AcsuB28r85TPMn1FmrgPWAaxatSpXr1495wZu2LCB+TxuodrI7VNf28rtU19nyz2toYOqNp1c5bz7w5/gTzeOZlU2lQmj6etg3ihzp2e38b5Ozx1F3ihzZ8tuyqg3418GnFp+PxX4xMD8kyJih4jYl+pAvC+UTf33RsRh5Sj8Fw88RpLUEI9+75bGPg5HxEeA1cCeEXEL8GbgHODSiHgp1Sb6EwEy87qIuBT4BrAFOCMz7y9P9TKqI/t3Aj5dJknqvKliu+mc41puiSZdY8U+M39zlruOnGX5s4GzZ5h/DXBgjU2TNAKOCqXx4Rn0JEnqOIu9JEkdN+qj8SVJNXFXiYblyF6S1Ft9+daBxV6SpI6z2EuS1HEWe0mSOs5iL0lSx1nsJUnqOIu9JEkdZ7GXJKnjLPaSJHWcxV6SpI7zdLmStEB9OAPblD71tUsc2UuS1HEWe0mSOs5iL0lSx1nsJUnqOIu9JEkdZ7GXJKnjLPaSNKQVay/3q2eaSH7PXpI0NqY+TG0657ixaEdXOLKXJKnjLPaSJHWcxV7SgrgfWxp/FntJkjrOYi9JUsd5NL6kWrlJXxo/FntJE2nqQ8VrD2q5IdIEcDO+JEkd58hekubJXRaaFBZ7SfNioWvPKF/7FWsv57UHbWH1yBLVBIu9JGls+aGyHu6zlySp4yz2kjTm+niWwhVrL2fjrfe03YzOsNhLktRxFntJkjrOA/QkSdvU9d0IU/3bdM5xLbekGY7sJdWiT/tY+9RXdYPFXpImhB8yNF8We0mSiq5+oHKfvaStmr6vtqv7NKUus9hLmlHXD8iS+sTN+JIkdZwje0k/Z1sjekf80uSx2EsCLOJSl7kZX5KkjnNkL/WcI3qp+yam2EfEscC7gEXA+zPznJabJI1cnaf0tMhL/TERxT4iFgHnA88EbgG+GBGXZeY32m2Z+q6u4jvKwmuRl/pnIoo9cChwY2b+G0BErAdOACz2PTIOF6qoq1Au9Hnmc6Ibi7zUX5NS7JcB3xm4fQvwlFE2YLZCM9tKd1vLz7Zynn5/nSvo6Zkbb72H09ZePmub68qe/nwXHbvzgp5/Po/b1vu2ULM932sP2sJpIyiy0/NHlStpMkRmtt2GbYqIE4FjMvO3y+1TgEMz8xXTljsdOL3c3A/41jzi9gS+t4DmzlcbuX3qa1u5feprW7l96mtbuX3qa5u5C/WIzNxrpjsmZWR/C7DPwO3lwG3TF8rMdcC6hQRFxDWZuWohzzEpuX3qa1u5feprW7l96mtbuX3qa5u5TZqU79l/EVgZEftGxIOBk4DLWm6TJEkTYSJG9pm5JSJeDlxB9dW7D2TmdS03S5KkiTARxR4gMz8FfGoEUQvaDTBhuX3qa1u5feprW7l96mtbuX3qa5u5jZmIA/QkSdL8Tco+e0mSNE8We0mSOs5i3wHlGwpt5C6NiO3byG5LRESfcvukT69xH/rqevHnWexrEhHHRMSrWsg9ATg3InYZ5T9wRPwq1dcfdy23G8+OiFUR8ZsRsV9EjOxvNyIeGREHAuQID3KJiEdFxJNGmRsRi0eRM0a5KyLiIBjpa7xfRDw1InYcRd5A7uER8Tyo+jqq9UUb68Y+rReHZbGvQUQcDfwv4Ksjzj2y5P5tZt47wpXV0cD/BPYq+Y2vKCPi2cAHqa6J8D6qM1w1LiKeA3wSeHNEXBIRz4+IXUaQ+2vA3wNvi4jLIuIFEbFHw5nHAX8TEU9vMmfMcj8JvCMiroyIpWV+YyvoUgz+GjgL+NzUh5yGMyMiHgL8H+DlEfES+FnBb7QGtLFu7NN6cS4s9gsUEUdQfSXwRZl5VUTsHhEPG9FmnIOBd2Tm5SXzyIjYv8miEBFrqK5A+DvA44Gdp0a9Ta2wImJv4NXASZl5EtV1Eg4rr/UOTWQO5L4SeEFmnki1wnozcEpE7N5g7k5UJ446OTOPoSpIhwEnN/XeRsQTgA8ANwKvHlXhbTH3cOBPgZdm5jOp/qbeBc2toCPiV0rGb2fmc6jOAnpwk5lTMvM/gEuBq4D9ozq1OJn506YyW1w3HkwP1otzZbFfuBuAzcAR5Y/441Tf0fxkRDyriTd64Dl/CXhY+XT+l8DvAf8DeGNEPLSB3O2A3YFTMvPLwEOAnYAjodEV1g+B+4DHliL7TOA04GLg9AY3Af8Q+C/KVoTM/FOqUzc/BngSNPaPvIXqdT645K4D/gnYF3h6Q7k3AW8A/jvwaeDMERXekeeW1+7BwFsy8/+V2WcBTY/Cfkj14eJfImIZ8DSqkfb6sq5oZH088H+5ierv6ktUBf+tEfHGMvJvIvsG4F5GtG7s4XpxbjLTaZ4TsKj83Ae4g6ownF7mvYbqU+3iBvP3o9rU+1HgJWXeYcD7gac03Pftys9fAf4NeFLDeacAVwKfB95Y5p1INVrZt8HctVQjwFOoNtH9BVVx+ssGsmLgb+oY4D3A4QP3vxy4vIHMqfdyKnsJ1QWlLgdWl3nLpparM3ta7h6jyB3I32vgNVgGfAVYUubt3uDf1HbAmQN/x68G/g7YpanMkvMU4L+X398C/CfwJw1lPaj83Ae4c5Trxj6tF+cyObKfh4g4KiLOA94SEU/PzO8ATwTOzGoURma+oyz+8LpzyyfyIzLzW1Sj28dRjfrIzM9TjVoeXXPu+SX3aWX2TyPiQZn5Wap/qieUZRfVmDnV10Mz84PA86iK/TcAMvNjwA5U/a/FtNwnUo1EvgmspioEL8nMPwa21LkLISKOoVoZvats/vxn4FbguLLJmcw8D9guIh5bc+Y7y9/x/SXnLqpR2CeA34uIC4B3U73WdeQeFxHvB/4oIlYP5P5gBLkXRsRbyv/Pd8td21EVvv/KzLuiuqrmH9fx/k71tWQ+HarTfwPvysyzy+0/A+4HHrHQvK3lUv0dPywiTqb68HoesEtEvLjuXKpjXI4q68aDaXDdOK2vzyjrxYuo1guPLJlNrBcH/55Wl9n3N7leXJC2P21M2gQcB3yNapR1JtU+sMfOsNwLgC9TRg8N5e5DNRr6XapR72uAF1KNUGoZ7U7LfV3JfdS0ZU6lWpHs0GBfV5b7jgfOAY6mOljva8DDG8h9fcldPsNyL6EqxjvXlPus8p69sLzGF5f5K6k2P15IdezAycC/1vE3NVvmDMt9FPh34OCa+noo1aWnX0S1efV7wG+MQy7VVpuzgC8Aj28o8wUzLPcbVJvWH9pgX19Y7nsvcDPwq+X284G9G8w9eYblals3zpL5XKrN6L9X/oebWC8O8/dU63pxwW1uuwGTNAEPBT7GA5sYd6E6MvxJA8ssKn8A1wMHNJi7biqXauSzH3Au8Cd1rKiG6S9lM2z5fR2wYgSZ+wNvotrseWXDff3Zazyw3G8B3wYOrCl3KXAJ8PRy+2jgb6i2Yjy+zFtNte/xL4AnNpT58ZL51IHljqE6cK2WvpbnfDZwycDtZ1JtqXl+m7nA9lRbUm4GHjOizF2oCsJ1da0rtpF7NNXWg8FdQ4tG+Bo/mPrXjdMzjy6ZJ5Tb+1NtHaptvTjs33GZX8t6sZY2t92ASZqoDmg8hoERHfDnwNppyz0L2G8EuWdNW24RAwV4VP0t82vJnUNfd6bsXx3xe7uMspWhptwAHlF+X0K1m+Iiqq0LdwHHDLy32zececa0zN2p+XgIqs2aFzKwxaSsKL8LPK2l3KeX26+o+f92a5mHl/f016npw8U2co8puU8Z+BuobV0xh/f22Dr7O0vm0SXzV6b6Wmc/h+jrLw/Mqz17vpP77IcUEZGZP83MKzLzvoEjP79NdfQ0EfGciHhsZn4qq/1GTef+ZCD3wMy8P8tfWMO5g/09AOo54nQOfT0oM+/Lat/ygs2hr4/PzFsz84YaczMzv11m/RLVh5rTsto//yaqlTTlvf1Jw5nnA28Ejin7He/OzJsWmjnNdVQjvD+MiO1Kez5DdfDjYQAt5K4qy7y3rv/bITKfmtWxCn+Tmf9aY+ZsuVcAbwV+Gar/17rWFdvI/bn3Frii5v7OlPn3JfPJ0NjR8Fvr61OmFmooe14s9kOIiEUzvGlTr913gDsi4niqI1y3tJD7R1QHGLWR++MWMn9UR+Y8cv+jydzMvDEzrxqYtXNdeXPIXFzm1/796/IBYgvw21THI7ybcmAp1SbtR9SdOWTuCoA6PkzNI7PW13kbuYtp/zWurfiN8d9TI7kL1uZmhXGfgDU8cGDY1NeEHjRtmd+i+p79F6lvP1RvcvvU12Fzy7wXAtcAj5vEzPJ821N2PfDA1+sePPWTaiV5EdUZ5b4OHDSpuX3qq6/xaHLrnlpvwLhOVCdE+BFwBbBrmTe1olwDvL38fgLVEZe17MftU26f+jpk7v8uvx9bllnwAWptZJbnOx74MNV35g8q86a+e/1Mqt0FQXWCoqOo6SCmNnL71Fdf49HkNjG13oBxnIBfpfoqzO9Tnd7ycQP3HUD11ZypI0y3Y4avZ5k7fpkTlLsI2HMSM8tzHQlcW1Z+bwQ+OHDf46m2lPzCV9AmMbdPffU1Hk1uU1PrDRi3ieqrGhsoX0+hunjExwbuPwI4rPxey9HRfcvtU1/nkVvL2eLayBx47jdRnUQFqtMKf5Tqu/xPKNPUkdl1Hw0+8tw+9dXXeDS5TU0eoPeLbqOcv7rcfgWwOCKeCZCZ/5TV2ZjIGg/o6Vlun/o619y6DvBsI3PKD4DHRcTvAh+h+v76Q4D1VPs8/3nqWwEdyO1TX9vK7VNfG7Nd2w0YFxHxKOCnwC2ZeXeZtz3VBViuBw4BPlP3m9un3D71ta3clvv6k8y8GfgQ1YmeHgV8JTNfU5b5EfC6iHhR1nQUehu5feprW7l96uuoWOxh6trhbwLuAa6NiOsy86IyuvtJRHwU+KuI+Fxmfs7cycjsW+449DUivgp8LjP/LCIeAbxs4IPFXcDd1HR1uTZy+9TXtnL71NeRyjHYl9DmBOxKdQaxw4GHUZ3R6mPAq6YtdzbVQRp17VPtTW6f+uprzK9Tndr3lVQH/X0WuAB4O9XX+uo62n/kuX3qq6/xaHJHOTmyr06CcytwW2b+e0RcQXVRgzMi4ruZ+eGy3D8A12d9+zf7lNunvraVO259fSXVucKfR/X9/Z2BF2XmNyc4t099bSu3T30drbY/bYzDBLyNaiW4S7m9C/CbZX5tR2X3ObdPffU1/lnu25vKbCu3T331NR5N7qim1hvQaucfOLnIIuAdVGdBmnqjl1NdVW2puZOV2bfcMe/rw7qQ26e++hqPJnfUUy+/ehdRXegky7uZ1QUp/ozqikWfjojHAM+g+prF/eZORmbfciekr3VeK2LkuX3qa1u5feprm6Y+0fRCRCwH7gXuy7LPMiK2z8yfRMSKct8rgEcCD6c6uOkr5o53Zt9y+9TXtnL71Ne2cvvU17HQ9qaFUU3Ac4CrqM4kthY4duC+I6nOC/6YfGBzzg7mjn9m33L71Fdf4+7m9qmv4zK13oCRdLL6dPZV4EBgP+DVwD8Cv1buvxp4nrmTldm33D711de4u7l96us4TX356t3OwPcy8+sAEbEEeDpwSkTcAByVmfcNnDTB3MnI7Ftun/raVm6f+tpWbp/6OjZ6cYBeZl4P3B0R74uInYFnAzcC/wSsysz7ynK1vsF9yu1TX9vK7VNf28rtU1/byu1TX8dJZ4t9RKyMiKUDs94A7An8BbBfZr4O2Ag8KyIWmTsZmX3L7VNf28rtU1/byu1TX8dWZvv7EuqegBOAm4HzgUdMu28x8KDy+2lUb3pdpw7tTW6f+upr3N3cPvXV13g0ueM6de6rdxGxG/Bh4JvAnVTnOX5nVlcxGlzudOAM4JTM/Jq5453Zt9w+9bWt3D71ta3cPvV13HWu2ANExL7A94HHUn262wk4NzM3DSzzIuCLmfktcycjs2+5feprW7l96mtbuX3q6zjrTLGPiIcDd1BtirlvYP5TqN7oHYHXAauAb2bmD80d/8y+5fapr23l9qmvbeX2qa8To+39CHVMwHHA14F1wKVUB14M3v8U4EzgX6iuQ/xwc8c/s2+5feqrr3F3c/vU10maWm/AAt/cAPahOppyNbAUeC1wG3DAtGXfCdxEPddc7k1un/rqa9zd3D711dd4NLmTNrXegBre6EVUn+SW8cBuiVdSXZt46rSHS6jOnPREcycjs2+5feqrr3F3c/vU10mbWm/AAt7cRwNPBn4J+Cjw+mn3v57qUoUPKbd3NHf8M/uW26e++hp3N7dPfZ3UqfUGzPMNPh74GtV5jc+jurjBJuCsgWVWAO/jgU95Ye54Z/Ytt0999TXubm6f+jrJU+sNmMcbfDjVdyefWG6vA94K/DeqEyi8ierT3mnANcASc8c/s2+5feqrr3F3c/vU10mfWm/APN/k0wZu7wVcXn5/JPAB4ILyBh9k7mRk9i23T331Ne5ubp/6OulT6w2Yx5u8CNh14PflwJeBvcu8RwDbAbuZOzmZfcvtU199jbub26e+Tvo0cRfCycz784ETIQTV9yV/kJm3l7Mh/SGwfWbeY+7kZPYtt099bSu3T31tK7dPfZ10nTiDXkRcBNwOHE21aWejuZOf2bfcPvW1rdw+9bWt3D71dZJMdLGPiAC2B64vP4/MzBvMnezMvuX2qa9t5fapr23l9qmvk2iii/2UiDiN6mIG15nbncy+5fapr23l9qmvbeX2qa+TpCvFPrKFjvQpt099bSu3T31tK7dPfW0rt099nSSdKPaSJGl2E3c0viRJmhuLvSRJHWexlySp4yz2krYpIu6PiK9ExHUR8dWIeE1EbHX9ERErIuKFo2qjpNlZ7CUN40eZeXBmHgA8E3gW8OZtPGYFYLGXxoBH40vapojYnJmLB24/EvgisCfVecg/COxc7n55Zv5LRHweeBxwE3AxcC5wDrAa2AE4PzPfN7JOSD1msZe0TdOLfZl3F/BY4F7gp5n5nxGxEvhIZq6KiNXA6zLz+LL86cBDM/OtEbED8M/AiZl50yj7IvXRdm03QNLEivJze+C8iDgYuB94zCzLHw08PiKeX27vBqykGvlLapDFXtKclc349wN3Uu27vwN4AtVxQP8528OAV2TmFSNppKSf8QA9SXMSEXsB7wXOK6cn3Q24PTN/CpxCdX1xqDbv7zLw0CuAl0XE9uV5HhMROyOpcY7sJQ1jp4j4CtUm+y1UB+S9o9x3AfBXEXEicBVwX5n/NWBLRHwVuAh4F9UR+l8qVyr7LvDc0TRf6jcP0JMkqePcjC9JUsdZ7CVJ6jiLvSRJHWexlySp4yz2kiR1nMVekqSOs9hLktRxFntJkjru/wMWfXAF/CgIgAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# plot distribution of all events & dates\n", "\n", "from pandas.plotting import register_matplotlib_converters\n", "register_matplotlib_converters()\n", "\n", "plt.figure(figsize=(8, 5))\n", "\n", "plt.hist(logs['datetime'], bins=200)\n", "\n", "plt.xlabel('Date')\n", "plt.ylabel('Count')\n", "plt.grid(True)\n", "\n", "plt.xticks(logs['date'].unique(), rotation=45)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is a very low number of event prior to August 1st. Maybe this was a stage where the app hadn't been fully launched yet or in was still in beta testing. We should ignore data prior to this time period by creating a new filtered DataFrame" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventuiddatetimegrouptimedate
2828Tutorial37374620466226217202019-08-01 00:07:2824600:07:282019-08-01
2829MainScreenAppear37374620466226217202019-08-01 00:08:0024600:08:002019-08-01
2830MainScreenAppear37374620466226217202019-08-01 00:08:5524600:08:552019-08-01
2831OffersScreenAppear37374620466226217202019-08-01 00:08:5824600:08:582019-08-01
2832MainScreenAppear14338408838240888902019-08-01 00:08:5924700:08:592019-08-01
\n", "
" ], "text/plain": [ " event uid datetime group \\\n", "2828 Tutorial 3737462046622621720 2019-08-01 00:07:28 246 \n", "2829 MainScreenAppear 3737462046622621720 2019-08-01 00:08:00 246 \n", "2830 MainScreenAppear 3737462046622621720 2019-08-01 00:08:55 246 \n", "2831 OffersScreenAppear 3737462046622621720 2019-08-01 00:08:58 246 \n", "2832 MainScreenAppear 1433840883824088890 2019-08-01 00:08:59 247 \n", "\n", " time date \n", "2828 00:07:28 2019-08-01 \n", "2829 00:08:00 2019-08-01 \n", "2830 00:08:55 2019-08-01 \n", "2831 00:08:58 2019-08-01 \n", "2832 00:08:59 2019-08-01 " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# new filtered DataFrame\n", "\n", "filtered_logs = logs[(logs['date'] >= pd.to_datetime('2019-08-01', infer_datetime_format=True))]\n", "filtered_logs.head()" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAEtCAYAAADa/h2tAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeaElEQVR4nO3de7xcZX3v8c+XcJU7TcCYgBs1IBcVISIFLSByEbSAigaVy3nZpnJARAUN4jnaV805tBVQsKCpIGAVxAoFGy5FDtXScttguITLi1QoRCJERUlpRRN+54/1bBh3Zu892Zln1npmvu/Xa1575pk1s7+zZs381vOsNWspIjAzM8thnboDmJlZ/3KRMTOzbFxkzMwsGxcZMzPLxkXGzMyycZExM7Ns1q07QC5Tp06NoaGhumOYmRVj6tSp3HDDDTdExCHdes6+LTJDQ0MMDw/XHcPMrCiSpnbz+TxcZmZm2WQrMpK2lXSzpAclLZb0sdT+eUk/lbQoXQ5teczpkpZIeljSwS3te0i6L913riTlym1mZt2Tc7hsJfDJiLhb0qbAXZJuTPedExFfbJ1Y0s7AHGAX4BXADyTtEBGrgAuAucBtwLXAIcB1GbObmVkXZOvJRMSyiLg7XV8BPAjMGOchhwOXR8TzEfEosATYU9J0YLOIuDWqA61dChyRK7eZmXVPT7bJSBoC3gjcnppOknSvpIskbZnaZgBPtDxsaWqbka6Pbjczs4bLXmQkbQJ8DzglIp6lGvp6NbAbsAw4a2TSNg+Pcdrb/a+5koYlDS9fvnxto5uZ2VrKWmQkrUdVYL4VEVcCRMRTEbEqIl4A/hbYM02+FNi25eEzgSdT+8w27auJiAURMTsiZk+bNq27L8bMzNZYzr3LBFwIPBgRZ7e0T2+Z7Ejg/nT9GmCOpA0kbQ/MAu6IiGXACkl7pec8Frg6V24zM+uenHuX7QMcA9wnaVFq+wxwtKTdqIa8HgP+DCAiFku6AniAas+0E9OeZQAnABcDG1HtVeY9y8xqNjRvIQCPnXlYzUmsybIVmYi4hfbbU64d5zHzgflt2oeBXbuXzszMesG/+Dczs2xcZMzMLBsXGTMzy8ZFxszWysgOAE03NG9hMVn7iYuMmZll4yJTo9a1Kq9hWcncS7CxuMiYmVk2LjJmZpaNi4yZmWXjImNmZtm4yJiZWTYuMmZmlo2LjJmZZeMiY2YDx7/r6R0XGTMzy8ZFxszMsnGRMTOzbFxkzMwsGxcZ61veuGtWPxcZM7MGK31lyUXGzMyycZExM7NsXGTMzCwbFxkr3siYdcnj1v3C74GN5iJjZmbZuMiYmRWg1F6ii4yZmWXjImNmZtm4yNga8QZ2M1sTLjJmZpaNi0wPjF7zd2/AzAaFi4yZDRSv4PWWi4xZjdyrtX6XrchI2lbSzZIelLRY0sdS+1aSbpT0SPq7ZctjTpe0RNLDkg5uad9D0n3pvnMlKVfuXPxlYmaDKGdPZiXwyYjYCdgLOFHSzsA84KaImAXclG6T7psD7AIcApwvaUp6rguAucCsdDkkY26zbLyiYYMmW5GJiGURcXe6vgJ4EJgBHA5ckia7BDgiXT8cuDwino+IR4ElwJ6SpgObRcStERHApS2PMbMauFhap3qyTUbSEPBG4HZgm4hYBlUhArZOk80Anmh52NLUNiNdH91uZpkNSjEZlNdZh+xFRtImwPeAUyLi2fEmbdMW47S3+19zJQ1LGl6+fPmah7Vx+YPYHd4+Z4Nk3ZxPLmk9qgLzrYi4MjU/JWl6RCxLQ2FPp/alwLYtD58JPJnaZ7ZpX01ELAAWAMyePbttIbI15y9EM5usnHuXCbgQeDAizm656xrguHT9OODqlvY5kjaQtD3VBv470pDaCkl7pec8tuUxZhNykbSxuFeZX87hsn2AY4C3SVqULocCZwIHSnoEODDdJiIWA1cADwDXAydGxKr0XCcAX6faGeDfgesy5rYC9PsXg7/8rJ0Sl4tsw2URcQvtt6cAHDDGY+YD89u0DwO7di+dmZn1gn/xb5NS2tqU9U6Ja9uWj4tMA/kDatZ7TSyOTcszGVn3LrM10w8LlHXO77cNAvdkzMwsGxcZs4ZxD8f6iYuMDRR/gZv1louMmZll4yLTY16TNrNB4iJjZh3xCpJNhouMmZll4yJjZmbZuMjUrCm/Mm5CBrNcvHzXx0XGzMyy8WFlrK2RNb/Hzjys5iTWJO4R2JpyT8bMzLJxkWmopmyrKU2/zbd+ei02mFxkzMwsGxcZMzPLxhv+Mxmat7C4jeYemmkm74RhJXORGXAuLM3Q5JUSLyO2NjxcZmZm2bgnY8XyGrZZ87knY33JBWiw9Nuu6/3EPRkrymS/SOr+Aurk/9edcUSTtw9ZedyTMbO+4l5Ns7gnY2bFGJTdufupSLonY2vNa45mkzMInxv3ZDLqhwWoH16DWUn67TPnnoyZraYfeqel5+8X7snYpPlDbGYTcU+mi/ylm5fnr1l53JPpMn8RmtXHn7/mcU/GzKxgTS+s2YqMpIskPS3p/pa2z0v6qaRF6XJoy32nS1oi6WFJB7e07yHpvnTfuZKUK7MNhn7YqG0v8XvZbDl7MhcDh7RpPycidkuXawEk7QzMAXZJjzlf0pQ0/QXAXGBWurR7TjMza6Bs22Qi4keShjqc/HDg8oh4HnhU0hJgT0mPAZtFxK0Aki4FjgCu635iM7P69GuPrI4N/ydJOhYYBj4ZEc8AM4DbWqZZmtp+l66Pbjczm1C/fnGXpNcb/i8AXg3sBiwDzkrt7bazxDjtbUmaK2lY0vDy5cvXMqqZma2tnhaZiHgqIlZFxAvA3wJ7pruWAtu2TDoTeDK1z2zTPtbzL4iI2RExe9q0ad0NXxOviZlZyXo6XCZpekQsSzePBEb2PLsG+Laks4FXUG3gvyMiVklaIWkv4HbgWOC8XmY2W1teUbDRBmmZyLkL82XArcCOkpZK+jDwV2l35HuB/YGPA0TEYuAK4AHgeuDEiFiVnuoE4OvAEuDf8UZ/MxtwJRWpnHuXHd2m+cJxpp8PzG/TPgzs2sVoZkXyGSutRB31ZCTt00mbmZlZq057MucBu3fQZmZma6Ckoa/JGLfISPpDYG9gmqRPtNy1GTCl/aOsFP2+cJtZ/SbqyawPbJKm27Sl/VngvblCWV45i8ugnIPdzDozbpGJiB8CP5R0cUT8R48yWaFK6xl5Q3peuedvruXNy0V3dbpNZgNJC4Ch1sdExNtyhDIzs/7QaZH5LvBVqt+rrJpgWjOjvJ5dDh4+tU6LzMqIuCBrEjMz6zud/uL/+5L+p6TpkrYauWRNZmZmxeu0J3Nc+ntaS1sAr+puHDMz6ycdFZmI2D53EDMz6z8dFZl0krHVRMSl3Y1jZnUpaUeFkrIOuk6Hy97Ucn1D4ADgbsBFxszMxtTpcNlHW29L2hz4ZpZEZmbWNyZ7Ppn/ojqxmJmZ2Zg63Sbzfaq9yaA6MOZOVCcZM7NRvL3AxjNoy0en22S+2HJ9JfAfEbE0Qx6znvIv0q1UpRSrjobL0oEyH6I6EvOWwG9zhjIzs/7Q6XDZ+4C/Bv4ZEHCepNMi4u8zZitGKWsUZma91ulw2RnAmyLiaQBJ04AfAANfZFxgzMzG1mmRWWekwCS/YPJ7pplZQ5S0klRSVntJp0Xmekk3AJel2+8Hrs0TyczM+sW4RUbSa4BtIuI0Se8G3kK1TeZW4Fs9yGfmNVizgk3Uk/kS8BmAiLgSuBJA0ux037syZjMz6xuDurI00XaVoYi4d3RjRAxTnYrZzMxsTBMVmQ3HuW+jbgYxs4kNzVs4sGvEVqaJisydkv50dKOkDwN35YlkZtZfBnnFYKJtMqcAV0n6IC8VldnA+sCRGXNZH/AhW6xUXna7Z9wiExFPAXtL2h/YNTUvjIj/lz2ZmZkVr9PzydwM3Jw5i5lZXxnkYbIR/tW+mZll4yJjZmbZuMhYo3m4waxs2YqMpIskPS3p/pa2rSTdKOmR9HfLlvtOl7RE0sOSDm5p30PSfem+cyUpV2YzM+uunD2Zi4FDRrXNA26KiFnATek2knYG5gC7pMecL2lKeswFwFxgVrqMfk4zM2uobEUmIn4E/HJU8+HAJen6JcARLe2XR8TzEfEosATYU9J0YLOIuDUiAri05TFmjePhPbPf1+ttMttExDKA9Hfr1D4DeKJluqWpbUa6PrrdzNaSC6L1QlM2/LfbzhLjtLd/EmmupGFJw8uXL+9aODMzm5xeF5mn0hAY6e/I2TaXAtu2TDcTeDK1z2zT3lZELIiI2RExe9q0aV0NbmaTN0i9pjoOYtrkA6f2ushcAxyXrh8HXN3SPkfSBpK2p9rAf0caUlshaa+0V9mxLY8ZGN1cgJq6IJpZf+r09MtrTNJlwH7AVElLgc8BZwJXpKM4Pw4cBRARiyVdATwArAROjIhV6alOoNpTbSPgunQxM7MCZCsyEXH0GHcdMMb084H5bdqHeengnGZmVpCmbPg3M7M+5CLTp7ztxcyawEXGsnCRy8vz10qRbZuMmVk3uKCWzT0ZMzPLxkXGzMyycZExM7NsvE1mkjxObGY2MfdkzMwsG/dkrJHcUzTrDy4yA8Jf2ma95c9cxcNlZmaWjXsyZl3gtdaJjcyjx848rOYk1kvuyUyCv1D6j99TszxcZMzMLBsXGTMzy8ZFxszMsnGRMTOzbFxk+og3XptZ07jImJlZNi4yZmaWjYuMmZll4yJjNmCG5i309rsOeT6tPRcZMzPLxkVmAHhtzMzq4iJjZmbZuMgUxD0SMyuNi4yZmWXjImNmZtn4pGVrwMNVZr3lz1z53JMxM7Ns3JPpY14L7A3PZxvhZWF17slY4/iDatY/XGTMzCybWoqMpMck3SdpkaTh1LaVpBslPZL+btky/emSlkh6WNLBdWQ2M7M1V+c2mf0j4uctt+cBN0XEmZLmpduflrQzMAfYBXgF8ANJO0TEqt5HbiYPLw2m1vf9sTMPqzGJ2diaNFx2OHBJun4JcERL++UR8XxEPAosAfbsfTwzs2Zr4gpnXUUmgH+SdJekualtm4hYBpD+bp3aZwBPtDx2aWpbjaS5koYlDS9fvjxTdDMz61Rdw2X7RMSTkrYGbpT00DjTqk1btJswIhYACwBmz57ddhozM+udWnoyEfFk+vs0cBXV8NdTkqYDpL9Pp8mXAtu2PHwm8GTv0pqZ2WT1vMhI2ljSpiPXgYOA+4FrgOPSZMcBV6fr1wBzJG0gaXtgFnBHb1Obmdlk1DFctg1wlaSR///tiLhe0p3AFZI+DDwOHAUQEYslXQE8AKwETvSeZWZmZeh5kYmInwBvaNP+C+CAMR4zH5ifOZoNuJE9c7w7cP2auJeUTU6TdmE2a4ROv+CG5i0s7suwCXlLnG82eS4yZmaWjYuMmZll4yJjZmbZuMiYmVk2LjJmZpaNi4yZmWXjImO18u6s1nReRtdOneeTMbMe8Zek1cVFpgNN+oBO9Kv0JmVdE6XmNrPxebjMsnMBMRtcLjJmZpaNi4yZmWXjbTJmZmvJQ8Jjc0/GrA94N1trKheZCfiDa2Y2eS4yZmZrwSui43ORMTOzbFxkCuUxeCudl9/B4CJjNgn+gjTrjHdhLpy/7HqrtPldWl7rP+7JtOEPpllveNi3/7nIjMELfx6ep2aDxcNl1hMuLjaRpi8jQ/MWjnn0cxubi4xZh5r+JQi/n9FfiNYEHi4zM7Ns3JMxa2Oik8OVoISe14iSso4oMXMd3JOx2vhDatb/3JMxm4CLoY3ohx5ur7knY2Zm2bjImI3DvRhrx8tF5zxcZj3nD6jZ4HBPxszMsimmyEg6RNLDkpZImld3HjMzm1gRRUbSFOBvgHcAOwNHS9q53lRmZjaRIooMsCewJCJ+EhG/BS4HDq85k5mZTaCUIjMDeKLl9tLUZmZmDVbK3mVq0xarTSTNBeamm/8p6eFJ/r+pwM8n+dg6OG9epeWF8jI7b5foL9s2d5q366+plCKzFNi25fZM4MnRE0XEAmDB2v4zScMRMXttn6dXnDev0vJCeZmdN68685YyXHYnMEvS9pLWB+YA19ScyczMJlBETyYiVko6CbgBmAJcFBGLa45lZmYTKKLIAETEtcC1Pfp3az3k1mPOm1dpeaG8zM6bV215FbHa9nMzM7OuKGWbjJmZFchFxgCQ1G43cesiz+M80s5AxZC0jaT16s7RKy4yXSRpk7ozrAlJQ5JeBxAFjJtK2lHSH0rasO4snZK0t6T3QDWPm15oJB0s6ZS6c3RK0uHAuZI2bfq8BZD0Dqo9YzdLt0vIPFvS0enzt8Y1w0WmSyQdBvyDpH3rztKJlPcfgbMl3SRpm9TeyIU+fTivAk4Hbhkp6A3OK0kvA74NnCTpf8CLhaaRnztJBwH/B7in7iydkHQAVd7vR8SKpq8opfn7F8A0qtyNX7mT9C7gm1SH8foa1Y8610gjF/bSSHoDcBGwBPh40wuNpL2Bs4APR8SBVIfs+TI0c6GX9EdU+f4kIv6Y6oe4u0Ez846IiP8CrgBuBnZOR6QgIl6oNVgbkt5KtffmhyLiZklbSHp5w4d1dgPOjoiFKesBknaWtFXdwUaTtD/VQX7/FHg9sLGkXdN9TV1Rmg58HJgTEXOovif2SsvGBp0+j4tMdzwKfBr4X8B1wGlNLTRpgV4f+POIuD01n06bw/Q0yLNUBfHfJM0A9qHqHVwu6dAm9gxait9jwBbA3VSF5guSzkg9nSblfgRYAbw1FZYrqXZ7/cc0jxvzRdiS5Q+Al6f5+PfAR4D/DZwhaeu68o0maV2qZeCYiPgx8DJgI+AAaPSK0rPAc8BrJW0BHAgcD1wCzO1080CTFvIiSVJEPAt8MyKWA9+lGnP9lKT90jQz0oJWu6j8M/ADePEDuw6wk6QtU9sWtQVsIyIWRcS/pHn4Aaq11znA7cDJwMa1BhzfXcAzEXEZ8GvgVGDL9D40okcjaZ2I+BnVGvYXqL5YLk+9xhuAk2jQPG75Ur4E2Be4DLgwIo4CvgRsDmxfT7rVRcTKiLgqIm6TtG5EPE3VM/+YpN3rzjeWiHiO6vtsLnA9cF5EvBv4O+CtVMN+E3KRmSRJh0n6OvB5SftFxCqAiPgl1Vrg1cBHJJ0PnAd03L3MIeW9UNKfS3prKohQ/SD3N8BvI+IZSccAf7km3eEcRuZvyrsvVB9W4MsRMT/dPgdYBbyyxqgvapcZeIhqbfuDwDHAV4BNJR1bW9CkZRn+nKS3R8QTVENQp6XjABIRZ6fJt6sp5otGzd+3RcTDwMXATsCrACLiNqqe+mvqS1oZ9ZnbLzWvSkX9R8B3gDekaafUFPP3jJrH+0TEpcB7gNuABwEi4rtU32c7dfSkEeHLGl6ozm/zMPAhqi76z4H3tZnuO8DPgN2anhf4BtWw2R3A6xuY9/1tpnsf1TDU1g1dJj6Q7vsq8DjwjnT7vcD0Bub9YJvp3g/8GJjWwLxHUA07fYRqu9cnqHq6i4DtG5h39GfuOKqVkA3qzDpG5hNS5jnpvj8G/go4iGongHuB7Tp63rpfWIkX4F3ApS23DwQeAN7b0nYw1YayXZueF1gP+Gn6ItyhgLybpg/oYmCXuvNOkPkgqp7W3i33TWlw3pF5vH76snmwCfO4Td6DUt7D0+2dqUYMvkjNK0mdzN+W9gXAUN15J8h8ZPrMfY5q2OymNZnHtb+wEi9UXdwLgZmj3pDlwD7p9hbUvDbVYd590+2PAjvWnbWDvHtTHST13U0oiBNkPjhlfnO6LdKhnOq+dLgMH9KUeTxG3oNS3j8amb915+xw/r6lpa3pmUfm8e7p9kZU2xQ7fl4fu2wS0gbob1DtjXMysCoiQtLJwHoRcVatAUfpJK+k9SLid7UGTTrMu040ZMM5jJv5Y8C6hS4TioZ8QfTjZ67WgG2Mk/kUqt73pDJ7w/8aSl9uK4E/AWZRddFH9mTZlIZshB7RQd4hgAYVmE7zNqnAjJd5E8pdJppSYPrtM9eovDBh5o1Zi8zuyUxg5MdoEfE7SVMiYpWk9SPit6qOmXQW1YKzOdWbc3RE3Oe8/ZkXysvsvM47Wk8z1z0O2OQL8E7gW8BC4HWpbZ14aXz1DKpx9h2At1PzBjzndWbndd6mZa71xTb5QvVr3LvSTD6D6seWI/e9nuqU0KvtVuu8/Zm3xMzO67xNyOzhsjFI+izwfET8dfpV7qep3oAb0ySbRMS/NmXjqPPmV1pm582rtLxQT2Zv+B/bL6kOtfJnVIet+CnVMYcup9rTolELD87bC6Vldt68SssLNWRuxPG0mkLSq4HfRcTjVMfn2QB4NbAoIj6Rpvlv4FRJH4qa93By3vxKy+y8eZWWN+WpNbOLTCLpSOCzwK8l3QPcEhHnSHolcEJLdX8G+BU1H7XYefMrLbPz5lVaXmhGZm+TASRtBvwT1bGPfkL1q/IPAD+iOgfEzcD9VEen3R84PiLuryet8/ZCaZmdN6/S8kJzMrsnU1lJNTb5ZET8TNINVAeHO5nq2D3voXpzNqY6qdNDtSWtOG9+pWV23rxKywsNyeyeTCLp/wJvpjrg3gpJm1LtT757RJxWb7rVOW9+pWV23rxKywvNyDzwe5dJL55l77NUhwg/T9KmEbEC+Bdgd0kvryvfaM6bX2mZnTev0vJCszIPbJEZeRNGdtWL6qRj51AdcfQ6STsAb6PavW9lXTlHOG9+pWV23rxKywvNzDxww2WSZlIdZfS5qA4Ih9IRiCUNpfs+SnWmve2AUyJiUU1xnbcHSsvsvHmVlhcanjkacKiDXl2ozu52M/BtYB5wSMt9B1Cdz3yHdHsKNZ+xznmd2Xmdt/TMtc6cHr8R2wH3ALsCOwIfB34IHJnuvxV4T905ndeZndd5+ynzIO3CvDHw80j7gUvaEtgXOEbSI8DbI+K5lh8n1c158ysts/PmVVpeKCDzwGz4j4gHgV9J+pqkjanOZ72Eak+L2RHxXJquEQuP8+ZXWmbnzau0vFBG5r4uMpJmSdqmpenTwFSqU4zuGBGnAvcBh0qaUkfGVs6bX2mZnTev0vJCgZnrHKvLeQEOBx6nOnzCK0fdtwkvnaTneKo3Z13n7d+8JWZ2Xufth8x9uQuzpM2pzvz2EPA08HLgS1EdhbR1urnAicAxEXFvz4O+lMN5Mysts/PmVVrelKW4zNDHv5ORtD3wC+C1VNV/I+DciHisZZoPAXdGxMO1hGzhvPmVltl58yotLxSauZ+KjKTtgKeouojPtbS/meoN2RA4FZgNPBQRz9YS9KVczptZaZmdN6/S8kKZmVv1zYZ/SYcB1wLnAd+QtOPIfRFxO3A1sAy4herw11vUEPNFzptfaZmdN6/S8kKZmVdT90ahtb0AAral2ptiP2Ab4JPAk8Auo6b9EvAosKvz9mfeEjM7r/P2Q+YxX0vdAbr0hkwBFgAzeGkI8GSqcymMHE5hS6pfxr7Refs7b4mZndd5+yFz29dRd4C1fBNeA7wJ+APgO8CnRt3/KeBi4GXp9obO2795S8zsvM7bD5nHfT11B1iLN+KdwL1Ux+n5CtVB4h4DTm+ZZgj4WstagJy3P/OWmNl5nbcfMk90KfLYZZL2Br4IHB0RP5a0ANiT6hzWt6VfuV4OvAXYg2pj2DOR3hHn7a+8UF5m53Xe0UrM3JG6q9wkq/3ewPEtt6cBC9P1VwEXAecDw8DrnLe/85aY2Xmdtx8yd/S66g4wyTdjCrBZy/WZwI+B6antlcC6wOZ1Z3VeZ3Ze5+3XzJ1civydTESsipd+cCTgV8AvI2JZ+rXrZ4D1IuLXdWVs5bz5lZbZefMqLS+UmbkTffOLf0kXU/0o6SCqLud99SYan/PmV1pm582rtLxQZubRii8ykgSsBzyY/h4QEY/Um2pszptfaZmdN6/S8kKZmcdSfJEZIel4qoPCLa47SyecN7/SMjtvXqXlhTIzj9ZPRaZJp0SdkPPmV1pm582rtLxQZubR+qbImJlZ8xS5d5mZmZXBRcbMzLJxkTEzs2xcZMzMLBsXGbMukLRK0iJJiyXdI+kTksb9fEkakvSBXmU0q4OLjFl3/HdE7BYRuwAHAocCn5vgMUOAi4z1Ne/CbNYFkv4zIjZpuf0q4E5gKtWBDb8JbJzuPiki/k3SbcBOVKfOvQQ4FziT6nS7GwB/ExFf69mLMMvARcasC0YXmdT2DPBaYAXwQkT8RtIs4LKImC1pP+DUiHhnmn4usHVEfEHSBsC/AkdFxKO9fC1m3VTkScvMCqH0dz3gK5J2A1YBO4wx/UHA6yW9N93eHJhF1dMxK5KLjFkGabhsFfA01baZp4A3UG0H/c1YDwM+GhE39CSkWQ94w79Zl0maBnwV+Eo67tTmwLKIeAE4huqEVFANo23a8tAbgBMkrZeeZwdJG2NWMPdkzLpjI0mLqIbGVlJt6D873Xc+8D1JRwE3A8+l9nuBlZLuAS4Gvky1x9nd6VDvy4EjehPfLA9v+Dczs2w8XGZmZtm4yJiZWTYuMmZmlo2LjJmZZeMiY2Zm2bjImJlZNi4yZmaWjYuMmZll8/8Bu97TooHiRT8AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# plot distribution of filtered events & dates\n", "\n", "plt.hist(filtered_logs['datetime'], bins=200)\n", "\n", "plt.xlabel('Date')\n", "plt.ylabel('Count')\n", "\n", "plt.xticks(rotation=45)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Number of events & users with filtered data" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There were 241298 total events after filtering\n", "The total number of events decreased by 1.2 % after filtering\n" ] } ], "source": [ "# count number of events after filter\n", "\n", "filtered_total_events = filtered_logs['event'].count()\n", "print(f'There were {filtered_total_events} total events after filtering')\n", "\n", "percent_lost_events = (((total_events - filtered_total_events) / total_events) * 100).round(1)\n", "\n", "print(f'The total number of events decreased by {percent_lost_events} % after filtering')" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There were 7534 unique users after filtering\n", "The total number of users decreased by 0 % after filtering\n" ] } ], "source": [ "# count number of users after filtering\n", "\n", "filtered_total_users = filtered_logs['uid'].nunique()\n", "print(f'There were {filtered_total_users} unique users after filtering')\n", "\n", "percent_lost_users = round((((total_users - filtered_total_users) / total_users) * 100))\n", "\n", "print(f'The total number of users decreased by {percent_lost_users} % after filtering')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Checking groups after filtering" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['246' '247' '248']\n" ] } ], "source": [ "# checking groups in filtered data\n", "\n", "print(filtered_logs.group.unique())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We still have users present in all 3 groups after filtering our data to dates prior to August 1st, 2019. We can proceed with the analysis now using filtered data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 4: Study the event funnel" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Frequency of events" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "event\n", "MainScreenAppear 117431\n", "OffersScreenAppear 46350\n", "CartScreenAppear 42365\n", "PaymentScreenSuccessful 34113\n", "Tutorial 1039\n", "Name: uid, dtype: int64" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# sort events by frequency\n", "\n", "event_counts = filtered_logs.groupby('event')['uid'].count().sort_values(ascending=False)\n", "event_counts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Events by number of users" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "event\n", "MainScreenAppear 7419\n", "OffersScreenAppear 4593\n", "CartScreenAppear 3734\n", "PaymentScreenSuccessful 3539\n", "Tutorial 840\n", "Name: uid, dtype: int64" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# calculate number of unique users who triggered each event\n", "\n", "event_user_counts = filtered_logs.groupby('event')['uid'].nunique().sort_values(ascending=False)\n", "event_user_counts" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "event\n", "MainScreenAppear 98.473586\n", "OffersScreenAppear 60.963632\n", "CartScreenAppear 49.561986\n", "PaymentScreenSuccessful 46.973719\n", "Tutorial 11.149456\n", "Name: uid, dtype: float64" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# calculate the percentage of total users who triggered each event\n", "\n", "percent_users_events = (event_user_counts / filtered_total_users) * 100\n", "percent_users_events" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Sequence of events\n", "It looks like the main sequence is:\n", "1. MainScreenAppear\n", "2. OffersScreenAppear\n", "3. CartScreenAppear\n", "4. PaymentScreenSuccessful\n", " \n", "The tutorial is most likely optional and could appear just before or after the main screen. We can leave it out of the funnel for now, since it isn't part of the critical sequence of events." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plotting the funnel" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "type": "funnel", "x": [ 7419, 4593, 3734, 3539 ], "y": [ "MainScreenAppear", "OffersScreenAppear", "CartScreenAppear", "PaymentScreenSuccessful" ] } ], "layout": { "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# plot funnel using plotly library\n", "\n", "fig = go.Figure(go.Funnel(\n", " y = event_user_counts.reset_index()['event'][:4],\n", " x = event_user_counts.reset_index()['uid'][:4]\n", " ))\n", "\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Conclusion\n", "Plotly makes it easy to plot funnel diagrams, and even does the calculations for you. Here, we can see that initially we had 7419 users who triggered the MainScreenAppear event.\n", "\n", "Of those initial users, roughly 62% made it to the next step which was the offer screen. This represents a 38% loss in users between these two stage, the biggest in the funnel. Maybe we can try to boost conversions at this stage to help the overall conversion rate.\n", "\n", "Of the initial 7419 users, about 48% of them get to the successful payment page. Nearly half of the users become customers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 5: Study the test results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Number of users in each test group" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "group\n", "246 2484\n", "247 2513\n", "248 2537\n", "Name: uid, dtype: int64" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# unique users per test group\n", "\n", "users_per_group = filtered_logs.groupby('group')['uid'].nunique()\n", "users_per_group" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Statistical significance of differences in conversion rates between control groups" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can write a function that uses a z-test to compare conversion rates at every stage and determine whether or not the differences between samples is statistically significant. The arguments for this function are the two sample groups, a list of events that we want to compare conversion rates for, and the critical significance level for the z-test. The function prints the null and alternative hypotheses and returns a DataFrame summarizing the results for each test." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "# function computes statistical significance between conversion rates at any given stage in the funnel with z-test\n", "\n", "def conversions_z_test(sample_A, sample_B, events, alpha):\n", " \n", " import math\n", " \n", " event_list = []\n", " sample_a_conv_list = []\n", " sample_b_conv_list = []\n", " conv_diff_list = []\n", " reject_null_list = []\n", " \n", " for event in events:\n", " # calculate total unique users for both samples\n", " total_users_A = sample_A['uid'].nunique()\n", " total_users_B = sample_B['uid'].nunique()\n", " \n", " # calculate unique users per each event for both samples\n", " users_per_event_A = sample_A.groupby('event')['uid'].nunique().sort_values(ascending=False)\n", " users_per_event_B = sample_B.groupby('event')['uid'].nunique().sort_values(ascending=False)\n", " \n", " # calculate the percentage of users who triggered each event for both samples\n", " percent_users_events_A = users_per_event_A / total_users_A\n", " percent_users_events_B = users_per_event_B / total_users_B\n", " \n", " # run z-test\n", " p1 = percent_users_events_A[event]\n", " p2 = percent_users_events_B[event] \n", " p = (users_per_event_A[event] + users_per_event_B[event]) / (total_users_A + total_users_B)\n", " \n", " # calculate z for samples\n", " z = ((p1 - p2) - 0) / (math.sqrt((p*(1-p))*((1/total_users_A)+(1/total_users_B))))\n", " \n", " # get z-score from alpha\n", " z_score = st.norm.ppf(1 - alpha)\n", " \n", " # get results\n", " if z >= z_score:\n", " reject_null = False\n", " else:\n", " reject_null = True\n", " \n", " # append data & results to lists\n", " event_list.append(event)\n", " sample_a_conv_list.append((percent_users_events_A[event]*100).round(2))\n", " sample_b_conv_list.append((percent_users_events_B[event]*100).round(2))\n", " conv_diff_list = np.array(sample_a_conv_list) - np.array(sample_b_conv_list)\n", " reject_null_list.append(reject_null)\n", " \n", " # create dictionary of results for DataFrame\n", " result_dict = {'event': event_list,\n", " 'sample_a_conv': sample_a_conv_list,\n", " 'sample_b_conv': sample_b_conv_list,\n", " 'difference': conv_diff_list,\n", " 'reject_null': reject_null_list\n", " }\n", " \n", " # print hypotheses \n", " print(f'Null hypothesis:\\n The difference in conversion rates between samples is statistically significant')\n", " print(f'Alt hypothesis:\\n The difference in conversion rates between samples is not statistically significant\\n')\n", " print('alpha =', alpha)\n", " \n", " # return results DataFrame\n", " return pd.DataFrame(result_dict)\n", " " ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "# split into groups, set events to compare, & set alpha\n", "\n", "group_A1 = filtered_logs.query('group ==\"246\"').drop('group', axis=1)\n", "group_A2 = filtered_logs.query('group ==\"247\"').drop('group', axis=1)\n", "group_B0 = filtered_logs.query('group ==\"248\"').drop('group', axis=1)\n", "group_A0 = filtered_logs.query('(group ==\"246\") | (group ==\"247\")')\n", "\n", "events = percent_users_events.index[:4]\n", "\n", "alpha = 0.05" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Null hypothesis:\n", " The difference in conversion rates between samples is statistically significant\n", "Alt hypothesis:\n", " The difference in conversion rates between samples is not statistically significant\n", "\n", "alpha = 0.05\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventsample_a_convsample_b_convdifferencereject_null
0MainScreenAppear98.6398.530.10True
1OffersScreenAppear62.0860.491.59True
2CartScreenAppear50.9749.261.71True
3PaymentScreenSuccessful48.3146.082.23True
\n", "
" ], "text/plain": [ " event sample_a_conv sample_b_conv difference \\\n", "0 MainScreenAppear 98.63 98.53 0.10 \n", "1 OffersScreenAppear 62.08 60.49 1.59 \n", "2 CartScreenAppear 50.97 49.26 1.71 \n", "3 PaymentScreenSuccessful 48.31 46.08 2.23 \n", "\n", " reject_null \n", "0 True \n", "1 True \n", "2 True \n", "3 True " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# run z-tests comparing conversion rates between control groups for all events\n", "\n", "conversions_z_test(group_A1, group_A2, events, alpha)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Conclusion\n", "After running our z-test comparing conversion rates for the control samples for each event, we can confirm that the differences in conversion rates were not statistically significant. Each difference at each stage of the funnel resulted in us being able to rejec the null hypothesis in favor of the alternative hypothesis. This is good news and tells us that the A/B test is most likely to be set up properly." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Statistical significance of differences in conversion rates between control groups & test group" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Null hypothesis:\n", " The difference in conversion rates between samples is statistically significant\n", "Alt hypothesis:\n", " The difference in conversion rates between samples is not statistically significant\n", "\n", "alpha = 0.05\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventsample_a_convsample_b_convdifferencereject_null
0MainScreenAppear98.6398.270.36True
1OffersScreenAppear62.0860.351.73True
2CartScreenAppear50.9748.482.49False
3PaymentScreenSuccessful48.3146.551.76True
\n", "
" ], "text/plain": [ " event sample_a_conv sample_b_conv difference \\\n", "0 MainScreenAppear 98.63 98.27 0.36 \n", "1 OffersScreenAppear 62.08 60.35 1.73 \n", "2 CartScreenAppear 50.97 48.48 2.49 \n", "3 PaymentScreenSuccessful 48.31 46.55 1.76 \n", "\n", " reject_null \n", "0 True \n", "1 True \n", "2 False \n", "3 True " ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# run z-tests for comparing conversion rates for control group A1 with test group B0 for all events\n", "\n", "conversions_z_test(group_A1, group_B0, events, alpha)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Null hypothesis:\n", " The difference in conversion rates between samples is statistically significant\n", "Alt hypothesis:\n", " The difference in conversion rates between samples is not statistically significant\n", "\n", "alpha = 0.05\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventsample_a_convsample_b_convdifferencereject_null
0MainScreenAppear98.5398.270.26True
1OffersScreenAppear60.4960.350.14True
2CartScreenAppear49.2648.480.78True
3PaymentScreenSuccessful46.0846.55-0.47True
\n", "
" ], "text/plain": [ " event sample_a_conv sample_b_conv difference \\\n", "0 MainScreenAppear 98.53 98.27 0.26 \n", "1 OffersScreenAppear 60.49 60.35 0.14 \n", "2 CartScreenAppear 49.26 48.48 0.78 \n", "3 PaymentScreenSuccessful 46.08 46.55 -0.47 \n", "\n", " reject_null \n", "0 True \n", "1 True \n", "2 True \n", "3 True " ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# run z-tests for comparing conversion rates for control group A2 with test group B0 for all events\n", "\n", "conversions_z_test(group_A2, group_B0, events, alpha)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Conclusion\n", "We have differing results for our test depending on which control group was used. Using group A1 (246) resulted in rejecting the null hypothesis for the conversion rate of the CartScreenAppear event. However, when we used group A2 (247), we failed to reject the null hypothesis. It appears that the difference of 2.49% for this combination of groups was statistically signicant." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Statistical significance of differences in conversion rates between combined control group & test group" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Null hypothesis:\n", " The difference in conversion rates between samples is statistically significant\n", "Alt hypothesis:\n", " The difference in conversion rates between samples is not statistically significant\n", "\n", "alpha = 0.05\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventsample_a_convsample_b_convdifferencereject_null
0MainScreenAppear98.5898.270.31True
1OffersScreenAppear61.2860.350.93True
2CartScreenAppear50.1148.481.63True
3PaymentScreenSuccessful47.1946.550.64True
\n", "
" ], "text/plain": [ " event sample_a_conv sample_b_conv difference \\\n", "0 MainScreenAppear 98.58 98.27 0.31 \n", "1 OffersScreenAppear 61.28 60.35 0.93 \n", "2 CartScreenAppear 50.11 48.48 1.63 \n", "3 PaymentScreenSuccessful 47.19 46.55 0.64 \n", "\n", " reject_null \n", "0 True \n", "1 True \n", "2 True \n", "3 True " ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# run z-tests for comparing conversion rates for control group A0 with test group B0 for all events\n", "\n", "conversions_z_test(group_A0, group_B0, events, alpha)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Conclusion\n", "When comparing combined control group A0 (246 & 247) conversion rates with the test group B0 (248) we are again able to reject the null hypothesis in favor of the alternative hypothesis. The difference in conversion rates between the combined control group and the test group is not significant at any stage of the funnel." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Explanation of critical significance level\n", "\n", "For theses tests, we used a critical significance level of 0.05. In total, we carried out 16 tests with 4 different combinations of samples (A1 vs A2, A1 vs B0, A2 vs B0, & A0 vs B0). As the number of tests increases, the probability of one of the results being false also increases. We can counteract this by adjusting our critical significance level.\n", "\n", "The probability of making at least one mistake in the course of k comparisons will be:\n", "\n", ">\\begin{align}\n", "\\ 1 - (1-\\alpha)^k \\\\ \\\\\n", "\\end{align}\n", "\n", "We carrier out 16 tests and our alpha was 0.05, so the probability of at least one of the results being incorrect is:\n", "\n", ">\\begin{align}\n", "\\ 1 - (1-0.05)^{16} \\\\ \\\\\n", "\\end{align}" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "56 %\n" ] } ], "source": [ "# calculate probability of false results\n", "\n", "p_false = 1 - (1 - alpha) ** 16\n", "print(round(p_false * 100), '%')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is roughly a 56% chance that at least one of our results was false! That's pretty high. Maybe we should adjust our critical significance level to a lower value in order to decrease the probablility that one of our test results is false. Let's change alpha to 0.01 and see what happens." ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "15 %\n" ] } ], "source": [ "# calculate probability of false results\n", "\n", "alpha = 0.01\n", "\n", "p_false = 1 - (1 - alpha) ** 16\n", "print(round(p_false * 100), '%')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A 15% chance of there being an error across 16 tests seems a little more reasonable. Now, let's rerun our tests with the new alpha value to see if this made any difference in the results of our 16 z-tests." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Null hypothesis:\n", " The difference in conversion rates between samples is statistically significant\n", "Alt hypothesis:\n", " The difference in conversion rates between samples is not statistically significant\n", "\n", "alpha = 0.01\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventsample_a_convsample_b_convdifferencereject_null
0MainScreenAppear98.6398.530.10True
1OffersScreenAppear62.0860.491.59True
2CartScreenAppear50.9749.261.71True
3PaymentScreenSuccessful48.3146.082.23True
\n", "
" ], "text/plain": [ " event sample_a_conv sample_b_conv difference \\\n", "0 MainScreenAppear 98.63 98.53 0.10 \n", "1 OffersScreenAppear 62.08 60.49 1.59 \n", "2 CartScreenAppear 50.97 49.26 1.71 \n", "3 PaymentScreenSuccessful 48.31 46.08 2.23 \n", "\n", " reject_null \n", "0 True \n", "1 True \n", "2 True \n", "3 True " ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# A1 vs A2 with new alpha\n", "\n", "conversions_z_test(group_A1, group_A2, events, alpha)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Null hypothesis:\n", " The difference in conversion rates between samples is statistically significant\n", "Alt hypothesis:\n", " The difference in conversion rates between samples is not statistically significant\n", "\n", "alpha = 0.01\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventsample_a_convsample_b_convdifferencereject_null
0MainScreenAppear98.6398.270.36True
1OffersScreenAppear62.0860.351.73True
2CartScreenAppear50.9748.482.49True
3PaymentScreenSuccessful48.3146.551.76True
\n", "
" ], "text/plain": [ " event sample_a_conv sample_b_conv difference \\\n", "0 MainScreenAppear 98.63 98.27 0.36 \n", "1 OffersScreenAppear 62.08 60.35 1.73 \n", "2 CartScreenAppear 50.97 48.48 2.49 \n", "3 PaymentScreenSuccessful 48.31 46.55 1.76 \n", "\n", " reject_null \n", "0 True \n", "1 True \n", "2 True \n", "3 True " ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# A1 vs B0 with new alpha\n", "\n", "conversions_z_test(group_A1, group_B0, events, alpha)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Null hypothesis:\n", " The difference in conversion rates between samples is statistically significant\n", "Alt hypothesis:\n", " The difference in conversion rates between samples is not statistically significant\n", "\n", "alpha = 0.01\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventsample_a_convsample_b_convdifferencereject_null
0MainScreenAppear98.5398.270.26True
1OffersScreenAppear60.4960.350.14True
2CartScreenAppear49.2648.480.78True
3PaymentScreenSuccessful46.0846.55-0.47True
\n", "
" ], "text/plain": [ " event sample_a_conv sample_b_conv difference \\\n", "0 MainScreenAppear 98.53 98.27 0.26 \n", "1 OffersScreenAppear 60.49 60.35 0.14 \n", "2 CartScreenAppear 49.26 48.48 0.78 \n", "3 PaymentScreenSuccessful 46.08 46.55 -0.47 \n", "\n", " reject_null \n", "0 True \n", "1 True \n", "2 True \n", "3 True " ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# A2 vs B0 with new alpha\n", "\n", "conversions_z_test(group_A2, group_B0, events, alpha)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Null hypothesis:\n", " The difference in conversion rates between samples is statistically significant\n", "Alt hypothesis:\n", " The difference in conversion rates between samples is not statistically significant\n", "\n", "alpha = 0.01\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
eventsample_a_convsample_b_convdifferencereject_null
0MainScreenAppear98.5898.270.31True
1OffersScreenAppear61.2860.350.93True
2CartScreenAppear50.1148.481.63True
3PaymentScreenSuccessful47.1946.550.64True
\n", "
" ], "text/plain": [ " event sample_a_conv sample_b_conv difference \\\n", "0 MainScreenAppear 98.58 98.27 0.31 \n", "1 OffersScreenAppear 61.28 60.35 0.93 \n", "2 CartScreenAppear 50.11 48.48 1.63 \n", "3 PaymentScreenSuccessful 47.19 46.55 0.64 \n", "\n", " reject_null \n", "0 True \n", "1 True \n", "2 True \n", "3 True " ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# A0 vs B0 with new alpha\n", "\n", "conversions_z_test(group_A0, group_B0, events, alpha)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Conclusion\n", "Changing alpha from 0.05 to 0.01 changed the result of the test between A1 and B0 for the CartScreenEvent. Previously, we were unable to reject the null hypothesis. We could say that the new font made a difference a this stage of the funnel. Now, we can reject the null hypothesis. It's likely that this was a false result that came from an alpha value that was initially too high.\n", "\n", "We can say with certainty now that the new font didn't make a statistically significant difference in conversion rates at any stage of the funnel for any group combination. It's safe to say that changing font size isn't an effective stratgey to increase conversion rates." ] } ], "metadata": { "ExecuteTimeLog": [ { "duration": 1964, "start_time": "2021-06-10T16:24:34.972Z" }, { "duration": 315, "start_time": "2021-06-10T16:24:36.938Z" }, { "duration": 37, "start_time": "2021-06-10T16:24:37.256Z" }, { "duration": 463, "start_time": "2021-06-10T16:24:37.296Z" }, { "duration": 39, "start_time": "2021-06-10T16:24:37.761Z" }, { "duration": 11, "start_time": "2021-06-10T16:24:37.803Z" }, { "duration": 230, "start_time": "2021-06-10T16:24:37.817Z" }, { "duration": 146, "start_time": "2021-06-10T16:24:38.052Z" }, { "duration": 15, "start_time": "2021-06-10T16:24:38.201Z" }, { "duration": 31, "start_time": "2021-06-10T16:24:38.219Z" }, { "duration": 35, "start_time": "2021-06-10T16:24:38.253Z" }, { "duration": 23, "start_time": "2021-06-10T16:24:38.290Z" }, { "duration": 554, "start_time": "2021-06-10T16:24:38.316Z" }, { "duration": -154, "start_time": "2021-06-10T16:24:39.027Z" }, { "duration": -155, "start_time": "2021-06-10T16:24:39.029Z" }, { "duration": -155, "start_time": "2021-06-10T16:24:39.031Z" }, { "duration": -155, "start_time": "2021-06-10T16:24:39.032Z" }, { "duration": -156, "start_time": "2021-06-10T16:24:39.034Z" }, { "duration": -156, "start_time": "2021-06-10T16:24:39.036Z" }, { "duration": -158, "start_time": "2021-06-10T16:24:39.039Z" }, { "duration": -161, "start_time": "2021-06-10T16:24:39.044Z" }, { "duration": -162, "start_time": "2021-06-10T16:24:39.046Z" }, { "duration": -162, "start_time": "2021-06-10T16:24:39.047Z" }, { "duration": -309, "start_time": "2021-06-10T16:24:39.196Z" }, { "duration": -309, "start_time": "2021-06-10T16:24:39.197Z" }, { "duration": -309, "start_time": "2021-06-10T16:24:39.199Z" }, { "duration": -309, "start_time": "2021-06-10T16:24:39.201Z" }, { "duration": -311, "start_time": "2021-06-10T16:24:39.204Z" }, { "duration": -315, "start_time": "2021-06-10T16:24:39.209Z" }, { "duration": -316, "start_time": "2021-06-10T16:24:39.212Z" }, { "duration": -316, "start_time": "2021-06-10T16:24:39.213Z" }, { "duration": -316, "start_time": "2021-06-10T16:24:39.215Z" }, { "duration": -316, "start_time": "2021-06-10T16:24:39.216Z" }, { "duration": -316, "start_time": "2021-06-10T16:24:39.218Z" }, { "duration": -316, "start_time": "2021-06-10T16:24:39.219Z" }, { "duration": 519, "start_time": "2021-06-10T16:25:03.455Z" }, { "duration": 277, "start_time": "2021-06-10T16:25:36.169Z" }, { "duration": 605, "start_time": "2021-06-10T16:25:42.559Z" }, { "duration": 131, "start_time": "2021-06-10T16:25:53.764Z" }, { "duration": 250, "start_time": "2021-06-10T16:25:54.401Z" }, { "duration": 472, "start_time": "2021-06-10T16:26:03.202Z" }, { "duration": 489, "start_time": "2021-06-10T16:28:00.333Z" }, { "duration": 502, "start_time": "2021-06-10T16:28:28.039Z" }, { "duration": 463, "start_time": "2021-06-10T16:28:50.507Z" }, { "duration": 545, "start_time": "2021-06-10T16:29:19.154Z" }, { "duration": 576, "start_time": "2021-06-10T16:29:30.181Z" }, { "duration": 519, "start_time": "2021-06-10T16:29:53.758Z" }, { "duration": 461, "start_time": "2021-06-10T16:30:32.958Z" }, { "duration": 478, "start_time": "2021-06-10T16:30:46.939Z" }, { "duration": 590, "start_time": "2021-06-10T16:31:00.832Z" }, { "duration": 482, "start_time": "2021-06-10T16:31:16.805Z" }, { "duration": 489, "start_time": "2021-06-10T16:31:26.638Z" }, { "duration": 503, "start_time": "2021-06-10T16:33:49.432Z" }, { "duration": 593, "start_time": "2021-06-10T16:34:16.901Z" }, { "duration": 476, "start_time": "2021-06-10T16:34:34.707Z" }, { "duration": 572, "start_time": "2021-06-10T16:34:51.769Z" }, { "duration": -446, "start_time": "2021-06-10T16:39:53.457Z" }, { "duration": 483, "start_time": "2021-06-10T16:40:01.458Z" }, { "duration": 532, "start_time": "2021-06-10T16:40:33.730Z" }, { "duration": 5308, "start_time": "2021-06-10T16:40:59.449Z" }, { "duration": 552, "start_time": "2021-06-10T16:42:25.660Z" }, { "duration": 804, "start_time": "2021-06-10T16:42:55.393Z" }, { "duration": 790, "start_time": "2021-06-10T16:43:23.684Z" }, { "duration": 847, "start_time": "2021-06-10T16:43:40.336Z" }, { "duration": 756, "start_time": "2021-06-10T16:44:09.855Z" }, { "duration": 764, "start_time": "2021-06-10T16:44:46.160Z" }, { "duration": 881, "start_time": "2021-06-10T16:45:00.615Z" }, { "duration": 811, "start_time": "2021-06-10T16:45:08.886Z" }, { "duration": 539, "start_time": "2021-06-10T16:45:21.297Z" }, { "duration": 543, "start_time": "2021-06-10T16:45:39.369Z" }, { "duration": 551, "start_time": "2021-06-10T16:45:45.361Z" }, { "duration": 539, "start_time": "2021-06-10T16:45:56.398Z" }, { "duration": 559, "start_time": "2021-06-10T16:46:02.811Z" }, { "duration": 543, "start_time": "2021-06-10T16:46:13.186Z" }, { "duration": 878, "start_time": "2021-06-10T16:46:28.632Z" }, { "duration": 872, "start_time": "2021-06-10T16:46:37.772Z" }, { "duration": 614, "start_time": "2021-06-10T16:46:58.157Z" }, { "duration": 8, "start_time": "2021-06-10T16:47:40.300Z" }, { "duration": 920, "start_time": "2021-06-10T16:47:40.482Z" }, { "duration": 782, "start_time": "2021-06-10T16:48:03.881Z" }, { "duration": 794, "start_time": "2021-06-10T16:48:10.647Z" }, { "duration": 770, "start_time": "2021-06-10T16:48:16.185Z" }, { "duration": 559, "start_time": "2021-06-10T16:48:25.485Z" }, { "duration": -277, "start_time": "2021-06-10T16:49:35.589Z" }, { "duration": 748, "start_time": "2021-06-10T16:49:45.675Z" }, { "duration": 627, "start_time": "2021-06-10T16:50:06.659Z" }, { "duration": 975, "start_time": "2021-06-10T16:50:15.455Z" }, { "duration": 613, "start_time": "2021-06-10T16:50:29.211Z" }, { "duration": 863, "start_time": "2021-06-10T16:52:12.351Z" }, { "duration": 785, "start_time": "2021-06-10T16:52:31.798Z" }, { "duration": 863, "start_time": "2021-06-10T16:52:42.323Z" }, { "duration": 1735, "start_time": "2021-06-10T16:53:01.768Z" }, { "duration": 302, "start_time": "2021-06-10T16:53:03.505Z" }, { "duration": 33, "start_time": "2021-06-10T16:53:03.810Z" }, { "duration": 459, "start_time": "2021-06-10T16:53:03.845Z" }, { "duration": 36, "start_time": "2021-06-10T16:53:04.307Z" }, { "duration": 5, "start_time": "2021-06-10T16:53:04.346Z" }, { "duration": 233, "start_time": "2021-06-10T16:53:04.353Z" }, { "duration": 154, "start_time": "2021-06-10T16:53:04.590Z" }, { "duration": 13, "start_time": "2021-06-10T16:53:04.747Z" }, { "duration": 47, "start_time": "2021-06-10T16:53:04.763Z" }, { "duration": 4, "start_time": "2021-06-10T16:53:04.812Z" }, { "duration": 11, "start_time": "2021-06-10T16:53:04.819Z" }, { "duration": 894, "start_time": "2021-06-10T16:53:04.832Z" }, { "duration": 136, "start_time": "2021-06-10T16:53:05.729Z" }, { "duration": 791, "start_time": "2021-06-10T16:53:05.882Z" }, { "duration": 20, "start_time": "2021-06-10T16:53:06.676Z" }, { "duration": 37, "start_time": "2021-06-10T16:53:06.698Z" }, { "duration": 17, "start_time": "2021-06-10T16:53:06.737Z" }, { "duration": 61, "start_time": "2021-06-10T16:53:06.757Z" }, { "duration": 466, "start_time": "2021-06-10T16:53:06.821Z" }, { "duration": 11, "start_time": "2021-06-10T16:53:07.289Z" }, { "duration": 903, "start_time": "2021-06-10T16:53:07.303Z" }, { "duration": 482, "start_time": "2021-06-10T16:53:08.210Z" }, { "duration": 13, "start_time": "2021-06-10T16:53:08.696Z" }, { "duration": 177, "start_time": "2021-06-10T16:53:08.712Z" }, { "duration": 892, "start_time": "2021-06-10T16:53:08.892Z" }, { "duration": 871, "start_time": "2021-06-10T16:53:09.786Z" }, { "duration": 965, "start_time": "2021-06-10T16:53:10.660Z" }, { "duration": 1443, "start_time": "2021-06-10T16:53:11.628Z" }, { "duration": 5, "start_time": "2021-06-10T16:53:13.083Z" }, { "duration": 8, "start_time": "2021-06-10T16:53:13.090Z" }, { "duration": 875, "start_time": "2021-06-10T16:53:13.100Z" }, { "duration": 944, "start_time": "2021-06-10T16:53:13.982Z" }, { "duration": 909, "start_time": "2021-06-10T16:53:14.928Z" }, { "duration": 1345, "start_time": "2021-06-10T16:53:15.839Z" }, { "duration": 815, "start_time": "2021-06-10T16:54:07.504Z" }, { "duration": 1841, "start_time": "2021-06-10T16:54:19.301Z" }, { "duration": 321, "start_time": "2021-06-10T16:54:21.144Z" }, { "duration": 37, "start_time": "2021-06-10T16:54:21.468Z" }, { "duration": 441, "start_time": "2021-06-10T16:54:21.508Z" }, { "duration": 41, "start_time": "2021-06-10T16:54:21.951Z" }, { "duration": 7, "start_time": "2021-06-10T16:54:21.994Z" }, { "duration": 214, "start_time": "2021-06-10T16:54:22.003Z" }, { "duration": 142, "start_time": "2021-06-10T16:54:22.220Z" }, { "duration": 22, "start_time": "2021-06-10T16:54:22.365Z" }, { "duration": 31, "start_time": "2021-06-10T16:54:22.389Z" }, { "duration": 4, "start_time": "2021-06-10T16:54:22.422Z" }, { "duration": 15, "start_time": "2021-06-10T16:54:22.429Z" }, { "duration": 883, "start_time": "2021-06-10T16:54:22.446Z" }, { "duration": 137, "start_time": "2021-06-10T16:54:23.331Z" }, { "duration": 808, "start_time": "2021-06-10T16:54:23.471Z" }, { "duration": 17, "start_time": "2021-06-10T16:54:24.284Z" }, { "duration": 37, "start_time": "2021-06-10T16:54:24.304Z" }, { "duration": 42, "start_time": "2021-06-10T16:54:24.344Z" }, { "duration": 40, "start_time": "2021-06-10T16:54:24.389Z" }, { "duration": 389, "start_time": "2021-06-10T16:54:24.432Z" }, { "duration": 11, "start_time": "2021-06-10T16:54:24.824Z" }, { "duration": 795, "start_time": "2021-06-10T16:54:24.837Z" }, { "duration": 592, "start_time": "2021-06-10T16:54:25.638Z" }, { "duration": 11, "start_time": "2021-06-10T16:54:26.233Z" }, { "duration": 184, "start_time": "2021-06-10T16:54:26.247Z" }, { "duration": 960, "start_time": "2021-06-10T16:54:26.434Z" }, { "duration": 887, "start_time": "2021-06-10T16:54:27.397Z" }, { "duration": 907, "start_time": "2021-06-10T16:54:28.287Z" }, { "duration": 1387, "start_time": "2021-06-10T16:54:29.196Z" }, { "duration": 5, "start_time": "2021-06-10T16:54:30.586Z" }, { "duration": 8, "start_time": "2021-06-10T16:54:30.593Z" }, { "duration": 938, "start_time": "2021-06-10T16:54:30.603Z" }, { "duration": 896, "start_time": "2021-06-10T16:54:31.543Z" }, { "duration": 908, "start_time": "2021-06-10T16:54:32.441Z" }, { "duration": 1675, "start_time": "2021-06-10T16:54:33.352Z" }, { "duration": 852, "start_time": "2021-06-10T16:55:02.488Z" }, { "duration": 930, "start_time": "2021-06-10T16:55:10.379Z" }, { "duration": 23798, "start_time": "2021-06-11T11:14:09.235Z" }, { "duration": 312, "start_time": "2021-06-11T11:14:33.036Z" }, { "duration": 41, "start_time": "2021-06-11T11:14:33.350Z" }, { "duration": 432, "start_time": "2021-06-11T11:14:33.394Z" }, { "duration": 30, "start_time": "2021-06-11T11:14:33.829Z" }, { "duration": 21, "start_time": "2021-06-11T11:14:33.862Z" }, { "duration": 258, "start_time": "2021-06-11T11:14:33.886Z" }, { "duration": 152, "start_time": "2021-06-11T11:14:34.148Z" }, { "duration": 17, "start_time": "2021-06-11T11:14:34.304Z" }, { "duration": 63, "start_time": "2021-06-11T11:14:34.324Z" }, { "duration": 5, "start_time": "2021-06-11T11:14:34.389Z" }, { "duration": 11, "start_time": "2021-06-11T11:14:34.396Z" }, { "duration": 1013, "start_time": "2021-06-11T11:14:34.410Z" }, { "duration": 139, "start_time": "2021-06-11T11:14:35.427Z" }, { "duration": 802, "start_time": "2021-06-11T11:14:35.568Z" }, { "duration": 20, "start_time": "2021-06-11T11:14:36.373Z" }, { "duration": 36, "start_time": "2021-06-11T11:14:36.396Z" }, { "duration": 16, "start_time": "2021-06-11T11:14:36.434Z" }, { "duration": 47, "start_time": "2021-06-11T11:14:36.484Z" }, { "duration": 393, "start_time": "2021-06-11T11:14:36.533Z" }, { "duration": 12, "start_time": "2021-06-11T11:14:36.929Z" }, { "duration": 859, "start_time": "2021-06-11T11:14:36.944Z" }, { "duration": 608, "start_time": "2021-06-11T11:14:37.809Z" }, { "duration": 14, "start_time": "2021-06-11T11:14:38.421Z" }, { "duration": 218, "start_time": "2021-06-11T11:14:38.439Z" }, { "duration": 924, "start_time": "2021-06-11T11:14:38.660Z" }, { "duration": 999, "start_time": "2021-06-11T11:14:39.587Z" }, { "duration": 1012, "start_time": "2021-06-11T11:14:40.588Z" }, { "duration": 1620, "start_time": "2021-06-11T11:14:41.603Z" }, { "duration": 6, "start_time": "2021-06-11T11:14:43.226Z" }, { "duration": 10, "start_time": "2021-06-11T11:14:43.235Z" }, { "duration": 936, "start_time": "2021-06-11T11:14:43.248Z" }, { "duration": 1005, "start_time": "2021-06-11T11:14:44.187Z" }, { "duration": 1019, "start_time": "2021-06-11T11:14:45.194Z" }, { "duration": 1485, "start_time": "2021-06-11T11:14:46.215Z" }, { "duration": 41965, "start_time": "2021-06-11T13:07:40.477Z" }, { "duration": 393, "start_time": "2021-06-11T13:08:22.445Z" }, { "duration": 52, "start_time": "2021-06-11T13:08:22.841Z" }, { "duration": 513, "start_time": "2021-06-11T13:08:22.896Z" }, { "duration": 32, "start_time": "2021-06-11T13:08:23.412Z" }, { "duration": 11, "start_time": "2021-06-11T13:08:23.446Z" }, { "duration": 279, "start_time": "2021-06-11T13:08:23.459Z" }, { "duration": 163, "start_time": "2021-06-11T13:08:23.743Z" }, { "duration": 20, "start_time": "2021-06-11T13:08:23.909Z" }, { "duration": 68, "start_time": "2021-06-11T13:08:23.932Z" }, { "duration": 11, "start_time": "2021-06-11T13:08:24.003Z" }, { "duration": 14, "start_time": "2021-06-11T13:08:24.017Z" }, { "duration": 1039, "start_time": "2021-06-11T13:08:24.035Z" }, { "duration": 162, "start_time": "2021-06-11T13:08:25.077Z" }, { "duration": 960, "start_time": "2021-06-11T13:08:25.242Z" }, { "duration": 18, "start_time": "2021-06-11T13:08:26.205Z" }, { "duration": 33, "start_time": "2021-06-11T13:08:26.226Z" }, { "duration": 24, "start_time": "2021-06-11T13:08:26.291Z" }, { "duration": 40, "start_time": "2021-06-11T13:08:26.318Z" }, { "duration": 466, "start_time": "2021-06-11T13:08:26.361Z" }, { "duration": 15, "start_time": "2021-06-11T13:08:26.829Z" }, { "duration": 924, "start_time": "2021-06-11T13:08:26.847Z" }, { "duration": 590, "start_time": "2021-06-11T13:08:27.775Z" }, { "duration": 25, "start_time": "2021-06-11T13:08:28.368Z" }, { "duration": 264, "start_time": "2021-06-11T13:08:28.395Z" }, { "duration": 1628, "start_time": "2021-06-11T13:08:28.662Z" }, { "duration": 1183, "start_time": "2021-06-11T13:08:30.292Z" }, { "duration": 1151, "start_time": "2021-06-11T13:08:31.478Z" }, { "duration": 1807, "start_time": "2021-06-11T13:08:32.632Z" }, { "duration": 6, "start_time": "2021-06-11T13:08:34.442Z" }, { "duration": 11, "start_time": "2021-06-11T13:08:34.451Z" }, { "duration": 1103, "start_time": "2021-06-11T13:08:34.486Z" }, { "duration": 1148, "start_time": "2021-06-11T13:08:35.591Z" }, { "duration": 1286, "start_time": "2021-06-11T13:08:36.742Z" }, { "duration": 1688, "start_time": "2021-06-11T13:08:38.032Z" }, { "duration": 21742, "start_time": "2021-06-11T22:33:05.533Z" }, { "duration": 361, "start_time": "2021-06-11T22:33:27.278Z" }, { "duration": 46, "start_time": "2021-06-11T22:33:27.642Z" }, { "duration": 501, "start_time": "2021-06-11T22:33:27.691Z" }, { "duration": 29, "start_time": "2021-06-11T22:33:28.195Z" }, { "duration": 4, "start_time": "2021-06-11T22:33:28.227Z" }, { "duration": 271, "start_time": "2021-06-11T22:33:28.233Z" }, { "duration": 165, "start_time": "2021-06-11T22:33:28.507Z" }, { "duration": 14, "start_time": "2021-06-11T22:33:28.675Z" }, { "duration": 35, "start_time": "2021-06-11T22:33:28.692Z" }, { "duration": 27, "start_time": "2021-06-11T22:33:28.729Z" }, { "duration": 10, "start_time": "2021-06-11T22:33:28.759Z" }, { "duration": 1109, "start_time": "2021-06-11T22:33:28.772Z" }, { "duration": 152, "start_time": "2021-06-11T22:33:29.884Z" }, { "duration": 884, "start_time": "2021-06-11T22:33:30.038Z" }, { "duration": 16, "start_time": "2021-06-11T22:33:30.924Z" }, { "duration": 45, "start_time": "2021-06-11T22:33:30.955Z" }, { "duration": 18, "start_time": "2021-06-11T22:33:31.003Z" }, { "duration": 65, "start_time": "2021-06-11T22:33:31.024Z" }, { "duration": 434, "start_time": "2021-06-11T22:33:31.091Z" }, { "duration": 12, "start_time": "2021-06-11T22:33:31.527Z" }, { "duration": 895, "start_time": "2021-06-11T22:33:31.542Z" }, { "duration": 562, "start_time": "2021-06-11T22:33:32.457Z" }, { "duration": 11, "start_time": "2021-06-11T22:33:33.022Z" }, { "duration": 188, "start_time": "2021-06-11T22:33:33.035Z" }, { "duration": 1105, "start_time": "2021-06-11T22:33:33.226Z" }, { "duration": 1151, "start_time": "2021-06-11T22:33:34.333Z" }, { "duration": 1148, "start_time": "2021-06-11T22:33:35.486Z" }, { "duration": 1688, "start_time": "2021-06-11T22:33:36.636Z" }, { "duration": 5, "start_time": "2021-06-11T22:33:38.327Z" }, { "duration": 24, "start_time": "2021-06-11T22:33:38.335Z" }, { "duration": 1105, "start_time": "2021-06-11T22:33:38.362Z" }, { "duration": 1120, "start_time": "2021-06-11T22:33:39.470Z" }, { "duration": 1147, "start_time": "2021-06-11T22:33:40.592Z" }, { "duration": 1598, "start_time": "2021-06-11T22:33:41.741Z" } ], "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }