{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VE mobile default test buckets analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Druing [initial check](https://phabricator.wikimedia.org/T229426#5468481), we uncovered, \"...53.4% of the users (both registered and anonymous) who were bucketed ended up in wikitext default bucket. It turns out that it would be incredibly unlikely (p << 10^-15) to get an imbalance this big if our random assignment was actually 50%–50%. So there's clearly an serious issue somewhere that we need to understand.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Previous bucketing results:**\n",
    "\n",
    "| bucket         | users     |\n",
    "|:---------------|-----------|\n",
    "| source default | 1,302,187 |\n",
    "| visual default | 1,214,917 |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plan for investigation\n",
    "\n",
    "What might be causing contributors not being assigned to test buckets in a more balanced way?\n",
    "1. See if imblance is present for any particular dimension. \n",
    "  + Registered vs anonymous contributors\n",
    "  + Country\n",
    "  + Browser\n",
    "  + Wiki\n",
    "  + Event.action (to confirm that imblance occurs at first action and throughout editing process)\n",
    "2. Report results back to team to determine what next steps make sense.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "shhh <- function(expr) suppressPackageStartupMessages(suppressWarnings(suppressMessages(expr)))\n",
    "shhh({\n",
    "    library(tidyverse); library(lubridate)\n",
    "    library(scales); library(data.table)\n",
    "})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collect Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "query <- \n",
    "\"\n",
    "SELECT\n",
    "    event.editing_session_id as edit_attempt_id, \n",
    "    wiki,  \n",
    "    event.platform as platform, \n",
    "    useragent.browser_family as browser_family, \n",
    "    useragent.os_family as os_family, \n",
    "    event.editor_interface as interface, \n",
    "    if(event.user_id != 0, concat(wiki, '-', event.user_id), event.anonymous_user_token) as user_id,\n",
    "    event.user_id = 0 as user_is_anonymous_byid, \n",
    "    if(event.anonymous_user_token is NULL, false, true) as user_is_anonymous_bytoken, \n",
    "    event.user_id != 0 as user_is_registered, \n",
    "    event.action as action,\n",
    "    event.init_timing as init_timing,\n",
    "    event.bucket,  \n",
    "    geocoded_data['country'] as country,\n",
    "    event.user_editcount as user_edit_count\n",
    "FROM event.editattemptstep\n",
    "WHERE\n",
    "    event.bucket in ('default-visual', 'default-source') and\n",
    "    year = 2019 and (\n",
    "        month = 7 and day >= 14 or\n",
    "        month >= 8)\n",
    "\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "sessions = wmf::query_hive(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>bucket</th><th scope=col>users</th><th scope=col>attempts</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>default-source</td><td>1907187</td><td>2721255</td></tr>\n",
       "\t<tr><td>default-visual</td><td>1784439</td><td>2539846</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 3\n",
       "\\begin{tabular}{r|lll}\n",
       " bucket & users & attempts\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t default-source & 1907187 & 2721255\\\\\n",
       "\t default-visual & 1784439 & 2539846\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 3\n",
       "\n",
       "| bucket &lt;chr&gt; | users &lt;int&gt; | attempts &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| default-source | 1907187 | 2721255 |\n",
       "| default-visual | 1784439 | 2539846 |\n",
       "\n"
      ],
      "text/plain": [
       "  bucket         users   attempts\n",
       "1 default-source 1907187 2721255 \n",
       "2 default-visual 1784439 2539846 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#recheck overall user bucket number to confirm any changes imbalance \n",
    "\n",
    "sessions_all <- sessions %>%\n",
    "  group_by(bucket) %>%\n",
    "  summarise(users = n_distinct(user_id),\n",
    "        attempts = n_distinct(edit_attempt_id))\n",
    "\n",
    "sessions_all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirmed that a larger percentage of users (about 51.6%) are still being included in the wikitext as default bucket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check if imbalance is present for just registered or anonymous users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>bucket</th><th scope=col>users</th><th scope=col>attempts</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>default-source</td><td>1891449</td><td>2663795</td></tr>\n",
       "\t<tr><td>default-visual</td><td>1769443</td><td>2482893</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 3\n",
       "\\begin{tabular}{r|lll}\n",
       " bucket & users & attempts\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t default-source & 1891449 & 2663795\\\\\n",
       "\t default-visual & 1769443 & 2482893\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 3\n",
       "\n",
       "| bucket &lt;chr&gt; | users &lt;int&gt; | attempts &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| default-source | 1891449 | 2663795 |\n",
       "| default-visual | 1769443 | 2482893 |\n",
       "\n"
      ],
      "text/plain": [
       "  bucket         users   attempts\n",
       "1 default-source 1891449 2663795 \n",
       "2 default-visual 1769443 2482893 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Break down by anonymous users to identify any discrepancy that occurs there\n",
    "\n",
    "sessions_anonymous <- sessions %>%\n",
    "  filter(user_is_anonymous_byid == 'true') %>%\n",
    "  group_by(bucket) %>%\n",
    "  summarise(users = n_distinct(user_id),\n",
    "        attempts = n_distinct(edit_attempt_id))\n",
    "\n",
    "sessions_anonymous"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also see an imbalance when looking at just anonymous users. About 51.7% of users are included in the wikitext as default bucket, similar to when looking at all users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>bucket</th><th scope=col>users</th><th scope=col>attempts</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>default-source</td><td>1891449</td><td>2663795</td></tr>\n",
       "\t<tr><td>default-visual</td><td>1769443</td><td>2482893</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 3\n",
       "\\begin{tabular}{r|lll}\n",
       " bucket & users & attempts\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t default-source & 1891449 & 2663795\\\\\n",
       "\t default-visual & 1769443 & 2482893\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 3\n",
       "\n",
       "| bucket &lt;chr&gt; | users &lt;int&gt; | attempts &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| default-source | 1891449 | 2663795 |\n",
       "| default-visual | 1769443 | 2482893 |\n",
       "\n"
      ],
      "text/plain": [
       "  bucket         users   attempts\n",
       "1 default-source 1891449 2663795 \n",
       "2 default-visual 1769443 2482893 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Look at user_isanonymous_bytoken to confirm it matches with user_is_anonymous_byid\n",
    "\n",
    "sessions_anonymous_bytoken <- sessions %>%\n",
    "  filter(user_is_anonymous_bytoken == \"true\") %>%\n",
    "  group_by(bucket) %>%\n",
    "  summarise(users = n_distinct(user_id),\n",
    "        attempts = n_distinct(edit_attempt_id))\n",
    "\n",
    "sessions_anonymous_bytoken"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirmed those two are equal. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>bucket</th><th scope=col>users</th><th scope=col>attempts</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>default-source</td><td>9137</td><td>47809</td></tr>\n",
       "\t<tr><td>default-visual</td><td>8819</td><td>47925</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 3\n",
       "\\begin{tabular}{r|lll}\n",
       " bucket & users & attempts\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t default-source & 9137 & 47809\\\\\n",
       "\t default-visual & 8819 & 47925\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 3\n",
       "\n",
       "| bucket &lt;chr&gt; | users &lt;int&gt; | attempts &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| default-source | 9137 | 47809 |\n",
       "| default-visual | 8819 | 47925 |\n",
       "\n"
      ],
      "text/plain": [
       "  bucket         users attempts\n",
       "1 default-source 9137  47809   \n",
       "2 default-visual 8819  47925   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Break down by registered users to identify any discrepancy that occurs there\n",
    "\n",
    "sessions_registered <- sessions %>%\n",
    "  filter(user_is_registered == \"true\") %>%\n",
    "  group_by(bucket) %>%\n",
    "  summarise(users = n_distinct(user_id),\n",
    "        attempts = n_distinct(edit_attempt_id))\n",
    "\n",
    "sessions_registered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unable to isolate imbalance to just anonymous or registred users. \n",
    "\n",
    "We also see an imbalance when looking at just anonymous users or registered users. About 51.7% of anonymous users and 50.9% of registered users are included in the wikitext as default bucket, similar to when looking at all users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check if imblance is present for a particular browser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>bucket</th><th scope=col>browser_family</th><th scope=col>users</th><th scope=col>attempts</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>default-source</td><td>Amazon Silk  </td><td> 262</td><td> 395</td></tr>\n",
       "\t<tr><td>default-visual</td><td>Amazon Silk  </td><td> 280</td><td> 402</td></tr>\n",
       "\t<tr><td>default-source</td><td>Android      </td><td>4667</td><td>6604</td></tr>\n",
       "\t<tr><td>default-visual</td><td>Android      </td><td>3887</td><td>6178</td></tr>\n",
       "\t<tr><td>default-source</td><td>Baidu Browser</td><td>   1</td><td>   1</td></tr>\n",
       "\t<tr><td>default-source</td><td>bingbot      </td><td> 398</td><td> 689</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped_df: 6 × 4\n",
       "\\begin{tabular}{r|llll}\n",
       " bucket & browser\\_family & users & attempts\\\\\n",
       " <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t default-source & Amazon Silk   &  262 &  395\\\\\n",
       "\t default-visual & Amazon Silk   &  280 &  402\\\\\n",
       "\t default-source & Android       & 4667 & 6604\\\\\n",
       "\t default-visual & Android       & 3887 & 6178\\\\\n",
       "\t default-source & Baidu Browser &    1 &    1\\\\\n",
       "\t default-source & bingbot       &  398 &  689\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 4\n",
       "\n",
       "| bucket &lt;chr&gt; | browser_family &lt;chr&gt; | users &lt;int&gt; | attempts &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| default-source | Amazon Silk   |  262 |  395 |\n",
       "| default-visual | Amazon Silk   |  280 |  402 |\n",
       "| default-source | Android       | 4667 | 6604 |\n",
       "| default-visual | Android       | 3887 | 6178 |\n",
       "| default-source | Baidu Browser |    1 |    1 |\n",
       "| default-source | bingbot       |  398 |  689 |\n",
       "\n"
      ],
      "text/plain": [
       "  bucket         browser_family users attempts\n",
       "1 default-source Amazon Silk     262   395    \n",
       "2 default-visual Amazon Silk     280   402    \n",
       "3 default-source Android        4667  6604    \n",
       "4 default-visual Android        3887  6178    \n",
       "5 default-source Baidu Browser     1     1    \n",
       "6 default-source bingbot         398   689    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sessions_bybrowser <- sessions %>%\n",
    "  group_by(bucket, browser_family) %>%\n",
    "  summarise(users = n_distinct(user_id),\n",
    "        attempts = n_distinct(edit_attempt_id)) %>%\n",
    "  arrange(browser_family)\n",
    "\n",
    "head(sessions_bybrowser)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "sessions_bybrowser$users <- as.numeric(sessions_bybrowser$users)\n",
    "sessions_bybrowser$attempts <- as.numeric(sessions_bybrowser$attempts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "sessions_bybrowser$bucket[sessions_bybrowser$bucket == \"default-source\"] <- \"default_source\"\n",
    "sessions_bybrowser$bucket[sessions_bybrowser$bucket == \"default-visual\"] <- \"default_visual\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAC91BMVEUAAAAAv8QBAQECAgID\nAwMEBAQFBQUGBgYHBwcICAgJCQkKCgoLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQV\nFRUWFhYXFxcYGBgZGRkaGhobGxsdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJyco\nKCgqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo8\nPDw9PT0/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5P\nT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBh\nYWFiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBxcXFycnJz\nc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp7e3t8fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISF\nhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWWlpaX\nl5eYmJiZmZmampqbm5ucnJydnZ2fn5+goKChoaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamq\nqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8\nvLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3O\nzs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g\n4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy\n8vLz8/P09PT19fX29vb39/f4dm34+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///94t25lAAAACXBI\nWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2deYBcVZno74MkbAoKIos6CurMm5k3OhP16Xvz\n1HFm1DcEJMEkpNMkQBARNI6APkD2bRBkEVxgDIzKFjIIlT2ELEggKySEIISQTpCkQ0LohEAm\nTZL+/nh1v6rqqu5UV9/vfFXV1VW/3x99u/vWV+fcc+6v6tx7zxIJALiJ+joDAPUAIgGUAUQC\nKAOIBFAGEAmgDCASQBlAJIAy0OgiRTm6/C//s9ypletFfUfp7NV45itHox53jt5E6v3EsJxY\nHpGqfYr2lB4iFaVRjztHsYov/B8i2fKBSA0KIhmzgUjFadTjztGl4tuvPP7A46/aXdCo62z3\n7f3FZ95zyGd/2ZHZ8865R0Wdb5BrGU7+p8MP+PgFbfE/7//iMYOOHTqrcG/2xbuvOP7Aj1/Z\nXpB0dvPU8A8POuJfZuT+sXNENOBnhe+5zzsVbDqT65KJLtksTKDrsRS8TxTtueqjB/zlvV2O\nq+DNux/D56Jf67/uij7fwxFmclCQ4HHRcpFdB0QfTv++PDo+aeZrn/6U10pQKFLHiXrujC4m\n0qmZX0Zn9gzNn9Kdr/hx5pe/2iZyaec/9zn9R+vfJxYkndncuH/nK+Mfm74QHTq1y3uWECmf\nXGFA12wWJtD1WAreLopO1z3TCo7r0m4JFx7Db6JP67/+R3RfD0c4tHvhnRndJvKH9B8vifwi\nOith5vsB/SmvlSCK8hX5u+jAW1o3/vSAIjcb7o0OvHnjhhsHRQ/o/454aHvBO+hmVvThBze/\ns+Cz0Q9FPhB9v+XdN1L/JF1Fjf+KU7jlwOhe6XoOz90vGr30rTcnZ0NWfiz66HNd37N4I1E3\n+eS6BRRksyCBbsdS8HZRdOQDba98OfrnghQKjmWfY2g/Opqf/s9j0Yff7eEINQeFCd4fDRO5\nIfpq9EuR0dGDyTLfH0CkvEhfia6M/3VpEZG+El0Rby6J/lH/d1eXd9DNiXpOyZrokyJHRJN3\nd93b+ZemcEX0Fel6Dp8YnVnwoumHRf+ztdt7lhApn1y3gIJsFiTQ7VgK3i6Kfpf+uSL6QEEK\nBcey7zFcGmsh/xJd09MR3tU9wc37HS1yUrR8v+Eix++3JVnm+wOIlP/9iGhNvHm5iEjZXauj\nI/V/f9r3HY6KBgzYf//9ougAke9E0SF/f8Wr3d8//iubwgek6zn8wWhlwYsGRMN3dn/PEiLl\nk+sWUJDNggS6HUvB20XRG+mf7dH+BSkUHMu+x7Bh0MBXZfX+B23u6Qj/tE+Cn47+KEf/tQw+\nSlqjv0uY+f4AIuV/HxDp6ftOEZE6dw3U/+3d9x0GFny37f7p36S3h8yVfU//Lm+Tjx8YvV3w\noihu83R7zxIi5ZPrFlCQzYIEBvSUiaJ/FBxLkWMYGf0/Oa/g27T7Ee7dJ8HzoztfjsbLxdHK\nSdGFCTPfH0Ck/O+9fyN1+Srp+g5HZl6RY/OkL0WfLfLagrfZL4qvLN4o8o303WjAb/d5z67v\nVBicT67ngH2/kfbNRFGRCo6lyDEsiI5sPTRa0cMRFiu86VHTf0ST5YnoZz+IZibMfH+gv+W3\n3BRW2Feiq+LN5YUn0sDMp+k/ZJr5lxZeV2TJvuKE6Pyu7/xm9J783s7UNIUr9QriCD0Bf1fk\nGind3tn/193fs+s7FQbnk+sS0CWbBQl0OZYu79NFpML0MsdS5BjkM9Hnoi/3dITZ9+uS4DsH\n/tnZg96SPe//xucPfCdh5vsD/S2/5aawwn4XHXRba+utBxaeVR+J7nxbMjf0NsY39O6X7pWc\nfcXc/aJxS7ftevHOL4j8/W9e3b3xIv0Azu7tTC1O4baD9KL+n6Mvv9L24JGdd+2W5e/adZwd\n7XdH1/fs9k6FwfnkugR0yWZBAl2OpfB9uoqUTa/gWIocg9yTbok90qU8C/Zm369LgvLF6Oi/\nT2+GHn6ACpgk8/2B/pbfclPYLu8YknncUXhWnZfd+c3Mq07N78mRe8XN+3e+U/aXywv25lJr\nyjxliZ9NTtVfT8/s/cl+XS6GOsZF0U1d3rPbOxUGFyRXGNA1mwUJFB5Ll0x0ESmbXsGbFzkG\n2XVU9PGuV4wFe3M5KExQro70C+quKLq2oAJKZ74f0N/yW24KRZL2K4474GOXFfZskLe/f9yg\n+Le9t//dwQcP/vne/J4cuVfI4qY/O+Dgv/rX5SJPnXHcoKO/+ECXvdnU3v3xxw447gp97i/3\n/uWgj1yxJ7v3iaFHDzxyyMzc23ekz+2rC9+z2zsVBhckVxjQ7VzMJ1B4LF0y0UWkbHqFb17k\nGOR/R7d0Lc+CvbkcdElwYRQ9md68GkWLJHHma5/+ll+oLWZG79nW13moCRAJHPz6wLibDyAS\neEg3iQ96pa8zURsgEoQTDfybGX2dhxoBkQDKACIBlAFEAigDiARQBhAJoAwgEkAZQCSAMoBI\nAGUAkQDKQN+JtHPdDnPM7nVt9oTWbbHHbNhoj3n91d5f052t6+wjqt9at8scU73C3tz7a7pT\nrcJ+s5KF3XcibU7Zp7d4J/WiPaEpy+wxc5+wxyyebo9ZlXq39xd1oyX1pjmGwpa4sNvNMetS\nW5O9EJGKgkh1WdiIlKUO6xaREMkHdSuIpCCSC+pWEElBJBfUrSCSgkguqFtBJAWRXFC3gkgK\nIrmgbgWRFERyQd0KIimI5IK6FURSEMkFdSuIpCCSC+pWEElBJBfUrSCSgkguqFtBJAWRXFC3\ngkgKIrmgbgWRFERyQd0KIimI5IK6FURSEMkFdSuIpCCSC+pWEElBJBfUrSCSgkguqFtBJAWR\nXFC3gkgKIrmgbgWRFERyQd0KIikNINLSi4c237Zd5ISY9N8dE0aOvKej920SqFtBJKUBRPrx\nkp1td1yRFin797Txra3jZ/S+TQJ1K4ikNIBIMe8Mz4t0wRKRJRf2vk1C+ev2v2yUSgiREEkp\no0hP/ygt0qhh5z66V2T4NpG2Eb1vk4BIgkhKY4i0ZuyaePPu6gvvEhmyR2TPib1vRc5ramp6\nZH4pHk/NLrm/GHNTM0vsNYpUKqGpU815mz99sj1mZmqeOeax1BxzTFhhzzLHzE9Nt8dMqenC\nfrzk/pVJRVoxNvfS1xN8E+W/kRApEYg0vzFEmn9a59f75maukUpC005o2vXAw6ev1+0NLbvX\nX/IrkanZu3K9bZOASIJISgOIpM+PTtgpfzjv5HG/aY+fE40YcXdH79skIJIgktIAIlUSRBJE\nUhDJBSIJIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QS\nRFIQyQUiCSIpiOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtE\nEkRSEMkFIgkiKYjkApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJIL\nRBJEUhDJBSIJIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCS\nC0QSRFIQyQUiCSIpiOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQg\nkgtEEkRSEMkFIgkiKYjkApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJEElBJBeIJIik\nIJILRBJEUhDJBSIJIimI5AKRBJEURHKBSIJICiK5QCRBJAWRemHF/FI8nppdcn8x5qZmlthr\nFKlUQlOnmvM2f/pke8zM1DxzzGOpOeaYsMKeZY6Zn5puj5lS04X9eMn9K6shUmn4RhK+kRS+\nkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkFIgkiKYjkApEEkRREcoFIgkgKIrlAJEEk\nBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJBSIJIimI5AKRBJEURHKBSIJICiK5QCRB\nJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQyQUiCSIpiOQCkQSRFERygUiCSAoiuUAk\nQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkFIgkiKYjkApEEkRREcoFIgkgKIrlA\nJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJBSIJIimI5AKRBJEURHKBSIJICiK5\nQCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQyQUiCSIpiOQCkQSRFERygUiCSAoi\nuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQ0gEhLLx7afNt2kY4JI0fe05F8mwREEkRSGkCk\nHy/Z2XbHFSLTxre2jp+RfJsERBJEUhpApJh3hotcsERkyYXJt0lAJEEkpUFEevpHIsO3ibSN\nSL5NAiIJIimNIdKasWtEhuwR2XNi8q3I1wYPHnx3qroYRapy7qAeWZRUpBVjV0rIN9Jt11xz\nzRPLS7EwtaDk/mIsS80vsdcoUqmEps8w52357Cn2mHmpZ8wxT6YWm2MWpp4yxywtWdg9MPlx\ne0xIYT9WtcJeVHL/2oQizT9Nv965RkoATTuhadcDD5++XrdTs3fjkm6TgEiCSEoDiHSCslM6\nJowYcbc+J0q2TQIiCSIpDSBSJUEkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkF\nIgkiKYjkApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJ\nBSIJIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQ\nyQUiCSIpiOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRS\nEMkFIgkiKYjkApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJE\nUhDJBSIJIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QS\nRFIQyQUiCSIpiOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgUi/s\naCtFS+qlkvuLsSm1vMReo0ilEpo915y3tgXT7DHPpLaYY15IvWqOCSvsFeaYtikL7THVK+zN\n5pg/ptaX3L+jGiKtWV6KhakFJfcXY1lqfom9RpFKJTR9hjlvy2dPscfMSz1jjnkytdgcszD1\nlDlmacnC7oHJj9tjQgr7saoV9qKS+9dWQ6TS0LQTmnYKTTsXiCSIpCCSC0QSRFIQyQUiCSIp\niOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkFIgki\nKYjkApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJBSIJ\nIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQyQUi\nCSIpiOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkF\nIgkiKYjkApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJ\nBSIJIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQ\nyQUiCSIpjSDSCSeckN3oLx0TRo68p6P3bRIQSRBJaQSR0g51/kgzbXxr6/gZvW+TgEiCSEoD\ninTBEpElF/a+TQIiCSIpjSTSqGHnPrpXZPg2kbYRvW9FVq9atWpzWylaUi+V3F+MTanlJfYa\nRSqV0Oy55ry1LZhmj3kmtcUc80LqVXNMWGGvMMe0TVloj6leYZc+I4vxx9T6kvt32EQSeXf1\nhXeJDNkjsufE3rciXxs8ePDdqepiFKnKuYN6ZJFVJJHXE3wT5b+Rpk6aNOn5daV4PrW85P5i\nvJxaVGKvUaRSCc2abc7buvlT7TFPp14xxyxLvWCOCSns1SULuwcmP2GPCSnseUGFvcYc80wv\nhf26XaTNzVwjlYRrJOEaqWdUpBtadq+/5Ffp75nsXbnetklAJEEkpRFEyj5A+sN5J4/7TXv8\nnGjEiLs7et8mAZEEkZRGEKmCIJIgkoJILhBJEElBJBeIJIikIJILRBJEUupOpNRQkUeOP/Dr\n283JhYBIgkhK3Yn0Dwtly3uiKLrInFwIiCSIpNSdSIe1y4RoxFvXftKcXAiIJIik1J1IB22R\nr0dLpe1gc3IhIJIgklJ3In3isvnRsSI7PmBOLgREEkRS6k6k89PXR+eIzBxmTi4ERBJEUupO\npO3fOPRLr4t8dY45uRAQSRBJqTuRqgsiCSIpdSdSVOR/lQORBJGUehVpz37m5EJAJEEkpU5F\nan/4febkQkAkQSSlvkSK8nzZnFwIiCSIpNSpSO/9++fNyYWASIJISn2JJNW+g4dIgkhK3YlU\nXRBJEElBJBeIJIik1J1Iu2/51Pv0MsmcXAiIJIik1J1I383dbzAnFwIiCSIpdSfS4Sct29Hj\nC8sOIgkiKXUn0mFbzOk4QCRBJKXuRBq70JyOA0QSRFLqTqS3z39wU8JpUssAIgkiKXUnUsTN\nhhyIhEgKIiFSIYgkPJAtAJEEkRRE6gSRBJEURHKBSIJISt2JxDVSJ4iESAoiIVIhiCRVbtp1\ntN73/bfNyYWASIJISl2KlOapsebkQkAkQSSlXkXacpg5uRAQSRBJqU+R3l42hLm/EUkQSXHd\nbBhvTi4ERBJEUupTpMM+dctuc3IhIJIgklJ3IlUXRBJEUhDJBSIJIin1J9K6MccO+tCY9ebU\ngkAkQSSl7kRad5ReJR39qjm5EBBJEEmpO5HGHH77+l3rbz+cB7KIJIikBIl07GTdpD5kTi4E\nRBJEUupOpIHbdLNtkDm5EBBJEEmpO5GOyX4jHWtOLgREEkRS6k6k5sPvSF8j3XH4GHNyISCS\nIJJSdyKtPVLv2h21zpxcCIgkiKTUnUiytvmYgcec1mJOLQhEEkRS6k+kqoJIgkhK3Yl0/2jd\njH7AnFwIiCSIpNSdSH+zWjer/9acXHFa15Xi+dTykvuL8XJqUYm9RpFKJTRrtjlv6+ZPtcc8\nnXrFHLMs9YI5JqSwV5cs7B6Y/IQ9JqSw5wUV9hpzzDO9FPbrxUQalH2OdAAiIRIiKUEiHX2f\nbu49qkwilYamndC0U+quaTf6sFvW7Vp3y2FN5uRCQCRBJKXuRFr7QX2O9MG15uRCQCRBJKXu\nRJKW044ZeMyY6jyPRaQYRJJ6FKmqIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJRfnr9r/Z\nKJUQIiGSgkiIVAgiSRVFGvB7czoOEEkQSak7kT63ypyOA0QSRFLqTqQ1Zz64sTrzfscgkiCS\nUnci9felLxGpRxBJEKkARBJEUmpapOqCSIJICiK5QCRBJKX+RPrjNz+4n8gXl5qTCwGRBJGU\nuhPp+cP0+uj+s83JhYBIgkhK3Yl00nG/35r+e8Nx5uRCQCRBJKXuRHr/Qv27vb9Ooo9IPYJI\nUkWR4lmE0tdIaw41JxcCIgkiKXUn0sfvF9lf5PLPmpMLAZEEkZS6E+n8w256acC6awbdaE4u\nBEQSRFLqTqStn9R+DYN3mZMLAZEEkZS6E0m2nPeRQR/6bps5tSAQSRBJqT+RqgoiCSIpdSdS\naqjII8cf+PXt5uRCQCRBJKXuRPqHhbLlPemLpIvMyYWASIJISt2JdFi7TIhGvHXtJ83JhYBI\ngkhK3Yl00Bb5erRU2g42JxcCIgkiKXUn0icumx8dK7LjA+bkQkAkQSSl7kQ6P319dI7IzGHm\n5EJAJEEkpe5E2v6NQ7/0ushX55iTCwGRBJGUuhOpuiCSIJKCSC4QSRBJqTuRmEWoE0RCJAWR\nEKkQRJIqN+06Wu/7/tvm5EJAJEEkpS5FSvPUWHNyISCSIJJSryJtOcycXAiIJIik1KdIby8b\nQs8GRBJEUlw3G8abkwsBkQSRlPoU6bBP3VKdpV0QSRBJqTuRqgsiCSIpiOQCkQSRFERygUiC\nSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSGkGkE044Id50TBg58p6O5Nsk\nIJIgktIIIqVVin9MG9/aOn5G8m0SEEkQSWkgkS5YIrLkwuTbJCCSIJLSQCIN3ybSNiL5VmTq\npEmTnl9XiudTy0vuL8bLqUUl9hpFKpXQrNnmvK2bP9Ue83TqFXPMstQL5piQwl5dsrB7YPIT\n9piQwp4XVNhrzDHP9FLYr9tEGrJHZM+JybciXxs8ePDdqepiFKnKuYN6ZFGlv5FWr1q1anNb\nKVpSL5XcX4xNqeUl9hpFKpXQ7LnmvLUtmGaPeSa1xRzzQupVc0xYYa8wx7RNWWiPqV5hlz4j\ni/HH1PqS+3fYROIaKQFcIwnXSD2jIk3N3o1Luk0CIgkiKY0g0gkn6JOkjgkjRtzdkXybBEQS\nRFIaQaQKgkiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkFIgkiKYjk\nApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJBSIJIimI\n5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQyQUiCSIp\niOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkFIgki\nKYjkApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJBSIJ\nIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQyQUi\nCSIpiOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUGkXlizvBQLUwtK7i/GstT8\nEnuNIpVKaPoMc96Wz55ij5mXesYc82RqsTlmYeopc8zSkoXdA5Mft8eEFPZjVSvsRSX3r62G\nSDvaStGSeqnk/mJsSi0vsdcoUqmEZs81561twTR7zDOpLeaYF1KvmmPCCnuFOaZtykJ7TPUK\ne7M55o+p9SX376iGSKWhaSc07RSadi4QSRBJQSQXiCSIpCCSC0QSRFIQyQUiCSIpiOQCkQSR\nFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkFIgkiKYjkApEE\nkRREcoFI0p9F+i8bpRJCJBeIJIikIJILRBJEUhDJBSIJIimI5AKRBJEURHKBSIJICiK5QCRB\nJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQyQUiCSIpiOQCkQSRFERygUiCSAoiuUAk\nQSQFkVwgkiCSgkguEEkQSUEkF4gkiKQgkgtEEkRSEMkFIgkiKYjkApEEkRREcoFIgkgKIrlA\nJEEkBZFcIJIgkoJILhBJEElBJBeIJIikIJILRBJEUhDJBSIJIimI5AKRBJEURHKBSIJICiK5\nQCRBJAWRXCCSIJKCSC4QSRBJQSQXiCSIpCCSC0QSRFIQyQUiCSIpiOQCkQSRFERygUiCSEoD\niXRCTHrbMWHkyHs6et8mAZEEkZRGEim7nTa+tXX8jN63SUAkQSSlAUW6YInIkgt73yYBkQSR\nlEYSadSwcx/dKzJ8m0jbiN63Irddc801TywvxcLUgpL7i7EsNb/EXqNIpRKaPsOct+Wzp9hj\n5qWeMcc8mVpsjlmYesocs7RkYRtFKpVQSGE/VrXCXlRy/1qbSCLvrr7wLpEhe0T2nNj7VuRr\ngwcPvjtVXYwiVTl3dYZRpL7ObqVYZBVJ5PUE30T5b6TeoWknNO2UBmraxWxu5hqpJIiESL1x\nQ8vu9Zf8SmRq9q5cb9skIJIgktJAIv3hvJPH/aY9fk40YsTdHb1vk4BIgkhKA4lUCRBJEElB\nJBeIJIikIJILRBJEUhDJBSIJIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4QSRBJ\nQSQXiCSIpCCSC0QSRFIQyQUiCSIpiOQCkQSRFERygUiCSAoiuUAkQSQFkVwgkiCSgkguEEkQ\nSUEkF4gkiKQgkgtEEkRSEMkFIgkiKYjkApEEkRREcoFIgkgKIrlAJEEkBZFcIJIgkoJILhBJ\nEElBJBeIJIikIJILRBJEUhDJBSIJIimI5AKRBJEURHKBSIJICiK5QCRBJAWRXCCSIJKCSC4Q\nSRBJQSQXiCT9WaR+WdiIlKUO6xaREMkHdSuIpCCSC+pWEElBJBfUrSCSgkguqFtBJAWRXFC3\ngkgKIrmgbgWRFERyQd0KIimI5IK6FURSEMkFdSuIpCCSC+pWEElBJBfUrSCSgkguqFtBJAWR\nXFC3gkgKIrmgbgWRFERyQd0KIimI5IK6FURSEMkFdSuIpCBSL6yYX4rHU7NL7i/G3NTMEnuN\ndVsqoalTzXmbP32yPWZmap455rHUHHNMWGHPKrG3jIU9paYL+/GS+1dWQ6TS8CEpfCMpfCO5\noG4FkZTyF3b5pjhCpFqr2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+Ig\nkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+Ig\nkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+Ig\nkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+Ig\nkiCSkwar2w7PGPkAAB5RSURBVOIgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar2+IgkiCSkwar\n2+IgkiCSkwar2+IgklSisMuXOUSqtbotDiIJIjlpsLotDiIJIjlpsLotDiIJIjlpsLotDiIJ\nIjlpsLotDiIJIjlpsLotDiIJIjlpsLotDiIJIjlpsLotDiIJIjlpsLotDiIJIjkpXbchnTxq\num6Lg0iCSE4QSRBJQSQXiCSIpCCSi34sUkjmioNIgkhOEEkQSUEkF4gkiKQgkgtEEkRSEMkF\nIgkiKYjkApEEkRREclG6bkOKonzFh0iCSAoiuYqvl7otX0KIJIjkBJEEkRREcoFIgkhKbYjk\nbK+XV6SOCSNH3tOR7LWIJIiklP+CNCRzNSXStPGtreNnJHttjYgUUnxBCRWlNkQKOYfKVwY1\nIpLzgMor0gVLRJZc2K+KIiBvNS1STRd2ccr/9V+tmALKK9LwbSJtI+LfzmtqanpkfgZjFqsa\nYwwKP6DizEzNK7HXKEXtF3bA8dT2Ac1fWRmRhuwR2XNi/NvYIUOGTJxdilmpGSX3F+Ox1HRz\nzOzUVHvMlCn2mKkpe8z0gJgZqVnmmJmpmeaY2i7syfaYaanHzDEzeim45ZURKf+N1Dvlv/7t\ngSnL7DGMR6rLwm43x/TR7e/i10jFoW4FkRRE2oepZbtrV5w6rFtEQqR96ZgwYsTdZXmOVJw6\nrFtEQiQf1K0gkoJILqhbQSQFkVxQt4JICiK5oG4FkRREckHdCiIpiOSCuhVEUhDJBXUriKQg\nkgvqVhBJQSQX1K0gkoJILqhbQSQFkVxQt4JICiK5oG4FkRREckHdCiIpiOSCuhVEUhDJBXUr\niKQgkgvqVhBJQSQX1K0gkoJILqhbQSQFkVxQt4JICiK5oG4FkRREckHdCiIpiOSCuhVEUhDJ\nBXUriKQgkouNE9eaY3ZMXGFPaNJT9pjps+wxf/i9PeaZibvMMS9N3GyOaZ34ijnm7Xor7GcD\nCnt10sLuO5EWDp5mjtkw+Ff2hD7/Y3vMKafZY37wJXvMLYO3mWMmDX7OHLNo8BRzTGtIYX/h\nEnvMN5vtMecHFPatg+1f5ZMGJ/w0QaSiIBIiKYiUB5EQSUGkThBJEElBJIB6BJEAykBfifRq\nwC1PgJqlr0TadMbUPkoZKsfe5/s6B31GnzXtatuk1p0hUdvLnY3+RutPEq4y141aLrhVyV7W\nd9dINWzSaxePWxQQtuHbe8qelTKQeZxfrab0hpCYahTcinky/SV72OIzk1nedyKtvm2I2aQl\n191s7/9lDnqhaXLYB+u1KWtEyAEZYzq+Nzf9843TrMmEFfbi5rnmmOoUXEvzLWdusiYjGy58\nIdkL+0ykP5z28H2nGk16cuzs6aclXOs590lsDErzztil8cb6IZl2b/Ppb9liQg7IfDwt8bm9\n9pzkAYHpxCw+c4XZJFvBhWfunhMetgWk2dF8SsL+eX0gUuaK9PR02/M1o0nnPCcvjl7aluzF\nmU9iY1DM9J+kf8z51knX2foK/+dv35RJd5lCgg7IfDxq0nM/TPzy4HQyn98tVpNMBefI3KML\nA74tW0b9Z7IX9oFImSvSYfEj/Um21t2It9OlJ9cmbIZna9QWlGbiHdJ2w7nP777c1sF469Bx\nt6757mummJADMh+PBj59lenlYelkPr+tJtkKLjhz2di2l+0xSeibpl26AK67L72d8W+PWsKu\nvTVden86bW/Cl2fKwBiUbmeM/eGIX6a/jf79d8lzNnuFyB2Pzjuv+erkMfa86QGZj0cDrxza\ndOrI4QnbNuHp6Od3S3NL4gh7wYVlLnNN1XLand9+KHFMJiihSX0i0o503t4YM3Hn+nHPmuI2\nNd28+8WzZyZ6bdyA1DKwBKWb4LtF3locj0HZPHp98pwtOvuGNzaetadjYfKr5tlTww7Idjwi\ny396+8vS0vT7tm3bd1QyHSVz1hmaXMaCC81c7ppq86+fTJ63bFCyz4W++UaKy2Lzld8YbmjY\ntUxtEdl41YjvzE32em1Aar0aglafP2zYFZmxpC1nP5I8cyJ7Hj3j4evnGAJW3xo3a0MOyBCT\nZl7zw/eNmmVobwWmIwsvuup5e7vOWHChmQu5puoMShTWR3fttLx3GwIeH31Ts3VQ5AZrva5p\nmtfRPmV0+nxYe3vT7MRhmWbD9jtHnps8qTljHnnAetPSfkBpmlel45peNLW3bOm8qveh54yd\n/8Tp023tupCCCyqEsGsqU1D1RdKmhrUsdpz2mrxhad5KpgFpS+eix+Kfy5u3Scu0txNHdd6K\n/VOipob2MuwY9UL6lDCaZD+gdFvoRL2p81NLe8uWzu6z4xe2j/lT+uP7tKmWdIwFF5K5HCEX\nfLagqouUbWqYPrlEVt6e/rE1oUm5O/8tmTJPns5JmRveP78/4eszN/KNzQbt0bHn5Ph65eEh\niR6FdD7JMB9Qmsv1pk6yS/mggmsdF/989ueyvmnp4wnvwgYUnKsQ7Bd8G6dusgVVXaRcU8Ny\nRTpv7FC9e7v120k6PmUeNay8/dctWubJ0tGqHfGG/v5U0jtImRv51maDmnRtLOus65oSPDfP\nH48YDijHxub4pk6iS+ygghP5/nSZ/lLLMx3nPi6zbkx2F9ZecI5CWHDOuQut11Rzmh9qMV3B\nVl2kfFMjOUvHLlt2un7UJetKGpf0gvQXX/Mc4yX2rRP096m3JAz609Nxi93cbIhNemPMxPZX\nxz07/dIEr88fj7FNo63ojRcPGZ7wXmJIwaWLLtP3Zv04efOMhP1pAgouuBCWjF2y5HTj1eim\nUa/Ju2tMD+SrLNIqU1Mjw4aLVhi7uKZL+vy440TzGtsl9tYx8fXLm2cknYbq6Yt2hNwn1l6G\nm684edgjsuJbSQLyx2Nq0+Ra0TsT93YKK7hM35udZ1zfnLAXQFDBBRbChoufs3aQ3i5zLpMn\nxo47O/l1ctVEyo7jW3zmdkNTI8OO5lPijwajSZmOEz83XmKvH3fl9EljJyUN2TP69YBbsble\nhjt3y67LJiQKyR+PpU3T2YpOTkjB5frevJFK3Lk6qOBCCmFV+uyJp+C0nD0bvr2nbexZ56yU\nyywTV1ZJpMyBaFdaQ1MjQ8soffkmSyeIllH6xXeDKZ101e6actMvEjZPYv79QVs7Q2ftzPcy\nnHbOLxM+Agg5npBWtD0hfXxkvotmLTjFXgjpD+6As+falOxYvVe2Nm80ZK5aTbvYpFxX2uRN\njUwz317k6UIfHX/xPW2LsVftt01Ry0fHH6UhvQyDjsfeirYnlH18ZC06Y8GF5S3zwW1MJtcX\nfePkMabua1W7RopNStyVtpNMMz/gEjud1tBzrIMA7FU7/iXDrdjlTXrX19zLMNPJJ+nx5Lv9\nW1vRAQXX+fjIej/aVHBhect9cNtqNdcXvWVi8szFVE0kvcIOfS4fcoltrdqYxDErL5+jdxAn\n/1ISt9gzHq0y9zLs7OSTKG8F94mNreiQgss/PkpYCCEFF5i3zjEQpqCQTvwx1RIpe4Ud+lw+\n5BLb2LNKSRqz+6nrRt38bIdsH5O4mbpiVOzRXefvsfYyzB1QwrwV3Ce2taLNBde6U2yPjySo\n4ILyliF7uiUOCuzEH1MtkXJX2JZPh1XWZn5LR9AldtCYanl75kVjJ6y7fmHS12ul3vUDvaWa\ntJdh0AEV3CdOHBKQTnZiC8vjowzGggusVNMYiBwBnfhzVEukzivs5B8p8bQTpmb+y8Nu6jC5\nFzJsuXDk8eaHzh2RfN7ldKWqR23zk0aYDyiXUOd94sqlk5vYIvHjo8CCCywD2xiITsyd+Dup\nlkj2cXyZaScszfyNw3+ULvTk7oUMWy68BNE0E+YtpqUpfsTXdu69SQOsB9SZkPE+cUA6+Ykt\nEj4+Ci240DIwjYHIoDc0jH3RO6mWSOZxfJ3TThia+ecvuCgu9MTuhQxb7nIJYkSHOif3yH5A\nkmvRGO8T29OxT2wRWnABZRBjHziRu6Fh6Yuepxoiaa/BpFfY+V6+5nvlMvnGXXGhJ3Rv7/NB\nw5YDLkEKYidbPDIeUEyuRWO7+W9PJ2Bii8CCs+YtO3+dfeBEQEeQAqogUq7XYKIr7EwLQNUz\nXSjqLH7bhr+TKfRExP1UQ4Ytmy9BCmNHJfUo4IBici2axNcGgemETGxhLrigvGXnrzPW6qqw\njiB5Ki+SsddgfGpn1TOYtGLYdx7aInLlbDEV+oaQYctB/XVyJG2yBx5QvkWTMKHQdEImtrAW\nXGDesvPXmWo1vrEV1BGkk4qLZO41mD61c+olv+XSMvpXv2i6ZPasS9Jtw+RjxON+qtXqf2RO\nIuSA7C2a0HSy0baJLYwFF5i3gPnrMp1AzTc0CqmkSJkPEXOvwZa8eslvuaR92PPUtd80fjG3\n2E3KXMwH9D8yEnBALz9sHNUZmE4O08QWIQUXmjfr/HXZG1vmGxqFVFCkFaN+ppeVtjO1ZWdQ\nd+9MKjuSfRLnH2lkTGpJmMb20McTIVgOSHlyzPRXjS2aoHTykUkntoivdYIKzp43vbq2zl/X\neWPL0hGkGxUU6b7rHjzj/MffNZm05pxLng3pPCqWoMJHGvrhlTCufVx7yOOJUKyl8C3bJIHB\n6Zg7gsTlFlhw5rxlb2wln79u/aUjbmoPO+O6UEGRtp66s2Ph906dsDH5x9D60fqYr9ImFT7S\nsHxE/mRK4Fy5YRhL4ZR4tNMmy4ILs3dZ0gldlEDLLbTgbGVgHw67vGnq5usmBJ5xhVTyGum6\n6ekPodkTmi5/M+nH0JW5CgpqOhnuTRQ80jB8RK45qyNoXqdQbKVwdXxn/Yn7DBG3vGZIJ3hR\nAi23sAmxkuYtZsU8+42ttEcrRRZevz3wjCugkiIt/178ISTvJl/N/pTO5+SGiso3NSz3JqzP\nglri3v+XLAiayNdGQdPJdLq2Nj/49pqzVhoibs/VTKJ0whYlyJab9S5IZt7WxHnLZs96da09\n8jsuH3rqNe96G+sVvf199siltoBTt+mm4yZDTNA6PvZnQSvOWCyy9AL7xbyVoOPR827jZSc1\nmcbd/nKuKZGgRQly5WYruNy8rdbsGVto+vI7x2/ac1XieTp6oqIiPfJvxoBb79HNyssNMQFN\nDR2/bn0WdPNJ12+Vc00DBoIImaY6d94ZnqjOuXPCby+4bOqyDQlvVAUuShBWbvl5Wy2EmaQ9\n8ifeaUtqXyoq0o5TjavsZubD2vqtxIN8JLx3ovVZ0LazHj57ymPXGiLCMB7Pc5vCzrutLy1I\nnXf9r2++IuFncdCiBDEh5Wabt7UT28OMTIj2yN96uvsDsrI9G261fmOuP+vqWRPHmHqrBvdO\nTF7kqy9ZmD6LZv3s7V/94Bv2QchGbMezPJ6rNfC8k3tsjx/N09dPGhZUbqvsA2+zWB5mFIac\n627ZVVikl8wZbJ/+swmvJH51vNaLqanRZbhl8iJfcvE5M3fLxS/KKxcEjfpKzOyptuPJzANh\nP+8yc4k8do8lb/bp6793b0i56Tri9oG3iuH7qGVnLsTWI78H+m5V8zKQWevF0NQIHW6Zjrxh\n7MQXLwxb7Dw5umiS4Xgy80A88Hry8y6zBkt26M0LyeZmDl6U4Oz1AeWWGdBpmbdVMqu2KUk/\nHDNP/pXkPfJL0Z9Fsq/1EjrcMmbTnc1NxvtIiclORGteNKlzHoik511mDZZc43Z7oqUwA+fW\nT3P6FjGXW25Ap2He1tyqbQZyT/6VsvRS6c8imdZ6yRA43DLDWw9alzpLSuYRYsCiSbm7TknP\nu8waLMahN0Fz66+YJ9L0jv5qK7eAAZ32VduuLHuX4/4rkm2tlyz2oaDVwb5oUpbMXafk6Bos\n1sZtwNz622PnhpubwvHDaHNvnYAPoFNMK00kod+KZF3rJXQoaNhcXWasiyZ1YjztMmuwWBu3\n5rn128e1pzN2yvRHHviPO3+2OnE6mYfRVpMCPoBCnvyXpr+KZF7rxTzcMmSurnB0IlrLokmd\nGE+7zPhRa+PW3BHkJ1PSGTvtxtvv+u3ExLOPdT6MtvZ7M30ArYgX7Ql58l+afiqSfa0X63DL\nkLm6wslORGtZNKkT22mXGz9qbNxaO4LE/VTD10xOXt46/sj0AbSyaUXYk//S9FORAgb/WYdb\nhszVFUznUi+GRZM6CXgEacE8f32un6rdJPPD9ez4I9MHkJoU8OS/NP1VpJBFQSzDLQPn6gpA\n10wqWOol+aJJdjIXfLais89f39lP1Tw2wdqPLz/+yPIBpCbZnvz3Tn8UKfsRGWZSQkLn6jKT\nWTPJPhFtCLkLPtP5HTB/fb6favKY7PoZpn58XcYfGT6AVjYlv/+RlH4oUvi6LcbhlkFzdRnJ\nrplknog2iM4LvqQLWwTOXx/QTzVs8lpjAz9+tqU8foE1pV7phyI51m2xuBc6V5eJ3JpJ1qVe\nQlhlveALmls/rH/vhoA5WONuQcaJdXKvbh9iSSgR/U8k14yYFvcChrdYya+ZlHypl1AWn7nd\neMEX0KFq6/iFIf1Ud4RMXqstujCTVgTNk1+S/iaSec0ksT5TDZ2rK4TCNZMqxl5tPMZ9Qa0X\nfAEdqi57xNxPddfuwMlrsya1GEIyJrWdtcCUUBL6mUjmNZPE+kw1dK6uMOxrJtnRIXmZvqDG\nC76ADlXrm7bZ+qmuPn/YsCv+ZF9nOf6xacxU+xp+97+xdNxEU0wi+pdI9jWTxPxMNXSurjDM\nayYFscHcFzSkQ5Vey9+pzbPE/VTXNM3raJ8y+nnzOstz481M81hG2Xj1qf9avsewefqVSCFr\nJtmfqQbO1RWIec0kM89tsg/JC5q/XlN4q6nFkrmLHot/Lm/eFnS1M+unFX5gkJz+IVLn0DJ7\nF/uAZ6qedVvslGmEZo/Eg9FbrCYFzF+/YkUmhSkXW3J3UqYb9s/vN+Vv49RNLc2PyRtnVKM/\ncTL6hUj5RZOsd9HMY9EzUY51W+yUZ4RmT2QHo1sHt9rnr4/7C8TJ7D3XciU/4g3dPHW1ppnw\nQdqc5odapGXcd0+t1PiwAPqFSJJfNMlmknksei69yq/bUkAl24+5wejWwa0B89fnTHp23LvJ\ng27NdOuZqoPeX0sWuGnUa/LumvZdz1gW8K00/UMkyS+aZHqmah6LLtkOSFVYt6U6dN5gN983\nsT9Cy5l0teGmWKYb9ptnrDCkM+cyeWLsONuAxorTT0SS/KJJho/VgLHouQ5IVVi3pTp0DkY3\nf++FmLQ6LriNow3jT9ePu3L6pLGm2abaxp51zkq57Alj7ipLvxAp7pdvHzdhHosedy3r7IBU\nhXVbqoN1MHpBpGH++gxxH7Z0wZmaXLum3PSL5EOC44te2bF6r2xtrqWGXb8QKTtzkvUD0jwW\nXbuWOZfkrUXCuzlZ5q9XKtCHrSuZi960qpPH1MyN7wy1L1LnzEnGvtvWsejZrmW+JXlrkop3\nva1kH7YudF70tkxsqWxKZmpfpPzMSba+29ax6NmuZb4leWuTKnTQqFgfti6EXPRWidoXqWDm\nJNPtW/tCtJmuZa4leWuUil3w7c0uYlS5PmwFBE3AVi1qX6SQmZN03RZLg6ZL17Jam/SuhtEO\nsUql+rDlMV/0VpXaFylg5qTsui2GaUFDupaBUpXldCXkoreq1L5IATMnmddtCehaBhl2VPxO\nRi4d+0VvNal9kewzJwWs22LvWgYZVlX+nmCGgIveatIPRDLOnBQyhjakaxnELD5ze9VMqlI6\nYfQHkUyEjKGNqe1qqlV0pGUVik4Xm67pKqo3kYLG0Co1XU01SnakZcWfU2UXm67lDpB1JlLY\nGNoMtVxNtUp2pGWFOyZ2LjZdwx0g60yksDG0WWq4mmqR6jS3WneGLzZdTepEpK5zaFmo0vpH\n9URmHdrO5tbcCib12sXjFoUsNl196kOk/Fh0q0nVWv+ojsisQ9vZ3Kpki/iFpsmZx+OBi5xX\nkfoQqWAsuvFap0rrH9UTmXVo882tyhXeO2O14/8e8yLnfUCdiFQwFt12rVOl9Y/qCl2HthrN\nrek/Sf+Y862Trmu3LXLeF9SJSC07WwqX+EhO5dc/qj8y69BWobk18Q5pu+Hc53dfXkOzBfVE\nXYikY2gD1vALmqsLsuvQVr65tXnsD0f8sl3k339X0WTKQj2IlB1Da71/FDpXF2TXoa18c+ut\nxZvTPzePXl/hdMpAPYiUG0NrMylori7IUM1uIC1nP1K1tMKpB5E6x9Ca7tfV8LDlmkafw1bP\npLW3N/WLgS31IFJ+DG3y+3U1PWy5lsk+h61ah6qWabU1EWRP1INIAWNoa3vYcg3T+RyWDlVd\nqQeR7GNoa3zYcq3SurNfdHvrE+pBJPMY2loftlybaL+3/tDtrU+oC5GMY2hrfthyTZLr91b7\n3d76hPoQyQyj+Kx09nur/W5vfULjiaTLtmCSlXy/t5rv9tYnNJxI2WVbGA9rpD/1e+sLGk6k\n3LIt3L610Z/6vfUFDSbSqjpctqVK9KN+b31BY4m0+MztdbhsSxXpH/3e+oKGEimeq6sel22p\nFv2l31tf0EgiZebqqsdlW6pEf+n31hc0kki5ubpYtgXKTiOIFD5XF8QwY1kCGkCk8Lm6Gpzs\nBxAzliWhAUQKn6urscl8ADFjWTIaQaTguboanOzXNzOWJaERRAqeq6uh2Zub2JsZy5JQ/yIF\nztXV8MTrLKtJzFiWhLoXKXCuLpB4nWUtNmYsS0DdixQ2VxdIZp1lii0hdS9S2FxdjU3uwVsL\nJiWm7kUKmaurwcnc9155+69bMia19HF++gV1L1LAXF0NT+zPguaH72ueE//GB1AS6l4k+1xd\nEJt0/iqR15rX8H2UkLoXyTxXF0hs0jAdAPlzvo8SUv8imefqAok7yusAyBv6Oh/9hgYQCUJo\nGR0PgHy6r7PRb0Ak6E52wrJRQ8+hz3diEAm6wYRlISASdIMJy0JAJMjT0iHChGVBIBJ08vKw\nm9ImMWFZCIgEnWwc/qO0SUxYFgIiQZ7zF1wUm8SEZXYQCfJMvnFXbBITltlBJFC2xz+2DX8n\nYxJYQSSIWTHsOw9tEblytmBSEIgEMS2jf/WLpktmz7pEZBfzeweASKC0NM/d89S13+TxUSCI\nBBl0TPkOJt0KBJEgC7MzeEAkyIFJDhAJOqG/dziIBHno7x0MIgGUAUQCKAOIBFAGEAmgDCAS\nQBlApNoiKmOF7PjuRwaWfDtNrJwpNjCUYm1RztP629FN7/SeGCKVBUqxtijnaf2haEeVU2xg\nKMXaopyndbL3QqSyQCnWFlG06asH//nv9bdXPj9AZN2YYwd9aOyr6X98LFoh8n++KPJsdJzI\n9gs+cfBhX0mJvHnBJw44/OQX8hH5t4qRtd858tBP3b5X//HWmMOOuUk2j3jfEf+6R/JNuz/u\n/943079tHThocx8ccn2ASLVFFJ374ot/PuDZ+LczV6Wk5ajDU9seff/R60XOiO6QHYccsl1+\nFp0p8o3o0s27Fp4sW/780Hu3LTzufS/mIgrfK/2j5W9nv7P2m9EN+vf3FmwZFj10Uvzzdim4\nRjo5uj792z3RN/rikOsDRKotoughkYnR8Pi3B9N/n5a2R9LqjBH5XTRCUv/rCw/LKdF9IodG\nf9CA86Ib0z9/n48ofK/cb1ujv9C/7xdZHr3n/vhL7W+lQKRF0YffFTkpYtH3YBCptoiiVpHW\nuO0WRRvTfx8Tpb+LZF30IZGN0YflvB9f8q30/9Kv+VIU/d+73xD5SPSSxKZ8MBdR+F7pH3t+\n8ulD0i28gfr3JpEd2Z/vkwKR5MvRb+TtQ47eXeWjrSMQqbaIovRXQ3t0UPxbfFoPjOKVkXdF\ng9I//yp65S/mzvnoS9Ffp/94/dvHRNF7UzIoGjBg//2iaL9cROF7pX98Pxq5ur09q0xHwc8u\nP2amv6H+M/pBNY+0zkCk2kK/L7LfSPHfmW+k9dGxErfirjq4fdfBF0Tfy7x26fDoePlQtC4f\n2/290j+OiN/wlYInRgU/C/77d9Hjo6PnKntsdQ0i1RZRNEnkocwVT/x3s14j3R5fI6UvhN7/\njyJfOTh6JPvit6ID03Zdk4/t/l7pH4dHb4rc3KtID0T/fPhnKnpkdQ4i1RZR9N0XX/zvA5bl\nTvC1Rx4+edvkw4+Kv3baBsQ3166LBsTj7746c8e266OvyZufHnT5K+0bH/hiDyKdGv1g+4yx\nvYq09xORKguBIFJtEUUb//GgT06STi3WNh8z8Ngxmebb56JF8R22z8W/zxn6gYM+8cO0Ujtv\n+Mx7B370zMU9iPTGqCPeP/atXkWSG6MDt1b+8OoXRALl5rg5CcEgEijfj5hg1QMiQcy6T5zS\n11no3yBSnRHlsEUdMnR7hTLUICASQBlAJIAygEgAZQCRAMoAIgGUAUQCKAOIBFAGEAmgDPx/\nT2soXbXCWf0AAAAASUVORK5CYII=",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Look at test bucket user counts for top browsers \n",
    "sessions_bybrowser_plot <- sessions_bybrowser %>%\n",
    "  filter(browser_family %in% c('Chrome Mobile', 'Mobile Safari', 'Samsung Internet', 'Chrome', \n",
    "                              'Chrome Mobile WebView', 'UC Browser', 'Opera Mobile', 'Firefox Mobile',\n",
    "                              'Chrome Mobile iOS', 'Facebook', 'Android', 'Mobile Safari UI/WKWebView',\n",
    "                              'Edge Mobile', 'Edge')) %>% # filter to top browsers for plot visibility\n",
    "  ggplot(aes(x= browser_family, y = users, fill = bucket)) +\n",
    "  geom_col() +\n",
    "   scale_y_continuous(\"user counts\") +\n",
    "    labs(title = \"Editor test bucket user counts by browser\") +\n",
    "    ggthemes::theme_tufte(base_size = 10, base_family = \"Gill Sans\") +\n",
    "    theme(axis.text.x=element_text(angle = 45, hjust = 1),\n",
    "        plot.title = element_text(hjust = 0.5),\n",
    "        panel.grid = element_line(\"gray70\"),\n",
    "        legend.position= \"none\")\n",
    "\n",
    "\n",
    "sessions_bybrowser_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the major browser types, there is a similar imbalance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 6 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>browser_family</th><th scope=col>default_source</th><th scope=col>default_visual</th><th scope=col>percent_wikitext_users</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>bingbot    </td><td>398</td><td>2</td><td>99.50000</td></tr>\n",
       "\t<tr><td>BingPreview</td><td>377</td><td>7</td><td>98.17708</td></tr>\n",
       "\t<tr><td>NetFront NX</td><td> 18</td><td>1</td><td>94.73684</td></tr>\n",
       "\t<tr><td>Chromium   </td><td>  7</td><td>1</td><td>87.50000</td></tr>\n",
       "\t<tr><td>oBot       </td><td>  3</td><td>1</td><td>75.00000</td></tr>\n",
       "\t<tr><td>Vivaldi    </td><td>  3</td><td>1</td><td>75.00000</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 6 × 4\n",
       "\\begin{tabular}{r|llll}\n",
       " browser\\_family & default\\_source & default\\_visual & percent\\_wikitext\\_users\\\\\n",
       " <chr> & <dbl> & <dbl> & <dbl>\\\\\n",
       "\\hline\n",
       "\t bingbot     & 398 & 2 & 99.50000\\\\\n",
       "\t BingPreview & 377 & 7 & 98.17708\\\\\n",
       "\t NetFront NX &  18 & 1 & 94.73684\\\\\n",
       "\t Chromium    &   7 & 1 & 87.50000\\\\\n",
       "\t oBot        &   3 & 1 & 75.00000\\\\\n",
       "\t Vivaldi     &   3 & 1 & 75.00000\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 6 × 4\n",
       "\n",
       "| browser_family &lt;chr&gt; | default_source &lt;dbl&gt; | default_visual &lt;dbl&gt; | percent_wikitext_users &lt;dbl&gt; |\n",
       "|---|---|---|---|\n",
       "| bingbot     | 398 | 2 | 99.50000 |\n",
       "| BingPreview | 377 | 7 | 98.17708 |\n",
       "| NetFront NX |  18 | 1 | 94.73684 |\n",
       "| Chromium    |   7 | 1 | 87.50000 |\n",
       "| oBot        |   3 | 1 | 75.00000 |\n",
       "| Vivaldi     |   3 | 1 | 75.00000 |\n",
       "\n"
      ],
      "text/plain": [
       "  browser_family default_source default_visual percent_wikitext_users\n",
       "1 bingbot        398            2              99.50000              \n",
       "2 BingPreview    377            7              98.17708              \n",
       "3 NetFront NX     18            1              94.73684              \n",
       "4 Chromium         7            1              87.50000              \n",
       "5 oBot             3            1              75.00000              \n",
       "6 Vivaldi          3            1              75.00000              "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sessions_bybrowser_percent_imbalance <- sessions_bybrowser %>%\n",
    " select(-4)  %>% ##look at user counts only\n",
    " spread(bucket,users) %>% \n",
    " #mutate(percent_diff = round(abs((default_source-default_visual)/default_visual *100), 3)) %>%\n",
    " mutate(percent_wikitext_users = default_source/(default_source + default_visual) *100) %>%\n",
    " arrange(desc(percent_wikitext_users)) \n",
    "\n",
    "head(sessions_bybrowser_percent_imbalance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do notice some much larger differences in buckets for the smaller browsers. Some of these are just due to the smaller population of overall users; however, there seems to be a signficant difference seeen for Bingbot and BingPreview browsers. These are both web crawling bot browsers, which might be a clue into what is occuring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check if imbalance is present for a particular country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>bucket</th><th scope=col>country</th><th scope=col>users</th><th scope=col>attempts</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>default-visual</td><td>Afghanistan</td><td> 53</td><td>  72</td></tr>\n",
       "\t<tr><td>default-source</td><td>Afghanistan</td><td> 56</td><td>  65</td></tr>\n",
       "\t<tr><td>default-visual</td><td>Åland      </td><td>221</td><td> 284</td></tr>\n",
       "\t<tr><td>default-source</td><td>Åland      </td><td>239</td><td> 313</td></tr>\n",
       "\t<tr><td>default-visual</td><td>Albania    </td><td>810</td><td>1073</td></tr>\n",
       "\t<tr><td>default-source</td><td>Albania    </td><td>871</td><td>1145</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped_df: 6 × 4\n",
       "\\begin{tabular}{r|llll}\n",
       " bucket & country & users & attempts\\\\\n",
       " <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t default-visual & Afghanistan &  53 &   72\\\\\n",
       "\t default-source & Afghanistan &  56 &   65\\\\\n",
       "\t default-visual & Åland       & 221 &  284\\\\\n",
       "\t default-source & Åland       & 239 &  313\\\\\n",
       "\t default-visual & Albania     & 810 & 1073\\\\\n",
       "\t default-source & Albania     & 871 & 1145\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 4\n",
       "\n",
       "| bucket &lt;chr&gt; | country &lt;chr&gt; | users &lt;int&gt; | attempts &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| default-visual | Afghanistan |  53 |   72 |\n",
       "| default-source | Afghanistan |  56 |   65 |\n",
       "| default-visual | Åland       | 221 |  284 |\n",
       "| default-source | Åland       | 239 |  313 |\n",
       "| default-visual | Albania     | 810 | 1073 |\n",
       "| default-source | Albania     | 871 | 1145 |\n",
       "\n"
      ],
      "text/plain": [
       "  bucket         country     users attempts\n",
       "1 default-visual Afghanistan  53     72    \n",
       "2 default-source Afghanistan  56     65    \n",
       "3 default-visual Åland       221    284    \n",
       "4 default-source Åland       239    313    \n",
       "5 default-visual Albania     810   1073    \n",
       "6 default-source Albania     871   1145    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sessions_bycountry <- sessions %>%\n",
    "  group_by(bucket, country) %>%\n",
    "  summarise(users = n_distinct(user_id),\n",
    "        attempts = n_distinct(edit_attempt_id)) %>%\n",
    "  arrange(country, users) \n",
    "\n",
    "head(sessions_bycountry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unable to isolate to a specific country. A large number of users for each country (around 51%) are added to the wikitext as default bucket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is an imbalance across all countries as well with a higher percentage of users added to the wikitext editor as default. bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check if imbalance is present for a particular wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 40 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>bucket</th><th scope=col>wiki</th><th scope=col>users</th><th scope=col>attempts</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>default-source</td><td>azwiki    </td><td> 30959</td><td> 45825</td></tr>\n",
       "\t<tr><td>default-visual</td><td>azwiki    </td><td> 27084</td><td> 39485</td></tr>\n",
       "\t<tr><td>default-source</td><td>bgwiki    </td><td> 36032</td><td> 54248</td></tr>\n",
       "\t<tr><td>default-visual</td><td>bgwiki    </td><td> 34397</td><td> 50781</td></tr>\n",
       "\t<tr><td>default-source</td><td>cawiki    </td><td> 17405</td><td> 23624</td></tr>\n",
       "\t<tr><td>default-visual</td><td>cawiki    </td><td> 16698</td><td> 22615</td></tr>\n",
       "\t<tr><td>default-source</td><td>dawiki    </td><td> 40500</td><td> 51981</td></tr>\n",
       "\t<tr><td>default-visual</td><td>dawiki    </td><td> 38869</td><td> 49837</td></tr>\n",
       "\t<tr><td>default-source</td><td>elwiki    </td><td> 70126</td><td>104277</td></tr>\n",
       "\t<tr><td>default-visual</td><td>elwiki    </td><td> 66026</td><td> 98229</td></tr>\n",
       "\t<tr><td>default-source</td><td>etwiki    </td><td>  9316</td><td> 12977</td></tr>\n",
       "\t<tr><td>default-visual</td><td>etwiki    </td><td>  8893</td><td> 11862</td></tr>\n",
       "\t<tr><td>default-source</td><td>fiwiki    </td><td> 82345</td><td>120946</td></tr>\n",
       "\t<tr><td>default-visual</td><td>fiwiki    </td><td> 79193</td><td>115314</td></tr>\n",
       "\t<tr><td>default-source</td><td>hrwiki    </td><td> 42845</td><td> 58747</td></tr>\n",
       "\t<tr><td>default-visual</td><td>hrwiki    </td><td> 40293</td><td> 55096</td></tr>\n",
       "\t<tr><td>default-source</td><td>huwiki    </td><td> 96830</td><td>140007</td></tr>\n",
       "\t<tr><td>default-visual</td><td>huwiki    </td><td> 93882</td><td>135732</td></tr>\n",
       "\t<tr><td>default-source</td><td>mlwiki    </td><td> 42919</td><td> 64669</td></tr>\n",
       "\t<tr><td>default-visual</td><td>mlwiki    </td><td> 37837</td><td> 55544</td></tr>\n",
       "\t<tr><td>default-source</td><td>mswiki    </td><td> 71688</td><td> 97154</td></tr>\n",
       "\t<tr><td>default-visual</td><td>mswiki    </td><td> 65501</td><td> 88663</td></tr>\n",
       "\t<tr><td>default-source</td><td>nowiki    </td><td> 52394</td><td> 68146</td></tr>\n",
       "\t<tr><td>default-visual</td><td>nowiki    </td><td> 50291</td><td> 65356</td></tr>\n",
       "\t<tr><td>default-source</td><td>ptwiki    </td><td>691554</td><td>994634</td></tr>\n",
       "\t<tr><td>default-visual</td><td>ptwiki    </td><td>645544</td><td>936121</td></tr>\n",
       "\t<tr><td>default-source</td><td>rowiki    </td><td> 76775</td><td>106667</td></tr>\n",
       "\t<tr><td>default-visual</td><td>rowiki    </td><td> 72683</td><td>100104</td></tr>\n",
       "\t<tr><td>default-source</td><td>srwiki    </td><td> 51947</td><td> 78163</td></tr>\n",
       "\t<tr><td>default-visual</td><td>srwiki    </td><td> 48594</td><td> 70192</td></tr>\n",
       "\t<tr><td>default-source</td><td>svwiki    </td><td>125415</td><td>172066</td></tr>\n",
       "\t<tr><td>default-visual</td><td>svwiki    </td><td>122535</td><td>167446</td></tr>\n",
       "\t<tr><td>default-source</td><td>tawiki    </td><td> 82113</td><td>124439</td></tr>\n",
       "\t<tr><td>default-visual</td><td>tawiki    </td><td> 76366</td><td>112996</td></tr>\n",
       "\t<tr><td>default-source</td><td>thwiki    </td><td>250324</td><td>353956</td></tr>\n",
       "\t<tr><td>default-visual</td><td>thwiki    </td><td>226800</td><td>319519</td></tr>\n",
       "\t<tr><td>default-source</td><td>urwiki    </td><td> 10885</td><td> 16939</td></tr>\n",
       "\t<tr><td>default-visual</td><td>urwiki    </td><td>  9567</td><td> 14704</td></tr>\n",
       "\t<tr><td>default-source</td><td>zh_yuewiki</td><td> 18214</td><td> 22139</td></tr>\n",
       "\t<tr><td>default-visual</td><td>zh_yuewiki</td><td> 17209</td><td> 21222</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped_df: 40 × 4\n",
       "\\begin{tabular}{r|llll}\n",
       " bucket & wiki & users & attempts\\\\\n",
       " <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t default-source & azwiki     &  30959 &  45825\\\\\n",
       "\t default-visual & azwiki     &  27084 &  39485\\\\\n",
       "\t default-source & bgwiki     &  36032 &  54248\\\\\n",
       "\t default-visual & bgwiki     &  34397 &  50781\\\\\n",
       "\t default-source & cawiki     &  17405 &  23624\\\\\n",
       "\t default-visual & cawiki     &  16698 &  22615\\\\\n",
       "\t default-source & dawiki     &  40500 &  51981\\\\\n",
       "\t default-visual & dawiki     &  38869 &  49837\\\\\n",
       "\t default-source & elwiki     &  70126 & 104277\\\\\n",
       "\t default-visual & elwiki     &  66026 &  98229\\\\\n",
       "\t default-source & etwiki     &   9316 &  12977\\\\\n",
       "\t default-visual & etwiki     &   8893 &  11862\\\\\n",
       "\t default-source & fiwiki     &  82345 & 120946\\\\\n",
       "\t default-visual & fiwiki     &  79193 & 115314\\\\\n",
       "\t default-source & hrwiki     &  42845 &  58747\\\\\n",
       "\t default-visual & hrwiki     &  40293 &  55096\\\\\n",
       "\t default-source & huwiki     &  96830 & 140007\\\\\n",
       "\t default-visual & huwiki     &  93882 & 135732\\\\\n",
       "\t default-source & mlwiki     &  42919 &  64669\\\\\n",
       "\t default-visual & mlwiki     &  37837 &  55544\\\\\n",
       "\t default-source & mswiki     &  71688 &  97154\\\\\n",
       "\t default-visual & mswiki     &  65501 &  88663\\\\\n",
       "\t default-source & nowiki     &  52394 &  68146\\\\\n",
       "\t default-visual & nowiki     &  50291 &  65356\\\\\n",
       "\t default-source & ptwiki     & 691554 & 994634\\\\\n",
       "\t default-visual & ptwiki     & 645544 & 936121\\\\\n",
       "\t default-source & rowiki     &  76775 & 106667\\\\\n",
       "\t default-visual & rowiki     &  72683 & 100104\\\\\n",
       "\t default-source & srwiki     &  51947 &  78163\\\\\n",
       "\t default-visual & srwiki     &  48594 &  70192\\\\\n",
       "\t default-source & svwiki     & 125415 & 172066\\\\\n",
       "\t default-visual & svwiki     & 122535 & 167446\\\\\n",
       "\t default-source & tawiki     &  82113 & 124439\\\\\n",
       "\t default-visual & tawiki     &  76366 & 112996\\\\\n",
       "\t default-source & thwiki     & 250324 & 353956\\\\\n",
       "\t default-visual & thwiki     & 226800 & 319519\\\\\n",
       "\t default-source & urwiki     &  10885 &  16939\\\\\n",
       "\t default-visual & urwiki     &   9567 &  14704\\\\\n",
       "\t default-source & zh\\_yuewiki &  18214 &  22139\\\\\n",
       "\t default-visual & zh\\_yuewiki &  17209 &  21222\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 40 × 4\n",
       "\n",
       "| bucket &lt;chr&gt; | wiki &lt;chr&gt; | users &lt;int&gt; | attempts &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| default-source | azwiki     |  30959 |  45825 |\n",
       "| default-visual | azwiki     |  27084 |  39485 |\n",
       "| default-source | bgwiki     |  36032 |  54248 |\n",
       "| default-visual | bgwiki     |  34397 |  50781 |\n",
       "| default-source | cawiki     |  17405 |  23624 |\n",
       "| default-visual | cawiki     |  16698 |  22615 |\n",
       "| default-source | dawiki     |  40500 |  51981 |\n",
       "| default-visual | dawiki     |  38869 |  49837 |\n",
       "| default-source | elwiki     |  70126 | 104277 |\n",
       "| default-visual | elwiki     |  66026 |  98229 |\n",
       "| default-source | etwiki     |   9316 |  12977 |\n",
       "| default-visual | etwiki     |   8893 |  11862 |\n",
       "| default-source | fiwiki     |  82345 | 120946 |\n",
       "| default-visual | fiwiki     |  79193 | 115314 |\n",
       "| default-source | hrwiki     |  42845 |  58747 |\n",
       "| default-visual | hrwiki     |  40293 |  55096 |\n",
       "| default-source | huwiki     |  96830 | 140007 |\n",
       "| default-visual | huwiki     |  93882 | 135732 |\n",
       "| default-source | mlwiki     |  42919 |  64669 |\n",
       "| default-visual | mlwiki     |  37837 |  55544 |\n",
       "| default-source | mswiki     |  71688 |  97154 |\n",
       "| default-visual | mswiki     |  65501 |  88663 |\n",
       "| default-source | nowiki     |  52394 |  68146 |\n",
       "| default-visual | nowiki     |  50291 |  65356 |\n",
       "| default-source | ptwiki     | 691554 | 994634 |\n",
       "| default-visual | ptwiki     | 645544 | 936121 |\n",
       "| default-source | rowiki     |  76775 | 106667 |\n",
       "| default-visual | rowiki     |  72683 | 100104 |\n",
       "| default-source | srwiki     |  51947 |  78163 |\n",
       "| default-visual | srwiki     |  48594 |  70192 |\n",
       "| default-source | svwiki     | 125415 | 172066 |\n",
       "| default-visual | svwiki     | 122535 | 167446 |\n",
       "| default-source | tawiki     |  82113 | 124439 |\n",
       "| default-visual | tawiki     |  76366 | 112996 |\n",
       "| default-source | thwiki     | 250324 | 353956 |\n",
       "| default-visual | thwiki     | 226800 | 319519 |\n",
       "| default-source | urwiki     |  10885 |  16939 |\n",
       "| default-visual | urwiki     |   9567 |  14704 |\n",
       "| default-source | zh_yuewiki |  18214 |  22139 |\n",
       "| default-visual | zh_yuewiki |  17209 |  21222 |\n",
       "\n"
      ],
      "text/plain": [
       "   bucket         wiki       users  attempts\n",
       "1  default-source azwiki      30959  45825  \n",
       "2  default-visual azwiki      27084  39485  \n",
       "3  default-source bgwiki      36032  54248  \n",
       "4  default-visual bgwiki      34397  50781  \n",
       "5  default-source cawiki      17405  23624  \n",
       "6  default-visual cawiki      16698  22615  \n",
       "7  default-source dawiki      40500  51981  \n",
       "8  default-visual dawiki      38869  49837  \n",
       "9  default-source elwiki      70126 104277  \n",
       "10 default-visual elwiki      66026  98229  \n",
       "11 default-source etwiki       9316  12977  \n",
       "12 default-visual etwiki       8893  11862  \n",
       "13 default-source fiwiki      82345 120946  \n",
       "14 default-visual fiwiki      79193 115314  \n",
       "15 default-source hrwiki      42845  58747  \n",
       "16 default-visual hrwiki      40293  55096  \n",
       "17 default-source huwiki      96830 140007  \n",
       "18 default-visual huwiki      93882 135732  \n",
       "19 default-source mlwiki      42919  64669  \n",
       "20 default-visual mlwiki      37837  55544  \n",
       "21 default-source mswiki      71688  97154  \n",
       "22 default-visual mswiki      65501  88663  \n",
       "23 default-source nowiki      52394  68146  \n",
       "24 default-visual nowiki      50291  65356  \n",
       "25 default-source ptwiki     691554 994634  \n",
       "26 default-visual ptwiki     645544 936121  \n",
       "27 default-source rowiki      76775 106667  \n",
       "28 default-visual rowiki      72683 100104  \n",
       "29 default-source srwiki      51947  78163  \n",
       "30 default-visual srwiki      48594  70192  \n",
       "31 default-source svwiki     125415 172066  \n",
       "32 default-visual svwiki     122535 167446  \n",
       "33 default-source tawiki      82113 124439  \n",
       "34 default-visual tawiki      76366 112996  \n",
       "35 default-source thwiki     250324 353956  \n",
       "36 default-visual thwiki     226800 319519  \n",
       "37 default-source urwiki      10885  16939  \n",
       "38 default-visual urwiki       9567  14704  \n",
       "39 default-source zh_yuewiki  18214  22139  \n",
       "40 default-visual zh_yuewiki  17209  21222  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "sessions_bywiki <- sessions %>%\n",
    "  group_by(bucket, wiki) %>%\n",
    "  summarise(users = n_distinct(user_id),\n",
    "        attempts = n_distinct(edit_attempt_id)) %>%\n",
    "  arrange(wiki) \n",
    "\n",
    "sessions_bywiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "sessions_bywiki$users <- as.numeric(sessions_bywiki$users)\n",
    "sessions_bywiki$attempts <- as.numeric(sessions_bywiki$attempts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "sessions_bywiki$bucket[sessions_bywiki$bucket == \"default-source\"] <- \"default_source\"\n",
    "sessions_bywiki$bucket[sessions_bywiki$bucket == \"default-visual\"] <- \"default_visual\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 6 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>wiki</th><th scope=col>default_source</th><th scope=col>default_visual</th><th scope=col>percent_wikitext_users</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>azwiki</td><td> 30959</td><td> 27084</td><td>53.33804</td></tr>\n",
       "\t<tr><td>urwiki</td><td> 10885</td><td>  9567</td><td>53.22218</td></tr>\n",
       "\t<tr><td>mlwiki</td><td> 42919</td><td> 37837</td><td>53.14652</td></tr>\n",
       "\t<tr><td>thwiki</td><td>250324</td><td>226800</td><td>52.46519</td></tr>\n",
       "\t<tr><td>mswiki</td><td> 71688</td><td> 65501</td><td>52.25492</td></tr>\n",
       "\t<tr><td>tawiki</td><td> 82113</td><td> 76366</td><td>51.81317</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 6 × 4\n",
       "\\begin{tabular}{r|llll}\n",
       " wiki & default\\_source & default\\_visual & percent\\_wikitext\\_users\\\\\n",
       " <chr> & <dbl> & <dbl> & <dbl>\\\\\n",
       "\\hline\n",
       "\t azwiki &  30959 &  27084 & 53.33804\\\\\n",
       "\t urwiki &  10885 &   9567 & 53.22218\\\\\n",
       "\t mlwiki &  42919 &  37837 & 53.14652\\\\\n",
       "\t thwiki & 250324 & 226800 & 52.46519\\\\\n",
       "\t mswiki &  71688 &  65501 & 52.25492\\\\\n",
       "\t tawiki &  82113 &  76366 & 51.81317\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 6 × 4\n",
       "\n",
       "| wiki &lt;chr&gt; | default_source &lt;dbl&gt; | default_visual &lt;dbl&gt; | percent_wikitext_users &lt;dbl&gt; |\n",
       "|---|---|---|---|\n",
       "| azwiki |  30959 |  27084 | 53.33804 |\n",
       "| urwiki |  10885 |   9567 | 53.22218 |\n",
       "| mlwiki |  42919 |  37837 | 53.14652 |\n",
       "| thwiki | 250324 | 226800 | 52.46519 |\n",
       "| mswiki |  71688 |  65501 | 52.25492 |\n",
       "| tawiki |  82113 |  76366 | 51.81317 |\n",
       "\n"
      ],
      "text/plain": [
       "  wiki   default_source default_visual percent_wikitext_users\n",
       "1 azwiki  30959          27084         53.33804              \n",
       "2 urwiki  10885           9567         53.22218              \n",
       "3 mlwiki  42919          37837         53.14652              \n",
       "4 thwiki 250324         226800         52.46519              \n",
       "5 mswiki  71688          65501         52.25492              \n",
       "6 tawiki  82113          76366         51.81317              "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sessions_bywiki_percent_imbalance <- sessions_bywiki %>%\n",
    " select(-4)  %>% ##look at user counts only\n",
    " spread(bucket,users) %>% \n",
    " #mutate(percent_diff = round(abs((default_source-default_visual)/default_visual *100), 3)) %>%\n",
    " mutate(percent_wikitext_users = default_source/(default_source + default_visual) *100) %>%\n",
    " arrange(desc(percent_wikitext_users)) \n",
    "\n",
    "head(sessions_bywiki_percent_imbalance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Unable to isolate the imblance to a particular wiki. Similar imblance trends seen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check if imbalance is present for a particular action\n",
    "\n",
    "Based on earlier result, we should see an imblance across all actions as well but just want to confirm.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 16 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>bucket</th><th scope=col>action</th><th scope=col>users</th><th scope=col>attempts</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>default-source</td><td>abort      </td><td>1387151</td><td>1763603</td></tr>\n",
       "\t<tr><td>default-visual</td><td>abort      </td><td> 835965</td><td> 998167</td></tr>\n",
       "\t<tr><td>default-source</td><td>init       </td><td>1906751</td><td>2720469</td></tr>\n",
       "\t<tr><td>default-visual</td><td>init       </td><td>1784041</td><td>2539217</td></tr>\n",
       "\t<tr><td>default-source</td><td>loaded     </td><td>1903903</td><td>2708243</td></tr>\n",
       "\t<tr><td>default-visual</td><td>loaded     </td><td>1256521</td><td>1708057</td></tr>\n",
       "\t<tr><td>default-source</td><td>ready      </td><td>1904072</td><td>2708547</td></tr>\n",
       "\t<tr><td>default-visual</td><td>ready      </td><td>1256752</td><td>1708386</td></tr>\n",
       "\t<tr><td>default-source</td><td>saveAttempt</td><td>  32690</td><td>  88018</td></tr>\n",
       "\t<tr><td>default-visual</td><td>saveAttempt</td><td>  27793</td><td>  73583</td></tr>\n",
       "\t<tr><td>default-source</td><td>saveFailure</td><td>   8501</td><td>  12529</td></tr>\n",
       "\t<tr><td>default-visual</td><td>saveFailure</td><td>   7635</td><td>  11097</td></tr>\n",
       "\t<tr><td>default-source</td><td>saveIntent </td><td>  37941</td><td>  98970</td></tr>\n",
       "\t<tr><td>default-visual</td><td>saveIntent </td><td>  31076</td><td>  79918</td></tr>\n",
       "\t<tr><td>default-source</td><td>saveSuccess</td><td>  28372</td><td>  78882</td></tr>\n",
       "\t<tr><td>default-visual</td><td>saveSuccess</td><td>  24132</td><td>  66269</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped_df: 16 × 4\n",
       "\\begin{tabular}{r|llll}\n",
       " bucket & action & users & attempts\\\\\n",
       " <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t default-source & abort       & 1387151 & 1763603\\\\\n",
       "\t default-visual & abort       &  835965 &  998167\\\\\n",
       "\t default-source & init        & 1906751 & 2720469\\\\\n",
       "\t default-visual & init        & 1784041 & 2539217\\\\\n",
       "\t default-source & loaded      & 1903903 & 2708243\\\\\n",
       "\t default-visual & loaded      & 1256521 & 1708057\\\\\n",
       "\t default-source & ready       & 1904072 & 2708547\\\\\n",
       "\t default-visual & ready       & 1256752 & 1708386\\\\\n",
       "\t default-source & saveAttempt &   32690 &   88018\\\\\n",
       "\t default-visual & saveAttempt &   27793 &   73583\\\\\n",
       "\t default-source & saveFailure &    8501 &   12529\\\\\n",
       "\t default-visual & saveFailure &    7635 &   11097\\\\\n",
       "\t default-source & saveIntent  &   37941 &   98970\\\\\n",
       "\t default-visual & saveIntent  &   31076 &   79918\\\\\n",
       "\t default-source & saveSuccess &   28372 &   78882\\\\\n",
       "\t default-visual & saveSuccess &   24132 &   66269\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 16 × 4\n",
       "\n",
       "| bucket &lt;chr&gt; | action &lt;chr&gt; | users &lt;int&gt; | attempts &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| default-source | abort       | 1387151 | 1763603 |\n",
       "| default-visual | abort       |  835965 |  998167 |\n",
       "| default-source | init        | 1906751 | 2720469 |\n",
       "| default-visual | init        | 1784041 | 2539217 |\n",
       "| default-source | loaded      | 1903903 | 2708243 |\n",
       "| default-visual | loaded      | 1256521 | 1708057 |\n",
       "| default-source | ready       | 1904072 | 2708547 |\n",
       "| default-visual | ready       | 1256752 | 1708386 |\n",
       "| default-source | saveAttempt |   32690 |   88018 |\n",
       "| default-visual | saveAttempt |   27793 |   73583 |\n",
       "| default-source | saveFailure |    8501 |   12529 |\n",
       "| default-visual | saveFailure |    7635 |   11097 |\n",
       "| default-source | saveIntent  |   37941 |   98970 |\n",
       "| default-visual | saveIntent  |   31076 |   79918 |\n",
       "| default-source | saveSuccess |   28372 |   78882 |\n",
       "| default-visual | saveSuccess |   24132 |   66269 |\n",
       "\n"
      ],
      "text/plain": [
       "   bucket         action      users   attempts\n",
       "1  default-source abort       1387151 1763603 \n",
       "2  default-visual abort        835965  998167 \n",
       "3  default-source init        1906751 2720469 \n",
       "4  default-visual init        1784041 2539217 \n",
       "5  default-source loaded      1903903 2708243 \n",
       "6  default-visual loaded      1256521 1708057 \n",
       "7  default-source ready       1904072 2708547 \n",
       "8  default-visual ready       1256752 1708386 \n",
       "9  default-source saveAttempt   32690   88018 \n",
       "10 default-visual saveAttempt   27793   73583 \n",
       "11 default-source saveFailure    8501   12529 \n",
       "12 default-visual saveFailure    7635   11097 \n",
       "13 default-source saveIntent    37941   98970 \n",
       "14 default-visual saveIntent    31076   79918 \n",
       "15 default-source saveSuccess   28372   78882 \n",
       "16 default-visual saveSuccess   24132   66269 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sessions_byaction <- sessions %>%\n",
    "  group_by(bucket, action) %>%\n",
    "  summarise(users = n_distinct(user_id),\n",
    "        attempts = n_distinct(edit_attempt_id)) %>%\n",
    "  arrange(action) \n",
    "\n",
    "sessions_byaction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirmed. There is an imbalance across all actions as well ranging from about 51% to 54%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of Initial Findings\n",
    "\n",
    "1. Unable to isolate the imbalance to one of the following subgroups  (user group (registered or anonymous), country, wiki or event action). Across all these groups, there were a similar imbalance between the wikitext and visual editor buckets  with higher number of users (around 51 to 54%) being placed in the wikitext bucket. \n",
    "2. These was also a similar imbalance across all of the major browsers; however, there seems to be a signficant difference between the wikitext and visual editor buckets for Bingbot and BingPreview browsers. For Bingbot, about 98% of 384 users were placed in the wikitext ad default bucket. These are both web crawling bot browsers. This imbalance wouldn't account for the total imbalance we are seeing in overall users but it might be a clue into where the issue is happening. \n",
    "3.   Since we're seeing the same imbalance towards the wikitext buckets across these various dimensions, it seems likely that this might be either a load time issue or an issue with the method of bucket assigment. \n",
    "\n",
    "Potential areas to investigate further: load time issue?\n",
    "Can we confirm that both buckets assigned on the server side?  Are they assigned at the same time an init event is recorded on the server? If for some reason, wikitext bucket is assigned on the server side and visual editor on the client side, than that delay might be why we are seeing fewing people in the visual editor buckets.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}