{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Post Deployment Language Switching QA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Task](https://phabricator.wikimedia.org/T275762)\n",
    "|\n",
    "[Schema](https://gerrit.wikimedia.org/r/plugins/gitiles/schemas/event/secondary/+/refs/heads/master/jsonschema/analytics/legacy/universallanguageselector/current.yaml)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contents\n",
    "1. [Language Links in the Sidebar](#Language-Links-in-the-Sidebar)\n",
    "2. [Language Switcher Button](#Language-Switcher-Button)\n",
    "3. [Language Button Context Events](#Language-Button-Context-Events)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "shhh <- function(expr) suppressPackageStartupMessages(suppressWarnings(suppressMessages(expr)))\n",
    "shhh({\n",
    "    library(tidyverse); \n",
    "    library(lubridate); \n",
    "    library(scales);\n",
    "    library(magrittr); \n",
    "    library(dplyr)\n",
    "})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language Links in the Siderbar\n",
    "\n",
    "[Instrumentation Task](https://phabricator.wikimedia.org/T275762)\n",
    "\n",
    "Instrumentation Notes:\n",
    "- Instrumentation limited to legacy sidebar in modern Vector.\n",
    "- Logged as `event.context = 'languages-list'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "query <- \n",
    "\"\n",
    "SELECT\n",
    "    TO_DATE(dt) AS `date`,\n",
    "    wiki,\n",
    "    event.web_session_id,\n",
    "    event.usereditbucket,\n",
    "    event.timetochangelanguage,\n",
    "    event.isanon,\n",
    "    event.interfacelanguage,\n",
    "    event.contentlanguage,\n",
    "    event.selectedinterfacelanguage,\n",
    "    Count(*) AS n_events\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND ((Month = 04 AND DAY > 26) OR (MONTH = 05))\n",
    "    AND event.context = 'languages-list'\n",
    "GROUP BY\n",
    "    TO_DATE(dt),\n",
    "    wiki,\n",
    "    event.web_session_id,\n",
    "    event.usereditbucket,\n",
    "    event.timetochangelanguage,\n",
    "    event.isanon,\n",
    "    event.interfacelanguage,\n",
    "    event.contentlanguage,\n",
    "    event.selectedinterfacelanguage\n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lang_link_events <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang_link_events$date <- as.Date(lang_link_events$date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Daily Langauge Link Events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 16 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>date</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;date&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>2021-04-28</td><td>   791</td><td>   636</td></tr>\n",
       "\t<tr><td>2021-04-29</td><td> 46934</td><td> 33173</td></tr>\n",
       "\t<tr><td>2021-04-30</td><td>274519</td><td>177001</td></tr>\n",
       "\t<tr><td>2021-05-01</td><td>243709</td><td>151370</td></tr>\n",
       "\t<tr><td>2021-05-02</td><td>258630</td><td>161650</td></tr>\n",
       "\t<tr><td>2021-05-03</td><td>303583</td><td>195465</td></tr>\n",
       "\t<tr><td>2021-05-04</td><td>300642</td><td>194931</td></tr>\n",
       "\t<tr><td>2021-05-05</td><td>295520</td><td>191876</td></tr>\n",
       "\t<tr><td>2021-05-06</td><td>291788</td><td>190268</td></tr>\n",
       "\t<tr><td>2021-05-07</td><td>265159</td><td>171239</td></tr>\n",
       "\t<tr><td>2021-05-08</td><td>225456</td><td>141143</td></tr>\n",
       "\t<tr><td>2021-05-09</td><td>244182</td><td>152870</td></tr>\n",
       "\t<tr><td>2021-05-10</td><td>301699</td><td>196353</td></tr>\n",
       "\t<tr><td>2021-05-11</td><td>296753</td><td>193206</td></tr>\n",
       "\t<tr><td>2021-05-12</td><td>278392</td><td>181382</td></tr>\n",
       "\t<tr><td>2021-05-13</td><td> 54123</td><td> 37258</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 16 × 3\n",
       "\\begin{tabular}{lll}\n",
       " date & n\\_events & n\\_sessions\\\\\n",
       " <date> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t 2021-04-28 &    791 &    636\\\\\n",
       "\t 2021-04-29 &  46934 &  33173\\\\\n",
       "\t 2021-04-30 & 274519 & 177001\\\\\n",
       "\t 2021-05-01 & 243709 & 151370\\\\\n",
       "\t 2021-05-02 & 258630 & 161650\\\\\n",
       "\t 2021-05-03 & 303583 & 195465\\\\\n",
       "\t 2021-05-04 & 300642 & 194931\\\\\n",
       "\t 2021-05-05 & 295520 & 191876\\\\\n",
       "\t 2021-05-06 & 291788 & 190268\\\\\n",
       "\t 2021-05-07 & 265159 & 171239\\\\\n",
       "\t 2021-05-08 & 225456 & 141143\\\\\n",
       "\t 2021-05-09 & 244182 & 152870\\\\\n",
       "\t 2021-05-10 & 301699 & 196353\\\\\n",
       "\t 2021-05-11 & 296753 & 193206\\\\\n",
       "\t 2021-05-12 & 278392 & 181382\\\\\n",
       "\t 2021-05-13 &  54123 &  37258\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 16 × 3\n",
       "\n",
       "| date &lt;date&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| 2021-04-28 |    791 |    636 |\n",
       "| 2021-04-29 |  46934 |  33173 |\n",
       "| 2021-04-30 | 274519 | 177001 |\n",
       "| 2021-05-01 | 243709 | 151370 |\n",
       "| 2021-05-02 | 258630 | 161650 |\n",
       "| 2021-05-03 | 303583 | 195465 |\n",
       "| 2021-05-04 | 300642 | 194931 |\n",
       "| 2021-05-05 | 295520 | 191876 |\n",
       "| 2021-05-06 | 291788 | 190268 |\n",
       "| 2021-05-07 | 265159 | 171239 |\n",
       "| 2021-05-08 | 225456 | 141143 |\n",
       "| 2021-05-09 | 244182 | 152870 |\n",
       "| 2021-05-10 | 301699 | 196353 |\n",
       "| 2021-05-11 | 296753 | 193206 |\n",
       "| 2021-05-12 | 278392 | 181382 |\n",
       "| 2021-05-13 |  54123 |  37258 |\n",
       "\n"
      ],
      "text/plain": [
       "   date       n_events n_sessions\n",
       "1  2021-04-28    791      636    \n",
       "2  2021-04-29  46934    33173    \n",
       "3  2021-04-30 274519   177001    \n",
       "4  2021-05-01 243709   151370    \n",
       "5  2021-05-02 258630   161650    \n",
       "6  2021-05-03 303583   195465    \n",
       "7  2021-05-04 300642   194931    \n",
       "8  2021-05-05 295520   191876    \n",
       "9  2021-05-06 291788   190268    \n",
       "10 2021-05-07 265159   171239    \n",
       "11 2021-05-08 225456   141143    \n",
       "12 2021-05-09 244182   152870    \n",
       "13 2021-05-10 301699   196353    \n",
       "14 2021-05-11 296753   193206    \n",
       "15 2021-05-12 278392   181382    \n",
       "16 2021-05-13  54123    37258    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_link_events_daily <- lang_link_events %>%\n",
    "    group_by(date) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_link_events_daily"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start recording events on 28 April 2021. There are an average 176,827 sessions per day including sessions by both logged in and logged out users. No unexpected spikes or drops so far."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clicks per session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Check to make sure there are duplicate session id. Some sessions should have more than one click event."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "FALSE"
      ],
      "text/latex": [
       "FALSE"
      ],
      "text/markdown": [
       "FALSE"
      ],
      "text/plain": [
       "[1] FALSE"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "length(unique(lang_link_events$web_session_id)) == nrow(lang_link_events)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>isanon</th><th scope=col>avg_clicks</th><th scope=col>max_clicks</th><th scope=col>min_clciks</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>false</td><td>1.000327</td><td>2</td><td>1</td></tr>\n",
       "\t<tr><td>true </td><td>1.000169</td><td>4</td><td>1</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 4\n",
       "\\begin{tabular}{llll}\n",
       " isanon & avg\\_clicks & max\\_clicks & min\\_clciks\\\\\n",
       " <chr> & <dbl> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t false & 1.000327 & 2 & 1\\\\\n",
       "\t true  & 1.000169 & 4 & 1\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 4\n",
       "\n",
       "| isanon &lt;chr&gt; | avg_clicks &lt;dbl&gt; | max_clicks &lt;int&gt; | min_clciks &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| false | 1.000327 | 2 | 1 |\n",
       "| true  | 1.000169 | 4 | 1 |\n",
       "\n"
      ],
      "text/plain": [
       "  isanon avg_clicks max_clicks min_clciks\n",
       "1 false  1.000327   2          1         \n",
       "2 true   1.000169   4          1         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_link_events_persession <- lang_link_events %>%\n",
    "    group_by(isanon) %>%\n",
    "    summarize(avg_clicks = mean(n_events),\n",
    "              max_clicks = max(n_events),\n",
    "              min_clciks = min(n_events))\n",
    "\n",
    "lang_link_events_persession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By Logged In Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>isanon</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>false</td><td> 113161</td><td>  46278</td></tr>\n",
       "\t<tr><td>true </td><td>3568719</td><td>2058676</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 3\n",
       "\\begin{tabular}{lll}\n",
       " isanon & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t false &  113161 &   46278\\\\\n",
       "\t true  & 3568719 & 2058676\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 3\n",
       "\n",
       "| isanon &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| false |  113161 |   46278 |\n",
       "| true  | 3568719 | 2058676 |\n",
       "\n"
      ],
      "text/plain": [
       "  isanon n_events n_sessions\n",
       "1 false   113161    46278   \n",
       "2 true   3568719  2058676   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_link_events_isanon <- lang_link_events %>%\n",
    "    group_by(isanon) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_link_events_isanon "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "97.8% of all sessions with clicks to the language links are by logged out users. That's high but expected because instrumentation was limited to legacy sidebar in modern Vector (not from legacy or other skins such as timeless). The new language switching functionality was made available to all logged-in users opted into the latest version of the Vector skin.\n",
    "\n",
    "Legacy sidebar in modern Vector would mostly appear to logged-out users on test wikis where Vector is deployed as default.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By Test Wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 13 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>wiki</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>bnwiki       </td><td>   8451</td><td>   5726</td></tr>\n",
       "\t<tr><td>dewikivoyage </td><td>    755</td><td>    604</td></tr>\n",
       "\t<tr><td>euwiki       </td><td>  31609</td><td>  22351</td></tr>\n",
       "\t<tr><td>fawiki       </td><td> 200781</td><td> 110092</td></tr>\n",
       "\t<tr><td>frwiki       </td><td>1933743</td><td>1091442</td></tr>\n",
       "\t<tr><td>frwiktionary </td><td>  21307</td><td>  13350</td></tr>\n",
       "\t<tr><td>hewiki       </td><td> 173526</td><td> 100165</td></tr>\n",
       "\t<tr><td>kowiki       </td><td> 146461</td><td>  81522</td></tr>\n",
       "\t<tr><td>ptwiki       </td><td> 555228</td><td> 328675</td></tr>\n",
       "\t<tr><td>ptwikiversity</td><td>     15</td><td>     13</td></tr>\n",
       "\t<tr><td>srwiki       </td><td>  98359</td><td>  56521</td></tr>\n",
       "\t<tr><td>trwiki       </td><td> 256347</td><td> 151107</td></tr>\n",
       "\t<tr><td>vecwiki      </td><td>   1348</td><td>   1133</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 13 × 3\n",
       "\\begin{tabular}{lll}\n",
       " wiki & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t bnwiki        &    8451 &    5726\\\\\n",
       "\t dewikivoyage  &     755 &     604\\\\\n",
       "\t euwiki        &   31609 &   22351\\\\\n",
       "\t fawiki        &  200781 &  110092\\\\\n",
       "\t frwiki        & 1933743 & 1091442\\\\\n",
       "\t frwiktionary  &   21307 &   13350\\\\\n",
       "\t hewiki        &  173526 &  100165\\\\\n",
       "\t kowiki        &  146461 &   81522\\\\\n",
       "\t ptwiki        &  555228 &  328675\\\\\n",
       "\t ptwikiversity &      15 &      13\\\\\n",
       "\t srwiki        &   98359 &   56521\\\\\n",
       "\t trwiki        &  256347 &  151107\\\\\n",
       "\t vecwiki       &    1348 &    1133\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 13 × 3\n",
       "\n",
       "| wiki &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| bnwiki        |    8451 |    5726 |\n",
       "| dewikivoyage  |     755 |     604 |\n",
       "| euwiki        |   31609 |   22351 |\n",
       "| fawiki        |  200781 |  110092 |\n",
       "| frwiki        | 1933743 | 1091442 |\n",
       "| frwiktionary  |   21307 |   13350 |\n",
       "| hewiki        |  173526 |  100165 |\n",
       "| kowiki        |  146461 |   81522 |\n",
       "| ptwiki        |  555228 |  328675 |\n",
       "| ptwikiversity |      15 |      13 |\n",
       "| srwiki        |   98359 |   56521 |\n",
       "| trwiki        |  256347 |  151107 |\n",
       "| vecwiki       |    1348 |    1133 |\n",
       "\n"
      ],
      "text/plain": [
       "   wiki          n_events n_sessions\n",
       "1  bnwiki           8451     5726   \n",
       "2  dewikivoyage      755      604   \n",
       "3  euwiki          31609    22351   \n",
       "4  fawiki         200781   110092   \n",
       "5  frwiki        1933743  1091442   \n",
       "6  frwiktionary    21307    13350   \n",
       "7  hewiki         173526   100165   \n",
       "8  kowiki         146461    81522   \n",
       "9  ptwiki         555228   328675   \n",
       "10 ptwikiversity      15       13   \n",
       "11 srwiki          98359    56521   \n",
       "12 trwiki         256347   151107   \n",
       "13 vecwiki          1348     1133   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# events and sessions that include link to language link by test wiki\n",
    "lang_link_events_testwiki <- lang_link_events %>%\n",
    "    filter(wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' )) %>%\n",
    "    group_by(wiki) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_link_events_testwiki\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'istestwiki' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 4 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>istestwiki</th><th scope=col>isanon</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>non_test_wiki</td><td>false</td><td>   3540</td><td>   1422</td></tr>\n",
       "\t<tr><td>non_test_wiki</td><td>true </td><td>    147</td><td>     80</td></tr>\n",
       "\t<tr><td>test_wiki    </td><td>false</td><td> 102371</td><td>  41836</td></tr>\n",
       "\t<tr><td>test_wiki    </td><td>true </td><td>3325559</td><td>1922334</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 4 × 4\n",
       "\\begin{tabular}{llll}\n",
       " istestwiki & isanon & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t non\\_test\\_wiki & false &    3540 &    1422\\\\\n",
       "\t non\\_test\\_wiki & true  &     147 &      80\\\\\n",
       "\t test\\_wiki     & false &  102371 &   41836\\\\\n",
       "\t test\\_wiki     & true  & 3325559 & 1922334\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 4 × 4\n",
       "\n",
       "| istestwiki &lt;chr&gt; | isanon &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| non_test_wiki | false |    3540 |    1422 |\n",
       "| non_test_wiki | true  |     147 |      80 |\n",
       "| test_wiki     | false |  102371 |   41836 |\n",
       "| test_wiki     | true  | 3325559 | 1922334 |\n",
       "\n"
      ],
      "text/plain": [
       "  istestwiki    isanon n_events n_sessions\n",
       "1 non_test_wiki false     3540     1422   \n",
       "2 non_test_wiki true       147       80   \n",
       "3 test_wiki     false   102371    41836   \n",
       "4 test_wiki     true   3325559  1922334   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# events and sessions that include link to language link by test wiki category and logged-in status\n",
    "lang_link_events_testwiki_isanon <- lang_link_events %>%\n",
    "    mutate(istestwiki = ifelse(wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' ), 'test_wiki', 'non_test_wiki')) %>%\n",
    "    group_by(istestwiki, isanon) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_link_events_testwiki_isanon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost all of the events recorded to date (99%) have been on test wikis. This is expected as the new language switcher button was deployed to all users opt'd in to the modern vector on all non test wikis. Users with modern vector on the test wikis have not been shown the new language switcher and still shown the language links in the sidebar.\n",
    "\n",
    "On non test wikis, the majority (94.67%) of sessions with clicks to the language list on modern vector come from logged-in users. Need to confirm if it's possible to have language link in sidebar if you are logged-in, on modern vector and on a non test wiki. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By User Edit Bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 6 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>usereditbucket</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>0 edits      </td><td>20092</td><td>10961</td></tr>\n",
       "\t<tr><td>1-4 edits    </td><td>11510</td><td> 5643</td></tr>\n",
       "\t<tr><td>100-999 edits</td><td>17264</td><td> 6533</td></tr>\n",
       "\t<tr><td>1000+ edits  </td><td>31482</td><td> 8932</td></tr>\n",
       "\t<tr><td>5-99 edits   </td><td>25556</td><td>11332</td></tr>\n",
       "\t<tr><td>NULL         </td><td>    7</td><td>    5</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 6 × 3\n",
       "\\begin{tabular}{lll}\n",
       " usereditbucket & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t 0 edits       & 20092 & 10961\\\\\n",
       "\t 1-4 edits     & 11510 &  5643\\\\\n",
       "\t 100-999 edits & 17264 &  6533\\\\\n",
       "\t 1000+ edits   & 31482 &  8932\\\\\n",
       "\t 5-99 edits    & 25556 & 11332\\\\\n",
       "\t NULL          &     7 &     5\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 6 × 3\n",
       "\n",
       "| usereditbucket &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| 0 edits       | 20092 | 10961 |\n",
       "| 1-4 edits     | 11510 |  5643 |\n",
       "| 100-999 edits | 17264 |  6533 |\n",
       "| 1000+ edits   | 31482 |  8932 |\n",
       "| 5-99 edits    | 25556 | 11332 |\n",
       "| NULL          |     7 |     5 |\n",
       "\n"
      ],
      "text/plain": [
       "  usereditbucket n_events n_sessions\n",
       "1 0 edits        20092    10961     \n",
       "2 1-4 edits      11510     5643     \n",
       "3 100-999 edits  17264     6533     \n",
       "4 1000+ edits    31482     8932     \n",
       "5 5-99 edits     25556    11332     \n",
       "6 NULL               7        5     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logged_in_editcount <- lang_link_events %>%\n",
    "    filter(isanon == 'false') %>%\n",
    "    group_by(usereditbucket) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "logged_in_editcount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>usereditbucket</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>5-99 edits</td><td>      1</td><td>      1</td></tr>\n",
       "\t<tr><td>NULL      </td><td>3325705</td><td>1922412</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 3\n",
       "\\begin{tabular}{lll}\n",
       " usereditbucket & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t 5-99 edits &       1 &       1\\\\\n",
       "\t NULL       & 3325705 & 1922412\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 3\n",
       "\n",
       "| usereditbucket &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| 5-99 edits |       1 |       1 |\n",
       "| NULL       | 3325705 | 1922412 |\n",
       "\n"
      ],
      "text/plain": [
       "  usereditbucket n_events n_sessions\n",
       "1 5-99 edits           1        1   \n",
       "2 NULL           3325705  1922412   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logged_out_editcount <- lang_link_events %>%\n",
    "    filter(isanon == 'true') %>%\n",
    "    group_by(usereditbucket) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "logged_out_editcount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are just a few instances (under 0.01%) of the event.usereditbucket field being populated for logged out users and recorded as NULL for logged-in users. Further investigation might be needed; however, the numbers of these events is not high enough to skew the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By Final Language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'selectedinterfacelanguage' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>selectedinterfacelanguage</th><th scope=col>n_sessions</th><th scope=col>pct_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>en</td><td>1443105</td><td>0.73470332</td></tr>\n",
       "\t<tr><td>es</td><td> 128358</td><td>0.06534871</td></tr>\n",
       "\t<tr><td>de</td><td> 125061</td><td>0.06367016</td></tr>\n",
       "\t<tr><td>it</td><td>  65010</td><td>0.03309743</td></tr>\n",
       "\t<tr><td>ar</td><td>  64408</td><td>0.03279094</td></tr>\n",
       "\t<tr><td>ru</td><td>  52428</td><td>0.02669177</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 6 × 3\n",
       "\\begin{tabular}{lll}\n",
       " selectedinterfacelanguage & n\\_sessions & pct\\_sessions\\\\\n",
       " <chr> & <int> & <dbl>\\\\\n",
       "\\hline\n",
       "\t en & 1443105 & 0.73470332\\\\\n",
       "\t es &  128358 & 0.06534871\\\\\n",
       "\t de &  125061 & 0.06367016\\\\\n",
       "\t it &   65010 & 0.03309743\\\\\n",
       "\t ar &   64408 & 0.03279094\\\\\n",
       "\t ru &   52428 & 0.02669177\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 3\n",
       "\n",
       "| selectedinterfacelanguage &lt;chr&gt; | n_sessions &lt;int&gt; | pct_sessions &lt;dbl&gt; |\n",
       "|---|---|---|\n",
       "| en | 1443105 | 0.73470332 |\n",
       "| es |  128358 | 0.06534871 |\n",
       "| de |  125061 | 0.06367016 |\n",
       "| it |   65010 | 0.03309743 |\n",
       "| ar |   64408 | 0.03279094 |\n",
       "| ru |   52428 | 0.02669177 |\n",
       "\n"
      ],
      "text/plain": [
       "  selectedinterfacelanguage n_sessions pct_sessions\n",
       "1 en                        1443105    0.73470332  \n",
       "2 es                         128358    0.06534871  \n",
       "3 de                         125061    0.06367016  \n",
       "4 it                          65010    0.03309743  \n",
       "5 ar                          64408    0.03279094  \n",
       "6 ru                          52428    0.02669177  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# test that you can switch from one language to the next.\n",
    "top_final_languages <- lang_link_events %>%\n",
    "    mutate(all_sessions = n_distinct(web_session_id)) %>%\n",
    "    group_by(selectedinterfacelanguage) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id),\n",
    "             pct_sessions = n_sessions/all_sessions) %>%\n",
    "    distinct() %>%\n",
    "    arrange(desc(n_sessions))\n",
    "\n",
    "head(top_final_languages )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most frequent language switches are to english (73% of sessions) followed by spanish (6.5%), and german (6.3%)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By Initial Language "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'interfacelanguage', 'contentlanguage' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>interfacelanguage</th><th scope=col>contentlanguage</th><th scope=col>n_sessions</th><th scope=col>pct_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>fr</td><td>fr</td><td>1103742</td><td>0.56192925</td></tr>\n",
       "\t<tr><td>pt</td><td>pt</td><td> 328012</td><td>0.16699513</td></tr>\n",
       "\t<tr><td>tr</td><td>tr</td><td> 150977</td><td>0.07686433</td></tr>\n",
       "\t<tr><td>fa</td><td>fa</td><td> 109989</td><td>0.05599681</td></tr>\n",
       "\t<tr><td>he</td><td>he</td><td> 100125</td><td>0.05097493</td></tr>\n",
       "\t<tr><td>ko</td><td>ko</td><td>  81429</td><td>0.04145655</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 6 × 4\n",
       "\\begin{tabular}{llll}\n",
       " interfacelanguage & contentlanguage & n\\_sessions & pct\\_sessions\\\\\n",
       " <chr> & <chr> & <int> & <dbl>\\\\\n",
       "\\hline\n",
       "\t fr & fr & 1103742 & 0.56192925\\\\\n",
       "\t pt & pt &  328012 & 0.16699513\\\\\n",
       "\t tr & tr &  150977 & 0.07686433\\\\\n",
       "\t fa & fa &  109989 & 0.05599681\\\\\n",
       "\t he & he &  100125 & 0.05097493\\\\\n",
       "\t ko & ko &   81429 & 0.04145655\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 4\n",
       "\n",
       "| interfacelanguage &lt;chr&gt; | contentlanguage &lt;chr&gt; | n_sessions &lt;int&gt; | pct_sessions &lt;dbl&gt; |\n",
       "|---|---|---|---|\n",
       "| fr | fr | 1103742 | 0.56192925 |\n",
       "| pt | pt |  328012 | 0.16699513 |\n",
       "| tr | tr |  150977 | 0.07686433 |\n",
       "| fa | fa |  109989 | 0.05599681 |\n",
       "| he | he |  100125 | 0.05097493 |\n",
       "| ko | ko |   81429 | 0.04145655 |\n",
       "\n"
      ],
      "text/plain": [
       "  interfacelanguage contentlanguage n_sessions pct_sessions\n",
       "1 fr                fr              1103742    0.56192925  \n",
       "2 pt                pt               328012    0.16699513  \n",
       "3 tr                tr               150977    0.07686433  \n",
       "4 fa                fa               109989    0.05599681  \n",
       "5 he                he               100125    0.05097493  \n",
       "6 ko                ko                81429    0.04145655  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "top_initial_languages <- lang_link_events %>%\n",
    "    mutate(all_sessions = n_distinct(web_session_id)) %>%\n",
    "    group_by(interfacelanguage, contentlanguage) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id),\n",
    "             pct_sessions = n_sessions/all_sessions) %>%\n",
    "    distinct() %>%\n",
    "    arrange(desc(n_sessions))\n",
    "\n",
    "head(top_initial_languages )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interfacelanguage and contentlanguage will usually be the same and should match for most instances, which is confirmed here.\n",
    "\n",
    "The top initial languages all from test wikis, which is expected since the language links are still shown to all logged-in and logged-out users on modern vector. The new language switcher, which replaces the lang links with a button, are show to all logged-in users opt'd into modern vector on non-test wikis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Most Frequent Switch Types\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'interfacelanguage', 'selectedinterfacelanguage' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>interfacelanguage</th><th scope=col>selectedinterfacelanguage</th><th scope=col>n_sessions</th><th scope=col>pct_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>fr</td><td>en</td><td>775769</td><td>0.39495398</td></tr>\n",
       "\t<tr><td>pt</td><td>en</td><td>264373</td><td>0.13459570</td></tr>\n",
       "\t<tr><td>tr</td><td>en</td><td>118824</td><td>0.06049483</td></tr>\n",
       "\t<tr><td>fa</td><td>en</td><td> 95622</td><td>0.04868239</td></tr>\n",
       "\t<tr><td>fr</td><td>de</td><td> 94915</td><td>0.04832245</td></tr>\n",
       "\t<tr><td>he</td><td>en</td><td> 84668</td><td>0.04310557</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 6 × 4\n",
       "\\begin{tabular}{llll}\n",
       " interfacelanguage & selectedinterfacelanguage & n\\_sessions & pct\\_sessions\\\\\n",
       " <chr> & <chr> & <int> & <dbl>\\\\\n",
       "\\hline\n",
       "\t fr & en & 775769 & 0.39495398\\\\\n",
       "\t pt & en & 264373 & 0.13459570\\\\\n",
       "\t tr & en & 118824 & 0.06049483\\\\\n",
       "\t fa & en &  95622 & 0.04868239\\\\\n",
       "\t fr & de &  94915 & 0.04832245\\\\\n",
       "\t he & en &  84668 & 0.04310557\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 4\n",
       "\n",
       "| interfacelanguage &lt;chr&gt; | selectedinterfacelanguage &lt;chr&gt; | n_sessions &lt;int&gt; | pct_sessions &lt;dbl&gt; |\n",
       "|---|---|---|---|\n",
       "| fr | en | 775769 | 0.39495398 |\n",
       "| pt | en | 264373 | 0.13459570 |\n",
       "| tr | en | 118824 | 0.06049483 |\n",
       "| fa | en |  95622 | 0.04868239 |\n",
       "| fr | de |  94915 | 0.04832245 |\n",
       "| he | en |  84668 | 0.04310557 |\n",
       "\n"
      ],
      "text/plain": [
       "  interfacelanguage selectedinterfacelanguage n_sessions pct_sessions\n",
       "1 fr                en                        775769     0.39495398  \n",
       "2 pt                en                        264373     0.13459570  \n",
       "3 tr                en                        118824     0.06049483  \n",
       "4 fa                en                         95622     0.04868239  \n",
       "5 fr                de                         94915     0.04832245  \n",
       "6 he                en                         84668     0.04310557  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "top_final_languages <- lang_link_events %>%\n",
    "    mutate(all_sessions = n_distinct(web_session_id)) %>%\n",
    "    group_by(interfacelanguage, selectedinterfacelanguage) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id),\n",
    "             pct_sessions = n_sessions/all_sessions) %>%\n",
    "    distinct() %>%\n",
    "    arrange(desc(n_sessions))\n",
    "\n",
    "head(top_final_languages )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "39% of all sessions are switches are from French to English. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Language Switcher Button"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Instrumentation Task](https://phabricator.wikimedia.org/T281928)\n",
    "\n",
    "- Instrumented as `event.action=\"compact-language-links-open\"`\n",
    "- Logged when a user open the new language switcher button. \n",
    "- The first click and subsequent clicks should be recorded per https://gerrit.wikimedia.org/r/c/mediawiki/extensions/UniversalLanguageSelector/+/688994/\n",
    "- Change merged on May 11th and backported/deployed on May 12th.\n",
    "- There should not be any clicks to the Language Switcher button by logged-in users on test wikis or logged-out users, as the new lanuage button has not been deployed to these wikis or these users.\n",
    "- Only deployed as logged-in users on the latest vector on a non test wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [],
   "source": [
    "query <- \n",
    "\"\n",
    "SELECT\n",
    "    TO_DATE(dt) AS `date`,\n",
    "    wiki,\n",
    "    event.web_session_id,\n",
    "    event.usereditbucket,\n",
    "    event.skin,\n",
    "    event.skinVersion,\n",
    "    event.timetochangelanguage,\n",
    "    event.isanon,\n",
    "    event.context,\n",
    "    event.interfacelanguage,\n",
    "    event.contentlanguage,\n",
    "    event.selectedinterfacelanguage,\n",
    "    Count(*) AS n_events\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month >= 05\n",
    "    AND event.action = 'compact-language-links-open'\n",
    "    AND useragent.is_bot = false\n",
    "GROUP BY\n",
    "    TO_DATE(dt),\n",
    "    wiki,\n",
    "    event.web_session_id,\n",
    "    event.usereditbucket,\n",
    "    event.skin,\n",
    "    event.skinVersion,\n",
    "    event.timetochangelanguage,\n",
    "    event.isanon,\n",
    "    event.context,\n",
    "    event.interfacelanguage,\n",
    "    event.contentlanguage,\n",
    "    event.selectedinterfacelanguage\n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lang_button_events <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Daily Lang Button Click Events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 49 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>date</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>2021-05-12</td><td>    12</td><td>    10</td></tr>\n",
       "\t<tr><td>2021-05-13</td><td>   194</td><td>   159</td></tr>\n",
       "\t<tr><td>2021-05-14</td><td>  1317</td><td>  1147</td></tr>\n",
       "\t<tr><td>2021-05-15</td><td>  1159</td><td>   928</td></tr>\n",
       "\t<tr><td>2021-05-16</td><td>  1358</td><td>  1042</td></tr>\n",
       "\t<tr><td>2021-05-17</td><td> 32668</td><td> 28541</td></tr>\n",
       "\t<tr><td>2021-05-18</td><td> 90811</td><td> 78945</td></tr>\n",
       "\t<tr><td>2021-05-19</td><td> 89299</td><td> 77108</td></tr>\n",
       "\t<tr><td>2021-05-20</td><td> 87644</td><td> 75820</td></tr>\n",
       "\t<tr><td>2021-05-21</td><td> 81498</td><td> 67973</td></tr>\n",
       "\t<tr><td>2021-05-22</td><td> 67460</td><td> 55270</td></tr>\n",
       "\t<tr><td>2021-05-23</td><td> 76222</td><td> 62416</td></tr>\n",
       "\t<tr><td>2021-05-24</td><td> 92724</td><td> 76384</td></tr>\n",
       "\t<tr><td>2021-05-25</td><td> 92351</td><td> 77546</td></tr>\n",
       "\t<tr><td>2021-05-26</td><td> 89947</td><td> 75196</td></tr>\n",
       "\t<tr><td>2021-05-27</td><td> 87440</td><td> 73193</td></tr>\n",
       "\t<tr><td>2021-05-28</td><td> 78177</td><td> 65185</td></tr>\n",
       "\t<tr><td>2021-05-29</td><td> 65494</td><td> 53538</td></tr>\n",
       "\t<tr><td>2021-05-30</td><td> 72238</td><td> 59545</td></tr>\n",
       "\t<tr><td>2021-05-31</td><td> 87384</td><td> 72570</td></tr>\n",
       "\t<tr><td>2021-06-01</td><td> 86754</td><td> 72234</td></tr>\n",
       "\t<tr><td>2021-06-02</td><td> 85835</td><td> 71774</td></tr>\n",
       "\t<tr><td>2021-06-03</td><td> 81864</td><td> 68127</td></tr>\n",
       "\t<tr><td>2021-06-04</td><td> 74580</td><td> 61948</td></tr>\n",
       "\t<tr><td>2021-06-05</td><td> 63824</td><td> 52160</td></tr>\n",
       "\t<tr><td>2021-06-06</td><td> 71082</td><td> 58131</td></tr>\n",
       "\t<tr><td>2021-06-07</td><td> 85287</td><td> 71501</td></tr>\n",
       "\t<tr><td>2021-06-08</td><td> 84667</td><td> 70399</td></tr>\n",
       "\t<tr><td>2021-06-09</td><td> 83472</td><td> 69111</td></tr>\n",
       "\t<tr><td>2021-06-10</td><td> 80794</td><td> 67411</td></tr>\n",
       "\t<tr><td>2021-06-11</td><td> 72647</td><td> 60353</td></tr>\n",
       "\t<tr><td>2021-06-12</td><td> 60176</td><td> 49825</td></tr>\n",
       "\t<tr><td>2021-06-13</td><td> 66600</td><td> 54534</td></tr>\n",
       "\t<tr><td>2021-06-14</td><td> 80895</td><td> 67718</td></tr>\n",
       "\t<tr><td>2021-06-15</td><td> 79705</td><td> 67001</td></tr>\n",
       "\t<tr><td>2021-06-16</td><td> 78153</td><td> 64678</td></tr>\n",
       "\t<tr><td>2021-06-17</td><td> 76308</td><td> 63395</td></tr>\n",
       "\t<tr><td>2021-06-18</td><td> 70039</td><td> 57090</td></tr>\n",
       "\t<tr><td>2021-06-19</td><td> 57473</td><td> 46714</td></tr>\n",
       "\t<tr><td>2021-06-20</td><td> 62298</td><td> 51153</td></tr>\n",
       "\t<tr><td>2021-06-21</td><td> 76513</td><td> 63711</td></tr>\n",
       "\t<tr><td>2021-06-22</td><td>103507</td><td> 80471</td></tr>\n",
       "\t<tr><td>2021-06-23</td><td>145211</td><td>103694</td></tr>\n",
       "\t<tr><td>2021-06-24</td><td>151335</td><td>106808</td></tr>\n",
       "\t<tr><td>2021-06-25</td><td>142435</td><td> 98600</td></tr>\n",
       "\t<tr><td>2021-06-26</td><td>128018</td><td> 85548</td></tr>\n",
       "\t<tr><td>2021-06-27</td><td>145148</td><td> 96486</td></tr>\n",
       "\t<tr><td>2021-06-28</td><td>180394</td><td>123458</td></tr>\n",
       "\t<tr><td>2021-06-29</td><td> 63085</td><td> 45605</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 49 × 3\n",
       "\\begin{tabular}{lll}\n",
       " date & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t 2021-05-12 &     12 &     10\\\\\n",
       "\t 2021-05-13 &    194 &    159\\\\\n",
       "\t 2021-05-14 &   1317 &   1147\\\\\n",
       "\t 2021-05-15 &   1159 &    928\\\\\n",
       "\t 2021-05-16 &   1358 &   1042\\\\\n",
       "\t 2021-05-17 &  32668 &  28541\\\\\n",
       "\t 2021-05-18 &  90811 &  78945\\\\\n",
       "\t 2021-05-19 &  89299 &  77108\\\\\n",
       "\t 2021-05-20 &  87644 &  75820\\\\\n",
       "\t 2021-05-21 &  81498 &  67973\\\\\n",
       "\t 2021-05-22 &  67460 &  55270\\\\\n",
       "\t 2021-05-23 &  76222 &  62416\\\\\n",
       "\t 2021-05-24 &  92724 &  76384\\\\\n",
       "\t 2021-05-25 &  92351 &  77546\\\\\n",
       "\t 2021-05-26 &  89947 &  75196\\\\\n",
       "\t 2021-05-27 &  87440 &  73193\\\\\n",
       "\t 2021-05-28 &  78177 &  65185\\\\\n",
       "\t 2021-05-29 &  65494 &  53538\\\\\n",
       "\t 2021-05-30 &  72238 &  59545\\\\\n",
       "\t 2021-05-31 &  87384 &  72570\\\\\n",
       "\t 2021-06-01 &  86754 &  72234\\\\\n",
       "\t 2021-06-02 &  85835 &  71774\\\\\n",
       "\t 2021-06-03 &  81864 &  68127\\\\\n",
       "\t 2021-06-04 &  74580 &  61948\\\\\n",
       "\t 2021-06-05 &  63824 &  52160\\\\\n",
       "\t 2021-06-06 &  71082 &  58131\\\\\n",
       "\t 2021-06-07 &  85287 &  71501\\\\\n",
       "\t 2021-06-08 &  84667 &  70399\\\\\n",
       "\t 2021-06-09 &  83472 &  69111\\\\\n",
       "\t 2021-06-10 &  80794 &  67411\\\\\n",
       "\t 2021-06-11 &  72647 &  60353\\\\\n",
       "\t 2021-06-12 &  60176 &  49825\\\\\n",
       "\t 2021-06-13 &  66600 &  54534\\\\\n",
       "\t 2021-06-14 &  80895 &  67718\\\\\n",
       "\t 2021-06-15 &  79705 &  67001\\\\\n",
       "\t 2021-06-16 &  78153 &  64678\\\\\n",
       "\t 2021-06-17 &  76308 &  63395\\\\\n",
       "\t 2021-06-18 &  70039 &  57090\\\\\n",
       "\t 2021-06-19 &  57473 &  46714\\\\\n",
       "\t 2021-06-20 &  62298 &  51153\\\\\n",
       "\t 2021-06-21 &  76513 &  63711\\\\\n",
       "\t 2021-06-22 & 103507 &  80471\\\\\n",
       "\t 2021-06-23 & 145211 & 103694\\\\\n",
       "\t 2021-06-24 & 151335 & 106808\\\\\n",
       "\t 2021-06-25 & 142435 &  98600\\\\\n",
       "\t 2021-06-26 & 128018 &  85548\\\\\n",
       "\t 2021-06-27 & 145148 &  96486\\\\\n",
       "\t 2021-06-28 & 180394 & 123458\\\\\n",
       "\t 2021-06-29 &  63085 &  45605\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 49 × 3\n",
       "\n",
       "| date &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| 2021-05-12 |     12 |     10 |\n",
       "| 2021-05-13 |    194 |    159 |\n",
       "| 2021-05-14 |   1317 |   1147 |\n",
       "| 2021-05-15 |   1159 |    928 |\n",
       "| 2021-05-16 |   1358 |   1042 |\n",
       "| 2021-05-17 |  32668 |  28541 |\n",
       "| 2021-05-18 |  90811 |  78945 |\n",
       "| 2021-05-19 |  89299 |  77108 |\n",
       "| 2021-05-20 |  87644 |  75820 |\n",
       "| 2021-05-21 |  81498 |  67973 |\n",
       "| 2021-05-22 |  67460 |  55270 |\n",
       "| 2021-05-23 |  76222 |  62416 |\n",
       "| 2021-05-24 |  92724 |  76384 |\n",
       "| 2021-05-25 |  92351 |  77546 |\n",
       "| 2021-05-26 |  89947 |  75196 |\n",
       "| 2021-05-27 |  87440 |  73193 |\n",
       "| 2021-05-28 |  78177 |  65185 |\n",
       "| 2021-05-29 |  65494 |  53538 |\n",
       "| 2021-05-30 |  72238 |  59545 |\n",
       "| 2021-05-31 |  87384 |  72570 |\n",
       "| 2021-06-01 |  86754 |  72234 |\n",
       "| 2021-06-02 |  85835 |  71774 |\n",
       "| 2021-06-03 |  81864 |  68127 |\n",
       "| 2021-06-04 |  74580 |  61948 |\n",
       "| 2021-06-05 |  63824 |  52160 |\n",
       "| 2021-06-06 |  71082 |  58131 |\n",
       "| 2021-06-07 |  85287 |  71501 |\n",
       "| 2021-06-08 |  84667 |  70399 |\n",
       "| 2021-06-09 |  83472 |  69111 |\n",
       "| 2021-06-10 |  80794 |  67411 |\n",
       "| 2021-06-11 |  72647 |  60353 |\n",
       "| 2021-06-12 |  60176 |  49825 |\n",
       "| 2021-06-13 |  66600 |  54534 |\n",
       "| 2021-06-14 |  80895 |  67718 |\n",
       "| 2021-06-15 |  79705 |  67001 |\n",
       "| 2021-06-16 |  78153 |  64678 |\n",
       "| 2021-06-17 |  76308 |  63395 |\n",
       "| 2021-06-18 |  70039 |  57090 |\n",
       "| 2021-06-19 |  57473 |  46714 |\n",
       "| 2021-06-20 |  62298 |  51153 |\n",
       "| 2021-06-21 |  76513 |  63711 |\n",
       "| 2021-06-22 | 103507 |  80471 |\n",
       "| 2021-06-23 | 145211 | 103694 |\n",
       "| 2021-06-24 | 151335 | 106808 |\n",
       "| 2021-06-25 | 142435 |  98600 |\n",
       "| 2021-06-26 | 128018 |  85548 |\n",
       "| 2021-06-27 | 145148 |  96486 |\n",
       "| 2021-06-28 | 180394 | 123458 |\n",
       "| 2021-06-29 |  63085 |  45605 |\n",
       "\n"
      ],
      "text/plain": [
       "   date       n_events n_sessions\n",
       "1  2021-05-12     12       10    \n",
       "2  2021-05-13    194      159    \n",
       "3  2021-05-14   1317     1147    \n",
       "4  2021-05-15   1159      928    \n",
       "5  2021-05-16   1358     1042    \n",
       "6  2021-05-17  32668    28541    \n",
       "7  2021-05-18  90811    78945    \n",
       "8  2021-05-19  89299    77108    \n",
       "9  2021-05-20  87644    75820    \n",
       "10 2021-05-21  81498    67973    \n",
       "11 2021-05-22  67460    55270    \n",
       "12 2021-05-23  76222    62416    \n",
       "13 2021-05-24  92724    76384    \n",
       "14 2021-05-25  92351    77546    \n",
       "15 2021-05-26  89947    75196    \n",
       "16 2021-05-27  87440    73193    \n",
       "17 2021-05-28  78177    65185    \n",
       "18 2021-05-29  65494    53538    \n",
       "19 2021-05-30  72238    59545    \n",
       "20 2021-05-31  87384    72570    \n",
       "21 2021-06-01  86754    72234    \n",
       "22 2021-06-02  85835    71774    \n",
       "23 2021-06-03  81864    68127    \n",
       "24 2021-06-04  74580    61948    \n",
       "25 2021-06-05  63824    52160    \n",
       "26 2021-06-06  71082    58131    \n",
       "27 2021-06-07  85287    71501    \n",
       "28 2021-06-08  84667    70399    \n",
       "29 2021-06-09  83472    69111    \n",
       "30 2021-06-10  80794    67411    \n",
       "31 2021-06-11  72647    60353    \n",
       "32 2021-06-12  60176    49825    \n",
       "33 2021-06-13  66600    54534    \n",
       "34 2021-06-14  80895    67718    \n",
       "35 2021-06-15  79705    67001    \n",
       "36 2021-06-16  78153    64678    \n",
       "37 2021-06-17  76308    63395    \n",
       "38 2021-06-18  70039    57090    \n",
       "39 2021-06-19  57473    46714    \n",
       "40 2021-06-20  62298    51153    \n",
       "41 2021-06-21  76513    63711    \n",
       "42 2021-06-22 103507    80471    \n",
       "43 2021-06-23 145211   103694    \n",
       "44 2021-06-24 151335   106808    \n",
       "45 2021-06-25 142435    98600    \n",
       "46 2021-06-26 128018    85548    \n",
       "47 2021-06-27 145148    96486    \n",
       "48 2021-06-28 180394   123458    \n",
       "49 2021-06-29  63085    45605    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_events_daily <- lang_button_events %>%\n",
    "    group_by(date) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_events_daily"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Confirmed we start recording events on May 12th following backport of the change, starting with just a few events and then more as change rolled out to additional wikis.\n",
    "-  There is an expected higher number of events than sessions, which is expected as a session can include multiple clicks to open the language button.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# By Context and Vector Version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'skinversion' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 4 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>skinversion</th><th scope=col>context</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>latest</td><td>header</td><td> 595196</td><td> 326328</td></tr>\n",
       "\t<tr><td>latest</td><td>other </td><td>  94351</td><td>  77145</td></tr>\n",
       "\t<tr><td>legacy</td><td>other </td><td>1362565</td><td>1093551</td></tr>\n",
       "\t<tr><td>NULL  </td><td>NULL  </td><td>  36761</td><td>  31132</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 4 × 4\n",
       "\\begin{tabular}{llll}\n",
       " skinversion & context & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t latest & header &  595196 &  326328\\\\\n",
       "\t latest & other  &   94351 &   77145\\\\\n",
       "\t legacy & other  & 1362565 & 1093551\\\\\n",
       "\t NULL   & NULL   &   36761 &   31132\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 4 × 4\n",
       "\n",
       "| skinversion &lt;chr&gt; | context &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| latest | header |  595196 |  326328 |\n",
       "| latest | other  |   94351 |   77145 |\n",
       "| legacy | other  | 1362565 | 1093551 |\n",
       "| NULL   | NULL   |   36761 |   31132 |\n",
       "\n"
      ],
      "text/plain": [
       "  skinversion context n_events n_sessions\n",
       "1 latest      header   595196   326328   \n",
       "2 latest      other     94351    77145   \n",
       "3 legacy      other   1362565  1093551   \n",
       "4 NULL        NULL      36761    31132   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_events_bycontext <- lang_button_events %>%\n",
    "# date new instrumentation was added\n",
    "    filter(date >= '2021-06-08')%>%\n",
    "    group_by(skinversion, context) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_events_bycontext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Conirmed we are only recording new button clicks on the latest vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Double Check to make sure there are some sessions with more than one event"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "FALSE"
      ],
      "text/latex": [
       "FALSE"
      ],
      "text/markdown": [
       "FALSE"
      ],
      "text/plain": [
       "[1] FALSE"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "length(unique(lang_button_events$web_session_id)) == nrow(lang_button_events)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lang Button Events Per Wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 6 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>wiki</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>enwiki</td><td>1719595</td><td>1341346</td></tr>\n",
       "\t<tr><td>frwiki</td><td> 455616</td><td> 280201</td></tr>\n",
       "\t<tr><td>ruwiki</td><td> 230135</td><td> 177510</td></tr>\n",
       "\t<tr><td>dewiki</td><td> 208276</td><td> 173340</td></tr>\n",
       "\t<tr><td>eswiki</td><td> 154449</td><td> 126942</td></tr>\n",
       "\t<tr><td>ptwiki</td><td> 145856</td><td>  91869</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 6 × 3\n",
       "\\begin{tabular}{lll}\n",
       " wiki & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t enwiki & 1719595 & 1341346\\\\\n",
       "\t frwiki &  455616 &  280201\\\\\n",
       "\t ruwiki &  230135 &  177510\\\\\n",
       "\t dewiki &  208276 &  173340\\\\\n",
       "\t eswiki &  154449 &  126942\\\\\n",
       "\t ptwiki &  145856 &   91869\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 6 × 3\n",
       "\n",
       "| wiki &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| enwiki | 1719595 | 1341346 |\n",
       "| frwiki |  455616 |  280201 |\n",
       "| ruwiki |  230135 |  177510 |\n",
       "| dewiki |  208276 |  173340 |\n",
       "| eswiki |  154449 |  126942 |\n",
       "| ptwiki |  145856 |   91869 |\n",
       "\n"
      ],
      "text/plain": [
       "  wiki   n_events n_sessions\n",
       "1 enwiki 1719595  1341346   \n",
       "2 frwiki  455616   280201   \n",
       "3 ruwiki  230135   177510   \n",
       "4 dewiki  208276   173340   \n",
       "5 eswiki  154449   126942   \n",
       "6 ptwiki  145856    91869   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "lang_button_events_bywiki <- lang_button_events %>%\n",
    "    group_by(wiki) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id)) %>%\n",
    "    arrange(desc(n_sessions))\n",
    "\n",
    "head(lang_button_events_bywiki)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost half (49.7%) of sessions with a click to the language button were recorded on English Wikipedia, followed by Russian, German and Spanish Wikipedia. Note: The language switcher button is currently was not available to logged-in users on test wikis until 22 June 2021. AB Test deployment on Fawiki was delayed until 28 June 2021. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lang Button Event on Test Wikis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 13 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>wiki</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>frwiki       </td><td>455616</td><td>280201</td></tr>\n",
       "\t<tr><td>ptwiki       </td><td>145856</td><td> 91869</td></tr>\n",
       "\t<tr><td>trwiki       </td><td> 67646</td><td> 46391</td></tr>\n",
       "\t<tr><td>hewiki       </td><td> 54807</td><td> 33156</td></tr>\n",
       "\t<tr><td>kowiki       </td><td> 39555</td><td> 24618</td></tr>\n",
       "\t<tr><td>srwiki       </td><td> 30836</td><td> 21933</td></tr>\n",
       "\t<tr><td>fawiki       </td><td> 15068</td><td> 11986</td></tr>\n",
       "\t<tr><td>frwiktionary </td><td>  6245</td><td>  4000</td></tr>\n",
       "\t<tr><td>bnwiki       </td><td>  4205</td><td>  2979</td></tr>\n",
       "\t<tr><td>euwiki       </td><td>  3972</td><td>  2944</td></tr>\n",
       "\t<tr><td>vecwiki      </td><td>   887</td><td>   794</td></tr>\n",
       "\t<tr><td>dewikivoyage </td><td>   259</td><td>   177</td></tr>\n",
       "\t<tr><td>ptwikiversity</td><td>     4</td><td>     3</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 13 × 3\n",
       "\\begin{tabular}{lll}\n",
       " wiki & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t frwiki        & 455616 & 280201\\\\\n",
       "\t ptwiki        & 145856 &  91869\\\\\n",
       "\t trwiki        &  67646 &  46391\\\\\n",
       "\t hewiki        &  54807 &  33156\\\\\n",
       "\t kowiki        &  39555 &  24618\\\\\n",
       "\t srwiki        &  30836 &  21933\\\\\n",
       "\t fawiki        &  15068 &  11986\\\\\n",
       "\t frwiktionary  &   6245 &   4000\\\\\n",
       "\t bnwiki        &   4205 &   2979\\\\\n",
       "\t euwiki        &   3972 &   2944\\\\\n",
       "\t vecwiki       &    887 &    794\\\\\n",
       "\t dewikivoyage  &    259 &    177\\\\\n",
       "\t ptwikiversity &      4 &      3\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 13 × 3\n",
       "\n",
       "| wiki &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| frwiki        | 455616 | 280201 |\n",
       "| ptwiki        | 145856 |  91869 |\n",
       "| trwiki        |  67646 |  46391 |\n",
       "| hewiki        |  54807 |  33156 |\n",
       "| kowiki        |  39555 |  24618 |\n",
       "| srwiki        |  30836 |  21933 |\n",
       "| fawiki        |  15068 |  11986 |\n",
       "| frwiktionary  |   6245 |   4000 |\n",
       "| bnwiki        |   4205 |   2979 |\n",
       "| euwiki        |   3972 |   2944 |\n",
       "| vecwiki       |    887 |    794 |\n",
       "| dewikivoyage  |    259 |    177 |\n",
       "| ptwikiversity |      4 |      3 |\n",
       "\n"
      ],
      "text/plain": [
       "   wiki          n_events n_sessions\n",
       "1  frwiki        455616   280201    \n",
       "2  ptwiki        145856    91869    \n",
       "3  trwiki         67646    46391    \n",
       "4  hewiki         54807    33156    \n",
       "5  kowiki         39555    24618    \n",
       "6  srwiki         30836    21933    \n",
       "7  fawiki         15068    11986    \n",
       "8  frwiktionary    6245     4000    \n",
       "9  bnwiki          4205     2979    \n",
       "10 euwiki          3972     2944    \n",
       "11 vecwiki          887      794    \n",
       "12 dewikivoyage     259      177    \n",
       "13 ptwikiversity      4        3    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "lang_button_events_bytestwiki <- lang_button_events %>%\n",
    "    filter(wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', \n",
    "                       'dewikivoyage', 'vecwiki' ))  %>%\n",
    "    group_by(wiki) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id)) %>%\n",
    "    arrange(desc(n_sessions))\n",
    "\n",
    "lang_button_events_bytestwiki"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About 8.4% of all sessions were recorded on the early adopter wikis. \n",
    "\n",
    "I reviewed the number of sessions on test wikis by logged-in status to determine if these were mostly logged-out or logged-in users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'istestwiki' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 4 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>istestwiki</th><th scope=col>isanon</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>non_test_wiki</td><td>false</td><td> 116409</td><td>  68458</td></tr>\n",
       "\t<tr><td>non_test_wiki</td><td>true </td><td>2892131</td><td>2324483</td></tr>\n",
       "\t<tr><td>test_wiki    </td><td>false</td><td>  27679</td><td>  13284</td></tr>\n",
       "\t<tr><td>test_wiki    </td><td>true </td><td> 797277</td><td> 508146</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 4 × 4\n",
       "\\begin{tabular}{llll}\n",
       " istestwiki & isanon & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t non\\_test\\_wiki & false &  116409 &   68458\\\\\n",
       "\t non\\_test\\_wiki & true  & 2892131 & 2324483\\\\\n",
       "\t test\\_wiki     & false &   27679 &   13284\\\\\n",
       "\t test\\_wiki     & true  &  797277 &  508146\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 4 × 4\n",
       "\n",
       "| istestwiki &lt;chr&gt; | isanon &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| non_test_wiki | false |  116409 |   68458 |\n",
       "| non_test_wiki | true  | 2892131 | 2324483 |\n",
       "| test_wiki     | false |   27679 |   13284 |\n",
       "| test_wiki     | true  |  797277 |  508146 |\n",
       "\n"
      ],
      "text/plain": [
       "  istestwiki    isanon n_events n_sessions\n",
       "1 non_test_wiki false   116409    68458   \n",
       "2 non_test_wiki true   2892131  2324483   \n",
       "3 test_wiki     false    27679    13284   \n",
       "4 test_wiki     true    797277   508146   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# events and sessions that include link to language link by test wiki category and logged-in status\n",
    "lang_button_events_testwiki_isanon <- lang_button_events %>%\n",
    "    mutate(istestwiki = ifelse(wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' ), 'test_wiki', 'non_test_wiki')) %>%\n",
    "    group_by(istestwiki, isanon) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_events_testwiki_isanon "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "0.29% of sessions with a click to the lang switcher button came from logged-in users on the early adopter wikis and 8.1% of sessions by logged-out users. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By anon status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>isanon</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>false</td><td> 144088</td><td>  81742</td></tr>\n",
       "\t<tr><td>true </td><td>3689408</td><td>2832629</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 3\n",
       "\\begin{tabular}{lll}\n",
       " isanon & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t false &  144088 &   81742\\\\\n",
       "\t true  & 3689408 & 2832629\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 3\n",
       "\n",
       "| isanon &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| false |  144088 |   81742 |\n",
       "| true  | 3689408 | 2832629 |\n",
       "\n"
      ],
      "text/plain": [
       "  isanon n_events n_sessions\n",
       "1 false   144088    81742   \n",
       "2 true   3689408  2832629   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_events_byanon <- lang_button_events %>%\n",
    "    group_by(isanon) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id)) \n",
    "\n",
    "head(lang_button_events_byanon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The majority of sessions with clicks to the language button are by logged out users (97.2%) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clicks per session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>isanon</th><th scope=col>avg_clicks</th><th scope=col>max_clicks</th><th scope=col>min_clciks</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>false</td><td>1.583506</td><td>285</td><td>1</td></tr>\n",
       "\t<tr><td>true </td><td>1.244324</td><td>798</td><td>1</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 4\n",
       "\\begin{tabular}{llll}\n",
       " isanon & avg\\_clicks & max\\_clicks & min\\_clciks\\\\\n",
       " <chr> & <dbl> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t false & 1.583506 & 285 & 1\\\\\n",
       "\t true  & 1.244324 & 798 & 1\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 4\n",
       "\n",
       "| isanon &lt;chr&gt; | avg_clicks &lt;dbl&gt; | max_clicks &lt;int&gt; | min_clciks &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| false | 1.583506 | 285 | 1 |\n",
       "| true  | 1.244324 | 798 | 1 |\n",
       "\n"
      ],
      "text/plain": [
       "  isanon avg_clicks max_clicks min_clciks\n",
       "1 false  1.583506   285        1         \n",
       "2 true   1.244324   798        1         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_events_persession <- lang_button_events %>%\n",
    "    group_by(isanon) %>%\n",
    "    summarize(avg_clicks = mean(n_events),\n",
    "              max_clicks = max(n_events),\n",
    "              min_clciks = min(n_events))\n",
    "\n",
    "lang_button_events_persession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most sessions include between 1 to 2 clicks to the language switcher button.\n",
    "\n",
    "Note: There are some sessions by anon users with over 600 clicks, which is is likely automated traffic from bots and can be filtered out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## User Edit Bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 6 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>usereditbucket</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>0 edits      </td><td>36100</td><td>26263</td></tr>\n",
       "\t<tr><td>1-4 edits    </td><td>15370</td><td>10060</td></tr>\n",
       "\t<tr><td>100-999 edits</td><td>21124</td><td>11002</td></tr>\n",
       "\t<tr><td>1000+ edits  </td><td>37444</td><td>15816</td></tr>\n",
       "\t<tr><td>5-99 edits   </td><td>34048</td><td>18811</td></tr>\n",
       "\t<tr><td>NULL         </td><td>    2</td><td>    2</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 6 × 3\n",
       "\\begin{tabular}{lll}\n",
       " usereditbucket & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t 0 edits       & 36100 & 26263\\\\\n",
       "\t 1-4 edits     & 15370 & 10060\\\\\n",
       "\t 100-999 edits & 21124 & 11002\\\\\n",
       "\t 1000+ edits   & 37444 & 15816\\\\\n",
       "\t 5-99 edits    & 34048 & 18811\\\\\n",
       "\t NULL          &     2 &     2\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 6 × 3\n",
       "\n",
       "| usereditbucket &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| 0 edits       | 36100 | 26263 |\n",
       "| 1-4 edits     | 15370 | 10060 |\n",
       "| 100-999 edits | 21124 | 11002 |\n",
       "| 1000+ edits   | 37444 | 15816 |\n",
       "| 5-99 edits    | 34048 | 18811 |\n",
       "| NULL          |     2 |     2 |\n",
       "\n"
      ],
      "text/plain": [
       "  usereditbucket n_events n_sessions\n",
       "1 0 edits        36100    26263     \n",
       "2 1-4 edits      15370    10060     \n",
       "3 100-999 edits  21124    11002     \n",
       "4 1000+ edits    37444    15816     \n",
       "5 5-99 edits     34048    18811     \n",
       "6 NULL               2        2     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logged_in_editcount <- lang_button_events %>%\n",
    "    filter(isanon == 'false') %>%\n",
    "    group_by(usereditbucket) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "logged_in_editcount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 1 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>usereditbucket</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>NULL</td><td>3689408</td><td>2832629</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 1 × 3\n",
       "\\begin{tabular}{lll}\n",
       " usereditbucket & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t NULL & 3689408 & 2832629\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 1 × 3\n",
       "\n",
       "| usereditbucket &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| NULL | 3689408 | 2832629 |\n",
       "\n"
      ],
      "text/plain": [
       "  usereditbucket n_events n_sessions\n",
       "1 NULL           3689408  2832629   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logged_out_editcount <- lang_button_events %>%\n",
    "    filter(isanon == 'true') %>%\n",
    "    group_by(usereditbucket) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "logged_out_editcount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirmed that we are not recording an edit count for logged out users as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Content Language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'interfacelanguage', 'contentlanguage' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>interfacelanguage</th><th scope=col>contentlanguage</th><th scope=col>n_sessions</th><th scope=col>pct_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>en</td><td>en</td><td>1361586</td><td>0.46746010</td></tr>\n",
       "\t<tr><td>fr</td><td>fr</td><td> 284033</td><td>0.09751429</td></tr>\n",
       "\t<tr><td>ru</td><td>ru</td><td> 178766</td><td>0.06137400</td></tr>\n",
       "\t<tr><td>de</td><td>de</td><td> 174512</td><td>0.05991351</td></tr>\n",
       "\t<tr><td>es</td><td>es</td><td> 127334</td><td>0.04371635</td></tr>\n",
       "\t<tr><td>pt</td><td>pt</td><td>  91875</td><td>0.03154255</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 6 × 4\n",
       "\\begin{tabular}{llll}\n",
       " interfacelanguage & contentlanguage & n\\_sessions & pct\\_sessions\\\\\n",
       " <chr> & <chr> & <int> & <dbl>\\\\\n",
       "\\hline\n",
       "\t en & en & 1361586 & 0.46746010\\\\\n",
       "\t fr & fr &  284033 & 0.09751429\\\\\n",
       "\t ru & ru &  178766 & 0.06137400\\\\\n",
       "\t de & de &  174512 & 0.05991351\\\\\n",
       "\t es & es &  127334 & 0.04371635\\\\\n",
       "\t pt & pt &   91875 & 0.03154255\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 4\n",
       "\n",
       "| interfacelanguage &lt;chr&gt; | contentlanguage &lt;chr&gt; | n_sessions &lt;int&gt; | pct_sessions &lt;dbl&gt; |\n",
       "|---|---|---|---|\n",
       "| en | en | 1361586 | 0.46746010 |\n",
       "| fr | fr |  284033 | 0.09751429 |\n",
       "| ru | ru |  178766 | 0.06137400 |\n",
       "| de | de |  174512 | 0.05991351 |\n",
       "| es | es |  127334 | 0.04371635 |\n",
       "| pt | pt |   91875 | 0.03154255 |\n",
       "\n"
      ],
      "text/plain": [
       "  interfacelanguage contentlanguage n_sessions pct_sessions\n",
       "1 en                en              1361586    0.46746010  \n",
       "2 fr                fr               284033    0.09751429  \n",
       "3 ru                ru               178766    0.06137400  \n",
       "4 de                de               174512    0.05991351  \n",
       "5 es                es               127334    0.04371635  \n",
       "6 pt                pt                91875    0.03154255  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "top_content_languages <- lang_button_events %>%\n",
    "    mutate(all_sessions = n_distinct(web_session_id)) %>%\n",
    "    group_by(interfacelanguage, contentlanguage) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id),\n",
    "             pct_sessions = n_sessions/all_sessions) %>%\n",
    "    distinct() %>%\n",
    "    arrange(desc(n_sessions))\n",
    "\n",
    "head(top_content_languages )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top current content and interface lang settings are english, russian, german and spanish, which fits with the top wikis we see where the language button switch clicks occur.\n",
    "\n",
    "Update: French is now on the list as of 22 June 2021 as AB test was deployed then"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Event Context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 3 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>context</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td> 595196</td><td> 326328</td></tr>\n",
       "\t<tr><td>NULL  </td><td>1781384</td><td>1439296</td></tr>\n",
       "\t<tr><td>other </td><td>1456916</td><td>1170691</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 3 × 3\n",
       "\\begin{tabular}{lll}\n",
       " context & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t header &  595196 &  326328\\\\\n",
       "\t NULL   & 1781384 & 1439296\\\\\n",
       "\t other  & 1456916 & 1170691\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 3 × 3\n",
       "\n",
       "| context &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| header |  595196 |  326328 |\n",
       "| NULL   | 1781384 | 1439296 |\n",
       "| other  | 1456916 | 1170691 |\n",
       "\n"
      ],
      "text/plain": [
       "  context n_events n_sessions\n",
       "1 header   595196   326328   \n",
       "2 NULL    1781384  1439296   \n",
       "3 other   1456916  1170691   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "lang_button_events_context <- lang_button_events %>%\n",
    "    group_by(context) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_events_context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are not recording `event.context` for these events. \n",
    "\n",
    "Note: This was changed with a fix deployed on 6 June 2021. We are now recorded new button clicks as event.context = 'header'. event.context = 'other' is recorded for people in control group (or using legacy skin) that click N more button in the sidebar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time to Change Language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning message in mean.default(timetochangelanguage):\n",
      "“argument is not numeric or logical: returning NA”\n",
      "Warning message in mean.default(sort(x, partial = half + 0L:1L)[half + 0L:1L]):\n",
      "“argument is not numeric or logical: returning NA”\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A data.frame: 1 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>avg_time</th><th scope=col>median_time</th><th scope=col>max_time</th><th scope=col>min_time</th></tr>\n",
       "\t<tr><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>NA</td><td>NA</td><td>NULL</td><td>NULL</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A data.frame: 1 × 4\n",
       "\\begin{tabular}{llll}\n",
       " avg\\_time & median\\_time & max\\_time & min\\_time\\\\\n",
       " <dbl> & <dbl> & <chr> & <chr>\\\\\n",
       "\\hline\n",
       "\t NA & NA & NULL & NULL\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A data.frame: 1 × 4\n",
       "\n",
       "| avg_time &lt;dbl&gt; | median_time &lt;dbl&gt; | max_time &lt;chr&gt; | min_time &lt;chr&gt; |\n",
       "|---|---|---|---|\n",
       "| NA | NA | NULL | NULL |\n",
       "\n"
      ],
      "text/plain": [
       "  avg_time median_time max_time min_time\n",
       "1 NA       NA          NULL     NULL    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "lang_button_events_time <- lang_button_events %>%\n",
    "    summarise(avg_time = mean(timetochangelanguage),\n",
    "             median_time = median(timetochangelanguage),\n",
    "             max_time = max(timetochangelanguage),\n",
    "             min_time = min(timetochangelanguage))\n",
    "\n",
    "lang_button_events_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: We don't record time to change language for the initial click to the button but we record when the user actually changes languages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confirm we can identify language switches for sessions where button is clicked"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rough query to confirm approach\n",
    "# will be refine in analysis\n",
    "query <- \n",
    "\"\n",
    "-- sessions where lang button was selected\n",
    "WITH button AS (\n",
    "SELECT\n",
    "    MIN(TO_DATE(dt)) as button_date,\n",
    "    event.web_session_id as session_id,\n",
    "    event.context as open_context,\n",
    "    wiki as wiki\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month >= 05\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.action = 'compact-language-links-open'\n",
    "GROUP BY \n",
    "    event.web_session_id,\n",
    "    event.context,\n",
    "    wiki\n",
    "),\n",
    "\n",
    "lang_switches AS (\n",
    "  SELECT\n",
    "    TO_DATE(dt) as switch_date,\n",
    "    event.web_session_id as session_id,\n",
    "    event.context as switch_context,\n",
    "    wiki as wiki\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month >= 05\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.action = 'language-change'\n",
    ")\n",
    "\n",
    "SELECT\n",
    "    button.button_date,\n",
    "    lang_switches.switch_date,\n",
    "    button.session_id,\n",
    "    button.wiki,\n",
    "    button.open_context,\n",
    "-- sessions with lang switch that occured after button clicks\n",
    "    IF(lang_switches.session_id IS NOT NULL AND switch_date >= button_date, 1, 0) AS language_switch,\n",
    "    lang_switches.switch_context\n",
    "FROM button\n",
    "LEFT JOIN lang_switches ON\n",
    "    button.session_id = lang_switches.session_id AND\n",
    "    button.wiki = lang_switches.wiki\n",
    "    \n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lang_button_switch_events <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reformat to date format\n",
    "lang_button_switch_events$button_date <- as.Date(lang_button_switch_events$button_date, format = \"%Y-%m-%d\")\n",
    "lang_button_switch_events$switch_date <- as.Date(lang_button_switch_events$switch_date, format = \"%Y-%m-%d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lang Button Sessions with Language Switch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 2</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>language_switch</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>no_switch</td><td>2106249</td></tr>\n",
       "\t<tr><td>switch   </td><td> 842170</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 2\n",
       "\\begin{tabular}{ll}\n",
       " language\\_switch & n\\_sessions\\\\\n",
       " <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t no\\_switch & 2106249\\\\\n",
       "\t switch    &  842170\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 2\n",
       "\n",
       "| language_switch &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|\n",
       "| no_switch | 2106249 |\n",
       "| switch    |  842170 |\n",
       "\n"
      ],
      "text/plain": [
       "  language_switch n_sessions\n",
       "1 no_switch       2106249   \n",
       "2 switch           842170   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_events_nsessions <- lang_button_switch_events %>%\n",
    "    mutate(language_switch = ifelse(language_switch == 0, \"no_switch\", \"switch\")) %>%\n",
    "    group_by(language_switch) %>%\n",
    "    summarise(n_sessions = n_distinct(session_id))\n",
    "\n",
    "lang_button_switch_events_nsessions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About 3.4% of sessions where the language button was clicked was followed by an event to switch the language. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lang Button Sessions with Switch by Context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'open_context' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>open_context</th><th scope=col>switch_context</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>content-language-switcher</td><td>112876</td></tr>\n",
       "\t<tr><td>header</td><td>interface                </td><td>     3</td></tr>\n",
       "\t<tr><td>header</td><td>languages-list           </td><td>   748</td></tr>\n",
       "\t<tr><td>other </td><td>content-language-switcher</td><td> 58041</td></tr>\n",
       "\t<tr><td>other </td><td>interface                </td><td>    38</td></tr>\n",
       "\t<tr><td>other </td><td>languages-list           </td><td>   446</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 6 × 3\n",
       "\\begin{tabular}{lll}\n",
       " open\\_context & switch\\_context & n\\_sessions\\\\\n",
       " <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t header & content-language-switcher & 112876\\\\\n",
       "\t header & interface                 &      3\\\\\n",
       "\t header & languages-list            &    748\\\\\n",
       "\t other  & content-language-switcher &  58041\\\\\n",
       "\t other  & interface                 &     38\\\\\n",
       "\t other  & languages-list            &    446\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 3\n",
       "\n",
       "| open_context &lt;chr&gt; | switch_context &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| header | content-language-switcher | 112876 |\n",
       "| header | interface                 |      3 |\n",
       "| header | languages-list            |    748 |\n",
       "| other  | content-language-switcher |  58041 |\n",
       "| other  | interface                 |     38 |\n",
       "| other  | languages-list            |    446 |\n",
       "\n"
      ],
      "text/plain": [
       "  open_context switch_context            n_sessions\n",
       "1 header       content-language-switcher 112876    \n",
       "2 header       interface                      3    \n",
       "3 header       languages-list               748    \n",
       "4 other        content-language-switcher  58041    \n",
       "5 other        interface                     38    \n",
       "6 other        languages-list               446    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_events_nsessions <- lang_button_switch_events %>%\n",
    "    filter(language_switch == 1,\n",
    "          button_date >= '2021-06-27') %>%\n",
    "    group_by(open_context, switch_context) %>%\n",
    "    summarise(n_sessions = n_distinct(session_id))\n",
    "\n",
    "lang_button_switch_events_nsessions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following 22 June 2021 fix, we are recording three types of language switch events after the new button is clicked:\n",
    "(1) 'interface': \n",
    "(2) 'languages-list'\n",
    "(3) 'content-language-switcher'\n",
    "\n",
    "\n",
    "TODO: Need to clarfiy differences.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Language Button Context Events\n",
    "\n",
    "Change deployed on 8 June 2021\n",
    "Rechecked data to confirm that events are recording as expected and that were able to distinguish between the following three event types:\n",
    "\n",
    "* People selecting language button on top of the page\n",
    "* People that are in the control bucket clicking the \"more\" button\n",
    "* People that have opted out of modern vector clicking the more button\n",
    "\n",
    "Update: Fix deployed on 10 June 2021 to address bug identified in post deployment QA, where links to switch languages after clicking the langauge switcher button were not instrumented. Data updated below to reflect changes following this fix.\n",
    "\n",
    "Note on instrumentation:\n",
    "If the user has opened the language switcher in the header and switched language, we should two events with the following properties:\n",
    "\n",
    "1. event.web_session_id=\"foo\" event.action=\"compact-language-links-open\" event.context=\"header\" event.skinversion=\"latest\"\n",
    "2. event.web_session_id=\"foo\" event.action=\"language-change\" event.context=\"interface\"`\n",
    "\n",
    "22 June 2021 Update: Fix deployed to add additional context to differentiate between the following events (both of these were previously marked as event.context = 'interface:\n",
    "- A user clicks the language button in the header and changes the display language setting\n",
    "- A user clicks the language button in the header and switches languages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By Skin Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'date', 'skin' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 50 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>date</th><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>2021-06-11</td><td>NULL  </td><td>NULL  </td><td>   18</td></tr>\n",
       "\t<tr><td>2021-06-11</td><td>vector</td><td>latest</td><td> 5467</td></tr>\n",
       "\t<tr><td>2021-06-11</td><td>vector</td><td>legacy</td><td>54874</td></tr>\n",
       "\t<tr><td>2021-06-12</td><td>NULL  </td><td>NULL  </td><td>   14</td></tr>\n",
       "\t<tr><td>2021-06-12</td><td>vector</td><td>latest</td><td> 4363</td></tr>\n",
       "\t<tr><td>2021-06-12</td><td>vector</td><td>legacy</td><td>45448</td></tr>\n",
       "\t<tr><td>2021-06-13</td><td>NULL  </td><td>NULL  </td><td>    5</td></tr>\n",
       "\t<tr><td>2021-06-13</td><td>vector</td><td>latest</td><td> 5053</td></tr>\n",
       "\t<tr><td>2021-06-13</td><td>vector</td><td>legacy</td><td>49477</td></tr>\n",
       "\t<tr><td>2021-06-14</td><td>NULL  </td><td>NULL  </td><td>    9</td></tr>\n",
       "\t<tr><td>2021-06-14</td><td>vector</td><td>latest</td><td> 5926</td></tr>\n",
       "\t<tr><td>2021-06-14</td><td>vector</td><td>legacy</td><td>61784</td></tr>\n",
       "\t<tr><td>2021-06-15</td><td>NULL  </td><td>NULL  </td><td>    6</td></tr>\n",
       "\t<tr><td>2021-06-15</td><td>vector</td><td>latest</td><td> 5952</td></tr>\n",
       "\t<tr><td>2021-06-15</td><td>vector</td><td>legacy</td><td>61043</td></tr>\n",
       "\t<tr><td>2021-06-16</td><td>NULL  </td><td>NULL  </td><td>    2</td></tr>\n",
       "\t<tr><td>2021-06-16</td><td>vector</td><td>latest</td><td> 5725</td></tr>\n",
       "\t<tr><td>2021-06-16</td><td>vector</td><td>legacy</td><td>58951</td></tr>\n",
       "\t<tr><td>2021-06-17</td><td>NULL  </td><td>NULL  </td><td>    1</td></tr>\n",
       "\t<tr><td>2021-06-17</td><td>vector</td><td>latest</td><td> 5622</td></tr>\n",
       "\t<tr><td>2021-06-17</td><td>vector</td><td>legacy</td><td>57772</td></tr>\n",
       "\t<tr><td>2021-06-18</td><td>NULL  </td><td>NULL  </td><td>    2</td></tr>\n",
       "\t<tr><td>2021-06-18</td><td>vector</td><td>latest</td><td> 4948</td></tr>\n",
       "\t<tr><td>2021-06-18</td><td>vector</td><td>legacy</td><td>52140</td></tr>\n",
       "\t<tr><td>2021-06-19</td><td>NULL  </td><td>NULL  </td><td>    4</td></tr>\n",
       "\t<tr><td>2021-06-19</td><td>vector</td><td>latest</td><td> 4288</td></tr>\n",
       "\t<tr><td>2021-06-19</td><td>vector</td><td>legacy</td><td>42424</td></tr>\n",
       "\t<tr><td>2021-06-20</td><td>NULL  </td><td>NULL  </td><td>    2</td></tr>\n",
       "\t<tr><td>2021-06-20</td><td>vector</td><td>latest</td><td> 4618</td></tr>\n",
       "\t<tr><td>2021-06-20</td><td>vector</td><td>legacy</td><td>46535</td></tr>\n",
       "\t<tr><td>2021-06-21</td><td>vector</td><td>latest</td><td> 5528</td></tr>\n",
       "\t<tr><td>2021-06-21</td><td>vector</td><td>legacy</td><td>58187</td></tr>\n",
       "\t<tr><td>2021-06-22</td><td>vector</td><td>latest</td><td>23023</td></tr>\n",
       "\t<tr><td>2021-06-22</td><td>vector</td><td>legacy</td><td>57450</td></tr>\n",
       "\t<tr><td>2021-06-23</td><td>vector</td><td>latest</td><td>47755</td></tr>\n",
       "\t<tr><td>2021-06-23</td><td>vector</td><td>legacy</td><td>55941</td></tr>\n",
       "\t<tr><td>2021-06-24</td><td>vector</td><td>latest</td><td>51067</td></tr>\n",
       "\t<tr><td>2021-06-24</td><td>vector</td><td>legacy</td><td>55742</td></tr>\n",
       "\t<tr><td>2021-06-25</td><td>NULL  </td><td>NULL  </td><td>    1</td></tr>\n",
       "\t<tr><td>2021-06-25</td><td>vector</td><td>latest</td><td>48562</td></tr>\n",
       "\t<tr><td>2021-06-25</td><td>vector</td><td>legacy</td><td>50041</td></tr>\n",
       "\t<tr><td>2021-06-26</td><td>vector</td><td>latest</td><td>44291</td></tr>\n",
       "\t<tr><td>2021-06-26</td><td>vector</td><td>legacy</td><td>41258</td></tr>\n",
       "\t<tr><td>2021-06-27</td><td>vector</td><td>latest</td><td>51986</td></tr>\n",
       "\t<tr><td>2021-06-27</td><td>vector</td><td>legacy</td><td>44501</td></tr>\n",
       "\t<tr><td>2021-06-28</td><td>NULL  </td><td>NULL  </td><td>    1</td></tr>\n",
       "\t<tr><td>2021-06-28</td><td>vector</td><td>latest</td><td>68627</td></tr>\n",
       "\t<tr><td>2021-06-28</td><td>vector</td><td>legacy</td><td>54833</td></tr>\n",
       "\t<tr><td>2021-06-29</td><td>vector</td><td>latest</td><td>23540</td></tr>\n",
       "\t<tr><td>2021-06-29</td><td>vector</td><td>legacy</td><td>22065</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 50 × 4\n",
       "\\begin{tabular}{llll}\n",
       " date & skin & skinversion & n\\_sessions\\\\\n",
       " <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t 2021-06-11 & NULL   & NULL   &    18\\\\\n",
       "\t 2021-06-11 & vector & latest &  5467\\\\\n",
       "\t 2021-06-11 & vector & legacy & 54874\\\\\n",
       "\t 2021-06-12 & NULL   & NULL   &    14\\\\\n",
       "\t 2021-06-12 & vector & latest &  4363\\\\\n",
       "\t 2021-06-12 & vector & legacy & 45448\\\\\n",
       "\t 2021-06-13 & NULL   & NULL   &     5\\\\\n",
       "\t 2021-06-13 & vector & latest &  5053\\\\\n",
       "\t 2021-06-13 & vector & legacy & 49477\\\\\n",
       "\t 2021-06-14 & NULL   & NULL   &     9\\\\\n",
       "\t 2021-06-14 & vector & latest &  5926\\\\\n",
       "\t 2021-06-14 & vector & legacy & 61784\\\\\n",
       "\t 2021-06-15 & NULL   & NULL   &     6\\\\\n",
       "\t 2021-06-15 & vector & latest &  5952\\\\\n",
       "\t 2021-06-15 & vector & legacy & 61043\\\\\n",
       "\t 2021-06-16 & NULL   & NULL   &     2\\\\\n",
       "\t 2021-06-16 & vector & latest &  5725\\\\\n",
       "\t 2021-06-16 & vector & legacy & 58951\\\\\n",
       "\t 2021-06-17 & NULL   & NULL   &     1\\\\\n",
       "\t 2021-06-17 & vector & latest &  5622\\\\\n",
       "\t 2021-06-17 & vector & legacy & 57772\\\\\n",
       "\t 2021-06-18 & NULL   & NULL   &     2\\\\\n",
       "\t 2021-06-18 & vector & latest &  4948\\\\\n",
       "\t 2021-06-18 & vector & legacy & 52140\\\\\n",
       "\t 2021-06-19 & NULL   & NULL   &     4\\\\\n",
       "\t 2021-06-19 & vector & latest &  4288\\\\\n",
       "\t 2021-06-19 & vector & legacy & 42424\\\\\n",
       "\t 2021-06-20 & NULL   & NULL   &     2\\\\\n",
       "\t 2021-06-20 & vector & latest &  4618\\\\\n",
       "\t 2021-06-20 & vector & legacy & 46535\\\\\n",
       "\t 2021-06-21 & vector & latest &  5528\\\\\n",
       "\t 2021-06-21 & vector & legacy & 58187\\\\\n",
       "\t 2021-06-22 & vector & latest & 23023\\\\\n",
       "\t 2021-06-22 & vector & legacy & 57450\\\\\n",
       "\t 2021-06-23 & vector & latest & 47755\\\\\n",
       "\t 2021-06-23 & vector & legacy & 55941\\\\\n",
       "\t 2021-06-24 & vector & latest & 51067\\\\\n",
       "\t 2021-06-24 & vector & legacy & 55742\\\\\n",
       "\t 2021-06-25 & NULL   & NULL   &     1\\\\\n",
       "\t 2021-06-25 & vector & latest & 48562\\\\\n",
       "\t 2021-06-25 & vector & legacy & 50041\\\\\n",
       "\t 2021-06-26 & vector & latest & 44291\\\\\n",
       "\t 2021-06-26 & vector & legacy & 41258\\\\\n",
       "\t 2021-06-27 & vector & latest & 51986\\\\\n",
       "\t 2021-06-27 & vector & legacy & 44501\\\\\n",
       "\t 2021-06-28 & NULL   & NULL   &     1\\\\\n",
       "\t 2021-06-28 & vector & latest & 68627\\\\\n",
       "\t 2021-06-28 & vector & legacy & 54833\\\\\n",
       "\t 2021-06-29 & vector & latest & 23540\\\\\n",
       "\t 2021-06-29 & vector & legacy & 22065\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 50 × 4\n",
       "\n",
       "| date &lt;chr&gt; | skin &lt;chr&gt; | skinversion &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| 2021-06-11 | NULL   | NULL   |    18 |\n",
       "| 2021-06-11 | vector | latest |  5467 |\n",
       "| 2021-06-11 | vector | legacy | 54874 |\n",
       "| 2021-06-12 | NULL   | NULL   |    14 |\n",
       "| 2021-06-12 | vector | latest |  4363 |\n",
       "| 2021-06-12 | vector | legacy | 45448 |\n",
       "| 2021-06-13 | NULL   | NULL   |     5 |\n",
       "| 2021-06-13 | vector | latest |  5053 |\n",
       "| 2021-06-13 | vector | legacy | 49477 |\n",
       "| 2021-06-14 | NULL   | NULL   |     9 |\n",
       "| 2021-06-14 | vector | latest |  5926 |\n",
       "| 2021-06-14 | vector | legacy | 61784 |\n",
       "| 2021-06-15 | NULL   | NULL   |     6 |\n",
       "| 2021-06-15 | vector | latest |  5952 |\n",
       "| 2021-06-15 | vector | legacy | 61043 |\n",
       "| 2021-06-16 | NULL   | NULL   |     2 |\n",
       "| 2021-06-16 | vector | latest |  5725 |\n",
       "| 2021-06-16 | vector | legacy | 58951 |\n",
       "| 2021-06-17 | NULL   | NULL   |     1 |\n",
       "| 2021-06-17 | vector | latest |  5622 |\n",
       "| 2021-06-17 | vector | legacy | 57772 |\n",
       "| 2021-06-18 | NULL   | NULL   |     2 |\n",
       "| 2021-06-18 | vector | latest |  4948 |\n",
       "| 2021-06-18 | vector | legacy | 52140 |\n",
       "| 2021-06-19 | NULL   | NULL   |     4 |\n",
       "| 2021-06-19 | vector | latest |  4288 |\n",
       "| 2021-06-19 | vector | legacy | 42424 |\n",
       "| 2021-06-20 | NULL   | NULL   |     2 |\n",
       "| 2021-06-20 | vector | latest |  4618 |\n",
       "| 2021-06-20 | vector | legacy | 46535 |\n",
       "| 2021-06-21 | vector | latest |  5528 |\n",
       "| 2021-06-21 | vector | legacy | 58187 |\n",
       "| 2021-06-22 | vector | latest | 23023 |\n",
       "| 2021-06-22 | vector | legacy | 57450 |\n",
       "| 2021-06-23 | vector | latest | 47755 |\n",
       "| 2021-06-23 | vector | legacy | 55941 |\n",
       "| 2021-06-24 | vector | latest | 51067 |\n",
       "| 2021-06-24 | vector | legacy | 55742 |\n",
       "| 2021-06-25 | NULL   | NULL   |     1 |\n",
       "| 2021-06-25 | vector | latest | 48562 |\n",
       "| 2021-06-25 | vector | legacy | 50041 |\n",
       "| 2021-06-26 | vector | latest | 44291 |\n",
       "| 2021-06-26 | vector | legacy | 41258 |\n",
       "| 2021-06-27 | vector | latest | 51986 |\n",
       "| 2021-06-27 | vector | legacy | 44501 |\n",
       "| 2021-06-28 | NULL   | NULL   |     1 |\n",
       "| 2021-06-28 | vector | latest | 68627 |\n",
       "| 2021-06-28 | vector | legacy | 54833 |\n",
       "| 2021-06-29 | vector | latest | 23540 |\n",
       "| 2021-06-29 | vector | legacy | 22065 |\n",
       "\n"
      ],
      "text/plain": [
       "   date       skin   skinversion n_sessions\n",
       "1  2021-06-11 NULL   NULL           18     \n",
       "2  2021-06-11 vector latest       5467     \n",
       "3  2021-06-11 vector legacy      54874     \n",
       "4  2021-06-12 NULL   NULL           14     \n",
       "5  2021-06-12 vector latest       4363     \n",
       "6  2021-06-12 vector legacy      45448     \n",
       "7  2021-06-13 NULL   NULL            5     \n",
       "8  2021-06-13 vector latest       5053     \n",
       "9  2021-06-13 vector legacy      49477     \n",
       "10 2021-06-14 NULL   NULL            9     \n",
       "11 2021-06-14 vector latest       5926     \n",
       "12 2021-06-14 vector legacy      61784     \n",
       "13 2021-06-15 NULL   NULL            6     \n",
       "14 2021-06-15 vector latest       5952     \n",
       "15 2021-06-15 vector legacy      61043     \n",
       "16 2021-06-16 NULL   NULL            2     \n",
       "17 2021-06-16 vector latest       5725     \n",
       "18 2021-06-16 vector legacy      58951     \n",
       "19 2021-06-17 NULL   NULL            1     \n",
       "20 2021-06-17 vector latest       5622     \n",
       "21 2021-06-17 vector legacy      57772     \n",
       "22 2021-06-18 NULL   NULL            2     \n",
       "23 2021-06-18 vector latest       4948     \n",
       "24 2021-06-18 vector legacy      52140     \n",
       "25 2021-06-19 NULL   NULL            4     \n",
       "26 2021-06-19 vector latest       4288     \n",
       "27 2021-06-19 vector legacy      42424     \n",
       "28 2021-06-20 NULL   NULL            2     \n",
       "29 2021-06-20 vector latest       4618     \n",
       "30 2021-06-20 vector legacy      46535     \n",
       "31 2021-06-21 vector latest       5528     \n",
       "32 2021-06-21 vector legacy      58187     \n",
       "33 2021-06-22 vector latest      23023     \n",
       "34 2021-06-22 vector legacy      57450     \n",
       "35 2021-06-23 vector latest      47755     \n",
       "36 2021-06-23 vector legacy      55941     \n",
       "37 2021-06-24 vector latest      51067     \n",
       "38 2021-06-24 vector legacy      55742     \n",
       "39 2021-06-25 NULL   NULL            1     \n",
       "40 2021-06-25 vector latest      48562     \n",
       "41 2021-06-25 vector legacy      50041     \n",
       "42 2021-06-26 vector latest      44291     \n",
       "43 2021-06-26 vector legacy      41258     \n",
       "44 2021-06-27 vector latest      51986     \n",
       "45 2021-06-27 vector legacy      44501     \n",
       "46 2021-06-28 NULL   NULL            1     \n",
       "47 2021-06-28 vector latest      68627     \n",
       "48 2021-06-28 vector legacy      54833     \n",
       "49 2021-06-29 vector latest      23540     \n",
       "50 2021-06-29 vector legacy      22065     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_events_byskin <- lang_button_events %>%\n",
    "    filter(date >= '2021-06-11') %>% # date fix deployed\n",
    "    group_by(date, skin, skinversion) %>%\n",
    "    summarise(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_switch_events_byskin "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are recording two skin types: 'NULL and 'vector' and recording both skin version types for vector:  'latest' and 'legacy'. I'm assuming NULL counts as all skin types not as vector. It's not needed for the AB test but I recommend revising to clarify the specific non-vector skin types at some point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "# By Context Field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'context', 'skin' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 4 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>context</th><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>vector</td><td>latest</td><td>325736</td></tr>\n",
       "\t<tr><td>NULL  </td><td>NULL  </td><td>NULL  </td><td>    65</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>latest</td><td> 62458</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>legacy</td><td>939722</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 4 × 4\n",
       "\\begin{tabular}{llll}\n",
       " context & skin & skinversion & n\\_sessions\\\\\n",
       " <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t header & vector & latest & 325736\\\\\n",
       "\t NULL   & NULL   & NULL   &     65\\\\\n",
       "\t other  & vector & latest &  62458\\\\\n",
       "\t other  & vector & legacy & 939722\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 4 × 4\n",
       "\n",
       "| context &lt;chr&gt; | skin &lt;chr&gt; | skinversion &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| header | vector | latest | 325736 |\n",
       "| NULL   | NULL   | NULL   |     65 |\n",
       "| other  | vector | latest |  62458 |\n",
       "| other  | vector | legacy | 939722 |\n",
       "\n"
      ],
      "text/plain": [
       "  context skin   skinversion n_sessions\n",
       "1 header  vector latest      325736    \n",
       "2 NULL    NULL   NULL            65    \n",
       "3 other   vector latest       62458    \n",
       "4 other   vector legacy      939722    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_events_bycontext <- lang_button_events %>%\n",
    "    filter(date >= '2021-06-11') %>% # date fix deployed\n",
    "    group_by(context, skin, skinversion) %>%\n",
    "    summarise(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_switch_events_bycontext "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Confirmed that with new instrumentation we can differentiate the following event types:\n",
    "  * People on vector selecting language button on top of the page:  `event.action = 'compact-language-links-open'`; `event.context = 'header'`; `event.skinVersion = 'latest'`\n",
    "  * People on vector that are in the control bucket clicking the \" N more\" button in the sidebar: `event.action = 'compact-language-links-open'` and `event.context = 'other'`, `event.skinVersion = 'latest'`\n",
    "  * People that have opted out of modern vector clicking the more button: `event.context = 'NULL'`; `event.skinVersion = 'NULL'`\n",
    "\n",
    "\n",
    "We are only recording clicks to the new button in the header on the latest vector as expected. \n",
    "\n",
    "There are clicks to the N More button by both legacy and latest as expected. The new language button was not deployed to logged-in users on test wikis so they would still have clicks to the N more button on the latest vector. When the AB test run, only users in the control group will be able to access the N more button on the latest vector.\n",
    "\n",
    "Note context value events are only set for user on vector; however, we can identify click from other skins to the N More button by the lack of values (`event.context = NULL`)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By logged-in and logged-out status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'context', 'skin', 'skinversion' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 8 × 5</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>context</th><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>isanon</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>vector</td><td>latest</td><td>false</td><td> 10298</td></tr>\n",
       "\t<tr><td>header</td><td>vector</td><td>latest</td><td>true </td><td>315691</td></tr>\n",
       "\t<tr><td>NULL  </td><td>NULL  </td><td>NULL  </td><td>false</td><td>     1</td></tr>\n",
       "\t<tr><td>NULL  </td><td>NULL  </td><td>NULL  </td><td>true </td><td>    64</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>latest</td><td>false</td><td>  2226</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>latest</td><td>true </td><td> 60257</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>legacy</td><td>false</td><td> 24755</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>legacy</td><td>true </td><td>915463</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 8 × 5\n",
       "\\begin{tabular}{lllll}\n",
       " context & skin & skinversion & isanon & n\\_sessions\\\\\n",
       " <chr> & <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t header & vector & latest & false &  10298\\\\\n",
       "\t header & vector & latest & true  & 315691\\\\\n",
       "\t NULL   & NULL   & NULL   & false &      1\\\\\n",
       "\t NULL   & NULL   & NULL   & true  &     64\\\\\n",
       "\t other  & vector & latest & false &   2226\\\\\n",
       "\t other  & vector & latest & true  &  60257\\\\\n",
       "\t other  & vector & legacy & false &  24755\\\\\n",
       "\t other  & vector & legacy & true  & 915463\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 8 × 5\n",
       "\n",
       "| context &lt;chr&gt; | skin &lt;chr&gt; | skinversion &lt;chr&gt; | isanon &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|---|\n",
       "| header | vector | latest | false |  10298 |\n",
       "| header | vector | latest | true  | 315691 |\n",
       "| NULL   | NULL   | NULL   | false |      1 |\n",
       "| NULL   | NULL   | NULL   | true  |     64 |\n",
       "| other  | vector | latest | false |   2226 |\n",
       "| other  | vector | latest | true  |  60257 |\n",
       "| other  | vector | legacy | false |  24755 |\n",
       "| other  | vector | legacy | true  | 915463 |\n",
       "\n"
      ],
      "text/plain": [
       "  context skin   skinversion isanon n_sessions\n",
       "1 header  vector latest      false   10298    \n",
       "2 header  vector latest      true   315691    \n",
       "3 NULL    NULL   NULL        false       1    \n",
       "4 NULL    NULL   NULL        true       64    \n",
       "5 other   vector latest      false    2226    \n",
       "6 other   vector latest      true    60257    \n",
       "7 other   vector legacy      false   24755    \n",
       "8 other   vector legacy      true   915463    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_events_byusertype <- lang_button_events %>%\n",
    "    filter(date >= '2021-06-11') %>% # date fix deployed\n",
    "    group_by(context, skin, skinversion, isanon) %>%\n",
    "    summarise(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_switch_events_byusertype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Totals by Anon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>isanon</th><th scope=col>n_events</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>false</td><td>11648</td><td>3378</td></tr>\n",
       "\t<tr><td>true </td><td>    9</td><td>   5</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 3\n",
       "\\begin{tabular}{lll}\n",
       " isanon & n\\_events & n\\_sessions\\\\\n",
       " <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t false & 11648 & 3378\\\\\n",
       "\t true  &     9 &    5\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 3\n",
       "\n",
       "| isanon &lt;chr&gt; | n_events &lt;int&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| false | 11648 | 3378 |\n",
       "| true  |     9 |    5 |\n",
       "\n"
      ],
      "text/plain": [
       "  isanon n_events n_sessions\n",
       "1 false  11648    3378      \n",
       "2 true       9       5      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_events_byusertype_all <- lang_button_events %>%\n",
    "    filter(date < '2021-06-22',\n",
    "          context == 'header') %>% # date fix deployed\n",
    "    group_by(isanon) %>%\n",
    "    summarize(n_events = sum(n_events),\n",
    "             n_sessions = n_distinct(web_session_id)) \n",
    "\n",
    "head(lang_button_events_byusertype_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirmed we are only recording clicks to the new language button by logged-in users. All new button clicks recorded so far have been on latest vector. \n",
    "\n",
    "Update: AB Test deployed on 22 June 2021. We start seeing a lot more logged-out users with clicks to the new button after that date."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Investigate logged-out new lang button events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A data.frame: 3 × 13</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>date</th><th scope=col>wiki</th><th scope=col>web_session_id</th><th scope=col>usereditbucket</th><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>timetochangelanguage</th><th scope=col>isanon</th><th scope=col>context</th><th scope=col>interfacelanguage</th><th scope=col>contentlanguage</th><th scope=col>selectedinterfacelanguage</th><th scope=col>n_events</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>2021-06-11</td><td>testwiki</td><td>bc29f4fc89d59f4bc99c</td><td>NULL</td><td>vector</td><td>latest</td><td>NULL</td><td>true</td><td>header</td><td>en</td><td>en</td><td>NULL</td><td>1</td></tr>\n",
       "\t<tr><td>2021-06-11</td><td>bnwiki  </td><td>4fa77beb78338661d19c</td><td>NULL</td><td>vector</td><td>latest</td><td>NULL</td><td>true</td><td>header</td><td>bn</td><td>bn</td><td>NULL</td><td>1</td></tr>\n",
       "\t<tr><td>2021-06-13</td><td>testwiki</td><td>1c87469b4eb16ead0427</td><td>NULL</td><td>vector</td><td>latest</td><td>NULL</td><td>true</td><td>header</td><td>en</td><td>en</td><td>NULL</td><td>3</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A data.frame: 3 × 13\n",
       "\\begin{tabular}{lllllllllllll}\n",
       " date & wiki & web\\_session\\_id & usereditbucket & skin & skinversion & timetochangelanguage & isanon & context & interfacelanguage & contentlanguage & selectedinterfacelanguage & n\\_events\\\\\n",
       " <chr> & <chr> & <chr> & <chr> & <chr> & <chr> & <chr> & <chr> & <chr> & <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t 2021-06-11 & testwiki & bc29f4fc89d59f4bc99c & NULL & vector & latest & NULL & true & header & en & en & NULL & 1\\\\\n",
       "\t 2021-06-11 & bnwiki   & 4fa77beb78338661d19c & NULL & vector & latest & NULL & true & header & bn & bn & NULL & 1\\\\\n",
       "\t 2021-06-13 & testwiki & 1c87469b4eb16ead0427 & NULL & vector & latest & NULL & true & header & en & en & NULL & 3\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A data.frame: 3 × 13\n",
       "\n",
       "| date &lt;chr&gt; | wiki &lt;chr&gt; | web_session_id &lt;chr&gt; | usereditbucket &lt;chr&gt; | skin &lt;chr&gt; | skinversion &lt;chr&gt; | timetochangelanguage &lt;chr&gt; | isanon &lt;chr&gt; | context &lt;chr&gt; | interfacelanguage &lt;chr&gt; | contentlanguage &lt;chr&gt; | selectedinterfacelanguage &lt;chr&gt; | n_events &lt;int&gt; |\n",
       "|---|---|---|---|---|---|---|---|---|---|---|---|---|\n",
       "| 2021-06-11 | testwiki | bc29f4fc89d59f4bc99c | NULL | vector | latest | NULL | true | header | en | en | NULL | 1 |\n",
       "| 2021-06-11 | bnwiki   | 4fa77beb78338661d19c | NULL | vector | latest | NULL | true | header | bn | bn | NULL | 1 |\n",
       "| 2021-06-13 | testwiki | 1c87469b4eb16ead0427 | NULL | vector | latest | NULL | true | header | en | en | NULL | 3 |\n",
       "\n"
      ],
      "text/plain": [
       "  date       wiki     web_session_id       usereditbucket skin   skinversion\n",
       "1 2021-06-11 testwiki bc29f4fc89d59f4bc99c NULL           vector latest     \n",
       "2 2021-06-11 bnwiki   4fa77beb78338661d19c NULL           vector latest     \n",
       "3 2021-06-13 testwiki 1c87469b4eb16ead0427 NULL           vector latest     \n",
       "  timetochangelanguage isanon context interfacelanguage contentlanguage\n",
       "1 NULL                 true   header  en                en             \n",
       "2 NULL                 true   header  bn                bn             \n",
       "3 NULL                 true   header  en                en             \n",
       "  selectedinterfacelanguage n_events\n",
       "1 NULL                      1       \n",
       "2 NULL                      1       \n",
       "3 NULL                      3       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_events_isanon <- lang_button_events %>%\n",
    "    filter(date >= '2021-06-11',\n",
    "          context == 'header',\n",
    "        isanon == \"true\")\n",
    "\n",
    "lang_button_events_isanon "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Two of these events occured on testwiki and one on bwiki, which is one of the early adopter wikis.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# By Test Wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'context', 'skin', 'skinversion' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 5 × 5</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>context</th><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>isanon</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>vector</td><td>latest</td><td>false</td><td>  4743</td></tr>\n",
       "\t<tr><td>header</td><td>vector</td><td>latest</td><td>true </td><td>258299</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>latest</td><td>false</td><td>   450</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>latest</td><td>true </td><td>  3781</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>legacy</td><td>false</td><td>   165</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 5 × 5\n",
       "\\begin{tabular}{lllll}\n",
       " context & skin & skinversion & isanon & n\\_sessions\\\\\n",
       " <chr> & <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t header & vector & latest & false &   4743\\\\\n",
       "\t header & vector & latest & true  & 258299\\\\\n",
       "\t other  & vector & latest & false &    450\\\\\n",
       "\t other  & vector & latest & true  &   3781\\\\\n",
       "\t other  & vector & legacy & false &    165\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 5 × 5\n",
       "\n",
       "| context &lt;chr&gt; | skin &lt;chr&gt; | skinversion &lt;chr&gt; | isanon &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|---|\n",
       "| header | vector | latest | false |   4743 |\n",
       "| header | vector | latest | true  | 258299 |\n",
       "| other  | vector | latest | false |    450 |\n",
       "| other  | vector | latest | true  |   3781 |\n",
       "| other  | vector | legacy | false |    165 |\n",
       "\n"
      ],
      "text/plain": [
       "  context skin   skinversion isanon n_sessions\n",
       "1 header  vector latest      false    4743    \n",
       "2 header  vector latest      true   258299    \n",
       "3 other   vector latest      false     450    \n",
       "4 other   vector latest      true     3781    \n",
       "5 other   vector legacy      false     165    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_events_byisanon_testwiki <- lang_button_events %>%\n",
    "    filter(wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' ), \n",
    "        date >= '2021-06-24') %>% # following AB test\n",
    "    group_by(context, skin, skinversion, isanon) %>%\n",
    "    summarise(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_switch_events_byisanon_testwiki"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numbers appear as expected. There are a limited number of events recorded on legacy vector since the new skin is deployed as opt-out.\n",
    "\n",
    "We're still seeing some clicks to the N More button on the latest vector. Not sure where these are coming frmo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# By Non Test Wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'context', 'skin', 'skinversion' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 4 × 6</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>context</th><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>isanon</th><th scope=col>n_sessions</th><th scope=col>n_events</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>vector</td><td>latest</td><td>false</td><td>4252</td><td>15217</td></tr>\n",
       "\t<tr><td>header</td><td>vector</td><td>latest</td><td>true </td><td>  62</td><td>  107</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>latest</td><td>false</td><td>   3</td><td>    3</td></tr>\n",
       "\t<tr><td>other </td><td>vector</td><td>latest</td><td>true </td><td>  19</td><td>   24</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 4 × 6\n",
       "\\begin{tabular}{llllll}\n",
       " context & skin & skinversion & isanon & n\\_sessions & n\\_events\\\\\n",
       " <chr> & <chr> & <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t header & vector & latest & false & 4252 & 15217\\\\\n",
       "\t header & vector & latest & true  &   62 &   107\\\\\n",
       "\t other  & vector & latest & false &    3 &     3\\\\\n",
       "\t other  & vector & latest & true  &   19 &    24\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 4 × 6\n",
       "\n",
       "| context &lt;chr&gt; | skin &lt;chr&gt; | skinversion &lt;chr&gt; | isanon &lt;chr&gt; | n_sessions &lt;int&gt; | n_events &lt;int&gt; |\n",
       "|---|---|---|---|---|---|\n",
       "| header | vector | latest | false | 4252 | 15217 |\n",
       "| header | vector | latest | true  |   62 |   107 |\n",
       "| other  | vector | latest | false |    3 |     3 |\n",
       "| other  | vector | latest | true  |   19 |    24 |\n",
       "\n"
      ],
      "text/plain": [
       "  context skin   skinversion isanon n_sessions n_events\n",
       "1 header  vector latest      false  4252       15217   \n",
       "2 header  vector latest      true     62         107   \n",
       "3 other   vector latest      false     3           3   \n",
       "4 other   vector latest      true     19          24   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_events_bynontest <- lang_button_events %>%\n",
    "    filter(skinversion == 'latest',\n",
    "           skin == 'vector',\n",
    "           date >= '2021-06-11',\n",
    "        !wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' ), \n",
    "        date >= '2021-06-08') %>% # date fix deployed\n",
    "    group_by(context, skin, skinversion, isanon) %>%\n",
    "    summarise(n_sessions = n_distinct(web_session_id),\n",
    "             n_events = sum(n_events))\n",
    "\n",
    "lang_button_switch_events_bynontest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On non-test wikis, we should only be recording clicks to the new button by logged-in users. There are two events by logged-out users which occurs on test wiki, see above.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Content Language Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'interfacelanguage', 'contentlanguage' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>interfacelanguage</th><th scope=col>contentlanguage</th><th scope=col>n_sessions</th><th scope=col>pct_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>en   </td><td>en</td><td>85</td><td>0.29513889</td></tr>\n",
       "\t<tr><td>ja   </td><td>ja</td><td>20</td><td>0.06944444</td></tr>\n",
       "\t<tr><td>zh-tw</td><td>zh</td><td>20</td><td>0.06944444</td></tr>\n",
       "\t<tr><td>de   </td><td>de</td><td>14</td><td>0.04861111</td></tr>\n",
       "\t<tr><td>ru   </td><td>ru</td><td>12</td><td>0.04166667</td></tr>\n",
       "\t<tr><td>ca   </td><td>ca</td><td>11</td><td>0.03819444</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 6 × 4\n",
       "\\begin{tabular}{llll}\n",
       " interfacelanguage & contentlanguage & n\\_sessions & pct\\_sessions\\\\\n",
       " <chr> & <chr> & <int> & <dbl>\\\\\n",
       "\\hline\n",
       "\t en    & en & 85 & 0.29513889\\\\\n",
       "\t ja    & ja & 20 & 0.06944444\\\\\n",
       "\t zh-tw & zh & 20 & 0.06944444\\\\\n",
       "\t de    & de & 14 & 0.04861111\\\\\n",
       "\t ru    & ru & 12 & 0.04166667\\\\\n",
       "\t ca    & ca & 11 & 0.03819444\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 4\n",
       "\n",
       "| interfacelanguage &lt;chr&gt; | contentlanguage &lt;chr&gt; | n_sessions &lt;int&gt; | pct_sessions &lt;dbl&gt; |\n",
       "|---|---|---|---|\n",
       "| en    | en | 85 | 0.29513889 |\n",
       "| ja    | ja | 20 | 0.06944444 |\n",
       "| zh-tw | zh | 20 | 0.06944444 |\n",
       "| de    | de | 14 | 0.04861111 |\n",
       "| ru    | ru | 12 | 0.04166667 |\n",
       "| ca    | ca | 11 | 0.03819444 |\n",
       "\n"
      ],
      "text/plain": [
       "  interfacelanguage contentlanguage n_sessions pct_sessions\n",
       "1 en                en              85         0.29513889  \n",
       "2 ja                ja              20         0.06944444  \n",
       "3 zh-tw             zh              20         0.06944444  \n",
       "4 de                de              14         0.04861111  \n",
       "5 ru                ru              12         0.04166667  \n",
       "6 ca                ca              11         0.03819444  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "top_content_languages <- lang_button_events %>%\n",
    "    filter(date >= '2021-06-11',\n",
    "              context == 'header') %>%\n",
    "    mutate(all_sessions = n_distinct(web_session_id)) %>%\n",
    "    group_by(interfacelanguage, contentlanguage) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id),\n",
    "             pct_sessions = n_sessions/all_sessions) %>%\n",
    "    distinct() %>%\n",
    "    arrange(desc(n_sessions))\n",
    "\n",
    "head(top_content_languages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No test wikis and larger size non-test wikis are listed as the top content langauges where a new search button was clicked."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Edit Count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'usereditbucket' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 6 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>usereditbucket</th><th scope=col>isanon</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>0 edits      </td><td>false</td><td> 93</td></tr>\n",
       "\t<tr><td>1-4 edits    </td><td>false</td><td> 64</td></tr>\n",
       "\t<tr><td>100-999 edits</td><td>false</td><td>135</td></tr>\n",
       "\t<tr><td>1000+ edits  </td><td>false</td><td>360</td></tr>\n",
       "\t<tr><td>5-99 edits   </td><td>false</td><td>181</td></tr>\n",
       "\t<tr><td>NULL         </td><td>true </td><td>  3</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 6 × 3\n",
       "\\begin{tabular}{lll}\n",
       " usereditbucket & isanon & n\\_sessions\\\\\n",
       " <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t 0 edits       & false &  93\\\\\n",
       "\t 1-4 edits     & false &  64\\\\\n",
       "\t 100-999 edits & false & 135\\\\\n",
       "\t 1000+ edits   & false & 360\\\\\n",
       "\t 5-99 edits    & false & 181\\\\\n",
       "\t NULL          & true  &   3\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 6 × 3\n",
       "\n",
       "| usereditbucket &lt;chr&gt; | isanon &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| 0 edits       | false |  93 |\n",
       "| 1-4 edits     | false |  64 |\n",
       "| 100-999 edits | false | 135 |\n",
       "| 1000+ edits   | false | 360 |\n",
       "| 5-99 edits    | false | 181 |\n",
       "| NULL          | true  |   3 |\n",
       "\n"
      ],
      "text/plain": [
       "  usereditbucket isanon n_sessions\n",
       "1 0 edits        false   93       \n",
       "2 1-4 edits      false   64       \n",
       "3 100-999 edits  false  135       \n",
       "4 1000+ edits    false  360       \n",
       "5 5-99 edits     false  181       \n",
       "6 NULL           true     3       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "lang_button_byeditcount <- lang_button_events %>%\n",
    "    filter(date >= '2021-06-11',\n",
    "              context == 'header') %>%\n",
    "    group_by(usereditbucket, isanon) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_button_byeditcount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three NULL events recorded. These are all for the logged-out user events with clicks to the header identified above.\n",
    "\n",
    "There appears to be some type of bug with these instances. Need to identify more info associated with these events to see if we can isolate what's happening here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check that you can determine language switches from sessions with clicks to new button"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 433,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rough query to confirm approach\n",
    "# will be refine in analysis\n",
    "query <- \n",
    "\"\n",
    "-- sessions where lang button was selected\n",
    "WITH button AS (\n",
    "SELECT\n",
    "    MIN(TO_DATE(dt)) as button_date,\n",
    "    event.web_session_id as session_id,\n",
    "    event.skinVersion as skinversion,\n",
    "    event.isAnon As isAnon,\n",
    "    event.context as button_type,\n",
    "    wiki as wiki\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month = 06\n",
    "    AND Day >= 23\n",
    "    AND Day <= 30\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.action = 'compact-language-links-open'\n",
    "GROUP BY \n",
    "    event.web_session_id,\n",
    "    event.context,\n",
    "    event.isAnon,\n",
    "    event.skinversion,\n",
    "    wiki\n",
    "),\n",
    "\n",
    "lang_switches AS (\n",
    "  SELECT\n",
    "    TO_DATE(dt) as switch_date,\n",
    "    event.web_session_id as session_id,\n",
    "    event.context as switch_context,\n",
    "    event.timetochangelanguage as switch_time,\n",
    "    wiki as wiki\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month = 06\n",
    "    AND day >= 23\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.action = 'language-change'\n",
    ")\n",
    "\n",
    "SELECT\n",
    "    button.button_date,\n",
    "    button.button_type,\n",
    "    button.skinversion,\n",
    "    button.isAnon,\n",
    "    lang_switches.switch_time,\n",
    "    lang_switches.switch_date,\n",
    "    button.session_id,\n",
    "    button.wiki,\n",
    "-- sessions with lang switch that occured after button clicks\n",
    "    IF(lang_switches.session_id IS NOT NULL, 1, 0) AS language_switch,\n",
    "    lang_switches.switch_context\n",
    "FROM button\n",
    "LEFT JOIN lang_switches ON\n",
    "    button.session_id = lang_switches.session_id\n",
    "    \n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 434,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lang_button_switch_events <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 435,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'button_type', 'wiki' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 20 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>button_type</th><th scope=col>wiki</th><th scope=col>content-language-switcher</th><th scope=col>interface</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>bnwiki      </td><td>  49</td><td>NA</td></tr>\n",
       "\t<tr><td>header</td><td>euwiki      </td><td> 105</td><td> 1</td></tr>\n",
       "\t<tr><td>header</td><td>frwiki      </td><td>2903</td><td> 1</td></tr>\n",
       "\t<tr><td>header</td><td>frwiktionary</td><td>  67</td><td>NA</td></tr>\n",
       "\t<tr><td>header</td><td>hewiki      </td><td> 660</td><td>NA</td></tr>\n",
       "\t<tr><td>header</td><td>kowiki      </td><td> 341</td><td> 4</td></tr>\n",
       "\t<tr><td>header</td><td>ptwiki      </td><td> 928</td><td> 1</td></tr>\n",
       "\t<tr><td>header</td><td>srwiki      </td><td> 135</td><td>NA</td></tr>\n",
       "\t<tr><td>header</td><td>trwiki      </td><td> 358</td><td> 1</td></tr>\n",
       "\t<tr><td>header</td><td>vecwiki     </td><td>   1</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>bnwiki      </td><td>   3</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>euwiki      </td><td>   8</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>frwiki      </td><td> 149</td><td> 1</td></tr>\n",
       "\t<tr><td>other </td><td>frwiktionary</td><td>   4</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>hewiki      </td><td>  15</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>kowiki      </td><td>  11</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>ptwiki      </td><td>  50</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>srwiki      </td><td>  35</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>trwiki      </td><td>  30</td><td>NA</td></tr>\n",
       "\t<tr><td>other </td><td>vecwiki     </td><td>   1</td><td>NA</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 20 × 4\n",
       "\\begin{tabular}{llll}\n",
       " button\\_type & wiki & content-language-switcher & interface\\\\\n",
       " <chr> & <chr> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t header & bnwiki       &   49 & NA\\\\\n",
       "\t header & euwiki       &  105 &  1\\\\\n",
       "\t header & frwiki       & 2903 &  1\\\\\n",
       "\t header & frwiktionary &   67 & NA\\\\\n",
       "\t header & hewiki       &  660 & NA\\\\\n",
       "\t header & kowiki       &  341 &  4\\\\\n",
       "\t header & ptwiki       &  928 &  1\\\\\n",
       "\t header & srwiki       &  135 & NA\\\\\n",
       "\t header & trwiki       &  358 &  1\\\\\n",
       "\t header & vecwiki      &    1 & NA\\\\\n",
       "\t other  & bnwiki       &    3 & NA\\\\\n",
       "\t other  & euwiki       &    8 & NA\\\\\n",
       "\t other  & frwiki       &  149 &  1\\\\\n",
       "\t other  & frwiktionary &    4 & NA\\\\\n",
       "\t other  & hewiki       &   15 & NA\\\\\n",
       "\t other  & kowiki       &   11 & NA\\\\\n",
       "\t other  & ptwiki       &   50 & NA\\\\\n",
       "\t other  & srwiki       &   35 & NA\\\\\n",
       "\t other  & trwiki       &   30 & NA\\\\\n",
       "\t other  & vecwiki      &    1 & NA\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 20 × 4\n",
       "\n",
       "| button_type &lt;chr&gt; | wiki &lt;chr&gt; | content-language-switcher &lt;int&gt; | interface &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| header | bnwiki       |   49 | NA |\n",
       "| header | euwiki       |  105 |  1 |\n",
       "| header | frwiki       | 2903 |  1 |\n",
       "| header | frwiktionary |   67 | NA |\n",
       "| header | hewiki       |  660 | NA |\n",
       "| header | kowiki       |  341 |  4 |\n",
       "| header | ptwiki       |  928 |  1 |\n",
       "| header | srwiki       |  135 | NA |\n",
       "| header | trwiki       |  358 |  1 |\n",
       "| header | vecwiki      |    1 | NA |\n",
       "| other  | bnwiki       |    3 | NA |\n",
       "| other  | euwiki       |    8 | NA |\n",
       "| other  | frwiki       |  149 |  1 |\n",
       "| other  | frwiktionary |    4 | NA |\n",
       "| other  | hewiki       |   15 | NA |\n",
       "| other  | kowiki       |   11 | NA |\n",
       "| other  | ptwiki       |   50 | NA |\n",
       "| other  | srwiki       |   35 | NA |\n",
       "| other  | trwiki       |   30 | NA |\n",
       "| other  | vecwiki      |    1 | NA |\n",
       "\n"
      ],
      "text/plain": [
       "   button_type wiki         content-language-switcher interface\n",
       "1  header      bnwiki         49                      NA       \n",
       "2  header      euwiki        105                       1       \n",
       "3  header      frwiki       2903                       1       \n",
       "4  header      frwiktionary   67                      NA       \n",
       "5  header      hewiki        660                      NA       \n",
       "6  header      kowiki        341                       4       \n",
       "7  header      ptwiki        928                       1       \n",
       "8  header      srwiki        135                      NA       \n",
       "9  header      trwiki        358                       1       \n",
       "10 header      vecwiki         1                      NA       \n",
       "11 other       bnwiki          3                      NA       \n",
       "12 other       euwiki          8                      NA       \n",
       "13 other       frwiki        149                       1       \n",
       "14 other       frwiktionary    4                      NA       \n",
       "15 other       hewiki         15                      NA       \n",
       "16 other       kowiki         11                      NA       \n",
       "17 other       ptwiki         50                      NA       \n",
       "18 other       srwiki         35                      NA       \n",
       "19 other       trwiki         30                      NA       \n",
       "20 other       vecwiki         1                      NA       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_bybuttontype <- lang_button_switch_events %>%\n",
    "    filter(\n",
    "          language_switch == 1, #only sessions with lang switch\n",
    "          wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' ),\n",
    "          skinversion == 'latest',\n",
    "          isanon == 'false',\n",
    "          switch_context %in% c('content-language-switcher', 'interface')) %>% \n",
    "    group_by(button_type, wiki, switch_context)  %>%\n",
    "    summarise(n_sessions = n_distinct(session_id))  %>%\n",
    "    spread(switch_context, n_sessions)\n",
    "\n",
    "lang_button_switch_bybuttontype"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Since deployment of the fix on 11 June 2021, were are now recording language switches that occur after the user clicks the new button. On 22 June 2021, context fields were added to differentiate click to change the display setting (event.context = interface) from clicks to languages.\n",
    "\n",
    "Data above restricted to only AB sessions. There are more langauge switches in the treatment group (button_type = header) but that's because the data for the control group is missing clicks to the language links in the sidebar. I'll run a query and add those numbers to confirm.\n",
    "\n",
    "ISSUES:\n",
    "* There are  language list selections associated with sessions where a header was clicked. Need to investigate further.\n",
    "\n",
    "Notes: \n",
    "*A NULL button_type means the user clicked from a skin type other than vector.\n",
    "* The language-list event is only recorded for direct clicks to links in the sidebar and does not require selecting the NMore button. The language-list sessions listed above include sessions where both a button was clicked and a user clicked the lang list in the sidebar.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lang Link Clicks to Sidebar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 438,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "query <- \n",
    "\n",
    "\"\n",
    "SELECT DISTINCT\n",
    "    event.web_session_id as session_id,\n",
    "    wiki as wiki\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month = 06\n",
    "-- first full day of events\n",
    "    AND Day >= 23\n",
    "    AND DAy <= 30\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.isAnon = false\n",
    "    AND wiki IN ('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki')\n",
    "    AND event.action = 'language-change'\n",
    "    AND event.context = 'languages-list'\n",
    "    AND event.skinversion = 'latest'\n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 439,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lang_link_sidebar <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 440,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 11 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>wiki</th><th scope=col>sidebar</th><th scope=col>button_type</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;chr&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>bnwiki      </td><td>  99</td><td>other</td></tr>\n",
       "\t<tr><td>dewikivoyage</td><td>   5</td><td>other</td></tr>\n",
       "\t<tr><td>euwiki      </td><td> 108</td><td>other</td></tr>\n",
       "\t<tr><td>frwiki      </td><td>6069</td><td>other</td></tr>\n",
       "\t<tr><td>frwiktionary</td><td> 176</td><td>other</td></tr>\n",
       "\t<tr><td>hewiki      </td><td> 986</td><td>other</td></tr>\n",
       "\t<tr><td>kowiki      </td><td> 627</td><td>other</td></tr>\n",
       "\t<tr><td>ptwiki      </td><td>1886</td><td>other</td></tr>\n",
       "\t<tr><td>srwiki      </td><td> 388</td><td>other</td></tr>\n",
       "\t<tr><td>trwiki      </td><td> 806</td><td>other</td></tr>\n",
       "\t<tr><td>vecwiki     </td><td>  39</td><td>other</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 11 × 3\n",
       "\\begin{tabular}{lll}\n",
       " wiki & sidebar & button\\_type\\\\\n",
       " <chr> & <int> & <chr>\\\\\n",
       "\\hline\n",
       "\t bnwiki       &   99 & other\\\\\n",
       "\t dewikivoyage &    5 & other\\\\\n",
       "\t euwiki       &  108 & other\\\\\n",
       "\t frwiki       & 6069 & other\\\\\n",
       "\t frwiktionary &  176 & other\\\\\n",
       "\t hewiki       &  986 & other\\\\\n",
       "\t kowiki       &  627 & other\\\\\n",
       "\t ptwiki       & 1886 & other\\\\\n",
       "\t srwiki       &  388 & other\\\\\n",
       "\t trwiki       &  806 & other\\\\\n",
       "\t vecwiki      &   39 & other\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 11 × 3\n",
       "\n",
       "| wiki &lt;chr&gt; | sidebar &lt;int&gt; | button_type &lt;chr&gt; |\n",
       "|---|---|---|\n",
       "| bnwiki       |   99 | other |\n",
       "| dewikivoyage |    5 | other |\n",
       "| euwiki       |  108 | other |\n",
       "| frwiki       | 6069 | other |\n",
       "| frwiktionary |  176 | other |\n",
       "| hewiki       |  986 | other |\n",
       "| kowiki       |  627 | other |\n",
       "| ptwiki       | 1886 | other |\n",
       "| srwiki       |  388 | other |\n",
       "| trwiki       |  806 | other |\n",
       "| vecwiki      |   39 | other |\n",
       "\n"
      ],
      "text/plain": [
       "   wiki         sidebar button_type\n",
       "1  bnwiki         99    other      \n",
       "2  dewikivoyage    5    other      \n",
       "3  euwiki        108    other      \n",
       "4  frwiki       6069    other      \n",
       "5  frwiktionary  176    other      \n",
       "6  hewiki        986    other      \n",
       "7  kowiki        627    other      \n",
       "8  ptwiki       1886    other      \n",
       "9  srwiki        388    other      \n",
       "10 trwiki        806    other      \n",
       "11 vecwiki        39    other      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_link_sidebar_bywiki <- lang_link_sidebar %>%\n",
    "    group_by(wiki)  %>%\n",
    "    summarise(sidebar = n_distinct(session_id),\n",
    "             button_type = 'other') # add column to specific this is the control group (button type will be other)\n",
    "\n",
    "lang_link_sidebar_bywiki"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll merge to have all lang switches together in the same location. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 21 × 5</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>button_type</th><th scope=col>wiki</th><th scope=col>content-language-switcher</th><th scope=col>interface</th><th scope=col>sidebar</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>bnwiki      </td><td>  49</td><td>NA</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>euwiki      </td><td> 105</td><td> 1</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>frwiki      </td><td>2903</td><td> 1</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>frwiktionary</td><td>  67</td><td>NA</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>hewiki      </td><td> 660</td><td>NA</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>kowiki      </td><td> 341</td><td> 4</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>ptwiki      </td><td> 928</td><td> 1</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>srwiki      </td><td> 135</td><td>NA</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>trwiki      </td><td> 358</td><td> 1</td><td>  NA</td></tr>\n",
       "\t<tr><td>header</td><td>vecwiki     </td><td>   1</td><td>NA</td><td>  NA</td></tr>\n",
       "\t<tr><td>other </td><td>bnwiki      </td><td>   3</td><td>NA</td><td>  99</td></tr>\n",
       "\t<tr><td>other </td><td>euwiki      </td><td>   8</td><td>NA</td><td> 108</td></tr>\n",
       "\t<tr><td>other </td><td>frwiki      </td><td> 149</td><td> 1</td><td>6069</td></tr>\n",
       "\t<tr><td>other </td><td>frwiktionary</td><td>   4</td><td>NA</td><td> 176</td></tr>\n",
       "\t<tr><td>other </td><td>hewiki      </td><td>  15</td><td>NA</td><td> 986</td></tr>\n",
       "\t<tr><td>other </td><td>kowiki      </td><td>  11</td><td>NA</td><td> 627</td></tr>\n",
       "\t<tr><td>other </td><td>ptwiki      </td><td>  50</td><td>NA</td><td>1886</td></tr>\n",
       "\t<tr><td>other </td><td>srwiki      </td><td>  35</td><td>NA</td><td> 388</td></tr>\n",
       "\t<tr><td>other </td><td>trwiki      </td><td>  30</td><td>NA</td><td> 806</td></tr>\n",
       "\t<tr><td>other </td><td>vecwiki     </td><td>   1</td><td>NA</td><td>  39</td></tr>\n",
       "\t<tr><td>other </td><td>dewikivoyage</td><td>  NA</td><td>NA</td><td>   5</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 21 × 5\n",
       "\\begin{tabular}{lllll}\n",
       " button\\_type & wiki & content-language-switcher & interface & sidebar\\\\\n",
       " <chr> & <chr> & <int> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t header & bnwiki       &   49 & NA &   NA\\\\\n",
       "\t header & euwiki       &  105 &  1 &   NA\\\\\n",
       "\t header & frwiki       & 2903 &  1 &   NA\\\\\n",
       "\t header & frwiktionary &   67 & NA &   NA\\\\\n",
       "\t header & hewiki       &  660 & NA &   NA\\\\\n",
       "\t header & kowiki       &  341 &  4 &   NA\\\\\n",
       "\t header & ptwiki       &  928 &  1 &   NA\\\\\n",
       "\t header & srwiki       &  135 & NA &   NA\\\\\n",
       "\t header & trwiki       &  358 &  1 &   NA\\\\\n",
       "\t header & vecwiki      &    1 & NA &   NA\\\\\n",
       "\t other  & bnwiki       &    3 & NA &   99\\\\\n",
       "\t other  & euwiki       &    8 & NA &  108\\\\\n",
       "\t other  & frwiki       &  149 &  1 & 6069\\\\\n",
       "\t other  & frwiktionary &    4 & NA &  176\\\\\n",
       "\t other  & hewiki       &   15 & NA &  986\\\\\n",
       "\t other  & kowiki       &   11 & NA &  627\\\\\n",
       "\t other  & ptwiki       &   50 & NA & 1886\\\\\n",
       "\t other  & srwiki       &   35 & NA &  388\\\\\n",
       "\t other  & trwiki       &   30 & NA &  806\\\\\n",
       "\t other  & vecwiki      &    1 & NA &   39\\\\\n",
       "\t other  & dewikivoyage &   NA & NA &    5\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 21 × 5\n",
       "\n",
       "| button_type &lt;chr&gt; | wiki &lt;chr&gt; | content-language-switcher &lt;int&gt; | interface &lt;int&gt; | sidebar &lt;int&gt; |\n",
       "|---|---|---|---|---|\n",
       "| header | bnwiki       |   49 | NA |   NA |\n",
       "| header | euwiki       |  105 |  1 |   NA |\n",
       "| header | frwiki       | 2903 |  1 |   NA |\n",
       "| header | frwiktionary |   67 | NA |   NA |\n",
       "| header | hewiki       |  660 | NA |   NA |\n",
       "| header | kowiki       |  341 |  4 |   NA |\n",
       "| header | ptwiki       |  928 |  1 |   NA |\n",
       "| header | srwiki       |  135 | NA |   NA |\n",
       "| header | trwiki       |  358 |  1 |   NA |\n",
       "| header | vecwiki      |    1 | NA |   NA |\n",
       "| other  | bnwiki       |    3 | NA |   99 |\n",
       "| other  | euwiki       |    8 | NA |  108 |\n",
       "| other  | frwiki       |  149 |  1 | 6069 |\n",
       "| other  | frwiktionary |    4 | NA |  176 |\n",
       "| other  | hewiki       |   15 | NA |  986 |\n",
       "| other  | kowiki       |   11 | NA |  627 |\n",
       "| other  | ptwiki       |   50 | NA | 1886 |\n",
       "| other  | srwiki       |   35 | NA |  388 |\n",
       "| other  | trwiki       |   30 | NA |  806 |\n",
       "| other  | vecwiki      |    1 | NA |   39 |\n",
       "| other  | dewikivoyage |   NA | NA |    5 |\n",
       "\n"
      ],
      "text/plain": [
       "   button_type wiki         content-language-switcher interface sidebar\n",
       "1  header      bnwiki         49                      NA          NA   \n",
       "2  header      euwiki        105                       1          NA   \n",
       "3  header      frwiki       2903                       1          NA   \n",
       "4  header      frwiktionary   67                      NA          NA   \n",
       "5  header      hewiki        660                      NA          NA   \n",
       "6  header      kowiki        341                       4          NA   \n",
       "7  header      ptwiki        928                       1          NA   \n",
       "8  header      srwiki        135                      NA          NA   \n",
       "9  header      trwiki        358                       1          NA   \n",
       "10 header      vecwiki         1                      NA          NA   \n",
       "11 other       bnwiki          3                      NA          99   \n",
       "12 other       euwiki          8                      NA         108   \n",
       "13 other       frwiki        149                       1        6069   \n",
       "14 other       frwiktionary    4                      NA         176   \n",
       "15 other       hewiki         15                      NA         986   \n",
       "16 other       kowiki         11                      NA         627   \n",
       "17 other       ptwiki         50                      NA        1886   \n",
       "18 other       srwiki         35                      NA         388   \n",
       "19 other       trwiki         30                      NA         806   \n",
       "20 other       vecwiki         1                      NA          39   \n",
       "21 other       dewikivoyage   NA                      NA           5   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_switched_AB <- full_join(lang_button_switch_bybuttontype, lang_link_sidebar_bywiki, \n",
    "                               by = c('wiki','button_type')\n",
    ")\n",
    "\n",
    "lang_switched_AB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lang Button Switches by Skin Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'button_type', 'switch_context', 'language_switch' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 10 × 5</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>button_type</th><th scope=col>switch_context</th><th scope=col>language_switch</th><th scope=col>skinversion</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>interface     </td><td>1</td><td>latest</td><td>  614</td></tr>\n",
       "\t<tr><td>header</td><td>NULL          </td><td>0</td><td>latest</td><td>  220</td></tr>\n",
       "\t<tr><td>NULL  </td><td>interface     </td><td>1</td><td>NULL  </td><td>    3</td></tr>\n",
       "\t<tr><td>NULL  </td><td>NULL          </td><td>0</td><td>NULL  </td><td>   36</td></tr>\n",
       "\t<tr><td>other </td><td>interface     </td><td>1</td><td>latest</td><td> 8197</td></tr>\n",
       "\t<tr><td>other </td><td>interface     </td><td>1</td><td>legacy</td><td>71475</td></tr>\n",
       "\t<tr><td>other </td><td>languages-list</td><td>1</td><td>latest</td><td> 5604</td></tr>\n",
       "\t<tr><td>other </td><td>languages-list</td><td>1</td><td>legacy</td><td>    6</td></tr>\n",
       "\t<tr><td>other </td><td>NULL          </td><td>0</td><td>latest</td><td> 3076</td></tr>\n",
       "\t<tr><td>other </td><td>NULL          </td><td>0</td><td>legacy</td><td>77735</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 10 × 5\n",
       "\\begin{tabular}{lllll}\n",
       " button\\_type & switch\\_context & language\\_switch & skinversion & n\\_sessions\\\\\n",
       " <chr> & <chr> & <int> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t header & interface      & 1 & latest &   614\\\\\n",
       "\t header & NULL           & 0 & latest &   220\\\\\n",
       "\t NULL   & interface      & 1 & NULL   &     3\\\\\n",
       "\t NULL   & NULL           & 0 & NULL   &    36\\\\\n",
       "\t other  & interface      & 1 & latest &  8197\\\\\n",
       "\t other  & interface      & 1 & legacy & 71475\\\\\n",
       "\t other  & languages-list & 1 & latest &  5604\\\\\n",
       "\t other  & languages-list & 1 & legacy &     6\\\\\n",
       "\t other  & NULL           & 0 & latest &  3076\\\\\n",
       "\t other  & NULL           & 0 & legacy & 77735\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 10 × 5\n",
       "\n",
       "| button_type &lt;chr&gt; | switch_context &lt;chr&gt; | language_switch &lt;int&gt; | skinversion &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|---|\n",
       "| header | interface      | 1 | latest |   614 |\n",
       "| header | NULL           | 0 | latest |   220 |\n",
       "| NULL   | interface      | 1 | NULL   |     3 |\n",
       "| NULL   | NULL           | 0 | NULL   |    36 |\n",
       "| other  | interface      | 1 | latest |  8197 |\n",
       "| other  | interface      | 1 | legacy | 71475 |\n",
       "| other  | languages-list | 1 | latest |  5604 |\n",
       "| other  | languages-list | 1 | legacy |     6 |\n",
       "| other  | NULL           | 0 | latest |  3076 |\n",
       "| other  | NULL           | 0 | legacy | 77735 |\n",
       "\n"
      ],
      "text/plain": [
       "   button_type switch_context language_switch skinversion n_sessions\n",
       "1  header      interface      1               latest        614     \n",
       "2  header      NULL           0               latest        220     \n",
       "3  NULL        interface      1               NULL            3     \n",
       "4  NULL        NULL           0               NULL           36     \n",
       "5  other       interface      1               latest       8197     \n",
       "6  other       interface      1               legacy      71475     \n",
       "7  other       languages-list 1               latest       5604     \n",
       "8  other       languages-list 1               legacy          6     \n",
       "9  other       NULL           0               latest       3076     \n",
       "10 other       NULL           0               legacy      77735     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_byskintype <- lang_button_switch_events %>%\n",
    "    group_by(button_type, switch_context, language_switch, skinversion)  %>%\n",
    "    summarise(n_sessions = n_distinct(session_id))\n",
    "\n",
    "lang_button_switch_byskintype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time to Change Language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'button_type', 'switch_context' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 28230 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>button_type</th><th scope=col>switch_context</th><th scope=col>switch_time</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>header</td><td>interface</td><td>10017.0           </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10019.0           </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10033.900146484375</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>100522.89990234375</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10064.39990234375 </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10088.89990234375 </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10102.900146484375</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>101213.60009765625</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10155.900146484375</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10161.5           </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10190.39990234375 </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10196.400146484375</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10205.800048828125</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10209.300048828125</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10210.0           </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10220.400146484375</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10240.39990234375 </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10260.2451171875  </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10282.699951171875</td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>102956.0          </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10310.7001953125  </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10314.89990234375 </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10350.10009765625 </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10404.0           </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10410.0           </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10436.0           </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10448.2998046875  </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10474.0           </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10489.30029296875 </td><td>1</td></tr>\n",
       "\t<tr><td>header</td><td>interface</td><td>10501.0           </td><td>1</td></tr>\n",
       "\t<tr><td>⋮</td><td>⋮</td><td>⋮</td><td>⋮</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9946.900146484375</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9948.900146484375</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9949.5           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9952.300048828125</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9955.0           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9956.2998046875  </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9958.199951171875</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>99629.5          </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9964.800048828125</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9965.0           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>99685.19995117188</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9969.0           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9969.10009765625 </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9970.0           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9970.7998046875  </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9972.0           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>997265.0         </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>99734.0          </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9977.119873046875</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9979.800048828125</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>998.0            </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9981.800048828125</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>998152.0         </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9982.39990234375 </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9988.0           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9989.0           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9992.400146484375</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>99969.30053710938</td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9997.0           </td><td>1</td></tr>\n",
       "\t<tr><td>other</td><td>languages-list</td><td>9999.0           </td><td>1</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 28230 × 4\n",
       "\\begin{tabular}{llll}\n",
       " button\\_type & switch\\_context & switch\\_time & n\\_sessions\\\\\n",
       " <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t header & interface & 10017.0            & 1\\\\\n",
       "\t header & interface & 10019.0            & 1\\\\\n",
       "\t header & interface & 10033.900146484375 & 1\\\\\n",
       "\t header & interface & 100522.89990234375 & 1\\\\\n",
       "\t header & interface & 10064.39990234375  & 1\\\\\n",
       "\t header & interface & 10088.89990234375  & 1\\\\\n",
       "\t header & interface & 10102.900146484375 & 1\\\\\n",
       "\t header & interface & 101213.60009765625 & 1\\\\\n",
       "\t header & interface & 10155.900146484375 & 1\\\\\n",
       "\t header & interface & 10161.5            & 1\\\\\n",
       "\t header & interface & 10190.39990234375  & 1\\\\\n",
       "\t header & interface & 10196.400146484375 & 1\\\\\n",
       "\t header & interface & 10205.800048828125 & 1\\\\\n",
       "\t header & interface & 10209.300048828125 & 1\\\\\n",
       "\t header & interface & 10210.0            & 1\\\\\n",
       "\t header & interface & 10220.400146484375 & 1\\\\\n",
       "\t header & interface & 10240.39990234375  & 1\\\\\n",
       "\t header & interface & 10260.2451171875   & 1\\\\\n",
       "\t header & interface & 10282.699951171875 & 1\\\\\n",
       "\t header & interface & 102956.0           & 1\\\\\n",
       "\t header & interface & 10310.7001953125   & 1\\\\\n",
       "\t header & interface & 10314.89990234375  & 1\\\\\n",
       "\t header & interface & 10350.10009765625  & 1\\\\\n",
       "\t header & interface & 10404.0            & 1\\\\\n",
       "\t header & interface & 10410.0            & 1\\\\\n",
       "\t header & interface & 10436.0            & 1\\\\\n",
       "\t header & interface & 10448.2998046875   & 1\\\\\n",
       "\t header & interface & 10474.0            & 1\\\\\n",
       "\t header & interface & 10489.30029296875  & 1\\\\\n",
       "\t header & interface & 10501.0            & 1\\\\\n",
       "\t ⋮ & ⋮ & ⋮ & ⋮\\\\\n",
       "\t other & languages-list & 9946.900146484375 & 1\\\\\n",
       "\t other & languages-list & 9948.900146484375 & 1\\\\\n",
       "\t other & languages-list & 9949.5            & 1\\\\\n",
       "\t other & languages-list & 9952.300048828125 & 1\\\\\n",
       "\t other & languages-list & 9955.0            & 1\\\\\n",
       "\t other & languages-list & 9956.2998046875   & 1\\\\\n",
       "\t other & languages-list & 9958.199951171875 & 1\\\\\n",
       "\t other & languages-list & 99629.5           & 1\\\\\n",
       "\t other & languages-list & 9964.800048828125 & 1\\\\\n",
       "\t other & languages-list & 9965.0            & 1\\\\\n",
       "\t other & languages-list & 99685.19995117188 & 1\\\\\n",
       "\t other & languages-list & 9969.0            & 1\\\\\n",
       "\t other & languages-list & 9969.10009765625  & 1\\\\\n",
       "\t other & languages-list & 9970.0            & 1\\\\\n",
       "\t other & languages-list & 9970.7998046875   & 1\\\\\n",
       "\t other & languages-list & 9972.0            & 1\\\\\n",
       "\t other & languages-list & 997265.0          & 1\\\\\n",
       "\t other & languages-list & 99734.0           & 1\\\\\n",
       "\t other & languages-list & 9977.119873046875 & 1\\\\\n",
       "\t other & languages-list & 9979.800048828125 & 1\\\\\n",
       "\t other & languages-list & 998.0             & 1\\\\\n",
       "\t other & languages-list & 9981.800048828125 & 1\\\\\n",
       "\t other & languages-list & 998152.0          & 1\\\\\n",
       "\t other & languages-list & 9982.39990234375  & 1\\\\\n",
       "\t other & languages-list & 9988.0            & 1\\\\\n",
       "\t other & languages-list & 9989.0            & 1\\\\\n",
       "\t other & languages-list & 9992.400146484375 & 1\\\\\n",
       "\t other & languages-list & 99969.30053710938 & 1\\\\\n",
       "\t other & languages-list & 9997.0            & 1\\\\\n",
       "\t other & languages-list & 9999.0            & 1\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 28230 × 4\n",
       "\n",
       "| button_type &lt;chr&gt; | switch_context &lt;chr&gt; | switch_time &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| header | interface | 10017.0            | 1 |\n",
       "| header | interface | 10019.0            | 1 |\n",
       "| header | interface | 10033.900146484375 | 1 |\n",
       "| header | interface | 100522.89990234375 | 1 |\n",
       "| header | interface | 10064.39990234375  | 1 |\n",
       "| header | interface | 10088.89990234375  | 1 |\n",
       "| header | interface | 10102.900146484375 | 1 |\n",
       "| header | interface | 101213.60009765625 | 1 |\n",
       "| header | interface | 10155.900146484375 | 1 |\n",
       "| header | interface | 10161.5            | 1 |\n",
       "| header | interface | 10190.39990234375  | 1 |\n",
       "| header | interface | 10196.400146484375 | 1 |\n",
       "| header | interface | 10205.800048828125 | 1 |\n",
       "| header | interface | 10209.300048828125 | 1 |\n",
       "| header | interface | 10210.0            | 1 |\n",
       "| header | interface | 10220.400146484375 | 1 |\n",
       "| header | interface | 10240.39990234375  | 1 |\n",
       "| header | interface | 10260.2451171875   | 1 |\n",
       "| header | interface | 10282.699951171875 | 1 |\n",
       "| header | interface | 102956.0           | 1 |\n",
       "| header | interface | 10310.7001953125   | 1 |\n",
       "| header | interface | 10314.89990234375  | 1 |\n",
       "| header | interface | 10350.10009765625  | 1 |\n",
       "| header | interface | 10404.0            | 1 |\n",
       "| header | interface | 10410.0            | 1 |\n",
       "| header | interface | 10436.0            | 1 |\n",
       "| header | interface | 10448.2998046875   | 1 |\n",
       "| header | interface | 10474.0            | 1 |\n",
       "| header | interface | 10489.30029296875  | 1 |\n",
       "| header | interface | 10501.0            | 1 |\n",
       "| ⋮ | ⋮ | ⋮ | ⋮ |\n",
       "| other | languages-list | 9946.900146484375 | 1 |\n",
       "| other | languages-list | 9948.900146484375 | 1 |\n",
       "| other | languages-list | 9949.5            | 1 |\n",
       "| other | languages-list | 9952.300048828125 | 1 |\n",
       "| other | languages-list | 9955.0            | 1 |\n",
       "| other | languages-list | 9956.2998046875   | 1 |\n",
       "| other | languages-list | 9958.199951171875 | 1 |\n",
       "| other | languages-list | 99629.5           | 1 |\n",
       "| other | languages-list | 9964.800048828125 | 1 |\n",
       "| other | languages-list | 9965.0            | 1 |\n",
       "| other | languages-list | 99685.19995117188 | 1 |\n",
       "| other | languages-list | 9969.0            | 1 |\n",
       "| other | languages-list | 9969.10009765625  | 1 |\n",
       "| other | languages-list | 9970.0            | 1 |\n",
       "| other | languages-list | 9970.7998046875   | 1 |\n",
       "| other | languages-list | 9972.0            | 1 |\n",
       "| other | languages-list | 997265.0          | 1 |\n",
       "| other | languages-list | 99734.0           | 1 |\n",
       "| other | languages-list | 9977.119873046875 | 1 |\n",
       "| other | languages-list | 9979.800048828125 | 1 |\n",
       "| other | languages-list | 998.0             | 1 |\n",
       "| other | languages-list | 9981.800048828125 | 1 |\n",
       "| other | languages-list | 998152.0          | 1 |\n",
       "| other | languages-list | 9982.39990234375  | 1 |\n",
       "| other | languages-list | 9988.0            | 1 |\n",
       "| other | languages-list | 9989.0            | 1 |\n",
       "| other | languages-list | 9992.400146484375 | 1 |\n",
       "| other | languages-list | 99969.30053710938 | 1 |\n",
       "| other | languages-list | 9997.0            | 1 |\n",
       "| other | languages-list | 9999.0            | 1 |\n",
       "\n"
      ],
      "text/plain": [
       "      button_type switch_context switch_time        n_sessions\n",
       "1     header      interface      10017.0            1         \n",
       "2     header      interface      10019.0            1         \n",
       "3     header      interface      10033.900146484375 1         \n",
       "4     header      interface      100522.89990234375 1         \n",
       "5     header      interface      10064.39990234375  1         \n",
       "6     header      interface      10088.89990234375  1         \n",
       "7     header      interface      10102.900146484375 1         \n",
       "8     header      interface      101213.60009765625 1         \n",
       "9     header      interface      10155.900146484375 1         \n",
       "10    header      interface      10161.5            1         \n",
       "11    header      interface      10190.39990234375  1         \n",
       "12    header      interface      10196.400146484375 1         \n",
       "13    header      interface      10205.800048828125 1         \n",
       "14    header      interface      10209.300048828125 1         \n",
       "15    header      interface      10210.0            1         \n",
       "16    header      interface      10220.400146484375 1         \n",
       "17    header      interface      10240.39990234375  1         \n",
       "18    header      interface      10260.2451171875   1         \n",
       "19    header      interface      10282.699951171875 1         \n",
       "20    header      interface      102956.0           1         \n",
       "21    header      interface      10310.7001953125   1         \n",
       "22    header      interface      10314.89990234375  1         \n",
       "23    header      interface      10350.10009765625  1         \n",
       "24    header      interface      10404.0            1         \n",
       "25    header      interface      10410.0            1         \n",
       "26    header      interface      10436.0            1         \n",
       "27    header      interface      10448.2998046875   1         \n",
       "28    header      interface      10474.0            1         \n",
       "29    header      interface      10489.30029296875  1         \n",
       "30    header      interface      10501.0            1         \n",
       "⋮     ⋮           ⋮              ⋮                  ⋮         \n",
       "28201 other       languages-list 9946.900146484375  1         \n",
       "28202 other       languages-list 9948.900146484375  1         \n",
       "28203 other       languages-list 9949.5             1         \n",
       "28204 other       languages-list 9952.300048828125  1         \n",
       "28205 other       languages-list 9955.0             1         \n",
       "28206 other       languages-list 9956.2998046875    1         \n",
       "28207 other       languages-list 9958.199951171875  1         \n",
       "28208 other       languages-list 99629.5            1         \n",
       "28209 other       languages-list 9964.800048828125  1         \n",
       "28210 other       languages-list 9965.0             1         \n",
       "28211 other       languages-list 99685.19995117188  1         \n",
       "28212 other       languages-list 9969.0             1         \n",
       "28213 other       languages-list 9969.10009765625   1         \n",
       "28214 other       languages-list 9970.0             1         \n",
       "28215 other       languages-list 9970.7998046875    1         \n",
       "28216 other       languages-list 9972.0             1         \n",
       "28217 other       languages-list 997265.0           1         \n",
       "28218 other       languages-list 99734.0            1         \n",
       "28219 other       languages-list 9977.119873046875  1         \n",
       "28220 other       languages-list 9979.800048828125  1         \n",
       "28221 other       languages-list 998.0              1         \n",
       "28222 other       languages-list 9981.800048828125  1         \n",
       "28223 other       languages-list 998152.0           1         \n",
       "28224 other       languages-list 9982.39990234375   1         \n",
       "28225 other       languages-list 9988.0             1         \n",
       "28226 other       languages-list 9989.0             1         \n",
       "28227 other       languages-list 9992.400146484375  1         \n",
       "28228 other       languages-list 99969.30053710938  1         \n",
       "28229 other       languages-list 9997.0             1         \n",
       "28230 other       languages-list 9999.0             1         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_button_switch_bytime<- lang_button_switch_events  %>%\n",
    "    filter(language_switch == 1,\n",
    "           skinversion == \"latest\") %>%\n",
    "    group_by(button_type, switch_context, switch_time)  %>%\n",
    "    summarise(n_sessions = n_distinct(session_id))\n",
    "\n",
    "lang_button_switch_bytime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Confirmed time to change is recorded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check Sidebar Clicks with added skin context fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {},
   "outputs": [],
   "source": [
    "query <- \n",
    "\"\n",
    "SELECT\n",
    "    TO_DATE(dt) AS `date`,\n",
    "    wiki,\n",
    "    event.web_session_id,\n",
    "    event.usereditbucket,\n",
    "    event.skin,\n",
    "    event.skinVersion,\n",
    "    event.timetochangelanguage,\n",
    "    event.isanon,\n",
    "    event.action,\n",
    "    event.interfacelanguage,\n",
    "    event.contentlanguage,\n",
    "    event.selectedinterfacelanguage,\n",
    "    Count(*) AS n_events\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month = 06\n",
    "    AND day >= 11\n",
    "    AND event.context = 'languages-list'\n",
    "    AND useragent.is_bot = false\n",
    "GROUP BY\n",
    "    TO_DATE(dt),\n",
    "    wiki,\n",
    "    event.web_session_id,\n",
    "    event.usereditbucket,\n",
    "    event.skin,\n",
    "    event.skinVersion,\n",
    "    event.timetochangelanguage,\n",
    "    event.isanon,\n",
    "    event.action,\n",
    "    event.interfacelanguage,\n",
    "    event.contentlanguage,\n",
    "    event.selectedinterfacelanguage\n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lang_sidebar_events <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'skin', 'skinversion' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 4 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>isanon</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>NULL  </td><td>NULL  </td><td>false</td><td>      5</td></tr>\n",
       "\t<tr><td>NULL  </td><td>NULL  </td><td>true </td><td>    162</td></tr>\n",
       "\t<tr><td>vector</td><td>latest</td><td>false</td><td>  44738</td></tr>\n",
       "\t<tr><td>vector</td><td>latest</td><td>true </td><td>1658646</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 4 × 4\n",
       "\\begin{tabular}{llll}\n",
       " skin & skinversion & isanon & n\\_sessions\\\\\n",
       " <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t NULL   & NULL   & false &       5\\\\\n",
       "\t NULL   & NULL   & true  &     162\\\\\n",
       "\t vector & latest & false &   44738\\\\\n",
       "\t vector & latest & true  & 1658646\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 4 × 4\n",
       "\n",
       "| skin &lt;chr&gt; | skinversion &lt;chr&gt; | isanon &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| NULL   | NULL   | false |       5 |\n",
       "| NULL   | NULL   | true  |     162 |\n",
       "| vector | latest | false |   44738 |\n",
       "| vector | latest | true  | 1658646 |\n",
       "\n"
      ],
      "text/plain": [
       "  skin   skinversion isanon n_sessions\n",
       "1 NULL   NULL        false        5   \n",
       "2 NULL   NULL        true       162   \n",
       "3 vector latest      false    44738   \n",
       "4 vector latest      true   1658646   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_sidebar_events_byskintype<- lang_sidebar_events %>%\n",
    "    group_by(skin, skinversion, isanon) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_sidebar_events_byskintype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are recording clicks to the language list in the sidebar either the latest version of vector or NULL. This is expected as instrumentation for sidebar clicks was limited to language list. Further instrumentation here to clarfiy NULL values would be helpful but it is assumed that any events identifed with a NULL skin type came from non latest vector.\n",
    "\n",
    "This instrumentation will be good for the AB Test since we will only be looking users on the latest vector skin; however, further clarification of these other skin types will be useful in the future in case we want to know percent of users clicking on these links from legacy vector vs other skin types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By Non-Test Wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'skin', 'skinversion' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 2 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>isanon</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>vector</td><td>latest</td><td>false</td><td>5</td></tr>\n",
       "\t<tr><td>vector</td><td>latest</td><td>true </td><td>4</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 2 × 4\n",
       "\\begin{tabular}{llll}\n",
       " skin & skinversion & isanon & n\\_sessions\\\\\n",
       " <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t vector & latest & false & 5\\\\\n",
       "\t vector & latest & true  & 4\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 2 × 4\n",
       "\n",
       "| skin &lt;chr&gt; | skinversion &lt;chr&gt; | isanon &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| vector | latest | false | 5 |\n",
       "| vector | latest | true  | 4 |\n",
       "\n"
      ],
      "text/plain": [
       "  skin   skinversion isanon n_sessions\n",
       "1 vector latest      false  5         \n",
       "2 vector latest      true   4         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_sidebar_events_byskintype_nontestwiki <- lang_sidebar_events %>%\n",
    "    filter(!wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' )) %>%\n",
    "    group_by(skin, skinversion, isanon) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_sidebar_events_byskintype_nontestwiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## By Test Wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'skin', 'skinversion' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 4 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>skin</th><th scope=col>skinversion</th><th scope=col>isanon</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>NULL  </td><td>NULL  </td><td>false</td><td>      5</td></tr>\n",
       "\t<tr><td>NULL  </td><td>NULL  </td><td>true </td><td>    162</td></tr>\n",
       "\t<tr><td>vector</td><td>latest</td><td>false</td><td>  44680</td></tr>\n",
       "\t<tr><td>vector</td><td>latest</td><td>true </td><td>1658560</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 4 × 4\n",
       "\\begin{tabular}{llll}\n",
       " skin & skinversion & isanon & n\\_sessions\\\\\n",
       " <chr> & <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t NULL   & NULL   & false &       5\\\\\n",
       "\t NULL   & NULL   & true  &     162\\\\\n",
       "\t vector & latest & false &   44680\\\\\n",
       "\t vector & latest & true  & 1658560\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 4 × 4\n",
       "\n",
       "| skin &lt;chr&gt; | skinversion &lt;chr&gt; | isanon &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| NULL   | NULL   | false |       5 |\n",
       "| NULL   | NULL   | true  |     162 |\n",
       "| vector | latest | false |   44680 |\n",
       "| vector | latest | true  | 1658560 |\n",
       "\n"
      ],
      "text/plain": [
       "  skin   skinversion isanon n_sessions\n",
       "1 NULL   NULL        false        5   \n",
       "2 NULL   NULL        true       162   \n",
       "3 vector latest      false    44680   \n",
       "4 vector latest      true   1658560   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_sidebar_events_byskintype_testwiki <- lang_sidebar_events %>%\n",
    "    filter(wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' )) %>%\n",
    "    group_by(skin, skinversion, isanon) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_sidebar_events_byskintype_testwiki"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the AB test has not started, we are not logging a lot of clicks by logged in or logged out users to the lang links on the sidebar on non-test wikis (the new search button was deployed as to all logged-in users on latest vector). \n",
    "\n",
    "Note: There are some sessions by both logged-in and logged-out users on the latest vector recorded as having click a lang list link. Not sure where these are coming from but it's such a small percentage it should not impact the AB test.\n",
    "\n",
    "The majority of clicks occur on test wikis by logged-out as expected. \n",
    "\n",
    "Once the AB test runs, any clicks to the lang links in the sidebar should only be recorded by users in the control group. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By Action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 1 × 2</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>action</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>language-change</td><td>1702128</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 1 × 2\n",
       "\\begin{tabular}{ll}\n",
       " action & n\\_sessions\\\\\n",
       " <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t language-change & 1702128\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 1 × 2\n",
       "\n",
       "| action &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|\n",
       "| language-change | 1702128 |\n",
       "\n"
      ],
      "text/plain": [
       "  action          n_sessions\n",
       "1 language-change 1702128   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_sidebar_events_byaction<- lang_sidebar_events %>%\n",
    "    group_by(action) %>%\n",
    "    summarize(n_sessions = n_distinct(web_session_id))\n",
    "\n",
    "lang_sidebar_events_byaction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only associated with `event.action = 'language-change'` as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check any follow-up actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rough query to confirm approach\n",
    "# will be refine in analysis\n",
    "query <- \n",
    "\"\n",
    "-- sessions where lang button was selected\n",
    "WITH button AS (\n",
    "SELECT\n",
    "    MIN(TO_DATE(dt)) as button_date,\n",
    "    event.web_session_id as session_id,\n",
    "    event.skinVersion as skinversion,\n",
    "    event.context as button_type,\n",
    "    wiki as wiki\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month = 06\n",
    "    AND Day >= 11\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.action = 'compact-language-links-open'\n",
    "GROUP BY \n",
    "    event.web_session_id,\n",
    "    event.context,\n",
    "    event.skinversion,\n",
    "    wiki\n",
    "),\n",
    "\n",
    "follow_actions AS (\n",
    "  SELECT\n",
    "    TO_DATE(dt) as action_date,\n",
    "    event.action as action_type,\n",
    "    event.web_session_id as session_id,\n",
    "    event.context as action_context,\n",
    "    wiki as wiki\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month = 06\n",
    "    AND Day >= 11\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.action != 'compact-language-links-open'\n",
    ")\n",
    "\n",
    "SELECT\n",
    "    button.button_date,\n",
    "    button.button_type,\n",
    "    button.skinversion,\n",
    "    follow_actions.action_date,\n",
    "    follow_actions.action_type,\n",
    "    button.session_id,\n",
    "    button.wiki,\n",
    "-- sessions with lang switch that occured after button clicks\n",
    "    IF(follow_actions.session_id IS NOT NULL, 1, 0) AS follow_action,\n",
    "    follow_actions.action_context\n",
    "FROM button\n",
    "LEFT JOIN follow_actions ON\n",
    "    button.session_id = follow_actions.session_id\n",
    "    \n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n",
      "Warning message in scan(file = file, what = what, sep = sep, quote = quote, dec = dec, :\n",
      "“embedded nul(s) found in input”\n"
     ]
    }
   ],
   "source": [
    "more_lang_actions <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By Action Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 279,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 11 × 2</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>action_type</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>font-change          </td><td>    13</td></tr>\n",
       "\t<tr><td>ime-change           </td><td>   263</td></tr>\n",
       "\t<tr><td>ime-disable          </td><td>    52</td></tr>\n",
       "\t<tr><td>ime-enable           </td><td>   143</td></tr>\n",
       "\t<tr><td>language-change      </td><td>284961</td></tr>\n",
       "\t<tr><td>more-languages-access</td><td>    94</td></tr>\n",
       "\t<tr><td>no-search-results    </td><td>  4078</td></tr>\n",
       "\t<tr><td>NULL                 </td><td> 45822</td></tr>\n",
       "\t<tr><td>settings-open        </td><td>   198</td></tr>\n",
       "\t<tr><td>webfonts-disable     </td><td>     3</td></tr>\n",
       "\t<tr><td>webfonts-enable      </td><td>    29</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 11 × 2\n",
       "\\begin{tabular}{ll}\n",
       " action\\_type & n\\_sessions\\\\\n",
       " <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t font-change           &     13\\\\\n",
       "\t ime-change            &    263\\\\\n",
       "\t ime-disable           &     52\\\\\n",
       "\t ime-enable            &    143\\\\\n",
       "\t language-change       & 284961\\\\\n",
       "\t more-languages-access &     94\\\\\n",
       "\t no-search-results     &   4078\\\\\n",
       "\t NULL                  &  45822\\\\\n",
       "\t settings-open         &    198\\\\\n",
       "\t webfonts-disable      &      3\\\\\n",
       "\t webfonts-enable       &     29\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 11 × 2\n",
       "\n",
       "| action_type &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|\n",
       "| font-change           |     13 |\n",
       "| ime-change            |    263 |\n",
       "| ime-disable           |     52 |\n",
       "| ime-enable            |    143 |\n",
       "| language-change       | 284961 |\n",
       "| more-languages-access |     94 |\n",
       "| no-search-results     |   4078 |\n",
       "| NULL                  |  45822 |\n",
       "| settings-open         |    198 |\n",
       "| webfonts-disable      |      3 |\n",
       "| webfonts-enable       |     29 |\n",
       "\n"
      ],
      "text/plain": [
       "   action_type           n_sessions\n",
       "1  font-change               13    \n",
       "2  ime-change               263    \n",
       "3  ime-disable               52    \n",
       "4  ime-enable               143    \n",
       "5  language-change       284961    \n",
       "6  more-languages-access     94    \n",
       "7  no-search-results       4078    \n",
       "8  NULL                   45822    \n",
       "9  settings-open            198    \n",
       "10 webfonts-disable           3    \n",
       "11 webfonts-enable           29    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# find what types of actions are recorded with a new button clicks\n",
    "new_button_actions <- more_lang_actions %>%\n",
    "    filter(button_type == 'header') %>%\n",
    "    group_by(action_type) %>%\n",
    "    summarise (n_sessions = n_distinct(session_id))\n",
    "\n",
    "new_button_actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have recorded 'ime-change' (user changed the input method), 'no-search-results' (User searched for a language with no results) and 'webfonts-enable' (webfonts-enable: User enabled the webfonts functionality via ULS settings) actions in sessions where clicks to the new button were logged. \n",
    "\n",
    "With the fix, we are now also recording language-change events."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By Action Type and Button Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'action_type' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 20 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>action_type</th><th scope=col>button_type</th><th scope=col>n_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>ime-change           </td><td>header</td><td>    3</td></tr>\n",
       "\t<tr><td>language-change      </td><td>header</td><td>  219</td></tr>\n",
       "\t<tr><td>more-languages-access</td><td>header</td><td>    1</td></tr>\n",
       "\t<tr><td>no-search-results    </td><td>header</td><td>    8</td></tr>\n",
       "\t<tr><td>NULL                 </td><td>header</td><td>   79</td></tr>\n",
       "\t<tr><td>webfonts-enable      </td><td>header</td><td>    1</td></tr>\n",
       "\t<tr><td>language-change      </td><td>NULL  </td><td>    2</td></tr>\n",
       "\t<tr><td>no-search-results    </td><td>NULL  </td><td>    2</td></tr>\n",
       "\t<tr><td>NULL                 </td><td>NULL  </td><td>   13</td></tr>\n",
       "\t<tr><td>font-change          </td><td>other </td><td>    1</td></tr>\n",
       "\t<tr><td>ime-change           </td><td>other </td><td>   71</td></tr>\n",
       "\t<tr><td>ime-disable          </td><td>other </td><td>   24</td></tr>\n",
       "\t<tr><td>ime-enable           </td><td>other </td><td>   47</td></tr>\n",
       "\t<tr><td>language-change      </td><td>other </td><td>26266</td></tr>\n",
       "\t<tr><td>more-languages-access</td><td>other </td><td>   32</td></tr>\n",
       "\t<tr><td>no-search-results    </td><td>other </td><td> 4449</td></tr>\n",
       "\t<tr><td>NULL                 </td><td>other </td><td>22555</td></tr>\n",
       "\t<tr><td>settings-open        </td><td>other </td><td> 1471</td></tr>\n",
       "\t<tr><td>ui-lang-revert       </td><td>other </td><td>    1</td></tr>\n",
       "\t<tr><td>webfonts-enable      </td><td>other </td><td>    7</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 20 × 3\n",
       "\\begin{tabular}{lll}\n",
       " action\\_type & button\\_type & n\\_sessions\\\\\n",
       " <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t ime-change            & header &     3\\\\\n",
       "\t language-change       & header &   219\\\\\n",
       "\t more-languages-access & header &     1\\\\\n",
       "\t no-search-results     & header &     8\\\\\n",
       "\t NULL                  & header &    79\\\\\n",
       "\t webfonts-enable       & header &     1\\\\\n",
       "\t language-change       & NULL   &     2\\\\\n",
       "\t no-search-results     & NULL   &     2\\\\\n",
       "\t NULL                  & NULL   &    13\\\\\n",
       "\t font-change           & other  &     1\\\\\n",
       "\t ime-change            & other  &    71\\\\\n",
       "\t ime-disable           & other  &    24\\\\\n",
       "\t ime-enable            & other  &    47\\\\\n",
       "\t language-change       & other  & 26266\\\\\n",
       "\t more-languages-access & other  &    32\\\\\n",
       "\t no-search-results     & other  &  4449\\\\\n",
       "\t NULL                  & other  & 22555\\\\\n",
       "\t settings-open         & other  &  1471\\\\\n",
       "\t ui-lang-revert        & other  &     1\\\\\n",
       "\t webfonts-enable       & other  &     7\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 20 × 3\n",
       "\n",
       "| action_type &lt;chr&gt; | button_type &lt;chr&gt; | n_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| ime-change            | header |     3 |\n",
       "| language-change       | header |   219 |\n",
       "| more-languages-access | header |     1 |\n",
       "| no-search-results     | header |     8 |\n",
       "| NULL                  | header |    79 |\n",
       "| webfonts-enable       | header |     1 |\n",
       "| language-change       | NULL   |     2 |\n",
       "| no-search-results     | NULL   |     2 |\n",
       "| NULL                  | NULL   |    13 |\n",
       "| font-change           | other  |     1 |\n",
       "| ime-change            | other  |    71 |\n",
       "| ime-disable           | other  |    24 |\n",
       "| ime-enable            | other  |    47 |\n",
       "| language-change       | other  | 26266 |\n",
       "| more-languages-access | other  |    32 |\n",
       "| no-search-results     | other  |  4449 |\n",
       "| NULL                  | other  | 22555 |\n",
       "| settings-open         | other  |  1471 |\n",
       "| ui-lang-revert        | other  |     1 |\n",
       "| webfonts-enable       | other  |     7 |\n",
       "\n"
      ],
      "text/plain": [
       "   action_type           button_type n_sessions\n",
       "1  ime-change            header          3     \n",
       "2  language-change       header        219     \n",
       "3  more-languages-access header          1     \n",
       "4  no-search-results     header          8     \n",
       "5  NULL                  header         79     \n",
       "6  webfonts-enable       header          1     \n",
       "7  language-change       NULL            2     \n",
       "8  no-search-results     NULL            2     \n",
       "9  NULL                  NULL           13     \n",
       "10 font-change           other           1     \n",
       "11 ime-change            other          71     \n",
       "12 ime-disable           other          24     \n",
       "13 ime-enable            other          47     \n",
       "14 language-change       other       26266     \n",
       "15 more-languages-access other          32     \n",
       "16 no-search-results     other        4449     \n",
       "17 NULL                  other       22555     \n",
       "18 settings-open         other        1471     \n",
       "19 ui-lang-revert        other           1     \n",
       "20 webfonts-enable       other           7     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# find what types of actions are recorded with all button clicks\n",
    "all_button_actions <- more_lang_actions %>%\n",
    "    group_by(action_type, button_type) %>%\n",
    "    summarise (n_sessions = n_distinct(session_id)) %>%\n",
    "    arrange(button_type)\n",
    "\n",
    "all_button_actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the actions not recorded for the new clicks button make sense. For example, we shouldn't be recording settings-open events in sessions with clicks to the new language button but I would anticipate there being clicks to switch languages from the button."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check new content-language-switcher event"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [],
   "source": [
    "query <- \n",
    "\"\n",
    "SELECT\n",
    "    event.web_session_id as session_id,\n",
    "    event.skinVersion as skinversion,\n",
    "    wiki as wiki\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month = 06\n",
    "    AND Day >= 22\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.context = 'content-language-switcher'\n",
    "    AND event.action = 'language-change'\n",
    "GROUP BY \n",
    "    event.web_session_id,\n",
    "    event.skinVersion,\n",
    "    wiki\n",
    ")\n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n",
      "Warning message in system(cmd, intern = TRUE):\n",
      "“running command 'export HADOOP_HEAPSIZE=1024 && ionice nice hive -S -f ./temp_query11634119c207.hql 2>&1 > ./temp_results116316d08b51.tsv' had status 20”\n"
     ]
    },
    {
     "ename": "ERROR",
     "evalue": "Error in read.table(file = file, header = header, sep = sep, quote = quote, : no lines available in input\n",
     "output_type": "error",
     "traceback": [
      "Error in read.table(file = file, header = header, sep = sep, quote = quote, : no lines available in input\nTraceback:\n",
      "1. wmfdata::query_hive(query)",
      "2. utils::read.delim(results_dump, sep = \"\\t\", quote = \"\", as.is = TRUE, \n .     header = TRUE)",
      "3. read.table(file = file, header = header, sep = sep, quote = quote, \n .     dec = dec, fill = fill, comment.char = comment.char, ...)",
      "4. stop(\"no lines available in input\")"
     ]
    }
   ],
   "source": [
    "lang_content_switches <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 301,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A data.frame: 6 × 6</caption>\n",
       "<thead>\n",
       "\t<tr><th></th><th scope=col>date</th><th scope=col>session_id</th><th scope=col>skinversion</th><th scope=col>isanon</th><th scope=col>action</th><th scope=col>wiki</th></tr>\n",
       "\t<tr><th></th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>2021-06-23</td><td>053a2ed57541904671e1</td><td>latest</td><td>true</td><td>language-change</td><td>trwiki</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>2021-06-23</td><td>28f7400c785a050e1466</td><td>latest</td><td>true</td><td>language-change</td><td>frwiki</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>2021-06-23</td><td>c2ebfd1bede689af758c</td><td>legacy</td><td>true</td><td>language-change</td><td>enwiki</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>2021-06-23</td><td>38d8c82b0d93a8db81b0</td><td>latest</td><td>true</td><td>language-change</td><td>hewiki</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>2021-06-23</td><td>11e1d5c96a48c08365b2</td><td>legacy</td><td>true</td><td>language-change</td><td>eswiki</td></tr>\n",
       "\t<tr><th scope=row>6</th><td>2021-06-23</td><td>8f1fb84ee0abe3ea118b</td><td>legacy</td><td>true</td><td>language-change</td><td>enwiki</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A data.frame: 6 × 6\n",
       "\\begin{tabular}{r|llllll}\n",
       "  & date & session\\_id & skinversion & isanon & action & wiki\\\\\n",
       "  & <chr> & <chr> & <chr> & <chr> & <chr> & <chr>\\\\\n",
       "\\hline\n",
       "\t1 & 2021-06-23 & 053a2ed57541904671e1 & latest & true & language-change & trwiki\\\\\n",
       "\t2 & 2021-06-23 & 28f7400c785a050e1466 & latest & true & language-change & frwiki\\\\\n",
       "\t3 & 2021-06-23 & c2ebfd1bede689af758c & legacy & true & language-change & enwiki\\\\\n",
       "\t4 & 2021-06-23 & 38d8c82b0d93a8db81b0 & latest & true & language-change & hewiki\\\\\n",
       "\t5 & 2021-06-23 & 11e1d5c96a48c08365b2 & legacy & true & language-change & eswiki\\\\\n",
       "\t6 & 2021-06-23 & 8f1fb84ee0abe3ea118b & legacy & true & language-change & enwiki\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A data.frame: 6 × 6\n",
       "\n",
       "| <!--/--> | date &lt;chr&gt; | session_id &lt;chr&gt; | skinversion &lt;chr&gt; | isanon &lt;chr&gt; | action &lt;chr&gt; | wiki &lt;chr&gt; |\n",
       "|---|---|---|---|---|---|---|\n",
       "| 1 | 2021-06-23 | 053a2ed57541904671e1 | latest | true | language-change | trwiki |\n",
       "| 2 | 2021-06-23 | 28f7400c785a050e1466 | latest | true | language-change | frwiki |\n",
       "| 3 | 2021-06-23 | c2ebfd1bede689af758c | legacy | true | language-change | enwiki |\n",
       "| 4 | 2021-06-23 | 38d8c82b0d93a8db81b0 | latest | true | language-change | hewiki |\n",
       "| 5 | 2021-06-23 | 11e1d5c96a48c08365b2 | legacy | true | language-change | eswiki |\n",
       "| 6 | 2021-06-23 | 8f1fb84ee0abe3ea118b | legacy | true | language-change | enwiki |\n",
       "\n"
      ],
      "text/plain": [
       "  date       session_id           skinversion isanon action          wiki  \n",
       "1 2021-06-23 053a2ed57541904671e1 latest      true   language-change trwiki\n",
       "2 2021-06-23 28f7400c785a050e1466 latest      true   language-change frwiki\n",
       "3 2021-06-23 c2ebfd1bede689af758c legacy      true   language-change enwiki\n",
       "4 2021-06-23 38d8c82b0d93a8db81b0 latest      true   language-change hewiki\n",
       "5 2021-06-23 11e1d5c96a48c08365b2 legacy      true   language-change eswiki\n",
       "6 2021-06-23 8f1fb84ee0abe3ea118b legacy      true   language-change enwiki"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(lang_content_switches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 305,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 8 × 2</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>date</th><th scope=col>num_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>2021-06-22</td><td>32193</td></tr>\n",
       "\t<tr><td>2021-06-23</td><td>66203</td></tr>\n",
       "\t<tr><td>2021-06-24</td><td>70480</td></tr>\n",
       "\t<tr><td>2021-06-25</td><td>66043</td></tr>\n",
       "\t<tr><td>2021-06-26</td><td>58343</td></tr>\n",
       "\t<tr><td>2021-06-27</td><td>66835</td></tr>\n",
       "\t<tr><td>2021-06-28</td><td>86242</td></tr>\n",
       "\t<tr><td>2021-06-29</td><td>46804</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 8 × 2\n",
       "\\begin{tabular}{ll}\n",
       " date & num\\_sessions\\\\\n",
       " <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t 2021-06-22 & 32193\\\\\n",
       "\t 2021-06-23 & 66203\\\\\n",
       "\t 2021-06-24 & 70480\\\\\n",
       "\t 2021-06-25 & 66043\\\\\n",
       "\t 2021-06-26 & 58343\\\\\n",
       "\t 2021-06-27 & 66835\\\\\n",
       "\t 2021-06-28 & 86242\\\\\n",
       "\t 2021-06-29 & 46804\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 8 × 2\n",
       "\n",
       "| date &lt;chr&gt; | num_sessions &lt;int&gt; |\n",
       "|---|---|\n",
       "| 2021-06-22 | 32193 |\n",
       "| 2021-06-23 | 66203 |\n",
       "| 2021-06-24 | 70480 |\n",
       "| 2021-06-25 | 66043 |\n",
       "| 2021-06-26 | 58343 |\n",
       "| 2021-06-27 | 66835 |\n",
       "| 2021-06-28 | 86242 |\n",
       "| 2021-06-29 | 46804 |\n",
       "\n"
      ],
      "text/plain": [
       "  date       num_sessions\n",
       "1 2021-06-22 32193       \n",
       "2 2021-06-23 66203       \n",
       "3 2021-06-24 70480       \n",
       "4 2021-06-25 66043       \n",
       "5 2021-06-26 58343       \n",
       "6 2021-06-27 66835       \n",
       "7 2021-06-28 86242       \n",
       "8 2021-06-29 46804       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_content_switches %>%\n",
    "    group_by(date) %>%\n",
    "    summarize(num_sessions = n_distinct(session_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirmed we start recording events on 22 June 2021 with the first full day of events recorded on 23 June 2021."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 306,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 1 × 2</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>action</th><th scope=col>num_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>language-change</td><td>464766</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 1 × 2\n",
       "\\begin{tabular}{ll}\n",
       " action & num\\_sessions\\\\\n",
       " <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t language-change & 464766\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 1 × 2\n",
       "\n",
       "| action &lt;chr&gt; | num_sessions &lt;int&gt; |\n",
       "|---|---|\n",
       "| language-change | 464766 |\n",
       "\n"
      ],
      "text/plain": [
       "  action          num_sessions\n",
       "1 language-change 464766      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_content_switches %>%\n",
    "    group_by(action) %>%\n",
    "    summarize(num_sessions = n_distinct(session_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirmed we are only recording for language-change actions as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## By Test Wiki Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 310,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 2 × 2</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>istest</th><th scope=col>num_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>non_test</td><td>134361</td></tr>\n",
       "\t<tr><td>test    </td><td>237847</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 2 × 2\n",
       "\\begin{tabular}{ll}\n",
       " istest & num\\_sessions\\\\\n",
       " <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t non\\_test & 134361\\\\\n",
       "\t test     & 237847\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 2 × 2\n",
       "\n",
       "| istest &lt;chr&gt; | num_sessions &lt;int&gt; |\n",
       "|---|---|\n",
       "| non_test | 134361 |\n",
       "| test     | 237847 |\n",
       "\n"
      ],
      "text/plain": [
       "  istest   num_sessions\n",
       "1 non_test 134361      \n",
       "2 test     237847      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "lang_content_switches %>%\n",
    "    filter(date > '2021-06-23',\n",
    "            wiki != 'fawiki') %>%\n",
    "    mutate(istest = ifelse(wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' ), 'test', 'non_test')) %>%\n",
    "    group_by(istest) %>%\n",
    "    summarize(num_sessions = n_distinct(session_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The majority (64%) of content-language-switcher events have occured on the test wikis, since 23 June 2021.  This is expected as these events should only fire with the new language button which is only available to logged-in users that opt-in on non test wikis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By Skin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` regrouping output by 'istest' (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A grouped_df: 4 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>istest</th><th scope=col>skinversion</th><th scope=col>num_sessions</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>non_test</td><td>latest</td><td>   984</td></tr>\n",
       "\t<tr><td>non_test</td><td>legacy</td><td>143948</td></tr>\n",
       "\t<tr><td>test    </td><td>latest</td><td>263198</td></tr>\n",
       "\t<tr><td>test    </td><td>legacy</td><td>    88</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A grouped\\_df: 4 × 3\n",
       "\\begin{tabular}{lll}\n",
       " istest & skinversion & num\\_sessions\\\\\n",
       " <chr> & <chr> & <int>\\\\\n",
       "\\hline\n",
       "\t non\\_test & latest &    984\\\\\n",
       "\t non\\_test & legacy & 143948\\\\\n",
       "\t test     & latest & 263198\\\\\n",
       "\t test     & legacy &     88\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A grouped_df: 4 × 3\n",
       "\n",
       "| istest &lt;chr&gt; | skinversion &lt;chr&gt; | num_sessions &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| non_test | latest |    984 |\n",
       "| non_test | legacy | 143948 |\n",
       "| test     | latest | 263198 |\n",
       "| test     | legacy |     88 |\n",
       "\n"
      ],
      "text/plain": [
       "  istest   skinversion num_sessions\n",
       "1 non_test latest         984      \n",
       "2 non_test legacy      143948      \n",
       "3 test     latest      263198      \n",
       "4 test     legacy          88      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "lang_content_switches %>%\n",
    "    filter(date > '2021-06-23',\n",
    "            wiki != 'fawiki') %>%\n",
    "    mutate(istest = ifelse(wiki %in% c('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' ), 'test', 'non_test')) %>%\n",
    "    group_by(istest, skinversion) %>%\n",
    "    summarize(num_sessions = n_distinct(session_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Potential issue: We're seeing events recorded for legacy. Need to confirm if this is possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AB Balance Check\n",
    "\n",
    "Looked at query documented in https://phabricator.wikimedia.org/T280825\n",
    "\n",
    "Adjusted to restrict to wikis in the AB test and also account for clicks to the language list in the sidebar for the control group.\n",
    "\n",
    "Unfortunately, it is not possible to accurately determine buckets as we do not have instrumentation to track disitinct users that visit the site during the time of the AB test. Further checks on the client will be done to confirm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare Clicks to New Header Button (Treatment) vs Clicks to N Other Button or Language Links in Sidebar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 430,
   "metadata": {},
   "outputs": [],
   "source": [
    "query <- \n",
    "\n",
    "\"\n",
    "SELECT\n",
    "    event.web_session_id as session_id,\n",
    "    wiki as wiki,\n",
    "    SUM(1) AS num_sessions,\n",
    "    SUM(if(event.action = 'compact-language-links-open' AND event.context = 'header', 1, 0)) as n_header,\n",
    "    SUM(if(event.action = 'language-change' AND event.context = 'languages-list', 1, 0)) as n_sidebar_link,\n",
    "    SUM(if(event.action = 'compact-language-links-open' AND event.context = 'other', 1, 0)) as n_other,\n",
    "    SUM(if(event.action = 'settings-open' AND event.context = 'interlanguage', 1, 0)) as n_sidebar_settings\n",
    "FROM event.universallanguageselector\n",
    "WHERE\n",
    "    year = 2021\n",
    "    AND month = 06\n",
    "-- first full day of events\n",
    "    AND Day >= 23\n",
    "    AND useragent.is_bot = false\n",
    "    AND event.skinVersion = 'latest'\n",
    "    AND event.isAnon = false\n",
    "    AND wiki IN ('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', \n",
    "    'euwiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki')\n",
    "GROUP BY\n",
    "    event.web_session_id,\n",
    "    wiki\n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 431,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ab_test_data <-  wmfdata::query_hive(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 432,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`summarise()` ungrouping output (override with `.groups` argument)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A tibble: 11 × 4</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>wiki</th><th scope=col>test</th><th scope=col>control</th><th scope=col>all</th></tr>\n",
       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>bnwiki      </td><td>  205</td><td>  220</td><td>  550</td></tr>\n",
       "\t<tr><td>dewikivoyage</td><td>    0</td><td>   10</td><td>   10</td></tr>\n",
       "\t<tr><td>euwiki      </td><td>  392</td><td>  176</td><td>  883</td></tr>\n",
       "\t<tr><td>frwiki      </td><td>10095</td><td>15302</td><td>32761</td></tr>\n",
       "\t<tr><td>frwiktionary</td><td>  270</td><td>  338</td><td>  807</td></tr>\n",
       "\t<tr><td>hewiki      </td><td> 2131</td><td> 3465</td><td> 7106</td></tr>\n",
       "\t<tr><td>kowiki      </td><td> 1641</td><td> 1711</td><td> 4351</td></tr>\n",
       "\t<tr><td>ptwiki      </td><td> 3301</td><td> 4436</td><td>10159</td></tr>\n",
       "\t<tr><td>srwiki      </td><td>  451</td><td>  967</td><td> 1818</td></tr>\n",
       "\t<tr><td>trwiki      </td><td> 1303</td><td> 1554</td><td> 3718</td></tr>\n",
       "\t<tr><td>vecwiki     </td><td>    3</td><td>  108</td><td>  113</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A tibble: 11 × 4\n",
       "\\begin{tabular}{llll}\n",
       " wiki & test & control & all\\\\\n",
       " <chr> & <int> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t bnwiki       &   205 &   220 &   550\\\\\n",
       "\t dewikivoyage &     0 &    10 &    10\\\\\n",
       "\t euwiki       &   392 &   176 &   883\\\\\n",
       "\t frwiki       & 10095 & 15302 & 32761\\\\\n",
       "\t frwiktionary &   270 &   338 &   807\\\\\n",
       "\t hewiki       &  2131 &  3465 &  7106\\\\\n",
       "\t kowiki       &  1641 &  1711 &  4351\\\\\n",
       "\t ptwiki       &  3301 &  4436 & 10159\\\\\n",
       "\t srwiki       &   451 &   967 &  1818\\\\\n",
       "\t trwiki       &  1303 &  1554 &  3718\\\\\n",
       "\t vecwiki      &     3 &   108 &   113\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A tibble: 11 × 4\n",
       "\n",
       "| wiki &lt;chr&gt; | test &lt;int&gt; | control &lt;int&gt; | all &lt;int&gt; |\n",
       "|---|---|---|---|\n",
       "| bnwiki       |   205 |   220 |   550 |\n",
       "| dewikivoyage |     0 |    10 |    10 |\n",
       "| euwiki       |   392 |   176 |   883 |\n",
       "| frwiki       | 10095 | 15302 | 32761 |\n",
       "| frwiktionary |   270 |   338 |   807 |\n",
       "| hewiki       |  2131 |  3465 |  7106 |\n",
       "| kowiki       |  1641 |  1711 |  4351 |\n",
       "| ptwiki       |  3301 |  4436 | 10159 |\n",
       "| srwiki       |   451 |   967 |  1818 |\n",
       "| trwiki       |  1303 |  1554 |  3718 |\n",
       "| vecwiki      |     3 |   108 |   113 |\n",
       "\n"
      ],
      "text/plain": [
       "   wiki         test  control all  \n",
       "1  bnwiki         205   220     550\n",
       "2  dewikivoyage     0    10      10\n",
       "3  euwiki         392   176     883\n",
       "4  frwiki       10095 15302   32761\n",
       "5  frwiktionary   270   338     807\n",
       "6  hewiki        2131  3465    7106\n",
       "7  kowiki        1641  1711    4351\n",
       "8  ptwiki        3301  4436   10159\n",
       "9  srwiki         451   967    1818\n",
       "10 trwiki        1303  1554    3718\n",
       "11 vecwiki          3   108     113"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ab_test_data %>%\n",
    "    group_by(wiki) %>%\n",
    "    summarise(test = sum(n_header),\n",
    "             control = sum(n_other + n_sidebar_link + n_sidebar_settings),\n",
    "             all = sum(num_sessions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirmed that we are logging both control and test events and the instrumentation allows us to distinguish these events.\n",
    "\n",
    "The splits are not perfectly balanced but there are no signficant differences indicating a regression or difference in sampling rate. Differences appear as expected based on a 50/50 split.\n",
    "\n",
    "Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 347,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A data.frame: 1 × 3</caption>\n",
       "<thead>\n",
       "\t<tr><th scope=col>test</th><th scope=col>control</th><th scope=col>all</th></tr>\n",
       "\t<tr><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><td>18425</td><td>26055</td><td>57645</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A data.frame: 1 × 3\n",
       "\\begin{tabular}{lll}\n",
       " test & control & all\\\\\n",
       " <int> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t 18425 & 26055 & 57645\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A data.frame: 1 × 3\n",
       "\n",
       "| test &lt;int&gt; | control &lt;int&gt; | all &lt;int&gt; |\n",
       "|---|---|---|\n",
       "| 18425 | 26055 | 57645 |\n",
       "\n"
      ],
      "text/plain": [
       "  test  control all  \n",
       "1 18425 26055   57645"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ab_test_data %>%\n",
    "    summarise(test = sum(n_header),\n",
    "             control = sum(n_other + n_sidebar_link + n_sidebar_settings),\n",
    "             all = sum(num_sessions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}