{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "34bc0694",
   "metadata": {},
   "source": [
    "# AB-test - Part 3/3(relationship metrics)\n",
    "\n",
    "> AB-test - Part 3(relationship metrics)\n",
    "\n",
    "- toc: true\n",
    "- branch: master\n",
    "- badges: true\n",
    "- comments: true\n",
    "- author: Zmey56\n",
    "- categories: [data analysis, ab-test, metrics]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6786bc4e",
   "metadata": {},
   "source": [
    "I continue my publications on analytics:\n",
    "\n",
    "- [The First Dashboard](https://alex.gladkikh.org/data%20analysis/mysql/superset/bi/2022/08/03/the-first-dashboard.html)\n",
    "- [Analyses of Product Metrics](https://alex.gladkikh.org/data%20analysis/mysql/superset/bi/2022/08/05/analyses-of-product-metrics.html)\n",
    "- [А/B-tests - Part 1/3(AA-test)](https://alex.gladkikh.org/data%20analysis/ab-test/aa-test/2022/08/09/aa-test-article.html)\n",
    "- [А/B-tests - Part 2/3(AB-test)](https://alex.gladkikh.org/data%20analysis/ab-test/aa-test/2022/08/11/ab-test-article.html)\n",
    "\n",
    "There are a huge number of metrics and relationship metrics that can be used for analysis. There are a large number of publications. For example, you can look at Nikita Marshalkin's materials.\n",
    "\n",
    "In 2018, Yandex researchers developed a method for analyzing tests over metrics $\\dfrac{x}{y}$  ratios.\n",
    "\n",
    "The idea of the method is that instead of pushing \"browser-based\" CTR into the test, you can construct another metric and analyze it, but at the same time it is guaranteed (unlike smoothed CTR) that if the test on this other metric sees changes, then there are changes in the original metric (that is, in likes per user and in user CTR)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c1a73cb",
   "metadata": {},
   "source": [
    "1. We calculate the total CTR in the control group $𝐶𝑇𝑅_{𝑐𝑜𝑛𝑡𝑟𝑜𝑙}=\\dfrac{𝑠𝑢𝑚(𝑙𝑖𝑘𝑒𝑠)}{𝑠𝑢𝑚(𝑣𝑖𝑒𝑤𝑠)}$\n",
    "2. Let's calculate the metric for users in both groups $𝑙𝑖𝑛𝑒𝑎𝑟𝑖𝑧𝑒𝑑𝑙𝑖𝑘𝑒𝑠=𝑙𝑖𝑘𝑒𝑠−𝐶𝑇𝑅_{𝑐𝑜𝑛𝑡𝑟𝑜𝑙} \\times 𝑣𝑖𝑒𝑤𝑠$\n",
    "3. After that, let's compare the differences in the groups by the metric linearizadlikes with the t-test\n",
    "\n",
    "This simple method guarantees that with a large sample size, it is possible to increase the sensitivity of the metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4c251b86",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pandahouse as ph\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "21fda9e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "connection = {\n",
    "    'host': 'https://clickhouse.lab.karpov.courses',\n",
    "    'password': '**********',\n",
    "    'user': '**********',\n",
    "    'database': 'simulator_20220620'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c8c90f6",
   "metadata": {},
   "source": [
    "The first test will be conducted between groups 0 and 3 according to the metric of linearized likes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f548fec0",
   "metadata": {},
   "outputs": [],
   "source": [
    "q = \"\"\"\n",
    "SELECT exp_group, \n",
    "    user_id,\n",
    "    sum(action = 'like') as likes,\n",
    "    sum(action = 'view') as views,\n",
    "    likes/views as ctr\n",
    "FROM {db}.feed_actions \n",
    "WHERE toDate(time) between '2022-05-24' and '2022-05-30'\n",
    "    and exp_group in (0,3)\n",
    "GROUP BY exp_group, user_id\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ff589e1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = ph.read_clickhouse(q, connection=connection)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1407c56b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsUAAAHxCAYAAACFyME+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA5L0lEQVR4nO3dfXxU1b3v8e+eh0zCkyJMDCDGtlqpniIqrQYtyJETUIjYgBb1CFVs4drCFV8XtUCx6qmi0tLaXnql9dpqHylKQY4EjBaqhUJLH7hasKCCApoMT0LCzCQzs+4flimTGTITkpk9M/vzfr14vZjf7Jn9y4Ik3+ysvZZljDECAAAAHMxldwMAAACA3QjFAAAAcDxCMQAAAByPUAwAAADHIxQDAADA8QjFAAAAcDxCMQAAABzPY3cDXeXQoWbFYrldcrlPnx46cKApp+csJIxPeoxR+xif9Bij9jE+6TFG6TFG7SuU8XG5LPXu3f2kzxdNKI7FTM5D8fHz4uQYn/QYo/YxPukxRu1jfNJjjNJjjNpXDOPD9AkAAAA4HqEYAAAAjlc00ycAAAAgRaMRHToUUCTSkpPzNTa6FIvFcnKuTHg8Jerd2y+3u2Mxl1AMAABQRA4dCqi0tJu6d6+QZVlZP5/H41Ikkh+h2Bij5uYjOnQooL59+3XotUyfAAAAKCKRSIu6d++Vk0CcbyzLUvfuvU7pKjmhGAAAoMg4MRAfd6ofO6EYAAAAjkcoBgAAgOMRigEAAOB4rD4BAADgAK+99jv95CdPKRJpVWlpqb7ylbu0cuVyBYNBPfTQAr399luaOXO6vv/9JXr55bXau3ePGhsbtH//fp133id1333z1L17j3bP8eyzP9Z///cKdevWTRdddIlefXWdli17Qd/85jd05MiH2rt3r4YNu1KTJ9+ub3/7Ue3Y8aYsy9Lllw/Tl7/8FXk8Hl155VCtWlWv008/XZLij99+e6d+8IPvqaKin959d5dKSnyaO/cbOuecj3XJ+HClGAAAoMi99967WrLkf2vhwu/q6ad/rtmz52ru3NmaNese7dz5D61evUr33/81zZx5dzxk/vWvf9aDDz6in/98mdxut55++kftnmPTpo1avfoF/fCHz+ipp36qY8eaE54PhcL66U+X6s47Z+o733lcvXqdpmee+ZV+9KNntXPnDv3iFz9N+3G8+eY2TZz4Bf3kJ7/U2LE1euih+ac+KG0QigEAAIrcH/+4SQcO7Nf//J936otfvFkPPvh1WZZLe/a8pwceeFiPPvpfuuCCf1N19TXx14wcebXOOKOPXC6Xxo0br82bN7Z7jo0bf6+RI0epZ8+esixLtbU3Jjw/ePBF8b//4Q8bNGHCjbIsSyUlJRo/foI2bdqQ9uM499zzdNFFF0uSxo4drx073tSHHx7uwEicHNMnAAAAilwsFtWll35WDz74SLzW0PCB+vb167e/rddpp52mHTveVGtrq7xeryTJ7XbHjzUmJpfLnfS+J3K73TLGxB+7XInXXsvKuiW834lLpxkTUyQSOeHxR+/T2tqadI62x6TrK1NcKQYAAChyl176WW3e/Aft3r1LkrRx42uaMuUm7d79jr773W9p0aL/rbPPPkc/+MET8de8+up6NTU1KRaLaeXK3+iKKz7X7jmGDbtS69e/oqamJknSf//3ipOuGfzZz1bpueeWyhijlpYWrVy5XJ/5zGWSpNNP763t2/8uSXrppbqE1+3Y8Q/t3LlDkrRy5fP6t38brJ49e3Z8QFLgSjEAAECR+9jHPq577pmr+++fI2OM3G63Hn3023r00W/qpptu1cc/fq7uvvteTZkySUOHfhROzzijj2bPnqnDhw9ryJBLNHnybe2e49JLP6Oamus1ffpt8vlK9bGPfVw+X2nKY++6639p0aLHNXnyF9TaGtHll1dp8uTb4899+9uPqWfPHho69DL16dM3/rozzuijJUsW64MP9ql37zP09a8/2EUjJFnmxOvcBezAgSbFYrn9UPz+ngoEjub0nIWE8UmPMWof45MeY9Q+xic9xii9QhujDz7YrYqKyk69x1NPPakPPzysu+++N+2xHo9LkUhM27f/Xf/v/23VDTdMkiT98pc/1d///kbClI3O+POf/6RFix7Ts88uTXtsqjFwuSz16XPy1TO4UgwAAICMzJ//Nb377u6EmmVJxkhz535Df/vbX7Ry5fOyLEtnnlmhe+6Za1OnHceV4k4otJ8cc43xSY8xah/jkx5j1D7GJz3GKL1CG6OuuFLcEcevFOcTrhQDXcynVplgKKFmlZUqLK9NHQEAgGwgFAPtMMGQ9q2tT6j1rx4llRGKAQAoJizJBgAAAMcjFAMAAMDxmD4BAAAA9T6tVJ6Srp8eGGlp1aEPQ+kPtBmhGAAAAPKUeLXtR890+HUul9XuCmCfumOypMxC8dq1dXrmmacUiUR0ww03acKEGzvcz6kiFAMAAMB2gUCjfvjDxXrqqWfl9ZZo+vTbdcklQ/Wxj308J+dnTjEAAABs96c/bdYllwxVr16nqaysTCNHXq11617O2fkJxQAAALDd/v0B9enTN/64T5++amxszNn5CcUAAACwXSwWk2VZ8cfGGLlcVjuv6FqEYgAAANiuvPxMHTiwP/744MED6tvXn7PzE4oBAABgu6FDP6stW/6oQ4cOKRQKad26V3TZZVU5Oz+rTwAAAECRltZ/Lp/W9e+bCb+/XF/60p2aOXOaWlsjqqkZrwsu+Lcu7+dkCMUAAAD45wYbHd9kw+NxKRKJdUkP1dVjVF09pkveq6OYPgEAAADHIxQDAADA8QjFAAAAcDxCMQAAAByPUAwAAADHy+rqE7feeqsOHjwoj+ej0zz44INqbm7WI488onA4rGuuuUazZs2SJG3btk1z585Vc3Ozhg4dqgceeCD+OgAAACCbspY6jTHatWuXfvvb38bDbSgU0pgxY/Tss8+qX79+mjZtmtavX68RI0Zo9uzZ+q//+i8NGTJEc+bM0dKlS3XzzTdnqz0AAACc4LTTu6nE6+7y921pjerDw8cyOvZHP/o/WrfuZUmWxo27TpMm/WeX93MyWQvFb7/9tiTp9ttv1+HDh3XjjTfqk5/8pCorKzVw4EBJUk1Njerq6nTuuecqFAppyJAhkqTa2lo98cQThGIAAIAcKfG69cPn/9bh17lclmIxc9Lnv1R7UUbv85e/bNGWLX/Uj3/8C0WjEf3nf96oYcOu1Nlnn9Phnk5F1kLxkSNHVFVVpa9//etqbW3V5MmTdccdd8jv/9ce1uXl5WpoaFBjY2NC3e/3q6GhoUPn69OnR5f13hF+f09bzlsoCn18Qo0hde/uS6j5Sr06rQs/rkIfo2xjfNJjjNrH+KTHGKVXSGPU2OiSx3Nqt425XFZWXpdJP5/5zGd08cVL5PF49f77+xWNRtWjR/dT+lhcLleH/82yFoovvvhiXXzxxfHHEydO1BNPPKFLL700XjPGyLIsxWIxWZaVVO+IAwea2v0pJRv8/p4KBI7m9JyFpBjGpyTUqubmcEItHGrV0S76uIphjLKJ8UmPMWof45MeY5ReoY1RLBY75R3mTiVLpbtSLKkD/bj15JM/0C9+8axGjhyl3r37ntLHEovFkv7NXC6r3YuoWVt94k9/+pM2btwYf2yM0YABAxQIBOK1QCCg8vJyVVRUJNT379+v8vLybLUGAACAPDV16jStWlWvxsYGrVy5PGfnzVooPnr0qB577DGFw2E1NTVp+fLluvvuu/XOO+9o9+7dikajWrVqlYYPH64BAwbI5/Npy5YtkqQVK1Zo+PDh2WoNAAAAeWb37l3aseNNSVJpaamGDx+pt97akbPzZ236xMiRI/W3v/1N119/vWKxmG6++WZdfPHFWrBggWbMmKFwOKwRI0ZozJgxkqSFCxdq3rx5ampq0oUXXqjJkydnqzUAAADkmX379uipp5Zo8eIfybIsvfbaeo0de13Ozp/VhYDvuusu3XXXXQm1qqoqrVy5MunYQYMGadmyZdlsBwAAAHmqqupK/f3vb+j222+Ry+XSiBH/rlGjRufs/OyOAQAAALW0RjNePq2j75upqVOnaerUaV3eQyYIxQAAAMh4g422PB7XKa92kU+ydqMdAAAAUCgIxQAAAHA8QjEAAECRMSa3G5rlk1P92AnFAAAARcTjKVFz8xFHBmNjjJqbj8jjKenwa7nRDgAAoIj07u3XoUMBNTUdzsn5XC6XYrH8udHO4ylR797+jr8uC70AAADAJm63R3379svZ+fz+ngoEjubsfNnC9AkAAAA4HqEYAAAAjkcoBgAAgOMRigEAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgOMRigEAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgOMRigEAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgOMRigEAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgON57G4AKAY+tcoEQwk1q6xUYXlt6ggAAHQEoRjoAiYY0r619Qm1/tWjpDJCMQAAhYDpEwAAAHA8QjEAAAAcj+kTQAd5vG4peDSh5lLMpm4AAEBXIBQDHRQLh7Wvfl1C7axRV9nSCwAA6BpMnwAAAIDjEYoBAADgeIRiAAAAOB6hGAAAAI7HjXZAjp24+12oMaSSUKs8ZSWKBFsSjmNHPAAAcodQDOTYibvfde/uU3NzWGeNuippRQt2xAMAIHeYPgEAAADHIxQDAADA8QjFAAAAcDzmFAO5ZkmRmJEkNQdb438HAAD2IRQDORaJGb2157Akyev1qLU1orMNwRgAADsxfQIAAACORygGAACA4xGKAQAA4HiEYgAAADgeoRgAAACOx+oTQJZ4vG4peDSpblmsNAEAQL4hFANZEguHta9+XVK9/9Ujct8MAABoF9MnAAAA4HhcKQbac8LucwAAoHgRioF2nLj73HHsPgcAQPFh+gQAAAAcjyvFQBdhmgUAAIWLUAx0gZhhmgUAAIWM6RMAAABwPK4UA1liWVbqKRWWlftmAABAuwjFQJaYFFMqJKZVAACQj5g+AQAAAMcjFAMAAMDxCMUAAABwPEIxAAAAHI9QDAAAAMcjFAMAAMDxWJINyFMer1sKHk2oWWWlCstrU0cAABQvQjGQp2LhsPbVr0uo9a8eJZURigEA6GpMnwAAAIDjcaUYyGNJ20SzQzQAAFlBKAbyVCzFNtH924ZkAADQJZg+AQAAAMfLeih+9NFHdd9990mSNmzYoJqaGlVXV2vRokXxY7Zt26ba2lqNHj1ac+fOVSQSyXZbAAAAQFxWQ/HGjRu1fPlySVIoFNKcOXO0ePFivfjii3r99de1fv16SdLs2bM1f/58rVmzRsYYLV26NJttAQAAAAmyFooPHz6sRYsWafr06ZKkrVu3qrKyUgMHDpTH41FNTY3q6uq0d+9ehUIhDRkyRJJUW1ururq6bLUFAAAAJMnajXbz58/XrFmz9P7770uSGhsb5ff748+Xl5eroaEhqe73+9XQ0NDh8/Xp06PzTZ8Cv7+nLectFIU+Pof3NMnrTfw0cVmWunf3JdXaHqdUtRR1r9eT8Xu63S71KfAx7ahC/z+UC4xR+xif9Bij9Bij9hXD+GQlFP/6179Wv379VFVVpeeff16SFIvFZFn/Wk/KGCPLsk5a76gDB5oUy/Gd+X5/TwUCR9Mf6FDFMD6uaEytrYlz3GPGqLk5nFA7zZik45Sq1qbu9XrU2hrJ+D2j0VjBj2lHFMP/oWxjjNrH+KTHGKXHGLWvUMbH5bLavYialVD84osvKhAIaPz48frwww917Ngx7d27V263O35MIBBQeXm5KioqFAgE4vX9+/ervLw8G20BAAAAKWUlFD/99NPxvz///PPavHmzHnjgAVVXV2v37t0666yztGrVKk2YMEEDBgyQz+fTli1bdOmll2rFihUaPnx4NtoC8pZlWSk26mCnDgAAciVnm3f4fD4tWLBAM2bMUDgc1ogRIzRmzBhJ0sKFCzVv3jw1NTXpwgsv1OTJk3PVFpAXTIqNOs42bNQBAECuZD0U19bWqra2VpJUVVWllStXJh0zaNAgLVu2LNutAAAAACmxox0AAAAcL2fTJwB0nsuy1ByJJdXLStxSjldfAQCgmBCKgQISNUZrNu5Kqo+uOkdlHn7xAwDAqeK7KAAAAByPUAwAAADHIxQDAADA8ZhTDHQQG20AAFB8CMVAB7HRBgAAxYfpEwAAAHA8QjEAAAAcj1AMAAAAxyMUAwAAwPEIxQAAAHA8QjEAAAAcj1AMAAAAxyMUAwAAwPHYvAMoAl6PS8FILKFWVuKW2u68BwAAUiIUA0Ug1BJV/abdCbXRVeeozMMvgwAAyATfMQEAAOB4hGIAAAA4HqEYAAAAjsecYjiOT60ywVBCzSorVVhemzoCAAB2IxTDcUwwpH1r6xNq/atHSWWEYgAAnIrpEwAAAHA8QjEAAAAcj+kTgNO5LAVbokllNv8AADgJoRhwuGBLVGs27kqqs/kHAMBJ+I4HAAAAxyMUAwAAwPEIxQAAAHA8QjEAAAAcj1AMAAAAxyMUAwAAwPFYkg0oUl6PS8FILKHG2sMAAKRGKAaKVKglqvpNuxNqrD0MAEBqfHcEAACA4xGKAQAA4HhMnwCKQC93VFWVZUk1AACQGUIxUARMOKSdK1Yn1CqmTrSpGwAACg/TJwAAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgOMRigEAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgOMRigEAAOB47GgHOIjX41IwEkssWpY9zQAAkEcIxYCDhFqiqt+0O6E26rJKm7oBACB/MH0CAAAAjseVYkCSx+uWgkeT6pZlbOgGAADkGqEYkBQLh7Wvfl1Svf/VI3LfDAAAyDmmTwAAAMDxuFIMFBCf162qyrKkupdpHgAAdAqhGCggsXBYO1esTqoPvO363DcDAEARYfoEAAAAHI9QDAAAAMcjFAMAAMDxmFOMouFTq0wwlFDzlJUoEmxJqLnUZptjAADgeIRiFA0TDGnf2vqE2lmjrkpaf/isUVflrikAAFAQmD4BAAAAxyMUAwAAwPEIxQAAAHA8QjEAAAAcj1AMAAAAxyMUAwAAwPEIxQAAAHA8QjEAAAAcj1AMAAAAxyMUAwAAwPEIxQAAAHC8rIbi7373u7r22ms1duxYPf3005KkDRs2qKamRtXV1Vq0aFH82G3btqm2tlajR4/W3LlzFYlEstkaAAAAEJe1ULx582b94Q9/0MqVK/Xcc8/p2Wef1fbt2zVnzhwtXrxYL774ol5//XWtX79ekjR79mzNnz9fa9askTFGS5cuzVZrgCP4vG5VVZYl/OnljnbuTV2WgpFYwh+5rK5pGAAAG3my9caf/exn9cwzz8jj8aihoUHRaFRHjhxRZWWlBg4cKEmqqalRXV2dzj33XIVCIQ0ZMkSSVFtbqyeeeEI333xzttoDil4sHNbOFasTahVTJ3bqPYMtUa3ZuCuhNrrqHJV5mIkFAChsWf1O5vV69cQTT2js2LGqqqpSY2Oj/H5//Pny8nI1NDQk1f1+vxoaGrLZGgAAABCXtSvFx82cOVNf+tKXNH36dO3atUuW9a9ftRpjZFmWYrFYynpH9OnTo8t67gi/v6ct5y0UuRyfUGNI3bv7Empul3XKNUlyWZa83jafJp2ppah7vZ4uf8+T1SwrxcfudqX82Et9XvnP6JZQazx4LOnYVMd1JT7H0mOM2sf4pMcYpccYta8Yxidrofitt95SS0uLPvWpT6msrEzV1dWqq6uT2+2OHxMIBFReXq6KigoFAoF4ff/+/SovL+/Q+Q4caFIsZrqs/0z4/T0VCBzN6TkLSa7HpyTUqubmcEKtd8ycck2STjNGra1tbvrsTK1N3ev1fPT3LnzP9mrGJH+c0Wgs5cceCrcm/fuFIsnHpjquq/A5lh5j1D7GJz3GKD3GqH2FMj4ul9XuRdSsTZ/Ys2eP5s2bp5aWFrW0tOjll1/WpEmT9M4772j37t2KRqNatWqVhg8frgEDBsjn82nLli2SpBUrVmj48OHZag0AAABIkLUrxSNGjNDWrVt1/fXXy+12q7q6WmPHjtUZZ5yhGTNmKBwOa8SIERozZowkaeHChZo3b56ampp04YUXavLkydlqDQAAAEiQ1TnFM2bM0IwZMxJqVVVVWrlyZdKxgwYN0rJly7LZDoAUermjqqosS6r3cEUVZX8fAIBDZPQdb86cOUm1mTNndnkzAHLPhEPauWJ10h8TCtrdGgAAOdPuleL7779fDQ0N2rJliw4ePBivRyIRvffee1lvDgAAAMiFdkPxxIkTtWPHDr355psaPXp0vO52u+MbbQCFKJLjlUoAAEB+azcUf/rTn9anP/1pDRs2TBUVFbnqCciqmDF6a8/hhNrZhpAMAICTZXSj3fvvv6/Zs2frww8/lDkhPLzwwgtZawwAAADIlYxC8fz581VbW6sLLrigwzvNAQAAAPkuo1Ds8Xh02223ZbsXAAAAwBYZLcl23nnn6c0338x2LwAAAIAtMrpS/N5772nChAnq37+/fD5fvM6cYgAAABSDjELxrFmzst0HAAAAYJuMQvEnP/nJbPcBAAAA2CajUHz55ZfLsiwZY+KrT/j9fv3ud7/LanMAAABALmQUirdv3x7/e0tLi1atWqV33nkna00BAAAAuZTR6hMnKikpUW1trX7/+99nox8AAAAg5zK6Unz48OH4340xev3113XkyJFs9QQAAADkVIfnFEtSnz59NHfu3Kw2BgAAAORKh+cUAwAAAMUmo1Aci8X01FNP6Xe/+50ikYiuuOIKTZ8+XR5PRi8HupRPrTLBUFLdpZgN3RQvl2WpOdJmTP+5+gwAAMUmo1T7rW99S9u3b9eUKVMUi8X0q1/9So899pjmzJmT7f6AJCYY0r619Un1s0ZdlftmiljUGK3ZuCuhNuqySnuaAQAgyzIKxa+++qqee+45eb1eSdJVV12l6667jlAMAACAopDRkmzGmHgglj5alu3ExwAAAEAhyygUDxo0SA8//LDeffddvffee3r44YfZ+hkAAABFI6NQfP/99+vIkSOaNGmSbrjhBh06dEhf//rXs90bAAAAkBPthuKWlhbde++92rhxoxYsWKANGzZo8ODBcrvd6tGjR656BAAAALKq3VD8xBNPqKmpSZdcckm89tBDD+nIkSP63ve+l/XmAAAAgFxoNxSvW7dO3/rWt9SnT5947cwzz9Rjjz2m+vrkJbEAAACAQtRuKPZ6vSotLU2q9+jRQyUlJVlrCgAAAMildkOxy+VSU1NTUr2pqUmRSCRrTQEAAAC51G4oHjdunObNm6djx47Fa8eOHdO8efNUXV2d9eYAAACAXGg3FE+ZMkU9e/bUFVdcoRtvvFETJ07UFVdcoV69eukrX/lKrnoEAAAAsqrdbZ5dLpceeughTZ8+XW+88YZcLpcGDx6s8vLyXPUHAAAAZF27ofi4AQMGaMCAAdnuBQAAALBFRjvaAQAAAMWMUAwAAADHy2j6BIDi4PO6VVVZllDzWqZT7+n1uBSMxBJqZSVuKda59wUAIJcIxYCDxMJh7VyxOqE28LbrO/WeoZao6jftTqiNrjpHZR5+EQUAKByEYkCSZVmKpLqyaVm5bwYAAOQcoRiQZIzRW3sOJ9XPNkwBAADACfj9JgAAAByPUAwAAADHIxQDAADA8ZhTjKKXdAMdN88BAIA2CMUoarEUN9Bx85yNXJaCLdGEEmsaAwDyAaEYQM4EW6Jas3FXQo01jQEA+YBQDCBjvdzRpB3xermjJzkaAIDCQSgGkDETDiXtiFcxdaJN3QAA0HX4nSUAAAAcjyvFALqc1+NSMBJLfoKVPwAAeYpQDKDLhVqiqt+0O6k+6rJKG7oBACA9pk8AAADA8QjFAAAAcDxCMQAAAByPUAwAAADHIxQDAADA8QjFAAAAcDxCMQAAAByPUAwAAADHY/MOACn5vG5VVZYl1LyWsakbAACyi1AMIKVYOKydK1Yn1Abedr09zQAAkGVMnwAAAIDjEYoBAADgeIRiAAAAOB6hGAAAAI5HKAYAAIDjEYoBAADgeIRiAAAAOB6hGAAAAI5HKAYAAIDjEYoBAADgeGzzjKISiZnEgmXZ0wgAACgohGIUjZgxemvP4YTa2cakPhgAAOAEhGIAneLzulVVWZZQ6+WO2tQNAACnJquh+Pvf/75Wr14tSRoxYoTuuecebdiwQY888ojC4bCuueYazZo1S5K0bds2zZ07V83NzRo6dKgeeOABeTxkdiDfxcJh7VyxOqFWMXWiTd10kMtSsCU5wJeVuKW2U3EAAEUtazfabdiwQa+99pqWL1+u3/zmN3rjjTe0atUqzZkzR4sXL9aLL76o119/XevXr5ckzZ49W/Pnz9eaNWtkjNHSpUuz1RoASJKCLVGt2bgr6U+qoAwAKG5ZC8V+v1/33XefSkpK5PV69YlPfEK7du1SZWWlBg4cKI/Ho5qaGtXV1Wnv3r0KhUIaMmSIJKm2tlZ1dXXZag0AAABIkLVQfN5558VD7q5du7R69WpZliW/3x8/pry8XA0NDWpsbEyo+/1+NTQ0ZKs1AAAAIEHWJ+3u2LFD06ZN0z333CO3261du3bFnzPGyLIsxWIxWScsnXW83hF9+vToqpY7xO/vact5C0U2xifUGFL37r6kusuy5PW2+S/dmVpnX5/he3q9noLosyM1y7JS/hu53a6keqnPK/8Z3ZLP/0/Z/BxrPHgsZZ/peso3fB1qH+OTHmOUHmPUvmIYn6yG4i1btmjmzJmaM2eOxo4dq82bNysQCMSfDwQCKi8vV0VFRUJ9//79Ki8v79C5DhxoUizHN8b4/T0VCBzN6TkLSbbGpyTUqubmcFL9NGPU2hpJLHam1tnXZ/CeXq/no7/neZ8drRljUv4bRaOxpHoo3HrS/yfZ/hwLRZL7SddTvuHrUPsYn/QYo/QYo/YVyvi4XFa7F1GzNn3i/fff11e+8hUtXLhQY8eOlSRddNFFeuedd7R7925Fo1GtWrVKw4cP14ABA+Tz+bRlyxZJ0ooVKzR8+PBstQYAAAAkyNqV4qeeekrhcFgLFiyI1yZNmqQFCxZoxowZCofDGjFihMaMGSNJWrhwoebNm6empiZdeOGFmjx5crZaAwAAABJkLRTPmzdP8+bNS/ncypUrk2qDBg3SsmXLstUOAAAAcFJZmz4BAAAAFApCMQAAAByPUAwAAADHIxQDAADA8bK+eQfQ5SwpkmpN6g5u+AIAAHAcoRgFJxIzemvP4aT62Sa3m7ega3g9LgUjsYRaWYlbyvFmPAAAZyMUA7BVqCWq+k27E2qjq85RmYfZXQCA3CEUI6/51CoTDCXULIsriAAAoGsRipHXTDCkfWvrE2r9rx5hUzcAAKBYEYqRN1JdFXYpdpKjAQAAug6hGHkj1VXhs0ZdZU8zAADAUQjFAHKmlzuqqsqypBoAAHYjFAPIGRMOaeeK1Qm1iqkTk447vkxb48FjCv1zuTaWaQMAZBOhGEDeOb5MW/fuPjU3hyWxTBsAILv4DgMAAADHIxQDAADA8Zg+AaDL+bzupBvqJMnLxisAgDxFKAbQ5WLhcNINdZI08LbrT/k9j9981xY34AEAugKhGEBBOH7zXVvcgAcA6Ap8JwEAAIDjEYoBAADgeIRiAAAAOB6hGAAAAI5HKAYAAIDjEYoBAADgeIRiAAAAOB6hGAAAAI5HKAYAAIDjEYoBAADgeIRiAAAAOB6hGAAAAI7nsbsBALBLD1dUVZVlKetRrhkAgKMQigE4lgkFtXPF6qR6v6kTpW5eGzoCANiFSyEAAABwPEIxAAAAHI9QDAAAAMcjFAMAAMDxCMUAAABwPEIxAAAAHI9QDAAAAMdjnWIAtvJ53UkbaPRyR23qBgDgVIRiALaKhcNJG2hUTJ1oUzcAAKdi+gQAAAAcj1AMAAAAx2P6BICC5vW4FIzEEmplJW4pZmzqCABQiAjFAApaqCWq+k27E2qjq85RmYdfhAEAMkcohi18apUJhhJqLsVOcjQAAEB2EYphCxMMad/a+oTaWaOusqcZAADgeIRiAHnn+NrFHo9LkchH0yBYuxgAkE2EYgB55/jaxV6vR62tEUmsXQwAyC7uRAEAAIDjEYoBAADgeIRiAAAAOB5zipH3Im03YbAsexoBAABFi1CMvBYzRm/tOZxQO9uwUxkAAOhaTJ8AAACA4xGKAQAA4HhMnwBQ0Hq5o6qqLEuo9XBFFeVnfgBABxCKARQ0Ew5p54rVCbV+UydK3bw2dQQAKESEYgAF4fjWz215LW68BAB0HqEYQEE4vvVzWwNvuz73zQAAig6T7gAAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgOMRigEAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgOOxeQeAouMr8Sh87EhCzdu9m8KGL3kAgNT4DoGs8qlVJhhKqrsUS3l8JNZmy17LykZbKHLRUEivPv2bhNrnpk6UuvWypyEAQN4jFCOrTDCkfWvrk+pnjboqqRYzRm/tOZxQO9uYpOMAAAC6GnOKAQAA4HhZDcVNTU0aN26c9uzZI0nasGGDampqVF1drUWLFsWP27Ztm2prazV69GjNnTtXkUgkm20BAAAACbIWiv/2t7/ppptu0q5duyRJoVBIc+bM0eLFi/Xiiy/q9ddf1/r16yVJs2fP1vz587VmzRoZY7R06dJstQUAAAAkyVooXrp0qe6//36Vl5dLkrZu3arKykoNHDhQHo9HNTU1qqur0969exUKhTRkyBBJUm1trerq6rLVFgAAAJAkazfaffOb30x43NjYKL/fH39cXl6uhoaGpLrf71dDQ0OHz9enT49Tb7YT/P6etpy3UPhKvere3ZdUd7uspLrLsuT1tvkvmWmtI8fm2Xt6vZ6C6NOu94w/18n3dLtd6tPm8/XwnqaU75nq2HzG16H2MT7pMUbpMUbtK4bxydnqE7FYTNYJy2sZY2RZ1knrHXXgQJNibZfzyjK/v6cCgaM5PWch8ft7KhxqVXNzOOm53jGTVD/NGLW2tplPnmmtI8fm0Xt6vZ6P/p7nfdr1nvHx6YL3jEZjSZ+vrmgs5XumOjZf8XWofYxPeoxReoxR+wplfFwuq92LqDkLxRUVFQoEAvHHgUBA5eXlSfX9+/fHp1wAQMFxWQq2RJPKZSVuKcc/uAMAMpezUHzRRRfpnXfe0e7du3XWWWdp1apVmjBhggYMGCCfz6ctW7bo0ksv1YoVKzR8+PBctQUApy5VALYsrdm4K+nQ0VXnqMzDKpgAkK9yFop9Pp8WLFigGTNmKBwOa8SIERozZowkaeHChZo3b56ampp04YUXavLkyblqCwBOWbAlmhSAR11WaU8zAIBOyXoofuWVV+J/r6qq0sqVK5OOGTRokJYtW5btVpBn2NIZAADkC7Z5hi3Y0hkAAOQTQjFOiU+tMsFQQs0qK1VYXps6ArqOy7LUHIkl1LhRDgCKG6EYp8QEQ9q3tj6h1r96lFRGKEbhixqTNFeYG+UAoLgRigEgA16PS8E2V4+ZBw8AxYNQjC7j8bql4L8W7w41huRSrJ1XAIUj1BJV/abdCTVWmgCA4kEoRpeJhcPaV78u/rh7d596V1XZ1xAAAECGmCAHAAAAxyMUAwAAwPGYPgEAp6iXO6qqyrKkGgCg8BCKAeAUmXBIO1esTqhVTJ1oUzcAgM5g+gS6VCRm4n+ag612twMAAJARrhSjy7Tdutnr9ag/WzcDAIACwJViAAAAOB6hGAAAAI5HKAYAAIDjMacYANrwed1dvtSa1+NSMJK47XlZiVuKMe8eAPIBoRhZZVmWIqm+6VtW7psBMhQLh7t8qbVQS1T1m3Yn1EZXnaMyD7+wA4B8QChGVpk2K1IcdzarUgAAgDzCJQoAAAA4HqEYAAAAjkcoBgAAgOMxpxgA8onLUrAlcaWLk65S4bLUePCYQiesasGKFgBwagjFAJBHgi1Rrdm4K6F2slUqgi1RvbZ1j5qbw2mPBQC0j1AMABlItXax1+KKLAAUC0IxAGQg1drFA2+73p5mAABdjt+xAQAAwPEIxQAAAHA8QjEAAAAcj1AMAAAAxyMUAwAAwPFYfaITmo61KHjCovmSgxbOt6RI24/TsuzpBQAAoJMIxZ1wLBTJeJH9YhOJGb2153BC7WzjgB8GgC7k9biSfrDmh0sAsAehGABsEmqJqn7T7oTaqMsqbeoGAJyNUAzAEVyWpeY2V2W7c1UWAPBPhGIAjhA1Jmm60+cv7WtPMwCAvEModiifWmWCoYSaVVaqsLyZHWsxfxhIxed1q6qyLKneyx3N6PW93NGk1/dwRRVlsSAAyCpCsUOZYEj71tYn1PpXj5LKkkNxymOvHpHV/oBCFQuHtXPF6qR65fQvJIXdVEHZhENJr+83daLULflzs4crqksqPIpEXAk1AjQAdByhGAByIFVYrpg6sVPvaUJBvbVyjVpbI/HayQI0AKB9hGLEebxuKXg0qe5SLMXRAJD/HL2ePIAOIRR3Qmmspajm/sXCYe2rX5dUP2vUVTnvBQC6gpPXkwfQMYTiToiFghnP/QNgr1Q3wHm5YRQA8E+EYgCOkGpO78DbrrenmQ5KtcayxDrLANCVCMVOZX20VTOA/JdqjWUpS+ssuywFWxJXxWAOLgAnIBQ7VCRm9Naewwm1sw3f9ACnC7ZEmYMLwJH4KgcAAADH40oxABQ7pkQAQFqEYgAockyJAID0CMUAUERSrlTBKhUAkBah2AF8apUJhhJqVor1WS3LYkUKoMClWqmidthZRbXREABkA6HYAUwwpH1r6xNq/a8ekXycSV6RQmJVCqDQmXCIjYYAIA1CMdJKeQWZX8cCnZbpLnupjjvZsQCAU0MoRlqpriBz9RjovEx32Ut13MmOBQCcGkIxAKBolcZamE8NICOEYgBAh6W8gbesVGHl1zzlWCjIfGoAGSEUAwA6LOUNvNWjpDLCJoDCRCgGALTL63Ep2Gbt4xLutQVQZAjFAFBEMl3RoiNCLVHVb9qdUKu9tG+n3hMA8g2hGACKSKYrWpxMD1c0KVT3ckeTjnOlXKox49NILkvBlsT3LStxS2wgBMAmhGIAQJxJcWNaxdSJScdFUyzV2L8DgTbYEk3aeW901Tkq87AqBAB78NUHAAAAjseVYgBAuzKdp+yyLDW3uSFPYloEgMJAKC5QqdYIlU6yTqgltmkGkOBkAbZ7iq8Nmc5TjhqTNCVCYloEgMJAKC5QqdYIlVKvExqJsU0zgEQnC7Cf78SqEqmuKEtSL09URyJtivxgDiDPEIoBwIFOFmA7s3xbqivK0kc36q3Zsj+hNuqyylM+DwBkA6G4yHi8bil4NKFmdXKNUgDF52QBtiPLt2UqVQBPtcwbANiJUFxkYuGw9tWvS6j1v3qEPc0AgFIH8FTLvAGAnQjFhSrVzXMAAAA4JYTiApXq5jmJG+gAFK4erqjcweaEWsoVdQAgCwjFAIC8YEJB7Xvp5YRaqhV1HCPFVtgS6z4D2UIozjOp1h+2upXpcGviGp+p1hIFgEKR6YYgsqSgQzcESbUVtsS6z0C2EIrzjAmF9G7dSwm1/mP+Q2v+FEiodWYtUQCwW6qb786+/fNJ90pEjGwNhikvVHRgSofPiqi1+VhCzdu9m8KGb79AvuGzMs+kmit8jseV2RUVSZZlsXsdgIJkTGZf/6TUG4J09upxygBsxbS7LnFKx8drxqikNfE4T5lPoWPJu4zKMnr1qecSSqP+xyS5Wro2KHc2vKeUYvqGE67Qw7nyKhS/8MIL+sEPfqBIJKIpU6bolltusbulvJDpFqtS6m8q3HwHoFB1ZEOQzl49TrVTaP+rRyR9Ta0Mh/Xu2t8m1M6uHqlX/+/ypPccmWo77FBIrz79m4TaiDtuUNDbPaHWwxVN+QNBD1dUUSV+nCl77+R87FTTN/Jy6gbhHV0kb0JxQ0ODFi1apOeff14lJSWaNGmSLrvsMp177rl2t5Y1qa9K8EkMAOmkmpOcKix2SKqlLlP8pi0bFx88HpeO7GtIqJ3m75byB4J+UydK3U4t7KaaznF4T5N8paVJV6pThfKUY2zzDYHuSIuO7Ev8AanHWX0VdTn0Bk2csrwJxRs2bNDll1+u008/XZI0evRo1dXV6atf/WpGr3e5cj9FwOV2q9vpPZNqyrSXcKvef21DQqn/8Kqk97Q8yedJVevIsbl4T4/HUxB92vmeHo9H3kgk7/u06z2Pj0++92nne5ad1jM+RvncZ1e/p4lGtG/9awm1c26pUaylNaHmcllJry8tK1FLOHHpt+PHvt+UGO4q3a6cfOypPp6zb6hO+Z4ej0dNbUK4z+uR1a1bQs1b6pXVZpqGpZi2PLcm6f0unTBaLV53Qr1bpDWzMfZ5dXD/4aQ++/Q/XS3hlsQ+y3wKB8MZ1SKhxNdKklXqU0ub6GKFW5L6HHDjtXKVlSTUShSRCSWeR5I8pV5FQokfk1VWqiORf30v3384KJfHlRzyXZZCrck/EPTymOSLXil6T/X6Um/urnKXWFFFjgUTap5uZWox7pO8oo1/9r//cFAt//w/mbJ/mz/OeBtp8pllTH78bv3JJ5/UsWPHNGvWLEnSr3/9a23dulUPPfSQzZ0BAACg2OXNxKBYLCbrhF9TGWMSHgMAAADZkjehuKKiQoHAv5YdCwQCKi8vt7EjAAAAOEXehOJhw4Zp48aNOnjwoILBoNauXavhw4fb3RYAAAAcIG9utDvzzDM1a9YsTZ48Wa2trZo4caIGDx5sd1sAAABwgLy50Q4AAACwS95MnwAAAADsQigGAACA4xGKAQAA4HiEYgAAADgeoTgDL7zwgq699lpVV1frZz/7WdLz27ZtU21trUaPHq25c+cqcsKWq06QbnyOu+eee/T888/nsLP8kW6M6uvrNX78eF133XW688479eGHH9rQpX3Sjc9LL72kmpoajR07Vvfdd59aWpK3fy12mX6erVu3Tv/+7/+ew87yQ7rx+f73v6+RI0dq/PjxGj9+fLtjWKzSjdHbb7+tW2+9Vdddd52mTp3K16ETbNu2Lf5/Z/z48frc5z6ncePG2dSpfdL9H3rjjTc0YcIEXXfddZo2bZqOHDliQ5edYNCuDz74wIwcOdIcOnTINDc3m5qaGrNjx46EY8aOHWv+8pe/GGOM+drXvmZ+9rOf2dCpPTIZnw8++MBMmzbNDB482Dz33HM2dWqfdGN09OhRc8UVV5gPPvjAGGPMd77zHfPQQw/Z1W7OpRuf5uZmc+WVV5pAIGCMMeauu+4yv/zlL+1q1xaZfJ4ZY0wgEDBjxowxI0eOtKFL+2QyPtOmTTN//vOfberQfunGKBaLmerqarN+/XpjjDGPP/64eeyxx+xqN+cy/Rwzxphjx46ZsWPHmj/+8Y857tJemYzRTTfdZNatW2eMMeaRRx4x3/72t+1o9ZRxpTiNDRs26PLLL9fpp5+ubt26afTo0aqrq4s/v3fvXoVCIQ0ZMkSSVFtbm/B8sUs3PtJHP1leffXVuuaaa2zq0l7pxqi1tVX333+/zjzzTEnS+eefr/fff9+udnMu3fh069ZNr7zyivr27atgMKgDBw6oV69eNnace5l8nknSvHnz9NWvftWGDu2Vyfi8/vrrevLJJ1VTU6MHH3xQ4XDYpm7tkW6M3njjDXXr1i2+adb06dN1yy232NVuzmX6OSZJTz75pD7zmc9o6NChOe7SXpmMUSwWU3NzsyQpGAyqtLTUjlZPGaE4jcbGRvn9/vjj8vJyNTQ0nPR5v9+f8HyxSzc+knTHHXfohhtuyHVreSPdGPXu3Vv/8R//IUkKhUJasmSJRo0alfM+7ZLJ/yGv16v169frqquu0qFDh3TllVfmuk1bZTJGzzzzjC644AJddNFFuW7PdunGp7m5WZ/61Kc0e/ZsLV++XEeOHNHixYvtaNU26cbo3XffVd++fTVnzhx9/vOf1/33369u3brZ0aotMvkck6SjR49q6dKljvzhM5Mxuu+++zRv3jxdeeWV2rBhgyZNmpTrNjuFUJxGLBaTZVnxx8aYhMfpni92Tv/4M5HpGB09elRf/vKXNWjQIH3+85/PZYu2ynR8RowYoU2bNmnkyJH6xje+kcMO7ZdujP7xj39o7dq1uvPOO+1oz3bpxqd79+764Q9/qE984hPyeDy6/fbbtX79ejtatU26MYpEItq8ebNuuukmLV++XAMHDtSCBQvsaNUWmX4dWrlypUaNGqU+ffrksr28kG6MQqGQ5s6dqx//+Md67bXXdPPNN+vee++1o9VTRihOo6KiQoFAIP44EAiovLz8pM/v378/4flil258kNkYNTY26uabb9b555+vb37zm7lu0Vbpxufw4cN67bXX4o9ramr05ptv5rRHu6Ubo7q6OgUCAU2YMEFf/vKX4/+fnCLd+Ozbt0/Lli2LPzbGyOPx5LRHu6UbI7/fr8rKSn3605+WJI0bN05bt27NeZ92yfR7WX19va699tpctpY30o3RP/7xD/l8Pg0ePFiS9IUvfEGbN2/OeZ+dQShOY9iwYdq4caMOHjyoYDCotWvXxudcSdKAAQPk8/m0ZcsWSdKKFSsSni926cYH6ccoGo1q+vTpuuaaazR37lzHXWlPNz7GGM2ePVv79u2T9FEAvOSSS+xq1xbpxmjmzJlas2aNVqxYoSVLlqi8vFw///nPbew4t9KNT2lpqR5//HG99957MsboZz/7WXzKklOkG6OLL75YBw8e1Pbt2yVJr7zyii688EK72s25TL6XGWP0xhtv6OKLL7apS3ulG6PKykp98MEHevvttyVJL7/8cvyHrIJhx919hWblypVm7Nixprq62ixZssQYY8wdd9xhtm7daowxZtu2bWbChAlm9OjR5u677zbhcNjOdnMu3fgcd++99zpy9Qlj2h+jtWvXmvPPP99cd9118T9z5syxuePcSvd/6KWXXjLjxo0zNTU1ZtasWebIkSN2tmuLTD/P3nvvPcetPmFM+vGpq6uLP3/fffc57uu0MenH6K9//auZMGGCufbaa83tt99u9u/fb2e7OZdufPbv32+GDRtmZ4u2SzdG69atMzU1NWbcuHFmypQp5t1337Wz3Q6zjDHG7mAOAAAA2InpEwAAAHA8QjEAAAAcj1AMAAAAxyMUAwAAwPEIxQAAAHA8QjEAFKHbb79dBw8etLsNACgYhGIAKEK///3v7W4BAAqKs/a5BIAitGzZMj399NNyuVzq3bu3BgwYIEmaMmWKlixZoltuuUWDBw/Wm2++qbvvvttxu7kBQCYIxQBQwLZv366FCxdq+fLl6tevn3784x/Ht1n9yU9+ojPOOEOSdN555+k73/mOjZ0CQH4jFANAAdu4caOuvPJK9evXT5L0xS9+UZL0q1/9KuG4oUOH5ro1ACgohGIAKGBut1uWZcUfh0Ih7d27N+m4bt265bItACg43GgHAAXssssu08aNG9XY2ChJ+uUvf6nHH39cbrdbkUjE5u4AoHBwpRgACtj555+v2bNn64477pAk+f1+Pfzww3r00Ud166236nvf+57NHQJAYbCMMcbuJgAAAAA7MX0CAAAAjkcoBgAAgOMRigEAAOB4hGIAAAA4HqEYAAAAjkcoBgAAgOMRigEAAOB4hGIAAAA43v8H3aisOUZsaJcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 842.4x595.44 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set(rc={'figure.figsize':(11.7,8.27)})\n",
    "\n",
    "groups = sns.histplot(data = df, \n",
    "              x='ctr', \n",
    "              hue='exp_group', \n",
    "              palette = ['r', 'b'],\n",
    "              alpha=0.5,\n",
    "              kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ef639b04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=-13.896870721904069, pvalue=1.055849414662529e-43)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.ttest_ind(df[df.exp_group == 0].ctr,\n",
    "                df[df.exp_group == 3].ctr,\n",
    "                equal_var=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8db84f3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "CTRcontrol_0 = (df[df.exp_group == 0]['likes'].sum())/(df[df.exp_group == 0]['views'].sum())\n",
    "CTRcontrol_3 = (df[df.exp_group == 3]['likes'].sum())/(df[df.exp_group == 3]['views'].sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c4ae3445",
   "metadata": {},
   "outputs": [],
   "source": [
    "linearized_likes_0 = df[df.exp_group == 0]['likes'] - (CTRcontrol_0*(df[df.exp_group == 0]['views']))\n",
    "linearized_likes_3 = df[df.exp_group == 3]['likes'] - (CTRcontrol_0*(df[df.exp_group == 3]['views']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e11759ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='Count'>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsUAAAHnCAYAAABQRyC4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA7JUlEQVR4nO3de3BU933//9c5u6tFErprF2HAJDEYYmob1/6GEHcg7uQHwrJiV6Ytcaa4ybQxmRQn7pTEBYrHuRIPGRKSOtNLfu43aactceNwsS2ndWrsGBJ/oWkoCfbP38QIENLuSiskVpfdPXvO7w+hRTd0Aa2OtOf5GGusz9mj3bc+SKvXfvZzPh/DcRxHAAAAgIeZbhcAAAAAuI1QDAAAAM8jFAMAAMDzCMUAAADwPEIxAAAAPI9QDAAAAM/LaSj+8Y9/rIaGBm3YsEFf/OIXJUlHjx5VfX291q1bp71792bPPX36tBoaGrR+/Xrt2LFDlmXlsjQAAAAgK2eh+Ny5c3riiSf09NNP6+DBg/rVr36lI0eOaPv27Xr66af1wgsv6NSpUzpy5Igkadu2bdq1a5deeuklOY6j/fv356o0AAAAYIicheJ///d/17333quamhoFAgHt3btXhYWFWrx4sRYtWiS/36/6+no1NjaqublZfX19WrlypSSpoaFBjY2NuSoNAAAAGMKfqztuampSIBDQli1b1NLSog9+8INaunSpQqFQ9pxwOKxIJKJoNDrkeCgUUiQSyVVpAAAAwBA5C8WZTEbHjx/X9773PRUVFemTn/yk5syZI8Mwsuc4jiPDMGTb9qjHAQAAgOmQs1BcXV2t1atXq7KyUpL0oQ99SI2NjfL5fNlzYrGYwuGwampqFIvFssfb2toUDocn9XgdHd2ybWdqindRVdVctbcn3C4jL9G3uUG/5gb9mhv0a27Qr7lBv04t0zRUUVF81dtzForvuecefe5zn1NXV5eKi4v12muvqba2Vn/7t3+rpqYmLVy4UIcPH9aDDz6oBQsWKBgM6sSJE7rzzjt14MABrVmzZlKPZ9tOXoRiSXnzfcxE9G1u0K+5Qb/mBv2aG/RrbtCv0ydnofj222/Xn/zJn+ihhx5SOp3W3XffrY985CN6z3veo61btyqZTGrt2rWqra2VJO3Zs0c7d+5UIpHQihUrtHnz5lyVBgAAAAxhOI6TFy9B2tsTefFqKhQqUSx2ye0y8hJ9mxv0a27Qr7lBv+YG/Zob9OvUMk1DVVVzr377NNYCAAAAzEiEYgAAAHgeoRgAAACeRygGAACA5xGKAQAA4HmEYgAAAHgeoRgAAACeRygGAACA5xGKAQAA4HmEYgAAAHgeoRgAAACeRygGAACA5xGKAQAA4HmEYgAAAHgeoRgAAACe53e7AAAzm98/9LWzZdkuVQIAQO4QigFcld9vqvFnZxXp6JEkzasoUu2qGwnGAIC8QygGMKZIR4+aowm3ywAAIKeYUwwAAADPIxQDAADA8wjFAAAA8DxCMQAAADyPUAwAAADPIxQDAADA8wjFAAAA8DxCMQAAADyPUAwAAADPIxQDAADA89jmGcC4TNNQTWWRQuWF8vlMOY6UydhulwUAwJQhFAMYV01lkX7n4v8o+es2XXirUMFQtco21BOMAQB5g1AMYEISrVH1tbaquLJYjiOVuV0QAABTiDnFAAAA8DxCMQAAADyPUAwAAADPIxQDAADA8wjFAAAA8DxCMQAAADyPUAwAAADPIxQDAADA8wjFAAAA8DxCMYDJM035fP0fkrL/BwBgtmKbZwCTFqyqVPzwAfVF29QeMOWvqFRV/f3KZGxZlu12eQAATBqhGMA1Scba1Nfaqo5LSRXUdOvo6+/INAzVrrqRYAwAmHUIxQCuS8rKyEjbisZ7ZJiG2+UAAHBNmAgIAAAAzyMUAwAAwPMIxQAAAPA8QjEAAAA8j1AMAAAAz2P1CQBTxjSNERt5sDwbAGA2IBQDyBocaDOZyYfZ6rJCPX+sSZF4tyRpXkUR6xYDAGYFQjEASf2BuPPFQ+qLtmlOuFplG+qv6X6i8R41RxNTXB0AALlFKAaQ1RdtU29LiySpzOVaAACYTlxoBwAAAM8jFAMAAMDzCMUAAADwPEIxAAAAPI9QDAAAAM8jFAMAAMDzCMUAAADwPEIxAAAAPI9QDAAAAM8jFAMAAMDzCMUAAADwPEIxgAkzZMgwNOQDAIB84He7AACzR8Bvqr2zT04iqWQipWR3aszzTdOQzzf0tbdl2bksEQCAa5LTUPxHf/RHisfj8vv7H+bzn/+8uru79ZWvfEXJZFIbNmzQY489Jkk6ffq0duzYoe7ubt1111168skns18HYOZIWbYsy5ZlZcYNuNVlhXr+WJMi8W5J0ryKItWuupFgDACYcXKWOh3H0ZkzZ/Sf//mf2XDb19en2tpafe9739P8+fP1yCOP6MiRI1q7dq22bdumL37xi1q5cqW2b9+u/fv366GHHspVeQCmSTTeo+Zowu0yAAAYU87mFP/mN7+RJH384x/Xhz/8Yf3jP/6jTp48qcWLF2vRokXy+/2qr69XY2Ojmpub1dfXp5UrV0qSGhoa1NjYmKvSAAAAgCFyFoq7urq0evVq/fVf/7X+4R/+Qf/yL/+iCxcuKBQKZc8Jh8OKRCKKRqNDjodCIUUikVyVBgAAAAyRs+kTd9xxh+64445se+PGjdq3b5/uvPPO7DHHcWQYhmzbljHoMvaB45NRVTX3+oueIUKhErdLyFv07djaA6YCAZ/8AVOVlcWSJL/fJ8dW/+9kdtUJI9uW+tt+v18+nymf31Qg0P/UMrzt9/tUUVHsyvc2G/Hzmhv0a27Qr7lBv06fnIXi48ePK51Oa/Xq1ZL6g+6CBQsUi8Wy58RiMYXDYdXU1Aw53tbWpnA4PKnHa29PyLadqSneRaFQiWKxS26XkZfo27H5fKastK10OiN/2lY83i3DkCwrI8d25DiO5EiO0//7PNCWJBmGKkoCKikKyLJspVOWWuM9ymRsZSxb6bQlqf++Ojq6udBuAvh5zQ36NTfo19ygX6eWaRpjDqLmbPrEpUuX9NRTTymZTCqRSOi5557Tn//5n+udd95RU1OTMpmMDh8+rDVr1mjBggUKBoM6ceKEJOnAgQNas2ZNrkoDMMWKw9Vace64ql75oRb/n+f1Oxf/RzWVRW6XBQDAhOVspPiee+7RL37xCz3wwAOybVsPPfSQ7rjjDu3evVtbt25VMpnU2rVrVVtbK0nas2ePdu7cqUQioRUrVmjz5s25Kg3AeExTPp8pw5BC5YX9o8LjTGnqbo0q2RpRyuofbVb59JQKAMBUyOlCwJ/5zGf0mc98Zsix1atX6+DBgyPOXb58uZ599tlclgNggoJVlYofPqBkrE2Lu3pVvuxmXfQZSrpdGAAAOcLuGABGlYy1qbelVZfi3QpWVbtdDgAAOZWzOcUAAADAbEEoBgAAgOcRigEAAOB5hGIAAAB4HqEYAAAAnkcoBgAAgOcRigEAAOB5hGIAAAB4HqEYAAAAnkcoBgAAgOcRigEMYRiGDEPZDwAAvMDvdgEAZg7DkOJdfVIiqWQipWR3ShLJGACQ/xgpBjBEysrIsmxZl/8PAIAXEIoBAADgeYRiAAAAeB6hGAAAAJ5HKAYAAIDnEYoBAADgeYRiAAAAeB6hGAAAAJ5HKAYAAIDnEYoBAADgeYRiAAAAeB6hGAAAAJ5HKAYAAIDnEYoBAADgeYRiAAAAeJ7f7QIAuMfnM0f9HAAAryEUAx7l85nqfPGQ+qJtkqTSZUtkGC4XBQCASwjFgIf1RdvU29IiSQqGql2uBgAA9xCKAUwb0zSGTNOwLNvFagAAuIJQDGDaVJcV6vljTYrEuzWvoki1q24kGAMAZgRCMYBpFY33qDmacLsMAACG4HJzAAAAeB6hGAAAAJ5HKAYAAIDnEYoBAADgeYRiAAAAeB6hGAAAAJ5HKAYAAIDnEYoBAADgeWzeAcAVw7d8ltj2GQDgHkIxAFcM3vJZEts+AwBcRSgG4Bq2fAYAzBTMKQYAAIDnEYoBAADgeYRiAAAAeB6hGAAAAJ5HKAYAAIDnsfoEgJwyDemG6mJJUnlJUI4jXWhjxQkAwMxCKAaQU5Vlhbr5Nz9VojWqojMBza+uVrTyFrfLAgBgCEIx4CGDd5AbvpvcVDJMU6GKQpWXBCVHSkRi6jzbLLuoQMGMLc0lFAMAZhZCMeARPp+pzhcPqS/aJkkqXbZEhpGbxyoOV2vFueNK/bxdZcuWqs2XowcCAGCKEIoBD+mLtqm3pUWSFAxV5/SxulujSrZGVFBdldPHAQBgKrD6BAAAADyPUAwAAADPIxQDAADA8wjFAAAA8DxCMQAAADyPUAwAAADPIxQDAADA8wjFAAAA8DxCMQAAADyPUAwAAADPIxQDAADA83Ieir/61a/q8ccflyQdPXpU9fX1Wrdunfbu3Zs95/Tp02poaND69eu1Y8cOWZaV67IAAACArJyG4mPHjum5556TJPX19Wn79u16+umn9cILL+jUqVM6cuSIJGnbtm3atWuXXnrpJTmOo/379+eyLAAAAGCInIXiixcvau/evdqyZYsk6eTJk1q8eLEWLVokv9+v+vp6NTY2qrm5WX19fVq5cqUkqaGhQY2NjbkqCwAAABghZ6F4165deuyxx1RaWipJikajCoVC2dvD4bAikciI46FQSJFIJFdlAQAAACP4c3Gn3//+9zV//nytXr1aP/jBDyRJtm3LMIzsOY7jyDCMqx6frKqquddf+AwRCpW4XULe8nrftgdMBQI+SZLPb8jnH9rO+EwZhpH9kCEZhsZsS2Pfbly+3RzU9vv98vnMy4/f/zTk9/tUUVHsQq/MXF7/ec0V+jU36NfcoF+nT05C8QsvvKBYLKb7779fnZ2d6unpUXNzs3w+X/acWCymcDismpoaxWKx7PG2tjaFw+FJP2Z7e0K27UxJ/W4KhUoUi11yu4y85PW+9flMWWlb6XRGkpSxHGWsoW07Y8txnOyHHMlxNGZbGvt25/Lt9qC2ZVnKZOzLj99/Ya1lZdTR0S3Lsl3onZnH6z+vuUK/5gb9mhv069QyTWPMQdSchOJnnnkm+/kPfvADvfHGG3ryySe1bt06NTU1aeHChTp8+LAefPBBLViwQMFgUCdOnNCdd96pAwcOaM2aNbkoCwAAABhVTkLxaILBoHbv3q2tW7cqmUxq7dq1qq2tlSTt2bNHO3fuVCKR0IoVK7R58+bpKgvADGGahny+oZc5MGoMAJguOQ/FDQ0NamhokCStXr1aBw8eHHHO8uXL9eyzz+a6FACjGJjDfw1T+adUdVmhnj/WpEi8W5I0r6JItatuJBgDAKbFtI0UA5h5DEOKd/UpZWWkRFJG0lL26jkXROM9ao4mXHt8AIB3EYoBj0tZGSVTGVmWLSMz+y9WBQDgWuR8m2cAAABgpiMUAwAAwPMIxQAAAPA8QjEAAAA8j1AMAAAAzyMUAwAAwPMIxQAAAPA8QjEAAAA8j1AMAAAAzyMUAwAAwPMIxQAAAPA8v9sFAJhehmFc/r/LhQAAMIMQigEPMQwp3tWnlJWREkkZSUsS6RgAAEIx4DEpK6NkKiPLsmVknGl/fMM0FaooVHlJUI4j2bat1njPtNcBAMBghGIA06ooVKUV544r9fN2lWZs3VxVpZ9U3up2WQAAjyMUA5h23a1RJVsjSlm20umMVO52RQAAr2P1CQAAAHgeoRgAAACeRygGAACA5xGKAQAA4HmEYgAAAHgeoRgAAACeRygGAACA5xGKAQAA4HmEYgAAAHgeO9oBcJVhmgpVFKq8JCjHkWzbVmu8x+2yAAAeQygG4KricLVWnDuu1M/bVZqxdXNVlX5SeavbZQEAPIZQDMB13a1RJVsjSlm20umMVO52RQAAr2FOMQAAADyPUAwAAADPIxQDAADA8wjFAAAA8DxCMQAAADyPUAwAAADPIxQDAADA8wjFAAAA8DxCMQAAADyPUAwAAADPIxQDAADA8wjFAAAA8Dy/2wUAyB2fzxz1cwAAMBShGMhTPp+pzhcPqS/aJkkqXbZEhuFyUQAAzFCEYiCP9UXb1NvSIkkKhqpdrgYAgJmL91MBAADgeYRiAAAAeB6hGAAAAJ5HKAYAAIDnEYoBAADgeYRiAAAAeB6hGAAAAJ7HOsVAnhnYuW6272BnmsaI78GybJeqAQDkO0IxkEcG72I323ewqy4r1PPHmhSJd0uS5lUUqXbVjQRjAEBOEIqBPDOwi10+7GAXjfeoOZpwuwwAgAfM7vdXAQAAgCkwoVC8ffv2EcceffTRKS8GAAAAcMOY0yeeeOIJRSIRnThxQvF4PHvcsiydO3cu58UBAAAA02HMULxx40a9/fbbeuutt7R+/frscZ/Pp5UrV+a6NgAAAGBajBmKb731Vt166636wAc+oJqamumqCQAAAJhWE1p9oqWlRdu2bVNnZ6ccx8keP3ToUM4KAwAAAKbLhELxrl271NDQoFtuuUXGbF74FAAAABjFhEKx3+/Xxz72sVzXAgAAALhiQkuyLV26VG+99VauawEAAABcMaGR4nPnzunBBx/UDTfcoGAwmD3OnGIAAADkgwmF4sceeyzXdQAAAACumVAovvnmm6/pzr/xjW/opZdekmEY2rhxoz72sY/p6NGj+spXvqJkMqkNGzZkA/fp06e1Y8cOdXd366677tKTTz4pv39C5QEAAADXZUKp8/3vf78Mw5DjONnVJ0KhkF599dWrfs0bb7yhn/70pzp48KAsy9K9996r1atXa/v27fre976n+fPn65FHHtGRI0e0du1abdu2TV/84he1cuVKbd++Xfv379dDDz00Nd8lAAAAMIYJheI333wz+3kqldLhw4f1zjvvjPk173vf+/Td735Xfr9fkUhEmUxGXV1dWrx4sRYtWiRJqq+vV2Njo5YsWaK+vr7sLnkNDQ3at28foRgAAADTYkKrTwxWUFCghoYGvf766+OeGwgEtG/fPtXV1Wn16tWKRqMKhULZ28PhsCKRyIjjoVBIkUhksqUBAAAA12RCI8UXL17Mfu44jk6dOqWurq4JPcCjjz6qP/3TP9WWLVt05syZIZt/DEzHsG171OOTUVU1d1Lnz2ShUInbJeQtL/Rte8BUIOCTz2/I5+//XJJ8fkMZnynDMGQaRv/vmCEZhka0jUm0pXHOv3y7Oaw91n37/X75fObl+vufpvx+nyoqit3pVJd44efVDfRrbtCvuUG/Tp9JzymWpKqqKu3YsWPMr/n1r3+tVCql9773vSosLNS6devU2Ngon8+XPScWiykcDqumpkaxWCx7vK2tTeFweFLfSHt7QrbtjH/iDBcKlSgWu+R2GXnJC33r85my0rbS6YwylqOMZcuybEmSnXFkZ2w5jiPbcfpffDqS42hE27nc1gTa0jjnX77dHtYe674ty1ImYytj2UqnLUmSZWXU0dGd/X7ynRd+Xt1Av+YG/Zob9OvUMk1jzEHUSc8pnqjz589r3759+ud//mdJ0ssvv6xNmzbpqaeeUlNTkxYuXKjDhw/rwQcf1IIFCxQMBnXixAndeeedOnDggNasWTPpxwQwlGFI8a4+payMlEjKSFrKDu8CAICsCYVi27b1ne98R6+++qosy9Ldd9+tLVu2jLlk2tq1a3Xy5Ek98MAD8vl8Wrdunerq6lRZWamtW7cqmUxq7dq1qq2tlSTt2bNHO3fuVCKR0IoVK7R58+ap+Q4Bj0tZGSVTGVmWLSMz+99NAQAgFyYUir/2ta/pzTff1MMPPyzbtvWv//qveuqpp7R9+/Yxv27r1q3aunXrkGOrV6/WwYMHR5y7fPlyPfvss5MoHQAAAJgaEwrFr732mv7t3/5NgUBAkvTBD35QH/7wh8cNxQAAAMBsMKEl2RzHyQZiqX9ZtsFtAAAAYDabUChevny5vvzlL+vs2bM6d+6cvvzlL1/z1s8AAADATDOhUPzEE0+oq6tLmzZt0u///u+ro6NDf/VXf5Xr2gAAAIBpMWYoTqVS+tznPqdjx45p9+7dOnr0qG677Tb5fD7NnZs/m2UAAADA28YMxfv27VMikdBv//ZvZ4994QtfUFdXl775zW/mvDgAAABgOowZil955RV97WtfU1VVVfbYvHnz9NRTT+k//uM/cl4cAAwwTUM+nym//8oHAABTZcwl2QKBgObMmTPi+Ny5c1VQUJCzogBguOqyQj1/rEmReLckaV5FkWpX3eiZbZ8BALk1Zig2TVOJRGLE/OFEIiHLsnJaGAAMF433qDmacLsMAEAeGvP9x/vuu087d+5UT09P9lhPT4927typdevW5bw4AAAAYDqMGYoffvhhlZSU6O6779Yf/MEfaOPGjbr77rtVWlqqT33qU9NVIwAAAJBT406f+MIXvqAtW7bol7/8pUzT1G233aZwODxd9QEAAAA5N2YoHrBgwQItWLAg17UAAAAArmBNIwAAAHgeoRgAAACeRygGAACA5xGKAQAA4HmEYgAAAHgeoRgAAACeRygGAACA5xGKAQAA4HmEYgAAAHgeoRgAAACeRygG8oxhSIZhyDDcrgQAgNnD73YBAKaO32+qszuleLxbSiRlJC1JpGMAAMZDKAbyTDqTUTKVkWXZMjKO2+UAADArMH0CAAAAnsdIMYC84fcPfZ1vWbZLlQAAZhtCMYC84PebavzZWUU6eiRJ8yqKVLvqRoIxAGBCCMUA8kako0fN0YTbZQAAZiHmFAMAAMDzCMUAAADwPKZPAJiVTNOQz3fldf3gzwEAmCxCMYBZqbqsUM8fa1Ik3i1JWr64UgYblQAArhGhGMCsFY1fubAuXFHkcjUAgNmMUAzMcoOnDZgmY6UAAFwLQjEwi/l8pjpfPKS+aJskqXTZEvl8PpermjqmId1QXSxJKi8JynGkC20J2TbbVwMAphahGJjl+qJt6m1pkSTNCVe7XM3Uqiwr1M2/+akSrVEVnQlofnW1opW36EJbt9ulAQDyDKEYmGW8tuJCojWqzrPNsosKFMzY0txb3C4JAJCHCMXALDLadAmDScQAAFw3QjEwywyeLhEM5dd0CQAA3JL/770CAAAA4yAUAwAAwPMIxQAAAPA8QjEAAAA8jwvtAMwohmkqVFGo8pKg5EgsrwEAmA6EYgAzSnG4WivOHVfq5+0qW7ZUbT5CMQAg9wjFAGac7taokq0RFVRXXfN9mKYxYnMTy7KvtzQAQJ4iFAPIS9VlhXr+WJMi8f4toedVFKl21Y0EYwDAqAjFAPJWNN6j5mjC7TIAALMAq08AAADA8wjFAAAA8DxCMQAAADyPOcXALGcYknF5LV8WLwMA4NoQioFZzDCkSz1pxS6vsFCQtFyuCACA2YlQDMxylu0omcpIkmzbcbkaAABmJ+YUAwAAwPMIxQAAAPA8QjEAAAA8j1AMAAAAzyMUAwAAwPMIxQAAAPA8QjEAAAA8j1AMAAAAzyMUAwAAwPMIxQAAAPC8nIbib33rW6qrq1NdXZ2eeuopSdLRo0dVX1+vdevWae/evdlzT58+rYaGBq1fv147duyQZVm5LA0AAADIylkoPnr0qH7yk5/oueee0w9/+EP98pe/1OHDh7V9+3Y9/fTTeuGFF3Tq1CkdOXJEkrRt2zbt2rVLL730khzH0f79+3NVGgAAADBEzkJxKBTS448/roKCAgUCAd100006c+aMFi9erEWLFsnv96u+vl6NjY1qbm5WX1+fVq5cKUlqaGhQY2NjrkoDAAAAhshZKF66dGk25J45c0YvvviiDMNQKBTKnhMOhxWJRBSNRoccD4VCikQiuSoNAAAAGMKf6wd4++239cgjj+izn/2sfD6fzpw5k73NcRwZhiHbtmUYxojjk1FVNXeqSnZdKFTidgl5Kx/6tj1gKhDwSZJMnyHDkMxBvy8DbcMwJOPa2sYk2v2POcb5l283h7Un9FijfK3f71cg4JfPZ8rnNxUI9D+Njdf2+32qqCiepn+lqZEPP68zEf2aG/RrbtCv0yenofjEiRN69NFHtX37dtXV1emNN95QLBbL3h6LxRQOh1VTUzPkeFtbm8Lh8KQeq709Idt2pqx2t4RCJYrFLrldRl7Kh771+UxZaVvpdEaSZGccOY5kO1d+9gfajuPIcK6t7VxuawLt/scc4/zLt9vD2hO579G+1rIspdOWMhlbGctWOt1/Ue54bcvKqKOjW5ZlT9O/1vXJh5/XmYh+zQ36NTfo16llmsaYg6g5mz7R0tKiT33qU9qzZ4/q6uokSbfffrveeecdNTU1KZPJ6PDhw1qzZo0WLFigYDCoEydOSJIOHDigNWvW5Ko0AB5kmoZ8PlN+/5UPAAAG5Gyk+Dvf+Y6SyaR2796dPbZp0ybt3r1bW7duVTKZ1Nq1a1VbWytJ2rNnj3bu3KlEIqEVK1Zo8+bNuSoNQJ4wDemG6v4pEeUlQTmOdKFt9HeNqssK9fyxJkXi3ZKkeRVFql1146wZOQYA5FbOQvHOnTu1c+fOUW87ePDgiGPLly/Xs88+m6tyAOShyrJC3fybnyrRGlXRmYDmV1crWnmLLrR1j3p+NN6j5mhimqsEAMwGOb/QDgByKdEaVefZZtlFBQpmbGnuLW6XBACYhZhUBwAAAM8jFAMAAMDzCMUAAADwPEIxAAAAPI9QDAAAAM8jFAMAAMDzCMUAAADwPEIxAAAAPI9QDAAAAM9jRztgljEMyTCM/s9drgUAgHxBKAZmOJ/vyhs6BQU+dXanFI9397eTlltlAQCQVwjFwAzm85nqfPGQ+qJtkqTSZUvkSEqmMpIk23ZcrA4AgPxBKAZmuL5om3pbWiRJc8LVLlcDAEB+4kI7AAAAeB6hGAAAAJ7H9AkAs4ZhmgpVFEqSykuCkqP+5TgAALhOhGIAs0ZRqEorzh3XxfOtKjoTUNmypWrzEYoBANePUAzMQAPLsA1ejg39uluj6jzbLLuoQAXVVW6XAwDIE4RiYIYZvAxb6bIlzA4AAGAaMAwFzEADy7Al4xfdLgUAAE8gFAMAAMDzCMUAAADwPEIxAAAAPI8L7QAXDF9VIpOxXaoEAABIhGJg2g1eXUKS5oSrVbahnmAMAICLCMWACwZWlxhQNsa5hiEZl9dlY3U2AAByg1AMzGCGIV3qSSsW75YkFSQtlysCACA/EYqBGc6yHSVTGUmSbTsuVwMAQH5i9QkAAAB4HqEYAAAAnkcoBgAAgOcRigEAAOB5hGIAAAB4HqtPAMgbhmkqVFEoSSovCar9Yq/LFQEAZgtCMYC8URSq0opzx3XxfKuqe29Qx4K7dDZyye2yAACzAKEYQF7pbo2q82yziuYEpAVuVwMAmC2YUwwAAADPIxQDAADA85g+AbjNNOXzXXl9OvhzAAAwPQjFgMuCVZWKHz6gvmibJKl02RIZhstFAQDgMYRiYAZIxtrU29IiSQqGql2uJj8YpqmKkqBuqC6WxBJtAICxEYoB5KXCUJUWvfm6CpsuSNKIJdpM0xgxVcWy7GmvEwAwMxCKAeSt7khMnWebJWnEEm3VZYV6/liTIvFuSdK8iiLVrrqRYAwAHkUoBmYYwxj4MMTU4tyKxnvUHE24XQYAYAYgFAMziGEYSvSm1Z1IqSPerYKk5XZJeYM5xgCAsRCKgRnGytiyrIySqYxs23G7nLwx3hxjAIC3EYoBeMZYc4xH4/dzIR4AeAWhGABG4febavzZWUU6eiRxIR4A5DtCMQBcRaSDC/EAwCvYTxYAAACeRygGAACA5xGKAQAA4HmEYsBlQzbrYLcOAABcwYV2gIuGb9ZRNCegUreLAgDAgwjFwDTw+cxRP5eGbtZR4PdNd2kAAECEYiDnfD5TnS8eUl+0TZJUumwJ0yQAAJhhCMXANOiLtqm3pUWSFAxVu1wNAAAYjgvtAAAA4HmMFAPwJMM0VVES1A3VxZKkUHnhmOebpjFyPjhbPgNA3iAUA/CkwlCVFr35ugqbLkiSyhfWyHf3u68adKvLCvX8sSZF4t2SpHkVRapddSPBGADyBKEYgGd1R2LqPNssSSoI+GSahvz+/tHg4aPCkhSN96g5mpjWGgEA04NQDACS/D5Tr51s0VtNcUnS8sWVMsQyIQDgFYRiALgs3tmXHQkOVxS5XA0AYDqx+gQAAAA8L6ehOJFI6L777tP58+clSUePHlV9fb3WrVunvXv3Zs87ffq0GhoatH79eu3YsUOWZeWyLCDnfD5zyAcAAJjZcvbX+he/+IU+8pGP6MyZM5Kkvr4+bd++XU8//bReeOEFnTp1SkeOHJEkbdu2Tbt27dJLL70kx3G0f//+XJUF5NzADnaR//2MIv/7GSVeP8IOdgAAzHA5C8X79+/XE088oXA4LEk6efKkFi9erEWLFsnv96u+vl6NjY1qbm5WX1+fVq5cKUlqaGhQY2NjrsoCpsXADna9LS1Kxi+6XQ4AABhHzi60+9KXvjSkHY1GFQqFsu1wOKxIJDLieCgUUiQSyVVZAAAAwAjTtvqEbdsyBr2H7DiODMO46vHJqqqaOyV1zgShUInbJeSt6erb9oCpQMAnSfL5Dfn8V29nJBmGIdMwpP7/rrQlGYayn0+mbQzc3zW0jUm0pXHOv8r3N6HHGqtvxmtPtu+M/qkvgUD/06LPZ17+dxq97ff7VFFRfI0/IRPDc0Fu0K+5Qb/mBv06faYtFNfU1CgWi2XbsVhM4XB4xPG2trbslIvJaG9PyLadKanVTaFQiWKxS26XkZemq299PlNW2lY6nZEkZSxHGau/bRiGuhIpdXf2qSN6SUVzAipV/4tB23Gk/v+utCU5jrKfT6btOI4M59razuW2JtCWxjn/Kt/fRO77al87ofYk+84wTJUWBRQqC0qSSooCirZ3K53uv/A3k7Ev/zv2ty0ro46O7pztaMdzQW7Qr7lBv+YG/Tq1TNMYcxB12i6Lv/322/XOO++oqalJmUxGhw8f1po1a7RgwQIFg0GdOHFCknTgwAGtWbNmusoCpp2VsWVZGSVTGbYInkEGtn1e+ebLWvnmy1p0+nVVlhW6XRYAYJpM20hxMBjU7t27tXXrViWTSa1du1a1tbWSpD179mjnzp1KJBJasWKFNm/ePF1lAUDW4G2fi+YEpAUuFwQAmDY5D8U//vGPs5+vXr1aBw8eHHHO8uXL9eyzz+a6FACYMqZpjFiDmpF/AJi92OYZAK5BdVmhnj/WpEi8W5I0r6JItatuJBgDwCxFKAaAaxSN96g5mnC7DADAFGD/WQAAAHgeI8UAMArDNFVREtQN1f1rEbNVNwDkN0IxAIxiYIm2wqYLmlsTVuxdaxWN97pdFgAgRwjFAHAVg5doAwDkN0IxMEUGlucavkyXMWT74uwmcAAAYAYhFANTwOcz1fniIfVF21S6bEl2/qlhGEr0ptWdSKkj3p3d1hkAAMwsrD4BTJG+aJt6W1qUjF8ccpxtnQEAmPkIxQAAAPA8QjEAAAA8j1AMAAAAz+NCOwAYh2GaKikuUHV5oWy7f1748M08TNMYsfIIc8gBYPYgFAPAOIrD1Sr+ryMyzrfohnRm1M08qssK9fyxJkXi3ZKkeRVFql11I8EYAGYJQjEATEBPJKpL5y4ombKuek403qPmaGIaqwIATBVCMXCNBr9VPvxtc4DpFAAwuxCKgWsweLMOSUM27ED+m8gcY6ZTAMDsQigGrtHAZh2SFAxVu1wNptNE5hhLTKcAgNmEUAwA12Aic4wBALMHoRiYIOYQAwCQvwjFwAQwhxjXiwvvAGBmIxQDE8QcYlwPLrwDgJmNUAxMkmEYMgxd/jDkOI7bJWGW4MI7AJi5mBgJTIJhGIp39eliIqmORErxrj6Z5tCQDAAAZh9GioFJSlkZWZYty8pIkto7++QkkkomUuru6lPFoIBsGBLjyAAAzHyEYuA6pSx7SEi+1JtWdyKljni3iuYEVOpyfQAAYHxMnwCmmJXpD8jJVIaLqAAAmCUIxQAAAPA8pk8AgwxeRzaTYZQXAACvIBQDlw3eoGNOuFplG+oJxgAAeAShGBhk8AYdZS7XgtnDME2VFBeourxQtm3LNAxVlAblOJJt22qN90zofvx+ZrQBgFsIxQBwnYrD1Sr+ryMyzrfohnRGod9aLvPNN1VyvkU3V1XpJ5W3jnsffr+pxp+dVaSjP0DfECrR/3PnAi7WBIBpQigGRmOaQ+YXD/4cGE1PJKpL5y4ombI0tyYso+uiLp27oHQ6I5VP7D4iHVd2vPP7fbkrFgAwAqEYGEWwqlLxwwfUF22TJJUuWyI2q8NUMk2DF14AMIMQioGrSMauzC8Ohqpdrgb5prqsUM8fa1Ik3i1JWr64UoZ45QUAbiEUA4BLovEr0yXCFUUuVwMA3kYoBoAZyDSMEVMquOgOAHKHUAwAM1BV2Zwh0yvmVRSpdtWNBGMAyBFCMQDMUIOnVwAAcotQDE/j6n9MN9M0VFNZpPKSyW/uMdzwzT4YRQaAa0coRt4ZHG7H2qZ58LbOEsuuYXrUVBbpdy7+j+xXOlSasSe8ucdwwzf7YHoFAFwfQjHyyuCgOydcrbIN9WMG48HbOl9t2TXDMGQYyn4A1yvRGpXdFlPKsmVlHIXeXZgdOZbjyDGk8rlBWZatC20J2bYz6v0M3uwDAHB9CMXIO4ODbtl13pdhGIp39UmJpJKJlJLdKYm1ZDGFisPVWnHuuFI/b1dpxtbty5YqffGiUv/ZrnBVtaKVt+hCW/e49zN8MxCJ6RQAMBmEYmAcKSsjy7JlXf4/MBmGaSpUcWUk2HHsEW85dLdGlWyNKGXZClZVKdnWpmQkoqBlS3NvkTT+DnjDNwNhOgUATA6hGJ4ykQvrmC6BqTR8JLh82VJd9BlKTvJ+JrIDHqtVAMC1IxTDU8a7sM4wxHQJTLnhI8HXih3wACB3CMXwlIlcWMd0CQAAvIdQDAB5aPgcZF7gAcDYCMXIX6Y5oXnDwEw1cJGepOyFemMt0TbY4DnIXHQHAOMjFCNvBasqFT98IDuHOPnemy8H4SvLrGV60vIZhhxn/JABTLeiUJVWnDuui+dbVXQmoPnVE1+iTeLCOwCYDEIx8loy1j+H2DAMpeaHsqPDA/OGM7Yjn9tFAmPobo2q82yz7KICBTNXlmgDAEwtQjFmvfGWWRsYGTY6+9THihLIY6Yh3VBdLGnodAs29gCA8RGKMasN3tZZGn2ZNenyyHDGYUUJzGrjzTGuLCvUzb/5qRKt0SHTLdjYAwDGRyjGrDeRZdaAfDBijnE4JOem31aso1flJUHJkRKR2KjTLZhfDABjIxRj1pnIrnQTZWhgBQqDlSgwKwyeY1xZPTQkly1bqjbflR/kwVtMS5IjR47tKFxZpEBg6Gx6Ro0BeB2hGLPKRKdLTHSrZp9pqL2zTykrIyWSMpKWmG+M2WRwSC6oHrpb3sDIcurn7XrPsqXq7Yr3B+g5Ab1zLqSTC+5U7GLvNU2n8PuZowwgvxCKMeuMN11isls1pyxbyVT/XGMjw9JsyC8DW0wXVFcNCdDpjK1oYc+El3cbzO831fizs4p09EiS5lcWacPqdymTuRKMCckAZhtCMfISWzUDE3e11SkGHxsceCUp0nFljnK4omjSF/Ix0gxgpiEUY8YZ/sd5+B9jAFNr+OoU8yuLVHf3uxU78EMlY20KhkOqrPtw9nfR5zNlDnvzZTIX8g0faWY1DAAzAaEYM8JAEPb5zCG70M2ZN/KP8WiMyxOHuVgOuDaDQ224okivnWxRya/P6dK5C1o4t0zG84N+L8PVqqq4Teci176axeCRZgCYCQjFcN3gi+dKly3J7kIn9c8ZHhySR7uwbmAOMRfLARM33prH8c4+zUn3z7eXpLazLYr9+qwkKZzKqOLGOUM2Cmm/2Ju9bzYLATAbEYqRE+NNgRi+rNrAxXOjXTg3PCSPJmVluFgOmIQRax5f3ujjahfeWRknG5ALKiu16M3XVdh0QZJU3XuDOhbcpbORS5KkUHmhXj/VqnhnrwxDqi6bow/cekM2cJumoVB5oRzbkWkYClUUyucz5Vz+1c1kbOYcA5h2hGJMueHLps0JV6tsQ/2QKRDjLas2eDrE1ZZcG7gdwLUZvBrF4I0+JvS1lzcJkaSiOQFpwZXbyuYGVfM/r2nO+RaFfmu5zAsJ/fLHEaUzGVXfskxG4pIWn23RDemMQr+1XM75Ll34f38sx7k8NaP+fj1/9MyYc44JzQCmGqEYV3U9F7wNXjZNksrGuD0Yqr4cfq+sLTx4OkSmJy2fYci5PIzEdAlg6g2fTiFHE37VaZimKkqC2ekUJcUF6on0B+65NWEZXRcVP3NeyZSlYFW1jK6LunTugpIpS3NrwrIvdih4KS7H6X9I0zTGnHM8FRfqjReqCd2A9xCKMarxRnsnxTTH3IXOMKRLPWnF4t3ZkDt47WDbduQftusc0yWAqTV8OsXw3fHGUhiqGjKdoqZwhdonsdvk4E10Sop71f5/22UMeqE7fI6yz2eOe6HeWKF2vFB9vaGbQA3MToRiXNV4o73DDV5BYmDkV5LmVFeOuFjONAdNgZBk2c5VQy67zgHTY6zd8cb92kHTKaoX3zDpxx54ITwnbau3OznktuFLxi1fXClDhkzTUE1lUXbb6oHnINM09NovLije1af2zl6FykeG2vFC9bWujsFyc8DsNaNC8aFDh/Ttb39blmXp4Ycf1kc/+lG3S8o7V1vSbPjIhjN88HXYaK80dDrF8BUkEr2XR34l+XvTKujuzAbsOeHqKyPDkgqS1rh1s+sc4A2GaaqkuEDV5YWybVumYaiiNKh4Z1KO7ag13qNwRZEkqaaySL9z8X9kn+nQL49LZcuWKH2xU8m2NpWkMgqFqvXm4v8lx5ACAV/2Ma72PDiYaSg7HSQ87EJAaeR0soHnUJ/PVOxij5zLFxXajiOfz5xUKGatdsAdMyYURyIR7d27Vz/4wQ9UUFCgTZs2adWqVVqyZInbpc14E3mCHzhv+IhtquOikrE2dXansn9QeiIxLbzjliHTCYNVQ0d7h0+n8PtNJWNt6mttVWG4OjvyK115Qh9tZFhS9op0ACgOV6v4v47IOH/lQjzzzTdVcr5Fy0LVevOm/5VdQs5xbCXeismORZWybJnlFUq2tSnZGlHKshWs6p8Skmpr1zvnQjq54E51JFJasqBMPsO4EnpHGWlesrBcoZOvKdES0fzb36v44ZPqjVz9+e+nv4oo3tWnd80v05KFFQqdfFWJlogCAVPt8UUTnn42kQuVByMwA1NnxoTio0eP6v3vf7/Ky8slSevXr1djY6P+7M/+zN3CpslYwXYyI7SpjosjQu/g9vDlzVJtbeptaVU83p39g3Lp/AWF3zXy7c/BX2v4TBUU+GTbpkzT0H//3zYZXb3qinePGPn1+Uxd6p3cyDAA7+qJRIdciDdwYV425P68XaUZW+XLluqiz1ByjPvqbo0q2RpROmMrWtijWGdSlSVBVZYV6ubf/FSJ1qiK5gTU0n2z+uId6rzQqjkFfr1nyRK1xfqnhJQvrJHRd+nKdDLTVOjy85/UH4rnn3pNc863yj8noPcsW5r92mCBT8nSwiE1jfXO3OAlKgeUDbptstd6jLVV99UwJxpeNWNCcTQaVSgUyrbD4bBOnjw54a83h+85Og2GP3EMZpoTv8jENA11nziuVFeXMuVlKrrjziHreQ7cJkkFpaUqvvOuIbf7i4tVUJaUr7BQgXRadrL/T8Ro7bnvXqxAaYkkqWjBfBWUFPe3e9IqffdCpavKVFxVoaIb5iswt3j0cyUFFt6oi2+8od6OTvn9pm6aV6POm29SQVmp5i5coIKSufLN7R+FKb1xoVJlJXIK+9/yHO32OWUl8s0tVuW7F0o9FfKVlMjKZK6rPfD9pMKVymSc/nZl6ZD2pL52gu2SxQtVWFaiYHmpggX+62oP7pvhfTVee8r77ipt25ZKrqE/ct2X19N3bvXl4HY6XKm5U/Sz5GZfTvnvdNcl+WxLBRlHhSVF8i+7aUJ9W1BRoZU3hZTotTS/uliOpKJ4iWSlFCzwy19UqEAqpWBZqQoK/ArOLVL1spsULC9V+eKFCiR75C/pf/6b++53qfP4/1FP+0VJUtmNC1RcWaZ0X5+Cw77W7/Np7qJ5Cgb9sm1Hpmno/zt/UYmetCQpVDFHlefeUqqz/zm+8IYalbxnsQrK+h+roLJyyNcOPN9Lkr+4OHvbeH9bhv/tuJrh9c0tCujmheUTekdvzpzAuOd4hW1P3QuJggLf+CfNQm682BovKxqOM2L2qCu+/e1vK5lM6jOf+Ywkaf/+/Tp16pQ+//nPu1sYAAAA8t7EhzNzrKamRrFYLNuOxWIKh8MuVgQAAACvmDGh+AMf+ICOHTumeDyu3t5e/ehHP9KaNWvcLgsAAAAeMGPmFM+bN0+PPfaYNm/erHQ6rY0bN+q2225zuywAAAB4wIyZUwwAAAC4ZcZMnwAAAADcQigGAACA5xGKAQAA4HmEYgAAAHgeoRgAAACeRyieIU6cOKGNGzfq/vvv18MPP6zm5mZJUldXlz7xiU9ow4YN+uhHPzpkgxNM3Ne//nV985vfzLbp1+t36NAh3XvvvVq3bp3+6Z/+ye1yZr1EIqH77rtP58+flyQdPXpU9fX1Wrdunfbu3etydbPTt771LdXV1amurk5PPfWUJPp1KnzjG9/Qvffeq7q6Oj3zzDOS6Nep9NWvflWPP/64JPp12jmYEe655x7n9OnTjuM4zve//31ny5YtjuM4zpNPPun8zd/8jeM4jvPcc885n/70p90qcVbq6upy/vIv/9K57bbbnH379mWP06/Xp7W11bnnnnucjo4Op7u726mvr3fefvttt8uatf77v//bue+++5wVK1Y4586dc3p7e521a9c6Z8+eddLptPPxj3/ceeWVV9wuc1Z5/fXXnT/8wz90ksmkk0qlnM2bNzuHDh2iX6/Tz372M2fTpk1OOp12ent7s3+76NepcfToUWfVqlXO5z73OZ4HXMBI8QyQSqX06U9/WsuXL5ckLVu2TC0tLZKkV155RfX19ZKk++67T6+++qrS6bRrtc42L7/8st71rnfpYx/72JDj9Ov1OXr0qN7//vervLxcRUVFWr9+vRobG90ua9bav3+/nnjiiezW9idPntTixYu1aNEi+f1+1dfX07+TFAqF9Pjjj6ugoECBQEA33XSTzpw5Q79ep/e973367ne/K7/fr/b2dmUyGXV1ddGvU+DixYvau3evtmzZIonnATcQimeAgoIC3X///ZIk27b1rW99Sx/60IckSdFoVKFQSJLk9/s1d+5cxeNx12qdbR544AF94hOfkM/nG3Kcfr0+g/tPksLhsCKRiIsVzW5f+tKXdNddd2Xb9O/1W7p0qVauXClJOnPmjF588UUZhkG/ToFAIKB9+/aprq5Oq1ev5ud1iuzatUuPPfaYSktLJfE84AZC8TR78cUXtWbNmiEff/zHfyypf8T4L/7iL2RZlh555JFRv95xHJkm/2zDjdWvE0G/To5t2zIMI9t2HGdIG9eH/p06b7/9tj7+8Y/rs5/9rBYtWkS/TpFHH31Ux44dU0tLi86cOUO/Xqfvf//7mj9/vlavXp09xvPA9PO7XYDXbNiwQRs2bBhxvLu7W5/85CdVXl6ub3/72woEApL6Xxm2tbWppqZGlmWpu7tb5eXl01z1zHe1fr0a+vX61NTU6Pjx49l2LBbLvvWP61dTUzPk4k/699qcOHFCjz76qLZv3666ujq98cYb9Ot1+vWvf61UKqX3vve9Kiws1Lp169TY2Djk3Tj6dfJeeOEFxWIx3X///ers7FRPT4+am5vp12nG0NgMsW3bNi1evFhf//rXVVBQkD2+du1a/fCHP5TU/0tz1113ZQMzrh39en0+8IEP6NixY4rH4+rt7dWPfvQjrVmzxu2y8sbtt9+ud955R01NTcpkMjp8+DD9O0ktLS361Kc+pT179qiurk4S/ToVzp8/r507dyqVSimVSunll1/Wpk2b6Nfr9Mwzz+jw4cM6cOCAHn30Uf3u7/6u/v7v/55+nWaMFM8Av/rVr/Tyyy9ryZIl+r3f+z1J/SOZf/d3f6dPf/rTevzxx1VXV6eSkhLt2bPH5WrzA/16febNm6fHHntMmzdvVjqd1saNG3Xbbbe5XVbeCAaD2r17t7Zu3apkMqm1a9eqtrbW7bJmle985ztKJpPavXt39timTZvo1+u0du1anTx5Ug888IB8Pp/WrVunuro6VVZW0q9TjOeB6Wc4juO4XQQAAADgJqZPAAAAwPMIxQAAAPA8QjEAAAA8j1AMAAAAzyMUAwAAwPMIxQAAAPA8QjEAAAA8j1AMAAAAz/v/ARaCA7SZq4mPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 842.4x595.44 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.histplot(linearized_likes_3, color='b')\n",
    "sns.histplot(linearized_likes_0, color='r')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "591fa611",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=-15.21499546090383, pvalue=5.4914249479687664e-52)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.ttest_ind(linearized_likes_0,\n",
    "                linearized_likes_3,\n",
    "                equal_var=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2f2a7e3",
   "metadata": {},
   "source": [
    "After applying the linearized likes method, the p-value decreased, which indicates that no statistically significant difference was detected."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b64b787d",
   "metadata": {},
   "source": [
    "Now let's run a test between the other groups (1 and 2) on the metric of linearized likes. We used the same data in the last article, where no statistically significant difference was found when using the T-test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6acf0ee5",
   "metadata": {},
   "outputs": [],
   "source": [
    "q = \"\"\"\n",
    "SELECT exp_group, \n",
    "    user_id,\n",
    "    sum(action = 'like') as likes,\n",
    "    sum(action = 'view') as views,\n",
    "    likes/views as ctr\n",
    "FROM {db}.feed_actions \n",
    "WHERE toDate(time) between '2022-05-24' and '2022-05-30'\n",
    "    and exp_group in (1,2)\n",
    "GROUP BY exp_group, user_id\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7ddf68c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = ph.read_clickhouse(q, connection=connection)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "48a8a30d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAssAAAHxCAYAAACbAfGNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvSklEQVR4nO3de5xdVX03/s+ZyWRwksglzhAaKaJSEeTSgpUAv0T62IRbvAyhhVBDLxZTEBQfo5hEUqRyE42XFl+l9UGlomIEEygEa5F4CRWlFx4gICoEmkIY7iRhQiaznz/4ORpgnzPJzJwzl/f79eLFnL3WOfs7Z++Z88matdeuFEVRBAAAeImmRhcAAADDlbAMAAAlhGUAACghLAMAQAlhGQAASgjLAABQQlgGAIAS4xpdwFB78smN6e2t71LSkydPzOOPb6jrPhk+HH+cAzgHxjbHf2Rpaqpk110nlLaP+rDc21vUPSz/ar+MXY4/zgGcA2Ob4z96mIYBAAAlhGUAACgx6qdhAACQbN3akyef7EpPz/ONLqUhmpqa84pXTMzEiTunUqn0+3nCMgDAGPDkk13Zaae2TJgwZbvC4mhQFEW2bu3Js88+lSef7Mpuu3X0+7mmYQAAjAE9Pc9nwoRXjrmgnCSVSiXjxrVkl10m5/nnu7frucIyAMAYMRaD8m+qVJqSbN9KJcIyAACUEJYBAKCEsAwAACWshgEAMIb98Iffz5e//MX09GzJTjvtlDPO+EBWrLg2zz33XM4//6L88pe/yFlnzc/f/u3l+dd//U7WrfvvPPro+jz22GPZZ5/fyTnnLM6ECROr7uPKK7+Uf/7n5Wlra8tBB/1efvCDW7Js2XX5xCf+Os8883TWrVuXww8/MvPm/Xk+/emLc99996ZSqeSwww7PaaedkXHjxuXIIw/N9dd/N7vsskuS9D3+5S9/ni984fOZMmWPPPjgAxk/vjWLFv11XvOavQfl/TGyDAAwRj300IO5/PK/y6WXfjZXXHFVFixYlEWLFuTssz+cn//8Z7nxxuuzZMlHc9ZZH+wLn//5n/+ej3/8wlx11bI0Nzfniiv+seo+fvzjW3PjjdflH/7hK/niF/8pmzZt3Ka9u3tz/umfrs7pp5+Vz3zmk3nlK3fOV77yjfzjP16Zn//8vnzta/9U8/u49941mTPnj/PlL389xx03O+eff+6OvykvIiwDAIxRP/nJj/P444/l/e8/PX/6p3Pz8Y9/LJVKU/77vx/KeeddkIsv/pvst9+bMnPmMX3POeqo/5XddpucpqamHH/8O3LbbbdW3cett/4oRx31tkyaNCmVSiWdnX+0TfuBBx7U9/W//dvqnHDCH6VSqWT8+PF5xztOyI9/vLrm9/H61++Tgw763STJcce9I/fdd2+efvqp7XgnypmGAQAwRvX2bs0hh/x+Pv7xC/u2rV//SF71qvZ873vfzc4775z77rs3W7ZsSUtLS5Kkubm5r29R9Kapqfklr/ubmpubUxS/Xq6tqWnbsdpXvKJtm9f7zeXtiqI3PT09v/H4hdfZsmXLS/bx4j616uovI8sAAGPUIYf8fm677d+ydu0DSZJbb/1hTj315Kxde38++9lPZenSv8tv//Zr8oUvfK7vOT/4waps2LAhvb29WbHi2zniiP+v6j4OP/zIrFp1czZs2JAk+ed/Xl663vPv//60fOtbV6coijz//PNZseLavPnNb0mS7LLLrrnnnruTJP/yLyu3ed599/0sP//5fUmSFSuuyZvedGAmTZq0/W/IyzCyDAAwRu2992vz4Q8vypIlC1MURZqbm3PxxZ/OxRd/Iief/O689rWvzwc/+JGceupJOfTQF0LrbrtNzoIFZ+Wpp57KwQf/XubN+7Oq+zjkkDdn9ux3Zv78P0tr607Ze+/XprV1p5ft+4EPfChLl34y8+b9cbZs6clhh03LvHl/3tf26U9fkkmTJubQQ9+SyZNf1fe83XabnMsvvyyPPPI/2XXX3fKxj318kN6hpFL85rj4KPT44xvS21vfb7G9fVK6up6t6z4ZPhx/nAM4B8a24Xr8H3lkbaZM2WtAr/HFL/59nn76qXzwgx/p93Puuefu/N//e0dOPPGkJMnXv/5Pufvuu7aZ+jEQ//7vP83SpZfkyiuv7lf/F78PTU2VTJ5cvpqHkWUAAAbk3HM/mgcfXPuybYsW/XX+67/+IytWXJNKpZLdd5+SD394UZ0r3HFGlofAcP0XJfXh+OMcwDkwtg3X4z8YI8ujgZFl6qZtQmvy8vPzX1AkmzZurls9AACDTVhmx1WSq25cU9r8J8ftl7aJreXPF6YBgGFOWGbIFEX1MD33mDfWsRoAgO1nnWUAAChhZBkAgFK77rxTxo1vGfTX7Xl+S558unvQX3ewCcsAAJQaN74la/7xK4P+um98z7wk/QvLGzduyPz5f55LLvlM9tjjtwa9lmpMwwAAYNi66647c/rp78lDDz3YkP0LywAADFvXXXdtPvjBj+RVr2pvyP5NwwAAYNg655yPNXT/RpYBAKCEsAwAACWEZQAAKGHOMgAApXqe3/L/L/M2+K87EgjLAACUeuHGIY2/eciyZdc1ZL+mYQAAQAlhGQAASgjLAABQQlgGAIASwjIAAJQQlgEAoISl4wAAKLXzLm0Z39I86K/7/JatefqpTTX7/Z//c3luvvm7SZLDDz8ip5/+/kGvpRphGQCAUuNbmvMP1/zXoL/uX3YeVLPPT37y4/zkJ/+WK674aiqVSv73/z4zq1Z9LzNmHDXo9ZQRlgEAGJYmT35Vzjjj7LS0tCRJ9trrNVm//pG61mDOMgAAw9JrX/u6vOlNByRJHnrowdx883czbdoRda1BWAYAYFj75S9/kbPPPiNnnPH+7Lnnb9d138IyAADD1h13/Gc+8IHTM3/++3LMMcfXff/mLAMAMCytX/9IFi78UM4778IccsibG1KDsAwAwLD0ta/9UzZvfj6f//zSvm3vfGdn3vnOOXWrQVgGAKDU81u29muZtx153Vo+8IEP5QMf+NCg73t7CMsAAJTqz41DRjMX+AEAQIkhDcsbNmzI8ccfn//+7/9OkqxevTqzZ8/OzJkzs3Tpr+eerFmzJp2dnZk1a1YWLVqUnp6eJMn//M//5JRTTsnRRx+dv/qrv8rGjRuHslwAANjGkIXl//qv/8rJJ5+cBx54IEnS3d2dhQsX5rLLLssNN9yQO++8M6tWrUqSLFiwIOeee25uuummFEWRq6++Okly3nnnZe7cuVm5cmXe9KY35bLLLhuqcgEARr2iKBpdQkMVRW+SynY9Z8jC8tVXX50lS5ako6MjSXLHHXdkr732yp577plx48Zl9uzZWblyZdatW5fu7u4cfPDBSZLOzs6sXLkyW7ZsyU9+8pPMmjVrm+0AAGy/cePGZ+PGZ8ZkYC6KIj09W/LUU49l/Pidtuu5Q3aB3yc+8YltHj/66KNpb2/ve9zR0ZH169e/ZHt7e3vWr1+fJ598MhMnTsy4ceO22Q4AwPbbddf2PPlkVzZseKrRpTREU1NzXvGKiZk4ceftel7dVsPo7e1NpfLrYe+iKFKpVEq3/+r/v+nFj/tj8uSJO170ALS3T2rIfuupe3NPJkxoLW2vpFK1vbmpadS+T6P1+6L/nAM4B8a24Xr8p0zZtdEljDh1C8tTpkxJV1dX3+Ourq50dHS8ZPtjjz2Wjo6O7Lbbbnn22WezdevWNDc39/XfXo8/viG9vfX9c0N7+6R0dT1b1302QtvE1mzcuLm0vUhRtX1rb++ofJ/GyvGnnHMA58DY5viPLE1NlaqDq3VbOu6ggw7K/fffn7Vr12br1q25/vrrM3369EydOjWtra25/fbbkyTLly/P9OnT09LSkkMPPTQ33HBDkuTb3/52pk+fXq9yAQCgfiPLra2tueiii3LmmWdm8+bNmTFjRo4++ugkyaWXXprFixdnw4YN2X///TNv3rwkyZIlS3LOOefkC1/4QvbYY498+tOfrle5AAAw9GH55ptv7vt62rRpWbFixUv67Lvvvlm2bNlLtk+dOjVXXnnlkNYHAABl3MEPAABKCMsAAFBCWAYAgBLCMgAAlBCWAQCghLAMAAAlhGUAACghLAMAQAlhGQAASgjLAABQQlgGAIASwjIAAJQQlgEAoISwDAAAJYRlAAAoISwDAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUEJYBAKCEsAwAACWEZQAAKCEsAwBACWEZAABKCMsAAFBCWAYAgBLCMgAAlBCWAQCghLAMAAAlhGUAACghLAMAQAlhGQAASgjLAABQQlgGAIASwjIAAJQQlgEAoISwDAAAJYRlAAAoISwDAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUEJYBAKCEsAwAACWEZQAAKCEsAwBACWEZAABKCMsAAFBCWAYAgBLCMgAAlBCWAQCghLAMAAAlhGUAACghLAMAQAlhGQAASgjLAABQQlgGAIASwjIAAJQQlgEAoISwDAAAJYRlAAAoMa7RBTA8tU1oTSrV+1QqNToAAIxwwjIvr5JcdeOaql1OOXa/OhUDANAYpmEAAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUaEhYXr58eY477rgcd9xxufjii5Mkq1evzuzZszNz5swsXbq0r++aNWvS2dmZWbNmZdGiRenp6WlEyQAAjEF1D8vPPfdcPvGJT+TKK6/M8uXL89Of/jQ333xzFi5cmMsuuyw33HBD7rzzzqxatSpJsmDBgpx77rm56aabUhRFrr766nqXDADAGFX3sLx169b09vbmueeeS09PT3p6ejJx4sTstdde2XPPPTNu3LjMnj07K1euzLp169Ld3Z2DDz44SdLZ2ZmVK1fWu2QAAMaout+UZOLEiXn/+9+fY445Jq94xSvy5je/OY8++mja29v7+nR0dGT9+vUv2d7e3p7169dv1/4mT544aLVvj/b2SQ3Z72Dp3tyTCRNaq/appFK1T6325qamEf8+lRmt3xf95xzAOTC2Of6jR93D8j333JNvfetb+d73vpdJkyblQx/6UB544IFtbp1cFEUqlUp6e3tfdvv2ePzxDentLQat/v5ob5+Urq5n67rPwdY2sTUbN26u2qdIUbVPrfatvb0j/n16OaPh+DMwzgGcA2Ob4z+yNDVVqg6u1n0axg9/+MNMmzYtkydPzvjx49PZ2Zkf//jH6erq6uvT1dWVjo6OTJkyZZvtjz32WDo6OupdMgAAY1Tdw/K+++6b1atXZ9OmTSmKIjfffHMOOuig3H///Vm7dm22bt2a66+/PtOnT8/UqVPT2tqa22+/PckLq2hMnz693iUDADBG1X0axpFHHpm77747nZ2daWlpyQEHHJAzzzwzRxxxRM4888xs3rw5M2bMyNFHH50kufTSS7N48eJs2LAh+++/f+bNm1fvkgEAGKPqHpaT5LTTTstpp522zbZp06ZlxYoVL+m77777ZtmyZfUqDQAA+riDHwAAlBCWAQCghLAMAAAlhGUAACghLAMAQAlhGQAASgjLAABQoiHrLMNI8cq2camkqNqnSCXPbOqpU0UAQD0Jy1BFJUV+8bXqN8V53clz6lQNAFBvpmEAAEAJYRkAAEqYhsGoVmvOsfnGAEA1wjKjWq05x+YbAwDVmIYBAAAlhGUAACghLAMAQAlhGQAASrjADwaoqakpO7f9+vHW7u7s3Nbc99iKGwAwcgnLMFDFtituTJjQmo0bN/c9tuIGAIxcpmEAAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUEJYBAKCEsAwAACWEZQAAKCEsAwBACWEZAABKCMsAAFBCWAYAgBLCMgAAlBjX6AJgtGtqasrObeXtRSp5ZlNP/QoCAPpNWIahVhT5xdeWlTa/7uQ5dSwGANgepmEAAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUsBoGY1qtZd2aKpX6FQMADDvCMmNbjWXd9pl7Yh2LAQCGG9MwAACghLAMAAAlhGUAACghLAMAQAkX+DGivbJtXCopStutZgEADISwzIhWidUsAIChYxoGAACUEJYBAKCEsAwAACWEZQAAKCEsAwBACWEZAABKCMsAAFBCWAYAgBLCMgAAlHAHvzGqbUJrUuVO0BW3iQYAEJbHrEpy1Y1rSptPOXa/OhYDADA8mYYBAAAlhGUAACghLAMAQAlhGQAASgjLAABQQlgGAIASwjIAAJSwzjI0WFNTU3ZuK28vUskzm3rqVxAA0EdYhkYrivzia8tKm1938pw6FgMA/CbTMAAAoISwDAAAJYRlAAAo0ZCwfPPNN6ezszPHHHNM/uZv/iZJsnr16syePTszZ87M0qVL+/quWbMmnZ2dmTVrVhYtWpSeHhc6AQBQH/0KywsXLnzJtrPOOmuHdvjQQw9lyZIlueyyy7JixYrcfffdWbVqVRYuXJjLLrssN9xwQ+68886sWrUqSbJgwYKce+65uemmm1IURa6++uod2i8AAGyvqqthLFmyJOvXr8/tt9+eJ554om97T09PHnrooR3a4b/8y7/k2GOPzZQpU5IkS5cuzdq1a7PXXntlzz33TJLMnj07K1euzOtf//p0d3fn4IMPTpJ0dnbmc5/7XObOnbtD+wYAgO1RNSzPmTMn9913X+69997MmjWrb3tzc3NfgN1ea9euTUtLS+bPn5+HH344b33rW7PPPvukvb29r09HR0fWr1+fRx99dJvt7e3tWb9+/Xbtb/LkiTtU50C1t09qyH77q3tzTyZMaC1tr6RStb0/fWq1Nzc1Dfh92trdXb3OSoa2vaTPNo8HuI/m5qa0t1dZiJlhabj/DmDoOQfGNsd/9Kgalg844IAccMABOfzww/tGggdq69at+elPf5orr7wybW1t+au/+qvstNNOqVQqfX2KokilUklvb+/Lbt8ejz++Ib29xaDU3l/t7ZPS1fVsXfe5vdomtmbjxs2l7UWKqu396VOrfWtv74Dfp53bmqvXWWRo21+mz4QJL3pvB7iPrVt788QwP5/Y1kj4HcDQcg6MbY7/yNLUVKk6uNqvm5I8/PDDWbBgQZ5++ukUxa+D53XXXbfdBb3qVa/KtGnTsttuuyVJ3va2t2XlypVpbm7u69PV1ZWOjo5MmTIlXV1dfdsfe+yxdHR0bPc+AQBgR/QrLJ977rnp7OzMfvvtt90juy921FFH5SMf+UieeeaZTJgwIT/4wQ9y9NFH5/LLL8/atWvz6le/Otdff31OOOGETJ06Na2trbn99ttzyCGHZPny5Zk+ffqA9g8AAP3Vr7A8bty4/Nmf/dmg7PCggw7Ke97znsydOzdbtmzJEUcckZNPPjmvfe1rc+aZZ2bz5s2ZMWNGjj766CTJpZdemsWLF2fDhg3Zf//9M2/evEGpAwAAaulXWN5nn31y77335g1veMOg7HTOnDmZM2fONtumTZuWFStWvKTvvvvum2XLlg3KfgEAYHv0Kyw/9NBDOeGEE/Jbv/VbaW399VX7OzJnGQAARop+heWzzz57qOsAAIBhp19h+Xd+53eGug4AABh2+hWWDzvssFQqlW3WOW5vb8/3v//9IS0OAAAaqV9h+Z577un7+vnnn8/111+f+++/f8iKAgCA4aBpe58wfvz4dHZ25kc/+tFQ1AMAAMNGv0aWn3rqqb6vi6LInXfemWeeeWaoagIAgGFhu+csJ8nkyZOzaNGiIS0MAAAabbvnLAMAwFjRr7Dc29ubL37xi/n+97+fnp6eHHHEEZk/f37GjevX0wEAYETqV9r91Kc+lXvuuSennnpqent7841vfCOXXHJJFi5cONT1MYo1NVXSNrG1vEORbNq4uX4FAQC8SL/C8g9+8IN861vfSktLS5LkrW99a97+9rcLywxIUSRX3bimtH3uMW+sYzUAAC/Vr6XjiqLoC8rJC8vH/eZjAAAYjfoVlvfdd99ccMEFefDBB/PQQw/lggsucAtsAABGvX6F5SVLluSZZ57JSSedlBNPPDFPPvlkPvaxjw11bQAA0FBVw/Lzzz+fj3zkI7n11ltz0UUXZfXq1TnwwAPT3NyciRMn1qtGAABoiKph+XOf+1w2bNiQ3/u93+vbdv755+eZZ57J5z//+SEvDgAAGqlqWL7lllvyqU99KpMnT+7btvvuu+eSSy7Jd7/73SEvDgAAGqlqWG5paclOO+30ku0TJ07M+PHjh6woAAAYDqqG5aampmzYsOEl2zds2JCenp4hKwoAAIaDqmH5+OOPz+LFi7Np06a+bZs2bcrixYszc+bMIS8OAAAaqWpYPvXUUzNp0qQcccQR+aM/+qPMmTMnRxxxRF75ylfmjDPOqFeNAADQEFVvd93U1JTzzz8/8+fPz1133ZWmpqYceOCB6ejoqFd9AADQMFXD8q9MnTo1U6dOHepaAABgWOlXWAYap6mpKTu3lbcXqeSZTS64BYChICzDcFcU+cXXlpU2v+7kOXUsBgDGlqoX+AEAwFgmLAMAQAlhGQAASgjLAABQQlgGAIASVsOAEa7W0nKJ5eUAYEcJyzDS1VhaLrG8HADsKNMwAACghLAMAAAlhGUAACghLAMAQAlhGQAASgjLAABQQlgGAIASwjIAAJQQlgEAoISwDAAAJYRlAAAoISwDAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUEJYBAKCEsAwAACWEZQAAKDGu0QVAmfGVIi1tzVX7NFUqdaoGABiLhGWGr6I3v/jasqpd9pl7Yp2KAQDGItMwAACghLAMAAAlhGUAACghLAMAQAlhGQAASgjLAABQQlgGAIASwjIAAJRwUxJgWGib0JpUuyFjkWzauLlu9QBAIiwDw0UluerGNaXNc495Yx2LAYAXmIYBAAAlhGUAACghLAMAQAlhGQAASgjLAABQoqFh+eKLL84555yTJFm9enVmz56dmTNnZunSpX191qxZk87OzsyaNSuLFi1KT09Po8oFAGCMaVhYvvXWW3PttdcmSbq7u7Nw4cJcdtllueGGG3LnnXdm1apVSZIFCxbk3HPPzU033ZSiKHL11Vc3qmQAAMaYhoTlp556KkuXLs38+fOTJHfccUf22muv7Lnnnhk3blxmz56dlStXZt26denu7s7BBx+cJOns7MzKlSsbUTIAAGNQQ25Kcu655+bss8/Oww8/nCR59NFH097e3tfe0dGR9evXv2R7e3t71q9fX/d6gepq3n0vcQc+AEakuoflb37zm9ljjz0ybdq0XHPNNUmS3t7eVCq//qQtiiKVSqV0+/aYPHni4BS+ndrbJzVkv/3VvbknEya0lrZXUqna3p8+A29PzRpSqdFnqNtL+mzzeBjU2NzclPb2tqp9BqJ7c0+uueXnVft0vvX1VX8uap2TzU1Nw/7n6jeNpFoZGs6Bsc3xHz3qHpZvuOGGdHV15R3veEeefvrpbNq0KevWrUtzc3Nfn66urnR0dGTKlCnp6urq2/7YY4+lo6Nju/b3+OMb0ttbDFr9/dHePildXc/WdZ/bq21iazZWGeUrUlRt70+fgbenZg0pavQZ6vaX6TNhwove22FQY1Ekz2/cVOUlKnlm045fPFvrfEqSrb29VX8uar1GrecPJyPhdwBDyzkwtjn+I0tTU6Xq4Grdw/IVV1zR9/U111yT2267Leedd15mzpyZtWvX5tWvfnWuv/76nHDCCZk6dWpaW1tz++2355BDDsny5cszffr0epcMI19R5BdfW1ba/LqT59SxGAAYORoyZ/nFWltbc9FFF+XMM8/M5s2bM2PGjBx99NFJkksvvTSLFy/Ohg0bsv/++2fevHkNrhYAgLGioWG5s7MznZ2dSZJp06ZlxYoVL+mz7777Ztmy8hExAAAYKu7gBwAAJYRlAAAoISwDAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUEJYBAKDEsLiDH9BYTU1N2bmtvL1IJc9s6qlfQQAwTAjLNExLenPijL1L2yt1rGXMK4r84mvld8p83clz6lgMAAwfwjINUyl6c8tnryhtP/6jp9WxGoZaU1MlbRNbS9srFf88AmD4EZaBuiiK5Kob15S2n3LsfnWsBgD6xwV+AABQQlgGAIASwjIAAJQwZxmoqeLiPADGKGEZ6BcX5wEwFpmGAQAAJYRlAAAoISwDAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUEJYBAKCEsAwAACWEZQAAKOF216NU24TWpFLeXqlUaQQAIImwPHpVkqtuXFPafMqx+9WxGEa6SqWSE2fsXdrekt46VgMA9SMsA7UVRW757BWlzcd/9LQ6FgMA9WPOMgAAlBCWAQCghLAMAAAlhGUAACghLAMAQAlhGQAASgjLAABQQlgGAIASwjIAAJQQlgEAoISwDAAAJYRlAAAoISwDAEAJYRkAAEoIywAAUEJYBgCAEsIyAACUEJYBAKCEsAwAACWEZQAAKCEsAwBACWEZAABKCMsAAFBCWAYAgBLCMgAAlBjX6AKAka9SqeTEGXtX7dOS3jpVAwCDR1gGBqwoitzy2Suq9jn+o6cNaQ1tE1qTSo1ORbJp4+YhrQOA0UVYBkaHSnLVjWuqdpl7zBvrVAwAo4U5ywAAUMLIMjAstKS36rzn8ZUim4a4hppTOUzjABhzhGVgWKgUvVXnPR83xHOeXyii+lQO0zgAxh7TMAAAoISRZYZMrT+rAwAMd8IyQ6bWn9WHeikxAICBMg0DAABKCMsAAFBCWAYAgBLCMgAAlBCWAQCghNUwgDGjqamStomtpe2VSrXb9wEwFjUkLP/t3/5tbrzxxiTJjBkz8uEPfzirV6/OhRdemM2bN+eYY47J2WefnSRZs2ZNFi1alI0bN+bQQw/Neeedl3HjZHxg+xVF9Tv0nXLsfnWsBoCRoO7TMFavXp0f/vCHufbaa/Ptb387d911V66//vosXLgwl112WW644YbceeedWbVqVZJkwYIFOffcc3PTTTelKIpcffXV9S4ZAIAxqu5hub29Peecc07Gjx+flpaWvO51r8sDDzyQvfbaK3vuuWfGjRuX2bNnZ+XKlVm3bl26u7tz8MEHJ0k6OzuzcuXKepdMA7WMH1f1v/irOQAwhOo+n2Gfffbp+/qBBx7IjTfemD/5kz9Je3t73/aOjo6sX78+jz766Dbb29vbs379+rrWS2Pd+fOuqu2vqU8ZAMAY1bDJv/fdd1/e+9735sMf/nCam5vzwAMP9LUVRZFKpZLe3t5tLrj51fbtMXnyxMEqebu0t09qyH5/pXtzTyZMqHIhUyoDau9vn5aW6qfYQNuTpLW1pbyxkuo1DrS9pM82j4e6hh2scXtfYzCO1UD2UUn1n6ta5/wLrzGw8765qanfP9uN/h1A4zkHxjbHf/RoSFi+/fbbc9ZZZ2XhwoU57rjjctttt6Wr69cjiF1dXeno6MiUKVO22f7YY4+lo6Nju/b1+OMb0ttbDFrt/dHePildXc/WdZ8v1jaxNRs3bi5tL1IMqL2/fbZs6RnS9iS5475HS9t+O6leYzHA9pfpM2HCi977ge5jCGrckdcYjGM1kH0USdWfq1rn/AuvMbDzfmtvb79+tofD7wAayzkwtjn+I0tTU6Xq4Grdw/LDDz+cM844I0uXLs20adOSJAcddFDuv//+rF27Nq9+9atz/fXX54QTTsjUqVPT2tqa22+/PYccckiWL1+e6dOn17tkYBBUKpWcOGPvRpcxpNomtCaVF0a5S5eoK5JNtf5xA8CwUfew/MUvfjGbN2/ORRdd1LftpJNOykUXXZQzzzwzmzdvzowZM3L00UcnSS699NIsXrw4GzZsyP7775958+bVu2RgEBRFkVs+e0Vp+/EfPa2O1QyRygtL073krwu/Ye4xb6xzUQAMRN3D8uLFi7N48eKXbVuxYsVLtu27775ZtmzZUJcFY1vlhZVHAIBt+XQEklRfeeQ19SujVCVx9z0A6k5YBkYMd98DoN7qflMSAAAYKYRlAAAoISwDAEAJYRkAAEoIywAAUEJYBgCAEpaOA+inpqaKtZ4BxhhhGaCfisJazwBjjWkYAABQQlgGAIASpmGww1rSmxNn7N3oMgAAhoywzA6rFL255bNXlLYf/9HT6lgNAMDgMw0DAABKCMsAAFBCWAYAgBLCMgAAlHCBH4wFlaRlvB93ANhePj1hjLjz512lba+pXxljXq1bZqdINm3cXL+CAKhKWAaoo1q3zJ57zBvrWA0AtZizDAAAJYwsj1BtE1qTSnl7pVKlEQCAfhGWR6pK9T/lnnLsfnUsBgBgdDINAwAAShhZBkaFlvTmxBl71+wDANtDWGbUq7q+sKndo0al6M0tn72iap/jP3panaoBYLQQlhn1rC8MAOwoc5YBAKCEsAwAACWEZQAAKGHOMsAw0tRUSdvE1vIORbJp4+b6FQQwxgnLAMNIUVS/4dDcY95Yx2oAEJZ5Wf1ZsxYAYLQTlseoWmHYmrUjSKXGWtIAwA7zCTtG1QrDYykIj4abllRbSzqxnnR/1fpHpDsAAow9wjJjnpuW8Cv+EQnAiwnLMFAvmgbR09O77Wh1rWkS/WgnqVQqA55HPxivMdy1TWitfc5YUQOg34RlGAS/OTrd0jIuW7b09D1+TWqPXhvdrq0oigGP+g7Gawx7leqraSRW1ADYHsIywAhSax3mSsWfIgAGk7AMMILUWof5lGP3G/Iaak71MM0DGEWEZajBsmzwIjWmepjmAYwmUgDUMByWZRPYAaAxfALDCOACQABojKZGFwAAAMOVsAwAACVMwwAYY2otP2c1C4BfE5YBxphay89ZzQLg10zDAACAEkaWh0DP5s3ZZWKVt7bSlKeefb5+BQEAsEOE5aHQ25t/vvDy0ubjPnpaHYsBAGBHCcsA/VSpVHLijL1L21vSW8dqAKgHYRmgn4qiyC2fvaK0ffbC9+bEGXun0lRJ0Vu8bB+BGmBkEZYBBsmvwnRLy7hs2dLzsn2ONw0LYEQRlgHqyFQOgJFFWAaoo1pTOYw8AwwvwvIo1ZLeqqNXwMhU62d7MEama93hr1KpDHgfACOFsDwMtU1oTWp8FtX6sKoUvUavYBSqx892rTv8nXLsfgPeB8BIISwPR5XqH1SJDysAgHpwu2sAAChhZBlgGKm1WsZgPH+g85przZseXymyaUB7ABg+hGWAYWSgq2XUen5/XqOWWvOmjxuEedM1r90okk0bN+/48/PChYy9JTeP6c8+gLFBWAZg+Klx7cbcY944oOcnL1z7MaB9AGOCsAxA3dUa+a214o/l7YB6EZYBqL8aI7+1VvyxvB1QL8IyALyMWqPXgzFv2rxoGP6E5RHKHfoAhlat0es/OW6/0jDdvbknlaZKvnrD3VX3YV40DH/C8gjlDn3AcFVJqo/IZnTMKa4WpidMaM07Z7yu5msMdPS6FqPbMHDCMsAYU2st5oGuw5wM/C6ktf56Nhg1Dge1Rq8HPPLcj1VBjG5DdcLyMDS+UphiAQyZga7lXA/+egYMFyMiLF933XX5whe+kJ6enpx66qk55ZRTGl3S0KrxIZH4oACGzkDvIsjgGeppGsOBqSIMd8M+LK9fvz5Lly7NNddck/Hjx+ekk07KW97ylrz+9a9vdGkAo1I9Rp6H+iLl/rx+rakcw2EqyEAuMkwGZ274UN9NsVIZ+gshB/o9MLYN+7C8evXqHHbYYdlll12SJLNmzcrKlSvzvve9r1/Pb2pqwEUklUradplUpbmSCZOq/HJLb9Xn92cfQ96uhtI+48aNS0tPz7CuUQ1D2/6Sc6ABNVQ1xPuoNDXlj496bfUSUuS2Ly8rbf+D0+cOaB+1Xv9X+5jY1rLDr/G/zjilvIamSsZXemu+Dy2V3uo1VFK1fVx681z386XtbW3ja9YwvqlId5XPykpTsvyWn5e2v+uofap/plUq+fb37ittf+dR+1T9Hl94jeqf569oG18zkFer4R1vff2g54WG5I9hoNaxSJE8t6n8nG2EWseqUhRFUadadsjf//3fZ9OmTTn77LOTJN/85jdzxx135Pzzz29wZQAAjHZNjS6glt7e3m3+jFQUxahYcggAgOFv2IflKVOmpKurq+9xV1dXOjo6GlgRAABjxbAPy4cffnhuvfXWPPHEE3nuuefyne98J9OnT290WQAAjAHD/gK/3XffPWeffXbmzZuXLVu2ZM6cOTnwwAMbXRYAAGPAsL/ADwAAGmXYT8MAAIBGEZYBAKCEsAwAACWEZQAAKCEsD8B1112XY489NjNnzsxXv/rVl7SvWbMmnZ2dmTVrVhYtWpSestvfMiLVOv7f/e538453vCNvf/vbc/rpp+fpp59uQJUMpVrnwK/ccsst+YM/+IM6VkY91Dr+v/zlL/Pud787b3/72/MXf/EXfgeMQrXOgbvuuisnnHBC3v72t+e9731vnnnmmQZUyYAV7JBHHnmkOOqoo4onn3yy2LhxYzF79uzivvvu26bPcccdV/zHf/xHURRF8dGPfrT46le/2oBKGQq1jv+zzz5bHHHEEcUjjzxSFEVRfOYznynOP//8RpXLEOjP74CiKIqurq7i6KOPLo466qgGVMlQqXX8e3t7i5kzZxarVq0qiqIoPvnJTxaXXHJJo8plCPTnd8DJJ59c3HLLLUVRFMWFF15YfPrTn25EqQyQkeUdtHr16hx22GHZZZdd0tbWllmzZmXlypV97evWrUt3d3cOPvjgJElnZ+c27YxstY7/li1bsmTJkuy+++5Jkje84Q15+OGHG1UuQ6DWOfArixcvzvve974GVMhQqnX877rrrrS1tfXdRGv+/Pk55ZRTGlUuQ6A/vwN6e3uzcePGJMlzzz2XnXbaqRGlMkDC8g569NFH097e3ve4o6Mj69evL21vb2/fpp2Rrdbx33XXXfOHf/iHSZLu7u5cfvnledvb3lb3Ohk6tc6BJPnKV76S/fbbLwcddFC9y2OI1Tr+Dz74YF71qldl4cKFede73pUlS5akra2tEaUyRPrzO+Ccc87J4sWLc+SRR2b16tU56aST6l0mg0BY3kG9vb2pVCp9j4ui2OZxrXZGtv4e32effTannXZa9t1337zrXe+qZ4kMsVrnwM9+9rN85zvfyemnn96I8hhitY5/T09Pbrvttpx88sm59tprs+eee+aiiy5qRKkMkVrnQHd3dxYtWpQvfelL+eEPf5i5c+fmIx/5SCNKZYCE5R00ZcqUdHV19T3u6upKR0dHaftjjz22TTsjW63jn7ww6jB37ty84Q1vyCc+8Yl6l8gQq3UOrFy5Ml1dXTnhhBNy2mmn9Z0PjA61jn97e3v22muvHHDAAUmS448/PnfccUfd62To1DoHfvazn6W1tTUHHnhgkuSP//iPc9ttt9W9TgZOWN5Bhx9+eG699dY88cQTee655/Kd73ynb25akkydOjWtra25/fbbkyTLly/fpp2Rrdbx37p1a+bPn59jjjkmixYt8leFUajWOXDWWWflpptuyvLly3P55Zeno6MjV111VQMrZjDVOv6/+7u/myeeeCL33HNPkuTmm2/O/vvv36hyGQK1zoG99torjzzySH75y18mSf71X/+17x9PjCzjGl3ASLX77rvn7LPPzrx587Jly5bMmTMnBx54YP7yL/8yZ511Vg444IBceumlWbx4cTZs2JD9998/8+bNa3TZDJJax/+RRx7J3Xffna1bt+amm25KkrzpTW8ywjyK9Od3AKNXf47/3/3d32Xx4sV57rnnMmXKlFxyySWNLptB1J9z4MILL8wHPvCBFEWRyZMn54ILLmh02eyASlEURaOLAACA4cg0DAAAKCEsAwBACWEZAABKCMsAAFBCWAYAgBLCMsAY8ud//ud54oknGl0GwIghLAOMIT/60Y8aXQLAiOKmJACj1LJly3LFFVekqakpu+66a6ZOnZokOfXUU3P55ZfnlFNOyYEHHph77703H/zgB/OHf/iHDa4YYPgRlgFGoXvuuSeXXnpprr322uyxxx750pe+1Hfb3S9/+cvZbbfdkiT77LNPPvOZzzSwUoDhTVgGGIVuvfXWHHnkkdljjz2SJH/6p3+aJPnGN76xTb9DDz203qUBjCjCMsAo1NzcnEql0ve4u7s769ate0m/tra2epYFMOK4wA9gFHrLW96SW2+9NY8++miS5Otf/3o++clPprm5OT09PQ2uDmDkMLIMMAq94Q1vyIIFC/Ke97wnSdLe3p4LLrggF198cd797nfn85//fIMrBBgZKkVRFI0uAgAAhiPTMAAAoISwDAAAJYRlAAAoISwDAEAJYRkAAEoIywAAUEJYBgCAEsIyAACU+H9tdpRMszOrbwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 842.4x595.44 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set(rc={'figure.figsize':(11.7,8.27)})\n",
    "\n",
    "groups = sns.histplot(data = df, \n",
    "              x='ctr', \n",
    "              hue='exp_group', \n",
    "              palette = ['r', 'b'],\n",
    "              alpha=0.5,\n",
    "              kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "caed8f07",
   "metadata": {},
   "outputs": [],
   "source": [
    "CTRcontrol_1 = (df[df.exp_group == 1]['likes'].sum())/(df[df.exp_group == 1]['views'].sum())\n",
    "CTRcontrol_2 = (df[df.exp_group == 2]['likes'].sum())/(df[df.exp_group == 2]['views'].sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8e55cb59",
   "metadata": {},
   "outputs": [],
   "source": [
    "linearized_likes_1 = df[df.exp_group == 1]['likes'] - (CTRcontrol_0*(df[df.exp_group == 1]['views']))\n",
    "linearized_likes_2 = df[df.exp_group == 2]['likes'] - (CTRcontrol_0*(df[df.exp_group == 2]['views']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "ef8147bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='Count'>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAskAAAHjCAYAAADR6oIgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAyrklEQVR4nO3df3DV9Z3v8df3/MiPA/GCcI5xKcOMVYGyVaydrehOUnfcEIwp3ZR2UWfxttdd8bqRhntpGUhxXN2WOpQs6tBxdl1763ZnLlItyGCo1Yra2PGa2SmXSjPeLSAyGBKCJiEk3+853+/9A84h+XJySML5nu/58XzMOJ58PuebvMMn55tXvufz/XwMx3EcAQAAAEgJ+F0AAAAAkG8IyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIWQDAAAALgQkgEAAACXkN8FZMvp02dk2/4u+Txr1nSdOjXoaw3IDca6tDDepYOxLi2Md+mYNWu6Tp8+o5kzp034mKIJybbt+B6Sk3WgNDDWpYXxLh2MdWlhvEvHZMea6RYAAACACyEZAAAAcCEkAwAAAC6EZAAAAMCFkAwAAAC4EJIBAAAAF0IyAAAA4EJIBgAAAFwIyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIWQDAAAALgQkgEAAAAXQjIAAADgQkgGAAAAXAjJAAAAgAshGQAAAHAJ+V0AAFxKJJCQLDN9Z7hMQ3YwtwUBAIoeIRlA/rNMdW3dlrZr/to1UrAyxwUBAIod0y0AAAAAF0IyAAAA4EJIBgAAAFwIyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIWQDAAAALgQkgEAAAAXQjIAAADg4mlIfv3119XU1KRly5bp8ccflyR1dHSosbFRdXV1amtrSz330KFDampq0tKlS7Vx40bF43EvSwMAAADG5VlIPnbsmB555BFt375du3fv1vvvv6/9+/drw4YN2r59u/bu3auDBw9q//79kqR169Zp06ZN2rdvnxzH0Y4dO7wqDQAAAMjIs5D86quv6s4771R1dbXC4bDa2tpUWVmpefPmae7cuQqFQmpsbFR7e7uOHz+u4eFhLV68WJLU1NSk9vZ2r0oDAAAAMgp59YmPHj2qcDis1atX68SJE/ryl7+s6667TtFoNPWcWCym7u5unTx5ckx7NBpVd3f3pL7erFnTs1b75YhGq/wuATnCWOfOSO+IwuFg2r5QKKDobO/HgvEuHYx1aWG8S8dks6JnITmRSOi9997T888/r0gkogcffFAVFRUyDCP1HMdxZBiGbNtO2z4Zp04NyradrNU/FdFolXp6BnytAbnBWOdWJGHLshJp++JxW/0ejwXjXToY69LCeJeOaLRKp04NTiooexaSZ8+erSVLlujKK6+UJN1xxx1qb29XMHjhalBPT49isZiqq6vV09OTau/t7VUsFvOqNAAAACAjz+Yk33777Xr77bfV39+vRCKht956S/X19Tp8+LCOHj2qRCKhPXv2qKamRnPmzFF5ebk6OzslSbt27VJNTY1XpQEAAAAZeXYl+cYbb9T999+ve+65R5Zl6bbbbtPdd9+ta665Rs3NzRoZGVFtba3q6+slSVu2bFFra6sGBwe1aNEirVq1yqvSAAAAgIw8C8mStGLFCq1YsWJM25IlS7R79+6LnrtgwQLt3LnTy3IAAACACWHHPQAAAMCFkAwAAAC4EJIBAAAAF0IyAAAA4EJIBgAAAFwIyQAAAIALIRkAAABw8XSdZAAoVKEQ1xAAoJQRkgHAJRQK6NXOj/Rx7xlVz56mb9wx3++SAAA5RkgGgDQ+7j2jY90DfpcBAPAJ7ycCAAAALoRkAAAAwIWQDAAAALgQkgEAAAAXQjIAAADgQkgGAAAAXFgCDkBBC4aDilhn03eGyzRkB3NbEACgKBCSARQ0x7TU1fZU2r75a9dIwcocVwQAKAZMtwAAAABcCMkAkEEgYEg6t1V1KMQpEwBKBdMtACCD6IxK/fzXH+h494CqZ0/TX978GcXjtt9lAQA8RkgGgEvoPjWkY90DfpcBAMgh3jsEAAAAXAjJAAAAgAshGQBGCYUCCgY5NQJAqWNOMgCcFwoF9GrnR7ryigoZhuF3OQAAHxGSAWCUj3vPyLEdv8sAAPiM9xQBAAAAF0IyAAAA4EJIBgAAAFwIyQAAAIALN+4BKFrBcFAR62z6znCZhuxgbgsCABQMQjKAouWYlrrankrbN3/tGilYmeOKAACFgukWAAAAgAshGQAAAHAhJAMAAAAuhGQAAADAhZAMAAAAuBCSAQAAABdCMgAAAOBCSAYAAABcCMkAAACACyEZAAAAcCEkAwAAAC6EZAAAAMCFkAwAAAC4hPwuAAD8Fgqdu14QDHLdAABwDiEZQEkLhQJ6tfMjfdx7Rp+7ZpYMw/C7JABAHuCyCYCS93HvGR3rHtCpT876XQoAIE8QkgEAAAAXQjIAAADgQkgGAAAAXLhxD0BeiAQSkmWm7QvIyXE1AIBSR0gGkB8sU11bt6XtWtjSnONiAACljukWAAAAgAshGQAAAHAhJAMAAAAuhGQAAADAhZAMAAAAuBCSAQAAABdPl4D7m7/5G/X19SkUOvdl/uEf/kFnzpzRD37wA42MjGjZsmVqaWmRJB06dEgbN27UmTNn9MUvflGPPvpo6jgAAAAglzxLoY7j6MiRI/r1r3+dCrvDw8Oqr6/X888/r6uvvloPPPCA9u/fr9raWq1bt06PP/64Fi9erA0bNmjHjh265557vCoPAAAAGJdn0y3++Mc/SpK+9a1v6Stf+Yr+7d/+TQcOHNC8efM0d+5chUIhNTY2qr29XcePH9fw8LAWL14sSWpqalJ7e7tXpQEAAAAZeXYlub+/X0uWLNH3vvc9WZalVatW6f7771c0Gk09JxaLqbu7WydPnhzTHo1G1d3dPamvN2vW9KzVfjmi0Sq/S0COMNbZNdI7onA4mLbPMJT1vlAooOjsqvOPgwqHQwqGggoGA6n/J9skKRwOKRQKaubMaZP91lBgeG2XFsa7dEw2K3oWkm+66SbddNNNqY9XrFihJ598UjfffHOqzXEcGYYh27ZlGMZF7ZNx6tSgbNu5/MIvQzRapZ6eAV9rQG4w1tkXSdiyrETaPsdR1vvicVv9PQMKhQKKxxOyrLgS8YQSCTv1/2SbJFlWXPF4QqdPn1E8bk/hO0Qh4LVdWhjv0hGNVunUqcFJBWXPplu89957euedd1IfO46jOXPmqKenJ9XW09OjWCym6urqMe29vb2KxWJelQYAAABk5FlIHhgY0BNPPKGRkRENDg7qpZde0tq1a3X48GEdPXpUiURCe/bsUU1NjebMmaPy8nJ1dnZKknbt2qWamhqvSgMAAAAy8my6xe23367f/e53+upXvyrbtnXPPffopptu0ubNm9Xc3KyRkRHV1taqvr5ekrRlyxa1trZqcHBQixYt0qpVq7wqDQAAAMjI04WIv/3tb+vb3/72mLYlS5Zo9+7dFz13wYIF2rlzp5flAAAAABPCbh0AciYSSEiWmbYvIH9vvAUAYDRCMoDcsUx1bd2WtmthS3OOiwEAYHye3bgHAAAAFCpCMgAAAOBCSAYAAABcCMkAAACACyEZAAAAcCEkAwAAAC6EZAAAAMCFkAwAAAC4EJIBAAAAF0IyAAAA4EJIBgAAAFwIyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIWQDAAAALgQkgEAAACXkN8FAIAfDEmhUEDB4MSvFQQCxpjnx+O2B5UBAPIBIRlAyTEMQwNnLf3vN7r0uWtmyTCMCR0XnVGpfe8e04neQVXPnqa/vPkzBGUAKFKEZAAlKR63dax7QFddGZnUcR+fOqNj3QMeVQUAyBfMSQYAAABcCMkAAACACyEZAAAAcCEkAwAAAC7cuAcAPosEEpJlpu8Ml2nIDua2IAAAIRkAfGeZ6tq6LW3X/LVrpGBljgsCADDdAgAAAHAhJAMAAAAuhGQAAADAhZAMAAAAuBCSAQAAABdWtwCALGEpNwAoHoRkAJiCQMBQMHjhzbh43GYpNwAoIoRkAJiC6IxK7Xv3mE70Dqp69jT95c2fkRJ+VwUAyBZCMgBM0cenzuhY94DfZQAAPMCNewAAAIALIRkAAABwISQDAAAALoRkAAAAwIUb9wAgjwXDQUWss+k7WXsZADxDSAaAHMgUdgNyxj3OMS11tT2Vto+1lwHAO4RkAMiBTGF3YUtzjqsBAFwKc5IBAAAAF0IyAAAA4MJ0CwAl6YorKvTf6+YpUhFWPDFNpjUz9TgRDGvXu+PcLAcAKAmEZAAlyYjH9bvNW1U1rUyWZWvYjKce37Thf/hdHgDAZ0y3AAAAAFwIyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIWQDAAAALiwBByAkmEYxvn/+1wIACDvEZIBlATDMHR6YFimZStSyakPAJAZvykAZFUkkJAsM21fQE6OqxnLPL9pSDjMTDMAQGaEZADZZZnq2rotbdfCluYcFwMAwNRwOQUAAABw8Twk//CHP9T69eslSR0dHWpsbFRdXZ3a2tpSzzl06JCampq0dOlSbdy4UfF43OuyAAAAgHF5GpLfeecdvfTSS5Kk4eFhbdiwQdu3b9fevXt18OBB7d+/X5K0bt06bdq0Sfv27ZPjONqxY4eXZQEAAAAZeRaSP/nkE7W1tWn16tWSpAMHDmjevHmaO3euQqGQGhsb1d7eruPHj2t4eFiLFy+WJDU1Nam9vd2rsgAAAIBL8iwkb9q0SS0tLbriiiskSSdPnlQ0Gk31x2IxdXd3X9QejUbV3d3tVVkAAADAJXmyusULL7ygq6++WkuWLNGLL74oSbJtO7WQvyQ5jiPDMMZtn6xZs6ZffuFZEI1W+V0CcoSxTm+kd0ThcDBtn2HI1z7DkAKGkTrHJB+PbjcMSYYUDAYUDocUDJ07Pvl4dHvycSgU1MyZ0zTSezan318oFFB0Nj+H2cZru7Qw3qVjslnRk5C8d+9e9fT0aPny5fr00081NDSk48ePKxi8cKLv6elRLBZTdXW1enp6Uu29vb2KxWKT/pqnTg3Ktv1dgzUarVJPz4CvNSA3GOvxRRK2LCuRts9x5FufYRhyHMl2HDnOuXNF8vHodseR5EiJhC3LiisRP3d88vHo9uTjeDyh06fPqCye2+89HrfVz89hVvHaLi2Md+mIRqt06tTgpIKyJyH5ueeeSz1+8cUX9e677+rRRx9VXV2djh49qs985jPas2ePvva1r2nOnDkqLy9XZ2enbr75Zu3atUs1NTVelAUAAABMSM42EykvL9fmzZvV3NyskZER1dbWqr6+XpK0ZcsWtba2anBwUIsWLdKqVatyVRYAFKxgOKiIdXb8J4TLNGSnn6oBAMjM85Dc1NSkpqYmSdKSJUu0e/fui56zYMEC7dy50+tSAKCoOKalrranxu2fv3aNFKzMYUUAUDzYcQ8AAABwISQDAAAALoRkAAAAwIWQDAAAALgQkgEAAAAXQjIAAADgQkgGAAAAXAjJAHCZAgFDwWBAhs5tfw0AKHyEZAC4TNEZldr37jGdHhjR6YFhgjIAFAFCMgBkwcenzsiMJ2Ratt+lAACygJAMAAAAuBCSAQAAABdCMgAAAOBCSAYAAABcQn4XAKDwRAIJyTLT9gXk5LgaAACyj5AMYPIsU11bt6XtWtjSnONiAADIPqZbAAAAAC6EZAAAAMCFkAwAAAC4EJIBAAAAF27cA4BsMiTDOP9AkuOw2gcAFCJCMgBkUTgUUF//iEwrobJwQDOrKgjKAFCACMkAio5x7lKub0wroWEzLk6xAFC4mJMMoKgYhqHTA8Pq7huSbTu+B2YAQGHiMgeAomNatobNuBxdmB9MVgYATAYhGUDRMqTU/OBIJac7AMDEMd0CQFFLzg+24rbfpQAACgghGQAAAHAhJAMAAAAuhGQAAADAhTtZAGASln3hKgXjMxWpCCuemCbTuvB4ekVIf/S7QABAVhCSAWASKo2E/s/mraqaVibr/FJzyce3PfI//S4PAJAlE5pusWHDhovaHn744awXAwAAAOSDjFeSH3nkEXV3d6uzs1N9fX2p9ng8rmPHjnleHAAAAOCHjCF5xYoV+uCDD9TV1aWlS5em2oPBoBYvXux1bQAAAIAvMobkz3/+8/r85z+vW2+9VdXV1bmqCQAAAPDVhG7cO3HihNatW6dPP/1UjuOk2l9++WXPCgMAAAD8MqGQvGnTJjU1Nelzn/ucDMPwuiYAAADAVxMKyaFQSN/85je9rgUAioshnbuuwMUFACg0E1oC7rrrrlNXV5fXtQBAUQmHAurrH9HpgWG/SwEATNKEriQfO3ZMX/va1/Qnf/InKi8vT7UzJxkAMjOthN8lAACmYEIhuaWlxes6AAAAgLwxoZB8/fXXe10HAAAAkDcmFJJvueUWGYYhx3FSq1tEo1G9+eabnhYHAAAA+GFCIfkPf/hD6rFpmtqzZ48OHz7sWVEAAACAnya0usVoZWVlampq0m9+8xsv6gEAAAB8N6EryZ988knqseM4OnjwoPr7+72qCQAAAPDVpOckS9KsWbO0ceNGTwsDAAAA/DLpOckAAABAsZtQSLZtW88++6zefPNNxeNx3XbbbVq9erVCoQkdDgAAABSUCd2496Mf/Ui//e1vdd999+mb3/ym/uM//kNPPPGE17UBAAAAvpjQpeC33npLP//5zxUOhyVJX/7yl/WVr3xFGzZs8LQ4AAAAwA8TupLsOE4qIEvnloEb/TEAAABQTCYUkhcsWKDvf//7+vDDD3Xs2DF9//vfZ6tqAAAAFK0JheRHHnlE/f39Wrlypb7+9a/r9OnT+t73vud1bQAAAIAvMoZk0zT13e9+V++88442b96sjo4O3XDDDQoGg5o+fXquagQAAAByKmNIfvLJJzU4OKgvfOELqbbHHntM/f39euqppzwvDgAAAPBDxpD8xhtv6Ec/+pFmzZqVarvqqqv0xBNP6Fe/+pXnxQEAAAB+yBiSw+GwKioqLmqfPn26ysrKPCsKAAAA8FPGkBwIBDQ4OHhR++DgoOLxuGdFAQAAAH7KGJLvuusutba2amhoKNU2NDSk1tZW1dXVXfKTb9u2TXfeeacaGhr03HPPSZI6OjrU2Niouro6tbW1pZ576NAhNTU1aenSpdq4cSMhHAAAAL7JGJLvu+8+VVVV6bbbbtM3vvENrVixQrfddpuuuOIKPfTQQxk/8bvvvqvf/va32r17t37+85/r+eef1x/+8Adt2LBB27dv1969e3Xw4EHt379fkrRu3Tpt2rRJ+/btk+M42rFjR/a+SwAAAGASMm5LHQgE9Nhjj2n16tX6/e9/r0AgoBtuuEGxWOySn/jP/uzP9NOf/lShUEjd3d1KJBLq7+/XvHnzNHfuXElSY2Oj2tvbde2112p4eFiLFy+WJDU1NenJJ5/UPffcc/nfIQAAADBJGUNy0pw5czRnzpxJf/JwOKwnn3xS//qv/6r6+nqdPHlS0Wg01R+LxdTd3X1RezQaVXd396S/HgAAAJANEwrJl+Phhx/W3/7t32r16tU6cuSIDMNI9TmOI8MwZNt22vbJmDUrPzY3iUar/C4BOVLKYz3SO6JwOJi2zzDke59hSAHDkIwLjw3DGPNYSt9uGJIMKRgMKBwOKRg697lHPx73OClDe27/XSQpFAooOrt0f06nqpRf26WI8S4dk82KnoXk//zP/5Rpmlq4cKEqKytVV1en9vZ2BYMXTug9PT2KxWKqrq5WT09Pqr23t3dCUzpGO3VqULbtZK3+qYhGq9TTM+BrDciNUh/rSMKWZSXS9jmOfO0zDEOOI9mOIzlKPXYcZ8xjKX2740hypETClmXFlYif+9yjH497nJShPbf/LpIUj9vqL+Gf06ko9dd2qWG8S0c0WqVTpwYnFZQz3rh3OT766CO1trbKNE2ZpqnXXntNK1eu1OHDh3X06FElEgnt2bNHNTU1mjNnjsrLy9XZ2SlJ2rVrl2pqarwqDQAAAMjIsyvJtbW1OnDggL761a8qGAyqrq5ODQ0NuvLKK9Xc3KyRkRHV1taqvr5ekrRlyxa1trZqcHBQixYt0qpVq7wqDQAAAMjI0znJzc3Nam5uHtO2ZMkS7d69+6LnLliwQDt37vSyHAAAAGBCPL9xD0DhiQQSkmWO2x+Qv/P/AQDwGiEZwMUsU11bt43bvbCledw+vyRXrJjkwjgAAKRFSAZQ8AzD0OmBYZmWrUglpzUAwOXzbHULAMgl07I1bMZlxW2/SwEAFAEuuQBAkQqGg4pYZ9N3hss0ZI+/EQkAlDpCMgAUKce01NX2VNq++WvXSMHKHFcEAIWD6RYAAACACyEZAAAAcCEkAwAAAC6EZAAAAMCFkAyg4LGBCAAg2wjJAAqWYRiybUf9Q+NvoQ0AwFQQkgEUNEdiAxEAQNYRkgEAAAAXQjIAAADgQkgGAAAAXAjJAAAAgAshGQC8Zpz7zzAMGaxXBwAFgZAMAB4LhwKyE466+4Z0emCYoAwABSDkdwEAUAocScNmXJx2AaAwcCUZAAAAcCEkAwAAAC687wcAJSgYDipinU3fGS7TkB3MbUEAkGcIyQBQghzTUlfbU2n75q9dIwUrc1wRAOQXplsAAAAALlxJBkpUJJCQLDNtX0BOjqsBACC/EJKBUmWZ6tq6LW3XwpbmHBcDAEB+YboFAAAA4EJIBgAAAFwIyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIWQDAAAALgQkgEAAAAXQjIAAADgQkgGAAAAXAjJAAAAgEvI7wIAoKQYkmGcfyDJcRxfywEApEdIBoAcCocC6usfkWklVBYOaGZVhd8lAQDSICQDQI6ZVkLDZlycggEgfzEnGQAAAHAhJAMAAAAuvNcHoOAY5+58O38DHAAA2UdIBlBQDMPQ6YFhmZatSCWnMACAN5huAaDgmJatYTMuK277XQoAoEgRkgEAAAAXQjIAAADgQkgGAAAAXAjJAAAAgAshGQAAAHAhJAMAAAAuhGQAAADAhZX4AQBjBMNBRayz6TvDZRqyg7ktCAB8QEgGAJfp08rVdPNsmdZMRSrCiiemyVz0XxSpCKuyrPjfgHNMS11tT6Xtm792jRSszHFFAJB7hGQAcLMs/d8ftmnYjKtqWpnicVtnR849/tLG/+F3dQCAHCj+SyIAAADAJBGSAQAAABdCMgAAAOBCSAYAAABcPA3JTz/9tBoaGtTQ0KAnnnhCktTR0aHGxkbV1dWpra0t9dxDhw6pqalJS5cu1caNGxWPx70sDQAAABiXZyG5o6NDb7/9tl566SX94he/0O9//3vt2bNHGzZs0Pbt27V3714dPHhQ+/fvlyStW7dOmzZt0r59++Q4jnbs2OFVaQAAAEBGnoXkaDSq9evXq6ysTOFwWJ/97Gd15MgRzZs3T3PnzlUoFFJjY6Pa29t1/PhxDQ8Pa/HixZKkpqYmtbe3e1UaAAAAkJFnIfm6665Lhd4jR47olVdekWEYikajqefEYjF1d3fr5MmTY9qj0ai6u7u9Kg0AAADIyPPNRD744AM98MAD+s53vqNgMKgjR46k+hzHkWEYsm1bhmFc1D4Zs2ZNz1bJlyUarfK7BORIoY/1SO+IwuH02wsbhsbtu1R/LvoMQwoYRuo8YRjGmLbkYxlK2548Ll27Yeii40Y/N+Nx49Qy/nFSKBTw5N8sm//eo4VCAUVnF/bP/qUU+msbk8N4l47JZkVPQ3JnZ6cefvhhbdiwQQ0NDXr33XfV09OT6u/p6VEsFlN1dfWY9t7eXsVisUl9rVOnBmXbTtZqn4potEo9PQO+1oDcKIaxjiRsWVYibZ/jaNy+S/V73WcYhhxHsh1HjuOc73fGtCUfy1Ha9uRx6dodRxcdN/q5GY8bp5bxj5PicduTf7Ns/Xu7xeO2+gv8Zz+TYnhtY+IY79IRjVbp1KnBSQVlz6ZbnDhxQg899JC2bNmihoYGSdKNN96ow4cP6+jRo0okEtqzZ49qamo0Z84clZeXq7OzU5K0a9cu1dTUeFUaAAAAkJFnV5KfffZZjYyMaPPmzam2lStXavPmzWpubtbIyIhqa2tVX18vSdqyZYtaW1s1ODioRYsWadWqVV6VBgAAAGTkWUhubW1Va2tr2r7du3df1LZgwQLt3LnTq3IAAACACfP8xj0A/okEEpJlpu0LyN85/AAA5DNCMlDMLFNdW7el7VrY0pzjYi7PhZUsfC4kmwyNWRUjeYMfAMB/hGQABeH0wLBMy1aksnhOW+FQQH39I0rYjk4PDGtmVQVBGQDyRPH8tgFQ1EzL1rAZVzjs2aI8vjCthBzHkWnZfpcyIcFwUBHrbPrOcJmG7PHXbQaAQkJIBgBMmGNa6mp7Km3f/LVrpGBljisCAG8U1yUZAAAAIAsIyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIXVLQDkLSO104a/dQAASg8hGUBeMgwjtYHI9TYbbAAAcovpFgDyVnIDEZGRAQA5xpVkoMBFAgnJMtP2BUiXAABMCSEZKHSWqa6t29J2LWxpznExAAAUB6ZbAAAAAC5cSQaAfGBI5xbzOLeUh+MwVQYA/ERIBoA8EA4F1Nc/ItNKqCwc0MyqCoIyAPiIkAwAecK0EudW8+DUDAC+Y04yAAAA4EJIBgAAAFwIyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIV1hgAg37g2FgEA5B5XkgEgzyQ3FunuG5JtOzIMwjIA5BohGQDyUHJjEfbcAwB/EJIBAAAAF0IyAAAA4EJIBgAAAFwIyQAAAIALS8ABQJ4bvRyc43ArHwDkAiEZAPKYIamvf0SmlVBZOKCZVRUEZQDIAUIyAN8l1wE2DIMAmEZyOThO2QCQO5xxgQIQCSQky0zbFyjwlXQNw9DpgWElbEenB4a5UgoAyAuEZKAQWKa6tm5L27WwpTnHxWSfadlyHEdm3E7Nv2WTOQCAnwjJAPJGcjtm00ooUsnpCQDgH5aAA5BXkvNvrbjtdykAgBLGpRoAQFYEw0FFrLPpO8NlGrKDuS0IAC4DIRkAkBWOaamr7am0ffPXrpGClTmuCACmjukWAAAAgAshGQAAAHAhJAMAAAAuhGQAAADAhZAMAAAAuBCSAQAAABdCMgAAAODCOskAfGEYxvn/+1wIAABpEJIB5JxhGDo9MCzTshWp5DRUCtiND0Ch4bcTAF+Ylq1hM65wmFlfpYDd+AAUGn47AQAAAC6EZAAAAMCFkAwAAAC4EJIBoFAY51YDMVgSBAA8x417AFAgwqGA+vpHJDmaWVXhdzkAUNQIyQBQQEwr4XcJAFASmG4BAAAAuHAlGcgTkUBCssy0fQE5Oa4GAIDSRkgG8oVlqmvrtrRdC1uac1wMAAClzdPpFoODg7rrrrv00UcfSZI6OjrU2Niouro6tbW1pZ536NAhNTU1aenSpdq4caPi8biXZQEAAAAZeRaSf/e73+nuu+/WkSNHJEnDw8PasGGDtm/frr179+rgwYPav3+/JGndunXatGmT9u3bJ8dxtGPHDq/KAgAAAC7Js5C8Y8cOPfLII4rFYpKkAwcOaN68eZo7d65CoZAaGxvV3t6u48ePa3h4WIsXL5YkNTU1qb293auyAAAAgEvybE7yP/7jP475+OTJk4pGo6mPY7GYuru7L2qPRqPq7u6e9NebNWv61IvNomi0yu8SkCPZHuuR3hGFw8G0fYahvOnL1uc1DClgGDIMQ8m9MZKPR7cHDCO1iYa7PbmpxlSPS9duGLrouNHPzXjcOLVM9bjxvgdJCoUCORmnXPSFQgFFZ+fPuZPzeGlhvEvHZLNizm7cs217zC5RjuPIMIxx2yfr1KlB2ba/KwBEo1Xq6RnwtQbkhhdjHUnYssZZA9dxlDd9l/t543FbhnHuse04chxHjpPsdy5qtx1Hci5+fvLx5RyXrt1xdNFxo5+b8bhxapnqceN9D9K5f0cvxymXffG4rf48OXdyHi8tjHfpiEardOrU4KSCcs5CcnV1tXp6elIf9/T0KBaLXdTe29ubmqIBoPicHhhWKMQS7QCA/Jaz31Q33nijDh8+rKNHjyqRSGjPnj2qqanRnDlzVF5ers7OTknSrl27VFNTk6uyAOSYadmy4rbfZRS281MwJE3pnbd8EwwHFUmcTf9fgB0GAfgjZ1eSy8vLtXnzZjU3N2tkZES1tbWqr6+XJG3ZskWtra0aHBzUokWLtGrVqlyVBQAFJxwKqK9/RAnb0emBYc2sqkhN6ShEjmmpq+2ptH3z166RgpU5rggAchCSX3/99dTjJUuWaPfu3Rc9Z8GCBdq5c6fXpQBA0TCthBzHkWlxVR4AvMDEQAAAAMCFkAwAAAC4EJIBAAAAF0IyAAAA4EJIBgAAAFwIyQAAAIALIRkAAABwISQDAAAALoRkAAAAwIWQDAAAALh4vi01AABTFQwHFbHOpu8rCythWukPDJdpyA56WBmAYkdIBoBCZkiGcf6BJMdxfC0n2xzTUlfbU2n7FrY0j9s3f+0aKVjpZWkAihwhGcihSCAhWWbavoCKK9yMZpxLcckchywKhwLq6x+RaSVUFg7oyisqVGQ5GQB8QUgGcsky1bV1W9quhS3NOS4mNwzD0OmBYZmWrett0psXTCuhYTOucLhsTGAGAEwdIRmA50zL1rAZVxFfLM8bycDM6R0ALg+XGgAAAAAXQjIAAADgQkgGAAAAXAjJAAAAgAshGQAAAHAhJAMAAAAuhGQAAADAhYU0AXgiucuewS57AIACREgGkHWjd9mLVHKa8YVx7r/kHysOe1UDwKTw2wvIskggIVlm2r5AkW85N/rqcXKXvTDbI/siHArITjjq7htSWTigmVUVBGUAmARCMpBtlqmurdvSdi1sac5xMbmTvHqcsB31D6X/IwG55UhsUQ0AU8QlHgBZY1q2HMeRFbf9LgUAgMvC5QUAl4Ub9AAAxYiQDGDKuEEPAFCs+K0G4LJwgx7yUTAcVMQ6m74zXKYhO5jbggAUHEIyABQ7IzkdpnSWg3NMS11tT6Xtm792jRSszHFFAAoNIRkAilw4FFBf/4hMK5FaDg4AkBkhGQBKgGklWA7uvExTMawB7kAFcA5nSwDAuIxRy5YUyzSNTFMxFn2nRVJ5bgsCkJcIycAUWAODiiTSX4kq9l31UDpGr17CNA0ApYaQDEyBPTJccrvqGSyEXJSS4zreVeLk6iX8ugBQajjrAbgk9xVFFDDjwsYvgYChvv5hSdLMqoqimU4BANlASAbGEQkkJMtM2+eU4EVVrigWh+RKFwnbUf+QKdNKv4W4YRiluYtiIKCIyfrKAPhtB4zPMsedUvG5tcU5pQKlwbQSchxHVvx8QB61jrJhSLbtqLtvqCR3UXRMU11bWV8ZACEZAEre6HWUI5UhOdLFuyiOmqZhGAZTMwAUPSYXApgcQ+cDU4m+HV+kkusop64uu4yepnF6YFiBgHH+Z4AfAgDFiSvJACYlHArITpTu2/GlLDlNw5Eu2sGPK8sAig2/4QBIuniJt0xLg6V9Ox4lhR38xsp0oy83/AGFibMbSlqmX2yltCnI6CXeIpWh1I1bZeGArryCDSSAS8pwoy83/AGFiZCM0pbhF1uxbgoynuQSb+FwYNSV4jL19Y8oFGLeKcYxamWMMc1FuJ01gNJCSAaQkWkl5HCPL8YxemWM620nFY5Hbz6TfDeiVFfFCIaDilisvQwUGkIyih5TKgBvJecnO7qwTNyFdybKxqyKUYo3+Tmmpa421l4GCg0hGcWPKRVAThhS2uk5yVUxxtvdr1BkuiLMH9xA8SEkAwCyppin52S6Iswf3EDxISQDAFCAWHYO8BYhGQCQG66VMEptbnLWsewc4ClCMgAgJ0avhMFOfQDyHSEZAJAz4+3UV6rrKmdcHk5i2gTgI0IyUALGCyDJdoO9QuCz0esql9IV5kw3A0pMmwD8REhGUSjFtZCNCSbb0VtOJzd2cBxJhsZsRQ34KbmuMr+WAOQLzkYoGBmDcMLRoa1Ppu0rxqWZRgff621HgYAh94W30Vfi3Bs7JHdHG70VNZBTo2/i8+idjOQfkqVyVRpAdhGSUTjYFGSMC1felAq+kcqQ4nFHkqOZVRXjHHd+Tii5AT5yb2edTYZhyLYddfcNXTR948IUo8LYItuLDUxYOg6YGEIykIcmexNTMviGwwFZBb6rGUpH2j/YXMvETZUjnfvcRij1+QxD6usfLqgtsj3ZwISl44AJISQj6zJdpQiWhZUwrXGPzdRfrHOL3dLNIU62T/gXunHhZjxuykMhcS8Tl83PF6kMybTsotgi2yuZrlxnOj9nPLdzdRoFipCM7LvEtIhMd3Jn6i+lKRXuOcTJK1/Jm+4uFXyTwSBhO+ofGudtVSBPpa4wG6Hzf/CNnVs8+p2WibzrMvqdllIx1Wkal7pyPZW+z313rSIJpneg8ORVSH755Zf14x//WPF4XPfdd5/uvfdev0sqesxN89ZEfoFfapUK00rIcRw5Gjv3+FKSx1lxrpihMIVDAdmJC3OLk++q9PWfv2nVcXR64MJV54n+ETlR6V6b7rZ0SypezhSObK0X7ck0DQ9qYXoH8lnehOTu7m61tbXpxRdfVFlZmVauXKkvfelLuvbaa/0ureBNdVWITH/9l8q0iMv5peeeNpHuRrrkc859cO7jTL/gS/GKGEpbcm5x8l2VUMi4cNOqM/o1UTapPyJHz312v76Tr8NkGE/eEJuwHQ2cNRWPOxcF8+SSipIutGlifxyPXq88+TXHO2dMVr5v0jLV6R2ZLuLk28WffKsHE5c3Ibmjo0O33HKLZsyYIUlaunSp2tvb9fd///cTOj4QyI+Jl/lSx2iGHdcfn/tfafuu/W//VWUzZ6Q/MGHrcIbjMvWN9zmNYHD8r3e+v/zKmb73GYbkBIIaLosoFApoWkVozBJrE/mchiFVhEcUjNsKhwIqryq/6LjUc0IBOca5r1ceDuqKUESV8YQqKkJKxB0FQiFNj81SIu6Maa+MJ2SEgrriqtlj2oMhY0rHJR8HgkHNvDp60ecbr3+ytbo/3+XWmo3vcfRzLjou4ajCmvr3HgwZl/1vNtnxnUyt+T6+7p/FUKZ/m3hC5RP4Hmb+SVTW+QCZfH0bhmSPeh1WhCKpzxeKO6nXb+j816moCMkyDMXjtmQEVXHllQqGjFRb8tyR7nV/Zjh+7rhAUPHIdMXj9pivOd45YzLntopZM1NfZ/R5LF/Os5KkhK0jP/lp2q7Pfuu+MX2jQ/51q+/XNHsk7XEB29H/G+f3k/u4kT5L03T+3bZQmc7a6S9AVAZsKZ5h6lqGYzP9Dr7m/m8qEMiPKJbxe8zw/RWSyWY0w8mTPy2feeYZDQ0NqaWlRZL0wgsv6MCBA3rsscd8rgwAAAClJm/+LLBt+6K3hSa6oxgAAACQTXkTkqurq9XT05P6uKenR7FYzMeKAAAAUKryJiTfeuuteuedd9TX16ezZ8/ql7/8pWpqavwuCwAAACUoP2aLS7rqqqvU0tKiVatWybIsrVixQjfccIPfZQEAAKAE5c2NewAAAEC+yJvpFgAAAEC+ICQDAAAALoRkAAAAwIWQDAAAALgQkrPk/fff15/+6Z+mPu7v79ff/d3fadmyZbr33nvHrAGNwtXZ2akVK1Zo+fLluu+++3T8+HFJjHexevnll3XnnXeqrq5OP/vZz/wuBx54+umn1dDQoIaGBj3xxBOSpI6ODjU2Nqqurk5tbW0+V4hs++EPf6j169dLYqyL2euvv66mpiYtW7ZMjz/+uKQpjLeDyzY0NOSsXLnSuf7661Ntjz76qPPMM884juM4L730krNmzRqfqkM23X777c6hQ4ccx3GcF154wVm9erXjOIx3Mfr444+d22+/3Tl9+rRz5swZp7Gx0fnggw/8LgtZ9Jvf/Mb567/+a2dkZMQxTdNZtWqV8/LLLzu1tbXOhx9+6FiW5XzrW99y3njjDb9LRZZ0dHQ4X/rSl5zvfve7ztmzZxnrIvXhhx86f/7nf+6cOHHCMU3Tufvuu5033nhj0uPNleQs2Lx5s+67774xbW+88YYaGxslSXfddZfefPNNWZblR3nIEtM0tWbNGi1YsECSNH/+fJ04cUIS412MOjo6dMstt2jGjBmKRCJaunSp2tvb/S4LWRSNRrV+/XqVlZUpHA7rs5/9rI4cOaJ58+Zp7ty5CoVCamxsZNyLxCeffKK2tjatXr1aknTgwAHGuki9+uqruvPOO1VdXa1wOKy2tjZVVlZOerwJyZfptdde0/DwsOrr68e0nzx5UtFoVJIUCoU0ffp09fX1+VEisqSsrEzLly+XJNm2raefflp33HGHJMa7GI0eU0mKxWLq7u72sSJk23XXXafFixdLko4cOaJXXnlFhmEw7kVq06ZNamlp0RVXXCGJ13gxO3r0qBKJhFavXq3ly5fr3//936c03nmz416+e+WVV/SDH/xgTNs111yjwcFB/eQnP7nk8Y7jKBDgb5JCMd54/+QnP5Fpmlq/fr3i8bgeeOCBtMcz3oXPtm0ZhpH62HGcMR+jeHzwwQd64IEH9J3vfEfBYFBHjhxJ9THuxeGFF17Q1VdfrSVLlujFF1+UxGu8mCUSCb333nt6/vnnFYlE9OCDD6qiomLS401InqBly5Zp2bJlY9peeOEFPfPMM7r33ntTbcuXL9fPfvYzxWIx9fb2qrq6WvF4XGfOnNGMGTNyXDWmKt14S9KZM2f04IMPasaMGfrxj3+scDgsSYx3EaqurtZ7772X+rinp0exWMzHiuCFzs5OPfzww9qwYYMaGhr07rvvjrnxlnEvDnv37lVPT4+WL1+uTz/9VENDQzp+/LiCwWDqOYx18Zg9e7aWLFmiK6+8UpJ0xx13qL29fdLjzaWuy/D1r39dv/rVr7Rr1y7t2rVLkrRr1y5Nnz5dtbW1+sUvfiHp3Ivzi1/8YipQoXCtW7dO8+bN0z/90z+prKws1c54F59bb71V77zzjvr6+nT27Fn98pe/VE1Njd9lIYtOnDihhx56SFu2bFFDQ4Mk6cYbb9Thw4dTb9fu2bOHcS8Czz33nPbs2aNdu3bp4Ycf1l/8xV/oX/7lXxjrInX77bfr7bffVn9/vxKJhN566y3V19dPery5kuyRNWvWaP369WpoaFBVVZW2bNnid0m4TO+//75ee+01XXvttfqrv/orSeeuIP/zP/8z412ErrrqKrW0tGjVqlWyLEsrVqzQDTfc4HdZyKJnn31WIyMj2rx5c6pt5cqV2rx5s5qbmzUyMqLa2tqL7jlBcSgvL2esi9SNN96o+++/X/fcc48sy9Jtt92mu+++W9dcc82kxttwHMfJUc0AAABAQWC6BQAAAOBCSAYAAABcCMkAAACACyEZAAAAcCEkAwAAAC6EZAAAAMCFkAwAAAC4EJIBAAAAl/8P0HU8A7xHRMYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 842.4x595.44 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.histplot(linearized_likes_1, color='b')\n",
    "sns.histplot(linearized_likes_2, color='r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c4fe6fb9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=6.1208039704412, pvalue=9.544973454280379e-10)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.ttest_ind(linearized_likes_1,\n",
    "                linearized_likes_2,\n",
    "                equal_var=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe581e73",
   "metadata": {},
   "source": [
    "As we can see this time, the T-test showed a statistically significant difference and the p-value dropped significantly.\n",
    "\n",
    "This is the end of the series of articles about AB testing and then you will find interesting articles about the automation of reporting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e920f4f9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}