{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "major-expansion",
   "metadata": {},
   "source": [
    "# _TriScale_ - Data Analysis\n",
    "\n",
    "> This notebook is intended for **live tutorial** sessions about _TriScale._  \n",
    "Here is the [self-study version](tutorial_data-analysis.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abandoned-springfield",
   "metadata": {},
   "source": [
    "To get started, we need to import a few Python modules. All the _TriScale_-specific functions are part of one module called `triscale`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "primary-secret",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import triscale"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "weighted-encounter",
   "metadata": {},
   "source": [
    "## _TriScale_ analysis API\n",
    "\n",
    "_TriScale_'s API contains three functions for data analysis\n",
    "- `analysis_metric()`\n",
    "- `analysis_kpi()`\n",
    "- `analysis_variability()`\n",
    "\n",
    "These functions have a similar structure. They take as input a data sample and the definition of the metric, KPI, or variability score to compute, respectively. Each function performs the corresponding analysis and returns its results, together with some optional data visualizations. \n",
    "\n",
    "Here are minimal examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "express-lotus",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some random data\n",
    "Y = np.random.sample(100)\n",
    "X = np.arange(len(Y))\n",
    "data = np.array([X,Y])\n",
    "df = pd.DataFrame(np.transpose(data), columns=['x','y'])\n",
    "\n",
    "# Minimal definition of a TriScale metric\n",
    "metric = {  \n",
    "    'measure': 50,   # Integer: interpreted as a percentile\n",
    "         }\n",
    "\n",
    "# Basic call of analysis_metric\n",
    "triscale.analysis_metric( \n",
    "    df,\n",
    "    metric,\n",
    "    plot=True,\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "received-scale",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Minimal definition of a TriScale KPI\n",
    "KPI = {\n",
    "    'percentile': 75,\n",
    "    'confidence': 95,\n",
    "    'bounds': [0,1]\n",
    "}\n",
    "\n",
    "# Basic call of analysis_metric\n",
    "triscale.analysis_kpi( \n",
    "    df.y.values,\n",
    "    KPI,\n",
    "    to_plot=['series','autocorr','horizontal']\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0746dd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Minimal definition of a TriScale KPI\n",
    "score = {\n",
    "    'percentile': 75, # the 25th-75th  percentiles range\n",
    "    'confidence': 95,\n",
    "    'bounds': [0,1]\n",
    "}\n",
    "\n",
    "# Basic call of analysis_metric\n",
    "triscale.analysis_variability( \n",
    "    df.y.values,\n",
    "    score,\n",
    "    to_plot=['horizontal']\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ce00180",
   "metadata": {},
   "source": [
    "For more details about any function, refer to the docstring, as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0015bfe9",
   "metadata": {},
   "outputs": [],
   "source": [
    "help(triscale.analysis_kpi)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71046f6c",
   "metadata": {},
   "source": [
    "> **Note.** The functions return the KPI/score values as well as **the result of the corresponding independence test.** One must not forget to get it and take extra care in the rest of the analysis shall the test returned `False`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11f1ee3f",
   "metadata": {},
   "source": [
    "## Analysis of Pantheon data (congestion control)\n",
    "\n",
    "We have collected data for a comparative evaluation of congestion-control schemes using the \n",
    "[Pantheon platform](https://pantheon.stanford.edu/). We use some of these data to illutrate the main analysis functions of _TriScale_.\n",
    "\n",
    "> For a more extensive description of the data collection and analysis, you can check the [complete case study notebook](casestudy_congestion-control.ipynb) or the [_TriScale_ paper](https://doi.org/10.5281/zenodo.3464273) itself.\n",
    "\n",
    "In a nutshell, the dataset contains\n",
    "- two metrics: the mean **throughput** and 95th percentile of the **one-way delay**\n",
    "- measured across **five series** of **ten runs** each\n",
    "- collected for **17 congestion-control schemes**\n",
    "- using an emulated environment.\n",
    "\n",
    "Let us first load and visualise the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87fb0112",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load and display the entire dataset\n",
    "df = pd.read_csv(Path('ExampleData/metrics_wo_convergence.csv'))\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "745f4ee4",
   "metadata": {},
   "source": [
    "We can easily extract the lists of `schemes` used and `dates` identifying each series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9ae6c95",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the list of congestion control schemes\n",
    "schemes = df.cc.unique()\n",
    "print(schemes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "582d65c9",
   "metadata": {},
   "source": [
    "Let's create a short `get_metrics` function to easily extract all metrics values for one scheme and one metric  \n",
    "(e.g., the `throughput` of `bbr`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22bf8815",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_metrics(df, scheme, metric):\n",
    "    '''Parse the dataset to extract the series of metric values f\n",
    "    or one scheme and all series of runs.\n",
    "    '''\n",
    "    # Initialize output\n",
    "    metric_data = []\n",
    "    \n",
    "    # List of dates (identifies the series)\n",
    "    dates = df.datetime.unique()\n",
    "    \n",
    "    # For each series\n",
    "    for date in dates:\n",
    "        \n",
    "        # Setup the data filter\n",
    "        filter = (\n",
    "            (df.cc == scheme) &\n",
    "            (df.datetime == date) \n",
    "        )\n",
    "\n",
    "        # Filter\n",
    "        df_filtered = df.where(filter).dropna()\n",
    "        \n",
    "        # Store metrics values for that series\n",
    "        metric_data.append(list(df_filtered[(metric+'_value')].values))        \n",
    "    \n",
    "    # Return the desired metric data\n",
    "    return metric_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78ddc147",
   "metadata": {},
   "source": [
    "The definition of KPI and variability score we use are provided below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c18649f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# KPIs\n",
    "KPI_tput  = {'percentile': 25,\n",
    "             'confidence': 75,\n",
    "             'name':       'KPI Throughput',\n",
    "             'unit':       'Mbit/s',\n",
    "             'bounds':     [0,120],          # expected value range\n",
    "             'tag':        'throughput'      # do not change the tag\n",
    "            }\n",
    "KPI_delay = {'percentile': 75,\n",
    "             'confidence': 75,\n",
    "             'name':       'KPI One-way delay',\n",
    "             'unit':       'ms',\n",
    "             'bounds':     [0,100],          # expected value range\n",
    "             'tag':        'delay'           # do not change the tag\n",
    "            }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fe5c0c6",
   "metadata": {},
   "source": [
    "\n",
    "> **Note.**  We aim to estimate the 25th percentile for the `throughput`, where higher is better.  It is the opposite for the delay. Thus, both KPIs aim to estimate the performance expected in at least 75% of the runs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07dbb640",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variability scores\n",
    "score_tput  = {'percentile': 50,\n",
    "               'confidence': 75,\n",
    "               'name':       'Throughput',\n",
    "               'unit':       'Mbit/s',\n",
    "               'bounds':     [0,120],        # expected value range\n",
    "               'tag':        'throughput'    # do not change the tag\n",
    "              }\n",
    "score_delay = {'percentile': 50,\n",
    "               'confidence': 75,\n",
    "               'name':       'One-way delay',\n",
    "               'unit':       'ms',\n",
    "               'bounds':     [0,100],        # expected value range\n",
    "               'tag':        'delay'         # do not change the tag\n",
    "              }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "836668d4",
   "metadata": {},
   "source": [
    "As an example, let us analyze the `throughput` of the `TCP BBR` scheme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79b32580",
   "metadata": {},
   "outputs": [],
   "source": [
    "#####################################\n",
    "# Extract the metrics values for the 5 series of 10 runs \n",
    "#####################################\n",
    "\n",
    "scheme = 'bbr'         # valid options: print(df.cc.unique())\n",
    "metric = 'throughput'  # valid options are 'throughput' and 'delay'\n",
    "metric_data = get_metrics(df, scheme, metric)\n",
    "\n",
    "# Initialize an empty list to collect the KPI values\n",
    "KPI_values = [] \n",
    "\n",
    "if metric == 'throughput':\n",
    "    KPI = KPI_tput\n",
    "    score = score_tput\n",
    "if metric == 'delay':\n",
    "    KPI = KPI_delay\n",
    "    score = score_delay\n",
    "\n",
    "#####################################\n",
    "## Step 1. Compute the KPIs\n",
    "#####################################\n",
    "\n",
    "for series_data in metric_data:\n",
    "    \n",
    "    indep_test_passed, KPI_value = triscale.analysis_kpi(\n",
    "        series_data,\n",
    "        KPI)\n",
    "    if indep_test_passed:\n",
    "        KPI_values.append(KPI_value)\n",
    "    \n",
    "# Print the (valid) KPI values\n",
    "s = '%i valid KPIs obtained\\n> ' % len(KPI_values)\n",
    "for k in KPI_values:\n",
    "    s += '%0.2f ' % k\n",
    "s += '\\n  in %s\\n' % KPI['unit']\n",
    "print(s)\n",
    "    \n",
    "#####################################    \n",
    "## Step 2. Compute the variability score\n",
    "#####################################\n",
    "\n",
    "(indep_test_passed, \n",
    " upper_bound, \n",
    " lower_bound, \n",
    " var_score, \n",
    " rel_score) = triscale.analysis_variability(\n",
    "    KPI_values,\n",
    "    score\n",
    ")\n",
    "\n",
    "if not indep_test_passed: \n",
    "    var_score *= -1\n",
    "\n",
    "print('Variability score: %0.2f %s' % (var_score, score['unit']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "876ebc25",
   "metadata": {},
   "source": [
    "## Your turn: time to practice"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41ca7832",
   "metadata": {},
   "source": [
    "First of all, can we change the KPI definition? Recall that we have 5 series of 10 runs in our dataset.\n",
    "- How many runs per series do we need to estimate the 90th percentile at 75% confidence?\n",
    "- Or the 75th percentile at 95% confidence?\n",
    "- With 10 runs per series, can we do much better than 75th percentile with 75% confidence level?\n",
    "- What is the trade-off when using \"better\" KPIs?\n",
    "\n",
    "> **Hint.** Remember the `triscale.experiment_sizing` function? :-) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8092d0b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "########## YOUR CODE HERE ###########\n",
    "# ...\n",
    "#####################################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfcc2a96",
   "metadata": {},
   "source": [
    "Let's now explore the dataset a little further!\n",
    "\n",
    "- What is the variability score of the `delay` metric of the `bbr` congestion-control scheme?\n",
    "\n",
    "Modify the definition of the variability score to estimate the median `'percentile': 50` instead of  \n",
    "the 25-75th percentile range. \n",
    "\n",
    "- What are the values of the variability scores now? Does this make sense to you?\n",
    "\n",
    "_Optional (and harder) questions:_ \n",
    "\n",
    "- Compute the scores for all the schemes. Do they vary a lot? \n",
    "- Do the variability scores seem \"big\" with respect to the range of KPI values? \n",
    "- Would you say that these experiments appear to be replicable?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7d08e0c",
   "metadata": {},
   "source": [
    "#### Solutions\n",
    "\n",
    "<details>\n",
    "<summary><br/>Click here show the solution: <strong>Changing the KPI definition</strong></summary>\n",
    "\n",
    "```python\n",
    "triscale.experiment_sizing(90,75,verbose=True);\n",
    "triscale.experiment_sizing(75,95,verbose=True);\n",
    "\n",
    "# \"Best\" options\n",
    "triscale.experiment_sizing(87,75,verbose=True);\n",
    "triscale.experiment_sizing(75,94,verbose=True);\n",
    "    \n",
    "```\n",
    "With 10 runs, \n",
    "- if we set the confidence level to 75%, the largest (resp. smallest) percentile one can estimate is the 87th (resp. 13th) percentile;\n",
    "- if we set the percentile to 75th, the best confidence level we can get is 94%.\n",
    "    \n",
    "The trade-off is using these \"best\" KPIs is that there is no margin for poor runs: the KPI estimate will always be the largest (resp. smallest) collected value. Moreover, if one run should fail, or not converge, then one would not have enough runs left to compute the desired KPI!\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8be21b1c",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary><br/>Click here show the solution: <strong>BBR delay</strong></summary>\n",
    "    \n",
    "Simply change the metric definition in the code bloc above from `throughput` to `delay`.\n",
    "It leads to the following output:\n",
    "    \n",
    "```\n",
    "5 valid KPIs obtained\n",
    "> 87.08 86.51 86.31 87.17 85.74 \n",
    "  in ms\n",
    "\n",
    "Variability score: 1.43 ms\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96eaf35f",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary><br/>Click here show the solution: <strong>BBR with median score</strong></summary>\n",
    "    \n",
    "Change the percentile in score definitions from 75 to 50, and re-run the analysis. \n",
    "The output is the same: the scores are not affected by the change in score definitions.\n",
    "    \n",
    "One would expect that a two-sided estimate of the median would be narrower than the estimate of the 25-75th percentile internal. While this is generally true, having only 5 series of runs is not enough to make a difference. This can be seen with the `robustness` parameter from `triscale.experiment_sizing`:\n",
    "    \n",
    "```python\n",
    ">>> triscale.experiment_sizing(50,75,robustness=1)\n",
    "(5,6)\n",
    "```\n",
    "Hence, for a two-sided confidence interval for the median, one needs at least 6 samples in order to \"exclude\" one.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6a78e6b",
   "metadata": {},
   "source": [
    "---\n",
    "Next step: [Seasonal Components](live_seasonal-comp.ipynb)  \n",
    "[Back to repo](.)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.1 64-bit ('triscale': conda)",
   "language": "python",
   "name": "python391jvsc74a57bd0684f90775fb1f43db0d8eed0780ba829f42a97c2f4b9a1bd592c18e47e5c272e"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}