{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# TriScale - Data Analysis\n", "\n", "> This notebook is intended for **self-study** of _TriScale._ \n", "Here is the [version for live sessions](live_data-analysis.ipynb).\n", "\n", "This notebook contains tutorial materials for _TriScale_. \n", "\n", "More specifically, this notebook presents _TriScale_'s data analysis functions, \n", "leading to the computation of variability scores, which serve to quantify replicability.\n", "\n", "> If you don't know about Jupyter Notebooks and how to interact with them, \n", "fear not! We compiled everything that you need to know here: [Notebook Basics](tutorial_notebook-basics.ipynb) :-) \n", "\n", "\n", "For more details about _TriScale,_ you may refer to [the paper](https://doi.org/10.5281/zenodo.3464273).\n", "\n", "---\n", "- [Data analysis](#Data-analysis)\n", " - [Runs and Metrics](#Runs-and-Metrics)\n", " - [Series and KPIs](#Series-and-KPIs)\n", " - [Sequels and Variability Scores](#Sequels-and-Variability-Scores)\n", "- [Your turn: time to practice](#Your-turn:-time-to-practice)\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get started, we need to import a few Python modules. \n", "All the _TriScale_-specific functions are part of one module called `triscale`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "import numpy as np\n", "\n", "import triscale" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alright, we are ready to analyse some data!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data analysis\n", "\n", "In this notebook, we consider that experiments have been designed, that the \n", "corresponding data has been collected, and we focus on the data analysis.\n", "\n", "_TriScale_'s methodology is structured around three time scales:\n", "- **Runs** which lead to the computation of performance **metrics**;\n", "- **Series** which lead to the computation of **key performance indicators (KPIs)**;\n", "- **Sequels** which lead to the computation of **variability scores**.\n", "\n", "_TriScale_'s API provides one function per time scale, which we will look at in the next sections.\n", "\n", "### Runs and Metrics\n", "\n", "In _TriScale_, metrics measure performance dimension during one run. The computation of metrics \n", "is implmented in the `analysis_metric()` function, which takes two compulsory arguments:\n", "- the raw data;\n", "- the metric definition.\n", "\n", "The raw data can be passed as a file path (i.e., a string) or as a Pandas DataFrame. \n", "- If a string is passed, the function tries to read the file as a csv file (comma separated) \n", "where `x` data is expected in the first column and `y` data in the second column. \n", "- If a pandas DataFrame is passed, `data` must contain columns named `x` and `y`.\n", "\n", "The metric definition is provided as a dictionary, with only the `measure` key being compulsory. \n", "This key defines \"what is the computation to be performed\" on the data; in other word, what the \"metric\" is. \n", "Currently supported measures are:\n", "- Any percentile ($0 **Note.** As presented here, the `analysis_metric()` function is not very interesting: \n", "it \"only\" returns some percentile of an array... The function is more useful when the metric \n", "attempts to estimate the _long-term performance_ of the system; that is, the value one \n", "would obtain shall the run last longer/more data points be collected. \n", "When this is the case, _TriScale_ performs a convergence test on the data, which can be \n", "triggered in `analysis_metric()` function by passing the optional `convergence` parameter. \n", "The study of convergence goes beyond the scope of this tutorial; refer to the [paper](https://doi.org/10.5281/zenodo.3464273) for more details. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Series and KPIs\n", "\n", "In _TriScale_, key performance indicators (KPIs) measure performance dimensions across a series of runs. \n", "Performing multiple runs allows to mitigate the inherent variability of the experimental conditions. \n", "KPIs capture this variability by estimating percentiles of the (unknown) metric distributions. \n", "\n", "Concretely, **a _TriScale_ KPI is a one-sided confidence interval of a percentile** \n", "e.g., a lower bound for the 25th percentile of a metric, estimated with a 95% confidence level. \n", "The computation of KPIs is implmented in the `analysis_kpi()` function, which takes \n", "two compulsory arguments:\n", "- the metric data;\n", "- the KPI definition.\n", "\n", "The metric data can be passed as a list or an NumPy array. The KPI definition is provided \n", "as a dictionary with three compulsory keys: \n", "- `percentile` ($0 **Why the independence test?** The metric data must be iid for the KPI to be a valid \n", "estimate of the underlying metric distribution. Note however that, in general, \n", "independence is a property of the data collection process, not of the data. \n", "Unfortunately though, in many practical cases in networking, independence cannot be \n", "guaranteed; for example, because there is some correlations in the interference \n", "conditions between sucessive experiments. \n", "In such a case, one can perform an _empirical_ test for independence; essentially, \n", "this test assesses whether the level of correlation in the data appears sufficiently \n", "low such that the data can be reasonably assumed to be iid.\n", "\n", "The following cell illustrates the basic usage of `analysis_kpi()` function." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "# Input data file path\n", "data = Path('ExampleData/metric_data.csv') # Failure recovery time, in seconds\n", "\n", "# Read data in a Pandas DataFrame\n", "df = pd.read_csv(data, header=0, names=['metric'])\n", "\n", "# KPI definition\n", "KPI = {\n", " 'percentile': 75,\n", " 'confidence': 95,\n", " 'bounds': [0,10],\n", " 'unit': 's'\n", "}\n", "\n", "# Computes the KPI\n", "indep_test_passed, KPI_value = triscale.analysis_kpi(\n", " df.metric.values,\n", " KPI,\n", ")\n", "\n", "# Output\n", "if indep_test_passed:\n", " print('The metric data appears iid.')\n", " print('KPI value: %0.2f %s' % (KPI_value, KPI['unit']))\n", "else:\n", " print('The metric data does not appear iid.')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since the metric data appears to be iid, we can interpret the KPI value as follows: \n", "> __With a confidence level of 95%, the 75th percentile on the metric is smaller or equal to 1.92s.__ \n", "In other words, with a probability of 95%, the performance metric is smaller or equal to 1.92s \n", "in at least three quarters of the runs. \n", "\n", "Even when the independence test fails, the KPI value is computed and returned. However, \n", "the user must then __be aware that the resulting KPI is not a trustworthy estimate__ of the \n", "corresponding percentile (i.e., it is not a valid confidence interval).\n", "\n", "> For more details about the implementation of the empirical independence test, refer to the [paper](https://doi.org/10.5281/zenodo.3464273). \n", "\n", "Moreover, if there are not enough data points to compute the desired KPI, the function returns `np.nan` (not-a-number).\n", "\n", "Optionally, the `analysis_kpi()` function produces and displays 3 plots: \n", "- the metric data series (`series`)\n", "- the autocorrelation plot (`autocorr`)\n", "- the metric data and the corresponding KPI value (`horizontal`)\n", "\n", "This is illustrated below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Computes the KPI and plot\n", "indep_test_passed, KPI_value = triscale.analysis_kpi(\n", " df.metric.values,\n", " KPI,\n", " to_plot=['series','autocorr','horizontal']\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Sequels and Variability Scores\n", "\n", "Sequels are repetitions of series of runs. _TriScale_'s variability scores measure the variations \n", "of KPI values across sequels. Hence, sequels aim to detect long-term variations of KPIs and, \n", "ultimately, to quantify the replicability of an experiment. \n", "\n", "Concretely, a __variability score is composed of two one-sided CI for a symmetric pair of percentiles;__ \n", "e.g., a 75% confidence interval for the 25-75th percentiles. The underlying computations are \n", "the same as for the [KPIs values](#Series-and-KPIs). The computation of variability scores is implmented in the \n", "`analysis_variability()` function, which takes two compulsory arguments:\n", "- the KPI data;\n", "- the variability score definition.\n", "\n", "The KPI data can be passed as a list or an NumPy array. The variability score definition is provided \n", "as a dictionary with three compulsory keys: \n", "- `percentile` ($0 __With a confidence level of 95%, the inter-quartile (25th-75th perc) range on the KPIs is smaller or equal to 0.4s.__ \n", "In other words, with a probability of 95%, across all series, the middle 50% of KPI values differ by 0.4s or less.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Your turn: time to practice\n", "\n", "We have collected data for a comparative evaluation of congestion-control schemes using the \n", "[Pantheon platform.](https://pantheon.stanford.edu/) More details about the experiment setup can be found in the [TriScale paper](https://doi.org/10.5281/zenodo.3464273).\n", "\n", "We performed five series of ten runs each, for 16 different schemes. For the purpose of this tutorial, \n", "we provide a dataset containing pre-computed metric values for each run. It contains two metrics:\n", "- the mean **throughput;** \n", "- the 95th percentile of the **one-way delay.**\n", "\n", "You task consists in analysing this dataset using _TriScale._ The goal is to compute and compare the \n", "variability scores of different congestion-control schemes. We will focus on the throughput metric only \n", "(an arbitrary choice).\n", "\n", "Let us first load and visualise the dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load and display the entire dataset\n", "df = pd.read_csv(Path('ExampleData/metrics_wo_convergence.csv'))\n", "display(df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can easily extract the lists of `schemes` used and `dates` identifying each series." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Extract the list of congestion control schemes\n", "schemes = df.cc.unique()\n", "print(schemes)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following cell contains a simple function to filter this dataset and extract metric values \n", "per scheme and per series." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def get_metrics(df, scheme, metric):\n", " '''Parse the dataset to extract the series of metric values f\n", " or one scheme and all series of runs.\n", " '''\n", " # Initialize output\n", " metric_data = []\n", " \n", " # List of dates (identifies the series)\n", " dates = df.datetime.unique()\n", " \n", " # For each series\n", " for date in dates:\n", " \n", " # Setup the data filter\n", " filter = (\n", " (df.cc == scheme) &\n", " (df.datetime == date) \n", " )\n", "\n", " # Filter\n", " df_filtered = df.where(filter).dropna()\n", " \n", " # Store metrics values for that series\n", " metric_data.append(list(df_filtered[(metric+'_value')].values)) \n", " \n", " # Return the desired metric data\n", " return metric_data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will use this function to easily extract all metrics values for one scheme and one metric \n", "(e.g., the `throughput` of `bbr`).\n", "\n", "The definition of KPI and variability score to use for our analysis are provided below. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# KPI\n", "KPI = {'percentile': 25,\n", " 'confidence': 75,\n", " 'name': 'KPI Throughput',\n", " 'unit': 'Mbit/s',\n", " 'bounds':[0,120], # expected value range\n", " }\n", "\n", "# Variability score\n", "score = {'percentile': 75,\n", " 'confidence': 75,\n", " 'name': 'Throughput',\n", " 'unit': 'Mbit/s',\n", " 'bounds':[0,120], # expected value range\n", " }" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "> **Note.** We aim to estimate the 25th percentile for the `throughput`, where higher is better. \n", "Thus, the KPI provides the performance expected in at least 75% of the runs.\n", "\n", "You are now all set to analyse this dataset! \n", "\n", "In the following cell, we provide a skeletton code which you should complete and excute in order to \n", "answer the following questions:\n", "\n", "- What is the variability score of the `throughput` metric of the `bbr` congestion-control scheme?\n", "\n", "Modify the definition of the variability score to estimate the median `'percentile': 50` instead of \n", "the 25-75th percentile range. \n", "\n", "- What is the value of the variability score now? Does this make sense to you?\n", "\n", "_Optional (and harder) questions:_ \n", "\n", "- Compute the scores for all the schemes. Do they vary a lot? \n", "- Do the variability scores seem \"big\" with respect to the range of KPI values? \n", "- Would you say that these experiment appear to be replicable?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "# Extract and display the metrics values for the 5 series of 10 runs \n", "scheme = 'bbr'\n", "metric = 'throughput' # valid options are 'throughput' and 'delay'\n", "metric_data = get_metrics(df, scheme, metric)\n", "\n", "# Initialize an empty list to collect the KPI values\n", "KPI_values = [] \n", "\n", "## Step 1. Compute the KPIs\n", "for series_data in metric_data:\n", " \n", " ########## YOUR CODE HERE ###########\n", " # - compute the KPI value \n", " # - if the independence test is passed, \n", " # store the value in the KPI list\n", " #####################################\n", "\n", " \n", "# Print the (valid) KPI values\n", "s = '%i valid KPIs obtained\\n> ' % len(KPI_values)\n", "for k in KPI_values:\n", " s += '%0.2f ' % k\n", "s += '\\n in %s\\n' % KPI['unit']\n", "print(s)\n", " \n", " \n", "## Step 2. Compute the variability score\n", "\n", "########## YOUR CODE HERE ###########\n", "# - compute the variability score \n", "# - print the result depending on the outcome\n", "# - if there are not enough KPIs, print `nan`\n", "# - if the independence test fails, print the score value *(-1)\n", "# - else, print the score value\n", "#####################################\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Solutions\n", "\n", "
\n", "
Click here show the solutions\n", " \n", "```python\n", " \n", "########## YOUR CODE HERE ########### \n", "# - compute the KPI value \n", "# - if the independence test is passed, \n", "# store the value in the KPI list\n", "\n", "indep_test_passed, KPI_value = triscale.analysis_kpi(\n", " series_data,\n", " KPI)\n", "if indep_test_passed:\n", " KPI_values.append(KPI_value)\n", " \n", "#\n", "#####################################\n", " \n", "########## YOUR CODE HERE ###########\n", "# - compute the variability score \n", "# - print the result depending on the outcome\n", "# - if there are not enough KPIs, print `nan`\n", "# - if the independence test fails, print the score value *(-1)\n", "# - else, print the score value\n", "\n", "(indep_test_passed, \n", " upper_bound, \n", " lower_bound, \n", " var_score, \n", " rel_score) = triscale.analysis_variability(\n", " KPI_values,\n", " score\n", ")\n", "\n", "if not indep_test_passed: \n", " var_score *= -1\n", "\n", "print('Variability score: %0.2f %s' % (var_score, score['unit']))\n", " \n", "#\n", "#####################################\n", " \n", "```\n", "These code blocks lead to the following output:\n", " \n", "```\n", "5 valid KPIs obtained\n", "> 105.07 105.04 104.68 104.92 105.08 \n", " in Mbit/s\n", "\n", "Variability score: 0.41 Mbit/s\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "Next step: [Seasonal Components](tutorial_seasonal-comp.ipynb) \n", "[Back to repo](.)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.9.1 64-bit ('triscale': conda)", "language": "python", "name": "python391jvsc74a57bd0684f90775fb1f43db0d8eed0780ba829f42a97c2f4b9a1bd592c18e47e5c272e" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" }, "metadata": { "interpreter": { "hash": "684f90775fb1f43db0d8eed0780ba829f42a97c2f4b9a1bd592c18e47e5c272e" } } }, "nbformat": 4, "nbformat_minor": 2 }