{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# _TriScale_ - Seasonal Components\n", "\n", "\n", "> This notebook is intended for **live tutorial** sessions about _TriScale._ \n", "Here is the [self-study version](tutorial_seasonal-comp.ipynb)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get started, we need to import a few Python modules. All the _TriScale_-specific functions are part of one module called `triscale`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "from pathlib import Path\n", "import datetime\n", "\n", "import pandas as pd\n", "import numpy as np\n", "\n", "import triscale" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analysis of Glossy data (low-power wireless)\n", "\n", "We consider performance data from [Glossy](https://ieeexplore.ieee.org/document/5779066) collected on [the FlockLab testbed](http://flocklab.ethz.ch/) as experiment environment. \n", "\n", "[Glossy](https://ieeexplore.ieee.org/document/5779066) is a low-power wireless protocol based on synchronous transmissions and a flooding strategy. One important tuning parameter of Glossy is the number of times $N$ that each node transmit each packet.\n", "\n", "The literature reports that a larger $N$ yields better reliability; that is, a larger packet reception ratio (PRR). We performed a short experimental study to validate this observation. \n", "More specifically, we test two values:\n", "- $N=1$\n", "- $N=2$\n", "\n", "> For a more extensive description of the data collection and analysis, you can check the [complete case study notebook](casestudy_glossy.ipynb) or the [_TriScale_ paper](https://doi.org/10.5281/zenodo.3464273) itself.\n", "\n", "In the nutshell, the dataset contains:\n", "- one metric: the median **packet reception ratio** between all nodes, or **PRR** for short\n", "- measured **24 times** per day, scheduled randomly\n", "- collected during **three weeks**\n", "- using both $N=1$ and $N=2$" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load the PRR results from the test\n", "glossy = pd.read_csv('ExampleData/metrics_glossy.csv', index_col=0, parse_dates=True)\n", "\n", "# Display a random sample\n", "glossy.sample(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Note.** To avoid bias during the analysis, the dataset has been \"anonymized;\" that is, we randomly replaced the value of $N$ with a letter ($A$ or $B$).\n", "\n", "The FlockLab testbed is located in an office building, where we expect more wireless interference during the day than during the night. Thus, for a fair comparison, the time span of a series of runs should be at least one day (24 hours).\n", "\n", "Let us select two days and compare the PRR of $A$ and $B$ on those days. The KPI definition is given below. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Days considered for the data analysis\n", "day_A = '2019-08-24'\n", "day_B = '2019-08-26'\n", "\n", "# Fitering the dataset for the data or interest\n", "data_A = glossy.loc[day_A].PRR_A.dropna().values\n", "data_B = glossy.loc[day_B].PRR_B.dropna().values\n", "\n", "# KPI definition\n", "KPI = {'name': 'PRR',\n", " 'unit': '\\%',\n", " 'percentile': 50,\n", " 'confidence': 95,\n", " 'class': 'one-sided',\n", " 'bounds': [0,100],\n", " 'bound': 'lower'}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use the `triscale.analysis_kpi()` function to compute the KPI value for each group. \n", "\n", "- Which group seems to perform best?\n", "- What confidence to you have in this result?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_A, KPI_A = triscale.analysis_kpi(data_A, KPI)\n", "test_B, KPI_B = triscale.analysis_kpi(data_B, KPI)\n", "\n", "print('KPI group A: {}'.format(KPI_A))\n", "print('KPI group B: {}'.format(KPI_B))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$A$ seems to perform better than $B$. \n", "\n", "But, even if the KPI has been defined with a high level of confidence, it **does not** mean that the experimental conditions during the two days were actually comparable... \n", "\n", "As a matter of fact, $A$ corresponds to $N=1$ which is highly unlikely to perform better than $N=2$..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What about seasonality? \n", "\n", "In the previous analysis, we (randomly?) picked some days for each group. But what\n", "do we know about the possible correlation between those two days? \n", "- Maybe we got unlucky on the day $B$ was tested?\n", "- Or maybe we omitted some hidden factor?\n", "\n", "To investigate that, we can look at the [wireless link quality data for FlockLab](https://doi.org/10.5281/zenodo.3354717), which is collected by the FlockLab maintainers and made publicly available. They ran the link quality tests every two hours, resulting in 12 measurement points per day.\n", "\n", "In this tutorial, we look at the data from August 2019, which has a large overlap with\n", "our data collection period. Let's load this dataset and have a look..." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "link_quality = pd.read_csv('ExampleData/flocklab_link_quality.csv', index_col=0, parse_dates=True)\n", "link_quality.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The dataset is simple: every two hours, we have one value representing the \"average \n", "link quality\" on the testbed (the computation that led to this average is irrelevant here).\n", "\n", "_TriScale_'s `network_profiling()` generates an autocorellation plot based on such data, as illustrated below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "link_quality_bounds = [0,100]\n", "link_quality_name = 'PRR [%]'\n", "fig_theil, fig_autocorr = triscale.network_profiling(\n", " link_quality, \n", " link_quality_bounds, \n", " link_quality_name,\n", ")\n", "fig_autocorr.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One can clearly see from the autocorrelation plot that the average link quality on FlockLab has strong seasonal components. The **first pic at lag 12 (i.e., 24h)** reveals the daily seasonal component. \n", "\n", "But there is also **a second main peak at lag 84**; which corresponds to one week.\n", "Indeed, there is less interference in the weekends than on weekdays, which creates a weekly seasonal component.\n", "\n", "Due to this weekly component, it becomes problematic (aka, potentially wrong) to\n", "compare results from different time periods which span less than a week.\n", "In other word, the time span for series of runs must be at least one week long\n", "to be fairly comparable.\n", "\n", "Let us quickly check which days of the week we picked for our first analysis..." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def weekday(str):\n", " '''Simple function printing the weekday\n", " from a date given as a string\n", " '''\n", " year, month, day = (int(x) for x in str.split('-')) \n", " ans = datetime.date(year, month, day)\n", " return ans.strftime(\"%A\")\n", " \n", "print('Data from group A was from a {}.'.format(weekday(day_A)))\n", "print('Data from group A was from a {}.'.format(weekday(day_B)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bingo! $B$ was tested on a weekday, while $A$ was tested on a weekend...\n", "\n", "> **Takeaway.** The day of the week was a \"hidden\" factor in our first analysis. Neglecting it led to wrong conclusions. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Your turn: time to practice\n", "\n", "Let us now use the entire Glossy dataset and analyse it as one series (with a span of three weeks)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_A = glossy.PRR_A.dropna().values\n", "data_B = glossy.PRR_B.dropna().values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use again the `triscale.analysis_kpi()` function to compute the KPI value for each group.\n", "- Which group seems to perform best now?\n", "- What about independence? Do you think the results are trustworthy?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "########## YOUR CODE HERE ###########\n", "# ...\n", "#####################################" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Solution\n", "\n", "
\n", "
Click here to show the solution.\n", " \n", "```python\n", ">>> triscale.analysis_kpi(data_A, KPI)\n", "(False, 80.0)\n", ">>> triscale.analysis_kpi(data_B, KPI)\n", "(False, 88.0)\n", "```\n", "Now, we do obtain the expected result: $N=2$ (group $B$) performs better than $N=1$.\n", "Note however that the independence test fails. This is due to the ordering of the tests:\n", "We scheduled tests randomly every day individually, not over the 3 weeks time span.\n", "Therefore, the data are affected by the (strong) weekly correlation on the environment.\n", " \n", "We can observe this correlation bt plotting the data and/or it's autocorellation function:\n", "```python\n", ">>> plots=['series','autocorr']\n", ">>> triscale.analysis_kpi(data_A, KPI, plots)\n", "```\n", "> **Note.** Contrary to the `link_quality` dataset, the Glossy data (`data_A` and `data_B`) is not ordered in time (which is intentional, since we try to discard the time factor from the analysis). Therefore, it is difficult to interpret the location of the peaks in the autocorrelation plot; mainly, we can observe that there is a strong correlation structure in the dataset. \n", " \n", "We can try to emulate the fact that we'd have properly randomized the run epochs by shuffling the data.\n", " \n", "```python\n", ">>> import random\n", ">>> random.shuffle(data_A)\n", ">>> to_plot=['autocorr']\n", ">>> triscale.analysis_kpi(data_A, KPI, to_plot)\n", "``` \n", " \n", "As you can see, the correlation structure significantly flattens. In some cases, the independence test might even pass... But keep in mind that this it is only an artifact! To make a strong statement, the run epochs should have been truly randomized.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "--- \n", "[Back to main repository](.)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.9.1 64-bit ('triscale': conda)", "language": "python", "name": "python391jvsc74a57bd0684f90775fb1f43db0d8eed0780ba829f42a97c2f4b9a1bd592c18e47e5c272e" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" }, "metadata": { "interpreter": { "hash": "684f90775fb1f43db0d8eed0780ba829f42a97c2f4b9a1bd592c18e47e5c272e" } } }, "nbformat": 4, "nbformat_minor": 2 }