{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## TriScale\n", "# Case Study - Video Streaming\n", "\n", "This notebook presents a case study of the TriScale framework. It revisits the analysis of [Pensieve](https://dl.acm.org/doi/10.1145/3098822.3098843), a system that generates adaptive bitrate algorithms for video streaming using reinforcement learning. Parts of this case study are described in the [TriScale paper](https://doi.org/10.5281/zenodo.3464273).\n", "\n", "## Evaluation objectives\n", "\n", "In this case study, various adaptive bitrate algorithms are compared using a user quality of experimence (QoE) as metric.\n", "\n", "The experiment has been designed and performed by the authors of [the Pensieve paper](https://dl.acm.org/doi/10.1145/3098822.3098843). In this case study, we show how _TriScale_ can be used to provide confidence intervals not only on single KPIs, but on entire cumulative distribution functions (CDFs).\n", "\n", "## List of Imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import copy\n", "from pathlib import Path\n", "import zipfile\n", "\n", "import pandas as pd\n", "import numpy as np\n", "import plotly.graph_objects as go\n", "\n", "import triscale\n", "import triplots" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download Source Files and Data\n", "[[Back to top](#TriScale)]\n", "\n", "The dataset for this case study is available on Zenodo: \n", "\n", "[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3451417.svg)](https://doi.org/10.5281/zenodo.3451417)\n", "\n", "\n", "The wget commands below download the required files to reproduce this case study.\n", "\n", "> **The .zip file is ~620 kB**" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Nothing to download\n" ] } ], "source": [ "# Set `download = True` to download (and extract) the data from this case study\n", "# Eventually, adjust the record_id for the file version you are interested in.\n", "\n", "# For reproducing the results of the TriScale paper, set `record_id = 3666724`\n", "\n", "download = True\n", "record_id = 3666724 # v3.0.1 (https://doi.org/10.5281/zenodo.3666724)\n", "\n", "files= ['UseCase_VideoStreaming.zip']\n", "if download:\n", " for file in files:\n", " print(file)\n", " url = 'https://zenodo.org/record/'+str(record_id)+'/files/'+file \n", " os.system('wget %s' %url)\n", " if file[-4:] == '.zip': \n", " with zipfile.ZipFile(file,\"r\") as zip_file:\n", " zip_file.extractall()\n", " print('Done.')\n", "else: \n", " print('Nothing to download')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now import the custom module for the case study. " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import UseCase_VideoStreaming.videostreaming as vs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Compute KPIs\n", "\n", "The metric values are given (retrieved from the Pensieve paper experiments). For each algorithm, we compute a set of KPIs which range from the 5th to the 98th percentile. Since our metric is QoE (larger is better) we compute lower-bounds for all KPIs." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Output retrieved from file. Skipping computation.\n" ] } ], "source": [ "# Construct the path to the different test results\n", "result_dir_path = Path('UseCase_VideoStreaming/FCC/linear')\n", "protocol_list = [x.stem for x in result_dir_path.iterdir()]\n", "protocol_list = list(set(protocol_list))\n", "config_file = Path('UseCase_VideoStreaming/config.yml')\n", "\n", "# Define the KPIs\n", "KPI_percentiles = np.arange(5,100,2) # percentiles\n", "KPI_confidence = 95 # confidence level\n", "KPI_base = {'confidence': KPI_confidence,\n", " 'bound': 'lower',\n", " 'unit': '',\n", " }\n", "KPI_list = []\n", "for p in KPI_percentiles:\n", " kpi = copy.deepcopy(KPI_base)\n", " kpi['percentile'] = p\n", " kpi['name'] = 'P%d'%p\n", " KPI_list.append(kpi)\n", " \n", "# Compute and store KPIs\n", "out_name = Path('UseCase_VideoStreaming') / 'kpis.csv'\n", "QoE = vs.compute_kpi(\n", " protocol_list,\n", " KPI_list,\n", " result_dir_path,\n", " out_name=out_name\n", ")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ProtocolNetworkQoEPercentileKPI
0robust_mpcFCClinear2-0.760
1robust_mpcFCClinear4-0.033
2robust_mpcFCClinear60.114
3robust_mpcFCClinear80.140
4robust_mpcFCClinear100.163
..................
338bufferFCClinear901.073
339bufferFCClinear921.134
340bufferFCClinear941.197
341bufferFCClinear961.614
342bufferFCClinear982.237
\n", "

343 rows × 5 columns

\n", "
" ], "text/plain": [ " Protocol Network QoE Percentile KPI\n", "0 robust_mpc FCC linear 2 -0.760\n", "1 robust_mpc FCC linear 4 -0.033\n", "2 robust_mpc FCC linear 6 0.114\n", "3 robust_mpc FCC linear 8 0.140\n", "4 robust_mpc FCC linear 10 0.163\n", ".. ... ... ... ... ...\n", "338 buffer FCC linear 90 1.073\n", "339 buffer FCC linear 92 1.134\n", "340 buffer FCC linear 94 1.197\n", "341 buffer FCC linear 96 1.614\n", "342 buffer FCC linear 98 2.237\n", "\n", "[343 rows x 5 columns]" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(QoE)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "line": { "shape": "hv" }, "marker": { "color": "rgba(237, 65, 29, 1)" }, "mode": "lines", "name": "95% CI on CDF", "type": "scatter", "x": [ 0.076, 0.165, 0.2, 0.23, 0.255, 0.28, 0.289, 0.305, 0.314, 0.332, 0.406, 0.471, 0.499, 0.537, 0.574, 0.588, 0.606, 0.625, 0.633, 0.64, 0.652, 0.6579999999999999, 0.665, 0.6709999999999999, 0.68, 0.6940000000000001, 0.711, 0.7390000000000001, 0.779, 0.8059999999999999, 0.8440000000000001, 0.871, 0.893, 0.951, 1.103, 1.138, 1.167, 1.217, 1.278, 1.2990000000000002, 1.311, 1.342, 1.42, 1.483, 1.571, 1.891, 2.17, 2.6460000000000004, 2.842 ], "y": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98 ] }, { "fill": "tozeroy", "fillcolor": "rgba(237, 65, 29, 0.2)", "line": { "shape": "hv" }, "mode": "none", "name": "Sample CDF", "opacity": 0.2, "type": "scatter", "x": [ 0.076, 0.087, 0.136, 0.165, 0.175, 0.187, 0.2, 0.213, 0.22399999999999998, 0.23, 0.239, 0.247, 0.255, 0.263, 0.26899999999999996, 0.27699999999999997, 0.28, 0.28300000000000003, 0.285, 0.289, 0.295, 0.29600000000000004, 0.302, 0.305, 0.308, 0.311, 0.314, 0.32, 0.322, 0.325, 0.332, 0.342, 0.35700000000000004, 0.406, 0.415, 0.428, 0.466, 0.47100000000000003, 0.479, 0.484, 0.494, 0.499, 0.511, 0.523, 0.537, 0.5489999999999999, 0.5589999999999999, 0.568, 0.574, 0.5760000000000001, 0.5820000000000001, 0.585, 0.588, 0.594, 0.596, 0.602, 0.606, 0.61, 0.616, 0.619, 0.625, 0.628, 0.628, 0.633, 0.634, 0.635, 0.64, 0.64, 0.643, 0.6459999999999999, 0.6459999999999999, 0.652, 0.652, 0.652, 0.657, 0.6579999999999999, 0.6579999999999999, 0.662, 0.665, 0.665, 0.667, 0.6709999999999999, 0.6709999999999999, 0.6709999999999999, 0.675, 0.677, 0.677, 0.68, 0.6829999999999999, 0.685, 0.6890000000000001, 0.6940000000000001, 0.696, 0.7020000000000001, 0.705, 0.711, 0.716, 0.722, 0.728, 0.7390000000000001, 0.75, 0.754, 0.774, 0.779, 0.7809999999999999, 0.787, 0.79, 0.8059999999999999, 0.8140000000000001, 0.825, 0.838, 0.8440000000000001, 0.852, 0.858, 0.8640000000000001, 0.871, 0.877, 0.884, 0.887, 0.893, 0.8959999999999999, 0.899, 0.905, 0.951, 0.975, 0.982, 1.075, 1.095, 1.103, 1.127, 1.129, 1.135, 1.138, 1.146, 1.15, 1.158, 1.167, 1.177, 1.185, 1.203, 1.217, 1.238, 1.255, 1.2670000000000001, 1.278, 1.287, 1.2930000000000001, 1.296, 1.298, 1.2990000000000002, 1.304, 1.305, 1.306, 1.311, 1.311, 1.314, 1.317, 1.3419999999999999, 1.3519999999999999, 1.354, 1.3980000000000001, 1.4140000000000001, 1.42, 1.439, 1.45, 1.474, 1.483, 1.5030000000000001, 1.531, 1.5390000000000001, 1.548, 1.571, 1.651, 1.665, 1.6869999999999998, 1.838, 1.891, 1.909, 2.052, 2.11, 2.17, 2.256, 2.4090000000000003, 2.499, 2.5980000000000003, 2.6460000000000004, 2.6689999999999996, 2.7089999999999996, 2.736, 2.773, 2.801, 2.842, 2.852, 2.862, 2.863, 2.8680000000000003, 2.873, 2.875, 2.8810000000000002 ], "y": [ 0.5, 1, 1.5, 2, 2.5, 3, 3.5000000000000004, 4, 4.5, 5, 5.5, 6, 6.5, 7.000000000000001, 7.5, 8, 8.5, 9, 9.5, 10, 10.5, 11, 11.5, 12, 12.5, 13, 13.5, 14.000000000000002, 14.499999999999998, 15, 15.5, 16, 16.5, 17, 17.5, 18, 18.5, 19, 19.5, 20, 20.5, 21, 21.5, 22, 22.5, 23, 23.5, 24, 24.5, 25, 25.5, 26, 26.5, 27, 27.500000000000004, 28.000000000000004, 28.499999999999996, 28.999999999999996, 29.5, 30, 30.5, 31, 31.5, 32, 32.5, 33, 33.5, 34, 34.5, 35, 35.5, 36, 36.5, 37, 37.5, 38, 38.5, 39, 39.5, 40, 40.5, 41, 41.5, 42, 42.5, 43, 43.5, 44, 44.5, 45, 45.5, 46, 46.5, 47, 47.5, 48, 48.5, 49, 49.5, 50, 50.5, 51, 51.5, 52, 52.5, 53, 53.5, 54, 54.50000000000001, 55.00000000000001, 55.50000000000001, 56.00000000000001, 56.49999999999999, 56.99999999999999, 57.49999999999999, 57.99999999999999, 58.5, 59, 59.5, 60, 60.5, 61, 61.5, 62, 62.5, 63, 63.5, 64, 64.5, 65, 65.5, 66, 66.5, 67, 67.5, 68, 68.5, 69, 69.5, 70, 70.5, 71, 71.5, 72, 72.5, 73, 73.5, 74, 74.5, 75, 75.5, 76, 76.5, 77, 77.5, 78, 78.5, 79, 79.5, 80, 80.5, 81, 81.5, 82, 82.5, 83, 83.5, 84, 84.5, 85, 85.5, 86, 86.5, 87, 87.5, 88, 88.5, 89, 89.5, 90, 90.5, 91, 91.5, 92, 92.5, 93, 93.5, 94, 94.5, 95, 95.5, 96, 96.5, 97, 97.5, 98, 98.5, 99, 99.5 ] } ], "layout": { "template": { "data": { "scatter": [ { "type": "scatter" } ] } }, "xaxis": { "title": { "text": "Mean QoE" } }, "yaxis": { "title": { "text": "CDF" } } } }, "text/html": [ "
\n", " \n", " \n", "
\n", " \n", "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "sample = dict(\n", " sample_cdf=True,\n", " protocol=['pensieve']\n", ")\n", "figure = vs.plot_cdf(\n", " QoE,\n", " config_file,\n", " result_dir_path,\n", " sample=sample\n", ")\n", "figure.show()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "line": { "shape": "hv" }, "marker": { "color": "rgb(102, 49, 160)" }, "mode": "lines", "name": "Buffer-based", "type": "scatter", "x": [ -2.699, -2.317, -1.7990000000000002, -1.569, -1.4469999999999998, -1.369, -1.241, -0.932, -0.821, -0.7609999999999999, -0.596, -0.481, -0.424, -0.281, -0.238, -0.181, -0.104, -0.056, 0.014, 0.038, 0.064, 0.09, 0.129, 0.155, 0.188, 0.205, 0.242, 0.282, 0.322, 0.353, 0.382, 0.428, 0.444, 0.465, 0.511, 0.586, 0.657, 0.7, 0.7709999999999999, 0.861, 0.872, 0.898, 0.94, 0.987, 1.073, 1.134, 1.197, 1.614, 2.237 ], "y": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98 ] }, { "line": { "shape": "hv" }, "marker": { "color": "rgb(247, 0, 247)" }, "mode": "lines", "name": "Rate-based", "type": "scatter", "x": [ -0.438, -0.235, -0.024, 0.014, 0.032, 0.05, 0.1, 0.121, 0.165, 0.19, 0.196, 0.197, 0.203, 0.224, 0.244, 0.246, 0.252, 0.252, 0.272, 0.302, 0.358, 0.417, 0.457, 0.48, 0.507, 0.536, 0.5529999999999999, 0.5710000000000001, 0.5870000000000001, 0.604, 0.609, 0.62, 0.6409999999999999, 0.6779999999999999, 0.706, 0.7240000000000001, 0.73, 0.747, 0.8109999999999999, 0.87, 0.989, 1.081, 1.147, 1.178, 1.185, 1.391, 1.694, 2.083, 2.525 ], "y": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98 ] }, { "line": { "shape": "hv" }, "marker": { "color": "rgb(255, 192, 0)" }, "mode": "lines", "name": "BOLA", "type": "scatter", "x": [ -0.32299999999999995, -0.152, 0.009000000000000001, 0.087, 0.125, 0.13699999999999998, 0.13699999999999998, 0.154, 0.171, 0.199, 0.218, 0.227, 0.242, 0.25, 0.261, 0.273, 0.273, 0.28, 0.29600000000000004, 0.313, 0.331, 0.344, 0.3670000000000001, 0.405, 0.447, 0.489, 0.516, 0.541, 0.5589999999999999, 0.581, 0.593, 0.624, 0.664, 0.6920000000000001, 0.72, 0.733, 0.7659999999999999, 0.777, 0.8290000000000001, 0.973, 1.026, 1.1, 1.163, 1.185, 1.197, 1.261, 1.6, 1.957, 2.417 ], "y": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98 ] }, { "line": { "shape": "hv" }, "marker": { "color": "rgb(0, 212, 97)" }, "mode": "lines", "name": "MPC", "type": "scatter", "x": [ -1.87, -1.249, -0.8, -0.615, -0.464, -0.337, -0.311, -0.243, -0.203, -0.145, -0.052000000000000005, 0.006, 0.059, 0.08800000000000001, 0.109, 0.129, 0.136, 0.154, 0.16699999999999998, 0.184, 0.202, 0.213, 0.228, 0.24, 0.252, 0.262, 0.279, 0.315, 0.353, 0.391, 0.425, 0.453, 0.473, 0.493, 0.525, 0.56, 0.59, 0.61, 0.6509999999999999, 0.6859999999999999, 0.703, 0.732, 0.821, 0.905, 1.12, 1.18, 1.217, 1.295, 2.071 ], "y": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98 ] }, { "line": { "shape": "hv" }, "marker": { "color": "rgb(0, 112, 192)" }, "mode": "lines", "name": "robustMPC", "type": "scatter", "x": [ -0.76, -0.033, 0.114, 0.14, 0.163, 0.19, 0.215, 0.228, 0.242, 0.258, 0.273, 0.29600000000000004, 0.313, 0.33, 0.3720000000000001, 0.396, 0.422, 0.48, 0.528, 0.555, 0.569, 0.58, 0.5870000000000001, 0.593, 0.599, 0.608, 0.622, 0.633, 0.654, 0.688, 0.7170000000000001, 0.762, 0.7859999999999999, 0.8109999999999999, 0.867, 0.9, 0.926, 0.972, 1.042, 1.104, 1.168, 1.195, 1.205, 1.222, 1.251, 1.3259999999999998, 1.628, 2.186, 2.428 ], "y": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98 ] }, { "line": { "shape": "hv" }, "marker": { "color": "rgba(237, 65, 29, 1)" }, "mode": "lines", "name": "Pensieve", "type": "scatter", "x": [ 0.076, 0.165, 0.2, 0.23, 0.255, 0.28, 0.289, 0.305, 0.314, 0.332, 0.406, 0.471, 0.499, 0.537, 0.574, 0.588, 0.606, 0.625, 0.633, 0.64, 0.652, 0.6579999999999999, 0.665, 0.6709999999999999, 0.68, 0.6940000000000001, 0.711, 0.7390000000000001, 0.779, 0.8059999999999999, 0.8440000000000001, 0.871, 0.893, 0.951, 1.103, 1.138, 1.167, 1.217, 1.278, 1.2990000000000002, 1.311, 1.342, 1.42, 1.483, 1.571, 1.891, 2.17, 2.6460000000000004, 2.842 ], "y": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98 ] }, { "line": { "shape": "hv" }, "marker": { "color": "black" }, "mode": "lines", "name": "Offline optimal", "type": "scatter", "x": [ 0.264, 0.298, 0.332, 0.34700000000000003, 0.361, 0.382, 0.41, 0.422, 0.428, 0.438, 0.464, 0.505, 0.5329999999999999, 0.61, 0.6409999999999999, 0.6779999999999999, 0.703, 0.711, 0.7170000000000001, 0.7290000000000001, 0.741, 0.748, 0.755, 0.76, 0.7859999999999999, 0.81, 0.83, 0.863, 0.907, 0.937, 0.963, 0.986, 1.027, 1.148, 1.249, 1.28, 1.317, 1.344, 1.3630000000000002, 1.399, 1.459, 1.602, 1.655, 1.72, 1.827, 1.954, 2.167, 2.748, 2.969 ], "y": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98 ] } ], "layout": { "template": { "data": { "scatter": [ { "type": "scatter" } ] } }, "xaxis": { "title": { "text": "Mean QoE" } }, "yaxis": { "title": { "text": "CDF" } } } }, "text/html": [ "
\n", " \n", " \n", "
\n", " \n", "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "sample = dict(\n", " sample_cdf=False,\n", ")\n", "figure = vs.plot_cdf(\n", " QoE,\n", " config_file,\n", " result_dir_path,\n", " sample=sample\n", ")\n", "figure.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 }