{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Prototype to Production: Python Tools for Rapid Web Interface Development\n", "\n", "## Authors\n", "**Scott D Christensen and Marvin S Brown**\n", "\n", "Contact: scott.d.christensen@usace.army.mil\n", "\n", "![COVID Tethys App Demo](TethysCovid.gif)\n", "\n", "## Abstract\n", "An interactive web application can take a powerful and complex Python workflow to the next level, drastically improve its utility, and enhance its ability to visualize results. The increased capabilities of open-source Python libraries have made the transition from Jupyter notebooks to production-ready web applications easier than ever. This presentation will demonstrate how to prototype a web application in the notebook environment, and then easily deploy it as a stand-alone web application with Panel. Next, we will show how Panel applications can be transitioned to a fully-featured web application in the Tethys Platform.\n", "\n", "## Introduction\n", "Our goal is to be able to rapidly develop web interfaces for engineering and modeling workflows. Jupyter notebooks provide a good starting point to iterate on ideas and start to prototype visualizations. However, we ultimately want a customized, self-contained, interactive web app. Enhancements that have been made to the [Panel](https://panel.holoviz.org/) library have made it easy to transition from prototyping in a Jupyter environment to a stand-alone Bokeh app. Additional enhancements to [Bokeh](https://docs.bokeh.org/) and to [Tethys Platform](http://docs.tethysplatform.org/) have made it possible to then transition your app into the Tethys environment.\n", "\n", "![App Spectrum](Spectrum.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Demonstration\n", "\n", "To demonstrate the use of these tools we will show a dashboard for exploring the COVID-19 data from [The Johns Hopkins Center for Systems Science and Engineering](https://github.com/CSSEGISandData/COVID-19).\n", "\n", "We start with reading in the data and creating a visualization in a notebook:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "import pandas as pd\n", "import numpy as np\n", "import xarray as xr\n", "from pyproj import Transformer\n", "import holoviews as hv\n", "import param\n", "import panel as pn\n", "\n", "hv.extension('bokeh')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Read in Data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "VARS = ['Confirmed', 'Deaths', 'Recovered']\n", "\n", "data_dir = Path('../COVID-19/csse_covid_19_data').resolve()\n", "\n", "time_series_path = data_dir / 'csse_covid_19_time_series' / 'time_series_covid19_{VAR}_global.csv'\n", "\n", "dfs = list()\n", "dims = ['Aggregation', 'Quantity', 'Country', 'Date']\n", "transformer = Transformer.from_crs(\"epsg:4326\", \"epsg:3857\")\n", "for var in VARS:\n", " df = pd.read_csv(time_series_path.as_posix().format(VAR=var.lower()))\n", " df.rename({'Country/Region': 'Country'}, axis=1, inplace=True)\n", " lat_lon = df[['Country', 'Lat', 'Long']]\n", " lat_lon = lat_lon.groupby('Country').first()\n", " lat_lon['x_y'] = lat_lon.apply(lambda row: transformer.transform(row.Lat, row.Long), axis=1)\n", " lat_lon['x'] = lat_lon.x_y.apply(lambda z: z[0])\n", " lat_lon['y'] = lat_lon.x_y.apply(lambda z: z[1])\n", " df.drop(['Lat', 'Long'], axis=1, inplace=True)\n", " df = df.groupby('Country').sum()\n", " df.insert(0, 'y', lat_lon['y'])\n", " df.insert(1, 'x', lat_lon['x'])\n", " df.sort_values(by='Country', inplace=True)\n", " dfs.append(df)\n", "data_vars = [df.iloc[:, 2:].values for df in dfs]\n", "data_vars.append(data_vars[0] - data_vars[1] - data_vars[2]) # active cases\n", "VARS.append('Active')\n", "data_vars = np.stack(data_vars)\n", "data_vars = np.stack((data_vars, np.diff(data_vars, axis=-1, n=1, prepend=0))) # daily changes\n", "data_vars = (data_vars + np.absolute(data_vars)) / 2\n", "data_vars = {'counts': (dims, data_vars)}\n", "\n", "coords = dict(\n", " Country=('Country', df.index),\n", " Date=pd.to_datetime(df.columns.tolist()[2:]),\n", " x=('Country', df['x']),\n", " y=('Country', df['y']),\n", " Quantity=VARS,\n", " Aggregation=['Totals', 'Daily']\n", ")\n", "data = xr.Dataset(data_vars=data_vars, coords=coords)\n", "data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Plot" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "kdims = ['Date', 'Quantity', 'Aggregation']\n", "grouped_data = data.sum('Country')\n", "selected_data = hv.Dataset(\n", " grouped_data, \n", " kdims=kdims, \n", " vdims=['counts']).aggregate(dimensions=['Date', 'Quantity'], function=np.sum)\n", " \n", "curves = selected_data.to(hv.Curve, 'Date', ).options(tools=['hover'], show_grid=True)\n", "overlay = ['Quantity']\n", "\n", "if overlay:\n", " curves = curves.overlay(overlay).options(legend_position='top_left', )\n", "curves.options(\n", " responsive=True,\n", " height=800,\n", " xrotation=60,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Panel Dashboard\n", "Using Panel we can convert our visualization into an interactive dashboard." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class CovidPlotter(param.Parameterized):\n", " plot_type = param.ObjectSelector(default='Curve', objects=['Bar', 'Curve'], precedence=0.1)\n", " aggregation = param.ObjectSelector(default='Totals', objects=data.coords['Aggregation'].values, precedence=0.2)\n", " quantity = param.ObjectSelector(default='All', objects=['All'] + list(data.coords['Quantity'].values), precedence=0.2)\n", " groupby = param.ObjectSelector(default='Global', precedence=0.4, objects=['Global', 'Country'], label='Group By')\n", " countries = param.ListSelector(default=['China', 'US', 'Italy', 'France'], objects=np.unique(data.coords['Country'].values))\n", " # states = param.ListSelector(default=['Utah', 'Mississippi', 'California'], objects=[])\n", " selected_data = param.ClassSelector(hv.Dataset, precedence=-1)\n", "\n", " def __init__(self, **params):\n", " super().__init__(**params)\n", " self.dimensions = None\n", " # self.update_states()\n", " self.select_data()\n", " \n", " @param.depends('plot_type', watch=True)\n", " def update_defaults(self):\n", " if self.plot_type == 'Bar':\n", " self.aggregation = 'Daily'\n", " self.quantity = 'Confirmed'\n", " self.groupby = 'Global'\n", " else:\n", " self.aggregation = 'Totals'\n", " self.quantity = 'All'\n", " \n", " @param.depends('groupby', 'aggregation', 'quantity', 'countries', 'plot_type', watch=True)\n", " def select_data(self):\n", " kdims = ['Date', 'Quantity', 'Aggregation']\n", " self.dimensions = ['Date']\n", " select_kwargs = dict(Aggregation=self.aggregation)\n", " self.param.countries.precedence = -1\n", " # self.param.states.precedence = -1\n", " \n", " if self.groupby == 'Global':\n", " grouped_data = data.sum('Country')\n", " elif self.groupby == 'Country':\n", " grouped_data = data\n", " kdims.append('Country')\n", " self.param.countries.precedence = 1\n", " if len(self.countries) > 1 or self.plot_type == 'Curve':\n", " self.dimensions.append('Country')\n", " select_kwargs['Country'] = self.countries\n", " \n", " if self.quantity == 'All':\n", " self.dimensions.append('Quantity')\n", " else:\n", " select_kwargs['Quantity'] = self.quantity\n", " \n", " selected_data = hv.Dataset(grouped_data, kdims=kdims, vdims=['counts']).select(**select_kwargs).aggregate(dimensions=self.dimensions, function=np.sum)\n", " self.selected_data = selected_data\n", " \n", " @param.depends('selected_data')\n", " def curves(self):\n", " curves = self.selected_data.to(hv.Curve, 'Date', ).options(tools=['hover'], show_grid=True)\n", " overlay = list()\n", " if self.quantity == 'All':\n", " overlay.append('Quantity')\n", " if self.groupby == 'Country':\n", " overlay.append('Country')\n", " if self.groupby == 'State':\n", " overlay.append('Province_State')\n", " \n", " if overlay:\n", " curves = curves.overlay(overlay).options(legend_position='top_left', )\n", " return curves.options(\n", " responsive=True,\n", " height=800,\n", " xrotation=60,\n", " )\n", " \n", " @param.depends('selected_data')\n", " def bars(self):\n", " kdims = self.dimensions\n", " return hv.Bars(self.selected_data, kdims=kdims).opts(\n", " responsive=True,\n", " height=800,\n", " stacked=False, \n", " show_legend=False, \n", " xrotation=60,\n", " tools=['hover'],\n", " )\n", " \n", " @param.depends('plot_type')\n", " def plot(self):\n", " return dict(\n", " Bar=self.bars,\n", " Curve=self.curves,\n", " )[self.plot_type]\n", " \n", " def panel(self):\n", " return pn.Row(\n", " pn.Param(self,\n", " widgets={\n", " 'countries': {'height': 550},\n", " },\n", " show_name=False),\n", " self.plot,\n", " )\n", "\n", "CovidPlotter().panel()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Continuing this process we created several visualizations that are all tied into a single Panel app. We exported the code into external Python files that can then be simply imported and executed in a Notebook:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from covid_dashboard import CovidDashboard" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Complete Panel App" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "CovidDashboard().panel()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Converting to a Stand-alone Bokeh App\n", "\n", "When running a notebook locally you can launch a stand-alone Bokeh app right from the notebook by adding a `.show()` to any Panel output.\n", "\n", "In our case this would be:\n", "\n", "```python\n", "CovidDashboard().panel().show()\n", "```\n", "\n", "Alternatively you can create a Python script, `main.py`, similar to the following:\n", "\n", "```python\n", "# filename: main.py\n", "\n", "from covid_dashboard import CovidDashboard\n", "\n", "cd = CovidDashboard().panel()\n", "cd.servable()\n", "```\n", " \n", "If the `main.py` script were located in a directory called `bokeh_app/` then this Bokeh application could be launched from the commandline like this:\n", "\n", "```bash\n", "panel serve /path/to/bokeh_app/\n", "```\n", "\n", "To see this served as a Bokeh app in Binder see the following link:\n", "\n", "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/sdc50/covid-19-dashboard/master?urlpath=/proxy/5006/panel_app)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Converting to a Production Tethys App\n", "\n", "Converting the app to Tethys is not quite as simple as moving from the notebook to a Bokeh app, but it is still very easy and requires minimal code.\n", "\n", "After creating a [Tethys App Scaffold](http://docs.tethysplatform.org/en/stable/tutorials/key_concepts/new_app_project.html), then a `handlers.py` file needs to be added that is similar to the `main.py` file mentioned above. It should contain code that is something like this:\n", "\n", "```python\n", "from bokeh.document import Document\n", "\n", "from covid_dashboard import CovidDashboard\n", "\n", "def handler(doc: Document) -> None:\n", " cd = CovidDashboard().panel()\n", " cd.server_doc(doc)\n", "```\n", "\n", "Next the default home controller in the `controllers.py` file should be modified to contain the following:\n", "\n", "```python\n", "def home(request):\n", " \"\"\"\n", " Controller for the app home page.\n", " \"\"\"\n", " script = server_document(request.get_full_path())\n", "\n", " context = {\n", " 'script': script,\n", " }\n", "\n", " return render(request, 'covid/home.html', context)\n", "```\n", "\n", "Finally, the handler needs to be registered with the default home `UrlMap` in `app.py`:\n", "\n", "```python\n", " UrlMap(\n", " name='home',\n", " url='covid',\n", " controller='covid.controllers.home',\n", " handler='covid.handlers.handler',\n", " handler_type='bokeh',\n", " )\n", "```\n", "\n", "With those changes the app will then be integrated into a Tethys app. Additional changes can then be made to take advantage of the Tethys framework.\n", "\n", "![COVID Tethys App Demo](TethysCovid.gif)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }