{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "
\n", "\n", "
\n", "\"Unidata\n", "
\n", "\n", "

python-awips: How to Access Data

\n", "

Unidata AMS 2021 Student Conference

\n", "\n", "
\n", "
\n", "\n", "---\n", "\n", "### Focuses\n", "\n", "* Cover the relevant methods for accessing EDEX and investigating what data is available. \n", "* This example we look at the \"grid\" data type and investigate the Global Forcast System (GFS) model. \n", "* We will talk quite a bit about the **DataAccessLayer** utility, and its [online documentation](http://unidata.github.io/python-awips/api/DataAccessLayer.html#) might be a helpful reference.\n", "\n", "### Objectives\n", "\n", "1. [Connect to EDEX](#1.-Connect-to-EDEX)\n", "1. [Get a List of Supported Data Types](#2.-Get-a-List-of-Supported-Data-Types)\n", "1. [Create a New Data Request and Set the Type](#3.-Create-a-New-Data-Request-and-Set-the-Type)\n", "1. [Get Available Locations](#4.-Get-Available-Locations)\n", "1. [Get Available Parameters](#5.-Get-Available-Parameters)\n", "1. [Get Available Levels](#6.-Get-Available-Levels)\n", "1. [Get Available Times](#7.-Get-Available-Times)\n", "1. [Get the Data!](#8.-Get-the-Data!)\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Imports\n", "\n", "Start by importing the DataAccessLayer package from python-awips:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from awips.dataaccess import DataAccessLayer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 1. Connect to EDEX\n", "\n", "Define a url for your EDEX connection, and then point python-awips at that EDEX" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Unidata's cloud EDEX instance is used in this example\n", "edex_url = \"edex-cloud.unidata.ucar.edu\"\n", "DataAccessLayer.changeEDEXHost(edex_url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 2. Get a List of Supported Data Types\n", "\n", "[***DataAccessLayer.getSupportedDatatypes()***](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getSupportedDatatypes) returns a list of supported data types offered by the EDEX server defined above. The code below shows how to populate, sort, and print out that list." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dataTypes = DataAccessLayer.getSupportedDatatypes()\n", "dataTypes.sort()\n", "list(dataTypes)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---\n", "## 3. Create a New Data Request and Set the Type\n", "\n", "Now create a new data request using [***DataAccessLayer.newDataRequest()***](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.newDataRequest), and set the data type using [***request.setDatatype()***](http://unidata.github.io/python-awips/api/IDataRequest.html#awips.dataaccess.IDataRequest.setDatatype). Below we create a few different requests with different data types to show some differences with other methods.\n", "\n", "For this example we are going to look at the \"grid\" data type, which is the model data can be found, along with some other datasets (such as MRMS)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create a request for data type grid\n", "grid_request = DataAccessLayer.newDataRequest()\n", "grid_request.setDatatype(\"grid\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---\n", "## 4. Get Available Locations\n", "\n", "Use the [***DataAccessLayer.getAvailableLocationNames(request)***](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getAvailableLocationNames) method to find out what locations are available for the given dataset. Typically these will be geographic locations or NWS sites, although in some instances it will be something else. Take a look at what's outputted for the grid_request, for example." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Grid Locations\n", "grid_locations = DataAccessLayer.getAvailableLocationNames(grid_request)\n", "grid_locations.sort()\n", "list(grid_locations)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---\n", "## 5. Get Available Parameters\n", "\n", "We're setting the \"location\" (in this case, what model we are interested in) to specify our request before we look at the available parameters.\n", "\n", "Take a look at the available parameters for the data set by using [***DataAccessLayer.getAvailableParameters(request)***](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getAvailableParameters)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Pick a model and set the location for the grid request -- we'll be using the Global Forecast System (GFS)\n", "grid_request.setLocationNames(\"GFS\")\n", "grid_params = DataAccessLayer.getAvailableParameters(grid_request)\n", "grid_params.sort()\n", "list(grid_params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---\n", "## 6. Get Available Levels\n", "\n", "\n", "Setting the parameters is just an option, you do not need to filter the data if you do not wish to. Also, although we are only setting one parameter in this example, you can set multiple parameters by using an array:\n", "```\n", "params = (\"param1\", \"param2\", \"param3\"...)\n", "request.setParameters(params)\n", "```\n", "\n", "Set a parameter, from the output above and take a look at what \"levels\" are available for the data set you're looking at using [***DataAccessLayer.getAvailableLevels(request)***](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getAvailableLevels).\n", "\n", ">Note: Not all datasets support levels. If you are trying this with another dataset and run into an exception (error), it's most likely because levels are not supported for that data type." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "# For grid data we'll use the temperature parameter (\"T\")\n", "grid_request.setParameters(\"T\")\n", "grid_levels = DataAccessLayer.getAvailableLevels(grid_request)\n", "for lvl in grid_levels:\n", " print(lvl)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* **0.0SFC** is the Surface level\n", "* **FHAG** stands for Fixed Height Above Ground (in meters)\n", "* **NTAT** stands for Nominal Top of the ATmosphere\n", "* **BL** stands for Boundary Layer, where **0.0_30.0BL** reads as *0-30 mb above ground level* \n", "* **TROP** is the Tropopause level" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# We'll set the level to surface level\n", "grid_request.setLevels(\"0.0SFC\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---\n", "## 7. Get Available Times\n", "\n", "Take a look at what time options are available for the data you're looking at using the [***DataAccessLayer.getAvailableTimes()***](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getAvailableTimes) method:\n", "\n", "* **getAvailableTimes(request, True)** will return an object of *run times* - formatted as `YYYY-MM-DD HH:MM:SS`\n", "* **getAvailableTimes(request)** will return an object of all times - formatted as `YYYY-MM-DD HH:MM:SS (F:ff)`\n", "* **getForecastRun(cycle, times)** will return a DataTime array for a single forecast cycle." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Available grid times\n", "grid_cycles = DataAccessLayer.getAvailableTimes(grid_request, True)\n", "grid_times = DataAccessLayer.getAvailableTimes(grid_request)\n", "## Using -1 in an array will access the last element of the array \n", "## (using -2 will access the second last element, and so on)\n", "grid_fcstRun = DataAccessLayer.getForecastRun(grid_cycles[-1], grid_times)\n", "\n", "## print out the time instances\n", "times = []\n", "for fcst in grid_fcstRun:\n", " print(fcst.getRefTime(), ':', fcst.getFcstTime(), 'seconds')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---\n", "## 8. Get the Data!\n", "\n", "Now that we have our `request` and DataTime `fcstRun` arrays ready, it's time to request the data array from EDEX. Depending on what kind of data we're working with, we'll either use [***DataAccessLayer.getGridData()***](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getGridData) or [***DataAccessLayer.getGeometryData()***](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getGeometryData)\n", "\n", "> Note: We have more, detailed notebooks about how analyze and visualize the data once you have what you want." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Grid Data\n", "grid_response = DataAccessLayer.getGridData(grid_request, [grid_fcstRun[-1]])\n", "for grid in grid_response:\n", " grid_data = grid.getRawData()\n", " lons, lats = grid.getLatLonCoords()\n", " print('Time :', grid.getDataTime(), \"-\", grid.getDataTime().getFcstTime(), 'seconds')\n", "\n", "## Take a look at some information in our data\n", "print('Model:', str(grid.getLocationName()))\n", "print('Parm :', str(grid.getParameter()))\n", "print('Unit :', str(grid.getUnit()))\n", "print(grid_data.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---\n", "\n", "## See also\n", "\n", "Several functions are used throughout this notebook from the DataAccessLayer class in python-awips, to see full documentation for these functions vist [**here**](http://unidata.github.io/python-awips/api/DataAccessLayer.html#).\n", "\n", "### Related Notebooks\n", "\n", "* [python-awips: Working with Surface Obs](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/visualization/python-awips-WorkingWithSurfaceObs.ipynb)\n", "* [python-awips: Working with Models](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/visualization/python-awips-WorkingWithModels.ipynb)\n", "* [python-awips: Working with Satellite Data](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/visualization/python-awips-WorkingWithSatelliteData.ipynb)\n", "* [python-awips: Working with Upper Air Obs](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/visualization/python-awips-WorkingWithUpperAirObs.ipynb)\n", "* [python-awips: Working with Maps and Topography Databases](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/visualization/python-awips-WorkingWithMapsTopoDatabases.ipynb)\n", "\n", "Top" ] } ], "metadata": { "kernelspec": { "display_name": "Python [conda env:pyaos-ams-2021]", "language": "python", "name": "conda-env-pyaos-ams-2021-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 1 }