{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Using the IMOS User Code Library with Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
This document intends to present how to load IMOS NetCDF data into a Python environment, and offers some suggestions about how to use the data once loaded. All the examples below will use the netCDF4 Python module (http://code.google.com/p/netcdf4-python/)\n", "\n", "The examples provided in this document only represent a tiny bit of the content of most of the NetCDF files. There are usually many more variables available in a NetCDF file, and therefore many other ways to display data.
\n", "***\n", "\n", "**Content:**\n", "- [Installation of the IMOS User Code Library (Python) and required packages](#install-imos)\n", "- [Finding an IMOS NetCDF file](#netcdf-imos)\n", "- [General Features of the netCDF4 module](#netcdf4)\n", "- [Output structure](#output)\n", "- [Discover Metadata](#metadata)\n", "- [Discover Variables](#variables)\n", "- [Export the salinity variable](#salinity)\n", "- [Plotting the Salinity-Temperature relationship](#plotting)\n", "\n", "***\n", "\n", "A list of interesting IMOS based Python library functions is available from their [Github website](https://github.com/aodn/imos-user-code-library/wiki/Using-the-IMOS-User-Code-Library-with-Python#12-finding-an-imos-netcdf-file)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Installation of the IMOS User Code Library (Python) and required packages\n", "\n", "The examples that will be used rely on the following Python packages, which need to be installed:\n", "\n", "+ **numpy** – standard package for scientific computing in Python, provides versatile numerical array objects (http://www.numpy.org/).\n", "+ **matplotlib** – for plotting (http://matplotlib.org/).\n", "+ **netCDF4** – for accessing netCDF files (http://code.google.com/p/netcdf4-python/, follow the Documentation link for installation instructions).\n", "\n", "The examples that will be use in this course have been tested in Python 2.7.3, with numpy 1.6.1, matplotlib 1.1.1, and version 1.0.2 of netCDF4. Later versions of these packages will usually work. Earlier versions may work also." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Finding an IMOS NetCDF File\n", "\n", "In order to find a dataset you are interested in, please refer to the portal help: http://help.aodn.org.au/help/?q=node/6. This is a how-to guide that can help users find an IMOS NetCDF file. When downloading your chosen dataset from the portal, choose one of the download options “List of URLs”, or “All source NETCDF files” to obtain netCDF files.\n", "\n", "For users who are already familiar with IMOS facilities and datasets, IMOS NetCDF files are also directly accessible via an OPeNDAP catalog at : http://thredds.aodn.org.au/thredds/catalog/IMOS/catalog.html\n", "\n", "Most of the examples in the following sections use the ‘Data URL’ of a dataset. If you have downloaded your dataset from the portal, the data URL is the file path to the file on your local machine. If you are using the THREDDS catelog, the file does not have to be downloaded to your local machine first – the OPeNDAP data URL can be parsed into Python. The OPeNDAP data URL is found on the ‘OPeNDAP Dataset Access Form’ page (see http://help.aodn.org.au/help/?q=node/11), inside the box labelled ‘Data URL’ just above the ‘Global Attributes’ field.\n", "\n", "Note: the list of URL’s generated by the IMOS portal when choosing that download option can be converted to a list of OPeNDAP data URL’s by replacing string http://data.aodn.org.au/IMOS/opendap with http://thredds.aodn.org.au/thredds/dodsC/IMOS." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## General Features of the netCDF4 module\n", "\n", "The first step consists of opening a NetCDF file, whether this file is available locally or remotely on an OPeNDAP server.\n", "\n", "Type in your Python command window (or script):\n", "\n", "```python\n", "from netCDF4 import Dataset\n", "\n", "aatams_URL = 'http://thredds.aodn.org.au/thredds/dodsC/IMOS/eMII/demos/AATAMS/marine_mammal_ctd-tag/2009_2011_ct64_Casey_Macquarie/ct64-M746-09/IMOS_AATAMS-SATTAG_TSP_20100205T043000Z_ct64-M746-09_END-20101029T071000Z_FV00.nc'\n", "aatams_DATA = Dataset(aatams_URL)\n", "```\n", "This creates a netCDF Dataset object, through which you can access all the contents of the file.\n", "\n", "Let's do it below - as this is a Python commands we will be using a `code` cell this time and not a `markdown` (text) one." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from netCDF4 import Dataset\n", "\n", "aatams_URL = 'http://thredds.aodn.org.au/thredds/dodsC/IMOS/eMII/demos/AATAMS/marine_mammal_ctd-tag/2009_2011_ct64_Casey_Macquarie/ct64-M746-09/IMOS_AATAMS-SATTAG_TSP_20100205T043000Z_ct64-M746-09_END-20101029T071000Z_FV00.nc'\n", "aatams_DATA = Dataset(aatams_URL)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Output structure\n", "\n", "Please refer to the netCDF4 module documentation for a complete description of the Dataset object:: 'http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4.Dataset-class.html' (or type help(Dataset) at the Python prompt)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "help(aatams_DATA)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Discover Metadata\n", "\n", "In order to see all the global attributes and some other information about the file, type in your command window:\n", "\n", "```python\n", "print aatams_DATA\n", "```\n", "\n", "The output will look something like this:\n", "\n", "```python\n", " root group (NETCDF3_64BIT file format):\n", " project: Integrated Marine Observing System (IMOS)\n", " conventions: IMOS-1.2\n", " date_created: 2012-09-13T07:27:03Z\n", " title: Temperature, Salinity and Depth profiles in near real time\n", " institution: AATAMS\n", " site: CTD Satellite Relay Data Logger\n", " abstract: CTD Satellite Relay Data Loggers are used to explore how marine mammal behaviour relates to their oceanic environment. Loggers developped at the University of St Andrews Sea Mammal Research Unit transmit data in near real time via the Argo satellite system\n", " source: SMRU CTD Satellite relay Data Logger on marine mammals\n", "…\n", " dimensions: obs, profiles\n", " variables: TIME, LATITUDE, LONGITUDE, TEMP, PRES, PSAL, parentIndex, TIME_quality_control, LATITUDE_quality_control, LONGITUDE_quality_control, TEMP_quality_control, PRES_quality_control, PSAL_quality_control\n", " groups:\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(aatams_DATA)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Global attributes in the netCDF file become attributes of the Dataset object. A list of global attribute names is returned by the ncattrs() method of the object. The __dict__ attribute of the object is a dictionary of all netCDF attribute names and values.\n", "\n", "```python\n", "# store the dataset's title in a local variable\n", "title_str = aatams_DATA.title\n", "\n", "# list all global attribute names\n", "aatams_DATA.ncattrs()\n", "\n", "# store the complete set of attributes in a dictionary (OrderedDict) object (similar to a standard Python dict, but\n", "# maintains the order in which items are entered)\n", "globalAttr = aatams_DATA.__dict__\n", "\n", "# now you can also do (same effect as first command above)\n", "title_str = globalAttr['title']\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# store the dataset's title in a local variable\n", "title_str = aatams_DATA.title\n", "\n", "# list all global attribute names\n", "aatams_DATA.ncattrs()\n", "\n", "# store the complete set of attributes in a dictionary (OrderedDict) object (similar to a standard Python dict, but\n", "# maintains the order in which items are entered)\n", "globalAttr = aatams_DATA.__dict__\n", "\n", "# now you can also do (same effect as first command above)\n", "title_str = globalAttr['title']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(title_str)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "globalAttr" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Discover Variables\n", "\n", "To list all the variables available in the NetCDF file, type:\n", "\n", "```python\n", "aatams_DATA.variables.keys()\n", "```\n", "\n", "Output:\n", "\n", "```python\n", "[u'TIME',\n", " u'LATITUDE',\n", " u'LONGITUDE',\n", " u'TEMP',\n", " u'PRES',\n", " u'PSAL',\n", " u'parentIndex',\n", " u'TIME_quality_control',\n", " u'LATITUDE_quality_control',\n", " u'LONGITUDE_quality_control',\n", " u'TEMP_quality_control',\n", " u'PRES_quality_control',\n", " u'PSAL_quality_control']\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "aatams_DATA.variables.keys()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(The 'u' means each variable name is represented by a Unicode string.)\n", "\n", "Each variable is accessed via a `Variable` object, in a similar way to the `Dataset` object. To access the `Temperature` variable :\n", "\n", "```python\n", "# netCDF4 Variable object\n", "TEMP = aatams_DATA.variables['TEMP']\n", "\n", "# now you can print the variable's attributes and other info\n", "print TEMP\n", "\n", "# access variable attributes, e.g. its standard_name\n", "TEMP.standard_name\n", "\n", "# extract the data values (as a numpy array)\n", "TEMP[:]\n", "\n", "# the variable's dimensions (as a tuple)\n", "TEMP.dimensions\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# netCDF4 Variable object\n", "TEMP = aatams_DATA.variables['TEMP']\n", "\n", "# now you can print the variable's attributes and other info\n", "print(TEMP)\n", "\n", "# access variable attributes, e.g. its standard_name\n", "TEMP.standard_name\n", "\n", "# extract the data values (as a numpy array)\n", "TEMP[:]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(TEMP[:])\n", "print(min(TEMP),max(TEMP))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export the salinity variable\n", "\n", "Following the same approach as the one proposed for the temperature variable extract the salinity one.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# netCDF4 Variable object\n", "PSAL = aatams_DATA.variables['PSAL']\n", "\n", "# now you can print the variable's attributes and other info\n", "print(PSAL)\n", "\n", "# access variable attributes, e.g. its standard_name\n", "PSAL.standard_name\n", "\n", "# extract the data values (as a numpy array)\n", "PSAL[:]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(PSAL[:])\n", "print(min(PSAL),max(PSAL))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plotting the Salinity-Temperature relationship\n", "\n", "We can now work with and plot these variables and get some information/relationship regarding their temporal and/or spatial evalution for example.\n", "\n", "To do so we need to import some useful Python libraries..." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from statsmodels.nonparametric.smoothers_lowess import lowess\n", "\n", "%matplotlib inline\n", "%config InlineBackend.figure_format = 'svg'\n", "plt.rcParams['mathtext.fontset'] = 'cm'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let us define the temperature and salinity variables more simply:\n", "- `PSAL[:]` to `sal` for salinity\n", "- `TEMP[:]` to `temp` for temperature" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sal = np.asarray(PSAL[:])\n", "temp = np.asarray(TEMP[:])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We then use a smoothing function `lowess` [Locally Weighted Scatterplot Smoothing](http://www.statsmodels.org/dev/generated/statsmodels.nonparametric.smoothers_lowess.lowess.html) so get a general trend of salinity evolution in relation to temperature from the existing scattered dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ys = lowess(sal, temp, it=5, frac=0.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we plot the result with **matplotlib**:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.figure(figsize=(10,7))\n", "\n", "plt.scatter(temp, sal, s=2, marker='o', facecolor='r', lw = 1,label='T-S dataset')\n", "plt.xlabel('Temperature in Celcius')\n", "plt.ylabel('Salinity units of parts per thousand (1.e-3)')\n", "plt.title(title_str)\n", "\n", "plt.plot(ys[:,0],ys[:,1],'k',linewidth=2,label='smoothing')\n", "\n", "plt.legend(loc=0, fontsize=10)\n", "plt.show()\n", "plt.close()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.15" } }, "nbformat": 4, "nbformat_minor": 2 }