{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Summarize Laser Optical Plankton Count data ###\n", "\n", "*Extract LOPC data from a Dorado mission, explore it, and add total LOPC count Parameters to your STOQS database.* \n", "\n", "Executing this Notebook requires a personal STOQS database. Follow the [steps to build your own development system](https://github.com/stoqs/stoqs/blob/master/README.md) — this will take a few hours and depends on a good connection to the Internet. Once your server is up log into it (after a `cd ~/Vagrants/stoqsvm`) and activate your virtual environment with the usual commands:\n", "\n", " vagrant ssh -- -X\n", " cd /vagrant/dev/stoqsgit\n", " source venv-stoqs/bin/activate\n", " \n", "Then load the `stoqs_simz_aug2013` database with the commands:\n", "\n", " cd stoqs\n", " ln -s mbari_campaigns.py campaigns.py\n", " export DATABASE_URL=postgis://stoqsadm:CHANGEME@127.0.0.1:5432/stoqs\n", " loaders/load.py --db stoqs_simz_aug2013\n", " loaders/load.py --db stoqs_simz_aug2013 --updateprovenance\n", " \n", "Loading this database takes an hour or so. Once it's finished you can interact with the data quite efficiently, as this Notebook demonstrates. Launch Jupyter Notebook with:\n", "\n", " cd contrib/notebooks\n", " ../../manage.py shell_plus --notebook\n", " \n", "navigate to this file and open it. You will then be able to execute the cells and experiment with different settings and code." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Sample LOPC Data\n", "\n", "We will look in detail at the LOPC data that is rendered in the left-hand panels of this quick look plot from Survey Dorado389_2013_225_01_225_01:\n", "![LOPC data from Dorado389_2013_225_01_225_01](http://dods.mbari.org/data/auvctd/surveys/2013/images/Dorado389_2013_225_01_225_01_lopc.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Let's write some Python code...\n", "\n", "Set `db` and `survey` variables and construct a Django query set template for getting LOPC data (where the `dataarray` field is not null) from the STOQS database. We using the [database diagram](https://raw.githubusercontent.com/stoqs/stoqs/master/doc/stoqs_model.png) for help in navigating the relationships and constructing the list of values we want to retrieve." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "db = 'stoqs_simz_aug2013'\n", "survey = 'Dorado389_2013_225_01_225_01'\n", "lopc = MeasuredParameter.objects.filter(dataarray__isnull=False).values(\n", " 'measurement__instantpoint__timevalue', 'measurement__depth',\n", " 'measurement__geom', 'parameter__domain', 'dataarray')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's look at the LOPC's Single Element Plankton count data which is loaded into STOQS as the sepCountList Parameter. We modify the the `lopc` query set template with our database name, survey name (which in STOQS is part of the Activity name), and Parameter name constraints and feed the records into a Pandas DataFrame object named `sep`:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import pandas as pd\n", "sep = pd.DataFrame.from_records(lopc.using(db).filter(\n", " measurement__instantpoint__activity__name__contains=survey,\n", " parameter__name='sepCountList'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's see how many records we got and look at the first 2:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "676\n" ] }, { "data": { "text/html": [ "
\n", " | dataarray | \n", "measurement__depth | \n", "measurement__geom | \n", "measurement__instantpoint__timevalue | \n", "parameter__domain | \n", "
---|---|---|---|---|---|
0 | \n", "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2585.0, 1261.0,... | \n", "0.042208 | \n", "[-121.88311757276813, 36.90482998210013] | \n", "2013-08-13 22:08:45 | \n", "[108.0, 123.0, 138.0, 153.0, 168.0, 183.0, 198... | \n", "
1 | \n", "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2539.0, 1112.0,... | \n", "0.571604 | \n", "[-121.88314317796421, 36.90485415120313] | \n", "2013-08-13 22:08:38 | \n", "[108.0, 123.0, 138.0, 153.0, 168.0, 183.0, 198... | \n", "