{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DRMS Basics\n",
    "\n",
    "DRMS data are organized in data series containing multiple records. The metadata for each individual record is stored in a SQL database, while the actual data are stored as separate files (e.g. FITS files) and usually do not contain any metadata.\n",
    "\n",
    "More information on DRMS series and JSOC data access is available on the JSOC website:\n",
    "\n",
    "- http://jsoc.stanford.edu/jsocwiki/JsocDataAccess\n",
    "- http://jsoc.stanford.edu/ajax/RecordSetHelp.html\n",
    "- http://jsoc.stanford.edu/jsocwiki/DataSeries\n",
    "\n",
    "An interactive webinterface for accessing the DRMS can be found at:\n",
    "\n",
    "- http://jsoc.stanford.edu/ajax/lookdata.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import drms\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import dates\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a [`drms.Client`](http://drms.readthedocs.io/en/stable/generated/drms.Client.html) instance to access the JSOC DRMS server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "client = drms.Client()\n",
    "print(client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List DRMS series\n",
    "\n",
    "A list of available series can be obtained using the [`drms.Client.series()`](http://drms.readthedocs.io/en/stable/generated/drms.Client.series.html) method. By default this method returns all available series (more than 200 publicly available from JSOC). A subset can be selected by specifying a regular expression.\n",
    "\n",
    "Note that series names are case insensitive and that you might need to escape characters like `'.'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "client.series('45s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "client.series('aia\\.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "client.series('hmi\\.v_')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Get detailed series information\n",
    "\n",
    "Information about a series can be retrieved using [`drms.Client.info()`](http://drms.readthedocs.io/en/stable/generated/drms.Client.info.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info_dopp = client.info('hmi.v_45s')\n",
    "print(info_dopp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info_dopp.primekeys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "info_dopp.keywords.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "info_dopp.keywords.loc[['T_REC', 'CRLT_OBS', 'OBS_VR']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info_dopp.segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info_sharp = client.info('hmi.sharp_720s')\n",
    "print(info_sharp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info_sharp.primekeys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info_sharp.segments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There also exist shortcuts to obtain lists of keywords ([`drms.Client.keys()`](http://drms.readthedocs.io/en/stable/generated/drms.Client.keys.html)) and primekeys ([`drms.Client.pkeys()`](http://drms.readthedocs.io/en/stable/generated/drms.Client.pkeys.html)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(client.pkeys('hmi.v_45s'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(client.keys('hmi.v_45s'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying metadata\n",
    "\n",
    "The central command for querying the DRMS is given by the [`drms.Client.query()`](http://drms.readthedocs.io/en/stable/generated/drms.Client.query.html) function. This function can be used to obtain metadata, as well as the location of locally stored data segments. Here we are look at the metadata.\n",
    "\n",
    "A DRMS recordset can be specified using the series name, followed by one or more pairs of square brackets, which correspond to the primekeys of the specific series. Each bracket group can be used to select a range of records using a primekey expression. A few examples for primekeys that represent a timestamp are:\n",
    "\n",
    "- `\"[2016.01.01_12:00_TAI]\"` -> One record at Noon Jan 01, 2016\n",
    "- `\"[2016.01.01_12:00_TAI/1h]\"` -> All records for one hour starting at Noon on Jan 01, 2016\n",
    "- `\"[2016.01.01_TAI/7d@1d]\"` -> One record per day, for a whole week, starting at Midnight on Jan 01, 2016\n",
    "- `\"[2016.01.01_TAI-2017.01.01_TAI@12h]\"` -> One record every 12 hours, for the whole year of 2016\n",
    "\n",
    "More details and more examples can be found on the [JSOC RecordSet Help](http://jsoc.stanford.edu/ajax/RecordSetHelp.html) page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "keys = client.query('hmi.m_720s[2017.01.01_TAI/1d@6h]', key=['T_REC', 'OBS_VR', 'DSUN_OBS'])\n",
    "print(keys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "keys = client.query('hmi.m_720s[2017.01.01_TAI/1d]', key=['T_REC', 'OBS_VR', 'DSUN_OBS'])\n",
    "t_rec = drms.to_datetime(keys['T_REC'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(2, 1, sharex=True, figsize=(8, 6))\n",
    "\n",
    "ax[0].plot(t_rec, keys['DSUN_OBS'])\n",
    "ax[0].set_ylabel('Distance [m]')\n",
    "\n",
    "ax[1].plot(t_rec, keys['OBS_VR'])\n",
    "ax[1].set_ylabel('Radial velocity [m/s]')\n",
    "\n",
    "ax[1].set_xlabel('Time')\n",
    "ax[1].xaxis.set_major_formatter(dates.DateFormatter('%H:%M'))\n",
    "fig.tight_layout()\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "keys = client.query('hmi.m_720s[2015.01.01_TAI/365d@3h]', key=['T_REC', 'DSUN_OBS'])\n",
    "t_rec = drms.to_datetime(keys['T_REC'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 5))\n",
    "ax.plot(t_rec, keys['DSUN_OBS'])\n",
    "ax.set_xlabel('Time')\n",
    "ax.set_ylabel('Distance [m]')\n",
    "fig.tight_layout()\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More interesting metadata\n",
    "\n",
    "This is a simplified version of [`plot_polarfield.py`](https://github.com/kbg/drms/blob/master/examples/plot_polarfield.py) which is included in the `drms` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "series = 'hmi.meanpf_720s'\n",
    "tsel = '2010.05.01_TAI-2017.01.01_TAI@12h'\n",
    "\n",
    "qstr = '{}[{}]'.format(series, tsel)\n",
    "print('Querying \"{}\"...'.format(qstr))\n",
    "\n",
    "keys = client.query(qstr, key=['T_REC', 'CAPN2', 'CAPS2'])\n",
    "print('-> {} lines retrieved.'.format(len(keys)))\n",
    "\n",
    "t_rec = drms.to_datetime(keys['T_REC'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info_meanpf = client.info(series)\n",
    "info_meanpf.keywords.loc[['CAPN2', 'CAPS2']].note"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 5))\n",
    "ax.plot(t_rec, keys['CAPN2'], label='North')\n",
    "ax.plot(t_rec, keys['CAPS2'], label='South')\n",
    "ax.set_ylim(-6.2, 6.2)\n",
    "ax.set_xlabel('Year')\n",
    "ax.set_ylabel('Mean radial field strength [G]')\n",
    "ax.legend()\n",
    "fig.tight_layout()\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}