{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Swarm access through VirES\n",
    "\n",
    "> Abstract: VirES is a server/client architecture to help access time series of Swarm data and models. Access is enabled through a token generated on the website, and a Python package, viresclient, which provides the connection with the Python ecosystem (e.g. xarray)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display important package versions used\n",
    "%load_ext watermark\n",
    "%watermark -i -v -p viresclient,pandas,xarray,matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**VirES** (Virtual environments for Earth Scientists) is a platform for data & model access, analysis, and visualisation for ESA's magnetic mission, **Swarm**\n",
    "\n",
    "This tutorial introduces the Python interface to VirES, **`viresclient`**. We demonstrate usage of the primary Swarm magnetic dataset (`SW_OPER_MAGA_LR_1B`) and geomagnetic field models produced as part of the Swarm mission. Some knowledge of `pandas` and `matplotlib` is assumed.\n",
    "\n",
    "Run this on the **[VRE](https://vre.vires.services/)** (Virtual Research Environment), where viresclient is already installed, or [check the instructions](https://viresclient.readthedocs.io/en/latest/installation.html) to set it up on your own Python environment.\n",
    "\n",
    "For more information see:\n",
    "- https://vires.services/ (Web interface)\n",
    "- https://viresclient.readthedocs.io (Python interface)\n",
    "- https://notebooks.vires.services (Guide to Virtual Research Environment / JupyterLab)\n",
    "- https://earth.esa.int/eogateway/missions/swarm (Swarm mission)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "In order to authenticate access to the VirES server, `viresclient` requires an access token - this ties communications between the server and the client to your account. **If you are using the VRE, this is handled automatically so you can skip this step.**\n",
    "\n",
    "If you are running `viresclient` on a different machine, you will need to follow these instructions:\n",
    "\n",
    "1. Create a user account at https://vires.services if you haven't already done so\n",
    "2. Install viresclient in your Python environment - see https://viresclient.readthedocs.io/en/latest/installation.html\n",
    "3. Create a new code cell here and execute the following:\n",
    "\n",
    "   ```python\n",
    "   from viresclient import set_token\n",
    "   set_token(\"https://vires.services/ows\", set_default=True)\n",
    "   ```\n",
    "\n",
    "\n",
    "4. You will now be directed to the [VirES token management page](https://vires.services/accounts/tokens/), and prompted to generate a new token and enter it here\n",
    "\n",
    "Your access token should now have been saved to your environment and you won't need to provide it again. The token and its associated access URL are stored in a file: `~/.viresclient.ini` (this file can also be edited directly). You may generate and set a new token, or revoke old tokens, at any point. These are similar to passwords, so **should be kept secret** - if you accidentally leak a token, you can revoke it at the token management page and generate a new one. It is also possible to set access tokens via CLI. For more information, see https://viresclient.readthedocs.io/en/latest/config_details.html\n",
    "\n",
    "To remove the configuration (assuming you left it in its default location), you can use the CLI command: `viresclient clear_credentials`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetching some data\n",
    "\n",
    "Import the `SwarmRequest` object which provides the VirES interface, and `datetime` which gives convenient time objects which can be used by `SwarmRequest.get_between()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from viresclient import SwarmRequest\n",
    "import datetime as dt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code will fetch one day (i.e. around 15 orbits) of the scalar (`F`) measurements from Swarm Alpha. \n",
    "\n",
    "Several keyword arguments can be provided to `.set_products()` to specify the type of data you want. The `measurements` available depend on the collection chosen in `.set_collection()`. The same set of `auxiliaries` are available for all collections - here we also choose to fetch the `MLT` - magnetic local time.\n",
    "\n",
    "`sampling_step=\"PT10S\"` downsamples the data to 10 seconds, from the `MAGx_LR` default of 1 second. If no `sampling_step` is provided, the data will be accessed in its original form (i.e. here, 1-second sampling). These strings to choose the `sampling_step` should be provided as [ISO 8601 durations](https://en.wikipedia.org/wiki/ISO_8601#Durations) (e.g. `\"PT1M\"` for 1-minute sampling).\n",
    "\n",
    "`start_time` and `end_time` in `.get_between()` together provide the time window you want to fetch data for - executing this line causes the request to be processed on the server and the data returned to you. NB: this returns data *up to but not including* `end_time`. Alternatively we can provide the start and end times as ISO_8601 strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up connection with server\n",
    "request = SwarmRequest()\n",
    "# Set collection to use\n",
    "request.set_collection(\"SW_OPER_MAGA_LR_1B\")\n",
    "# Set mix of products to fetch:\n",
    "#   measurements (variables from the given collection)\n",
    "#   models (magnetic model predictions at spacecraft sampling points)\n",
    "#   auxiliaries (variables available with any collection)\n",
    "# Reference: https://viresclient.readthedocs.io/en/latest/available_parameters.html\n",
    "# Also set additional configuration such as:\n",
    "#   sampling_step\n",
    "request.set_products(\n",
    "    measurements=[\"F\"],\n",
    "    sampling_step=\"PT10S\",\n",
    "    auxiliaries=[\"MLT\"]\n",
    ")\n",
    "# Fetch data from a given time interval\n",
    "data = request.get_between(\n",
    "    start_time=dt.datetime(2016,1,1),\n",
    "    end_time=dt.datetime(2016,1,2)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is now contained within the object which we called `data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is temporarily stored on disk and not yet loaded into memory - the `ReturnedData` object is actually a wrapper around a temporary CDF file which could be written to disk directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data.to_file(\"test_file.cdf\", overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... but it is possible to directly transfer it to a `pandas.DataFrame` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = data.as_dataframe()\n",
    "print(type(df))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... or a `xarray.Dataset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = data.as_xarray()\n",
    "print(type(ds))\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try plotting some things to visualise the data. The following shows the variation in field strength measured by the satellite as it passes between high and low latitudes, varying from one orbit to the next as it samples a different longitude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot(y=\"F\")\n",
    "df.plot(y=\"F\", x=\"Latitude\")\n",
    "df.plot(y=\"Latitude\", x=\"Longitude\")\n",
    "df.plot(y=\"Latitude\", x=\"Longitude\", c=\"F\", kind=\"scatter\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetching model evaluations at the same time\n",
    "\n",
    "Various (spherical harmonic) [models of the geomagnetic field](https://magneticearth.org/pages/models.html) are produced as Swarm mission products and these are available through VirES. They are evaluated on demand at the same points and times as the data sample points. Here we ask for the `MCO_SHA_2D` model, a dedicated core field model produced from Swarm data. By supplying `residuals=True` we will get the data-model residuals, named in the dataframe as `F_res_MCO_SHA_2D`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request = SwarmRequest()\n",
    "request.set_collection(\"SW_OPER_MAGA_LR_1B\")\n",
    "request.set_products(\n",
    "    measurements=[\"F\"],\n",
    "    models=[\"MCO_SHA_2D\"],\n",
    "    residuals=True,\n",
    "    sampling_step=\"PT10S\"\n",
    ")\n",
    "\n",
    "data = request.get_between(\n",
    "    start_time=dt.datetime(2016,1,1),\n",
    "    end_time=dt.datetime(2016,1,2)\n",
    ")\n",
    "\n",
    "df = data.as_dataframe()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The core field has been removed from the data so the amplitudes are much smaller. Can you interpret the new signals in terms of external fields, i.e. from the ionosphere and magnetosphere?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot(y=\"F_res_MCO_SHA_2D\")\n",
    "df.plot(y=\"F_res_MCO_SHA_2D\", x=\"Latitude\")\n",
    "df.plot(y=\"Latitude\", x=\"Longitude\", c=\"F_res_MCO_SHA_2D\", kind=\"scatter\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More complex model handling\n",
    "\n",
    "We can also remove a magnetospheric field model at the same time, by specifying a new model (which we call `MCO_MMA` here, but can be named whatever you like) which is the sum of core and magnetospheric models.\n",
    "\n",
    "> See [Swarm_notebooks/notebooks/02b__viresclient-Available-Data](https://notebooks.vires.services/notebooks/02b__viresclient-available-data) for more examples of this - it is also possible to specify the spherical harmonic degrees (min/max) to use, and to provide your own `.shc` model.\n",
    "\n",
    "The remaining signal is now primarily due to the ionosphere.\n",
    "\n",
    "Note that here we are instead using the CI (comprehensive inversion) core and magnetosphere models (indicated by `2C` in the product names). The magnetospheric model (`MMA`) is split into two parts - \"primary\": external to the Earth, and \"secondary\": the induced counterpart internal to the Earth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request = SwarmRequest()\n",
    "request.set_collection(\"SW_OPER_MAGA_LR_1B\")\n",
    "request.set_products(\n",
    "    measurements=[\"F\"],\n",
    "    models=[\"MCO_MMA = 'MCO_SHA_2C' + 'MMA_SHA_2C-Primary' + 'MMA_SHA_2C-Secondary'\"],\n",
    "    residuals=True,\n",
    "    sampling_step=\"PT10S\",\n",
    "    auxiliaries=[\"MLT\"]\n",
    ")\n",
    "\n",
    "data = request.get_between(\n",
    "    start_time=dt.datetime(2016,1,1),\n",
    "    end_time=dt.datetime(2016,1,2)\n",
    ")\n",
    "\n",
    "df = data.as_dataframe()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot(y=\"F_res_MCO_MMA\")\n",
    "df.plot(y=\"F_res_MCO_MMA\", x=\"Latitude\")\n",
    "df.plot(y=\"Latitude\", x=\"Longitude\", c=\"F_res_MCO_MMA\", kind=\"scatter\");"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}