{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to FAST data\n",
    "\n",
    "> Abstract: Some datasets are available more rapidly than others (for space weather monitoring). These are delivered through a special \"FAST\" processing chain, as opposed to the normal operational \"OPER\" chain. They are available under different collection names containing the string \"FAST\". Here we compare the availability and quality of FAST data in contrast to OPER data.\n",
    "\n",
    "NB: Data is currently only available on the DISC machine for select users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SERVER_URL = 'https://vires.services/ows'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext watermark\n",
    "%watermark -i -v -p viresclient,pandas,xarray,matplotlib,tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from viresclient import SwarmRequest\n",
    "import datetime as dt\n",
    "from tqdm import tqdm\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "from operator import or_\n",
    "from functools import reduce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Which data are available FAST?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request = SwarmRequest(SERVER_URL)\n",
    "\n",
    "all_collections = request.available_collections(details=False)\n",
    "# Identify collections with FAST in the name\n",
    "collections = {group: [c for c in colls if \"FAST\" in c] for (group, colls) in all_collections.items()}\n",
    "# Filter the empty groups\n",
    "collections = {group: colls for group, colls in collections.items() if len(colls) > 0}\n",
    "collections\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use `'SW_FAST_MAGA_LR_1B'` as an example and compare to `'SW_OPER_MAGA_LR_1B'`.\n",
    "\n",
    "We can query the availability times of the underlying products:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fast_availability = request.available_times(\"SW_FAST_MAGA_LR_1B\")\n",
    "fast_availability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note: VirES only keeps a rolling store of the most recent month of FAST data.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oper_availability = request.available_times(\"SW_OPER_MAGA_LR_1B\")\n",
    "oper_availability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Latest OPER: {oper_availability['endtime'].iloc[-1]}\")\n",
    "print(f\"Latest FAST: {fast_availability['endtime'].iloc[-1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, `OPER` data is delivered in full-day chunks with a few days delay. `FAST` data are typically delivered in several-hour chunks and available as soon as possible, subject to downlink opportunities from the satellite passes over the ground stations and subsequent processing time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing FAST data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`FAST` data are identical in format to `OPER` data so can be accessed and used in the same way through VirES.\n",
    "\n",
    "In the following example, we access the latest available day of `OPER` data and compare to the available `FAST` data, for each of Swarm A, B, C."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_root = \"SW_{}_MAG{}_LR_1B\"\n",
    "\n",
    "def find_latest_oper(spacecraft=\"A\"):\n",
    "    \"\"\"Identify the latest availability time for OPER data\"\"\"\n",
    "    collection = collection_root.format(\"OPER\", spacecraft)\n",
    "    request = SwarmRequest(SERVER_URL)\n",
    "    times = request.available_times(collection)\n",
    "    return times['endtime'].iloc[-1].to_pydatetime()\n",
    "\n",
    "def fetch_data(spacecraft=\"A\", type=\"OPER\", start=None, end=None):\n",
    "    \"\"\"Fetch either OPER or FAST data\"\"\"\n",
    "    collection = collection_root.format(type, spacecraft)\n",
    "    request = SwarmRequest(SERVER_URL)\n",
    "    request.set_collection(collection)\n",
    "    request.set_products(\n",
    "        measurements=[\"B_NEC\", \"Flags_B\"],\n",
    "        models=[\"CHAOS\"],\n",
    "    )\n",
    "    data = request.get_between(start, end, asynchronous=False, show_progress=False)\n",
    "    ds = data.as_xarray()\n",
    "    ds[\"B_NEC_res_CHAOS\"] = ds[\"B_NEC\"] - ds[\"B_NEC_CHAOS\"]\n",
    "    return ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the latest availability times of OPER data per-satellite\n",
    "oper_end_times = {sc: find_latest_oper(sc) for sc in \"ABC\"}\n",
    "oper_end_times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_fast = {}\n",
    "data_oper = {}\n",
    "for sc in tqdm(\"ABC\"):\n",
    "    # Fetch the latest available day of OPER data\n",
    "    oper_end = oper_end_times[sc] + dt.timedelta(seconds=1)\n",
    "    oper_start = oper_end - dt.timedelta(days=1)\n",
    "    data_oper[sc] = fetch_data(sc, \"OPER\", oper_start, oper_end)\n",
    "    # Fetch all FAST data from then until now\n",
    "    data_fast[sc] = fetch_data(sc, \"FAST\", oper_end, dt.datetime.now())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def flag_filter(ds):\n",
    "    \"\"\"A simplistic filter for close-to-nominal data\"\"\"\n",
    "    # Filtering by Flags_B\n",
    "    # For flag meanings, see https://swarmhandbook.earth.esa.int/catalogue/SW_MAGx_LR_1B\n",
    "    # This includes Charlie data, where the ASM was lost\n",
    "    nominal = 0b0000\n",
    "    asm_off = 0b0001\n",
    "    vfm_asm_discrepency = 0b1000\n",
    "    bitmask_filters = (nominal, asm_off, vfm_asm_discrepency)\n",
    "    flags = ds[\"Flags_B\"]\n",
    "    flags_masked = reduce(or_, [flags & x for x in bitmask_filters])\n",
    "    return ds.where(flags == flags_masked)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identify the minimal and maximal times in the datasets\n",
    "tmin = min(data_oper[sc][\"Timestamp\"].data[0] for sc in \"ABC\")\n",
    "tmax = max(data_fast[sc][\"Timestamp\"].data[-1] for sc in \"ABC\")\n",
    "\n",
    "fig, axes = plt.subplots(nrows=3, figsize=(15, 10), sharex=True)\n",
    "for ax, sc in zip(axes, \"ABC\"):\n",
    "    # Remove bad data before plotting\n",
    "    ds_oper_nominal = flag_filter(data_oper[sc])\n",
    "    ds_fast_nominal = flag_filter(data_fast[sc])\n",
    "    ax.set_prop_cycle(\"color\", [\"tab:blue\", \"tab:orange\", \"tab:green\"])\n",
    "    ds_oper_nominal[\"B_NEC_res_CHAOS\"].plot.line(x=\"Timestamp\", ax=ax)\n",
    "    ds_fast_nominal[\"B_NEC_res_CHAOS\"].plot.line(x=\"Timestamp\", ax=ax)\n",
    "    ax.set_xlim(tmin, tmax)\n",
    "    # Add shading behind OPER data section\n",
    "    oper_start = data_oper[sc][\"Timestamp\"].data[0]\n",
    "    oper_end = data_oper[sc][\"Timestamp\"].data[-1]\n",
    "    ymin, ymax = ax.get_ylim()\n",
    "    ax.fill_betweenx((ymin, ymax), oper_start, oper_end, alpha=0.3, color=\"grey\")\n",
    "    # Add and fix some labelling\n",
    "    ax.text(oper_end, ymax, \"⬅️ OPER    FAST ➡️\", ha=\"center\", va=\"top\")\n",
    "    ax.set_xlim(oper_start, ax.get_xlim()[-1])\n",
    "    ax.set_xlabel(\"\")\n",
    "    ax.set_ylabel(f\"Swarm {sc}\\n{ax.get_ylabel()}\")\n",
    "\n",
    "ax.xaxis.set_major_formatter(mpl.dates.DateFormatter(\"%Y-%m-%d\\n%H:%M\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  },
  "vscode": {
   "interpreter": {
    "hash": "6a693262f012b0764ce1a509731aa7af644c1d221d30b43c6fed6082691c75d0"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}