{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# QC Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objective: \n",
    "Show different ways to configure a quality control (QC) procedure - explicit inline or calling a pre-set configuration.\n",
    "\n",
    "\n",
    "For CoTeDe, the most important component is the human operator, hence it should be easy to control which tests to apply and the specific parameters of each test. CoTeDe is based on the principle of a single engine for multiple applications by using a dictionary to describe the QC procedure to be used, since 2011."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CoTeDe version: 0.21.0\n"
     ]
    }
   ],
   "source": [
    "# A different version of CoTeDe might give slightly different outputs.\n",
    "# Please let me know if you see something that I should update.\n",
    "\n",
    "import cotede\n",
    "print(\"CoTeDe version: {}\".format(cotede.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### load_cfg(), just for demonstration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we will import the load_cfg() function to illustrate different procedures. This is typically not necessary since ProfileQC does that for us. The cfgname argument for load_cfg is the same for ProfileQC, thus when we call\n",
    "\n",
    "ProfileQC(dataset, cfgname='argo')\n",
    "\n",
    "the procedure applied to dataset is the same shown by\n",
    "\n",
    "load_cfg(cfgname='argo')\n",
    "\n",
    "We will take advantage on that and simplify this notebook by inspecting only the configuration without actually applying it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cotede.utils import load_cfg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Built-in tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to configure a QC procedure is by using one of the built-in tests, for example the GTSPP procedure for realtime data, here named 'gtspp_realtime'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['revision', 'common', 'variables']\n"
     ]
    }
   ],
   "source": [
    "cfg = load_cfg('gtspp_realtime')\n",
    "\n",
    "print(list(cfg.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output cfg is a dictionary type of object, more specifically it is an ordered dictionary. The configuration has:\n",
    "\n",
    "* A revision to help to determine how to handle this configuration.\n",
    "\n",
    "* A common item with the common tests for the whole dataset, i.e. the tests that are valid for all variables. For instance, a valid date and time is the same if we are evaluating temperature, salinity, or chlorophyll fluorescence.\n",
    "\n",
    "* A variables, with a list of the variables to evaluate.\n",
    "\n",
    "Let's check each item:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.21'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cfg['revision']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['valid_datetime', 'valid_position', 'location_at_sea']\n"
     ]
    }
   ],
   "source": [
    "print(list(cfg['common'].keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, for GTSSP realtime assessement, all variables must be associated with a valid time and a valid location that is at sea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['sea_water_temperature', 'sea_water_salinity']\n"
     ]
    }
   ],
   "source": [
    "print(list(cfg['variables'].keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GTSPP evaluates temperature and salinity. Here we use CF standard names, so temperature is sea_water_temperature. But which tests are applied on temperature measurements?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['global_range', 'gradient', 'spike', 'profile_envelop']\n"
     ]
    }
   ],
   "source": [
    "print(list(cfg['variables']['sea_water_temperature'].keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the spike test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('threshold', 2.0)])\n"
     ]
    }
   ],
   "source": [
    "print(cfg['variables']['sea_water_temperature']['spike'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one single item, the threshold, here defined as 2, so that any measured temperature with a spike greater than this threshold will fail on this spike test.\n",
    "\n",
    "Let's check the global range test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['minval', 'maxval']\n"
     ]
    }
   ],
   "source": [
    "print(list(cfg['variables']['sea_water_temperature']['global_range']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here there are two limit values, the minimum acceptable value and the maximum one. Anything beyond these limits will fail this test.\n",
    "\n",
    "Check CoTeDe's manual to see what each test does and the possible parameters for each one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explicit inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A QC procedure can also be explicitly defined with a dictionary. For instance, let's consider that we want to evaluate the temperature of a dataset with a single test, the spike test, using a threshold equal to one,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('revision', '0.21'), ('variables', OrderedDict([('sea_water_temperature', {'spike': {'threshold': 1}})]))])\n"
     ]
    }
   ],
   "source": [
    "my_config = {\"sea_water_temperature\":\n",
    "               {\"spike\": {\n",
    "                   \"threshold\": 1\n",
    "                   }\n",
    "               }\n",
    "           }\n",
    "\n",
    "cfg = load_cfg(my_config)\n",
    "print(cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that load_cfg took care for us to format it with the 0.21 standard, adding the revision and variables. If a revision is not defined, it is assumed a pre-0.21."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compound procedure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many of the recommended QC procedures share several tests in common. One way to simplify a QC procedure definition is by using inheritance to define a QC procedure to be used as a template. For example, let's create a new QC procedure that is based on GTSPP realtime and add a new test to that, the World Ocean Atlas Climatology comparison for temperature, with a threshold of 3 standard deviations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "odict_keys(['revision', 'common', 'variables', 'inherit'])\n"
     ]
    }
   ],
   "source": [
    "my_config = {\"inherit\": \"gtspp_realtime\",\n",
    "             \"sea_water_temperature\":\n",
    "               {\"woa_normbias\": {\n",
    "                   \"threshold\": 3\n",
    "                   }\n",
    "               }\n",
    "           }\n",
    "\n",
    "cfg = load_cfg(my_config)\n",
    "print(cfg.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a new item, inherit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['gtspp_realtime']\n"
     ]
    }
   ],
   "source": [
    "print(cfg['inherit'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now sea_water_temperature has all the GTSPP realtime tests plus the WOA comparison,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "odict_keys(['global_range', 'gradient', 'spike', 'profile_envelop', 'woa_normbias'])\n"
     ]
    }
   ],
   "source": [
    "print(cfg['variables']['sea_water_temperature'].keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This new definition is actually the GTSPP recommended procedure for non-realtime data, i.e. the delayed mode. The built-in GTSPP procedure is actually written by inheriting the GTSPP realtime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['gtspp_realtime']\n"
     ]
    }
   ],
   "source": [
    "cfg = load_cfg('gtspp')\n",
    "print(cfg['inherit'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inheritance can also be used to modify any parameter from the parent template procedure. For example, let's use the GTSPP recommended procedure but with a more restricted threshold, equal to 1,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('threshold', 1)])\n"
     ]
    }
   ],
   "source": [
    "my_config = {\"inherit\": \"gtspp_realtime\",\n",
    "             \"sea_water_temperature\":\n",
    "               {\"spike\": {\n",
    "                   \"threshold\": 1\n",
    "                   }\n",
    "               }\n",
    "           }\n",
    "\n",
    "cfg = load_cfg(my_config)\n",
    "print(cfg['variables']['sea_water_temperature']['spike'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom collection of QC procedures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different datasets might require different procedures to best identify the bad measurements. Therefore it is convenient to be able to create a personal toolbox of procedures. For instance, I work with Spray underwater gliders at Scripps Institution of Oceanography, and I have a generic QC procedure for spray. I also have a modified version of that specifically for the California Underwater Glider Network operations, as well as another procedure for the deployments in the Mediterranean Sea. I saved those as spray.json, spray_CUGN.json, and spray_med.json.\n",
    "\n",
    "Any CoTeDe user can create its own collection of QC procedures. When CoTeDe doesn't find a built-in standard with that name, it searches in your home directory at:\n",
    "\n",
    "~/.config/cotederc/cfg/\n",
    "\n",
    "So for me, I placed those 3 json files at\n",
    "\n",
    "/home/guilherme/.config/cotederc/cfg/{spray.json,spray_CUGN.json,spray_med.json}\n",
    "\n",
    "Now I can call load_cfg('spray_CUGN').\n",
    "\n",
    "You can change where CoTeDe search for the configuration files by defining the environment variable COTEDE_DIR. If you use bash, you could do:\n",
    "\n",
    "export COTEDE_DIR='/my/much/better/place/to/save/these/'\n",
    "\n",
    "but keep in mind that CoTeDe will look for the directory cfg inside $COTEDE_DIR to look for the JSON files. I use this approach on my servers to keep everything tidy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Actual QC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remmember that there is no reason for most users to use load_cfg(), but I used it here to better illustrate how a QC procedure can be defined.\n",
    "\n",
    "The cfgname used in load_cfg is the same cfgname of ProfileQC. Therefore, you can apply everything that you learned here with ProfileQC. For instance, if you want to evaluate the temperature measurements from a profile by comparing with the World Ocean Atlas using 10 standard deviations as the tolerance, you could:\n",
    "\n",
    "pqced = ProfileQC(my_data, {\"sea_water_temperature\": {\"woa_normbias\": {\"threshold\": 10}}})\n",
    "\n",
    "Or if you want to evaluate a profile (my_data) using the euroGOOS recommended QC procedure (another built-in standard) you could\n",
    "\n",
    "pqced = ProfileQC(my_data, \"eurogoos\")\n",
    "\n",
    "Note that my_data must satisfy the CoTeDe's data model. Check the manual if you don't know what I'm talking about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excercise\n",
    "\n",
    "CoTede has a built-in QC procedure based on Argo recommendations, named 'argo'. Which variables, tests, and thresholds are applied on that setup? hint: line 3 loads the config for 'gtspp_realtime'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}