{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding a custom reader to satpy\n",
    "\n",
    "\n",
    "In order to add a reader to satpy, you will need to create two files:\n",
    " - a YAML file for describing the files to read and the datasets that are available\n",
    " - a python file implementing the actual reading of the datasets and metadata\n",
    "\n",
    "For this tutorial, we will implement a reader for the Eumetsat NetCDF format for SEVIRI data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The YAML file\n",
    "\n",
    "The yaml file is composed of three sections:\n",
    "- the `reader` section, that provides basic parameters for the reader\n",
    "- the `file_types` section, which gives the patterns of the files this reader can handle\n",
    "- the `datasets` section, describing the datasets available from this reader \n",
    "\n",
    "### The `reader` section\n",
    "The `reader` section, that provides basic parameters for the reader.\n",
    "\n",
    "The parameters to provide in this section are:\n",
    "- description: General description of the reader\n",
    "- name: this is the name of the reader, it should be the same as the filename (without the .yaml extension obviously).  This is the name used interactively in satpy, so choose it well! A loose convention is to use `<format>_<instrument>_<level>` as a template for the name\n",
    "- sensors: the list of sensors this reader will support\n",
    "- reader: the metareader to use, in most cases the `FileYAMLReader` is a good choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```yaml\n",
    "reader:\n",
    "  description: NetCDF4 reader for the Eumetsat MSG format\n",
    "  name: nc_seviri_l1b\n",
    "  sensors: [seviri]\n",
    "  reader: !!python/name:satpy.readers.yaml_reader.FileYAMLReader\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `file_types` section\n",
    "\n",
    "Each file type needs to provide:\n",
    " - `file_reader`, the class that will handle the files for this reader, that you will implement in the corresponding python file (see next section)\n",
    " - `file_patterns`, the patterns to match to find files this reader can handle. The syntax to use is basically the same as `format` with the addition of time. See the [trollsift package documentation](https://trollsift.readthedocs.io/en/latest/usage.html) for more details.\n",
    " - Optionally, a file type can have a `requires` field: it is a list of file types that the current file types needs to function. For example, the HRIT MSG format segment files each need a prologue and epilogue file to be read properly, hence in this case we have added `requires: [HRIT_PRO, HRIT_EPI]` to the file type definition.\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```yaml\n",
    "file_types:\n",
    "    nc_seviri_l1b:\n",
    "        file_reader: !!python/name:satpy.readers.nc_seviri_l1b.NCSEVIRIFileHandler\n",
    "        file_patterns: ['W_XX-EUMETSAT-Darmstadt,VIS+IR+IMAGERY,{satid:4s}+SEVIRI_C_EUMG_{processing_time:%Y%m%d%H%M%S}.nc']\n",
    "    nc_seviri_l1b_hrv:\n",
    "        file_reader: !!python/name:satpy.readers.nc_seviri_l1b.NCSEVIRIHRVFileHandler\n",
    "        file_patterns: ['W_XX-EUMETSAT-Darmstadt,HRV+IMAGERY,{satid:4s}+SEVIRI_C_EUMG_{processing_time:%Y%m%d%H%M%S}.nc']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `datasets` section\n",
    "\n",
    "The datasets section describes each dataset available in the files. The parameters provided are made available to the methods of the implementing class.\n",
    "\n",
    "Parameters you can define for example are:\n",
    " - name\n",
    " - sensor\n",
    " - resolution\n",
    " - wavelength\n",
    " - polarization\n",
    " - standard_name: the name used for the dataset, that will be used for knowing what kind of data it is and handle it appropriately\n",
    " - units: the units of the data, important to get consistent processing across multiple platforms/instruments\n",
    " - modifiers: what modification have already been applied to the data, eg `sunz_corrected`\n",
    " - file_type\n",
    " - coordinates: this tells which datasets to load to navigate the current dataset\n",
    " - and any other field that is relevant for the reader\n",
    "\n",
    "This section can be copied and adapted simply from existing seviri readers, like for example the `msg_native` reader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```yaml\n",
    "\n",
    "datasets:\n",
    "  HRV:\n",
    "    name: HRV\n",
    "    resolution: 1000.134348869\n",
    "    wavelength: [0.5, 0.7, 0.9]\n",
    "    calibration:\n",
    "      reflectance:\n",
    "        standard_name: toa_bidirectional_reflectance\n",
    "        units: \"%\"\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b_hrv\n",
    "\n",
    "  IR_016:\n",
    "    name: IR_016\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [1.5, 1.64, 1.78]\n",
    "    calibration:\n",
    "      reflectance:\n",
    "        standard_name: toa_bidirectional_reflectance\n",
    "        units: \"%\"\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "    nc_key: 'ch3'\n",
    "\n",
    "  IR_039:\n",
    "    name: IR_039\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [3.48, 3.92, 4.36]\n",
    "    calibration:\n",
    "      brightness_temperature:\n",
    "        standard_name: toa_brightness_temperature\n",
    "        units: K\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "    nc_key: 'ch4'\n",
    "\n",
    "  IR_087:\n",
    "    name: IR_087\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [8.3, 8.7, 9.1]\n",
    "    calibration:\n",
    "      brightness_temperature:\n",
    "        standard_name: toa_brightness_temperature\n",
    "        units: K\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "  IR_097:\n",
    "    name: IR_097\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [9.38, 9.66, 9.94]\n",
    "    calibration:\n",
    "      brightness_temperature:\n",
    "        standard_name: toa_brightness_temperature\n",
    "        units: K\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "  IR_108:\n",
    "    name: IR_108\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [9.8, 10.8, 11.8]\n",
    "    calibration:\n",
    "      brightness_temperature:\n",
    "        standard_name: toa_brightness_temperature\n",
    "        units: K\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "  IR_120:\n",
    "    name: IR_120\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [11.0, 12.0, 13.0]\n",
    "    calibration:\n",
    "      brightness_temperature:\n",
    "        standard_name: toa_brightness_temperature\n",
    "        units: K\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "  IR_134:\n",
    "    name: IR_134\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [12.4, 13.4, 14.4]\n",
    "    calibration:\n",
    "      brightness_temperature:\n",
    "        standard_name: toa_brightness_temperature\n",
    "        units: K\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "  VIS006:\n",
    "    name: VIS006\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [0.56, 0.635, 0.71]\n",
    "    calibration:\n",
    "      reflectance:\n",
    "        standard_name: toa_bidirectional_reflectance\n",
    "        units: \"%\"\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "  VIS008:\n",
    "    name: VIS008\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [0.74, 0.81, 0.88]\n",
    "    calibration:\n",
    "      reflectance:\n",
    "        standard_name: toa_bidirectional_reflectance\n",
    "        units: \"%\"\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "  WV_062:\n",
    "    name: WV_062\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [5.35, 6.25, 7.15]\n",
    "    calibration:\n",
    "      brightness_temperature:\n",
    "        standard_name: toa_brightness_temperature\n",
    "        units: \"K\"\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "  WV_073:\n",
    "    name: WV_073\n",
    "    resolution: 3000.403165817\n",
    "    wavelength: [6.85, 7.35, 7.85]\n",
    "    calibration:\n",
    "      brightness_temperature:\n",
    "        standard_name: toa_brightness_temperature\n",
    "        units: \"K\"\n",
    "      radiance:\n",
    "        standard_name: toa_outgoing_radiance_per_unit_wavelength\n",
    "        units: W m-2 um-1 sr-1\n",
    "      counts:\n",
    "        standard_name: counts\n",
    "        units: count\n",
    "    file_type: nc_seviri_l1b\n",
    "\n",
    "    \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The YAML file is now ready, let's go on with the corresponding python file.\n",
    "## The python file\n",
    "The python files needs to implement a file handler class for each file type that we want to read.\n",
    "Such a class needs to implement a few methods:\n",
    "- the `__init__` method, that takes as arguments \n",
    "  - the filename (string)\n",
    "  - the filename info (dict) that we get by parsing the filename using the pattern defined in the yaml file\n",
    "  - the filetype info that we get from the filetype definition in the yaml file\n",
    "  \n",
    "  This method can also recieve other file handler instances as parameter if the filetype at hand has requirements. (See the explanation in the YAML file filetype section above)\n",
    "- the `get_dataset` method, which takes as arguments \n",
    "  - the dataset ID of the dataset to load\n",
    "  - the dataset info that is the description of the channel in the YAML file\n",
    "  \n",
    "  This method has to return an xarray.DataArray instance if the loading is successful, containing the data and metadata of the loaded dataset, or return None if the loading was unsuccessful.\n",
    "- the `get_area_def` method, that takes as single argument the dataset ID for which we want the area. For the data that cannot be geolocated with an area definition, the pixel coordinates need to be loadable from `get_dataset` for the resulting scene to be navigated. That is, if the data cannot be geolocated with an area definition then the dataset section should specify `coordinates: [longitude_dataset, latitude_dataset]`\n",
    "- Optionally, the `get_bounding_box` method can be implemented if filtering files by area is desirable for this data type \n",
    "\n",
    "On top of that, two attributes need to be defined: `start_time` and `end_time`, that define the start and end times of the sensing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is nc_seviri_l1b.py\n",
    "class NCSEVIRIFileHandler():\n",
    "    def __init__(self, filename, filename_info, filetype_info):\n",
    "        super(NCSEVIRIFileHandler, self).__init__(filename, filename_info, filetype_info)\n",
    "        self.nc = None\n",
    "    \n",
    "    def get_dataset(self, dataset_id, dataset_info):\n",
    "        if dataset_id.calibration != 'radiance':\n",
    "            # TODO: implement calibration to relfectance or brightness temperature\n",
    "            return\n",
    "        if self.nc is None:\n",
    "            self.nc = xr.open_dataset(self.filename,\n",
    "                                      decode_cf=True,\n",
    "                                      mask_and_scale=True,\n",
    "                                      chunks={'num_columns_vis_ir': CHUNK_SIZE,\n",
    "                                              'num_rows_vis_ir': CHUNK_SIZE})\n",
    "            self.nc = self.nc.rename({'num_columns_vir_ir': 'x', 'num_rows_vir_ir': 'y'})\n",
    "        dataset = self.nc[dataset_info['nc_key']]\n",
    "        dataset.attrs.update(dataset_info)\n",
    "        return dataset\n",
    "    \n",
    "    def get_area_def(self, dataset_id):\n",
    "        # TODO\n",
    "        pass\n",
    "    \n",
    "class NCSEVIRIHRVFileHandler():\n",
    "  # left as an exercise to the reader :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}