{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Combine calibrated dark images for use in later reduction steps\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step is to combine the individual calibrated dark images into a single\n",
    "combined image. That combined image will have less noise than the individual\n",
    "images, minimizing the noise added to the remaining images when the dark is\n",
    "subtracted.\n",
    "\n",
    "Regardless of which path you took through the calibration of the biases (with\n",
    "overscan or without) there should be a folder named either `example1-reduced` or\n",
    "`example2-reduced` that contains the calibrated bias and dark images. If there\n",
    "is not, please run the previous notebook before continuing with this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "from astropy.nddata import CCDData\n",
    "from astropy.stats import mad_std\n",
    "\n",
    "import ccdproc as ccdp\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from convenience_functions import show_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use custom style for larger fonts and figures\n",
    "plt.style.use('guide.mplstyle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recommended settings for image combination\n",
    "\n",
    "As discussed in the [notebook about combining images](01-06-Image-combination.ipynb), the recommendation is\n",
    "that you combine by averaging the individual images but sigma clip to remove\n",
    "extreme values.\n",
    "\n",
    "[ccdproc](https://ccdproc.readthedocs.org) provides two ways to combine:\n",
    "\n",
    "+ An object-oriented interface built around the `Combiner` object, described in\n",
    "the [ccdproc documentation on image combination](https://ccdproc.readthedocs.io/en/latest/image_combination.html).\n",
    "+ A function called [`combine`](https://ccdproc.readthedocs.io/en/latest/api/ccdproc.combine.html#ccdproc.combine), which we will use here because the function\n",
    "allows you to specify the maximum amount of memory that should be used during\n",
    "combination. That feature can be essential depending on how many images you need\n",
    "to combine, how big they are, and how much memory your computer has.\n",
    "\n",
    "*NOTE: If using a version of ccdproc lower than 2.0, set the memory limit a\n",
    "factor of 2-3 lower than you want the maximum memory consumption to be.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1: Cryogenically-cooled camera"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The remainder of this section assumes that the calibrated bias images are in the\n",
    "folder `example1-reduced` which was created in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calibrated_path = Path('example1-reduced')\n",
    "reduced_images = ccdp.ImageFileCollection(calibrated_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make a combined image for each exposure time in Example 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several dark exposure times in this data set. By converting the times\n",
    "in the summary table to a set it returns only the unique values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "darks = reduced_images.summary['imagetyp'] == 'DARK'\n",
    "dark_times = set(reduced_images.summary['exptime'][darks])\n",
    "print(dark_times)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below loops over the dark exposure times and, for each exposure time:\n",
    "\n",
    "+ selects the relevant calibrated dark images,\n",
    "+ combines them using the `combine` function,\n",
    "+ adds the keyword `COMBINED` to the header so that later calibration steps can\n",
    "easily identify which bias to use, and\n",
    "+ writes the file whose name includes the exposure time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for exp_time in sorted(dark_times):\n",
    "    calibrated_darks = reduced_images.files_filtered(imagetyp='dark', exptime=exp_time,\n",
    "                                                     include_path=True)\n",
    "\n",
    "    combined_dark = ccdp.combine(calibrated_darks,\n",
    "                                 method='average',\n",
    "                                 sigma_clip=True, sigma_clip_low_thresh=5, sigma_clip_high_thresh=5,\n",
    "                                 sigma_clip_func=np.ma.median, sigma_clip_dev_func=mad_std,\n",
    "                                 mem_limit=350e6\n",
    "                                )\n",
    "\n",
    "    combined_dark.meta['combined'] = True\n",
    "\n",
    "    dark_file_name = 'combined_dark_{:6.3f}.fit'.format(exp_time)\n",
    "    combined_dark.write(calibrated_path / dark_file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Result for Example 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A single calibrated 300 second dark image and the combined 300 second image are\n",
    "shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))\n",
    "\n",
    "show_image(CCDData.read(calibrated_darks[0]).data, cmap='gray', ax=ax1, fig=fig)\n",
    "ax1.set_title('Single calibrated dark')\n",
    "show_image(combined_dark.data, cmap='gray', ax=ax2, fig=fig)\n",
    "ax2.set_title('{} dark images combined'.format(len(calibrated_darks)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2: Thermoelectrically-cooled camera\n",
    "\n",
    "The process for combining the images is exactly the same as in example 1. The\n",
    "only difference is the directory that contains the calibrated bias frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calibrated_path = Path('example2-reduced')\n",
    "reduced_images = ccdp.ImageFileCollection(calibrated_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make a combined image for each exposure time in Example 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example there are only darks of a single exposure time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "darks = reduced_images.summary['imagetyp'] == 'DARK'\n",
    "dark_times = set(reduced_images.summary['exptime'][darks])\n",
    "print(dark_times)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite the fact that there is only one exposure time, we might as well reuse\n",
    "the code from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for exp_time in sorted(dark_times):\n",
    "    calibrated_darks = reduced_images.files_filtered(imagetyp='dark', exptime=exp_time,\n",
    "                                                     include_path=True)\n",
    "\n",
    "    combined_dark = ccdp.combine(calibrated_darks,\n",
    "                                 method='average',\n",
    "                                 sigma_clip=True, sigma_clip_low_thresh=5, sigma_clip_high_thresh=5,\n",
    "                                 sigma_clip_func=np.ma.median, signma_clip_dev_func=mad_std,\n",
    "                                 mem_limit=350e6\n",
    "                                )\n",
    "\n",
    "    combined_dark.meta['combined'] = True\n",
    "\n",
    "    dark_file_name = 'combined_dark_{:6.3f}.fit'.format(exp_time)\n",
    "    combined_dark.write(calibrated_path / dark_file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Result for Example 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference between a single calibrated bias image and the combined bias\n",
    "image is much clearer in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))\n",
    "\n",
    "show_image(CCDData.read(calibrated_darks[0]).data, cmap='gray', ax=ax1, fig=fig)\n",
    "ax1.set_title('Single calibrated dark')\n",
    "show_image(combined_dark.data, cmap='gray', ax=ax2, fig=fig)\n",
    "ax2.set_title('{} dark images combined'.format(len(calibrated_darks)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}