{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reducing science images\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"from matplotlib import pyplot as plt\n",
"import numpy as np\n",
"\n",
"from astropy import units as u\n",
"import ccdproc as ccdp\n",
"\n",
"from convenience_functions import show_image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Use custom style for larger fonts and figures\n",
"plt.style.use('guide.mplstyle')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Overview\n",
"\n",
"[ccdproc](https://ccdproc.readthedocs.io) provides a couple of ways to approach\n",
"calibration of the science images:\n",
"\n",
"+ Perform each of the individual steps manually using `subtract_bias`,\n",
"`subtract_dark`, and `flat_correct`.\n",
"+ Use the [`ccd_process` function](https://ccdproc.readthedocs.io/en/latest/api/ccdproc.ccd_process.html#ccdproc.ccd_process) to perform all of the reduction steps.\n",
"\n",
"This notebook will do each of these in separate sections below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def find_nearest_dark_exposure(image, dark_exposure_times, tolerance=0.5):\n",
" \"\"\"\n",
" Find the nearest exposure time of a dark frame to the exposure time of the image,\n",
" raising an error if the difference in exposure time is more than tolerance.\n",
" \n",
" Parameters\n",
" ----------\n",
" \n",
" image : astropy.nddata.CCDData\n",
" Image for which a matching dark is needed.\n",
" \n",
" dark_exposure_times : list\n",
" Exposure times for which there are darks.\n",
" \n",
" tolerance : float or ``None``, optional\n",
" Maximum difference, in seconds, between the image and the closest dark. Set\n",
" to ``None`` to skip the tolerance test.\n",
" \n",
" Returns\n",
" -------\n",
" \n",
" float\n",
" Closest dark exposure time to the image.\n",
" \"\"\"\n",
"\n",
" dark_exposures = np.array(list(dark_exposure_times))\n",
" idx = np.argmin(np.abs(dark_exposures - image.header['exptime']))\n",
" closest_dark_exposure = dark_exposures[idx]\n",
"\n",
" if (tolerance is not None and \n",
" np.abs(image.header['exptime'] - closest_dark_exposure) > tolerance):\n",
" \n",
" raise RuntimeError('Closest dark exposure time is {} for flat of exposure '\n",
" 'time {}.'.format(closest_dark_exposure, a_flat.header['exptime']))\n",
" \n",
" \n",
" return closest_dark_exposure"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example 1\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data in this example is from chip 0 of the Large Format Camera at Palomar\n",
"Observatory. In earlier notebooks we determined that:\n",
"\n",
"+ This CCD has a [useful overscan region](01.08-Overscan.ipynb#Case-1:-Cryogenically-cooled-Large-Format-Camera-(LFC)-at-Palomar).\n",
"+ There is [no need to scale dark frames](05-03-Calibrating-the-flats.ipynb#Subtracting-bias-is-not-necessary-in-this-example) in this data, and so no need to subtract bias.\n",
"\n",
"First, we create [ImageFileCollection](https://ccdproc.readthedocs.io/en/latest/ccdproc/image_management.html)s for the raw data and the calibrated\n",
"data, and define variables for the image type of the flat field and the science\n",
"images (there are, unfortunately, no standard names for these)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"reduced_path = Path('example1-reduced')\n",
"\n",
"science_imagetyp = 'object'\n",
"flat_imagetyp = 'flatfield'\n",
"exposure = 'exptime'\n",
"\n",
"ifc_reduced = ccdp.ImageFileCollection(reduced_path)\n",
"ifc_raw = ccdp.ImageFileCollection('example-cryo-LFC')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, a quick look at the science images in this data set."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lights = ifc_raw.summary[ifc_raw.summary['imagetyp'] == science_imagetyp.upper()]\n",
"lights['date-obs', 'file', 'object', 'filter', exposure]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, a look at the calibrated combined images available to us."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"combo_calibs = ifc_reduced.summary[ifc_reduced.summary['combined'].filled(False).astype('bool')]\n",
"combo_calibs['date-obs', 'file', 'imagetyp', 'object', 'filter', exposure]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Both of these images will use the 300 second dark exposure to remove dark\n",
"current and there is a flat field image for each of the two filters present in\n",
"the data.\n",
"\n",
"Though we could hard code which dark to use, it is possible to set up a\n",
"dictionary for accessing the darks, with a separate one for accessing the science\n",
"images. The goal is to write code that is more conveniently reused."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"combined_darks = {ccd.header[exposure]: ccd for ccd in ifc_reduced.ccds(imagetyp='dark', combined=True)}\n",
"combined_flats = {ccd.header['filter']: ccd for ccd in ifc_reduced.ccds(imagetyp=flat_imagetyp, combined=True)}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The calibration steps for each of these science images are:\n",
"\n",
"+ Subtract the overscan from the science image.\n",
"+ Trim the overscan.\n",
"+ Subtract the dark current (which in this case also includes the bias).\n",
"+ Flat correct the image.\n",
"\n",
"These are the minimum steps needed to calibrate. Gain correcting the data and\n",
"identifying cosmic rays are additional steps that might often be done at this\n",
"stage. Identification of cosmic rays will be discussed in [this notebook](08-03-Cosmic-ray-removal.ipynb)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# These two lists are created so that we have copies of the raw and calibrated images\n",
"# to later in the notebook. They are not ordinarily required.\n",
"all_reds = []\n",
"light_ccds = []\n",
"for light, file_name in ifc_raw.ccds(imagetyp=science_imagetyp, return_fname=True, ccd_kwargs=dict(unit='adu')):\n",
" light_ccds.append(light)\n",
" \n",
" reduced = ccdp.subtract_overscan(light, overscan=light[:, 2055:], median=True)\n",
" \n",
" reduced = ccdp.trim_image(reduced[:, :2048])\n",
"\n",
" closest_dark = find_nearest_dark_exposure(reduced, combined_darks.keys())\n",
" reduced = ccdp.subtract_dark(reduced, combined_darks[closest_dark],\n",
" exposure_time=exposure, exposure_unit=u.second\n",
" )\n",
" good_flat = combined_flats[reduced.header['filter']]\n",
" reduced = ccdp.flat_correct(reduced, good_flat)\n",
" all_reds.append(reduced)\n",
" reduced.write(reduced_path / file_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Before and after for Example 1\n",
"\n",
"The cell below displays each science image before and after calibration."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fig, axes = plt.subplots(2, 2, figsize=(10, 20), tight_layout=True)\n",
"\n",
"for row, raw_science_image in enumerate(light_ccds):\n",
" filt = raw_science_image.header['filter']\n",
" axes[row, 0].set_title('Uncalibrated image {}'.format(filt))\n",
" show_image(raw_science_image, cmap='gray', ax=axes[row, 0], fig=fig, percl=90)\n",
"\n",
" axes[row, 1].set_title('Calibrated image {}'.format(filt))\n",
" show_image(all_reds[row].data, cmap='gray', ax=axes[row, 1], fig=fig, percl=90)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In each reduced image the overall pattern in the flat field has been removed,\n",
"and the sensor glow along the left edge and in the bottom left corner of the\n",
"uncalibrated images has been removed. The background in the calibrated images is\n",
"still not perfectly uniform, though that background can be removed using\n",
"photutils if desired. The partial bad column visible in both uncalibrated images\n",
"is also still present in the calibrated images, particularly in the `g'` filter.\n",
"\n",
"Those pixels will need to be masked, as described in the [notebook on bad\n",
"pixels](08-01-Identifying-hot-pixels.ipynb)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The camera in this example is a thermoelectrically-cooled Andor Aspen CG16M. In\n",
"previous notebooks we decided that:\n",
"\n",
"+ The overscan region of this camera is not useful.\n",
"+ For this data set the dark frames needed to be scaled, so we need to subtract\n",
"the bias frame from each science image.\n",
"\n",
"First, we create [ImageFileCollection](https://ccdproc.readthedocs.io/en/latest/ccdproc/image_management.html)s for the raw data and the\n",
"calibrated data, and define variables for the image type of the flat field and\n",
"the science images (there are, unfortunately, no standard names for these)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"reduced_path = Path('example2-reduced')\n",
"\n",
"science_imagetyp = 'light'\n",
"flat_imagetyp = 'flat'\n",
"exposure = 'exposure'\n",
"\n",
"ifc_reduced = ccdp.ImageFileCollection(reduced_path)\n",
"ifc_raw = ccdp.ImageFileCollection('example-thermo-electric')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we check what science images are available."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lights = ifc_raw.summary[ifc_raw.summary['imagetyp'] == science_imagetyp.upper()]\n",
"lights['date-obs', 'file', 'object', 'filter', exposure]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are only two science images, both in the same filter and with the same\n",
"exposure time.\n",
"\n",
"The combined calibration images available are listed below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"combo_calibs = ifc_reduced.summary[ifc_reduced.summary['combined'].filled(False).astype('bool')]\n",
"combo_calibs['date-obs', 'file', 'imagetyp', 'filter', exposure]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Although there is only one of each type of combined calibration image, the code\n",
"below sets up a dictionary for the flats and darks. That code will work on a\n",
"more typical night when there might be several filters and multiple exposure\n",
"times."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"combined_darks = {ccd.header[exposure]: ccd for ccd in ifc_reduced.ccds(imagetyp='dark', combined=True)}\n",
"combined_flats = {ccd.header['filter']: ccd for ccd in ifc_reduced.ccds(imagetyp=flat_imagetyp, combined=True)}\n",
"\n",
"# There is only on bias, so no need to set up a dictionary.\n",
"combined_bias = [ccd for ccd in ifc_reduced.ccds(imagetyp='bias', combined=True)][0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The calibration steps for each of these science images is:\n",
"\n",
"+ Trim the overscan.\n",
"+ Subtract bias, because bias was subtracted from the dark frames.\n",
"+ Subtract dark, using the dark closest in exposure time to the science image.\n",
"+ Flat correct the images, using the combined flat whose filter matches the\n",
"science image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"all_reds = []\n",
"light_ccds = []\n",
"for light, file_name in ifc_raw.ccds(imagetyp=science_imagetyp, return_fname=True):\n",
" light_ccds.append(light)\n",
"\n",
" reduced = ccdp.trim_image(light[:, :4096])\n",
" \n",
" # Note that the first argument in the remainder of the ccdproc calls is\n",
" # the *reduced* image, so that the calibration steps are cumulative.\n",
" reduced = ccdp.subtract_bias(reduced, combined_bias)\n",
" \n",
" closest_dark = find_nearest_dark_exposure(reduced, combined_darks.keys())\n",
"\n",
" reduced = ccdp.subtract_dark(reduced, combined_darks[closest_dark], \n",
" exposure_time='exptime', exposure_unit=u.second)\n",
" \n",
" good_flat = combined_flats[reduced.header['filter']]\n",
" reduced = ccdp.flat_correct(reduced, good_flat)\n",
" all_reds.append(reduced)\n",
" reduced.write(reduced_path / file_name)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fig, axes = plt.subplots(2, 2, figsize=(10, 10), tight_layout=True)\n",
"\n",
"for row, raw_science_image in enumerate(light_ccds):\n",
" filt = raw_science_image.header['filter']\n",
" axes[row, 0].set_title('Uncalibrated image {}'.format(filt))\n",
" show_image(raw_science_image, cmap='gray', ax=axes[row, 0], fig=fig, percl=90)\n",
"\n",
" axes[row, 1].set_title('Calibrated image {}'.format(filt))\n",
" show_image(all_reds[row].data, cmap='gray', ax=axes[row, 1], fig=fig, percl=90)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In both images, most of the nonunifomity in the background has been removed along\n",
"the bright column and the sensor glow on the left and right edges of the\n",
"uncalibrated images.\n",
"\n",
"The calibrated image in the top row looks like flat correction has led to a\n",
"fairly uniform background. The background in the image in the second row is not\n",
"that uniform. As it turns out the second image was taken much closer to dawn\n",
"than the first image, so the sky background really was not uniform when the\n",
"image was taken."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}