{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Combining flats\n",
"\n",
"There is one step in combining flats that is different from most other image\n",
"combination: the flats should be scaled to a common value before combining them.\n",
"This is particularly important if the flats are twilight flats in which the\n",
"average image value typically changes significantly as the images are being\n",
"taken.\n",
"\n",
"Flats are typically grouped by filter when combining them. That is, one combined\n",
"flat is produced for each filter in which flats were taken.\n",
"\n",
"Combination will be done for each of the two examples in the previous notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"\n",
"import ccdproc as ccdp\n",
"from astropy.stats import mad_std\n",
"from astropy.visualization import hist\n",
"\n",
"from convenience_functions import show_image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Use custom style for larger fonts and figures\n",
"plt.style.use('guide.mplstyle')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example 1\n",
"\n",
"We begin by setting up an image collection for the reduced data. This data is\n",
"from chip 0 of the cryogenically-cooled Large Format Camera at Palomar\n",
"Observatory."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"calibrated_path = Path('example1-reduced')\n",
"\n",
"flat_imagetyp = 'flatfield'\n",
"\n",
"ifc = ccdp.ImageFileCollection(calibrated_path)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll first check what filters are present."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"flat_filters = set(h['filter'] for h in ifc.headers(imagetyp=flat_imagetyp))\n",
"flat_filters"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"These flats are dome flats, essentially pictures of a screen in the dome\n",
"illuminated by a light source, so you would not expect there to be much variable\n",
"in the typical pixel value between different exposures. There is typically\n",
"*some* variation, though, so we graph it below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"median_count = [np.median(data) for data in ifc.data(imagetyp=flat_imagetyp)]\n",
"mean_count = [np.mean(data) for data in ifc.data(imagetyp=flat_imagetyp)]\n",
"plt.plot(median_count, label='median')\n",
"plt.plot(mean_count, label='mean')\n",
"plt.xlabel('Image number')\n",
"plt.ylabel('Count (ADU)')\n",
"plt.title('Pixel value in calibrated flat frames')\n",
"plt.legend()\n",
"print(median_count)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Although this is less frame-to-frame variation than we will see in Example 2, it\n",
"is about 5%. If we were to combine these without scaling the flats to a common\n",
"value then the images with higher counts would effectively get more weight than\n",
"the images.\n",
"\n",
"There is a substantial difference between the mean and median of this data.\n",
"Typically it is better to use the median because extreme values do not affect\n",
"the median as much as the mean.\n",
"\n",
"To scale the frames so that they have the same median value, we need to define a\n",
"function that can calculate the inverse of the median given the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def inv_median(a):\n",
" return 1 / np.median(a)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This function is passed into the `scale` argument of `combine` below. One\n",
"combined flat is created for each filter in the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for filt in flat_filters:\n",
" to_combine = ifc.files_filtered(imagetyp=flat_imagetyp, filter=filt, include_path=True)\n",
" combined_flat = ccdp.combine(to_combine,\n",
" method='average', scale=inv_median,\n",
" sigma_clip=True, sigma_clip_low_thresh=5, sigma_clip_high_thresh=5,\n",
" sigma_clip_func=np.ma.median, signma_clip_dev_func=mad_std,\n",
" mem_limit=350e6\n",
" )\n",
"\n",
" combined_flat.meta['combined'] = True\n",
" dark_file_name = 'combined_flat_filter_{}.fit'.format(filt.replace(\"''\", \"p\"))\n",
" combined_flat.write(calibrated_path / dark_file_name)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Discussion of Example 1"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We will begin by checking that the right number of combined flats have been\n",
"created. There were two filters, `g'` and `i'` in the raw data so there should\n",
"be two combined flats. We need to refresh the `ImageFileCollection` for the\n",
"reduced data because new files, our flats, have been added to them."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ifc.refresh()\n",
"ifc.files_filtered(imagetyp=flat_imagetyp, combined=True)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"That looks good. The two flats are displayed below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10, 20), tight_layout=True)\n",
"\n",
"for ccd, axis in zip(ifc.ccds(imagetyp=flat_imagetyp, combined=True), axes):\n",
" show_image(ccd.data, cmap='gray', fig=fig, ax=axis)\n",
" title = \"Filter {}\".format(ccd.header['filter'])\n",
" axis.set_title(title)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The first thing to notice is that the flats are different in these two filters.\n",
"That is expected because one of the elements in the optical path, the filter, is\n",
"different.\n",
"\n",
"The pattern of electronics in the flat images is because this is a\n",
"backside-illuminated CCD. The light-detecting pixels are on the under side of\n",
"the chip and the light needs to pass through the chip to reach the sensor. The\n",
"small dark spots are places where the chip wasn't thinned uniformly.\n",
"\n",
"Compare this with [Example 2](#discussion-of-example-2) below, which shows a flat\n",
"taken with a frontside-illuminated camera."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example 2\n",
"\n",
"The data in this example is from a thermoelectrically-cooled Andor Aspen CG16M.\n",
"These flats are twilight flats, taken just after sunset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"calibrated_path = Path('example2-reduced')\n",
"\n",
"flat_imagetyp = 'flat'\n",
"\n",
"ifc = ccdp.ImageFileCollection(calibrated_path)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"flat_filters = set(h['filter'] for h in ifc.headers(imagetyp=flat_imagetyp))\n",
"flat_filters"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Twilight flats typically differ more frame-to-frame than dome flats, as\n",
"shown in the figure below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"median_count = [np.median(hdu.data) for hdu in ifc.hdus(imagetyp=flat_imagetyp)]\n",
"mean_count = [np.mean(data) for data in ifc.data(imagetyp=flat_imagetyp)]\n",
"plt.plot(median_count, label='median')\n",
"plt.plot(mean_count, label='mean')\n",
"plt.xlabel('Image number')\n",
"plt.ylabel('Count (ADU)')\n",
"plt.title('Pixel value in calibrated flat frames')\n",
"plt.legend()\n",
"print(median_count)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def inv_median(a):\n",
" return 1 / np.median(a)\n",
"\n",
"for filt in flat_filters:\n",
" to_combine = ifc.files_filtered(imagetyp=flat_imagetyp, filter=filt, include_path=True)\n",
" combined_flat = ccdp.combine(to_combine,\n",
" method='average', scale=inv_median,\n",
" sigma_clip=True, sigma_clip_low_thresh=5, sigma_clip_high_thresh=5,\n",
" sigma_clip_func=np.ma.median, signma_clip_dev_func=mad_std,\n",
" mem_limit=350e6\n",
" )\n",
"\n",
" combined_flat.meta['combined'] = True\n",
" dark_file_name = 'combined_flat_filter_{}.fit'.format(filt.replace(\"''\", \"p\"))\n",
" combined_flat.write(calibrated_path / dark_file_name)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Discussion of Example 2"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We expect only one combined flat because there was only one filter. The\n",
"`ImageFileCollection` is refreshed before we query it because the combined flats\n",
"were added after the collection was created."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ifc.refresh()\n",
"ifc.files_filtered(imagetyp=flat_imagetyp, combined=True)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The combined flat is displayed below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"show_image(combined_flat, cmap='gray', figsize=(10, 20))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This flat looks very different than the one in [Example 1](#discussion-of-example-1)\n",
"because this CCD is frontside-illuminated and the previous one is backside-illuminated. \n",
"That means the sensor is on the top of the chip and the light does\n",
"not pass through the sensor chip itself to reach the sensors. Though only one\n",
"filter is shown here, the flat field looks slightly different through other\n",
"filters on this camera."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": true,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": true,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 4
}