{ "cells": [ { "cell_type": "markdown", "id": "8c8138ef-e687-444e-95b1-9797654f3f62", "metadata": {}, "source": [ "# On-The-Fly (OTF) Data Reduction\n", "\n", "This notebook shows how to use dysh to calibrate and grid an *On-The-Fly* (OTF) observation. See \n", "[Mangum et al. (2007)](https://ui.adsabs.harvard.edu/abs/2007A%26A...474..679M) \n", "for the background on this method.\n", "\n", "OTF observations can be calibrated in dysh, however, generating FITS cubes requires the use of additional applications.\n", "\n", "The workflow to go from raw data to a FITS cube would then require the following steps:\n", "\n", "1. Calibrate the data using dysh. This can include baseline subtraction, altough in some cases baseline removal is more effective in a FITS cube.\n", "\n", "2. Write the calibrated spectra to a format compatible with the gridding tool being used. For GBT observations the recommended format is SDFITS and the tool to grid the data is the\n", "[gbtgridder](https://github.com/GreenBankObservatory/gbtgridder/tree/release_3.0).\n", "\n", "4. Baseline subtraction in the FITS cube. This is optional, depending on the data quality, and dysh does not provide convenience functions for this.\n", "\n", "## Dysh commands\n", "\n", "The following dysh commands are introduced (leaving out all the function arguments):\n", "\n", " filename = dysh_data()\n", " sdf = GBTFITSLoad()\n", " sb = sdf.getsigref()\n", " ta = sb.timeaverage()\n", " ta.baseline()\n", " ta.plot()\n", "\n", "## Loading Modules\n", "We start by loading the modules we will use for the data reduction. \n" ] }, { "cell_type": "code", "execution_count": 1, "id": "b4967550-2ca1-4931-b53b-6f9868718490", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "import" ] }, "outputs": [], "source": [ "# We use dysh_data to retrieve the example data set.\n", "from dysh.util.files import dysh_data\n", "from pathlib import Path\n", "\n", "# This is required to load and calibrate the example data set.\n", "from dysh.fits.gbtfitsload import GBTFITSLoad\n", "from dysh.log import init_logging" ] }, { "cell_type": "markdown", "id": "06ed5124-ef17-40e4-8fff-d13900aca5e3", "metadata": {}, "source": [ "## Setup\n", "We start the dysh logging, so we get more information about what is happening.\n", "This is only needed if working on a notebook.\n", "If using the CLI through the ``dysh`` command, then logging is setup for you." ] }, { "cell_type": "code", "execution_count": 2, "id": "8d0b3528-5da1-4390-8bf2-e7268eb2e92a", "metadata": {}, "outputs": [], "source": [ "init_logging(2)\n", "\n", "# also create a local \"output\" directory where temporary notebook files can be stored.\n", "output_dir = Path.cwd() / \"output\"\n", "output_dir.mkdir(exist_ok=True)" ] }, { "cell_type": "markdown", "id": "87669763-8d96-4521-9e81-f69d7213e133", "metadata": {}, "source": [ "## Data Retrieval\n", "\n", "Download the example SDFITS data, if necessary." ] }, { "cell_type": "code", "execution_count": 3, "id": "6bc88bc5-986d-4eae-b1c7-6398cc9ddd5a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "wget" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "12:03:27.099 I Resolving example=otf1 -> mapping-L/data/TGBT17A_506_11.raw.vegas/TGBT17A_506_11.raw.vegas.A.fits\n" ] } ], "source": [ "filename = dysh_data(example='otf1')" ] }, { "cell_type": "code", "execution_count": 4, "id": "93a62e3a-c95d-475b-8602-b5b8b7934733", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "load" ] }, "outputs": [], "source": [ "sdfits = GBTFITSLoad(filename)" ] }, { "cell_type": "code", "execution_count": 5, "id": "90d482e1-3953-49da-9be6-49605ae9cf64", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
| SCAN | \n", "OBJECT | \n", "VELOCITY | \n", "PROC | \n", "PROCSEQN | \n", "RESTFREQ | \n", "DOPFREQ | \n", "# IF | \n", "# POL | \n", "# INT | \n", "# FEED | \n", "AZIMUTH | \n", "ELEVATION | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 6 | \n", "3C286 | \n", "0.0 | \n", "OnOff | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "13 | \n", "1 | \n", "248.3657 | \n", "72.5531 | \n", "
| 7 | \n", "3C286 | \n", "0.0 | \n", "OnOff | \n", "2 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "13 | \n", "1 | \n", "250.0385 | \n", "73.6777 | \n", "
| 8 | \n", "SgrB2M | \n", "57.0 | \n", "Track | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "142.3845 | \n", "12.2545 | \n", "
| 9 | \n", "W33B | \n", "64.0 | \n", "Track | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "132.2057 | \n", "17.9786 | \n", "
| 10 | \n", "G30.589-0.044 | \n", "38.0 | \n", "Track | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "115.4811 | \n", "25.5523 | \n", "
| 11 | \n", "G31.412+0.307 | \n", "97.0 | \n", "Track | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "115.8034 | \n", "27.0986 | \n", "
| 12 | \n", "G35.577-0.029 | \n", "49.0 | \n", "Track | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "112.3139 | \n", "29.0337 | \n", "
| 13 | \n", "G40.622-0.137 | \n", "32.0 | \n", "Track | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "107.7736 | \n", "31.3159 | \n", "
| 14 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "38.9626 | \n", "41.3644 | \n", "
| 15 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "2 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "38.9910 | \n", "41.4488 | \n", "
| 16 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "3 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "38.9918 | \n", "41.5360 | \n", "
| 17 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "4 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0193 | \n", "41.6203 | \n", "
| 18 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "5 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0198 | \n", "41.7075 | \n", "
| 19 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "6 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0468 | \n", "41.7919 | \n", "
| 20 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "7 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0465 | \n", "41.8791 | \n", "
| 21 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "8 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0730 | \n", "41.9637 | \n", "
| 22 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "9 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0722 | \n", "42.0508 | \n", "
| 23 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "10 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0983 | \n", "42.1355 | \n", "
| 24 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "11 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0968 | \n", "42.2227 | \n", "
| 25 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "12 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.1224 | \n", "42.3073 | \n", "
| 26 | \n", "NGC6946 | \n", "45.0 | \n", "DecLatMap | \n", "13 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.1202 | \n", "42.3945 | \n", "
| 27 | \n", "NGC6946 | \n", "45.0 | \n", "Track | \n", "1 | \n", "1.6168 | \n", "1.420406 | \n", "5 | \n", "2 | \n", "61 | \n", "1 | \n", "39.0881 | \n", "42.1493 | \n", "
\n",
" \n",
" for i,_s in enumerate(sb0):\n",
" for j,_c in enumerate(_s._calibrated):\n",
" if math.isnan(_c.data.sum()):\n",
" # If the sum is NaN, then skip (continue) this item.\n",
" continue\n",
" s_i = getspec(j)\n",
" s_i.baseline(1, model=\"poly\", exclude=exclude, remove=True)\n",
" _s._calibrated[j] -= s_i.baseline_model(s_i.spectral_axis).value\n",
" \n",
" \n",
" This can be speed up if we only create a Spectrum once, and then update its data attribute with the data for each integration before computing the baseline. That would be:\n",
" \n",
" \n",
" sp0 = sb0.timeaverage() \n",
" for i,_s in enumerate(sb0):\n",
" for j,_c in enumerate(_s._calibrated):\n",
" if math.isnan(_c.data.sum()):\n",
" # If the sum is NaN, then skip (continue) this item.\n",
" continue\n",
" sp0._data = _s._calibrated[j] # Update the data of the `Spectrum`.\n",
" sp0.baseline(1, model=\"poly\", exclude=exclude, remove=True)\n",
" _s._calibrated[j] -= sp0.baseline_model(sp0.spectral_axis).value\n",
" \n",
" \n",
"