{ "cells": [ { "cell_type": "markdown", "id": "da18aeb2-71a2-4ed9-ab28-564d79542f91", "metadata": {}, "source": [ "# Modifying Metadata\n", "\n", "This notebook shows how to modify the metadata of an SDFITS file using dysh.\n", "\n", "## Background\n", "\n", "We will use a practical example. For observations with the GBT it is recommended to observe a flux density calibrator (see e.g., [Perley & Butler 2017](https://ui.adsabs.harvard.edu/abs/2017ApJS..230....7P/abstract) for a list of calibrator sources and their flux densities) with the same configuration as that used for the science observations. The reason being that the values of the temperature equivalent power of the noise diodes stored in the SDFITS files can be out of date.\n", "\n", "In this example we won't use observations of a flux density calibrator, but instead we will use the analysis of [Goddy et al (2020)](https://ui.adsabs.harvard.edu/abs/2020RNAAS...4....3G/abstract). They find that the temperature stored in the SDFITS files is on average lower than the measured values, so that the temperature must be corrected by 20%\n", "$$\n", "T_{\\rm{CAL,corr}}=1.2\\ T_{\\rm{CAL,file}}\n", "$$\n", "Here we use this to correct the temperature of the noise diode.\n", "\n", "## Dysh commands\n", "\n", "The following dysh commands are introduced (leaving out all the function arguments):\n", "\n", " filename = dysh_data()\n", " sdf = GBTFITSLoad()\n", " sdf.select()\n", " sb = sdf.getps()\n", " ta = sb.timeaverage()\n", " ta.baseline()\n", " ta.average()\n", " ta.plot()\n", " ta_plt.savefig()\n", "\n", "\n", "## Loading Modules\n", "We start by loading the modules we will use for this example. " ] }, { "cell_type": "code", "execution_count": 1, "id": "3de88ab1-3ed0-417f-a3a9-33ef711ce5aa", "metadata": {}, "outputs": [], "source": [ "# These modules are required for working with the data.\n", "from dysh.fits.gbtfitsload import GBTFITSLoad\n", "from dysh.log import init_logging\n", "from astropy import units as u\n", "\n", "# These modules are used for file I/O\n", "from dysh.util.files import dysh_data\n", "from pathlib import Path" ] }, { "cell_type": "markdown", "id": "22b6d78d-1489-40a3-ad9b-e313af808f3f", "metadata": {}, "source": [ "## Setup\n", "We start the dysh logging, so we get more information about what is happening.\n", "This is only needed if working on a notebook.\n", "If using the CLI through the ``dysh`` command, then logging is setup for you." ] }, { "cell_type": "code", "execution_count": 2, "id": "de4e81c0-6dcd-47ad-8351-5008860e4917", "metadata": {}, "outputs": [], "source": [ "init_logging(2)\n", "\n", "# also create a local \"output\" directory where temporary notebook files can be stored.\n", "output_dir = Path.cwd() / \"output\"\n", "output_dir.mkdir(exist_ok=True)" ] }, { "cell_type": "markdown", "id": "aad31cf1-092d-48d0-8468-8d50756ecdaa", "metadata": {}, "source": [ "### Data retrieval\n", "\n", "We download the data we will use for this example, if necessary." ] }, { "cell_type": "code", "execution_count": 3, "id": "f610f64f-f951-4ee4-aee4-6d450c2ef851", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "12:26:17.411 I Resolving test=getps -> AGBT05B_047_01/AGBT05B_047_01.raw.acs/\n" ] } ], "source": [ "filename = dysh_data(test=\"getps\")" ] }, { "cell_type": "markdown", "id": "248462cf-06d3-48ff-9cc5-5fa6ad9f8d47", "metadata": {}, "source": [ "### Data loading\n", "\n", "We load the data and inspect its contents." ] }, { "cell_type": "code", "execution_count": 4, "id": "ca5004bd-fbda-4ba3-ab6b-6da604c419ce", "metadata": {}, "outputs": [], "source": [ "sdfits = GBTFITSLoad(filename)" ] }, { "cell_type": "code", "execution_count": 5, "id": "24e047ab-a4bf-447c-90f6-246ee7f9413e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SCANOBJECTVELOCITYPROCPROCSEQNRESTFREQDOPFREQ# IF# POL# INT# FEEDAZIMUTHELEVATION
51NGC52914386.0OnOff11.4204051.42040512111198.343118.6427
52NGC52914386.0OnOff21.4204051.42040512111198.930618.7872
53NGC52914386.0OnOff11.4204051.42040512111199.330518.3561
54NGC52914386.0OnOff21.4204051.42040512111199.915718.4927
55NGC52914386.0OnOff11.4204051.42040512111200.304218.0575
56NGC52914386.0OnOff21.4204051.42040512111200.890618.1860
57NGC52914386.0OnOff11.4204051.42040512111202.327517.3853
58NGC52914386.0OnOff21.4204051.42040512111202.919217.4949
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "sdfits.summary()" ] }, { "cell_type": "markdown", "id": "257826a2-8124-47f3-af42-fedb9fd9a888", "metadata": {}, "source": [ "### Metadata inspection\n", "\n", "Now we inspect the current noise diode temperature stored in the SDFITS file, and some additional related parameters." ] }, { "cell_type": "code", "execution_count": 6, "id": "df3cad8b-34f8-4bbd-b950-e8c85c344ee0", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TCALPLNUMCALINTNUM
01.4242921T0
11.4242921F0
21.4526500T0
31.4526500F0
41.4242921T1
...............
3471.4526500F9
3481.4242921T10
3491.4242921F10
3501.4526500T10
3511.4526500F10
\n", "

352 rows × 4 columns

\n", "
" ], "text/plain": [ " TCAL PLNUM CAL INTNUM\n", "0 1.424292 1 T 0\n", "1 1.424292 1 F 0\n", "2 1.452650 0 T 0\n", "3 1.452650 0 F 0\n", "4 1.424292 1 T 1\n", ".. ... ... .. ...\n", "347 1.452650 0 F 9\n", "348 1.424292 1 T 10\n", "349 1.424292 1 F 10\n", "350 1.452650 0 T 10\n", "351 1.452650 0 F 10\n", "\n", "[352 rows x 4 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sdfits[\"TCAL\", \"PLNUM\", \"CAL\", \"INTNUM\"]" ] }, { "cell_type": "markdown", "id": "227bec08-e2d3-488b-9f38-83d788d2a7bc", "metadata": {}, "source": [ "For polarization 0 the noise diode temperature is 1.452650 K and for polarization 1 it is 1.424292 K.\n", "\n", "We will calibrate the data using these values to compare after we update the noise diode temperature. We use position switching calibration, then we time average all the scans and remove an order 1 polynomial." ] }, { "cell_type": "code", "execution_count": 7, "id": "01c287c8-30fe-4949-9595-d7b4a050c2dc", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "12:26:18.554 I EXCLUDING None\n" ] } ], "source": [ "ps_original = sdfits.getps(plnum=0, ifnum=0, fdnum=0).timeaverage()\n", "ps_original.baseline(degree=1, remove=True) " ] }, { "cell_type": "markdown", "id": "b3e0a6f9-209d-4eaa-9785-9ea3b621193e", "metadata": {}, "source": [ "### Metadata update\n", "\n", "Now we update the temperature of the noise diode by multiplying by 1.2." ] }, { "cell_type": "code", "execution_count": 8, "id": "ecdbe355-f310-4cac-9952-d1608778700f", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/teuben/GBT/dysh/src/dysh/fits/gbtfitsload.py:3348: UserWarning: Changing an existing SDFITS column TCAL\n", " warnings.warn(f\"Changing an existing SDFITS column {items}\") # noqa: B028\n", "/home/teuben/GBT/dysh/src/dysh/fits/sdfitsload.py:1116: UserWarning: Changing an existing SDFITS column TCAL\n", " warnings.warn(f\"Changing an existing SDFITS column {items}\") # noqa: B028\n" ] } ], "source": [ "sdfits[\"TCAL\"] *= 1.2" ] }, { "cell_type": "markdown", "id": "de6c5eb6-7a5e-4b9e-bf2e-f96db35287bf", "metadata": {}, "source": [ "Now we check that the values were updated." ] }, { "cell_type": "code", "execution_count": 9, "id": "c9377048-1a0a-4ff6-bdd4-1bcf08f9ca58", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TCALPLNUM
01.709151
11.709151
21.743180
31.743180
41.709151
.........
3471.743180
3481.709151
3491.709151
3501.743180
3511.743180
\n", "

352 rows × 2 columns

\n", "
" ], "text/plain": [ " TCAL PLNUM\n", "0 1.70915 1\n", "1 1.70915 1\n", "2 1.74318 0\n", "3 1.74318 0\n", "4 1.70915 1\n", ".. ... ...\n", "347 1.74318 0\n", "348 1.70915 1\n", "349 1.70915 1\n", "350 1.74318 0\n", "351 1.74318 0\n", "\n", "[352 rows x 2 columns]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sdfits[\"TCAL\", \"PLNUM\"]" ] }, { "cell_type": "markdown", "id": "50febab3-1fc3-4763-afef-de92d6c61340", "metadata": {}, "source": [ "The values were updated. We proceed with the data reduction." ] }, { "cell_type": "code", "execution_count": 10, "id": "d435b22b-6fa2-4966-b04b-d9eebe04f745", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "12:26:19.294 I EXCLUDING None\n" ] } ], "source": [ "ps_updated = sdfits.getps(plnum=0, ifnum=0, fdnum=0).timeaverage()\n", "ps_updated.baseline(degree=1, remove=True) " ] }, { "cell_type": "markdown", "id": "80b89085-88f3-4212-84d0-fff9c0418c9a", "metadata": {}, "source": [ "Now plot and compare the result. Since the antenna temperature is directly proportional to the temperature of the noise diode, now the line profile after the update should be 20% brighter than without the update." ] }, { "cell_type": "code", "execution_count": 11, "id": "7b1b2b5d-353f-4526-b3e3-c50a0ca89182", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d7e75d3e1a1540078baf05c0c57ca414", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HBox(children=(Button(description='Clear All Regions', style=ButtonStyle(), tooltip='Clear all …" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "19a6ae35723e420488329650a52757bf", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HBox(children=(Button(description='Clear All Regions', style=ButtonStyle(), tooltip='Clear all …" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ps_original.plot(ymin=-0.5, ymax=0.3)\n", "ps_updated.plot(ymin=-0.5, ymax=0.3)" ] }, { "cell_type": "markdown", "id": "f7b0f07d-186a-4a26-a069-8ee452cc15f5", "metadata": {}, "source": [ "## Final Stats\n", "\n", "Finally, at the end we compute some statistics over a spectrum, merely as a checksum if the notebook is reproducible." ] }, { "cell_type": "code", "execution_count": 12, "id": "690ce4ba-0fac-4c28-a5d0-6d461a9b00b3", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "12:26:19.417 I rms is OK \n" ] } ], "source": [ "ps_updated.check_stats(0.07192832 * u.K)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.9" } }, "nbformat": 4, "nbformat_minor": 5 }