{ "cells": [ { "cell_type": "markdown", "id": "7bf35116-8ea6-4356-8966-050eff36ca7c", "metadata": {}, "source": [ "# Merging SDFITS Files\n", "\n", "This notebook shows how to merge SDFITS files. \n", "There is no built-in function to merge SDFITS files (yet), so we will put the files we want to merge in the same directory, and then load them and write them to a single SDFITS file.\n", "For this recipe we will use a single SDFITS file to generate two files from subsets of its data. We will need to use `getsigref` instead of `getps` to pull out a spectrum.\n", "\n", "The following dysh commands are (re)introduced (leaving out all the function arguments):\n", "\n", " filename = dysh_data()\n", " sdf = GBTFITSLoad()\n", " sdf.summary()\n", " sdf.getsigref()\n", " sdf.write()\n", " \n", "\n", "## Loading Modules\n", "We start by loading the modules we will use for this recipe. " ] }, { "cell_type": "code", "execution_count": 1, "id": "921bae33-6f92-4e2d-bdea-47722539a5bb", "metadata": {}, "outputs": [], "source": [ "# These modules are required for loading and writing data.\n", "from dysh.fits import GBTFITSLoad\n", "from dysh.log import init_logging\n", "from astropy import units as u\n", "\n", "# These modules are used for file I/O\n", "from dysh.util.files import dysh_data\n", "from pathlib import Path" ] }, { "cell_type": "markdown", "id": "1457f872-a7b6-4409-98a5-7d3930e92ed1", "metadata": {}, "source": [ "## Setup\n", "We start the dysh logging, so we get more information about what is happening.\n", "This is only needed if working on a notebook.\n", "If using the CLI through the ``dysh`` command, then logging is setup for you." ] }, { "cell_type": "code", "execution_count": 2, "id": "303bbc82-060e-4ab0-bf68-d7355cae0f87", "metadata": {}, "outputs": [], "source": [ "init_logging(2)\n", "\n", "# also create a local \"output\" directory where temporary notebook files can be stored.\n", "output_dir = Path.cwd() / \"output\"\n", "output_dir.mkdir(exist_ok=True)" ] }, { "cell_type": "markdown", "id": "aae89b6c-caf4-45d0-a3cf-fcca531f40fd", "metadata": {}, "source": [ "## Data Retrieval\n", "\n", "Download the example SDFITS data, if necessary." ] }, { "cell_type": "code", "execution_count": 3, "id": "4da076d0-e4de-416b-ab64-69987323c662", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "18:46:35.715 I Resolving test=getps -> AGBT05B_047_01/AGBT05B_047_01.raw.acs/\n" ] } ], "source": [ "filename = dysh_data(test=\"getps\")" ] }, { "cell_type": "markdown", "id": "61448bb0-9c34-4a44-a681-cf5e6fc54022", "metadata": {}, "source": [ "## Data Loading\n", "\n", "Next, we use \n", "[GBTFITSLoad](https://dysh.readthedocs.io/en/latest/reference/modules/dysh.fits.html#dysh.fits.GBTFITSLoad)\n", "to load the data, and then its \n", "[summary](https://dysh.readthedocs.io/en/latest/reference/modules/dysh.fits.html#dysh.fits.GBTFITSLoad.summary)\n", "method to inspect its contents." ] }, { "cell_type": "code", "execution_count": 4, "id": "54a6b302-1d99-4d38-9c83-17dd02b2c273", "metadata": {}, "outputs": [], "source": [ "sdfits = GBTFITSLoad(filename)" ] }, { "cell_type": "code", "execution_count": 5, "id": "4bc8d280-ec1c-498f-943d-7ec8b6e6602a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SCANOBJECTVELOCITYPROCPROCSEQNRESTFREQDOPFREQ# IF# POL# INT# FEEDAZIMUTHELEVATION
51NGC52914386.0OnOff11.4204051.42040512111198.343118.6427
52NGC52914386.0OnOff21.4204051.42040512111198.930618.7872
53NGC52914386.0OnOff11.4204051.42040512111199.330518.3561
54NGC52914386.0OnOff21.4204051.42040512111199.915718.4927
55NGC52914386.0OnOff11.4204051.42040512111200.304218.0575
56NGC52914386.0OnOff21.4204051.42040512111200.890618.1860
57NGC52914386.0OnOff11.4204051.42040512111202.327517.3853
58NGC52914386.0OnOff21.4204051.42040512111202.919217.4949
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "sdfits.summary()" ] }, { "cell_type": "markdown", "id": "2f78fdb1-d2ea-4494-90d5-eedcbdddebb2", "metadata": {}, "source": [ "## Writing SDFITS Files\n", "\n", "To show how to merge SDFITS files, we will create two separate SDFITS files first.\n", "We use the \n", "[GBTFITSLoad.write](https://dysh.readthedocs.io/en/latest/reference/modules/dysh.fits.html#dysh.fits.GBTFITSLoad.write)\n", "method to write the data.\n", "We will write scans 51 and 52 to two separate files.\n", "\n", "### Create an Output Directory\n", "\n", "The approach we will use to merge SDFITS files relies on having the files to be merged in the same directory.\n", "We create a new temporary directory ``./output/merge_sdfits`` " ] }, { "cell_type": "code", "execution_count": 6, "id": "9f05bcd4-44cd-497b-b322-4a6640db4204", "metadata": {}, "outputs": [], "source": [ "tmp_dir = output_dir / \"merge_sdfits\"\n", "tmp_dir.mkdir(exist_ok=True, parents=True) # Create the output directory if it does not exist." ] }, { "cell_type": "markdown", "id": "67444e8c-2fc2-4fa3-a73d-b1444dafc34d", "metadata": {}, "source": [ "### Write the Intermediate SDFITS Files\n", "\n", "Now we write scans 51 and 52 to the directory we created." ] }, { "cell_type": "code", "execution_count": 7, "id": "78a7a00b-01c8-4153-a3cf-10a5fab459db", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " ID TAG SCAN # SELECTED\n", "--- --------- ---- ----------\n", " 0 7a1da3548 51 44\n", " ID TAG SCAN # SELECTED\n", "--- --------- ---- ----------\n", " 0 8c391ccc8 52 44\n" ] } ], "source": [ "sdfits.write(tmp_dir / \"scan51.fits\", scan=51, overwrite=True)\n", "sdfits.write(tmp_dir / \"scan52.fits\", scan=52, overwrite=True)" ] }, { "cell_type": "markdown", "id": "eea51254-5686-4b88-b275-21ed1496e73a", "metadata": {}, "source": [ "## Merge SDFITS Files\n", "\n", "Now that we have two SDFITS files we want to merge, we can do this by loading them using \n", "[GBTFITSLoad](https://dysh.readthedocs.io/en/latest/reference/modules/dysh.fits.html#dysh.fits.GBTFITSLoad)\n", "by passing the directory containing the files we want to merge.\n", "In this case, that would be ``./output/merge_sdfits``." ] }, { "cell_type": "code", "execution_count": 8, "id": "27b10c0a-1286-4ac5-83a4-111185ac74df", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "18:46:36.214 I Loaded 2 FITS files\n" ] }, { "data": { "text/plain": [ "[PosixPath('/home/teuben/GBT/dysh/notebooks/examples/output/merge_sdfits/scan51.fits'),\n", " PosixPath('/home/teuben/GBT/dysh/notebooks/examples/output/merge_sdfits/scan52.fits')]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sdfits_merged = GBTFITSLoad(tmp_dir)\n", "sdfits_merged.files" ] }, { "cell_type": "code", "execution_count": 9, "id": "34b1523d-5652-4691-92c4-b5adb20788bb", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SCANOBJECTVELOCITYPROCPROCSEQNRESTFREQDOPFREQ# IF# POL# INT# FEEDAZIMUTHELEVATION
51NGC52914386.0OnOff11.4204051.42040512111198.343118.6427
52NGC52914386.0OnOff21.4204051.42040512111198.930618.7872
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "sdfits_merged.summary()" ] }, { "cell_type": "markdown", "id": "2133873f-a414-4529-97c7-b744853e2d47", "metadata": {}, "source": [ "### Write the Merged SDFITS Files to a New File\n", "\n", "Now we can write the merged SDFITS files to a new file using \n", "[GBTFITSLoad.write](https://dysh.readthedocs.io/en/latest/reference/modules/dysh.fits.html#dysh.fits.GBTFITSLoad.write)\n", "We tell it to write to a single file with the ``multifile=False`` argument.\n", "We write to a new directory to show that this is only one file now." ] }, { "cell_type": "code", "execution_count": 10, "id": "2b6a95f6-5ca2-4dff-997e-fb41512e8114", "metadata": {}, "outputs": [], "source": [ "new_dir = output_dir / \"merged_sdfits\"\n", "new_dir.mkdir(exist_ok=True)" ] }, { "cell_type": "code", "execution_count": 11, "id": "79a9e175-82d8-4c73-be60-65d1454d4d02", "metadata": {}, "outputs": [], "source": [ "sdfits_merged.write(new_dir / \"merged_sdfits.fits\", multifile=False, overwrite=True)" ] }, { "cell_type": "markdown", "id": "28adc061-b482-4755-a34a-00ec5be0f50a", "metadata": {}, "source": [ "Load the merged SDFITS file. " ] }, { "cell_type": "code", "execution_count": 12, "id": "bf6201fd-2d70-4c0a-9b58-ba48fb95c574", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SCANOBJECTVELOCITYPROCPROCSEQNRESTFREQDOPFREQ# IF# POL# INT# FEEDAZIMUTHELEVATION
51NGC52914386.0OnOff11.4204051.42040512111198.343118.6427
52NGC52914386.0OnOff21.4204051.42040512111198.930618.7872
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "sdfits_merged = GBTFITSLoad(new_dir)\n", "sdfits_merged.summary()" ] }, { "cell_type": "markdown", "id": "83cb7126-c668-4c08-a889-18431d9421fa", "metadata": {}, "source": [ "You can check that it is indeed a single file by looking at the ``files`` attribute of the ``GBTFITSLoad`` object." ] }, { "cell_type": "code", "execution_count": 13, "id": "f86f32f0-afd5-47c7-b704-1aa8ad028948", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[PosixPath('/home/teuben/GBT/dysh/notebooks/examples/output/merged_sdfits/merged_sdfits.fits')]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sdfits_merged.files" ] }, { "cell_type": "markdown", "id": "fa7617a6-fa80-4207-bdca-849c5bc1cfa0", "metadata": {}, "source": [ "The same principles can be applied to more complex examples, like selecting an IF for certain scans, and another for others, and then merging them all in a single file." ] }, { "cell_type": "markdown", "id": "d4b561f6-a255-406e-9c46-297e98a2eafc", "metadata": {}, "source": [ "## Final Stats\n", "\n", "We then use\n", "[getsigref](https://dysh.readthedocs.io/en/latest/reference/modules/dysh.fits.html#dysh.fits.GBTFITSLoad.getsigref)\n", "to obtain two spectra, which using the \n", "[Spectrum.stats](https://dysh.readthedocs.io/en/latest/reference/modules/dysh.spectra.html#dysh.spectra.spectrum.Spectrum.stats)\n", "should give the same answer \n", "\n", "A note on this, the\n", "[getps](https://dysh.readthedocs.io/en/latest/reference/modules/dysh.fits.html#dysh.fits.GBTFITSLoad.getps)\n", "procedure does not work here, because after the merge, the ON and OFF are in different BINTABLES, and `getps` does not support this mode." ] }, { "cell_type": "code", "execution_count": 14, "id": "4a105e82-977a-4e3b-91eb-8d37d562d88d", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "18:46:37.343 I rms is OK \n" ] } ], "source": [ "sdfits.getsigref(51,52,ifnum=0,plnum=0,fdnum=0)[0].timeaverage().check_stats(0.09095205 * u.K)" ] }, { "cell_type": "code", "execution_count": 15, "id": "7efbbff7-4e28-487e-8bec-1bc324fd6ddc", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "18:46:37.817 I rms is OK \n" ] } ], "source": [ "\n", "sdfits_merged.getsigref(51,52,ifnum=0,plnum=0,fdnum=0).timeaverage().check_stats(0.09095205 * u.K)" ] }, { "cell_type": "markdown", "id": "bacadb9d-cfcc-4af5-af70-cf416ef021a4", "metadata": {}, "source": [ "## See Also\n", "\n", "There is also an example of writing multiple SDFITS files in the `dataIO` notebook\n", "[here](https://github.com/GreenBankObservatory/dysh/blob/main/notebooks/examples/dataIO.ipynb#Writing-SDFITS-to-Multiple-Files)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.9" } }, "nbformat": 4, "nbformat_minor": 5 }