{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Fermi-LAT data with Gammapy\n", "\n", "## Introduction\n", "\n", "This tutorial will show you how to work with Fermi-LAT data with Gammapy. As an example, we will look at the Galactic center region using the high-energy dataset that was used for the 3FHL catalog, in the energy range 10 GeV to 2 TeV.\n", "\n", "We note that support for Fermi-LAT data analysis in Gammapy is very limited. For most tasks, we recommend you use \n", "[Fermipy](http://fermipy.readthedocs.io/), which is based on the [Fermi Science Tools](https://fermi.gsfc.nasa.gov/ssc/data/analysis/software/) (Fermi ST).\n", "\n", "Using Gammapy with Fermi-LAT data could be an option for you if you want to do an analysis that is not easily possible with Fermipy and the Fermi Science Tools. For example a joint likelihood fit of Fermi-LAT data with data e.g. from H.E.S.S., MAGIC, VERITAS or some other instrument, or analysis of Fermi-LAT data with a complex spatial or spectral model that is not available in Fermipy or the Fermi ST.\n", "\n", "Besides Gammapy, you might want to look at are [Sherpa](http://cxc.harvard.edu/sherpa/) or [3ML](https://threeml.readthedocs.io/). Or just using Python to roll your own analyis using several existing analysis packages. E.g. it it possible to use Fermipy and the Fermi ST to evaluate the likelihood on Fermi-LAT data, and Gammapy to evaluate it e.g. for IACT data, and to do a joint likelihood fit using e.g. [iminuit](http://iminuit.readthedocs.io/) or [emcee](http://dfm.io/emcee).\n", "\n", "To use Fermi-LAT data with Gammapy, you first have to use the Fermi ST to prepare an event list (using ``gtselect`` and ``gtmktime``, exposure cube (using ``gtexpcube2`` and PSF (using ``gtpsf``). You can then use [gammapy.data.EventList](https://docs.gammapy.org/dev/api/gammapy.data.EventList.html), [gammapy.maps](https://docs.gammapy.org/dev/maps/index.html) and the [gammapy.irf.EnergyDependentTablePSF](https://docs.gammapy.org/dev/api/gammapy.irf.EnergyDependentTablePSF.html) to read the Fermi-LAT maps and PSF, i.e. support for these high-level analysis products from the Fermi ST is built in. To do a 3D map analyis, you can use Fit for Fermi-LAT data in the same way that it's use for IACT data. This is illustrated in this notebook. A 1D region-based spectral analysis is also possible, this will be illustrated in a future tutorial.\n", "\n", "## Setup\n", "\n", "**IMPORTANT**: For this notebook you have to get the prepared ``3fhl`` dataset provided in your $GAMMAPY_DATA.\n", "\n", "Note that the ``3fhl`` dataset is high-energy only, ranging from 10 GeV to 2 TeV." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Check that you have the prepared Fermi-LAT dataset\n", "# We will use diffuse models from here\n", "!ls -1 $GAMMAPY_DATA/fermi_3fhl" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from astropy import units as u\n", "from astropy.coordinates import SkyCoord\n", "from gammapy.data import EventList\n", "from gammapy.irf import EnergyDependentTablePSF, EnergyDispersion\n", "from gammapy.maps import Map, MapAxis, WcsNDMap, WcsGeom\n", "from gammapy.spectrum.models import TableModel, PowerLaw\n", "from gammapy.image.models import SkyPointSource, SkyDiffuseConstant\n", "from gammapy.cube.models import SkyModel, SkyDiffuseCube, SkyModels\n", "from gammapy.cube import MapDataset, PSFKernel, MapEvaluator\n", "from gammapy.utils.fitting import Fit" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Events\n", "\n", "To load up the Fermi-LAT event list, use the [gammapy.data.EventList](https://docs.gammapy.org/dev/api/gammapy.data.EventList.html) class:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "events = EventList.read(\n", " \"$GAMMAPY_DATA/fermi_3fhl/fermi_3fhl_events_selected.fits.gz\"\n", ")\n", "print(events)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The event data is stored in a [astropy.table.Table](http://docs.astropy.org/en/stable/api/astropy.table.Table.html) object. In case of the Fermi-LAT event list this contains all the additional information on positon, zenith angle, earth azimuth angle, event class, event type etc." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "events.table.colnames" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "events.table[:5][[\"ENERGY\", \"RA\", \"DEC\"]]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(events.time[0].iso)\n", "print(events.time[-1].iso)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "energy = events.energy\n", "energy.info(\"stats\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As a short analysis example we will count the number of events above a certain minimum energy: " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for e_min in [10, 100, 1000] * u.GeV:\n", " n = (events.energy > e_min).sum()\n", " print(\"Events above {0:4.0f}: {1:5.0f}\".format(e_min, n))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Counts\n", "\n", "Let us start to prepare things for an 3D map analysis of the Galactic center region with Gammapy. The first thing we do is to define the map geometry. We chose a TAN projection centered on position ``(glon, glat) = (0, 0)`` with pixel size 0.1 deg, and four energy bins." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "gc_pos = SkyCoord(0, 0, unit=\"deg\", frame=\"galactic\")\n", "energy_axis = MapAxis.from_edges(\n", " [10, 30, 100, 300, 2000], name=\"energy\", unit=\"GeV\", interp=\"log\"\n", ")\n", "counts = Map.create(\n", " skydir=gc_pos,\n", " npix=(100, 80),\n", " proj=\"TAN\",\n", " coordsys=\"GAL\",\n", " binsz=0.1,\n", " axes=[energy_axis],\n", " dtype=float,\n", ")\n", "# We put this call into the same Jupyter cell as the Map.create\n", "# because otherwise we could accidentally fill the counts\n", "# multiple times when executing the ``fill_by_coord`` multiple times.\n", "counts.fill_by_coord(\n", " {\n", " \"skycoord\": events.radec,\n", " # The coord-based interface doesn't use Quantity,\n", " # so we need to pass energy in the same unit as\n", " # used for the map axis\n", " \"energy\": events.energy,\n", " }\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "counts.geom.axes[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "counts.sum_over_axes().smooth(2).plot(stretch=\"sqrt\", vmax=30);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exposure\n", "\n", "The Fermi-LAT datatset contains the energy-dependent exposure for the whole sky as a HEALPix map computed with ``gtexpcube2``. This format is supported by ``gammapy.maps`` directly.\n", "\n", "Interpolating the exposure cube from the Fermi ST to get an exposure cube matching the spatial geometry and energy axis defined above with Gammapy is easy. The only point to watch out for is how exactly you want the energy axis and binning handled.\n", "\n", "Below we just use the default behaviour, which is linear interpolation in energy on the original exposure cube. Probably log interpolation would be better, but it doesn't matter much here, because the energy binning is fine. Finally, we just copy the counts map geometry, which contains an energy axis with `node_type=\"edges\"`. This is non-ideal for exposure cubes, but again, acceptable because exposure doesn't vary much from bin to bin, so the exact way interpolation occurs in later use of that exposure cube doesn't matter a lot. Of course you could define any energy axis for your exposure cube that you like." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "exposure_hpx = Map.read(\n", " \"$GAMMAPY_DATA/fermi_3fhl/fermi_3fhl_exposure_cube_hpx.fits.gz\"\n", ")\n", "# Unit is not stored in the file, set it manually\n", "exposure_hpx.unit = \"cm2 s\"\n", "exposure_hpx.geom.axes[0].unit = \"GeV\"\n", "print(exposure_hpx.geom)\n", "print(exposure_hpx.geom.axes[0])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "exposure_hpx.plot();" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# For exposure, we choose a geometry with node_type='center',\n", "# whereas for counts it was node_type='edge'\n", "axis = MapAxis.from_nodes(\n", " counts.geom.axes[0].center, name=\"energy\", unit=\"GeV\", interp=\"log\"\n", ")\n", "geom = WcsGeom(wcs=counts.geom.wcs, npix=counts.geom.npix, axes=[axis])\n", "\n", "coord = counts.geom.get_coord()\n", "data = exposure_hpx.interp_by_coord(coord)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "exposure = WcsNDMap(geom, data, unit=exposure_hpx.unit, dtype=float)\n", "print(exposure.geom)\n", "print(exposure.geom.axes[0])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Exposure is almost constant accross the field of view\n", "exposure.slice_by_idx({\"energy\": 0}).plot(add_cbar=True);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Exposure varies very little with energy at these high energies\n", "energy = [10, 100, 1000] * u.GeV\n", "exposure.get_by_coord({\"skycoord\": gc_pos, \"energy\": energy})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Galactic diffuse background" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Fermi-LAT collaboration provides a galactic diffuse emission model, that can be used as a background model for\n", "Fermi-LAT source analysis.\n", "\n", "Diffuse model maps are very large (100s of MB), so as an example here, we just load one that represents a small cutout for the Galactic center region." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "diffuse_galactic_fermi = Map.read(\n", " \"$GAMMAPY_DATA/fermi_3fhl/gll_iem_v06_cutout.fits\"\n", ")\n", "# Unit is not stored in the file, set it manually\n", "diffuse_galactic_fermi.unit = \"cm-2 s-1 MeV-1 sr-1\"\n", "print(diffuse_galactic_fermi.geom)\n", "print(diffuse_galactic_fermi.geom.axes[0])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Interpolate the diffuse emission model onto the counts geometry\n", "# The resolution of `diffuse_galactic_fermi` is low: bin size = 0.5 deg\n", "# We use ``interp=3`` which means cubic spline interpolation\n", "coord = counts.geom.get_coord()\n", "\n", "data = diffuse_galactic_fermi.interp_by_coord(\n", " {\n", " \"skycoord\": coord.skycoord,\n", " \"energy\": coord[\"energy\"]\n", " },\n", " interp=3,\n", ")\n", "diffuse_galactic = WcsNDMap(\n", " exposure.geom, data, unit=diffuse_galactic_fermi.unit\n", ")\n", "\n", "print(diffuse_galactic.geom)\n", "print(diffuse_galactic.geom.axes[0])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "diffuse_galactic.slice_by_idx({\"energy\": 0}).plot(add_cbar=True);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Exposure varies very little with energy at these high energies\n", "energy = np.logspace(1, 3, 10) * u.GeV\n", "dnde = diffuse_galactic.interp_by_coord(\n", " {\"skycoord\": gc_pos, \"energy\": energy}, interp=\"linear\", fill_value=None\n", ")\n", "plt.plot(energy.value, dnde, \"*\")\n", "plt.loglog()\n", "plt.xlabel(\"Energy (GeV)\")\n", "plt.ylabel(\"Flux (cm-2 s-1 MeV-1 sr-1)\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# TODO: show how one can fix the extrapolate to high energy\n", "# by computing and padding an extra plane e.g. at 1e3 TeV\n", "# that corresponds to a linear extrapolation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Isotropic diffuse background\n", "\n", "To load the isotropic diffuse model with Gammapy, use the [gammapy.spectrum.models.TableModel](https://docs.gammapy.org/dev/api/gammapy.spectrum.models.TableModel.html). We are using `'fill_value': 'extrapolate'` to extrapolate the model above 500 GeV:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "filename = \"$GAMMAPY_DATA/fermi_3fhl/iso_P8R2_SOURCE_V6_v06.txt\"\n", "interp_kwargs = {\"fill_value\": None}\n", "diffuse_iso = TableModel.read_fermi_isotropic_model(\n", " filename=filename, interp_kwargs=interp_kwargs\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can plot the model in the energy range between 50 GeV and 2000 GeV:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "erange = [50, 2000] * u.GeV\n", "diffuse_iso.plot(erange, flux_unit=\"1 / (cm2 MeV s sr)\");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## PSF\n", "\n", "Next we will tke a look at the PSF. It was computed using ``gtpsf``, in this case for the Galactic center position. Note that generally for Fermi-LAT, the PSF only varies little within a given regions of the sky, especially at high energies like what we have here. We use the [gammapy.irf.EnergyDependentTablePSF](https://docs.gammapy.org/dev/api/gammapy.irf.EnergyDependentTablePSF.html) class to load the PSF and use some of it's methods to get some information about it." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "psf = EnergyDependentTablePSF.read(\n", " \"$GAMMAPY_DATA/fermi_3fhl/fermi_3fhl_psf_gc.fits.gz\"\n", ")\n", "print(psf)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get an idea of the size of the PSF we check how the containment radii of the Fermi-LAT PSF vari with energy and different containment fractions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.figure(figsize=(8, 5))\n", "psf.plot_containment_vs_energy(linewidth=2, fractions=[0.68, 0.95])\n", "plt.xlim(50, 2000)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In addition we can check how the actual shape of the PSF varies with energy and compare it against the mean PSF between 50 GeV and 2000 GeV:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.figure(figsize=(8, 5))\n", "\n", "for energy in [100, 300, 1000] * u.GeV:\n", " psf_at_energy = psf.table_psf_at_energy(energy)\n", " psf_at_energy.plot_psf_vs_rad(label=\"PSF @ {:.0f}\".format(energy), lw=2)\n", "\n", "erange = [50, 2000] * u.GeV\n", "spectrum = PowerLaw(index=2.3)\n", "psf_mean = psf.table_psf_in_energy_band(energy_band=erange, spectrum=spectrum)\n", "psf_mean.plot_psf_vs_rad(label=\"PSF Mean\", lw=4, c=\"k\", ls=\"--\")\n", "\n", "plt.xlim(1e-3, 0.3)\n", "plt.ylim(1e3, 1e6)\n", "plt.legend();" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Let's compute a PSF kernel matching the pixel size of our map\n", "psf_kernel = PSFKernel.from_table_psf(psf, counts.geom, max_radius=\"1 deg\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "psf_kernel.psf_kernel_map.sum_over_axes().plot(stretch=\"log\", add_cbar=True);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Energy Dispersion\n", "For simplicity we assume a diagonal energy dispersion:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "e_true = exposure.geom.axes[0].edges\n", "e_reco = counts.geom.axes[0].edges\n", "edisp = EnergyDispersion.from_diagonal_response(e_true=e_true, e_reco=e_reco)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Background\n", "\n", "Let's compute a background cube, with predicted number of background events per pixel from the diffuse Galactic and isotropic model components. For this, we use the use the [gammapy.cube.MapEvaluator](https://docs.gammapy.org/dev/api/gammapy.cube.MapEvaluator.html) to multiply with the exposure and apply the PSF. The Fermi-LAT energy dispersion at high energies is small, we neglect it here." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_diffuse = SkyDiffuseCube(diffuse_galactic, name=\"diffuse\")\n", "eval_diffuse = MapEvaluator(model=model_diffuse, exposure=exposure, psf=psf_kernel, edisp=edisp)\n", "\n", "background_gal = eval_diffuse.compute_npred()\n", "\n", "background_gal.sum_over_axes().plot()\n", "print(\n", " \"Background counts from Galactic diffuse: \", background_gal.data.sum()\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_iso = SkyModel(SkyDiffuseConstant(), diffuse_iso, name=\"diffuse-iso\")\n", "\n", "eval_iso = MapEvaluator(model=model_iso, exposure=exposure, edisp=edisp)\n", "\n", "background_iso = eval_iso.compute_npred()\n", "\n", "background_iso.sum_over_axes().plot(add_cbar=True)\n", "print(\n", " \"Background counts from isotropic diffuse: \", background_iso.data.sum()\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "background_total = background_iso + background_gal" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Excess and flux\n", "\n", "Let's compute an excess and flux image, by subtracting the background, and summing over the energy axis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "excess = counts.copy()\n", "excess.data -= background_total.data\n", "excess.sum_over_axes().smooth(\"0.1 deg\").plot(\n", " cmap=\"coolwarm\", vmin=-5, vmax=5, add_cbar=True\n", ")\n", "print(\"Excess counts: \", excess.data.sum())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "flux = excess.copy()\n", "flux.data /= exposure.data\n", "flux.unit = excess.unit / exposure.unit\n", "flux.sum_over_axes().smooth(\"0.1 deg\").plot(stretch=\"sqrt\", add_cbar=True);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fit\n", "\n", "Finally, the big finale: let's do a 3D map fit for the source at the Galactic center, to measure it's position and spectrum. We keep the background normalisation free." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model = SkyModel(\n", " SkyPointSource(\"0 deg\", \"0 deg\"),\n", " PowerLaw(index=2.5, amplitude=\"1e-11 cm-2 s-1 TeV-1\", reference=\"100 GeV\"),\n", ")\n", "\n", "model_total = SkyModels([model, model_diffuse, model_iso])\n", "\n", "dataset = MapDataset(\n", " model=model_total,\n", " counts=counts,\n", " exposure=exposure,\n", " psf=psf_kernel,\n", ")\n", "fit = Fit(dataset)\n", "result = fit.run()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(result)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dataset.parameters.to_table()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "residual = counts - dataset.npred()\n", "residual.sum_over_axes().smooth(\"0.1 deg\").plot(\n", " cmap=\"coolwarm\", vmin=-3, vmax=3, add_cbar=True\n", ");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercises\n", "\n", "- Fit the position and spectrum of the source [SNR G0.9+0.1](http://gamma-sky.net/#/cat/tev/110).\n", "- Make maps and fit the position and spectrum of the [Crab nebula](http://gamma-sky.net/#/cat/tev/25)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "In this tutorial you have seen how to work with Fermi-LAT data with Gammapy. You have to use the Fermi ST to prepare the exposure cube and PSF, and then you can use Gammapy for any event or map analysis using the same methods that are used to analyse IACT data.\n", "\n", "This works very well at high energies (here above 10 GeV), where the exposure and PSF is almost constant spatially and only varies a little with energy. It is not expected to give good results for low-energy data, where the Fermi-LAT PSF is very large. If you are interested to help us validate down to what energy Fermi-LAT analysis with Gammapy works well (e.g. by re-computing results from 3FHL or other published analysis results), or to extend the Gammapy capabilities (e.g. to work with energy-dependent multi-resolution maps and PSF), that would be very welcome!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0" }, "toc": { "base_numbering": 1, "nav_menu": { "height": "237px", "width": "253px" }, "number_sections": false, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": "block", "toc_window_display": false }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }