{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "bYYGJvqP-Csl" }, "source": [ "# Removing scattered light from *TESS* light curves using linear regression (`RegressionCorrector`)" ] }, { "cell_type": "markdown", "metadata": { "id": "gOFEJAKe-vjA" }, "source": [ "## Learning Goals\n", "\n", "By the end of this tutorial, you will:\n", " - Be familiar with the Lightkurve [RegressionCorrector](https://docs.lightkurve.org/reference/api/lightkurve.correctors.RegressionCorrector.html?highlight=regressioncorrector).\n", " - Understand how to create regressors from a [TargetPixelFile](https://docs.lightkurve.org/reference/targetpixelfile.html?highlight=targetpixelfile) object.\n", " - Be able to remove the scattered light background signal from *TESS* data." ] }, { "cell_type": "markdown", "metadata": { "id": "7NGNkiQB-sKQ" }, "source": [ "## Introduction\n", "\n", "Lightkurve offers several tools to the community for removing instrument noise and systematics from data from the *Kepler*, *K2*, and *TESS* missions. This tutorial will demonstrate the use of Lightkurve's `RegressionCorrector` class to remove the scattered light and spacecraft motion noise from [*TESS* Full Frame Images (FFIs)](https://heasarc.gsfc.nasa.gov/docs/tess/data-products.html#full-frame-images).\n", "\n", "*TESS* FFIs have an additive scattered light background that has not been removed by the pipeline. This scattered light must be removed by the user. This can be done in a few ways, including a basic median subtraction. In this tutorial, we'll show you how to use Lightkurve's corrector tools to remove the scattered light. " ] }, { "cell_type": "markdown", "metadata": { "id": "EAzRgQPpAcBR" }, "source": [ "## Imports\n", "\n", "This tutorial requires the [**Lightkurve**](http://docs.lightkurve.org/) package, and also makes use of **[NumPy](https://numpy.org/)** and **[Matplotlib](https://matplotlib.org/)**." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "azlcSd2lAjhy" }, "outputs": [], "source": [ "import lightkurve as lk\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": { "id": "a53oRxa12SI7" }, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": { "id": "c03Y0DPv-Csl" }, "source": [ "## 1. Using `RegressionCorrector` on TESSCut FFI Cutouts\n", "\n", "For this tutorial we will use the *TESS* Sector 15 data of [KIC 8462852](https://en.wikipedia.org/wiki/Tabby%27s_Star) (also known as Boyajian's Star).\n", "We'll start by downloading the FFI data using MAST's TESSCut service, querying it through Lightkurve." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "X9i17V-K-Csp" }, "outputs": [], "source": [ "target = \"KIC 8462852\" # Boyajian's Star\n", "tpf = lk.search_tesscut(target, sector=15).download(cutout_size=(50, 50))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "executionInfo": { "elapsed": 21563, "status": "ok", "timestamp": 1601341100142, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "bCe-AaDr-Csr", "outputId": "de3a9f2c-5a31-4411-ec28-c0a35697cc33" }, "outputs": [], "source": [ "tpf" ] }, { "cell_type": "markdown", "metadata": { "id": "DU3iTJ03-Csu" }, "source": [ "This cutout works the same as any Lightkurve target pixel file (TPF). *TESS* FFI cutouts do not have aperture masks created by the pipeline. Instead, users must create their own apertures. There are many methods we could use to do this, but for now we can create a threshold aperture, using Lightkurve's [create_threshold_mask()](https://docs.lightkurve.org/reference/api/lightkurve.KeplerTargetPixelFile.create_threshold_mask.html?highlight=create_threshold_mask#lightkurve.KeplerTargetPixelFile.create_threshold_mask) method." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Vanh1KjQ-Csu" }, "outputs": [], "source": [ "aper = tpf.create_threshold_mask()" ] }, { "cell_type": "markdown", "metadata": { "id": "6fzg_SI2-Csw" }, "source": [ "Let's plot the aperture to make sure it selected the star in the center and has a reasonable number of pixels." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 420 }, "executionInfo": { "elapsed": 1762, "status": "ok", "timestamp": 1601341300620, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "qhTr4hF8-Csx", "outputId": "35efec0a-ea85-474a-ebb8-d10788bfb376" }, "outputs": [], "source": [ "tpf.plot(aperture_mask=aper);" ] }, { "cell_type": "markdown", "metadata": { "id": "L7HKo9k--Csz" }, "source": [ "Looks good. We can sum up the pixels in this aperture, and create an uncorrected light curve." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "eMMxpcQJ-Csz" }, "outputs": [], "source": [ "uncorrected_lc = tpf.to_lightcurve(aperture_mask=aper)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 1010, "status": "ok", "timestamp": 1601341307101, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "c77kQm4a-Cs2", "outputId": "f2898b18-e7f4-4c62-d16c-ea554228f233" }, "outputs": [], "source": [ "uncorrected_lc.plot();" ] }, { "cell_type": "markdown", "metadata": { "id": "HcMtgqnyJDnC" }, "source": [ "## 2. Creating a `DesignMatrix` from Pixel Regressors" ] }, { "cell_type": "markdown", "metadata": { "id": "XCivwPyP-Cs4" }, "source": [ "The flux in the aperture appears to be dominated by scattered light. We can tell because *TESS* orbits Earth twice in each sector, thus patterns which appear twice within a sector are typically related to the *TESS* orbit (such as the scattered light effect).\n", "\n", "To remove this light, we are going to detrend the light curve against some vectors which we think are predictive of this systematic noise.\n", "\n", "In this case, we can use the **pixels outside the aperture** as vectors that are highly predictive of the systematic noise, that is, we will make the assumption that these pixels do not contain any flux from our target.\n", "\n", "We can select these pixels by specifying flux outside of the aperture using Python's bitwise invert operator `~` to take the inverse of the aperture mask." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "1aXsrIix-Cs5" }, "outputs": [], "source": [ "regressors = tpf.flux[:, ~aper]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "executionInfo": { "elapsed": 279, "status": "ok", "timestamp": 1601341310383, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "2vv7gYfL-Cs6", "outputId": "a9b048f7-2542-4016-b130-af5b1f4bea56" }, "outputs": [], "source": [ "regressors.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "OcA2bJKk-Cs8" }, "source": [ "`regressors` is now an array with shape *ntime* x *npixels outside of the aperture*. If we plot the first 30 of these pixels, we can see that they contain mostly scattered light, with some offset terms." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 265 }, "executionInfo": { "elapsed": 663, "status": "ok", "timestamp": 1601341312851, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "s29KGgyI-Cs9", "outputId": "b6b44254-4e45-42a3-9424-28744ac6c229" }, "outputs": [], "source": [ "plt.plot(regressors[:, :30]);" ] }, { "cell_type": "markdown", "metadata": { "id": "LsENQCtA-Cs_" }, "source": [ "In linear regression problems, it is common to refer to the matrix of regressors as the design matrix (also known as model matrix or regressor matrix). Lightkurve provides a convenient `DesignMatrix` class which is designed to help you work with detrending vectors. \n", "\n", "The [DesignMatrix](https://docs.lightkurve.org/reference/api/lightkurve.correctors.DesignMatrix.html?highlight=designmatrix#lightkurve.correctors.DesignMatrix) class has several convenience functions, and can be passed into Lightkurve's corrector objects. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "am6YyH2K-Cs_" }, "outputs": [], "source": [ "from lightkurve.correctors import DesignMatrix\n", "dm = DesignMatrix(regressors, name='regressors')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "executionInfo": { "elapsed": 456, "status": "ok", "timestamp": 1601341318263, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "9vPETGCO-CtB", "outputId": "b70f4c15-c378-46e6-e4a7-92813daef337" }, "outputs": [], "source": [ "dm" ] }, { "cell_type": "markdown", "metadata": { "id": "1jVWgySe-CtD" }, "source": [ "As shown above, `dm` is now a design matrix with the same shape as the input pixels. Currently, we have 2,541 pixels that we are using to detrend our light curve against. Rather than using all of the pixels, we can reduce these to their principal components using Principal Component Analysis (PCA). We do this for several reasons:\n", "\n", "1. By reducing to a smaller number of vectors, we can remove some of the stochastic noise in our detrending vectors.\n", "2. By reducing to the principal components, we can avoid pixels that have intrinsic variability (for example, from astrophysical long-period variables) that can be confused with the true astrophysical signal of our target.\n", "3. By reducing the number of vectors, our detrending will be faster (although in this case, the detrending will still take seconds).\n", "\n", "The choice of the number of components is a tricky issue, but in general you should choose a number that is much smaller than the number of vectors." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "qr2wexvz-CtD" }, "outputs": [], "source": [ "dm = dm.pca(5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "executionInfo": { "elapsed": 430, "status": "ok", "timestamp": 1601341321863, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "nACVUHBg-CtF", "outputId": "dd7cc1e6-e3d1-4c7b-b11e-a5ca1d7d4a8f" }, "outputs": [], "source": [ "dm" ] }, { "cell_type": "markdown", "metadata": { "id": "QJd-0Kf3-CtH" }, "source": [ "Using the `pca()` method, we have now reduced the number of components in our design matrix to five. These vectors show a combination of scattered light and spacecraft motion, which makes them suited to detrend our input light curve." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 265 }, "executionInfo": { "elapsed": 421, "status": "ok", "timestamp": 1601341323931, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "676s7Unp-CtI", "outputId": "0829d0ae-a1e7-43d2-a70b-d3c98ec33487" }, "outputs": [], "source": [ "plt.plot(tpf.time.value, dm.values + np.arange(5)*0.2, '.');" ] }, { "cell_type": "markdown", "metadata": { "id": "zOOPOhn3-CtJ" }, "source": [ "Note: the `DesignMatrix` object provides a convenient `plot()` method to visualize the vectors:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 410 }, "executionInfo": { "elapsed": 917, "status": "ok", "timestamp": 1601341326386, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "UNqudAAJ-CtK", "outputId": "2ed57c05-bbe6-45e9-8b7b-72477575d5d0" }, "outputs": [], "source": [ "dm.plot();" ] }, { "cell_type": "markdown", "metadata": { "id": "EVmE9d6r-CtM" }, "source": [ "We can now detrend the uncorrected light curve against these vectors. Lightkurve's [RegressionCorrector](https://docs.lightkurve.org/reference/api/lightkurve.correctors.RegressionCorrector.html?highlight=regressioncorrector#lightkurve.correctors.RegressionCorrector) will use linear algebra to find the combination of vectors that makes the input light curve **closest to zero**. To do this, we need one more component; we need an \"offset\" term, to be able to fit the mean level of the light curve. We can do this by appending a \"constant\" to our design matrix." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "VjxyTIHW-CtM" }, "outputs": [], "source": [ "dm = dm.append_constant()" ] }, { "cell_type": "markdown", "metadata": { "id": "6qYkny8tJU2f" }, "source": [ "## 3. Removing Background Scattered Light Using Linear Regression" ] }, { "cell_type": "markdown", "metadata": { "id": "l5aQXZjw-CtR" }, "source": [ "Now that we have a design matrix, we only need to pass it into a [lightkurve.Corrector](https://docs.lightkurve.org/reference/api/lightkurve.correctors.corrector.Corrector.html?highlight=lightkurve%20corrector#lightkurve.correctors.corrector.Corrector). To use our design matrix, we can pass it to the [RegressionCorrector](https://docs.lightkurve.org/reference/api/lightkurve.correctors.RegressionCorrector.html?highlight=regressioncorrector), which will detrend the input light curve against the vectors we've built." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "gKiAq33b-CtR" }, "outputs": [], "source": [ "from lightkurve.correctors import RegressionCorrector\n", "corrector = RegressionCorrector(uncorrected_lc)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "executionInfo": { "elapsed": 383, "status": "ok", "timestamp": 1601341331785, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "22-ftOZX-CtT", "outputId": "13a74208-dbc3-4395-9125-03cd2da955af" }, "outputs": [], "source": [ "corrector" ] }, { "cell_type": "markdown", "metadata": { "id": "tcTlNruW-CtV" }, "source": [ "To correct the light curve, we pass in our design matrix." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "DZJLhAsL-CtW" }, "outputs": [], "source": [ "corrected_lc = corrector.correct(dm)" ] }, { "cell_type": "markdown", "metadata": { "id": "f6v_OWxe-CtY" }, "source": [ "Now we can plot the results:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 951, "status": "ok", "timestamp": 1601341335862, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "H1zoqGnW-CtY", "outputId": "fee676f5-aa50-4f46-a8a8-dbf94d5b8823" }, "outputs": [], "source": [ "ax = uncorrected_lc.plot(label='Original light curve')\n", "corrected_lc.plot(ax=ax, label='Corrected light curve');" ] }, { "cell_type": "markdown", "metadata": { "id": "wJPoKJop-Cta" }, "source": [ "As shown above, the scattered light from the background has been removed. If we want to take a more in-depth look at the correction, we can use the [diagnose()](https://docs.lightkurve.org/reference/api/lightkurve.correctors.RegressionCorrector.diagnose.html?highlight=diagnose#lightkurve.correctors.RegressionCorrector.diagnose) method to see what the `RegressionCorrector` found as the best fitting solution." ] }, { "cell_type": "markdown", "metadata": { "id": "bl_TGn7DJmvy" }, "source": [ "## 4. Diagnosing the Correction" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 538 }, "executionInfo": { "elapsed": 1586, "status": "ok", "timestamp": 1601341339709, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "_Pav_N1y-Cta", "outputId": "7676cb49-2c9a-4f18-87f2-13cc11abdf31" }, "outputs": [], "source": [ "corrector.diagnose();" ] }, { "cell_type": "markdown", "metadata": { "id": "9Boc_bF0-Ctc" }, "source": [ "The `RegressionCorrector` has clipped out some outliers during the fit of the trend. You can read more about the outlier removal, how to pass a cadence mask, and error propagation in the [docs](https://docs.lightkurve.org/reference/api/lightkurve.correctors.RegressionCorrector.html?highlight=regressioncorrector)." ] }, { "cell_type": "markdown", "metadata": { "id": "tjbwIoe--Ctc" }, "source": [ "**Watch Out!**\n", "\n", "The `RegressionCorrector` assumes that you want to remove the trend and set the light curve to the **mean** level of the **uncorrected light curve**. This isn't true for *TESS* scattered light. *TESS* FFI light curves have **additive background**, and so we want to reduce the flux to the lowest recorded level, assuming that at that point the contribution from scattered light is approximately zero.\n", "\n", "To do this, we will first need to look at the model of the background that `RegressionCorrector` built. We can access that in the `corrector` object." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 519 }, "executionInfo": { "elapsed": 400, "status": "ok", "timestamp": 1601341343056, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "z7rPM0cg-Ctd", "outputId": "d75d18df-608f-4900-acee-b60824a522d8" }, "outputs": [], "source": [ "corrector.model_lc" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 1028, "status": "ok", "timestamp": 1601341344768, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "y4SwUwcu-Ctf", "outputId": "b1b596bf-64d4-4c23-8a8e-5c756f3814ef" }, "outputs": [], "source": [ "model = corrector.model_lc\n", "model.plot();" ] }, { "cell_type": "markdown", "metadata": { "id": "8GYiVNZ7-Ctg" }, "source": [ "As you can see above, the model drops below zero flux. This is impossible; the scattered light can't be removing flux from our target! \n", "\n", "To rectify this, we can subtract the model flux value at the 5th percentile." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "VglBeToo-Cth" }, "outputs": [], "source": [ "# Normalize to the 5th percentile of model flux\n", "model -= np.percentile(model.flux, 5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 869, "status": "ok", "timestamp": 1601341348908, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "gJ13M9VH-Cti", "outputId": "ea9a0d96-d24b-43e2-ec87-e9db7b1696fd" }, "outputs": [], "source": [ "model.plot();" ] }, { "cell_type": "markdown", "metadata": { "id": "-F6cejiT-Ctk" }, "source": [ "This looks better. Now we can remove this model from our uncorrected light curve." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "wS-dvhF5-Ctk" }, "outputs": [], "source": [ "corrected_lc = uncorrected_lc - model" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 1132, "status": "ok", "timestamp": 1601341352869, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "04rj9Ecu-Ctm", "outputId": "fe034baa-4e7e-4f17-a1ce-b70ab95347f3" }, "outputs": [], "source": [ "ax = uncorrected_lc.plot(label='Original light curve')\n", "corrected_lc.plot(ax=ax, label='Corrected light curve');" ] }, { "cell_type": "markdown", "metadata": { "id": "cVInnt_9-Cto" }, "source": [ "This looks great. As a final test, let's investigate how the light curve we obtained using `RegressionCorrector` compares against a light curve obtained using a more basic median background removal method." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 1067, "status": "ok", "timestamp": 1601341355285, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "iBL2ZsL1-Cto", "outputId": "749bb113-f989-42cb-dfb6-7a31e7be5df5" }, "outputs": [], "source": [ "bkg = np.median(regressors, axis=1)\n", "bkg -= np.percentile(bkg, 5)\n", "\n", "npix = aper.sum()\n", "median_subtracted_lc = uncorrected_lc - npix * bkg\n", "\n", "ax = median_subtracted_lc.plot(label='Median background subtraction')\n", "corrected_lc.plot(ax=ax, label='RegressionCorrector');" ] }, { "cell_type": "markdown", "metadata": { "id": "VvE4PU8K-Ctq" }, "source": [ "Lastly, let's show how you can do all of the above in a single cell." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "WriD6CBJ-Ctq" }, "outputs": [], "source": [ "# Make an aperture mask and an uncorrected light curve\n", "aper = tpf.create_threshold_mask()\n", "uncorrected_lc = tpf.to_lightcurve(aperture_mask=aper)\n", "\n", "# Make a design matrix and pass it to a linear regression corrector\n", "dm = DesignMatrix(tpf.flux[:, ~aper], name='regressors').pca(5).append_constant()\n", "rc = RegressionCorrector(uncorrected_lc)\n", "corrected_ffi_lc = rc.correct(dm)\n", "\n", "# Optional: Remove the scattered light, allowing for the large offset from scattered light\n", "corrected_ffi_lc = uncorrected_lc - rc.model_lc + np.percentile(rc.model_lc.flux, 5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 916, "status": "ok", "timestamp": 1601341365642, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "ateW8trM-Cts", "outputId": "d148e4aa-72d1-411a-81b5-3e105e13973f" }, "outputs": [], "source": [ "ax = uncorrected_lc.plot(label='Original light curve')\n", "corrected_ffi_lc.plot(ax=ax, label='Corrected light curve');" ] }, { "cell_type": "markdown", "metadata": { "id": "oBGQPJnx-Ctu" }, "source": [ "## 5. Using `RegressionCorrector` on *TESS* Two-Minute Cadence Target Pixel Files\n", "\n", "*TESS* releases high-time resolution TPFs of interesting targets. These higher time resolution TPFs have background removed for users by the pipeline. However, there are still common trends in TPF pixels that are not due to scattered light, but could be from, for example, spacecraft motion.\n", "\n", "`RegressionCorrector` can be used in exactly the same way to remove these common trends." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "zExzJuMk-Ctu" }, "outputs": [], "source": [ "# Download a 2-minute cadence Target Pixel File (TPF)\n", "tpf_2min = lk.search_targetpixelfile(target, author='SPOC', cadence=120, sector=15).download()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "executionInfo": { "elapsed": 10343, "status": "ok", "timestamp": 1601341381938, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "PLktwfqZ-Ctw", "outputId": "ca93195c-64c9-42f4-93f2-e001333cb58d", "scrolled": true }, "outputs": [], "source": [ "tpf_2min" ] }, { "cell_type": "markdown", "metadata": { "id": "xXAa6wHT-Cty" }, "source": [ "Note, unlike the FFI data, the TPF has been processed by the SPOC pipeline, and includes an aperture mask." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "FV0hq9IN-Cty" }, "outputs": [], "source": [ "# Use the pipeline aperture and an uncorrected light curve\n", "aper = tpf_2min.pipeline_mask\n", "uncorrected_lc = tpf_2min.to_lightcurve()\n", "\n", "# Make a design matrix\n", "dm = DesignMatrix(tpf_2min.flux[:, ~aper], name='pixels').pca(5).append_constant()\n", "\n", "# Regression Corrector Object\n", "reg = RegressionCorrector(uncorrected_lc)\n", "corrected_lc = reg.correct(dm)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 1747, "status": "ok", "timestamp": 1601341412387, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "Nip1bpKA-Ct0", "outputId": "bb55568a-13e8-4487-dc9f-678885d95d99" }, "outputs": [], "source": [ "ax = uncorrected_lc.errorbar(label='Original light curve')\n", "corrected_lc.errorbar(ax=ax, label='Corrected light curve');" ] }, { "cell_type": "markdown", "metadata": { "id": "PQqpVvn6-Ct2" }, "source": [ "As you can see, the corrected light curve has removed long-term trends and some motion noise, for example, see time around 1720 Barycentric *TESS* Julian Date (BTJD). We can use the same `diagnose()` method to understand the model that has been fit and subtracted by `RegressionCorrector`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 538 }, "executionInfo": { "elapsed": 1962, "status": "ok", "timestamp": 1601341416716, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "Q3rLiGJA-Ct3", "outputId": "89bc9cd1-27c2-4150-8003-56cd65fefb50" }, "outputs": [], "source": [ "reg.diagnose();" ] }, { "cell_type": "markdown", "metadata": { "id": "eY_K5YDH-Ct6" }, "source": [ "To show the corrected version has improved, we can use the Combined Differential Photometric Precision (CDPP) metric. As shown below, the corrected light curve has a lower CDPP, showing it is less noisy." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 37 }, "executionInfo": { "elapsed": 389, "status": "ok", "timestamp": 1601341418995, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "d5BmzAl4-Ct7", "outputId": "f12a2eca-9674-4a5d-c582-b047855276c8" }, "outputs": [], "source": [ "uncorrected_lc.estimate_cdpp()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 37 }, "executionInfo": { "elapsed": 368, "status": "ok", "timestamp": 1601341420061, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "mgtE24fn-Ct9", "outputId": "6b498080-9f3f-43db-a3e7-3c1306af90df" }, "outputs": [], "source": [ "corrected_lc.estimate_cdpp()" ] }, { "cell_type": "markdown", "metadata": { "id": "zbTGkfTQRUAe" }, "source": [ "## 6. Should I use `RegressionCorrector` or `PLDCorrector`?\n", "\n", "In addition to the corrector demonstrated in this tutorial, Lightkurve has a special case of `RegressionCorrector` called [PLDCorrector](https://docs.lightkurve.org/reference/api/lightkurve.correctors.PLDCorrector.html?highlight=pldcorrector). PLD, or Pixel Level Decorrelation, is a method of removing systematic noise from light curves using linear regression, with a design matrix constructed from a combination of pixel-level light curves. \n", "\n", "For more information about the `PLDCorrector`, please see the tutorial specifically on removing instrumental noise from *K2* and *TESS* light curves using PLD.\n", "\n", "For *TESS*, the `PLDCorrector` works in a very similar way to the `RegressionCorrector`. The major difference between them is that `PLDCorrector` constructs its own design matrix, making it a streamlined, versatile tool to apply to any *TESS* or *K2* light curve.\n", "\n", "Here, we perform PLD and diagnose the correction in just three lines. To make a more direct comparison to the `RegressionCorrector`, we pass in arguments to set the number of components to five (as in section 2), as well as remove the spline fit." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 764 }, "executionInfo": { "elapsed": 3516, "status": "ok", "timestamp": 1601341427208, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "exFLkUmFKBLx", "outputId": "4ae28425-1b30-4071-f82c-f911c578e85d" }, "outputs": [], "source": [ "from lightkurve.correctors import PLDCorrector\n", "pld = PLDCorrector(tpf)\n", "pld_corrected_lc = pld.correct(restore_trend=False, pca_components=5)\n", "pld.diagnose();" ] }, { "cell_type": "markdown", "metadata": { "id": "gbdhv908S1If" }, "source": [ "And there we go!\n", "\n", "Now let's compare the performance of these two corrections." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 387 }, "executionInfo": { "elapsed": 1139, "status": "ok", "timestamp": 1601341433754, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "ODE2wR6_QC52", "outputId": "5f75900a-6eff-4725-ddda-95802d0f8deb" }, "outputs": [], "source": [ "ax = corrected_ffi_lc.normalize().plot(label='RegressionCorrector')\n", "pld_corrected_lc.normalize().plot(label='PLDCorrector', ax=ax);" ] }, { "cell_type": "markdown", "metadata": { "id": "M_G2hNyQTfej" }, "source": [ "`PLDCorrector` offers an additional diagnostic plot, named [diagnose_masks](https://docs.lightkurve.org/reference/api/lightkurve.correctors.PLDCorrector.diagnose_masks.html?highlight=diagnose_masks#lightkurve.correctors.PLDCorrector.diagnose_masks). This allows you to inspect the pixels that were used to create your design matrix." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 338 }, "executionInfo": { "elapsed": 5992, "status": "ok", "timestamp": 1601341446372, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "owF7pU9RKtkE", "outputId": "746b78f7-a95f-49cc-bade-75dfe4dfd001" }, "outputs": [], "source": [ "pld.diagnose_masks();" ] }, { "cell_type": "markdown", "metadata": { "id": "WMJJIRYYT0Zl" }, "source": [ "While it is more convenient to apply to light curves and generally works well with default parameters, the `PLDCorrector` is less flexible than the `RegressionCorrector`, which allows you to create your own custom design matrix. However, the `PLDCorrector` also allows you to create \"higher order\" PLD regressors by taking the products of existing pixel regressors, which improves the performance of corrections to *K2* data (see the paper by [Luger et al. 2016](https://arxiv.org/abs/1607.00524) for more information).\n", "\n", "When considering which corrector to use, remember that `PLDCorrector` is minimal and designed to be effective at removing both background scattered light from *TESS* and motion noise from *K2*, while `RegressionCorrector` is flexible and gives you more control over the creation of the design matrix and the correction." ] }, { "cell_type": "markdown", "metadata": { "id": "aG4fgysD_Brp" }, "source": [ "## About this Notebook\n", "\n", "**Authors:** Christina Hedges (christinalouisehedges@gmail.com), Nicholas Saunders (nksaun@hawaii.edu), Geert Barentsen\n", "\n", "**Updated On:** 2020-09-28" ] }, { "cell_type": "markdown", "metadata": { "id": "MptPsdBQ_Qju" }, "source": [ "## Citing Lightkurve and its Dependencies\n", "\n", "If you use `lightkurve` or its dependencies for published research, please cite the authors. Click the buttons below to copy BibTeX entries to your clipboard." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 144 }, "executionInfo": { "elapsed": 519, "status": "ok", "timestamp": 1601341461397, "user": { "displayName": "Geert Barentsen", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64", "userId": "05704237875861987058" }, "user_tz": 420 }, "id": "1khvwNHx_QDz", "outputId": "5f2928fe-5150-4696-86e2-852097b54f56" }, "outputs": [], "source": [ "lk.show_citation_instructions()" ] }, { "cell_type": "markdown", "metadata": { "id": "ZhdRVU3B_Zn2" }, "source": [ "\"Space" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "Removing scattered light from TESS data using the Lightkurve RegressionCorrector", "provenance": [ { "file_id": "https://github.com/KeplerGO/lightkurve/blob/master/docs/source/tutorials/04-how-to-remove-tess-scattered-light-using-regressioncorrector.ipynb", "timestamp": 1599958717967 } ] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10" } }, "nbformat": 4, "nbformat_minor": 4 }