{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Find DM Stack Documentation\n",
    "\n",
    "<br>Author(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
    "<br>Maintainer(s): **Alex Drlica-Wagner** ([@kadrlica](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@kadrlica))\n",
    "<br>Level: **Introductory**\n",
    "<br>Last Verified to Run: **2021-03-31**\n",
    "<br>Verified Stack Release: **21.0.0**\n",
    "\n",
    "### Learning Objectives:\n",
    "\n",
    "In this notebook we will look at a few different ways to find the documentation on a given DM Stack function or class. \n",
    "After working through this tutorial you should be able to: \n",
    "1. Use the jupyter notebook built-in functions to read the docstrings of Stack classes and functions \n",
    "2. Use the `where_is` Stack Club utility to locate DM Stack web documentation.\n",
    "\n",
    "### Logistics\n",
    "This notebook is intended to be runnable on `lsst-lsp-stable.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
    "\n",
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:47.950602Z",
     "iopub.status.busy": "2021-04-23T20:41:47.947329Z",
     "iopub.status.idle": "2021-04-23T20:41:50.162055Z",
     "shell.execute_reply": "2021-04-23T20:41:50.160797Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nb-kadrlica-r21-0-0\r\n",
      "lsst_distrib          21.0.0+973e4c9e85 \tcurrent v21_0_0 setup\r\n"
     ]
    }
   ],
   "source": [
    "# What version of the Stack are we using?\n",
    "! echo $HOSTNAME\n",
    "! eups list -s | grep lsst_distrib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll need the `stackclub` package to be installed. If you are not developing this package, and you have permission to write to your base python site-packages, you can install it using `pip`, like this:\n",
    "```\n",
    "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
    "```\n",
    "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing), you'll need to make a local, editable installation, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:50.174478Z",
     "iopub.status.busy": "2021-04-23T20:41:50.171348Z",
     "iopub.status.idle": "2021-04-23T20:41:52.212871Z",
     "shell.execute_reply": "2021-04-23T20:41:52.213991Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/kadrlica/notebooks/.beavis/StackClub/GettingStarted\r\n"
     ]
    }
   ],
   "source": [
    "! cd .. && python setup.py -q develop --user && cd -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When editing the `stackclub` package files, we want the latest version to be imported when we re-run the import command. To enable this, we need the %autoreload magic command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:52.224786Z",
     "iopub.status.busy": "2021-04-23T20:41:52.223541Z",
     "iopub.status.idle": "2021-04-23T20:41:52.254595Z",
     "shell.execute_reply": "2021-04-23T20:41:52.253509Z"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Notebook Help\n",
    "\n",
    "Command line tasks have usage information - try running them with no arguments, or `--help`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:52.269758Z",
     "iopub.status.busy": "2021-04-23T20:41:52.268673Z",
     "iopub.status.idle": "2021-04-23T20:41:56.132810Z",
     "shell.execute_reply": "2021-04-23T20:41:56.131486Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: imageDifference.py input [options]\r\n",
      "\r\n",
      "positional arguments:\r\n",
      "  input                 path to input data repository, relative to $PIPE_INPUT_ROOT\r\n",
      "\r\n",
      "optional arguments:\r\n",
      "  -h, --help            show this help message and exit\r\n",
      "  --calib RAWCALIB      path to input calibration repository, relative to $PIPE_CALIB_ROOT\r\n",
      "  --output RAWOUTPUT    path to output data repository (need not exist), relative to $PIPE_OUTPUT_ROOT\r\n",
      "  --rerun [INPUT:]OUTPUT\r\n",
      "                        rerun name: sets OUTPUT to ROOT/rerun/OUTPUT; optionally sets ROOT to ROOT/rerun/INPUT\r\n",
      "  -c [NAME=VALUE [NAME=VALUE ...]], --config [NAME=VALUE [NAME=VALUE ...]]\r\n",
      "                        config override(s), e.g. -c foo=newfoo bar.baz=3\r\n",
      "  -C [CONFIGFILE [CONFIGFILE ...]], --configfile [CONFIGFILE [CONFIGFILE ...]]\r\n",
      "                        config override file(s)\r\n",
      "  -L [LEVEL|COMPONENT=LEVEL [LEVEL|COMPONENT=LEVEL ...]], --loglevel [LEVEL|COMPONENT=LEVEL [LEVEL|COMPONENT=LEVEL ...]]\r\n",
      "                        logging level; supported levels are [trace|debug|info|warn|error|fatal]\r\n",
      "  --longlog             use a more verbose format for the logging\r\n",
      "  --debug               enable debugging output?\r\n",
      "  --doraise             raise an exception on error (else log a message and continue)?\r\n",
      "  --noExit              Do not exit even upon failure (i.e. return a struct to the calling script)\r\n",
      "  --profile PROFILE     Dump cProfile statistics to filename\r\n",
      "  --show SHOW [SHOW ...]\r\n",
      "                        display the specified information to stdout and quit (unless run is specified); information is\r\n",
      "                        (config[=PATTERN]|history=PATTERN|tasks|data|run)\r\n",
      "  -j PROCESSES, --processes PROCESSES\r\n",
      "                        Number of processes to use\r\n",
      "  -t TIMEOUT, --timeout TIMEOUT\r\n",
      "                        Timeout for multiprocessing; maximum wall time (sec)\r\n",
      "  --clobber-output      remove and re-create the output directory if it already exists (safe with -j, but not all other forms of\r\n",
      "                        parallel execution)\r\n",
      "  --clobber-config      backup and then overwrite existing config files instead of checking them (safe with -j, but not all other\r\n",
      "                        forms of parallel execution)\r\n",
      "  --no-backup-config    Don't copy config to file~N backup.\r\n",
      "  --clobber-versions    backup and then overwrite existing package versions instead of checkingthem (safe with -j, but not all\r\n",
      "                        other forms of parallel execution)\r\n",
      "  --no-versions         don't check package versions; useful for development\r\n",
      "  --id [KEY=VALUE1[^VALUE2[^VALUE3...] [KEY=VALUE1[^VALUE2[^VALUE3...] ...]]\r\n",
      "                        data ID, e.g. --id visit=12345 ccd=1,2\r\n",
      "  --templateId [KEY=VALUE1[^VALUE2[^VALUE3...] [KEY=VALUE1[^VALUE2[^VALUE3...] ...]]\r\n",
      "                        Template data ID in case of calexp template, e.g. --templateId visit=6789\r\n",
      "\r\n",
      "Notes:\r\n",
      "            * --config, --configfile, --id, --loglevel and @file may appear multiple times;\r\n",
      "                all values are used, in order left to right\r\n",
      "            * @file reads command-line options from the specified file:\r\n",
      "                * data may be distributed among multiple lines (e.g. one option per line)\r\n",
      "                * data after # is treated as a comment and ignored\r\n",
      "                * blank lines and lines starting with # are ignored\r\n",
      "            * To specify multiple values for an option, do not use = after the option name:\r\n",
      "                * right: --configfile foo bar\r\n",
      "                * wrong: --configfile=foo bar\r\n"
     ]
    }
   ],
   "source": [
    "! imageDifference.py --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pipeline task python code also contains useful docstrings, accessible in various ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:56.143101Z",
     "iopub.status.busy": "2021-04-23T20:41:56.141804Z",
     "iopub.status.idle": "2021-04-23T20:41:58.610755Z",
     "shell.execute_reply": "2021-04-23T20:41:58.609554Z"
    }
   },
   "outputs": [],
   "source": [
    "from lsst.pipe.tasks.imageDifference import ImageDifferenceTask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:58.617728Z",
     "iopub.status.busy": "2021-04-23T20:41:58.616468Z",
     "iopub.status.idle": "2021-04-23T20:41:58.679596Z",
     "shell.execute_reply": "2021-04-23T20:41:58.680824Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class ImageDifferenceTask in module lsst.pipe.tasks.imageDifference:\n",
      "\n",
      "class ImageDifferenceTask(lsst.pipe.base.cmdLineTask.CmdLineTask, lsst.pipe.base.pipelineTask.PipelineTask)\n",
      " |  ImageDifferenceTask(butler=None, **kwargs)\n",
      " |  \n",
      " |  Subtract an image from a template and measure the result\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      ImageDifferenceTask\n",
      " |      lsst.pipe.base.cmdLineTask.CmdLineTask\n",
      " |      lsst.pipe.base.pipelineTask.PipelineTask\n",
      " |      lsst.pipe.base.task.Task\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, butler=None, **kwargs)\n",
      " |      !Construct an ImageDifference Task\n",
      " |      \n",
      " |      @param[in] butler  Butler object to use in constructing reference object loaders\n",
      " |  \n",
      " |  fitAstrometry(self, templateSources, templateExposure, selectSources)\n",
      " |      Fit the relative astrometry between templateSources and selectSources\n",
      " |      \n",
      " |      Todo\n",
      " |      ----\n",
      " |      \n",
      " |      Remove this method. It originally fit a new WCS to the template before calling register.run\n",
      " |      because our TAN-SIP fitter behaved badly for points far from CRPIX, but that's been fixed.\n",
      " |      It remains because a subtask overrides it.\n",
      " |  \n",
      " |  getSchemaCatalogs(self)\n",
      " |      Return a dict of empty catalogs for each catalog dataset produced by this task.\n",
      " |  \n",
      " |  run(self, exposure=None, selectSources=None, templateExposure=None, templateSources=None, idFactory=None, calexpBackgroundExposure=None, subtractedExposure=None)\n",
      " |      PSF matches, subtract two images and perform detection on the difference image.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      exposure : `lsst.afw.image.ExposureF`, optional\n",
      " |          The science exposure, the minuend in the image subtraction.\n",
      " |          Can be None only if ``config.doSubtract==False``.\n",
      " |      selectSources : `lsst.afw.table.SourceCatalog`, optional\n",
      " |          Identified sources on the science exposure. This catalog is used to\n",
      " |          select sources in order to perform the AL PSF matching on stamp images\n",
      " |          around them. The selection steps depend on config options and whether\n",
      " |          ``templateSources`` and ``matchingSources`` specified.\n",
      " |      templateExposure : `lsst.afw.image.ExposureF`, optional\n",
      " |          The template to be subtracted from ``exposure`` in the image subtraction.\n",
      " |          The template exposure should cover the same sky area as the science exposure.\n",
      " |          It is either a stich of patches of a coadd skymap image or a calexp\n",
      " |          of the same pointing as the science exposure. Can be None only\n",
      " |          if ``config.doSubtract==False`` and ``subtractedExposure`` is not None.\n",
      " |      templateSources : `lsst.afw.table.SourceCatalog`, optional\n",
      " |          Identified sources on the template exposure.\n",
      " |      idFactory : `lsst.afw.table.IdFactory`\n",
      " |          Generator object to assign ids to detected sources in the difference image.\n",
      " |      calexpBackgroundExposure : `lsst.afw.image.ExposureF`, optional\n",
      " |          Background exposure to be added back to the science exposure\n",
      " |          if ``config.doAddCalexpBackground==True``\n",
      " |      subtractedExposure : `lsst.afw.image.ExposureF`, optional\n",
      " |          If ``config.doSubtract==False`` and ``config.doDetection==True``,\n",
      " |          performs the post subtraction source detection only on this exposure.\n",
      " |          Otherwise should be None.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      results : `lsst.pipe.base.Struct`\n",
      " |          ``subtractedExposure`` : `lsst.afw.image.ExposureF`\n",
      " |              Difference image.\n",
      " |          ``matchedExposure`` : `lsst.afw.image.ExposureF`\n",
      " |              The matched PSF exposure.\n",
      " |          ``subtractRes`` : `lsst.pipe.base.Struct`\n",
      " |              The returned result structure of the ImagePsfMatchTask subtask.\n",
      " |          ``diaSources``  : `lsst.afw.table.SourceCatalog`\n",
      " |              The catalog of detected sources.\n",
      " |          ``selectSources`` : `lsst.afw.table.SourceCatalog`\n",
      " |              The input source catalog with optionally added Qa information.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      The following major steps are included:\n",
      " |      \n",
      " |      - warp template coadd to match WCS of image\n",
      " |      - PSF match image to warped template\n",
      " |      - subtract image from PSF-matched, warped template\n",
      " |      - detect sources\n",
      " |      - measure sources\n",
      " |      \n",
      " |      For details about the image subtraction configuration modes\n",
      " |      see `lsst.ip.diffim`.\n",
      " |  \n",
      " |  runDataRef(self, sensorRef, templateIdList=None)\n",
      " |      Subtract an image from a template coadd and measure the result.\n",
      " |      \n",
      " |      Data I/O wrapper around `run` using the butler in Gen2.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      sensorRef : `lsst.daf.persistence.ButlerDataRef`\n",
      " |          Sensor-level butler data reference, used for the following data products:\n",
      " |      \n",
      " |          Input only:\n",
      " |          - calexp\n",
      " |          - psf\n",
      " |          - ccdExposureId\n",
      " |          - ccdExposureId_bits\n",
      " |          - self.config.coaddName + \"Coadd_skyMap\"\n",
      " |          - self.config.coaddName + \"Coadd\"\n",
      " |          Input or output, depending on config:\n",
      " |          - self.config.coaddName + \"Diff_subtractedExp\"\n",
      " |          Output, depending on config:\n",
      " |          - self.config.coaddName + \"Diff_matchedExp\"\n",
      " |          - self.config.coaddName + \"Diff_src\"\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      results : `lsst.pipe.base.Struct`\n",
      " |          Returns the Struct by `run`.\n",
      " |  \n",
      " |  runDebug(self, exposure, subtractRes, selectSources, kernelSources, diaSources)\n",
      " |      Make debug plots and displays.\n",
      " |      \n",
      " |      Todo\n",
      " |      ----\n",
      " |      Test and update for current debug display and slot names\n",
      " |  \n",
      " |  runQuantum(self, butlerQC: lsst.pipe.base.butlerQuantumContext.ButlerQuantumContext, inputRefs: lsst.pipe.base.connections.InputQuantizedConnection, outputRefs: lsst.pipe.base.connections.OutputQuantizedConnection)\n",
      " |      Method to do butler IO and or transforms to provide in memory\n",
      " |      objects for tasks run method\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      butlerQC : `ButlerQuantumContext`\n",
      " |          A butler which is specialized to operate in the context of a\n",
      " |          `lsst.daf.butler.Quantum`.\n",
      " |      inputRefs : `InputQuantizedConnection`\n",
      " |          Datastructure whose attribute names are the names that identify\n",
      " |          connections defined in corresponding `PipelineTaskConnections`\n",
      " |          class. The values of these attributes are the\n",
      " |          `lsst.daf.butler.DatasetRef` objects associated with the defined\n",
      " |          input/prerequisite connections.\n",
      " |      outputRefs : `OutputQuantizedConnection`\n",
      " |          Datastructure whose attribute names are the names that identify\n",
      " |          connections defined in corresponding `PipelineTaskConnections`\n",
      " |          class. The values of these attributes are the\n",
      " |          `lsst.daf.butler.DatasetRef` objects associated with the defined\n",
      " |          output connections.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Static methods defined here:\n",
      " |  \n",
      " |  makeIdFactory(expId, expBits)\n",
      " |      Create IdFactory instance for unique 64 bit diaSource id-s.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      expId : `int`\n",
      " |          Exposure id.\n",
      " |      \n",
      " |      expBits: `int`\n",
      " |          Number of used bits in ``expId``.\n",
      " |      \n",
      " |      Note\n",
      " |      ----\n",
      " |      The diasource id-s consists of the ``expId`` stored fixed in the highest value\n",
      " |      ``expBits`` of the 64-bit integer plus (bitwise or) a generated sequence number in the\n",
      " |      low value end of the integer.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      idFactory: `lsst.afw.table.IdFactory`\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes defined here:\n",
      " |  \n",
      " |  ConfigClass = <class 'lsst.pipe.tasks.imageDifference.ImageDifferenceC...\n",
      " |      Config for ImageDifferenceTask\n",
      " |  \n",
      " |  RunnerClass = <class 'lsst.pipe.tasks.imageDifference.ImageDifferenceT...\n",
      " |      A `TaskRunner` for `CmdLineTask`\\ s that require a ``butler`` keyword\n",
      " |      argument to be passed to their constructor.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from lsst.pipe.base.cmdLineTask.CmdLineTask:\n",
      " |  \n",
      " |  writeConfig(self, butler, clobber=False, doBackup=True)\n",
      " |      Write the configuration used for processing the data, or check that\n",
      " |      an existing one is equal to the new one if present.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      butler : `lsst.daf.persistence.Butler`\n",
      " |          Data butler used to write the config. The config is written to\n",
      " |          dataset type `CmdLineTask._getConfigName`.\n",
      " |      clobber : `bool`, optional\n",
      " |          A boolean flag that controls what happens if a config already has\n",
      " |          been saved:\n",
      " |      \n",
      " |          - `True`: overwrite or rename the existing config, depending on\n",
      " |            ``doBackup``.\n",
      " |          - `False`: raise `TaskError` if this config does not match the\n",
      " |            existing config.\n",
      " |      doBackup : `bool`, optional\n",
      " |          Set to `True` to backup the config files if clobbering.\n",
      " |  \n",
      " |  writeMetadata(self, dataRef)\n",
      " |      Write the metadata produced from processing the data.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      dataRef\n",
      " |          Butler data reference used to write the metadata.\n",
      " |          The metadata is written to dataset type\n",
      " |          `CmdLineTask._getMetadataName`.\n",
      " |  \n",
      " |  writePackageVersions(self, butler, clobber=False, doBackup=True, dataset='packages')\n",
      " |      Compare and write package versions.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      butler : `lsst.daf.persistence.Butler`\n",
      " |          Data butler used to read/write the package versions.\n",
      " |      clobber : `bool`, optional\n",
      " |          A boolean flag that controls what happens if versions already have\n",
      " |          been saved:\n",
      " |      \n",
      " |          - `True`: overwrite or rename the existing version info, depending\n",
      " |            on ``doBackup``.\n",
      " |          - `False`: raise `TaskError` if this version info does not match\n",
      " |            the existing.\n",
      " |      doBackup : `bool`, optional\n",
      " |          If `True` and clobbering, old package version files are backed up.\n",
      " |      dataset : `str`, optional\n",
      " |          Name of dataset to read/write.\n",
      " |      \n",
      " |      Raises\n",
      " |      ------\n",
      " |      TaskError\n",
      " |          Raised if there is a version mismatch with current and persisted\n",
      " |          lists of package versions.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      Note that this operation is subject to a race condition.\n",
      " |  \n",
      " |  writeSchemas(self, butler, clobber=False, doBackup=True)\n",
      " |      Write the schemas returned by\n",
      " |      `lsst.pipe.base.Task.getAllSchemaCatalogs`.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      butler : `lsst.daf.persistence.Butler`\n",
      " |          Data butler used to write the schema. Each schema is written to the\n",
      " |          dataset type specified as the key in the dict returned by\n",
      " |          `~lsst.pipe.base.Task.getAllSchemaCatalogs`.\n",
      " |      clobber : `bool`, optional\n",
      " |          A boolean flag that controls what happens if a schema already has\n",
      " |          been saved:\n",
      " |      \n",
      " |          - `True`: overwrite or rename the existing schema, depending on\n",
      " |            ``doBackup``.\n",
      " |          - `False`: raise `TaskError` if this schema does not match the\n",
      " |            existing schema.\n",
      " |      doBackup : `bool`, optional\n",
      " |          Set to `True` to backup the schema files if clobbering.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      If ``clobber`` is `False` and an existing schema does not match a\n",
      " |      current schema, then some schemas may have been saved successfully\n",
      " |      and others may not, and there is no easy way to tell which is which.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from lsst.pipe.base.cmdLineTask.CmdLineTask:\n",
      " |  \n",
      " |  applyOverrides(config) from builtins.type\n",
      " |      A hook to allow a task to change the values of its config *after*\n",
      " |      the camera-specific overrides are loaded but before any command-line\n",
      " |      overrides are applied.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      config : instance of task's ``ConfigClass``\n",
      " |          Task configuration.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      This is necessary in some cases because the camera-specific overrides\n",
      " |      may retarget subtasks, wiping out changes made in\n",
      " |      ConfigClass.setDefaults. See LSST Trac ticket #2282 for more\n",
      " |      discussion.\n",
      " |      \n",
      " |      .. warning::\n",
      " |      \n",
      " |         This is called by CmdLineTask.parseAndRun; other ways of\n",
      " |         constructing a config will not apply these overrides.\n",
      " |  \n",
      " |  parseAndRun(args=None, config=None, log=None, doReturnResults=False) from builtins.type\n",
      " |      Parse an argument list and run the command.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      args : `list`, optional\n",
      " |          List of command-line arguments; if `None` use `sys.argv`.\n",
      " |      config : `lsst.pex.config.Config`-type, optional\n",
      " |          Config for task. If `None` use `Task.ConfigClass`.\n",
      " |      log : `lsst.log.Log`-type, optional\n",
      " |          Log. If `None` use the default log.\n",
      " |      doReturnResults : `bool`, optional\n",
      " |          If `True`, return the results of this task. Default is `False`.\n",
      " |          This is only intended for unit tests and similar use. It can\n",
      " |          easily exhaust memory (if the task returns enough data and you\n",
      " |          call it enough times) and it will fail when using multiprocessing\n",
      " |          if the returned data cannot be pickled.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      struct : `lsst.pipe.base.Struct`\n",
      " |          Fields are:\n",
      " |      \n",
      " |          ``argumentParser``\n",
      " |              the argument parser (`lsst.pipe.base.ArgumentParser`).\n",
      " |          ``parsedCmd``\n",
      " |              the parsed command returned by the argument parser's\n",
      " |              `~lsst.pipe.base.ArgumentParser.parse_args` method\n",
      " |              (`argparse.Namespace`).\n",
      " |          ``taskRunner``\n",
      " |              the task runner used to run the task (an instance of\n",
      " |              `Task.RunnerClass`).\n",
      " |          ``resultList``\n",
      " |              results returned by the task runner's ``run`` method, one entry\n",
      " |              per invocation (`list`). This will typically be a list of\n",
      " |              `Struct`, each containing at least an ``exitStatus`` integer\n",
      " |              (0 or 1); see `Task.RunnerClass` (`TaskRunner` by default) for\n",
      " |              more details.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      Calling this method with no arguments specified is the standard way to\n",
      " |      run a command-line task from the command-line. For an example see\n",
      " |      ``pipe_tasks`` ``bin/makeSkyMap.py`` or almost any other file in that\n",
      " |      directory.\n",
      " |      \n",
      " |      If one or more of the dataIds fails then this routine will exit (with\n",
      " |      a status giving the number of failed dataIds) rather than returning\n",
      " |      this struct;  this behaviour can be overridden by specifying the\n",
      " |      ``--noExit`` command-line option.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes inherited from lsst.pipe.base.cmdLineTask.CmdLineTask:\n",
      " |  \n",
      " |  canMultiprocess = True\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from lsst.pipe.base.pipelineTask.PipelineTask:\n",
      " |  \n",
      " |  getResourceConfig(self)\n",
      " |      Return resource configuration for this task.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      Object of type `~config.ResourceConfig` or ``None`` if resource\n",
      " |      configuration is not defined for this task.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from lsst.pipe.base.task.Task:\n",
      " |  \n",
      " |  __reduce__(self)\n",
      " |      Pickler.\n",
      " |  \n",
      " |  emptyMetadata(self)\n",
      " |      Empty (clear) the metadata for this Task and all sub-Tasks.\n",
      " |  \n",
      " |  getAllSchemaCatalogs(self)\n",
      " |      Get schema catalogs for all tasks in the hierarchy, combining the\n",
      " |      results into a single dict.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      schemacatalogs : `dict`\n",
      " |          Keys are butler dataset type, values are a empty catalog (an\n",
      " |          instance of the appropriate `lsst.afw.table` Catalog type) for all\n",
      " |          tasks in the hierarchy, from the top-level task down\n",
      " |          through all subtasks.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      This method may be called on any task in the hierarchy; it will return\n",
      " |      the same answer, regardless.\n",
      " |      \n",
      " |      The default implementation should always suffice. If your subtask uses\n",
      " |      schemas the override `Task.getSchemaCatalogs`, not this method.\n",
      " |  \n",
      " |  getFullMetadata(self)\n",
      " |      Get metadata for all tasks.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      metadata : `lsst.daf.base.PropertySet`\n",
      " |          The `~lsst.daf.base.PropertySet` keys are the full task name.\n",
      " |          Values are metadata for the top-level task and all subtasks,\n",
      " |          sub-subtasks, etc.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      The returned metadata includes timing information (if\n",
      " |      ``@timer.timeMethod`` is used) and any metadata set by the task. The\n",
      " |      name of each item consists of the full task name with ``.`` replaced\n",
      " |      by ``:``, followed by ``.`` and the name of the item, e.g.::\n",
      " |      \n",
      " |          topLevelTaskName:subtaskName:subsubtaskName.itemName\n",
      " |      \n",
      " |      using ``:`` in the full task name disambiguates the rare situation\n",
      " |      that a task has a subtask and a metadata item with the same name.\n",
      " |  \n",
      " |  getFullName(self)\n",
      " |      Get the task name as a hierarchical name including parent task\n",
      " |      names.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      fullName : `str`\n",
      " |          The full name consists of the name of the parent task and each\n",
      " |          subtask separated by periods. For example:\n",
      " |      \n",
      " |          - The full name of top-level task \"top\" is simply \"top\".\n",
      " |          - The full name of subtask \"sub\" of top-level task \"top\" is\n",
      " |            \"top.sub\".\n",
      " |          - The full name of subtask \"sub2\" of subtask \"sub\" of top-level\n",
      " |            task \"top\" is \"top.sub.sub2\".\n",
      " |  \n",
      " |  getName(self)\n",
      " |      Get the name of the task.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      taskName : `str`\n",
      " |          Name of the task.\n",
      " |      \n",
      " |      See also\n",
      " |      --------\n",
      " |      getFullName\n",
      " |  \n",
      " |  getTaskDict(self)\n",
      " |      Get a dictionary of all tasks as a shallow copy.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      taskDict : `dict`\n",
      " |          Dictionary containing full task name: task object for the top-level\n",
      " |          task and all subtasks, sub-subtasks, etc.\n",
      " |  \n",
      " |  makeSubtask(self, name, **keyArgs)\n",
      " |      Create a subtask as a new instance as the ``name`` attribute of this\n",
      " |      task.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      name : `str`\n",
      " |          Brief name of the subtask.\n",
      " |      keyArgs\n",
      " |          Extra keyword arguments used to construct the task. The following\n",
      " |          arguments are automatically provided and cannot be overridden:\n",
      " |      \n",
      " |          - \"config\".\n",
      " |          - \"parentTask\".\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      The subtask must be defined by ``Task.config.name``, an instance of\n",
      " |      `~lsst.pex.config.ConfigurableField` or\n",
      " |      `~lsst.pex.config.RegistryField`.\n",
      " |  \n",
      " |  timer(self, name, logLevel=10000)\n",
      " |      Context manager to log performance data for an arbitrary block of\n",
      " |      code.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      name : `str`\n",
      " |          Name of code being timed; data will be logged using item name:\n",
      " |          ``Start`` and ``End``.\n",
      " |      logLevel\n",
      " |          A `lsst.log` level constant.\n",
      " |      \n",
      " |      Examples\n",
      " |      --------\n",
      " |      Creating a timer context:\n",
      " |      \n",
      " |      .. code-block:: python\n",
      " |      \n",
      " |          with self.timer(\"someCodeToTime\"):\n",
      " |              pass  # code to time\n",
      " |      \n",
      " |      See also\n",
      " |      --------\n",
      " |      timer.logInfo\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from lsst.pipe.base.task.Task:\n",
      " |  \n",
      " |  makeField(doc) from builtins.type\n",
      " |      Make a `lsst.pex.config.ConfigurableField` for this task.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      doc : `str`\n",
      " |          Help text for the field.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      configurableField : `lsst.pex.config.ConfigurableField`\n",
      " |          A `~ConfigurableField` for this task.\n",
      " |      \n",
      " |      Examples\n",
      " |      --------\n",
      " |      Provides a convenient way to specify this task is a subtask of another\n",
      " |      task.\n",
      " |      \n",
      " |      Here is an example of use:\n",
      " |      \n",
      " |      .. code-block:: python\n",
      " |      \n",
      " |          class OtherTaskConfig(lsst.pex.config.Config):\n",
      " |              aSubtask = ATaskClass.makeField(\"brief description of task\")\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from lsst.pipe.base.task.Task:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(ImageDifferenceTask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can follow up on the methods and attributes listed in the `help()` output, with further `help()` commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:58.687378Z",
     "iopub.status.busy": "2021-04-23T20:41:58.686073Z",
     "iopub.status.idle": "2021-04-23T20:41:58.727857Z",
     "shell.execute_reply": "2021-04-23T20:41:58.729088Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function getName in module lsst.pipe.base.task:\n",
      "\n",
      "getName(self)\n",
      "    Get the name of the task.\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    taskName : `str`\n",
      "        Name of the task.\n",
      "    \n",
      "    See also\n",
      "    --------\n",
      "    getFullName\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(ImageDifferenceTask.getName)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `help()` function mostly prints out the `__doc__` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:58.735565Z",
     "iopub.status.busy": "2021-04-23T20:41:58.734124Z",
     "iopub.status.idle": "2021-04-23T20:41:58.770919Z",
     "shell.execute_reply": "2021-04-23T20:41:58.772151Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Get the name of the task.\n",
      "\n",
      "        Returns\n",
      "        -------\n",
      "        taskName : `str`\n",
      "            Name of the task.\n",
      "\n",
      "        See also\n",
      "        --------\n",
      "        getFullName\n",
      "        \n"
     ]
    }
   ],
   "source": [
    "print(ImageDifferenceTask.getName.__doc__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Jupyter/IPython `?` magic command gives a different, condensed view that may sometimes be helpful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:58.778553Z",
     "iopub.status.busy": "2021-04-23T20:41:58.777124Z",
     "iopub.status.idle": "2021-04-23T20:41:58.843763Z",
     "shell.execute_reply": "2021-04-23T20:41:58.842446Z"
    }
   },
   "outputs": [],
   "source": [
    "? ImageDifferenceTask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Online Resources: Searchable GitHub-hosted Source Code\n",
    "\n",
    "All the DM code is housed in GitHub repositories in the `lsst` organization.\n",
    "It's nice to provide hyperlinks to the code you are demonstrating, so people can quickly go read the source. We can construct the GitHub URL from the module name, using the `stackclub.where_is` utility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:58.849761Z",
     "iopub.status.busy": "2021-04-23T20:41:58.848316Z",
     "iopub.status.idle": "2021-04-23T20:41:58.926950Z",
     "shell.execute_reply": "2021-04-23T20:41:58.925632Z"
    }
   },
   "outputs": [],
   "source": [
    "from stackclub import where_is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:58.932691Z",
     "iopub.status.busy": "2021-04-23T20:41:58.931251Z",
     "iopub.status.idle": "2021-04-23T20:41:58.965309Z",
     "shell.execute_reply": "2021-04-23T20:41:58.964035Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[`lsst.pipe.tasks.imageDifference`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/imageDifference.py)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[`lsst.pipe.tasks.imageDifference`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/imageDifference.py)\n"
     ]
    }
   ],
   "source": [
    "from lsst.pipe.tasks.imageDifference import ImageDifferenceTask\n",
    "where_is(ImageDifferenceTask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `where_is` looks for the named object in the source code on GitHub. You can specify this behavior explitly with the `in_the` kwarg:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:58.970929Z",
     "iopub.status.busy": "2021-04-23T20:41:58.969602Z",
     "iopub.status.idle": "2021-04-23T20:41:58.996819Z",
     "shell.execute_reply": "2021-04-23T20:41:58.995756Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[`lsst.daf.persistence.butler`](https://github.com/lsst/daf_persistence/blob/master/python/lsst/daf/persistence/butler.py)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[`lsst.daf.persistence.butler`](https://github.com/lsst/daf_persistence/blob/master/python/lsst/daf/persistence/butler.py)\n"
     ]
    }
   ],
   "source": [
    "from lsst.daf.persistence import Butler\n",
    "where_is(Butler.get, in_the='source')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In case you're interested in what the `where_is` function is doing, paste the following into a python cell: \n",
    "```\n",
    "%load ../stackclub/where_is\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GitHub search is pretty powerful. Here's an example, using the search string `user:lsst ImageDifferenceTask` and selecting \"Code\" results (in python):\n",
    "\n",
    "https://github.com/search?l=Python&q=user%3Alsst+ImageDifferenceTask&type=Code\n",
    "\n",
    "You can also generate search strings like this one with `where_is`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:59.002095Z",
     "iopub.status.busy": "2021-04-23T20:41:59.000905Z",
     "iopub.status.idle": "2021-04-23T20:41:59.025762Z",
     "shell.execute_reply": "2021-04-23T20:41:59.026619Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[searching for `Butler` in the `lsst` repo](https://github.com/search?l=Python&q=org%3Alsst+Butler&type=Code)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[searching for `Butler` in the `lsst` repo](https://github.com/search?l=Python&q=org%3Alsst+Butler&type=Code)\n"
     ]
    }
   ],
   "source": [
    "where_is(Butler, in_the='repo')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, here's how to generate a search within the LSST DM technotes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-23T20:41:59.031256Z",
     "iopub.status.busy": "2021-04-23T20:41:59.030222Z",
     "iopub.status.idle": "2021-04-23T20:41:59.054950Z",
     "shell.execute_reply": "2021-04-23T20:41:59.055762Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[searching for `ImageDifferenceTask` in the `lsst-dm` technotes](https://github.com/search?l=reStructuredText&q=org%3Alsst-dm+ImageDifferenceTask&type=Code)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[searching for `ImageDifferenceTask` in the `lsst-dm` technotes](https://github.com/search?l=reStructuredText&q=org%3Alsst-dm+ImageDifferenceTask&type=Code)\n"
     ]
    }
   ],
   "source": [
    "where_is(ImageDifferenceTask, in_the='technotes')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "In this tutorial we have explored two general ways to read more about the DM Stack code objects: the built-in notebook `help` and magic '?' commands, and the `stackclub.where_is`  utility for locating the relevant part of the Stack source code. \n",
    "\n",
    "Both of the above methods focus on the python code, which for many purposes will be sufficient. However, to understand the Stack's C++ primitives, we'll need to dig deeper into the DM Stack's [doxygen documentation](http://doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/), as linked from https://pipelines.lsst.io. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}