{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"toc-hr-collapsed": false
},
"source": [
"# Quickstart LSST Verify Demo\n",
"\n",
"<br>Owner: **Keith Bechtol** ([@bechtol](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@bechtol))\n",
"<br>Minor updates by: Douglas Tucker ([@douglasleetucker](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@douglasleetucker))\n",
"<br>Last Verified to Run: **2021-04-13**\n",
"<br>Verified Stack Release: **v21.0.0**\n",
"\n",
"This notebook demonstrates basic functionality of the LSST Verify python package: https://github.com/lsst/verify . The notebook is based on the documentation at https://sqr-019.lsst.io/ . \n",
"\n",
"Another example from the LSST Systems Engineering team can be found [here](https://github.com/mareuter/notebooks/blob/master/LSST/Systems_Engineering/System_Verification_SQuaSH/System_Verification_Demo.ipynb), for which the metrics are defined [here](https://github.com/mareuter/notebooks/tree/master/LSST/Systems_Engineering/System_Verification_SQuaSH).\n",
"\n",
"### Learning Objectives\n",
"\n",
"After working through and studying this notebook you should be able to\n",
" 1. Create custom metrics for your science cases and associate multiple specifications with those metrics\n",
" 2. Evaluate metrics and store the output for subsequent analysis\n",
" 3. Create customized summary displays for the performance on those metrics relative to your specifications\n",
"\n",
"### Logistics\n",
"\n",
"This notebook is intended to be runnable on `lsst-lsp-stable.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:32.480000Z",
"iopub.status.busy": "2021-04-23T20:50:32.474847Z",
"iopub.status.idle": "2021-04-23T20:50:34.341507Z",
"shell.execute_reply": "2021-04-23T20:50:34.342758Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"nb-kadrlica-r21-0-0\r\n",
" 21.0.0+973e4c9e85 \tcurrent v21_0_0 setup\r\n"
]
}
],
"source": [
"# What version of the Stack am I using?\n",
"! echo $HOSTNAME\n",
"! eups list -s lsst_distrib"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:34.354075Z",
"iopub.status.busy": "2021-04-23T20:50:34.352743Z",
"iopub.status.idle": "2021-04-23T20:50:35.579731Z",
"shell.execute_reply": "2021-04-23T20:50:35.578413Z"
}
},
"outputs": [],
"source": [
"import json\n",
"import numpy as np\n",
"from matplotlib import pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"import astropy.units as u\n",
"\n",
"import lsst.verify"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Defining metrics and specifications\n",
"\n",
"There are specific rules for the directory structure and naming. In particular, there are required folders for \"metrics\" and \"specs\". The metrics are defined in a set of yaml files in the metrics folder. For each yaml file of metrics, there is a corresponding directory with the same name in the specifications directory. The specifications are defined in their own yaml files. An example directory structure appears below."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:35.586968Z",
"iopub.status.busy": "2021-04-23T20:50:35.585480Z",
"iopub.status.idle": "2021-04-23T20:50:36.456744Z",
"shell.execute_reply": "2021-04-23T20:50:36.455679Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"verify_demo\r\n",
"├── metrics\r\n",
"│ ├── demo_astrometry.yaml\r\n",
"│ └── demo_photometry.yaml\r\n",
"└── specs\r\n",
" ├── demo_astrometry\r\n",
" │ └── specs.yaml\r\n",
" └── demo_photometry\r\n",
" └── specs.yaml\r\n",
"\r\n",
"4 directories, 4 files\r\n"
]
}
],
"source": [
"!tree verify_demo"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take a look at both the metric and specification yaml files"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:36.466179Z",
"iopub.status.busy": "2021-04-23T20:50:36.464951Z",
"iopub.status.idle": "2021-04-23T20:50:37.297079Z",
"shell.execute_reply": "2021-04-23T20:50:37.295831Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"AstrometricRMS:\r\n",
" unit: mas\r\n",
" description:\r\n",
" Astrometric residual RMS.\r\n",
" reference:\r\n",
" url: https://example.com/AstroRMS\r\n",
" tags:\r\n",
" - astrometry\r\n",
" - demo\r\n",
" "
]
}
],
"source": [
"!cat verify_demo/metrics/demo_astrometry.yaml"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:37.306894Z",
"iopub.status.busy": "2021-04-23T20:50:37.305511Z",
"iopub.status.idle": "2021-04-23T20:50:38.278676Z",
"shell.execute_reply": "2021-04-23T20:50:38.277423Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"name: \"minimum\"\r\n",
"metric: \"AstrometricRMS\"\r\n",
"threshold:\r\n",
" operator: \"<=\"\r\n",
" unit: \"mas\"\r\n",
" value: 20.0\r\n",
"tags:\r\n",
" - \"minimum\"\r\n",
"---\r\n",
"name: \"design\"\r\n",
"metric: \"AstrometricRMS\"\r\n",
"threshold:\r\n",
" operator: \"<=\"\r\n",
" unit: \"mas\"\r\n",
" value: 10.0\r\n",
"tags:\r\n",
" - \"design\"\r\n",
" "
]
}
],
"source": [
"!cat verify_demo/specs/demo_astrometry/specs.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the definition of specifications, note that the \"---\" line between specifications is required.\n",
"\n",
"Next, we create instances of the `MetricSet` and `SpecificationSet`."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.289014Z",
"iopub.status.busy": "2021-04-23T20:50:38.287649Z",
"iopub.status.idle": "2021-04-23T20:50:38.298851Z",
"shell.execute_reply": "2021-04-23T20:50:38.299910Z"
}
},
"outputs": [],
"source": [
"METRIC_PACKAGE = \"verify_demo\"\n",
"metrics = lsst.verify.MetricSet.load_metrics_package(METRIC_PACKAGE)\n",
"specs = lsst.verify.SpecificationSet.load_metrics_package(METRIC_PACKAGE)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"View the metrics that have been defined:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.316453Z",
"iopub.status.busy": "2021-04-23T20:50:38.315243Z",
"iopub.status.idle": "2021-04-23T20:50:38.321372Z",
"shell.execute_reply": "2021-04-23T20:50:38.322403Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<i>Table length=2</i>\n",
"<table id=\"table139878895570896\" class=\"table-striped table-bordered table-condensed\">\n",
"<thead><tr><th>Name</th><th>Description</th><th>Units</th><th>Reference</th><th>Tags</th></tr></thead>\n",
"<thead><tr><th>str30</th><th>str28</th><th>str15</th><th>str28</th><th>str16</th></tr></thead>\n",
"<tr><td>demo_astrometry.AstrometricRMS</td><td>Astrometric residual RMS.</td><td>$\\mathrm{mas}$</td><td>https://example.com/AstroRMS</td><td>astrometry, demo</td></tr>\n",
"<tr><td>demo_photometry.ZeropointRMS</td><td>Photometric calibration RMS.</td><td>$\\mathrm{mmag}$</td><td>https://example.com/PhotRMS</td><td>demo, photometry</td></tr>\n",
"</table>"
],
"text/plain": [
"<lsst.verify.metricset.MetricSet at 0x7f3817e16650>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"metrics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"View the specifications that have been defined:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.330558Z",
"iopub.status.busy": "2021-04-23T20:50:38.329454Z",
"iopub.status.idle": "2021-04-23T20:50:38.333290Z",
"shell.execute_reply": "2021-04-23T20:50:38.334238Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<i>Table length=4</i>\n",
"<table id=\"table139878895600592\" class=\"table-striped table-bordered table-condensed\">\n",
"<thead><tr><th>Name</th><th>Test</th><th>Tags</th></tr></thead>\n",
"<thead><tr><th>str38</th><th>str27</th><th>str7</th></tr></thead>\n",
"<tr><td>demo_astrometry.AstrometricRMS.design</td><td>$x$ <= 10.0 $\\mathrm{mas}$</td><td>design</td></tr>\n",
"<tr><td>demo_astrometry.AstrometricRMS.minimum</td><td>$x$ <= 20.0 $\\mathrm{mas}$</td><td>minimum</td></tr>\n",
"<tr><td>demo_photometry.ZeropointRMS.design</td><td>$x$ <= 10.0 $\\mathrm{mmag}$</td><td>design</td></tr>\n",
"<tr><td>demo_photometry.ZeropointRMS.minimum</td><td>$x$ <= 20.0 $\\mathrm{mmag}$</td><td>minimum</td></tr>\n",
"</table>"
],
"text/plain": [
"<lsst.verify.specset.SpecificationSet at 0x7f3817e16f50>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"specs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Computing and storing metrics\n",
"\n",
"For the purpose of illustration, let's make up some measurement values corresponding to our metrics. The following lines are placeholders for the analysis that we would want to do. In this example, we choose measurement values that are intermediate between the specifications defined above so that we can see what happens when some specifications are met and others are not. Notice that the measurements can have dimensions."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.341110Z",
"iopub.status.busy": "2021-04-23T20:50:38.340075Z",
"iopub.status.idle": "2021-04-23T20:50:38.343118Z",
"shell.execute_reply": "2021-04-23T20:50:38.343986Z"
}
},
"outputs": [],
"source": [
"zp_rms = 15.*u.mmag\n",
"zp_meas = lsst.verify.Measurement('demo_photometry.ZeropointRMS', zp_rms)\n",
"\n",
"astro_rms = 15.*u.mas\n",
"astro_meas = lsst.verify.Measurement('demo_astrometry.AstrometricRMS', astro_rms)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is possible to include extra information along with the measurements. These are made up values only for the purpose of illustration."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.352000Z",
"iopub.status.busy": "2021-04-23T20:50:38.351025Z",
"iopub.status.idle": "2021-04-23T20:50:38.354166Z",
"shell.execute_reply": "2021-04-23T20:50:38.353329Z"
}
},
"outputs": [],
"source": [
"zp_meas.extras['x'] = lsst.verify.Datum(np.random.random(10) * u.mag, label=\"x\", description=\"x-values\")\n",
"zp_meas.extras['y'] = lsst.verify.Datum(np.random.random(10) * u.mag, label=\"y\", description=\"y-values\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create an LSST verify job and add the measurements."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.360439Z",
"iopub.status.busy": "2021-04-23T20:50:38.359460Z",
"iopub.status.idle": "2021-04-23T20:50:38.361584Z",
"shell.execute_reply": "2021-04-23T20:50:38.362390Z"
}
},
"outputs": [],
"source": [
"job = lsst.verify.Job(metrics=metrics, specs=specs)\n",
"job.measurements.insert(zp_meas)\n",
"job.measurements.insert(astro_meas)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Provide metadata about the job. This could be used to capture information about the analysis configuration, software version, dataset, etc."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.367911Z",
"iopub.status.busy": "2021-04-23T20:50:38.367000Z",
"iopub.status.idle": "2021-04-23T20:50:38.369013Z",
"shell.execute_reply": "2021-04-23T20:50:38.369776Z"
}
},
"outputs": [],
"source": [
"job.meta.update({'version': 'test'})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When we are done, write the output to a file. This can be exported to metric aggregators at a later time."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.374692Z",
"iopub.status.busy": "2021-04-23T20:50:38.373814Z",
"iopub.status.idle": "2021-04-23T20:50:38.378508Z",
"shell.execute_reply": "2021-04-23T20:50:38.379194Z"
}
},
"outputs": [],
"source": [
"job.write('demo.json')"
]
},
{
"cell_type": "markdown",
"metadata": {
"toc-hr-collapsed": false
},
"source": [
"## Creating reports\n",
"\n",
"Create a report to visualize the outcome of our analysis. We already have the job in memory, but for the purpose of illustration, let's read in the file that we just wrote to show how one could examine the results at a later time."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.384560Z",
"iopub.status.busy": "2021-04-23T20:50:38.383682Z",
"iopub.status.idle": "2021-04-23T20:50:38.387185Z",
"shell.execute_reply": "2021-04-23T20:50:38.387834Z"
}
},
"outputs": [],
"source": [
"with open('demo.json') as f:\n",
" job = lsst.verify.Job.deserialize(**json.load(f))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Display a summary report"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.395756Z",
"iopub.status.busy": "2021-04-23T20:50:38.394966Z",
"iopub.status.idle": "2021-04-23T20:50:38.397939Z",
"shell.execute_reply": "2021-04-23T20:50:38.398579Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<i>Table length=4</i>\n",
"<table id=\"table139878895670096-130198\" class=\"table-striped table-bordered table-condensed\">\n",
"<thead><tr><th>Status</th><th>Specification</th><th>Measurement</th><th>Test</th><th>Metric Tags</th><th>Spec. Tags</th></tr></thead>\n",
"<tr><td>❌</td><td>demo_astrometry.AstrometricRMS.design</td><td>15.0 $\\mathrm{mas}$</td><td>$x$ <= 10.0 $\\mathrm{mas}$</td><td>astrometry, demo</td><td>design</td></tr>\n",
"<tr><td>✅</td><td>demo_astrometry.AstrometricRMS.minimum</td><td>15.0 $\\mathrm{mas}$</td><td>$x$ <= 20.0 $\\mathrm{mas}$</td><td>astrometry, demo</td><td>minimum</td></tr>\n",
"<tr><td>❌</td><td>demo_photometry.ZeropointRMS.design</td><td>15.0 $\\mathrm{mmag}$</td><td>$x$ <= 10.0 $\\mathrm{mmag}$</td><td>demo, photometry</td><td>design</td></tr>\n",
"<tr><td>✅</td><td>demo_photometry.ZeropointRMS.minimum</td><td>15.0 $\\mathrm{mmag}$</td><td>$x$ <= 20.0 $\\mathrm{mmag}$</td><td>demo, photometry</td><td>minimum</td></tr>\n",
"</table><style>table.dataTable {clear: both; width: auto !important; margin: 0 !important;}\n",
".dataTables_info, .dataTables_length, .dataTables_filter, .dataTables_paginate{\n",
"display: inline-block; margin-right: 1em; }\n",
".paginate_button { margin-right: 5px; }\n",
"</style>\n",
"<script>\n",
"\n",
"var astropy_sort_num = function(a, b) {\n",
" var a_num = parseFloat(a);\n",
" var b_num = parseFloat(b);\n",
"\n",
" if (isNaN(a_num) && isNaN(b_num))\n",
" return ((a < b) ? -1 : ((a > b) ? 1 : 0));\n",
" else if (!isNaN(a_num) && !isNaN(b_num))\n",
" return ((a_num < b_num) ? -1 : ((a_num > b_num) ? 1 : 0));\n",
" else\n",
" return isNaN(a_num) ? -1 : 1;\n",
"}\n",
"\n",
"require.config({paths: {\n",
" datatables: 'https://cdn.datatables.net/1.10.12/js/jquery.dataTables.min'\n",
"}});\n",
"require([\"datatables\"], function(){\n",
" console.log(\"$('#table139878895670096-130198').dataTable()\");\n",
" \n",
"jQuery.extend( jQuery.fn.dataTableExt.oSort, {\n",
" \"optionalnum-asc\": astropy_sort_num,\n",
" \"optionalnum-desc\": function (a,b) { return -astropy_sort_num(a, b); }\n",
"});\n",
"\n",
" $('#table139878895670096-130198').dataTable({\n",
" order: [],\n",
" pageLength: 50,\n",
" lengthMenu: [[10, 25, 50, 100, 500, 1000, -1], [10, 25, 50, 100, 500, 1000, 'All']],\n",
" pagingType: \"full_numbers\",\n",
" columnDefs: [{targets: [], type: \"optionalnum\"}]\n",
" });\n",
"});\n",
"</script>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"job.report().show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice that because of the measurement values we used in this example, some of the specifications are met, while others are not.\n",
"\n",
"It is possible to select particular tags to customize the report. The example below shows a selection on specification tags."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.405737Z",
"iopub.status.busy": "2021-04-23T20:50:38.404974Z",
"iopub.status.idle": "2021-04-23T20:50:38.407830Z",
"shell.execute_reply": "2021-04-23T20:50:38.408442Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<i>Table length=2</i>\n",
"<table id=\"table139878895670032-864770\" class=\"table-striped table-bordered table-condensed\">\n",
"<thead><tr><th>Status</th><th>Specification</th><th>Measurement</th><th>Test</th><th>Metric Tags</th><th>Spec. Tags</th></tr></thead>\n",
"<tr><td>✅</td><td>demo_astrometry.AstrometricRMS.minimum</td><td>15.0 $\\mathrm{mas}$</td><td>$x$ <= 20.0 $\\mathrm{mas}$</td><td>astrometry, demo</td><td>minimum</td></tr>\n",
"<tr><td>✅</td><td>demo_photometry.ZeropointRMS.minimum</td><td>15.0 $\\mathrm{mmag}$</td><td>$x$ <= 20.0 $\\mathrm{mmag}$</td><td>demo, photometry</td><td>minimum</td></tr>\n",
"</table><style>table.dataTable {clear: both; width: auto !important; margin: 0 !important;}\n",
".dataTables_info, .dataTables_length, .dataTables_filter, .dataTables_paginate{\n",
"display: inline-block; margin-right: 1em; }\n",
".paginate_button { margin-right: 5px; }\n",
"</style>\n",
"<script>\n",
"\n",
"var astropy_sort_num = function(a, b) {\n",
" var a_num = parseFloat(a);\n",
" var b_num = parseFloat(b);\n",
"\n",
" if (isNaN(a_num) && isNaN(b_num))\n",
" return ((a < b) ? -1 : ((a > b) ? 1 : 0));\n",
" else if (!isNaN(a_num) && !isNaN(b_num))\n",
" return ((a_num < b_num) ? -1 : ((a_num > b_num) ? 1 : 0));\n",
" else\n",
" return isNaN(a_num) ? -1 : 1;\n",
"}\n",
"\n",
"require.config({paths: {\n",
" datatables: 'https://cdn.datatables.net/1.10.12/js/jquery.dataTables.min'\n",
"}});\n",
"require([\"datatables\"], function(){\n",
" console.log(\"$('#table139878895670032-864770').dataTable()\");\n",
" \n",
"jQuery.extend( jQuery.fn.dataTableExt.oSort, {\n",
" \"optionalnum-asc\": astropy_sort_num,\n",
" \"optionalnum-desc\": function (a,b) { return -astropy_sort_num(a, b); }\n",
"});\n",
"\n",
" $('#table139878895670032-864770').dataTable({\n",
" order: [],\n",
" pageLength: 50,\n",
" lengthMenu: [[10, 25, 50, 100, 500, 1000, -1], [10, 25, 50, 100, 500, 1000, 'All']],\n",
" pagingType: \"full_numbers\",\n",
" columnDefs: [{targets: [], type: \"optionalnum\"}]\n",
" });\n",
"});\n",
"</script>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"job.report(spec_tags=['minimum']).show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is also possible to see what tags are available."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.413236Z",
"iopub.status.busy": "2021-04-23T20:50:38.412512Z",
"iopub.status.idle": "2021-04-23T20:50:38.415206Z",
"shell.execute_reply": "2021-04-23T20:50:38.415853Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'astrometry', 'demo'}"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"job.metrics['demo_astrometry.AstrometricRMS'].tags"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"View metadata."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.420325Z",
"iopub.status.busy": "2021-04-23T20:50:38.419630Z",
"iopub.status.idle": "2021-04-23T20:50:38.422190Z",
"shell.execute_reply": "2021-04-23T20:50:38.422783Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"{\n",
" \"version\": \"test\"\n",
"}"
],
"text/plain": [
"ChainMap({'version': 'test'}, {}, {})"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"job.meta"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A lot of information is available for plotting if we want to dig deeper into the results. These are the extra data that we saved together with the metric values. "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"execution": {
"iopub.execute_input": "2021-04-23T20:50:38.447105Z",
"iopub.status.busy": "2021-04-23T20:50:38.446431Z",
"iopub.status.idle": "2021-04-23T20:50:38.573456Z",
"shell.execute_reply": "2021-04-23T20:50:38.574715Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"(0.0, 1.0)"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbI0lEQVR4nO3df5RcZZ3n8fcnTSI9gDSSgKQTSNQQQHCNNgwe2BUFTcCdJIujwoLADMIOsyAqkyFRRxAdQbOOyoiDETwI/gDEbE9Goo1KgDPMhKGhhRCgNUYg6QSTAM0y0pAmfPePe5uuFNW3q5vcqtvVn9c5fVL13KdufeueTn36Ps/9oYjAzMxsKBPqXYCZmRWbg8LMzDI5KMzMLJODwszMMjkozMwsk4PCzMwyOSjMykj6tKRr6l2HWVE4KKwmJJ0m6T8r/ISkz9W7vlIR8aWI+Fg1fSVdKun7ZW13SHoh/XzbJC2XdEDZa0LShWWvuzBtv7Sk7dOSfp+ua6Okm6r9HJIek3RCtf0z1nOWpH99reuxsctBYTURET+IiD1Lf4BPAH8AvjPS9UnabVfXuIudn37GtwB7Av+nbPlvgDPK2s5M2wGQdCbwUeCEdF1twK9yq9hsCA4KqwtJc4CvA6dExOa0bW9J10raLKlH0hclNaXLzpJ0t6SvSXoKuDTtf72krZIel/RZSRPK+n9T0rOSHpV0fMn7T5W0QtLTktZJOqdk2St7CZJmpH/lnynpiXQP4TPpsnnAp4GPpH/xP1D+OSOiF2gH3l626F7gTyS9NV3XW4Hd0/YBRwIdEfG7dF1PRsSykjoXS/rpENv3BuBA4F/S2v42bT9a0r9J6pX0gKTjSl5zlqT1kp5L92JOk3QocDXwrnQ9vZXezxpb0f8qswYkqQW4BfhCRNxRsug6YAvJX+F7AD8FNgDfTpf/KXAjsD8wMW3fG3gTsC9wG7AZuLak/y3AZOBkYLmkmRHxdLqeh4CpwCHALyT9LiJuH6LsY4HZwMHAf0haHhE/l/Ql4C0RcfoQn3Xf9L3XVVh8A8lexcUkexM3AG8tWb4auFJSD7AK6IqIHQMLI+KKIWolIj4q6b8CH4uIX6a1tAK3kuyl/Bw4HviJpEOA54ErgSMjojsdKntDRDwi6a/S9Rw71PtZY/MehdWUJAHXk3xJf6WkfX/gJOATEfHHiNgCfA04peTlmyLiHyPiJWB7umxJRDwXEY8BXyX5EhywBfh6RPRHxE1AN/ABSdOBY4CLI+KFiPg1cA2vHgoq9fmI6IuIB4AHgP8yzEe9UtKzwDaSoLqgQp/vA6dKmph+lp3mOiLi++nr5gJ3AlskXTzM+2Y5HVgZESsj4uWI+AXQSbLdAV4GDpfUHBGbI2Lta3gvayAOCqu1i0n+aj4zdr4i5UEkewmb02GRXpI9hv1K+mwoeTw57f94SdvjQGvJ856y93icZA9iKvB0RDyX8dpyT5Y8fp5k3iHLxyNib+BtwD7AtPIOEfEEyZ7Gl4DfRsSGCn1+EBEnAC3AXwFfkDR3mPceykHAhwa2b7qNjwUOiIg/Ah9J32OzpFvTPQ0zB4XVTjoe/hngz9Ox+1IbgBeByRHRkv68PiJKh2JKv/S3Af0kX34DDgR6Sp63pnswpcs3pT9vkLRXxmurlXn55YhYA3wRuKqslgHXAxel/2atpz8ifgw8CBw+yto2ADeUbN+WiNhjYAgrIjoi4n3AAcCjDB5k4EtMj3MOCquJdMz7RpKhpa7y5emE9m3AVyW9XtIESW+W9O5K60vH6m8G/l7SXpIOAj7FzsM3+wEflzRR0oeAQ0mGXjYA/wZcLml3SW8Dzi57bbX+AMwYmEQfwvdI5lXmV1h2E/D+9LPsJJ1c/kD6+SZIOpFkb+yedPmlku4YprY3lTz/PvBnkuZKako/+3GSpknaX9ICSXuQBPZ/kgxFDaxnmqRJGe9lDcxBYbVyDsmX5Tf06nMprk77nAFMAh4GniGZiD6g8uqAZPz+j8B64F+BHwLfLVl+DzCLZO/j70n2ZJ5Kl50KzCDZu/i/wCUDk74j9OP036ck3V+pQ0RsB74B/F2FZX0R8cuI6Kvw0v9HclTVE0AvyZzOeRExcE7DdODujNouBz6bDjP9TRqQC9J1biXZw1hE8j0wgSRoNwFPA+8GzkvXczuwFnhS0raM97MGJd+4yBqRpLNo8CN1JP0aOL4k/Mxy4cNjzcaoiHh7vWuw8SG3oSdJ35W0RdJDQyyXpCvTk50elPSOvGoxM7PRy3OO4jpgXsbyE0nGj2cB5wL/lGMtNs5ExHWNPOxkVku5BUVE3EUyKTaUBcD1kVgNtKjkwmlmZlYM9ZyjaGXnE6g2pm2byztKOpdkr4M99tjjnYcc4vOAzKw4Hn3yOfp3vPyq9olNEzjkjXtVeEXt3XfffdsiYspoXjsmJrPTC6EtA2hra4vOzs46V2RmeWjv6mFpRzebevuY2tLMormzWTgn64T5Ypi5+NaKZyUK6LziA7UupyJJjw/fq7J6nkfRQ3Ic+IBpjO7MWDNrAO1dPSxZvoae3j4C6OntY8nyNbR3Ff9rYWpL84jax5p6BsUK4Iz06KejgWcHLjdtZuPP0o5u+vp37NTW17+DpR3ddaqoeovmzqZ5YtNObc0Tm1g0d3adKtq1cht6kvQj4DhgsqSNwCUkF3EjIq4GVpJctXIdyUXW/iKvWsys+Db1Vjo5fej2IhkYHhuLw2bVyC0oIuLUYZYH8L/zen8zG1umtjTTUyEUxsrwzcI5rQ0TDOV8rSczK4RGH74Zy8bEUU9m1vgaffhmLHNQmFlhNPLwzVjmoSczM8vkoDAzs0wOCjMzy+SgMDOzTA4KMzPL5KAwM7NMDgozM8vkoDAzs0wOCjMzy+SgMDOzTA4KMzPL5Gs9mVmhjNXboTYyB4WZFcbA7VAH7nQ3cDtUwGFRRx56MrPCGMu3Q21kDgozK4yxfDvURuagMLPCGOq2p2PldqiNykFhZoXh26EWkyezzawwfDvUVyvCUWAOCjMrFN8OdVBRjgLz0JOZWUEV5SgwB4WZWUEV5SgwDz01gCKMYZrZrje1pZmeCqFQ66PAvEcxxg2MYfb09hEMjmG2d/XUuzQze42KchSYg2KMK8oYppntegvntHL5yUfQ2tKMgNaWZi4/+Qgf9WQjU5QxTDPLRxGOAvMexRjnM1nNLG8OijGuKGOYZta4PPQ0xvlMVjPLm4OiARRhDNPMGpeHnszMLJODwszMMjkozMwsU65BIWmepG5J6yQtrrD8QEmrJHVJelDSSXnWY2ZmI5dbUEhqAq4CTgQOA06VdFhZt88CN0fEHOAU4Ft51WNmZqOT5x7FUcC6iFgfEduBG4EFZX0CeH36eG9gU471mJnZKOQZFK3AhpLnG9O2UpcCp0vaCKwELqi0IknnSuqU1Ll169Y8ajUzsyHUezL7VOC6iJgGnATcIOlVNUXEsohoi4i2KVOm1LxIM7PxLM+g6AGmlzyflraVOhu4GSAi/h3YHZicY01mZjZCeQbFvcAsSTMlTSKZrF5R1ucJ4HgASYeSBIXHlszMCiS3oIiIl4DzgQ7gEZKjm9ZKukzS/LTbRcA5kh4AfgScFRGRV01mZjZyuV7rKSJWkkxSl7Z9ruTxw8AxedZgZmavTb0ns83MrOAcFGZmlslBYWZmmRwUZmaWyUFhZmaZHBRmZpbJQWFmZpkcFGZmlslBYWZmmRwUZmaWyUFhZmaZHBRmZpbJQWFmZpkcFGZmlslBYWZmmRwUZmaWyUFhZmaZHBRmZpbJQWFmZpkcFGZmlslBYWZmmXardwFmZu1dPSzt6GZTbx9TW5pZNHc2C+e01rssSzkozKyu2rt6WLJ8DX39OwDo6e1jyfI1AA6LgvDQk5nV1dKO7ldCYkBf/w6WdnTXqSIr56Aws7ra1Ns3onarPQeFmdXV1JbmEbVb7TkozKyuFs2dTfPEpp3amic2sWju7DpVZOU8mW1mdTUwYe2jnorLQWGZfNii1cLCOa3+vSowB4UNyYctmhl4jsIy+LBFMwMHhWXwYYtmBg4Ky+DDFs0MHBSWwYctmhnkHBSS5knqlrRO0uIh+nxY0sOS1kr6YZ712MgsnNPK5ScfQWtLMwJaW5q5/OQjPJFtNs7kdtSTpCbgKuB9wEbgXkkrIuLhkj6zgCXAMRHxjKT98qrHRseHLZpZnnsURwHrImJ9RGwHbgQWlPU5B7gqIp4BiIgtOdZjZmajkGdQtAIbSp5vTNtKHQwcLOluSaslzau0IknnSuqU1Ll169acyjUzs0rqPZm9GzALOA44FfiOpJbyThGxLCLaIqJtypQpta3QzGycyzMoeoDpJc+npW2lNgIrIqI/In4P/IYkOMzMrCDyDIp7gVmSZkqaBJwCrCjr006yN4GkySRDUetzrMnMzEao6qCQtEd6JFNVIuIl4HygA3gEuDki1kq6TNL8tFsH8JSkh4FVwKKIeKr68s3MLG+KiMoLpAkkewGnAUcCLwKvA7YBtwLfjoh1NarzFW1tbdHZ2VnrtzUzG9Mk3RcRbaN5bdYexSrgzSTnObwxIqZHxH7AscBq4MuSTh/Nm5qZ2diRdcLdCRHRX94YEU8DPwF+ImlibpWZmVkhDBkUAyEh6Q0VFj+XHqn0qiAxM7PGUs1k9v3AVpJDV3+bPn5M0v2S3plncWZmVn/VBMUvgJMiYnJE7AucCPwU+GvgW3kWZ2Zm9VdNUBwdER0DTyLiNuBdEbGa5CgoMzNrYNVcPXazpItJLuoH8BHgD+k5FS/nVpmZmRVCNXsU/5Pk8hvt6c+BaVsT8OG8CjMzs2IYdo8iIrYBFwyxuOYn3JmZWW0NGxSSpgB/C7wV2H2gPSLem2NdZmZWENUMPf0AeBSYCXweeIzkgn9mZjYOVDOZvW9EXCvpwoi4E7hTkoNinGnv6mFpRzebevuY2tLMormzfYtUs3GimqAYOPt6s6QPAJuASmdrW4Nq7+phyfI19PXvAKCnt48ly9cAOCzMxoFqhp6+KGlv4CLgb4BrgE/mWpUVytKO7ldCYkBf/w6WdnTXqSIzq6Vqjnr6afrwWeA9+ZZjRbSpt29E7WbWWKo56mkmyeGxM0r7R8T8oV5jjWVqSzM9FUJhaktzHaoxs1qrZo6iHbgW+Bd8Jva4tGju7J3mKACaJzaxaO7sOlZlZrVSTVC8EBFX5l6JFdbAhLWPejIbn6oJim9IugS4jeR2qABExP25VWWFs3BOq4PBbJyqJiiOAD4KvJfBoadIn5uZWYOrJig+BLwpIrbnXYyZmRVPNedRPAS05FyHmZkVVDV7FC3Ao+llO0rnKHx4rJnZOFBNUFySexVmZlZYQwaFJEXizuH65FOamZkVQdYcxSpJF0g6sLRR0iRJ75X0PeDMfMszM7N6yxp6mgf8JfCj9DIevUAzSbjcBnw9Irpyr9DMzOpqyKCIiBeAbwHfkjQRmAz0RURvjWozM7MCqGYym4joBzbnXIuZmRVQNedRmJnZOOagMDOzTMMGRXrk0z61KMbMzIqnmj2K/YF7Jd0saZ4k5V2UmZkVx7BBERGfBWaR3LzoLOC3kr4k6c0512ZmZgVQ1RxFevb1k+nPS8A+wC2SvpJjbWZmVgDVzFFcKOk+4CvA3cAREXEe8E7gg8O8dp6kbknrJC3O6PdBSSGpbYT1m5lZzqo5j+INwMkR8XhpY0S8LOm/D/UiSU3AVcD7gI0k8xwrIuLhsn57ARcC94y0eDMzy181cxSXlIdEybJHMl56FLAuItanNz26EVhQod8XgC8DL1RRr5mZ1Vie51G0AhtKnm9M214h6R3A9Ii4NWtFks6V1Cmpc+vWrbu+UrMM7V09HHPF7cxcfCvHXHE77V099S7JrKbqdsKdpAnAPwAXDdc3IpZFRFtEtE2ZMiX/4sxS7V09LFm+hp7ePgLo6e1jyfI1DgsbV/IMih5gesnzaWnbgL2Aw4E7JD0GHA2s8IS2FcnSjm76+nfs1NbXv4OlHd11qsis9vIMinuBWZJmSpoEnAKsGFgYEc9GxOSImBERM4DVwPyI6MyxJrMR2dTbN6J2s0aUW1BExEvA+UAH8Ahwc0SslXSZJN9v28aEqS3NI2o3a0RVXWZ8tCJiJbCyrO1zQ/Q9Ls9azEZj0dzZLFm+Zqfhp+aJTSyaO7uOVZnVVq5BYTbWLZyTHKi3tKObTb19TG1pZtHc2a+0m40HDgqzYSyc0+pgsHHN96MwM7NMDgozM8vkoDAzs0wOCjMzy+SgMDOzTA4KMzPL5KAwM7NMDgozM8vkoDAzs0wOCjMzy+SgMDOzTA4KMzPL5KAwM7NMDgozM8vkoDAzs0wOCjMzy+SgMDOzTA4KMzPL5KAwM7NMDgozM8vkoDAzs0wOCjMzy+SgMDOzTA4KMzPL5KAwM7NMDgozM8vkoDAzs0wOCjMzy+SgMDOzTA4KMzPL5KAwM7NMDgozM8uUa1BImiepW9I6SYsrLP+UpIclPSjpV5IOyrMeMzMbudyCQlITcBVwInAYcKqkw8q6dQFtEfE24BbgK3nVY2Zmo5PnHsVRwLqIWB8R24EbgQWlHSJiVUQ8nz5dDUzLsR4zMxuFPIOiFdhQ8nxj2jaUs4GfVVog6VxJnZI6t27dugtLNDOz4RRiMlvS6UAbsLTS8ohYFhFtEdE2ZcqU2hZnZjbO7ZbjunuA6SXPp6VtO5F0AvAZ4N0R8WKO9ZiZ2SjkuUdxLzBL0kxJk4BTgBWlHSTNAb4NzI+ILTnWYmZmo5RbUETES8D5QAfwCHBzRKyVdJmk+Wm3pcCewI8l/VrSiiFWZ2ZmdZLn0BMRsRJYWdb2uZLHJ+T5/mZm9toVYjLbzMyKy0FhZmaZHBRmZpbJQWFmZplyncw2s9euvauHpR3dbOrtY2pLM4vmzmbhnKyLHJjtWg4KswJr7+phyfI19PXvAKCnt48ly9cAOCysZjz0ZFZgSzu6XwmJAX39O1ja0V2nimw8clCYFdim3r4RtZvlwUFhVmBTW5pH1G6WBweFWYEtmjub5olNO7U1T2xi0dzZdarIxiNPZpsV2MCEtY96snpyUJgV3MI5rQ4GqysPPZmZWSYHhZmZZXJQmJlZJgeFmZllclCYmVkmB4WZmWVyUJiZWSYHhZmZZXJQmJlZJgeFmZllclCYmVkmB4WZmWVyUJiZWSYHhZmZZXJQmJlZJgeFmZllclCYmVkmB4WZmWVyUJiZWSYHhZmZZXJQmJlZJgeFmZllclCYmVmmXINC0jxJ3ZLWSVpcYfnrJN2ULr9H0ow86zEzs5HLLSgkNQFXAScChwGnSjqsrNvZwDMR8Rbga8CX86rHzMxGJ889iqOAdRGxPiK2AzcCC8r6LAC+lz6+BTheknKsyczMRmi3HNfdCmwoeb4R+NOh+kTES5KeBfYFtpV2knQucG769EVJD+VS8dgzmbJtNY55WwzythjkbTFo9mhfmGdQ7DIRsQxYBiCpMyLa6lxSIXhbDPK2GORtMcjbYpCkztG+Ns+hpx5gesnzaWlbxT6SdgP2Bp7KsSYzMxuhPIPiXmCWpJmSJgGnACvK+qwAzkwf/zlwe0REjjWZmdkI5Tb0lM45nA90AE3AdyNiraTLgM6IWAFcC9wgaR3wNEmYDGdZXjWPQd4Wg7wtBnlbDPK2GDTqbSH/AW9mZll8ZraZmWVyUJiZWabCBoUv/zGoim3xKUkPS3pQ0q8kHVSPOmthuG1R0u+DkkJSwx4aWc22kPTh9HdjraQf1rrGWqni/8iBklZJ6kr/n5xUjzrzJum7krYMda6ZElem2+lBSe+oasURUbgfksnv3wFvAiYBDwCHlfX5a+Dq9PEpwE31rruO2+I9wJ+kj88bz9si7bcXcBewGmird911/L2YBXQB+6TP96t33XXcFsuA89LHhwGP1bvunLbFfwPeATw0xPKTgJ8BAo4G7qlmvUXdo/DlPwYNuy0iYlVEPJ8+XU1yzkojqub3AuALJNcNe6GWxdVYNdviHOCqiHgGICK21LjGWqlmWwTw+vTx3sCmGtZXMxFxF8kRpENZAFwfidVAi6QDhltvUYOi0uU/WofqExEvAQOX/2g01WyLUmeT/MXQiIbdFumu9PSIuLWWhdVBNb8XBwMHS7pb0mpJ82pWXW1Vsy0uBU6XtBFYCVxQm9IKZ6TfJ8AYuYSHVUfS6UAb8O5611IPkiYA/wCcVedSimI3kuGn40j2Mu+SdERE9NazqDo5FbguIr4q6V0k528dHhEv17uwsaCoexS+/MegarYFkk4APgPMj4gXa1RbrQ23LfYCDgfukPQYyRjsigad0K7m92IjsCIi+iPi98BvSIKj0VSzLc4GbgaIiH8Hdie5YOB4U9X3SbmiBoUv/zFo2G0haQ7wbZKQaNRxaBhmW0TEsxExOSJmRMQMkvma+REx6ouhFVg1/0faSfYmkDSZZChqfQ1rrJVqtsUTwPEAkg4lCYqtNa2yGFYAZ6RHPx0NPBsRm4d7USGHniK/y3+MOVVui6XAnsCP0/n8JyJift2KzkmV22JcqHJbdADvl/QwsANYFBENt9dd5ba4CPiOpE+STGyf1Yh/WEr6EckfB5PT+ZhLgIkAEXE1yfzMScA64HngL6pabwNuKzMz24WKOvRkZmYF4aAwM7NMDgozM8vkoDAzs0wOCjMzy+SgMKuCpGZJd0pqyvE9Jkm6Kz2B1KwwHBRm1flLYHlE7MjrDdIL2v0K+Ehe72E2Gg4KG9ckHZlel393SXuk9204vELX04B/Tl9zXLp38c+S1ku6QtJpkv5D0hpJb077/Vl6r5QuSb+UtH/aPkXSL9L3ukbS4+mZ05CcTX1aDT66WdV8wp2Ne5K+SHJJh2ZgY0RcXrZ8EsnZ7m9Mnx9H8oV+KMlVAdYD10TEJZIuBGZGxCck7QP0RkRI+hhwaERcJOmbQE9EXJ5e0fVnwJSI2JYObT0ZEVNq8NHNquKxUDO4jOR6QS8AH6+wfDLQW9Z278A1ciT9DrgtbV9DciMpSC64dlN6vf9JwO/T9mOB/wEQET+X9MzASiNih6TtkvaKiOde6wcz2xU89GSW3MdkT5Krz+5eYXlfhfbSK/S+XPL8ZQb/APtH4JsRcQTwv4ZYdyWvo7FvumRjjIPCLLny7t8BPyC5M95O0jvENUmq9ot+wN4MXsL5zJL2u4EPA0h6P7DPwAJJ+wLbIqJ/hO9llhsHhY1rks4A+iPih8AVwJGS3luh620kQ0YjcSnJFX3vA7aVtH+e5KquDwEfAp4EBoaZ3gM0+t35bIzxZLZZFdJbrH4yIj66C9b1OmBHennsdwH/FBFvT5ctBxZHxG9e6/uY7SqezDarQkTcL2mVpKZdcC7FgcDN6a1btwPnwCtHV7U7JKxovEdhZmaZPEdhZmaZHBRmZpbJQWFmZpkcFGZmlslBYWZmmf4/OP0Zmw7ftwAAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"m = job.measurements['demo_photometry.ZeropointRMS']\n",
"\n",
"plt.figure()\n",
"plt.scatter(m.extras['x'].quantity, m.extras['y'].quantity)\n",
"plt.xlabel('%s (%s)'%(m.extras['x'].label, m.extras['x'].unit.name))\n",
"plt.ylabel('%s (%s)'%(m.extras['y'].label, m.extras['y'].unit.name))\n",
"plt.title('%s; %s'%(m.metric_name.metric, job.meta[\"version\"]))\n",
"plt.xlim(0, 1)\n",
"plt.ylim(0, 1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, the particular values used in this example are just for demonstration purposes."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "LSST",
"language": "python",
"name": "lsst"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}