{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "![SciUnit Logo](https://raw.githubusercontent.com/scidash/assets/master/logos/SciUnit/sci-unit-tag.png)\n", "\n", "\"Open\n", "\n", "# Chapter 6. Workshop Tutorial, Real Example\n", "(or [back to Chapter 5](chapter5.ipynb))" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Install SciUnit if necessary\n", "!pip install -q sciunit" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Import the package\n", "import sciunit" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Add some default CSS styles for these examples\n", "sciunit.utils.style()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "
Toy example: A brief history of cosmology

" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from IPython.display import HTML\n", "HTML(\"\"\"\"\"\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
Experimentalists
Modelers\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
BabyloniansBraheGalileoLe Verrier
Ptolemy
Copernicus
Kepler
Newton
Einstein
\n", "
\n", "\n", "\n", " \n", " \n", " \n", "
PassFailUnclear
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Model validation goals: " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### - Generate one unit tests for each experimental datum (or stylized fact about data)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### - Execute these tests against all models capable of taking them" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### - Programatically display the results as a “dashboard\" of model validity\n", " - Optionally record and display non-boolean test results, test artifacts, etc." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### High-level workflow for validation:" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "```python\n", "# Hypothetical examples of data-driven tests\n", "from cosmounit.tests import brahe_test, galileo_test, leverrier_test\n", "\n", "# Hypothetical examples of parameterized models\n", "from cosmounit.models import ptolemy_model, copernicus_model \n", "\n", "# Execute one test against one model and return a test score\n", "score = brahe_test.judge(copernicus_model) \n", "```\n", "\n", "This is the only code-like cell of the tutorial that **doesn't** contain executable code, since it is a high-level abstraction. Don't worry, you'll be running real code just a few cells down!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Q: How does a test “know\" how to test a model?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### A: Through guarantees that models provide to tests, called “Capabilities\"." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "[Code for **sciunit.capabilities** on GitHub](https://github.com/scidash/sciunit/tree/master/sciunit/capabilities.py)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### Next we show an example of a Capability relevant to the cosmology case outlined above." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: sympy in /opt/mambaforge/lib/python3.9/site-packages (1.8)\n", "Requirement already satisfied: mpmath>=0.19 in /opt/mambaforge/lib/python3.9/site-packages (from sympy) (1.2.1)\n" ] } ], "source": [ "!pip install sympy" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "# Some imports to make the code below run\n", "from math import pi, sqrt, sin, cos, tan, atan\n", "from datetime import datetime, timedelta\n", "import numpy as np\n", "# SymPy is needed because one of Kepler's equations\n", "# is in implicit form and must be solved numerically!\n", "from sympy import Symbol, sin as sin_\n", "from sympy.solvers.solvers import nsolve" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "class ProducesOrbitalPosition(sciunit.Capability):\n", " \"\"\"\n", " A model `capability`, i.e. a collection of methods that a test is allowed to invoke on a model.\n", " These methods are unimplemented by design, and the model must implement them.\n", " \"\"\"\n", " \n", " def get_position(self, t: datetime) -> tuple:\n", " \"\"\"Produce an orbital position from a time point\n", " in polar coordinates.\n", " \n", " Args:\n", " t (datetime): The time point to examine, relative to perihelion\n", " Returns:\n", " tuple: A pair of (r, theta) coordinates in the oribtal plane\n", " \"\"\"\n", " raise NotImplementedError(\"\")\n", " \n", " @property\n", " def perihelion(self) -> datetime:\n", " \"\"\"Return the time of last perihelion\"\"\"\n", " raise NotImplementedError(\"\")\n", " \n", " @property\n", " def period(self) -> float:\n", " \"\"\"Return the period of the orbit\"\"\"\n", " raise NotImplementedError(\"\")\n", " \n", " @property\n", " def eccentricity(self) -> float:\n", " \"\"\"Return the eccentricity of the orbit\"\"\"\n", " raise NotImplementedError(\"\")\n", " \n", " def get_x_y(self, t: datetime) -> tuple:\n", " \"\"\"Produce an orbital position from a time point, but in cartesian coordinates.\n", " This method does not require a model-specific implementation.\n", " Thus, a generic implementation can be provided in advance.\"\"\"\n", " raise NotImplementedError(\"\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### [SciUnit](http://sciunit.scidash.org) (and domain specific libraries that build upon it) also define their own capabilities" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# An extremely generic model capability\n", "from sciunit.capabilities import ProducesNumber\n", "\n", "# A specific model capability used in neurophysiology\n", "#from neuronunit.capabilities import HasMembranePotential" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Now we can define a model class that implements this `ProducesOrbitalPosition` capability by inheritance. " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### All models are subclasses of `sciunit.Model` and typically one or more subclasses of `sciunit.Capability`." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "class BaseKeplerModel(sciunit.Model, \n", " ProducesOrbitalPosition):\n", " \"\"\"A sciunit model class corresponding to a Kepler-type model\n", " of an object in the solar system. This model has the \n", " `ProducesOrbitalPosition` capability by inheritance,\n", " so it must implement all of the unimplemented methods of that capability\"\"\"\n", " \n", " def get_position(self, t):\n", " \"\"\"Implementation of polar coordinate position as a function of time\"\"\"\n", " r, theta = self.heliocentric_distance(t), self.true_anomaly(t)\n", " return r, theta\n", " \n", " @property\n", " def perihelion(self):\n", " \"\"\"Implementation of time of last perihelion\"\"\"\n", " return self.params['perihelion']\n", " \n", " @property\n", " def period(self):\n", " \"\"\"Implementation of period of the orbit\"\"\"\n", " return self.params['period']\n", " \n", " @property\n", " def eccentricity(self):\n", " \"\"\"Implementation of orbital eccentricity (assuming elliptic orbit)\"\"\"\n", " a, b = self.params['semimajor_axis'], self.params['semiminor_axis']\n", " return sqrt(1 - (b/a)**2)\n", " \n", " def get_x_y(self, t: datetime) -> tuple:\n", " \"\"\"Produce an orbital position from a time point, but in cartesian coordinates.\n", " This method does not require a model-specific implementation.\n", " Thus, a generic implementation can be provided in advance.\"\"\"\n", " r, theta = self.get_position(t)\n", " x, y = r*cos(theta), r*sin(theta)\n", " return x, y" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "class KeplerModel(BaseKeplerModel):\n", " \"\"\"This 'full' model contains all of the methods required\n", " to complete the implementation of the `ProducesOrbitalPosition` capability\"\"\"\n", " \n", " def mean_anomaly(self, t):\n", " \"\"\"How long into its period the object is at time `t`\"\"\"\n", " time_since_perihelion = t - self.perihelion\n", " return 2*pi*(time_since_perihelion % self.period)/self.period\n", " \n", " def eccentric_anomaly(self, t):\n", " \"\"\"How far the object has gone into its period at time `t`\"\"\"\n", " E = Symbol('E')\n", " M, e = self.mean_anomaly(t), self.eccentricity\n", " expr = E - e*sin_(E) - M\n", " return nsolve(expr, 0)\n", " \n", " def true_anomaly(self, t):\n", " \"\"\"Theta in a polar coordinate system at time `t`\"\"\"\n", " e, E = self.eccentricity, self.eccentric_anomaly(t)\n", " theta = 2*atan(sqrt(tan(E/2)**2 * (1+e)/(1-e)))\n", " return theta\n", " \n", " def heliocentric_distance(self, t):\n", " \"\"\"R in a polar coordinate system at time `t`\"\"\"\n", " a, e = self.params['semimajor_axis'], self.eccentricity\n", " E = self.eccentric_anomaly(t)\n", " return a*(1-e*cos(E))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Now we can instantiate a specific model from this class, e.g. one representing the orbital path of Earth (according to Kepler)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# The quantities module to put dimensional units on values\n", "import quantities as pq\n", "\n", "# `earth_model` will be a specific instance of KeplerModel, with its own parameters\n", "earth_model = KeplerModel(name = \"Kepler's Earth Model\",\n", " semimajor_axis=149598023 * pq.km, \n", " semiminor_axis=149577161 * pq.km, \n", " period=timedelta(365, 22118), # Period of Earth's orbit \n", " perihelion=datetime(2019, 1, 3, 0, 19), # Time and date of Earth's last perihelion\n", " )" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### We can use this model to make specific predictions, for example the current distance between Earth and the sun." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Heliocentric distance of Earth right now is predicted to be 152035067.4 km\n" ] } ], "source": [ "# The time right now\n", "t = datetime.now()\n", "# Predicted distance from the sun, right now\n", "r = earth_model.heliocentric_distance(t)\n", "print(\"Heliocentric distance of Earth right now is predicted to be %s\" % r.round(1))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Now let's build a test class that we might use to validate (i.e. unit test to produce test scores) with this (and hopefully other) models" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### First, what kind of scores do we want our test to return?" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Several score types available in SciUnit\n", "from sciunit.scores import BooleanScore, ZScore, RatioScore, PercentScore # etc., etc." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "[Code for **sciunit.scores** on GitHub](https://github.com/scidash/sciunit/tree/master/sciunit/scores)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Here's a first shot a test class for assessing the agreement between predicted and observed positions of orbiting objects. All test classes are subclasses of `sciunit.Test`." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class PositionTest(sciunit.Test):\n", " \"\"\"A test of a planetary position at some specified time\"\"\"\n", " \n", " # This test can only operate on models that implement\n", " # the `ProducesOrbitalPosition` capability.\n", " required_capabilities = (ProducesOrbitalPosition,)\n", " score_type = BooleanScore # This test's 'judge' method will return a BooleanScore.\n", " \n", " def generate_prediction(self, model):\n", " \"\"\"Generate a prediction from a model\"\"\"\n", " t = self.observation['t'] # Get the time point from the test's observation\n", " x, y = model.get_x_y(t) # Get the predicted x, y coordinates from the model\n", " return {'t': t, 'x': x, 'y': y} # Roll this into a model prediction dictionary\n", " \n", " def compute_score(self, observation, prediction):\n", " \"\"\"Compute a test score based on the agreement between\n", " the observation (data) and prediction (model)\"\"\"\n", " # Compare observation and prediction to get an error measure\n", " delta_x = observation['x'] - prediction['x']\n", " delta_y = observation['y'] - prediction['y']\n", " error = np.sqrt(delta_x**2 + delta_y**2)\n", " \n", " passing = bool(error < 1e5*pq.kilometer) # Turn this into a True/False score\n", " score = self.score_type(passing) # Create a sciunit.Score object\n", " score.set_raw(error) # Add some information about how this score was obtained\n", " score.description = (\"Passing score if the prediction is \"\n", " \"within < 100,000 km of the observation\") # Describe the scoring logic\n", " return score" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### We might want to include extra checks and constraints on observed data, test parameters, or other contingent testing logic." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class StricterPositionTest(PositionTest):\n", " # Optional observation units to validate against\n", " units = pq.meter\n", " \n", " # Optional schema for the format of observed data\n", " observation_schema = {'t': {'min': 0, 'required': True},\n", " 'x': {'units': True, 'required': True},\n", " 'y': {'units': True, 'required': True},\n", " 'phi': {'required': False}}\n", " \n", " def validate_observation(self, observation):\n", " \"\"\"Additional checks on the observation\"\"\"\n", " assert isinstance(observation['t'], datetime)\n", " return observation\n", " \n", " # Optional schema for the format of test parameters\n", " params_schema = {'rotate': {'required': False}}\n", "\n", " # Optional schema for the format of default test parameters\n", " default_params = {'rotate': False}\n", " \n", " def compute_score(self, observation, prediction):\n", " \"\"\"Optionally use additional information to compute model/data agreement\"\"\"\n", " observation_rotated = observation.copy()\n", " if 'phi' in observation:\n", " # Project x and y values onto the plane defined by `phi`.\n", " observation_rotated['x'] *= cos(observation['phi'])\n", " observation_rotated['y'] *= cos(observation['phi'])\n", " return super().compute_score(observation_rotated, prediction)\n", " " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Now we can instantiate a test. Each test instance is a combination of the test class, describing the testing logic and required capabilties, plus some 'observation', i.e. data. " ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# A single test instance, best on the test class `StricterPositionTest` combined with\n", "# a specific set of observed data (a time and some x, y coordinates)\n", "# N.B.: This data is made up for illustration purposes\n", "earth_position_test_march = StricterPositionTest(name = \"Earth Orbital Data on March 1st, 2019\",\n", " observation = {'t': datetime(2019, 3, 1),\n", " 'x': 7.905e7 * pq.km, \n", " 'y': 1.254e8 * pq.km})" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Finally, we can execute this one test against this one model" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "Pass" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Execute `earth_position_test` against `earth_model` and return a score\n", "score = earth_position_test_march.judge(earth_model)\n", "# Display the score\n", "score" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### And we can get additional information about the test, including intermediate objects computed in order to generate a score." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "'Passing score if the prediction is within < 100,000 km of the observation'" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Describe the score in plain language\n", "score.describe()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "({'t': datetime.datetime(2019, 3, 1, 0, 0),\n", " 'x': array(79046604.57417324) * km,\n", " 'y': array(1.25401809e+08) * km},\n", " {'t': datetime.datetime(2019, 3, 1, 0, 0),\n", " 'x': array(79050000.) * km,\n", " 'y': array(1.254e+08) * km})" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# What were the prediction and observation used to compute the score?\n", "score.prediction, score.observation" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "array(3847.28076371) * km" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# What was the raw error before the decision criterion was applied?\n", "score.get_raw()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### We may want to bundle many such tests into a `TestSuite`. This suite may contain test from multiple classes, or simply tests which differ only in the observation (data) used to instantiate them." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# A new test for a new month: same test class, new observation (data)\n", "# N.B. I deliberately picked \"observed\" values that will make the model fail this test\n", "earth_position_test_april = StricterPositionTest(name = \"Earth Orbital Data on April 1st, 2019\",\n", " observation = {'t': datetime(2019, 4, 1),\n", " 'x': 160000 * pq.km, \n", " 'y': 70000 * pq.km})\n", "\n", "# A test suite built from both of the tests that we have instantiated\n", "earth_position_suite = sciunit.TestSuite([earth_position_test_march,\n", " earth_position_test_april],\n", " name = 'Earth observations in Spring, 2019')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### We can then test our model against this whole suite of tests" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[38;2;60;169;88m\u001b[48;2;50;50;50mScore: Pass for Kepler's Earth Model on Earth Orbital Data on March 1st, 2019\u001b[0m\n", "\u001b[38;2;230;78;52m\u001b[48;2;50;50;50mScore: Fail for Kepler's Earth Model on Earth Orbital Data on April 1st, 2019\u001b[0m\n" ] } ], "source": [ "# Run the whole suite (two tests) against one model\n", "scores = earth_position_suite.judge(earth_model)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Rich HTML output is automatically produced when this score output is summarized" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", "
Earth Orbital Data on March 1st, 2019 Earth Orbital Data on April 1st, 2019
Kepler's Earth ModelPassFail
" ], "text/plain": [ " Earth Orbital Data on March 1st, 2019 \\\n", "Kepler's Earth Model Pass \n", "\n", " Earth Orbital Data on April 1st, 2019 \n", "Kepler's Earth Model Fail " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Display the returned `scores` object\n", "scores" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### We can then expand this to multiple models" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Just like the Kepler model, but returning a random orbital angle\n", "class RandomModel(KeplerModel):\n", " def get_position(self, t):\n", " r, theta = super().get_position(t)\n", " return r, 2*pi*np.random.rand()" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# A new model instance, using the same parameters but a different underlying model class\n", "random_model = RandomModel(name = \"Random Earth Model\",\n", " semimajor_axis=149598023 * pq.km, \n", " semiminor_axis=149577161 * pq.km, \n", " period=timedelta(365, 22118), # Period of Earth's orbit \n", " perihelion=datetime(2019, 1, 3, 0, 19), # Time and date of Earth's last perihelion\n", " )" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[38;2;60;169;88m\u001b[48;2;50;50;50mScore: Pass for Kepler's Earth Model on Earth Orbital Data on March 1st, 2019\u001b[0m\n", "\u001b[38;2;230;78;52m\u001b[48;2;50;50;50mScore: Fail for Kepler's Earth Model on Earth Orbital Data on April 1st, 2019\u001b[0m\n", "\u001b[38;2;230;78;52m\u001b[48;2;50;50;50mScore: Fail for Random Earth Model on Earth Orbital Data on March 1st, 2019\u001b[0m\n", "\u001b[38;2;230;78;52m\u001b[48;2;50;50;50mScore: Fail for Random Earth Model on Earth Orbital Data on April 1st, 2019\u001b[0m\n" ] } ], "source": [ "# Run the whole suite (two tests) against two models\n", "scores = earth_position_suite.judge([earth_model, random_model])" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Earth Orbital Data on March 1st, 2019 Earth Orbital Data on April 1st, 2019
Kepler's Earth ModelPassFail
Random Earth ModelFailFail
" ], "text/plain": [ " Earth Orbital Data on March 1st, 2019 \\\n", "Kepler's Earth Model Pass \n", "Random Earth Model Fail \n", "\n", " Earth Orbital Data on April 1st, 2019 \n", "Kepler's Earth Model Fail \n", "Random Earth Model Fail " ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Display the returned `scores` object\n", "scores" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Or extract just a slice:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "Earth Orbital Data on March 1st, 2019 Pass\n", "Earth Orbital Data on April 1st, 2019 Fail\n", "Name: Kepler's Earth Model, dtype: object" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# All the scores for just one model\n", "scores[earth_model]" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "Kepler's Earth Model Pass\n", "Random Earth Model Fail\n", "Name: Earth Orbital Data on March 1st, 2019, dtype: object" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# All the scores for just one test\n", "scores[earth_position_test_march]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### What about models that can't take a certain test? Some models aren't capable (even in principle) of doing what the test is asking of them." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# A simple model which has some capabilities, \n", "# but not the ones needed for the orbital position test\n", "class SimpleModel(sciunit.Model,\n", " sciunit.capabilities.ProducesNumber):\n", " pass\n", "\n", "simple_model = SimpleModel()" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[38;2;60;169;88m\u001b[48;2;50;50;50mScore: Pass for Kepler's Earth Model on Earth Orbital Data on March 1st, 2019\u001b[0m\n", "\u001b[38;2;230;78;52m\u001b[48;2;50;50;50mScore: Fail for Kepler's Earth Model on Earth Orbital Data on April 1st, 2019\u001b[0m\n", "\u001b[38;2;230;78;52m\u001b[48;2;50;50;50mScore: Fail for Random Earth Model on Earth Orbital Data on March 1st, 2019\u001b[0m\n", "\u001b[38;2;230;78;52m\u001b[48;2;50;50;50mScore: Fail for Random Earth Model on Earth Orbital Data on April 1st, 2019\u001b[0m\n", "\u001b[38;2;128;128;128m\u001b[48;2;50;50;50mScore: N/A for SimpleModel on Earth Orbital Data on March 1st, 2019\u001b[0m\n", "\u001b[38;2;128;128;128m\u001b[48;2;50;50;50mScore: N/A for SimpleModel on Earth Orbital Data on April 1st, 2019\u001b[0m\n" ] } ], "source": [ "# Run the whole suite (two tests) against two models\n", "scores = earth_position_suite.judge([earth_model, random_model, simple_model])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Incapable models don't fail, they get the equivalent of 'incomplete' grades" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Earth Orbital Data on March 1st, 2019 Earth Orbital Data on April 1st, 2019
Kepler's Earth ModelPassFail
Random Earth ModelFailFail
SimpleModelN/AN/A
" ], "text/plain": [ " Earth Orbital Data on March 1st, 2019 \\\n", "Kepler's Earth Model Pass \n", "Random Earth Model Fail \n", "SimpleModel N/A \n", "\n", " Earth Orbital Data on April 1st, 2019 \n", "Kepler's Earth Model Fail \n", "Random Earth Model Fail \n", "SimpleModel N/A " ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Display the returned `scores` object\n", "scores" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### [SciUnit](http://sciunit.scidash.org) is in use in several multiscale modeling projects including:\n", "#### - [The Human Brain Project](https://www.humanbrainproject.eu/en/) (neurophysiology, neuroanatomy, neuroimaging)\n", "#### - [OpenWorm](http://openworm.org/) (biophysics, network dynamics, animal behavior)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### [NeuronUnit](http://neuronunit.scidash.org) is a reference implementation in the domain of neurophysiology of:\n", "#### - model classes\n", "#### - test classes\n", "#### - capability classes\n", "#### - tools for constructing tests from several public neurophysiology databases\n", "#### - tools for implementing capabilities from standard model exchange formats\n", "#### - tools for executing simulations underlying testing using popular simulators\n", "#### - test-driven model optimization" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### [SciDash](http://dash.scidash.org) is a web application for creating, scheduling, and viewing the results of SciUnit tests without writing a single line of code." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "
\n", "
\n", "Links:\n", "
\n", "\n", "\n", " \n", " \n", "\n", "\n", " \n", " \n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "
\n", "
\n", "Funded by:\n", "

\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "
R01DC018455 (NIDCD)R01MH106674 (NIMH)
R01EB021711 (NIBIB)Human Brain Project
\n", "\n", "### Thanks also to Sharon Crook, Justas Birgiolas, Russell Jarvis, and Cyrus Omar" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.4" } }, "nbformat": 4, "nbformat_minor": 4 }