{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![SciUnit Logo](https://raw.githubusercontent.com/scidash/assets/master/logos/sciunit.png)\n", "\n", "# SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests. \n", "\n", "\"Open\n", "\n", "# Chapter 2. Writing a `model` and `test` in SciUnit from scratch\n", "(or [back to Chapter 1](chapter1.ipynb))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### If you are using this file in Google Colab, this block of code can help you install sciunit from PyPI in Colab environment." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "try:\n", " import google.colab\n", " IN_COLAB = True\n", "except:\n", " IN_COLAB = False\n", "if IN_COLAB:\n", " !pip install -q sciunit" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import sciunit" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### SciUnit works by making models declare and implement capabilities that tests use to interact with those models. \n", "Each `capability` is a subclass of `sciunit.Capability`, and contains one or more unimplemented methods. Here we define a simple capability through which a model can return a single number. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class ProducesNumber(sciunit.Capability):\n", " \"\"\"An example capability for producing some generic number.\"\"\"\n", "\n", " def produce_number(self):\n", " \"\"\"The implementation of this method should return a number.\"\"\"\n", " raise NotImplementedError(\"Must implement produce_number.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### SciUnit models subclass `sciunit.Model` as well as each `sciunit.Capability` they aim to implement. \n", "Here we create a trivial model class that is instantiated with a single constant. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sciunit.capabilities import ProducesNumber # One of many potential model capabilities." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class ConstModel(sciunit.Model, \n", " ProducesNumber):\n", " \"\"\"A model that always produces a constant number as output.\"\"\"\n", " \n", " def __init__(self, constant, name=None):\n", " self.constant = constant \n", " super(ConstModel, self).__init__(name=name, constant=constant)\n", "\n", " def produce_number(self):\n", " return self.constant" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A `model` we want to test is always an instance (with specific model arguments) of a more generic `model` class. \n", "Here we create an instance of `ConstModel` that will always produce the number 37 and give it a name. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "const_model_37 = ConstModel(37, name=\"Constant Model 37\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A SciUnit test class must contain:\n", "1. the capabilities a model requires to take the test. \n", "2. the type of score that it will return\n", "3. an implementation of `generate_prediction`, which will use the model's capabilities to get some values out of the model.\n", "4. an implementaiton of `compute_score`, to use the provided observation and the generated prediction to compute a sciunit `Score`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sciunit.scores import BooleanScore # One of several SciUnit score types. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class EqualsTest(sciunit.Test):\n", " \"\"\"Tests if the model predicts \n", " the same number as the observation.\"\"\" \n", " \n", " required_capabilities = (ProducesNumber,) # The one capability required for a model to take this test. \n", " score_type = BooleanScore # This test's 'judge' method will return a BooleanScore. \n", " \n", " def generate_prediction(self, model):\n", " return model.produce_number() # The model has this method if it inherits from the 'ProducesNumber' capability.\n", " \n", " def compute_score(self, observation, prediction):\n", " score = self.score_type(observation['value'] == prediction) # Returns a BooleanScore. \n", " score.description = 'Passing score if the prediction equals the observation'\n", " return score" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A SciUnit test is a specific instance of a `test` class, parameterized by the observation (i.e. the empirical data that the `model` aims to recapitulate). \n", "Here we create a test instance parameterized by the observation 37.0. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "equals_37_test = EqualsTest({'value':37}, name='=37')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Every test has a `judge` method which executes the test and returns a `score` for the provide model. \n", "Here we judge the model we just created using the test we just created. The `judge` method does a lot of things behind the scenes: \n", "1. It checks to makes sure that your `model` expresses each `capability` required to take the test. It doesn't check to see if they are implemented correctly (how could it know?) but it does check to make sure the `model` at least claims (through inheritance) to express each `capability`. The required capabilities are none other than those in the test's `required_capabilities` attribute. Since `ProducesNumber` is the only required capability, and the `ConstModel` class inherits from the corresponding capability class, that check passes.\n", "2. It calls the test's `generate_prediction` method, which uses the model's capabilities to make the model return some quantity of interest, in this case a characteristic number.\n", "3. It calls the test's `compute_score` method, which compares the observation the test was instantiated with against the prediction returned in the previous step. This comparison of quantities is cast into a score (in this case, a `BooleanScore`), bound to some `model` output of interest (in this case, the number produces by the `model`), and that `score` object is returned.\n", "4. The `score` returned is checked to make sure it is of the type promised in the class definition, i.e. that a `BooleanScore` is returned if a `BooleanScore` is listed in the `score_type` attribute of the `test`.\n", "5. The `score` is bound to the `test` that returned it, the `model` that took the `test`, and the prediction and observation that were used to compute it." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score = equals_37_test.judge(const_model_37)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A score is an object containing information about the result of the test, and the provenance of that result. \n", "Printing the `score` just prints a representation of its value (for a `BooleanScore`, `True` has the representation 'Pass')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also summarize the `score` in its entirety, printing information about the associated `model` and `test`. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score.summarize()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How was that score computed again? " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Several logically related tests can be grouped using a `TestSuite`. \n", "These can be instances of the same test class (instantiated with different observations) or instances of different test classes. Anything tests that you think belongs together can be part of a TestSuite. A test can be a part of many different suites at once. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "equals_1_test = EqualsTest({'value':1}, name='=1') # Test that model output equals 1. \n", "equals_2_test = EqualsTest({'value':2}, name='=2') # Test that model output equals 2. \n", "\n", "equals_suite = sciunit.TestSuite([equals_1_test, equals_2_test, equals_37_test], name=\"Equals test suite\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can test our model using this TestSuite, and display the results. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score_matrix = equals_suite.judge(const_model_37)\n", "score_matrix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can create more models and subject those to the test suite to get a more extensive score matrix." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "const_model_1 = ConstModel(1, name='Constant Model 1')\n", "const_model_2 = ConstModel(2, name='Constant Model 2')\n", "score_matrix = equals_suite.judge([const_model_1, const_model_2, const_model_37])\n", "score_matrix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also examine the results only for one of the tests in the suite." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score_matrix[equals_1_test]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or examine the results only for one of the models. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score_matrix[const_model_2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### In the next section we'll see how to build slightly more sophisticated tests using objects built-in to SciUnit. \n", "### Onto [Chapter 3](chapter3.ipynb)!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }