{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SciUnit Logo](https://raw.githubusercontent.com/scidash/assets/master/logos/sciunit.png)\n",
    "\n",
    "# SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests.  \n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "# Chapter 3. Testing with help from the SciUnit standard library\n",
    "(or [back to Chapter 2](chapter2.ipynb))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If you are using this file in Google Colab, this block of code can help you install sciunit from PyPI in Colab environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "    IN_COLAB = True\n",
    "except:\n",
    "    IN_COLAB = False\n",
    "if IN_COLAB:\n",
    "    !pip install -q sciunit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sciunit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In this chapter we will use the same toy model in Chapter 1 but write a more interesting test with additional features included in SciUnit. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sciunit.models.examples import ConstModel # One of many dummy models included for illustration.  \n",
    "const_model_37 = ConstModel(37, name=\"Constant Model 37\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's write a test that validates the `observation` and returns more informative `score` type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sciunit.capabilities import ProducesNumber\n",
    "from sciunit.scores import ZScore # One of many SciUnit score types.  \n",
    "from sciunit.errors import ObservationError # An exception class raised when a test is instantiated \n",
    "                                     # with an invalid observation.\n",
    "    \n",
    "class MeanTest(sciunit.Test):\n",
    "    \"\"\"Tests if the model predicts \n",
    "    the same number as the observation.\"\"\"   \n",
    "    \n",
    "    required_capabilities = (ProducesNumber,) # The one capability required for a model to take this test.  \n",
    "    score_type = ZScore # This test's 'judge' method will return a BooleanScore.  \n",
    "    \n",
    "    def validate_observation(self, observation):\n",
    "        if type(observation) is not dict:\n",
    "            raise ObservationError(\"Observation must be a python dictionary\")\n",
    "        if 'mean' not in observation:\n",
    "            raise ObservationError(\"Observation must contain a 'mean' entry\")\n",
    "        \n",
    "    def generate_prediction(self, model):\n",
    "        return model.produce_number() # The model has this method if it inherits from the 'ProducesNumber' capability.\n",
    "    \n",
    "    def compute_score(self, observation, prediction):\n",
    "        score = ZScore.compute(observation,prediction) # Compute and return a ZScore object.  \n",
    "        score.description = (\"A z-score corresponding to the normalized location of the observation \"\n",
    "                             \"relative to the predicted distribution.\")\n",
    "        return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've done two new things here:\n",
    "- The optional `validate_observation` method checks the `observation` to make sure that it is the right type, that it has the right attributes, etc.  This can be used to ensures that the `observation` is exactly as the other core test methods expect.  If we don't provide the right kind of observation:\n",
    "```python\n",
    "-> mean_37_test = MeanTest(37, name='=37')\n",
    "ObservationError: Observation must be a python dictionary\n",
    "```\n",
    "then we get an error.  In contrast, this is what our test was looking for:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "observation = {'mean':37.8, 'std':2.1}\n",
    "mean_37_test = MeanTest(observation, name='=37')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Instead of returning a `BooleanScore`, encoding a `True`/`False` value, we return a `ZScore` encoding a more quantitative summary of the relationship between the observation and the prediction.  When we execute the test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score = mean_37_test.judge(const_model_37)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we get a more quantitative summary of the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score.summarize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score.describe()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}