{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing notebooks\n",
    "\n",
    "> _The f🥇rst test is the hardest to write._\n",
    "\n",
    "Tests are investments and testing over time measures the return on investment.  Testing promotes\n",
    "\n",
    "* Longevity\n",
    "* Protection from upstream changes.  \n",
    "* Value to you and consumers of your software.\n",
    "\n",
    "Often authors defer to notebooks to experiment and test computational ideas.  There are testing tools for complete notebooks like _informal_ tests with nbconvert or _formal_ tests with nbval.  These tests approaches apply tests to static documents outside of the interactive computing context.\n",
    "\n",
    "> **_Remember_ 💭 [Restart and run all or it didn't happen](2018-07-16-Testing-restart-run-all.ipynb).**\n",
    "\n",
    "This essay discusses using formal testing tools during interactive computing contexts in modern computational notebooks.  These tools include: [__doctest__](https://docs.python.org/3/library/doctest.html), [__unittest__](https://docs.python.org/3/library/unittest.html), and [__pytest__](https://doc.pytest.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This topic has been discussed in some of our past presentations.\n",
    "\n",
    "* https://github.com/deathbeds/LitAF\n",
    "* https://github.com/deathbeds/nostalgiaforever\n",
    "* https://github.com/deathbeds/wtf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Assertions\n",
    "\n",
    "This document commonly uses `assert` statements to indicate a test is happening.  Assertions are useful in testing, but not in production.  [`assert` statements can introduce potential security vulnerabilites](https://hackernoon.com/10-common-security-gotchas-in-python-and-how-to-avoid-them-e19fbe265e03#91f9) for optimized python code.\n",
    "\n",
    "> [Only use assert statements to communicate with other developers, such as in unit tests or in to guard against incorrect API usage.](https://hackernoon.com/10-common-security-gotchas-in-python-and-how-to-avoid-them-e19fbe265e03#91f9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Doctest\n",
    "\n",
    "[`doctest`](https://docs.python.org/3/library/doctest.html) is the simplest Python testing tool to use in the notebook.  `doctest`s are tests that live in strings and Python docstrings, and `>>>` indicates a place where a test is run.\n",
    "\n",
    "[`doctest` discovers tests](https://docs.python.org/3/library/doctest.html#which-docstrings-are-examined) in the docstrings of functions, classes, modules, and a special dictionary `__test__`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def a_function_with_a_doctest(i): \n",
    "        \"\"\"Converts `i` into a string representation\n",
    "        \n",
    "        >>> assert all(isinstance(\n",
    "        ... a_function_with_a_doctest(object), str) for object in (range(10), 1))\n",
    "        \"\"\"\n",
    "        return str(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`__doc__` the module docstring is tested.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "    __doc__ = \"\"\"\n",
    "    For the sake of example, the docstring for the module is created if it doesn't exist\n",
    "    or appended to the original docstring.\n",
    "    \n",
    "    Doctest registers the statement below as a test.\n",
    "    \n",
    "    >>> assert True\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TestResults(failed=0, attempted=2)\n"
     ]
    }
   ],
   "source": [
    "    if __name__ == '__main__': print(__import__('doctest').testmod())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`__test__` is dictionary object that may contain named doctest-able objects like strings, functions, classes, and modules.  The key of the dictionary in the test name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "    __test__ = {\n",
    "        'a_doc_test': \"\"\"This is a docstring with tests that will be run.\n",
    "\n",
    "    >>> assert True\"\"\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conveniently, `__test__` is defined using a private namespace, but this means that a notebook author could easily transport tests within their notebooks and modules without any change to their syntactic choices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unittest\n",
    "\n",
    "[`unittest`](https://docs.python.org/3/library/unittest.html) is a buitlin unit testing framework for python.  It differs from `doctest` in that the tests are actualy python objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "    from unittest import TestCase, main as _main"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`unittest` collects classes subclasses as `TestCase`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "    class myTest(__import__('unittest').TestCase):\n",
    "        \"\"\"This is a unitest class with a docstring.  `doctest`\"\"\"\n",
    "        \n",
    "        def test_assertion(self): assert True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def main():\n",
    "        try: _main(argv=['discover'])\n",
    "        except SystemExit: ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.002s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The snippet below includes `doctest`s within `unittest`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "    from doctest import DocTestSuite\n",
    "    def load_tests(loader, tests, ignore):\n",
    "        tests.addTests(DocTestSuite(__import__(__name__)))\n",
    "        return tests\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.002s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*We can observe that more tests were collection because the `load_tests` object exists.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## pytest\n",
    "\n",
    "[`pytest`](http://doc.pytest.org/) is one of the most common frameworks in python used for testing.  It has an extensive plugin framework and ecosystem that allows non experts to create robust testing frameworks.  [`pytest`](http://doc.pytest.org/) generally requires less syntax that traditional doctests especially when combined with [`pytest.fixtures`](https://www.google.com/search?q=pytest+fixtures&rlz=1C1AVFC_enUS803US803&oq=pytest+fixtures&aqs=chrome..69i57j0l5.3743j0j7&sourceid=chrome&ie=UTF-8).\n",
    "\n",
    "`pytest` discovery is configurable, but by default tests beginning with `test_*` will be discovered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def test_thing(a: (range(10), 1)):\n",
    "        assert isinstance(a_function_with_a_doctest(a), str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### unittest, doctest, and pytest\n",
    "\n",
    "`pytest` automated collections unittests and has options for doctesting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running pytest in an IPython context\n",
    "\n",
    "A notebook author will often use IPython magics and other IPython specific API's.  These applications require the test be run with an active IPython context.  Most examples of `pytest` run tests using vanilla pytest by invoking either `pytest` or `python -m pytest` at the command line.  In these situations, the IPython context is unavailable.\n",
    "\n",
    "The code snippet below shares an application of running `pytest` through IPython using\n",
    "\n",
    "    ipython -m doctest --\n",
    "    \n",
    "where any argument following `--` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "    import pytest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As stated, `pytest` discovery is configurable; notebooks are not python files so we must modify the way tests are found using [`pytest_collect_file`](https://docs.pytest.org/en/latest/example/nonpython.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def pytest_collect_file(parent, path):\n",
    "        if path.ext == \".ipynb\": return pytest.Module(path, parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run `pytest` and collect notebook files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def _run_pytest_discovery():\n",
    "        pytest.main('--collect-only -p no:pytest-importnb 2018-07-31-Testing-notebooks.ipynb'.split(), [__import__(__name__)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Interactive testing with `pytest`\n",
    "\n",
    "> As far as we know, `pytest` can not run the current context in an IPython session.  A Pytest runner is required.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "* `doctest`s run tests in strings and __docstrings__ .\n",
    "* `unittest`s run tests on objects and may include `doctest`.\n",
    "* `pytest` runs tests on objects and may include `doctest` and `unittest` tests.\n",
    "\n",
    "In general, we end a lot of our essays with `doctest` to promote better documentation and reuse during an interactive session.  `pytest` is applied to our files or collections of files."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}