Testing Python Notebooks

Sun 01 December 2019

Filed under Testing

Testing Jupyter Notebooks¶

Introduction¶

I recently watched an excellent video on Python testing: Keynote - Preventing, Finding, and Fixing Bugs On a Time Budget | Raymond Hettinger @ PyBay2018: see it here.

As as result I was inspired to get pytest working with Jupyter Notebooks.

Implementation¶

The following are the imports and magics for this notebook.

It turns out that getting pytest working inside a notebook requires some support helper magics in a package called ipytest. Note that some versions of pytest are incompatible with the latest ipytest package. Be sure to get the latest pytest.

Imports¶

In [19]:

import pytest
import ipytest.magics


import warnings
import math
import random

import sys
import os
import subprocess
import datetime
import platform
import datetime

Magics¶

lab_black will format python cells in a standardized way.

In [20]:

%load_ext lab_black

The lab_black extension is already loaded. To reload it, use:
  %reload_ext lab_black

watermark documents the current environment.

In [21]:

%load_ext watermark

The watermark extension is already loaded. To reload it, use:
  %reload_ext watermark

Setup¶

pytest works (in part) by rewriting assert statements: we chose to suppress the warning messages about this.

In [22]:

print('pytest version = ', pytest.__version__)

# pytest rewrites Abstact Syntax Tree.  ignore warning about this
warnings.filterwarnings('ignore', category=UserWarning)

pytest version =  5.2.4

pytest magics needs to know the notebook file name.

In [23]:

# tell pytest our file name
__file__ = 'pytestnotebook.ipynb'

Using Pytest¶

First Steps¶

We define a function with an obvious error.

In [24]:

# trivial function with obvious error
def my_sum(a: float, b: float) -> float:
    return a


# end my_sum

We run pytest, cleaning all existing test results, and asking for verbose results.

pytest finds the test_my_sum function, executes it, and catches the assert failures.

In [25]:

%%run_pytest[clean] -v
def test_my_sum():
    assert 6==my_sum(6,0), 'Expected 6, got {}'.format(my_sum(6,0))
    assert 6==my_sum(2,4), 'Expected 6, got {}'.format(my_sum(2,4))

================================================= test session starts =================================================
platform win32 -- Python 3.7.1, pytest-5.2.4, py-1.7.0, pluggy-0.13.1 -- D:\Anaconda3\envs\ac5-py37\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\Users\\donrc\\Documents\\JupyterNotebooks\\PythonNotebookProject\\develop\\.hypothesis\\examples')
rootdir: C:\Users\donrc\Documents\JupyterNotebooks\PythonNotebookProject\develop
plugins: hypothesis-4.44.2, arraydiff-0.3, doctestplus-0.2.0, openfiles-0.3.1, remotedata-0.3.1
collecting ... collected 1 item

pytestnotebook.py::test_my_sum FAILED                                                                            [100%]

====================================================== FAILURES =======================================================
_____________________________________________________ test_my_sum _____________________________________________________

    def test_my_sum():
        assert 6==my_sum(6,0), 'Expected 6, got {}'.format(my_sum(6,0))
>       assert 6==my_sum(2,4), 'Expected 6, got {}'.format(my_sum(2,4))
E       AssertionError: Expected 6, got 2

<ipython-input-25-f05edba4f4f5>:3: AssertionError
================================================== 1 failed in 0.07s ==================================================

If we run the same test, but minimize output, we get:

In [26]:

%%run_pytest[clean] -qq
def test_my_sum():
    assert 6==my_sum(6,0)
    assert 6==my_sum(2,4)

F                                                                                                                [100%]
====================================================== FAILURES =======================================================
_____________________________________________________ test_my_sum _____________________________________________________

    def test_my_sum():
        assert 6==my_sum(6,0)
>       assert 6==my_sum(2,4)
E       AssertionError

<ipython-input-26-d901d1b3a70e>:3: AssertionError

More Realistic Testing¶

We define a quadratic equation solver (i.e.solves for axx+b*x+c = 0, given a, b, c). We test the input values and raise ValueError exceptions for invalid a, b, c . See here .

In [27]:

# more complicated function


def quadratic_solve(
    a: float, b: float, c: float
) -> (float, float):
    # set small value for testing input coefficients
    EPS = 1e-10

    # test if real roots possible
    if b * b < (4 * a * c):
        raise ValueError(
            'a={a}, b={b}, c={c}: b*b-4*a*c cannot be -ve'
        )
    # end if

    # test if power of x*x too small (ie have linear equation)
    if abs(a) > 1e-10:

        # choose formulas that minize round off errors
        if b > 0:
            x1 = (-b - math.sqrt(b * b - 4 * a * c)) / (
                2 * a
            )
            x2 = (2 * c) / (
                -b - math.sqrt(b * b - 4 * a * c)
            )
        else:  # b-nve
            x1 = (-b + math.sqrt(b * b - 4 * a * c)) / (
                2 * a
            )
            x2 = (2 * c) / (
                -b + math.sqrt(b * b - 4 * a * c)
            )
        # endif

    else:
        # solve linear equation, if possible
        if abs(b) > 1e-10:
            x1 = -c / b
            x2 = x1
        else:
            raise ValueError('a,b cannot both be zero')
        # end if
    # end if

    return x1, x2


# end quadratic_solve

Informally test solver in a case where round-off might cause problems.

In [28]:

print(quadratic_solve(1, 1e8, 1))

(-100000000.0, -1e-08)

Now test that the correct exceptions get thrown.

In [29]:

%%run_pytest[clean]
# test throws right exception if complex roots solve quadratic
def test_nve_discriminant():
    for n1 in range(1000):
        a = random.randint(2, 1_000_000)
        c = random.randint(2, 1_000_000)
        b_max = int(math.sqrt(4 * a * c)) - 1
        b = random.randint(-b_max, b_max + 1)
        b = b * random.choice([-1, 1])
        with pytest.raises(ValueError):
            x1, x2 = quadratic_solve(a, b, c)
        # end with
    # end for


# end test_nve_discriminant

# test throws right exception if a,b both 0
def test_ab_zero():
    for n1 in range(1000):
        a = 0
        b = 0
        c = random.randint(-1_000_000, 1_000_000)

        with pytest.raises(ValueError):
            x1, x2 = quadratic_solve(a, b, c)
        # end with
    # end for


# end test_ab_zero

================================================= test session starts =================================================
platform win32 -- Python 3.7.1, pytest-5.2.4, py-1.7.0, pluggy-0.13.1
rootdir: C:\Users\donrc\Documents\JupyterNotebooks\PythonNotebookProject\develop
plugins: hypothesis-4.44.2, arraydiff-0.3, doctestplus-0.2.0, openfiles-0.3.1, remotedata-0.3.1
collected 2 items

pytestnotebook.py ..                                                                                             [100%]

================================================== 2 passed in 0.07s ==================================================

Run test on a single test case.

In [30]:

%%run_pytest -v
# test quadratic actually solves equation
def test_quadratic_solve2():
    a = 1
    b = 2
    c = 1
    x1, x2 = quadratic_solve(a, b, c)
    assert x1 == -1 and x2 == -1


# end test_quadratic_solve2

================================================= test session starts =================================================
platform win32 -- Python 3.7.1, pytest-5.2.4, py-1.7.0, pluggy-0.13.1 -- D:\Anaconda3\envs\ac5-py37\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\Users\\donrc\\Documents\\JupyterNotebooks\\PythonNotebookProject\\develop\\.hypothesis\\examples')
rootdir: C:\Users\donrc\Documents\JupyterNotebooks\PythonNotebookProject\develop
plugins: hypothesis-4.44.2, arraydiff-0.3, doctestplus-0.2.0, openfiles-0.3.1, remotedata-0.3.1
collecting ... collected 3 items

pytestnotebook.py::test_nve_discriminant PASSED                                                                  [ 33%]
pytestnotebook.py::test_ab_zero PASSED                                                                           [ 66%]
pytestnotebook.py::test_quadratic_solve2 PASSED                                                                  [100%]

================================================== 3 passed in 0.11s ==================================================

Now run a test, chosing roots of the equation at random (with a normalized to 1).

In [31]:

%%run_pytest -v
def test_quadratic_solve3():

    for i1 in range(1000):
        n1 = random.randint(-1_000_000, 1_000_000)
        n2 = random.randint(-1_000_000, 1_000_000)

        a = 1
        c = n1 * n2
        b = n1 + n2
        if b * b > 4 * a * c:
            x1, x2 = quadratic_solve(a, b, c)
            assert (
                math.isclose(x1, -n1)
                and math.isclose(x2, -n2)
            ) or (
                math.isclose(x1, -n2)
                and math.isclose(x2, -n1)
            ), f'{n1}, {n2} -> {x1}, {x2}'

        # end if
    # end for


# end test_quadratic_solve3

================================================= test session starts =================================================
platform win32 -- Python 3.7.1, pytest-5.2.4, py-1.7.0, pluggy-0.13.1 -- D:\Anaconda3\envs\ac5-py37\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\Users\\donrc\\Documents\\JupyterNotebooks\\PythonNotebookProject\\develop\\.hypothesis\\examples')
rootdir: C:\Users\donrc\Documents\JupyterNotebooks\PythonNotebookProject\develop
plugins: hypothesis-4.44.2, arraydiff-0.3, doctestplus-0.2.0, openfiles-0.3.1, remotedata-0.3.1
collecting ... collected 4 items

pytestnotebook.py::test_nve_discriminant PASSED                                                                  [ 25%]
pytestnotebook.py::test_ab_zero PASSED                                                                           [ 50%]
pytestnotebook.py::test_quadratic_solve2 PASSED                                                                  [ 75%]
pytestnotebook.py::test_quadratic_solve3 PASSED                                                                  [100%]

================================================== 4 passed in 0.13s ==================================================

Run the test with no constraints on a.

In [32]:

%%run_pytest -v
def test_quadratic_solve4():
    for i1 in range(1000):
        n1 = random.randint(-1_000_000, 1_000_000)
        n2 = random.randint(-1_000_000, 1_000_000)
        n3 = random.randint(1, 1_000_000)

        a = n3 * 1
        c = n3 * n1 * n2
        b = n3 * (n1 + n2)
        if b * b > 4 * a * c:
            x1, x2 = quadratic_solve(a, b, c)
            assert (
                math.isclose(x1, -n1)
                and math.isclose(x2, -n2)
                or math.isclose(x1, -n2)
                and math.isclose(x2, -n1)
            ), f'{n1}, {n2} -> {x1}, {x2}'

        # end if
    # end for


# end test_quadratic_solve4

================================================= test session starts =================================================
platform win32 -- Python 3.7.1, pytest-5.2.4, py-1.7.0, pluggy-0.13.1 -- D:\Anaconda3\envs\ac5-py37\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\Users\\donrc\\Documents\\JupyterNotebooks\\PythonNotebookProject\\develop\\.hypothesis\\examples')
rootdir: C:\Users\donrc\Documents\JupyterNotebooks\PythonNotebookProject\develop
plugins: hypothesis-4.44.2, arraydiff-0.3, doctestplus-0.2.0, openfiles-0.3.1, remotedata-0.3.1
collecting ... collected 5 items

pytestnotebook.py::test_nve_discriminant PASSED                                                                  [ 20%]
pytestnotebook.py::test_ab_zero PASSED                                                                           [ 40%]
pytestnotebook.py::test_quadratic_solve2 PASSED                                                                  [ 60%]
pytestnotebook.py::test_quadratic_solve3 PASSED                                                                  [ 80%]
pytestnotebook.py::test_quadratic_solve4 PASSED                                                                  [100%]

================================================== 5 passed in 0.15s ==================================================

Test the case where a = 0 (i.e. we have a linear equation).

In [33]:

%%run_pytest -v
def test_quadratic_solve5():

    for i1 in range(1000):
        n1 = random.randint(-1_000_000, 1_000_000)
        n2 = random.randint(-1_000_000, 1_000_000)
        n3 = random.randint(1, 1_000_000)

        a = 0
        c = n2
        b = n1
        if b > 0:
            x1, x2 = quadratic_solve(a, b, c)
            assert math.isclose(
                x1, -float(n2) / float(n1)
            ), f'{n1}, {n2} -> {x1}, {x2}'
        # end if
    # end for


# end test_quadratic_solve5

================================================= test session starts =================================================
platform win32 -- Python 3.7.1, pytest-5.2.4, py-1.7.0, pluggy-0.13.1 -- D:\Anaconda3\envs\ac5-py37\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\Users\\donrc\\Documents\\JupyterNotebooks\\PythonNotebookProject\\develop\\.hypothesis\\examples')
rootdir: C:\Users\donrc\Documents\JupyterNotebooks\PythonNotebookProject\develop
plugins: hypothesis-4.44.2, arraydiff-0.3, doctestplus-0.2.0, openfiles-0.3.1, remotedata-0.3.1
collecting ... collected 6 items

pytestnotebook.py::test_nve_discriminant PASSED                                                                  [ 16%]
pytestnotebook.py::test_ab_zero PASSED                                                                           [ 33%]
pytestnotebook.py::test_quadratic_solve2 PASSED                                                                  [ 50%]
pytestnotebook.py::test_quadratic_solve3 PASSED                                                                  [ 66%]
pytestnotebook.py::test_quadratic_solve4 PASSED                                                                  [ 83%]
pytestnotebook.py::test_quadratic_solve5 PASSED                                                                  [100%]

================================================== 6 passed in 0.15s ==================================================

Reproducibility Details¶

In [34]:

%watermark --iversions

platform 1.0.8
pytest   5.2.4

In [35]:

%watermark

2019-12-02T14:27:40+10:00

CPython 3.7.1
IPython 7.2.0

compiler   : MSC v.1915 64 bit (AMD64)
system     : Windows
release    : 10
machine    : AMD64
processor  : Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
CPU cores  : 8
interpreter: 64bit

In [36]:

# show info to support reproducibility

theNotebook = __file__


def python_env_name():
    envs = subprocess.check_output(
        'conda env list'
    ).splitlines()
    # get unicode version of binary subprocess output
    envu = [x.decode('ascii') for x in envs]
    active_env = list(
        filter(lambda s: '*' in str(s), envu)
    )[0]
    env_name = str(active_env).split()[0]
    return env_name


# end python_env_name

print('python version : ' + sys.version)
print('python environment :', python_env_name())

print('current wkg dir: ' + os.getcwd())
print('Notebook name: ' + theNotebook)
print(
    'Notebook run at: '
    + str(datetime.datetime.now())
    + ' local time'
)
print(
    'Notebook run at: '
    + str(datetime.datetime.utcnow())
    + ' UTC'
)
print('Notebook run on: ' + platform.platform())

python version : 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
python environment : ac5-py37
current wkg dir: C:\Users\donrc\Documents\JupyterNotebooks\PythonNotebookProject\develop
Notebook name: pytestnotebook.ipynb
Notebook run at: 2019-12-02 14:27:44.703459 local time
Notebook run at: 2019-12-02 04:27:44.703459 UTC
Notebook run on: Windows-10-10.0.18362-SP0