{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# The complete notebook starter package.\n", "\n", "Modern notebooks are treated as disposable. The long term value of a notebook is lacking because of programming style, not the technology.\n", "\n", "Many scientists and journalists frequently open blank notebooks to informally test ideas. This document discussing the value introducing formal testing into the notebook development process sooner. In this approach, early ideas last longer.\n", "\n", "Individual contributors must enjoy the same software engineering benefits as members of organizations. To compete with larger research machines the individual needs:\n", "* Testing\n", "* Documentation\n", "* Software distribution\n", "\n", "A successful notebook project will mature notebook source to Python source, but it will try to retain \n", "as many notebooks as possible for testing.\n", "\n", "The rest of this document discusses tips for successfully maturing notebook projects." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ " %reload_ext pidgin" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "

Notebooks have the potential to create tests and documentation from the onset of a project.

\n", "
import nbval, importnb, nbsphinx, nbformat, nbconvert, pytest, pathlib, os\n",
       "
" ], "text/markdown": [ "* `nbval` will ensure that all of the code cells are executed.\n", "* `importnb` will discover named tests with `pytest`.\n", "* __[.travis.yml]({{name}}/{{name}}.travis.yml)__ will evaluate the tests in continuous integration.\n", "* `nbsphinx` is configured in __conf.py__ so that notebooks are our documentation and may be deployed on \n", "[readthedocs](https://readthedocs.org/).\n", "* __[setup.py]({{name}}/setup.py)__ is configured to make all source within the package usable.\n", "\n", " At the beginning of a project, there is no difference between source, tests, and documentation.\n", "\n", "Notebooks have the potential to create tests and documentation from the onset of a project.\n", "\n", " import nbval, importnb, nbsphinx, nbformat, nbconvert, pytest, pathlib, os" ], "text/plain": [ "'* `nbval` will ensure that all of the code cells are executed.\\n* `importnb` will discover named tests with `pytest`.\\n* __[.travis.yml]({{name}}/{{name}}.travis.yml)__ will evaluate the tests in continuous integration.\\n* `nbsphinx` is configured in __conf.py__ so that notebooks are our documentation and may be deployed on \\n[readthedocs](https://readthedocs.org/).\\n* __[setup.py]({{name}}/setup.py)__ is configured to make all source within the package usable.\\n\\n At the beginning of a project, there is no difference between source, tests, and documentation.\\n\\nNotebooks have the potential to create tests and documentation from the onset of a project.\\n\\n import nbval, importnb, nbsphinx, nbformat, nbconvert, pytest, pathlib, os'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "* `nbval` will ensure that all of the code cells are executed.\n", "* `importnb` will discover named tests with `pytest`.\n", "* __[.travis.yml]({{name}}/{{name}}.travis.yml)__ will evaluate the tests in continuous integration.\n", "* `nbsphinx` is configured in __conf.py__ so that notebooks are our documentation and may be deployed on \n", "[readthedocs](https://readthedocs.org/).\n", "* __[setup.py]({{name}}/setup.py)__ is configured to make all source within the package usable.\n", "\n", " At the beginning of a project, there is no difference between source, tests, and documentation.\n", "\n", "Notebooks have the potential to create tests and documentation from the onset of a project.\n", "\n", " import nbval, importnb, nbsphinx, nbformat, nbconvert, pytest, pathlib, os" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

Consider a sample project with the name boiler. This name is chosen because we are writing\n", "boiler plate code.

\n", "
name = 'boiler'\n",
       "
" ], "text/markdown": [ "Consider a sample project with the `name` __{{name}}__. _This name is chosen because we are writing\n", "__{{name}}__ plate code._\n", "\n", " name = 'boiler'" ], "text/plain": [ "\"Consider a sample project with the `name` __{{name}}__. _This name is chosen because we are writing\\n__{{name}}__ plate code._\\n\\n name = 'boiler'\"" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "Consider a sample project with the `name` __{{name}}__. _This name is chosen because we are writing\n", "__{{name}}__ plate code._\n", "\n", " name = 'boiler'" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['boiler/src/boiler', 'boiler/src/boiler/docs']" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "

The boiler project requires the packages are available in the \"src\" directory. Reason for the \"src\" directory\n", "layout, redirected from Good pytest integration practices.

\n", "
packages = F\"\"\"{name}/src/{name}\n",
       "{name}/src/{name}/docs\"\"\".splitlines()\n",
       "packages\n",
       "
" ], "text/markdown": [ "The __{{name}}__ project requires the `packages` are available in the `\"src\"` directory. Reason for the `\"src\"` [directory\n", "layout](https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure), redirected from [Good `pytest` integration practices](https://docs.pytest.org/en/latest/goodpractices.html#tests-outside-application-code).\n", "\n", " packages = F\"\"\"{name}/src/{name}\n", " {name}/src/{name}/docs\"\"\".splitlines()\n", " packages" ], "text/plain": [ "'The __{{name}}__ project requires the `packages` are available in the `\"src\"` directory. Reason for the `\"src\"` [directory\\nlayout](https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure), redirected from [Good `pytest` integration practices](https://docs.pytest.org/en/latest/goodpractices.html#tests-outside-application-code).\\n\\n packages = F\"\"\"{name}/src/{name}\\n {name}/src/{name}/docs\"\"\".splitlines()\\n packages'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "The __{{name}}__ project requires the `packages` are available in the `\"src\"` directory. Reason for the `\"src\"` [directory\n", "layout](https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure), redirected from [Good `pytest` integration practices](https://docs.pytest.org/en/latest/goodpractices.html#tests-outside-application-code).\n", "\n", " packages = F\"\"\"{name}/src/{name}\n", " {name}/src/{name}/docs\"\"\".splitlines()\n", " packages" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

For each of the packages we should make the corresponding directories.

\n", "
for package in packages: pathlib.Path(package).mkdir(exist_ok=True, parents=True)\n",
       "
" ], "text/markdown": [ "For each of the `packages` we should make the corresponding directories.\n", "\n", " for package in packages: pathlib.Path(package).mkdir(exist_ok=True, parents=True)" ], "text/plain": [ "'For each of the `packages` we should make the corresponding directories.\\n\\n for package in packages: pathlib.Path(package).mkdir(exist_ok=True, parents=True)'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "For each of the `packages` we should make the corresponding directories.\n", "\n", " for package in packages: pathlib.Path(package).mkdir(exist_ok=True, parents=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For computational narratives, __readme.ipynb__ is for people as __init__.py is for Python." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

Our init.py files will import the readme into its namespace.

\n", "
_init_file = \"\"\"with __import__('importnb').Notebook():\n",
       "    from . import readme\n",
       "    from .readme import *\n",
       "\"\"\";\n",
       "
\n", "

Any code in the default readme.ipynb files will be available within the module.

" ], "text/markdown": [ "Our __init__.py files will __import__ the readme into its namespace.\n", "\n", " _init_file = \"\"\"with __import__('importnb').Notebook():\n", " from . import readme\n", " from .readme import *\n", " \"\"\";\n", "\n", "Any code in the default __readme.ipynb__ files will be available within the module." ], "text/plain": [ "'Our __init__.py files will __import__ the readme into its namespace.\\n\\n _init_file = \"\"\"with __import__(\\'importnb\\').Notebook():\\n from . import readme\\n from .readme import *\\n \"\"\";\\n\\nAny code in the default __readme.ipynb__ files will be available within the module.'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "Our __init__.py files will __import__ the readme into its namespace.\n", "\n", " _init_file = \"\"\"with __import__('importnb').Notebook():\n", " from . import readme\n", " from .readme import *\n", " \"\"\";\n", "\n", "Any code in the default __readme.ipynb__ files will be available within the module." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

Configuring the docs

\n", "
for package in packages: \n",
       "    with pathlib.Path(package, 'readme.ipynb').open('w') as file:\n",
       "        nbformat.write(\n",
       "            nbformat.v4.new_notebook(cells=[nbformat.v4.new_markdown_cell(\n",
       "                \"\"\"[readme](readme.ipynb)\"\"\", metadata={\"nbsphinx-toctree\": {}})]), file)\n",
       "
" ], "text/markdown": [ "### Configuring the docs\n", "\n", " for package in packages: \n", " with pathlib.Path(package, 'readme.ipynb').open('w') as file:\n", " nbformat.write(\n", " nbformat.v4.new_notebook(cells=[nbformat.v4.new_markdown_cell(\n", " \"\"\"[readme](readme.ipynb)\"\"\", metadata={\"nbsphinx-toctree\": {}})]), file)" ], "text/plain": [ "'### Configuring the docs\\n\\n for package in packages: \\n with pathlib.Path(package, \\'readme.ipynb\\').open(\\'w\\') as file:\\n nbformat.write(\\n nbformat.v4.new_notebook(cells=[nbformat.v4.new_markdown_cell(\\n \"\"\"[readme](readme.ipynb)\"\"\", metadata={\"nbsphinx-toctree\": {}})]), file)'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "### Configuring the docs\n", "\n", " for package in packages: \n", " with pathlib.Path(package, 'readme.ipynb').open('w') as file:\n", " nbformat.write(\n", " nbformat.v4.new_notebook(cells=[nbformat.v4.new_markdown_cell(\n", " \"\"\"[readme](readme.ipynb)\"\"\", metadata={\"nbsphinx-toctree\": {}})]), file)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

Make the corresponding \"__init__.py\" files.

\n", "
for package in packages:  pathlib.Path(package, '__init__.py').write_text(_init_file)\n",
       "
" ], "text/markdown": [ "Make the corresponding `\"__init__.py\"` files.\n", " \n", " for package in packages: pathlib.Path(package, '__init__.py').write_text(_init_file)" ], "text/plain": [ "'Make the corresponding `\"__init__.py\"` files.\\n \\n for package in packages: pathlib.Path(package, \\'__init__.py\\').write_text(_init_file)'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "Make the corresponding `\"__init__.py\"` files.\n", " \n", " for package in packages: pathlib.Path(package, '__init__.py').write_text(_init_file)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

The top_level_readme['metadata']['nbsphinx-toctree'] object creates the index for our documentation.

\n", "

At the beginning of a project there is no different between the project source, tests, and documentation.

\n", "
\n", "
with pathlib.Path(name, 'readme.ipynb').open('w') as file:\n",
       "    nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)\n",
       "
" ], "text/markdown": [ "The `top_level_readme['metadata']['nbsphinx-toctree']` object creates the index for our documentation. \n", "> At the beginning of a project there is no different between the project source, tests, and documentation.\n", "\n", " \n", " with pathlib.Path(name, 'readme.ipynb').open('w') as file:\n", " nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)" ], "text/plain": [ "\"The `top_level_readme['metadata']['nbsphinx-toctree']` object creates the index for our documentation. \\n> At the beginning of a project there is no different between the project source, tests, and documentation.\\n\\n \\n with pathlib.Path(name, 'readme.ipynb').open('w') as file:\\n nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)\"" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "

Creating the top level readme and docs

\n", "
import nbformat, nbsphinx, pathlib\n",
       "
\n", "

nbsphinx provides the interface between a notebook based project and readthedocs.

\n", "
top_level_readme = nbformat.v4.new_markdown_cell(F\"\"\"\n",
       "* [Source](src/{name}/readme.ipynb)\n",
       "* [Docs](src/{name}/docs/readme.ipynb)\n",
       "\"\"\", metadata={\"nbsphinx-toctree\": {}})\n",
       "
\n", "

The top_level_readme['metadata']['nbsphinx-toctree'] object creates the index for our documentation.

\n", "

At the beginning of a project there is no different between the project source, tests, and documentation.

\n", "
\n", "
with pathlib.Path(name, 'readme.ipynb').open('w') as file:\n",
       "    nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)\n",
       "
" ], "text/markdown": [ "## Creating the top level readme and docs\n", " \n", " import nbformat, nbsphinx, pathlib\n", "\n", "`nbsphinx` provides the interface between a notebook based project and __readthedocs__.\n", "\n", " top_level_readme = nbformat.v4.new_markdown_cell(F\"\"\"\n", " * [Source](src/{name}/readme.ipynb)\n", " * [Docs](src/{name}/docs/readme.ipynb)\n", " \"\"\", metadata={\"nbsphinx-toctree\": {}})\n", " \n", "The `top_level_readme['metadata']['nbsphinx-toctree']` object creates the index for our documentation. \n", "> At the beginning of a project there is no different between the project source, tests, and documentation.\n", "\n", " \n", " with pathlib.Path(name, 'readme.ipynb').open('w') as file:\n", " nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)" ], "text/plain": [ "'## Creating the top level readme and docs\\n \\n import nbformat, nbsphinx, pathlib\\n\\n`nbsphinx` provides the interface between a notebook based project and __readthedocs__.\\n\\n top_level_readme = nbformat.v4.new_markdown_cell(F\"\"\"\\n * [Source](src/{name}/readme.ipynb)\\n * [Docs](src/{name}/docs/readme.ipynb)\\n \"\"\", metadata={\"nbsphinx-toctree\": {}})\\n \\nThe `top_level_readme[\\'metadata\\'][\\'nbsphinx-toctree\\']` object creates the index for our documentation. \\n> At the beginning of a project there is no different between the project source, tests, and documentation.\\n\\n \\n with pathlib.Path(name, \\'readme.ipynb\\').open(\\'w\\') as file:\\n nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "## Creating the top level readme and docs\n", " \n", " import nbformat, nbsphinx, pathlib\n", "\n", "`nbsphinx` provides the interface between a notebook based project and __readthedocs__.\n", "\n", " top_level_readme = nbformat.v4.new_markdown_cell(F\"\"\"\n", " * [Source](src/{name}/readme.ipynb)\n", " * [Docs](src/{name}/docs/readme.ipynb)\n", " \"\"\", metadata={\"nbsphinx-toctree\": {}})\n", " \n", "The `top_level_readme['metadata']['nbsphinx-toctree']` object creates the index for our documentation. \n", "> At the beginning of a project there is no different between the project source, tests, and documentation.\n", "\n", " \n", " with pathlib.Path(name, 'readme.ipynb').open('w') as file:\n", " nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "ename": "OSError", "evalue": "symbolic link privilege not held", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mOSError\u001b[0m Traceback (most recent call last)", "\u001b[1;32m\u001b[0m in \u001b[0;36m\u001b[1;34m\u001b[0m\n\u001b[0;32m 1\u001b[0m \u001b[1;34m\"\"\"Make a symbollic link between readme and index so the docs build.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mos\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msymlink\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpathlib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mPath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'readme.ipynb'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpathlib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mPath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'index.ipynb'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[1;31mOSError\u001b[0m: symbolic link privilege not held" ] } ], "source": [ "Make a symbollic link between readme and index so the docs build.\n", " \n", " os.symlink(pathlib.Path(name, 'readme.ipynb'), pathlib.Path(name, 'index.ipynb'))" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting boiler/.travis.yml\n" ] } ], "source": [ " %%file {name}/.travis.yml\n", " language: python\n", " python: ['3.6']\n", " install: [\"python -m pip install .\"]\n", " script: [\"python setup.py test\"]" ] }, { "cell_type": "code", "execution_count": 213, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
versions = ['3.6', '3.7']\n",
       "pathlib.Path(name, '.travis.yml').write_text(\n",
       "
\n", "

language: python\n", "python: {versions}\n", "install: [\"python -m pip install .\"]\n", "script: [\"python setup.py test\"]

\n", "
.format(versions=versions));\n",
       "
" ], "text/markdown": [ " versions = ['3.6', '3.7']\n", " pathlib.Path(name, '.travis.yml').write_text(\n", "language: python\n", "python: {versions}\n", "install: [\"python -m pip install .\"]\n", "script: [\"python setup.py test\"]\n", " \n", " .format(versions=versions));" ], "text/plain": [ "' versions = [\\'3.6\\', \\'3.7\\']\\n pathlib.Path(name, \\'.travis.yml\\').write_text(\\nlanguage: python\\npython: {versions}\\ninstall: [\"python -m pip install .\"]\\nscript: [\"python setup.py test\"]\\n \\n .format(versions=versions));'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ " versions = '3.6 3.7'\n", " pathlib.Path(name, '.travis.yml').write_text(\n", "language: python\n", "python: {versions.split()}\n", "install: [\"python -m pip install .\"]\n", "script: [\"python setup.py test\"]\n", " \n", " .format(versions=versions));" ] }, { "cell_type": "code", "execution_count": 129, "metadata": {}, "outputs": [], "source": [ " %%file {name}/requirements.txt\n", " importnb" ] }, { "cell_type": "code", "execution_count": 131, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting boiler/src/boiler/tests/requirements.txt\n" ] } ], "source": [ " %%file {name}/src/{name}/tests/requirements.txt\n", " nbval" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "_version Hourly and minute versioning" ] }, { "cell_type": "code", "execution_count": 133, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting boiler/src/boiler/_version.py\n" ] } ], "source": [ " %%file {name}/src/{name}/_version.py\n", " time = __import__('datetime').datetime.now()\n", "\n", " __version__ = '.'.join(\n", " str(getattr(time, object))\n", " for object in \"\"\"\n", " year month day hour minute\n", " \"\"\".strip().split())" ] }, { "cell_type": "code", "execution_count": 136, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting boiler/setup.py\n" ] } ], "source": [ " %%file {name}/setup.py\n", " import setuptools, pathlib, nbconvert\n", " name = 'boiler'\n", " here = pathlib.Path(__file__).parent\n", " with (here / 'src' / name / '_version.py').open('r') as file: exec(file.read())\n", " setuptools.setup(\n", " name=name, version=__version__,\n", " long_description=nbconvert.get_exporter('markdown')().from_filename(here / 'readme.ipynb')[0],\n", " long_description_content_type='text/markdown',\n", " packages=setuptools.find_packages(where='src'),\n", " package_dir={'': 'src',},\n", " setup_requires=[\"pytest-runner\", \"nbconvert\"],\n", " install_requires=(here /'requirements.txt').read_text().splitlines(),\n", " tests_require=(here /'src'/ name / 'tests' / 'requirements.txt').read_text().splitlines(),\n", " include_package_data=True)" ] }, { "cell_type": "code", "execution_count": 137, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting boiler/MANIFEST.in\n" ] } ], "source": [ " %%file {name}/MANIFEST.in\n", " include LICENSE readme.md changelog.ipynb\n", " recursive-include src/boiler *.ipynb *.md" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "https://docs.pytest.org/en/latest/goodpractices.html#integrating-with-setuptools-python-setup-py-test-pytest-runner\n", " \n", " python setup.py test" ] }, { "cell_type": "code", "execution_count": 138, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting boiler/setup.cfg\n" ] } ], "source": [ " %%file {name}/setup.cfg\n", " [tool:pytest]\n", " addopts = --nbval -p no:pytest-pidgin\n", "\n", " [aliases]\n", " test=pytest" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " python -m pytest --nbval -p no:pytest-pidgin" ] }, { "cell_type": "code", "execution_count": 140, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting boiler/_config.yaml\n" ] } ], "source": [ " %%file {name}/_config.yaml\n", " name: {name}" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.6" } }, "nbformat": 4, "nbformat_minor": 2 }