{ "cells": [ { "cell_type": "markdown", "id": "08f83295", "metadata": {}, "source": [ "# Notebook Best Practices\n", "\n", "> How to write great nbdev notebooks\n", "\n", "- order: 2" ] }, { "cell_type": "code", "execution_count": null, "id": "a5048e9c", "metadata": {}, "outputs": [], "source": [ "#| hide\n", "from __future__ import annotations\n", "import numpy as np\n", "from fastcore.test import *\n", "from nbdev.showdoc import *\n", "from nbdev.qmd import *" ] }, { "cell_type": "markdown", "id": "10471f7d-9f32-48c1-807f-936d19c1a87e", "metadata": {}, "source": [ "The flexibility offered by notebooks can be overwhelming. While there are industry standards for writing Python packages---like [numpy](https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard) and [sphinx](https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html) docstrings, and [pytest](https://docs.pytest.org/) and [unittest](https://docs.python.org/3/library/unittest.html) testing frameworks---they weren't designed for notebooks.\n", "\n", "This article walks you through the practices we've learned to leverage the full power of notebooks with nbdev[^contributing]. Our approach weaves code, tests, and docs into a single interactive context that invites experimentation. If you prefer to learn by example, you might want to start with [the annotated example](#putting-it-all-together-an-annotated-example) and branch out from there.\n", "\n", "[^contributing]: We're always open to improving our workflows and don't like to be too prescriptive about style. If you have any ideas, please feel free to post them in the [forum](https://forums.fast.ai/c/nbdev/48)." ] }, { "cell_type": "markdown", "id": "6f74b0a2-7587-4259-8cd0-fba35b15e68d", "metadata": {}, "source": [ "![Marie Curie's research notebook dated 19-21 January 1900 ([source](https://commons.wikimedia.org/wiki/File:Marie_Curie;_Holograph_Notebook._Wellcome_L0021265.jpg)).](images/marie-curie-notebook.jpg){.rounded .preview-image alt=\"Photo of an opened research notebook with diagrams and writing in French\"}" ] }, { "cell_type": "markdown", "id": "a48bdd2b", "metadata": {}, "source": [ "## Know which form of notebook you're writing" ] }, { "cell_type": "markdown", "id": "f64403de", "metadata": {}, "source": [ "First of all, decide which form of notebook you're writing. We're fans of the [Diátaxis system](https://diataxis.fr/) which classifies documentation into four forms: tutorials, how-to guides, explanations, and references. They've laid this out beautifully in the following diagram:" ] }, { "cell_type": "markdown", "id": "e6f017dc", "metadata": {}, "source": [ "![](https://documentation.divio.com/_images/overview.png){.rounded alt=\"A 2x2 matrix, from top-left to bottom-right, 'Tutorials (learning-oriented)', 'How-to guides (problem-oriented)', 'Explanation (understanding-oriented), and 'Reference (information-oriented)'. Horizontal axis reads 'Most useful when we're studying' on the left, and 'Most useful when we're working' on the right. Vertical axis reads 'Practical steps' on top, and 'Theoretical knowledge' below.\"}" ] }, { "cell_type": "markdown", "id": "360fa1dc", "metadata": {}, "source": [ "## Start with a great title and subtitle" ] }, { "cell_type": "markdown", "id": "077102ab", "metadata": {}, "source": [ "Start with a markdown cell at the top of your notebook with its title in an H1 header, and subtitle in a blockquote. For example:" ] }, { "cell_type": "markdown", "id": "2bf12c10", "metadata": {}, "source": [ "```markdown\n", "# Great title\n", "\n", "> And an even better subtitle\n", "```" ] }, { "cell_type": "markdown", "id": "dedb473d", "metadata": {}, "source": [ "The title will also be used to reference your page in the sidebar. You can also optionally add [frontmatter](/api/09_frontmatter.ipynb) to this cell to customize nbdev and Quarto." ] }, { "cell_type": "markdown", "id": "d22e2490", "metadata": {}, "source": [ "## Introduce your notebook" ] }, { "cell_type": "markdown", "id": "183b7e3e-d5a1-444b-90ae-540f2aa85d52", "metadata": {}, "source": [ "Introduce your notebook with markdown cells below the title. We recommend a slightly different approach depending on the [form of documentation](#know-which-form-of-notebook-youre-writing):\n", "\n", "- **Reference:** Start with a brief description of the technical component, and an overview that links to the main symbols in the page (you might want to [use doclinks](#reference-related-symbols-with-doclinks))\n", "- **Tutorials and how-to guides:** Describe what the reader will learn and how. Keep it short and get to the subject matter quickly\n", "- **Explanations:** Since these are typically very focused, a short description of the topic is often sufficient." ] }, { "cell_type": "markdown", "id": "3ada0a7c-ad5c-4dff-82d4-a1aaad8f3d29", "metadata": {}, "source": [ "::: {.callout-tip appearance=\"simple\"}\n", "Note that Markdown lists such as the one above require a blank line above them to be rendered as lists in the documentation, even though the notebook viewer will render lists that are not preceded by a blank line.\n", ":::" ] }, { "cell_type": "markdown", "id": "0c76aa13", "metadata": {}, "source": [ "## Use lots of code examples, pictures, plots, and videos" ] }, { "cell_type": "markdown", "id": "a00bcdff", "metadata": {}, "source": [ "Take advantage of the richness of notebooks by including code examples, pictures, plots, and videos. " ] }, { "cell_type": "markdown", "id": "b486143a", "metadata": {}, "source": [ "Here are a few examples to get you started:\n", "\n", "- fastai's documentation makes extensive use of code examples, plots, images, and tables, for example, the [computer vision intro](https://docs.fast.ai/tutorial.vision.html)\n", "- [`nbdev.release`](/api/18_release.ipynb) opens with a terminal screencast demo in SVG format created with [asciinema](https://asciinema.org/) and [svg-term-cli](https://github.com/marionebl/svg-term-cli)\n", "- The [documentation explanation](/explanations/docs.ipynb#overview) describes a complex data pipeline using a [Mermaid diagram](https://quarto.org/docs/authoring/diagrams.html)\n", "- The [directives explanation](/explanations/directives.ipynb) showcases all of nbdev's directives with executable examples in call-out cards (and makes great use of emojis too!)\n", "- [RDKit](https://www.rdkit.org/docs/Cookbook.html#drawing-molecules-jupyter) renders beautiful molecule diagrams" ] }, { "cell_type": "markdown", "id": "71c85ff3", "metadata": {}, "source": [ "## Keep docstrings short; elaborate in separate cells" ] }, { "cell_type": "markdown", "id": "3b0ee64c", "metadata": {}, "source": [ "While nbdev renders docstrings as markdown, they aren't rendered correctly when using `symbol?` or `help(symbol)` and they can't include executed code. By splitting longer docstrings across separate code and markdown cells you can [use code examples, pictures, plots, and videos](#use-lots-of-code-examples-pictures-plots-and-videos)." ] }, { "cell_type": "markdown", "id": "71f45249", "metadata": {}, "source": [ "We find a single-line summary sufficient for most docstrings." ] }, { "cell_type": "markdown", "id": "204169fe", "metadata": {}, "source": [ "## Document parameters with docments" ] }, { "cell_type": "markdown", "id": "5cd81e29", "metadata": {}, "source": [ "[`fastcore.docments`](https://fastcore.fast.ai/docments.html) is a concise way to document parameters that is beautifully rendered by nbdev. For example, this function:" ] }, { "cell_type": "code", "execution_count": null, "id": "08b3fc52", "metadata": {}, "outputs": [], "source": [ "def draw_n(n:int, # Number of cards to draw\n", " replace:bool=True # Draw with replacement?\n", " )->list: # List of cards\n", " \"Draw `n` cards.\"" ] }, { "cell_type": "markdown", "id": "4da5e594", "metadata": {}, "source": [ "...would include the following table as part of its documentation:" ] }, { "cell_type": "code", "execution_count": null, "id": "48dd5364", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "::: {.py-2 .px-3 .mb-4 .border .rounded .shadow-sm}\n", "\n", "| | **Type** | **Default** | **Details** |\n", "| -- | -------- | ----------- | ----------- |\n", "| n | int | | Number of cards to draw |\n", "| replace | bool | True | Draw with replacement? |\n", "| **Returns** | **list** | | **List of cards** |\n", "\n", ":::\n", "\n", "\n" ] } ], "source": [ "#| echo: false\n", "#| output: asis\n", "print(div(DocmentTbl(draw_n)._repr_markdown_(),\n", " classes='py-2 px-3 mb-4 border rounded shadow-sm'.split()))" ] }, { "cell_type": "markdown", "id": "7eb859b3", "metadata": {}, "source": [ "nbdev also supports some numpy docstring sections. For example, this code snippet would produce the same table (there's no need to include types like in the docstring if you already have annotations):" ] }, { "cell_type": "code", "execution_count": null, "id": "3d89283a", "metadata": {}, "outputs": [], "source": [ "def draw_n(n:int, replace:bool=True) -> Cards:\n", " \"\"\"\n", " Draw `n` cards.\n", " \n", " Parameters\n", " ----------\n", " n\n", " Number of cards to draw\n", " replace\n", " Draw with replacement?\n", " \n", " Returns\n", " -------\n", " cards\n", " List of cards\n", " \"\"\"" ] }, { "cell_type": "markdown", "id": "52616927", "metadata": {}, "source": [ "::: {.callout-tip appearance=\"simple\"}\n", "\n", "You can render a symbol's parameters table directly with `DocmentTbl`. In fact, that's how we rendered the table above.\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "f967dffc", "metadata": {}, "source": [ "## Consider turning code examples into tests by adding assertions" ] }, { "cell_type": "markdown", "id": "1ceddcab", "metadata": {}, "source": [ "nbdev blurs the lines between code, docs, and tests. _Every_ code cell is run as a test (unless it's explicitly marked otherwise), and any error in the cell fails the test." ] }, { "cell_type": "markdown", "id": "cc85415b", "metadata": {}, "source": [ "Consider turning your code examples into tests by adding assertions -- if they would make valuable tests and if it doesn't hurt readability. [`fastcore.test`](https://fastcore.fast.ai/test.html) provides a set of light wrappers around `assert` for better notebook tests (for example, they print both objects on error if they differ). " ] }, { "cell_type": "markdown", "id": "09c2fc5b", "metadata": {}, "source": [ "Here's an example using `fastcore.test.test_eq`:" ] }, { "cell_type": "code", "execution_count": null, "id": "f42cf559", "metadata": {}, "outputs": [], "source": [ "def inc(x): return x + 1\n", "test_eq(inc(3), 4)" ] }, { "cell_type": "markdown", "id": "39da5019", "metadata": {}, "source": [ "## Document error-cases as tests" ] }, { "cell_type": "markdown", "id": "1797812c", "metadata": {}, "source": [ "Docstring-driven approaches typically document the errors raised by an object using plaintext descriptions, for example, in a \"raises\" section." ] }, { "cell_type": "markdown", "id": "3f30a861", "metadata": {}, "source": [ "In nbdev, we recommend documenting errors with actual failing code using `fastcore.test.test_fail`. For example:" ] }, { "cell_type": "code", "execution_count": null, "id": "a2cd0a74", "metadata": {}, "outputs": [], "source": [ "def divide(x, y): return x / y\n", "test_fail(lambda: divide(1, 0), contains=\"division by zero\")" ] }, { "cell_type": "markdown", "id": "fd5460ff", "metadata": {}, "source": [ "The first argument is a `lambda` since we need to allow `test_fail` to control its execution and catch any errors." ] }, { "cell_type": "markdown", "id": "984750ab", "metadata": {}, "source": [ "## Reference related symbols with doclinks" ] }, { "cell_type": "markdown", "id": "12ce67f8", "metadata": {}, "source": [ "If you surround a symbol with backticks, nbdev will automatically link to that symbol's reference page. We call these [doclinks](/api/05_doclinks.ipynb)." ] }, { "cell_type": "markdown", "id": "e7c686c2", "metadata": {}, "source": [ "Prefer fully qualified symbol paths, like `package.module.symbol` instead of `symbol`. It may be more verbose but it helps users know which module a symbol originates from, which is especially important for third-party packages." ] }, { "cell_type": "markdown", "id": "9e542ed1", "metadata": {}, "source": [ "Any package created with nbdev will automatically support doclinks. Non-nbdev packages can be supported by creating a minimal nbdev-index package. [`nbdev-index`](https://github.com/fastai/nbdev-index) is a collection of such packages, which already supports django, numpy, pandas, pytorch, scipy, sphinx, the Python standard library, and even other programming languages like APL!" ] }, { "cell_type": "markdown", "id": "f7cdb4ee", "metadata": {}, "source": [ "## Add rich representations to your classes" ] }, { "cell_type": "markdown", "id": "8a27b373", "metadata": {}, "source": [ "This is another way to take advantage of the [rich display feature of notebooks](https://ipython.readthedocs.io/en/stable/config/integrating.html#rich-display). You can provide rich representations to your object by defining a `_repr_markdown_` method that returns markdown text (which may also include HTML/CSS)." ] }, { "cell_type": "markdown", "id": "5a1222a5", "metadata": {}, "source": [ "Here's a simple example to get you started:" ] }, { "cell_type": "code", "execution_count": null, "id": "2428f2a5", "metadata": {}, "outputs": [], "source": [ "class Color:\n", " def __init__(self, color): self.color = color\n", " def _repr_markdown_(self):\n", " style = f'background-color: {self.color}; width: 50px; height: 50px; margin: 10px'\n", " return f'
'" ] }, { "cell_type": "code", "execution_count": null, "id": "a1bd9c61", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "
" ], "text/plain": [ "<__main__.Color at 0x10e9cd940>" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Color('green')" ] }, { "cell_type": "code", "execution_count": null, "id": "1e7332e1", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "
" ], "text/plain": [ "<__main__.Color at 0x10e9c9d90>" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Color('blue')" ] }, { "cell_type": "markdown", "id": "20f79bf8", "metadata": {}, "source": [ "Also see [the earlier list of example projects](#use-lots-of-code-examples-pictures-plots-and-videos) that make use of beautiful visual representations." ] }, { "cell_type": "markdown", "id": "92bdf0e6", "metadata": {}, "source": [ "## Document class methods with `show_doc` or `fastcore.basics.patch`" ] }, { "cell_type": "markdown", "id": "c13da395", "metadata": {}, "source": [ "nbdev automatically documents exported function and class definitions with `show_doc`. However, it's up to you to document class methods. There are two ways to do that: calling `show_doc` on the method, or defining the method with the `fastcore.basics.patch` decorator." ] }, { "cell_type": "markdown", "id": "d99db795", "metadata": {}, "source": [ "::: {.panel-tabset}" ] }, { "cell_type": "markdown", "id": "8739d401", "metadata": {}, "source": [ "### Notebook (show_doc)" ] }, { "cell_type": "markdown", "id": "0ed2eb50", "metadata": {}, "source": [ "If your class is defined in a single cell, use `show_doc`. Here's what your notebook might look like:" ] }, { "cell_type": "markdown", "id": "6f084ffa", "metadata": {}, "source": [ "::: {.pt-4 .pb-1 .px-3 .mb-2 .border .rounded .shadow-sm}\n", "\n", "```python\n", "#| export\n", "class Number:\n", " \"A number.\"\n", " def __init__(self, num): self.num = num\n", " def __add__(self, other):\n", " \"Sum of this and `other`.\"\n", " return Number(self.num + other.num)\n", " def __repr__(self): return f'Number({self.num})'\n", "```\n", "\n", "For example, here is the number 5:\n", "\n", "```python\n", "Number(5)\n", "```\n", "\n", "```python\n", "show_doc(Number.__add__)\n", "```\n", "\n", "For example:\n", "\n", "```python\n", "Number(3) + Number(4)\n", "```\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "5c5d8036", "metadata": {}, "source": [ "### Notebook (@patch)" ] }, { "cell_type": "markdown", "id": "77fa89e2", "metadata": {}, "source": [ "If you split your class definition across cells with `fastcore.basics.patch`, here's what your notebook might look like:" ] }, { "cell_type": "markdown", "id": "8e058c39", "metadata": {}, "source": [ "::: {.pt-4 .pb-1 .px-3 .mb-2 .border .rounded .shadow-sm}\n", "\n", "```python\n", "#| export\n", "class Number:\n", " \"A number.\"\n", " def __init__(self, num): self.num = num\n", " def __repr__(self): return f'Number({self.num})'\n", "```\n", "\n", "For example, here is the number 5:\n", "\n", "```python\n", "Number(5)\n", "```\n", "\n", "```python\n", "#| export\n", "@patch\n", "def __add__(self:Number, other):\n", " \"Sum of this and `other`.\"\n", " return Number(self.num + other.num)\n", "```\n", "\n", "For example:\n", "\n", "```python\n", "Number(3) + Number(4)\n", "```\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "e0733e60", "metadata": {}, "source": [ "### Docs" ] }, { "cell_type": "markdown", "id": "38e418ca", "metadata": {}, "source": [ "In either case, this is how the documentation would be rendered:" ] }, { "cell_type": "markdown", "id": "0ab8682a", "metadata": {}, "source": [ "::: {}" ] }, { "cell_type": "code", "execution_count": null, "id": "3f8bd614", "metadata": {}, "outputs": [], "source": [ "#| hide\n", "class Number:\n", " \"A number.\"\n", " def __init__(self, num): self.num = num\n", " def __add__(self, other):\n", " \"Sum of this and `other`.\"\n", " return Number(self.num + other.num)\n", " def __repr__(self): return f'Number({self.num})'" ] }, { "cell_type": "code", "execution_count": null, "id": "9f49d293", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "---\n", "\n", "### Number\n", "\n", "> Number (num)\n", "\n", "A number." ], "text/plain": [ "---\n", "\n", "### Number\n", "\n", "> Number (num)\n", "\n", "A number." ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| eval: false\n", "show_doc(Number)" ] }, { "cell_type": "markdown", "id": "bb9fe1c7", "metadata": {}, "source": [ "For example, here is the number 5:" ] }, { "cell_type": "code", "execution_count": null, "id": "e11b7da9", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Number(5)" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Number(5)" ] }, { "cell_type": "code", "execution_count": null, "id": "e65aa0b9", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "---\n", "\n", "### Number.__add__\n", "\n", "> Number.__add__ (other)\n", "\n", "Sum of this and `other`." ], "text/plain": [ "---\n", "\n", "### Number.__add__\n", "\n", "> Number.__add__ (other)\n", "\n", "Sum of this and `other`." ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| eval: false\n", "show_doc(Number.__add__)" ] }, { "cell_type": "markdown", "id": "9c901d93", "metadata": {}, "source": [ "For example:" ] }, { "cell_type": "code", "execution_count": null, "id": "f27b5cc0", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Number(7)" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Number(3) + Number(4)" ] }, { "cell_type": "markdown", "id": "9b96d368", "metadata": {}, "source": [ ":::" ] }, { "cell_type": "markdown", "id": "9086592e", "metadata": {}, "source": [ ":::" ] }, { "cell_type": "markdown", "id": "840c0cc0", "metadata": {}, "source": [ "## Group symbols with H2 sections" ] }, { "cell_type": "markdown", "id": "7c645118", "metadata": {}, "source": [ "As your notebooks grow, consider grouping related symbols using markdown cells with level 2 headers. Since nbdev displays documented symbols as level 3 headers, this would group all symbols below your level 2 header.\n", "\n", "Here is the markdown syntax:" ] }, { "cell_type": "markdown", "id": "ad2629ad", "metadata": {}, "source": [ "```markdown\n", "## Section title\n", "```" ] }, { "cell_type": "markdown", "id": "b4e7f322", "metadata": {}, "source": [ "## Split long explanations with H4 sections" ] }, { "cell_type": "markdown", "id": "d0b8423b", "metadata": {}, "source": [ "Similar to the previous section, as a symbol's explanation grows, consider grouping its cells using level 4 headers. This is the recommended way to structure your reference docs, for example, to achieve numpy-style structures with sections like notes, examples, methods, and so on.\n", "\n", "Here's the markdown syntax:" ] }, { "cell_type": "markdown", "id": "81ed35a8", "metadata": {}, "source": [ "```markdown\n", "#### Section title\n", "```" ] }, { "cell_type": "markdown", "id": "bf975e5f-ee0d-4918-a5c6-e0ff52ee8860", "metadata": {}, "source": [ "## Putting it all together: an annotated example" ] }, { "cell_type": "markdown", "id": "6b0a54ad-8ca1-41aa-8589-1fc577dead91", "metadata": {}, "source": [ "In this section, we'll guide you through a full example of writing a documented and tested function in a notebook using all of the principles described above. We'll use the `numpy.all` function since it follows the widely-known numpy-docstring standard for .py files." ] }, { "cell_type": "markdown", "id": "417b5923-aa1f-4c14-bb65-1dbe0bb12d98", "metadata": {}, "source": [ "Below is the definition of the `numpy.all` function. Take note of how all of the information is included in the docstring. While this works well for .py files, it doesn't let us weave executable code with rich markdown as we can in notebooks:" ] }, { "cell_type": "code", "execution_count": null, "id": "a6a40d5e", "metadata": {}, "outputs": [], "source": [ "def all(a, axis=None, out=None, keepdims=np._NoValue, *, where=np._NoValue):\n", " \"\"\"\n", " Test whether all array elements along a given axis evaluate to True.\n", " Parameters\n", " ----------\n", " a : array_like\n", " Input array or object that can be converted to an array.\n", " axis : None or int or tuple of ints, optional\n", " Axis or axes along which a logical AND reduction is performed.\n", " The default (``axis=None``) is to perform a logical AND over all\n", " the dimensions of the input array. `axis` may be negative, in\n", " which case it counts from the last to the first axis.\n", " .. versionadded:: 1.7.0\n", " If this is a tuple of ints, a reduction is performed on multiple\n", " axes, instead of a single axis or all the axes as before.\n", " out : ndarray, optional\n", " Alternate output array in which to place the result.\n", " It must have the same shape as the expected output and its\n", " type is preserved (e.g., if ``dtype(out)`` is float, the result\n", " will consist of 0.0's and 1.0's). See :ref:`ufuncs-output-type` for more\n", " details.\n", " keepdims : bool, optional\n", " If this is set to True, the axes which are reduced are left\n", " in the result as dimensions with size one. With this option,\n", " the result will broadcast correctly against the input array.\n", " If the default value is passed, then `keepdims` will not be\n", " passed through to the `all` method of sub-classes of\n", " `ndarray`, however any non-default value will be. If the\n", " sub-class' method does not implement `keepdims` any\n", " exceptions will be raised.\n", " where : array_like of bool, optional\n", " Elements to include in checking for all `True` values.\n", " See `~numpy.ufunc.reduce` for details.\n", " .. versionadded:: 1.20.0\n", " Returns\n", " -------\n", " all : ndarray, bool\n", " A new boolean or array is returned unless `out` is specified,\n", " in which case a reference to `out` is returned.\n", " See Also\n", " --------\n", " ndarray.all : equivalent method\n", " any : Test whether any element along a given axis evaluates to True.\n", " Notes\n", " -----\n", " Not a Number (NaN), positive infinity and negative infinity\n", " evaluate to `True` because these are not equal to zero.\n", " Examples\n", " --------\n", " >>> np.all([[True,False],[True,True]])\n", " False\n", " >>> np.all([[True,False],[True,True]], axis=0)\n", " array([ True, False])\n", " >>> np.all([-1, 4, 5])\n", " True\n", " >>> np.all([1.0, np.nan])\n", " True\n", " >>> np.all([[True, True], [False, True]], where=[[True], [False]])\n", " True\n", " >>> o=np.array(False)\n", " >>> z=np.all([-1, 4, 5], out=o)\n", " >>> id(z), id(o), z\n", " (28293632, 28293632, array(True)) # may vary\n", " \"\"\"\n", " ..." ] }, { "cell_type": "markdown", "id": "bf63b97d-350c-42f7-87c3-e5e57b2dfc08", "metadata": {}, "source": [ "Alternatively, Here is how we'd write `numpy.all` in a notebook using nbdev. The first step is to define the function:" ] }, { "cell_type": "markdown", "id": "b1976a6c-ce95-48c3-befe-5cf5f4c10a41", "metadata": {}, "source": [ "::: {.column-screen-inset-right}\n", "\n", "```python\n", "#| export\n", "def all(a, # Input array or object that can be converted to an array.\n", " axis:int|tuple|None=None, # Axis or axes along which a logical AND reduction is performed (default: all).\n", " out:np.ndarray|None=None, # Alternate output array in which to place the result.\n", " keepdims:bool=np._NoValue, # Leave reduced one-dimensional axes in the result?\n", " where=np._NoValue, # Elements to include in reduction. See `numpy.ufunc.reduce` for details. New in version 1.20.0.\n", " ) -> np.ndarray|bool: # A new boolean or array, or a reference to `out` if its specified.\n", " \"Test whether all array elements along a given axis evaluate to `True`.\"\n", " ...\n", "```\n", "\n", "::: " ] }, { "cell_type": "markdown", "id": "631ec60f-615a-4e23-8f6f-5369479ee11f", "metadata": {}, "source": [ "We can observe the following differences between this code and numpy-docstrings:\n", "\n", "- The definition uses simple type annotations, which will be rendered in the function's parameters table below\n", "- Parameters are described with a short comment, called [docments](#document-parameters-with-docments) -- a concise alternative to numpy and sphinx docstring formats (although nbdev does support numpy docstrings see [this example](#document-parameters-with-docments))\n", "- The docstring and parameter descriptions are all short, single-line summaries. We prefer to [keep docstrings short and instead elaborate in separate cells](#keep-docstrings-short-elaborate-in-separate-cells), where we can use markdown and real code examples.\n", "\n", "Note: the use of `|` syntax for unions e.g. `int|tuple|None` (equivalent to `Union[int, tuple, None]`) requires using Python 3.10 or by treating all annotations as strings using `from __future__ import annotations` which is available from Python 3.7." ] }, { "cell_type": "markdown", "id": "39bb74cf-5ed1-432a-83a6-4fb9d8fd43dd", "metadata": {}, "source": [ "Our function definition is automatically rendered in the docs like this. Note that parameter names, types, defaults, and details are all parsed from the definition which means you don't have to repeat yourself." ] }, { "cell_type": "markdown", "id": "3173bb16-d65a-4df2-8b89-da27218cbc4d", "metadata": {}, "source": [ "::: {.pt-2 .pb-1 .px-3 .mt-2 .mb-4 .border .rounded .shadow-sm .overflow-auto}" ] }, { "cell_type": "code", "execution_count": null, "id": "0c20416d", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "---\n", "\n", "### all\n", "\n", "> all (a, axis:Union[int,tuple,NoneType]=None,\n", "> out:Optional[numpy.ndarray]=None, keepdims:bool=,\n", "> where=)\n", "\n", "Test whether all array elements along a given axis evaluate to `True`.\n", "\n", "| | **Type** | **Default** | **Details** |\n", "| -- | -------- | ----------- | ----------- |\n", "| a | | | Input array or object that can be converted to an array. |\n", "| axis | int \\| tuple \\| None | None | Axis or axes along which a logical AND reduction is performed (default: all). |\n", "| out | np.ndarray \\| None | None | Alternate output array in which to place the result. |\n", "| keepdims | bool | | Leave reduced one-dimensional axes in the result? |\n", "| where | _NoValueType | | Elements to include in reduction. See `numpy.ufunc.reduce` for details. New in version 1.20.0. |\n", "| **Returns** | **np.ndarray \\| bool** | | **A new boolean or array, or a reference to `out` if its specified.** |" ], "text/plain": [ "---\n", "\n", "### all\n", "\n", "> all (a, axis:Union[int,tuple,NoneType]=None,\n", "> out:Optional[numpy.ndarray]=None, keepdims:bool=,\n", "> where=)\n", "\n", "Test whether all array elements along a given axis evaluate to `True`.\n", "\n", "| | **Type** | **Default** | **Details** |\n", "| -- | -------- | ----------- | ----------- |\n", "| a | | | Input array or object that can be converted to an array. |\n", "| axis | int \\| tuple \\| None | None | Axis or axes along which a logical AND reduction is performed (default: all). |\n", "| out | np.ndarray \\| None | None | Alternate output array in which to place the result. |\n", "| keepdims | bool | | Leave reduced one-dimensional axes in the result? |\n", "| where | _NoValueType | | Elements to include in reduction. See `numpy.ufunc.reduce` for details. New in version 1.20.0. |\n", "| **Returns** | **np.ndarray \\| bool** | | **A new boolean or array, or a reference to `out` if its specified.** |" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| eval: false\n", "def all(a, # Input array or object that can be converted to an array.\n", " axis:int|tuple|None=None, # Axis or axes along which a logical AND reduction is performed (default: all).\n", " out:np.ndarray|None=None, # Alternate output array in which to place the result.\n", " keepdims:bool=np._NoValue, # Leave reduced one-dimensional axes in the result?\n", " where=np._NoValue, # Elements to include in reduction. See `numpy.ufunc.reduce` for details. New in version 1.20.0.\n", " ) -> np.ndarray|bool: # A new boolean or array, or a reference to `out` if its specified.\n", " \"Test whether all array elements along a given axis evaluate to `True`.\"\n", " ...\n", "\n", "show_doc(all)" ] }, { "cell_type": "markdown", "id": "a2f0f0b8-f16c-4f1b-a9c9-5b2250c353af", "metadata": {}, "source": [ ":::" ] }, { "cell_type": "markdown", "id": "aa21852c-7a41-4b9a-b764-452ddb83a5f1", "metadata": {}, "source": [ "Next, describe how to use your function using markdown cells and [lots of code examples](#use-lots-of-code-examples-pictures-plots-and-videos). This is the biggest benefit to developing in notebooks: instead of copying and pasting code examples into plaintext, you can include real executeable code examples.\n", "\n", "We start with basic usage first:" ] }, { "cell_type": "markdown", "id": "3525b5ad-99de-4bdf-bfe8-d92d1f3cc292", "metadata": {}, "source": [ "::: {.pt-3 .pb-1 .px-3 .mt-2 .mb-4 .border .rounded .shadow-sm}" ] }, { "cell_type": "markdown", "id": "0fb6fb1d-eade-476a-b1a4-30936f6f93a3", "metadata": {}, "source": [ "For example:" ] }, { "cell_type": "code", "execution_count": null, "id": "5cbee86a", "metadata": {}, "outputs": [], "source": [ "x = [[True,False],[True,True]]\n", "test_eq(np.all(x), False)" ] }, { "cell_type": "markdown", "id": "cda5c03c-7258-4b0f-b832-b9567ab634cb", "metadata": {}, "source": [ ":::" ] }, { "cell_type": "markdown", "id": "7cfdfdd7-7f44-46d5-9f2e-53cc1b3e0a48", "metadata": {}, "source": [ "Our code examples [use assertion functions from `fastcore.test`](#consider-turning-code-examples-into-tests-by-adding-assertions), so that they serve as both docs and tests. `nbdev_test` runs every code cell as a test (unless it’s explicitly marked otherwise), and any error in the cell fails the test." ] }, { "cell_type": "markdown", "id": "c48945f7-371d-4f73-b4f7-cffaa042c868", "metadata": {}, "source": [ "Having described basic usage, we now elaborate on more advanced functionality for each parameter. This differs from numpy's approach which includes all parameter docs in the table and where not all parameters have code examples." ] }, { "cell_type": "markdown", "id": "eeb26c41-7e1a-40db-b804-efa279008c83", "metadata": {}, "source": [ "::: {.pt-3 .pb-1 .px-3 .mt-2 .mb-4 .border .rounded .shadow-sm}" ] }, { "cell_type": "markdown", "id": "71746f8b-1a17-4054-a38f-2587f594db32", "metadata": {}, "source": [ "With `axis`:" ] }, { "cell_type": "code", "execution_count": null, "id": "c3e82a98", "metadata": {}, "outputs": [], "source": [ "test_eq(np.all(x, axis=0), [True,False])" ] }, { "cell_type": "markdown", "id": "2d47a9e4-1052-45b7-a9d3-04c4756ea79b", "metadata": {}, "source": [ "`axis` may be negative, in which case it counts from the last to the first axis:" ] }, { "cell_type": "code", "execution_count": null, "id": "b5eeac7d", "metadata": {}, "outputs": [], "source": [ "test_eq(np.all(x, axis=-1), [False,True])" ] }, { "cell_type": "markdown", "id": "8f52148d-98be-46ec-953c-b5dbef18ec89", "metadata": {}, "source": [ "If `axis` is a tuple of ints, a reduction is performed on multiple\n", "axes, instead of a single axis or all the axes as before." ] }, { "cell_type": "code", "execution_count": null, "id": "d88ff8f7", "metadata": {}, "outputs": [], "source": [ "test_eq(np.all(x, axis=(0,1)), False)" ] }, { "cell_type": "markdown", "id": "8802f2cc-e9e9-4d09-9176-b078f28a2e1d", "metadata": {}, "source": [ "Integers, floats, not a number (nan), and infinity all evaluate to `True` because they're not equal to zero:" ] }, { "cell_type": "code", "execution_count": null, "id": "140f54ff", "metadata": {}, "outputs": [], "source": [ "test_eq(np.all([-1, 1, -1.0, 1.0, np.nan, np.inf, -np.inf]), True)" ] }, { "cell_type": "markdown", "id": "75c0aef7-2440-4713-9309-a8c814934704", "metadata": {}, "source": [ "You can use `where` to test specific elements. For example, this tests only the second column:" ] }, { "cell_type": "code", "execution_count": null, "id": "77791ed8", "metadata": {}, "outputs": [], "source": [ "test_eq(np.all(x, where=[[False],[True]]), True)" ] }, { "cell_type": "markdown", "id": "1776798f-4956-4807-bd32-ac522b194105", "metadata": {}, "source": [ "The output can be stored in an optional `out` array. If provided, a reference to `out` will be returned:" ] }, { "cell_type": "code", "execution_count": null, "id": "a54a5c79", "metadata": {}, "outputs": [], "source": [ "o = np.array(False)\n", "z = np.all([-1, 4, 5], out=o)\n", "test_is(z, o)\n", "test_eq(z, True)" ] }, { "cell_type": "markdown", "id": "7c8e5694-c2f6-4629-95f7-b19e7b721037", "metadata": {}, "source": [ "`out` must have the same shape as the expected output and its type is preserved (e.g., if `dtype(out)` is float, the result will consist of 0.0's and 1.0's). See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more\n", "details." ] }, { "cell_type": "markdown", "id": "eb90daba-3bf8-4245-9a55-fd5d34bbbdff", "metadata": {}, "source": [ "With `keepdims`, the result will broadcast correctly against the input array." ] }, { "cell_type": "code", "execution_count": null, "id": "6f585fea", "metadata": {}, "outputs": [], "source": [ "test_eq(np.all(x, axis=0, keepdims=True), [[True, False]]) # Note the nested list" ] }, { "cell_type": "markdown", "id": "111981a9-c0dc-4697-8baf-598293988192", "metadata": {}, "source": [ "If the default value is passed, then `keepdims` will not be passed through to the `all` method of sub-classes of `ndarray`, however any non-default value will be. If the sub-class' method does not implement `keepdims` any exceptions will be raised." ] }, { "cell_type": "code", "execution_count": null, "id": "4f05c8e8", "metadata": {}, "outputs": [], "source": [ "class MyArray(np.ndarray):\n", " def all(self, axis=None, out=None): ...\n", "\n", "y = MyArray((2,2))\n", "y[:] = x\n", "np.all(y) # No TypeError since `keepdims` isn't passed\n", "test_fail(lambda: np.all(y, keepdims=True), contains=\"all() got an unexpected keyword argument 'keepdims'\")" ] }, { "cell_type": "markdown", "id": "4cf6b9b6-646b-46aa-9f37-20a9d83e9f17", "metadata": {}, "source": [ ":::" ] }, { "cell_type": "markdown", "id": "ab7948cb-355f-46f3-a68d-f03c22db6f18", "metadata": {}, "source": [ "Since we prefer to document via code examples, we also document error-cases with assertions using `fastcore.test.test_fail`. This differs from docstring-based approaches which usually document error-cases in prose, usually in a \"raises\" section of the docstring." ] }, { "cell_type": "markdown", "id": "3852101a-e3e5-42f7-89dd-c2f3f66ebdc3", "metadata": {}, "source": [ "Finally, we link to related symbols with [doclinks](#reference-related-symbols-with-doclinks) (symbols surrounded in backticks are automatically linked) and describe their relation using code examples." ] }, { "cell_type": "markdown", "id": "9d8c6de5-ebd6-4fa0-a69c-2512a8f54689", "metadata": {}, "source": [ "::: {.pt-3 .pb-1 .px-3 .mt-2 .mb-4 .border .rounded .shadow-sm}" ] }, { "cell_type": "markdown", "id": "f40e6aa8-9681-4265-b234-930b4801ccf0", "metadata": {}, "source": [ "The `numpy.ndarray.all` method is equivalent to calling `numpy.all` with the array:" ] }, { "cell_type": "code", "execution_count": null, "id": "3370e60e", "metadata": {}, "outputs": [], "source": [ "test_eq(np.array(x).all(), np.all(x))" ] }, { "cell_type": "markdown", "id": "485a591d-6b1c-45b8-8814-4bd76e82175d", "metadata": {}, "source": [ "In contrast, `numpy.any` tests whether _any_ element evaluates to `True` (rather than _all_ elements):" ] }, { "cell_type": "code", "execution_count": null, "id": "e6b24e25", "metadata": {}, "outputs": [], "source": [ "test_eq(np.any(x), True)" ] }, { "cell_type": "markdown", "id": "80b1fd1d-5031-4309-8e12-8db2b50d4dc4", "metadata": {}, "source": [ ":::" ] }, { "cell_type": "markdown", "id": "ac918fbc-6863-4552-b1ae-7b314fabfc02", "metadata": {}, "source": [ "### Recap" ] }, { "cell_type": "markdown", "id": "00a32927-2de1-4b27-9262-d3754ccb62c6", "metadata": {}, "source": [ "In summary, here is how the nbdev version of `numpy.all` differs from the numpy docstring. nbdev uses:\n", "\n", "- Type annotations and docments instead of the numpy docstring format (although nbdev supports numpy docstrings too)\n", "- Short parameter descriptions, with details in separate cells with markdown and code example\n", "- Doclinks to related symbols instead of a \"See also\" section\n", "- Lots of code examples (which are also tests) mixed with prose to describe how to use the function\n", "- Code examples with assertions to document error-cases instead of a \"Raises\" section." ] }, { "cell_type": "code", "execution_count": null, "id": "488448e2", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "python3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 5 }