{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": false }, "outputs": [], "source": [ "#hide\n", "#default_exp cli\n", "from nbdev.showdoc import show_doc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Command line functions\n", "\n", "> Console commands added by the nbdev library" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "from nbdev.imports import *\n", "from nbdev.export import *\n", "from nbdev.sync import *\n", "from nbdev.merge import *\n", "from nbdev.export2html import *\n", "from nbdev.clean import *\n", "from nbdev.test import *\n", "from fastcore.script import *\n", "from ghapi.all import GhApi\n", "from urllib.error import HTTPError\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`nbdev` comes with the following commands. To use any of them, you must be in one of the subfolders of your project: they will search for the `settings.ini` recursively in the parent directory but need to access it to be able to work. Their names all begin with nbdev so you can easily get a list with tab completion.\n", "- `nbdev_build_docs` builds the documentation from the notebooks\n", "- `nbdev_build_lib` builds the library from the notebooks\n", "- `nbdev_bump_version` increments version in `settings.py` by one\n", "- `nbdev_clean_nbs` removes all superfluous metadata form the notebooks, to avoid merge conflicts\n", "- `nbdev_detach` exports cell attachments to `dest` and updates references\n", "- `nbdev_diff_nbs` gives you the diff between the notebooks and the exported library\n", "- `nbdev_fix_merge` will fix merge conflicts in a notebook file\n", "- `nbdev_install_git_hooks` installs the git hooks that use the last two command automatically on each commit/merge\n", "- `nbdev_nb2md` converts a notebook to a markdown file\n", "- `nbdev_new` creates a new nbdev project\n", "- `nbdev_read_nbs` reads all notebooks to make sure none are broken\n", "- `nbdev_test_nbs` runs tests in notebooks\n", "- `nbdev_trust_nbs` trusts all notebooks (so that the HTML content is shown)\n", "- `nbdev_update_lib` propagates any change in the library back to the notebooks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Navigating from notebooks to script and back" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_build_lib[source]

\n", "\n", "> nbdev_build_lib(**`fname`**:\"A notebook name or glob to convert\"=*`None`*, **`bare`**:\"Omit nbdev annotation comments (may break some functionality).\"=*`False`*)\n", "\n", "Export notebooks matching `fname` to python modules" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_build_lib)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default (`fname` left to `None`), the whole library is built from the notebooks in the `lib_folder` set in your `settings.ini`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_update_lib[source]

\n", "\n", "> nbdev_update_lib(**`fname`**:\"A python filename or glob to convert\"=*`None`*, **`silent`**:\"Don't print results\"=*`False`*)\n", "\n", "Propagates any change in the modules matching `fname` to the notebooks that created them" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_update_lib)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default (`fname` left to `None`), the whole library is treated. Note that this tool is only designed for small changes such as typo or small bug fixes. You can't add new cells in notebook from the library." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_diff_nbs[source]

\n", "\n", "> nbdev_diff_nbs()\n", "\n", "Prints the diff between an export of the library in notebooks and the actual modules" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_diff_nbs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running tests" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_test_nbs[source]

\n", "\n", "> nbdev_test_nbs(**`fname`**:\"A notebook name or glob to convert\"=*`None`*, **`flags`**:\"Space separated list of flags\"=*`None`*, **`n_workers`**:\"Number of workers to use\"=*`None`*, **`verbose`**:\"Print errors along the way\"=*`True`*, **`timing`**:\"Timing each notebook to see the ones are slow\"=*`False`*, **`pause`**:\"Pause time (in secs) between notebooks to avoid race conditions\"=*`0.5`*)\n", "\n", "Test in parallel the notebooks matching `fname`, passing along `flags`" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_test_nbs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default (`fname` left to `None`), the whole library is tested from the notebooks in the `lib_folder` set in your `settings.ini`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Building documentation" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_build_docs[source]

\n", "\n", "> nbdev_build_docs(**`fname`**:\"A notebook name or glob to convert\"=*`None`*, **`force_all`**:\"Rebuild even notebooks that haven't changed\"=*`False`*, **`mk_readme`**:\"Also convert the index notebook to README\"=*`True`*, **`n_workers`**:\"Number of workers to use\"=*`None`*, **`pause`**:\"Pause time (in secs) between notebooks to avoid race conditions\"=*`0.5`*)\n", "\n", "Build the documentation by converting notebooks matching `fname` to html" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_build_docs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default (`fname` left to `None`), the whole documentation is build from the notebooks in the `lib_folder` set in your `settings.ini`, only converting the ones that have been modified since the their corresponding html was last touched unless you pass `force_all=True`. The index is also converted to make the README file, unless you pass along `mk_readme=False`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_nb2md[source]

\n", "\n", "> nbdev_nb2md(**`fname`**:\"A notebook file name to convert\", **`dest`**:\"The destination folder\"=*`'.'`*, **`img_path`**:\"Folder to export images to\"=*`''`*, **`jekyll`**:\"To use jekyll metadata for your markdown file or not\"=*`False`*)\n", "\n", "Convert the notebook in `fname` to a markdown file" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_nb2md)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_detach[source]

\n", "\n", "> nbdev_detach(**`path_nb`**:\"Path to notebook\", **`dest`**:\"Destination folder\"=*`''`*, **`use_img`**:\"Convert markdown images to img tags\"=*`False`*, **`replace`**:\"Write replacement notebook back to `path_bn`\"=*`True`*)\n", "\n", "Export cell attachments to `dest` and update references" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_detach)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Other utils" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_read_nbs[source]

\n", "\n", "> nbdev_read_nbs(**`fname`**:\"A notebook name or glob to convert\"=*`None`*)\n", "\n", "Check all notebooks matching `fname` can be opened" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_read_nbs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default (`fname` left to `None`), the all the notebooks in `lib_folder` are checked." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_trust_nbs[source]

\n", "\n", "> nbdev_trust_nbs(**`fname`**:\"A notebook name or glob to convert\"=*`None`*, **`force_all`**:\"Trust even notebooks that haven't changed\"=*`False`*)\n", "\n", "Trust notebooks matching `fname`" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_trust_nbs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default (`fname` left to `None`), the all the notebooks in `lib_folder` are trusted. To speed things up, only the ones touched since the last time this command was run are trusted unless you pass along `force_all=True`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

nbdev_fix_merge[source]

\n", "\n", "> nbdev_fix_merge(**`fname`**:\"A notebook filename to fix\", **`fast`**:\"Fast fix: automatically fix the merge conflicts in outputs or metadata\"=*`True`*, **`trust_us`**:\"Use local outputs/metadata when fast merging\"=*`True`*)\n", "\n", "Fix merge conflicts in notebook `fname`" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(nbdev_fix_merge)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you have merge conflicts after a `git pull`, the notebook file will be broken and won't open in jupyter notebook anymore. This command fixes this by changing the notebook to a proper json file again and add markdown cells to signal the conflict, you just have to open that notebook again and look for `>>>>>>>` to see those conflicts and manually fix them. The old broken file is copied with a `.ipynb.bak` extension, so is still accessible in case the merge wasn't successful.\n", "\n", "Moreover, if `fast=True`, conflicts in outputs and metadata will automatically be fixed by using the local version if `trust_us=True`, the remote one if `trust_us=False`. With this option, it's very likely you won't have anything to do, unless there is a real conflict." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "def bump_version(version, part=2):\n", " version = version.split('.')\n", " version[part] = str(int(version[part]) + 1)\n", " for i in range(part+1, 3): version[i] = '0'\n", " return '.'.join(version)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_eq(bump_version('0.1.1' ), '0.1.2')\n", "test_eq(bump_version('0.1.1', 1), '0.2.0')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "@call_parse\n", "def nbdev_bump_version(part:Param(\"Part of version to bump\", int)=2):\n", " \"Increment version in `settings.py` by one\"\n", " cfg = Config()\n", " print(f'Old version: {cfg.version}')\n", " cfg.d['version'] = bump_version(Config().version, part)\n", " cfg.save()\n", " update_version()\n", " print(f'New version: {cfg.version}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Git hooks" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "@call_parse\n", "def nbdev_install_git_hooks():\n", " \"Install git hooks to clean/trust notebooks automatically\"\n", " try: path = Config().config_file.parent\n", " except: path = Path.cwd()\n", " hook_path = path/'.git'/'hooks'\n", " fn = hook_path/'post-merge'\n", " hook_path.mkdir(parents=True, exist_ok=True)\n", " #Trust notebooks after merge\n", " fn.write_text(\"#!/bin/bash\\necho 'Trusting notebooks'\\nnbdev_trust_nbs\")\n", " os.chmod(fn, os.stat(fn).st_mode | stat.S_IEXEC)\n", " #Clean notebooks on commit/diff\n", " (path/'.gitconfig').write_text(\"\"\"# Generated by nbdev_install_git_hooks\n", "#\n", "# If you need to disable this instrumentation do:\n", "# git config --local --unset include.path\n", "#\n", "# To restore the filter\n", "# git config --local include.path .gitconfig\n", "#\n", "# If you see notebooks not stripped, checked the filters are applied in .gitattributes\n", "#\n", "[filter \"clean-nbs\"]\n", " clean = nbdev_clean_nbs --read_input_stream True\n", " smudge = cat\n", " required = true\n", "[diff \"ipynb\"]\n", " textconv = nbdev_clean_nbs --disp True --fname\n", "\"\"\")\n", " cmd = \"git config --local include.path ../.gitconfig\"\n", " print(f\"Executing: {cmd}\")\n", " run(cmd)\n", " print(\"Success: hooks are installed and repo's .gitconfig is now trusted\")\n", " try: nb_path = Config().path(\"nbs_path\")\n", " except: nb_path = Path.cwd()\n", " (nb_path/'.gitattributes').write_text(\"**/*.ipynb filter=clean-nbs\\n**/*.ipynb diff=ipynb\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This command installs git hooks to make sure notebooks are cleaned before you commit them to GitHub and automatically trusted at each merge. To be more specific, this creates:\n", "- an executable '.git/hooks/post-merge' file that contains the command `nbdev_trust_nbs`\n", "- a `.gitconfig` file that uses `nbev_clean_nbs` has a filter/diff on all notebook files inside `nbs_folder` and a `.gitattributes` file generated in this folder (copy this file in other folders where you might have notebooks you want cleaned as well)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Starting a new project" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "_template_git_repo = \"https://github.com/fastai/nbdev_template.git\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "import tarfile" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "def extract_tgz(url, dest='.'):\n", " with urlopen(url) as u: tarfile.open(mode='r:gz', fileobj=u).extractall(dest)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "#hide\n", "def _get_branch(owner, repo, default='main'):\n", " api = GhApi(owner=owner, repo=repo, token=os.getenv('GITHUB_TOKEN'))\n", " try: return api.repos.get().default_branch\n", " except HTTPError: \n", " msg= [f\"Could not access repo: {owner}/{repo} to find your default branch - `{default} assumed.\\n\",\n", " \"Edit `settings.ini` if this is incorrect.\\n\"\n", " \"In the future, you can allow nbdev to see private repos by setting the environment variable GITHUB_TOKEN as described here: https://nbdev.fast.ai/cli.html#Using-nbdev_new-with-private-repos \\n\", ]\n", " print(''.join(msg))\n", " return default" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Could not access repo: fastai/definitely-does-not-exist to find your default branch - `main assumed.\n", "Edit `docs/_config.yml` if this is incorrect.\n", "In the future, you can allow nbdev to see private repos by setting the environment variable GITHUB_TOKEN as described here: https://nbdev.fast.ai/cli.html#Using-nbdev_new-with-private-repos \n", "\n" ] } ], "source": [ "#hide\n", "# should see warning message.\n", "assert _get_branch('fastai', 'definitely-does-not-exist') == 'main';" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "@call_parse\n", "def nbdev_new():\n", " \"Create a new nbdev project from the current git repo\"\n", " url = run('git config --get remote.origin.url')\n", " if not url: raise Exception('This does not appear to be a cloned git directory with a remote')\n", " author = run('git config --get user.name').strip()\n", " email = run('git config --get user.email').strip()\n", " if not (author and email): raise Exception('User name and email not configured in git')\n", "\n", " # download and untar template, and optionally notebooks\n", " tgnm = urljson('https://api.github.com/repos/fastai/nbdev_template/releases/latest')['tag_name']\n", " FILES_URL = f\"https://github.com/fastai/nbdev_template/archive/{tgnm}.tar.gz\"\n", " extract_tgz(FILES_URL)\n", " path = Path()\n", " nbexists = True if first(path.glob('*.ipynb')) else False\n", " for o in (path/f'nbdev_template-{tgnm}').ls(): \n", " if o.name == '00_core.ipynb':\n", " if not nbexists: shutil.move(str(o), './')\n", " elif not Path(f'./{o.name}').exists(): shutil.move(str(o), './') \n", " shutil.rmtree(f'nbdev_template-{tgnm}')\n", "\n", " # auto-config settings.ini from git\n", " settings_path = Path('settings.ini')\n", " settings = settings_path.read_text()\n", " owner,repo = repo_details(url)\n", " branch = _get_branch(owner, repo)\n", " settings = settings.format(lib_name=repo, user=owner, author=author, author_email=email, branch=branch)\n", " settings_path.write_text(settings) \n", " nbdev_install_git_hooks()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`nbdev_new` is a command line tool that creates a new nbdev project from the current directory, which must be a cloned git repo.\n", "\n", "After you run `nbdev_new`, please check the contents of `settings.ini` look good, and then run `nbdev_build_lib`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Using `nbdev_new` with private repos\n", "\n", "`nbdev_new` attempts to find your repo on GitHub to determine your default branch. If your repo is private, you will need to set the environment variable `GITHUB_TOKEN` with a [personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) prior to running `nbdev_new`, or `nbdev_new` will use a default value instead. \n", "\n", "You will receive a warning message if nbdev is not able to find your repo on GitHub that will tell you to replace the default branch name in `settings.ini` (which is the root of your repo)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export -" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Converted 00_export.ipynb.\n", "Converted 01_sync.ipynb.\n", "Converted 02_showdoc.ipynb.\n", "Converted 03_export2html.ipynb.\n", "Converted 04_test.ipynb.\n", "Converted 05_merge.ipynb.\n", "Converted 06_cli.ipynb.\n", "Converted 07_clean.ipynb.\n", "Converted 99_search.ipynb.\n", "Converted example.ipynb.\n", "Converted index.ipynb.\n", "Converted tutorial.ipynb.\n" ] } ], "source": [ "#hide\n", "from nbdev.export import notebook2script\n", "notebook2script()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "jupytext": { "split_at_heading": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 4 }