{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#hide\n", "# default_exp script" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Script - command line interfaces\n", "\n", "> A fast way to turn your python function into a script." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Part of [fast.ai](https://www.fast.ai)'s toolkit for delightful developer experiences." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, you want to create a quick script, either for yourself, or for others. But in Python, that involves a whole lot of boilerplate and ceremony, especially if you want to support command line arguments, provide help, and other niceties. You can use [argparse](https://docs.python.org/3/library/argparse.html) for this purpose, which comes with Python, but it's complex and verbose.\n", "\n", "`fastcore.script` makes life easier. There are much fancier modules to help you write scripts (we recommend [Python Fire](https://github.com/google/python-fire), and [Click](https://click.palletsprojects.com/en/7.x/) is also popular), but fastcore.script is very fast and very simple. In fact, it's <50 lines of code! Basically, it's just a little wrapper around `argparse` that uses modern Python features and some thoughtful defaults to get rid of the boilerplate.\n", "\n", "For full details, see the [docs](https://fastcore.script.fast.ai) for `core`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here's a complete example (available in `examples/test_fastcore.py`):\n", "\n", "```python\n", "from fastcore.script import *\n", "@call_parse\n", "def main(msg:Param(\"The message\", str),\n", " upper:Param(\"Convert to uppercase?\", store_true)):\n", " \"Print `msg`, optionally converting to uppercase\"\n", " print(msg.upper() if upper else msg)\n", "````" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you copy that info a file and run it, you'll see:\n", "\n", "```\n", "$ examples/test_fastcore.py --help\n", "usage: test_fastcore.py [-h] [--upper] [--pdb PDB] [--xtra XTRA] msg\n", "\n", "Print `msg`, optionally converting to uppercase\n", "\n", "positional arguments:\n", " msg The message\n", "\n", "optional arguments:\n", " -h, --help show this help message and exit\n", " --upper Convert to uppercase? (default: False)\n", " --pdb PDB Run in pdb debugger (default: False)\n", " --xtra XTRA Parse for additional args (default: '')\n", "```\n", "\n", "As you see, we didn't need any `if __name__ == \"__main__\"`, we didn't have to parse arguments, we just wrote a function, added a decorator to it, and added some annotations to our function's parameters. As a bonus, we can also use this function directly from a REPL such as Jupyter Notebook - it's not just for command line scripts!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Param annotations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each parameter in your function should have an annotation `Param(...)` (as in the example above). You can pass the following when calling `Param`: `help`,`type`,`opt`,`action`,`nargs`,`const`,`choices`,`required` . Except for `opt`, all of these are just passed directly to `argparse`, so you have all the power of that module at your disposal. Generally you'll want to pass at least `help` (since this is provided as the help string for that parameter) and `type` (to ensure that you get the type of data you expect). `opt` is a bool that defines whether a param is optional or required (positional) - but you'll generally not need to set this manually, because fastcore.script will set it for you automatically based on *default* values.\n", "\n", "You should provide a default (after the `=`) for any *optional* parameters. If you don't provide a default for a parameter, then it will be a *positional* parameter." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## setuptools scripts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There's a really nice feature of pip/setuptools that lets you create commandline scripts directly from functions, makes them available in the `PATH`, and even makes your scripts cross-platform (e.g. in Windows it creates an exe). fastcore.script supports this feature too. The trick to making a function available as a script is to add a `console_scripts` section to your setup file, of the form: `script_name=module:function_name`. E.g. in this case we use: `test_fastcore.script=fastcore.script.test_cli:main`. With this, you can then just type `test_fastcore.script` at any time, from any directory, and your script will be called (once it's installed using one of the methods below).\n", "\n", "You don't actually have to write a `setup.py` yourself. Instead, just use [nbdev](https://nbdev.fast.ai). Then modify `settings.ini` as appropriate for your module/script. To install your script directly, you can type `pip install -e .`. Your script, when installed this way (it's called an [editable install](http://codumentary.blogspot.com/2014/11/python-tip-of-year-pip-install-editable.html), will automatically be up to date even if you edit it - there's no need to reinstall it after editing. With nbdev you can even make your module and script available for installation directly from pip and conda by running `make release`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## API details" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from fastcore.test import *" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "import inspect,functools\n", "import argparse\n", "from fastcore.imports import *\n", "from fastcore.utils import *" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "def store_true():\n", " \"Placeholder to pass to `Param` for `store_true` action\"\n", " pass" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "def store_false():\n", " \"Placeholder to pass to `Param` for `store_false` action\"\n", " pass" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "def bool_arg(v):\n", " \"Use as `type` for `Param` to get `bool` behavior\"\n", " return str2bool(v)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "class Param:\n", " \"A parameter in a function used in `anno_parser` or `call_parse`\"\n", " def __init__(self, help=None, type=None, opt=True, action=None, nargs=None, const=None,\n", " choices=None, required=None, default=None):\n", " if type==store_true: type,action,default=None,'store_true' ,False\n", " if type==store_false: type,action,default=None,'store_false',True\n", " store_attr()\n", " \n", " def set_default(self, d):\n", " if self.default is None:\n", " if d==inspect.Parameter.empty: self.opt = False\n", " else: self.default = d\n", " if self.default is not None: self.help += f\" (default: {self.default})\"\n", "\n", " @property\n", " def pre(self): return '--' if self.opt else ''\n", " @property\n", " def kwargs(self): return {k:v for k,v in self.__dict__.items()\n", " if v is not None and k!='opt' and k[0]!='_'}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each parameter in your function should have an annotation `Param(...)`. You can pass the following when calling `Param`: `help`,`type`,`opt`,`action`,`nargs`,`const`,`choices`,`required` (i.e. it takes the same parameters as `argparse.ArgumentParser.add_argument`, plus `opt`). Except for `opt`, all of these are just passed directly to `argparse`, so you have all the power of that module at your disposal. Generally you'll want to pass at least `help` (since this is provided as the help string for that parameter) and `type` (to ensure that you get the type of data you expect).\n", "\n", "`opt` is a bool that defines whether a param is optional or required (positional) - but you'll generally not need to set this manually, because fastcore.script will set it for you automatically based on *default* values. You should provide a default (after the `=`) for any *optional* parameters. If you don't provide a default for a parameter, then it will be a *positional* parameter." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "p = Param(help=\"help\", type=int)\n", "p.set_default(1)\n", "test_eq(p.kwargs, {'help': 'help (default: 1)', 'type': int, 'default': 1})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "def anno_parser(func, prog=None, from_name=False):\n", " \"Look at params (annotated with `Param`) in func and return an `ArgumentParser`\"\n", " p = argparse.ArgumentParser(description=func.__doc__, prog=prog)\n", " for k,v in inspect.signature(func).parameters.items():\n", " param = func.__annotations__.get(k, Param())\n", " param.set_default(v.default)\n", " p.add_argument(f\"{param.pre}{k}\", **param.kwargs)\n", " p.add_argument(f\"--pdb\", help=\"Run in pdb debugger (default: False)\", action='store_true')\n", " p.add_argument(f\"--xtra\", help=\"Parse for additional args (default: '')\", type=str)\n", " return p" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This converts a function with parameter annotations of type `Param` into an `argparse.ArgumentParser` object. Function arguments with a default provided are optional, and other arguments are positional." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: progname [-h] [--b B] [--pdb] [--xtra XTRA] required a\n", "\n", "my docs\n", "\n", "positional arguments:\n", " required Required param\n", " a param 1\n", "\n", "optional arguments:\n", " -h, --help show this help message and exit\n", " --b B param 2 (default: test)\n", " --pdb Run in pdb debugger (default: False)\n", " --xtra XTRA Parse for additional args (default: '')\n" ] } ], "source": [ "def f(required:Param(\"Required param\", int),\n", " a:Param(\"param 1\", bool_arg),\n", " b:Param(\"param 2\", str)=\"test\"):\n", " \"my docs\"\n", " ...\n", "\n", "p = anno_parser(f, 'progname')\n", "p.print_help()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "def args_from_prog(func, prog):\n", " \"Extract args from `prog`\"\n", " if prog is None or '#' not in prog: return {}\n", " if '##' in prog: _,prog = prog.split('##', 1)\n", " progsp = prog.split(\"#\")\n", " args = {progsp[i]:progsp[i+1] for i in range(0, len(progsp), 2)}\n", " for k,v in args.items():\n", " t = func.__annotations__.get(k, Param()).type\n", " if t: args[k] = t(v)\n", " return args" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes it's convenient to extract arguments from the actual name of the called program. `args_from_prog` will do this, assuming that names and values of the params are separated by a `#`. Optionally there can also be a prefix separated by `##` (double underscore)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "exp = {'a': False, 'b': 'baa'}\n", "test_eq(args_from_prog(f, 'foo##a#0#b#baa'), exp)\n", "test_eq(args_from_prog(f, 'a#0#b#baa'), exp)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#export\n", "def call_parse(func):\n", " \"Decorator to create a simple CLI from `func` using `anno_parser`\"\n", " mod = inspect.getmodule(inspect.currentframe().f_back)\n", " if not mod: return func\n", " \n", " @functools.wraps(func)\n", " def _f(*args, **kwargs):\n", " mod = inspect.getmodule(inspect.currentframe().f_back)\n", " if not mod: return func(*args, **kwargs)\n", " p = anno_parser(func)\n", " args = p.parse_args().__dict__\n", " xtra = otherwise(args.pop('xtra', ''), eq(1), p.prog)\n", " tfunc = trace(func) if args.pop('pdb', False) else func\n", " tfunc(**merge(args, args_from_prog(func, xtra)))\n", "\n", " if mod.__name__==\"__main__\":\n", " setattr(mod, func.__name__, _f)\n", " return _f()\n", " else: return _f" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "@call_parse\n", "def test_add(a:Param(\"param a\", int), b:Param(\"param 1\",int)): return a + b" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`call_parse` decorated functions work as regular functions and also as command-line interface functions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_eq(test_add(1,2), 3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is the main way to use `fastcore.script`; decorate your function with `call_parse`, add `Param` annotations as shown above, and it can then be used as a script." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export -" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Converted 00_test.ipynb.\n", "Converted 01_foundation.ipynb.\n", "Converted 02_utils.ipynb.\n", "Converted 03_dispatch.ipynb.\n", "Converted 04_transform.ipynb.\n", "Converted 05_logargs.ipynb.\n", "Converted 06_meta.ipynb.\n", "Converted 07_script.ipynb.\n", "Converted index.ipynb.\n" ] } ], "source": [ "#hide\n", "from nbdev.export import notebook2script\n", "notebook2script()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "jupytext": { "split_at_heading": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 2 }