{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# \"fastcore: An Underrated Python Library\"\n", "\n", "> A unique python library that extends the python programming language and provides utilities that enhance productivity.\n", "- author: \"Hamel Husain\"\n", "- toc: false\n", "- image: images/copied_from_nb/fastcore_imgs/td.png\n", "- comments: true\n", "- categories: [fastcore, fastai]\n", "- permalink: /fastcore/\n", "- badges: true" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![screenshot with code](fastcore_imgs/td.png)\n", "\n", "# Background\n", "\n", "I recently embarked on a journey to sharpen my python skills: I wanted to learn advanced patterns, idioms, and techniques. I started with reading books on advanced Python, however, the information didn't seem to stick without having somewhere to apply it. I also wanted the ability to ask questions from an expert while I was learning -- which is an arrangement that is hard to find! That's when it occurred to me: What if I could find an open source project that has fairly advanced python code and write documentation and tests? I made a bet that if I did this it would force me to learn everything very deeply, and the maintainers would be appreciative of my work and be willing to answer my questions. \n", "\n", "And that's exactly what I did over the past month! I'm pleased to report that it has been the most efficient learning experience I've ever experienced. I've discovered that writing documentation forced me to deeply understand not just what the code does but also _why the code works the way it does_, and to explore edge cases while writing tests. Most importantly, I was able to ask questions when I was stuck, and maintainers were willing to devote extra time knowing that their mentorship was in service of making their code more accessible! It turns out the library I choose, [fastcore](https://fastcore.fast.ai/) is some of the most fascinating Python I have ever encountered as its purpose and goals are fairly unique.\n", "\n", "For the uninitiated, [fastcore](https://fastcore.fast.ai/) is a library on top of which many [fast.ai](https://github.com/fastai) projects are built on. Most importantly, [fastcore](https://fastcore.fast.ai/) extends the python programming language and strives to eliminate boilerplate and add useful functionality for common tasks. In this blog post, I'm going to highlight some of my favorite tools that fastcore provides, rather than sharing what I learned about python. My goal is to pique your interest in this library, and hopefully motivate you to check out the documentation after you are done to learn more!\n", "\n", "# Why fastcore is interesting\n", "\n", "1. **Get exposed to ideas from other languages without leaving python:** I’ve always heard that it is beneficial to learn other languages in order to become a better programmer. From a pragmatic point of view, I’ve found it difficult to learn other languages because I could never use them at work. Fastcore extends python to include patterns found in languages as diverse as Julia, Ruby and Haskell. Now that I understand these tools I am motivated to learn other languages.\n", "2. **You get a new set of pragmatic tools**: fastcore includes utilities that will allow you to write more concise expressive code, and perhaps solve new problems.\n", "3. **Learn more about the Python programming language:** Because fastcore extends the python programming language, many advanced concepts are exposed during the process. For the motivated, this is a great way to see how many of the internals of python work. \n", " \n", "\n", "# A whirlwind tour through fastcore\n", "\n", "Here are some things you can do with fastcore that immediately caught my attention." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting git+git://github.com/fastai/fastcore@master\n", " Cloning git://github.com/fastai/fastcore (to revision master) to /private/var/folders/88/9mddczp92wg04x0ykmw940q00000gn/T/pip-req-build-fph52w2h\n", " Running command git clone -q git://github.com/fastai/fastcore /private/var/folders/88/9mddczp92wg04x0ykmw940q00000gn/T/pip-req-build-fph52w2h\n", "Requirement already satisfied, skipping upgrade: pip in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from fastcore==1.0.12) (20.1.1)\n", "Requirement already satisfied, skipping upgrade: packaging in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from fastcore==1.0.12) (20.4)\n", "Requirement already satisfied, skipping upgrade: six in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from packaging->fastcore==1.0.12) (1.15.0)\n", "Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from packaging->fastcore==1.0.12) (2.4.7)\n", "Building wheels for collected packages: fastcore\n", " Building wheel for fastcore (setup.py) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for fastcore: filename=fastcore-1.0.12-py3-none-any.whl size=40633 sha256=8275b4eaa351c9c867e9c1f83105bc6ec25f251caf1efcf247a8e9ecc0f16f76\n", " Stored in directory: /private/var/folders/88/9mddczp92wg04x0ykmw940q00000gn/T/pip-ephem-wheel-cache-x8fbj38b/wheels/9b/c8/80/1e03649fe9e96f2b8231a1f38d4e20389036c7cb204afd8563\n", "Successfully built fastcore\n", "Installing collected packages: fastcore\n", " Attempting uninstall: fastcore\n", " Found existing installation: fastcore 1.0.12\n", " Uninstalling fastcore-1.0.12:\n", " Successfully uninstalled fastcore-1.0.12\n", "Successfully installed fastcore-1.0.12\n", "Collecting git+git://github.com/fastai/nbdev@master\n", " Cloning git://github.com/fastai/nbdev (to revision master) to /private/var/folders/88/9mddczp92wg04x0ykmw940q00000gn/T/pip-req-build-whf785xv\n", " Running command git clone -q git://github.com/fastai/nbdev /private/var/folders/88/9mddczp92wg04x0ykmw940q00000gn/T/pip-req-build-whf785xv\n", "Requirement already satisfied, skipping upgrade: pip in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbdev==1.0.19) (20.1.1)\n", "Requirement already satisfied, skipping upgrade: packaging in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbdev==1.0.19) (20.4)\n", "Requirement already satisfied, skipping upgrade: fastcore>=1.0.10 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbdev==1.0.19) (1.0.12)\n", "Requirement already satisfied, skipping upgrade: nbformat>=4.4.0 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbdev==1.0.19) (5.0.7)\n", "Requirement already satisfied, skipping upgrade: nbconvert<6 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbdev==1.0.19) (5.6.1)\n", "Requirement already satisfied, skipping upgrade: pyyaml in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbdev==1.0.19) (5.3.1)\n", "Requirement already satisfied, skipping upgrade: jupyter_client in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbdev==1.0.19) (6.1.6)\n", "Requirement already satisfied, skipping upgrade: ipykernel in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbdev==1.0.19) (5.3.2)\n", "Requirement already satisfied, skipping upgrade: six in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from packaging->nbdev==1.0.19) (1.15.0)\n", "Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from packaging->nbdev==1.0.19) (2.4.7)\n", "Requirement already satisfied, skipping upgrade: ipython-genutils in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbformat>=4.4.0->nbdev==1.0.19) (0.2.0)\n", "Requirement already satisfied, skipping upgrade: jsonschema!=2.5.0,>=2.4 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbformat>=4.4.0->nbdev==1.0.19) (3.0.2)\n", "Requirement already satisfied, skipping upgrade: traitlets>=4.1 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbformat>=4.4.0->nbdev==1.0.19) (4.3.3)\n", "Requirement already satisfied, skipping upgrade: jupyter-core in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbformat>=4.4.0->nbdev==1.0.19) (4.6.3)\n", "Requirement already satisfied, skipping upgrade: pandocfilters>=1.4.1 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbconvert<6->nbdev==1.0.19) (1.4.2)\n", "Requirement already satisfied, skipping upgrade: entrypoints>=0.2.2 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbconvert<6->nbdev==1.0.19) (0.3)\n", "Requirement already satisfied, skipping upgrade: bleach in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbconvert<6->nbdev==1.0.19) (3.1.5)\n", "Requirement already satisfied, skipping upgrade: testpath in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbconvert<6->nbdev==1.0.19) (0.4.4)\n", "Requirement already satisfied, skipping upgrade: defusedxml in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbconvert<6->nbdev==1.0.19) (0.6.0)\n", "Requirement already satisfied, skipping upgrade: jinja2>=2.4 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbconvert<6->nbdev==1.0.19) (2.11.2)\n", "Requirement already satisfied, skipping upgrade: mistune<2,>=0.8.1 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbconvert<6->nbdev==1.0.19) (0.8.4)\n", "Requirement already satisfied, skipping upgrade: pygments in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from nbconvert<6->nbdev==1.0.19) (2.6.1)\n", "Requirement already satisfied, skipping upgrade: tornado>=4.1 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from jupyter_client->nbdev==1.0.19) (6.0.4)\n", "Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from jupyter_client->nbdev==1.0.19) (2.8.1)\n", "Requirement already satisfied, skipping upgrade: pyzmq>=13 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from jupyter_client->nbdev==1.0.19) (19.0.1)\n", "Requirement already satisfied, skipping upgrade: appnope; platform_system == \"Darwin\" in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from ipykernel->nbdev==1.0.19) (0.1.0)\n", "Requirement already satisfied, skipping upgrade: ipython>=5.0.0 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from ipykernel->nbdev==1.0.19) (7.16.1)\n", "Requirement already satisfied, skipping upgrade: setuptools in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev==1.0.19) (49.2.0.post20200714)\n", "Requirement already satisfied, skipping upgrade: attrs>=17.4.0 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev==1.0.19) (19.3.0)\n", "Requirement already satisfied, skipping upgrade: pyrsistent>=0.14.0 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev==1.0.19) (0.16.0)\n", "Requirement already satisfied, skipping upgrade: decorator in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from traitlets>=4.1->nbformat>=4.4.0->nbdev==1.0.19) (4.4.2)\n", "Requirement already satisfied, skipping upgrade: webencodings in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from bleach->nbconvert<6->nbdev==1.0.19) (0.5.1)\n", "Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from jinja2>=2.4->nbconvert<6->nbdev==1.0.19) (1.1.1)\n", "Requirement already satisfied, skipping upgrade: backcall in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from ipython>=5.0.0->ipykernel->nbdev==1.0.19) (0.2.0)\n", "Requirement already satisfied, skipping upgrade: jedi>=0.10 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from ipython>=5.0.0->ipykernel->nbdev==1.0.19) (0.17.1)\n", "Requirement already satisfied, skipping upgrade: pickleshare in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from ipython>=5.0.0->ipykernel->nbdev==1.0.19) (0.7.5)\n", "Requirement already satisfied, skipping upgrade: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from ipython>=5.0.0->ipykernel->nbdev==1.0.19) (3.0.5)\n", "Requirement already satisfied, skipping upgrade: pexpect; sys_platform != \"win32\" in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from ipython>=5.0.0->ipykernel->nbdev==1.0.19) (4.8.0)\n", "Requirement already satisfied, skipping upgrade: parso<0.8.0,>=0.7.0 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel->nbdev==1.0.19) (0.7.0)\n", "Requirement already satisfied, skipping upgrade: wcwidth in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel->nbdev==1.0.19) (0.2.5)\n", "Requirement already satisfied, skipping upgrade: ptyprocess>=0.5 in /Users/hamelsmu/anaconda3/lib/python3.8/site-packages (from pexpect; sys_platform != \"win32\"->ipython>=5.0.0->ipykernel->nbdev==1.0.19) (0.6.0)\n", "Building wheels for collected packages: nbdev\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Building wheel for nbdev (setup.py) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for nbdev: filename=nbdev-1.0.19-py3-none-any.whl size=51832 sha256=69a997325e38de84de9410724f59a24deeab6514bc4bea80bdfc90d12acfcff8\n", " Stored in directory: /private/var/folders/88/9mddczp92wg04x0ykmw940q00000gn/T/pip-ephem-wheel-cache-r2gv8q50/wheels/28/83/ee/f8ff336622886142e3bc00cfc9ba627f277f30f2ad3d7030f0\n", "Successfully built nbdev\n", "Installing collected packages: nbdev\n", " Attempting uninstall: nbdev\n", " Found existing installation: nbdev 1.0.19\n", " Uninstalling nbdev-1.0.19:\n", " Successfully uninstalled nbdev-1.0.19\n", "Successfully installed nbdev-1.0.19\n", "Collecting numpy\n", " Downloading numpy-1.19.2-cp38-cp38-macosx_10_9_x86_64.whl (15.3 MB)\n", "\u001b[K |████████████████████████████████| 15.3 MB 4.7 MB/s eta 0:00:01\n", "\u001b[?25hInstalling collected packages: numpy\n", " Attempting uninstall: numpy\n", " Found existing installation: numpy 1.18.5\n", " Uninstalling numpy-1.18.5:\n", " Successfully uninstalled numpy-1.18.5\n", "Successfully installed numpy-1.19.2\n" ] } ], "source": [ "#hide\n", "! pip install -U git+git://github.com/fastai/fastcore@master\n", "! pip install -U git+git://github.com/fastai/nbdev@master\n", "! pip install -U numpy\n", "from fastcore.foundation import *\n", "from fastcore.meta import *\n", "from fastcore.utils import *\n", "from fastcore.test import *\n", "from nbdev.showdoc import *\n", "from fastcore.dispatch import typedispatch\n", "from functools import partial\n", "import numpy as np\n", "import inspect" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Making **kwargs transparent\n", "\n", "Whenever I see a function that has the argument **kwargs, I cringe a little. This is because it means the API is obfuscated and I have to read the source code to figure out what valid parameters might be. Consider the below example:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def baz(a, b=2, c=3, d=4): return a + b + c\n", "\n", "def foo(c, a, **kwargs):\n", " return c + baz(a, **kwargs)\n", "\n", "inspect.signature(foo)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Without reading the source code, it might be hard for me to know that `foo` also accepts and additional parameters `b` and `d`. We can fix this with [`delegates`](https://fastcore.fast.ai/foundation.html#delegates):" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def baz(a, b=2, c=3, d=4): return a + b + c\n", "\n", "@delegates(baz) # this decorator will pass down keyword arguments from baz\n", "def foo(c, a, **kwargs):\n", " return c + baz(a, **kwargs)\n", "\n", "inspect.signature(foo)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can customize the behavior of this decorator. For example, you can have your cake and eat it too by passing down your arguments and also keeping `**kwargs`:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@delegates(baz, keep=True)\n", "def foo(c, a, **kwargs):\n", " return c + baz(a, **kwargs)\n", "\n", "inspect.signature(foo)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also exclude arguments. For example, we exclude argument `d` from delegation:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def basefoo(a, b=2, c=3, d=4): pass\n", "\n", "@delegates(basefoo, but=['d']) # exclude `d`\n", "def foo(c, a, **kwargs): pass\n", "\n", "inspect.signature(foo)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also delegate between classes:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class BaseFoo:\n", " def __init__(self, e, c=2): pass\n", "\n", "@delegates()# since no argument was passsed here we delegate to the superclass\n", "class Foo(BaseFoo):\n", " def __init__(self, a, b=1, **kwargs): super().__init__(**kwargs)\n", " \n", "inspect.signature(Foo)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more information, read the [docs on delegates](https://fastcore.fast.ai/foundation.html#delegates).\n", "\n", "___" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Avoid boilerplate when setting instance attributes\n", "\n", "Have you ever wondered if it was possible to avoid the boilerplate involved with setting attributes in `__init__`?" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "class Test:\n", " def __init__(self, a, b ,c): \n", " self.a, self.b, self.c = a, b, c" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ouch! That was painful. Look at all the repeated variable names. Do I really have to repeat myself like this when defining a class? Not Anymore! Checkout [store_attr](https://fastcore.fast.ai/basics.html#store_attr):" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "class Test:\n", " def __init__(self, a, b, c): \n", " store_attr()\n", " \n", "t = Test(5,4,3)\n", "assert t.b == 4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also exclude certain attributes:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "class Test:\n", " def __init__(self, a, b, c): \n", " store_attr(but=['c'])\n", " \n", "t = Test(5,4,3)\n", "assert t.b == 4\n", "assert not hasattr(t, 'c')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are many more ways of customizing and using `store_attr` than I highlighted here. Check out [the docs](https://fastcore.fast.ai/basics.html#store_attr) for more detail.\n", "\n", "P.S. you might be thinking that Python [dataclasses](https://docs.python.org/3/library/dataclasses.html) also allow you to avoid this boilerplate. While true in some cases, `store_attr` is more flexible.{% fn 1 %}\n", "\n", "{{ \"For example, store_attr does not rely on inheritance, which means you won't get stuck using multiple inheritance when using this with your own classes. Also, unlike dataclasses, store_attr does not require python 3.7 or higher. Furthermore, you can use store_attr anytime in the object lifecycle, and in any location in your class to customize the behavior of how and when variables are stored.\" | fndetail: 1 }}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Avoiding subclassing boilerplate\n", "\n", "One thing I hate about python is the `__super__().__init__()` boilerplate associated with subclassing. For example:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "class ParentClass:\n", " def __init__(self): self.some_attr = 'hello'\n", " \n", "class ChildClass(ParentClass):\n", " def __init__(self):\n", " super().__init__()\n", "\n", "cc = ChildClass()\n", "assert cc.some_attr == 'hello' # only accessible b/c you used super" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can avoid this boilerplate by using the metaclass [PrePostInitMeta](https://fastcore.fast.ai/foundation.html#PrePostInitMeta). We define a new class called `NewParent` that is a wrapper around the `ParentClass`:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "class NewParent(ParentClass, metaclass=PrePostInitMeta):\n", " def __pre_init__(self, *args, **kwargs): super().__init__()\n", "\n", "class ChildClass(NewParent):\n", " def __init__(self):pass\n", " \n", "sc = ChildClass()\n", "assert sc.some_attr == 'hello' " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Type Dispatch\n", "\n", "Type dispatch, or [Multiple dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch#Julia), allows you to change the way a function behaves based upon the input types it receives. This is a prominent feature in some programming languages like Julia. For example, this is a [conceptual example](https://en.wikipedia.org/wiki/Multiple_dispatch#Julia) of how multiple dispatch works in Julia, returning different values depending on the input types of x and y:\n", "\n", "```julia\n", "collide_with(x::Asteroid, y::Asteroid) = ... \n", "# deal with asteroid hitting asteroid\n", "\n", "collide_with(x::Asteroid, y::Spaceship) = ... \n", "# deal with asteroid hitting spaceship\n", "\n", "collide_with(x::Spaceship, y::Asteroid) = ... \n", "# deal with spaceship hitting asteroid\n", "\n", "collide_with(x::Spaceship, y::Spaceship) = ... \n", "# deal with spaceship hitting spaceship\n", "```\n", "\n", "Type dispatch can be especially useful in data science, where you might allow different input types (i.e. Numpy arrays and Pandas dataframes) to a function that processes data. Type dispatch allows you to have a common API for functions that do similar tasks.\n", "\n", "Unfortunately, Python does not support this out-of-the box. Fortunately, there is the [@typedispatch](https://fastcore.fast.ai/dispatch.html#typedispatch-Decorator) decorator to the rescue. This decorator relies upon type hints in order to route inputs the correct version of the function:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "@typedispatch\n", "def f(x:str, y:str): return f'{x}{y}'\n", "\n", "@typedispatch\n", "def f(x:np.ndarray): return x.sum()\n", "\n", "@typedispatch\n", "def f(x:int, y:int): return x+y" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Below is a demonstration of type dispatch at work for the function `f`:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Hello World!'" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "f('Hello ', 'World!')" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "5" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "f(2,3)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "20" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "f(np.array([5,5,5,5]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are limitations of this feature, as well as other ways of using this functionality that [you can read about here](https://fastcore.fast.ai/dispatch.html). In the process of learning about typed dispatch, I also found a python library called [multipledispatch](https://github.com/mrocklin/multipledispatch) made by [Mathhew Rocklin](https://github.com/mrocklin) (the creator of Dask). \n", "\n", "After using this feature, I am now motivated to learn languages like Julia to discover what other paradigms I might be missing.\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A better version of functools.partial\n", "\n", "`functools.partial` is a great utility that creates functions from other functions that lets you set default values. Lets take this function for example that filters a list to only contain values >= `val`:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[3, 4, 5, 6]" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_input = [1,2,3,4,5,6]\n", "def f(arr, val): \n", " \"Filter a list to remove any values that are less than val.\"\n", " return [x for x in arr if x >= val]\n", "\n", "f(test_input, 3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can create a new function out of this function using `partial` that sets the default value to 5:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[5, 6]" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "filter5 = partial(f, val=5)\n", "filter5(test_input)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One problem with `partial` is that it removes the original docstring and replaces it with a generic docstring:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'partial(func, *args, **keywords) - new function with partial application\\n of the given arguments and keywords.\\n'" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "filter5.__doc__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[fastcore.utils.partialler](https://fastcore.fast.ai/basics.html#partialler) fixes this, and makes sure the docstring is retained such that the new API is transparent:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Filter a list to remove any values that are less than val.'" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "filter5 = partialler(f, val=5)\n", "filter5.__doc__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Composition of functions\n", "\n", "A technique that is pervasive in functional programming languages is function composition, whereby you chain a bunch of functions together to achieve some kind of result. This is especially useful when applying various data transformations. Consider a toy example where I have three functions: (1) Removes elements of a list less than 5 (from the prior section) (2) adds 2 to each number (3) sums all the numbers:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "15" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def add(arr, val): return [x + val for x in arr]\n", "def arrsum(arr): return sum(arr)\n", "\n", "# See the previous section on partialler\n", "add2 = partialler(add, val=2)\n", "\n", "transform = compose(filter5, add2, arrsum)\n", "transform([1,2,3,4,5,6])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But why is this useful? You might me thinking, I can accomplish the same thing with:\n", "\n", "```py\n", "arrsum(add2(filter5([1,2,3,4,5,6])))\n", "```\n", "You are not wrong! However, composition gives you a convenient interface in case you want to do something like the following:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[7.5, 7.5]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def fit(x, transforms:list):\n", " \"fit a model after performing transformations\"\n", " x = compose(*transforms)(x)\n", " y = [np.mean(x)] * len(x) # its a dumb model. Don't judge me\n", " return y\n", "\n", "# filters out elements < 5, adds 2, then predicts the mean\n", "fit(x=[1,2,3,4,5,6], transforms=[filter5, add2])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more information about `compose`, read [the docs](https://fastcore.fast.ai/basics.html#compose).\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A more useful __repr__\n", "\n", "In python, `__repr__` helps you get information about an object for logging and debugging. Below is what you get by default when you define a new class. (Note: we are using `store_attr`, which was discussed earlier)." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "<__main__.Test at 0x7ffcd766cee0>" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class Test:\n", " def __init__(self, a, b=2, c=3): store_attr() # `store_attr` was discussed previously\n", " \n", "Test(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can use [basic_repr](https://fastcore.fast.ai/basics.html#basic_repr) to quickly give us a more sensible default:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Test(a=2, b=2, c=3)" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class Test:\n", " def __init__(self, a, b=2, c=3): store_attr() \n", " __repr__ = basic_repr('a,b,c')\n", " \n", "Test(2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Monkey Patching With A Decorator\n", "\n", "It can be convenient to [monkey patch](https://www.geeksforgeeks.org/monkey-patching-in-python-dynamic-behavior/) with a decorator, which is especially helpful when you want to patch an external library you are importing. We can use the [decorator @patch](https://fastcore.fast.ai/foundation.html#patch) from `fastcore.foundation` along with type hints like so:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "class MyClass(int): pass \n", "\n", "@patch\n", "def func(self:MyClass, a): return self+a\n", "\n", "mc = MyClass(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, `MyClass` has an additional method named `func`:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "13" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mc.func(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Still not convinced? I'll show you another example of this kind of patching in the next section." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## A better pathlib.Path\n", "\n", "When you see [these extensions](https://fastcore.fast.ai/xtras.html#Extensions-to-Pathlib.Path) to pathlib.path you won't ever use vanilla pathlib again! A number of additional methods have been added to pathlib, such as:\n", "\n", "- `Path.readlines`: same as `with open('somefile', 'r') as f: f.readlines()`\n", "- `Path.read`: same as `with open('somefile', 'r') as f: f.read()`\n", "- `Path.save`: saves file as pickle\n", "- `Path.load`: loads pickle file\n", "- `Path.ls`: shows the contents of the path as a list. \n", "- etc.\n", "\n", "[Read more about this here](https://fastcore.fast.ai/xtras.html#Extensions-to-Pathlib.Path). Here is a demonstration of `ls`: " ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(#7) [Path('2020-09-01-fastcore.ipynb'),Path('README.md'),Path('fastcore_imgs'),Path('2020-02-20-test.ipynb'),Path('.ipynb_checkpoints'),Path('2020-02-21-introducing-fastpages.ipynb'),Path('my_icons')]" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from fastcore.utils import *\n", "from pathlib import Path\n", "p = Path('.')\n", "p.ls() # you don't get this with vanilla Pathlib.Path!!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Wait! What's going on here? We just imported `pathlib.Path` - why are we getting this new functionality? Thats because we imported the `fastcore.utils` module, which patches this module via the `@patch` decorator discussed earlier. Just to drive the point home on why the `@patch` decorator is useful, I'll go ahead and add another method to `Path` right now:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'This is fun!'" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@patch\n", "def fun(self:Path): return \"This is fun!\"\n", "\n", "p.fun()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That is magical, right? I know! That's why I'm writing about it!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## An Even More Concise Way To Create Lambdas\n", "\n", "\n", "`Self`, with an uppercase S, is an even more concise way to create lambdas that are calling methods on an object. For example, let's create a lambda for taking the sum of a Numpy array:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "arr=np.array([5,4,3,2,1])\n", "f = lambda a: a.sum()\n", "assert f(arr) == 15" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can use `Self` in the same way:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "f = Self.sum()\n", "assert f(arr) == 15" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's create a lambda that does a groupby and max of a Pandas dataframe:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Another Column
Some Column
a6
b60
\n", "
" ], "text/plain": [ " Another Column\n", "Some Column \n", "a 6\n", "b 60" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "df=pd.DataFrame({'Some Column': ['a', 'a', 'b', 'b', ], \n", " 'Another Column': [5, 7, 50, 70]})\n", "\n", "f = Self.groupby('Some Column').mean()\n", "f(df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Read more about `Self` in [the docs](https://fastcore.fast.ai/basics.html#Self-(with-an-uppercase-S))." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Notebook Functions\n", "\n", "These are simple but handy, and allow you to know whether or not code is executing in a Jupyter Notebook, Colab, or an Ipython Shell:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(True, False, True)" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from fastcore.imports import in_notebook, in_colab, in_ipython\n", "in_notebook(), in_colab(), in_ipython()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is useful if you are displaying certain types of visualizations, progress bars or animations in your code that you may want to modify or toggle depending on the environment." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## A Drop-In Replacement For List\n", "\n", "You might be pretty happy with Python's `list`. This is one of those situations that you don't know you needed a better list until someone showed one to you. Enter `L`, a list like object with many extra goodies. \n", "\n", "The best way I can describe `L` is to pretend that `list` and `numpy` had a pretty baby:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "define a list (check out the nice `__repr__` that shows the length of the list!)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(#3) [1,2,3]" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "L(1,2,3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Shuffle a list:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(#20) [8,7,5,12,14,16,2,15,19,6...]" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "p = L.range(20).shuffle()\n", "p" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Index into a list:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(#3) [5,14,2]" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "p[2,4,6]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "L has sensible defaults, for example appending an element to a list:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(#4) [1,2,3,4]" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "1 + L(2,3,4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is much more `L` has to offer. Read [the docs](https://fastcore.fast.ai/foundation.html#Class-L-Methods) to learn more." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# But Wait ... There's More!\n", "\n", "![](fastcore_imgs/bwtm.jpg)\n", "\n", "There are more things I would like to show you about fastcore, but there is no way they would reasonably fit into a blog post. Here is a list of some of my favorite things that I didn't demo in this blog post:\n", "\n", "## Utilities\n", "\n", "The [Basics](https://fastcore.fast.ai/basics.html) section contain many shortcuts to perform common tasks or provide an additional interface to what standard python provides. \n", "\n", "- [mk_class](https://fastcore.fast.ai/basics.html#mk_class): quickly add a bunch of attributes to a class\n", "- [wrap_class](https://fastcore.fast.ai/basics.html#wrap_class): add new methods to a class with a simple decorator\n", "- [groupby](https://fastcore.fast.ai/basics.html#groupby): similar to Scala's groupby\n", "- [merge](https://fastcore.fast.ai/basics.html#merge): merge dicts\n", "- [fasttuple](https://fastcore.fast.ai/basics.html#fastuple): a tuple on steroids\n", "- [Infinite Lists](https://fastcore.fast.ai/basics.html#Infinite-Lists): useful for padding and testing\n", "- [chunked](https://fastcore.fast.ai/basics.html#chunked): for batching and organizing stuff\n", "\n", "## Multiprocessing\n", "\n", "The [Multiprocessing section](http://fastcore.fast.ai/xtras.html#Multiprocessing) extends python's multiprocessing library by offering features like:\n", "\n", "- progress bars\n", "- ability to pause to mitigate race conditions with external services\n", "- processing things in batches on each worker, ex: if you have a vectorized operation to perform in chunks\n", "\n", "## Functional Programming\n", "\n", "The [functional programming section](https://fastcore.fast.ai/basics.html#Functions-on-Functions) is my favorite part of this library. \n", "\n", "- [maps](https://fastcore.fast.ai/basics.html#maps): a map that also composes functions\n", "- [mapped](https://fastcore.fast.ai/xtras.html#mapped): A more robust `map` \n", "- [using_attr](https://fastcore.fast.ai/basics.html#using_attr): compose a function that operates on an attribute\n", "\n", "## Transforms\n", "\n", "[Transforms](https://fastcore.fast.ai/transform.html) is a collection of utilities for creating data transformations and associated pipelines. These transformation utilities build upon many of the building blocks discussed in this blog post.\n", "\n", "\n", "## Further Reading\n", "\n", "**It should be noted that you should read the [main page of the docs](https://fastcore.fast.ai/) first, followed by the section on [tests](https://fastcore.fast.ai/) to fully understand the documentation.**\n", "\n", "- The [fastcore documentation site](https://fastcore.fast.ai/).\n", "- The [fastcore GitHub repo](https://github.com/fastai/fastcore).\n", "- Blog post on [delegation](https://www.fast.ai/2019/08/06/delegation/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Shameless plug: fastpages\n", "\n", "This blog post was written entirely in a Jupyter Notebook, which GitHub automatically converted into to a blog post! Sound interesting? [Check out fastpages](https://github.com/fastai/fastpages)." ] } ], "metadata": { "kernelspec": { "display_name": "fastai", "language": "python", "name": "fastai" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }