{ "metadata": { "name": "MetaClasses" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "A Primer on Python Metaclasses" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*This notebook was originally a blog post at*\n", "[*Pythonic Perambulations*](http://jakevdp.github.com/blog/2012/12/01/a-primer-on-python-metaclasses/)\n", "*by Jake Vanderplas*" ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Python, Classes, and Objects" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Most readers are aware that Python is an object-oriented language. By\n", "object-oriented, we mean that Python can define *classes*, which bundle\n", "**data** and **functionality** into one entity. For example, we may\n", "create a class ``IntContainer`` which stores an integer and allows\n", "certain operations to be performed:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class IntContainer(object):\n", " def __init__(self, i):\n", " self.i = int(i)\n", "\n", " def add_one(self):\n", " self.i += 1" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 1 }, { "cell_type": "code", "collapsed": false, "input": [ "ic = IntContainer(2)\n", "ic.add_one()\n", "print(ic.i)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "3\n" ] } ], "prompt_number": 2 }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is a bit of a silly example, but shows the fundamental nature of\n", "classes: their ability to bundle data and operations into a single\n", "*object*, which leads to cleaner, more manageable, and more adaptable code.\n", "Additionally, classes can inherit properties from parents and add or\n", "specialize attributes and methods. This *object-oriented*\n", "approach to programming can be very intuitive and powerful.\n", "\n", "What many do not realize, though, is that quite literally\n", "[*everything*](http://www.diveintopython.net/getting_to_know_python/everything_is_an_object.html)\n", "in the Python language is an object.\n", "\n", "For example, integers are simply instances of\n", "the built-in ``int`` type:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print type(1)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n" ] } ], "prompt_number": 3 }, { "cell_type": "markdown", "metadata": {}, "source": [ "To emphasize that the ``int`` type really is an object, let's derive from it\n", "and specialize the ``__add__`` method (which is the machinery underneath\n", "the ``+`` operator):\n", "\n", "*(Note: We'll used the* ``super`` *syntax to call methods from the parent class: if you're unfamiliar with this, take a look at*\n", "[*this StackOverflow question*](http://stackoverflow.com/questions/576169/understanding-python-super-and-init-methods)*).*" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class MyInt(int):\n", " def __add__(self, other):\n", " print \"specializing addition\"\n", " return super(MyInt, self).__add__(other)\n", "\n", "i = MyInt(2)\n", "print(i + 2)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "specializing addition\n", "4\n" ] } ], "prompt_number": 4 }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Using the ``+`` operator on our derived type goes through our ``__add__``\n", "method, as expected.\n", "We see that ``int`` really is an object that can be subclassed and extended\n", "just like user-defined classes. The same is true\n", "of ``float``s, ``list``s, ``tuple``s, and everything else in the Python\n", "language. They're all objects." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Down the Rabbit Hole: Classes as Objects" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We said above that *everything* in python is an object: it turns out that this\n", "is true of *classes themselves*. Let's look at an example.\n", "\n", "We'll start by defining a class that does nothing" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class DoNothing(object):\n", " pass" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we instantiate this, we can use the ``type`` operator to see the type\n", "of object that it is:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "d = DoNothing()\n", "type(d)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 6, "text": [ "__main__.DoNothing" ] } ], "prompt_number": 6 }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that our variable ``d`` is an instance of the class\n", "``__main__.DoNothing``.\n", "\n", "We can do this similarly for built-in types:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "L = [1, 2, 3]\n", "type(L)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 7, "text": [ "list" ] } ], "prompt_number": 7 }, { "cell_type": "markdown", "metadata": {}, "source": [ "A list is, as you may expect, an object of type ``list``.\n", "\n", "But let's take this a step further: what is the type\n", "of ``DoNothing`` itself?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "type(DoNothing)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 8, "text": [ "type" ] } ], "prompt_number": 8 }, { "cell_type": "markdown", "metadata": {}, "source": [ "The type of ``DoNothing`` is ``type``. This tells us that the *class*\n", "``DoNothing`` is itself an object, and that object is of type ``type``.\n", "\n", "It turns out that this is the same for built-in datatypes:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "type(tuple), type(list), type(int), type(float)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 9, "text": [ "(type, type, type, type)" ] } ], "prompt_number": 9 }, { "cell_type": "markdown", "metadata": {}, "source": [ "What this shows is that in Python, *classes are objects*, and they are objects of\n", "type ``type``. ``type`` is a *metaclass*: a class which instantiates classes.\n", "All [new-style classes](http://www.python.org/doc/newstyle/)\n", "in Python are instances of the ``type`` metaclass, including ``type`` itself:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "type(type)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 10, "text": [ "type" ] } ], "prompt_number": 10 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Yes, you read that correctly:\n", "the type of ``type`` is ``type``. In other words, ``type`` is *an\n", "instance of itself*. This sort of circularity cannot (to my knowledge)\n", "be duplicated in pure Python, and the behavior is created through a bit of a\n", "hack at the implementation level of Python." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Metaprogramming: Creating Classes on the Fly" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we've stepped back and considered the fact that classes in Python\n", "are simply objects like everything else, we can think about what is known\n", "as *metaprogramming*. You're probably used to creating functions which\n", "return objects. We can think of these functions as an object factory: they\n", "take some arguments, create an object, and return it. Here is a simple example\n", "of a function which creates an ``int`` object:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def int_factory(s):\n", " i = int(s)\n", " return i\n", "\n", "i = int_factory('100')\n", "print(i)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "100\n" ] } ], "prompt_number": 11 }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is overly-simplistic, but any function you write in the course\n", "of a normal program can be boiled down to this: take some arguments,\n", "do some operations, and create & return an object.\n", "With the above discussion in mind, though, there's nothing to stop\n", "us from creating an object of type ``type`` (that is, a class), \n", "and returning that instead -- this is a *metafunction:*" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def class_factory():\n", " class Foo(object):\n", " pass\n", " return Foo\n", "\n", "F = class_factory()\n", "f = F()\n", "print(type(f))" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n" ] } ], "prompt_number": 12 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Just as the function ``int_factory`` constructs an returns an instance of\n", "``int``,\n", "the function ``class_factory`` constructs and returns an instance of ``type``:\n", "that is, a class.\n", "\n", "But the above construction is a bit awkward: especially if we were going to do some\n", "more complicated logic when constructing ``Foo``, it would be nice to avoid all the\n", "nested indentations and define the class in a more dynamic way.\n", "We can accomplish this by instantiating ``Foo`` from ``type`` directly:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def class_factory():\n", " return type('Foo', (), {})\n", "\n", "F = class_factory()\n", "f = F()\n", "print(type(f))" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n" ] } ], "prompt_number": 13 }, { "cell_type": "markdown", "metadata": {}, "source": [ "In fact, the construct" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class MyClass(object):\n", " pass" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 14 }, { "cell_type": "markdown", "metadata": {}, "source": [ "is identical to the construct" ] }, { "cell_type": "code", "collapsed": false, "input": [ "MyClass = type('MyClass', (), {})" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 15 }, { "cell_type": "markdown", "metadata": {}, "source": [ "``MyClass`` is an instance of type ``type``, and that can be seen\n", "explicitly in the second version of the definition.\n", "A potential confusion arises from the more common use of ``type`` as\n", "a function to determine the type of an object, but you should strive\n", "to separate these two uses of the keyword in your mind:\n", "here ``type`` is a class (more precisely, a *metaclass*),\n", "and ``MyClass`` is an instance of ``type``.\n", "\n", "The arguments to the ``type`` constructor are:" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "type(name, bases, dct)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- ``name`` is a string giving the name of the class to be constructed\n", "- ``bases`` is a tuple giving the parent classes of the class to be constructed\n", "- ``dct`` is a dictionary of the attributes and methods of the class to be constructed\n", "\n", "So, for example, the following two pieces of code have identical results:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class Foo(object):\n", " i = 4\n", "\n", "class Bar(Foo):\n", " def get_i(self):\n", " return self.i\n", " \n", "b = Bar()\n", "print(b.get_i())" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "4\n" ] } ], "prompt_number": 16 }, { "cell_type": "code", "collapsed": false, "input": [ "Foo = type('Foo', (), dict(i=4))\n", "\n", "Bar = type('Bar', (Foo,), dict(get_i = lambda self: self.i))\n", "\n", "b = Bar()\n", "print(b.get_i())" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "4\n" ] } ], "prompt_number": 17 }, { "cell_type": "markdown", "metadata": {}, "source": [ "This perhaps seems a bit over-complicated in the case of this contrived\n", "example, but it can be very powerful as a means of dynamically creating\n", "new classes on-the-fly." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Making Things Interesting: Custom Metaclasses" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now things get really fun. Just as we can inherit from and extend a class we've\n", "created, we can also inherit from and extend the ``type`` metaclass, and create\n", "custom behavior in our metaclass." ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Example 1: Modifying Attributes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's use a simple example where we want to create an API in which the user can\n", "create a set of interfaces which contain a file object. Each interface should\n", "have a unique string ID, and contain an open file object. The user could then write\n", "specialized methods to accomplish certain tasks. There are certainly good\n", "ways to do this without delving into metaclasses, but such a simple example will\n", "(hopefully) elucidate what's going on.\n", "\n", "First we'll create our interface meta class, deriving from ``type``:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class InterfaceMeta(type):\n", " def __new__(cls, name, parents, dct):\n", " # create a class_id if it's not specified\n", " if 'class_id' not in dct:\n", " dct['class_id'] = name.lower()\n", " \n", " # open the specified file for writing\n", " if 'file' in dct:\n", " filename = dct['file']\n", " dct['file'] = open(filename, 'w')\n", " \n", " # we need to call type.__new__ to complete the initialization\n", " return super(InterfaceMeta, cls).__new__(cls, name, parents, dct)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 18 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that we've modified the input dictionary (the attributes and\n", "methods of the class) to add a class id if it's not present, and to\n", "replace the filename with a file object pointing to that file name.\n", "\n", "Now we'll use our ``InterfaceMeta`` class to construct and instantiate\n", "an Interface object:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "Interface = InterfaceMeta('Interface', (), dict(file='tmp.txt'))\n", "\n", "print(Interface.class_id)\n", "print(Interface.file)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "interface\n", "\n" ] } ], "prompt_number": 19 }, { "cell_type": "markdown", "metadata": {}, "source": [ "This behaves as we'd expect: the ``class_id`` class variable is created,\n", "and the ``file`` class variable is replaced with an open file object.\n", "Still, the creation of the ``Interface`` class\n", "using ``InterfaceMeta`` directly is a bit clunky and difficult to read.\n", "This is where ``__metaclass__`` comes in\n", "and steals the show. We can accomplish the same thing by\n", "defining ``Interface`` this way:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class Interface(object):\n", " __metaclass__ = InterfaceMeta\n", " file = 'tmp.txt'\n", " \n", "print(Interface.class_id)\n", "print(Interface.file)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "interface\n", "\n" ] } ], "prompt_number": 20 }, { "cell_type": "markdown", "metadata": {}, "source": [ "by defining the ``__metaclass__`` attribute of the class, we've told the\n", "class that it should be constructed using ``InterfaceMeta`` rather than\n", "using ``type``. To make this more definite, observe that the type of\n", "``Interface`` is now ``InterfaceMeta``:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "type(Interface)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 21, "text": [ "__main__.InterfaceMeta" ] } ], "prompt_number": 21 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Furthermore, any class derived from ``Interface`` will now be constructed\n", "using the same metaclass:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class UserInterface(Interface):\n", " file = 'foo.txt'\n", " \n", "print(UserInterface.file)\n", "print(UserInterface.class_id)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n", "userinterface\n" ] } ], "prompt_number": 22 }, { "cell_type": "markdown", "metadata": {}, "source": [ "This simple example shows how metaclasses can be used to create powerful and\n", "flexible APIs for projects. For example, the\n", "[Django project](https://www.djangoproject.com/)\n", "makes use of these sorts of constructions to allow concise declarations\n", "of very powerful extensions to their basic classes." ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Example 2: Registering Subclasses" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another possible use of a metaclass is to automatically register all\n", "subclasses derived from a given base class. For example, you may have\n", "a basic interface to a database and wish for the user to be able to\n", "define their own interfaces, which are automatically stored in a master registry.\n", "\n", "You might proceed this way:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class DBInterfaceMeta(type):\n", " # we use __init__ rather than __new__ here because we want\n", " # to modify attributes of the class *after* they have been\n", " # created\n", " def __init__(cls, name, bases, dct):\n", " if not hasattr(cls, 'registry'):\n", " # this is the base class. Create an empty registry\n", " cls.registry = {}\n", " else:\n", " # this is a derived class. Add cls to the registry\n", " interface_id = name.lower()\n", " cls.registry[interface_id] = cls\n", " \n", " super(DBInterfaceMeta, cls).__init__(name, bases, dct)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 23 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our metaclass simply adds a ``registry`` dictionary if it's not already\n", "present, and adds the new class to the registry if the registry is already\n", "there. Let's see how this works:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class DBInterface(object):\n", " __metaclass__ = DBInterfaceMeta\n", " \n", "print(DBInterface.registry)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "{}\n" ] } ], "prompt_number": 24 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's create some subclasses, and double-check that they're added to\n", "the registry:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class FirstInterface(DBInterface):\n", " pass\n", "\n", "class SecondInterface(DBInterface):\n", " pass\n", "\n", "class SecondInterfaceModified(SecondInterface):\n", " pass\n", "\n", "print(DBInterface.registry)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "{'firstinterface': , 'secondinterface': , 'secondinterfacemodified': }\n" ] } ], "prompt_number": 25 }, { "cell_type": "markdown", "metadata": {}, "source": [ "It works as expected! This could be used in conjunction with\n", "a function that chooses implementations from the registry,\n", "and any user-defined ``Interface``-derived objects would be\n", "automatically accounted for, without the user having to remember\n", "to manually register the new types." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Conclusion: When Should You Use Metaclasses?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I've gone through some examples of what metaclasses are, and some ideas \n", "about how they might be used to create very powerful and flexible APIs. \n", "Although metaclasses are in the background of everything you do in Python,\n", "the average coder rarely has to think about them.\n", "\n", "But the question remains: when should you think about using custom\n", "metaclasses in your project? It's a complicated question, but\n", "there's a quotation floating around the web\n", "that addresses it quite succinctly:\n", "\n", "> Metaclasses are deeper magic than 99% of users should ever worry about.\n", "> If you wonder whether you need them, you don\u2019t (the people who actually\n", "> need them know with certainty that they need them, and don\u2019t need an\n", "> explanation about why).\n", ">\n", "> \u2013 Tim Peters\n", "\n", "In a way, this is a very unsatisfying answer: it's a bit reminiscent of\n", "the wistful and cliched explanation of the border between attraction\n", "and love: \"well, you just... know!\"\n", "\n", "But I think Tim is right: in general,\n", "I've found that most tasks in Python that can be accomplished through\n", "use of custom metaclasses can also be accomplished more cleanly and with\n", "more clarity by other means. As programmers, we should always be careful\n", "to avoid being clever for the sake of cleverness alone, though\n", "it is admittedly an ever-present temptation.\n", "\n", "I personally spent six years doing science with Python, writing code\n", "nearly on a daily basis, before I found a problem for which metaclasses\n", "were the natural solution. And it turns out Tim was right:\n", "\n", "I just knew." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This post was written entirely in an IPython Notebook: the notebook file is available for\n", "download [here](http://jakevdp.github.com/downloads/notebooks/MetaClasses.ipynb).\n", "For more information on blogging with notebooks in octopress, see my\n", "[previous post](http://jakevdp.github.com/blog/2012/10/04/blogging-with-ipython/)\n", "on the subject." ] } ], "metadata": {} } ] }