{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "import import\n", "=============\n", "\n", "- dive into Python import statement\n", "-----------------------------------\n", "\n", "Konrad Ha\u0142as (@konradhalas)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Few words about me\n", "==================\n", "\n", "* PyWaw co-organizer\n", "* developer in SUNSCRAPERS\n", "* MutPy author" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Agenda\n", "======\n", "\n", "* presentation purpose\n", "* basic info\n", "* syntax history\n", "* inside machinery\n", "* best practices\n", "* common problems" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Presentation purpose\n", "====================\n", "\n", "* understand how import statement works\n", "* learn how to override standard import behavior\n", "* prepare to meet with \"magic\" import code in other projects\n", "* do it well and avoid problems" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "What is purpose of import statement?\n", "====================================\n", "\n", "* it allows to use code from other modules\n", "* it allows to split code into multiple modules" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "What does import statement do?\n", "==============================\n", "\n", "* find a module\n", "* initialize it if necessary\n", "* define a name or names in the local namespace " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Syntax history\n", "==============" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Python 1.0.0 - 1994\n", "===================\n", "\n" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "
\n",
      "    import_stmt:  \"import\" identifier (\",\" identifier)*\n",
      "                  | \"from\" identifier \"import\" identifier (\",\" identifier)*\n",
      "                  | \"from\" identifier \"import\" \"*\"\n",
      "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Python 1.5.0 - 1997\n", "===================" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "
    \n",
      "    import_stmt:  \"import\" module (\",\" module)* \n",
      "                  | \"from\" module \"import\" identifier (\",\" identifier)*\n",
      "                  | \"from\" module \"import\" \"*\" \n",
      "    module:       (identifier \".\")* identifier\n",
      "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Python 2.0.0 - 2000\n", "===================" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "
\n",
      "    import_stmt:  \"import\" module [\"as\" name] (\",\" module [\"as\" name] )* \n",
      "                  | \"from\" module \"import\" identifier [\"as\" name]\n",
      "                  (\",\" identifier [\"as\" name] )*\n",
      "                  | \"from\" module \"import\" \"*\" \n",
      "    module:       (identifier \".\")* identifier\n",
      "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Python 2.5.0 - 2006\n", "===================" ] }, { "cell_type": "raw", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "
\n",
      "    import_stmt:       \"import\" module [\"as\" name] ( \",\" module [\"as\" name] )*\n",
      "                       | \"from\" relative_module \"import\" identifier [\"as\" name]\n",
      "                       ( \",\" identifier [\"as\" name] )*\n",
      "                       | \"from\" relative_module \"import\" \"(\" identifier [\"as\" name]\n",
      "                       ( \",\" identifier [\"as\" name] )* [\",\"] \")\"\n",
      "                       | \"from\" module \"import\" \"*\"\n",
      "    module :           (identifier \".\")* identifier\n",
      "    relative_module :  \".\"* module | \".\"+\n",
      "    name :             identifier\n",
      "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "import statement machinery\n", "==========================\n", "\n", "* defined in **PEP 302** (\"New Import Hooks\")\n", "* introduced in **Python 2.3** (2003)\n", "* complete integration in **Python 3.3** (2012)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Let's start...\n", "==============" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Machinery structures\n", "====================\n", "\n", "1. `sys.modules`\n", "2. `sys.meta_path`\n", "3. `sys.path`\n", "4. `sys.path_hooks`\n", "5. `sys.path_importer_cache`" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`sys.modules`\n", "=============\n", "\n", "This is a dictionary that maps module names to modules which have already been loaded." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import sys\n", "\n", "sys.modules" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 1, "text": [ "{'IPython': ,\n", " 'IPython.config': ,\n", " 'IPython.config.application': ,\n", " 'IPython.config.configurable': ,\n", " 'IPython.config.loader': ,\n", " 'IPython.core': ,\n", " 'IPython.core.alias': ,\n", " 'IPython.core.application': ,\n", " 'IPython.core.autocall': ,\n", " 'IPython.core.builtin_trap': ,\n", " 'IPython.core.compilerop': ,\n", " 'IPython.core.completer': ,\n", " 'IPython.core.completerlib': ,\n", " 'IPython.core.crashhandler': ,\n", " 'IPython.core.debugger': ,\n", " 'IPython.core.display': ,\n", " 'IPython.core.display_trap': ,\n", " 'IPython.core.displayhook': ,\n", " 'IPython.core.displaypub': ,\n", " 'IPython.core.error': ,\n", " 'IPython.core.excolors': ,\n", " 'IPython.core.extensions': ,\n", " 'IPython.core.fakemodule': ,\n", " 'IPython.core.formatters': ,\n", " 'IPython.core.getipython': ,\n", " 'IPython.core.history': ,\n", " 'IPython.core.hooks': ,\n", " 'IPython.core.inputsplitter': ,\n", " 'IPython.core.inputtransformer': ,\n", " 'IPython.core.interactiveshell': ,\n", " 'IPython.core.logger': ,\n", " 'IPython.core.macro': ,\n", " 'IPython.core.magic': ,\n", " 'IPython.core.magic_arguments': ,\n", " 'IPython.core.magics': ,\n", " 'IPython.core.magics.auto': ,\n", " 'IPython.core.magics.basic': ,\n", " 'IPython.core.magics.code': ,\n", " 'IPython.core.magics.config': ,\n", " 'IPython.core.magics.deprecated': ,\n", " 'IPython.core.magics.display': ,\n", " 'IPython.core.magics.execution': ,\n", " 'IPython.core.magics.extension': ,\n", " 'IPython.core.magics.history': ,\n", " 'IPython.core.magics.logging': ,\n", " 'IPython.core.magics.namespace': ,\n", " 'IPython.core.magics.osm': ,\n", " 'IPython.core.magics.pylab': ,\n", " 'IPython.core.magics.script': ,\n", " 'IPython.core.oinspect': ,\n", " 'IPython.core.page': ,\n", " 'IPython.core.payload': ,\n", " 'IPython.core.payloadpage': ,\n", " 'IPython.core.prefilter': ,\n", " 'IPython.core.profiledir': ,\n", " 'IPython.core.prompts': ,\n", " 'IPython.core.pylabtools': ,\n", " 'IPython.core.release': ,\n", " 'IPython.core.shadowns': ,\n", " 'IPython.core.shellapp': ,\n", " 'IPython.core.splitinput': ,\n", " 'IPython.core.ultratb': ,\n", " 'IPython.core.usage': ,\n", " 'IPython.display': ,\n", " 'IPython.extensions': ,\n", " 'IPython.extensions.storemagic': ,\n", " 'IPython.external': ,\n", " 'IPython.external.argparse': ,\n", " 'IPython.external.decorator': ,\n", " 'IPython.external.decorator._decorator': ,\n", " 'IPython.external.path': ,\n", " 'IPython.external.path._path': ,\n", " 'IPython.external.pexpect': ,\n", " 'IPython.external.pexpect._pexpect': ,\n", " 'IPython.external.simplegeneric': ,\n", " 'IPython.external.simplegeneric._simplegeneric': ,\n", " 'IPython.external.ssh': ,\n", " 'IPython.external.ssh.tunnel': ,\n", " 'IPython.kernel': ,\n", " 'IPython.kernel.blocking': ,\n", " 'IPython.kernel.blocking.channels': ,\n", " 'IPython.kernel.blocking.client': ,\n", " 'IPython.kernel.channels': ,\n", " 'IPython.kernel.channelsabc': ,\n", " 'IPython.kernel.client': ,\n", " 'IPython.kernel.clientabc': ,\n", " 'IPython.kernel.connect': ,\n", " 'IPython.kernel.inprocess': ,\n", " 'IPython.kernel.inprocess.blocking': ,\n", " 'IPython.kernel.inprocess.channels': ,\n", " 'IPython.kernel.inprocess.client': ,\n", " 'IPython.kernel.inprocess.manager': ,\n", " 'IPython.kernel.inprocess.socket': ,\n", " 'IPython.kernel.launcher': ,\n", " 'IPython.kernel.manager': ,\n", " 'IPython.kernel.managerabc': ,\n", " 'IPython.kernel.multikernelmanager': ,\n", " 'IPython.kernel.zmq': ,\n", " 'IPython.kernel.zmq.datapub': ,\n", " 'IPython.kernel.zmq.displayhook': ,\n", " 'IPython.kernel.zmq.heartbeat': ,\n", " 'IPython.kernel.zmq.iostream': ,\n", " 'IPython.kernel.zmq.ipkernel': ,\n", " 'IPython.kernel.zmq.kernelapp': ,\n", " 'IPython.kernel.zmq.parentpoller': ,\n", " 'IPython.kernel.zmq.serialize': ,\n", " 'IPython.kernel.zmq.session': ,\n", " 'IPython.kernel.zmq.zmqshell': ,\n", " 'IPython.lib': ,\n", " 'IPython.lib.backgroundjobs': ,\n", " 'IPython.lib.deepreload': ,\n", " 'IPython.lib.display': ,\n", " 'IPython.lib.inputhook': ,\n", " 'IPython.lib.latextools': ,\n", " 'IPython.lib.pretty': ,\n", " 'IPython.lib.security': ,\n", " 'IPython.terminal': ,\n", " 'IPython.terminal.embed': ,\n", " 'IPython.terminal.interactiveshell': ,\n", " 'IPython.terminal.ipapp': ,\n", " 'IPython.testing': ,\n", " 'IPython.testing.skipdoctest': ,\n", " 'IPython.utils': ,\n", " 'IPython.utils.PyColorize': ,\n", " 'IPython.utils._process_common': ,\n", " 'IPython.utils._process_posix': ,\n", " 'IPython.utils._sysinfo': ,\n", " 'IPython.utils._tokenize_py3': ,\n", " 'IPython.utils.codeutil': ,\n", " 'IPython.utils.coloransi': ,\n", " 'IPython.utils.contexts': ,\n", " 'IPython.utils.data': ,\n", " 'IPython.utils.decorators': ,\n", " 'IPython.utils.dir2': ,\n", " 'IPython.utils.encoding': ,\n", " 'IPython.utils.frame': ,\n", " 'IPython.utils.generics': ,\n", " 'IPython.utils.importstring': ,\n", " 'IPython.utils.io': ,\n", " 'IPython.utils.ipstruct': ,\n", " 'IPython.utils.jsonutil': ,\n", " 'IPython.utils.localinterfaces': ,\n", " 'IPython.utils.module_paths': ,\n", " 'IPython.utils.nested_context': ,\n", " 'IPython.utils.openpy': ,\n", " 'IPython.utils.path': ,\n", " 'IPython.utils.pickleshare': ,\n", " 'IPython.utils.pickleutil': ,\n", " 'IPython.utils.process': ,\n", " 'IPython.utils.py3compat': ,\n", " 'IPython.utils.rlineimpl': ,\n", " 'IPython.utils.strdispatch': ,\n", " 'IPython.utils.sysinfo': ,\n", " 'IPython.utils.syspathcontext': ,\n", " 'IPython.utils.terminal': ,\n", " 'IPython.utils.text': ,\n", " 'IPython.utils.timing': ,\n", " 'IPython.utils.tokenize2': ,\n", " 'IPython.utils.traitlets': ,\n", " 'IPython.utils.ulinecache': ,\n", " 'IPython.utils.version': ,\n", " 'IPython.utils.warn': ,\n", " 'IPython.utils.wildcard': ,\n", " 'IPython.utils.zmqrelated': ,\n", " '__future__': ,\n", " '__main__': ,\n", " '_ast': ,\n", " '_bisect': ,\n", " '_bz2': ,\n", " '_codecs': ,\n", " '_collections': ,\n", " '_compat_pickle': ,\n", " '_ctypes': ,\n", " '_curses': ,\n", " '_datetime': ,\n", " '_frozen_importlib': ,\n", " '_functools': ,\n", " '_hashlib': ,\n", " '_heapq': ,\n", " '_imp': ,\n", " '_io': ,\n", " '_json': ,\n", " '_locale': ,\n", " '_lsprof': ,\n", " '_multiprocessing': ,\n", " '_osx_support': ,\n", " '_pickle': ,\n", " '_posixsubprocess': ,\n", " '_random': ,\n", " '_socket': ,\n", " '_sqlite3': ,\n", " '_sre': ,\n", " '_string': ,\n", " '_strptime': ,\n", " '_struct': ,\n", " '_thread': ,\n", " '_virtualenv_distutils': ,\n", " '_warnings': ,\n", " '_weakref': ,\n", " '_weakrefset': ,\n", " 'abc': ,\n", " 'argparse': ,\n", " 'array': ,\n", " 'ast': ,\n", " 'atexit': ,\n", " 'base64': ,\n", " 'bdb': ,\n", " 'binascii': ,\n", " 'bisect': ,\n", " 'builtins': ,\n", " 'bz2': ,\n", " 'cProfile': ,\n", " 'calendar': ,\n", " 'cmd': ,\n", " 'code': ,\n", " 'codecs': ,\n", " 'codeop': ,\n", " 'collections': ,\n", " 'collections.abc': ,\n", " 'concurrent': ,\n", " 'concurrent.futures': ,\n", " 'concurrent.futures._base': ,\n", " 'concurrent.futures.process': ,\n", " 'concurrent.futures.thread': ,\n", " 'contextlib': ,\n", " 'copy': ,\n", " 'copyreg': ,\n", " 'ctypes': ,\n", " 'ctypes._endian': ,\n", " 'ctypes.macholib': ,\n", " 'ctypes.macholib.dyld': ,\n", " 'ctypes.macholib.dylib': ,\n", " 'ctypes.macholib.framework': ,\n", " 'ctypes.util': ,\n", " 'curses': ,\n", " 'datetime': ,\n", " 'dis': ,\n", " 'distutils': ,\n", " 'distutils.debug': ,\n", " 'distutils.dep_util': ,\n", " 'distutils.dist': ,\n", " 'distutils.errors': ,\n", " 'distutils.fancy_getopt': ,\n", " 'distutils.log': ,\n", " 'distutils.spawn': ,\n", " 'distutils.sysconfig': ,\n", " 'distutils.text_file': ,\n", " 'distutils.util': ,\n", " 'distutils.version': ,\n", " 'encodings': ,\n", " 'encodings.aliases': ,\n", " 'encodings.idna': ,\n", " 'encodings.latin_1': ,\n", " 'encodings.utf_8': ,\n", " 'errno': ,\n", " 'fcntl': ,\n", " 'fnmatch': ,\n", " 'functools': ,\n", " 'gc': ,\n", " 'genericpath': ,\n", " 'getopt': ,\n", " 'getpass': ,\n", " 'gettext': ,\n", " 'glob': ,\n", " 'grp': ,\n", " 'hashlib': ,\n", " 'heapq': ,\n", " 'hmac': ,\n", " 'html': ,\n", " 'html.entities': ,\n", " 'imp': ,\n", " 'importlib': ,\n", " 'importlib._bootstrap': ,\n", " 'importlib.machinery': ,\n", " 'inspect': ,\n", " 'io': ,\n", " 'itertools': ,\n", " 'json': ,\n", " 'json.decoder': ,\n", " 'json.encoder': ,\n", " 'json.scanner': ,\n", " 'keyword': ,\n", " 'linecache': ,\n", " 'locale': ,\n", " 'logging': ,\n", " 'logging.handlers': ,\n", " 'marshal': ,\n", " 'math': ,\n", " 'multiprocessing': ,\n", " 'multiprocessing.connection': ,\n", " 'multiprocessing.forking': ,\n", " 'multiprocessing.process': ,\n", " 'multiprocessing.queues': ,\n", " 'multiprocessing.reduction': ,\n", " 'multiprocessing.synchronize': ,\n", " 'multiprocessing.util': ,\n", " 'numbers': ,\n", " 'opcode': ,\n", " 'operator': ,\n", " 'os': ,\n", " 'os.path': ,\n", " 'pdb': ,\n", " 'pickle': ,\n", " 'pkgutil': ,\n", " 'platform': ,\n", " 'posix': ,\n", " 'posixpath': ,\n", " 'pprint': ,\n", " 'pstats': ,\n", " 'pty': ,\n", " 'pwd': ,\n", " 'pydoc': ,\n", " 'queue': ,\n", " 'random': ,\n", " 're': ,\n", " 'readline': ,\n", " 'reprlib': ,\n", " 'resource': ,\n", " 'runpy': ,\n", " 'select': ,\n", " 'shlex': ,\n", " 'shutil': ,\n", " 'signal': ,\n", " 'site': ,\n", " 'socket': ,\n", " 'sqlite3': ,\n", " 'sqlite3.dbapi2': ,\n", " 'sre_compile': ,\n", " 'sre_constants': ,\n", " 'sre_parse': ,\n", " 'stat': ,\n", " 'storemagic': ,\n", " 'string': ,\n", " 'stringprep': ,\n", " 'struct': ,\n", " 'subprocess': ,\n", " 'sys': ,\n", " 'tarfile': ,\n", " 'tempfile': ,\n", " 'termios': ,\n", " 'textwrap': ,\n", " 'threading': ,\n", " 'time': ,\n", " 'token': ,\n", " 'tokenize': ,\n", " 'tornado': ,\n", " 'tornado.concurrent': ,\n", " 'tornado.escape': ,\n", " 'tornado.ioloop': ,\n", " 'tornado.log': ,\n", " 'tornado.platform': ,\n", " 'tornado.platform.auto': ,\n", " 'tornado.platform.interface': ,\n", " 'tornado.platform.posix': ,\n", " 'tornado.stack_context': ,\n", " 'tornado.util': ,\n", " 'traceback': ,\n", " 'tty': ,\n", " 'types': ,\n", " 'unicodedata': ,\n", " 'urllib': ,\n", " 'urllib.parse': ,\n", " 'uuid': ,\n", " 'warnings': ,\n", " 'weakref': ,\n", " 'zipimport': ,\n", " 'zlib': ,\n", " 'zmq': ,\n", " 'zmq.core': ,\n", " 'zmq.core._device': ,\n", " 'zmq.core._poll': ,\n", " 'zmq.core._version': ,\n", " 'zmq.core.constants': ,\n", " 'zmq.core.context': ,\n", " 'zmq.core.error': ,\n", " 'zmq.core.message': ,\n", " 'zmq.core.socket': ,\n", " 'zmq.core.stopwatch': ,\n", " 'zmq.devices': ,\n", " 'zmq.devices.basedevice': ,\n", " 'zmq.devices.monitoredqueue': ,\n", " 'zmq.devices.monitoredqueuedevice': ,\n", " 'zmq.devices.proxydevice': ,\n", " 'zmq.error': ,\n", " 'zmq.eventloop': ,\n", " 'zmq.eventloop.ioloop': ,\n", " 'zmq.eventloop.minitornado': ,\n", " 'zmq.eventloop.minitornado.stack_context': ,\n", " 'zmq.eventloop.minitornado.util': ,\n", " 'zmq.eventloop.zmqstream': ,\n", " 'zmq.sugar': ,\n", " 'zmq.sugar.attrsettr': ,\n", " 'zmq.sugar.backend': ,\n", " 'zmq.sugar.constants': ,\n", " 'zmq.sugar.context': ,\n", " 'zmq.sugar.frame': ,\n", " 'zmq.sugar.poll': ,\n", " 'zmq.sugar.socket': ,\n", " 'zmq.sugar.tracker': ,\n", " 'zmq.sugar.version': ,\n", " 'zmq.utils': ,\n", " 'zmq.utils.initthreads': ,\n", " 'zmq.utils.jsonapi': ,\n", " 'zmq.utils.strtypes': }" ] } ], "prompt_number": 1 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`sys.meta_path`\n", "===============\n", "\n", "A list of *finder* objects that have their `find_module()` methods called to see if one of the objects can find the module to be imported." ] }, { "cell_type": "code", "collapsed": false, "input": [ "sys.meta_path" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 2, "text": [ "[_frozen_importlib.BuiltinImporter,\n", " _frozen_importlib.FrozenImporter,\n", " _frozen_importlib.PathFinder]" ] } ], "prompt_number": 2 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Module finder\n", "=============\n", "\n", "* it must implement `find_module(fullname, paths=None)` method\n", " * `fullname` - fully qualified name of the module\n", " * `paths` - list of paths\n", "* it should return a *loader* object if the module was found, or `None` if it wasn't" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`DummyFinder` - example\n", "=====================" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class DummyFinder:\n", " \n", " def find_module(self, fullname, paths=None):\n", " print('find_module - fullname: {}, paths: {}'.format(fullname, paths))\n", " return None\n", " \n", "sys.meta_path.insert(0, DummyFinder())" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 3 }, { "cell_type": "code", "collapsed": false, "input": [ "sys.path.append('./workspace/')" ], "language": "python", "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "import eggs" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "find_module - fullname: eggs, paths: None\n" ] } ], "prompt_number": 5 }, { "cell_type": "code", "collapsed": false, "input": [ "from spam import bacon" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "find_module - fullname: spam, paths: None\n", "find_module - fullname: spam.bacon, paths: ['./workspace/spam']\n" ] } ], "prompt_number": 6 }, { "cell_type": "code", "collapsed": false, "input": [ "del sys.modules['spam']\n", "del sys.modules['spam.bacon']" ], "language": "python", "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "prompt_number": 7 }, { "cell_type": "code", "collapsed": false, "input": [ "import spam.bacon" ], "language": "python", "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "find_module - fullname: spam, paths: None\n", "find_module - fullname: spam.bacon, paths: ['./workspace/spam']\n" ] } ], "prompt_number": 8 }, { "cell_type": "code", "collapsed": false, "input": [ "'eggs' in sys.modules" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 9, "text": [ "True" ] } ], "prompt_number": 9 }, { "cell_type": "code", "collapsed": false, "input": [ "'spam' in sys.modules" ], "language": "python", "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 10, "text": [ "True" ] } ], "prompt_number": 10 }, { "cell_type": "code", "collapsed": false, "input": [ "'spam.bacon' in sys.modules" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 11, "text": [ "True" ] } ], "prompt_number": 11 }, { "cell_type": "code", "collapsed": false, "input": [ "del sys.meta_path[0]\n", "del sys.modules['eggs']\n", "del sys.modules['spam']\n", "del sys.modules['spam.bacon']" ], "language": "python", "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "prompt_number": 12 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Module finder should return a module loader object\n", "==================================================" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Module loader\n", "=============\n", "\n", "* loader is typically returned by a finder\n", "* it must define `load_module(fullname)` method\n", "* this method should return the loaded module or raise `ImportError` exception" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`DummyLoader` - example\n", "=====================" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class DummyFinder:\n", " \n", " def find_module(self, fullname, paths=None):\n", " print('find_module - fullname: {}, paths: {}'.format(fullname, paths))\n", " if fullname == 'eggs':\n", " return DummyLoader()\n", " else:\n", " return None\n", " \n", "class DummyLoader:\n", " \n", " def load_module(self, fullname):\n", " print('load_module - fullname: {}'.format(fullname))\n", " raise ImportError('spam spam spam')\n", " \n", "sys.meta_path.insert(0, DummyFinder())" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 13 }, { "cell_type": "code", "collapsed": false, "input": [ "import eggs" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "ename": "ImportError", "evalue": "spam spam spam", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0meggs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mload_module\u001b[0;34m(self, fullname)\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mload_module\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfullname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'load_module - fullname: {}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfullname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mImportError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'spam spam spam'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 16\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmeta_path\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mDummyFinder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mImportError\u001b[0m: spam spam spam" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "find_module - fullname: eggs, paths: None\n", "load_module - fullname: eggs\n" ] } ], "prompt_number": 14 }, { "cell_type": "code", "collapsed": false, "input": [ "del sys.meta_path[0]" ], "language": "python", "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "find_module - fullname: zmq.core.zmq, paths: ['/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/core']\n", "find_module - fullname: zmq.core.zmq, paths: ['/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/core']\n" ] } ], "prompt_number": 15 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`finder + loader == importer`\n", "=============================" ] }, { "cell_type": "code", "collapsed": false, "input": [ "class DummyImporter:\n", " \n", " def find_module(self, fullname, paths=None):\n", " print('find_module - fullname: {}, paths: {}'.format(fullname, paths))\n", " if fullname == 'eggs':\n", " return self\n", " else:\n", " return None\n", " \n", " def load_module(self, fullname):\n", " print('load_module - fullname: {}'.format(fullname))\n", " raise ImportError('spam spam spam')\n", " \n", "sys.meta_path.insert(0, DummyImporter())" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 16 }, { "cell_type": "code", "collapsed": false, "input": [ "del sys.meta_path[0]" ], "language": "python", "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "prompt_number": 17 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`load_module` method responsibilities\n", "=====================================\n", "\n", "1. it must create empty module and add it to `sys.modules` or use existing one\n", "2. it may set the `__file__` attribute of the module\n", "3. it may set the `__name__` attribute of the module\n", "4. if the module is a package, it must set the module `__path__` attribute\n", "5. it must set the module `__loader__` attribute\n", "6. it may set the `__package__` attribute of the module\n", "7. if the module is a Python module, it should execute the module\u2019s code\n", "8. it must return created module" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`DynamicImporter` - example\n", "===========================" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import types\n", "\n", "class DynamicImporter:\n", "\n", " def __init__(self, name, code):\n", " self.name = name\n", " self.code = code\n", "\n", " def find_module(self, name, path):\n", " if name == self.name:\n", " return self\n", " else:\n", " return None\n", "\n", " def load_module(self, name):\n", " if name in sys.modules:\n", " module = sys.modules[name]\n", " else:\n", " module = types.ModuleType(name)\n", " sys.modules[name] = module\n", " module.__loader__ = self\n", " exec(self.code, module.__dict__)\n", " return module" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 18 }, { "cell_type": "code", "collapsed": false, "input": [ "sys.meta_path.insert(0, DynamicImporter(name='spam', code='eggs = True'))\n", "\n", "import spam\n", "\n", "print(spam.eggs)" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "True\n" ] } ], "prompt_number": 19 }, { "cell_type": "code", "collapsed": false, "input": [ "del sys.meta_path[0]" ], "language": "python", "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "prompt_number": 20 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Circular dependency example\n", "===========================" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "# a.py\n", "\n", "import b\n", "\n", "X = 1\n", "\n", "# b.py\n", "\n", "import a\n", "\n", "print(a.X)\n", "```" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import a" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "AttributeError", "evalue": "'module' object has no attribute 'X'", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/Users/konradhalas/dev/workspace/personal/import_import_presentation/workspace/a.py\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Users/konradhalas/dev/workspace/personal/import_import_presentation/workspace/b.py\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mAttributeError\u001b[0m: 'module' object has no attribute 'X'" ] } ], "prompt_number": 21 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Circular dependency solution\n", "============================\n", "\n", "* merge modules\n", "* extract common artefacts\n", "* defer `import` (put it into method body)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "How \"normal\" `import` works?\n", "==========================" ] }, { "cell_type": "code", "collapsed": false, "input": [ "sys.meta_path" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 22, "text": [ "[_frozen_importlib.BuiltinImporter,\n", " _frozen_importlib.FrozenImporter,\n", " _frozen_importlib.PathFinder]" ] } ], "prompt_number": 22 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Machinery structures\n", "====================\n", "\n", "1. `sys.modules`\n", "2. `sys.meta_path`\n", "3. **`sys.path`**\n", "4. **`sys.path_hooks`**\n", "5. **`sys.path_importer_cache`**" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`PathFinder` - path based finder\n", "================================\n", "\n", "1. iterates over `sys.path`\n", "2. for each *path entry* in `sys.path` iterates over every callable in `sys.path_hooks`\n", "3. each of the *path entry hooks* in `sys.path_hooks` is called with a single argument - path\n", "4. path entry hook should return *path entry finder* or raise `ImportError`\n", "5. returend path entry finder is cached in `sys.path_importer_cache`\n", "6. call `find_loader` method on returned path entry finder with a single argument - fully qualified module name\n", "7. this method should return loader (and namespace portion) or raise `ImportError`" ] }, { "cell_type": "code", "collapsed": false, "input": [ "sys.path_hooks" ], "language": "python", "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 23, "text": [ "[zipimport.zipimporter, ]" ] } ], "prompt_number": 23 }, { "cell_type": "code", "collapsed": false, "input": [ "sys.path_importer_cache" ], "language": "python", "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 24, "text": [ "{'.': FileFinder('.'),\n", " './workspace/': FileFinder('./workspace/'),\n", " './workspace/spam': FileFinder('./workspace/spam'),\n", " '/Users/konradhalas/.ipython/extensions': FileFinder('/Users/konradhalas/.ipython/extensions'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/collections': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/collections'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/encodings': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/encodings'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/lib-dynload': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/lib-dynload'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/plat-darwin': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python3.3/plat-darwin'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/bin/../lib/python33.zip': None,\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/importlib': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/importlib'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/lib-dynload': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/lib-dynload'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/plat-darwin': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/plat-darwin'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/config': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/config'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/core': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/core'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/core/magics': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/core/magics'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/extensions': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/extensions'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/decorator': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/decorator'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/path': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/path'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/pexpect': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/pexpect'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/simplegeneric': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/simplegeneric'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/ssh': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/external/ssh'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/kernel': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/kernel'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/kernel/blocking': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/kernel/blocking'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/kernel/inprocess': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/kernel/inprocess'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/kernel/zmq': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/kernel/zmq'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/lib': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/lib'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/terminal': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/terminal'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/testing': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/testing'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/utils': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/IPython/utils'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/distribute-0.6.31-py3.3.egg': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/distribute-0.6.31-py3.3.egg'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/pip-1.2.1-py3.3.egg': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/pip-1.2.1-py3.3.egg'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/tornado': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/tornado'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/tornado/platform': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/tornado/platform'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/core': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/core'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/devices': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/devices'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/eventloop': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/eventloop'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/eventloop/minitornado': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/eventloop/minitornado'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/sugar': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/sugar'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/utils': FileFinder('/Users/konradhalas/dev/virtualenvs/import_import/lib/python3.3/site-packages/zmq/utils'),\n", " '/Users/konradhalas/dev/virtualenvs/import_import/lib/python33.zip': None,\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/concurrent': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/concurrent'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/concurrent/futures': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/concurrent/futures'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/ctypes': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/ctypes'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/ctypes/macholib': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/ctypes/macholib'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/html': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/html'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/logging': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/logging'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/multiprocessing': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/multiprocessing'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/plat-darwin': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/plat-darwin'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/sqlite3': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/sqlite3'),\n", " '/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib': FileFinder('/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib')}" ] } ], "prompt_number": 24 }, { "cell_type": "code", "collapsed": false, "input": [ "def http_path_entry_hook(path):\n", " if path.startswith('http://') or path.startswith('https://'):\n", " return HttpPathEntryFinder(path)\n", " else:\n", " raise ImportError()" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "prompt_number": 25 }, { "cell_type": "code", "collapsed": false, "input": [ "import requests\n", "\n", "class HttpPathEntryFinder:\n", "\n", " def __init__(self, path):\n", " self.path = path\n", "\n", " def find_loader(self, name):\n", " url = self.path + name + '.py'\n", " if requests.head(url).status_code == 200:\n", " return HttpLoader(url), None\n", " else:\n", " raise ImportError()" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "prompt_number": 26 }, { "cell_type": "code", "collapsed": false, "input": [ "class HttpLoader:\n", "\n", " def __init__(self, url):\n", " self.url = url\n", "\n", " def load_module(self, name):\n", " if name in sys.modules:\n", " module = sys.modules[name]\n", " else:\n", " module = types.ModuleType(name)\n", " sys.modules[name] = module\n", " module.__loader__ = self\n", " module.__file__ = self.url\n", " response = requests.get(self.url)\n", " exec(response.content, module.__dict__)\n", " return module" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "prompt_number": 27 }, { "cell_type": "code", "collapsed": false, "input": [ "sys.path_hooks.insert(0, http_path_entry_hook)\n", "sys.path.append('https://raw.github.com/jcrocholl/pep8/master/')" ], "language": "python", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "prompt_number": 28 }, { "cell_type": "code", "collapsed": false, "input": [ "import pep8" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 29 }, { "cell_type": "code", "collapsed": false, "input": [ "pep8" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 30, "text": [ "" ] } ], "prompt_number": 30 }, { "cell_type": "code", "collapsed": false, "input": [ "pep8.Checker(lines=['x + y\\n']).check_all()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "stdin:1:2: E221 multiple spaces before operator\n", "stdin:1:5: E222 multiple spaces after operator\n" ] }, { "metadata": {}, "output_type": "pyout", "prompt_number": 31, "text": [ "2" ] } ], "prompt_number": 31 }, { "cell_type": "code", "collapsed": false, "input": [ "del sys.path[-1]\n", "del sys.path_hooks[0]\n", "del sys.modules['pep8']" ], "language": "python", "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "prompt_number": 32 }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Good import practices\n", "=====================\n", "\n", "* put all imports at the top of the file\n", "* divide imports into 3 sections by blank lines:\n", " * Python modules\n", " * third-party modules\n", " * your modules\n", "* put two blank lines after last section\n", "* each section should be sorted alphabetically\n", "* only one import should be on each line\n", "* avoid *star* imports\n", "* don't import single artefact, import whole module\n", "* use relative imports if you can\n", "* avoid `from-import-as`\n", "* do not hack `__init__.py`" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Bad example\n", "===========" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "import requests, ast\n", "from xml import *\n", "from json import dump\n", "from spam import eggs # current package\n", "from os import path as os_path\n", "X = 1\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Good example\n", "============" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "```\n", "import ast\n", "import json\n", "import os.path\n", "from xml import sax\n", "\n", "import requests\n", "\n", "from . import eggs\n", "\n", "\n", "X = 1\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Not covered topics\n", "==================\n", "\n", "* `importlib` and `imp` module\n", "* namespace packages\n", "* `BuiltinImporter` and `FrozenImporter`" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Summary\n", "=======\n", "\n", "* you could do magic with `import` statement\n", "* `import` machinery is quite complicated, but also reasonable\n", "* follow good practices\n", "* don't do that at home" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Thank you!\n", "==========\n", "\n", "Q&A\n", "\n", "Presentation source code: [bitbucket.org/khalas/import_import_presentation](https://bitbucket.org/khalas/import_import_presentation)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "`reload` function\n", "=================\n", "\n", "* <= Python 2.7 - `reload` (builtin function)\n", "* <= Python 3.3 - `imp.reload`\n", "* \\>= Python 3.4 - `importlib.reload`" ] } ], "metadata": {} } ] }