{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import cv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Performance, HPC and IPython Parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Performance issues\n",
    "\n",
    "- Python looping in big arrays can be **slow**\n",
    "- However\n",
    "    - OpenCV operations are efficient machine code\n",
    "    - NumPy operations on arrays are efficient machine code\n",
    "    - SciPy Stack relies in C, C++ and Fortran implementations for numerical software"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Low-level code written in Python, as looping in big arrays, can be slow, mainly because Python is dynamically typed and interpreted. However, in the scientific computing environment described above, this is rarely a problem: the OpenCV interface just access optimized C/C++ code, and most of the software in the SciPy Stack relies in a base of numerical software implemented in C, C++ and Fortran, including the efficient NumPy arrays."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cython\n",
    "\n",
    "- Cython is a **static compiler**\n",
    "- It works on a **super-set of the Python language** that supports **C-like static type declarations**\n",
    "- Compiles Python code to C\n",
    "    - Produces a module that can be imported by the Python interpreter\n",
    "- Useful to\n",
    "    - speed-up low-level looping in arrays;\n",
    "    - access external C/C++ libraries\n",
    "- **Pareto Principle**\n",
    "    - 80% of the run-time is spent in 20% of the source code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "But in the few situations where a low-level looping must be implemented (if the task\n",
    "cannot be implemented using NumPy capabilities) or the functionality of a external library is needed, [Cython](http://cython.org) raises as an alternative. Cython is a static compiler capable of working in a super-set of the Python language that supports C-like static type declarations. It compiles Python code to C, further producing a Python module that can be imported and used from the interpreter. As noted by [Behnel *et al.*]((http://dx.doi.org/10.1109/MCSE.2010.118)), the key idea behind Cython is the **Pareto Principle**, also known as the \"80/20 rule\": 80% of the run-time is spent in 20% of the source code. Cython’s goal is to speed up the critical parts of the code while avoiding too much overhead on coding by the programmer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## IPython.parallel\n",
    "\n",
    "- Other performance issues can be addressed by **parallelization**\n",
    "- IPython.parallel allows parallel and distributed computing\n",
    "    - Single Program, Multiple Data (SPMD) \n",
    "    - Multiple Program, Multiple Data (MPMD). \n",
    "- Parallel applications can be developed, executed and monitored from the IPython shell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Computer vision tasks can involve large sets of images or big point clouds\n",
    "- However, the parallelization of these tasks is trivial \n",
    "- In IPython.parallel, those tasks can be implemented in a few lines of code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Other performance issues can be addressed by parallelization. IPython.parallel is a powerful architecture for parallel and distributed computing supporting different styles of parallelism, such as single program, multiple data (SPMD) and multiple program, multiple data (MPMD). Parallel applications can be easily developed, executed and monitored interactively from the IPython shell. Computer vision tasks can involve large sets of images or big point clouds, but many times the parallelization of these tasks is trivial and, using IPython.parallel, implemented in a few lines of code. The dynamic load balancing feature allows the use of all the available processing threads in the computer or all the processing power available in a cluster, but keeping the interactive computing environment free from large amounts of specific code for parallel computing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Example 9 - Process a bundle of images in parallel\n",
    "\n",
    "In this example, SIFT descriptors of a reference image $I_1$ are computed. Then, descriptors are extracted for every image $I_n$ in a list, and the matches to $I_1$ descriptors are computed. The processing of the list is done in parallel, using all the available cores in the user’s machine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let $D_1$ be an array containing the descriptors of $I_1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "T1 = cv2.imread('data/templeRing/templeR0001.png', cv2.IMREAD_GRAYSCALE)\n",
    "sift = cv2.xfeatures2d.SIFT_create(nfeatures=5000)\n",
    "_, D_1 = sift.detectAndCompute(T1, mask=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In a system shell, an IPython cluster for parallel computing is started using:\n",
    "\n",
    "    ipcluster start --n=8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Eight *nodes* are started (in this example, the number of clusters is selected based on the number of cores available in the user’s machine). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note: if `ipcluster` is not available, it can be installed using, for example, `pip`:\n",
    "\n",
    "    $ pip install ipyparallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Back to the IPython shell, the next step is the creation of a `Client` object. A `LoadBalancedView` object is created to provide a load-balanced parallel execution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from ipyparallel import Client\n",
    "rc = Client()\n",
    "lview = rc.load_balanced_view()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- The decorator `@lview.parallel` defines a **parallel, load-balanced** funtion\n",
    "- The arguments are:\n",
    "    - The image file absolute path in the filesystem\n",
    "    - The reference descriptor set, $D_1$\n",
    "- `get_num_matches` will:\n",
    "    - read the image;\n",
    "    - compute SIFT features and their descriptors;\n",
    "    - perform matching using OpenCV's *brute force matching* `BFMatcher` and\n",
    "    - return the number of martches found"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Next, a Python *decorator* is used to define a parallel function that computes the descriptors and the matches (the decorator starts with a \"`@`\" symbol). The function below takes a path to an image in the file system, computes the SIFT features and uses OpenCV’s `BFMatcher` to get the matches to $D_1$, returning the number of matches found and the image’s path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "@lview.parallel()\n",
    "def get_num_matches(arg):    \n",
    "    fname, D_src = arg\n",
    "    import cv2\n",
    "    frame = cv2.imread(fname, cv2.IMREAD_GRAYSCALE)\n",
    "    print frame.shape\n",
    "    sift = cv2.SIFT(nfeatures=5000)\n",
    "    _, D = sift.detectAndCompute(frame, mask=None)\n",
    "    matcher = cv2.BFMatcher(cv2.NORM_L2, crossCheck=True)\n",
    "    matches = matcher.match(D_src, D)\n",
    "    return fname, len(matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- File paths and $D_1$ are assembled in an **arguments list** \n",
    "- The `map` function starts the parallelized call\n",
    "- Load balancing is automatically performed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "IPython capability to access the system’s shell is employed to list all the files in a directory and store the file paths in a list of strings, `fnames`. Finally, the `map` function calls `get_num_matches` to every string in the `fnames` list, automatically performing the load balance on the nodes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "BadYieldError",
     "evalue": "yielded unknown object <Future at 0x7f10477efb50 state=pending>",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mBadYieldError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-0ade8fe7f65c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mD_1\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mfname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mfnames\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0masync_res\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_num_matches\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.pyc\u001b[0m in \u001b[0;36mmap\u001b[0;34m(self, *sequences)\u001b[0m\n\u001b[1;32m    283\u001b[0m         \u001b[0;32mand\u001b[0m \u001b[0mmismatched\u001b[0m \u001b[0msequence\u001b[0m \u001b[0mlengths\u001b[0m \u001b[0mwill\u001b[0m \u001b[0mbe\u001b[0m \u001b[0mpadded\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    284\u001b[0m         \"\"\"\n\u001b[0;32m--> 285\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msequences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m__ipp_mapping\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    286\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    287\u001b[0m \u001b[0m__all__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'remote'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'parallel'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'RemoteFunction'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'ParallelFunction'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-128>\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *sequences, **kwargs)\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.pyc\u001b[0m in \u001b[0;36msync_view_results\u001b[0;34m(f, self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     77\u001b[0m     \u001b[0mview\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_sync_results\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     78\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 79\u001b[0;31m         \u001b[0mret\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     80\u001b[0m     \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     81\u001b[0m         \u001b[0mview\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_sync_results\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *sequences, **kwargs)\u001b[0m\n\u001b[1;32m    257\u001b[0m             \u001b[0mview\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mview\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mbalanced\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    258\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mview\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtemp_flags\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mblock\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 259\u001b[0;31m                 \u001b[0mar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mview\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    260\u001b[0m                 \u001b[0mar\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mowner\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    261\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.pyc\u001b[0m in \u001b[0;36mapply\u001b[0;34m(self, f, *args, **kwargs)\u001b[0m\n\u001b[1;32m    209\u001b[0m         \u001b[0;34m`\u001b[0m\u001b[0;34m`\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m`\u001b[0m\u001b[0;34m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    210\u001b[0m         \"\"\"\n\u001b[0;32m--> 211\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_really_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    212\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    213\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-144>\u001b[0m in \u001b[0;36m_really_apply\u001b[0;34m(self, f, args, kwargs, block, track, after, follow, timeout, targets, retries)\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.pyc\u001b[0m in \u001b[0;36msync_results\u001b[0;34m(f, self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     45\u001b[0m     \u001b[0;34m\"\"\"sync relevant results from self.client to our results attribute.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     46\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_sync_results\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 47\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     48\u001b[0m     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_sync_results\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     49\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-143>\u001b[0m in \u001b[0;36m_really_apply\u001b[0;34m(self, f, args, kwargs, block, track, after, follow, timeout, targets, retries)\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.pyc\u001b[0m in \u001b[0;36msave_ids\u001b[0;34m(f, self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     33\u001b[0m     \u001b[0mn_previous\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhistory\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 35\u001b[0;31m         \u001b[0mret\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     36\u001b[0m     \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m         \u001b[0mnmsgs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhistory\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mn_previous\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.pyc\u001b[0m in \u001b[0;36m_really_apply\u001b[0;34m(self, f, args, kwargs, block, track, after, follow, timeout, targets, retries)\u001b[0m\n\u001b[1;32m   1035\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1036\u001b[0m         future = self.client.send_apply_request(self._socket, f, args, kwargs, track=track,\n\u001b[0;32m-> 1037\u001b[0;31m                                 metadata=metadata)\n\u001b[0m\u001b[1;32m   1038\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1039\u001b[0m         ar = AsyncResult(self.client, future, fname=getname(f),\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/client.pyc\u001b[0m in \u001b[0;36msend_apply_request\u001b[0;34m(self, socket, f, args, kwargs, metadata, track, ident)\u001b[0m\n\u001b[1;32m   1397\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1398\u001b[0m         future = self._send(socket, \"apply_request\", buffers=bufs, ident=ident,\n\u001b[0;32m-> 1399\u001b[0;31m                             metadata=metadata, track=track)\n\u001b[0m\u001b[1;32m   1400\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1401\u001b[0m         \u001b[0mmsg_id\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfuture\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmsg_id\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/ipyparallel/client/client.pyc\u001b[0m in \u001b[0;36m_send\u001b[0;34m(self, socket, msg_type, content, parent, ident, buffers, track, header, metadata)\u001b[0m\n\u001b[1;32m    967\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmetadata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg_id\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    968\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 969\u001b[0;31m         \u001b[0mmulti_future\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfutures\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_done_callback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcleanup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    970\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    971\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0m_really_send\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/tornado/gen.pyc\u001b[0m in \u001b[0;36mmulti_future\u001b[0;34m(children, quiet_exceptions)\u001b[0m\n\u001b[1;32m    776\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    777\u001b[0m         \u001b[0mkeys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 778\u001b[0;31m     \u001b[0mchildren\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconvert_yielded\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mchildren\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    779\u001b[0m     \u001b[0;32massert\u001b[0m \u001b[0mall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mis_future\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mchildren\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    780\u001b[0m     \u001b[0munfinished_children\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/singledispatch.pyc\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m    208\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    209\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 210\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mdispatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    211\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m     \u001b[0mregistry\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mobject\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/tornado/gen.pyc\u001b[0m in \u001b[0;36mconvert_yielded\u001b[0;34m(yielded)\u001b[0m\n\u001b[1;32m   1227\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0m_wrap_awaitable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0myielded\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1228\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1229\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mBadYieldError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"yielded unknown object %r\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0myielded\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1230\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1231\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0msingledispatch\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mBadYieldError\u001b[0m: yielded unknown object <Future at 0x7f10477efb50 state=pending>"
     ]
    }
   ],
   "source": [
    "fnames = !ls data/templeRing/temple*.png\n",
    "\n",
    "args = [(fname, D_1) for fname in fnames]\n",
    "async_res = get_num_matches.map(args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/templeRing/templeR0001.png 802\n",
      "/tmp/templeRing/templeR0002.png 549\n",
      "/tmp/templeRing/templeR0003.png 491\n",
      "/tmp/templeRing/templeR0004.png 482\n",
      "/tmp/templeRing/templeR0005.png 470\n",
      "/tmp/templeRing/templeR0006.png 441\n",
      "/tmp/templeRing/templeR0007.png 401\n",
      "/tmp/templeRing/templeR0008.png 358\n",
      "/tmp/templeRing/templeR0009.png 393\n",
      "/tmp/templeRing/templeR0010.png 438\n",
      "/tmp/templeRing/templeR0011.png 456\n",
      "/tmp/templeRing/templeR0012.png 455\n",
      "/tmp/templeRing/templeR0013.png 444\n",
      "/tmp/templeRing/templeR0014.png 405\n",
      "/tmp/templeRing/templeR0015.png 436\n",
      "/tmp/templeRing/templeR0016.png 421\n",
      "/tmp/templeRing/templeR0017.png 410\n",
      "/tmp/templeRing/templeR0018.png 398\n",
      "/tmp/templeRing/templeR0019.png 408\n",
      "/tmp/templeRing/templeR0020.png 451\n",
      "/tmp/templeRing/templeR0021.png 430\n",
      "/tmp/templeRing/templeR0022.png 444\n",
      "/tmp/templeRing/templeR0023.png 454\n",
      "/tmp/templeRing/templeR0024.png 476\n",
      "/tmp/templeRing/templeR0025.png 475\n",
      "/tmp/templeRing/templeR0026.png 509\n",
      "/tmp/templeRing/templeR0027.png 509\n",
      "/tmp/templeRing/templeR0028.png 546\n",
      "/tmp/templeRing/templeR0029.png 558\n",
      "/tmp/templeRing/templeR0030.png 662\n",
      "/tmp/templeRing/templeR0031.png 577\n",
      "/tmp/templeRing/templeR0032.png 465\n",
      "/tmp/templeRing/templeR0033.png 477\n",
      "/tmp/templeRing/templeR0034.png 479\n",
      "/tmp/templeRing/templeR0035.png 457\n",
      "/tmp/templeRing/templeR0036.png 456\n",
      "/tmp/templeRing/templeR0037.png 473\n",
      "/tmp/templeRing/templeR0038.png 477\n",
      "/tmp/templeRing/templeR0039.png 473\n",
      "/tmp/templeRing/templeR0040.png 454\n",
      "/tmp/templeRing/templeR0041.png 461\n",
      "/tmp/templeRing/templeR0042.png 431\n",
      "/tmp/templeRing/templeR0043.png 442\n",
      "/tmp/templeRing/templeR0044.png 454\n",
      "/tmp/templeRing/templeR0045.png 448\n",
      "/tmp/templeRing/templeR0046.png 454\n",
      "/tmp/templeRing/templeR0047.png 459\n"
     ]
    }
   ],
   "source": [
    "for f, n in async_res:\n",
    "    print f, n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "This simple example is able to explore all the available cores in the local machine, just asking for a few extra lines of code. But the parallel computing capabilities in IPython go farbeyond, supporting SPMD and MPMD parallelism and the use of StarCluster for execution\n",
    "in Amazon’s Elastic Compute Cloud (EC$_2$). The interested reader is referred to the section [Using IPython for parallel computing in the IPython](http://ipyparallel.readthedocs.io/en/latest/) documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}