{ "metadata": { "name": "parallel python" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Parallel computing with Python\n", "\n", "## `multiprocessing`\n", "\n", "The built in module `multiprocessing` provides functionality to create _processes_ which runs given tasks.\n", "\n", "http://docs.python.org/2/library/multiprocessing.html\n", "\n", "All strategies for paralleliztion has a rathe large overhead compared to lower level languages such as C or FORTRAN.\n", "\n", "The way `multiprocessing` runs code in parallel is by launching subprocesses with a seperate interpretor for for each process. This means that in order to gain speed the computation we want to perform should be relatively substantial.\n", "\n", "(In case you are familiar with threads: It should be noted that Python has a `threading` module for working with threads, however, all threads will be run on a single CPU.)\n", "\n", "Byt using multiprocessing we can utilize the machines we are running code on more efficiently" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import multiprocessing" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 1 }, { "cell_type": "code", "collapsed": false, "input": [ "multiprocessing.cpu_count()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 2, "text": [ "16" ] } ], "prompt_number": 2 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before talking about some more advanced featuers, let's describe the most typical use pattern of `multiprocessing`.\n", "\n", "Note: `multiprocessing` _can_ be used in the IPython Notebook, but there are sometimes issues with printing from subprocesses. To make things clearer and avoid complications we shall run external scripts in stead." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `Process`\n", "Processes share nothing\n", "\n", "To spawn a process, initiate it with a target function and call the `.start()` method.\n", "\n", "This method will arrange things so that given code will be run in a seperate process from the parent process. To get the parent process to wait until a process has finished before moving on one need to call the `.join()` method." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import os\n", "os.getpid()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 4, "text": [ "83928" ] } ], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "%%file mp.py\n", "from multiprocessing import Process\n", "import os\n", "\n", "def worker():\n", " print(\"Worker process {}\".format(os.getpid()))\n", "\n", "if __name__ == \"__main__\":\n", " proc1 = Process(target=worker)\n", " proc1.start()\n", " proc2 = Process(target=worker)\n", " proc2.start()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Overwriting mp.py\n" ] } ], "prompt_number": 7 }, { "cell_type": "code", "collapsed": false, "input": [ "%%bash\n", "python mp.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Worker process 87461\n", "Worker process 87462\n" ] } ], "prompt_number": 8 }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get the target function to actually work on some input, you need to provide the arguments in the constructur of the `Process`." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file mp.py\n", "from multiprocessing import Process\n", "import os\n", "\n", "def worker(arg):\n", " print(\"Worker process {}, argument was {}\".format(os.getpid(), arg))\n", "\n", "if __name__ == \"__main__\":\n", " proc1 = Process(target=worker, args=(10,))\n", " proc1.start()\n", " proc2 = Process(target=worker, args=(11,))\n", " proc2.start()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Overwriting mp.py\n" ] } ], "prompt_number": 9 }, { "cell_type": "code", "collapsed": false, "input": [ "%%bash\n", "python mp.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Worker process 87490, argument was 10\n", "Worker process 87491, argument was 11\n" ] } ], "prompt_number": 10 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Processes communicate over interprocess communication channel\n", "\n", "- `Queue`\n", "- `Pipe`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `Pipe`\n", "\n", "Gives a pair of connection objects which are connected by a pipe. Uses `send` and `recv` methods on objects to comminucate between processes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `Queue`\n", "\n", "Gives a thread and process safe queue shared between processes. Can contain any pickle-able object." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file mp2.py\n", "from multiprocessing import Process, Queue\n", "import os\n", "\n", "def worker(tasks, results):\n", " t = tasks.get()\n", " result = t * 2\n", " results.put([os.getpid(), t, \"->\", result])\n", "\n", "if __name__ == \"__main__\":\n", " n = 20\n", " my_tasks = Queue()\n", " my_results = Queue()\n", " \n", " workers = [Process(target=worker, args=(my_tasks, my_results)) for i in range(n)]\n", " \n", " for proc in workers:\n", " proc.start()\n", " \n", " for i in range(n):\n", " my_tasks.put(i)\n", " \n", " for i in range(n):\n", " result = my_results.get()\n", " print(result)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Overwriting mp2.py\n" ] } ], "prompt_number": 11 }, { "cell_type": "code", "collapsed": false, "input": [ "%%bash\n", "python mp2.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[87680, 0, '->', 0]\n", "[87681, 1, '->', 2]\n", "[87682, 2, '->', 4]\n", "[87683, 3, '->', 6]\n", "[87685, 5, '->', 10]\n", "[87684, 4, '->', 8]\n", "[87687, 7, '->', 14]\n", "[87686, 6, '->', 12]\n", "[87688, 8, '->', 16]\n", "[87689, 9, '->', 18]\n", "[87691, 11, '->', 22]\n", "[87690, 10, '->', 20]\n", "[87693, 13, '->', 26]\n", "[87692, 12, '->', 24]\n", "[87694, 14, '->', 28]\n", "[87695, 15, '->', 30]\n", "[87696, 16, '->', 32]\n", "[87697, 17, '->', 34]\n", "[87699, 19, '->', 38]\n", "[87698, 18, '->', 36]\n" ] } ], "prompt_number": 12 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Because the processes are executed in parallel we can never know the order of results being put in the `Queue`." ] }, { "cell_type": "code", "collapsed": false, "input": [ "from multiprocessing import Queue" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 13 }, { "cell_type": "code", "collapsed": false, "input": [ "q = Queue()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 14 }, { "cell_type": "code", "collapsed": false, "input": [ "q.get?" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 15 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `Manager`\n", "\n", "A special `Process` which holds Python objects so that other processes can manipulate them.\n", "When a managed object is manipulated somewhere, the manager will make sure the change is propagated to any other processes using the managed object." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file mp3.py\n", "from multiprocessing import Process, Manager\n", "import os\n", "\n", "def worker(l):\n", " p = os.getpid()\n", " l[int(str(p)[-2:])] = p\n", "\n", "\n", "if __name__ == \"__main__\":\n", " n = 100\n", " manager = Manager()\n", " l = manager.list()\n", " l.extend([0] * n)\n", " \n", " processes = [Process(target=worker, args=(l,)) for i in range(20)]\n", " \n", " for proc in processes:\n", " proc.start()\n", " \n", " for proc in processes:\n", " proc.join()\n", " \n", " print(l)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Overwriting mp3.py\n" ] } ], "prompt_number": 16 }, { "cell_type": "code", "collapsed": false, "input": [ "%%bash\n", "python mp3.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 87824, 87825, 87826, 87827, 87828, 87829, 87830, 87831, 87832, 87833, 87834, 87835, 87836, 87837, 87838, 87839, 87840, 87841, 87842, 87843, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n" ] } ], "prompt_number": 17 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `Pool`\n", "\n", "The `Pool` class distributes work between workers and collects the results as a list. It is _extremely_ handy for quickly implementing some simple parallelization." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file mp.py\n", "import multiprocessing\n", "import os\n", "\n", "\n", "def task(args):\n", " print \"Running process\", os.getpid(), \"with args\", args\n", " return os.getpid(), args\n", "\n", "\n", "if __name__ == \"__main__\":\n", " pool = multiprocessing.Pool(processes=4)\n", " result = pool.map(task, [1,2,3,4]*3)\n", " print(result)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Overwriting mp.py\n" ] } ], "prompt_number": 20 }, { "cell_type": "code", "collapsed": false, "input": [ "%%bash\n", "python mp.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Running process 88078 with args 4\n", "Running process 88077 with args 3\n", "Running process 88077 with args 3\n", "Running process 88077 with args 2\n", "Running process 88075 with args 1\n", "Running process 88075 with args 1\n", "Running process 88075 with args 4\n", "Running process 88075 with args 3\n", "Running process 88076 with args 2\n", "Running process 88076 with args 2\n", "Running process 88076 with args 1\n", "Running process 88076 with args 4\n", "[(88075, 1), (88076, 2), (88077, 3), (88078, 4), (88075, 1), (88076, 2), (88077, 3), (88075, 4), (88076, 1), (88077, 2), (88075, 3), (88076, 4)]\n" ] } ], "prompt_number": 21 }, { "cell_type": "markdown", "metadata": {}, "source": [ "The method `.map()` works like the built in function `map()`, but will send data from the iterable to different processes. By default it will send one element at a time, but this can be changed with the `chunksize` parameter.\n", "\n", "A similar method called `.map_async()` usually performs better in parallel, but in that case one has to fetch the results using a `.get()` method of the returned value of `.map_async()` (which is an instance of the class `AsyncResult`)." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file mp.py\n", "import multiprocessing\n", "import os\n", "\n", "def task(args):\n", " print \"Running process\", os.getpid(), \"with args\", args\n", " return os.getpid(), args\n", "\n", "\n", "if __name__ == \"__main__\":\n", " pool = multiprocessing.Pool(processes=4)\n", " result = pool.map_async(task, [1,2,3,4])\n", " print(result.get())" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Overwriting mp.py\n" ] } ], "prompt_number": 22 }, { "cell_type": "code", "collapsed": false, "input": [ "%%bash\n", "python mp.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Running process 88108 with args 1\n", "Running process 88109 with args 2\n", "Running process 88110 with args 3\n", "Running process 88111 with args 4\n", "[(88108, 1), (88109, 2), (88110, 3), (88111, 4)]\n" ] } ], "prompt_number": 23 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## IPython Parallel\n", "\n", "http://ipython.org/ipython-doc/dev/parallel/\n", "\n", "The strategy for achieving parallelization in IPython Parallel differs from `multiprocessing` in that you need to start _engine_ processes and a _hub_ process before you want to run code.\n", "\n", "One then connect to the hub, which will aid in the distrubution of work across the engines.\n", "\n", "When IPython is installed, a program `ipcluster` is also installed which simplifies starting engines and hub. To start a cluster with 4 workers one would run \n", "\n", " $ ipcluster start -n 4\n", "\n", "\n", "In the IPython Notebook, clusters can also be started from the IPython Dashboard." ] }, { "cell_type": "code", "collapsed": false, "input": [ "from IPython.parallel import Client" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 24 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's try initiating the client before we have started the cluster!" ] }, { "cell_type": "code", "collapsed": false, "input": [ "cli = Client()" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "IOError", "evalue": "Connection file not found: u'/Users/vale/.ipython/profile_default/security/ipcontroller-client.json'", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mIOError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcli\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mClient\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/Users/vale/.virtualenvs/devel/lib/python2.7/site-packages/IPython/parallel/client/client.pyc\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, url_or_file, profile, profile_dir, ipython_dir, context, debug, exec_key, sshserver, sshkey, password, paramiko, timeout, **extra_args)\u001b[0m\n\u001b[1;32m 409\u001b[0m \u001b[0murl_or_file\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_cd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msecurity_dir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl_or_file\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 410\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl_or_file\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 411\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mIOError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Connection file not found: %r\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0murl_or_file\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 412\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl_or_file\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 413\u001b[0m \u001b[0mcfg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mIOError\u001b[0m: Connection file not found: u'/Users/vale/.ipython/profile_default/security/ipcontroller-client.json'" ] } ], "prompt_number": 25 }, { "cell_type": "markdown", "metadata": {}, "source": [ "IPython keeps track of addresses and such of local engines in the `.ipython` directory in your home folder. But `Client()` can also take a URL or a path to a configuration file with information on how to connect to the cluster.\n", "\n", "Now start the cluster (either in terminal or on the Dashboard)" ] }, { "cell_type": "code", "collapsed": false, "input": [ "cli = Client()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 26 }, { "cell_type": "code", "collapsed": false, "input": [ "cli.ids" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 27, "text": [ "[0, 1, 2, 3, 4, 5, 6, 7]" ] } ], "prompt_number": 27 }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `ids` field lists the engines in the cluster.\n", "\n", "Say we want to have a look at the pid like we have done for all the `multiprocessing` exampels." ] }, { "cell_type": "code", "collapsed": false, "input": [ "def get_pid():\n", " import os\n", " return os.getpid()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 28 }, { "cell_type": "code", "collapsed": false, "input": [ "cli[0].apply_sync(get_pid)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 29, "text": [ "88707" ] } ], "prompt_number": 29 }, { "cell_type": "code", "collapsed": false, "input": [ "get_pid()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 30, "text": [ "83928" ] } ], "prompt_number": 30 }, { "cell_type": "markdown", "metadata": {}, "source": [ "We need to `import os` in the function we send to the engine. That engine was started without any imported functions. Every time we want the engines to have some data, we need to explicitly send it to them. Or instructions on how to get it in the case of `import`.\n", "\n", "There is a method for syncing import across engines." ] }, { "cell_type": "code", "collapsed": false, "input": [ "cli[:].apply_sync(get_pid)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 33, "text": [ "[88707, 88708, 88706, 88709, 88710, 88711, 88712, 88713]" ] } ], "prompt_number": 33 }, { "cell_type": "markdown", "metadata": {}, "source": [ "A collection of engines is referred to as an _engine pool_, the standard _view_ to an engine pool is called a _direct view_." ] }, { "cell_type": "code", "collapsed": false, "input": [ "dview = cli[:]" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 34 }, { "cell_type": "markdown", "metadata": {}, "source": [ "The direct view provides a decorator which can be used to make any function parallelized." ] }, { "cell_type": "code", "collapsed": false, "input": [ "@dview.parallel(block=True)\n", "def pstr(x):\n", " return str(x)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 35 }, { "cell_type": "markdown", "metadata": {}, "source": [ "After the `.parallel` decorater have been used on a function, it is changed in such a way that arguments given to the function will be split up, sent to engines, then put together again after the calculation is done.\n", "\n", "The new `ParallelFunction` also has a `.map()` method which will run the function for each of the values in an iterable on different engines, and return the list of results when it's done." ] }, { "cell_type": "code", "collapsed": false, "input": [ "str(range(10))" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 36, "text": [ "'[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]'" ] } ], "prompt_number": 36 }, { "cell_type": "code", "collapsed": false, "input": [ "pstr(range(10))" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 37, "text": [ "['[0, 1]', '[2, 3]', '[4]', '[5]', '[6]', '[7]', '[8]', '[9]']" ] } ], "prompt_number": 37 }, { "cell_type": "code", "collapsed": false, "input": [ "pstr.map(range(10))" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 38, "text": [ "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']" ] } ], "prompt_number": 38 }, { "cell_type": "code", "collapsed": false, "input": [ "from numpy import random" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 39 }, { "cell_type": "code", "collapsed": false, "input": [ "@dview.parallel(block=True)\n", "def task(delay):\n", " import os, time\n", " \n", " t0 = time.time()\n", " pid = os.getpid()\n", " time.sleep(delay)\n", " t1 = time.time()\n", " \n", " return [pid, t0, t1]" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 40 }, { "cell_type": "code", "collapsed": false, "input": [ "delays = random.rand(4)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 43 }, { "cell_type": "code", "collapsed": false, "input": [ "task.map(delays)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 44, "text": [ "[[88707, 1361974304.03308, 1361974304.959194],\n", " [88708, 1361974304.034218, 1361974304.233133],\n", " [88706, 1361974304.035217, 1361974304.954711],\n", " [88709, 1361974304.036044, 1361974304.276678]]" ] } ], "prompt_number": 44 }, { "cell_type": "code", "collapsed": false, "input": [ "def visualize_tasks(results):\n", " res = np.array(results)\n", " fig, ax = plt.subplots(figsize=(10, res.shape[1]))\n", " \n", " yticks = []\n", " yticklabels = []\n", " tmin = min(res[:,1])\n", " for n, pid in enumerate(np.unique(res[:,0])):\n", " yticks.append(n)\n", " yticklabels.append(\"%d\" % pid)\n", " for m in np.where(res[:,0] == pid)[0]:\n", " ax.add_patch(Rectangle((res[m,1] - tmin, n-0.25),\n", " res[m,2] - res[m,1], 0.5, color=\"green\", alpha=0.5))\n", " \n", " ax.set_ylim(-.5, n+.5)\n", " ax.set_xlim(0, max(res[:,2]) - tmin + 0.)\n", " ax.set_yticks(yticks)\n", " ax.set_yticklabels(yticklabels)\n", " ax.set_ylabel(\"PID\")\n", " ax.set_xlabel(\"seconds\")" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 45 }, { "cell_type": "code", "collapsed": false, "input": [ "delays = random.rand(100) / 4." ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 46 }, { "cell_type": "code", "collapsed": false, "input": [ "result = task.map(delays)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 47 }, { "cell_type": "code", "collapsed": false, "input": [ "visualize_tasks(result)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAmwAAADOCAYAAACD3l7pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9UVPed//HnCGSFrZhAtkVHJ6MJlED4MVSIMaadtvFg\n7CYGixuELPFgsmxim5xUAknt8fiju3ZPu9HYrm31eIzLohOIp4Y9pu4qBjj+gEEZAz3GLdJMDER7\nmm2gKpREmO8f+TobigKRO3JneD3O4Rwvcz/3835/uGfO23vvzNvi8/l8iIiIiIhpTRrvAERERERk\neCrYRERERExOBZuIiIiIyalgExERETG58PEOIFAsFst4hyAiIiIyasN9DjRkCzYYPnGRkaxdu5a1\na9eOdxgSpHT+yFjpHJpYRrrQpFuiIiIiIiangk1ERETE5FSwiVyH0+kc7xAkiOn8kbHSOSSfZQnV\nTgcWi0XPsImIiEhQGKluCekPHQC8eOhFLly6MN5hjNqZD8+QeHvieIcxaoGIdzzXINjW/1pCIYfr\nCbbcPk+8wZbbWEykXIczmnW4uk/cF+L40YM/ukmRiRmFfMF24dIF7LfaxzuMUTty7siEj3c81yDY\n1v9aQiGH6wm23D5PvMGW21hMpFyHM5p1uLqPt8t7U2IS8wrIM2wbN24kOTmZlJQU8vPz6evrw+12\nk5WVhcPhIDMzk6amJgAqKipwOBz+n7CwMFpaWgBYvXo1NpuNKVOmDDr+L37xC1JTU3E4HNx33328\n/fbbgUhDRERExBQML9i8Xi/bt2+nubmZ1tZW+vv7cblclJWVsWHDBjweD+vXr6e0tBSAgoICPB4P\nHo+H8vJyZs+eTWpqKgCLFy/G7XYPmaOgoICWlhY8Hg/f//73WbVqldFpiIiIiJiG4bdEo6OjiYiI\noKenh7CwMHp6epg+fTpxcXF0d3cD0NXVhdVqHTJ29+7d5OXl+bezsrKuOcdnr7hdunSJ22+/3eAs\nRERERMzD8IItJiaGVatWYbPZiIyMJDs7mwULFpCQkMD8+fMpKSlhYGCA48ePDxlbWVlJdXX1qObZ\nunUrL7/8MpcvX+bYsWPX3Gft2rWcOnMK72Qv9nQ79nT7WFITERERMURtbS21tbWj3t/wgq29vZ3N\nmzfj9XqZOnUqS5cupaKigp07d7JlyxZycnKoqqqiqKiIgwcP+sc1NjYSFRVFUlLSqOZ55plneOaZ\nZ9izZw9FRUW89dZbQ/ZZu3Yt3n1ePdwqIiIipuJ0Ogd91966deuG3d/wZ9hOnDjBvHnziI2NJTw8\nnCVLlnD06FHcbjc5OTkA5ObmDnk2zeVykZ+f/7nne+yxx2hubjYkdhEREREzMrxgS0xMpKGhgd7e\nXnw+HzU1NSQlJREfH09dXR0Ahw8fJiEhwT9mYGCAqqqqQc+vDefs2bP+f+/fv9//IQURERGRUGT4\nLdG0tDQKCwuZM2cOkyZNIiMjg+LiYubOncvKlSvp6+sjMjKSbdu2+cfU19djs9mw2+2DjlVaWsqe\nPXvo7e1l5syZPPXUU6xZs4af/exnHDp0iIiICP7mb/6GnTt3Gp2GiIiIiGmEfGsqdToILHU6MJ9Q\nyOF6gi03dTq4tomU63DU6UA+a6TWVCFfsImIiIiY3Uh1S0A6HYiIiIiIcVSwiYiIiJicCjYRERER\nk1PBJiIiImJyKthERERETE4Fm4iIiIjJqWATERERMTkVbCIiIiImp4JNRERExOQM7yVqNmZqTRVq\n7VhCLZ/hhFquNzOfYFm7YInzs24kZrPmada4xttY1iXY11TtuAYL+YLtwqUL2G+1j3cYABw5d8Q0\nsRgh1PIZTqjlejPzCZa1C5Y4P+tGYjZrnmaNa7yNZV2CfU29Xd7xDsFUAnJLdOPGjSQnJ5OSkkJ+\nfj59fX243W6ysrJwOBxkZmbS1NQEQEVFBQ6Hw/8TFhZGS0sLAKtXr8ZmszFlypRBx3/55ZdJTk4m\nLS2NBx98kHPnzgUiDRERERFTMLxg83q9bN++nebmZlpbW+nv78flclFWVsaGDRvweDysX7+e0tJS\nAAoKCvB4PHg8HsrLy5k9ezapqakALF68GLfbPWSOjIwMTp48ydtvv01ubq7/WCIiIiKhyPCCLTo6\nmoiICHp6erhy5Qo9PT1Mnz6duLg4uru7Aejq6sJqtQ4Zu3v3bvLy8vzbWVlZxMXFDdnP6XQyefJk\nAO699146OjqMTkNERETENAx/hi0mJoZVq1Zhs9mIjIwkOzubBQsWkJCQwPz58ykpKWFgYIDjx48P\nGVtZWUl1dfXnmm/Hjh0sWrTomq+tXbuWU2dO4Z3sxZ5ux55uv5GURERERAxVW1tLbW3tqPc3vGBr\nb29n8+bNeL1epk6dytKlS6moqGDnzp1s2bKFnJwcqqqqKCoq4uDBg/5xjY2NREVFkZSUNOq5/uM/\n/oPm5mY2bdp0zdfXrl2Ld583qB+6FBERkdDjdDpxOp3+7XXr1g27v+G3RE+cOMG8efOIjY0lPDyc\nJUuWcPToUdxuNzk5OQDk5uYOeTbN5XKRn58/6nkOHTrEP//zP1NdXU1ERIShOYiIiIiYieEFW2Ji\nIg0NDfT29uLz+aipqSEpKYn4+Hjq6uoAOHz4MAkJCf4xAwMDVFVVDXp+bTgej4d//Md/5D//8z+5\n/fbbjU5BRERExFQMvyWalpZGYWEhc+bMYdKkSWRkZFBcXMzcuXNZuXIlfX19REZGsm3bNv+Y+vp6\nbDYbdrt90LFKS0vZs2cPvb29zJw5k6eeeoo1a9ZQWlrK5cuXyc3NBeCOO+5g3759RqciIiIiYgoW\nn8/nG+8gAsFiseDz+dTpIIBCLZ/hhFqu6nQwVLDE+VnqdBD61Olg4nQ6uFq3XPf1UC/YRERERMxu\npLpFzd9FRERETE4Fm4iIiIjJqWATERERMTkVbCIiIiImp4JNRERExORUsImIiIiYnAo2EREREZNT\nwSYiIiJicoa3pjIbM3U6MKNg/ybszwq2XIIt3qs+b9yBznO81tHoeccjj9HOGaiOChPtm+xFxiLk\nC7YLly5gv9U+3mGY1pFzR0JmfYItl2CL96rPG3eg8xyvdTR63vHIY7Rz3khsoxnj7fJ+rmOKTGS6\nJSoiIiJicgEp2DZu3EhycjIpKSnk5+fT19eH2+0mKysLh8NBZmYmTU1NAFRUVOBwOPw/YWFhtLS0\nALB69WpsNhtTpkwZdPz6+noyMjKIiIhg7969gUhBRERExDQML9i8Xi/bt2+nubmZ1tZW+vv7cblc\nlJWVsWHDBjweD+vXr6e0tBSAgoICPB4PHo+H8vJyZs+eTWpqKgCLFy/G7XYPmeOOO+5g165d5Ofn\nGx2+iIiIiOkY/gxbdHQ0ERER9PT0EBYWRk9PD9OnTycuLo7u7m4Aurq6sFqtQ8bu3r2bvLw8/3ZW\nVtY157jjjjsAmDRJd3RFREQk9BlesMXExLBq1SpsNhuRkZFkZ2ezYMECEhISmD9/PiUlJQwMDHD8\n+PEhYysrK6murjYslrVr13LqzCm8k73Y0+3Y0+2GHVtERETkRtXW1lJbWzvq/Q0v2Nrb29m8eTNe\nr5epU6eydOlSKioq2LlzJ1u2bCEnJ4eqqiqKioo4ePCgf1xjYyNRUVEkJSUZFsvatWvx7vMG5Sfx\nREREJHQ5nU6cTqd/e926dcPub/g9xRMnTjBv3jxiY2MJDw9nyZIlHD16FLfbTU5ODgC5ublDnk1z\nuVw39EyaxWIxJG4RERERszK8YEtMTKShoYHe3l58Ph81NTUkJSURHx9PXV0dAIcPHyYhIcE/ZmBg\ngKqqqkHPr42Gz+fD5/MZGr+IiIiI2RhesKWlpVFYWMicOXNITU3F5/NRXFzML3/5S0pLS0lPT+cH\nP/gB27Zt84+pr6/HZrNht9sHHau0tJSZM2fS29vLzJkzWb9+PQBNTU3MnDmT119/neLiYlJSUoxO\nQ0RERMQ0LL4QvURlsVjw+XxqTTWCYG2PdC3BlkuwxXuVWlMFZl61phKZ2K7WLdd9PdQLNhERERGz\nG6lu0ReZiYiIiJicCjYRERERk1PBJiIiImJyKthERERETE4Fm4iIiIjJqWATERERMTkVbCIiIiIm\np4JNRERExOTCxzuAQFKXg/8TrN+qP1qByi/Y1m24eIMtl0DSWph7DUKhe4WZ1zfQJkLu49GlI6QL\ntguXLmC/1T7eYZjCkXNHQnotApVfsK3bcPEGWy6BpLUw9xqMV2xGzmvm9Q20iZC7t8t70+fULVER\nERERkwtIwbZx40aSk5NJSUkhPz+fvr4+3G43WVlZOBwOMjMzaWpqAqCiogKHw+H/CQsLo6WlBYDV\nq1djs9mYMmXKoOP39fXx2GOPER8fz9y5c3nvvfcCkYaIiIiIKRhesHm9XrZv305zczOtra309/fj\ncrkoKytjw4YNeDwe1q9fT2lpKQAFBQV4PB48Hg/l5eXMnj2b1NRUABYvXozb7R4yx44dO4iNjaWt\nrY3nn3+esrIyo9MQERERMQ3DC7bo6GgiIiLo6enhypUr9PT0MH36dOLi4uju7gagq6sLq9U6ZOzu\n3bvJy8vzb2dlZREXFzdkv+rqap544gkAvv3tb1NTU2N0GiIiIiKmYfiHDmJiYli1ahU2m43IyEiy\ns7NZsGABCQkJzJ8/n5KSEgYGBjh+/PiQsZWVlVRXV484R2dnJzNnzvw0gfBwpk6dyh//+EdiYmIG\n7XfKdQrvZC8A9nQ79nT7mPMTERERGava2lpqa2tHvb/hBVt7ezubN2/G6/UydepUli5dSkVFBTt3\n7mTLli3k5ORQVVVFUVERBw8e9I9rbGwkKiqKpKQkw2JJz0sP+U+qiIiISPBxOp04nU7/9rp164bd\n3/BboidOnGDevHnExsYSHh7OkiVLOHr0KG63m5ycHAByc3OHPJvmcrnIz88f1RxWq5Vz584BcOXK\nFbq7u4dcXRMREREJFYYXbImJiTQ0NNDb24vP56OmpoakpCTi4+Opq6sD4PDhwyQkJPjHDAwMUFVV\nNej5teE88sgj7Nq1C4DXX3+db37zm0anISIiImIaht8STUtLo7CwkDlz5jBp0iQyMjIoLi5m7ty5\nrFy5kr6+PiIjI9m2bZt/TH19PTabDbvdPuhYpaWl7Nmzh97eXmbOnMlTTz3FmjVrWLFiBX//939P\nfHw8sbGxuFwuo9MQERERMQ2Lz+fzXe/FV199lS1btnDmzBkAkpKS+O53v+v/hKaZWSwWyg6WqTXV\n/xfqrULUmupTak01OloLc6+BWlMFt4mQeyBaU1ksFoYpya5/hW3Xrl288sorvPzyyzgcDnw+Hx6P\nhxdeeAGLxUJhYaGhgQbCze7zJSIiIhII173Cdu+99+JyuZg1a9ag33u9Xh577DEaGxtvSoA3aqRK\nVURERMQsRqpbrvuhg4sXLw4p1gDsdjsXL140JjoRERERGdF1C7bJkydfd9Bwr4mIiIiIsa57SzQy\nMpK77rrrmoPa29vp6ekJaGBjpVuiIiIiEixu+EMH77zzTkACEhEREZHPZ9iv9QhmusImIiIiweKG\nr7B94QtfwGKxXPegf/rTn8YenYiIiIiMSFfYRERERMbZDV9h6+3t5Re/+AXt7e2kpKSwYsUKwsMN\n72QVUC8eetE0nQ4mwjc/j8TMa2Dm2EbLqBwCvRY3a61D4W961VhzCaW1GM5EyfOqQOQbzGsYiO4D\nZnLdCuyJJ57glltuYf78+bz55pucPn2aV1555WbGNmYXLl3Afqt9vMMA4Mi5I6aJZbyYeQ3MHNto\nGZVDoNfiZq11KPxNrxprLqG0FsOZKHleFYh8g3kNvV3e8Q4hoIb9lGhraysATz75JJmZmTctKBER\nERH5P9f94tzP3v78vLdCN27cSHJyMikpKeTn59PX14fb7SYrKwuHw0FmZiZNTU0AVFRU4HA4/D9h\nYWG0tLQAcPLkSVJSUoiPj+e5557zH/+9997jm9/8JmlpaXz961+ns7Pzc8UnIiIiEkyuW7C1tLQw\nZcoU/09ra6v/39HR0dc9oNfrZfv27TQ3N9Pa2kp/fz8ul4uysjI2bNiAx+Nh/fr1lJaWAlBQUIDH\n48Hj8VBeXs6sWbNITU0F4Omnn2bHjh20tbXR1tbGgQMHACgpKWH58uW8/fbbrFmzhpdeesnINRER\nERExlesWbP39/Vy8eNH/c+XKFf+/h/tKj+joaCIiIujp6eHKlSv09PQwffp04uLi6O7uBqCrqwur\n1Tpk7O7du1m2bBkA58+f5+LFi2RlZQFQWFjIvn37gE9v137jG98AwOl08sYbb9xg+iIiIiLmZ/jH\nPmNiYli1ahU2m43IyEiys7NZsGABCQkJzJ8/n5KSEgYGBjh+/PiQsZWVlVRXVwPQ2dnJjBkz/K9Z\nrVb/rc+0tDT27t3Ls88+y69+9SsuXrzIRx99xG233TboeKdcp/BO9gJgT7djT7cbna6IiIjI51Zb\nW0ttbe2o9ze8YGtvb2fz5s14vV6mTp3K0qVLqaioYOfOnWzZsoWcnByqqqooKiri4MGD/nGNjY1E\nRUWRlJQ04hw/+clP+M53vsOrr77KV7/6VaxWK2FhYUP2S89LD9pPu4iIiEjocjqdOJ1O//a6deuG\n3d/wgu3EiRPMmzeP2NhYAJYsWcLRo0dxu90cOnQIgNzcXJ588slB41wuF/n5+f5tq9VKR0eHf7uj\no8N/G3XatGns3bsXgEuXLrF3795hn6sTERERCWbXfYbtRiUmJtLQ0EBvby8+n4+amhqSkpKIj4+n\nrq4OgMOHD5OQkOAfMzAwQFVVFXl5ef7fTZs2jejoaBobG/H5fJSXl/Poo48C8L//+78MDAwAn34i\ndcWKFUanISIiImIahl9hS0tLo7CwkDlz5jBp0iQyMjIoLi5m7ty5rFy5kr6+PiIjI9m2bZt/TH19\nPTabDbvdPuhYW7duZfny5fT29rJo0SIWLlwIfHrf96WXXsJisfC1r32Nf/u3fzM6DRERERHTCOle\nomUHy9SaykTMvAZmjm201JpqfOa5GdSaanQmSp5XqTXVYMHemmqkXqIhXbCFaGoiIiISYkaqWwx/\nhk1EREREjKWCTURERMTkVLCJiIiImJwKNhERERGTU8EmIiIiYnIq2ERERERMTgWbiIiIiMmpYBMR\nERExOcNbU5nJ8n3LxzsE0zLzt1nfaGxmzul6gjFmI03k/G9m7sG8zsEc+1iZMfdAxHTod4d4cPaD\nhh7zLwV7FwQI8YLNfqt9vEMwrSPnjph2fW40NjPndD3BGLORJnL+NzP3YF7nYI59rMyYeyBi6vmk\nJ+B5eru8AT3+zaBboiIiIiImF5CCbePGjSQnJ5OSkkJ+fj59fX243W6ysrJwOBxkZmbS1NQEQEVF\nBQ6Hw/8TFhZGS0sLACdPniQlJYX4+Hiee+45//HPnj3LAw88gMPhIC0tjV//+teBSENERETEFAwv\n2LxeL9u3b6e5uZnW1lb6+/txuVyUlZWxYcMGPB4P69evp7S0FICCggI8Hg8ej4fy8nJmzZpFamoq\nAE8//TQ7duygra2NtrY2Dhw4AMAPf/hDHn/8cTweDy6Xi2eeecboNERERERMw/Bn2KKjo4mIiKCn\np4ewsDB6enqYPn06cXFxdHd3A9DV1YXVah0ydvfu3SxbtgyA8+fPc/HiRbKysgAoLCxk3759LFy4\nkGnTpo14LIDaV2v9/7an27Gn2w3MVEREROTG1NbWUltbO+r9DS/YYmJiWLVqFTabjcjISLKzs1mw\nYAEJCQnMnz+fkpISBgYGOH78+JCxlZWVVFdXA9DZ2cmMGTP8r1mtVjo7OwF46aWXuO+++/jpT3/K\n5cuXqampuWYszuVOo9MTERERGTOn04nT6fRvr1u3btj9Db8l2t7ezubNm/F6vXzwwQdcunSJiooK\nVqxYwZYtWzh37hybNm2iqKho0LjGxkaioqJISkoacY7vfe97PPnkk7z//vu8+eabPP7440anISIi\nImIahhdsJ06cYN68ecTGxhIeHs6SJUs4evQobrebnJwcAHJzc3G73YPGuVwu8vPz/dtWq5WOjg7/\ndkdHh/+K27Fjx/i7v/s7AObOncuf//xnPvzwQ6NTERERETEFwwu2xMREGhoa6O3txefzUVNTQ1JS\nEvHx8dTV1QFw+PBhEhIS/GMGBgaoqqoiLy/P/7tp06YRHR1NY2MjPp+P8vJyFi9e7J/j0KFDALzz\nzjv8+c9/5vbbbzc6FRERERFTMPwZtrS0NAoLC5kzZw6TJk0iIyOD4uJi5s6dy8qVK+nr6yMyMpJt\n27b5x9TX12Oz2bDb7YOOtXXrVpYvX05vby+LFi1i4cKFAPz4xz9mxYoVbNq0CYvFwq5du4xOQ0RE\nRMQ0LD6fzzfeQQSCxWLhiV89Md5hmJYZW55cpdZUE8dEzl+tqUYnmGMfKzPmrtZUgWOxWBiuJAvp\ngi1EUxMREZEQM1LdotZUIiIiIiangk1ERETE5FSwiYiIiJicCjYRERERk1PBJiIiImJyKthERERE\nTE4Fm4iIiIjJqWATERERMTnDW1OZyfJ9ywN6fDN+C/VnmT2+QDrz4RmAkM9/Iv+Nr2WirYcZ8jVD\nDEYJpVw+KxjyCmSMZsz/RjovhHTBZr/VHtDjHzl3JOBzjIXZ4wukI+eOAIE/B8bbRP4bX8tEWw8z\n5GuGGIwSSrl8VjDkFcgYzZi/t8v7ucfolqiIiIiIyQWkYNu4cSPJycmkpKSQn59PX18fbrebrKws\nHA4HmZmZNDU1AVBRUYHD4fD/hIWF0dLSAsDJkydJSUkhPj6e5557zn/8733ve/79v/zlL3PbbbcF\nIg0RERERUzC8YPN6vWzfvp3m5mZaW1vp7+/H5XJRVlbGhg0b8Hg8rF+/ntLSUgAKCgrweDx4PB7K\ny8uZNWsWqampADz99NPs2LGDtrY22traOHDgAAAvv/yyf8x3v/tdvv3tbxudhoiIiIhpGP4MW3R0\nNBEREfT09BAWFkZPTw/Tp08nLi6O7u5uALq6urBarUPG7t69m2XLlgFw/vx5Ll68SFZWFgCFhYXs\n27ePhQsXDhmzYcOGa8ZS+2qt/9/2dDv2dLsBGYqIiIiMTW1tLbW1taPe3/CCLSYmhlWrVmGz2YiM\njCQ7O5sFCxaQkJDA/PnzKSkpYWBggOPHjw8ZW1lZSXV1NQCdnZ3MmDHD/5rVaqWzs3PQ/u+99x5e\nr5dvfOMb14zFudxpXGIiIiIiBnE6nTidTv/2unXrht3f8Fui7e3tbN68Ga/XywcffMClS5eoqKhg\nxYoVbNmyhXPnzrFp0yaKiooGjWtsbCQqKoqkpKRRz+VyuVi6dCkWi8XoNERERERMw/CC7cSJE8yb\nN4/Y2FjCw8NZsmQJR48exe12k5OTA0Bubi5ut3vQOJfLRX5+vn/barXS0dHh3+7o6BhyG/W1117z\n30IVERERCVWGF2yJiYk0NDTQ29uLz+ejpqaGpKQk4uPjqaurA+Dw4cMkJCT4xwwMDFBVVUVeXp7/\nd9OmTSM6OprGxkZ8Ph/l5eU8+uij/tfPnDnDRx99xNy5c41OQURERMRUDH+GLS0tjcLCQubMmcOk\nSZPIyMiguLiYuXPnsnLlSvr6+oiMjGTbtm3+MfX19dhsNux2+6Bjbd26leXLl9Pb28uiRYsGfeBA\nV9dERERkorD4fD7feAcRCBaLhSd+9URA5zBju4vPMnt8gaTWVBPTRFsPM+RrhhiMEkq5fFYw5KXW\nVJ/WLcOVZCFdsIVoaiIiIhJiRqpb1JpKRERExORUsImIiIiYnAo2EREREZNTwSYiIiJicirYRERE\nRExOBZuIiIiIyalgExERETE5FWwiIiIiJmd4ayozefHQi1y4dOGmzWfGb1MeTxNxPYIt52CLdyzG\nkmsorVOw5hKscQ8nUDlNtK4BgWSmfEO6YLtw6QL2W+03bb4j547c1PnMbiKuR7DlHGzxjsVYcg2l\ndQrWXII17uEEKqdArlUo/h2GY6Z8dUtURERExOQCUrBt3LiR5ORkUlJSyM/Pp6+vD7fbTVZWFg6H\ng8zMTJqamgCoqKjA4XD4f8LCwmhpaQHg5MmTpKSkEB8fz3PPPTdojsrKSpKTk7nnnnsoKCgIRBoi\nIiIipmB4web1etm+fTvNzc20trbS39+Py+WirKyMDRs24PF4WL9+PaWlpQAUFBTg8XjweDyUl5cz\na9YsUlNTAXj66afZsWMHbW1ttLW1ceDAAQDa2tr40Y9+xLFjx/jNb37DK6+8YnQaIiIiIqZh+DNs\n0dHRRERE0NPTQ1hYGD09PUyfPp24uDi6u7sB6Orqwmq1Dhm7e/duli1bBsD58+e5ePEiWVlZABQW\nFrJv3z4WLlzI9u3b+c53vsPUqVMBuP32268ZyynXKbyTvQDY0+3Y0+0GZysiIiLy+XlPefGe8o56\nf8MLtpiYGFatWoXNZiMyMpLs7GwWLFhAQkIC8+fPp6SkhIGBAY4fPz5kbGVlJdXV1QB0dnYyY8YM\n/2tWq5XOzk7g0ytsFouF+fPn09/fz9q1a8nOzh5yvPS8dNM8LCgiIiJy1V9eSKrbVTfs/obfEm1v\nb2fz5s14vV4++OADLl26REVFBStWrGDLli2cO3eOTZs2UVRUNGhcY2MjUVFRJCUljTjHJ598wtmz\nZ6mrq2PPnj089dRT/qt3IiIiIqHG8ILtxIkTzJs3j9jYWMLDw1myZAlHjx7F7XaTk5MDQG5uLm63\ne9A4l8tFfn6+f9tqtdLR0eHf7ujo8F9xmzlzJg8//DBhYWHY7XYSEhI4e/as0amIiIiImILhBVti\nYiINDQ309vbi8/moqakhKSmJ+Ph46uo+vdx3+PBhEhIS/GMGBgaoqqoiLy/P/7tp06YRHR1NY2Mj\nPp+P8vJyFi9eDMCjjz5KbW0tAB9++CG//e1vmT17ttGpiIiIiJiC4c+wpaWlUVhYyJw5c5g0aRIZ\nGRkUFxczd+5cVq5cSV9fH5GRkWzbts0/pr6+HpvNht1uH3SsrVu3snz5cnp7e1m0aBELFy4EIDs7\nm//+7/9kH5imAAAG7ElEQVQmOTmZsLAwfvKTn3DbbbcZnYqIiIiIKVh8Pp9vvIMIBIvFQtnBMrWm\nGkcTcT2CLedgi3cs1JrqU8GaS7DGPRy1pjK/m5nvrpxdDFeShXTBFqKpyU1SW1uL0+kc7zAkSOn8\nkbHSOTSxjFS3qDWVyHVcfU5S5Ebo/JGx0jkkn6WCTURERMTkVLCJiIiImFxIP8MmIiIiEiyGK8kM\n/1oPswjROlREREQmIN0SFRERETE5FWwiIiIiJqeCTURERMTkgr5gO3DgAImJicTHx/Mv//Iv19zn\n2WefJT4+nrS0NDwez02OUMxupHOotraWqVOn4nA4cDgc/PCHPxyHKMWsioqK+NKXvkRKSsp199F7\nkFzPSOeP3n/EzxfErly54rvzzjt97777ru/jjz/2paWl+U6fPj1on/379/seeughn8/n8zU0NPju\nvffe8QhVTGo059Bbb73le/jhh8cpQjG7+vp6X3Nzs++ee+655ut6D5LhjHT+6P1HrgrqK2xut5u7\n7roLu91OREQEeXl5vPHGG4P2qa6u5oknngDg3nvvpauri9///vfjEa6Y0GjOIdCnjuX6HnjgAW67\n7bbrvq73IBnOSOcP6P1HPhXUBVtnZyczZ870b8+YMYPOzs4R9+no6LhpMYq5jeYcslgsHDt2jLS0\nNBYtWsTp06dvdpgSxPQeJGOh9x+5Kqi/h220X477l/870ZfqylWjORcyMjJ4//33iYqK4te//jWP\nPvoov/3tb29CdBIq9B4kN0rvP3JVUF9hs1qtvP/++/7t999/nxkzZgy7T0dHB1ar9abFKOY2mnNo\nypQpREVFAfDQQw/xySef8Mc//vGmxinBS+9BMhZ6/5GrgrpgmzNnDm1tbXi9Xj7++GNee+01Hnnk\nkUH7PPLII/z7v/87AA0NDdx666186UtfGo9wxYRGcw79/ve/918hcbvd+Hw+YmJixiNcCUJ6D5Kx\n0PuPXBXUt0TDw8P52c9+RnZ2Nv39/axYsYK7776bX/7ylwAUFxezaNEi3nzzTe666y7++q//mp07\nd45z1GImozmHXn/9dX7+858THh5OVFQULpdrnKMWM1m2bBl1dXV8+OGHzJw5k3Xr1vHJJ58Aeg+S\nkY10/uj9R64K2ebvIiIiIqEiqG+JioiIiEwEKthERERETE4Fm4iIiIjJqWATERERMTkVbCIiAVRb\nW8vDDz883mGISJBTwSYiIiJicirYRCSkXb58mW9961ukp6eTkpJCZWUlJ0+exOl0MmfOHBYuXMiF\nCxcAOHv2LA8++CDp6el85Stf4d133wXghRdeICUlhdTUVCorK4FPr5w5nU6WLl3K3XffzeOPP+6f\n88CBA9x999185Stf4Ve/+pX/93V1dTgcDhwOBxkZGVy6dOkmroSIBLOg/uJcEZGRHDhwAKvVyv79\n+wH405/+xEMPPUR1dTWxsbG89tprrF69mh07dlBQUMD3v/99Fi9ezMcff0x/fz979+7l7bffpqWl\nhT/84Q9kZmby1a9+FYBTp05x+vRppk2bxv3338+xY8fIyMjgH/7hH3jrrbe48847eeyxx/y9Q//1\nX/+VrVu3ct9999HT08Nf/dVfjdu6iEhw0RU2EQlpqampHDx4kBdffJEjR45w7tw5fvOb3/Dggw/i\ncDj4p3/6Jzo7O7l06RIffPABixcvBuCWW24hMjKSo0ePkp+fj8Vi4Ytf/CJf+9rXaGpqwmKxkJWV\nxfTp07FYLKSnp/Puu+9y5swZZs2axZ133gnA448/7m8tdP/99/P888/z05/+lI8++oiwsLBxWxcR\nCS66wiYiIS0+Ph6Px8P+/fv5wQ9+wNe//nWSk5M5duzYoP0uXrx43WP8ZUOYq1fMPnuFLCwsjCtX\nrvhfu9bYsrIy/vZv/5b9+/dz//3381//9V98+ctfvuHcRGTi0BU2EQlp58+fZ/LkyRQUFFBSUoLb\n7ebDDz+koaEBgE8++YTTp08zZcoUZsyYwRtvvAFAX18fvb29PPDAA7z22msMDAzwhz/8gfr6erKy\nsoYUcfBpIZeYmIjX6+V3v/sdAHv27PG/3t7eTnJyMqWlpWRmZvI///M/N2EFRCQU6AqbiIS01tZW\nXnjhBSZNmsQtt9zCz3/+c8LCwnj22Wfp7u7mypUrPP/88yQlJVFeXk5xcTFr1qwhIiKC119/nZyc\nHI4fP05aWhoWi4Uf//jHfPGLX+Sdd94ZcjUNPr3qtm3bNr71rW8RFRXFAw88wOXLlwF45ZVXeOut\nt5g0aRL33HMPDz300M1eDhEJUmr+LiIiImJyuiUqIiIiYnIq2ERERERMTgWbiIiIiMmpYBMREREx\nORVsIiIiIiangk1ERETE5P4fUPipv8DkdaQAAAAASUVORK5CYII=\n" } ], "prompt_number": 48 }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are more views to engine pools, one of them is the `load_balanced_view` which automatically helps utilize engines which normally would sit aroudn idly waiting for other engines to finish." ] }, { "cell_type": "code", "collapsed": false, "input": [ "lbview = cli.load_balanced_view()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 49 }, { "cell_type": "code", "collapsed": false, "input": [ "@lbview.parallel(block=True)\n", "def lb_task(delay):\n", " import os, time\n", " \n", " t0 = time.time()\n", " pid = os.getpid()\n", " time.sleep(delay)\n", " t1 = time.time()\n", " \n", " return [pid, t0, t1]" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 50 }, { "cell_type": "code", "collapsed": false, "input": [ "result = lb_task.map(delays)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 51 }, { "cell_type": "code", "collapsed": false, "input": [ "visualize_tasks(result)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAmwAAADOCAYAAACD3l7pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9w1PWdx/HnksSaHAQlXrsQiIuaFBOTsCGJEWO7bWGC\n9BSDcGLiRQb0ckqrY8EFa8fjR+/gaFWkPdrCMOjlAmuiU8yNljt+mGSOkCyQxaSjXmPOFRNNp16F\nEpNGSfb+cHabkEB+8N3sd5fXYyYz+e5+P9/v+/35frK8+X6/+/1YfD6fDxERERExrQmhDkBERERE\nLk0Fm4iIiIjJqWATERERMTkVbCIiIiImFx3qAILFYrGEOgQRERGREbvU90AjtmCDSycucinr169n\n/fr1oQ5DwpjGkFwujaEry3AnmnRJVERERMTkVLCJiIiImJwKNpEhOByOUIcgYU5jSC6XxpD0Z4nU\nmQ4sFovuYRMREZGwMFzdEtFfOvBbd2gdHZ0doQ7DNKwTrWyZt+WK75dw7YfxiDtc+2Yo1olWgLDP\nYzhXSp4QnFzDbcxHwvEOtz43gj/nsbgiCraOzg5s19hCHYZpeM94AfVLuPbDeMQdrn0zFH8u4Z7H\ncK6UPCE4uYbbmI+E4x1ufW4Ef85jEZR72DZv3kxaWhrp6ekUFRXR09OD2+0mNzcXu91OTk4Ox48f\nB6C8vBy73R74iYqKoqmpCYCnn36apKQkJk2aNGD7v/zlL8nIyMBut3Pbbbfx1ltvBSMNEREREVMw\nvGDzer3s2rWLxsZGmpub6e3txeVysXbtWjZt2oTH42Hjxo04nU4AiouL8Xg8eDweysrKuOGGG8jI\nyABg0aJFuN3uQfsoLi6mqakJj8fDD3/4Q1avXm10GiIiIiKmYfgl0fj4eGJiYujq6iIqKoquri6m\nTZuG1Wrl7NmzAJw5c4bExMRBbffu3cuyZcsCy7m5uUPuo/8Zt87OTq677jqDsxARERExD8MLtilT\nprB69WqSkpKIjY2loKCA+fPnk5KSQn5+PmvWrKGvr49jx44NaltRUUFVVdWI9rNjxw6ee+45Pvvs\nM+rq6oZcx/+E6FPvnoI8sM22jTUtEREREcNUV1dTXV094vUNL9haW1vZtm0bXq+XyZMns3TpUsrL\ny9mzZw/bt2+nsLCQyspKVqxYwcGDBwPtGhoaiIuLIzU1dUT7efTRR3n00UfZt28fK1as4M033xy0\njr9g8+73XjE3NIqIiIj5ORyOAc/a27BhwyXXN/wethMnTjB37lwSEhKIjo5m8eLFHD16FLfbTWFh\nIQBLliwZdG+ay+WiqKho1Pu77777aGxsNCR2ERERETMyvGCbNWsW9fX1dHd34/P5OHz4MKmpqSQn\nJ1NTUwPAkSNHSElJCbTp6+ujsrJywP1rl/Lee+8Ffn/99dcDX1IQERERiUSGXxLNzMykpKSE7Oxs\nJkyYQFZWFqWlpeTl5bFq1Sp6enqIjY1l586dgTa1tbUkJSVhs9kGbMvpdLJv3z66u7uZMWMGDz/8\nMM888ww///nPOXToEDExMfz1X/81e/bsMToNEREREdMIyoNznU5n4LEdftnZ2TQ0NAy5vsPhGPKL\nA1u3bmXr1q2DXt+2bduo4rFOtF7Ww+oijf8J2Vd6v4RrP4xH3OHaN0Px5xLueQznSskTgpNruI35\nSDje4dbnRvDnPBaaS1REREQkxIarW4Iy04GIiIiIGEcFm4iIiIjJqWATERERMTkVbCIiIiImp4JN\nRERExORUsImIiIiYnAo2EREREZNTwSYiIiJicirYREREREwuKFNTmcXy/cuDun3rRCtb5m1h3aF1\ndHR2BHVfoeDPDxj3HIO570g/bpcSqmM6nn1+JR/foag/RiYU/eSfpkjHZbD+n1XypYgu2GzX2IK6\nff/cZx2dHUHfVyj0n9ttvHMM5r4j/bhdSqiO6Xj2+ZV8fIei/hiZUPSTf586LoNdKXOLjkZQLolu\n3ryZtLQ00tPTKSoqoqenB7fbTW5uLna7nZycHI4fPw5AeXk5drs98BMVFUVTUxMATz/9NElJSUya\nNGnA9p977jnS0tLIzMxk3rx5nD59OhhpiIiIiJiC4QWb1+tl165dNDY20tzcTG9vLy6Xi7Vr17Jp\n0yY8Hg8bN27E6XQCUFxcjMfjwePxUFZWxg033EBGRgYAixYtwu12D9pHVlYWJ0+e5K233mLJkiWB\nbYmIiIhEIsMLtvj4eGJiYujq6uL8+fN0dXUxbdo0rFYrZ8+eBeDMmTMkJiYOart3716WLVsWWM7N\nzcVqtQ5az+FwcPXVVwNw66230tbWZnQaIiIiIqZh+D1sU6ZMYfXq1SQlJREbG0tBQQHz588nJSWF\n/Px81qxZQ19fH8eOHRvUtqKigqqqqlHtb/fu3SxcuHDI96pfrA78bpttwzbbNqpti4iIiARDdXU1\n1dXVI17f8IKttbWVbdu24fV6mTx5MkuXLqW8vJw9e/awfft2CgsLqaysZMWKFRw8eDDQrqGhgbi4\nOFJTU0e8r3//93+nsbGR559/fsj3Hcsdl5uOiIiIiOEcDgcOhyOwvGHDhkuub/gl0RMnTjB37lwS\nEhKIjo5m8eLFHD16FLfbTWFhIQBLliwZdG+ay+WiqKhoxPs5dOgQ//zP/0xVVRUxMTGG5iAiIiJi\nJoYXbLNmzaK+vp7u7m58Ph+HDx8mNTWV5ORkampqADhy5AgpKSmBNn19fVRWVg64f+1SPB4P//AP\n/8B//Md/cN111xmdgoiIiIipGH5JNDMzk5KSErKzs5kwYQJZWVmUlpaSl5fHqlWr6OnpITY2lp07\ndwba1NbWkpSUhM1mG7Atp9PJvn376O7uZsaMGTz88MM888wzOJ1OPvvsM5YsWQLA9ddfz/79+41O\nRURERMQUgvLgXKfTOehRG9nZ2TQ0NAy5vsPhoK6ubtDrW7duZevWrYNe73/v26UE+8F7/qdUWyda\nI/Ihf/78/L+PZ47B3HekH7dLCdUxHc8+v5KP71DUHyMTin7y71PHZbD+n1XyJYvP5/OFOohgsFgs\nRGhqIiIiEmGGq1s0+buIiIiIyalgExERETE5FWwiIiIiJqeCTURERMTkVLCJiIiImJwKNhERERGT\nU8EmIiIiYnIq2ERERERMLigzHZjFukPr6OjsCHUYIWGdaGXLvC2B5bH2hX874dyXocqh/zEI1r4j\n4fiMVrByHo/jFSz+p8KHU8xGGu2YiNS/m3AcB+EYc6hEdMHW0dmB7RpbqMMIiQunOhlrX/i3E859\nGaoc+h+DYO07Eo7PaAUr5/E4XsHijz2cYjbSaMdEpP7dhOM4CMeYQ0WXREVERERMLigF2+bNm0lL\nSyM9PZ2ioiJ6enpwu93k5uZit9vJycnh+PHjAJSXl2O32wM/UVFRNDU1AfD000+TlJTEpEmTBmy/\ntraWrKwsYmJiePXVV4ORgoiIiIhpGF6web1edu3aRWNjI83NzfT29uJyuVi7di2bNm3C4/GwceNG\nnE4nAMXFxXg8HjweD2VlZdxwww1kZGQAsGjRItxu96B9XH/99bz00ksUFRUZHb6IiIiI6Rh+D1t8\nfDwxMTF0dXURFRVFV1cX06ZNw2q1cvbsWQDOnDlDYmLioLZ79+5l2bJlgeXc3Nwh93H99dcDMGGC\nruiKiIhI5DO8YJsyZQqrV68mKSmJ2NhYCgoKmD9/PikpKeTn57NmzRr6+vo4duzYoLYVFRVUVVUZ\nFssp1ym8V3sBsM22YZttM2zbIiIiImPlPeXFe8o74vUNL9haW1vZtm0bXq+XyZMns3TpUsrLy9mz\nZw/bt2+nsLCQyspKVqxYwcGDBwPtGhoaiIuLIzU11bBYZi+brW+eiIiIiOlceCKp5qWaS65v+DXF\nEydOMHfuXBISEoiOjmbx4sUcPXoUt9tNYWEhAEuWLBl0b5rL5RrTPWkWi8WQuEVERETMyvCCbdas\nWdTX19Pd3Y3P5+Pw4cOkpqaSnJxMTc2X1eORI0dISUkJtOnr66OysnLA/Wsj4fP58Pl8hsYvIiIi\nYjaGF2yZmZmUlJSQnZ1NRkYGPp+P0tJSfvWrX+F0Opk9ezY/+tGP2LlzZ6BNbW0tSUlJ2Gy2Adty\nOp3MmDGD7u5uZsyYwcaNGwE4fvw4M2bM4JVXXqG0tJT09HSj0xARERExjaDMdOB0OgOP7fDLzs6m\noaFhyPUdDgd1dXWDXt+6dStbt24d9HpOTg4ffvjhsHFYJ1oHPfH/SuGf7qP/8lj6wr+dcO7LUOXQ\n/xgEa9+RcHxGK1g5j8fxChZ/7OEUs5FGOyYi9e8mHMdBOMYcKhZfhF5TtFgsulwqIiIiYWG4ukUP\nMhMRERExORVsIiIiIiangk1ERETE5FSwiYiIiJicCjYRERERk1PBJiIiImJyKthERERETE4Fm4iI\niIjJBWWmA7NZd2gdHZ0dIY3BOtHKlnlbTBHLcPyxgjn6LhQu93j170OIzH4crzEdTn87Rgh1vqHe\n/1iNNW7/k/bDKdfLEa7H90IXfsZeCa6Igq2jswPbNbaQxuCfdsMMsQyn/xQh4RBvMFzu8bpwmpVI\n7MfxGtPh9LdjhFDnG+r9j9VY4/a3C6dcL0e4Ht8LXYlTWemSqIiIiIjJBaVg27x5M2lpaaSnp1NU\nVERPTw9ut5vc3Fzsdjs5OTkcP34cgPLycux2e+AnKiqKpqYmAJ5++mmSkpKYNGnSgO339PRw3333\nkZycTF5eHh988EEw0hARERExBcMLNq/Xy65du2hsbKS5uZne3l5cLhdr165l06ZNeDweNm7ciNPp\nBKC4uBiPx4PH46GsrIwbbriBjIwMABYtWoTb7R60j927d5OQkEBLSwtPPPEEa9euNToNEREREdMw\nvGCLj48nJiaGrq4uzp8/T1dXF9OmTcNqtXL27FkAzpw5Q2Ji4qC2e/fuZdmyZYHl3NxcrFbroPWq\nqqp48MEHAbj33ns5fPiw0WmIiIiImIbhXzqYMmUKq1evJikpidjYWAoKCpg/fz4pKSnk5+ezZs0a\n+vr6OHbs2KC2FRUVVFVVDbuP9vZ2ZsyY8WUC0dFMnjyZP/7xj0yZMmXAeuvXrwfg1LunIA9ss22X\nnZ+IiIjI5aqurqa6unrE6xtesLW2trJt2za8Xi+TJ09m6dKllJeXs2fPHrZv305hYSGVlZWsWLGC\ngwcPBto1NDQQFxdHamqqYbH4Czbvfm9YfxtGREREIovD4cDhcASWN2zYcMn1Db8keuLECebOnUtC\nQgLR0dEsXryYo0eP4na7KSwsBGDJkiWD7k1zuVwUFRWNaB+JiYmcPn0agPPnz3P27NlBZ9dERERE\nIoXhBdusWbOor6+nu7sbn8/H4cOHSU1NJTk5mZqaGgCOHDlCSkpKoE1fXx+VlZUD7l+7lLvvvpuX\nXnoJgFdeeYXvfOc7RqchIiIiYhqGXxLNzMykpKSE7OxsJkyYQFZWFqWlpeTl5bFq1Sp6enqIjY1l\n586dgTa1tbUkJSVhs9kGbMvpdLJv3z66u7uZMWMGDz/8MM888wwrV67k7/7u70hOTiYhIQGXy2V0\nGiIiIiKmccmC7cUXX2T79u28++67AKSmpvL9738/8A3Ni3E6nYHHdvhlZ2fT0NAw5PoOh4O6urpB\nr2/dupWtW7cOev0rX/kKFRUVl4yhP+tEa8ifiuyf/sQMsQzHH6v/d7PHGwyXe7z69+HlbMfMxmtM\nh9PfjhFCnW+o9z9WY43b3y6ccr0c4Xp8L3ThZ+yVwOLz+XxDvfHSSy+xbds2nnvuOex2Oz6fD4/H\nw5NPPsnjjz9OSUnJeMc6KhaLhYukJiIiImIqw9UtFy3Ybr31VlwuFzNnzhzwutfr5b777rvo2TKz\nUMEmIiIi4WK4uuWiXzo4d+7coGINwGazce7cOWOiExEREZFhXbRgu/rqqy/a6FLviYiIiIixLnpJ\nNDY2lptuumnIRq2trXR1dQU1sMulS6IiIiISLoarWy76LdF33nknKAGJiIiIyOhc9AxbuNMZNhER\nEQkXYz7DNnHiRCwWy0U3+qc//enyoxMRERGRYekMm4iIiEiIjfkMW3d3N7/85S9pbW0lPT2dlStX\nEh1t+ExWQbfu0Do6OjtCHUZQWSda2TJvyxWR61ipj4amfvmL8egL/9PZ1dfjM+5G09+R+rcQzDEX\njuPZf5zD0UUrsAcffJCrrrqK/Px83njjDd5++21eeOGF8YzNEB2dHdiusYU6jKDyTy9yJeQ6Vuqj\noalf/mI8+sK/D/W1Fwj+uBtNf0fq30Iwx1w4judwno7rkt8SbW5uBuChhx4iJydn3IISERERkb+4\n6INz+1/+HO2l0M2bN5OWlkZ6ejpFRUX09PTgdrvJzc3FbreTk5PD8ePHASgvL8dutwd+oqKiaGpq\nAuDkyZOkp6eTnJzM448/Htj+Bx98wHe+8x0yMzP51re+RXt7+6jiExEREQknFy3YmpqamDRpUuCn\nubk58Ht8fPxFN+j1etm1axeNjY00NzfT29uLy+Vi7dq1bNq0CY/Hw8aNG3E6nQAUFxfj8XjweDyU\nlZUxc+ZMMjIyAHjkkUfYvXs3LS0ttLS0cODAAQDWrFnD8uXLeeutt3jmmWd46qmnjOwTEREREVO5\naMHW29vLuXPnAj/nz58P/H6pR3rEx8cTExNDV1cX58+fp6uri2nTpmG1Wjl79iwAZ86cITExcVDb\nvXv3cv/99wPw8ccfc+7cOXJzcwEoKSlh//79wJeXa7/97W8D4HA4eO2118aYvoiIiIj5Gf61zylT\nprB69WqSkpKIjY2loKCA+fPnk5KSQn5+PmvWrKGvr49jx44NaltRUUFVVRUA7e3tTJ8+PfBeYmJi\n4NJnZmYmr776Ko899hi//vWvOXfuHJ9++inXXnvtgO2tX7+eU++ewnu1F9tsG7bZNqPTFRERERm1\n6upqqqurR7y+4QVba2sr27Ztw+v1MnnyZJYuXUp5eTl79uxh+/btFBYWUllZyYoVKzh48GCgXUND\nA3FxcaSmpg67j5/+9Kd873vf48UXX+Qb3/gGiYmJREVFDVpv/fr1ePd7w+obLCIiIhL5HA4HDocj\nsLxhw4ZLrm94wXbixAnmzp1LQkICAIsXL+bo0aO43W4OHToEwJIlS3jooYcGtHO5XBQVFQWWExMT\naWtrCyy3tbUFLqNOnTqVV199FYDOzk5effXVS95XJyIiIhLOLnoP21jNmjWL+vp6uru78fl8HD58\nmNTUVJKTk6mpqQHgyJEjpKSkBNr09fVRWVnJsmXLAq9NnTqV+Ph4Ghoa8Pl8lJWVcc899wDwf//3\nf/T19QFffiN15cqVRqchIiIiYhqGn2HLzMykpKSE7OxsJkyYQFZWFqWlpeTl5bFq1Sp6enqIjY1l\n586dgTa1tbUkJSVhs9kGbGvHjh0sX76c7u5uFi5cyIIFC4Avr/s+9dRTWCwWvvnNb/Kv//qvRqch\nIiIiYhpBmWvK6XQGHtvhl52dTUNDw5DrOxwO6urqBr0+Z86cwMN7+7v33nu59957RxSLdaI1rJ9s\nPBL+6UGuhFzHSn00NPXLX4xHX/j3ob4en3E3mv6O1L+FYI65cBzP/pjDkSZ/FxEREQmx4eoWw+9h\nExERERFjqWATERERMTkVbCIiIiImp4JNRERExORUsImIiIiYnAo2EREREZNTwSYiIiJicirYRERE\nREwuKDMdmM26Q+vo6OwIdRjjxjrRypZ5WwLLZs7fH6tZY/Q/FduMsY1F/7Fhpj43ahxEwvEy+98E\nhH8/m7GPjYjJjHmNxGjG04X/vl1JroiCraOzA9s1tlCHMW4unCbEzPn7YzVrjP74zBjbWPQfG2bq\nc6PGQSQcL7P/TUD497MZ+9iImMyY10iMZjyF0zRYRtMlURERERGTC0rBtnnzZtLS0khPT6eoqIie\nnh7cbje5ubnY7XZycnI4fvw4AOXl5djt9sBPVFQUTU1NAJw8eZL09HSSk5N5/PHHA9t/7733uOOO\nO7Db7WRmZvKb3/wmGGmIiIiImILhBZvX62XXrl00NjbS3NxMb28vLpeLtWvXsmnTJjweDxs3bsTp\ndAJQXFyMx+PB4/FQVlbGzJkzycjIAOCRRx5h9+7dtLS00NLSwoEDBwD48Y9/zAMPPIDH48HlcvHo\no48anYaIiIiIaRh+D1t8fDwxMTF0dXURFRVFV1cX06ZNw2q1cvbsWQDOnDlDYmLioLZ79+7l/vvv\nB+Djjz/m3Llz5ObmAlBSUsL+/ftZsGABU6dOHXZbAOvXrwfg1LunIA9ss20GZysiIiIyetXV1VRX\nV494fcMLtilTprB69WqSkpKIjY2loKCA+fPnk5KSQn5+PmvWrKGvr49jx44NaltRUUFVVRUA7e3t\nTJ8+PfBeYmIi7e3tADz11FPcdttt/OxnP+Ozzz7j8OHDQ8biL9i8+71hdQOmiIiIRDaHw4HD4Qgs\nb9iw4ZLrG35JtLW1lW3btuH1evnoo4/o7OykvLyclStXsn37dk6fPs3zzz/PihUrBrRraGggLi6O\n1NTUYffxgx/8gIceeogPP/yQN954gwceeMDoNERERERMw/CC7cSJE8ydO5eEhASio6NZvHgxR48e\nxe12U1hYCMCSJUtwu90D2rlcLoqKigLLiYmJtLW1BZbb2toCZ9zq6ur427/9WwDy8vL485//zCef\nfGJ0KiIiIiKmYHjBNmvWLOrr6+nu7sbn83H48GFSU1NJTk6mpqYGgCNHjpCSkhJo09fXR2VlJcuW\nLQu8NnXqVOLj42loaMDn81FWVsaiRYsC+zh06BAA77zzDn/+85+57rrrjE5FRERExBQMv4ctMzOT\nkpISsrOzmTBhAllZWZSWlpKXl8eqVavo6ekhNjaWnTt3BtrU1taSlJSEzWYbsK0dO3awfPlyuru7\nWbhwIQsWLADgJz/5CStXruT555/HYrHw0ksvGZ2GiIiIiGkEZaYDp9MZeGyHX3Z2Ng0NDUOu73A4\nqKurG/T6nDlzaG5uHvT6jTfeOKpvVlgnWq+opyP7p/nov2zW/P2xmjVGf3xmjG0s+o8NM/W5UeMg\nEo6X2f8mIPz72Yx9bERMZsxrJEYzni789+1KYvH5fL5QBxEMFouFCE1NREREIsxwdYumphIREREx\nORVsIiIiIiangk1ERETE5FSwiYiIiJicCjYRERERk1PBJiIiImJyKthERERETE4Fm4iIiIjJBWWm\nA7NYvn95qEO4KOtEK1vmbWHdoXV0dHYEdR9AUPcTLP37CAh6fw2133Drs7EKh5zDIUajBSPnSOzH\nUOfkf/r+aPYd6piHM5acwk3/fyPDQUQXbLZrbKEO4aL8U3B0dHYELc7+03wEcz/B0r+P/MYjj/E4\nNmYTDjmHQ4xGC0bOkdiPoc7Jv//R7DvUMQ9nLDmFm3Cavgt0SVRERETE9IJSsG3evJm0tDTS09Mp\nKiqip6cHt9tNbm4udrudnJwcjh8/DkB5eTl2uz3wExUVRVNTEwAnT54kPT2d5ORkHn/88cD2f/CD\nHwTW//rXv861114bjDRERERETMHwgs3r9bJr1y4aGxtpbm6mt7cXl8vF2rVr2bRpEx6Ph40bN+J0\nOgEoLi7G4/Hg8XgoKytj5syZZGRkAPDII4+we/duWlpaaGlp4cCBAwA899xzgTbf//73uffee41O\nQ0RERMQ0DL+HLT4+npiYGLq6uoiKiqKrq4tp06ZhtVo5e/YsAGfOnCExMXFQ271793L//fcD8PHH\nH3Pu3Dlyc3MBKCkpYf/+/SxYsGBQm02bNg0ZS/WL1YHfbbNt2GbbDMhQRERE5PJUV1dTXV094vUN\nL9imTJnC6tWrSUpKIjY2loKCAubPn09KSgr5+fmsWbOGvr4+jh07NqhtRUUFVVVVALS3tzN9+vTA\ne4mJibS3tw9Y/4MPPsDr9fLtb397yFgcyx3GJSYiIiJiEIfDgcPhCCxv2LDhkusbfkm0tbWVbdu2\n4fV6+eijj+js7KS8vJyVK1eyfft2Tp8+zfPPP8+KFSsGtGtoaCAuLo7U1NQR78vlcrF06VIsFovR\naYiIiIiYhuEF24kTJ5g7dy4JCQlER0ezePFijh49itvtprCwEIAlS5bgdrsHtHO5XBQVFQWWExMT\naWtrCyy3tbUNuoz68ssvBy6hioiIiEQqwwu2WbNmUV9fT3d3Nz6fj8OHD5OamkpycjI1NTUAHDly\nhJSUlECbvr4+KisrWbZsWeC1qVOnEh8fT0NDAz6fj7KyMu65557A+++++y6ffvopeXl5RqcgIiIi\nYiqG38OWmZlJSUkJ2dnZTJgwgaysLEpLS8nLy2PVqlX09PQQGxvLzp07A21qa2tJSkrCZrMN2NaO\nHTtYvnw53d3dLFy4cMAXDnR2TURERK4UQZnpwOl0Bh7b4ZednU1DQ8OQ6zscDurq6ga9PmfOHJqb\nm4ds84//+I/DxmHmpxj7p/2wTrQGLU7/PoK9n2Dp30f9Xwt2HuNxbMwmHHIOhxiNFoycI7EfQ52T\nf/+j2XeoYx7OWHIKN/3/bQkHFp/P5wt1EMFgsViI0NREREQkwgxXt2hqKhERERGTU8EmIiIiYnIq\n2ERERERMTgWbiIiIiMmpYBMRERExORVsIiIiIiangk1ERETE5FSwiYiIiJhcUGY6MJN1h9YFfu/o\n7AhhJMaxTrSyZd4W1h1aZ7qczBzbePL3g1+49ofRxzMSx8dIcvI/UT1Scr5cYx0Hoe7HUO//coQy\n9tEe7/6xXvhZeiWL+IKt/+CwXWMLXSAG8k8V0tHZYbqczBzbeLpwOpdw7Q+jj2ckjo+R5ORfJ1Jy\nvlxjHQeh7sdQ7/9yhDL20R7v/rFG8tRYo6VLoiIiIiImF5SCbfPmzaSlpZGenk5RURE9PT243W5y\nc3Ox2+3k5ORw/PhxAMrLy7Hb7YGfqKgompqaADh58iTp6ekkJyfz+OOPD9hHRUUFaWlp3HLLLRQX\nFwcjDRERERFTMLxg83q97Nq1i8bGRpqbm+nt7cXlcrF27Vo2bdqEx+Nh48aNOJ1OAIqLi/F4PHg8\nHsrKypgQgHzVAAAJsUlEQVQ5cyYZGRkAPPLII+zevZuWlhZaWlo4cOAAAC0tLWzZsoW6ujp++9vf\n8sILLxidhoiIiIhpGH4PW3x8PDExMXR1dREVFUVXVxfTpk3DarVy9uxZAM6cOUNiYuKgtnv37uX+\n++8H4OOPP+bcuXPk5uYCUFJSwv79+1mwYAG7du3ie9/7HpMnTwbguuuuGzKW9evXc+rdUwBYb7Fi\ny7cZna6IiIjIqFVXV1NdXT3i9Q0v2KZMmcLq1atJSkoiNjaWgoIC5s+fT0pKCvn5+axZs4a+vj6O\nHTs2qG1FRQVVVVUAtLe3M3369MB7iYmJtLe3A1+eYbNYLOTn59Pb28v69espKCgYtL3169fj3e81\nOkURERGRy+JwOHA4HIHlDRs2XHJ9wy+Jtra2sm3bNrxeLx999BGdnZ2Ul5ezcuVKtm/fzunTp3n+\n+edZsWLFgHYNDQ3ExcWRmpo67D6++OIL3nvvPWpqati3bx8PP/xw4OydiIiISKQxvGA7ceIEc+fO\nJSEhgejoaBYvXszRo0dxu90UFhYCsGTJEtxu94B2LpeLoqKiwHJiYiJtbW2B5ba2tsAZtxkzZnDX\nXXcRFRWFzWYjJSWF9957z+hUREREREzB8IJt1qxZ1NfX093djc/n4/Dhw6SmppKcnExNTQ0AR44c\nISUlJdCmr6+PyspKli1bFnht6tSpxMfH09DQgM/no6ysjEWLFgFwzz33BK77fvLJJ/zud7/jhhtu\nMDoVEREREVMw/B62zMxMSkpKyM7OZsKECWRlZVFaWkpeXh6rVq2ip6eH2NhYdu7cGWhTW1tLUlIS\nNpttwLZ27NjB8uXL6e7uZuHChSxYsACAgoIC/uu//ou0tDSioqL46U9/yrXXXmt0KiIiIiKmEJSZ\nDpxOZ+CxHX7Z2dk0NDQMub7D4aCurm7Q63PmzKG5uXnINs8++yzPPvvssLH4p7iAwU+fD1f+nKwT\nrabLycyxjaf+486/HI79YfTxjMTxMZKc/OtESs6Xa6zjINT9GOr9X45Qxj7a490/1gs/S69kFp/P\n5wt1EMFgsViI0NRkHFRXVw/49o7IaGkMyeXSGLqyDFe3aGoqkSGM5tk4IkPRGJLLpTEk/algExER\nETE5FWwiIiIiJhfR97CJiIiIhItLlWRB+ZaoGURoHSoiIiJXIF0SFRERETE5FWwiIiIiJqeCTURE\nRMTkwr5gO3DgALNmzSI5OZl/+Zd/GXKdxx57jOTkZDIzM/F4POMcoZjdcGOourqayZMnY7fbsdvt\n/PjHPw5BlGJGK1as4Gtf+xrp6ekXXUefP3Ipw40hff5IgC+MnT9/3nfjjTf63n//fd/nn3/uy8zM\n9L399tsD1nn99dd9d955p8/n8/nq6+t9t956ayhCFZMayRh68803fXfddVeIIhQzq62t9TU2Nvpu\nueWWId/X548MZ7gxpM8f8QvrM2xut5ubbroJm81GTEwMy5Yt47XXXhuwTlVVFQ8++CAAt956K2fO\nnOH3v/99KMIVExrJGAJ961iGdscdd3Dttdde9H19/shwhhtDoM8f+VJYF2zt7e3MmDEjsDx9+nTa\n29uHXaetrW3cYhRzG8kYslgs1NXVkZmZycKFC3n77bfHO0wJU/r8kculzx/xC+vnsI304bgX/u9E\nD9UVv5GMhaysLD788EPi4uL4zW9+wz333MPvfve7cYhOIoE+f+Ry6PNH/ML6DFtiYiIffvhhYPnD\nDz9k+vTpl1ynra2NxMTEcYtRzG0kY2jSpEnExcUBcOedd/LFF1/wxz/+cVzjlPCkzx+5XPr8Eb+w\nLtiys7NpaWnB6/Xy+eef8/LLL3P33XcPWOfuu+/m3/7t3wCor6/nmmuu4Wtf+1oowhUTGskY+v3v\nfx84S+J2u/H5fEyZMiUU4UqY0eePXC59/ohfWF8SjY6O5uc//zkFBQX09vaycuVKbr75Zn71q18B\nUFpaysKFC3njjTe46aab+Ku/+iv27NkT4qjFTEYyhl555RV+8YtfEB0dTVxcHC6XK8RRi1ncf//9\n1NTU8MknnzBjxgw2bNjAF198AejzR0ZmuDGkzx/xi9jJ30VEREQiRVhfEhURERG5EqhgExERETE5\nFWwiIiIiJqeCTURERMTkVLCJiARRdXU1d911V6jDEJEwp4JNRERExORUsIlIRPvss8/47ne/y+zZ\ns0lPT6eiooKTJ0/icDjIzs5mwYIFdHR0APDee+8xb948Zs+ezZw5c3j//fcBePLJJ0lPTycjI4OK\nigrgyzNnDoeDpUuXcvPNN/PAAw8E9nngwAFuvvlm5syZw69//evA6zU1Ndjtdux2O1lZWXR2do5j\nT4hIOAvrB+eKiAznwIEDJCYm8vrrrwPwpz/9iTvvvJOqqioSEhJ4+eWXefrpp9m9ezfFxcX88Ic/\nZNGiRXz++ef09vby6quv8tZbb9HU1MQf/vAHcnJy+MY3vgHAqVOnePvtt5k6dSq33347dXV1ZGVl\n8fd///e8+eab3Hjjjdx3332B+UOfffZZduzYwW233UZXVxdf+cpXQtYvIhJedIZNRCJaRkYGBw8e\nZN26dfz3f/83p0+f5re//S3z5s3DbrfzT//0T7S3t9PZ2clHH33EokWLALjqqquIjY3l6NGjFBUV\nYbFY+OpXv8o3v/lNjh8/jsViITc3l2nTpmGxWJg9ezbvv/8+7777LjNnzuTGG28E4IEHHghMLXT7\n7bfzxBNP8LOf/YxPP/2UqKiokPWLiIQXnWETkYiWnJyMx+Ph9ddf50c/+hHf+ta3SEtLo66ubsB6\n586du+g2LpwQxn/GrP8ZsqioKM6fPx94b6i2a9eu5W/+5m94/fXXuf322/nP//xPvv71r485NxG5\ncugMm4hEtI8//pirr76a4uJi1qxZg9vt5pNPPqG+vh6AL774grfffptJkyYxffp0XnvtNQB6enro\n7u7mjjvu4OWXX6avr48//OEP1NbWkpubO6iIgy8LuVmzZuH1evnf//1fAPbt2xd4v7W1lbS0NJxO\nJzk5OfzP//zPOPSAiEQCnWETkYjW3NzMk08+yYQJE7jqqqv4xS9+QVRUFI899hhnz57l/PnzPPHE\nE6SmplJWVkZpaSnPPPMMMTExvPLKKxQWFnLs2DEyMzOxWCz85Cc/4atf/SrvvPPOoLNp8OVZt507\nd/Ld736XuLg47rjjDj777DMAXnjhBd58800mTJjALbfcwp133jne3SEiYUqTv4uIiIiYnC6JioiI\niJicCjYRERERk1PBJiIiImJyKthERERETE4Fm4iIiIjJqWATERERMbn/BzTleJbh/gVBAAAAAElF\nTkSuQmCC\n" } ], "prompt_number": 52 }, { "cell_type": "markdown", "metadata": {}, "source": [ "(Load balancing visualization example from http://nbviewer.ipython.org/urls/raw.github.com/jrjohansson/scientific-python-lectures/master/Lecture-6B-HPC.ipynb )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To make sure that engines have the same modules loaded, one can use the method `.sync_imports()` of an engine pool view." ] }, { "cell_type": "code", "collapsed": false, "input": [ "with dview.sync_imports():\n", " import numpy\n", " import os" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "importing numpy on engine(s)\n", "importing os on engine(s)\n" ] } ], "prompt_number": 53 }, { "cell_type": "code", "collapsed": false, "input": [ "%%px\n", "os.getpid()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "display_data", "text": [ "\u001b[0;31mOut[0:1]: \u001b[0m88707" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[1:1]: \u001b[0m88708" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[2:1]: \u001b[0m88706" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[3:1]: \u001b[0m88709" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[4:1]: \u001b[0m88710" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[5:1]: \u001b[0m88711" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[6:1]: \u001b[0m88712" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[7:1]: \u001b[0m88713" ] } ], "prompt_number": 54 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since the engines will not have the same namespace as any interpreter that might want to use them, if we want them to have the same data to work on one need to send that data to the engines.\n", "\n", "This is done with the `.push()` method of an engine pool view." ] }, { "cell_type": "code", "collapsed": false, "input": [ "dview.push(dict(a=np.arange(5), b=np.zeros(5)), block=True)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 55, "text": [ "[None, None, None, None, None, None, None, None]" ] } ], "prompt_number": 55 }, { "cell_type": "code", "collapsed": false, "input": [ "dview.pull(\"a\", block=True)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 56, "text": [ "[array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4])]" ] } ], "prompt_number": 56 }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the sake of convenience, pushing and pulling to the namespaces on the engines can be done through a dictionary like syntax" ] }, { "cell_type": "code", "collapsed": false, "input": [ "dview[\"a\"]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 57, "text": [ "[array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4]),\n", " array([0, 1, 2, 3, 4])]" ] } ], "prompt_number": 57 }, { "cell_type": "code", "collapsed": false, "input": [ "dview[\"c\"] = \"See\"" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 58 }, { "cell_type": "code", "collapsed": false, "input": [ "dview.pull(\"c\", block=True)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 59, "text": [ "['See', 'See', 'See', 'See', 'See', 'See', 'See', 'See']" ] } ], "prompt_number": 59 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Something which might be more useful is to push _different_ sections of data to various engines, just like when we applied the parallel function above.\n", "\n", "The direct view has the methods `.scatter()` and `.gather()`." ] }, { "cell_type": "code", "collapsed": false, "input": [ "dview.scatter('x', np.arange(95), block=True)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 60 }, { "cell_type": "markdown", "metadata": {}, "source": [ "This will divide the array to four sections, and divide those out to different engines." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%px\n", "x" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "display_data", "text": [ "\u001b[0;31mOut[0:2]: \u001b[0marray([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[1:2]: \u001b[0marray([12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23])" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[2:2]: \u001b[0marray([24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35])" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[3:2]: \u001b[0marray([36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47])" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[4:2]: \u001b[0marray([48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59])" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[5:2]: \u001b[0marray([60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71])" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[6:2]: \u001b[0marray([72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83])" ] }, { "output_type": "display_data", "text": [ "\u001b[0;31mOut[7:2]: \u001b[0marray([84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94])" ] } ], "prompt_number": 61 }, { "cell_type": "code", "collapsed": false, "input": [ "def x_mean():\n", " return x.mean()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 62 }, { "cell_type": "code", "collapsed": false, "input": [ "dview.apply_sync(x_mean)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 63, "text": [ "[5.5, 17.5, 29.5, 41.5, 53.5, 65.5, 77.5, 89.0]" ] } ], "prompt_number": 63 }, { "cell_type": "code", "collapsed": false, "input": [ "dview.gather(\"x\", targets=1, block=True)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 64, "text": [ "array([12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23])" ] } ], "prompt_number": 64 }, { "cell_type": "markdown", "metadata": {}, "source": [ "An alternative to importing modules in the definition of a function, there is a `@require` decoratir in `IPython.parallel` which will cause the required module to be imported before the code in the function is executed." ] }, { "cell_type": "code", "collapsed": false, "input": [ "from IPython.parallel import require" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 65 }, { "cell_type": "code", "collapsed": false, "input": [ "@require(\"os\")\n", "def get_pid():\n", " return os.getpid()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 66 }, { "cell_type": "code", "collapsed": false, "input": [ "dview.apply_sync(get_pid)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 67, "text": [ "[88707, 88708, 88706, 88709, 88710, 88711, 88712, 88713]" ] } ], "prompt_number": 67 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Assignment 7\n", "\n", "This is a very naive function for factoring integers" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def factorize(n):\n", " if n < 2:\n", " return []\n", " factors = []\n", " p = 2\n", "\n", " while True:\n", " if n == 1:\n", " return factors\n", " r = n % p\n", " if r == 0:\n", " factors.append(p)\n", " n = n / p\n", " elif p * p >= n:\n", " factors.append(n)\n", " return factors\n", " elif p > 2:\n", " p += 2\n", " else:\n", " p += 1" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The task is to factorize all numbers from 2 to 500000, and count the number of _unique_ factors. Then count how many times a factor count occurs. That is, make a histogram of factor counts.\n", "\n", "For the sake of clarity, this is what the task would be for the numbers up to 10:\n", "\n", " number factors unique num_unique_factors\n", " 2 [2] 1\n", " 3 [3] 1\n", " 4 [2, 2] [2] 1\n", " 5 [5] 1\n", " 6 [2, 3] 2\n", " 7 [7] 1\n", " 8 [2, 2, 2] [2] 1\n", " 9 [3, 3] [3] 1\n", " 10 [2, 5] 2\n", "\n", "Which gives us\n", "\n", " {1: 7, 2: 2}\n", "\n", "\n", "### 1.\n", "First implement this serially, that is, how you normally would. (It should take in the order of 15 seconds for 500000 numbers).\n", "\n", "\n", "### 2.\n", "Then make an implementation which uses `multiprocessing` for parallelization, (the parallelization strategy should be to factor some numbers on some processes, don't worry about the implementation of the factoring function).\n", "\n", "### 3.\n", "Implement the same thing using `IPython.parallel`. (Remember that you need to start the `ipcluster` before running a script using `IPython.parallel`, this is something I have a tendency to forget.)\n", "\n", "For this assignment, put _everything_ in one script. Call the script `num_factors.py`. Put it in the `scripts` directory of your repository.\n", "\n", "The script should take a character as an argument to determining how it will be run.\n", "So to run it serially, you run\n", "\n", " $ num_factors.py s\n", "\n", "With `multiprocessing`\n", "\n", " $ num_factors.py m\n", "\n", "and with `IPython.parallel`\n", "\n", " $ num_factors.py i\n", "\n", "The script should print a dictionary like in the example above" ] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] } ], "metadata": {} } ] }