{
 "metadata": {
  "name": "",
  "signature": "sha256:dd660ce669809bc71efd233ff4f0e4bf3058643050e31480a60e8801d1758fe7"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Using Parallel Magics"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "IPython has a few magics for working with your engines.\n",
      "\n",
      "This assumes you have started an IPython cluster, either with the notebook interface,\n",
      "or the `ipcluster/controller/engine` commands."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from IPython import parallel\n",
      "rc = parallel.Client()\n",
      "dv = rc[:]\n",
      "rc.ids"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Creating a Client registers the parallel magics `%px`, `%%px`, `%pxresult`, `pxconfig`, and `%autopx`.  \n",
      "These magics are initially associated with a DirectView always associated with all currently registered engines."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we can execute code remotely with `%px`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px a=5"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px print a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with dv.sync_imports():\n",
      "    import sys"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px from __future__ import print_function\n",
      "%px print(\"ERROR\", file=sys.stderr)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You don't have to wait for results.  The `%pxconfig` magic lets you change the default blocking/targets for the `%px` magics:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxconfig --noblock"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px import time\n",
      "%px time.sleep(5)\n",
      "%px time.time()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "But you will notice that this didn't output the result of the last command.\n",
      "For this, we have `%pxresult`, which displays the output of the latest request:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxresult"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Remember, an IPython engine is IPython, so you can do magics remotely as well!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "%matplotlib inline\n",
      "import numpy as np\n",
      "import matplotlib.pyplot as plt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`%%px` can also be used as a cell magic, for submitting whole blocks.\n",
      "This one acceps `--block` and `--noblock` flags to specify\n",
      "the blocking behavior, though the default is unchanged.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dv.scatter('id', dv.targets, flatten=True)\n",
      "dv['stride'] = len(dv)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px --block\n",
      "print(\"    id=%s\" % id)\n",
      "print(\"stride=%s\" % stride)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px --noblock\n",
      "x = np.linspace(0,np.pi,1000)\n",
      "for n in range(id, 12, stride):\n",
      "    print(n)\n",
      "    plt.plot(x,np.sin(n*x))\n",
      "plt.title(\"Plot %i\" % id);"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxresult"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Parallel Exceptions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you raise exceptions with the parallel exception,\n",
      "the CompositeError raised locally will display your remote traceback."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "from numpy.random import random\n",
      "A = random((100,100,'invalid shape'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Remote Cell Magics"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "%%bash\n",
      "echo \"remote parallel interactive bash!\"\n",
      "hostname\n",
      "date\n",
      "echo $$"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "%%ruby\n",
      "puts 'hello from ruby'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dv.scatter('rank', dv.targets, flatten=True)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "%%timeit\n",
      "from numpy.random import random\n",
      "from numpy.linalg import norm\n",
      "N = 100 * (rank + 1)\n",
      "A = random((N,N))\n",
      "norm(A, 2)    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Debugging Engines"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Since the IPython engine is precisely the same object used for the notebook and qtconsole,\n",
      "we can connect other fronteds directly to the engine.\n",
      "\n",
      "The first step is to bind the engine's sockets, so its connection pattern looks like a regular kernel"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "from IPython.parallel import bind_kernel\n",
      "bind_kernel()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we can raise an exception on the engines"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "\n",
      "def foo(a, b):\n",
      "    return a/(1-b)\n",
      "\n",
      "def bar(b):\n",
      "    return foo(2, b)\n",
      "\n",
      "bar(1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we can connect a qtconsole to the engine(s)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px %qtconsole"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}