---
aliases:
- /2014/03/python-on-gordon
categories:
- hpc
- python
date: 2014-03-20 19:30
layout: post
slug: setup-ipython-notebook-parallel-Gordon
title: Python on Gordon

---

Gordon has already a `python` environment setup which can be activated by loading the `python` module:

    module load python # add this to .bashrc to load it at every login

### Install virtualenv

Then we need to setup a sandboxed local environment to install other packages, by using `virtualenv`, get the link to the latest version from <https://pypi.python.org/pypi/virtualenv>, then download it on gordon and unpack it, e.g.

    wget --no-check-certificate https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.11.2.tar.gz
    tar xzvf virtualenv*tar.gz

Then create your own virtualenv and load it:

	mkdir ~/venv
    python virtualenv-*/virtualenv.py ~/venv/py
    source ~/venv/py/bin/activate # add this to .bashrc to load it at every login

you can restore your previous environment by deactivating the virtualenv:

	deactivate # from your bash prompt
    
### Install IPython

Using `pip` you can install `IPython` and all dependencies for the notebook and parallel tools running:

    pip install ipython pyzmq tornado jinja

### Configure the IPython notebook
For interactive data exploration, you can run the `IPython` notebook in a computing node on Gordon and export the web interface to your local machine, which also embeds all the plots.
Configuring the tunnelling over SSH is complicated, so I created a script, takes a little time to setup but then is very easy to use, see https://github.com/pyHPC/ipynbhpc.

### Configure IPython parallel
[IPython parallel](http://ipython.org/ipython-doc/stable/parallel/) on Gordon allows to launch a `PBS` job with tens (or hundreds) of Python engines and then easily submit hundreds (or thousands) of serial jobs to be executed with automatic load balancing.
First of all create the default configuration files:

    ipython profile create --parallel 
Then, in `~/.ipython/profile_default/ipcluster_config.py`, you need to setup:

    c.IPClusterStart.controller_launcher_class = 'LocalControllerLauncher' 
    c.IPClusterStart.engine_launcher_class = 'PBS' 
    c.PBSLauncher.batch_template_file = u'/home/REPLACEWITHYOURUSER/.ipython/profile_default/pbs.engine.template' # "~" does not work
    
You also need to allow connections to the controller from other hosts, setting  in `~/.ipython/profile_default/ipcontroller_config.py`: 

    c.HubFactory.ip = '*'
    c.HubFactory.engine_ip = '*'

Finally create the PBS template `~/.ipython/profile_default/pbs.engine.template`:

    #!/bin/bash
    #PBS -q normal
    #PBS -N ipcluster
    #PBS -l nodes={n/16}:ppn={n}:native
    #PBS -l walltime=01:00:00
    #PBS -o ipcluster.out
    #PBS -e ipcluster.err
    #PBS -m abe
    #PBS -V
    mpirun_rsh -np {n} -hostfile $PBS_NODEFILE ipengine

Here we chose to run 16 IPython engines per Gordon node, so each has access to 4GB of ram, if you need more just change 16 to 8 for example.

### Run IPython parallel

You can submit a job to the queue running, `n` is equal to the number of processes you want to use, so it needs to be a multiple of the `ppn` chosen in the PBS template:

    ipcluster start --n=32 &
   
in this case we are requesting 2 nodes, with 16 IPython engines each, check with:

    qstat -u $USER
   
basically `ipcluster` runs an `ipcontroller` on the login node and submits a job to PBS for running the `ipengines` on the computing nodes.

Once the PBS job is running, check that the engines are connected by opening a IPython on the login node and print the `ids`:

    In [1]: from IPython.parallel import Client
    In [2]: rc = Client()
    In [3]: rc.ids

You can stop the cluster (kills `ipcontroller` and runs `qdel` on the PBS job) either by sending CTRL-c to `ipcluster` or running:

    ipcluster stop # from bash console
    
### Submit jobs to IPython parallel

As soon as `ipcluster` is executed, `ipcontroller` is ready to queue jobs up, which will be then consumed by the engines once they will be running.
The easiest method to submit jobs with automatic load balancing is to create a load balanced view:

    In [1]: from IPython.parallel import Client
    In [2]: rc = Client()
    In [3]: lview = rc.load_balanced_view() # default load-balanced view

and then use its `map` method:
    
    def exp_10(x):
    	return x**10
        
    list_of_args = range(100)
    result = lview.map(exp_10, list_of_args)

In this code `IPython` will distribute uniformly the list of arguments to the engines and the function will be evalutated for each of them and the result copied back to the connecting client running on the login node.

### Submit non-python jobs to IPython parallel
Let's assume you have a list of commands you want to run in a text file, one command per line, those could be implemented in any programming language, e.g.:

    date &> date.log
    hostname &> hostname.log
 
Then you create a function that executes one of those commands:

	def run_command(command):
        import subprocess
        subprocess.Popen(command, shell = True)
   
Then apply this function to the list of commands:

    list_of_commands = open("commands.txt").readlines()
    lview.map(run_command, list_of_commands)

I created a script that automates this process, see https://gist.github.com/zonca/8994544, you can run as:

    ./ipcluster_run_commands.py commands.txt