{ "metadata": { "kernelspec": { "codemirror_mode": { "name": "ipython", "version": 2 }, "display_name": "IPython (Python 2)", "language": "python", "name": "python2" }, "name": "", "signature": "sha256:53d1f434bf40ebc115f8bec6111b4b1e1dc3aa9600f74262d564ee5682abc464" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Pair programming with an IPython Notebook Server" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is a brief tutorial on how to configure a secure, single-user IPython notebook server to be shared between two users, with manual synchronization of contents. IPython currently does *not* provide real-time, Google Docs-style live synchronization (see note below).\n", "\n", "We will assume the host machine is a Linux or OSX machine with standard POSIX user management semantics. This may also be done with a Windows host, but some of the details of this specific tutorial will need modification. Also, the actual commands were typed on Linux, so a few command-line flags may be slightly different if you are \n", "on OSX.\n", "\n", "**Note:** This tutorial is aimed at users who need/want to self-host their server, for data privacy/size reasons, computational resources requirements, etc. If you can use an external service, [Cloud SageMath](https://cloud.sagemath.com) offers hosted IPython services with real-time live sync." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create a new isolated user for shared work\n", "\n", "With the `useradd` command, make a new user called `shareipython` (or whatever you want to call it):\n", "\n", " sudo useradd -m -s /bin/bash shareipython\n", "\n", "Next, set a regular password for this user:\n", "\n", " alpamayo[~]> sudo passwd shareipython\n", " Enter new UNIX password: \n", " Retype new UNIX password: \n", " passwd: password updated successfully\n", " \n", "and log in as `shareipython`:\n", " \n", " alpamayo[~]> su - shareipython\n", " Password: \n", " shareipython@alpamayo:~$ \n", "\n", "From now on, you will be logged in as `shareipython`. Unless you know for a fact that the system version of IPython is up to date, you may want to update IPython itself (plus any other libraries you will need):\n", "\n", " pip install --user --upgrade ipython[all]\n", "\n", "Next, start the IPython console once, just to verify that the basics work OK: \n", "\n", " shareipython@alpamayo:~$ ipython\n", " Python 2.7.6 (default, Mar 22 2014, 22:59:56) \n", " Type \"copyright\", \"credits\" or \"license\" for more information.\n", "\n", " IPython 2.2.0 -- An enhanced Interactive Python.\n", " ? -> Introduction and overview of IPython's features.\n", " %quickref -> Quick reference.\n", " help -> Python's own help system.\n", " object? -> Details about 'object', use 'object??' for extra details.\n", "\n", " In [1]: exit\n", " shareipython@alpamayo:~$ \n", "\n", "OK, at this point, you have a working IPython, in this case version 2.2.0. Now you can proceed to creating a shared notebook server." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating a secure, shared notebook server\n", "\n", "You can find the full details on how to secure the IPython notebook [in the official documentation](http://ipython.org/ipython-doc/2/notebook/public_server.html#running-a-public-notebook-server). This document is only a brief summary of the key points, organized for easy copy-paste.\n", "\n", "Start by creating a special profile to run the notebook server with:\n", "\n", " ipython profile create nbserver\n", "\n", "which should produce output like:\n", "\n", " shareipython@alpamayo:~$ ipython profile create nbserver\n", " [ProfileCreate] Generating default config file: u'/home/shareipython/.ipython/profile_nbserver/ipython_config.py'\n", " [ProfileCreate] Generating default config file: u'/home/shareipython/.ipython/profile_nbserver/ipython_qtconsole_config.py'\n", " [ProfileCreate] Generating default config file: u'/home/shareipython/.ipython/profile_nbserver/ipython_notebook_config.py'\n", " [ProfileCreate] Generating default config file: u'/home/shareipython/.ipython/profile_nbserver/ipython_nbconvert_config.py'\n", "\n", "And now change to the directory where those files were created:\n", "\n", " cd /home/shareipython/.ipython/profile_nbserver/\n", "\n", "*Note:* You can always locate this directory with this command:\n", "\n", " shareipython@alpamayo:~/.ipython/profile_nbserver$ ipython profile locate nbserver\n", " /home/shareipython/.ipython/profile_nbserver\n", "\n", "Next, create a self-signed SSL certificate that you will use to secure the Notebook connection. Again, at the system command line (not IPython), type:\n", "\n", " openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem\n", " \n", "You will get a bunch of prompts, you can simply hit `Enter` to all of them and provide empty answers, they don't matter. It will look like this:\n", "\n", " shareipython@alpamayo:~/.ipython/profile_nbserver$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem\n", " Generating a 1024 bit RSA private key\n", " .......................................++++++\n", " ...........................++++++\n", " writing new private key to 'mycert.pem'\n", " -----\n", " You are about to be asked to enter information that will be incorporated\n", " into your certificate request.\n", " What you are about to enter is what is called a Distinguished Name or a DN.\n", " There are quite a few fields but you can leave some blank\n", " For some fields there will be a default value,\n", " If you enter '.', the field will be left blank.\n", " -----\n", " Country Name (2 letter code) [AU]:\n", " State or Province Name (full name) [Some-State]:\n", " Locality Name (eg, city) []:\n", " Organization Name (eg, company) [Internet Widgits Pty Ltd]:\n", " Organizational Unit Name (eg, section) []:\n", " Common Name (e.g. server FQDN or YOUR name) []:\n", " Email Address []:\n", "\n", "Now, verify that you have a certificate file:\n", "\n", " shareipython@alpamayo:~/.ipython/profile_nbserver$ ls *.pem\n", " mycert.pem\n", "\n", "Next, create a password for the notebook server (this need not be the same Unix password of the user). This should be done inside of a Python or IPython shell:\n", "\n", " In [1]: from IPython.lib import passwd\n", "\n", " In [2]: passwd()\n", " Enter password: \n", " Verify password: \n", " Out[2]: 'sha1:9b18ae137d41:004295f4352d691c295a5ef193aaf7b54a4a2864'\n", " \n", "Save the output above, as you'll need to paste it next in a configuration file.\n", "\n", "You now have all the pieces to set up the server. Next, you will need to edit a file named `ipython_notebook_config.py` in this same directory, with content like the following. Note that if you named your user something other than `shareipython` you'll need to adjust the path, and the hashed password should be the string **you** actually created. Use the content below only as a reference. IPython has already created that file for you, and there are many more parameters. The reference below is the *minimal* amount of configuration needed for this particular task:\n", "\n", "```python\n", "c = get_config()\n", "\n", "# The full path to an SSL/TLS certificate file.\n", "c.NotebookApp.certfile = u'/home/shareipython/.ipython/profile_nbserver/mycert.pem'\n", "\n", "# Listen on all IP addresses, so it can be reached over the public internet:\n", "c.NotebookApp.ip = '*'\n", "\n", "# Don't open the web browser when the server starts:\n", "c.NotebookApp.open_browser = False\n", "\n", "# Hashed notebook password that was previously created:\n", "c.NotebookApp.password = u'sha1:9b18ae137d41:004295f4352d691c295a5ef193aaf7b54a4a2864'\n", "\n", "# It is a good idea to put it on a known, fixed port\n", "c.NotebookApp.port = 8989\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Turning off autosave\n", "\n", "You **must** deactivate autosave, so that multiple users don't overwrite each other accidentally. For that, you must edit the file `custom.js` located in the `static/custom` subdirectory of the `nbserver` profile (in this case, `~/.ipython/profile_nbserver/static/custom` and add at the bottom:\n", "\n", "```javascript\n", "$([IPython.events]).on(\"notebook_loaded.Notebook\", function () {\n", " IPython.notebook.set_autosave_interval(0);\n", " });\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running the public notebook server\n", "\n", "Now you should be ready to run your shared notebook server. The following assumes that:\n", "\n", "1. Your network has no firewalling that will prevent external users from connecting on port 8989, which you selected above. You will need to talk to your network/system administration staff if that's not the case.\n", "\n", "2. No other process is running on that port. If some other user in your organization is doing so, simply select a different port number.\n", "\n", "Since you'll need to have the server running persistently even if you log out from your computer, you should start the server process inside a persistent terminal system, like [tmux](http://tmux.sourceforge.net/) or [Screen](http://www.gnu.org/software/screen/). \n", "\n", "**NOTE:** From now on, you should be *inside* a tmux (or screen) session.\n", "\n", "First, go to the highest-level directory that you want to have available over the notebook. That may be the home directory of the special `shareipython` user, or a deeper one if you prefer. The notebook server can drill down deeper in the file system, but it can *not* go higher (for security reasons). \n", "\n", "In that directory, start the notebook server with the specific profile you created above:\n", "\n", " ipython notebook --profile nbserver\n", " \n", "and you should see something along the following lines:\n", " \n", " shareipython@alpamayo:~$ ipython notebook --profile nbserver\n", " 2014-09-19 10:31:53.348 [NotebookApp] Using existing profile dir: u'/home/shareipython/.ipython/profile_nbserver'\n", " 2014-09-19 10:31:53.353 [NotebookApp] Using MathJax from CDN: https://cdn.mathjax.org/mathjax/latest/MathJax.js\n", " 2014-09-19 10:31:53.364 [NotebookApp] Serving notebooks from local directory: /home/shareipython\n", " 2014-09-19 10:31:53.364 [NotebookApp] 0 active kernels \n", " 2014-09-19 10:31:53.364 [NotebookApp] The IPython Notebook is running at: https://[all ip addresses on your system]:8989/\n", " 2014-09-19 10:31:53.364 [NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).\n", " \n", "You can now leave this notebook running for as long as you want." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Connecting to your server\n", "\n", "Test it by connecting to port 8989 **over the https protocol**, by typing in a web browser `https://IP.OF.YOUR.SERVER:8989`.\n", "\n", "**WARNING FOR OSX USERS:** you can *not* use Safari for this, as it will not connect over `https` with a self-signed certificate. You *must* use Chrome or Firefox.\n", "\n", "Because you are using a self-signed certificate, you will see a page similar to the following. This example shows the Chrome browser, but Firefox has a similar one:\n", "\n", "![img](images/self-signed-cert-warning.png)\n", "\n", "Click on the `advanced` link at the bottom, and proceed to the destination:\n", "\n", "![img](images/self-signed-cert-proceed.png)\n", "\n", "This should take you to a login screen, where you will type the password you had created above:\n", "\n", "![img](images/notebook-login.png)\n", "\n", "Once you log in, you will see a standard IPython Notebook, dashboard, with *one* new property: it now has a \"Logout\" button on the upper right hand:\n", "\n", "![img](images/notebook-dashboard-secure.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using your new server in shared, collaborative mode\n", "\n", "Now that you have a public server, you can invite colleagues to work under this shared space. Note that everyone who logs in with that password is acting as the same system user. \n", "\n", "The trick to using the system is that there is *zero* automatic synchronization of the content. So you should be face to face (or on the phone/skype/g+) talking to your colleague(s) as you work, and only one person at a time can write to the file, holding a 'write lock' on the document. When the writer is ready to hand it off to someone else, they *must manually save*, and tell the others, who then must refresh their browser page.\n", "\n", "### Tips\n", "\n", "* Interactive widget controls (sliders, menus, etc) don't automatically reload on page refresh, so you will need to manually re-execute any cell with a live widget that you want to use after you've reloaded the page.\n", "\n", "* It can be very useful to keep one or more 'scratch' notebooks in the shared server, where various users can quickly prototype things out while the person holding the 'write lock' works on the main notebook. Then, content can be transfered from the scratch one to the main one when ready.\n", "\n", "* If two people want to work in parallel on a single notebook, another useful trick is to use the `File -> Make a copy` item from the menu and create a temporary copy. That has all the material from the original, which makes it easy to make further edits in isolation from the user holding the 'write lock'. Once ready, those edits can be transfered back to the main document.\n", "\n", "The workflow is not perfect, but it works.\n", "\n", "### *Happy collaborating!*" ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Known issues" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When behind a proxy, especially if your system or browser is set to autodetect the proxy, the notebook web application might fail to connect to the server's websockets, and present you with a warning at startup. In this case, you need to configure your system not to use the proxy for the server's address.\n", "\n", "For example, in Firefox, go to the Preferences panel, Advanced section,\n", "Network tab, click 'Settings...', and add the address of the notebook server\n", "to the 'No proxy for' field." ] } ], "metadata": {} } ] }