{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running Neurokernel on NVIDIA Jetson Embedded Platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we show step by step how to run Neurokernel on [Jetson TK1 Embedded Development Kit](http://www.nvidia.com/object/jetson-tk1-embedded-dev-kit.html). It can be applied to the latest [Jetson TX1 platform](http://www.nvidia.com/object/jetson-tx1-dev-kit.html) with minor modification. Before we start, you must have a Jetson TK1 Embedded Development Kit and have enrolled in [NVIDIA Embedded Developer Program](https://developer.nvidia.com/embedded-developer-program).\n",
    "To install Jetson Development Pack (JetPack), you also need access to a Ubuntu OS. Follow [these steps](http://docs.nvidia.com/jetpack-l4t/2_0/index.html#developertools/mobile/jetpack/jetpack_l4t/2.0/jetpack_l4t_install.htm) to install JetPack. This notebook assumes that JetPack is already installed on your TK1 Kit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='files/files/tk1.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Neurokernel on Jetson TK1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please execute the following commands in terminal on the TK1. Note that executing command at a time is preferred to ensure complete installation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first maximize CPU performance to make running the installlation faster. For the 4 main CPU cores to always run at max performance until reboot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sudo su\n",
    "echo 0 > /sys/devices/system/cpu/cpuquiet/tegra_cpuquiet/enable\n",
    "echo 1 > /sys/devices/system/cpu/cpu0/online\n",
    "echo 1 > /sys/devices/system/cpu/cpu1/online\n",
    "echo 1 > /sys/devices/system/cpu/cpu2/online\n",
    "echo 1 > /sys/devices/system/cpu/cpu3/online\n",
    "echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor\n",
    "exit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update system and install necessary libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sudo apt-mark hold xserver-xorg-core\n",
    "sudo apt-get update\n",
    "sudo apt-get upgrade\n",
    "sudo apt-get install bash-completion command-not-found\n",
    "sudo apt-get install emacs24 libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev \n",
    "sudo apt-get install libgdbm-dev libc6-dev libbz2-dev liblapack-dev libhdf5-dev libxml2-dev libxslt1-dev\n",
    "sudo apt-get install python-dev tmux gfortran git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install Latest Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mkdir packages\n",
    "cd packages\n",
    "wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz\n",
    "tar -xvf Python-2.7.11.tgz\n",
    "cd Python-2.7.11\n",
    "./configure --prefix=/home/ubuntu/opt\n",
    "make\n",
    "make install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cd\n",
    "\n",
    "echo -e \"export PATH=/home/ubuntu/opt/bin:\\$PATH\\n\"\\\n",
    "\"export LD_LIBRARY_PATH=/home/ubuntu/opt/lib:\\$LD_LIBRARY_PATH\\n\"\\\n",
    "\"export PYTHONPATH=/home/ubuntu/opt/lib/python2.7/site-packages:\\$PYTHONPATH\" | tee -a ~/.bashrc\n",
    "\n",
    "source .bashrc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install Open-mpi:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cd packages\n",
    "wget http://www.open-mpi.org/software/ompi/v1.10/downloads/openmpi-1.10.1.tar.gz\n",
    "tar -xvf openmpi-1.10.1.tar.gz\n",
    "cd openmpi-1.10.1\n",
    "\n",
    "./configure --with-cuda=/usr/local/cuda --with-threads=posix\\\n",
    "  --disable-mca-dso --prefix=/home/ubuntu/opt\n",
    "\n",
    "make -j 4\n",
    "make install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install virtualenv:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cd ~/packages\n",
    "wget https://pypi.python.org/packages/source/v/virtualenv/virtualenv-13.1.2.tar.gz\n",
    "tar -xvf virtualenv-13.1.2.tar.gz\n",
    "cd virtualenv-13.1.2\n",
    "python setup.py install\n",
    "\n",
    "cd \n",
    "virtualenv NK\n",
    "source NK/bin/activate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install required Python packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pip install numpy scipy ipython cython numexpr pycuda\n",
    "pip install bidict pandas networkx mpi4py h5py dill lxml markupsafe shutilwhich ply psutil futures twiggy matplotlib\n",
    "pip install sphinx sphinx_rtd_theme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step will take a lot of time. If some installation fails with error message: No matching distribution found, you can install it manually by replacing link_to_package with the url for the package in the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pip install -Iv link_to_package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install Neurokernel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cd\n",
    "git clone https://github.com/neurokernel/neurokernel.git\n",
    "cd neurokernel\n",
    "git pull\n",
    "python setup.py develop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test Neurokernel by running one of the examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cd ~/neurokernel/examples/intro/data\n",
    "python gen_generic_lpu.py -s 0 -l lpu_0 generic_lpu_0.gexf.gz generic_lpu_0_input.h5                \n",
    "python gen_generic_lpu.py -s 1 -l lpu_1 generic_lpu_1.gexf.gz generic_lpu_1_input.h5\n",
    "cd ../\n",
    "python intro_demo.py --gpu_dev 0 0 --log file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can ignore warning messages like these:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    ---------------------------------------------------------\n",
    "    The call to cuMemHostRegister failed.\n",
    "      Host:  tegra-ubuntu\n",
    "      cuMemHostRegister return value:  801\n",
    "      Memory Pool:  smcuda\n",
    "    ---------------------------------------------------------\n",
    "    ---------------------------------------------------------\n",
    "    Sorry!  You were supposed to get help about:\n",
    "        cuMemHostRegister during init failed\n",
    "    from the file:\n",
    "        help-mpi-common-cuda.txt\n",
    "    But I couldn't find that topic in the file.  Sorry!\n",
    "    ---------------------------------------------------------\n",
    "\n",
    "or\n",
    "\n",
    "    983 more processes have sent help message help-mpi-common-cuda.txt / cuIpcGetMemHandle failed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is caused by the fact that cudaHostRegister is not supported in armv7 devices.\n",
    "Inspect 'neurokernel.log' after the example finishes execution. If there is no error in the log file, the installation is complete."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}