{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Machine Learning\n", "# 5. Programming Neural Networks with Theano" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [Luis Martí](http://lmarti.com)\n", "#### [Instituto de Computação](http://www.ic.uff)\n", "#### [Universidade Federal Fluminense](http://www.uff.br)\n", "$\\newcommand{\\vec}[1]{\\boldsymbol{#1}}$" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os, sys, gzip\n", "import six.moves.cPickle as pickle\n", "\n", "import numpy\n", "\n", "import theano\n", "import theano.tensor as T\n", "import theano.d3viz as d3viz\n", "\n", "from IPython.display import Image, SVG, IFrame\n", "\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "import matplotlib.cm as cm\n", "\n", "from mpl_toolkits.mplot3d import Axes3D" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import seaborn\n", "seaborn.set(style='whitegrid')\n", "seaborn.set_context('talk')\n", "\n", "%matplotlib inline\n", "%config InlineBackend.figure_format = 'retina'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# What about `theano`?\n", "\n", "* Mathematical symbolic expression compiler\n", " * Easy to define expressions\n", " * Expressions are similar numpy's syntax and semantics\n", "* Possible to manipulate those expressions\n", " * Substitutions\n", " * **Gradient**, R operator\n", "* Stability optimizations\n", "Fast to compute values for those expressions\n", "* Speed optimizations\n", "* Use fast back-ends (GPUs (CUDA), BLAS, custom C code)\n", "* Tools to inspect and check for correctness" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Theano defines a *language*, a *compiler*, and a *library*.\n", " * Define a symbolic expression\n", " * Compile a function that can compute values\n", " * Execute that function on numeric values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The graph that represents mathematical operations is bipartite, and has two sorts of nodes:\n", "* **Variable nodes**, that represent data\n", "* **Apply nodes**, that represent the application of mathematical operations \n", "\n", "In practice:\n", "* Variables values are used for the graph inputs and outputs, and intermediate values\n", "* Variables will hold data during the function execution phase\n", "* An Apply node has inputs and outputs, which are Variables\n", "* An Apply node represents the specific application of an Op on these input variables\n", "* The same Variable can be used as inputs by several Apply nodes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Shared variables\n", "* Symbolic variables, with a value associated to them\n", "* The value is persistent across function calls\n", "* The value is shared among all functions\n", "* The variable has to be an input variable\n", "* The variable is an implicit input to all functions using it" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An optimization replaces a part of the graph with different nodes.\n", "* The types of the replaced nodes have to match\n", "* Merge equivalent computations\n", "* Simplify expressions: $\\frac{x}{x}$ becomes $1$\n", "* Numerical stability: For example, gives the right answer for $\\log(1 + x)$ even if $x$ is really small.\n", "* Insert in-place an destructive versions of operations\n", "* Use specialized, high-performance versions (Elemwise loop fusion, GEMV,\n", "GEMM)\n", "* Shape inference\n", "* Constant folding\n", "* Transfer to GPU" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using GPUs\n", "\n", "Theano tries to make the use of GPUs as transparent as possible,\n", "* Currently limited to `float32` `dtype`\n", "* Not easy to interact in Python with `CudaNdarrays`\n", "\n", "Select GPU by setting the device flag to ’gpu’ or 'gpu{0,1,2,...}'.\n", "* All `float32` shared variables will be created in GPU memory\n", "* Enables optimizations moving supported operations to GPU\n", "\n", "You want to make sure you use `float32`\n", "* `floatX` is the default type of all tensors and sparse matrices.\n", "* By default, aliased to ’float64’ for double precision on CPU\n", "* Can be set to `float32` by a configuration flag\n", "* You can always explicitly use `T.fmatrix()` or `T.matrix(dtype=’float32’)`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Configuration flags can be set in a couple of ways:\n", "* In the shell:\n", "```bash\n", "$ THEANO_FLAGS=device=gpu0,floatX=float32\n", "```\n", "* In Python:\n", "```python\n", "theano.config.device = 'gpu0'\n", "theano.config.floatX = 'float32'\n", "```\n", "* In the `.theanorc` configuration file:\n", "```\n", "[global]\n", " device = gpu0\n", " floatX = float32\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# A first contact with `theano`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Defining theano variables and expressions" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "x = T.dscalar('x')\n", "y = x ** 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You might need to install `pydot` by running `pip install pydot`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "Software | Version |
---|---|
Python | 3.6.1 64bit [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] |
IPython | 6.1.0 |
OS | Darwin 16.6.0 x86_64 i386 64bit |
scipy | 0.19.0 |
numpy | 1.12.1 |
matplotlib | 2.0.2 |
sklearn | 0.18.1 |
theano | 0.9.0 |
Sun Jun 11 23:47:32 2017 CEST |