{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theano tensor 模块：nnet 子模块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`nnet` 是 `tensor` 模块中与神经网络 `Neural Networks` 相关的子模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using gpu device 1: Tesla C2075 (CNMeM is disabled)\n"
     ]
    }
   ],
   "source": [
    "import theano\n",
    "from theano import tensor as T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sigmoid 函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "共有三种 `sigmoid`：\n",
    "\n",
    "- `T.nnet.sigmoid(x)`\n",
    "- `T.nnet.ultra_sigmoid(x)`\n",
    "- `T.nnet.hard_sigmoid(x)`\n",
    "\n",
    "精度和时间：\n",
    "\n",
    "`sigmoid > ultra_fast_sigmoid > hard_sigmoid`\n",
    "\n",
    "函数图像：\n",
    "\n",
    "![](http://deeplearning.net/software/theano/_images/sigmoid_prec.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sigmoid(((W \\dot x) + b))\n"
     ]
    }
   ],
   "source": [
    "x, y, b = T.dvectors('x', 'y', 'b')\n",
    "W = T.dmatrix('W')\n",
    "y = T.nnet.sigmoid(T.dot(W, x) + b)\n",
    "\n",
    "print theano.pprint(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 其他"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`T.nnet.softplus(x)` 返回 \n",
    "\n",
    "$$\\operatorname{softplus}(x) = \\log_e{\\left(1 + \\exp(x)\\right)}$$\n",
    "\n",
    "会解决在 1 附近自定义函数值不准的问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "softplus(((W \\dot x) + b))\n"
     ]
    }
   ],
   "source": [
    "x,y,b = T.dvectors('x','y','b')\n",
    "W = T.dmatrix('W')\n",
    "y = T.nnet.softplus(T.dot(W,x) + b)\n",
    "\n",
    "print theano.pprint(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`T.nnet.softplus(x)` 返回 \n",
    "\n",
    "$$\n",
    "\\operatorname{softmax}_{ij}(x) = \\frac{\\exp{x_{ij}}}{\\sum_k\\exp(x_{ik})}\n",
    "$$\n",
    "\n",
    "当 `softmax` 作用到矩阵时，它会按照行进行计算。\n",
    "\n",
    "不过，下面\n",
    "的代码计算性能上更加稳定：\n",
    "\n",
    "```\n",
    "e_x = exp(x - x.max(axis=1, keepdims=True))\n",
    "out = e_x / e_x.sum(axis=1, keepdims=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Softmax(((W \\dot x) + b))\n"
     ]
    }
   ],
   "source": [
    "x,y,b = T.dvectors('x','y','b')\n",
    "W = T.dmatrix('W')\n",
    "y = T.nnet.softmax(T.dot(W,x) + b)\n",
    "\n",
    "print theano.pprint(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`T.nnet.relu(x, alpha=0)` 返回这样一个函数：\n",
    "\n",
    "$$\n",
    "f(x_i) = \\left\\{\n",
    "\\begin{aligned}\n",
    "x_i, & \\ x_i > 0 \\\\\n",
    "\\alpha x_i, & \\ otherwise\n",
    "\\end{aligned}\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 损失函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`T.nnet.binary_crossentropy(output, target)` 二类交叉熵：\n",
    "\n",
    "$$\n",
    "\\text{crossentropy}(t,o) = -(t\\cdot log(o) + (1 - t) \\cdot log(1 - o))\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x, y, b, c = T.dvectors('x', 'y', 'b', 'c')\n",
    "W = T.dmatrix('W')\n",
    "V = T.dmatrix('V')\n",
    "h = T.nnet.sigmoid(T.dot(W, x) + b)\n",
    "x_recons = T.nnet.sigmoid(T.dot(V, h) + c)\n",
    "recon_cost = T.nnet.binary_crossentropy(x_recons, x).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`T.nnet.categorical_crossentropy(coding_dist, true_dist)` 多类交叉熵\n",
    "\n",
    "$$\n",
    "H(p,q) = - \\sum_x p(x) \\log(q(x))\n",
    "$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}