\n",
"
\n",
"
\n",
"
\n",
"![](\"images/nn.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Overview of Neural Networks\n",
"- A **loss function** is defined over the *output units* and *desired outputs* (i.e., labels)\n",
"$$\\mathcal{L}\\left( \\textbf{y}, \\hat{\\textbf{y}} \\right) \\mbox{ where } \\hat{\\textbf{y}}=f(\\textbf{x};\\theta)$$\n",
"- The parameter of the network is trained to minimize the loss function based on gradient descent methods\n",
"$$\\min_{\\theta} \\mathbb{E}_{(\\textbf{x},\\textbf{y}) \\sim \\mbox{data}} \\left[ \\mathcal{L}\\left( \\textbf{y}, \\hat{\\textbf{y}} \\right) \\right] \\mbox{ where } \\hat{\\textbf{y}}=f(\\textbf{x};\\theta) $$\n",
"![](\"images/nn_forward.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Forward Propagation\n",
"![](\"images/nn_forward2.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Forward Propagation\n",
"- The activation of each unit is computed based on **the previous layer** and **parameters (or weights)** associated with edges\n",
"$$\\underbrace{\\textbf{h}^{(l)}}_{l\\mbox{-th layer}}=f^{(l)}(\\underbrace{\\textbf{h}^{(l-1)}}_{(l-1)\\mbox{-th layer}}; \\underbrace{\\theta^{(l)}}_{\\mbox{weights}}) \\mbox { where } \\textbf{h}^{(0)} \\equiv \\textbf{x}, \\textbf{h}^{(L)} \\equiv \\hat{\\textbf{y}}$$\n",
"$$\\hat{\\textbf{y}}=f(\\textbf{x};\\theta)=f^{(L)} \\circ f^{(L-1)} \\cdots f^{(2)} \\circ f^{(1)}\\left(\\textbf{x} ; \\theta^{(1)} \\right) $$\n",
"![](\"images/linear.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Types of Layers: Non-linear Activation Function\n",
"- Applies a non-linear function to individual units.\n",
"- There is no weight.\n",
"- Allows neural networks to learn non-linear features.\n",
"- ex) Sigmoid, Hyperbolic Tangent, Rectified Linear Function\n",
"![](\"images/activation.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Non-linear Activation: Sigmoid\n",
"$$ h_i=\\sigma (x_i) = \\frac{1}{1+\\exp\\left(-x_i\\right)} $$\n",
"$$ \\textbf{h} = \\sigma \\left(\\textbf{x} \\right) $$ "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Multi-layer Neural Network\n",
"- Consists of multiple (linear + non-linear activation) layers.\n",
"- Each layer learns non-linear features from its previous layer.\n",
"- Often called Multi-Layer Perceptron (MLP).\n",
"- 2-layer MLP with infinite number of hidden units can approximate any functions.\n",
"![](\"images/mlp.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Multi-layer Neural Network\n",
"- Simplified illustration that only shows edges with weights.\n",
"- We assume that each layer is followed by a non-linear activation function (except for the output layer).\n",
"![](\"images/mlp_standard.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Multi-layer Neural Network\n",
"- More simplified illustration\n",
"![](\"images/mlp_simple.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Multi-layer Neural Network\n",
"- Even more simplified illustration\n",
"![](\"images/mlp_simplest.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Outline\n",
"\n",
"- Motivation\n",
"- Basics of Neural Networks\n",
" - Forward Propagation\n",
" - **Backward Propagation**\n",
"- Deep Neural Networks\n",
" - Convolutional Neural Networks\n",
" - Recurrent Neural Networks\n",
"- Applications\n",
" - Computer Vision\n",
" - Natural Language Processing\n",
" - Reinforcement Learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Training Neural Networks\n",
"- Repeat until convergence\n",
" - $(\\textbf{x},\\textbf{y}) \\leftarrow$ Sample an example (or a mini-batch) from data\n",
" - $\\hat{\\textbf{y}} \\leftarrow f\\left( \\textbf{x} ; \\theta \\right)$ Forward propagation\n",
" - Compute $\\mathcal{L}\\left(\\textbf{y},\\hat{\\textbf{y}}\\right)$\n",
" - $\\nabla_{\\theta}\\mathcal{L} \\leftarrow$ Backward propagation\n",
" - Update weights using (stochastic) gradient descent\n",
" - $\\theta \\leftarrow \\theta - \\alpha \\nabla_{\\theta}\\mathcal{L}$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Types of Losses: Squared Loss\n",
"$$ \\mathcal{L}\\left(\\textbf{y},\\hat{\\textbf{y}}\\right) = \\frac{1}{2}\\left\\Vert \\textbf{y}-\\hat{\\textbf{y}}\\right\\Vert^2_2 $$\n",
"$$ \\frac{\\partial\\mathcal{L}}{\\partial \\hat{y}_i}=\\hat{y}_i-y_i \\iff \\nabla_{\\hat{\\textbf{y}}}\\mathcal{L}=\\hat{\\textbf{y}}-\\textbf{y} $$\n",
"- Used for regression problems"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Types of Losses: Cross Entropy\n",
"$$ \\mathcal{L}\\left(\\textbf{y},\\hat{\\textbf{y}}\\right) = -\\sum_{i}y_i\\log\\hat{y}_i $$\n",
"$$ \\frac{\\partial\\mathcal{L}}{\\partial \\hat{y}_i}=-\\frac{y_i}{\\hat{y}_i} \\iff \\nabla_{\\hat{\\textbf{y}}}\\mathcal{L}=-\\frac{\\textbf{y}}{\\hat{\\textbf{y}}}$$\n",
"\n",
"- Measure a distance between two multinomial distributions\n",
"- Used for classification problems \n",
" - ex) $\\textbf{y}=$[0 0 1], $\\hat{\\textbf{y}}=$[0.3 0.2 0.5], $\\mathcal{L}\\left(\\textbf{y},\\hat{\\textbf{y}}\\right)=-\\log 0.5$\n",
"- When cross entropy is used with a softmax output, the last layer is equivalent to softmax regression (multi-class version of logistic regression)."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Training Neural Networks\n",
"![](\"images/forward_backward.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Idea of Back-Propagation\n",
"- Denote $x,h,\\theta$ is the input, output, and parameter of a layer.\n",
"- It is non-trivial to derive the gradient of loss w.r.t. parameters in intermediate layers\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Idea of Back-Propagation\n",
"- Assuming that $\\frac{\\partial \\mathcal{L}}{\\partial h}$ is given, use the **chain rule** to compute the gradients\n",
"$$\\frac{\\partial \\mathcal{L}}{\\partial \\theta}=\\underbrace{\\frac{\\partial \\mathcal{L}}{\\partial h}}_{\\mbox{given}} \\underbrace{\\frac{\\partial h}{\\partial \\theta}}_{\\mbox{easy}} \\mbox{ , } \\frac{\\partial \\mathcal{L}}{\\partial x}=\\underbrace{\\frac{\\partial \\mathcal{L}}{\\partial h}}_{\\mbox{given}} \\underbrace{\\frac{\\partial h}{\\partial x}}_{\\mbox{easy}}$$\n",
"- We need only $\\frac{\\partial \\mathcal{L}}{\\partial \\theta}$ for gradient descent. Why compute $\\frac{\\partial \\mathcal{L}}{\\partial x}$? \n",
" - The previous layer needs it because $x$ is the output of the previous layer.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Idea of Back-Propagation\n",
"![](\"images/nn_backward.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Idea of Back-Propagation\n",
"![](\"images/nn_backward_2.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Idea of Back-Propagation\n",
"![](\"images/nn_backward_3.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Idea of Back-Propagation\n",
"![](\"images/nn_backward_4.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Back-Propagation Algorithm\n",
"- Compute $\\nabla_{\\textbf{y}} \\mathcal{L}=\\left[\\frac{\\partial\\mathcal{L}}{\\partial y_1}, ..., \\frac{\\partial\\mathcal{L}}{\\partial y_n}\\right]$ directly from the loss function.\n",
"- For each layer (from top to bottom) with output $\\textbf{h}$, input $\\textbf{x}$, and weights $\\textbf{W}$,\n",
" - Assuming that $\\nabla_{\\textbf{h}}\\mathcal{L}$ is given, compute gradients using the **chain rule** as follows: \n",
"$$\\nabla_{\\textbf{W}}\\mathcal{L}=\\nabla_{\\textbf{h}}\\mathcal{L}\\nabla_{\\textbf{W}}\\textbf{h}$$\n",
"$$\\nabla_{\\textbf{x}}\\mathcal{L}=\\nabla_{\\textbf{h}}\\mathcal{L}\\nabla_{\\textbf{x}}\\textbf{h}$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Practice: Linear\n",
"- Forward: $h_i=\\sum_{j}w_{ij}x_j + b_i \\iff \\textbf{h} = \\textbf{W}\\textbf{x} + \\textbf{b}$ \n",
"- Gradient w.r.t. parameters\n",
"$$ \\frac{\\partial \\mathcal{L}}{\\partial w_{ij}} = \\frac{\\partial \\mathcal{L}}{\\partial h_{i}}\\frac{\\partial h_i}{\\partial w_{ij}}=\\frac{\\partial \\mathcal{L}}{\\partial h_{i}}x_j \\iff \\nabla_{\\textbf{W}} \\mathcal{L} = \\nabla_{\\textbf{h}}\\mathcal{L}\\textbf{x}^{\\top}$$\n",
"$$ \\nabla_{\\textbf{b}} \\mathcal{L} = \\nabla_{\\textbf{h}}\\mathcal{L} $$\n",
"- Gradient w.r.t. inputs\n",
"$$ \\frac{\\partial \\mathcal{L}}{\\partial x_{j}} = \\sum_{i} \\frac{\\partial \\mathcal{L}}{\\partial h_{i}}\\frac{\\partial h_i}{\\partial x_{j}}=\\sum_{i} \\frac{\\partial \\mathcal{L}}{\\partial h_{i}}w_{ij} \\iff \\nabla_{\\textbf{x}} \\mathcal{L} = \\textbf{W}^{\\top}\\nabla_{\\textbf{h}}\\mathcal{L} $$\n",
"![](\"images/linear_back.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Practice: Non-Linear Activation (Sigmoid) \n",
"- Forward: $ h_i=\\sigma (x_i) = \\frac{1}{1+\\exp\\left(-x_i\\right)} \\iff \\textbf{h} = \\sigma \\left(\\textbf{x} \\right) $ \n",
"- Gradient w.r.t. inputs\n",
"$$ \\frac{\\partial \\mathcal{L}}{\\partial x_{i}} = \\frac{\\partial \\mathcal{L}}{\\partial h_{i}}\\frac{\\partial h_i}{\\partial x_{i}}=\\frac{\\partial \\mathcal{L}}{\\partial h_{i}}\\sigma(x_i)(1-\\sigma(x_i))=\\frac{\\partial \\mathcal{L}}{\\partial h_{i}}h_i(1-h_i)$$\n",
"$$ \\nabla_{\\textbf{x}} \\mathcal{L} = \\nabla_{\\textbf{h}}\\mathcal{L} \\odot \\textbf{h} \\odot (\\textbf{1} - \\textbf{h}) $$\n",
"![](\"images/activation_back.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Example Code (Torch/Lua)\n",
"```lua\n",
"require \"torch\"\n",
"require \"nn\"\n",
"\n",
"dataX = torch.Tensor(1000, 3)\n",
"dataY = torch.Tensor(1000):random(2)\n",
"-- network construction\n",
"-- 3 -> 4 -> 3 -> 2\n",
"model = nn.Sequential()\n",
"model:add(nn.Linear(3,4))\n",
"model:add(nn.Sigmoid())\n",
"model:add(nn.Linear(4,3))\n",
"model:add(nn.Sigmoid())\n",
"model:add(nn.Linear(3,2))\n",
"model:add(nn.LogSoftMax())\n",
"-- loss function (cross entropy)\n",
"criterion = nn.ClassNLLCriterion() \n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"```lua\n",
"for iter=1,10000 do\n",
" -- sample an example from a dataset\n",
" idx = torch.random(1000)\n",
" x = dataX[idx]\n",
" y = dataY[idx]\n",
" -- forward propagation\n",
" y_pred = model:forward(x)\n",
" loss = criterion:forward(y_pred, y)\n",
" -- backward propagation\n",
" model:zeroGradParameters() -- dL/dW = 0\n",
" -- compute dL/dy\n",
" dL_dy = criterion:backward(y_pred, y) \n",
" -- compute gradient w.r.t. weights\n",
" model:backward(x, dL_dy) \n",
" -- SGD update with learning rate of 0.001\n",
" model:updateParameters(0.001) \n",
"end\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### FAQ: Does a neural network have to be always layer-structured?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- No. It can be any directed acyclic graph (DAG).\n",
"- Example of a complex neural network\n",
"![](\"images/dag.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### FAQ: Can we define any arbitrary layers?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- We can define any layer as long as it is differentiable.\n",
"- Example) Addition layer\n",
" - Forward: $\\textbf{h} = \\textbf{x}_1 + \\textbf{x}_2$\n",
" - Backward\n",
" - $\\nabla_{\\textbf{x}_1}\\mathcal{L} = \\nabla_{\\textbf{h}}\\mathcal{L}\\nabla_{\\textbf{x}_1}\\textbf{h}=\\nabla_{\\textbf{h}}\\mathcal{L}$\n",
" - $\\nabla_{\\textbf{x}_2}\\mathcal{L} = \\nabla_{\\textbf{h}}\\mathcal{L}\\nabla_{\\textbf{x}_2}\\textbf{h}=\\nabla_{\\textbf{h}}\\mathcal{L}$\n",
"![](\"images/addition.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### FAQ: How to handle shared weights?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- To constrain $W_1=W_2=W$, we need $\\Delta W_1 = \\Delta W_2$.\n",
"- Compute $\\nabla_{W_1}\\mathcal{L}$ and $\\nabla_{W_2}\\mathcal{L}$ separately.\n",
"- Use $\\nabla_{W}\\mathcal{L}=\\nabla_{W_1}\\mathcal{L}+\\nabla_{W_2}\\mathcal{L}$ to update the shared weight.\n",
"- In practice, we accumulate gradients to the shared memory space for $\\nabla_{W}\\mathcal{L}$ during back-propagation.\n",
"- Weight sharing is used in *convolutional neural networks* and *recurrent neural networks*.\n",
"![](\"images/weight_sharing.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Outline\n",
"\n",
"- Motivation\n",
"- Basics of Neural Networks\n",
" - Forward Propagation\n",
" - Backward Propagation\n",
"- **Deep Neural Networks**\n",
" - Convolutional Neural Networks\n",
" - Recurrent Neural Networks\n",
"- Applications\n",
" - Computer Vision\n",
" - Natural Language Processing\n",
" - Reinforcement Learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### What is \"deep\" neural network?\n",
"- A neural network is considered to be **deep** if it has more than two (non-linear) hidden layers..\n",
"- Higher layers extract more abstract and hierarchical features.\n",
" - In a classification network, features become more *linearly separable* as layer goes up\n",
"![](\"images/deep_nn.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### What is \"deep\" neural network?\n",
"- Difficulties in training deep neural networks\n",
" - Easily overfit (The number of parameters is large)\n",
" - Hard to optimize (highly non-convex optimization)\n",
" - Computationally expensive (many matrix multiplications)\n",
"- Recent Advances\n",
" - Large-scale dataset (e.g., 1M images in ImageNet)\n",
" - Better regularization (e.g., Dropout)\n",
" - Better optimization (e.g., RMSProp, ReLU) \n",
" - Better hardware (GPU for matrix computation)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Popular Deep Architectures\n",
"- Convolutional Neural Network (CNN)\n",
" - Widely used for image modeling\n",
" - ex) object recognition, segmentation, vision-based reinforcement learning problems\n",
"- Recurrent Neural Network (RNN)\n",
" - Widely used for sequential data modeling\n",
" - ex) machine translation, image caption generation"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Outline\n",
"\n",
"- Motivation\n",
"- Basics of Neural Networks\n",
" - Forward Propagation\n",
" - Backward Propagation\n",
"- Deep Neural Networks\n",
" - **Convolutional Neural Networks**\n",
" - Recurrent Neural Networks\n",
"- Applications\n",
" - Computer Vision\n",
" - Natural Language Processing\n",
" - Reinforcement Learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Convolutional Neural Network\n",
"- A special kind of multi-layer neural network\n",
"- Designed to recognize visual patterns directly from raw pixels"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Multi-layer Perceptron (MLP)\n",
"- Consider 100x100 input pixels\n",
"- 1 hidden layer with 10000 hidden units\n",
"- **100M parameters** $\\rightarrow$ infeasible!\n",
"$\\rightarrow$ Pixels are locally correlated!\n",
"![](\"images/cnn_ex1.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Locally-connected Neural Network\n",
"- Consider 100x100 input pixels\n",
"- Each unit is connected to 10x10 pixels.\n",
"- Each unit extracts a local pattern from the image.\n",
"- 10000 hidden units\n",
"- **1M parameters** $\\rightarrow$ still too large\n",
"![](\"images/cnn_ex2.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Convolutional Neural Network (one filter)\n",
"- Consider 100x100 input pixels\n",
"- Apply the **same filter (weight)** over the entire image.\n",
"- Hidden units form a 100x100 **feature map**.\n",
"- 10x10 parameters. \n", "$\\rightarrow$ **only captures a single local pattern.**\n", "
![](\"images/cnn_ex3.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Convolutional Neural Network (multiple filters)\n",
"- 100x100 input pixels\n",
"- Apply K number of 10x10 filters. \n",
"- Hidden units form a Kx100x100 feature map.\n",
"- Kx10x10 parameters. \n", "- Num of filters and size of filters are hyperparameters.\n", "
![](\"images/cnn_ex4.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Typical Deep CNN Architecture\n",
"![](\"images/typical_cnn.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Details of Convolution\n",
"![](\"images/convolution_step1.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Details of Convolution: Convolutional Filtering\n",
"- A filter has $\\textbf{W} \\in \\mathbb{R}^{h \\times w}$ weights (bias is omitted for simplicity).\n",
"- Compute inner products between $\\textbf{W}$ and $h \\times w$ input patches by sliding window.\n",
" - The same weight is shared across the entire image.\n",
"- The following animation shows the simplest case: one-channel input, one filter\n",
"
\n",
"(Figure from Stanford UFLDL Tutorial)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Details of Convolution: Convolutional Filtering\n",
"- In general, an input consists of multiple channels.\n",
" - Each filter has $\\textbf{W} \\in \\mathbb{R}^{c \\times h \\times w}$ weight ($c$: \\#channels of input).\n",
"- Applying $K$ different filters $\\rightarrow$ produces a 3D feature map (stacked through channels).\n",
"- Hyperparameters: num of filters, size of filters\n",
"- The following figure shows the general case: multi-channel input, mutilple filters\n",
"
\n",
"(Figure from Yann LeCun)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Details of Convolution: Non-linearity\n",
"- Method: Just apply non-linear function (e.g., Sigmoid, ReLU)\n",
"- ReLU is preferred because it is easier to optimize.\n",
"![](\"images/convolution_step2.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Details of Convolution: Pooling\n",
"![](\"images/convolution_step3.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Details of Convolution: Pooling\n",
"- Method: Take average or maximum over HxW region of input feature.\n",
"- Outcome\n",
" - Shrink the number of hidden units.$\\rightarrow$ reduces the number of parameters at the end.\n",
" - Make features robust to small translations of image.\n",
"- Hyperparameters: pooling method (avg or max), pooling size\n",
"- Often called \"sub-sampling\"\n",
"
\n",
"(Figure from Stanford UFLDL Tutorial)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Illustration of Deep CNN\n",
"- Feature maps become smaller in higher layers due to pooling.\n",
"- Hidden units in higher layers capture patterns from larger input patches.\n",
"
\n",
"(Figure from Yann LeCun)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Example Code (Torch/Lua)\n",
"```lua\n",
"require \"torch\"\n",
"require \"nn\"\n",
"model = nn.Sequential()\n",
"-- 1st convolution: 3 -> 64 (5x5 filters)\n",
"model:add(nn.SpatialConvolution(3, 64, 5, 5))\n",
"model:add(nn.ReLU())\n",
"model:add(nn.SpatialMaxPooling(2,2,2,2)) -- 2x2 pooling\n",
"-- 2nd convolution: 64 -> 128 (5x5 filters)\n",
"model:add(nn.SpatialConvolution(64, 128, 5, 5))\n",
"model:add(nn.ReLU())\n",
"model:add(nn.SpatialMaxPooling(2,2,2,2))\n",
"-- Reshape 3D map (128*5*5) to a long vector\n",
"model:add(nn.View(-1):setNumInputDims(3)) \n",
"-- 3rd linear: 128*5*5 -> 128 hidden units\n",
"model:add(nn.Linear(128*5*5, 128))\n",
"model:add(nn.ReLU())\n",
"-- 4th linear: 128 -> 10 classes\n",
"model:add(nn.Linear(128, 10))\n",
"model:add(nn.LogSoftMax()))\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### What is learned by CNN? Filter Visualization\n",
"- Train a deep CNN on *ImageNet* (1.2M images, 1000 classes)\n",
"- Perform forward propagations from many examples\n",
"- Find image patches that strongly activate a specific feature map (filter)\n",
"- Reconstruct the input patch from the feature map\n",
"- Proposed by Zeiler and Fergus (ECCV 2014)\n",
"\n",
"(Figure from Yann LeCun)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Filter Visualization: 1st and 2nd Layer\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Filter Visualization: 3rd Layer\n",
"- Shows more complex patterns\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Filter Visualization: 4th Layer\n",
"- More class-specific\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Filter Visualization: 5th Layer\n",
"- Shows entire objects with pose variations\n",
"- Each filter can be viewed as a part detector. (e.g., dog face, text, animal leg)\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Comparison to Traditional Approach\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### ImageNet Challenge 2012\n",
"- AlexNet (7 layers, Krizhevsky et al.) achieved 16.4% error.\n",
"- The next best model (non-CNN) achieved 26.2% error.\n",
"\n",
"\n",
"(Figure from Alex Krizhevsky et al.)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### ImageNet Challenge\n",
"- ImageNet 2013: Clarifi (7 layers) $\\rightarrow$ 14.8% error"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- ImageNet 2014: GoogLeNet (22 layers) $\\rightarrow$ 4.9% error (Human: 5.1% error)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- ImageNet 2015: ResNet (152 layers!) $\\rightarrow$ 3.5% error"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"source": [
"### Feature Generalization: Dataset\n",
"- Pre-train a CNN on a large-scale dataset (ImageNet) and **train only the final linear layer** on another dataset.\n",
"- Achieves state-of-the-art results on small datasets (e.g., Caltech-101).\n",
"- The learned features can generalize to any dataset.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Feature Generalization: Task\n",
"- There is a large-scale dataset (ImageNet) for classification, but there are only small-scale datasets for other tasks (e.g., detection, segmentation).\n",
"- Pre-train a CNN on a large-scale classification dataset.\n",
"- Use the pre-trained CNN as \"feature extraction\" method for other tasks.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Feature Generalization: Task\n",
"- Achieves state-of-the-art results on many other vision tasks.\n",
" - ex) object detection, segmentation, depth map prediction, image caption generation\n",
"- A CNN trained on a large-scale classification dataset learns a generic feature that can be used for many different vision tasks.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Summary of Convolutional Neural Network\n",
"- Convolutional Neural Network: a special kind of neural network with local connectivity and weight sharing.\n",
"- Achieves state-of-the-art performances on many different computer visoin tasks.\n",
"- Higher layers extract high-level features (e.g., dog face).\n",
"- Learned features can be generally used for other vision tasks."
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.11"
}
},
"nbformat": 4,
"nbformat_minor": 0
}