{ "cells": [ { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0, "slideshow": { "slide_type": "slide" } }, "source": [ "\n", "# MODERN DEEP LEARNING ERA" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0, "slideshow": { "slide_type": "slide" } }, "source": [ "# Breakthrough Paper\n", "\n", "\"ImageNet Classification with Deep Convolutional Neural Networks\", Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, 2013\n", "\n", "- substantially better performance than previous methods\n", "- effective use of GPUs for convolutional neural networks\n", "- ReLU nonlinearity\n", "- max pooling\n", "- data augmentation\n", "- dropout, local response normalization\n", "- much bigger than previous networks\n", "\n", "None of these were new ideas, but this paper brought it all together." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# This took a lot of planning\n", "\n", "Deep Learning didn't come out of nothing.\n", "\n", "The major people in the field today tried to push it for 20 years compared to other approaches.\n", "\n", "Finally succeeded in the 2010's due to Alexnet." ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0, "slideshow": { "slide_type": "slide" } }, "source": [ "# Alexnet Architecture\n", "\n", "![alexnet](figs/alexnet.png)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from torch import nn\n", "from torchmore import flex, layers" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "# Alexnet Architecture (approximately)\n", "\n", "model = nn.Sequential(\n", " layers.Input(\"BDHW\", sizes=(None, 3, 224, 224)),\n", " flex.Conv2d(2*48, (11, 11)),\n", " nn.ReLU(),\n", " nn.MaxPool2d((2, 2)),\n", " flex.Conv2d(2*192, (3, 3)),\n", " nn.ReLU(),\n", " flex.Conv2d(2*192, (3, 3)),\n", " nn.ReLU(),\n", " flex.Conv2d(2*128, (3, 3)),\n", " layers.Reshape(0, [1, 2, 3]),\n", " flex.Linear(4096),\n", " nn.ReLU(),\n", " flex.Linear(4096),\n", " nn.ReLU(),\n", " flex.Linear(1000)\n", ")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "\n", "# Alexnet Learned Features\n", "\n", "![alexnet features](figs/alexnet-features.png)\n", "\n", "NB: these are standard PCA/Gabor-jet style features from vision>" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0, "slideshow": { "slide_type": "slide" } }, "source": [ "# ReLU\n", "\n", "\n", "\n", "$\\sigma(x) = (1 + e^{-x})^{-1}$, $\\rho(x) = \\max(0, x)$" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0, "slideshow": { "slide_type": "slide" } }, "source": [ "# ReLU Derivatives\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0, "slideshow": { "slide_type": "slide" } }, "source": [ "# Nonlinearity Properties\n", "\n", "
property | sigmoid | ReLU |
---|---|---|
derivatives | infinite | f': discontinuous, f'': zero |
monotonicity> | monotonic | monotonic |
range | $(0, 1)$ | $(0, \\infty)$ |
zero derivative | none | $(-\\infty, 0)$ |