{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Replacing Fully-Connnected by Equivalent Convolutional Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assume we have a 2x2 input image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 1, 2, 2])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs = torch.tensor([[[[1., 2.],\n",
    "                         [3., 4.]]]])\n",
    "\n",
    "inputs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fully Connected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A fully connected layer, which maps the 4 input features two 2 outputs, would be computed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc = torch.nn.Linear(4, 2)\n",
    "\n",
    "weights = torch.tensor([[1.1, 1.2, 1.3, 1.4],\n",
    "                        [1.5, 1.6, 1.7, 1.8]])\n",
    "bias = torch.tensor([1.9, 2.0])\n",
    "fc.weight.data = weights\n",
    "fc.bias.data = bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[14.9000, 19.0000]], grad_fn=<ReluBackward0>)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.relu(fc(inputs.view(-1, 4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolution with Kernels equal to the input size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../images/fc-to-conv/fc-to-conv-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can obtain the same outputs if we use convolutional layers where the kernel size is the same size as the input feature array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 1, 2, 2])\n",
      "torch.Size([2])\n"
     ]
    }
   ],
   "source": [
    "conv = torch.nn.Conv2d(in_channels=1,\n",
    "                       out_channels=2,\n",
    "                       kernel_size=inputs.squeeze(dim=(0)).squeeze(dim=(0)).size())\n",
    "print(conv.weight.size())\n",
    "print(conv.bias.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv.weight.data = weights.view(2, 1, 2, 2)\n",
    "conv.bias.data = bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[14.9000]],\n",
       "\n",
       "         [[19.0000]]]], grad_fn=<ReluBackward0>)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.relu(conv(inputs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolution with 1x1 Kernels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../images/fc-to-conv/fc-to-conv-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can replace the fully connected layer using a convolutional layer when we reshape the input image into a num_inputs x 1 x 1 image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[14.9000]],\n",
       "\n",
       "         [[19.0000]]]], grad_fn=<ReluBackward0>)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conv = torch.nn.Conv2d(in_channels=4,\n",
    "                       out_channels=2,\n",
    "                       kernel_size=(1, 1))\n",
    "\n",
    "conv.weight.data = weights.view(2, 4, 1, 1)\n",
    "conv.bias.data = bias\n",
    "torch.relu(conv(inputs.view(1, 4, 1, 1)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}