{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "4263500859b02fc63f33bc5461337abc",
     "grade": false,
     "grade_id": "cell-b830011d03d5028e",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Part 4: More advanced networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "1f5b883c3113868b6b5aa1b0043cb459",
     "grade": false,
     "grade_id": "cell-fdfa95dffc89d2d0",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "__Before starting, we recommend you enable GPU acceleration if you're running on Colab.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute this code block to install dependencies when running on colab\n",
    "try:\n",
    "    import torch\n",
    "except:\n",
    "    from os.path import exists\n",
    "    from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
    "    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
    "    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\\.\\([0-9]*\\)\\.\\([0-9]*\\)$/cu\\1\\2/'\n",
    "    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'\n",
    "\n",
    "    !pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision\n",
    "\n",
    "try: \n",
    "    import torchbearer\n",
    "except:\n",
    "    !pip install torchbearer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "88d2e37fa6a6583c9e38d5921d9991aa",
     "grade": false,
     "grade_id": "cell-05c1755ee9c9a7d5",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Recent network models, such as the deep residual network (ResNet) and GoogLeNet architectures, do not follow a straight path from input to output. Instead, these models incorporate branches and merges to create a computation graph. Branching and merging is easy to implement in PyTorch as shown in the following code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "ed43e83bd5a43023f803ee59d9d8fd36",
     "grade": false,
     "grade_id": "cell-12d0f2493e03d080",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import torch \n",
    "import torch.nn.functional as F\n",
    "from torch import nn\n",
    "\n",
    "class BranchModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(BranchModel, self).__init__()\n",
    "        self.left  = nn.Conv2d(1, 16, (1, 1), padding=0)\n",
    "        self.right = nn.Conv2d(1, 16, (5, 5), padding=2)\n",
    "        self.fc1 = nn.Linear(16*14*14, 128)\n",
    "        self.fc2 = nn.Linear(128, 10)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        out_l = self.left(x)\n",
    "        out_l = F.relu(out_l)\n",
    "\n",
    "        out_r = self.right(x)\n",
    "        out_r = F.relu(out_r)\n",
    "\n",
    "        out = out_l + out_r\n",
    "        \n",
    "        out = F.max_pool2d(out, (2,2))\n",
    "        out = F.dropout(out, 0.2)\n",
    "        out = out.view(out.shape[0], -1)\n",
    "        out = self.fc1(out)\n",
    "        out = F.relu(out)\n",
    "        out = self.fc2(out)\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "c79212397c10e559b0fc063595a4cfb3",
     "grade": false,
     "grade_id": "cell-9a097578d70c702d",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "This defines a variant of our initial simple CNN model in which the input is split into two paths and then merged again; the left hand path consists of a 1x1 convolution layer, whilst the right-hand path has a 5x5 convolutional layer. The 1x1 convolutions will have the effect of increasing the number of bands in the input from 1 to 16 (with each band a (potentially different) scalar multiple of the input). Padding is used to ensure the feature maps have the same shape on the left and right branches. In this case the left and right branches are merged by summing them together (element-wise, layer by layer).\n",
    "\n",
    "__Use the code block below to train and evaluate the above model.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "fde8d9e6d7b897c164a58dea2fc73adc",
     "grade": false,
     "grade_id": "cell-b3c56ef27392430d",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "0041b3fafbad76d0e75e0ef14a21dc36",
     "grade": false,
     "grade_id": "cell-83daaee56b0f4107",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Going further\n",
    "\n",
    "None of the network topology we have experimented with thus far are optimised. Nor are they reproductions of network topologies from recent papers.\n",
    "\n",
    "__There is a lot of opportunity for you to tune and improve upon these models. What is the best error rate score you can achieve?__\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}