{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "295b7519",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import IPython\n",
    "\n",
    "import openpifpaf\n",
    "openpifpaf.show.Canvas.show = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "507ff862",
   "metadata": {},
   "source": [
    "# NuScenes 2D detection\n",
    "<div style=\"text-align: right\"> by <a href=\"https://www.linkedin.com/in/duncan-zauss/\">Duncan Zauss</a>, 10/08/2021</div> <br />\n",
    "This is an extension to OpenPifPaf to make 2D detections with the objects from the NuScenes dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9af39c74",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "We provide a pretrained checkpoint to predict the objects from NuScenes. The required checkpoint will be automatically downloaded if you provide \"--checkpoint=shufflenetv2k16-nuscenes\" as a CLI argument. Below an example prediction is shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c48bff6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python -m openpifpaf.predict nuscenes/n013-2018-08-28-10-41-15+0800__CAM_FRONT__1535424448912407.jpg \\\n",
    "    --checkpoint=shufflenetv2k16-nuscenes --image-output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cf35743",
   "metadata": {},
   "outputs": [],
   "source": [
    "IPython.display.Image('nuscenes/n013-2018-08-28-10-41-15+0800__CAM_FRONT__1535424448912407.jpg.predictions.jpeg')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62896a2f",
   "metadata": {},
   "source": [
    "## Data download and preprocessing\n",
    "If you wish to train or evaluate a model, you will need create a NuScenes account and then download the NuScenes dataset from [this link](https://www.nuscenes.org/download). Only the metadata and the sample images are required, there is no need to download the Sweep images (unannotated previous and following frames).\n",
    "Subsequently, you need to process the image annotations to MS COCO style annotations with [this script](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/data_converter/nuimage_converter.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdba73a6",
   "metadata": {},
   "source": [
    "## Train\n",
    "\n",
    "Our model was trained with the following command:<br/>\n",
    "```\n",
    "python3 -m openpifpaf.train --dataset=nuscenes --lr=0.0001 --momentum=0.95 --b-scale=10.0 --clip-grad-value=10 --epochs=150 --lr-decay 120 140 --lr-decay-epochs=10 --batch-size=32 --weight-decay=1e-5 --nuscenes-upsample=2 --nuscenes-extended-scale --nuscenes-orientation-invariant=0.1 --basenet=shufflenetv2k16\n",
    "```\n",
    "\n",
    "## Evaluation\n",
    "To evaluate your network you can use the following command:<br/>\n",
    "```\n",
    "python3 -m openpifpaf.eval --seed-threshold=0.3 --instance-threshold=0.3 --dataset=nuscenes --checkpoint=shufflenetv2k16-nuscenes\n",
    "```\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}