{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import openpifpaf\n",
    "\n",
    "openpifpaf.show.Canvas.show = True\n",
    "openpifpaf.show.Canvas.image_min_dpi = 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prediction API\n",
    "\n",
    "This page documents how you can use OpenPifPaf from your own Python code.\n",
    "It focuses on single-image prediction.\n",
    "This API interface is for more advanced use cases. Please refer to \n",
    "{doc}`Getting Started: Prediction <predict_cli>` for documentation on \n",
    "the command line interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import numpy as np\n",
    "import PIL\n",
    "import requests\n",
    "import torch\n",
    "\n",
    "print('OpenPifPaf version', openpifpaf.__version__)\n",
    "print('PyTorch version', torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load an Example Image\n",
    "\n",
    "Image credit: \"[Learning to surf](https://www.flickr.com/photos/fotologic/6038911779/in/photostream/)\" by fotologic which is licensed under [CC-BY-2.0].\n",
    "\n",
    "[CC-BY-2.0]: https://creativecommons.org/licenses/by/2.0/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_response = requests.get('https://raw.githubusercontent.com/openpifpaf/openpifpaf/main/docs/coco/000000081988.jpg')\n",
    "pil_im = PIL.Image.open(io.BytesIO(image_response.content)).convert('RGB')\n",
    "im = np.asarray(pil_im)\n",
    "\n",
    "with openpifpaf.show.image_canvas(im) as ax:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the Predictor API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = openpifpaf.Predictor(checkpoint='shufflenetv2k16')\n",
    "predictions, gt_anns, image_meta = predictor.pil_image(pil_im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can immediately visualize the predicted annotations (`predictions`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotation_painter = openpifpaf.show.AnnotationPainter()\n",
    "with openpifpaf.show.image_canvas(im) as ax:\n",
    "    annotation_painter.annotations(ax, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each prediction in the `predictions` list above is of type `Annotation`. You can access the joint coordinates in the `data` attribute. It is a numpy array that contains the $x$ and $y$ coordinates and the confidence for every joint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions[0].data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Predictor` class can also be created with `json_data=True` and then it will\n",
    "return JSON serializable dicts and list instead of `Annotation` objects.\n",
    "\n",
    "The other items that are returned are ground truth annotations (`gt_anns`) which\n",
    "are not provided for this image and meta information about the image (`image_meta`)\n",
    "which is useful to understand the transformations that were applied before\n",
    "passing the image through the neural network. Usually, you don't need `image_meta`\n",
    "as the inverse transform has already been applied to ground truth and predictions\n",
    "in the `Predictor` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gt_anns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(predict-fields)=\n",
    "## Fields\n",
    "\n",
    "Below are visualizations of the fields.\n",
    "When using the API here, the visualization types are individually enabled.\n",
    "Then, the index for every field to visualize must be specified. In the example below, the fifth CIF (left shoulder) and the fifth CAF (left shoulder to left hip) are activated.\n",
    "\n",
    "These plots are also accessible from the command line: use `--debug-indices cif:5 caf:5` to select which joints and connections to visualize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "openpifpaf.visualizer.Base.set_all_indices(['cif,caf:5:confidence'])\n",
    "_ = predictor.pil_image(pil_im)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "openpifpaf.visualizer.Base.set_all_indices(['cif,caf:5:regression'])\n",
    "_ = predictor.pil_image(pil_im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the CIF field, a high resolution accumulation (in the code it's called `CifHr`) is generated.\n",
    "This is also the basis for the seeds. Both are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "openpifpaf.visualizer.Base.set_all_indices(['cif:5:hr', 'seeds'])\n",
    "_ = predictor.pil_image(pil_im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting from a seed, the poses are constructed. At every joint position, an occupancy map marks whether a previous pose was already constructed here. This reduces the number of poses that are constructed from multiple seeds for the same person. The final occupancy map is below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "openpifpaf.visualizer.Base.set_all_indices(['occupancy:5'])\n",
    "_ = predictor.pil_image(pil_im)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ea6946363a43e80d241452ab397f4c58bdd3d2517da174158e9c46ce6717422a"
  },
  "kernelspec": {
   "display_name": "Python 3.9.4 64-bit ('venv3': venv)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}