{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Object pose estimation"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"## Overview\n",
"\n",
"In this tutorial, we will show how to optimize the pose of an object while correctly accounting for the visibility discontinuities. We are going to optimize several latent variables that control the translation and rotation of the object.\n",
"\n",
"In differentiable rendering, we aim to evaluate the derivative of a pixel intensity integral with respect to a scene parameter $\\pi$ as follows:\n",
"\n",
"\\begin{equation}\n",
"\\partial_\\pi I(\\pi) = \\partial_\\pi \\int_P f(\\textbf{x}, \\pi) ~ d\\textbf{x}\n",
"\\end{equation}\n",
"\n",
"where $\\textbf{x}$ is a light path in the path space $P$.\n",
"\n",
"When the function $f(\\cdot)$ is continuous w.r.t. $\\pi$, we can move the derivative into the integral and then apply Monte Carlo integration. Under this assumption, differentiating the rendering process via automatic differentiation, as in the previous tutorials, is correct.\n",
"\n",
"However, if $f(\\cdot)$ has discontinuities w.r.t. $\\pi$, direct application of automatic differentiation is not correct anymore, as it omits an integral term given by the [Reynolds transport theorem](https://en.wikipedia.org/wiki/Reynolds_transport_theorem). This needs to be considered when differentiating shape-related parameters (e.g., position), as the discontinuities in the visiblity function (the silhouette of the object) are then dependent on the differentiated parameter.\n",
"\n",
"In the last years, several works tried to address this issue (e.g., Li et al. (2018), Zhang et al. (2020), Loubet et al. (2019), Bangaru et al. (2020), ...). Mitsuba provides dedicated integrators implementing the *reparameterization*-based approach (Loubet et al. (2019), Bangaru et al. (2020), Zeltner et al. (2021)):\n",
"\n",
"- [prb_reparam
][1]: reparameterized Path Replay Backpropagation (PRB) integrator\n",
"- [direct_reparam
][2]: reparameterized direct illumination integrator\n",
"\n",
"\n",
"In this tutorial, we will optimize the position and rotation of a mesh in order to match a target rendering. To keep things simple, we will use the `direct_reparam` integrator.\n",
"\n",
"\n",
"
mitsuba.ad.Optimizer
](https://mitsuba.readthedocs.io/en/latest/src/api_reference.html#mitsuba.ad.Optimizer)\n",
"- [prb_reparam
plugin](https://mitsuba.readthedocs.io/en/latest/src/generated/plugins_integrators.html#reparameterized-path-replay-backpropagation-integrator-prb-reparam)\n",
"- [direct_reparam
plugin](https://mitsuba.readthedocs.io/en/latest/src/generated/plugins_integrators.html#reparameterized-direct-integrator-direct-reparam)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
},
"vscode": {
"interpreter": {
"hash": "524ef5c892e16adde62d9febd56e921248026795a369fc4d8cb46b781ffa4996"
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}