{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Object pose estimation"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"## Overview\n",
"\n",
"In this tutorial, we will show how to optimize the pose of an object while correctly accounting for the visibility discontinuities. We are going to optimize several latent variables that control the translation and rotation of the object.\n",
"\n",
"In differentiable rendering, we aim to evaluate the derivative of a pixel intensity integral with respect to a scene parameter $\\pi$ as follows:\n",
"\n",
"\\begin{equation}\n",
"\\partial_\\pi I(\\pi) = \\partial_\\pi \\int_P f(\\textbf{x}, \\pi) ~ d\\textbf{x}\n",
"\\end{equation}\n",
"\n",
"where $\\textbf{x}$ is a light path in the path space $P$.\n",
"\n",
"When the function $f(\\cdot)$ is continuous w.r.t. $\\pi$, we can move the derivative into the integral and then apply Monte Carlo integration. Under this assumption, differentiating the rendering process via automatic differentiation, as in the previous tutorials, is correct.\n",
"\n",
"However, if $f(\\cdot)$ has discontinuities w.r.t. $\\pi$, direct application of automatic differentiation is not correct anymore, as it omits an integral term given by the [Reynolds transport theorem](https://en.wikipedia.org/wiki/Reynolds_transport_theorem). This needs to be considered when differentiating shape-related parameters (e.g., position), as the discontinuities in the visiblity function (the silhouette of the object) are then dependent on the differentiated parameter.\n",
"\n",
"In the last years, several works tried to address this issue (e.g., Li et al. (2018), Zhang et al. (2020), Loubet et al. (2019), Bangaru et al. (2020), Zhang et al. (2023), ...). Mitsuba provides dedicated integrators implementing the \"*projective sampling*\"-based approach (Zhang et al. (2023)).\n",
"\n",
"- [direct_projective
][1]: projective sampling direct illumination integrator\n",
"- [prb_projective
][2]: projective sampling wth Path Replay Backpropagation (PRB) integrator\n",
"\n",
"In this tutorial, we will optimize the position and rotation of a mesh in order to match a target rendering. To keep things simple, we will use the `direct_projective` integrator.\n",
"You will learn more about this integrator in the following tutorials.\n",
"\n",
"\n",
"