{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Pandora: a new stereo matching framework\n",
"<img src=\"img/logo-cnes-triangulaire.jpg\" width=\"200\" height=\"200\">\n",
"\n",
"*Cournet, M., Sarrazin, E., Dumas, L., Michel, J., Guinet, J., Youssefi, D., Defonte, V., Fardet, Q., 2020. Ground-truth generation and disparity estimation for optical satellite imagery. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction and basic usage"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Imports and external functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"from IPython.display import Image, display\n",
"import io\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import os\n",
"from pathlib import Path\n",
"import rasterio"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from snippets.utils import *"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_image(img, title=None, output_dir=None, cmap=\"viridis\"):\n",
" fig = plt.figure()\n",
" plt.title(title)\n",
" plt.imshow(img, cmap=cmap, vmin=np.min(img), vmax=np.max(img))\n",
" plt.colorbar()\n",
" if output_dir is not None:\n",
" fig.savefig(os.path.join(output_dir,title + '.pdf'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_state_machine(machine):\n",
" stream = io.BytesIO()\n",
" try:\n",
" pandora_machine.get_graph().draw(stream, prog='dot', format='png')\n",
" display(Image(stream.getvalue()))\n",
" except:\n",
" print(\"It is not possible to show the graphic of the state machine. To solve it, please install graphviz on your system (apt-get install graphviz if operating in Linux) and install python package with pip install graphviz\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# What is Pandora ?\n",
"\n",
"* Pandora is a Toolbox to estimate disparity \n",
"* It is inspired by the work of [Scharstein 2002]\n",
"* Pandora embeds several state-of-art algorithms\n",
"* It is easy to configure and modular\n",
"* Will be used in the 3D reconstruction pipeline CARS for CO3D mission \n",
"\n",
"[Scharstein 2002] *A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms*, D. Scharstein and R. Szeliski, \n",
"vol. 47, International Journal of Computer Vision}, 2002"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"img/logo_diagram.png\" width=\"500\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Inputs\n",
"\n",
"* Stereo rectified image pair (with associated masks)\n",
"* Disparity range to explore \n",
"* Configuration file\n",
"\n",
"## Outputs\n",
"\n",
"* Disparity and validity maps in left image geometry \n",
"* Disparity and validity maps in righ image geometry (*optional*)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pandora's pipeline\n",
"\n",
"Pandora provides the following steps:\n",
"* matching cost computation (**mandatory**)\n",
"* cost aggregation\n",
"* cost optimization\n",
"* disparity computation (**mandatory**)\n",
"* subpixel disparity refinement\n",
"* disparity filtering\n",
"* validation\n",
"* multiscale processing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"img/schema_complet.png\" width=\"1000\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Available implementations for each step\n",
"\n",
"| Step | Algorithms implemented |\n",
"|:--------------------------|:-----------------------|\n",
"| Matching cost computation | Census / SAD / SSD / ZNNC / MC-CNN |\n",
"| Cost aggregation | Cross Based Cost Aggregation |\n",
"| Cost optimization | SGM |\n",
"| Disparity computation | Winner-Take-All |\n",
"| Subpixel disparity refinement | Vfit / Quadratic |\n",
"| Disparity filtering | Median / Bilateral |\n",
"| Validation | Cross checking |\n",
"| Multiscale | Fixed zoom pyramid |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Pandora execution options with state machine"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Imports of pandora "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Load pandora imports\n",
"import pandora\n",
"from pandora.img_tools import read_img\n",
"from pandora.check_json import check_pipeline_section, concat_conf, memory_consumption_estimation\n",
"from pandora.state_machine import PandoraMachine\n",
"from pandora import import_plugin, check_conf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### (Optional) If Pandora plugins are to be used, import them"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Available Pandora Plugins include :\n",
"- MC-CNN Matching cost computation\n",
"- SGM Optimization"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Load plugins\n",
"import_plugin()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Load and visualize input data "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Provide image path"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Paths to left and right images\n",
"img_left_path = \"data/Cones_LEFT.tif\"\n",
"img_right_path = \"data/Cones_RIGHT.tif\"\n",
"# Paths to masks (None if not provided)\n",
"left_mask_path = None\n",
"right_mask_path = None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Provide image configuration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"image_cfg = {'image': {'no_data_left': np.nan, 'no_data_right': np.nan}}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Provide output directory to write results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"output_dir = os.path.join(os.getcwd(),\"output\")\n",
"# If necessary, create output dir\n",
"Path(output_dir).mkdir(exist_ok=True,parents=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Convert input data to dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_left = read_img(img_left_path, no_data=image_cfg['image']['no_data_left'],\n",
" mask=left_mask_path)\n",
"img_right = read_img(img_right_path, no_data=image_cfg['image']['no_data_right'],\n",
" mask=right_mask_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize input data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_image(img_left.im, \"Left input image\", output_dir, cmap=\"gray\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Option 1 : trigger all the steps of the machine at ones"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Instantiate the machine"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine = PandoraMachine()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Define pipeline configuration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"user_pipeline_cfg = {\n",
" 'pipeline':{\n",
" \"right_disp_map\": {\n",
" \"method\": \"accurate\"\n",
" },\n",
" \"matching_cost\" : {\n",
" \"matching_cost_method\": \"zncc\",\n",
" \"window_size\": 5,\n",
" \"subpix\": 4\n",
" },\n",
" \"disparity\": {\n",
" \"disparity_method\": \"wta\",\n",
" \"invalid_disparity\": \"NaN\"\n",
" },\n",
" \"refinement\": {\n",
" \"refinement_method\": \"quadratic\"\n",
" },\n",
" \"filter\": {\n",
" \"filter_method\": \"median\"\n",
" },\n",
" \"validation\": {\n",
" \"validation_method\": \"cross_checking\"\n",
" },\n",
" \"filter.this_time_after_validation\" : {\n",
" \"filter_method\": \"median\",\n",
" \"filter_size\": 3\n",
" } \n",
" }\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Disparity interval used"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"disp_min = -60\n",
"disp_max = 0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Check the configuration and sequence of steps"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"checked_cfg = check_pipeline_section(user_pipeline_cfg, pandora_machine)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pipeline_cfg = checked_cfg['pipeline']\n",
"print(pipeline_cfg)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Estimate the memory consumption of the pipeline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"min_mem_consump, max_mem_consump = memory_consumption_estimation(user_pipeline_cfg, [img_left_path, disp_min, disp_max], pandora_machine)\n",
"print(\"Estimated maximum memory consumption between {:.2f} GiB and {:.2f} GiB\".format(min_mem_consump, max_mem_consump))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Prepare the machine "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine.run_prepare(pipeline_cfg, img_left, img_right, disp_min, disp_max)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Trigger all the steps of the machine at ones"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"left_disparity, right_disparity = pandora.run(pandora_machine, img_left, img_right, disp_min, disp_max, pipeline_cfg)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize output disparity map"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_image(left_disparity.disparity_map, \"Left disparity map\", output_dir, cmap=pandora_cmap())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Option 2 : trigger the machine step by step"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The implementation of Pandora with a state machine makes it possible to set up a more flexible pipeline, which makes it possible to choose via a configuration file the steps as well as the order of the steps that one wishes to follow in Pandora.\n",
"\n",
"Moreover, the state machine allows to run each step of the pipeline independently, giving the possibility to save and visualize the results after each step. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The state machine has three states : \n",
"* Begin\n",
"* Cost volume\n",
"* Disparity map \n",
"\n",
"Being the connections between them the different steps of the pipeline."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"../doc/sources/Images/Machine_state_diagram.png\" width=\"700\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Instantiate the machine"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine = PandoraMachine()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Define pipeline configuration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"user_pipeline_cfg = {\n",
" 'pipeline':{\n",
" \"right_disp_map\": {\n",
" \"method\": \"accurate\"\n",
" },\n",
" \"matching_cost\" : {\n",
" \"matching_cost_method\": \"zncc\",\n",
" \"window_size\": 5,\n",
" \"subpix\": 4\n",
" },\n",
" \"aggregation\": {\n",
" \"aggregation_method\": \"cbca\"\n",
" },\n",
" \"disparity\": {\n",
" \"disparity_method\": \"wta\",\n",
" \"invalid_disparity\": \"NaN\"\n",
" },\n",
" \"refinement\": {\n",
" \"refinement_method\": \"quadratic\"\n",
" },\n",
" \"filter\": {\n",
" \"filter_method\": \"median\"\n",
" },\n",
" \"validation\": {\n",
" \"validation_method\": \"cross_checking\"\n",
" }\n",
" }\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Disparity interval used"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"disp_min = -60\n",
"disp_max = 0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Check the pipeline configuration and sequence of steps"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"checked_cfg = check_pipeline_section(user_pipeline_cfg, pandora_machine)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(checked_cfg)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pipeline_cfg = checked_cfg['pipeline']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Estimate the memory consumption of the pipeline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"min_mem_consump, max_mem_consump = memory_consumption_estimation(user_pipeline_cfg, [img_left_path, disp_min, disp_max], pandora_machine)\n",
"print(\"Estimated maximum memory consumption between {:.2f} GiB and {:.2f} GiB\".format(min_mem_consump, max_mem_consump))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Prepare the machine "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine.run_prepare(pipeline_cfg, img_left, img_right, disp_min, disp_max)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Trigger the machine step by step"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_state_machine(pandora_machine)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run matching cost "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine.run('matching_cost', pipeline_cfg)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_state_machine(pandora_machine)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine.run('aggregation', pipeline_cfg)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_state_machine(pandora_machine)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run disparity "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine.run('disparity', pipeline_cfg)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_state_machine(pandora_machine)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run refinement "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine.run('refinement', pipeline_cfg)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_state_machine(pandora_machine)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run filter"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine.run('filter', pipeline_cfg)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_state_machine(pandora_machine)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run validation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandora_machine.run('validation', pipeline_cfg)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_state_machine(pandora_machine)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize output disparity map"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_image(pandora_machine.left_disparity.disparity_map, \"Left disparity map\", output_dir, cmap=pandora_cmap())"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}