{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p align=\"center\"><img src=\"https://raw.githubusercontent.com/IntelRealSense/librealsense/master/doc/img/realsense.png\" width=\"70%\" /><br><br></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZkVvgf-z4D1f"
   },
   "source": [
    "## Getting the Data\n",
    "This notebook is intended to showcase effect of post processing filters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "y2jplwqU4OHe",
    "outputId": "7a1685ad-c7ed-4fb9-9d76-eb728609f97f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data is Ready\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'wget' is not recognized as an internal or external command,\n",
      "operable program or batch file.\n"
     ]
    }
   ],
   "source": [
    "!wget -N https://librealsense.intel.com/rs-tests/TestData/stairs.bag\n",
    "print(\"Data is Ready\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to extract and visualise the data we will need to import python libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Environment Ready\n"
     ]
    }
   ],
   "source": [
    "import numpy as np                        # fundamental package for scientific computing\n",
    "import matplotlib.pyplot as plt           # 2D plotting library producing publication quality figures\n",
    "import pyrealsense2 as rs                 # Intel RealSense cross-platform open-source API\n",
    "print(\"Environment Ready\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CwiY9raM5nCH"
   },
   "source": [
    "## Capturing Depth Frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "cV1OkBYn55VH",
    "outputId": "0fbae060-91a8-4f2f-acf8-c20a0cf33dcd"
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Failed to resolve request. Request to enable_device_from_file(\"stairs.bag\") was invalid, Reason: Failed to create ros reader: Error opening file: stairs.bag",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-6-ac59184d4988>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mcfg\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconfig\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mcfg\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0menable_device_from_file\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"stairs.bag\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mprofile\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpipe\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcfg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;31m# Skip 5 first frames to give the Auto-Exposure time to adjust\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mRuntimeError\u001b[0m: Failed to resolve request. Request to enable_device_from_file(\"stairs.bag\") was invalid, Reason: Failed to create ros reader: Error opening file: stairs.bag"
     ]
    }
   ],
   "source": [
    "# Setup:\n",
    "pipe = rs.pipeline()\n",
    "cfg = rs.config()\n",
    "cfg.enable_device_from_file(\"stairs.bag\")\n",
    "profile = pipe.start(cfg)\n",
    "\n",
    "# Skip 5 first frames to give the Auto-Exposure time to adjust\n",
    "for x in range(5):\n",
    "  pipe.wait_for_frames()\n",
    "  \n",
    "# Store next frameset for later processing:\n",
    "frameset = pipe.wait_for_frames()\n",
    "depth_frame = frameset.get_depth_frame()\n",
    "\n",
    "# Cleanup:\n",
    "pipe.stop()\n",
    "print(\"Frames Captured\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JxlrWpLj6mkJ"
   },
   "source": [
    "## Visualising the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 321
    },
    "colab_type": "code",
    "id": "5ajjn7Wq6nEg",
    "outputId": "737674f4-5314-470a-99c0-662f35444e02"
   },
   "outputs": [],
   "source": [
    "colorizer = rs.colorizer()\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(depth_frame).get_data())\n",
    "\n",
    "plt.rcParams[\"axes.grid\"] = False\n",
    "plt.rcParams['figure.figsize'] = [8, 4]\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying Filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decimation\n",
    "When using Depth-from-Stereo solution, z-accuracy is related to original spacial resolution. \n",
    "\n",
    "If you are satisfied with lower spatial resolution, the [**Decimation Filter**](https://github.com/IntelRealSense/librealsense/blob/master/doc/post-processing-filters.md#decimation-filter) will reduce spatial resolution preserving z-accuracy and performing some rudamentary hole-filling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decimation = rs.decimation_filter()\n",
    "decimated_depth = decimation.process(depth_frame)\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(decimated_depth).get_data())\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can control the amount of decimation (liniar scale factor) via `filter_magnitude` option.\n",
    "\n",
    "> Note the changing image resolution\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decimation.set_option(rs.option.filter_magnitude, 4)\n",
    "decimated_depth = decimation.process(depth_frame)\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(decimated_depth).get_data())\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spatial Filter\n",
    "\n",
    "[**Spatial Filter**](https://github.com/IntelRealSense/librealsense/blob/master/doc/post-processing-filters.md#spatial-edge-preserving-filter) is a fast implementation of [Domain-Transform Edge Preserving Smoothing](http://inf.ufrgs.br/~eslgastal/DomainTransform/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spatial = rs.spatial_filter()\n",
    "filtered_depth = spatial.process(depth_frame)\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(filtered_depth).get_data())\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can emphesize the effect of the filter by cranking-up `smooth_alpha` and `smooth_delta` options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spatial.set_option(rs.option.filter_magnitude, 5)\n",
    "spatial.set_option(rs.option.filter_smooth_alpha, 1)\n",
    "spatial.set_option(rs.option.filter_smooth_delta, 50)\n",
    "filtered_depth = spatial.process(depth_frame)\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(filtered_depth).get_data())\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The filter also offers some basic spatial **hole filling** capabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spatial.set_option(rs.option.holes_fill, 3)\n",
    "filtered_depth = spatial.process(depth_frame)\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(filtered_depth).get_data())\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Temporal Filter\n",
    "\n",
    "Our implementation of [**Temporal Filter**](https://github.com/IntelRealSense/librealsense/blob/master/doc/post-processing-filters.md#temporal-filter) does basic temporal smoothing and hole-filling. It is meaningless when applied to a single frame, so let's capture several consecutive frames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profile = pipe.start(cfg)\n",
    "\n",
    "frames = []\n",
    "for x in range(10):\n",
    "    frameset = pipe.wait_for_frames()\n",
    "    frames.append(frameset.get_depth_frame())\n",
    "\n",
    "pipe.stop()\n",
    "print(\"Frames Captured\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to \"feed\" the frames to the filter one by one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temporal = rs.temporal_filter()\n",
    "for x in range(10):\n",
    "    temp_filtered = temporal.process(frames[x])\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(temp_filtered).get_data())\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can modify filter options to fine tune the result (with any temporal filtering there is a trade-off between smoothing and motion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hole Filling\n",
    "[Hole Filling](https://github.com/IntelRealSense/librealsense/blob/master/doc/post-processing-filters.md#holes-filling-filter) filter offers additional layer of depth exterpolation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hole_filling = rs.hole_filling_filter()\n",
    "filled_depth = hole_filling.process(depth_frame)\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(filled_depth).get_data())\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting Everything Together\n",
    "These filters work best when applied sequentially one after another. At longer range, it also helps using `disparity_transform` to switch from depth representation to disparity form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "depth_to_disparity = rs.disparity_transform(True)\n",
    "disparity_to_depth = rs.disparity_transform(False)\n",
    "\n",
    "for x in range(10):\n",
    "    frame = frames[x]\n",
    "    frame = decimation.process(frame)\n",
    "    frame = depth_to_disparity.process(frame)\n",
    "    frame = spatial.process(frame)\n",
    "    frame = temporal.process(frame)\n",
    "    frame = disparity_to_depth.process(frame)\n",
    "    frame = hole_filling.process(frame)\n",
    "\n",
    "colorized_depth = np.asanyarray(colorizer.colorize(frame).get_data())\n",
    "plt.imshow(colorized_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "> **Want to learn more?** Visit [realsense.intel.com](http://realsense.intel.com) and [github.com/IntelRealSense](http://www.github.com/IntelRealSense/librealsense)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of Intel RealSense API",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}