{ "cells": [ { "cell_type": "markdown", "id": "b8e61c5c-4f3e-4980-be22-4010a247babd", "metadata": {}, "source": [ "## Streaming Video: Live video streams made easy\n", "\n", "In this example we will demonstrate how to develop a general tool to transform live image streams from the users web cam. We will be using Panels\n", "[`VideoStream`](https://panel.holoviz.org/reference/widgets/VideoStream.html) widget to record and stream the images.\n", "\n", "We will also show how to apply *blur*, *grayscale*, *sobel* and *face recognition* models to the video stream." ] }, { "cell_type": "markdown", "id": "5a3760b6-f8a7-46ad-8831-620a131c1512", "metadata": {}, "source": [ "## Imports and Settings\n", "\n", "Among other things we will be using [numpy](https://numpy.org/), [PIL](https://pillow.readthedocs.io/en/stable/) and [scikit-image](https://scikit-image.org/) to work with the images." ] }, { "cell_type": "code", "execution_count": null, "id": "02b862bc-f710-456b-9a34-f1b74a76df2c", "metadata": {}, "outputs": [], "source": [ "import base64\n", "import io\n", "import time\n", "\n", "import numpy as np\n", "import param\n", "import PIL\n", "import skimage\n", "\n", "from PIL import Image, ImageFilter\n", "from skimage import data, filters\n", "from skimage.color.adapt_rgb import adapt_rgb, each_channel\n", "from skimage.draw import rectangle\n", "from skimage.exposure import rescale_intensity\n", "from skimage.feature import Cascade\n", "\n", "import panel as pn\n", "\n", "pn.extension(design='material', sizing_mode=\"stretch_width\")" ] }, { "cell_type": "markdown", "id": "43e68289-6c33-435f-b279-cf89327266ac", "metadata": {}, "source": [ "We define the *height* and *width* of the images to transform. Smaller is faster.\n", "We also define the *timeout*, i.e. how often the videostream takes and streams a new image." ] }, { "cell_type": "code", "execution_count": null, "id": "7b46542f-fa1a-4dde-a8c6-f023f3358a6c", "metadata": {}, "outputs": [], "source": [ "HEIGHT = 500 # pixels\n", "WIDTH = 500 # pixels\n", "TIMEOUT = 500 # milliseconds" ] }, { "cell_type": "markdown", "id": "bb8b1066-c799-4d6a-94d0-361dd3cc6007", "metadata": {}, "source": [ "## Base Image Models\n", "\n", "We will need to define some *base image models* components. The base models are custom Panel components that inherit from Panels [`Viewer`](https://panel.holoviz.org/how_to/custom_components/custom_viewer.html) class.\n", "\n", "The *base models* makes it easy to later turn *image to image* algorithms into interactive UIs like the `FaceDetectionModel` shown in the image just below.\n", "\n", "\n", "\n", "Please note we restrict our selves to working with `.jpg` images. The `VideoStream` widget also support `.png` images. But `.png` images are much bigger and slower to work with." ] }, { "cell_type": "code", "execution_count": null, "id": "a507eb66-b242-441c-8d89-dba504ec79a3", "metadata": {}, "outputs": [], "source": [ "class ImageModel(pn.viewable.Viewer):\n", " \"\"\"Base class for image models.\"\"\"\n", "\n", " def __init__(self, **params):\n", " super().__init__(**params)\n", "\n", " with param.edit_constant(self):\n", " self.name = self.__class__.name.replace(\"Model\", \"\")\n", " self.view = self.create_view()\n", "\n", " def __panel__(self):\n", " return self.view\n", "\n", " def apply(self, image: str, height: int = HEIGHT, width: int = WIDTH) -> str:\n", " \"\"\"Transforms a base64 encoded jpg image to a base64 encoded jpg BytesIO object\"\"\"\n", " raise NotImplementedError()\n", "\n", " def create_view(self):\n", " \"\"\"Creates a view of the parameters of the transform to enable the user to configure them\"\"\"\n", " return pn.Param(self, name=self.name)\n", "\n", " def transform(self, image):\n", " \"\"\"Transforms the image\"\"\"\n", " raise NotImplementedError()" ] }, { "cell_type": "markdown", "id": "be8256eb-f2bb-4f24-84dc-8b6b47d64958", "metadata": {}, "source": [ "Lets define a base model for working with **`PIL`** images" ] }, { "cell_type": "code", "execution_count": null, "id": "5ebf5560-29f6-4b3f-815e-a70010998ba1", "metadata": {}, "outputs": [], "source": [ "class PILImageModel(ImageModel):\n", " \"\"\"Base class for PIL image models\"\"\"\n", "\n", " @staticmethod\n", " def to_pil_img(value: str, height=HEIGHT, width=WIDTH):\n", " \"\"\"Converts a base64 jpeg image string to a PIL.Image\"\"\"\n", " encoded_data = value.split(\",\")[1]\n", " base64_decoded = base64.b64decode(encoded_data)\n", " image = Image.open(io.BytesIO(base64_decoded))\n", " image.draft(\"RGB\", (height, width))\n", " return image\n", "\n", " @staticmethod\n", " def from_pil_img(image: Image):\n", " \"\"\"Converts a PIL.Image to a base64 encoded JPG BytesIO object\"\"\"\n", " buff = io.BytesIO()\n", " image.save(buff, format=\"JPEG\")\n", " return buff\n", "\n", " def apply(self, image: str, height: int = HEIGHT, width: int = WIDTH) -> io.BytesIO:\n", " pil_img = self.to_pil_img(image, height=height, width=width)\n", "\n", " transformed_image = self.transform(pil_img)\n", "\n", " return self.from_pil_img(transformed_image)\n", "\n", " def transform(self, image: PIL.Image) -> PIL.Image:\n", " \"\"\"Transforms the PIL.Image image\"\"\"\n", " raise NotImplementedError()" ] }, { "cell_type": "markdown", "id": "63cff6f1-f652-4413-b230-b4602e115560", "metadata": {}, "source": [ "Lets define a base model for working with **`Numpy`** images." ] }, { "cell_type": "code", "execution_count": null, "id": "128c8e1e-8695-4710-a7c1-40476ced300a", "metadata": {}, "outputs": [], "source": [ "class NumpyImageModel(ImageModel):\n", " \"\"\"Base class for np.ndarray image models\"\"\"\n", "\n", " @staticmethod\n", " def to_np_ndarray(image: str, height=HEIGHT, width=WIDTH) -> np.ndarray:\n", " \"\"\"Converts a base64 encoded jpeg string to a np.ndarray\"\"\"\n", " pil_img = PILImageModel.to_pil_img(image, height=height, width=width)\n", " return np.array(pil_img)\n", "\n", " @staticmethod\n", " def from_np_ndarray(image: np.ndarray) -> io.BytesIO:\n", " \"\"\"Converts np.ndarray jpeg image to a jpeg BytesIO instance\"\"\"\n", " if image.dtype == np.dtype(\"float64\"):\n", " image = (image * 255).astype(np.uint8)\n", " pil_img = PIL.Image.fromarray(image)\n", " return PILImageModel.from_pil_img(pil_img)\n", "\n", " def apply(self, image: str, height: int = HEIGHT, width: int = WIDTH) -> io.BytesIO:\n", " np_array = self.to_np_ndarray(image, height=height, width=width)\n", "\n", " transformed_image = self.transform(np_array)\n", "\n", " return self.from_np_ndarray(transformed_image)\n", "\n", " def transform(self, image: np.ndarray) -> np.ndarray:\n", " \"\"\"Transforms the np.array image\"\"\"\n", " raise NotImplementedError()" ] }, { "cell_type": "markdown", "id": "77bf1e99-701e-4dad-bd8a-faba4bdd446a", "metadata": {}, "source": [ "## Timer\n", "\n", "Lets define a timer component to visualize the stats of the live videostream and the image transformations\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "id": "92a4ec91-3a6d-43be-8337-690614bb10cb", "metadata": {}, "outputs": [], "source": [ "class Timer(pn.viewable.Viewer):\n", " \"\"\"Helper Component used to show duration trends\"\"\"\n", "\n", " _trends = param.Dict()\n", "\n", " def __init__(self, **params):\n", " super().__init__()\n", "\n", " self.last_updates = {}\n", " self._trends = {}\n", "\n", " self._layout = pn.Row(**params)\n", "\n", " def time_it(self, name, func, *args, **kwargs):\n", " \"\"\"Measures the duration of the execution of the func function and reports it under the\n", " name specified\"\"\"\n", " start = time.time()\n", " result = func(*args, **kwargs)\n", " end = time.time()\n", " duration = round(end - start, 2)\n", " self._report(name=name, duration=duration)\n", " return result\n", "\n", " def inc_it(self, name):\n", " \"\"\"Measures the duration since the last time `inc_it` was called and reports it under the\n", " specified name\"\"\"\n", " start = self.last_updates.get(name, time.time())\n", " end = time.time()\n", " duration = round(end - start, 2)\n", " self._report(name=name, duration=duration)\n", " self.last_updates[name] = end\n", "\n", " def _report(self, name, duration):\n", " if not name in self._trends:\n", " self._trends[name] = pn.indicators.Trend(\n", " name=name,\n", " data={\"x\": [1], \"y\": [duration]},\n", " height=100,\n", " width=150,\n", " sizing_mode=\"fixed\",\n", " )\n", " self.param.trigger(\"_trends\")\n", " else:\n", " trend = self._trends[name]\n", " next_x = max(trend.data[\"x\"]) + 1\n", " trend.stream({\"x\": [next_x], \"y\": [duration]}, rollover=10)\n", "\n", " @param.depends(\"_trends\")\n", " def _panel(self):\n", " self._layout[:] = list(self._trends.values())\n", " return self._layout\n", "\n", " def __panel__(self):\n", " return pn.panel(self._panel)" ] }, { "cell_type": "markdown", "id": "44851482-f580-49a7-ac53-2132d3f29bc0", "metadata": {}, "source": [ "## VideoStreamInterface\n", "\n", "The `VideoStreamInterface` will be putting things together in a nice UI.\n", "\n", "\n", "\n", "Lets define a helper function first" ] }, { "cell_type": "code", "execution_count": null, "id": "b4de82da-81e6-4799-a1d3-a54bf8e017b1", "metadata": {}, "outputs": [], "source": [ "def to_instance(value, **params):\n", " \"\"\"Converts the value to an instance\n", "\n", " Args:\n", " value: A param.Parameterized class or instance\n", "\n", " Returns:\n", " An instance of the param.Parameterized class\n", " \"\"\"\n", " if isinstance(value, param.Parameterized):\n", " value.param.update(**params)\n", " return value\n", " return value(**params)" ] }, { "cell_type": "markdown", "id": "d74ba396-7972-4711-8ff5-b4d27b31120c", "metadata": {}, "source": [ "The `VideoStreamInterface` will take a list of `ImageModel`s. The user can the select and apply the models to the images from the `VideoStream`." ] }, { "cell_type": "code", "execution_count": null, "id": "c6dc71b7-34b2-4171-8a72-46ab9edeba91", "metadata": {}, "outputs": [], "source": [ "class VideoStreamInterface(pn.viewable.Viewer):\n", " \"\"\"An easy to use interface for a VideoStream and a set of transforms\"\"\"\n", "\n", " video_stream = param.ClassSelector(\n", " class_=pn.widgets.VideoStream, constant=True, doc=\"The source VideoStream\"\n", " )\n", "\n", " height = param.Integer(\n", " default=HEIGHT,\n", " bounds=(10, 2000),\n", " step=10,\n", " doc=\"\"\"The height of the image converted and shown\"\"\",\n", " )\n", " width = param.Integer(\n", " default=WIDTH,\n", " bounds=(10, 2000),\n", " step=10,\n", " doc=\"\"\"The width of the image converted and shown\"\"\",\n", " )\n", "\n", " model = param.Selector(doc=\"The currently selected model\")\n", "\n", " def __init__(\n", " self,\n", " models,\n", " timeout=TIMEOUT,\n", " paused=False,\n", " **params,\n", " ):\n", " super().__init__(\n", " video_stream=pn.widgets.VideoStream(\n", " name=\"Video Stream\",\n", " timeout=timeout,\n", " paused=paused,\n", " height=0,\n", " width=0,\n", " visible=False,\n", " format=\"jpeg\",\n", " ),\n", " **params,\n", " )\n", " self.image = pn.pane.JPG(\n", " height=self.height, width=self.width, sizing_mode=\"fixed\"\n", " )\n", " self._updating = False\n", " models = [to_instance(model) for model in models]\n", " self.param.model.objects = models\n", " self.model = models[0]\n", " self.timer = Timer(sizing_mode=\"stretch_width\")\n", " self.settings = self._create_settings()\n", " self._panel = self._create_panel()\n", "\n", " def _create_settings(self):\n", " return pn.Column(\n", " pn.Param(\n", " self.video_stream,\n", " parameters=[\"timeout\", \"paused\"],\n", " widgets={\n", " \"timeout\": {\n", " \"widget_type\": pn.widgets.IntSlider,\n", " \"start\": 10,\n", " \"end\": 2000,\n", " \"step\": 10,\n", " }\n", " },\n", " ),\n", " self.timer,\n", " pn.Param(self, parameters=[\"height\", \"width\"], name=\"Image\"),\n", " pn.Param(\n", " self,\n", " parameters=[\"model\"],\n", " expand_button=False,\n", " expand=False,\n", " widgets={\n", " \"model\": {\n", " \"widget_type\": pn.widgets.RadioButtonGroup,\n", " \"orientation\": \"vertical\",\n", " \"button_type\": \"primary\",\n", " \"button_style\": \"outline\"\n", " }\n", " },\n", " name=\"Model\",\n", " ),\n", " self._get_transform,\n", " )\n", "\n", " def _create_panel(self):\n", " return pn.Row(\n", " self.video_stream,\n", " pn.layout.HSpacer(),\n", " self.image,\n", " pn.layout.HSpacer(),\n", " sizing_mode=\"stretch_width\",\n", " align=\"center\",\n", " )\n", "\n", " @param.depends(\"height\", \"width\", watch=True)\n", " def _update_height_width(self):\n", " self.image.height = self.height\n", " self.image.width = self.width\n", "\n", " @param.depends(\"model\")\n", " def _get_transform(self):\n", " # Hack: returning self.transform stops working after browsing the transforms for a while\n", " return self.model.view\n", "\n", " def __panel__(self):\n", " return self._panel\n", "\n", " @param.depends(\"video_stream.value\", watch=True)\n", " def _handle_stream(self):\n", " if self._updating:\n", " return\n", "\n", " self._updating = True\n", " if self.model and self.video_stream.value:\n", " value = self.video_stream.value\n", " try:\n", " image = self.timer.time_it(\n", " name=\"Model\",\n", " func=self.model.apply,\n", " image=value,\n", " height=self.height,\n", " width=self.width,\n", " )\n", " self.image.object = image\n", " except PIL.UnidentifiedImageError:\n", " print(\"unidentified image\")\n", "\n", " self.timer.inc_it(\"Last Update\")\n", " self._updating = False" ] }, { "cell_type": "markdown", "id": "9afa4c9b-a541-4a77-bb38-68d85f0a82d4", "metadata": {}, "source": [ "## Custom Image Models\n", "\n", "We will now make specific image to image algorithms interactive.\n", "\n", "Let us start with the [Gaussian Blur](https://pillow.readthedocs.io/en/stable/reference/ImageFilter.html#PIL.ImageFilter.GaussianBlur) algorithm." ] }, { "cell_type": "code", "execution_count": null, "id": "f389b022-28b0-4ce2-a629-0df728a235bb", "metadata": {}, "outputs": [], "source": [ "class GaussianBlurModel(PILImageModel):\n", " \"\"\"Gaussian Blur Model\n", "\n", " https://pillow.readthedocs.io/en/stable/reference/ImageFilter.html#PIL.ImageFilter.GaussianBlur\n", " \"\"\"\n", "\n", " radius = param.Integer(default=2, bounds=(0, 10))\n", "\n", " def transform(self, image: Image):\n", " return image.filter(ImageFilter.GaussianBlur(radius=self.radius))" ] }, { "cell_type": "markdown", "id": "42d93547-bfce-4deb-a7e5-387cab9fac8e", "metadata": {}, "source": [ "Lets implement a [Grayscale](https://scikit-image.org/docs/0.15.x/auto_examples/color_exposure/plot_rgb_to_gray.html) algorithm." ] }, { "cell_type": "code", "execution_count": null, "id": "13f7f769-af74-4027-be8d-a22208e31800", "metadata": {}, "outputs": [], "source": [ "class GrayscaleModel(NumpyImageModel):\n", " \"\"\"GrayScale Model\n", "\n", " https://scikit-image.org/docs/0.15.x/auto_examples/color_exposure/plot_rgb_to_gray.html\n", " \"\"\"\n", "\n", " def transform(self, image: np.ndarray):\n", " grayscale = skimage.color.rgb2gray(image[:, :, :3])\n", " return skimage.color.gray2rgb(grayscale)" ] }, { "cell_type": "markdown", "id": "b3461593-e63d-424a-b402-cbd6fcc75ffe", "metadata": {}, "source": [ "Lets implement the [Sobel](https://scikit-image.org/docs/0.15.x/auto_examples/color_exposure/plot_adapt_rgb.html) algorithm." ] }, { "cell_type": "code", "execution_count": null, "id": "0991dd1c-0557-4dfa-b940-7918b0363d6c", "metadata": {}, "outputs": [], "source": [ "class SobelModel(NumpyImageModel):\n", " \"\"\"Sobel Model\n", "\n", " https://scikit-image.org/docs/0.15.x/auto_examples/color_exposure/plot_adapt_rgb.html\n", " \"\"\"\n", " def transform(self, image):\n", "\n", "\n", " @adapt_rgb(each_channel)\n", " def sobel_each(image):\n", " return filters.sobel(image)\n", "\n", " return rescale_intensity(1 - sobel_each(image))" ] }, { "cell_type": "markdown", "id": "cadae0c9-0191-4ca4-a770-c3be0b0987b7", "metadata": {}, "source": [ "Lets implement the [face detection model](https://scikit-image.org/docs/0.15.x/auto_examples/applications/plot_face_detection.html) of scikit-image." ] }, { "cell_type": "code", "execution_count": null, "id": "57423cdc-e23e-4236-ace8-452e8068970c", "metadata": {}, "outputs": [], "source": [ "@pn.cache()\n", "def get_detector():\n", " \"\"\"Returns the Cascade detector\"\"\"\n", " trained_file = data.lbp_frontal_face_cascade_filename()\n", " return Cascade(trained_file)" ] }, { "cell_type": "code", "execution_count": null, "id": "5baa7f38-6368-4ed1-883c-7a0247d1c08a", "metadata": {}, "outputs": [], "source": [ "class FaceDetectionModel(NumpyImageModel):\n", " \"\"\"Face detection using a cascade classifier.\n", "\n", " https://scikit-image.org/docs/0.15.x/auto_examples/applications/plot_face_detection.html\n", " \"\"\"\n", "\n", " scale_factor = param.Number(default=1.4, bounds=(1.0, 2.0), step=0.1)\n", " step_ratio = param.Integer(default=1, bounds=(1, 10))\n", " size_x = param.Range(default=(60, 322), bounds=(10, 500))\n", " size_y = param.Range(default=(60, 322), bounds=(10, 500))\n", "\n", " def transform(self, image):\n", " detector = get_detector()\n", " detected = detector.detect_multi_scale(\n", " img=image,\n", " scale_factor=self.scale_factor,\n", " step_ratio=self.step_ratio,\n", " min_size=(self.size_x[0], self.size_y[0]),\n", " max_size=(self.size_x[1], self.size_y[1]),\n", " )\n", "\n", " for patch in detected:\n", " rrr, ccc = rectangle(\n", " start=(patch[\"r\"], patch[\"c\"]),\n", " extent=(patch[\"height\"], patch[\"width\"]),\n", " shape=image.shape[:2],\n", " )\n", " image[rrr, ccc, 0] = 200\n", "\n", " return image" ] }, { "cell_type": "markdown", "id": "253e15c1-0097-456f-b979-fbf42de8d270", "metadata": {}, "source": [ "Please note that these models are just examples. You can also implement your own models using Scikit-Image, Pytorch, Tensorflow etc and use the `VideoStreamInterface` to work interactively with them." ] }, { "cell_type": "markdown", "id": "1e6ffe9c-7885-4cb8-9793-97c00862e8ba", "metadata": {}, "source": [ "## Its alive!\n", "\n", "Lets define an instance of the `VideoStreamInterface`" ] }, { "cell_type": "code", "execution_count": null, "id": "47d8118f-77b0-4edf-a733-2b9ba877e25e", "metadata": {}, "outputs": [], "source": [ "component = VideoStreamInterface(\n", " models=[\n", " GaussianBlurModel,\n", " GrayscaleModel,\n", " SobelModel,\n", " FaceDetectionModel,\n", " ]\n", ")\n", "pn.Row(pn.Row(component.settings, max_width=400), component)" ] }, { "cell_type": "markdown", "id": "651ea40a-4d91-4a17-8761-c8b5cc446197", "metadata": {}, "source": [ "## Wrap it in a template\n", "\n", "What makes Panel unique is that our components work very well in both the notebook and as standalone data apps.\n", "\n", "We can wrap the component in the nicely styled [`FastListTemplate`](https://panel.holoviz.org/reference/templates/FastListTemplate.html) to make it *ready for production*." ] }, { "cell_type": "code", "execution_count": null, "id": "dab6006d-001d-4217-ade7-ff2d0d77752f", "metadata": {}, "outputs": [], "source": [ "pn.template.FastListTemplate(\n", " site=\"Panel\",\n", " title=\"VideoStream Interface\",\n", " sidebar=[component.settings],\n", " main=[component],\n", ").servable(); # We add ; to not show the template in the notebook as it does not display well." ] }, { "cell_type": "markdown", "id": "9b00da3e-87fd-4e3e-bc81-e8bf827e2818", "metadata": {}, "source": [ "## Serve it as a server side app" ] }, { "cell_type": "markdown", "id": "7191262b-f610-494f-a797-9186a496c633", "metadata": {}, "source": [ "It is now possible to serve the live app via the command `panel serve streaming_videostream.ipynb`. The app is the available at http://localhost:5006/streaming_videostream.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "01fb5b28-6e49-4aaa-a0ca-2eaa301106c2", "metadata": {}, "source": [ "## Serve it as a client side app\n", "\n", "You can also [`panel convert`](https://panel.holoviz.org/how_to/wasm/convert.html) this app to web assembly for even better performance.\n", "\n", "First you will need to create a `requirements.txt` file with the following content\n", "\n", "```bash\n", "panel\n", "numpy\n", "scikit-image\n", "```\n", "\n", "Then you can\n", "\n", "- Run `panel convert streaming_videostream.ipynb --to pyodide-worker --out pyodide --requirements requirements.txt`\n", "- Run `python3 -m http.server` to start a web server locally\n", "- Open http://localhost:8000/pyodide/streaming_videostream.html to try out the app." ] } ], "metadata": { "language_info": { "name": "python", "pygments_lexer": "ipython3" } }, "nbformat": 4, "nbformat_minor": 5 }