{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# FIRST import all the necessary libraries and modules!\n", "import cv2 # import OpenCV\n", "import numpy as np # import NumPy\n", "\n", "# import instructor made functions \n", "import sys\n", "sys.path.insert(0, '../..')\n", "from utils import * " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Painter Lab\n", "\n", "

\n", " Let's learn how to paint on live video using colored objects! \n", "
To do this, we will need to combine everything we've learned so far: drawing, color tracking, and contours!\n", "

\n", "\n", "![multiple draw](multiple_draw.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# HSV Masking\n", " \n", "

\n", " HSV tends to be better for tracking an object by color than BGR. \n", "
Do you remember why this is? Discuss with your partner!\n", "

\n", "

\n", " We can mask video frames using lower and upper bounds for hue (H), saturation (S), and value (V) to track objects by their color.\n", "

\n", "\n", "![HSV](hsv.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Isolating Objects\n", "\n", "

\n", " First, we need to detect the colored objects we will be drawing with! \n", "

\n", "\n", "

\n", " Here is an example of a GOOD MASK:\n", "

\n", "\n", "![hsv_good_example](hsv_good_example.png)\n", "\n", "

\n", " Here is an example of a BAD MASK:\n", "

\n", "\n", "\n", "\n", "

\n", " Exercise:\n", "

\n", "

\n", " " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hsv_select_live()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Masking the Video\n", "\n", "

\n", " Exercise:\n", "
Let's create a function that returns a masked frame using input HSV values:\n", "

    \n", "
  1. Convert our frames from BGR to HSV via cv2.cvtColor
    2. \n", "
  2. Use cv2.inRange to create a mask using the given inputs\n", "
\n", "

" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def mask_frame(frame, hsv_lower, hsv_upper):\n", " # TASK #1: Convert \"frame\" to from BGR to HSV using cv2.cvtColor\n", " \n", " \n", " # TASK #2: Mask the frame using cv2.inRange, with \"hsv_lower\" and \"hsv_upper\" as inputs\n", " # Assume hsv_lower and hsv_upper are the correct data types!\n", " \n", " \n", " # TASK #3: Return the mask\n", " \n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Exercise:\n", "
Let's test our mask on a colored object! \n", "
Set hsv_lower and hsv_upper using the values you recorded for one of your colored objects.\n", "

" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# TASK: Define the color ranges for your object\n", "hsv_lower = (None, None, None) # Replace NONE with integers!\n", "hsv_upper = (None, None, None) # Replace NONE with integers!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Run the code below to see your mask work in real time!\n", "

\n", " \n", "

\n", " It should look something like this:\n", "

\n", "\n", "![mask](mask.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def show_mask(frame):\n", " mask = mask_frame(frame, hsv_lower, hsv_upper) # calculates the mask\n", " cv2.imshow('Mask', mask) # displays the mask\n", "\n", "video(show_mask)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Finding Contours\n", "\n", "

\n", " We can track the location of your colored object using contours. \n", "
We can calculate a list of contours based on a given mask like this:\n", "

\n", "\n", "```python\n", "contours = cv2.findContours(mask, 3, 2)[0]\n", "```\n", "\n", "

\n", " Exercise:\n", "
In the function below, find and return the list of all contours.\n", "

" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def find_contours(mask):\n", " # TASK #1: Find all contours\n", " \n", " \n", " # TASK #2: Return the list of contours\n", " \n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Run the code block below to see your contours drawn on the frame!\n", "

\n", "\n", "

\n", " It should look similar to this:\n", "

\n", "\n", "![contours](contours.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def show_contours(frame):\n", " mask = mask_frame(frame, hsv_lower, hsv_upper) # calculates the mask\n", " contours = find_contours(mask) # finds the contours\n", " cv2.drawContours(frame, contours, -1, (0, 255, 0), 3) # draw contours over the frame\n", " cv2.imshow('Contours', frame)\n", "\n", "video(show_contours)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Isolating Object Contour" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " As you can see, there are extra contours :( \n", "
We just want ONE contour for our colored object! \n", "

\n", "\n", "

\n", " Let's only consider contours with area larger than 1000 pixels. \n", "
We can find the area of a single contour like this:\n", "

\n", "\n", "```python\n", "area = cv2.contourArea(contour)\n", "```\n", "\n", "

\n", " Exercise:\n", "
Fill in the function below to return only contours larger than 1000 pixels:\n", "

\n", "

" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def object_contour(mask):\n", " contours = find_contours(mask) # finds and saves all the contours\n", " obj_contour = None # keeps track of object contour (if unchanged, function will return None)\n", " max_area = 0 # keeps track of maximum area\n", " \n", " if len(contours) > 0:\n", " for contour in contours:\n", " # TASK #1: Find and save the area using cv2.contourArea\n", " \n", " \n", " # TASK #2: If area is larger than 1000 AND area is larger than max_area,\n", " # save the current contour in obj_contour, and update max_area\n", " \n", " \n", " \n", " \n", " # TASK #3: Return the object contour\n", " \n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Run the code below to display this contour!\n", "

\n", " \n", "

\n", " It should look similar to this:\n", "

\n", " \n", "![object contour](obj_contour.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def show_object_contour(frame):\n", " mask = mask_frame(frame, hsv_lower, hsv_upper) # calculates the mask\n", " obj_contour = object_contour(mask) # finds the object contour\n", " if obj_contour is not None: # if the contour exists,\n", " cv2.drawContours(frame, [obj_contour], -1, (0, 255, 0), 3) # draws the contour\n", " cv2.imshow('Object Contour', frame)\n", "\n", "video(show_object_contour)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Smallest Enclosing Circle" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " The instructor-made functions find_center(<contour>) and find_radius(<contour>) find the center coordinates and radius of the smallest circle that fits around our object.\n", "

\n", "\n", "

\n", " Exercise:\n", "
Using these functions, draw a circle around the object. \n", "
Choose the color and thickness of the circle yourself!\n", "

" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def draw_circle(frame, contour):\n", " # TASK #1: Find and save the center using find_center\n", " \n", "\n", " # TASK #2: Find and save the radius using find_radius\n", " \n", "\n", " # TASK #3: Draw a circle (on the frame) that encloses the object using cv2.circle\n", " \n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Run the code below to see your circle drawn on the frame in real time!\n", "

\n", "\n", "

\n", " It should look like this:\n", "

\n", "\n", "![circle](circle.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def display_circle(frame):\n", " mask = mask_frame(frame, hsv_lower, hsv_upper) # calculates the mask\n", " contour = object_contour(mask) # find the object contour\n", " if contour is not None: # if the contour exists\n", " draw_circle(frame, contour) # draws circle around object\n", " cv2.imshow('Circle Tracking', frame)\n", "\n", "video(display_circle)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Drawing with an Object" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " In the draw function below, we will:\n", "

    \n", "
  1. Draw a line from the previous position of our object to the current position.
    2. \n", "
  2. Save our lines in a list.
    3. \n", "
  3. Draw the saved lines one EVERY frame (otherwise, they will disappear).
    4. \n", "
  4. Save the start coordinates, stop coordinates, and color of each line.
    5. \n", "
\n", "

\n", "\n", "

\n", " Exercise:\n", "
Fill in the draw function below to draw with your object!\n", "

\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def draw(frame, contour, color, prev_pos):\n", " prev_x = prev_pos[0] # gets px from prev_pos\n", " prev_y = prev_pos[1] # gets py from prev_pos\n", " drawing = prev_pos != [-1, -1] # True if we are already drawing\n", " \n", " if contour is not None:\n", " draw_circle(frame, contour) # displays circle around object\n", " \n", " # TASK #1: Find and save the center coordinates using find_center\n", " \n", " \n", " # TASK #2: The center is a tuple of the form (x, y)\n", " # Use indexing to find and save x and y\n", " \n", " \n", " # TASK #3: If we are already drawing (use an if statement)\n", " \n", " \n", " # TASK #4: Append to lines list a tuple of: (previous coordinates, current coordinates, color)\n", " # Hint: Use (prev_x,prev_y) for previous coords, (x,y) for current coords, and use the input color\n", " # Hint: Assume \"lines\" is a given list\n", " \n", " \n", " # TASK #5: Update prev_x and prev_y\n", " \n", " \n", " \n", " # TASK #6: If we are not drawing (use an elif or else statement)\n", " \n", " \n", " # TASK #7: Change the drawing variable\n", " # Hint: If we see the object, should we be drawing or not drawing?\n", " \n", " \n", " # TASK #8: Update prev_x and prev_y\n", " \n", " \n", " \n", " else:\n", " # TASK #9: Change the drawing variable\n", " # Hint: if we no longer see the object, should we be drawing or not drawing?\n", " \n", " \n", " # updates prev_pos based on drawing variable, prev_x, and prev_y\n", " if drawing == False:\n", " prev_pos[0] = -1\n", " prev_pos[1] = -1\n", " else:\n", " prev_pos[0] = prev_x\n", " prev_pos[1] = prev_y" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Exercise:\n", "
In the function below, call draw on a color of your choice! \n", "
Then draw the lines saved in lines onto the frame.\n", "

" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "lines = [] # stores the lines\n", "prev_pos = [-1, -1] # stores previous position (starts in not already drawing state)\n", "\n", "def show_draw(frame):\n", " mask = mask_frame(frame, hsv_lower, hsv_upper) # calculates the mask\n", " contour = object_contour(mask) # finds the object contour\n", " \n", " # TASK #1: Choose and save a color of your choice\n", " \n", " \n", " draw(frame, contour, color, prev_pos) # calls draw using chosen color\n", " \n", " for line in lines:\n", " # TASK #2: Draw each line using the (previous position, current position, and color) we stored in \"line\"\n", " # Choose the line thickness yourself! Use cv2.line and the line list.\n", " \n", " \n", " cv2.imshow('Draw', frame)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Run the code below. You should be able to draw on top of the video with your object!\n", "

\n", " \n", "

\n", " It should look something like this:\n", "

\n", "\n", "![draw](draw.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "lines = [] # reset lines list\n", "video(show_draw)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# BONUS: Drawing with Multiple Objects" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Exercise:\n", "
To draw with multiple objects, we need to:\n", "

\n", "

" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# TASK #1: Replace with name of first color\n", "_prev_pos = [-1, -1]\n", "\n", "# TASK #2: Replace , , etc. with your values\n", "_hsv_lower = np.array([, , ])\n", "_hsv_upper = np.array([, , ])\n", "\n", "# TASK #3: Repeat TASKS #1 and #2 with EACH COLOR you are using\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Exercise:\n", "
Now we need to update show_draw to include multiple colors.\n", "

" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "def show_draw(frame):\n", " # TASK #1: Replace with your first color.\n", " _color =\n", " _mask = mask_frame(frame, _hsv_lower, _hsv_upper) # calculates the mask\n", " _contour = object_contour(_mask) # finds the object contour\n", " draw(frame, _contour, _color, _prev_pos) # calls draw for this color\n", " \n", " # TASK #2: Repeat TASK #1 with EACH COLOR you are using\n", " \n", " \n", " \n", " \n", " for line in lines:\n", " # TASK #3: Draw each line using the (previous position, current position, and color) we stored in \"line\"\n", " # Choose the line thickness yourself! Use cv2.line and the line list.\n", " \n", " \n", " cv2.imshow('Multiple Draw', frame)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " Run the code below. Each object should now draw a different color!\n", "

\n", " \n", "

\n", " It should look something like this:\n", "

\n", "\n", "![multiple draw](multiple_draw.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "lines = [] # reset lines list\n", "video(show_draw)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bonus Challenge Ideas!!!!!\n", "- Add more colors!!!!!\n", "\n", "### Make a mask for a color \"tool\" that, when introduced in frame, \n", "- switches/cycles through colors! So you can have one tool for multiple colors\n", "- stops the drawing so that you can reposition your paintbrush without covering it or moving it off the screen\n", "- changes your drawings from lines to shapes! (circles, rectangles)\n", "\n", "\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.1" } }, "nbformat": 4, "nbformat_minor": 4 }