{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transformers installation\n",
    "! pip install transformers datasets\n",
    "# To install from source instead of the last release, comment the command above and uncomment the following one.\n",
    "# ! pip install git+https://github.com/huggingface/transformers.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Zero-shot object detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Traditionally, models used for [object detection](https://huggingface.co/docs/transformers/main/en/tasks/object_detection) require labeled image datasets for training,\n",
    "and are limited to detecting the set of classes from the training data.\n",
    "\n",
    "Zero-shot object detection is supported by the [OWL-ViT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/owlvit) model which uses a different approach. OWL-ViT\n",
    "is an open-vocabulary object detector. It means that it can detect objects in images based on free-text queries without\n",
    "the need to fine-tune the model on labeled datasets.\n",
    "\n",
    "OWL-ViT leverages multi-modal representations to perform open-vocabulary detection. It combines [CLIP](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/clip) with\n",
    "lightweight object classification and localization heads. Open-vocabulary detection is achieved by embedding free-text queries with the text encoder of CLIP and using them as input to the object classification and localization heads.\n",
    "associate images and their corresponding textual descriptions, and ViT processes image patches as inputs. The authors\n",
    "of OWL-ViT first trained CLIP from scratch and then fine-tuned OWL-ViT end to end on standard object detection datasets using\n",
    "a bipartite matching loss.\n",
    "\n",
    "With this approach, the model can detect objects based on textual descriptions without prior training on labeled datasets.\n",
    "\n",
    "In this guide, you will learn how to use OWL-ViT:\n",
    "- to detect objects based on text prompts\n",
    "- for batch object detection\n",
    "- for image-guided object detection\n",
    "\n",
    "Before you begin, make sure you have all the necessary libraries installed:\n",
    "\n",
    "```bash\n",
    "pip install -q transformers\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Zero-shot object detection pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest way to try out inference with OWL-ViT is to use it in a [pipeline()](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.pipeline). Instantiate a pipeline\n",
    "for zero-shot object detection from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "checkpoint = \"google/owlvit-base-patch32\"\n",
    "detector = pipeline(model=checkpoint, task=\"zero-shot-object-detection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, choose an image you'd like to detect objects in. Here we'll use the image of astronaut Eileen Collins that is\n",
    "a part of the [NASA](https://www.nasa.gov/multimedia/imagegallery/index.html) Great Images dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import skimage\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "\n",
    "image = skimage.data.astronaut()\n",
    "image = Image.fromarray(np.uint8(image)).convert(\"RGB\")\n",
    "\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"flex justify-center\">\n",
    "     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png\" alt=\"Astronaut Eileen Collins\"/>\n",
    "</div>\n",
    "\n",
    "Pass the image and the candidate object labels to look for to the pipeline.\n",
    "Here we pass the image directly; other suitable options include a local path to an image or an image url. We also pass text descriptions for all items we want to query the image for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'score': 0.3571370542049408,\n",
       "  'label': 'human face',\n",
       "  'box': {'xmin': 180, 'ymin': 71, 'xmax': 271, 'ymax': 178}},\n",
       " {'score': 0.28099656105041504,\n",
       "  'label': 'nasa badge',\n",
       "  'box': {'xmin': 129, 'ymin': 348, 'xmax': 206, 'ymax': 427}},\n",
       " {'score': 0.2110239565372467,\n",
       "  'label': 'rocket',\n",
       "  'box': {'xmin': 350, 'ymin': -1, 'xmax': 468, 'ymax': 288}},\n",
       " {'score': 0.13790413737297058,\n",
       "  'label': 'star-spangled banner',\n",
       "  'box': {'xmin': 1, 'ymin': 1, 'xmax': 105, 'ymax': 509}},\n",
       " {'score': 0.11950037628412247,\n",
       "  'label': 'nasa badge',\n",
       "  'box': {'xmin': 277, 'ymin': 338, 'xmax': 327, 'ymax': 380}},\n",
       " {'score': 0.10649408400058746,\n",
       "  'label': 'rocket',\n",
       "  'box': {'xmin': 358, 'ymin': 64, 'xmax': 424, 'ymax': 280}}]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = detector(\n",
    "    image,\n",
    "    candidate_labels=[\"human face\", \"rocket\", \"nasa badge\", \"star-spangled banner\"],\n",
    ")\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import ImageDraw\n",
    "\n",
    "draw = ImageDraw.Draw(image)\n",
    "\n",
    "for prediction in predictions:\n",
    "    box = prediction[\"box\"]\n",
    "    label = prediction[\"label\"]\n",
    "    score = prediction[\"score\"]\n",
    "\n",
    "    xmin, ymin, xmax, ymax = box.values()\n",
    "    draw.rectangle((xmin, ymin, xmax, ymax), outline=\"red\", width=1)\n",
    "    draw.text((xmin, ymin), f\"{label}: {round(score,2)}\", fill=\"white\")\n",
    "\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"flex justify-center\">\n",
    "     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_2.png\" alt=\"Visualized predictions on NASA image\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text-prompted zero-shot object detection by hand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you've seen how to use the zero-shot object detection pipeline, let's replicate the same\n",
    "result manually.\n",
    "\n",
    "Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit).\n",
    "Here we'll use the same checkpoint as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection\n",
    "\n",
    "model = AutoModelForZeroShotObjectDetection.from_pretrained(checkpoint)\n",
    "processor = AutoProcessor.from_pretrained(checkpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a different image to switch things up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "url = \"https://unsplash.com/photos/oj0zeY2Ltk4/download?ixid=MnwxMjA3fDB8MXxzZWFyY2h8MTR8fHBpY25pY3xlbnwwfHx8fDE2Nzc0OTE1NDk&force=true&w=640\"\n",
    "im = Image.open(requests.get(url, stream=True).raw)\n",
    "im"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"flex justify-center\">\n",
    "     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_3.png\" alt=\"Beach photo\"/>\n",
    "</div>\n",
    "\n",
    "Use the processor to prepare the inputs for the model. The processor combines an image processor that prepares the\n",
    "image for the model by resizing and normalizing it, and a [CLIPTokenizer](https://huggingface.co/docs/transformers/main/en/model_doc/clip#transformers.CLIPTokenizer) that takes care of the text inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_queries = [\"hat\", \"book\", \"sunglasses\", \"camera\"]\n",
    "inputs = processor(text=text_queries, images=im, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pass the inputs through the model, post-process, and visualize the results. Since the image processor resized images before\n",
    "feeding them to the model, you need to use the [post_process_object_detection()](https://huggingface.co/docs/transformers/main/en/model_doc/owlvit#transformers.OwlViTImageProcessor.post_process_object_detection) method to make sure the predicted bounding\n",
    "boxes have the correct coordinates relative to the original image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "    target_sizes = torch.tensor([im.size[::-1]])\n",
    "    results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)[0]\n",
    "\n",
    "draw = ImageDraw.Draw(im)\n",
    "\n",
    "scores = results[\"scores\"].tolist()\n",
    "labels = results[\"labels\"].tolist()\n",
    "boxes = results[\"boxes\"].tolist()\n",
    "\n",
    "for box, score, label in zip(boxes, scores, labels):\n",
    "    xmin, ymin, xmax, ymax = box\n",
    "    draw.rectangle((xmin, ymin, xmax, ymax), outline=\"red\", width=1)\n",
    "    draw.text((xmin, ymin), f\"{text_queries[label]}: {round(score,2)}\", fill=\"white\")\n",
    "\n",
    "im"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"flex justify-center\">\n",
    "     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png\" alt=\"Beach photo with detected objects\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass multiple sets of images and text queries to search for different (or same) objects in several images.\n",
    "Let's use both an astronaut image and the beach image together.\n",
    "For batch processing, you should pass text queries as a nested list to the processor and images as lists of PIL images,\n",
    "PyTorch tensors, or NumPy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images = [image, im]\n",
    "text_queries = [\n",
    "    [\"human face\", \"rocket\", \"nasa badge\", \"star-spangled banner\"],\n",
    "    [\"hat\", \"book\", \"sunglasses\", \"camera\"],\n",
    "]\n",
    "inputs = processor(text=text_queries, images=images, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously for post-processing you passed the single image's size as a tensor, but you can also pass a tuple, or, in case\n",
    "of several images, a list of tuples. Let's create predictions for the two examples, and visualize the second one (`image_idx = 1`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "    target_sizes = [x.size[::-1] for x in images]\n",
    "    results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)\n",
    "\n",
    "image_idx = 1\n",
    "draw = ImageDraw.Draw(images[image_idx])\n",
    "\n",
    "scores = results[image_idx][\"scores\"].tolist()\n",
    "labels = results[image_idx][\"labels\"].tolist()\n",
    "boxes = results[image_idx][\"boxes\"].tolist()\n",
    "\n",
    "for box, score, label in zip(boxes, scores, labels):\n",
    "    xmin, ymin, xmax, ymax = box\n",
    "    draw.rectangle((xmin, ymin, xmax, ymax), outline=\"red\", width=1)\n",
    "    draw.text((xmin, ymin), f\"{text_queries[image_idx][label]}: {round(score,2)}\", fill=\"white\")\n",
    "\n",
    "images[image_idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"flex justify-center\">\n",
    "     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png\" alt=\"Beach photo with detected objects\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image-guided object detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to zero-shot object detection with text queries, OWL-ViT offers image-guided object detection. This means\n",
    "you can use an image query to find similar objects in the target image.\n",
    "Unlike text queries, only a single example image is allowed.\n",
    "\n",
    "Let's take an image with two cats on a couch as a target image, and an image of a single cat\n",
    "as a query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n",
    "image_target = Image.open(requests.get(url, stream=True).raw)\n",
    "\n",
    "query_url = \"http://images.cocodataset.org/val2017/000000524280.jpg\"\n",
    "query_image = Image.open(requests.get(query_url, stream=True).raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a quick look at the images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots(1, 2)\n",
    "ax[0].imshow(image_target)\n",
    "ax[1].imshow(query_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"flex justify-center\">\n",
    "     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_5.png\" alt=\"Cats\"/>\n",
    "</div>\n",
    "\n",
    "In the preprocessing step, instead of text queries, you now need to use `query_images`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = processor(images=image_target, query_images=query_image, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For predictions, instead of passing the inputs to the model, pass them to [image_guided_detection()](https://huggingface.co/docs/transformers/main/en/model_doc/owlvit#transformers.OwlViTForObjectDetection.image_guided_detection). Draw the predictions\n",
    "as before except now there are no labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    outputs = model.image_guided_detection(**inputs)\n",
    "    target_sizes = torch.tensor([image_target.size[::-1]])\n",
    "    results = processor.post_process_image_guided_detection(outputs=outputs, target_sizes=target_sizes)[0]\n",
    "\n",
    "draw = ImageDraw.Draw(image_target)\n",
    "\n",
    "scores = results[\"scores\"].tolist()\n",
    "boxes = results[\"boxes\"].tolist()\n",
    "\n",
    "for box, score, label in zip(boxes, scores, labels):\n",
    "    xmin, ymin, xmax, ymax = box\n",
    "    draw.rectangle((xmin, ymin, xmax, ymax), outline=\"white\", width=4)\n",
    "\n",
    "image_target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"flex justify-center\">\n",
    "     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_6.png\" alt=\"Cats with bounding boxes\"/>\n",
    "</div>\n",
    "\n",
    "If you'd like to interactively try out inference with OWL-ViT, check out this demo:\n",
    "\n",
    "<iframe\n",
    "\tsrc=\"https://adirik-owl-vit.hf.space\"\n",
    "\tframeborder=\"0\"\n",
    "\twidth=\"850\"\n",
    "\theight=\"450\"\n",
    "></iframe>"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 4
}