Accelerate Image Annotation with SAM and Grounding DINO | Python Tutorial

Introduction

The speaker introduces the topic of automating image labeling using computer vision and outlines the benefits of using recent advancements in object detection models.

• Recent advancements in computer vision have made it possible to automate most of the image labeling process.

• A Python script can be used to do 95% of the work, with annotations added or removed at the end.

• This saves a lot of time and money.

Using Python to Automate Image Annotation

The speaker explains how to use Python to automatically annotate raw images or convert bounding boxes into segmentations.

• Grounding Dyno is a powerful zero-shot object detector that can be used for most use cases.

• Segment Anything model produces masks based on prompts, which can include bounding boxes.

• Output from Grounding Dyno is used as a prompt for Segment Anything model.

• Nvidia SMI should be run first to confirm access to GPU. If not, follow instructions in "Before You Start" section.

Models Used

The speaker provides more information about the models used in this tutorial.

• Grounding Dyno is a zero-shot object detector that can detect objects without prior training on those objects specifically.

• Segment Anything model produces masks based on prompts, which can include bounding boxes as prompts.

Setting Up Python Environment

The speaker explains how to set up the Python environment needed for this tutorial.

• Install necessary dependencies by following instructions provided in video or notebook description.

• Supervision needs to be uninstalled and reinstalled with version 0.6.0

• Roboflow is also installed as data sets will be used from there and automated annotations will be uploaded into their backend.

Downloading Model Weights

The speaker explains how to download the necessary model weights.

• Grounding Dyno requires two files: a configuration file (already in repository) and a weights file (downloaded using wget).

• Segment Anything model also requires downloading checkpoint using wget.

• Both models are loaded into memory, starting with Grounding Dyno.

Setting up the Python Environment

In this section, the speaker sets up the Python environment and explains how to annotate data using images.

Annotating Data Using Images

• Drag and drop your images into Google Colab's File Explorer on the left side.

• Remember to put them into the data subdirectory for easier management of file paths.

• Use grounding Dyno to produce zero-shot detections for selected classes.

• Adding "all" before the name of a class can improve detection quality, especially with longer class lists.

• Run our first zero-shot detection on an example image using grounding Dyno.

Prompting Models with Bounding Boxes

In this section, we learn how to prompt models with bounding boxes and run segmentation logic.

Running Segmentation Logic

• Wrap segmentation logic into a separate function.

• Encoding runs only once for every image.

• Loop over bounding boxes to prompt the model.

• Obtain corresponding masks for every bounding box extracted using grounding Dyno.

Plotting Results in Different Ways

In this section, we learn how to plot results in different ways and handle multiple masks produced by some bounding boxes.

Handling Multiple Masks Produced by Some Bounding Boxes

• Some bounding boxes may produce multiple masks that need handling when saving those masks into some common annotation format (in our case, pascalvok XML).

• Save each detached mask as a separate object behind the scenes.

Automatically Annotating All Images

In this section, we learn how to automatically annotate all images using zero-shot detection and grounding Dyno.

Automatically Annotating All Images

• List all images in the directory.

• Load the image using OpenCV.

• Run grounding Dyno to extract bounding boxes.

• Filter out all bounding boxes that don't have an assigned class.

• Discard those bounding boxes to prevent errors in later parts of the notebook.

• Extract masks associated with every image using sum.

• Plot all masks on corresponding images.

PascalVOC XML and Annotation Quality

In this section, the speaker discusses how to use PascalVOC XML to save predictions and customize parameters for annotation quality.

Customizing Parameters for Annotation Quality

• Three important parameters at this stage are mean and max image area percentage and the approximation percentage.

• The first two filtering parameters filter out masks based on their area in relation to the whole image.

• The third parameter steers the amount of points that will constitute your annotation poly.

Annotation Polygon

• Most annotation formats convert masks into polygons that pretty much cover the same area.

• The amount of points in a polygon does not directly translate into quality of a mask.

• It is better to keep 25 or 50 percent of the points in your original polygon and keep that geometry simpler.

Uploading Annotations into Roboflow

In this section, the speaker shows how to upload annotations into Roboflow using PascalVOC XML.

Creating Separate XML Files for Every Image

• When saving masks to annotation files, separate XML files are created for every image.

• Each object has a separate tag in it with bounding box and polygon information.

Uploading Annotations into Roboflow

• Use Roboflow package to upload annotations into Roboflow for further refinement.

Uploading Images and Annotations to Roboflow

In this section, the speaker demonstrates how to upload images and annotations to Roboflow.

Uploading Images and Annotations

• Sign in to the Roboflow account.

• Open the Jupyter notebook and wait for a few seconds for the images and automatically generated annotations to be uploaded into the Roboflow account.

• Go back to the project, navigate to the data set, and verify that both images and annotations are present.

Data Cleaning

• Examine each image in detail.

• Zoom in on each image to check for any irregular edges caused by low-quality masks being converted into polygons.

• To smooth out polygon edges, drop a larger number of points from the polygon.

Examining Scenes with Multiple Polygons

In this section, the speaker examines scenes with multiple polygons using layers view.

Layers View

• Switch to layers view for examining scenes with multiple polygons of multiple classes.

• Observe different objects such as shoes, backpacks, noses of dogs, ears part of people etc.

• Delete one of two detections for dogs by right-clicking on one of them.

Converting Bounding Boxes into Instant Segmentation

In this section, the speaker explains how to convert bounding boxes into instant segmentation using Python.

Blueberry Dataset Example

• Download blueberry dataset available at Roboflow Universe which has 78 images annotated with bounding boxes.

• Locate dataset using dataset.location property.

• Loop over train directory only but can process all directories similarly.

Converting Bounding Boxes into Segmentations

• Use samp instead of grounding Dyno to produce initial bounding boxes.

• Loop over the bounding boxes and query some with every bounding box separately.

• Observe the result by taking a look at it.

Saving Masks in Pascal VOC XML Format

In this section, the speaker explains how to save masks in Pascal VOC XML format and change the approximation percentage from 75 to 50.

Saving Masks

• To save masks in Pascal VOC XML format, use the same process as before.

• Change the approximation percentage from 75 to 50 to keep more of the points in polygons.

• This is necessary for low-resolution images where there is a risk of having triangles and rectangles.

Creating an Empty Project

In this section, the speaker explains how to create an empty project and add data sets.

Creating a Project

• Change the project description so that it's unique.

• Create an empty project by pressing shift+enter.

• Go back to Roboflow workspace and see one more data set called "blueberry" which is empty.

Looping Over Images and Annotations

In this section, the speaker explains how to loop over images and annotations and send them one by one to Roboflow backend.

Looping Over Images

• Press shift+enter to loop over images and annotations.

• Slowly send them one by one to Roboflow backend.

• The number will grow while waiting for the process to complete.

• Look at images and annotations that are already there.

Using Smart Poly Tool for Annotation

In this section, the speaker explains how to use smart poly tool for annotation.

Using Smart Poly Tool

• Pick smart poly tool from toolbox on right-hand side.

• Hover over fruit to see preview of mask geometry.

• Click when geometry is acceptable.

• Right-click and select proper class if needed.

Grounding Dino and Sam for Annotation

In this section, the speaker explains how using grounding Dino and Sam can speed up the annotation process.

Using Grounding Dino and Sam

• Using grounding Dino and Sam can speed up your annotation process.

• It can convert object detection data set into instant segmentation.

• The process is not perfect, but it reduces time and money investment.

• You still need to have a human in the loop.

Announcement of Python Library

In this section, the speaker announces a new python library that will allow users to automatically annotate their data sets with minimum effort.

Announcement of Python Library

• A new python library is being built that will allow users to automatically annotate their data sets with minimum effort.

• The library will use grounding Dino or some other zero-shot detectors to transfer knowledge through training process to real-time detectors like yellow V8.

• Stay tuned for release.