Symptom

In an object detection task, a bounding box of an image may be overlapped by another bounding box, which can be reflected by the overlap degree of bounding boxes. A larger overlap degree indicates that more parts of a bounding box are overlapped by other bounding boxes. Object detection models have different detection effects for datasets with different overlap degrees. The following figure shows the scenario where many bounding boxes are overlapping. You are advised to refer to the following algorithms and technical description to understand how to reduce the sensitivity of object detection models to bounding box overlap degrees.

Figure 1 Scenario where many bounding boxes are overlapping

Solution

In the model inference phase, the NMS uses the IoU between objects and the corresponding thresholds for processing. In this case, multiple objects with a high overlap degree cannot be detected. The following describes the processing process of the NMS in the traditional object detection model, where B indicates the bounding box.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

  Input: B={(Bi, Si)}_(i=1 to N), where Si is the score of Bi. D = Empty set  
  Step 1 Select the box M with the highest score from B.
  Step 2 Add M and its score to D, and delete M and its score from B.
  Step 3    for Bi in B:  
               if  IoU(M, Bi) >= NMS_threshold  
                  Delete Bi and its score from B.
               end if  
             end for  
  Step 4 Repeat steps 1 to 3 until B is an empty set.  
  Output: D

The disadvantages of the traditional NMS are as follows:

1. The boxes that are greater than the NMS thresholds are all forcibly deleted. The following shows the formula, where si indicates the class prediction probability:

   

2. The calculation focuses on the IoU, that is, the overlapping part of two bounding boxes. Other positions of the two bounding boxes are not fully considered.

   The Soft NMS uses a smoother filtering rule to overcome the shortcoming of the traditional NMS:

   

   Another Gaussian expression:

   

   The soft NMS can reduce the sensitivity of object detection models to the overlap degrees of bounding boxes.

Verification

The open source dataset fruit is used for verification. The following table analyzes the sensitivity to the overlap degrees of bounding boxes before the soft NMS is used.

The feature distribution indicates that the entire dataset overlap degree is divided based on percentage intervals, and the corresponding values are recall rates. The sensitivity of Banana is 0.3811.

The following figure shows the change of the sensitivity to bounding box overlap degrees after the soft NMS is used. The sensitivity of the Banana type to bounding box overlap degrees decreases from 0.3811 to 0.2487. This verifies that the algorithm can optimize the model in terms of bounding box overlap.

Suggestions

In the model inference result, if the detected classes are very sensitive to the overlap degrees of bounding boxes, you are advised to use the soft NMS for model optimization and enhancement during training and inference.