SimDeduplication Operator Overview

Image deduplication is a common method for image data processing. Image duplication means that the image content is completely the same, or the scale, displacement, color, or brightness changes slightly, or a small amount of other content is added. The SimDeduplication operator can implement image deduplication based on the similarity threshold you set.

Figure 1 SimDeduplication effect

Parameters

Operator Input Requirements

The following two types of operator input are available:

• Datasets: Select a dataset and its version created on the ModelArts console from the drop-down list. Ensure that the dataset type be the same as the scenario type selected in this task.
• OBSCatalog: Select either of the following storage structures:
  • Only images: If the directory contains only images, the JPG, JPEG, PNG, and BMP formats are supported, and all images in the nested subdirectories are read.
  • Images and labels: The structure varies depending on the scenario type.

    The following shows the directory structure in the image classification scenario. The following directory structure supports only single-label scenarios.

    input_path/
        --label1/
            ----1.jpg
        --label2/
            ----2.jpg
        --../

    The following shows the directory structure in the object detection scenario. Images in JPG, JPEG, PNG, and BMP formats are supported. XML files are standard PACAL VOC files.

    input_path/
        --1.jpg
        --1.xml
        --2.jpg
        --2.xml
        ...

Output Description

• Image classification

  The output directory structure is as follows:

  output_path/
      --Data/
          ----class1/  # If the input data has labeling information, the information is also output. class1 indicates the labeling class.
              ------1.jpg
          ----class2/
              ------2.jpg
              ------3.jpg
      --output.manifest

  The following shows a manifest file example.

  {
  	"id": "xss", 
  	"source": "obs://home/fc8e2688015d4a1784dcbda44d840307_14.jpg",
  	"usage": "train", 
  	"annotation": [
  		{
  			"name": "Cat", 
  			"type": "modelarts/image_classification"
  		}
  	]
  }

• Object detection
  The output directory structure is as follows:

  output_path/
      --Data/
          ----1.jpg
          ----1.xml  # If the input data has labeling information, the information is also output. xml indicates the label file.
          ----2.jpg
          ----3.jpg
      --output.manifest

  The following shows a manifest file example.

  {
  	"source":"obs://fake/be462ea9c5abc09f.jpg",
  	"annotation":[
  		{
  			"annotation-loc":"obs://fake/be462ea9c5abc09f.xml",
  			"type":"modelarts/object_detection",
  			"annotation-format":"PASCAL VOC",
  			"annotated-by":"modelarts/hard_example_algo"
  			}
  	]
  }