Data Selection (Hard Examples)

Algorithm Overview

In actual service scenarios, model maintenance is a long-term process. For example, data retraining is performed weekly or monthly, or periodic retraining is required when data accumulates to a certain volume. Full-data retraining often consumes a large amount of labeling manpower and training time. To improve the model maintenance efficiency, hard examples-based retraining can be adopted.

A hard example filtering algorithm is used to analyze and filter full data and output only a small amount of valuable data for model maintenance. In this case, retraining with the filtered data effectively reduces the labeling manpower and training time.

Multiple methods are integrated in the hard example filtering algorithm. To achieve the optimal effect, you need to select some or all methods and adjust their weights based on your needs.

Parameters

Operator Input Requirements

The following two types of operator input are available:

• Datasets: Select a dataset and its version created on the ModelArts console from the drop-down list. Ensure that the dataset type be the same as the scenario type selected in this task.
• OBSCatalog: The directory must contain the raw image for inference and the inference result file.

  The directory structure is as follows:

  input_path/
     --images/ #The folder name must be images.
          ----1.jpg
          ----2.jpg
      --inference_results/ # The folder name must be inference_results.
          ----1.jpg_result.txt
          ----2.jpg_result.txt

  The .txt inference result file must meet the following requirements: If you use the model trained by the ModelArts built-in algorithm for inference, the default inference result meets the requirements.

  • Image classification

    {
        "predicted_label": "dog",
        "scores": [
            [
                "dog",
                "0.589"
            ],
            [
                "cat",
                "0.411"
            ]
        ]
    }

  • Object detection

    {
        "detection_classes": [
            "cat",
            "cat"
            ],
        "detection_boxes": [
            [
                117.56356048583984,
                335.9902648925781,
                270.50848388671875,
                469.0136413574219
            ],
            [
                18.747316360473633,
                13.10757064819336,
                217.25146484375,
                108.3551025390625
            ]
        ],
        "detection_scores": [
                0.5179755091667175,
                0.46941104531288147
            ]
    }

Output Description

• Object detection

  The output directory structure is as follows:

  output_path :
      --Data
          ----1.jpg
          ----1.xml     # Export the filtering result to this directory.
      --output.manifest

  A manifest file example is as follows:

  {"source":"/tmp/test_out/object_detection/images/be462ea9c5abc09f.jpg",
  "hard":"True",
  "hard-reasons":"0", # Reason why the sample is determined as a hard example. The specific reason is displayed only in the auto labeling module.
  "hard-coefficient":"1.0", # Hard example coefficient obtained using the hard example algorithm. A larger value indicates a higher probability that the sample is a hard example.
  "annotation":[
  {"annotation-loc":"/tmp/test_out/object_detection/annotations/be462ea9c5abc09f.xml",
  "type":"modelarts/object_detection",
  "annotation-format":"PASCAL VOC",
  "annotated-by":"modelarts/hard_example_algo"}]}

• Image classification

  The output directory structure is as follows:

  output_path :
      --Data
          ----class1
              ------1.jpg
          ----class2
              ------2.jpg
      --output.manifest

  A manifest file example is as follows:

  {"source":"obs://obs_bucket_name/folder_name/catDog/5.jpg",
  "hard":true,
  "hard-reasons":"1-20-2-19-21-3",
  "hard-coefficient":1.0,
  "annotation":[
  {"name":"cat",
  "type":"modelarts/image_classification",
  "confidence":0.599,
  "annotated-by":"modelarts/hard_example_algo"}]}

Log File Description

task_summary_file is the output file path of the simplified log. The content is as follows:

{
"task_status": 'SUCCEED', # Algorithm execution status
"total_sample": integer,  # Total input samples
"hard_sample": integer # Total output samples
}

or

{
"task_status": 'FAILED',
"error_message": 'xxxxxx' # Error information that causes the algorithm execution failure
}