Precautions for Code Compilation

Pay attention to the following when using user code:

• ModelArts passes trial_id as the unique identifier for running the user code once. Ensure that the unique identifier is used to save the file to avoid file read/write conflicts when multiple tasks are concurrently executed.
• In the boot code, a method named train needs to be provided, so that ModelArts can call the method to perform complete training, evaluation, and model export. If training and evaluation are separated, you can provide a method named eval in the code as the entry for model evaluation and export. The AutoSearch framework calls eval in the same process after train is executed. The train and eval methods do not have input parameters.
• Note the following when using argparse: ModelArts passes extra parameters when running user code. Therefore, you need to use non-exclusive input parameters to parse APIs. For example, use parse_known_args of argparse to replace parse_args.

Using autosearch.config

The search space defined in YAML is processed by the search algorithm and sent to users in the dict format. You can obtain the search space from the autosearch.config file in the user code.

Generally, you can obtain the configuration corresponding to this training using autosearch.config['key'], for example:

search_space:
  - type: discrete
    params:
      - name: resnet50
        values: ["1-11111111-2111121111-211111",
                 "1-1112-1111111111121-11111112111",
        ]

That is, you can obtain the configuration used in this training by calling autosearch.config['resnet50']. In general, the architecture code in config may vary according to the search algorithm. For details, see Example: Replacing the Original ResNet-50 with a Better Network Architecture, Example: Searching for Hyperparameters Using Classic Hyperparameter Algorithms, Example: Searching for Network Architectures Using the MBNAS Algorithm, Example: Implementing Auto Data Augmentation Using a Preset Data Augmentation Policy, and Example: Using Multisearch.

Using autosearch.reporter

You can use autosearch.reporter to feed back training results.

Generally, a search algorithm requires a reward_attr. The larger the reward_attr value, the better. In addition, the search algorithm needs to support mathematical expressions to facilitate multi-object search.

search_algotirhm:
    type: xxx
    reward_attr: accuracy - 0.1 * latency

The two metrics need to be reported to the AutoSearch framework in the training code.

autosearch.reporter(accuracy=foo, latency=bar)

The search algorithm calculates the reward based on the reported result and updates the algorithm status. Additional information you report will not be used.

Modifying Reporter Without Modifying Code

In addition to using autosearch.reporter to feed back the training result, you can analyze the printed logs to automatically obtain the feedback information. After specifying reward_attr, you need to configure a regular expression pointing to metrics in reward_attr in the YAML configuration file.

For example, if the result information is as follows:

I can see atdout in screen
result=-1.000000;

The YAML configuration file as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

search_algorithm:
  type: anneal_search
  max_concurrent: 2
  reward_attr: result
  report_keys:
    - name: result
      regex: (?<=result=).+(?=;)
  save_model_count: 3
  num_samples: 6
  mode: max

The system uses the regular expression to search for the corresponding metrics in the printed information and automatically provides feedback. In this way, you do not need to modify the service code or maintain additional code.

Using a Preset Decoder

A decoder converts the input code (character string) into the code of an executable algorithm framework (such as TensorFlow or PyTorch). ModelArts provides several types of decoders. You can use these decoders in the code module to greatly reduce your workload. You do not need to modularize your code or parse network architectures or augmentation policies in the code. You only need to embed the preset decoder code.

• ResNet50 decoder for NAS

  The search space corresponding to this preset decoder is used to search for the number of blocks in each stage of ResNet and search for the block position when channels are doubled. The encoding character string format of the backbone architecture is 111-2111-211111-211, where - is used to separate each downsampling phase by using different resolutions, 1 indicates a regular block when channels are not changed, and 2 indicates a block when the basic channels are doubled. The character string indicates a standard ResNet50 architecture. Therefore, each stage can have any number of 1s, and a total of three 2s are required.

  Other ResNet decoders are similar to the ResNet50 decoder.

• Decoder for data augmentation

  For each of the NumPy, TensorFlow, and PyTorch frameworks or libraries (commonly used on the preprocessing node), a data augmentation decoder is provided to decode character strings into data augmentation operations on the corresponding frameworks or libraries.

Built-in Environment Information Reference

Table 2 describes the built-in environment of ModelArts for auto search jobs. The environment version cannot be modified. You can refer to it when using auto search jobs.