Creating a Hyperparameter Search Job

Background

If the AI engine is pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64 or tensorflow_2.1.0-cuda_10.1-py_3.7-ubuntu_18.04-x86_64 and the hyperparameter to be optimized is of the float type, you can use hyperparameter search on ModelArts.

You can perform the hyperparameter search without any code modification. The procedure is as follows:

1. Preparations
2. Creating an Algorithm
3. Creating a Training Job
4. Viewing Details About a Hyperparameter Search Job

Preparations

• Data has been prepared. Specifically, you have created an available dataset in ModelArts, or you have uploaded the dataset used for training to the OBS directory.
• Prepare the training script and upload it to the OBS directory. For details about how to develop a training script, see Developing a Custom Script.
• In the training code, print the search indicator parameters.
• At least one empty folder has been created in OBS for storing the training output.
• The account is not in arrears because resources are consumed when training jobs are running.
• The OBS directory you use and ModelArts are in the same region.

Creating an Algorithm

Log in to the ModelArts management console and create an algorithm by referring to Creating an Algorithm. The image must use the pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64 or tensorflow_2.1.0-cuda_10.1-py_3.7-ubuntu_18.04-x86_64 engine.

Hyperparameters that you want to optimize need to be defined when you configure Hyperparameters. You can specify the name, type, default value, and constraints. For details, see Defining Hyperparameters.

Select autoSearch(S) to enable auto search for the algorithm. During an auto search, ModelArts uses a regular expression to obtain the search indicator parameters and performs hyperparameter optimization based on the optimization direction. Print search parameters in the code and set the following parameters.

Figure 1 Enabling auto search

• Search Indicator

  The search indicator is the value of the objective function, which can be loss, accuracy, and so on. By optimizing and converging this value based on the optimization direction, the optimal hyperparameter can be found to improve the model precision and convergence speed.

  Table 1 Search indicator parameters

  Parameter	Description
  Name	search indicator name. This parameter must be identical to the search indicator parameter in the code.
  Optimization Direction	This parameter can be max or min.
  Counter regularization	A regular expression needs to be entered. You can click Generate Intelligently to generate a regular expression automatically.

• Setting Automatic Search Parameters

  Select hyperparameters that can be used for search optimization from what you configured for Hyperparameters. Only hyperparameters of the float type are supported. After autoSearch(S) is selected, set the value range.

• Search Algorithm Configuration

  ModelArts has three built-in algorithms for hyperparameter search. You can select one or more algorithms as needed. The algorithms and their parameter description are as follows:

  • bayes_opt_search:Bayesian Optimization (SMAC)
  • tpe_search:TPE Algorithm
  • anneal_search:Simulated Annealing Algorithm

After the algorithm is created, use it to create a training job.

Creating a Training Job

Log in to the ModelArts console and create a training job by referring to Creating a Training Job. Pay attention to operations described in this section before you enable the hyperparameter search.

Viewing Details About a Hyperparameter Search Job

After a training job is completed, you can review the results of the automated hyperparameter search to evaluate the job's performance.

If the training job is a hyperparameter search job, go to the training job details page and click the Auto Search Results tab to view the hyperparameter search results.

Figure 3 Hyperparameter search results