Creating a Production Training Job (New Version)

Input

Click Add and configure training inputs.

• Parameter name

  The algorithm code reads the training input data based on the input parameter name.

  The recommended value is data_url. The training input parameters must match the input parameters of the selected algorithm.

• Input Source: The training input supports OBS storage and datasets.
  • Click next to Input Source and select the storage path to the training input data from an OBS bucket. Files must not exceed 10 GB in total size, 1,000 in number, or 1 GB per file.
  • Click Dataset and select the target dataset and its version in the ModelArts dataset list.

  When the training job is started, ModelArts automatically downloads the data in the input path to the training container.

• Obtained from

  The following uses training input data_path as an example.

  • If you select Hyperparameters, use this code to obtain the data:

    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--data_path')
    args, unknown = parser.parse_known_args()
    data_path = args.data_path 

  • If you select Environment variables, use this code to obtain the data:

    import os
    data_path = os.getenv("data_path", "")

Output

Click Add and configure training outputs.

• Parameter name

  The algorithm code reads the training output data based on the output parameter name.

  The recommended value is train_url. The training output parameters must match the output parameters of the selected algorithm.

• Data path

  Click next to Data path and select the storage path to the training output data from an OBS bucket. Files must not exceed 1 GB in total size, 128 in number, or 128 MB per file.

  During training, the system automatically synchronizes files from the local code directory of the training container to the data path.

  Only OBS can be used to store output data. Choose an empty folder for your output files to prevent issues with stored training inputs.

• Obtained from

  The following uses the training output train_url as an example.

  • If you select Hyperparameters, use this code to obtain the data:

    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--train_url')
    args, unknown = parser.parse_known_args()
    train_url = args.train_url 

  • If you select Environment variables, use this code to obtain the data:

    import os
    train_url = os.getenv("train_url", "")

• Predownload to Container Directory

  Choose whether to pre-download files in the output directory to the training container.

  • If you set this parameter to No, the system does not download the files in the training output path to the local directory of the training container when the training job is started.
  • If you set this parameter to Yes, the system automatically downloads the files in the training output path to the local directory of the training container when the training job is started. The larger the file size, the longer the download time. To avoid excessive training time, remove any unneeded files as soon as possible. Select Yes for Incremental Model Training or Resumable Training.

Hyperparameter

Used for tuning. This parameter is determined by the selected algorithm. If hyperparameters have been defined in the algorithm, all hyperparameters in the algorithm are displayed.

Hyperparameters can be modified and deleted. The status depends on the hyperparameter constraint settings in the algorithm. For details, see Table 6.

• Click Add to add hyperparameters. The total number of hyperparameters cannot exceed 100.
• To import hyperparameters in batches, click Upload. You will need to fill in the hyperparameters based on the provided template. The total number of hyperparameters should not exceed 100, or the import will fail.

Environment Variable

Add environment variables based on service requirements. For details about the environment variables preset in the training container, see Managing Environment Variables of a Training Container.

• Click Add to add environment variables. The total number of environment variables cannot exceed 100.
• To import environment variables in batches, click Upload. You will need to fill in the environment variables based on the provided template. The total number of environment variables should not exceed 100, or the import will fail.

Automated Hyperparameter Search

If you select My algorithm for Algorithm Type and the selected algorithm supports the autoSearch(S) policy, you can click More Configurations to show Automated Hyperparameter Search.

Selecting it enables automated hyperparameter search during training, potentially increasing the time needed.

For details, see Overview.

Source of resources

• Public resource pool: The public resource pool is available for all tenants and does not require user creation.
• Dedicated resource pool: Dedicated resource pools are created separately and used exclusively. For details, see Creating a Standard Dedicated Resource Pool.

Resource Pool

After selecting Dedicated resource pool, click Select Resource Pool to select the target dedicated resource pool. You can view the status, node specifications, number of idle/fragmented nodes, number of available/total nodes, and number of cards of the dedicated resource pool. Hover over View in the Idle/Fragmented Nodes column to check fragment details and check whether the resource pool meets the training requirements.

Specification Type

This parameter is displayed when you select a dedicated resource pool. The following specifications types are supported:

• Preset: Select preset instance specifications in the dedicated resource pool.
• Customized Specifications: Training jobs allow custom resource specifications using dedicated pools to enhance their utilization. Custom specifications must match or stay below the node specifications of the dedicated resource pool. For CPU specifications, you can only customize the number of vCPUs and memory. For GPU and Ascend specifications, you can customize the number of vCPUs, memory, and cards.

Figure 2 Specifications

Specifications

This parameter is mandatory when Specification Type is set to Preset. Select resource specifications, including server type and model.

• If a resource type has been defined in the training code, select a proper resource type based on algorithm constraints. For example, if the resource type defined in the training code is CPU and you select other types, the training fails.
• If some resource types are invisible or unavailable for selection, they are not supported.
• If Input is configured, click Check Input Size next to resource pool specifications to ensure the storage is larger than the input data size.

Compute Nodes

Select the number of instances as required. The default value is 1.

• If only one instance is used, a single-node training job is created. ModelArts starts one training container on this node. The training container exclusively uses the compute resources of the selected specifications.
• If more than one instance is used, a distributed training job is created. For more information about distributed training configurations, see Overview.

  Before creating a distributed training job, pre-install all required pip dependencies (see Installing pip Dependencies in an Image). If there are more than 10 nodes, the system automatically deletes the pip source configuration. Executing pip install commands during training may cause training failures.

Storage Mounting

When you select a dedicated resource pool, you can mount multiple storage types to improve data access efficiency.

Figure 3 Storage mounting

• Add SFS Turbo

  When ModelArts and SFS Turbo are directly connected, multiple SFS Turbo file systems can be mounted to a training job to store training data. You can mount a file system multiple times, but each mount path must be distinct. A maximum of five disks can be mounted to a training job. For details about how to configure the network connection between ModelArts and SFS Turbo, see Creating a Network.

  Figure 4 SFS Turbo
  
  • Name: Select an SFS Turbo file system.
  • Mount Path: Enter the SFS Turbo mount path in the training container. The path cannot be a / directory or a system-mounted directory like /cache or /home/ma-user/modelarts.
  • Directory: Specify the SFS Turbo storage location. If you have configured the folder control permission, select a storage location. If you have not configured the folder control permission, retain the default value / or customize a location.
  • Mounting Mode: Permission on the mounted SFS Turbo file system. This parameter is displayed as Read/Write or Read-only based on the permission of the SFS Turbo storage location. If you have not configured the folder control permission, this parameter is unavailable. For details about how to set permissions for SFS Turbo folders, see Permissions Management.
  • Mount Options: Configure SFS mount parameters to accelerate and optimize training. For details about the parameters, see Configuring SFS Turbo Mount Options. Alternatively, retain the default settings below:

    mountOptions:
    - vers=3 
    - timeo=600 
    - nolock 
    - hard

• Add SFS 3.0 Capacity-Oriented: A training job can mount an SFS 3.0 file system to store training data. Set the parameters below.
  • Name: Enter the name of the file system. The name must be the same as that in SFS. Otherwise, the mounting fails.
  • Mount Path: Enter the cloud mount path in the training container.
  NOTE:

  To add SFS 3.0 Capacity-Oriented, submit a service ticket.

• Add PFS: A training job can mount an OBS parallel file system to store training data. Set the parameters below.
  Figure 5 PFS
  
  • Storage Configuration: Select a parallel file system.
  • Mount Path: Enter the cloud mount path in the training container.
  NOTE:

  To add PFS, submit a service ticket.

Supernode Affinity Group Instances

• Selecting a dedicated resource pool and an Snt9b23 flavor allows you to set Supernode Affinity Group Instances.
• You can set this parameter if you select a supernode resource pool. If Supernode Affinity Group Instances is set to N, every N pods are scheduled to the same supernode to schedule affinity jobs. In distributed training, affinity job scheduling ensures uniformity in the architecture of the allocated compute resources.
• You must set the number of instances as an integral multiple of Supernode Affinity Group Instances. Otherwise, the training job cannot be created.
• For more information about supernode affinity group instances, see Configuring Supernode Affinity Group Instances.

Training Mode

ModelArts offers various training modes when using a MindSpore preset image with Ascend specifications.

• Common mode: It is the default training scenario.
• High-performance mode: Certain O&M functions will be adjusted or even disabled to maximally accelerate the running speed, but this will deteriorate fault locating. This mode is suitable for stable networks requiring high performance.
• Fault diagnosis mode: Certain O&M functions will be enabled or adjusted to collect more information for locating faults.

Auto Restart

Choose whether to enable automatic restart for a training job.

• This function is disabled by default. If a training exception occurs, the job is directly stopped.
• If this function is enabled, the system will handle any exceptions caused by environmental or suspension issues during a training job. The system automatically detects faults and processes them according to the corresponding policies, thereby increasing the training success rate. Training job recovery policies enable automatic restarts at the process, container, and job levels. These policies require no manual configuration as they are automatically applied and upgraded as needed.

  To avoid losing training progress, ensure your code can resume training from where it is interrupted, and then enable unconditional auto restart to optimize compute usage. For details, see Resumable Training.

  If auto restart is triggered during training, the system records the restart information. You can check the fault recovery details on the training job details page. For details, see Training Job Fault Tolerance Check.

Maximum Restarts

This parameter is available when Auto Restart is enabled.

The training job will stop if it is still abnormal after maximum automatic restarts.

• Default value: 3
• Value range: 1 to 128

The value cannot be changed once the training job is created. Set this parameter based on your needs.

Unconditional Auto Restart

This parameter is available when Auto Restart is enabled. If Unconditional auto restart is selected, the training job will be restarted unconditionally once the system detects a training exception. To prevent invalid restarts, the system limits unconditional restarts to three consecutive attempts.

Restart Upon Suspension

This parameter is available when Auto Restart is enabled. ModelArts continuously monitors job processes to detect suspension and optimize resource usage. When this feature is enabled, suspended jobs can be automatically restarted at the process level.

However, ModelArts does not verify code logic, and suspension detection is periodic, which may result in false reports. By enabling this feature, you acknowledge the possibility of false positives. To prevent unnecessary restarts, ModelArts limits consecutive restarts to three.

Increase job scheduling priority

• Jobs join the queue based on their submission time by default. Enabling this feature increases job priorities in the queue.
• When using a dedicated resource pool, you can increase the scheduling priority of the training job. This parameter is not supported when a public resource pool is used.
• The priority can be set to 1, 2, or 3. A larger number indicates a higher priority. The default priority is 1, and the highest priority is 3.
• To set the priority to 3, you will also need the permission. For details about how to set the permission, see Assigning the Permission to Set the Highest Job Priority to an IAM User.
• If a training job is in the Pending state for a long time, you can change the job priority to reduce the queuing duration. For details, see Priority of a Training Job.

Preemption

When using a dedicated resource pool, you can set this parameter. This parameter is not supported when a public resource pool is used.

When enabled, jobs that allow preemption may be terminated and re-queued if resource pool capacity is insufficient. To avoid losing training progress, configure resumable training before enabling this function. For details, see Resumable Training.

Auto Stop

Choose whether to enable Auto Stop.

• This function is disabled by default, and the training job keeps running until the training is completed.
• If you enable this function, set the auto stop time. The value can be 1 hour, 2 hours, 4 hours, 6 hours, or Customize. The customized time must range from 1 hour to 720 hours. When you enable this function, the training stops automatically when the time limit is reached. The time limit does not count down when the training is paused.

Persistent Log Saving

This function is enabled by default when Ascend specifications are selected and cannot be modified.

This function is available when CPU or GPU specifications are selected.

• If this function is enabled (default), configure Log Path. The platform permanently stores training logs to the specified OBS path.
• If this function is disabled, ModelArts automatically stores the logs for 30 days. You can download all logs on the job details page to a local path.

Log Path

When Persistent Log Saving is enabled, you must configure a log path to store log files generated by the training job.

Ensure that you have read and write permissions to the selected OBS directory.

Event Notification

Choose whether to enable event notification for the training job.

• This function is deselected by default, which means SMN is disabled.
• If this function is enabled, you will be notified of specific events, such as job status changes or suspected suspensions, via an SMS or email. Notifications will be billed based on SMN pricing. In this case, you must configure the topic name and events.
  • Topic: topic of event notifications. Click Create Topic to go to the SMN console to create a topic and add a subscription to the topic. You will receive event notifications only after the subscription status changes to Confirmed. For details, see Adding a Subscription.
  • Event: events you want to subscribe to. Examples: JobStarted, JobCompleted, JobFailed, JobTerminated, JobHanged, JobRestarted, and JobPreempted.

Password-free SSH Between Nodes

Choose whether to enable password-free SSH between nodes.

• This function is disabled by default, which means the password-free SSH file is not generated.
• If this function is enabled, the password-free SSH file is generated. To enable distributed training with a custom MPI or Horovod-based image, set up password-free SSH trust between nodes for seamless communication.

  Configure the password-free SSH file directory. This directory stores the auto-generated SSH key files in the training container. By default, it is set to /home/ma-user/.ssh. For details, see Configuring Password-free SSH Mutual Trust Between Instances for a Training Job Created Using a Custom Image.

Tags

TMS's predefined tags are recommended for adding the same tag to different cloud resources. For details about how to use tags, see Using TMS Tags to Manage Resources by Group.

You can add up to 20 tags to a training job.