Enabling and Viewing HA Configurations

In this section, you will learn about ModelArts HA configuration methods, recommendations, and how to view HA task details.

Because there is a lot of content, a quick overview is provided below to help you navigate directly to the topics you care about. Refer to the following categories to read the relevant sections:

To enable HA configurations in ModelArts, you can set them when creating a training job on the console or by using the job creation API. For details, see Enabling HA Configuration on the Console and Enabling HA Configuration Using an API.
To understand which detection items will trigger HA actions once enabled, see Check Items and Conditions.
To learn the complete workflow for training jobs with HA configurations enabled, see Recommended HA Configuration Workflow.
To select the optimal HA configurations for different use cases, see Common HA Configuration Combinations.
To review how fault tolerance environment checks performed after completing a job with HA enabled, see Effect of a Fault Tolerance Check.
To check whether HA is enabled for a job and view its status, see Checking Fault Tolerance and Recovery Details.

Constraints

Suspension can be detected only for training jobs that run on GPUs or NPUs.

Enabling HA Configuration on the Console

Follow these steps to enable HA configurations using the console:

Log in to the ModelArts console.
In the navigation pane, choose Model Build > Training.
Click Create Training Job. In the HA Settings section, check Fault Tolerance and Recovery to configure HA settings, as shown in Figure 1 or Figure 2.
Figure 1 Enabling HA configuration (new console)

Figure 2 Enabling HA configuration (old console)

When creating a training job, you can configure the capabilities in the HA Settings section as listed in Table 1.

**Table 1** HA configuration items
Configuration Item	Description
Fault Tolerance and Recovery (new console)/Auto Restart (old console)	If a training job fails due to environment issues, process suspensions, or other abnormalities, the system automatically detects the fault and applies recovery strategies to improve the success rate. For details about the check items, see Check Items and Conditions. Checking this option enables HA configuration. The system supports process-level, container-level, and job-level automatic restarts out of the box without extra setup, automatically matching and upgrading recovery policies.
Maximum Restarts	Controls the maximum number of automatic restarts allowed for a job to prevent infinite restart loops. The value ranges from 1 to 128. Once Fault Tolerance and Recovery (new console)/Auto Restart (old console) is checked, the auto-restart feature is enabled with a default maximum restart limit of 3. You can also manually customize this value.
Unconditional Auto Restart	When ModelArts detects a training job anomaly, it unconditionally restarts the job. To avoid invalid restart, the system supports only three consecutive unconditional restarts. Note: Unconditional auto restart occupies Maximum number of restarts. For example, Maximum number of restarts is set to 10 and Unconditional auto restart is enabled. If a training job unconditionally restarts three times in a row, Unconditional auto restart will be triggered. If the training job continues to repeatedly restart after the restart, Unconditional auto restart will no longer be triggered, leaving 7 remaining restart attempts.
Restart Upon Suspension	ModelArts continuously monitors job processes to detect suspension and optimize resource usage. When this feature is enabled, suspended jobs can be automatically restarted at the process level. To prevent unnecessary restarts, ModelArts limits consecutive restarts to three. Note: Restart Upon Suspension does not occupy Maximum number of restarts. For example, Maximum number of restarts is set to 10 and Restart Upon Suspension is enabled. If a training job is suspended and restarted for three consecutive times, Restart Upon Suspension will be triggered. If the training job still restarts repeatedly after the restart, Restart Upon Suspension will no longer be triggered. The remaining number of restart times is 10.
Re-execute operators	Automatically attempts to re-execute failed communication operators, primarily used in distributed training scenarios. For details, see Operator Re-execution.

Enabling HA Configuration Using an API

When creating a training job via the API, you can enable HA capabilities by configuring the corresponding parameters under annotations.

The following is an example request snippet. For specific field names, value ranges, and support details, see the API documentation for Creating a Training Job.

{
  "metadata": {
    "annotations": {
      "fault-tolerance/job-retry-num": "3",
      "fault-tolerance/job-unconditional-retry": "true",
      "fault-tolerance/hang-retry": "true",
      "fault-tolerance/pod-retry-num": "2"
    }
  }
}

**Table 2** API parameters for HA configuration
Parameter	Example Value	Description
fault-tolerance/job-retry-num	"3"	Number of job-level automatic restarts. If the fault-tolerance/job-retry-num field is added, auto restart is enabled. The value can be an integer ranging from 1 to 128. If this hyperparameter is not specified, the default value 0 is used, indicating that the job will not be re-issued and the environment pre-check will not be enabled.
fault-tolerance/job-unconditional-retry	"true"	Specifies whether to enable unconditional automatic restart.
fault-tolerance/hang-retry	"true"	Specifies whether to enable restart upon suspension.
fault-tolerance/pod-retry-num	"2"	Number of pod-level reschedules.

Check Items and Conditions

Table 3 helps you clearly understand under what conditions an automatic restart will be triggered.

**Table 3** Automatic restart check items
Check Item	Item (Log Keyword)	Execution Condition	Requirements for a Check
Domain name detection	dns	None	The domain names of the volcano containers in the .host file in /etc/volcano are successfully resolved.
Disk size – Container root directory	disk-size root	None	The directory is greater than 32 GB.
Disk size – /dev/shm	disk-size shm	None	The directory is greater than 1 GB.
Disk size – /cache	disk-size cache	None	The directory is greater than 32 GB.
ulimit check	ulimit	An IB network is used.	max locked memory > 16000 open files > 1000000 stack size > 8000 max user processes > 1000000
GPU check	gpu-check	GPU and the v2 training engine are used.	GPUs are detected.

Recommended HA Configuration Workflow

Below is the configuration workflow for enabling HA features for training jobs in ModelArts. You can follow these steps to complete your HA setup:

Ensure your training script already supports checkpoint saving and loading.
Confirm that the checkpoint path is located in a location accessible after a job restarts.
Enable automatic restart on the job creation page in the ModelArts console or via API parameters.
Configure an appropriate value for the maximum number of restarts.
For multi-node multi-PU tasks, enable job suspension detection or restart upon suspension.
For Ascend supernode communication failure scenarios, evaluate whether to enable operator re-execution.
Submit a training job.
Monitor logs, events, and recovery details while the job is running.

Common HA Configuration Combinations

This section provides recommended HA configurations for common training scenarios. You can refer to the settings according to your specific training scenario. For details, see Table 4.

**Table 4** Common configuration combinations
Training Scenario	Configuration Description	Applicable Scenario
Single-node training	{ "metadata": { "annotations": { "fault-tolerance/job-retry-num": "2" } } }	Single-node training. The training script supports checkpoints. Automatic restart is required after an abnormal job failure.
Multi-node multi-PU training	{ "metadata": { "annotations": { "fault-tolerance/job-retry-num": "3", "fault-tolerance/job-unconditional-retry": "true", "fault-tolerance/hang-retry": "true" } } }	Multi-node distributed training. Long training cycle. Automatic recovery is required for abnormal exits and suspension scenarios.
Long-term training	{ "metadata": { "annotations": { "fault-tolerance/job-retry-num": "5", "fault-tolerance/job-unconditional-retry": "true", "fault-tolerance/hang-retry": "true", "fault-tolerance/pod-retry-num": 3 (number of pod reschedules) } } }	Foundation model pre-training or long-context fine-tuning. Able to save checkpoints periodically. Able to tolerate the additional restart time caused by automatic job recovery.

Effect of a Fault Tolerance Check

If the fault tolerance check is passed, the logs of the check items will be recorded, indicating that the check items are successful. You can search for the keyword item in the log file. A fault tolerance check minimizes reported runtime faults.
If a fault tolerance check fails, check failure logs will be recorded. You can search for the keyword item in the log file to view the failure information.

If the number of job restarts does not reach the specified time, the job will be automatically issued again. You can search for keywords error,exiting to obtain the logs recording a restarted job that ends with a failure.

Checking Fault Tolerance and Recovery Details

When a training job fault occurs (such as in-place NPU recovery and job-level rescheduling), the Fault Recovery Details tab appears on the job details page, recording the start and stop details of the training job.

If you enable auto restart when creating a training job, you can view the number of restart times on the training job details page. Restarts displays the current number of restart times and the maximum number of restart times. In the Fault tolerance and recovery tab of the training job details page, you can view the restart details of the training job.

Figure 3 Restarts