Model High Availability (HA) Feature Overview

What Is the Model Training HA Feature?

During long-running and large-scale model training, job interruptions are inevitable due to hardware stability issues, network fluctuations, and other factors. If a training job can recover quickly after an interruption, it significantly improves compute device utilization while reducing time costs and compute losses for users.

ModelArts provides a model training high availability (HA) feature. It includes capabilities such as resumable training, fault detection, automatic restart, faulty node isolation, suspension detection, operator re-execution, and failure log analysis. These features enhance the stability of long-running, multi-node multi-PU training jobs for LLMs. In the event of a failure, the system automatically restores the training state and resumes execution, minimizing compute loss.

Model HA primarily addresses the following issues:

• Job failures caused by infrastructure anomalies in nodes, chips, networks, or storage during training.
• Overall task failure triggered by an anomaly in a single node during multi-node multi-PU training.
• Training suspension, where the training process does not exit but makes no meaningful progress for an extended period.
• Distributed training failure caused by sporadic issues in communication links.

1. Enabling ModelArts HA configurations improves automatic recovery capabilities, but it cannot replace fault-tolerant code design. You should still thoroughly incorporate fault tolerance into your training scripts.
2. For long-running tasks, especially LLM training, it is strongly recommended that you implement checkpoint saving and resumable training from checkpoints.
3. If HA is not enabled, ModelArts will not handle exceptions detected during training.

Model Training HA Feature Map

Table 1 lists the HA capabilities supported by ModelArts and their corresponding scenarios. Select the features that best match your requirements.