Help Center/ ModelArts/ Model Training/ Fault Tolerance and Recovery/ HA Configuration Scenarios

Updated on 2026-08-27 GMT+08:00

HA Configuration Scenarios

This section recommends HA configurations for common scenarios, providing a reference for setting up your HA training jobs.

**Table 1** Recommended HA configurations
Scenario	Description	Recommended Configuration	Configuration Combination	Recommendation
Single-node training	Suitable for single-node single-PU and single-node multi-PU training.	Configure the checkpoint save path. Set up resuming from checkpoint in the training script. Enable auto restart. Set a reasonable maximum number of restarts. Troubleshoot issues via failure log analysis after job failure.	Resumable training	Recommended
			Auto restart	Recommended
			Job suspension detection	Optional
			Restart upon suspension	Optional
			Operator re-execution	Usually not needed
Multi-node multi-PU training	Suitable for distributed training and large-scale GPU/NPU training.	Ensure training scripts support checkpoint saving and resumption. Save checkpoints in reliable shared storage. Ensure training scripts support repeated launches (i.e., re-entrancy capability). Enable auto restart. Enable unconditional auto restart based on service needs. Enable job suspension detection. Enable restart upon suspension for training tasks requiring long-term stability. Monitor job recovery through events, logs, and recovery details.	Resumable training	Required
			Auto restart	Recommended
			Unconditional auto restart	Recommended
			Job suspension detection	Recommended
			Restart upon suspension	Recommended
			Pod-level rescheduling	Based on resource pool capabilities
			Isolated job-level rescheduling	Based on resource pool capabilities
LLM long-term stable training	Suitable for tasks with long durations and large resource scales, such as LLM pre-training, fine-tuning, and reinforcement learning.	Use stable shared storage to save checkpoints. Periodically save training states including models, optimizers, learning rate schedulers, and random states. Support automatic recovery from the most recent valid checkpoint. Enable auto restart and fault recovery. Enable job suspension detection and restart upon suspension. Evaluate whether to enable operator re-execution based on hardware scenarios. Configure logs, events, and failure analysis loops.	Resumable training	Required
			Auto restart	Required
			Unconditional auto restart	Recommended
			Suspension detection	Recommended
			Restart upon suspension	Recommended
			Operator re-execution	Recommended for specific Ascend supernode scenarios
			Log failure analysis	Recommended

Parent topic: Fault Tolerance and Recovery

Previous topic: Model High Availability (HA) Feature Overview

Next topic: High Reliability Preparation: Resumable Training