Operator Re-execution

Overview

When a communication operator fails because of a faulty link, the HCCL feature can rerun the operator to keep the training job running smoothly. During large-scale model training, if the cluster's communication link drops occasionally, executing the operator again avoids job failure caused by these interruptions. This enhances the overall stability of the training process.

Scenario

In a supernode cluster constructed with Ascend's next-generation hardware Snt9b23, the NPUs of the servers within each supernode are interconnected via the HCCS protocol using UnifiedBus, forming an efficient "hyperplane" network. Meanwhile, servers between two supernodes are interconnected via a parameter-plane network. However, jobs running in the supernode cluster rely on both networks for distributed communication. An intermittent interruption in either network can cause communication failure, ultimately leading to task failure. To solve this problem, ModelArts introduces a configurable operator re-execution capability, providing you with a reliable solution.

It mainly involves operator re-execution capabilities for two network failure scenarios: hyperplane and parameter plane:

Scenario 1: Parameter-Plane Network Failure

The Snt9b23 hardware features a single-PU, two-DIE architecture, where the two DIEs are connected via a high-speed SIO channel. If the parameter-plane link of the DIE-1 device completely fails, DIE-1 can borrow DIE-0's parameter-plane link via the SIO channel to communicate; enabling operator re-execution allows for this path-borrowing communication. If the DIE-1 device's parameter-plane link suffers a temporary interruption, enabling operator re-execution allows communication to continue over the existing link.

Figure 1 Path-borrowing communication

Scenario 2: Hyperplane Network Failure

In a supernode cluster composed of Ascend Snt9b23 hardware, NPUs in the hyperplane network are interconnected via a two-layer switch architecture. If an L1-L2 link fails, the switch's BGP path-switching algorithm can be leveraged to replan the communication path, ensuring successful operator re-execution and uninterrupted service.

Figure 2 BGP path switching

Constraints

• Only dedicated resource pools with Ascend Snt9b23 supernodes are supported.
• Only preset image environments using the PyTorch framework are supported.
• Only training jobs using full PU allocations are supported.
• Depends on Ascend components, with the following version constraints: HDK version 24.1.RC3.5 or later, and CANN version 8.1.RC1 or later.
• The MindSpore framework currently does not support operator re-execution.

Performance Impact

Once operator re-execution is enabled, it will impact performance and cannot be turned off. You can enable operator re-execution first to evaluate its performance impact on your workloads before deciding whether to keep it enabled.

Enabling operator re-execution affects workload performance. The performance impact based on tests with selected open-source models is shown in the table below. The actual impact depends on the communication operators used in your training model; if the performance impact on your workload model is significant, exercise caution when enabling this feature.

Enabling Operator Re-execution

There are two ways to enable operator re-execution: through the console or via the API.

• Console settings

  On the training job creation page, Re-execute operators can only be enabled (checked) when the training job configuration meets the criteria specified in Constraints.

  When operator re-execution is enabled, Open will be displayed on both the training job confirmation page and the training job details page. This parameter is not displayed if it is not configured or enabled.

  

  Do not enable operator re-execution by configuring the environment variable HCCL_OP_RETRY_ENABLE. Because platform components cannot detect this setting, enabling operator re-execution via this method will fail.

  Once operator re-execution is enabled, it will impact performance and cannot be turned off. You can enable operator re-execution first to evaluate its performance impact on your workloads before deciding whether to keep it enabled.

• API settings

  When creating a training job via the API, pass the fault-tolerance/hccl_op_retry field in annotations under the metadata field. Valid values for fault-tolerance/hccl_op_retry are true or false. Setting it to true enables operator re-execution. For details about the parameters, see Table 2.

  Table 2 Parameters

  Parameter	Mandatory	Type	Description
  kind	Yes	String	Description: Type of a training job. Constraints: N/A. Options: job: common job edge_job: edge job hetero_job: heterogeneous job mrs_job: MRS job autosearch_job: auto search job diag_job: diagnosis job visualization_job: visualization job Default Value: job
  annotations	No	Map<String,String>	Description: Advanced functions of a training job. Constraints: The options are as follows: job_template: Template RL (heterogeneous job) fault-tolerance/job-retry-num: 3 (number of retries upon a fault) fault-tolerance/job-unconditional-retry: true (unconditional restart) fault-tolerance/hang-retry: true (restart upon suspension) jupyter-lab/enable: true (JupyterLab training application) tensorboard/enable: true (TensorBoard training application) mindstudio-insight/enable: true (MindStudio Insight training application) fault-tolerance/hccl_op_retry: true (operator re-execution)

  For details about how to set request parameters, see the following example. For a training job with operator re-execution enabled, querying the training job details will also return the field "fault-tolerance/hccl_op_retry": "true".

  {
      "kind": "job",
      "metadata": {
          "annotations": {
              "fault-tolerance/hccl_op_retry": "true"
          }
      }
  }