Rectifying a Faulty Node in a Standard Dedicated Resource Pool

Handling Faulty Nodes

• Replacing a node: The node name will be changed and the original node will be deleted.

  In the Nodes tab, locate the node to be replaced, and click Replace in the Operation column. No fee is charged for this operation.

  Check the node replacement records on the Records page. Running indicates that the node is being replaced. After the replacement, you can check the new node in the node list.

  The replacement can last no longer than 24 hours. If no suitable resource is found after the replacement times out, the status changes to Failed. Hover over to check the failure cause.

  

  • The number of replacements per day cannot exceed 20% of the total nodes in the resource pool. The number of nodes to be replaced cannot exceed 5% of the total nodes in the resource pool.
  • Ensure that there are idle node resources. Otherwise, the replacement may fail.
  • If there are any nodes in the Resetting state in the operation records, nodes in the resource pool cannot be replaced.
• HA redundant node

  An HA redundant node is the backup node in the dedicated resource pool. It can automatically replace a faulty common node to improve the Service Level Agreement (SLA) of the resource pool and prevent service loss. You can set the number of HA nodes based your requirements.

  

  HA redundant nodes cannot be used for service running, which will affect the number of actual available nodes in the resource pool. When a resource pool delivers a task, select the number of actual available nodes with caution. If the HA redundant nodes are included, the task will keep waiting.

  The running mechanism of HA redundant nodes are as follows:

  • HA redundant nodes are isolated and cannot be scheduled by default. Workloads cannot be scheduled to the nodes.
  • HA redundant nodes function as standby nodes and work with node fault detection to ensure that the faulty nodes are automatically replaced by normal nodes in a resource pool. The replacement normally takes only a few minutes. After the replacement, the original HA redundant nodes will be de-isolated and become normal nodes, and the faulty nodes will be labeled as HA redundant nodes. Rectify the fault for future automatic switchover. After the faulty nodes are rectified, they become the new HA redundant nodes.

  HA redundant nodes can free you from paying attention to node status and reduce O&M costs. However, you need to purchase backup nodes as HA redundant nodes. The resource costs are higher.

  How do I set HA nodes?

  • Enabling HA redundancy

    Select a node without any service as the HA redundant node. On the resource pool details page, click the Nodes tab, locate the target node, and click More > Enable HA Redundancy in the Operation column.

    If you need to enable HA redundancy for nodes in batches, select the target nodes, and click Enable HA Redundancy above the list.

    Figure 1 Enabling HA redundancy
    
    Figure 2 HA redundant node
    
    

    • 5% of the total nodes in a resource pool should be HA redundant nodes. For example, there are 20 nodes in a resource pool, then there should be one HA redundant node.
    • To check whether a node has any service, go to the resource pool details page. In the Nodes tab, check whether all GPUs and Ascend chips are available. If yes, the node has no services.
      Figure 3 Checking whether a node has services
      

  • Disabling HA redundancy

    On the resource pool details page, click the Nodes tab. Locate the target node and click More > Disable HA Redundancy in the Operation column.

    Once disabled, the nodes will be de-isolated and no longer used as standby nodes. Workloads can be scheduled to the nodes properly.

    If you need to disable HA redundancy for nodes in batches, select the target nodes, and click Disable HA Redundancy above the list.

    Figure 4 Disabling HA redundancy
    
    Figure 5 Non-HA redundancy
    

• Resetting a node: Perform this operation to upgrade the OS of the node. If an error occurs during node configuration update, reset the node to rectify the fault.

  In the Nodes tab, locate the node you want to reset. Click Reset in the Operation column to reset a node. You can also select multiple nodes, and click Reset to reset multiple nodes.

  Configure the parameters described in the table below.

  Table 1 Parameters

  Parameter	Description
  Operating system	Select an OS from the drop-down list box.
  Configuration Mode	Select a configuration mode for resetting the node. By node percentage: the maximum ratio of nodes that can be reset if there are multiple nodes in the reset task By node quantity: the maximum number of nodes that can be reset if there are multiple nodes in the reset task

  Check the node reset records on the Records page. If the node is being reset, its status is Resetting. After the reset is complete, the node status changes to Available. Resetting a node will not be charged.

  Figure 6 Resetting a node
  
  

  • Resetting a node will affect running services.
  • Only nodes in the Available state can be reset.
  • A single node can be in only one reset task at a time. Multiple reset tasks cannot be delivered to the same node at a time.
  • If there are any nodes in the Replacing state in the operation records, nodes in the resource pool cannot be reset.
  • When the driver of a resource pool is being upgraded, nodes in this resource pool cannot be reset.
  • For GPU and NPU specifications, after the node is reset, the driver of the node may be upgraded. Wait patiently.
  Figure 7 Viewing resource pool nodes
  
  Figure 8 Operation records
  

• Restarting the node:

  In the Nodes tab, locate the target node, and click More > Reboot in the Operation column. You can also select multiple nodes and click Reboot above the list to reboot multiple nodes.

  When delivering a node restart task, select the corresponding nodes. Restarting a node will affect running services. Exercise caution.

  Check the node operation records in the Records dialog box. If the node is being restarted, its status is Rebooting. After it is restarted, the status changes to Available. Restarting a node will not be charged.

  Figure 9 Restarting a node
  
  Figure 10 Operation records
  
  

  • Restarting a node will affect running services.
  • Only nodes in the Available or Unavailable state can be restarted.
  • A single node can be in only one restart task at a time. Multiple restart tasks cannot be delivered to the same node at a time.
  • If the node is being replaced, reset, or deleted, it cannot be restarted.
  • When the driver of a resource pool is being upgraded, nodes in this resource pool cannot be restarted.
  • After the node is restarted, it will be unavailable for a short time. The service is being started and the health status is being checked. Wait patiently.
• Deleting, unsubscribing from, or releasing a node
  • For a pay-per-use resource pool, click Delete in the Operation column.

    To delete nodes in batches, select the check boxes next to the node names, and click Delete.

  • For a yearly/monthly resource pool whose resources are not expired, click Unsubscribe in the Operation column.
  • For a yearly/monthly resource pool whose resources are expired (in the grace period), click Release in the Operation column.

  If the delete button is available for a yearly/monthly node, click the button to delete the node.

  

  • Before deleting, unsubscribing from, or releasing a node, ensure that there are no running jobs on this node. Otherwise, the jobs will be interrupted.
  • Delete, unsubscribe from, or release abnormal nodes in a resource pool and add new ones for substitution.
  • If there is only one node, it cannot be deleted, unsubscribed from, or released.

Authorizing Technical Support to Locate Faults

During fault locating and performance diagnosis performed by Huawei technical support, you need to authorize certain O&M operations. To do so, go to the resource pool details page, click the Nodes tab, locate the target node, and click More > Authorize in the Operation column. In the displayed dialog box, click OK.

Normally, the Authorize button is unavailable. It will become available after the Huawei technical support applies for O&M.

After the O&M, Huawei technical support will disable the authorization. No further operations are required.