Lite Server Node Fault Diagnosis

Scenario

Lite Server task center provides one-click fault diagnosis, including parameter plane network diagnosis and Ascend software diagnosis. You can quickly diagnose the network and Ascend software on Lite Server without learning about specific diagnosis commands.

For the parameter plane network diagnosis, you can query the network status, IP address, and mask information of the PU. For the Ascend device diagnosis, you can check the driver firmware version compatibility and implement automatic in-band check. You can batch start diagnosis tasks on multiple servers at the same time, improving efficiency to a large extent.

Constraints

• Only Snt9b nodes and Snt9b23 supernodes are supported.
• You can select a maximum of 50 common nodes or supernode subnodes for the same task.
• The NodeTaskHub plug-in is required for the node where the task is to be created. Ensure that the plug-in is installed before task creation. For details, see Managing Lite Server AI Plug-ins.
• Only one diagnosis task can be executed on a node at the same time. The task cannot be interrupted once started. Plan the task priority.
• Ensure that no services are running on the nodes you are going to diagnose. Running commands during diagnosis can cause service interruptions or errors.
• Install the MCU, driver, and firmware for Ascend HDK 23.0.0 or later before starting the diagnosis. A preconfigured OS is already installed. If you use a custom OS, ensure that the software has been installed correctly.
• The diagnosis requires the Ascend-docker-runtime development kit. This software is pre-installed on the default OS. If you use a custom OS, ensure the software has been installed correctly.

Procedure

1. Log in to the ModelArts console.
2. In the navigation pane on the left, choose Lite Servers under Resource Management. On the displayed page, click the Task Center tab.
   Figure 1 Task center
   

3. Click Create Task in the upper left corner. On the displayed Job Templates page, locate Ascend Fault Diagnosis, and click Create Task.
   Figure 2 Task templates
   

4. On the Ascend Fault Diagnosis page, enter the task name and description. Set server model and type, select a diagnosis item, select the notice, and click Create now.

   Table 1 Parameters for creating a task

   Parameter	Description
   Name	The system automatically generates a task name. You can change the name as required.
   Description	Enter the task description for quick search.
   Server Model	Select a server model and select nodes in the node list. You can search for node information using keywords. Snt9b nodes and Snt9b23 supernodes are supported.
   Diagnosis Item	You can select Parameter Plane Network Diagnosis, Ascend Device Diagnosis, or both. Parameter Plane Network Diagnosis: Check and record parameter-plane network metrics and information. Ascend Device Diagnosis: Check the health and compatibility of Ascend software and chip metrics.

5. View the task execution status in the Task Center tab.

6. Click the task name to access its details page, where you can view the task details.
   Figure 3 Checking the task details
   

7. On the task details page, locate the target node and click View Logs in the Operation column. In the displayed window on the right, view the detailed log about task execution. All check results are displayed in the task logs, and basic log analysis is provided.
   Figure 4 Viewing logs
   

In-band Automatic Check Items

The table below lists the in-band automatic check items in the Ascend device check task.

hccn_tool -i $i -udp  -g

timeout 20s npu-smi info -t health -i "$i" | grep OK -c

timeout 20s npu-smi info -t board -i "$i" | grep Version

lspci | grep d8 / lspci | grep d8 -c

hccn_tool -i $i -link -g

hccn_tool -i $i -net_health -g

hccn_tool -i $i -pfc -g

hccn_tool -i $i -tls -g | grep switch

ascend-dmi  -ci