Help Center/ Cloud Container Engine/ User Guide/ Scheduling/ NPU Scheduling/ NPU Fault Handling
Updated on 2026-05-26 GMT+08:00
View PDF
Share

NPU Fault Handling

Fault Locating

The vLLM inference service response times out.

If the vLLM inference service does not respond for more than 3 minutes, the service will enter the suspended state. As a result, the ongoing inference task is interrupted, and tokens cannot be output. This problem is usually caused by residual connection status in the hardware for vNPUs or by service process blocking. You need to log in to the corresponding pod as the root user and perform the following operations:

  1. Terminates the vLLM server process.
  2. Check whether the vllm and python processes related to inference exist. 
    ps aux | grep python
ps aux | grep vllm

    If the processes exist, wait for 3 minutes. The program will automatically clear the processes. Then, run the preceding commands again to check the processes. If no command output is displayed, no vllm or python process related to inference is running.

  3. Manually clear vNPU connection status files.

    The vLLM service depends on the FlexNPU module for hardware acceleration. If the service exits abnormally, status files such as *_connect and *_disconnect may remain. As a result, the subsequent startup fails or the connection is abnormal.

    1. Check the current entity_id. 
      ls /etc/flexnpu/shm/
    2. Go to the corresponding entity_id directory and clear the status files. 
      cd /etc/flexnpu/shm/${entity_id}/
rm -f *_connect
rm -f *_disconnect

      Replace ${entity_id} with the entity_id value obtained in the previous step.

    3. Run the ls command once to ensure that no residual file exists.
  4. Run the vLLM inference service again. The inference task output is restored.

Manually Clearing Residual flexnpu-server Pod and Node Resources

During the use of FlexNPU, if a workload has been created or deleted, some resources may remain due to the current resource clearing mechanism. To prevent resource leakage and ensure environment consistency, you are advised to manually check and clear the following residual resources after completing related operations.

  1. Delete all client pods.
  2. Access the flexnpu-server pod.
    1. Search for pods whose names start with flexnpu-server. 
      kubectl get pods -A
    2. Access the pod. 
      kubectl exec -it ${flexnpu-server} -n kube-system -- bash

      Replace ${flexnpu-server} with the pod name obtained in the previous step.

  3. Manually clear the residual key directories on the node. 
    rm -rf /etc/supervisor/conf.d/*
rm -rf /etc/flexnpu/persistent/*
rm -rf /etc/flexnpu/server/tmp/*
rm -rf /etc/flexnpu/shm/*

    After the clearing is complete, verify that the directories are cleared.

    ls /etc/supervisor/conf.d/
ls /etc/flexnpu/persistent/
ls /etc/flexnpu/server/tmp/
ls /etc/flexnpu/shm/
  4. Restart and update the flexnpud service.

    After the clearing is complete, run the following commands to restart and update the flexnpud service so that the new configuration file can be loaded and the status can be updated:

    supervisorctl -c /config.conf restart flexnpud
supervisorctl -c /config.conf update
Parent Topic: NPU Scheduling