What Can I Do If a Pod Cannot Be Started After the CCE AI Suite (Ascend NPU) Add-on Is Upgraded from 1.x.x to 2.x.x?

Symptom

After upgrading the CCE AI Suite (Ascend NPU) add-on from 1.x.x to 2.x.x but deploying services using the original configuration, a service pod remains in the pending state and cannot start.

Running the kubectl describe pod command reveals an error message, which indicates that the container creation has failed.

Error response from daemon: Duplicate mount point: /usr/local/bin/npu-smi

Possible Cause

In version 1.x.x, the add-on cannot automatically mount drivers and npu-smi to a service pod. This must be done manually. In version 2.x.x, the add-on can automatically mount drivers and npu-smi to service pods. This leads to a conflict between the npu-smi directories automatically mounted to a service pod and those manually mounted before the upgrade. As a result, the Duplicate mount point: /usr/local/bin/npu-smi error occurs.

CCE AI Suite (Ascend NPU) 2.x.x automatically mounts the following directories to service pods:

/usr/local/Ascend/driver/lib64/common
/usr/local/Ascend/driver/lib64/driver
/usr/local/bin/npu-smi

The way CCE AI Suite (Ascend NPU) mounts drivers and npu-smi to service pods differs depending on its version. The following table shows the details.

Solution

When using a CCE AI Suite (Ascend NPU) of version 2.x.x, remove the fields for mounting drivers and npu-smi directories from the YAML file of a service pod. The add-on will automatically handle the mounting of drivers and npu-smi directories.

To manually mount them to a service pod, change the mount path /usr/local/bin/npu-smi to /usr/local/sbin/npu-smi to resolve the path conflict.