What Do I Do If the Driver Is Unavailable After the Kernel Is Upgraded?

Symptom

• When you run the nvidia-smi command, the error message "failed to initialize NVML: Driver/library version mismatch" is displayed.
• When you run the nvidia-smi command, the error message "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA diver" is displayed.

Checking the Kernel Version

1. Run the following command to check the current kernel version:

   uname -r

2. Run the corresponding command on the server to check the version of the kernel when the driver was installed:
   • CentOS: find /usr/lib/modules -name nvidia.ko
   • Ubuntu: find /lib/modules -name nvidia.ko

   If the current kernel version is different from the kernel version when the driver was installed, the driver is unavailable after the kernel is upgraded.

Solution

1. Run the following commands in sequence to remove the NVIDIA driver:

   rmmod nvidia_drm

   rmmod nvidia_modeset

   rmmod nvidia

2. Run the following command to query the GPU information:

   nvidia-smi

   • If the command output is normal, the problem has been resolved.
   • If the fault persists in the command output, refer to Why Is the GPU Driver Unavailable?