Manually Upgrading the Driver Version of a GPU Node

Prerequisites

The cluster can be accessed using kubectl. For details, see Connecting to a Cluster Using kubectl.

Procedure

To use a specific NVIDIA driver version, perform the following operations to install the GPU driver of the latest version on the node:

1. Take the node offline and manually evict pods on the node to prevent all programs from using GPUs.
   1. Take the node offline.

      kubectl cordon <NODE_NAME>

      After the node is offline, it becomes unschedulable.

      NAME            STATUS                     ROLES    AGE   VERSION
      192.168.1.xx    Ready,SchedulingDisabled   <none>   20m   v1.25.5-r20-25.1.31.1

   2. Manually evict pods on the node.

      kubectl drain 192.168.1.xx --ignore-daemonsets=true --delete-emptydir-data

      Expected result

      node/192.168.1.xx drained

   3. Check whether there are any DaemonSet pods are using GPUs. If so, stop kubelet and the runtime.

      Log in to the node where the GPU driver needs to be upgraded, for example, the node with IP address 192.168.1.xx.

      • Stop containerd.

        systemctl stop kubelet kubelet-monit containerd containerd-monit

      • Stop Docker.

        systemctl stop kubelet kubelet-monit docker docker-monit

   4. Check whether there are any programs are using GPUs.

      Run the sudo fuser -v /dev/nvidia* command.

      If this command is unavailable (for example, in an RPM-based Linux distribution), run the yum install psmisc command to install the Psmisc package.

      # sudo fuser -v /dev/nvidia*
                           USER        PID ACCESS COMMAND
      /dev/nvidia0:        root      12192 F.... nvidia-gpu-devi
      /dev/nvidiactl:      root      12192 F.... nvidia-gpu-devi

      Run the sudo kill 12192 command to delete the process. In this example, process ID 12192 is for reference only.

      # sudo kill 12192
      # sudo fuser -v /dev/nvidia*      # Recheck whether there are any programs are using GPUs.

2. Install the NVIDIA driver of a specified version on the node.
   1. Download the driver of the specified version at the official NVIDIA website. For details about how to select a proper driver version, see Selecting a GPU Driver Version for Nodes.
   2. Record the driver information of the current version. The command for obtaining a driver varies depending on the CCE AI Suite (NVIDIA GPU) add-on version.
      • Version 1.x.x: /opt/cloud/cce/nvidia/bin/nvidia-smi
      • Versions from 2.0.0 to 2.5.3: /usr/local/nvidia/bin/nvidia-smi
      • Versions later than 2.5.4: nvidia-smi

      # nvidia-smi
      Tue Feb 20 15:06:54 2024
      +-----------------------------------------------------------------------------+
      | NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.4     |
      |-------------------------------+----------------------+----------------------+
      | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
      | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
      |                               |                      |               MIG M. |
      |===============================+======================+======================|
      |   0  Tesla V100-SXM2...  Off  | 00000000:21:01.0 Off |                    0 |
      | N/A   31C    P0    23W / 300W |      0MiB / 16160MiB |      0%      Default |
      |                               |                      |                  N/A |
      +-------------------------------+----------------------+----------------------+
      
      +-----------------------------------------------------------------------------+
      | Processes:                                                                  |
      |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
      |        ID   ID                                                   Usage      |
      |=============================================================================|
      |  No running processes found                                                 |
      +-----------------------------------------------------------------------------+

   3. Prepare for the installation.
      Obtain the kernel version of the current node and install its dependencies (such as gcc, make, and kernel-devel). The following is an example using a RPM-based Linux distribution (whose OS is EulerOS, HCE, or CentOS).

      yum install -y perl kernel-devel-$(uname -r) zlib-devel binutils binutils-extra binutils-devel elfutils-libelf-devel gcc make

   4. Install the downloaded driver and verify the installation.
      The NVIDIA-Linux-x86_64-535.54.03.run driver is used as an example.

      # Obtain the kernel version.
      kerVersion=`uname -r`
      
      # Logs for installation details
      touch /root/nvidia-installer.log
      
      # Install the driver downloaded at the official NVIDIA website.
      sh NVIDIA-Linux-x86_64-535.54.03.run --silent --kernel-source-path=/usr/src/kernels/$kerVersion --log-file-name=/root/nvidia-installer.log --no-install-compat32-libs --utility-prefix="/usr/local/nvidia" --opengl-prefix="/usr/local/nvidia"

      

      If the following information is displayed during the installation:

      ERROR: An NVIDIA kernel module 'nvidia' appears to already be loaded in your kernel.  This may be because it is in use (for example, by an X server, a CUDA program, or the NVIDIA Persistence Daemon), but this may also happen if your kernel was configured without support for module unloading.  Please be sure to exit any programs that may be using the GPU(s) before attempting to upgrade your driver.  If no GPU-based programs are running, you know that your kernel supports module unloading, and you still receive this message, then an error may have occurred that has corrupted an NVIDIA kernel module's usage count, for which the simplest remedy is to reboot your computer.
      ERROR: Installation has failed.  Please see the file '/root/nvidia-installer.log' for details.  You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.

      Perform the following operations to handle this issue:

      1. Disable kubelet and the runtime.
         • Disable containerd.

           systemctl disable kubelet kubelet-monit containerd containerd-monit

         • Disable Docker.

           systemctl disable kubelet kubelet-monit docker docker-monit

      2. Restart the affected node.
      3. Run the driver installation command again.
      4. Enable kubelet and the runtime.
         • Enable containerd.

           systemctl enable kubelet kubelet-monit containerd containerd-monit

         • Enable Docker.

           systemctl enable kubelet kubelet-monit docker docker-monit

   5. Run the nvidia-smi command to check whether the new-version GPU driver has been installed.

      # nvidia-smi
      Tue Feb 20 15:13:58 2024
      +---------------------------------------------------------------------------------------+
      | NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
      |-----------------------------------------+----------------------+----------------------+
      | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
      | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
      |                                         |                      |               MIG M. |
      |=========================================+======================+======================|
      |   0  Tesla V100-SXM2-16GB           Off | 00000000:21:01.0 Off |                    0 |
      | N/A   34C    P0              38W / 300W |      0MiB / 16384MiB |      2%      Default |
      |                                         |                      |                  N/A |
      +-----------------------------------------+----------------------+----------------------+
      
      +---------------------------------------------------------------------------------------+
      | Processes:                                                                            |
      |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
      |        ID   ID                                                             Usage      |
      |=======================================================================================|
      |  No running processes found                                                           |
      +---------------------------------------------------------------------------------------+

   6. Delete driver files and historical commands.

      rm NVIDIA-Linux-x86_64-535.54.03.run /root/nvidia-installer.log && history -c

3. Restart the node.
   1. Restart kubelet and the runtime.
      • Restart containerd.

        systemctl start kubelet kubelet-monit containerd containerd-monit

      • Restart Docker.

        systemctl start kubelet kubelet-monit docker docker-monit

   2. Remove scheduling restrictions on the node.

      kubectl uncordon <NODE_NAME>