Installing and Using xGPU

Constraints

• xGPU is supported only on NVIDIA Tesla T4 and V100.
• The HCE kernel version is 5.10 or later.
• NVIDIA driver 535.54.03 has been installed on GPU-accelerated ECSs.
• Docker 18.09.0-300 or later has been installed on GPU-accelerated ECSs.
• Limited by GPU virtualization, when applications in a container are initialized, the GPU compute monitored by NVIDIA System Management Interface (nvidia-smi) may exceed the upper limit of the GPU compute available for the container.
• When the first CUDA application is created, a percentage of the GPU memory (about 3 MiB on NVIDIA Tesla T4) is requested from each GPU. This GPU memory is considered as the management overhead and is not controlled by the xGPU service.
• xGPU cannot be used on bare metal servers and VMs configured with a passthrough NIC at the same time.
• GPU memory isolation provided by xGPU cannot be used for GPU memory applications through unified virtual memory (UVM) by calling CUDA API cudaMallocManaged(). For more information, see NVIDIA official document. Use other methods to request GPU memory. For example, call cudaMalloc().
  

  xGPU allows users to apply for GPU memory by dynamically disabling the UVM. For details about how to disable the UVM, see the description of the uvm_disable interface.

Installing xGPU

To install xGPU, contact customer service.

You are advised to use xGPU through CCE. For details, see GPU Virtualization.

Using xGPU

The following table describes the environment variables of xGPU. When creating a container, you can specify the environment variables to configure the GPU compute and memory that a container engine can obtain using xGPU.

The following example describes how you can use xGPU to allow two NVIDIA-Docker containers to share one GPU.

Example:

docker run --rm -it --runtime=nvidia -e GPU_CONTAINER_QUOTA_PERCENT=50 -e GPU_CONTAINER_MEM=5120 -e GPU_IDX=0 -e GPU_POLICY=1 -e GPU_CONTAINER_PRIORITY=1 --shm-size 16g -v /mnt/:/mnt nvcr.io/nvidia/tensorrt:19.07-py3 bash 
docker run --rm -it --runtime=nvidia -e GPU_CONTAINER_QUOTA_PERCENT=30 -e GPU_CONTAINER_MEM=1024 -e GPU_IDX=0 -e GPU_POLICY=1 -e GPU_CONTAINER_PRIORITY=0 --shm-size 16g -v /mnt/:/mnt nvcr.io/nvidia/tensorrt:19.07-py3 bash

Viewing procfs Directory

At runtime, xGPU generates and manages multiple proc file systems (procfs) in the /proc/xgpu directory. The following are operations for you to view xGPU information and configure xGPU settings.

1. Run the following command to view the node information:

   ls /proc/xgpu/
   0 container version uvm_disable

   The following table describes the information about the procfs nodes.

   Directory	Read/Write Type	Description
   0	Read and write	xGPU generates a directory for each GPU on a GPU-accelerated ECS. Each directory uses a number as its name, for example, 0, 1, and 2. In this example, there is only one GPU, and the directory ID is 0.
   container	Read and write	xGPU generates a directory for each container running in the GPU-accelerated ECSs.
   version	Read-only	xGPU version.
   uvm_disable	Read and write	Controls whether to disable the option that allows all containers to request GPU memory through UVM. The default value is 0. 0: This option is enabled. 1: This option is disabled.

2. Run the following command to view the items in the directory of the first GPU:

   ls /proc/xgpu/0/
   max_inst meminfo policy quota utilization_line utilization_rate xgpu1 xgpu2

   The following table describes the information about the GPU.

   Directory	Read/Write Type	Description
   max_inst	Read and write	Specifies the maximum number of containers. The value ranges from 1 to 25. This parameter can be modified only when no container is running.
   meminfo	Read-only	Specifies the total available GPU memory.
   policy	Read and write	Specifies the GPU compute isolation policy used by the GPU. The default value is 1. 0: native scheduling. The GPU compute is not isolated. 1: fixed scheduling 2: average scheduling 3: preemptive scheduling 4: weight-based preemptive scheduling 5: hybrid scheduling 6: weighted scheduling For details, see GPU Compute Scheduling Examples.
   quota	Read-only	Specifies the total weight of the GPU compute.
   utilization_line	Read and write	Specifies the GPU usage threshold for online containers to suppress offline containers. If the GPU usage exceeds the value of this parameter, online containers completely suppress offline containers. If the GPU usage does not exceed the value of this parameter, online containers partially suppress offline containers.
   utilization_rate	Read only	Specifies the GPU usage.
   xgpuIndex	Read and write	Specifies the xgpu subdirectory of the GPU. In this example, xgpu1 and xgpu2 belong to the first GPU.

3. Run the following command to view the directory of a container:

   ls /proc/xgpu/container/
   9418 9582

   The following table describes the content in the directory.

   Directory	Read/Write Type	Description
   containerID	Read and write	Specifies the container ID. xGPU generates an ID for each container during container creation.

4. Run the following commands to view the directory of each container:

   ls /proc/xgpu/container/9418/
   xgpu1 uvm_disable
   ls /proc/xgpu/container/9582/
   xgpu2 uvm_disable

   The following table describes the content in the directory.

   Directory	Read/Write Type	Description
   xgpuIndex	Read and write	Specifies the xgpu subdirectory of the container. In this example, xgpu1 is the subdirectory of container 9418, and xgpu2 is the subdirectory of container 9582.
   uvm_disable	Read and write	Controls whether to disable the option that only allows a container to request GPU memory through UVM. The default value is 0. 0: This option is enabled. 1: This option is disabled.

5. Run the following command to view the xgpuIndex directory:

   ls /proc/xgpu/container/9418/xgpu1/
   meminfo priority quota

   The following table describes the content in the directory.

   Directory	Read/Write Type	Description
   meminfo	Read-only	Specifies the GPU memory allocated using xGPU and the remaining GPU memory. For example, 3308MiB/5120MiB, 64% free indicates that 5,120 MiB of memory is allocated and 64% is available.
   priority	Read and write	Specifies the priority of a container. The default value is 0. 0: low priority 1: high priority This parameter is used in hybrid scenarios where there are both online and offline containers. High-priority containers can preempt the GPU compute of low-priority containers.
   quota	Read-only	Specifies the percentage of the GPU compute allocated using xGPU. For example, 50 indicates that 50% of the GPU compute is allocated using xGPU.

Upgrading xGPU

You can perform cold upgrades on xGPU.

1. Run the following command to stop all running containers:

   docker ps -q | xargs -I {} docker stop {}

2. Run the following command to upgrade the RPM package of xGPU:

   rpm -U hce_xgpu

Uninstalling xGPU

1. Run the following command to stop all running containers:

   docker ps -q | xargs -I {} docker stop {}

2. Run the following command to uninstall the RPM package:

   rpm -e hce_xgpu

Monitoring Containers Using xgpu-smi

You can use xgpu-smi to view xGPU information, including the container ID, GPU compute usage and allocation, and GPU memory usage and allocation.

The following shows the xgpu-smi monitoring information.

You can run the xgpu-smi -h command to view the help information about the xgpu-smi tool.