On this page

Show all

Help Center/ Cloud Container Engine/ User Guide/ Scheduling/ GPU Scheduling/ Configuring NVIDIA GPU to Use DCGM-Exporter for GPU Metric Monitoring

Configuring NVIDIA GPU to Use DCGM-Exporter for GPU Metric Monitoring

Updated on 2025-02-18 GMT+08:00

View PDF

It is essential for O&M personnel to monitor large-scale Kubernetes-based GPU devices. By tracking GPU metrics, the O&M personnel can gain valuable insights into the overall utilization, health status, and workload performance of the entire cluster, thereby facilitating swift issue resolution, optimized GPU resource allocation, and enhanced resource efficiency. Moreover, data scientists and AI algorithm engineers can leverage relevant monitoring metrics to learn about the organization's GPU usage patterns, ultimately supporting informed capacity planning and task scheduling decisions.

The latest NVIDIA offerings enable the utilization of the Data Center GPU Manager (DCGM) for managing large-scale GPU clusters. The CCE AI Suite (NVIDIA GPU) add-on, version 2.7.32 or later, is built on NVIDIA DCGM, providing advanced GPU monitoring functionalities. DCGM offers a broad spectrum of GPU monitoring metrics, featuring:

GPU behavior monitoring
GPU configuration management
GPU policy management
GPU health diagnosis
Statistics on GPUs and threads
Configuration and monitoring for NVSwitches

This section leverages CCE Cloud Native Cluster Monitoring and DCGM Exporter to monitor GPUs in comprehensive scenarios. For details about commonly used metrics, see GPU Metrics. For more information about DCGM-Exporter, see DCGM-Exporter.

Prerequisites

A NVIDIA GPU node is running properly in the cluster.

Step 1: Enable DCGM-Exporter

Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Add-ons, locate CCE AI Suite (NVIDIA GPU) on the right, and click Install.
Enable Use DCGM-Exporter to Observe DCGM Metrics. Then, DCGM-Exporter will be deployed as a DaemonSet on GPU nodes.

NOTICE:

DCGM-Exporter is seamlessly integrated into NVIDIA GPU of version 2.1.24, 2.7.40, or later versions. However, it is not supported in earlier versions.

To send GPU monitoring data to AOM, enable Report Monitoring Data to AOM in Cloud Native Cluster Monitoring after enabling DCGM-Exporter. GPU metrics reported to AOM are custom metrics and you will be billed on a pay-per-use basis for them. For details, see Pricing Details.
Configure other parameters for the add-on and click Install. For details about parameter settings, see CCE AI Suite (NVIDIA GPU).

Step 2: Collect DCGM Metrics

NOTICE:

By default, the metrics exposed by DCGM-Exporter are not collected and reported by Prometheus. To use Prometheus or Grafana to view these metrics, enable ServiceMonitor for DCGM-Exporter.

ServiceMonitor for DCGM-Exporter has been preset in Cloud Native Cluster Monitoring (kube-prometheus-stack) of version 3.12.0 or later. You can enable it in Settings.

Log in to the CCE console and click the cluster name to access the cluster console.
In the navigation pane, choose Add-ons and install the Cloud Native Cluster Monitoring. Then, enable Report Monitoring Data to AOM and select the target AOM instance. For details about other settings, see Cloud Native Cluster Monitoring.
In the navigation pane, choose Settings. Then, click the Monitoring tab.
In the Collection configuration area, find ServiceMonitor and click Manage.
Search for ServiceMonitor of DCGM-Exporter and enable it.

Step 3: View DCGM Metrics on AOM

Go to the AOM console and select the target AOM instance in the instance list.
Choose Metric Management in the navigation pane and check DCGM metrics.

Step 4: Use Grafana to View DCGM Metrics

In the navigation pane, choose Add-ons, install Grafana, enable Interconnecting Data Sources with AOM, and select the interconnected AOM instance.
Go to the AOM console and select the target AOM instance in the instance list.
Choose Settings in the navigation pane. In the Grafana Data Source Info area, obtain the AOM instance URL, username, and password. If Grafana Data Source Info is unavailable for the AOM instance, click Add Access Code in the Credential area to generate the Grafana data source configuration.
Access Grafana and add the Prometheus data source to Grafana.
Configure AOM settings, including enabling Basic auth and Skip TLS Verify.
Import NVIDIA DCGM Exporter Dashboard, which is provided by NVIDIA for displaying DCGM metrics. For details about how to import dashboards to Grafana, see Manage dashboards.
View the imported dashboard.

Appendix: Troubleshooting DCGM-Exporter Faults

Check the running status.

On the NVIDIA GPU details page, check whether the target pod is running.
Check pod logs for the HTTP server listening status.
Run the curl command in the cluster to access DCGM-Exporter and check whether data can be obtained.
1. Check DCGM-Exporter's pod IP address.
```
kubectl get po -A -owide | grep dcgm
```
2. Check data. In the following command, 10.1.1.15 is the obtained pod IP address:
```
curl 10.1.1.15:9400/metrics | head
```

Helpful Links

GPU Metrics

Parent Topic: GPU Scheduling

Previous topic: Monitoring GPU Metrics

Next topic: Configuring Workload Scaling Based on GPU Monitoring Metrics

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot