Monitoring GPU Metrics

Prerequisites

• The Cloud Native Cluster Monitoring add-on has been installed in the cluster.
• The CCE AI Suite (NVIDIA GPU) add-on has been installed in the cluster, and the add-on version is 2.0.10 or later.
• To monitor GPU virtualization metrics, ensure Volcano Scheduler has been installed in the cluster and the add-on version is 1.10.5 or later.

Accessing Prometheus

After the Prometheus add-on is installed, you can deploy workloads and Services. The Prometheus server will be deployed as a StatefulSet in the monitoring namespace.

You can create a public network LoadBalancer Service so that Prometheus can be accessed from an external network.

1. Log in to the CCE console and click the name of the cluster with Prometheus installed to access the cluster console. In the navigation pane, choose Services & Ingresses.
2. Click Create from YAML in the upper right corner to create a public network LoadBalancer Service.

   apiVersion: v1
   kind: Service
   metadata:
     name: prom-lb     # Service name, which is customizable.
     namespace: monitoring
     labels:
       app: prometheus
       component: server
     annotations:
       kubernetes.io/elb.id: 038ff***     # Replace it with the ID of the public network load balancer in the VPC that the cluster belongs to.
   spec:
     ports:
       - name: cce-service-0
         protocol: TCP
         port: 88             # Service port, which is customizable.
         targetPort: 9090     # Default Prometheus port. Retain the default value.
     selector:                # The label selector can be adjusted based on the label of a Prometheus server instance.
       app.kubernetes.io/name: prometheus
       prometheus: server
     type: LoadBalancer

3. After the Service is created, visit Public IP address of the load balancer:Service port to access Prometheus.
   Figure 1 Accessing Prometheus
   

4. Choose Status > Targets to view the targets monitored by Prometheus.
   Figure 2 Viewing monitored targets
   

Monitoring GPU Metrics

Create a GPU workload. After the workload runs properly, access Prometheus and view GPU metrics on the Graph page.

For more details, see GPU Metrics.

Figure 3 Viewing GPU metrics

Accessing Grafana

The Prometheus add-on has had Grafana (an open-source visualization tool) installed and interconnected. You can create a public network LoadBalancer Service so that you can access Grafana from the public network and view Prometheus monitoring data on Grafana.

Click the access address to access Grafana and select a proper dashboard to view the aggregated content.

1. Log in to the CCE console and click the name of the cluster with Prometheus installed to access the cluster console. In the navigation pane, choose Services & Ingresses.
2. Click Create from YAML in the upper right corner to create a public network LoadBalancer Service for Grafana.

   apiVersion: v1
   kind: Service
   metadata:
     name: grafana-lb     # Service name, which is customizable
     namespace: monitoring
     labels:
       app: grafana
     annotations:
       kubernetes.io/elb.id: 038ff***     # Replace it with the ID of the public network load balancer in the VPC to which the cluster belongs.
   spec:
     ports:
       - name: cce-service-0
         protocol: TCP
         port: 80     # Service port, which is customizable
         targetPort: 3000     # Default Grafana port. Retain the default value.
     selector:
       app: grafana
     type: LoadBalancer

3. After the Service is created, visit Public IP address of the load balancer:Service port to access Grafana and select a proper dashboard to view xGPU resources.
   Figure 4 Viewing xGPU resources