Monitoring Center FAQ

Indexes

Why Is There No Data on Monitoring Center?
How Do I Disable Monitoring Center?
Why Are Custom Metrics Not Displayed on Monitoring Center?
Why Is the Resource Information Not Displayed in the Node List for a Short Time (1 to 2 Minutes) After the prometheus-server Instance Is Restarted When the Cloud Native Cluster Monitoring Add-on Is Deployed ?
Why Is Some Data Doubled After the kube-state-metrics Instance Is Restarted When the Cloud Native Cluster Monitoring Add-on Is Deployed ?
Why Does the Cloud Native Cluster Monitoring Add-on Fail to Report Metrics?
Why Does the Workload/Node CPU Usage of Monitoring Center Exceed 100%?

Why Is There No Data on Monitoring Center?

Possible cause 1: The Cloud Native Cluster Monitoring add-on is abnormal.
Access the Add-ons page on the cluster console and check whether the Cloud Native Cluster Monitoring add-on is in the Running state.

Figure 1 Checking the add-on status

If the add-on is not running normally, locate the fault based on the events.

Figure 2 Viewing add-on events
Possible cause 2: The AOM instance interconnected with the Cloud Native Cluster Monitoring add-on is deleted.
Access the Add-ons page on the cluster console and check the configuration of the Cloud Native Cluster Monitoring add-on.

Figure 3 Editing add-on configuration

Ensure that AOM Instance is not left empty.

How Do I Disable Monitoring Center?

To disable cluster monitoring, uninstall the Cloud Native Cluster Monitoring add-on on the Add-ons page or disable the option for interconnecting with AOM.

Why Are Custom Metrics Not Displayed on Monitoring Center?

Monitoring Center currently does not display custom metrics. To view custom metrics, you can create a dashboard for custom metrics on the dashboard of AOM.

Why Is the Resource Information Not Displayed in the Node List for a Short Time (1 to 2 Minutes) After the prometheus-server Instance Is Restarted When the Cloud Native Cluster Monitoring Add-on Is Deployed ?

After a prometheus-server instance is restarted, the UID tag values of its metrics change. During the rolling restart of the prometheus-server instance, metrics overlap because data is stored locally. This means the Cloud Native Cluster Monitoring add-on reports metrics from both the old and new prometheus-server instances, resulting in inaccurate resource information in the node list. When the metrics overlap, the resource information in the node list is not displayed. Unless otherwise specified, you are advised to connect the Cloud Native Cluster Monitoring add-on to AOM .

Why Is Some Data Doubled After the kube-state-metrics Instance Is Restarted When the Cloud Native Cluster Monitoring Add-on Is Deployed ?

When the kube-state-metrics instance is scheduled to a new node, the instance tag values of the metrics collected by the kube-state-metrics instance change. During the rolling restart of the kube-state-metrics instance, metrics overlap because data is stored locally. This means the Cloud Native Cluster Monitoring add-on reports metrics from both the old and new kube-state-metrics instances. In addition, the instance label values are inconsistent, so all metrics are considered valid. As a result, the number of nodes, the number of workloads, the number of pods, the number of namespaces, and the number of control plane components displayed on the Clusters tab (Monitoring Center > Clusters) are all doubled. Unless otherwise specified, you are advised to connect the Cloud Native Cluster Monitoring add-on to AOM .

Why Does the Cloud Native Cluster Monitoring Add-on Fail to Report Metrics?

The add-on pod has run out of storage space on the PV. As a result, metrics cannot be written.

Go to the Add-ons page, select the prometheus-server-x instance, and view its logs. If the log contains information similar to "no space left on device", the space of the disk mounted to this add-on pod is insufficient.

Figure 4 Viewing the add-on pod logs
Click to enlarge

Solutions

Solution 1: You are advised to connect the add-on to the AOM instance. If AOM is used to manage metrics, storage management is not required.
Solution 2: In the navigation pane, choose Storage. On the displayed page, switch to the monitoring namespace, select the pvc-prometheus-server-0 disk, and click More > Scale-out in the Operation column. After the scale-out is complete, go to the StatefulSets tab and restart the prometheus-server-0 instance.
Figure 5 Expanding the PVC capacity

Insufficient disk space will prevent Prometheus metrics from being written. As a result, data cannot be collected. This means that any monitoring data generated during the scale-out and subsequent restart will be lost.

Why Does the Workload/Node CPU Usage of Monitoring Center Exceed 100%?

The workload CPU usage is calculated using container_cpu_usage_seconds_total. The system periodically updates the used CPU and the time point at which the used CPU is collected. By default, Prometheus collects metrics at the collection time point instead of the time point specified by container_cpu_usage_seconds_total. As a result, the time point at which the used CPU is collected is inaccurate, and there is a short latency.

Assume that the system updates the used CPU every 6 seconds, and the collection period is 15 seconds, Prometheus collected data at 18:30:14 for the first time and at 18:30:29 for the second time. However, the time point specified by container_cpu_usage_seconds_total is 18:30:10 for the first and 18:30:28 for the second time.

Used CPU	Time Point
100,000	18:30:10
150,000	18:30:16
200,000	18:30:22
250,000	18:30:28
300,000	18:30:34

Actual used CPU per second: (150000-100000)/(18:30:16-18:30:10) = 8333.33
Used CPU per second collected by Prometheus: (250000-100000)/(18:30:29-18:30:14) = 10000

The preceding data values are manually amplified and are only examples. The actual difference is small.

Solution

You can configure honorTimestamps to use the time point specified by container_cpu_usage_seconds_total to avoid this problem. Weigh the pros and cons before deciding whether to configure honorTimestamps.

Configure honorTimestamps	Pros	Cons
No (default behavior of Prometheus)	The metric compression ratio is higher, the total number of stored metrics decreases, and the query performance is excellent. The collection intervals of different metrics are the same, which is driven by the Prometheus collection interval. Generally, breakpoints do not occur. There is a low probability that deviations occur when multiple metrics are combined for PromQL calculation.	Metrics such as the CPU usage may be slightly distorted.
Yes	The time points at which metrics are collected are consistent with the actual time points. In scenarios such as pressure tests, the calculated results are more authentic.	The total number of stored metrics increases, the metric compression ratio decreases, and the query performance deteriorates. Not all metrics carry correct timestamps. As a result, a large deviation may occur when PromQL calculation is performed on multiple metrics. In extreme cases, breakpoints may occur on metrics.

To configure honorTimestamps, take the following steps:

Cloud Native Cluster Monitoring 3.11.0 or later has been installed in the cluster, and the preset collection function has been enabled.

Log in to the CCE console and click the cluster name to access the cluster console.
In the navigation pane, choose ConfigMaps and Secrets, switch to the monitoring namespace, and locate the persistent-user-config configuration item.

Click Edit YAML, search for kubelet-cadvisor, and add honorTimestamps: true.

...
    - customBlacklist: []
      customWhitelist: []
      destNamespace: kube-system
      name: kubelet-cadvisor
      namespace: monitoring
      scrapeAllMetrics: false
      honorTimestamps: true
      scrapeInterval: ""
      status: "on"
      type: ServiceMonitor
...