Monitoring Center FAQ

Indexes

• Why Is There No Data on Monitoring Center?
• How Do I Disable Monitoring Center?
• Why Are Custom Metrics Not Displayed on Monitoring Center?
• Why Is the Resource Information Not Displayed in the Node List for a Short Time (1 to 2 Minutes) After the prometheus-server Instance Is Restarted When the Cloud Native Cluster Monitoring Add-on Is Deployed with Local Data Storage Enabled?
• Why Is Some Data Doubled After the kube-state-metrics Instance Is Restarted When the Cloud Native Cluster Monitoring Add-on Is Deployed with Local Data Storage Enabled?
• Why Does the Cloud Native Cluster Monitoring Add-on with Local Data Storage Enabled Fail to Report Metrics?
• Why Does the Workload/Node CPU Usage of Monitoring Center Exceed 100%?
• Why Is 403 Displayed in Collection Endpoint Access? How Do I Handle It?

Why Is There No Data on Monitoring Center?

• Possible cause 1: The Cloud Native Cluster Monitoring add-on is abnormal.

  Access the Add-ons page on the cluster console and check whether the Cloud Native Cluster Monitoring add-on is in the Running state.

  Figure 1 Checking the add-on status
  

  If the add-on is not running normally, locate the fault based on the events.

  Figure 2 Viewing add-on events
  
• Possible cause 2: The AOM instance interconnected with the Cloud Native Cluster Monitoring add-on is deleted.

  Access the Add-ons page on the cluster console and check the configuration of the Cloud Native Cluster Monitoring add-on.

  Figure 3 Editing add-on configuration
  

  Ensure that AOM Instance is not left empty.

  Figure 4 Viewing the AOM instance
  

How Do I Disable Monitoring Center?

To disable cluster monitoring, uninstall the Cloud Native Cluster Monitoring add-on on the Add-ons page or disable the option for interconnecting with AOM.

Why Are Custom Metrics Not Displayed on Monitoring Center?

Monitoring Center currently does not display custom metrics. To view custom metrics, you can create a dashboard for custom metrics on the dashboard of AOM.

Why Is the Resource Information Not Displayed in the Node List for a Short Time (1 to 2 Minutes) After the prometheus-server Instance Is Restarted When the Cloud Native Cluster Monitoring Add-on Is Deployed with Local Data Storage Enabled?

After a prometheus-server instance is restarted, the UID tag values of its metrics change. During the rolling restart of the prometheus-server instance, metrics overlap because data is stored locally. This means the Cloud Native Cluster Monitoring add-on reports metrics from both the old and new prometheus-server instances to AOM, resulting in inaccurate resource information in the node list. When the metrics overlap, the resource information in the node list is not displayed. Unless otherwise specified, you are advised to connect the Cloud Native Cluster Monitoring add-on to AOM with local data storage disabled.

Why Is Some Data Doubled After the kube-state-metrics Instance Is Restarted When the Cloud Native Cluster Monitoring Add-on Is Deployed with Local Data Storage Enabled?

When the kube-state-metrics instance is scheduled to a new node, the instance tag values of the metrics collected by the kube-state-metrics instance change. During the rolling restart of the kube-state-metrics instance, metrics overlap because data is stored locally. This means the Cloud Native Cluster Monitoring add-on reports metrics from both the old and new kube-state-metrics instances. In addition, the instance label values are inconsistent, so all metrics are considered valid. As a result, the number of nodes, the number of workloads, the number of pods, the number of namespaces, and the number of control plane components displayed on the Clusters tab (Monitoring Center > Clusters) are all doubled. Unless otherwise specified, you are advised to connect the Cloud Native Cluster Monitoring add-on to AOM with local data storage disabled.

Why Does the Cloud Native Cluster Monitoring Add-on with Local Data Storage Enabled Fail to Report Metrics?

The add-on pod with local data storage enabled has run out of storage space on the PV. As a result, metrics cannot be written.

Go to the Add-ons page, select the prometheus-server-x instance, and view its logs. If the log contains information similar to "no space left on device", the space of the disk mounted to this add-on pod is insufficient.

Figure 5 Viewing the add-on pod logs

Solutions

• Solution 1: You are advised to connect the add-on with local data storage disabled to the AOM instance. If AOM is used to manage metrics, storage management is not required.
• Solution 2: In the navigation pane, choose Storage. On the displayed page, switch to the monitoring namespace, select the pvc-prometheus-server-0 disk, and click More > Scale-out in the Operation column. After the scale-out is complete, go to the StatefulSets tab and restart the prometheus-server-0 instance.
  Figure 6 Expanding the PVC capacity
  
  

  Insufficient disk space will prevent Prometheus metrics from being written. As a result, data cannot be collected. This means that any monitoring data generated during the scale-out and subsequent restart will be lost.

Why Does the Workload/Node CPU Usage of Monitoring Center Exceed 100%?

The workload CPU usage is calculated using container_cpu_usage_seconds_total. The system periodically updates the used CPU and the time point at which the used CPU is collected. By default, Prometheus collects metrics at the collection time point instead of the time point specified by container_cpu_usage_seconds_total. As a result, the time point at which the used CPU is collected is inaccurate, and there is a short latency.

Assume that the system updates the used CPU every 6 seconds, and the collection period is 15 seconds, Prometheus collected data at 18:30:14 for the first time and at 18:30:29 for the second time. However, the time point specified by container_cpu_usage_seconds_total is 18:30:10 for the first and 18:30:28 for the second time.

• Actual used CPU per second: (150000-100000)/(18:30:16-18:30:10) = 8333.33
• Used CPU per second collected by Prometheus: (250000-100000)/(18:30:29-18:30:14) = 10000

The preceding data values are manually amplified and are only examples. The actual difference is small.

Solution

You can configure honorTimestamps to use the time point specified by container_cpu_usage_seconds_total to avoid this problem. Weigh the pros and cons before deciding whether to configure honorTimestamps.

To configure honorTimestamps, take the following steps:

Cloud Native Cluster Monitoring 3.11.0 or later has been installed in the cluster, and the preset collection function has been enabled.

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose ConfigMaps and Secrets, switch to the monitoring namespace, and locate the persistent-user-config configuration item.
3. Click Edit YAML, search for kubelet-cadvisor, and add honorTimestamps: true.

   ...
       - customBlacklist: []
         customWhitelist: []
         destNamespace: kube-system
         name: kubelet-cadvisor
         namespace: monitoring
         scrapeAllMetrics: false
         honorTimestamps: true
         scrapeInterval: ""
         status: "on"
         type: ServiceMonitor
   ...

4. Click OK to save the configuration. The configuration takes effect in about 1 minute.

Why Is 403 Displayed in Collection Endpoint Access? How Do I Handle It?

Root Cause

Authentication has been configured for collection tasks in the ServiceMonitor or PodMonitor format corresponding to your collection endpoint. For security purposes, the endpoint to be authenticated cannot be accessed by default.

Solution: You can configure to allow access to endpoints with authentication.

If you allow access to endpoints with authentication, the endpoints to be authenticated can be directly accessed by accessing the prometheus-lightweight service in the cluster. For this reason, do not expose the prometheus-lightweight service port outside the cluster.

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose ConfigMaps and Secrets. Then, set Namespace to All namespaces and locate the persistent-user-config configuration item.
3. Click Update to edit lightweight-user-config.yaml and add - --target-response-auto-auth=true under operatorConfigOverride.

   customSettings:
     operatorEnvOverride: []
     operatorConfigOverride:
     - --target-response-auto-auth=true
     promAdapterConfigOverride: []

4. Click OK to save the configuration. The configuration takes effect in about 1 minute.