Cloud Native Cluster Monitoring

Introduction

kube-prometheus-stack uses Prometheus-operator and Prometheus to provide easy-to-use, end-to-end Kubernetes cluster monitoring.

This add-on allows the monitoring data to be interconnected with Container Intelligent Analysis (CIA) so that you can view monitoring data and configure alarms on the CIA console.

Open source community: https://github.com/prometheus/prometheus

Notes and Constraints

• By default, the kube-state-metrics component of the add-on does not collect labels and annotations of Kubernetes resources. To collect these labels and annotations, manually enable the collection function in the startup parameters and check whether the corresponding metrics are added to the collection trustlist of ServiceMonitor named kube-state-metrics. For details, see Collecting All Labels and Annotations of a Pod.
• In 3.8.0 and later versions, component metrics in the kube-system and monitoring namespaces are not collected by default. If you have workloads in the two namespaces, use Pod Monitor or Service Monitor to collect these metrics.
• In 3.8.0 and later versions, etcd-server, kube-controller, kube-scheduler, autoscaler, fluent-bit, volcano-agent, volcano-scheduler and otel-collector metrics are not collected by default. Enable the collection as required.

  To enable this function, on the ConfigMaps and Secrets page, expand the dropdown list of Namespace, and select monitoring. Locate the row that contains the configuration item named persistent-user-config, and click Edit YAML in the operation column. Remove the serviceMonitorDisable or podMonitorDisable configuration in the customSettings field as required or set the configuration to an empty array.

  ...
     customSettings:
        podMonitorDisable: []
        serviceMonitorDisable: []

• In versions later than 3.9.0, Grafana are removed from the kube-prometheus-stack add-on and used an independent Grafana add-on. Therefore, before rolling kube-prometheus-stack from a version later than 3.9.0 to a version earlier than 3.9.0, uninstall Grafana from the add-on.

Permissions

The node-exporter component of the kube-prometheus-stack add-on needs to read the Docker info data from the /var/run/docker.sock directory on the host for monitoring the Docker disk space.

The following permission is required for running node-exporter:

• cap_dac_override: reads the Docker info data.

Installing the Add-on

The Cloud Native Cluster Monitoring add-on automatically selects a deployment mode based on Data Storage Configuration. This is supported by Cloud Native Cluster Monitoring 3.7.1 or later.

• Original agent mode: Disable Local Data Storage and enable at least one of Report Monitoring Data to AOM and Report Monitoring Data to a Third-Party Platform.

• Original server mode: Enable Local Data Storage and Report Monitoring Data to AOM or Report Monitoring Data to a Third-Party Platform.

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Add-ons, locate Cloud Native Cluster Monitoring on the right, and click Install.
2. On the Install Add-on page, enable at least one item in the Data Storage Configuration area.
   • Reporting Monitoring Data to AOM: Report Prometheus data to AOM. After this function is enabled, you can select the corresponding AOM instance. The collected basic metrics are free of charge. Custom metrics are charged by AOM. For details, see Product Pricing Details. To interconnect with AOM, you must have certain permissions. Only Huawei Cloud accounts, HUAWEI IDs, and IAM users in the admin user group can perform this operation.
   • Reporting Monitoring Data to a Third-Party Monitoring Platform: To report Prometheus data to a third-party monitoring system, you need to enter the address and token of the third-party monitoring system and determine whether to skip certificate authentication.
   • Local data storage: Select the type and size of a disk for storing monitoring data to store Prometheus data in PVCs in the cluster. Storage volumes are not deleted along with the add-on. If local data storage is enabled, all components will be deployed. For details, see Components.
     

     An available PVC named pvc-prometheus-server exists in namespace monitoring and will be used as the storage source.

3. Configure the add-on specifications as needed.
   • Add-on Specifications:
     • If you selected Preset, the system will configure the number of pods and resource quotas for the add-on based on the preset specifications. You can see the configurations on the console.
     • If you selected Custom, you can adjust the number of pods and resource quotas as needed. High availability is not possible with a single pod. If an error occurs on the node where the add-on instance runs, the add-on will fail.
   • Prometheus HA: The Prometheus-server, Prometheus-operator, thanos-query, custom-metrics-apiserver, alertmanager, and kube-state-metrics components are deployed in multi-instance mode in the cluster.
   • Number of collected shards (available after Local data storage is disabled): When there is a lot of Prometheus data, you can configure this parameter to spread the data across a specific number of Prometheus instances. This will help with storage and querying. Increasing the number of shards reduces the data volume carried by each shard. This can increase the upper limit of the metric collection throughput and cause more resources to be consumed. You are advised to increase shards when the cluster scale is large to improve the collection performance. In addition, you need to consider the impact of resource usage and optimize it based on specific scenarios.
   • Install Grafana: Use Grafana to visualize monitoring data. Grafana creates a 5 GiB storage volume by default. Uninstalling the add-on will not delete this volume. The default username and password for the first login are admin. You will be asked to change the password immediately after login.

     In versions later than 3.9.0, Grafana is removed from Cloud Native Cluster Monitoring and has become an independent Grafana add-on. So this option is no longer displayed.

4. Configure the add-on parameters.
   • Custom Metric Collection: Application metrics are automatically collected in the form of service discovery. After this function is enabled, you need to add related configurations to the target application. For details, see Monitoring Custom Metrics Using Cloud Native Cluster Monitoring.
   • Collection Interval: Configure the collection interval.
   • Data Retention (available only after Local data storage is enabled): Configure how long the monitoring data can be kept.
   • Listening port of node-exporter: This port uses the host network to listen to and expose metrics on the node for Prometheus collection. The default port number is 9100, but it can be modified if there is a conflict with an existing port.

5. Click Install.

   After the add-on is installed, you may need to perform the following operations:

   • To use custom metrics to create an auto scaling policy, ensure that local data storage is enabled for the add-on and then perform the following steps:
     1. Collect custom metrics reported by applications to Prometheus. For details, see Monitoring Custom Metrics Using Cloud Native Cluster Monitoring.
     2. Aggregate these custom metrics collected by Prometheus to the API server for the HPA policy to use. For details, see Creating an HPA Policy Using Custom Metrics.
   • To use this add-on to provide system resource metrics (such as CPU and memory usage) for workload auto scaling, enable the Metric API. For details, see Providing Resource Metrics Through the Metrics API. After the configuration, you can use Prometheus to collect system resource metrics. (This configuration is not recommended).

Components

All Kubernetes resources created during Cloud Native Cluster Monitoring add-on installation are created in the namespace named monitoring.

Providing Resource Metrics Through the Metrics API

Resource metrics of containers and nodes, such as CPU and memory usage, can be obtained through the Kubernetes Metrics API. Resource metrics can be directly accessed, for example, by using the kubectl top command, or used by HPA or CustomedHPA policies for auto scaling.

The add-on can provide the Kubernetes Metrics API that is disabled by default. To enable the API, create the following APIService object:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    app: custom-metrics-apiserver
    release: cceaddon-prometheus
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: custom-metrics-apiserver
    namespace: monitoring
    port: 443
  version: v1beta1
  versionPriority: 100

You can save the object as a file, name it metrics-apiservice.yaml, and run the following command:

kubectl create -f metrics-apiservice.yaml

Run the kubectl top pod -n monitoring command. If the following information is displayed, the Metrics API can be accessed:

# kubectl top pod -n monitoring
NAME                                                      CPU(cores)   MEMORY(bytes)
......
custom-metrics-apiserver-d4f556ff9-l2j2m                  38m          44Mi
......

To uninstall the add-on, run the following kubectl command and delete the APIService object. Otherwise, the metrics-server add-on cannot be installed due to residual APIService resources.

kubectl delete APIService v1beta1.metrics.k8s.io

Creating an HPA Policy Using Custom Metrics

HPA policies can only be used when Cloud Native Cluster Monitoring is deployed with local data storage enabled. You can configure custom metrics required by HPA policies in the user-adapter-config ConfigMap.

To use Prometheus to monitor custom metrics, the application needs to provide a metric monitoring API. For details, see Prometheus Monitoring Data Collection.

In this section, the nginx metric (nginx_connections_accepted) in Monitoring Custom Metrics Using Cloud Native Cluster Monitoring is used as an example.

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose ConfigMaps and Secrets.
2. Click the ConfigMaps tab, select the monitoring namespace, locate the row containing user-adapter-config (or adapter-config), and click Update.
   Figure 1 Updating a ConfigMap
   

3. In Data, click Edit for the config.yaml file to add a custom metric collection rule under the rules field. Click OK.

   You can add multiple collection rules by adding multiple configurations under the rules field. For details, see Metrics Discovery and Presentation Configuration.

   Example custom metric rule:

   rules:
   # Match the metric whose name is nginx_connections_accepted. The metric name must be confirmed. Otherwise, the HPA controller cannot get the metric.
   - seriesQuery: '{__name__=~"nginx_connections_accepted",container!="POD",namespace!="",pod!=""}'
     resources:
       # Specify pod and namespace resources.
       overrides:
         namespace:
           resource: namespace
         pod:
           resource: pod
     name:
       #Use nginx_connections_accepted"
       matches: "nginx_connections_accepted"
       #Use nginx_connections_accepted_per_second to represent the metric. The name is the custom metric name in a custom HPA policy.
       as: "nginx_connections_accepted_per_second"
       # Calculate rate(nginx_connections_accepted[2m]) to specify the number of requests received per second.
     metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[2m])'

   Figure 2 Modifying ConfigMap data
   

4. Redeploy the custom-metrics-apiserver workload in the monitoring namespace.
   Figure 3 Redeploying custom-metrics-apiserver
   

5. In the navigation pane, choose Workloads. Locate the workload for which you want to create an HPA policy and choose More > Auto Scaling. In the Custom Policy area, you can select the preceding parameters to create an auto scaling policy.
   Figure 4 Creating an HPA policy
   

Collecting All Labels and Annotations of a Pod

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Workloads.
2. Switch to the monitoring namespace, click the Deployments tab, and click the name of the kube-state-metrics workload. On the page displayed, click the Containers tab and click Edit on the right.
3. In the Lifecycle area of the container settings, edit the startup command.
   Figure 5 Editing the startup command
   
   To collect labels, add the following information to the end of the original kube-state-metrics startup parameter:

   --metric-labels-allowlist=pods=[*],nodes=[node,failure-domain.beta.kubernetes.io/zone,topology.kubernetes.io/zone]

   To collect annotations, add parameters in the startup parameters in the same way.

   --metric-annotations-allowlist=pods=[*],nodes=[node,failure-domain.beta.kubernetes.io/zone,topology.kubernetes.io/zone]

   

   When editing the startup command, do not modify other original startup parameters. Otherwise, the component may be abnormal.

4. kube-state-metrics starts to collect the labels/annotations of pods and nodes and checks whether kube_pod_labels/kube_pod_annotations is in the collection task of CloudScope.

   kubectl get servicemonitor kube-state-metrics -nmonitoring -oyaml | grep kube_pod_labels

For more kube-state-metrics startup parameters, see kube-state-metrics/cli-arguments.

Change History