Cloud Native Cluster Monitoring

Introduction

The Cluster Native Cluster Monitoring add-on (kube-prometheus-stack) uses Prometheus-operator and Prometheus to provide easy-to-use, end-to-end Kubernetes cluster monitoring.

This add-on allows the monitoring data to be interconnected with Monitoring Center so that you can view monitoring data and configure alarms in Monitoring Center.

Open-source community: https://github.com/prometheus/prometheus

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane on the left, choose Add-ons. Locate Cloud Native Cluster Monitoring on the right and click Install.
2. In the Data Storage Configuration area, enable Report Monitoring Data to AOM and select an AOM instance.

   Reporting Monitoring Data to AOM: If this option is enabled, Prometheus data will be reported to AOM, and you need to select an AOM instance. Basic metrics are free, but custom metrics are billed based on the standard pricing of AOM. For details, see AOM Pricing Details. To interconnect with AOM, you must have certain permissions. Only Huawei Cloud accounts, HUAWEI IDs, and IAM users in the admin user group can perform this operation.

3. Configure the add-on specifications as needed.
   • Prometheus HA: The prometheus-lightweight, prometheus-operator, custom-metrics-apiserver, and kube-state-metrics components are deployed in multi-instance mode in the cluster.
   • Configuration List: You can configure vCPU and memory quotas for each component as needed.

4. Configure the add-on parameters as needed.
   • User-defined indicator HPA: Application metrics are automatically collected in the form of service discovery. If this option is enabled, you need to add related configurations to the target application. For details, see Creating an HPA Policy Using Custom Metrics.
   • Collection Interval: The value ranges from 10 to 60.

5. Click Install.

   After the add-on is installed, you may need to perform the following operations:

   If you want to use custom metrics to create an HPA policy, you need to aggregate the custom metrics collected by Prometheus to the API server. For details, see Creating an HPA Policy Using Custom Metrics.

Components

All Kubernetes resources created during kube-prometheus-stack add-on installation are created in the namespace named monitoring.

Table 1 kube-prometheus-stack components

Component	Description	Resource Type
prometheusOperator (workload name: prometheus-operator)	Deploys and manages the Prometheus Server based on CustomResourceDefinitions (CRDs), and monitors and processes the events related to these CRDs. It is the control center of the entire system.	Deployment
prometheus-lightweight (workload name: prometheus-lightweight)	A Prometheus Server cluster deployed by the operator based on the Prometheus CRDs that can be regarded as StatefulSets.	StatefulSet
kubeStateMetrics (workload name: kube-state-metrics)	Converts the Prometheus metric data into a format that can be identified by Kubernetes APIs. NOTE: If the components run in multiple pods, only metrics in one pod are collected.	Deployment
adapter (workload name: custom-metrics-apiserver)	Aggregates custom metrics to the native Kubernetes API Server. This is closely related to custom metric HPA. The adapter component needs to be installed only when custom metric HPA is enabled.	Deployment

Providing Resource Metrics Through the Metrics API

Resource metrics of containers and nodes, such as CPU and memory usage, can be obtained through the Kubernetes Metrics API. Resource metrics can be directly accessed, for example, by using the kubectl top command, or used by HPA or CustomedHPA policies for auto scaling.

The add-on can provide the Kubernetes Metrics API that is disabled by default. To enable the API, create the following APIService object:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    app: custom-metrics-apiserver
    release: cceaddon-prometheus
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: custom-metrics-apiserver
    namespace: monitoring
    port: 443
  version: v1beta1
  versionPriority: 100

You can save the object as a file, name it metrics-apiservice.yaml, and run the following command:

kubectl create -f metrics-apiservice.yaml

Run the kubectl top pod -n monitoring command. If information similar to the following is displayed, the Metrics API can be accessed:

# kubectl top pod -n monitoring
NAME                                                      CPU(cores)   MEMORY(bytes)
......
custom-metrics-apiserver-d4f556ff9-l2j2m                  38m          44Mi
......

NOTICE:

To uninstall the add-on, run the following kubectl command and delete the APIService object. Otherwise, the metrics-server add-on cannot be installed due to residual APIService resources.

kubectl delete APIService v1beta1.metrics.k8s.io

Creating an HPA Policy Using Custom Metrics

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. You can configure collection rules. The collected metrics can be used as monitoring metrics of HPA policies to control workload auto scaling.
   • To create a ConfigMap for collection rules, perform the following steps:

     In the navigation pane on the left, choose ConfigMaps and Secrets. Then click Create from YAML.

     You can add multiple collection rules by adding multiple configurations under the rules field. For details, see Metrics Discovery and Presentation Configuration. The following is an example of a custom collection rule:

     kind: ConfigMap
     apiVersion: v1
     metadata:
       name: user-adapter-config  # Configuration item name, which cannot be changed.
       namespace: monitoring # Namespace of the ConfigMap
     data:
       config.yaml: |-
         rules: # Collection rule
         - seriesQuery: 'container_network_receive_bytes_total{namespace!="",pod!=""}' # Original metrics required for scale-out (metrics from kubelet)
           seriesFilters: []
           resources:
             overrides:  # Specifies pod and namespace resources.
               namespace:
                 resource: namespace
               pod:
                 resource: pod
           name:
             matches: container_(.*)_total  # Matches metrics that start with container_ and end with _total.
             as: "pod_${1}_per_second"  # Metric alias
           metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[30s])) by (<<.GroupBy>>) # Metric change rate of all containers for a workload within 30s
     NOTE:

     In the preceding example, the aggregation time is 30s. If this time is set to a value less than 15s (a collection period), the metrics may be inaccurate.

   • If a ConfigMap already exists, perform the following steps to modify the collection rule:

     In the navigation pane on the left, choose ConfigMaps and Secrets. Switch to the monitoring namespace, locate the target ConfigMap on the ConfigMaps tab, and click Update.

     On the Update ConfigMap page, locate the target data, click Edit to modify the collection rule.

     Figure 1 Updating a ConfigMap
     

3. In the navigation pane, choose Add-ons. Locate the Cloud Native Cluster Monitoring add-on, click Edit, and enable the custom metric HPA.

   Click OK.

   NOTE:

   You need to create collection rules described in 2 and then enable the custom metric HPA to trigger add-on redeployment for custom metric collection to take effect.

4. In the navigation pane on the left, choose Workloads. Locate the workload for which you want to create an HPA policy and click Auto Scaling. In the Custom Policy area, you can create the auto scaling policy in 2.
   NOTE:

   After a workload is created, create an HPA policy unless all pods for that workload are ready and the metrics of the first collection period are collected.

   Figure 2 Creating an HPA policy