Cloud Native Cluster Monitoring

Introduction

The Cluster Native Cluster Monitoring add-on (kube-prometheus-stack) uses Prometheus-operator and Prometheus to provide easy-to-use, end-to-end Kubernetes cluster monitoring.

This add-on allows the monitoring data to be interconnected with Monitoring Center so that you can view monitoring data and configure alarms in Monitoring Center.

Open source community: https://github.com/prometheus/prometheus

Permissions

The node-exporter component of the Cluster Native Cluster Monitoring add-on needs to read the Docker info data from the /var/run/docker.sock directory on the host for monitoring the Docker disk space.

The following permission is required for running node-exporter:

• cap_dac_override: reads the Docker info data.

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane on the left, choose Add-ons. Locate Cloud Native Cluster Monitoring on the right and click Install.
2. On the Install Add-on page, select a version.
3. Configure the parameters.
   • Interconnecting with AOM: Prometheus data will be reported to AOM. If this option is enabled, you need to select an AOM instance. Basic metrics are free of charge. Custom metrics are billed based on the standard pricing of AOM. For details, see Product Pricing Details. To interconnect with AOM, you must have certain permissions. Only Huawei Cloud accounts, HUAWEI IDs, and IAM users in the admin user group can perform this operation.
   • User-defined indicator HPA: Application metrics are automatically collected in the form of service discovery. After this function is enabled, you need to add related configurations to the target application. For details, see Creating an HPA Policy Using Custom Metrics.

4. Click Install.

   After the add-on is installed, you may need to perform the following operations:

   • To use this add-on to provide system resource metrics (such as CPU and memory usage) for workload auto scaling, enable the Metric API. For details, see Providing Resource Metrics Through the Metrics API. After the configuration, you can use Prometheus to collect system resource metrics.

Components

All Kubernetes resources created during kube-prometheus-stack add-on installation are created in the namespace named monitoring.

Providing Resource Metrics Through the Metrics API

Resource metrics of containers and nodes, such as CPU and memory usage, can be obtained through the Kubernetes Metrics API. Resource metrics can be directly accessed, for example, by using the kubectl top command, or used by HPA or CustomedHPA policies for auto scaling.

The add-on can provide the Kubernetes Metrics API that is disabled by default. To enable the API, create the following APIService object:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    app: custom-metrics-apiserver
    release: cceaddon-prometheus
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: custom-metrics-apiserver
    namespace: monitoring
    port: 443
  version: v1beta1
  versionPriority: 100

You can save the object as a file, name it metrics-apiservice.yaml, and run the following command:

kubectl create -f metrics-apiservice.yaml

Run the kubectl top pod -n monitoring command. If the following information is displayed, the Metrics API can be accessed:

# kubectl top pod -n monitoring
NAME                                                      CPU(cores)   MEMORY(bytes)
......
custom-metrics-apiserver-d4f556ff9-l2j2m                  38m          44Mi
......

To uninstall the add-on, run the following kubectl command and delete the APIService object. Otherwise, the metrics-server add-on cannot be installed due to residual APIService resources.

kubectl delete APIService v1beta1.metrics.k8s.io

Creating an HPA Policy Using Custom Metrics

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane on the left, choose ConfigMaps and Secrets. Click Create from YAML.

   You can add multiple collection rules by adding multiple configurations under the rules field. For details, see Metrics Discovery and Presentation Configuration. The following is an example of a custom collection rule:

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: user-adapter-config
     namespace: monitoring
   data:
     config.yaml: |-
       rules:
       - seriesQuery: 'container_network_receive_bytes_total{namespace!="",pod!=""}' # Original metrics required for scale-out (metrics from kubelet)
         seriesFilters: []
         resources:
           overrides:
             namespace:
               resource: namespace
             pod:
               resource: pod
         name:
           matches: container_(.*)_total
           as: "pod_${1}_per_second"  # Metric alias
         metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[30s])) by (<<.GroupBy>>) # Metric change rate of all containers for a workload within 30s

   

   In the preceding example, the aggregation time is 30s. If this time is set to a value less than 15s (a collection period), the metrics may be inaccurate.

3. In the navigation pane on the left, choose Add-ons. In the right pane, locate the Cloud Native Cluster Monitoring add-on, click Edit in the Operation column, and enable the custom metric HPA.

   Click Install.

   

   You need to create collection rules described in 2 and then enable the custom metric HPA to trigger add-on redeployment for custom metric collection to take effect.

4. In the navigation pane on the left, choose Workloads. Locate the workload for which you want to create an HPA policy and choose More > Auto Scaling. In the Custom Policy area, you can select the preceding parameters to create an auto scaling policy.
   

   After a workload is created, create an HPA policy unless all pods for that workload are ready and the metrics of the first collection period are collected.

   Figure 1 Creating an HPA policy