Auto Scaling Based on Prometheus Metrics

Kubernetes' default HPA policy only allows for auto scaling based on CPU and memory usage. However, in more complex service scenarios, this may not be sufficient to meet routine O&M requirements. To address this, CCE offers a Cloud Native Cluster Monitoring (kube-prometheus-stack) add-on that integrates into the open-source Prometheus ecosystem. This add-on allows for monitoring of various components and provides multiple preset monitoring dashboards that are ready to use out-of-the-box. This document describes how to convert Huawei Cloud Prometheus metrics into metrics that can be used by HPA, providing a more convenient scaling mechanism for applications.

Prerequisites

A cluster has been created.
You can access the cluster using kubectl. For details, see Connecting to a Cluster Using kubectl.

Step 1: Install Cloud Native Cluster Monitoring

Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Add-ons.
Locate the Cloud Native Cluster Monitoring add-on and click Install.
You are advised to focus on the following configurations, and adjust any other configurations as necessary. For details, see Cloud Native Cluster Monitoring.
- Local Data Storage: Enable this option to store the monitoring data in local storage. You can determine whether to report the monitoring data to AOM or a third-party monitoring platform.
- Custom Metric Collection: Enable this option in this practice. If this option is not enabled, custom metrics cannot be collected.
Click Install.

Step 2: Obtain Prometheus Monitoring Data

In this section, HPA is performed based on metrics related to pods, for example, pod metrics. You can also perform HPA based on metrics irrelevant to pods, for example, external load balancer metrics. For details, see Auto Scaling Based on ELB Monitoring Metrics.

The following describes how to deploy a sample-app application and expose the container_memory_working_set_bytes_per_second metric in Prometheus standard mode to identify the number of bytes per second in the working set of the container memory. For more information about Prometheus metrics, see METRIC TYPES.

Deploy the test application.

Write a sample-app.yaml file. The file content is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata: 
      labels:
        app: sample-app
    spec:
      containers: 
      - image: swr.cn-east-3.myhuaweicloud.com/container/autoscale-demo:v0.1.2 # Sample image
        name: metrics-provider 
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
          limits:
            cpu: 250m
            memory: 512Mi
        ports:
        - name: http
          containerPort: 8080   #Container port
      imagePullSecrets:
        - name: default-secret
---
apiVersion: v1
kind: Service
metadata:
  name: sample-app
  namespace: default
  labels: 
    app: sample-app
spec:
  ports: 
    - port: 80
      name: http
      protocol: TCP
      targetPort: 8080
  selector:
    app: sample-app 
  type: ClusterIP

The application exposes container_memory_working_set_bytes_per_second, which is used to check the working memory size of a container per second.

Create a workload.
```
kubectl apply -f sample-app.yaml
```

Create a ServiceMonitor to monitor custom metrics.

Create a servicemonitor.yaml file. The file content is as follows:

apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor
metadata:
  name: sample-app  # ServiceMonitor name
  namespace: default
spec:
  endpoints:        # Endpoints of the service to be monitored, including the name, port number, path, protocol, and more
  - interval: 30s   # Prometheus operator checks whether a service needs to be added to the monitored target list every 30 seconds.
    port: http
    path: /metrics
  namespaceSelector:
    any: true
  selector: 
    matchLabels:
      app: sample-app  # Label of the object whose data needs to be collected

Create ServiceMonitor.
```
kubectl apply -f servicemonitor.yaml
```

Step 3: Modify Configuration Files

Modify Prometheus' adapter-config.

You can modify the rules field in adapter-config to convert the metrics exposed by Prometheus to metrics that can be associated with HPA.
```
kubectl -n monitoring edit configmap user-adapter-config
```
Add a custom metric collection rule to the rules field. You can add multiple collection rules by adding multiple configurations under the rules field. For details, see Metrics Discovery and Presentation Configuration.
The following is an example of a custom collection rule:
```
rules:
- seriesQuery: container_memory_working_set_bytes{namespace!="",pod!=""}
  resources:
    overrides:
      namespace: 
        resource: namespace
      pod: 
	resource: pod
  name:
    matches: ^(.*)_bytes
    as: ${1}_bytes_per_second #The value of ${1} is the value that matches ^(.*) in matches:"^(.*)_bytes".
  metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
```
- seriesQuery: indicates the PromQL request data. It specifies the metrics to be obtained by users. You can configure this parameter as needed.
- metricsQuery: aggregates the data requested by PromQL in seriesQuery.
- resources: specifies data label in PromQL, which is used to match resources. The resources here refer to the api-resource in a cluster, such as pods, namespaces, and nodes. You can run kubectl api-resources -o wide to check the resources. The key corresponds to LabelName in the Prometheus data. You have to ensure that the Prometheus metric data contains LabelName.
- name: indicates that Prometheus metric names are converted to readable metric names based on regular expression matching. In this example, container_memory_working_set_bytes is converted to container_memory_working_set_bytes_per_second.
Redeploy the custom-metrics-apiserver workload in the monitoring namespace.
```
kubectl -nmonitoring delete pod -l app=custom-metrics-apiserver
```

Run the following command to check whether the metric is added:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/container_memory_working_set_bytes_per_second

Click to enlarge

Step 4: Create an HPA Policy

Create an HPA policy using custom metrics.

Create an hpa.yaml file. The file content is as follows:

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2
metadata:
  name: sample-app-memory-high
spec:
# Description of an HPA object. HPA dynamically changes the number of pods of the object.
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
# Minimum number of pods and maximum number of pods of the HPA
  minReplicas: 1
  maxReplicas: 10
#Monitored metric array. Multiple types of metrics can coexist.
  metrics:
  - type: Pods
    pods:
      metric: 
        name: container_memory_working_set_bytes_per_second   # Use the custom container metrics.
      target:
        type: AverageValue  # Target value of the AverageValue type. For the pod metrics, only the target values of the AverageValue type are supported.
        averageValue: 1024000m   # 1024000m specifies 1 KB.