Scaling Based on Prometheus Metrics

How It Works

CCE fully integrates with the open-source Prometheus ecosystem. It supports monitoring of various custom metrics and auto scaling based on those metrics. The following figure shows the architecture.

Figure 1 Using an HPA policy with custom metrics

The core process for implementing HPA auto scaling with custom metrics is as follows:

1. Exposing metrics: Applications instrument their code to expose key metrics (such as queue length, online user count, and requests per second).
2. Metric collection: If an application cannot directly expose metrics, deploy an Exporter to scrape data and convert it into a Prometheus-compatible format.
3. Metric storage: Prometheus periodically scrapes metrics from Exporters or applications and stores them as time-series data for querying and analysis.
4. Metric translation: defines the mapping between PromQL queries and Kubernetes custom metric API names in the adapter configuration file (user-adapter-config).
5. API registration: registers the custom metric API with the Kubernetes API server so that HPA can query custom metrics through a unified interface, like CPU metrics.
6. Scaling: HPA periodically queries custom metric values, calculates the desired replica count, and scales the workload accordingly.

Prerequisites

• A cluster is available.
• You can access the cluster using kubectl. For details, see Accessing a Cluster Using kubectl.

Step 1: Install Cloud Native Cluster Monitoring

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Add-ons.
2. Locate the Cloud Native Cluster Monitoring add-on and click Install.
   Prioritize the configurations listed below and adjust any other configurations as needed. For details, see Cloud Native Cluster Monitoring.

   • Local Data Storage: Enable this option to store the monitoring data in local storage. You can determine whether to report the monitoring data to AOM or a third-party monitoring platform.
   • Custom Metric Collection: Enable this option in this practice. If this option is not enabled, custom metrics cannot be collected.

3. Click Install.

Step 2: Obtain Prometheus Monitoring Data

In this section, HPA scaling is performed based on metrics related to pods, for example, pod metrics. You can also perform HPA scaling based on metrics unrelated to pods, for example, external load balancer metrics. For details, see Scaling Based on Load Balancer Monitoring Metrics.

The following describes how to deploy a sample-app application and expose the container_memory_working_set_bytes_per_second metric in Prometheus standard mode to identify the number of bytes per second in the working set of the container memory. For details about Prometheus metrics, see Metric types.

1. Deploy a test application.
   1. Write a sample-app.yaml file. The file content is as follows:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: sample-app
        labels:
          app: sample-app
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: sample-app
        template:
          metadata: 
            labels:
              app: sample-app
          spec:
            containers: 
            - image: swr.cn-east-3.myhuaweicloud.com/container/autoscale-demo:v0.1.2 # Sample image
              name: metrics-provider 
              resources:
                requests:
                  cpu: 250m
                  memory: 512Mi
                limits:
                  cpu: 250m
                  memory: 512Mi
              ports:
              - name: http
                containerPort: 8080   #Container port
            imagePullSecrets:
              - name: default-secret
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: sample-app
        namespace: default
        labels: 
          app: sample-app
      spec:
        ports: 
          - port: 80
            name: http
            protocol: TCP
            targetPort: 8080
        selector:
          app: sample-app 
        type: ClusterIP 

      

      The application exposes container_memory_working_set_bytes_per_second to check the working memory size of a container per second.

   2. Create the workload.

      kubectl apply -f sample-app.yaml

2. Create a ServiceMonitor to monitor custom metrics.
   1. Create a servicemonitor.yaml file. The file content is as follows:

      apiVersion: monitoring.coreos.com/v1 
      kind: ServiceMonitor
      metadata:
        name: sample-app  # ServiceMonitor name
        namespace: default
      spec:
        endpoints:        # Endpoints of the service to be monitored, including the name, port number, path, protocol, and more
        - interval: 30s   # Prometheus operator checks whether a service needs to be added to the monitored target list every 30 seconds.
          port: http
          path: /metrics
        namespaceSelector:
          any: true
        selector: 
          matchLabels:
            app: sample-app  # Label of the object whose data needs to be collected

   2. Create the ServiceMonitor.

      kubectl apply -f servicemonitor.yaml

Step 3: Modify Configuration Files

1. Modify Prometheus' adapter-config.

   You can modify the rules field in adapter-config to convert the metrics exposed by Prometheus to metrics that can be associated with the HPA.

   kubectl -n monitoring edit configmap user-adapter-config

2. Add a custom metric collection rule to the rules field. You can add multiple collection rules by adding multiple configurations under the rules field. For details, see Metrics Discovery and Presentation Configuration.
   The following is an example of a custom collection rule:

   rules:
   - seriesQuery: container_memory_working_set_bytes{namespace!="",pod!=""}
     resources:
       overrides:
         namespace: 
           resource: namespace
         pod: 
   	resource: pod
     name:
       matches: ^(.*)_bytes
       as: ${1}_bytes_per_second #The value of ${1} is the value that matches ^(.*) in matches:"^(.*)_bytes".
     metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)

   • seriesQuery: indicates the PromQL request data. It specifies the metrics to be obtained by users. You can configure this parameter as needed.
   • metricsQuery: aggregates the data requested by PromQL in seriesQuery.
   • resources: specifies data label in PromQL, which is used to match resources. The resources here refer to the api-resource in a cluster, such as pods, namespaces, and nodes. You can run kubectl api-resources -o wide to check the resources. The key corresponds to LabelName in the Prometheus data. You have to ensure that the Prometheus metric data contains LabelName.
   • name: indicates that Prometheus metric names are converted to readable metric names based on regular expression matching. In this example, container_memory_working_set_bytes is converted to container_memory_working_set_bytes_per_second.

3. Redeploy the custom-metrics-apiserver workload in the monitoring namespace.

   kubectl -nmonitoring delete pod -l app=custom-metrics-apiserver

4. Check whether the metric has been added.

   kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/container_memory_working_set_bytes_per_second

   

Step 4: Create an HPA Policy

1. Create an HPA policy using custom metrics.
   1. Create an hpa.yaml file. The file content is as follows:

      kind: HorizontalPodAutoscaler
      apiVersion: autoscaling/v2
      metadata:
        name: sample-app-memory-high
      spec:
      # Description of an HPA target. The HPA dynamically changes the number of pods of the target.
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: sample-app
      # Minimum number of pods and maximum number of pods of the HPA
        minReplicas: 1
        maxReplicas: 10
      #Monitored metric array. Multiple types of metrics can coexist.
        metrics:
        - type: Pods
          pods:
            metric: 
              name: container_memory_working_set_bytes_per_second   # Use the custom container metrics.
            target:
              type: AverageValue  # Target value of the AverageValue type. For the pod metrics, only the target values of the AverageValue type are supported.
              averageValue: 1024000m   # 1024000m specifies 1 KB.

   2. Create the HPA policy.

      kubectl apply -f hpa.yaml

2. Check whether the HPA policy takes effect.

   kubectl get hpa sample-app-memory-high

   

   The number of replicas has been increased from 1 to 10.

Helpful Links

• If the application uses ELB to provide external access, you can perform auto scaling based on ELB monitoring metrics (such as QPS). For details, see Scaling Based on Load Balancer Monitoring Metrics.
• If the application uses the Nginx ingress to route and forward traffic, you can perform auto scaling based on Nginx ingress monitoring metrics. For details, see Scaling Multiple Applications Using Nginx Ingresses.