Auto Scaling Based on ELB Monitoring Metrics

Background

By default, Kubernetes scales a workload based on resource usage metrics such as CPU and memory. However, this mechanism cannot reflect the real-time resource usage when traffic bursts arrive, because the collected CPU and memory usage data lags behind the actual load balancer traffic metrics. For some services (such as flash sale and social media) that require fast auto scaling, scaling based on this rule may not be performed in a timely manner and cannot meet these services' actual needs. In this case, auto scaling based on ELB QPS data can respond to service requirements more timely.

Solution

This section describes an auto scaling solution based on ELB monitoring metrics. Compared with CPU/memory usage-based auto scaling, auto scaling based on ELB QPS data is more targeted and timely.

The key of this solution is to obtain the ELB metric data and report the data to Prometheus, convert the data in Prometheus to the metric data that can be identified by HPA, and then perform auto scaling based on the converted data.

The implementation scheme is as follows:

1. Develop a Prometheus exporter to obtain ELB metric data, convert the data into the format required by Prometheus, and report it to Prometheus. This section uses cloudeye-exporter as an example.
2. Convert the Prometheus data into the Kubernetes metric API for the HPA controller to use.
3. Set an HPA rule to use ELB monitoring data as auto scaling metrics.

Figure 1 ELB traffic flows and monitoring data

Other metrics can be collected in the similar way.

Prerequisites

• You are familiar with Prometheus.
• You have the Cloud Native Cluster Monitoring (kube-prometheus-stack) add-on installed in your cluster. This add-on supports clusters of v1.17 or later.
  

  You need to set the deployment mode of Cloud Native Cluster Monitoring to the server mode.

Building an Exporter Image

This section uses cloudeye-exporter to monitor load balancer metrics. To develop an exporter, see Appendix: Developing an Exporter.

1. Log in to a VM that can access the Internet and has Docker installed and write a Dockerfile.

   vi Dockerfile

   The content is as follows:

   FROM ubuntu:18.04
   RUN apt-get update \
     && apt-get install -y git ca-certificates curl \
       && update-ca-certificates \
         && curl -O  https://dl.google.com/go/go1.14.14.linux-amd64.tar.gz \
           && tar -zxf go1.14.14.linux-amd64.tar.gz -C /usr/local \
             && git clone https://github.com/huaweicloud/cloudeye-exporter \
               && export PATH=$PATH:/usr/local/go/bin \
                 && export GO111MODULE=on \
                   && export GOPROXY=https://goproxy.cn,direct \
                     && export GONOSUMDB=* \
                       && cd cloudeye-exporter \
                         && go build 
   CMD ["/cloudeye-exporter/cloudeye-exporter -config=/tmp/clouds.yml"]

2. Build an image. The image name is cloudeye-exporter and the image version is 1.0.

   docker build --network host . -t cloudeye-exporter:1.0

3. Push the image to SWR.
   1. (Optional) Log in to the SWR console, choose Organizations in the navigation pane, and click Create Organization in the upper right corner of the page.

      Skip this step if you already have an organization.

   2. In the navigation pane, choose My Images and then click Upload Through Client. On the page displayed, click Generate a temporary login command and click to copy the command.
   3. Run the login command copied in the previous step on the cluster node. If the login is successful, the message "Login Succeeded" is displayed.
   4. Tag the cloudeye-exporter image.

      docker tag {Image name 1:Tag 1}/{Image repository address}/{Organization name}/{Image name 2:Tag 2}

      • {Image name 1:Tag 1}: name and tag of the local image to be uploaded.
      • {Image repository address}: The domain name at the end of the login command in 3.b is the image repository address, which can be obtained on the SWR console.
      • {Organization name}: name of the organization created in 3.a.
      • {Image name 2:Tag 2}: desired image name and tag to be displayed on the SWR console.

      The following is an example:

      docker tag cloudeye-exporter:1.0 swr.ap-southeast-1.myhuaweicloud.com/cloud-develop/cloudeye-exporter:1.0

   5. Push the image to the image repository.

      docker push {Image repository address}/{Organization name}/{Image name 2:Tag 2}

      The following is an example:

      docker push swr.ap-southeast-1.myhuaweicloud.com/cloud-develop/cloudeye-exporter:1.0

      The following information will be returned upon a successful push:

      ... 
      030***: Pushed 
      1.0: digest: sha256:eb7e3bbd*** size: **

      To view the pushed image, go to the SWR console and refresh the My Images page.

Deploying the Exporter

Prometheus can dynamically monitor pods if you add Prometheus annotations to the pods (the default path is /metrics). This section uses cloudeye-exporter as an example.

Common annotations in Prometheus are as follows:

• prometheus.io/scrape: If the value is true, the pod will be monitored.
• prometheus.io/path: URL from which the data is collected. The default value is /metrics.
• prometheus.io/port: port number of the endpoint to collect data from.
• prometheus.io/scheme: Defaults to http. If HTTPS is configured for security purposes, change the value to https.

Interconnecting with Prometheus

After collecting monitoring data, Prometheus needs to convert the data into the Kubernetes metric API for the HPA controller to perform auto scaling.

In this example, the ELB metrics associated with the workload need to be monitored. Therefore, the target workload must use the Service or ingress of the LoadBalancer type.

1. View the access mode of the workload to be monitored and obtain the ELB listener ID.
   1. On the CCE cluster console, choose Networking. On the Services or Ingresses tab page, view the Service or ingress of the LoadBalancer type and click the load balancer to access the load balancer page.

      

   2. On the Listeners tab, view the listener corresponding to the workload and copy the listener ID.

      

2. Use kubectl to connect to the cluster and add Prometheus configurations. In this example, collect load balancer metrics. For details about advanced usage, see Configuration.
   1. Create the prometheus-additional.yaml file, add the following content to the file, and save the file:

      - job_name: elb_metric
        params:
          services: ['SYS.ELB']
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
          - action: keep
            regex: '8087'
            source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_port
          - action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            source_labels:
              - __address__
              - __meta_kubernetes_service_annotation_prometheus_io_port
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - action: replace
            source_labels:
              - __meta_kubernetes_namespace
            target_label: kubernetes_namespace
          - action: replace
            source_labels:
              - __meta_kubernetes_service_name
            target_label: kubernetes_service

   2. Use the preceding configuration file to create a secret named additional-scrape-configs.

      kubectl create secret generic additional-scrape-configs --from-file prometheus-additional.yaml -n monitoring --dry-run=client -o yaml | kubectl apply -f -

   3. Modify the Prometheus node object.

      kubectl edit prometheus server -n monitoring

      Add the following content to the spec field and save the file:

      spec:
        additionalScrapeConfigs:
          key: prometheus-additional.yaml
          name: additional-scrape-configs

   4. Check whether the modification has taken effect.

      kubectl get secret prometheus-server -n monitoring -o jsonpath="{.data['prometheus\.yaml\.gz']}" | base64 --decode | gzip -d | grep -A3 elb

      If any command output is displayed, the modification has taken effect.

3. Modify the user-adapter-config ConfigMap. (In earlier versions of this add-on, the ConfigMap name is adapter-config.)

   kubectl edit configmap user-adapter-config -nmonitoring

   Add the following content under the rules field and save the file. Replace the listener ID obtained in 1 with the value of seriesQuery.

   apiVersion: v1
   data:
     config.yaml: |-
       rules:
       - metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
         resources:
           overrides:
             kubernetes_namespace:
               resource: namespace
             kubernetes_service:
               resource: service
         name:
           matches: huaweicloud_sys_elb_(.*)
           as: "elb01_${1}"
         seriesQuery: '{lbaas_listener_id="*****"}' #ELB listener ID
       ...

4. Redeploy the custom-metrics-apiserver workload in the monitoring namespace.

   

Creating an HPA Policy

After the data reported by the exporter to Prometheus is converted into the Kubernetes metric API by using the Prometheus adapter, you can create an HPA policy for auto scaling.

1. Create an HPA policy. The inbound traffic of the ELB load balancer is used to trigger scale-out. When the value of m7_in_Bps (inbound traffic rate) exceeds 1000, the nginx Deployment will be scaled.

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: nginx
     namespace: default
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: nginx
     minReplicas: 1
     maxReplicas: 10
     metrics:
       - type: Object
         object:
           metric:
             name: elb01_listener_m7_in_Bps  #Monitoring metric name
           describedObject:
             apiVersion: v1
             kind: Service
             name: cloudeye-exporter
           target:
             type: Value
             value: 1000

   Figure 2 Created HPA Policy
   

2. After the HPA policy is created, perform a pressure test on the workload (accessing the pods through ELB). Then, the HPA controller determines whether scaling is required based on the configured value.
   In the Events dialog box, obtain scaling records in the Kubernetes Event column.

   Figure 3 Scaling events
   

ELB Listener Metrics

The following table lists the ELB listener metrics that can be collected using the method described in this section.

Appendix: Developing an Exporter

Prometheus periodically calls the /metrics API of the exporter to obtain metric data. Applications only need to report monitoring data through /metrics. You can select a Prometheus client in a desired language and integrate it into applications to implement the /metrics API. For details about the client, see Prometheus CLIENT LIBRARIES. For details about how to write the exporter, see WRITING EXPORTERS.

The monitoring data must be in the format that Prometheus supports. Each data record provides the ELB ID, listener ID, namespace where the Service is located, Service name, and Service UID as labels, as shown in the following figure.

To obtain the preceding data, perform the following steps:

1. Obtain all Services.

   The annotations field in the returned information contains the ELB associated with the Service.

   • kubernetes.io/elb.id
   • kubernetes.io/elb.class

2. Use APIs in Querying Listeners to get the listener ID based on the load balancer ID obtained in the previous step.
3. Obtain the ELB monitoring data.

   The ELB monitoring data is obtained using the CES APIs described in Querying Monitoring Data of Multiple Metrics . For details about ELB monitoring metrics, see Monitoring Metrics. Example:

   • m1_cps: number of concurrent connections
   • m5_in_pps: number of incoming data packets
   • m6_out_pps: number of outgoing data packets
   • m7_in_Bps: incoming rate
   • m8_out_Bps: outgoing rate

4. Aggregate data in the format that Prometheus supports and expose the data through the /metrics API.

   The Prometheus client can easily call the /metrics API. For details, see CLIENT LIBRARIES. For details about how to develop an exporter, see WRITING EXPORTERS.