Using KEDA for Event-Driven Auto Scaling

How It Works

Figure 1 KEDA architecture

KEDA deploys three components in a Kubernetes cluster:

• keda-operator monitors KEDA resources (for example, ScaledObjects) and manages the HPA lifecycle. It directly handles scaling between 0 and 1 (and vice versa), activating idle Deployments when events arrive and scaling their pod replicas to zero when idle.
• keda-metrics-apiserver exposes external metrics from event sources (for example, Kafka, RabbitMQ, Elasticsearch) to the Kubernetes Horizontal Pod Autoscaler (HPA) through the Kubernetes metrics API. The HPA queries these metrics to calculate the required pod count.
• keda-admission-webhooks validates KEDA resources (for example, ScaledObject) on creation or update to detect configuration errors (such as multiple ScaledObjects targeting the same Deployment) and prevent runtime issues.

KEDA handles scaling through two complementary mechanisms:

• 0 <-> 1 scaling is handled directly by keda-operator. When events arrive, the operator scales the target Deployment from 0 to 1 replica. When no events remain, it scales back to 0 replicas.
• 1 <-> N scaling is delegated to the HPA. The keda-operator creates and manages the HPA resource, which queries external metrics via keda-metrics-apiserver and adjusts the pod count accordingly.

Preparations

• Create a CCE cluster (v1.30 or later) and ensure that an EIP has been bound to a node for downloading images and external access.
• Prepare an ECS, bind an EIP to it, and configure kubectl to connect to the cluster. For details, see Connecting to a Cluster Using kubectl.
• Install the Helm client (3.8 or later) on the ECS. For details, see Deploying an Application Using Helm v3.

Installing KEDA

1. Log in to the ECS.
2. Run the command below to install KEDA (v2.20.1 is used as an example). You can also visit the KEDA official website and follow the instructions to manually install KEDA.

   helm install keda kedacore/keda \
     --namespace keda \
     --create-namespace \
     --version 2.20.1 \

3. Run the following command to check whether KEDA is successfully installed:

   kubectl get pod -n keda

   Expected command output

   

Using a Redis Task Queue to Trigger KEDA Scaling

1. Deploy a single-node Redis.

   # Create a Redis Deployment.
   kubectl create deployment redis --image=redis:7-alpine

   Check whether the Redis pod is ready.

   kubectl get pod | grep "redis"

   

2. Expose the ClusterIP Service for KEDA and consumer programs to access.

   kubectl expose deployment redis --port=6379 --target-port=6379

   The intra-cluster access address of Redis is redis:6379.

   Check the ClusterIP Service.

   kubectl get svc redis

3. Deploy a test application.
   Create the worker-deploy.yaml file. The file content is as follows:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: redis-worker
   spec:
     replicas: 0  # The initial number of replicas is 0. No pod is running when there is no message.
     selector:
       matchLabels:
         app: redis-worker
     template:
       metadata:
         labels:
           app: redis-worker
       spec:
         containers:
         - name: worker
           image: nginx:alpine
           ports:
           - containerPort: 80

   Create application resources.

   kubectl apply -f worker-deploy.yaml

   Check the number of replicas.

   kubectl get deploy redis-worker

   

4. Create the core configuration of the Redis trigger ScaledObject.

   Create the redis-scaledobject.yaml file. The file content is as follows:

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: redis-list-scaler
     namespace: default
   spec:
     scaleTargetRef:
       name: redis-worker          # Scaling object
     minReplicaCount: 0            # Minimum number of replicas
     maxReplicaCount: 10           # Maximum number of replicas
     cooldownPeriod: 30           # Scaling cooldown period
     pollingInterval: 5           # Polling interval
     triggers:
     - type: redis
       metadata:
         address: redis:6379        # Redis access address
         listName: task_queue      # Redis queue name
         listLength: "5"            # Redis queue length threshold

   Application trigger:

   kubectl apply -f redis-scaledobject.yaml

   Check the trigger status. If READY is True, the configuration is successful.

   kubectl get scaledobject

   

5. Log in to the Redis client.

   kubectl exec -it deploy/redis -- redis-cli

6. Write multiple task records to task_queue.

   # Write eight messages at a time. The threshold 5 is exceeded, triggering scale-out.
   LPUSH task_queue task1 task2 task3 task4 task5 task6 task7 task8
   # Check the queue length.
   LLEN task_queue

   

7. Observe the automatic scale-out of application replicas on another terminal.

   kubectl get deploy redis-worker

   

   The number of replicas changes from 0 to 2, indicating that the automatic scale-out is successful.

8. Return to the redis-cli window and display the queue message.

   DEL task_queue

   

9. Wait for 30 seconds and observe the automatic scale-in of application replicas.

   kubectl get deploy redis-worker

   

   The number of replicas changes from 2 to 0, indicating that the automatic scale-in is successful.