Help Center> Cloud Container Engine> Best Practices> Networking> Performing Graceful Rolling Upgrade for CCE Turbo Cluster Applications

Performing Graceful Rolling Upgrade for CCE Turbo Cluster Applications

Scenario

In a CCE Turbo cluster, when you deploy a workload to run an application, ENIs or supplementary ENIs are used for container networking. The application is exposed as a LoadBalancer Service or ingress, connected to a dedicated ELB load balancer to allow access traffic to reach the containers directly. When rolling upgrade or auto scaling is performed on the application, your pods may fail to work with ELB and 5xx errors may occur. This section guides you to configure container probes and readiness time to achieve graceful upgrade and elastic scaling.

Procedure

The following uses an Nginx Deployment as an example.

  1. On the CCE console, choose Workloads > Deployments in the navigation pane, and click Create Deployment in the upper right corner.

    Figure 1 Creating an Nginx Deployment

  2. In the Add Container step, expand Health Check, and configure the readiness probe as below.

    The probe checks whether your container is ready. If not, requests will not be forwarded to the container.
    Figure 2 Configuring the readiness probe

  3. Expand Lifecycle and configure pre-stop command for the container.

    This configuration ensures that the container can provide services for external systems during its exit.
    Figure 3 Configuring the pre-stop command

  4. Click Next: Set Application Access and set the Service and port as below.

    Figure 4 Creating a Service

  5. Click Next. Configure the minimum readiness time. This parameter specifies the minimum interval for a newly created pod to be ready without any of its containers crashing. The recommended value is the expected time for starting the service container plus the duration from the time when the ELB service delivers the member to the time when the member takes effect.

    Figure 5 Configuring the minimum readiness time

  6. For an existing workload, you can change the readiness time setting by clicking Edit YAML on the workload details page.

    Figure 6 Configuring the minimum readiness time by editing the YAML file

    A pod is considered available only when the minimum readiness time is exceeded without any of its containers crashing.

  7. Create an ENI LoadBalancer Service.

    Figure 7 Creating an ENI LoadBalancer Service

  8. After the configuration is complete, test the application upgrade and auto scaling. Prepare a client node outside the cluster, configure the detection script detection_script.sh, and copy the following content to the script. 192.168.12.82:80 indicates the URL for accessing the Service. (If you use a node in the cluster to access the Service, the request is forwarded within the cluster without passing through the ELB load balancer.)

    #! /bin/bash 
    for (( ; ; ))
    do
             curl -I 192.168.12.82:80 | grep "200 OK"
             if [ $? -ne 0 ]; then
                      echo "response error!"
                      exit 1
             fi
    done

  9. Run the detection script (bash detection_script.sh) and trigger the rolling upgrade of the application on the CCE console. In the following example, an environment variable is added for the container to trigger the rolling upgrade of the application.

    Figure 8 Triggering the application rolling upgrade

  10. Check the upgrade state on the CCE console. If the access to the application is not interrupted, and the returned responses are all 200 ok, the graceful upgrade is successfully triggered.

    Figure 9 Graceful upgrade status