Using Istio-based Canary Release in a Pipeline

Application Scenario

During service release, the following issues often arise:

• Although the test environment is designed to be as similar as possible to the production environment, there are still inherent differences due to factors such as resource configuration and network environment. As a result, it is not feasible to fully verify all service scenarios using the test environment alone.
• If the test is performed after the release is completed in the production environment, there exist high service risks and online users are easily affected.

Solution Architecture

How can we complete the release and verification of the live environment while mitigating service risks? Grayscale release is the solution.

Grayscale release involves creating a workload that is identical to the current online service in the production environment and updating only the program package version of the workload, including the service code and configuration. By default, the newly created gray load does not access the live network traffic and has no impact on online user services. Function verification can be performed directly in the production environment. After the gray verification is passed, gradually shift the real online service traffic to the gray load. When all traffic has been shifted, the gray load takes over all live network traffic, and the original online workload no longer processes service requests. In this case, the old workload can be safely taken offline, and the new workload can be retained to complete the overall release process.

Grayscale release can have multiple modes based on how traffic is diverted, for example, blue-green release with 100% instantaneous traffic shift and canary release with metered traffic shift based on traffic proportion or request content. Different grayscale release modes have varying requirements on cluster resources and technical capabilities. Some modes also require that certain add-ons or agent components be installed in the cluster.

Canary release provides the following capabilities:

• Performs risk-free gray verification in the production environment.
• Supports fast traffic switchover between old and new workloads.
• If an exception occurs during the verification, quickly rolls back the online service to the status before the release.
• Supports grayscale release operations on the pipeline page.

During system upgrade, users often face problems such as service interruption during deployment and gray verification failure. This practice describes how to configure release policies in a pipeline to implement smooth system upgrade without interrupting services.

Prerequisites

• A project is available. If no project is available, create one.
• You have permissions to create applications. For details, see Permission Management.
• A pipeline is available. If no pipeline is available, create one.
• The service contains the following resources and they are defined as the current version:
  • A CCE or Kubernetes cluster is available, for example, cce-demo.
  • The service has been rolled out for the first time in the cluster. A Deployment and a Service have been created in the cluster, for example, deployment-demo and service-demo.
  • For details about Deployments and Services, see Deployments and Service.
  • The Service must be associated with the Deployment. Check whether the spec.selector of the Service is configured with only the common labels of the application, for example, the app label, and whether the selector value is the same as the spec.selector.matchLabels.app value of the Deployment.
    Figure 1 spec.selector configuration of the example Service
    
    Figure 2 spec.selector configuration of the example Deployment
    
  • Istio has been installed and configured in the cluster.
  • Ensure that automatic Sidecar injection has been enabled for the cluster, and Istio Sidecar has been injected into the namespace where the deployment is executed and each pod of the workload.
  • Istio resources VirtualService and DestinationRule have been properly configured, for example, service-demo-virtual-service and service-demo-destination-rule.
  • For details about the restrictions on Services that support Istio-based canary release, see Fixing Items Manually and Fixing Items Automatically

Step 1: Creating a New Stage-Level Application in the Pipeline

1. Go to the CodeArts home page and click the target project name to access the project.
2. Choose CICD > Pipeline and click Edit in the Operation column of the target pipeline. The Task Orchestration page is displayed.
3. Click Stage to create a stage.
4. Create a grayscale release job, as shown in Figure 3.
   Figure 3 Creating a grayscale release job
   
   1. Choose New Job > From Scratch, search for Initiate Workload as shown in the figure below, click Add, set parameters, and click OK. Table 1 describes the parameters.
      Figure 4 Searching for "Initiate Workload"
      
      Figure 5 Initiate Workload

      

      Table 1 Parameter description

      Parameter	Description	Example
      Name	Name displayed in the pipeline job step.	Retain the default value.
      Step Name	Step name.	Retain the default value.
      CCE	Select an existing CCE cluster.	cce
      Namespace	Select the namespace of the Kubernetes cluster on CCE or the namespace of the user-defined Kubernetes cluster.	Retain the default value.
      Service Name	Select the name of the Service created in the CCE or user-defined Kubernetes cluster.	service-demo
      Grayscale Release Version Type	Grayscale release version type: Custom: a grayscale release version entered by the user Task execution timestamp: timestamp generated based on the execution time Random number: a number randomly generated	Retain the default value.
      Grayscale Release Version	When Grayscale Release Version Type is set to Custom, manually enter a grayscale release version.	v2
      Image Address	Configure the pull address of the image to be deployed.	Retain the default value.

   2. Click + under the job created in the previous step, select From Scratch, search for ManualReview as shown in the figure below, click Add, set parameters, and click OK. Table 2 describes the parameters.
      Figure 6 Searching for "ManualReview"
      
      Figure 7 ManualReview
      

      Table 2 Parameter description

      Parameter	Description	Example
      Name	Name displayed in the pipeline job step.	Retain the default value.
      Review Source	Review source: Project Members: members of the project to which the pipeline belongs Project Roles: roles configured for the project to which the pipeline belongs	Retain the default value.
      Reviewer	Reviewers to be configured when the Review Source is set to Project Members. Multiple reviewers can be selected.	-
      Review Mode	The review mode of multiple reviewers that need to be configured when Review Source is set to Project Members: Review by all: The application is approved after being approved by all the reviewers. Review by anyone: The application is approved after being approved by any of the reviewers.	Retain the default value.
      Review Role	Review role that needs to be configured when Review Source is set to Project Roles. Note: Use a union of people corresponding to the roles and can be reviewed by anyone. (In IPD projects, the project creator will not be involved in the reviewers as the role of project manager.)	-
      Timeout Processing	Review timeout processing: Review failed and pipeline stopped Review result ignored and pipeline continues	Retain the default value.
      Review Duration	The review waiting duration can be configured. The review duration cannot exceed three days.	01:00:00
      Description	Supplementary description of the review result.	-

   3. Click + under the job created in the previous step, select From Scratch, search for Traffic Diversion as shown in the figure below, click Add, set parameters, and click OK. For details about the parameters, see Table 3. (Set Traffic Diversion Type to Istio-based canary release and Traffic Takeover to Traffic diversion.)
      

      This step can be performed multiple times to configure different weights.

      Figure 8 Searching for "Traffic Diversion"
      
      Figure 9 Traffic Diversion
      

      Table 3 Parameter description

      Parameter	Description	Example
      Name	Name displayed in the pipeline job step.	Retain the default value.
      Step Name	Step name.	Retain the default value.
      Traffic Diversion Type	Traffic diversion type: Service-based blue-green traffic diversion Istio-based canary release When Istio-based canary release is used, Traffic Diversion Type must be set to Istio-based canary release.	Istio-based canary release
      Traffic Takeover	Traffic takeover: Traffic diversion Official release When Istio-based canary release is used, Traffic Takeover must be set to Traffic diversion for the non-last traffic diversion job and must be set to Official release for the last traffic diversion job.	Retain the default value.
      VirtualService Name	Name of the Istio resource VirtualService of the application	service-demo-virtual-service
      DestinationRule Name	Name of the Istio resource DestinationRule of the application.	service-demo-destination-rule
      Grayscale Release Policy	Grayscale release policy that needs to be configured when Traffic Takeover is set to Traffic diversion: Based on traffic ratio: Traffic is diverted to the grayscale edge based on the request traffic ratio. Based on request content-Cookie: Traffic is diverted to the grayscale edge based on the cookie content of the request. Based on request content-Header: Traffic is diverted to the grayscale edge based on the request header.	Based on traffic ratio
      Grayscale Traffic Percentage (%)	When Grayscale Release Policy is set to Based on traffic ratio, configure the traffic ratio for traffic diverted to the grayscale edge.	20
      Cookie Content	When Grayscale Release Policy is set to Based on request content-Cookie, configure the cookie content for traffic diverted to the grayscale edge.	-
      Custom Header	When Grayscale Release Policy is set to Based on request content-Header, configure the request header content for traffic diverted to the grayscale edge.	-

   4. Click + under the job created in the previous step, select From Scratch, search for ManualReview, click Add, set parameters, and click OK. Table 2 describes the parameters.
   5. Click + under the job created in the previous step, select From Scratch, search for Traffic Diversion, click Add, set parameters, and click OK. For details about the parameters, see Table 3. (Set Traffic Diversion Type to Istio-based canary release and Traffic Takeover to Official release.)
      Figure 10 Traffic Diversion
      

   6. Click + under the job created in the previous step, select From Scratch, search for ManualReview, click Add, set parameters, and click OK. Table 2 describes the parameters.
   7. Click + under the job created in the previous step, select From Scratch, search for Bring Old Version Offline, click Add, set parameters, and click OK. Table 4 describes the parameters.
      Figure 11 Searching for "Bring Old Version Offline"
      
      Figure 12 Bring Old Version Offline
      

      Table 4 Parameter description

      Parameter	Description	Example
      Name	Name displayed in the pipeline job step.	Retain the default value.
      Step Name	Step name.	Retain the default value.

   8. Click Save and return in the upper right corner to complete the creation of the canary traffic diversion pipeline job.

Step 2: (Optional) Configure the Rollback Capability

To configure the rollback capability, perform the following steps to configure the rollback add-ons. The following figure shows the complete configuration process.

Figure 13 Configuring the rollback add-on

Scenarios where the rollback capability can be configured:

• Roll back to the status before deployment when all release steps are successfully executed.

  

• Roll back to the status before deployment when any release step fails.

  

• Roll back to the deployment status when any release step is successfully executed.

  

1. Edit the pipeline created in Step 1: Creating a New Stage-Level Application in the Pipeline.
2. Click + under Bring Old Version Offline, select From Scratch, search for ManualReview, and click Add.
3. After setting the parameters, click Job Configuration, as shown in Figure 14. Set Execute to Always (needs to run pre-task), as shown in Figure 15, and click OK. Table 2 describes the parameters.
   Figure 14 Job configuration
   
   Figure 15 ManualReview
   

4. Click + under the job created in the previous step, select From Scratch, search for Grayscale Release Rollback, and click Add.
   Figure 16 Grayscale Release Rollback
   

5. After setting the parameters, click Job Configuration, as shown in Figure 17. Set Execute to When previous job succeeds, as shown in Figure 18, and click OK.
   Figure 17 Job configuration
   
   Figure 18 Execute
   

Step 3: Execute the Pipeline

1. Execute the "Initiate Workload" step.

   Click in the pipeline operation area, select all created serial jobs, and click OK to start gray deployment. During the execution, each step can be verified.

   1. After the "Initiate Workload" step is successfully executed, check whether a new workload is generated in the namespace.
      Figure 19 Deployments
      
   2. After the step is executed, two groups of workloads (old and new) are generated in the namespace. Switch to the CCE console, locate the target Service (for example, service-demo), and click the button for associating instances in the Operation column. In the instance list, check whether the instances corresponding to the old and new workloads exist.
      Figure 20 Services

      

2. Perform the "Traffic Diversion" operation after the review is approved.
   1. Check whether traffic is forwarded to the workloads of the new and old versions as expected based on the configured traffic diversion policy on the live network.
   2. After the steps are complete, check whether the YAML configurations of the DestinationRule and VirtualService are the same as those shown in the figure below. If the v1 and v2 versions of the same API have different return results or log outputs, call the service access address to initiate a request and check whether the traffic distribution meets the configured proportion requirements based on the return value or log information of the API.
      Figure 21 DestinationRule YAML example
      
      Figure 22 VirtualService YAML Example
      

3. Release the grayscale version.
   1. After traffic is diverted based on the policy and the grayscale version is approved, release the grayscale version. Then, check whether all traffic is diverted to the upgraded version on the live network.
   2. After the preceding steps are complete, check the YAML configurations of DestinationRule and VirtualService. Ensure that the configurations are the same as those shown in the figure below. If the v1 and v2 versions of the same API have different return results or log output, access the service access address and check whether all traffic has been switched to the new workload based on the API return value or log information.
      Figure 23 DestinationRule YAML example
      
      Figure 24 VirtualService YAML Example
      

4. Bring the old version offline.
   1. After the new version is officially released and approved, bring the old version offline. You can check whether the service system has been updated to the target version on the live network.
      Figure 25 Workloads
      
      Figure 26 Services
      
   2. After the step is executed, check whether only the latest workload is retained in the namespace and the old workload has been deleted.

Step 4: (Optional) Perform a Rollback

You can perform a rollback when any step in Step 3: Execute the Pipeline is successfully executed or fails.

1. Roll back the pipeline step to the previous ManualReview of the rollback add-on, as shown in Figure 27.
   Figure 27 ManualReview
   

2. If a rollback is required, set Audit Operation to Pass. The system automatically runs the rollback add-on. If no rollback is required, set Audit Operation to Reject.
3. After the rollback add-on is executed, check the target namespace and ensure that only the original workload is retained.
4. Access the service address to verify that all traffic has been switched back to the original workload.