Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
Help Center/ Cloud Container Engine/ Best Practices/ Release/ Using Nginx Ingress to Implement Grayscale Release and Blue-Green Deployment

Using Nginx Ingress to Implement Grayscale Release and Blue-Green Deployment

Updated on 2025-01-08 GMT+08:00

This section describes the scenarios and practices of using Nginx Ingress to implement grayscale release and blue-green deployment.

Application Scenarios

Nginx Ingress supports three traffic division policies based on the header, cookie, and service weight. Based on these policies, the following two release scenarios can be implemented:

  • Scenario 1: Split some user traffic to the new version.

    Assume that Service A that provides layer-7 networking is running. A new version is ready to go online, but you do not want to replace the original Service A. You want to forward the user requests whose header or cookie contains foo=bar to the new version of Service A. After the new version runs stably for a period of time, you can gradually bring the new version online and smoothly bring the old version offline. The following figure shows the process:

  • Scenario 2: Split a certain proportion of traffic to the new version.

    Assume that Service B that provides layer-7 services is running. After some problems are resolved, a new version of Service B needs to be released. However, you do not want to replace the original Service B. Instead, you want to switch 20% traffic to the new version of Service B. After the new version runs stably for a period of time, you can switch all traffic from the old version to the new version and smoothly bring the old version offline.

Annotations

Nginx Ingress supports release and testing in different scenarios by configuring annotations for grayscale release, blue-green deployment, and A/B testing. The implementation process is as follows: Create two ingresses for the service. One is a common ingress, and the other is an ingress with the annotation nginx.ingress.kubernetes.io/canary: "true", which is called a canary ingress. Configure a traffic division policy for the canary ingress. The two ingresses cooperate with each other to implement release and testing in multiple scenarios. The annotation of Nginx Ingress supports the following rules:

  • nginx.ingress.kubernetes.io/canary-by-header

    Header-based traffic division, which is applicable to grayscale release. If the request header contains the specified header name and the value is always, the request is forwarded to the backend service defined by the canary ingress. If the value is never, the request is not forwarded and a rollback to the source version can be performed. If other values are used, the annotation is ignored and the request traffic is allocated according to other rules based on the priority.

  • nginx.ingress.kubernetes.io/canary-by-header-value

    This rule must be used together with canary-by-header. You can customize the value of the request header, including but not limited to always or never. If the value of the request header matches the specified custom value, the request is forwarded to the corresponding backend service defined by the canary ingress. If the values do not match, the annotation is ignored and the request traffic is allocated according to other rules based on the priority.

  • nginx.ingress.kubernetes.io/canary-by-header-pattern

    This rule is similar to canary-by-header-value. The only difference is that this annotation uses a regular expression, not a fixed value, to match the value of the request header. If this annotation and canary-by-header-value exist at the same time, this one will be ignored.

  • nginx.ingress.kubernetes.io/canary-by-cookie

    Cookie-based traffic division, which is applicable to grayscale release. Similar to canary-by-header, this annotation is used for cookies. Only always and never are supported, and the value cannot be customized.

  • nginx.ingress.kubernetes.io/canary-weight

    Traffic is divided based on service weights, which is applicable to blue-green deployment. This annotation indicates the percentage of traffic allocated by the canary ingress. The value ranges from 0 to 100. For example, if the value is set to 100, all traffic is forwarded to the backend service backing the canary ingress.

NOTE:
  • The preceding annotation rules are evaluated based on the priority. The priority is as follows: canary-by-header -> canary-by-cookie -> canary-weight.
  • When an ingress is marked as a canary ingress, all non-canary annotations except nginx.ingress.kubernetes.io/load-balance and nginx.ingress.kubernetes.io/upstream-hash-by are ignored.
  • For more information, see Annotations.

Prerequisites

  • To use Nginx Ingress to implement grayscale release of a cluster, install the nginx-ingress add-on as the Ingress Controller and expose a unified traffic entrance externally. For details, see Installing the Add-on.
  • The Nginx image has been uploaded to SWR. The Nginx images have two versions. The welcome pages are Old Nginx and New Nginx.

Resource Creation

You can use YAML to deploy Deployments and Services in either of the following ways:

  • On the Create Deployment page, click Create YAML on the right and edit the YAML file in the window.
  • Save the sample YAML file in this section as a file and use kubectl to specify the YAML file. For example, run the kubectl create -f xxx.yaml command.

Step 1: Deploy Services of Two Versions

Two versions of Nginx are deployed in the cluster, and Nginx Ingress is used to provide layer-7 domain name access for external systems.

  1. Create a Deployment and Service for the first version. This section uses old-nginx as an example. Example YAML:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: old-nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: old-nginx
      template:
        metadata:
          labels:
            app: old-nginx
        spec:
          containers:
          - image: {your_repository}/nginx:old  # The image used by the container is nginx:old.
            name: container-0
            resources:
              limits:
                cpu: 100m
                memory: 200Mi
              requests:
                cpu: 100m
                memory: 200Mi
          imagePullSecrets:
          - name: default-secret
    
    ---
    
    apiVersion: v1
    kind: Service
    metadata:
      name: old-nginx
    spec:
      selector:
        app: old-nginx
      ports:
      - name: service0
        targetPort: 80
        port: 8080
        protocol: TCP
      type: NodePort

  2. Create a Deployment and Service for the second version. This section uses new-nginx as an example. Example YAML:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: new-nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: new-nginx
      template:
        metadata:
          labels:
            app: new-nginx
        spec:
          containers:
          - image: {your_repository}/nginx:new  # The image used by the container is nginx:new.
            name: container-0
            resources:
              limits:
                cpu: 100m
                memory: 200Mi
              requests:
                cpu: 100m
                memory: 200Mi
          imagePullSecrets:
          - name: default-secret
    
    ---
    
    apiVersion: v1
    kind: Service
    metadata:
      name: new-nginx
    spec:
      selector:
        app: new-nginx
      ports:
      - name: service0
        targetPort: 80
        port: 8080
        protocol: TCP
      type: NodePort

    You can log in to the CCE console to view the deployment status.

  3. Create an ingress to expose the service and point to the service of the old version. Example YAML:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: gray-release
      namespace: default
      annotations:
        kubernetes.io/elb.port: '80'
    spec:
      rules:
        - host: www.example.com
          http:
            paths:
              - path: /
                backend:
                  service:
                    name: old-nginx      # Set the back-end service to old-nginx.
                    port:
                      number: 80
                property:
                  ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
                pathType: ImplementationSpecific
      ingressClassName: nginx   # Nginx ingress is used.

  4. Run the following command to verify the access:

    curl -H "Host: www.example.com"  http://<EXTERNAL_IP>

    In the preceding command, <EXTERNAL_IP> indicates the external IP address of the Nginx ingress.

    Expected outputs:

    Old Nginx

Step 2: Launch the New Version of the Service in Grayscale Release Mode

Set the traffic division policy for the service of the new version. CCE supports the following policies for grayscale release and blue-green deployment:

Header-based, cookie-based, and weight-based traffic division rules

Grayscale release can be implemented based on all these policies. Blue-green deployment can be implemented by adjusting the new service weight to 100%. For details, see the following examples.

CAUTION:

Pay attention to the following:

  • Only one canary ingress can be defined for the same service so that the backend service supports a maximum of two versions.
  • Even if the traffic is completely switched to the canary ingress, the old version service must still exist. Otherwise, an error is reported.
  • Header-based rules

    In the following example, only the request whose header contains Region set to bj or gz can be forwarded to the service of the new version.

    1. Create a canary ingress, set the backend service to the one of the new versions, and add annotations.
      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: canary-ingress
        namespace: default
        annotations:
          nginx.ingress.kubernetes.io/canary: "true"                       # Enable canary.
          nginx.ingress.kubernetes.io/canary-by-header: "Region"
          nginx.ingress.kubernetes.io/canary-by-header-pattern: "bj|gz"    # Requests whose header contains Region with the value bj or gz are forwarded to the canary ingress.
          kubernetes.io/elb.port: '80'
      spec:
        rules:
          - host: www.example.com
            http:
              paths:
                - path: /
                  backend:
                    service:
                      name: new-nginx      # Set the back-end service to new-nginx.
                      port:
                        number: 80
                  property:
                    ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
                  pathType: ImplementationSpecific
        ingressClassName: nginx   # Nginx ingress is used.
    2. Run the following command to test the access:
      $ curl -H "Host: www.example.com" -H "Region: bj" http://<EXTERNAL_IP>
      New Nginx
      $ curl -H "Host: www.example.com" -H "Region: sh" http://<EXTERNAL_IP>
      Old Nginx
      $ curl -H "Host: www.example.com" -H "Region: gz" http://<EXTERNAL_IP>
      New Nginx
      $ curl -H "Host: www.example.com" http://<EXTERNAL_IP>
      Old Nginx

      In the preceding command, <EXTERNAL_IP> indicates the external IP address of the Nginx ingress.

      Only requests whose header contains Region with the value bj or gz are responded by the service of the new version.

  • Cookie-based rules

    In the following example, only the request whose cookie contains user_from_bj can be forwarded to the service of the new version.

    1. Create a canary ingress, set the backend service to the one of the new versions, and add annotations.
      NOTE:

      If you have created a canary ingress in the preceding steps, delete it and then perform this step to create a canary ingress.

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: canary-ingress
        namespace: default
        annotations:
          nginx.ingress.kubernetes.io/canary: "true"                       # Enable canary.
          nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_bj"    # Requests whose cookie contains user_from_bj are forwarded to the canary ingress.
          kubernetes.io/elb.port: '80'
      spec:
        rules:
          - host: www.example.com
            http:
              paths:
                - path: /
                  backend:
                    service:
                      name: new-nginx      # Set the back-end service to new-nginx.
                      port:
                        number: 80
                  property:
                    ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
                  pathType: ImplementationSpecific
        ingressClassName: nginx   # Nginx ingress is used.
    2. Run the following command to test the access:
      $ curl -s -H "Host: www.example.com" --cookie "user_from_bj=always" http://<EXTERNAL_IP>
      New Nginx
      $ curl -s -H "Host: www.example.com" --cookie "user_from_gz=always" http://<EXTERNAL_IP>
      Old Nginx
      $ curl -s -H "Host: www.example.com" http://<EXTERNAL_IP>
      Old Nginx

      In the preceding command, <EXTERNAL_IP> indicates the external IP address of the Nginx ingress.

      Only requests whose cookie contains user_from_bj with the value always are responded by the service of the new version.

  • Service weight-based rules

    Example 1: Only 20% of the traffic is allowed to be forwarded to the service of the new version to implement grayscale release.

    1. Create a canary ingress and add annotations to import 20% of the traffic to the backend service of the new version.
      NOTE:

      If you have created a canary ingress in the preceding steps, delete it and then perform this step to create a canary ingress.

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: canary-ingress
        namespace: default
        annotations:
          nginx.ingress.kubernetes.io/canary: "true"         # Enable canary.
          nginx.ingress.kubernetes.io/canary-weight: "20"    # Forward 20% of the traffic to the canary ingress.
          kubernetes.io/elb.port: '80'
      spec:
        rules:
          - host: www.example.com
            http:
              paths:
                - path: /
                  backend:
                    service:
                      name: new-nginx      # Set the back-end service to new-nginx.
                      port:
                        number: 80
                  property:
                    ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
                  pathType: ImplementationSpecific
        ingressClassName: nginx   # Nginx ingress is used.
    2. Run the following command to test the access:
      $ for i in {1..20}; do curl -H "Host: www.example.com" http://<EXTERNAL_IP>; done;
      Old Nginx
      Old Nginx
      Old Nginx
      New Nginx
      Old Nginx
      New Nginx
      Old Nginx
      New Nginx
      Old Nginx
      Old Nginx
      Old Nginx
      Old Nginx
      Old Nginx
      New Nginx
      Old Nginx
      Old Nginx
      Old Nginx
      Old Nginx
      Old Nginx
      Old Nginx

      In the preceding command, <EXTERNAL_IP> indicates the external IP address of the Nginx ingress.

      It can be seen that there is a 4/20 probability that the service of the new version responds, which complies with the setting of the service weight of 20%.

      NOTE:

      After traffic is divided based on the weight (20%), the probability of accessing the new version is close to 20%. The traffic ratio may fluctuate within a small range, which is normal.

    Example 2: Allow all traffic to be forwarded to the service of the new version to implement blue-green deployment.

    1. Create a canary ingress and add annotations to import 100% of the traffic to the backend service of the new version.
      NOTE:

      If you have created a canary ingress in the preceding steps, delete it and then perform this step to create a canary ingress.

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: canary-ingress
        namespace: default
        annotations:
          nginx.ingress.kubernetes.io/canary: "true"          # Enable canary.
          nginx.ingress.kubernetes.io/canary-weight: "100"    # All traffic is forwarded to the canary ingress.
          kubernetes.io/elb.port: '80'
      spec:
        rules:
          - host: www.example.com
            http:
              paths:
                - path: /
                  backend:
                    service:
                      name: new-nginx      # Set the back-end service to new-nginx.
                      port:
                        number: 80
                  property:
                    ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
                  pathType: ImplementationSpecific
        ingressClassName: nginx   # Nginx ingress is used.
    2. Run the following command to test the access:
      $ for i in {1..10}; do curl -H "Host: www.example.com" http://<EXTERNAL_IP>; done;
      New Nginx
      New Nginx
      New Nginx
      New Nginx
      New Nginx
      New Nginx
      New Nginx
      New Nginx
      New Nginx
      New Nginx

      In the preceding command, <EXTERNAL_IP> indicates the external IP address of the Nginx ingress.

      All access requests are responded by the service of the new version, and the blue-green deployment is successfully implemented.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback