Help Center/Cloud Container Engine/Best Practices/Release/Using an Nginx Ingress for Grayscale Release and Blue-Green Deployment

Updated on 2026-03-10 GMT+08:00

Using an Nginx Ingress for Grayscale Release and Blue-Green Deployment

This section describes the scenarios and practices of using an Nginx ingress for grayscale release and blue-green deployment.

Application Scenarios

Nginx ingresses support traffic division policies based on the header, cookie, and service weight. Based on these policies, the following release scenarios can be implemented:

Scenario 1: Split some user traffic to the new version.
Assume that there is a Service A that provides Layer 7 networking. A new version is ready to go online, but you do not want to replace the original Service A. You want to forward the user requests whose header or cookie contains foo=bar to the new version of Service A. After the new version runs stably for a period of time, you can gradually bring the new version online and smoothly bring the old version offline. The figure below shows the process.
Scenario 2: Split a certain proportion of traffic to the new version.
Assume that there is a Service B that provides Layer 7 networking. After some problems are resolved, a new version of Service B needs to be released. However, you do not want to replace the original Service B. Instead, you want to switch 20% traffic to the new version of Service B. After the new version runs stably for a period of time, you can switch all traffic from the old version to the new version and smoothly bring the old version offline.

Annotations

Nginx ingresses support release and testing in different scenarios like grayscale release, blue-green deployment, and A/B testing by configuring annotations. The implementation works as follows: create two ingresses for a service, one regular ingress and another ingress annotated with nginx.ingress.kubernetes.io/canary: "true", referred to as the canary ingress. Configure traffic-splitting annotations for the canary ingress. The two ingresses cooperate with each other to implement release and testing in multiple scenarios. The Nginx ingress annotations support the following rules:

nginx.ingress.kubernetes.io/canary-by-header
Header-based traffic division, which is applicable to grayscale release. If a request header contains the specified header name and the value is always, the request is forwarded to the backend service defined by the canary ingress. If the value is never, the request is not forwarded and a rollback to the source version will be performed. If other values are used, the annotation will be ignored and the request traffic will be forwarded according to other rules based on the priority.
nginx.ingress.kubernetes.io/canary-by-header-value
This rule must be used together with canary-by-header. You can customize the value of a request header, including but not limited to always or never. If the value of the request header matches the specified custom value, the request will be forwarded to the corresponding backend service defined by the canary ingress. If the values do not match, the annotation will be ignored and the request traffic will be forwarded according to other rules based on the priority.
nginx.ingress.kubernetes.io/canary-by-header-pattern
This rule is similar to canary-by-header-value. The only difference is that this annotation uses a regular expression, not a fixed value, to match the value of a request header. If this annotation and canary-by-header-value exist in parallel, this one will be ignored.
nginx.ingress.kubernetes.io/canary-by-cookie
Cookie-based traffic division, which is applicable to grayscale release. Similar to canary-by-header, this annotation is used for cookies. Only always and never are supported, and the value cannot be customized.
nginx.ingress.kubernetes.io/canary-weight
Traffic is divided based on service weights. This approach is applicable to blue-green deployment. This annotation indicates the percentage of traffic allocated by the canary ingress. The value ranges from 0 to 100. For example, if the value is set to 100, all traffic will be forwarded to the backend service of the canary ingress.

The preceding annotation rules are evaluated based on the priority. The priority is as follows: canary-by-header -> canary-by-cookie -> canary-weight.
When an ingress is marked as a canary ingress, all non-canary annotations except nginx.ingress.kubernetes.io/load-balance and nginx.ingress.kubernetes.io/upstream-hash-by are ignored.
For details, see Annotations.

Prerequisites

To use an Nginx ingress for grayscale release in a cluster, install the NGINX Ingress Controller add-on and expose a unified traffic entry externally. For details, see NGINX Ingress Controller.
The Nginx image has been uploaded to SWR. There are two versions of the Nginx image. The welcome pages are Old Nginx and New Nginx.

Resource Creation

You can use YAML to deploy Deployments and Services in either of the following ways:

On the Create Deployment page, click Create from YAML on the right and edit the YAML file in the sliding window.
Save the sample YAML file in this section as a file and use kubectl to specify the YAML file. For example, run the kubectl create -f xxx.yaml command.

Step 1: Deploy Services of Two Versions

Two versions of Nginx are deployed in the cluster, and an Nginx ingress is used to provide Layer 7 domain name access for external systems.

Create a Deployment and Service for the first version. old-nginx is used as an example. Example YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: old-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: old-nginx
  template:
    metadata:
      labels:
        app: old-nginx
    spec:
      containers:
      - image: {your_repository}/nginx:old  # The image used by the container is nginx:old.
        name: container-0
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
      imagePullSecrets:
      - name: default-secret

---

apiVersion: v1
kind: Service
metadata:
  name: old-nginx
spec:
  selector:
    app: old-nginx
  ports:
  - name: service0
    targetPort: 80
    port: 8080
    protocol: TCP
  type: NodePort

Create a Deployment and Service for the second version. new-nginx is used as an example. Example YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: new-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: new-nginx
  template:
    metadata:
      labels:
        app: new-nginx
    spec:
      containers:
      - image: {your_repository}/nginx:new  # The image used by the container is nginx:new.
        name: container-0
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
      imagePullSecrets:
      - name: default-secret

---

apiVersion: v1
kind: Service
metadata:
  name: new-nginx
spec:
  selector:
    app: new-nginx
  ports:
  - name: service0
    targetPort: 80
    port: 8080
    protocol: TCP
  type: NodePort

You can log in to the CCE console to view the deployment status.

Create an ingress to expose the service and point to the service of the old version. Example YAML:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release
  namespace: default
  annotations:
    kubernetes.io/elb.port: '80'
spec:
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            backend:
              service:
                name: old-nginx      # Set the backend service to old-nginx.
                port:
                  number: 80
            property:
              ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
            pathType: ImplementationSpecific
  ingressClassName: nginx   # Nginx ingress is used.

Test the connectivity.
```
curl -H "Host: www.example.com"  http://<EXTERNAL_IP>
```
In the command, <EXTERNAL_IP> indicates the external IP address of an Nginx ingress.
Expected outputs:
```
Old Nginx
```

Step 2: Launch the New Version of the Service in Grayscale Release Mode

Set the traffic division policy for the service of the new version. CCE supports the following policies for grayscale release and blue-green deployment:

Header-based, cookie-based, and weight-based traffic division

Grayscale release can be implemented based on all these policies. Blue-green deployment can be implemented by adjusting the new service weight to 100%. For details, see the examples below.

Pay attention to the following:

Only one canary ingress can be defined for the same service so that the backend service supports a maximum of two versions.
Even if the traffic is completely switched to the canary ingress, the old version service must still exist. Otherwise, an error is reported.

Header-based rules

In this example, only the request whose header contains Region set to bj or gz can be forwarded to the service of the new version.

Create a canary ingress, set the backend service to the one of the new version, and add annotations.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"                       # Enable canary.
    nginx.ingress.kubernetes.io/canary-by-header: "Region"
    nginx.ingress.kubernetes.io/canary-by-header-pattern: "bj|gz"    # Requests whose header contains Region with the value bj or gz are forwarded to the canary ingress.
    kubernetes.io/elb.port: '80'
spec:
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            backend:
              service:
                name: new-nginx      # Set the backend service to new-nginx.
                port:
                  number: 80
            property:
              ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
            pathType: ImplementationSpecific
  ingressClassName: nginx   # Nginx ingress is used.

Test the connectivity.

$ curl -H "Host: www.example.com" -H "Region: bj" http://<EXTERNAL_IP>
New Nginx
$ curl -H "Host: www.example.com" -H "Region: sh" http://<EXTERNAL_IP>
Old Nginx
$ curl -H "Host: www.example.com" -H "Region: gz" http://<EXTERNAL_IP>
New Nginx
$ curl -H "Host: www.example.com" http://<EXTERNAL_IP>
Old Nginx

In the command, <EXTERNAL_IP> indicates the external IP address of an Nginx ingress.

The service of the new version responds only to requests whose header contains Region with the value bj or gz.

Cookie-based rules

In this example, only the requests whose cookie contains user_from_bj can be forwarded to the service of the new version.

Create a canary ingress, set the backend service to the one of the new versions, and add annotations.

If you have created a canary ingress in the preceding steps, delete it and then perform this step to create a canary ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"                       # Enable canary.
    nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_bj"    # Requests whose cookie contains user_from_bj are forwarded to the canary ingress.
    kubernetes.io/elb.port: '80'
spec:
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            backend:
              service:
                name: new-nginx      # Set the backend service to new-nginx.
                port:
                  number: 80
            property:
              ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
            pathType: ImplementationSpecific
  ingressClassName: nginx   # An Nginx ingress is used.

Test the connectivity.

$ curl -s -H "Host: www.example.com" --cookie "user_from_bj=always" http://<EXTERNAL_IP>
New Nginx
$ curl -s -H "Host: www.example.com" --cookie "user_from_gz=always" http://<EXTERNAL_IP>
Old Nginx
$ curl -s -H "Host: www.example.com" http://<EXTERNAL_IP>
Old Nginx

In the command, <EXTERNAL_IP> indicates the external IP address of an Nginx ingress.

The service of the new version responds only to the requests whose cookie contains user_from_bj with the value always.

Service weight-based rules

Example 1: Only 20% of the traffic is allowed to be forwarded to the service of the new version for a grayscale release.

Create a canary ingress and add annotations to import 20% of the traffic to the backend service of the new version.

If you have created a canary ingress in the preceding steps, delete it and then perform this step to create a canary ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"         # Enable canary.
    nginx.ingress.kubernetes.io/canary-weight: "20"    # Forward 20% of the traffic to the canary ingress.
    kubernetes.io/elb.port: '80'
spec:
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            backend:
              service:
                name: new-nginx      # Set the backend service to new-nginx.
                port:
                  number: 80
            property:
              ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
            pathType: ImplementationSpecific
  ingressClassName: nginx   # Nginx ingress is used.

Test the connectivity.
```
$ for i in {1..20}; do curl -H "Host: www.example.com" http://<EXTERNAL_IP>; done;
Old Nginx
Old Nginx
Old Nginx
New Nginx
Old Nginx
New Nginx
Old Nginx
New Nginx
Old Nginx
Old Nginx
Old Nginx
Old Nginx
Old Nginx
New Nginx
Old Nginx
Old Nginx
Old Nginx
Old Nginx
Old Nginx
Old Nginx
```
In the command, <EXTERNAL_IP> indicates the external IP address of an Nginx ingress.
It can be seen that there is a 4/20 probability that the service of the new version responds, which complies with the setting of the service weight of 20%.
After traffic is divided based on the weight (20%), the probability of accessing the new version is close to 20%. The traffic ratio may fluctuate within a small range, which is normal.

Example 2: Allow all traffic to be forwarded to the service of the new version for blue-green deployment.

Create a canary ingress and add annotations to import 100% of the traffic to the backend service of the new version.

If you have created a canary ingress in the preceding steps, delete it and then perform this step to create a canary ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"          # Enable canary.
    nginx.ingress.kubernetes.io/canary-weight: "100"    # All traffic is forwarded to the canary ingress.
    kubernetes.io/elb.port: '80'
spec:
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            backend:
              service:
                name: new-nginx      # Set the backend service to new-nginx.
                port:
                  number: 80
            property:
              ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
            pathType: ImplementationSpecific
  ingressClassName: nginx   # Nginx ingress is used.

Test the connectivity.
```
$ for i in {1..10}; do curl -H "Host: www.example.com" http://<EXTERNAL_IP>; done;
New Nginx
New Nginx
New Nginx
New Nginx
New Nginx
New Nginx
New Nginx
New Nginx
New Nginx
New Nginx
```
In the command, <EXTERNAL_IP> indicates the external IP address of an Nginx ingress.
The service of the new version responds to all requests. This means that the blue-green deployment has been implemented.

Parent Topic: Release

Previous topic: Using Services to Implement Simple Grayscale Release and Blue-Green Deployment

Next topic: Batch Computing

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

For any further questions, feel free to contact us through the chatbot.

Chatbot