Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Configuring Grayscale Release for a LoadBalancer Ingress

Updated on 2025-02-18 GMT+08:00

Dedicated LoadBalancer ingresses support grayscale release by:

  • Proportion
  • HTTP request header
  • Cookie
NOTE:

ELB is necessary for grayscale release of ingresses. To use grayscale release, submit a service ticket to request the enabling of ELB grayscale release.

Prerequisites

  • A CCE standard or Turbo cluster is available, and the cluster version meets the following requirements:
    • v1.23: v1.23.14-r0 or later
    • v1.25: v1.25.9-r0 or later
    • v1.27: v1.27.6-r0 or later
    • v1.28: v1.28.4-r0 or later
    • Other clusters of later versions

Notes and Constraints

  • After a grayscale ingress is created, do not delete the original ingress.
  • When configuring multiple grayscale release policies for ingresses associated with a load balancer listener, the policy based on HTTP request headers takes the highest priority. The cookie-based policy has the second highest priority. The proportion-based policy has the lowest priority.
  • This feature relies on the ELB advanced forwarding policies. Once enabled, the priority is not determined by the domain name or path matching. You can customize the forwarding policy priority as needed. For details about the forwarding policy priority, see Forwarding Policy Priorities of LoadBalancer Ingresses.

Using the CCE Console

  1. Log in to the CCE console and click the cluster name to access the cluster console.
  2. Deploy the original service.

    1. In the navigation pane, choose Workloads. Then, click Create Workload in the upper right corner and deploy a workload named origin-server. For details about parameter settings, see Creating a Deployment.
    2. In the navigation pane, choose Services & Ingresses. On the Services tab page, click Create Service in the upper right corner. Create a Service named origin-service and associate it with the created origin-server workload. For a CCE standard cluster, create a NodePort Service, and for a CCE Turbo cluster, create a ClusterIP Service.
    3. In the navigation pane, choose Services & Ingresses. Click the Ingresses tab page and then Create Ingress in the upper right corner. Create an ingress named origin-ingress and associate it with the created origin-service Service. For details about parameter settings, see Creating a LoadBalancer Ingress on the Console.

  3. Grayscale release a new version for the Service.

    1. In the navigation pane, choose Services & Ingresses, switch to the Ingresses tab, locate the row containing the ingress for which you want to configure grayscale release, and choose More > Create Grayscale Release in the Operation column.
    2. Configure grayscale release parameters.

      Table 1 Key parameters

      Parameter

      Description

      Example

      Gray release

      • HTTP request header: When the header key-value pair in an HTTP request is matched, the grayscale release service can be accessed.
        • If domain names and paths are configured for an ingress forwarding policy, a maximum of eight values can be configured.
        • If only paths are configured for an ingress forwarding policy, a maximum of nine values can be configured.
      • Cookie: When the cookie key-value pair in an HTTP request is matched, the grayscale release service can be accessed.
      • Proportion: percentage of the requests for accessing the grayscale release service to the total number of requests

      HTTP request header

      • Key: a
      • Value: b

      Forwarding Policy

      • Domain Name: Enter an actual domain name to be accessed. If it is left blank, the ingress can be accessed through the IP address. Ensure that the domain name has been registered and licensed. Once a forwarding policy is configured with a domain name specified, you must use the domain name for access.
      • Grayscale Release: After grayscale release is enabled for a domain name, external access traffic of the cluster will be forwarded to the target Service using the selected traffic policy.
      • Traffic Policy: Select an option from HTTP request header, Cookie, and Proportion as needed. (The percentage of a grayscale request is 0%.)
      • Destination Service: Select an existing Service or create a Service. Any Services that do not match the search criteria will be filtered out automatically.
      • Destination Service Port: Select the access port of the destination Service.
      • Domain Name: You do not need to configure this parameter.
      • Grayscale Release: enabled
      • Traffic Policy: HTTP request header
      • Destination Service: nginx
      • Destination Service Port: 80
    3. Click OK.
    4. Check whether the new version is grayscale released.

  4. Terminate the old-version Service.

    After the Service of the new version runs stably, terminate the Service of the old version. During the Service termination, change the original Service to the new-version Service so that all traffic will be routed to the new-version Service. Then, delete the grayscale released ingress.

Using kubectl

  1. Deploy the original service.

    1. Deploy a workload named origin-server.
    2. Deploy a Service named origin-service and associate it with the created origin-server workload. (Create a NodePort Service in a CCE standard cluster or a ClusterIP Service in a CCE Turbo cluster.)
    3. Deploy an ingress named origin-ingress and associate it with the created origin-service Service.
      Configure the ingress as follows:
      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: origin-ingress
        namespace: default
        annotations:
          kubernetes.io/elb.port: '81'
          kubernetes.io/elb.id: e491f4e7-2351-4984-ad0a-8569e5e964a3
          kubernetes.io/elb.class: performance
      spec:
        rules:
          - host: example.com
            http:
              paths:
                - path: /
                  backend:
                    service:
                      name: origin-service
                      port:
                        number: 81
                  property:
                    ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
                  pathType: ImplementationSpecific
        ingressClassName: cce

  2. Grayscale release a new version for the Service.

    Deploy the workload, Service, and ingress of a new version so that traffic can be routed to the Service of the new version by weight or header.

    1. Deploy a workload named canary-server.
    2. Deploy a Service named canary-service and associate it with the created canary-server workload. (Create a NodePort Service in a CCE standard cluster or a ClusterIP Service in a CCE Turbo cluster.)
    3. Create a grayscale released ingress. For details about parameter settings, see Parameters.
      1. Scenario 1, by HTTP request header: When the header key-value pair in an HTTP request is matched, the grayscale released Service is accessed.

        Deploy an ingress named canary-header-ingress and associate it with the created canary-service Service for grayscale release by header.

        Configure the ingress as follows:
        apiVersion: networking.k8s.io/v1
        kind: Ingress
        metadata:
          name: canary-header-ingress
          namespace: default
          annotations:
            kubernetes.io/elb.canary: 'true'                        # Set this parameter to true, indicating that canary annotation is enabled.
            kubernetes.io/elb.canary-by-header: 'location'          # Set the header key to location.
            kubernetes.io/elb.canary-by-header-value: '{"values":["hz","sz","sh"]}'  # Set the header values to hz, sz, and sh.
            kubernetes.io/elb.class: performance         # Only dedicated load balancers support grayscale release.
            kubernetes.io/elb.id: e491f4e7-2351-4984-ad0a-8569e5e964a3
            kubernetes.io/elb.port: '81'
        spec:
          ingressClassName: cce
          rules:
            - host: example.com
              http:
                paths:
                  - path: /
                    pathType: ImplementationSpecific
                    backend:
                      service:
                        name: canary-service
                        port:
                          number: 80
                    property:
                      ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
      2. Scenario 2: by proportion: Configure the proportion of requests for accessing the grayscale released Service.

        Deploy an ingress named canary-weight-ingress and associate it with the created canary-service Service for grayscale release by weight.

        Configure the ingress as follows:
        apiVersion: networking.k8s.io/v1
        kind: Ingress
        metadata:
          name: canary-weight-ingress
          namespace: default
          annotations:
            kubernetes.io/elb.canary: 'true'             # Set this parameter to true, indicating that canary annotation is enabled.
            kubernetes.io/elb.canary-weight: '40'        # Configure a weight, indicating that 40% of the request traffic will be routed to the Service of the new version.
            kubernetes.io/elb.class: performance         # Only dedicated load balancers support grayscale release.
            kubernetes.io/elb.id: e491f4e7-2351-4984-ad0a-8569e5e964a3
            kubernetes.io/elb.port: '81'
        spec:
          ingressClassName: cce
          rules:
            - host: example.com
              http:
                paths:
                  - path: /
                    pathType: ImplementationSpecific
                    backend:
                      service:
                        name: canary-service
                        port:
                          number: 81
                    property:
                      ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
      3. Scenario 3, by cookie: When the cookie key-value pair in a request is matched, the grayscale released Service is accessed.

        Deploy an ingress named canary-cookie-ingress and associate it with the created canary-service Service for grayscale release by cookie.

        Configure the ingress as follows:
        apiVersion: networking.k8s.io/v1
        kind: Ingress
        metadata:
          name: canary-cookie-ingress
          namespace: default
          annotations:
            kubernetes.io/elb.canary: 'true'                   # Set this parameter to true, indicating that canary annotation is enabled.
            kubernetes.io/elb.canary-by-cookie: 'location'     # Set the header key to location.
            kubernetes.io/elb.canary-by-cookie-value: '{"values":["hz","sz","sh"]}'  // Set the header values to hz, sz, and sh.
            kubernetes.io/elb.class: performance                  # Only dedicated load balancers support grayscale release.
            kubernetes.io/elb.id: e491f4e7-2351-4984-ad0a-8569e5e964a3
            kubernetes.io/elb.port: '81'
        spec:
          ingressClassName: cce
          rules:
            - host: example.com
              http:
                paths:
                  - path: /
                    pathType: ImplementationSpecific
                    backend:
                      service:
                        name: canary-service
                        port:
                          number: 81
                    property:
                      ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
    4. Check whether the new version is grayscale released.
      1. Check the creation of the ingress.
        $ kubectl get ingress
        NAME                    CLASS   HOSTS        ADDRESS        PORTS   AGE
        canary-cookie-ingress   cce     example.com   88.88.88.165   80      16s
        canary-header-ingress   cce     example.com   88.88.88.165   80      46s
        canary-weight-ingress   cce     example.com   88.88.88.165   80      117s
        origin-ingress          cce     example.com   88.88.88.165   80      2m25s
      2. Check ELB rules.

      3. Check the result of a service request.
        1. Use the location: hz request header to access the Service.
          $ curl -H "Host: example.com" -H "location: hz" http://88.88.88.165:81/
          Expected result:
          $ new

          Attempt the access for multiple times. All the returned results are new.

        2. Access the Service without using a request header.
          $ curl -H "Host: example.com" http://88.88.88.165:81/

          Expected result: 40% for new, and 60% for old

  3. Terminate the old-version Service.

    After the Service of the new version runs stably, terminate the Service of the old version. During the Service termination, change the original Service to the new-version Service so that all traffic will be routed to the new-version Service. Then, delete the grayscale released ingress.

    1. Change the original Service to the new-version Service.
      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: origin-ingress
        namespace: default
        annotations:
          kubernetes.io/elb.port: '81'
          kubernetes.io/elb.id: e491f4e7-2351-4984-ad0a-8569e5e964a3
          kubernetes.io/elb.class: performance
      spec:
        rules:
          - host: example.com
            http:
              paths:
                - path: /
                  backend:
                    service:
                      name: canary-service
                      port:
                        number: 81
                  property:
                    ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
                  pathType: ImplementationSpecific
        ingressClassName: cce
    2. Check whether the old-version Service has been terminated.
      1. Initiate a request with a header and a request without a header to access the Service.
        $ curl -H "Host: example.com" -H "location: hz" http://88.88.88.165:81/
        $ curl -H "Host: example.com" http://88.88.88.165:81/
      2. Expected result: All results are returned from the new-version Service.
    3. Delete the grayscale released ingress.
      $ kubectl delete ingress canary-weight-ingress canary-header-ingress

Parameters

Table 2 Parameters for grayscale release

Parameter

Mandatory

Type

Description

kubernetes.io/elb.canary

No

String

Grayscale release status for an ingress. If this parameter is set to true, the implementation of grayscale release varies depending on annotation configurations.

Option: true

  • Grayscale release applies only to dedicated load balancers.
  • If this parameter is set to true, the configuration cannot be deleted or modified.

kubernetes.io/elb.canary-weight

No

String

Weight of a grayscale release. After this parameter is configured, the ingress will be grayscale released based on the weight.

  • The value is a positive integer ranging from 0 to 100. It is a percentage of traffic for routing.
  • This parameter is mandatory when an ingress is grayscale released by weight.
  • Do not configure this parameter with other grayscale release functions.

kubernetes.io/elb.session-affinity-mode

No

String

After weight-based grayscale release is enabled, configure sticky session.

This parameter can only be set to HTTP_COOKIE for grayscale release.

kubernetes.io/

elb.session-affinity-option

No

String

Sticky session timeout after sticky session is enabled for weight-based grayscale release.

The parameter value is a JSON string in the following format:

{"persistence_timeout": "1440"}

Parameters:

  • Timeout range: 1 to 1440
  • Default value: 1440

kubernetes.io/elb.canary-by-header

No

String

Key of an ingress header used for grayscale release, indicating the name of a request header. This parameter must be used with kubernetes.io/elb.canary-by-header-value.

Parameters:

Enter 1 to 40 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.

kubernetes.io/elb.canary-by-header-value

No

String

Value of an ingress header used for grayscale release. This parameter must be used with kubernetes.io/elb.canary-by-header.

The parameter value is an array in JSON format, for example:

'{"values":["a","b"]}'

Parameters:

  • Array length: At least one value must be configured.
    • If domain names and paths are configured for an ingress forwarding policy, a maximum of eight values can be configured.
    • If only paths are configured for an ingress forwarding policy, a maximum of nine values can be configured.
  • Enter 1 to 128 characters. Asterisks (*) and question marks (?) are allowed, but spaces and double quotation marks are not allowed. An asterisk can match zero or more characters, and a question mark can match one character.

kubernetes.io/elb.canary-by-cookie

No

String

Key of cookie-based grayscale release, indicating the name of a request cookie. This parameter must be used with kubernetes.io/elb.canary-by-cookie-value.

Parameters:

Enter 1 to 100 characters, including letters, digits, and other characters (!%'"()*+,./:=?@^\\-_`~).

kubernetes.io/elb.canary-by-cookie-value

No

String

Value of cookie-based grayscale release. This parameter must be used with kubernetes.io/elb.canary-by-cookie.

The parameter value is an array in JSON format, for example:

'{"values":["a","b"]}'

Parameters:

  • Array length: At least one value must be configured.
    • If domain names and paths are configured for an ingress forwarding policy, a maximum of eight values can be configured.
    • If only paths are configured for an ingress forwarding policy, a maximum of nine values can be configured.
  • Enter 1 to 100 characters, including letters, digits, and other characters (!%'"()*+,./:=?@^\\-_`~). Spaces are not allowed.

kubernetes.io/elb.canary-related-ingress-uid

No

String

UID of the original ingress associated with the grayscale release ingress, which is used to display the association between the original ingress and the grayscale release ingress.

● Format: character string

● Value: metadata.uid of the original ingress

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback