Help Center> Cloud Container Engine> User Guide> User Guide for Standard and Turbo Clusters> Network> Ingresses> LoadBalancer Ingresses> Configuring a LoadBalancer Ingress Using Annotations
Updated on 2024-06-26 GMT+08:00
View PDF

Configuring a LoadBalancer Ingress Using Annotations

You can add annotations to a YAML file for more advanced ingress functions. This section describes the annotations that can be used when you create a LoadBalancer ingress.

Interconnection with ELB

Table 1 Annotations for interconnecting with ELB

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.class

String

Select a proper load balancer type.

The value can be:

v1.9 or later

kubernetes.io/ingress.class

String
  • cce: A proprietary LoadBalancer ingress is used.
  • nginx: Nginx ingress is used.

This parameter is mandatory when an ingress is created by calling the API.

For clusters of v1.23 or later, use the parameter ingressClassName. For details, see Using kubectl to Create a LoadBalancer Ingress.

Only clusters of v1.21 or earlier

kubernetes.io/elb.port

String

This parameter indicates the external port registered with the address of the LoadBalancer Service.

The value ranges from 1 to 65535.

NOTE:

Some ports are high-risk ports and are blocked by default, for example, port 21.

v1.9 or later

kubernetes.io/elb.id

String

Mandatory when an existing load balancer is to be interconnected.

ID of a load balancer.

How to obtain:

On the management console, click Service List, and choose Networking > Elastic Load Balance. Click the name of the target load balancer. On the Summary tab page, find and copy the ID.

v1.9 or later

kubernetes.io/elb.ip

String

Mandatory when an existing load balancer is to be interconnected.

Service address of a load balancer. The value can be the public IP address of a public network load balancer or the private IP address of a private network load balancer.

v1.9 or later

kubernetes.io/elb.autocreate

Table 11 Object

Mandatory when load balancers are automatically created.

Example

  • If a public network load balancer will be automatically created, set this parameter to the following value:

    '{"type":"public","bandwidth_name":"cce-bandwidth-1551163379627","bandwidth_chargemode":"bandwidth","bandwidth_size":5,"bandwidth_sharetype":"PER","eip_type":"5_bgp","name":"james"}'

  • If a private network load balancer will be automatically created, set this parameter to the following value:

    {"type":"inner","name":"A-location-d-test"}

v1.9 or later

kubernetes.io/elb.enterpriseID

String

Optional when load balancers are automatically created.

Clusters of v1.15 and later versions support this field. In clusters earlier than v1.15, load balancers are created in the default project by default.

This parameter indicates the ID of the enterprise project in which the ELB load balancer will be created.

If this parameter is not specified or is set to 0, resources will be bound to the default enterprise project.

How to obtain:

Log in to the management console and choose Enterprise > Project Management on the top menu bar. In the list displayed, click the name of the target enterprise project, and copy the ID on the enterprise project details page.

v1.15 or later

kubernetes.io/elb.subnet-id

String

Optional when load balancers are automatically created.

ID of the subnet where the cluster is located. The value can contain 1 to 100 characters.

  • Mandatory when a cluster of v1.11.7-r0 or earlier is to be automatically created.
  • Optional for clusters later than v1.11.7-r0.

Mandatory for clusters earlier than v1.11.7-r0

Discarded in clusters later than v1.11.7-r0
The following shows how to use the preceding annotations:

Using HTTP/2

Table 2 Annotations of using HTTP/2

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.http2-enable

String

Whether HTTP/2 is enabled. Request forwarding using HTTP/2 improves the access performance between your application and the load balancer. However, the load balancer still uses HTTP/1.x to forward requests to the backend server.

Options:

  • true: enabled
  • false: disabled (default value)

Note: HTTP/2 can be enabled or disabled only when the listener uses HTTPS. This parameter is invalid and defaults to false when the listener protocol is HTTP.

v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later

For details, see Configuring HTTP/2 for a LoadBalancer Ingress.

Configuring ELB Certificates

Table 3 ELB certificate annotations

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.tls-certificate-ids

String

ELB certificate IDs, which are separated by comma (,). The list length is greater than or equal to 1. The first ID in the list is the server certificate, and the other IDs are SNI certificates in which a domain name must be contained.

To obtain the certificate, log in to the CCE console, choose Service List > Networking > Elastic Load Balance, and click Certificates in the navigation pane. In the load balancer list, copy the ID under the target certificate name.

v1.19.16-r2, v1.21.5-r0, v1.23.3-r0, or later

For details, see Using the ELB Certificate.

Interconnecting with HTTPS Backend Services

Table 4 Annotations for interconnecting with HTTPS backend services

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.pool-protocol

String

To interconnect with HTTPS backend services, set this parameter to https.

v1.23.8, v1.25.3, or later

For details, see Configuring HTTPS Backend Services for a LoadBalancer Ingress.

Configuring Timeout for an Ingress

Table 5 Annotations of configuring ingress redirection rules

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.keepalive_timeout

String

Timeout for client connections. If there are no requests reaching the load balancer during the timeout duration, the load balancer will disconnect the connection from the client and establish a new connection when there is a new request.

Value:

  • For TCP listeners, the value ranges from 10 to 4000 (in seconds). The default value is 300.
  • For HTTP or HTTPS listeners, the value ranges from 0 to 4000 (in seconds). The default value is 60.

For UDP listeners, this parameter does not take effect.

Dedicated load balancers: v1.19.16-r30, v1.21.10-r10, v1.23.8-r10, v1.25.3-r10, or later

Shared load balancers: v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later

kubernetes.io/elb.client_timeout

String

Timeout for waiting for a request from a client. There are two cases:

  • If the client fails to send a request header to the load balancer during the timeout duration, the request will be interrupted.
  • If the interval between two consecutive request bodies reaching the load balancer is greater than the timeout duration, the connection will be disconnected.

The value ranges from 1 to 300 (in seconds). The default value is 60.

This parameter is available only for HTTP and HTTPS listeners.

Minimum value: 1

Maximum value: 300

Default value: 60

Dedicated load balancers: v1.19.16-r30, v1.21.10-r10, v1.23.8-r10, v1.25.3-r10, or later

Shared load balancers: v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later

kubernetes.io/elb.member_timeout

String

Timeout for waiting for a response from a backend server. After a request is forwarded to the backend server, if the backend server does not respond within the duration specified by member_timeout, the load balancer will stop waiting and return HTTP 504 Gateway Timeout.

The value ranges from 1 to 300 (in seconds). The default value is 60.

This parameter is available only for HTTP and HTTPS listeners.

Minimum value: 1

Maximum value: 300

Default value: 60

Dedicated load balancers: v1.19.16-r30, v1.21.10-r10, v1.23.8-r10, v1.25.3-r10, or later

Shared load balancers: v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later

For details, see Configuring Timeout for a LoadBalancer Ingress.

Adding Resource Tags

Table 6 Annotations

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.tags

String

Add resource tags to a load balancer. This parameter can be configured only when a load balancer is automatically created.

A tag is in the format of "key=value". Use commas (,) to separate multiple tags.

v1.23.11-r0, v1.25.6-r0, v1.27.3-r0, or later

For details, see Creating an Ingress - Automatically Creating a Load Balancer.

Blocklist/Trustlist

Table 7 Annotations for ELB access control

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.acl-id

String
  • If this parameter is not specified, CCE does not modify access control on the ELB.
  • If this parameter is left empty, all IP addresses are allowed to access the load balancer.
  • If this parameter is set to the IP address group ID of the load balancer, access control is enabled and you need to configure an IP address blocklist or trustlist for the load balancer. Additionally, you need to configure both kubernetes.io/elb.acl-status and kubernetes.io/elb.acl-type.

    How to obtain:

    Log in to the console. In the Service List, choose Networking > Elastic Load Balance. On the Network Console, choose Elastic Load Balance > IP Address Groups and copy the ID of the target IP address group. For details, see Creating an IP Address Group.

v1.23.12-r0, v1.25.7-r0, v1.27.4-r0, v1.28.2-r0, or later

kubernetes.io/elb.acl-status

String

Access control status. This parameter is mandatory when you configure an IP address blocklist or trustlist for a load balancer. Options:

  • on: Access control is enabled.
  • off: Access control is disabled.

v1.23.12-r0, v1.25.7-r0, v1.27.4-r0, v1.28.2-r0, or later

kubernetes.io/elb.acl-type

String

IP address list type. This parameter is mandatory when you configure an IP address blocklist or trustlist for a load balancer. Options:

  • black: indicates a blocklist. The selected IP address group cannot access the load balancer.
  • white: indicates a trustlist. Only the selected IP address group can access the load balancer.

v1.23.12-r0, v1.25.7-r0, v1.27.4-r0, v1.28.2-r0, or later

For details, see Configuring a Blocklist/Trustlist Access Policy for a LoadBalancer Ingress.

Configuring a Custom Listening Port

A custom listening port can be configured for an ingress. In this way, both ports 80 and 443 can be exposed.

Table 8 Annotations for a custom listening port

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.listen-ports

String

Create multiple listening ports for an ingress. The port number ranges from 1 to 65535.

The following is an example for JSON characters:

kubernetes.io/elb.listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
  • Only the listening ports that comply with both HTTP and HTTPS are allowed.
  • Only newly created ingresses are allowed. Additionally, after multiple listening ports are configured, annotations cannot be modified or deleted.
  • If both kubernetes.io/elb.listen-ports and kubernetes.io/elb.port are configured, kubernetes.io/elb.listen-ports takes a higher priority.
  • Ingress configuration items such as the blocklist, trustlist, and timeout concurrently take effect on multiple listening ports.
  • Advanced forwarding policies are not supported.

v1.23.14-r0, v1.25.9-r0, v1.27.6-r0, v1.28.4-r0, or later

For example, if an existing ELB is used, the configuration is as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/elb.id: 2c623150-17bf-45f1-ae6f-384b036f547e     # ID of an existing load balancer
    kubernetes.io/elb.class: performance    # Load balancer type
    kubernetes.io/elb.listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'    # Multi-listener configuration
    kubernetes.io/elb.tls-certificate-ids: 6cfb43c9de1a41a18478b868e34b0a82,6cfb43c9de1a41a18478b868e34b0a82   # HTTPS certificate configuration
  name: test-https
  namespace: default
spec:
  ingressClassName: cce
  rules:
  - host: xxx.com
    http:
      paths:
      - backend:
          service:
            name: test
            port:
              number: 8888
        path: /
        pathType: ImplementationSpecific
        property:
          ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH

Enabling GZIP

Table 9 Annotations for enabling GZIP

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.gzip-enabled

String

LoadBalancer ingresses support data compression, which reduces the size of files to be transferred, improves file transfer efficiency, and reduces the bandwidth needed for the transmission.

If this function is enabled, specific files will be compressed. If you do not enable this function, files will not be compressed. By default, data compression is disabled.

The files in the following format can be compressed:

  • Brotli can compress all file formats.
  • GZIP can compress the files in the following format: text, xml text, plain text, css application, javascript application, x-javascript application, rss+xml application, atom+xml application, xml application, or json

This function is available only for HTTP/HTTPS listeners of dedicated load balancers. If the advanced configuration for enabling data compression or the target annotation is deleted, the ELB configuration will not be modified.

v1.23.14-r0, v1.25.9-r0, v1.27.6-r0, v1.28.4-r0, or later

For details, see Configuring GZIP Data Compression for a LoadBalancer Ingress.

Configuring the Priorities of Forwarding Rules

When ingresses use the same load balancer listener, forwarding rules can be prioritized based on the following rules:

  • Forwarding rules of different ingresses: The rules are sorted based on the priorities (ranging from 1 to 1000) of the kubernetes.io/elb.ingress-order annotation. A smaller value indicates a higher priority.
  • Forwarding rules of an ingress: If the kubernetes.io/elb.priority-enable annotation is set to true, the forwarding rules are sorted based on the sequence in which they are added during ingress creation. A forwarding rule added earlier indicates a higher priority. If the kubernetes.io/elb.priority-enable annotation is not configured, the default sorting of the forwarding rules on the load balancer will be used.

If the preceding annotations are not configured, the default sorting of the forwarding rules on the load balancer will be used, regardless of whether the forwarding rules are of the same ingress or different ingresses under the same load balancer listener.

Table 10 Annotations for forwarding rule priorities

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.ingress-order

String

Specifies the sequence of forwarding rules of different ingresses. The value ranges from 1 to 1000. A smaller value indicates a higher priority. The priority of a forwarding rule must be unique under the same load balancer listener.

This parameter is available only for dedicated load balancers.

NOTE:

When this annotation is configured, the kubernetes.io/elb.priority-enable annotation is enabled by default. The forwarding rules of each ingress will be sorted.

v1.23.15-r0, v1.25.10-r0, v1.27.7-r0, v1.28.5-r0, v1.29.1-r10, or later

kubernetes.io/elb.priority-enable

String

This parameter can only be set to true, indicating to sort the forwarding rules of an ingress. The priorities of the forwarding rules are determined based on the sequence in which they are added during ingress creation. A forwarding rule added earlier indicates a higher priority.

If this parameter is not configured, the default sorting of the forwarding rules on the load balancer will be used. After this parameter is enabled, it cannot be disabled.

This parameter is available only for dedicated load balancers.

The following shows an example configuration using an existing load balancer:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test
  namespace: default
  annotations:
    kubernetes.io/elb.port: '88'
    kubernetes.io/elb.id: 2c623150-17bf-45f1-ae6f-384b036f547e    # ID of an existing load balancer
    kubernetes.io/elb.class: performance    # Load balancer type
    kubernetes.io/elb.ingress-order: '1'    # Priority of a forwarding rule of different ingresses
    kubernetes.io/elb.priority-enable: 'true'   # The forwarding rules of an ingress are sorted based on the forwarding rule sequence in paths.
spec:
  rules:
    - host: ''
      http:
        paths:
          - path: /test1
            backend:
              service:
                name: test1
                port:
                  number: 8081
            property:
              ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
            pathType: ImplementationSpecific
          - path: /test2
            backend:
              service:
                name: test2
                port:
                  number: 8081
            property:
              ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
            pathType: ImplementationSpecific
  ingressClassName: cce

Parameters for Automatically Creating a Load Balancer

Table 11 elb.autocreate data structure

Parameter

Mandatory

Type

Description

name

No

String

Name of the automatically created load balancer.

The value can contain 1 to 64 characters. Only letters, digits, underscores (_), hyphens (-), and periods (.) are allowed.

Default: cce-lb+service.UID

type

No

String

Network type of the load balancer.

  • public: public network load balancer
  • inner: private network load balancer

Default: inner

bandwidth_name

Yes for public network load balancers

String

Bandwidth name. The default value is cce-bandwidth-******.

The value can contain 1 to 64 characters. Only letters, digits, underscores (_), hyphens (-), and periods (.) are allowed.

bandwidth_chargemode

No

String

Bandwidth billing mode.

  • bandwidth: billed by bandwidth
  • traffic: billed by traffic

Default: bandwidth

bandwidth_size

Yes for public network load balancers

Integer

Bandwidth size. The value ranges from 1 Mbit/s to 2000 Mbit/s by default. Configure this parameter based on the bandwidth range allowed in your region.

The minimum increment for bandwidth adjustment varies depending on the bandwidth range.
  • The minimum increment is 1 Mbit/s if the allowed bandwidth does not exceed 300 Mbit/s.
  • The minimum increment is 50 Mbit/s if the allowed bandwidth ranges from 300 Mbit/s to 1000 Mbit/s.
  • The minimum increment is 500 Mbit/s if the allowed bandwidth exceeds 1000 Mbit/s.

bandwidth_sharetype

Yes for public network load balancers

String

Bandwidth sharing mode.

  • PER: dedicated bandwidth

eip_type

Yes for public network load balancers

String

EIP type.

  • 5_telcom: China Telecom
  • 5_union: China Unicom
  • 5_bgp: dynamic BGP
  • 5_sbgp: static BGP

The specific type varies with regions. For details, see the EIP console.

vip_subnet_cidr_id

No

String

Subnet where a load balancer is located. The subnet must belong to the VPC where the cluster resides.

If this parameter is not specified, the ELB load balancer and the cluster are in the same subnet.

This field can be specified only for clusters of v1.21 or later.

vip_address

No

String

Private IP address of the load balancer. Only IPv4 addresses are supported.

The IP address must be in the ELB CIDR block. If this parameter is not specified, an IP address will be automatically assigned from the ELB CIDR block.

This parameter is available only in clusters of v1.23.11-r0, v1.25.6-r0, v1.27.3-r0, or later versions.

available_zone

Yes

Array of strings

AZ where the load balancer is located.

You can obtain all supported AZs by getting the AZ list.

This parameter is available only for dedicated load balancers.

l4_flavor_name

Yes

String

Flavor name of the layer-4 load balancer.

You can obtain all supported types by getting the flavor list.

This parameter is available only for dedicated load balancers.

l7_flavor_name

No

String

Flavor name of the layer-7 load balancer.

You can obtain all supported types by getting the flavor list.

This parameter is available only for dedicated load balancers. The value of this parameter must be the same as that of l4_flavor_name, that is, both are elastic specifications or fixed specifications.

elb_virsubnet_ids

No

Array of strings

Subnet where the backend server of the load balancer is located. If this parameter is left blank, the default cluster subnet is used. Load balancers occupy different number of subnet IP addresses based on their specifications. Do not use the subnet CIDR blocks of other resources (such as clusters and nodes) as the load balancer CIDR block.

This parameter is available only for dedicated load balancers.

Example:

"elb_virsubnet_ids": [
   "14567f27-8ae4-42b8-ae47-9f847a4690dd"
 ]

ipv6_vip_virsubnet_id

No

String

Specifies the ID of the IPv6 subnet where the load balancer resides. IPv6 must be enabled for the corresponding subnet. This parameter is mandatory only when the dual-stack clusters are used.

This parameter is available only for dedicated load balancers.
Parent topic: LoadBalancer Ingresses