Help Center> Cloud Container Engine> User Guide> User Guide for Standard and Turbo Clusters> Network> Service> LoadBalancer> Using Annotations to Balance Load
Updated on 2024-07-02 GMT+08:00
View PDF

Using Annotations to Balance Load

You can add annotations to a YAML file to use some CCE advanced functions. This section describes the available annotations when a LoadBalancer service is created.

Interconnection with ELB

Table 1 Annotations for interconnecting with ELB

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.class

String

Select a proper load balancer type.

The value can be:

v1.9 or later

kubernetes.io/elb.id

String

Mandatory when an existing load balancer is to be associated.

ID of a load balancer.

How to obtain:

On the management console, click Service List, and choose Networking > Elastic Load Balance. Click the name of the target load balancer. On the Summary tab page, find and copy the ID.

NOTE:

The system preferentially connects to the load balancer based on the kubernetes.io/elb.id field. If this field is not specified, the spec.loadBalancerIP field is used (optional and available only in 1.23 and earlier versions).

Do not use the spec.loadBalancerIP field to connect to the load balancer. This field will be discarded by Kubernetes. For details, see Deprecation.

v1.9 or later

kubernetes.io/elb.autocreate

Table 14

Mandatory when load balancers are automatically created.

Example:

  • If a public network load balancer will be automatically created, set this parameter to the following value:

    '{"type":"public","bandwidth_name":"cce-bandwidth-1551163379627","bandwidth_chargemode":"bandwidth","bandwidth_size":5,"bandwidth_sharetype":"PER","eip_type":"5_bgp","name":"james"}'

  • If a private network load balancer will be automatically created, set this parameter to the following value:

    {"type":"inner","name":"A-location-d-test"}

v1.9 or later

kubernetes.io/elb.enterpriseID

String

Optional when load balancers are automatically created.

Clusters of v1.15 and later versions support this field. In clusters earlier than v1.15, load balancers are created in the default project by default.

This parameter indicates the ID of the enterprise project in which the ELB load balancer will be created.

If this parameter is not specified or is set to 0, resources will be bound to the default enterprise project.

How to obtain:

Log in to the management console and choose Enterprise > Project Management on the top menu bar. In the list displayed, click the name of the target enterprise project, and copy the ID on the enterprise project details page.

v1.15 or later

kubernetes.io/elb.subnet-id

String

Optional when load balancers are automatically created.

ID of the subnet where the cluster is located. The value can contain 1 to 100 characters.

  • Mandatory when a cluster of v1.11.7-r0 or earlier is to be automatically created.
  • Optional for clusters later than v1.11.7-r0.

Mandatory for clusters earlier than v1.11.7-r0

Discarded in clusters later than v1.11.7-r0

kubernetes.io/elb.lb-algorithm

String

Specifies the load balancing algorithm of the backend server group. The default value is ROUND_ROBIN.

Options:

  • ROUND_ROBIN: weighted round robin algorithm
  • LEAST_CONNECTIONS: weighted least connections algorithm
  • SOURCE_IP: source IP hash algorithm
NOTE:

If this parameter is set to SOURCE_IP, the weight setting (weight field) of backend servers bound to the backend server group is invalid, and sticky session cannot be enabled.

v1.9 or later

The following shows how to use the preceding annotations:

  • Associate an existing load balancer. For details, see Using kubectl to Create a Service (Using an Existing Load Balancer).
    apiVersion: v1 
kind: Service 
metadata: 
  name: nginx
  annotations:
    kubernetes.io/elb.id: <your_elb_id>                         # Load balancer ID. Replace it with the actual value.
    kubernetes.io/elb.class: performance                        # Load balancer type
    kubernetes.io/elb.lb-algorithm: ROUND_ROBIN                   # Load balancer algorithm
spec:
  selector: 
     app: nginx
  ports: 
  - name: service0 
    port: 80
    protocol: TCP 
    targetPort: 80
  type: LoadBalancer
  • Automatically create a load balancer. For details, see Using kubectl to Create a Service (Automatically Creating a Load Balancer).
    Shared load balancer: 
    apiVersion: v1 
kind: Service 
metadata: 
  annotations:   
    kubernetes.io/elb.class: union
    kubernetes.io/elb.autocreate: '{
      "type": "public",
      "bandwidth_name": "cce-bandwidth-1551163379627",
      "bandwidth_chargemode": "bandwidth",
      "bandwidth_size": 5,
      "bandwidth_sharetype": "PER",
      "eip_type": "5_bgp"
    }'
    kubernetes.io/elb.enterpriseID: '0'               # ID of the enterprise project to which the load balancer belongs
    kubernetes.io/elb.lb-algorithm: ROUND_ROBIN     # Load balancer algorithm
  labels: 
    app: nginx 
  name: nginx 
spec: 
  ports: 
  - name: service0 
    port: 80
    protocol: TCP 
    targetPort: 80
  selector: 
    app: nginx 
  type: LoadBalancer
    Dedicated load balancer: 
    apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
  namespace: default
  annotations:
    kubernetes.io/elb.class: performance
    kubernetes.io/elb.autocreate: '{
      "type": "public",
      "bandwidth_name": "cce-bandwidth-1626694478577",
      "bandwidth_chargemode": "bandwidth",
      "bandwidth_size": 5,
      "bandwidth_sharetype": "PER",
      "eip_type": "5_bgp",
      "available_zone": [
         ""
      ],
      "l4_flavor_name": "L4_flavor.elb.s1.small"
    }'
    kubernetes.io/elb.enterpriseID: '0'               # ID of the enterprise project to which the load balancer belongs
    kubernetes.io/elb.lb-algorithm: ROUND_ROBIN     # Load balancer algorithm
spec:
  selector:
    app: nginx
  ports:
  - name: cce-service-0
    targetPort: 80
    nodePort: 0
    port: 80
    protocol: TCP
  type: LoadBalancer

Sticky Session

Table 2 Annotations for sticky session

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.session-affinity-mode

String

Source IP address-based sticky session means that access requests from the same IP address are forwarded to the same backend server.

  • Disabling sticky session: Do not configure this parameter.
  • Enabling sticky session: Set this parameter to SOURCE_IP, indicating that the sticky session is based on the source IP address.
NOTE:

When kubernetes.io/elb.lb-algorithm is set to SOURCE_IP (source IP hash), sticky session cannot be enabled.

v1.9 or later

kubernetes.io/elb.session-affinity-option

Table 17

Sticky session timeout.

v1.9 or later
The following shows how to use the preceding annotations: 
apiVersion: v1 
kind: Service 
metadata: 
  name: nginx
  annotations:
    kubernetes.io/elb.id: <your_elb_id>                         # Load balancer ID. Replace it with the actual value.
    kubernetes.io/elb.class: performance                   # Load balancer type
    kubernetes.io/elb.session-affinity-mode: SOURCE_IP          # The sticky session type is source IP address.
    kubernetes.io/elb.session-affinity-option: '{"persistence_timeout": "30"}'     # Stickiness duration (min)
spec:
  selector: 
     app: nginx
  ports: 
  - name: service0 
    port: 80
    protocol: TCP 
    targetPort: 80
  type: LoadBalancer

Health Check

Table 3 Annotations for health check

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.health-check-flag

String

Whether to enable the ELB health check.

  • Enabling health check: Leave blank this parameter or set it to on.
  • Disabling health check: Set this parameter to off.

If this parameter is enabled, the kubernetes.io/elb.health-check-option field must also be specified at the same time.

v1.9 or later

kubernetes.io/elb.health-check-option

Table 15

ELB health check configuration items.

v1.9 or later

kubernetes.io/elb.health-check-options

Table 16

ELB health check configuration items. Each Service port can be configured separately, and you can configure only some ports.

NOTE:

Either kubernetes.io/elb.health-check-option or kubernetes.io/elb.health-check-options can be configured.

v1.19.16-r5 or later

v1.21.8-r0 or later

v1.23.6-r0 or later

v1.25.2-r0 or later
  • The following shows how to use kubernetes.io/elb.health-check-option:
    apiVersion: v1 
kind: Service 
metadata: 
  name: nginx
  annotations:
    kubernetes.io/elb.id: <your_elb_id>                         # Load balancer ID. Replace it with the actual value.
    kubernetes.io/elb.class: performance                   # Load balancer type
    kubernetes.io/elb.health-check-flag: 'on'                   # Enable ELB health check.
    kubernetes.io/elb.health-check-option: '{
      "protocol":"TCP",
      "delay":"5",
      "timeout":"10",
      "max_retries":"3"
    }'
spec:
  selector: 
     app: nginx
  ports: 
  - name: service0 
    port: 80
    protocol: TCP 
    targetPort: 80
  type: LoadBalancer
  • For details about how to use kubernetes.io/elb.health-check-options, see Configuring Health Check on Multiple Ports of a LoadBalancer Service.

HTTP or HTTPS

Table 4 Annotations for using HTTP or HTTPS

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.protocol-port

String

If a Service is HTTP/HTTPS-compliant, configure the protocol and port number in the format of "protocol:port".

where,

  • protocol: specifies the protocol used by the listener port. The value can be http or https.
  • ports: Service ports specified by spec.ports[].port.

v1.19.16 or later

kubernetes.io/elb.cert-id

String

ID of an ELB certificate, which is used as the HTTPS server certificate.

To obtain the certificate, log in to the CCE console, choose Service List > Networking > Elastic Load Balance, and click Certificates in the navigation pane. In the load balancer list, copy the ID under the target certificate name.

v1.19.16 or later

For details, see Configuring HTTP/HTTPS for a LoadBalancer Service.

SNI

Table 5 Annotations for using SNIs

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.tls-certificate-ids

String

In ELB, the IDs of SNI certificates that must contain a domain name are separated by commas (,).

To obtain the certificate, log in to the CCE console, choose Service List > Networking > Elastic Load Balance, and click Certificates in the navigation pane. In the load balancer list, copy the ID under the target certificate name.

v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later

HTTPS must be enabled. For details, see Configuring SNI for a LoadBalancer Service.

Dynamic Adjustment of the Weight of the Backend ECS

Table 6 Annotations for dynamically adjusting the weight of the backend ECS

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.adaptive-weight

String

Dynamically adjust the weight of the load balancer backend server based on the number pods on the server. In this way, the requests received by each pod are more balanced.

  • true: enabled
  • false: disabled

This parameter applies only to clusters of v1.21 or later and is invalid in passthrough networking.

v1.21 or later
The following shows how to use the preceding annotations: 
apiVersion: v1 
kind: Service 
metadata: 
  name: nginx
  annotations:
    kubernetes.io/elb.id: <your_elb_id>                         # Load balancer ID. Replace it with the actual value.
    kubernetes.io/elb.class: performance                   # Load balancer type
    kubernetes.io/elb.adaptive-weight: 'true'                   # Enable dynamic adjustment of the weight of the backend ECS.
spec:
  selector: 
     app: nginx
  ports: 
  - name: service0 
    port: 80
    protocol: TCP 
    targetPort: 80
  type: LoadBalancer

Passthrough Capability

Table 7 Annotations for passthrough capability

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.pass-through

String

Whether the access requests from within the cluster to the Service pass through the ELB load balancer.

v1.19 or later

For details, see Configuring Passthrough Networking for a LoadBalancer Service.

Blocklist/Trustlist

Table 8 Annotations for ELB access control

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.acl-id

String
  • If this parameter is not specified, CCE does not modify access control on the ELB.
  • If this parameter is left empty, all IP addresses are allowed to access the load balancer.
  • If this parameter is set to the IP address group ID of the load balancer, access control is enabled and you need to configure an IP address blocklist or trustlist for the load balancer. Additionally, you need to configure both kubernetes.io/elb.acl-status and kubernetes.io/elb.acl-type.

    How to obtain:

    Log in to the console. In the Service List, choose Networking > Elastic Load Balance. On the Network Console, choose Elastic Load Balance > IP Address Groups and copy the ID of the target IP address group. For details, see Creating an IP Address Group.

v1.23.12-r0, v1.25.7-r0, v1.27.4-r0, v1.28.2-r0, or later

kubernetes.io/elb.acl-status

String

This parameter is mandatory when you configure an IP address blocklist or trustlist for a load balancer. Options:

  • on: Access control is enabled.
  • off: Access control is disabled.

v1.23.12-r0, v1.25.7-r0, v1.27.4-r0, v1.28.2-r0, or later

kubernetes.io/elb.acl-type

String

This parameter is mandatory when you configure an IP address blocklist or trustlist for a load balancer. Options:

  • black: indicates a blocklist. The selected IP address group cannot access the load balancer.
  • white: indicates a trustlist. Only the selected IP address group can access the load balancer.

v1.23.12-r0, v1.25.7-r0, v1.27.4-r0, v1.28.2-r0, or later
The following shows how to use the preceding annotations: 
apiVersion: v1 
kind: Service 
metadata: 
  name: nginx
  annotations:
    kubernetes.io/elb.id: <your_elb_id>                    # Load balancer ID. Replace it with the actual value.
    kubernetes.io/elb.class: performance                   # Load balancer type
    kubernetes.io/elb.acl-id: <your_acl_id>               # ID of an IP address group for accessing a load balancer
    kubernetes.io/elb.acl-status: 'on'                    # Enable access control.
    kubernetes.io/elb.acl-type: 'white'                   # Trustlist for access control
spec:
  selector: 
     app: nginx
  ports: 
  - name: service0 
    port: 80
    protocol: TCP 
    targetPort: 80
  type: LoadBalancer

Host Network

Table 9 Annotations for host network

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/hws-hostNetwork

String

If the pod uses hostNetwork, the ELB forwards the request to the host network after this annotation is used.

Options:

  • true: enabled
  • false (default): disabled

v1.9 or later
The following shows how to use the preceding annotations: 
apiVersion: v1 
kind: Service 
metadata: 
  name: nginx
  annotations:
    kubernetes.io/elb.id: <your_elb_id>                         # Load balancer ID. Replace it with the actual value.
    kubernetes.io/elb.class: performance                   # Load balancer type
    kubernetes.io/hws-hostNetwork: 'true'                     # The load balancer forwards the request to the host network.
spec:
  selector: 
     app: nginx
  ports: 
  - name: service0 
    port: 80
    protocol: TCP 
    targetPort: 80
  type: LoadBalancer

Timeout

Table 10 Annotation for configuring timeout

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.keepalive_timeout

String

Timeout for client connections. If there are no requests reaching the load balancer during the timeout duration, the load balancer will disconnect the connection from the client and establish a new connection when there is a new request.

Value:

  • For TCP listeners, the value ranges from 10 to 4000 (in seconds). The default value is 300.
  • For HTTP, HTTPS, and TERMINATED_HTTPS listeners, the value ranges from 0 to 4000 (in seconds). The default value is 60.
  • For UDP listeners, this parameter does not take effect.

Dedicated load balancers: v1.19.16-r30, v1.21.10-r10, v1.23.8-r10, v1.25.3-r10, or later

Shared load balancers: v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later

For details, see Configuring Timeout for a LoadBalancer Service.

Resource Tags

Table 11 Annotations

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.tags

String

Add resource tags to a load balancer. This parameter can be configured only when a load balancer is automatically created.

A tag is in the format of "key=value". Use commas (,) to separate multiple tags.

v1.23.11-r0, v1.25.6-r0, v1.27.3-r0, or later

For details, see Using kubectl to Create a Service (Automatically Creating a Load Balancer).

HTTP/2

Table 12 Annotations of using HTTP/2

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.http2-enable

String

Whether HTTP/2 is enabled. Request forwarding using HTTP/2 improves the access performance between your application and the load balancer. However, the load balancer still uses HTTP/1.x to forward requests to the backend server.

Options:

  • true: enabled
  • false: disabled (default value)

Note: HTTP/2 can be enabled or disabled only when the listener uses HTTPS. This parameter is invalid and defaults to false when the listener protocol is HTTP.

v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later

For details, see Configuring HTTP/2 for a LoadBalancer Service.

Enabling GZIP

Table 13 Annotations for enabling GZIP

Parameter

Type

Description

Supported Cluster Version

kubernetes.io/elb.gzip-enabled

String

LoadBalancer Services support data compression, which reduces the size of files to be transferred, improves file transfer efficiency, and reduces the bandwidth needed for the transmission.

If this function is enabled, specific files will be compressed. If you do not enable this function, files will not be compressed. By default, data compression is disabled.

The files in the following format can be compressed:

  • Brotli: all file formats
  • GZIP: text, xml text, plain text, css application, javascript application, x-javascript application, rss+xml application, atom+xml application, xml application, or json

This function is available only for HTTP/HTTPS listeners of dedicated load balancers. If the advanced configuration for enabling data compression or the target annotation is deleted, the ELB configuration will not be modified.

v1.23.14-r0, v1.25.9-r0, v1.27.6-r0, v1.28.4-r0, or later

For details, see Configuring GZIP Data Compression for a LoadBalancer Service.

Parameters for Automatically Creating a Load Balancer

Table 14 elb.autocreate data structure

Parameter

Mandatory

Type

Description

name

No

String

Name of the automatically created load balancer.

The value can contain 1 to 64 characters. Only letters, digits, underscores (_), hyphens (-), and periods (.) are allowed.

Default: cce-lb+service.UID

type

No

String

Network type of the load balancer.

  • public: public network load balancer
  • inner: private network load balancer

Default: inner

bandwidth_name

Yes for public network load balancers

String

Bandwidth name. The default value is cce-bandwidth-******.

The value can contain 1 to 64 characters. Only letters, digits, underscores (_), hyphens (-), and periods (.) are allowed.

bandwidth_chargemode

No

String

Bandwidth billing mode.

  • bandwidth: billed by bandwidth
  • traffic: billed by traffic

Default: bandwidth

bandwidth_size

Yes for public network load balancers

Integer

Bandwidth size. The value ranges from 1 Mbit/s to 2000 Mbit/s by default. Configure this parameter based on the bandwidth range allowed in your region.

The minimum increment for bandwidth adjustment varies depending on the bandwidth range.
  • The minimum increment is 1 Mbit/s if the allowed bandwidth does not exceed 300 Mbit/s.
  • The minimum increment is 50 Mbit/s if the allowed bandwidth ranges from 300 Mbit/s to 1000 Mbit/s.
  • The minimum increment is 500 Mbit/s if the allowed bandwidth exceeds 1000 Mbit/s.

bandwidth_sharetype

Yes for public network load balancers

String

Bandwidth sharing mode.

  • PER: dedicated bandwidth

eip_type

Yes for public network load balancers

String

EIP type.

  • 5_telcom: China Telecom
  • 5_union: China Unicom
  • 5_bgp: dynamic BGP
  • 5_sbgp: static BGP

The specific type varies with regions. For details, see the EIP console.

vip_subnet_cidr_id

No

String

Subnet where a load balancer is located. The subnet must belong to the VPC where the cluster resides.

If this parameter is not specified, the ELB load balancer and the cluster are in the same subnet.

This field can be specified only for clusters of v1.21 or later.

vip_address

No

String

Private IP address of the load balancer. Only IPv4 addresses are supported.

The IP address must be in the ELB CIDR block. If this parameter is not specified, an IP address will be automatically assigned from the ELB CIDR block.

This parameter is available only in clusters of v1.23.11-r0, v1.25.6-r0, v1.27.3-r0, or later versions.

available_zone

Yes

Array of strings

AZ where the load balancer is located.

You can obtain all supported AZs by getting the AZ list.

This parameter is available only for dedicated load balancers.

l4_flavor_name

Yes

String

Flavor name of the layer-4 load balancer.

You can obtain all supported types by getting the flavor list.

This parameter is available only for dedicated load balancers.

l7_flavor_name

No

String

Flavor name of the layer-7 load balancer.

You can obtain all supported types by getting the flavor list.

This parameter is available only for dedicated load balancers. The value of this parameter must be the same as that of l4_flavor_name, that is, both are elastic specifications or fixed specifications.

elb_virsubnet_ids

No

Array of strings

Subnet where the backend server of the load balancer is located. If this parameter is left blank, the default cluster subnet is used. Load balancers occupy different number of subnet IP addresses based on their specifications. Do not use the subnet CIDR blocks of other resources (such as clusters and nodes) as the load balancer CIDR block.

This parameter is available only for dedicated load balancers.

Example:

"elb_virsubnet_ids": [
   "14567f27-8ae4-42b8-ae47-9f847a4690dd"
 ]

ipv6_vip_virsubnet_id

No

String

Specifies the ID of the IPv6 subnet where the load balancer resides. IPv6 must be enabled for the corresponding subnet. This parameter is mandatory only when the dual-stack clusters are used.

This parameter is available only for dedicated load balancers.
Table 15 elb.health-check-option data structure

Parameter

Mandatory

Type

Description

delay

No

String

Health check interval (s)

Value range: 1 to 50. Default value: 5

timeout

No

String

Health check timeout, in seconds.

Value range: 1 to 50. Default value: 10

max_retries

No

String

Maximum number of health check retries.

Value range: 1 to 10. Default value: 3

protocol

No

String

Health check protocol.

Value options: TCP or HTTP

path

No

String

Health check URL. This parameter needs to be configured when the protocol is HTTP.

Default value: /

Value range: 1-80 characters
Table 16 elb.health-check-options

Parameter

Mandatory

Type

Description

target_service_port

Yes

String

Port for health check specified by spec.ports. The value consists of the protocol and port number, for example, TCP:80.

monitor_port

No

String

Re-specified port for health check. If this parameter is not specified, the service port is used by default.

NOTE:

Ensure that the port is in the listening state on the node where the pod is located. Otherwise, the health check result will be affected.

delay

No

String

Health check interval (s)

Value range: 1 to 50. Default value: 5

timeout

No

String

Health check timeout, in seconds.

Value range: 1 to 50. Default value: 10

max_retries

No

String

Maximum number of health check retries.

Value range: 1 to 10. Default value: 3

protocol

No

String

Health check protocol.

Default value: protocol of the associated Service

Value options: TCP, UDP, or HTTP

path

No

String

Health check URL. This parameter needs to be configured when the protocol is HTTP.

Default value: /

Value range: 1-80 characters
Table 17 elb.session-affinity-option data structure

Parameter

Mandatory

Type

Description

persistence_timeout

Yes

String

Sticky session timeout, in minutes. This parameter is valid only when elb.session-affinity-mode is set to SOURCE_IP.

Value range: 1 to 60. Default value: 60
Parent topic: LoadBalancer