LoadBalancer Service Annotations
You can add annotations to a YAML file to enable advanced CCE functions. This section describes the available annotations when a LoadBalancer Service is created.
Indexes
| Category | ELB Annotation |
|---|---|
| Load balancer settings | |
| Port or protocol settings | |
| Advanced features of ELB listeners |
Interconnection with ELB
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.class | String | Select a proper load balancer type. This parameter cannot be modified after the Service is created. Options:
| v1.9 or later |
| kubernetes.io/elb.id | String | Mandatory when an existing load balancer is associated. This parameter cannot be modified after the Service is created. ID of a load balancer. How to obtain: Log in to the ELB console and click the load balancer name. On the Summary tab of the load balancer details page, find the ID field and copy it. NOTE: The system preferentially connects to the load balancer based on the kubernetes.io/elb.id field. If this field is not specified, the spec.loadBalancerIP field is used (optional and available only in 1.23 and earlier versions). Do not use the spec.loadBalancerIP field to connect to the load balancer. This field will be discarded by Kubernetes. For details, see Deprecation. | v1.9 or later |
| kubernetes.io/elb.autocreate | Mandatory when load balancers are automatically created. This parameter cannot be modified after the Service is created. CAUTION: kubernetes.io/elb.id and kubernetes.io/elb.autocreate cannot be specified concurrently. If both annotations are specified in a Service, CCE will use an existing load balancer instead of automatically creating one. If this Service is deleted, the existing load balancer is considered as an automatically created one. If no other resources are associated with the load balancer, the load balancer will also be deleted. Example:
| v1.9 or later | |
| kubernetes.io/elb.enterpriseID | String | Optional when load balancers are automatically created. This parameter is available in clusters v1.15 or later. In clusters earlier than v1.15, load balancers are created in the default project by default. This parameter indicates the ID of the enterprise project in which the ELB load balancer will be created. If this parameter is not specified or is set to 0, resources will be bound to the default enterprise project. How to obtain: Log in to the EPS console, click the name of the target enterprise project. On the enterprise project details page, find the ID field and copy the ID. | v1.15 or later |
| kubernetes.io/elb.subnet-id | String | Optional when load balancers are automatically created. ID of the subnet where the cluster is located. The value can contain 1 to 100 characters.
| Mandatory for clusters of a version earlier than v1.11.7-r0 Discarded in clusters of a version later than v1.11.7-r0 |
| kubernetes.io/elb.lb-algorithm | String | Specifies the load balancing algorithm of the backend server group. The default value is ROUND_ROBIN. Options:
NOTE: If this parameter is set to SOURCE_IP, the weight setting (weight field) of backend servers bound to the backend server group is invalid, and sticky session cannot be enabled. | v1.9 or later |
The following shows how to use the preceding annotations:
- Associate an existing load balancer. For details, see Using kubectl to Create a Service (Using an Existing Load Balancer).
apiVersion: v1 kind: Service metadata: name: nginx annotations: kubernetes.io/elb.id: <your_elb_id> # Load balancer ID. Replace it with the actual value. kubernetes.io/elb.class: performance # Load balancer type kubernetes.io/elb.lb-algorithm: ROUND_ROBIN # Load balancer algorithm spec: selector: app: nginx ports: - name: service0 port: 80 protocol: TCP targetPort: 80 type: LoadBalancer - Automatically create a load balancer. For details, see Using kubectl to Create a Service (Automatically Creating a Load Balancer).
Sticky Session
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.session-affinity-mode | String | Source IP address-based sticky session means that access requests from the same IP address are forwarded to the same backend server.
NOTE: When kubernetes.io/elb.lb-algorithm is set to SOURCE_IP (source IP hash), sticky session cannot be enabled. | v1.9 or later |
| kubernetes.io/elb.session-affinity-option | Sticky session timeout. | v1.9 or later |
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
kubernetes.io/elb.id: <your_elb_id> # Load balancer ID. Replace it with the actual value.
kubernetes.io/elb.class: # Load balancer type
kubernetes.io/elb.session-affinity-mode: SOURCE_IP # The sticky session type is source IP address.
kubernetes.io/elb.session-affinity-option: '{"persistence_timeout": "30"}' # Stickiness duration, which is measured in minutes
spec:
selector:
app: nginx
ports:
- name: service0
port: 80
protocol: TCP
targetPort: 80
type: LoadBalancer Health Check
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.health-check-flag | String | Whether to enable the ELB health check. If this annotation is not added, the health check is enabled by default.
If this function is enabled, you must set kubernetes.io/elb.health-check-option or kubernetes.io/elb.health-check-options. Otherwise, ELB health checks will fail. | v1.9 or later |
| kubernetes.io/elb.health-check-option | ELB health check configuration option. | v1.9 or later | |
| kubernetes.io/elb.health-check-options | ELB health check configuration options. Each Service port can be configured separately, and you can configure only some ports. NOTE: Either kubernetes.io/elb.health-check-option or kubernetes.io/elb.health-check-options can be configured. | v1.19.16-r5 or later v1.21.8-r0 or later v1.23.6-r0 or later v1.25.2-r0 or later |
- The following shows how to use kubernetes.io/elb.health-check-option:
apiVersion: v1 kind: Service metadata: name: nginx annotations: kubernetes.io/elb.id: <your_elb_id> # Load balancer ID. Replace it with the actual value. kubernetes.io/elb.class: # Load balancer type kubernetes.io/elb.health-check-flag: 'on' # Enable ELB health check. kubernetes.io/elb.health-check-option: '{ "protocol":"TCP", "delay":"5", "timeout":"10", "max_retries":"3" }' spec: selector: app: nginx ports: - name: service0 port: 80 protocol: TCP targetPort: 80 type: LoadBalancer - For details about how to use kubernetes.io/elb.health-check-options, see Configuring Health Check on Multiple LoadBalancer Service Ports.
Passthrough Capability
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.pass-through | String | Whether the access requests from within the cluster to the Service pass through the ELB load balancer. | v1.19 or later |
For details about applications and use cases, see Configuring Passthrough Networking for a LoadBalancer Service.
Resource Tags
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.tags | String | Add resource tags to a load balancer. This parameter can be configured only when a load balancer is automatically created. A tag is in the format of "key=value". Use commas (,) to separate multiple tags. | v1.23.11-r0, v1.25.6-r0, v1.27.3-r0, v1.23.8-r1, v1.25.3-r1, or later |
For details about applications and use cases, see Using kubectl to Create a Service (Automatically Creating a Load Balancer).
Changing a Custom EIP
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.custom-eip-id | String | ID of the custom EIP, which can be seen on the EIP console. The EIP must be bindable. | v1.23.18-r0, v1.25.13-r0, v1.27.10-r0, v1.28.8-r0, v1.29.4-r0, v1.30.1-r0, or later |
For details about applications and use cases, see Changing a Custom EIP for a LoadBalancer Service.
Configuring a Reclaim Policy for an Auto-created Load Balancer
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.instance-reclaim-policy | String | Optional when load balancers are automatically created. Configure a reclaim policy for an auto-created load balancer. The options are as follows:
NOTE:
| v1.28.15-r60, v1.29.15-r20, v1.30.14-r20, v1.31.10-r20, v1.32.6-r20, v1.33.5-r10, or later |
The following shows how to use the preceding annotation:
apiVersion: v1
kind: Service
metadata:
name: service-0
labels:
app: test
version: v1
namespace: default
annotations:
kubernetes.io/elb.class: performance
kubernetes.io/elb.autocreate: '{"type":"inner","available_zone":["******"],"elb_virsubnet_ids":["******"],"ipv6_vip_virsubnet_id":"******","l7_flavor_name":"","l4_flavor_name":"L4_flavor.elb.pro.max","vip_subnet_cidr_id":"******"}'
kubernetes.io/elb.enterpriseID: '0'
kubernetes.io/elb.lb-algorithm: ROUND_ROBIN
kubernetes.io/elb.health-check-flag: 'on'
kubernetes.io/elb.health-check-option: '{"protocol":"TCP","delay":"5","timeout":"10","max_retries":"3"}'
kubernetes.io/elb.instance-reclaim-policy: retain # Configure a reclaim policy for an auto-created load balancer.
spec:
selector:
app: nginx
version: v1
externalTrafficPolicy: Cluster
ports:
- name: cce-service-0
targetPort: 80
nodePort: 0
port: 80
protocol: TCP
type: LoadBalancer
loadBalancerIP: '' HTTP/HTTPS
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.protocol-port | String | If a Service is HTTP/HTTPS-compliant, configure the protocol and port in the format of protocol:port. where,
| v1.19.16 or later |
| kubernetes.io/elb.cert-id | String | ID of an ELB certificate, which is used as the HTTPS server certificate. How to obtain: Log in to the ELB console and choose Certificates. In the certificate list, copy the ID under the target certificate name. | v1.19.16 or later |
For details about applications and use cases, see Configuring HTTP/HTTPS for LoadBalancer Services.
SNI
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.tls-certificate-ids | String | In ELB, the IDs of SNI certificates that must contain a domain name are separated by commas (,). How to obtain: Log in to the ELB console and choose Certificates. In the certificate list, copy the ID under the target certificate name. | v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later |
HTTPS must be enabled. For details, see Configuring SNI for a LoadBalancer Service.
HTTP/2
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.http2-enable | String | Whether HTTP/2 is enabled. Request forwarding using HTTP/2 improves access performance between your application and the load balancer. However, the load balancer still uses HTTP/1.x to forward requests to the backend server. Options:
Note: HTTP/2 can be enabled or disabled only when the listener uses HTTPS. This parameter is invalid and defaults to false when the listener protocol is HTTP. | v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later |
For details about applications and use cases, see Configuring HTTP/2 for LoadBalancer Services.
Dynamically Adjusting the Weight of the Backend Server
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.adaptive-weight | String | Dynamically adjust the weight of the load balancer backend server based on the number of pods on the server. In this way, the requests received by each pod are more balanced.
| v1.21 or later |
This parameter is invalid in passthrough networking, where dedicated load balancers are used in the CCE Turbo cluster.
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
kubernetes.io/elb.id: <your_elb_id> # Load balancer ID. Replace it with the actual value.
kubernetes.io/elb.class: # Load balancer type
kubernetes.io/elb.adaptive-weight: 'true' # Enable dynamic adjustment of the weight of the backend ECS.
spec:
selector:
app: nginx
ports:
- name: service0
port: 80
protocol: TCP
targetPort: 80
type: LoadBalancer Host Network
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/hws-hostNetwork | String | This annotation can be used by the Service only if the pod uses the host network. After this annotation is used, ELB forwards requests to the host network. Options:
| v1.9 or later |
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
kubernetes.io/elb.id: <your_elb_id> # Load balancer ID. Replace it with the actual value.
kubernetes.io/elb.class: # Load balancer type
kubernetes.io/hws-hostNetwork: 'true' # The load balancer forwards the request to the host network.
spec:
selector:
app: nginx
ports:
- name: service0
port: 80
protocol: TCP
targetPort: 80
type: LoadBalancer Timeout
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.keepalive_timeout | String | Timeout for client connections. If there are no requests reaching the load balancer during this period, the load balancer will disconnect the connection from the client and establish a new connection when there is a new request. Value:
| Dedicated load balancers: v1.19.16-r30, v1.21.10-r10, v1.23.8-r10, v1.25.3-r10, or later Shared load balancers: v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later |
| kubernetes.io/elb.client_timeout | String | Timeout for waiting for a request from a client. There are two cases:
The value ranges from 1 to 300 (in seconds). The default value is 60. | v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later |
| kubernetes.io/elb.member_timeout | String | Timeout for backend server response. After a request is forwarded to the backend server, if the backend server does not respond within the duration specified by member_timeout, the load balancer will stop waiting and return HTTP 504 Gateway Timeout. The value ranges from 1 to 300 (in seconds). The default value is 60. | v1.23.13-r0, v1.25.8-r0, v1.27.5-r0, v1.28.3-r0, or later |
For details about applications and use cases, see Configuring Timeout for a LoadBalancer Service.
Enabling GZIP
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.gzip-enabled | String | LoadBalancer Services support data compression, which reduces the size of files to be transferred, improves file transfer efficiency, and reduces the bandwidth needed for the transmission. If this function is enabled, specific files will be compressed. If it is not enabled, files will not be compressed. By default, data compression is disabled. The files in the following format can be compressed:
This function is available only for HTTP/HTTPS listeners of dedicated load balancers. Deleting the advanced configuration for data compression or the target annotations will not modify the existing ELB settings. | v1.23.14-r0, v1.25.9-r0, v1.27.6-r0, v1.28.4-r0, or later |
For details about applications and use cases, see Configuring GZIP Data Compression for a LoadBalancer Service.
Configuring Port Ranges for a Listener
| Parameter | Type | Description | Supported Cluster Version |
|---|---|---|---|
| kubernetes.io/elb.port-ranges | String | If a dedicated load balancer is used and the TCP, UDP, or TLS protocol is selected, you can create a listener that listens to ports within a certain range from 1 to 65535. You can add a maximum of 10 port ranges that do not overlap for each listener. The parameter value is in the following format, where ports_name and port must be unique: '{"<ports_name_1>":["<port_1>,<port_2>","<port_3>,<port_4>"], "<ports_name_2>":["<port_5>,<port_6>","<port_7>,<port_8>"]}' For example, the port names are cce-service-0 and cce-service-1, and the listener listens to ports 100–200 and 300–400, and 500–600 and 700–800, respectively. '{"cce-service-0":["100,200", "300,400"], "cce-service-1":["500,600", "700,800"]}' NOTE: This function requires ELB. Before using this function, check whether ELB supports full-port listening and forwarding for layer-4 protocols in the current region. | v1.23.18-r0, v1.25.13-r0, v1.27.10-r0, v1.28.8-r0, v1.29.4-r0, v1.30.1-r0, or later |
For details about applications and use cases, see Configuring a Range of Listening Ports for LoadBalancer Services.
Parameters for Automatically Creating a Load Balancer
| Parameter | Mandatory | Type | Description |
|---|---|---|---|
| name | No | String | Name of the automatically created load balancer. The value can contain 1 to 64 characters. Only letters, digits, underscores (_), hyphens (-), and periods (.) are allowed. Default: cce-lb+service.UID |
| type | No | String | Network type of the load balancer.
Default: inner |
| bandwidth_name | Yes for public network load balancers | String | Bandwidth name. The default value is cce-bandwidth-******. The value can contain 1 to 64 characters. Only letters, digits, underscores (_), hyphens (-), and periods (.) are allowed. |
| bandwidth_chargemode | Yes for public network load balancers | String | Bandwidth mode.
|
| bandwidth_size | Yes for public network load balancers | Integer | Bandwidth size. The value ranges from 1 Mbit/s to 2000 Mbit/s by default. Configure this parameter based on the bandwidth range allowed in your region. The minimum increment for bandwidth adjustment varies depending on the bandwidth range.
|
| bandwidth_sharetype | Yes for public network load balancers | String | Bandwidth sharing mode.
|
| eip_type | Yes for public network load balancers | String | EIP type.
The types vary by region. For details, see the EIP console. |
| vip_subnet_cidr_id | No | String | The ID of the IPv4 subnet where the load balancer resides. This subnet is used to allocate IP addresses for the load balancer to provide external services. The IPv4 subnet must belong to the cluster's VPC. If this parameter is not specified, the load balancer and the cluster will be in the same subnet by default. This field can be specified only for clusters v1.21 or later. How to obtain: Log in to the VPC console. In the navigation pane, choose Subnets. Filter the target subnet by the cluster's VPC name, click the subnet name, and copy the IPv4 Subnet ID on the Summary tab. |
| elb_virsubnet_ids | No | Array of strings | The network ID of the subnet where the load balancer is located. This subnet is used to allocate IP addresses for accessing the backend server. If this parameter is not specified, the subnet specified by vip_subnet_cidr_id will be used by default. Load balancers occupy varying numbers of subnet IP addresses based on their specifications. Do not use the subnet CIDR blocks of other resources (such as clusters or nodes) as the load balancer's CIDR block. This parameter is available only for dedicated load balancers. Example: "elb_virsubnet_ids": [ "14567f27-8ae4-42b8-ae47-9f847a4690dd" ] How to obtain: Log in to the VPC console. In the navigation pane, choose Subnets. Filter the target subnet by the cluster's VPC name, click the subnet name, and copy the Network ID on the Summary tab. |
| vip_address | No | String | Private IP address of the load balancer. Only IPv4 addresses are supported. The IP address must be in the ELB CIDR block. If this parameter is not specified, an IP address will be automatically assigned from the ELB CIDR block. This parameter is only available in clusters v1.23.11-r0, v1.25.6-r0, v1.27.3-r0, or later versions. |
| available_zone | Yes | Array of strings | AZ where the load balancer is located. This parameter is available only for dedicated load balancers. |
| l4_flavor_name | No | String | Flavor name of the Layer 4 load balancer. This parameter is mandatory when TCP or UDP is used. This parameter is available only for dedicated load balancers. |
| l7_flavor_name | No | String | Flavor name of the Layer 7 load balancer. This parameter is mandatory when HTTP or HTTPS is used. This parameter is available only for dedicated load balancers. Its value must match that of l4_flavor_name, both of which must be either elastic or fixed. |
| Parameter | Mandatory | Type | Description |
|---|---|---|---|
| delay | No | String | Health check interval (s) Value range: 1 to 50. Default value: 5 |
| timeout | No | String | Health check timeout, in seconds. Value range: 1 to 50. Default value: 10 |
| max_retries | No | String | Maximum number of health check retries. Value range: 1 to 10. Default value: 3 |
| protocol | No | String | Health check protocol. Value options: HTTPS, TLS, gRPC, TCP, UDP, and HTTP To use HTTPS, TLS, or gRPC, you need to use a dedicated load balancer, and the cluster version must be v1.28.15-r90, v1.29.15-r50, v1.30.14-r50, v1.31.14-r10, v1.32.9-r10, v1.33.7-r10, v1.34.3-r0, or later. gRPC and TLS depend on the ELB capabilities. Before using this function, submit a service ticket to enable the required ELB capabilities.
CAUTION: To change the health check protocol (including the switchover between global check and custom check), disable the health check, enable it again, and change the protocol. |
| path | No | String | Health check URL. This parameter needs to be configured when the protocol is HTTP. Default value: / Value range: 1-80 characters |
| Parameter | Mandatory | Type | Description |
|---|---|---|---|
| target_service_port | Yes | String | Port for health check specified by spec.ports. The value consists of the protocol and port number, for example, TCP:80. |
| monitor_port | No | String | Re-specified port for health check. If this parameter is not specified, the service port is used by default. NOTE: Ensure that the port is in the listening state on the node where the pod is located. Otherwise, the health check result will be affected. |
| delay | No | String | Health check interval (s) Value range: 1 to 50. Default value: 5 |
| timeout | No | String | Health check timeout, in seconds. Value range: 1 to 50. Default value: 10 |
| max_retries | No | String | Maximum number of health check retries. Value range: 1 to 10. Default value: 3 |
| protocol | No | String | Health check protocol. Default value: protocol of the associated Service Value options: HTTPS, TLS, gRPC, TCP, UDP, and HTTP To use HTTPS, TLS, or gRPC, you need to use a dedicated load balancer, and the cluster version must be v1.28.15-r90, v1.29.15-r50, v1.30.14-r50, v1.31.14-r10, v1.32.9-r10, v1.33.7-r10, v1.34.3-r0, or later. gRPC and TLS depend on the ELB capabilities. Before using this function, submit a service ticket to enable the required ELB capabilities.
CAUTION: To change the health check protocol (including the switchover between global check and custom check), disable the health check, enable it again, and change the protocol. |
| path | No | String | Health check URL. This parameter needs to be configured when the protocol is HTTP, HTTPS, or gRPC. Default value: / Value range: 1-80 characters |
| expected_codes | No | String | Expected response status code. This parameter is supported only by clusters v1.19.16-r50, v1.21.11-r10, v1.23.9-r10, v1.25.4-r10, v1.27.1-r10, or later. Value:
Default value: 200. The status code ranges from 200 to 599. For gRPC health checks, the default value is 0, and the status code ranges from 0 to 99. This parameter is available only for HTTP, HTTPS, or gRPC. It has no effect on other protocols. |