Configuring Multiple Cross-Cluster Backend Servers for a LoadBalancer Service

Notes and Constraints

• This feature works only on the Services created using an existing load balancer, but not on those created using an automatically created load balancer.
• This feature works only on the Services created using a YAML file, not through the console.
• Cross-cluster access Services connect to the same listener. The backend server group under the listener uses the configurations (such as the health check configuration) of the first Service created among the clusters.
• When Services in different clusters connect to the same listener, the cross-cluster access must be configured for all Services.

Precautions

• By default, deleting a cross-cluster Service only deletes the ELB backend server in the current cluster. You can also configure a cascading deletion policy to delete the load balancer resources. Note that when you configure a cascading deletion policy, deleting a Service or cluster will forcibly reclaim the listener and backend server group.
• When using a load balancer for cross-cluster access, ensure cross-cluster access is enabled for all the Services bound to the load balancer. Otherwise, the backend server group of the load balancer may be overwritten.
• After a cross-cluster access Service is created, CCE automatically adds the kubernetes.io/elb.listener-port-ranges-auto-generated or kubernetes.io/elb.mark, and kubernetes.io/elb.listener-port annotations for internal status management. Do not manually specify, modify, or delete these annotations. Doing so will cause load balancer resource reclamation to fail.

Adding Multiple Cross-Cluster Backend Servers That Are in the Same VPC to a Load Balancer

The cluster version must be v1.23.18-r0, v1.25.13-r0, v1.27.10-r0, v1.28.8-r0, v1.29.4-r0, v1.30.1-r0, or later.

If cluster A and cluster B are both located in the same VPC and there is already a load balancer, configure cross-cluster access with the following settings: Set the weight of the Service in cluster A to 80 and that of the Service in cluster B to 20 for traffic distribution. To do so, perform the following operations:

1. Create a Service in cluster A.

   apiVersion: v1 
   kind: Service 
   metadata: 
     annotations:
       kubernetes.io/elb.id: 5083f225-9bf8-48fa-9c8b-67bd9693c4c0   # Load balancer ID. Replace it with the actual value.
       kubernetes.io/elb.class: performance  # Load balancer type
       kubernetes.io/elb.multicluster: 'true'  # Enable cross-cluster access for the Service.
       kubernetes.io/elb.multicluster-loadbalancer-weight: '80'  # Weight of the cluster Service for receiving traffic
       kubernetes.io/elb.multicluster-resource-recycle-policy: member-only  # Policy for deleting load balancer resources. The default value is member-only, indicating that only the backend servers of the load balancer are deleted when the Service is deleted. The cascade policy forcibly reclaims the ELB listener and all associated resources.
     name: nginx 
   spec: 
     ports: 
     - name: service0 
       port: 80   # ELB listener port
       protocol: TCP 
       targetPort: 80
     selector: 
       app: nginx 
     type: LoadBalancer

   Table 1 Key annotation parameters

   Parameter	Mandatory	Type	Description
   kubernetes.io/elb.id	Yes	String	Load balancer ID. This parameter works only with existing load balancers. This parameter cannot be modified after the Service is created.
   kubernetes.io/elb.class	Yes	String	Load balancer type. The value can be union or performance. This parameter cannot be modified after the Service is created.
   kubernetes.io/elb.multicluster	Yes	String	Value true indicates cross-cluster Service access is enabled.
   kubernetes.io/elb.multicluster-loadbalancer-weight	No	String	Weight of the traffic received by the cluster Service. The value ranges from 0 to 100. A larger value indicates more traffic distributed. If this annotation is not specified, the default weight is 1. NOTE: Deleting kubernetes.io/elb.multicluster-loadbalancer-weight will not change the configured weight.
   kubernetes.io/elb.multicluster-resource-recycle-policy	Yes	String	Policy for deleting ELB resources. The default value of this parameter is member-only. member-only: Only the backend servers of the load balancer will be deleted when the Service is deleted. cascade: The ELB listener and all associated resources are forcibly reclaimed when the Service is deleted. Configure this policy only on the first Service that was created.

2. Create a Service in cluster B.

   apiVersion: v1 
   kind: Service 
   metadata: 
     annotations:
       kubernetes.io/elb.id: 5083f225-9bf8-48fa-9c8b-67bd9693c4c0   # Load balancer ID. Replace it with the actual value.
       kubernetes.io/elb.class: performance  # Load balancer type
       kubernetes.io/elb.multicluster: 'true'  # Enable cross-cluster access for the Service.
       kubernetes.io/elb.multicluster-loadbalancer-weight: '20'  # Weight of the cluster Service for receiving traffic
       kubernetes.io/elb.multicluster-resource-recycle-policy: member-only  # Policy for deleting load balancer resources. The default value is member-only, indicating that only the backend servers of the load balancer are deleted when the Service is deleted. The cascade policy forcibly reclaims the ELB listener and all associated resources.
     name: nginx 
   spec: 
     ports: 
     - name: service0 
       port: 80   # The ELB listener port, which must match the Service port in cluster A
       protocol: TCP 
       targetPort: 80
     selector: 
       app: nginx 
     type: LoadBalancer

Adding Multiple Cross-Cluster Backend Servers That Are in Different VPCs to a Load Balancer

• The cluster version must be v1.23.18-r0, v1.25.13-r0, v1.27.10-r0, v1.28.8-r0, v1.29.4-r0, v1.30.1-r0, or later.
• Only existing dedicated load balancers can be used.
• IP as a Backend must be enabled on the ELB console. For details, see Adding Backend Servers in a Different VPC from a Backend Server Group.
• The VPC CIDR blocks cannot conflict with each other, and the VPCs are connected through VPC peering. For details, see VPC Peering Connection.
• For a cluster that is not in the same VPC as the load balancer, the cluster node security group (which is a network interface security group for Cloud Native Network 2.0) must allow the traffic with the source IP addresses in the VPC CIDR blocks of the load balancer to pass.
• When a cross-VPC backend server is added to a load balancer, the security group of the cluster must allow the access from the backend CIDR blocks of the load balancer, and UDP listeners are not allowed. For details, see Restrictions and Limitations.

If cluster A and cluster B are located in different VPCs and there is already a load balancer in one of the VPCs, configure cross-cluster access.

1. Create a Service in cluster A.

   apiVersion: v1 
   kind: Service 
   metadata: 
     annotations:
       kubernetes.io/elb.id: 5083f225-9bf8-48fa-9c8b-67bd9693c4c0   # Load balancer ID. Replace it with the actual value.
       kubernetes.io/elb.class: performance  # Load balancer type
       kubernetes.io/elb.multicluster: 'true'  # Enable cross-cluster access for the Service.
       kubernetes.io/elb.multivpc: 'true'
       kubernetes.io/elb.multicluster-loadbalancer-weight: '80'  # Weight of the cluster Service for receiving traffic
       kubernetes.io/elb.multicluster-resource-recycle-policy: member-only  # Policy for deleting load balancer resources. The default value is member-only, indicating that only the backend servers of the load balancer are deleted when the Service is deleted. The cascade policy forcibly reclaims the ELB listener and all associated resources.
     name: nginx 
   spec: 
     ports: 
     - name: service0 
       port: 80   # ELB listener port
       protocol: TCP 
       targetPort: 80
     selector: 
       app: nginx 
     type: LoadBalancer

   Table 2 Key annotation parameters

   Parameter	Mandatory	Type	Description
   kubernetes.io/elb.id	Yes	String	Load balancer ID. This parameter works only with existing load balancers. This parameter cannot be modified after the Service is created.
   kubernetes.io/elb.class	Yes	String	Load balancer type. Only dedicated load balancers can be used, meaning that the value of this parameter must be performance. This parameter cannot be modified after the Service is created.
   kubernetes.io/elb.multicluster	Yes	String	Value true indicates cross-cluster Service access is enabled.
   kubernetes.io/elb.multivpc	No	String	Value true indicates cross-cluster access in different VPCs is enabled. This function is available only if kubernetes.io/elb.multicluster is set to true. This parameter cannot be modified after the Service is created.
   kubernetes.io/elb.multicluster-loadbalancer-weight	No	String	Weight of the traffic received by the cluster Service. The value ranges from 0 to 100. A larger value indicates more traffic distributed. If this annotation is not specified, the default weight is 1. NOTE: Deleting kubernetes.io/elb.multicluster-loadbalancer-weight will not change the configured weight.
   kubernetes.io/elb.multicluster-resource-recycle-policy	Yes	String	Policy for deleting ELB resources. The default value of this parameter is member-only. member-only: Only the backend servers of the load balancer will be deleted when the Service is deleted. cascade: The ELB listener and all associated resources are forcibly reclaimed when the Service is deleted. Configure this policy only on the first Service that was created.

2. Create a Service in cluster B.

   apiVersion: v1 
   kind: Service 
   metadata: 
     annotations:
       kubernetes.io/elb.id: 5083f225-9bf8-48fa-9c8b-67bd9693c4c0   # Load balancer ID. Replace it with the actual value.
       kubernetes.io/elb.class: performance  # Load balancer type
       kubernetes.io/elb.multicluster: 'true'  # Enable cross-cluster access for the Service.
       kubernetes.io/elb.multivpc: 'true'
       kubernetes.io/elb.multicluster-loadbalancer-weight: '20'  # Weight of the cluster Service for receiving traffic
       kubernetes.io/elb.multicluster-resource-recycle-policy: member-only  # Policy for deleting load balancer resources. The default value is member-only, indicating that only the backend servers of the load balancer are deleted when the Service is deleted. The cascade policy forcibly reclaims the ELB listener and all associated resources.
     name: nginx 
   spec: 
     ports: 
     - name: service0 
       port: 80   # The ELB listener port, which must match the Service port in cluster A
       protocol: TCP 
       targetPort: 80
     selector: 
       app: nginx 
     type: LoadBalancer

Adding Multiple Cluster Backend Servers and ECSs That Are in the Same VPC to a Load Balancer Listener

The cluster version must be v1.23.18-r0, v1.25.13-r0, v1.27.10-r0, v1.28.8-r0, v1.29.4-r0, v1.30.1-r0, or later.

If cluster A and two ECSs are both located in the same VPC and there is already a load balancer, configure access by setting the weight of the Service in cluster A to 80 and that of each ECS to 20 for traffic distribution. To do so, perform the following operations:

1. On the ELB console, create a listener (for example, frontend port 80) and a backend server group, manually add the ECSs to the group, and set the total weight to 20. For details, see Adding Backend Servers in the Same VPC as a Backend Server Group.

   

2. Create a Service in cluster A, connect the Service to the created listener, and set the Service weight to 80.

   apiVersion: v1 
   kind: Service 
   metadata: 
     annotations:
       kubernetes.io/elb.id: 5083f225-9bf8-48fa-9c8b-67bd9693c4c0   # Load balancer ID. Replace it with the actual value.
       kubernetes.io/elb.class: performance  # Load balancer type
       kubernetes.io/elb.multicluster: 'true'  # Enable cross-cluster access for the Service.
       kubernetes.io/elb.multicluster-loadbalancer-weight: '80'  # Weight of the cluster Service for receiving traffic
       kubernetes.io/elb.multicluster-resource-recycle-policy: member-only  # Policy for deleting load balancer resources. The default value is member-only, indicating that only the backend servers of the load balancer are deleted when the Service is deleted. The cascade policy forcibly reclaims the ELB listener and all associated resources.
     name: nginx 
   spec: 
     ports: 
     - name: service0 
       port: 80   # Listener port 80 for connecting to the load balancer
       protocol: TCP 
       targetPort: 80
     selector: 
       app: nginx 
     type: LoadBalancer

   Table 3 Key annotation parameters

   Parameter	Mandatory	Type	Description
   kubernetes.io/elb.id	Yes	String	Load balancer ID. This parameter works only with existing load balancers. This parameter cannot be modified after the Service is created.
   kubernetes.io/elb.class	Yes	String	Load balancer type. The value can be union or performance. This parameter cannot be modified after the Service is created.
   kubernetes.io/elb.multicluster	Yes	String	Value true indicates cross-cluster Service access is enabled.
   kubernetes.io/elb.multicluster-loadbalancer-weight	No	String	Weight of the cluster Service for receiving traffic. If this annotation is not specified, the default weight is 1. NOTE: Deleting kubernetes.io/elb.multicluster-loadbalancer-weight will not change the configured weight.
   kubernetes.io/elb.multicluster-resource-recycle-policy	Yes	String	Policy for deleting ELB resources. The default value of this parameter is member-only. member-only: Only the backend servers of the load balancer will be deleted when the Service is deleted. cascade: The ELB listener and all associated resources are forcibly reclaimed when the Service is deleted. Configure this policy only on the first Service that was created.

3. Check whether the pods have been added to the ELB backend server group and whether the weight is 80.

   

Helpful Links

With cross-cluster backend servers enabled (kubernetes.io/elb.multicluster: 'true'), you can use kubernetes.io/elb.custom-backend-weights to configure weights for individual StatefulSet backend pods. For details about the applications and configurations of cross-cluster backend servers, see Configuring Multiple Cross-Cluster Backend Servers for a LoadBalancer Service. You only need to replace the kubernetes.io/elb.multicluster-loadbalancer-weight settings with the kubernetes.io/elb.custom-backend-weights settings.

The StatefulSet settings in kubernetes.io/elb.custom-backend-weights apply only to a Service's backend servers. Therefore, when using cross-cluster backend servers, configure both kubernetes.io/elb.multicluster and kubernetes.io/elb.custom-backend-weights for Services in each cluster.