What Should I Do If An Error Occurred During a LoadBalancer Update?

Symptom

An error occurs when a LoadBalancer is updated. The error information is as follows:

(combined from similar events):Details:Update member of listener/pool(dc9098a3-e004-4e60-ac6c-44a9a04bd8f8/539490e1-51c2-4c09-b4df-10730f77e35f) error: Failed to create member :(error_msg":"Quota exceeded for resources: members_per_pool","error_code":"ELB.8905","request_id":"e064fd46211318ff57f455b29c07c821"},status code: 409

Check Item 1: Check Whether the Number of Backend Servers Reaches the Upper Limit

By default, a maximum of 500 backend servers can be added to a backend server group of a load balancer. When a dedicated load balancer is used for creating a Service in a CCE Turbo cluster, a backend server is created on the ELB console for each Service pod. If the number of backend servers exhausts the upper limit, the preceding error occurs.

Solution: Properly plan the backend servers of the load balancer based on service requirements. For details, see the following documents:

Check Item 2: Check Whether the Backend Server Health Check Is Abnormal

To ensure uninterrupted service, a new backend server is added first during the load balancer backend server update. The original backend server will be deleted only after the new backend server is available.

However, if the backend server quota is used up, no more backend servers can be added. As a result, the preceding error occurs, and the existing backend servers will be updated directly. If the health checks of all updated backend servers failed due to incorrect configuration during the Service update, the original normal backend servers will not be deleted to ensure normal services. In this case, the incorrect configurations apply only to some backend servers, while the other backend servers still keep the original configurations.

Solution: If the backend server quota is used up, configure the correct health check protocol and port when updating the Service and then check whether the health check is performed successfully.