How Do I Resolve Issues with a LoadBalancer Service?
LoadBalancer Services offer services through load balancers. When you create a LoadBalancer Service, you have the option to choose an existing load balancer or have one created automatically. If you choose an existing load balancer, CCE will assign resources like the load balancer listeners and backend server groups to the Service. If you choose to automatically create a load balancer, CCE will assign resources like the load balancer listeners and backend server groups to the Service and create a load balancer. The following describes how to resolve issues with a LoadBalancer Service.
Troubleshooting Process
- Log in to the CCE console.
- Click the cluster name to access the cluster console. In the navigation pane, choose Services & Ingresses.
- Locate the row containing the target Services and click View Events in the Operation column to check whether there is any abnormal event.
- If there is, you can resolve the problem based on the events and Troubleshooting Based on the Abnormal Events.
- If there is not, the problem may be due to access issues or peripheral service configurations. You can resolve the problem by referring to Troubleshooting Based on the Common Problems.
Troubleshooting Based on the Abnormal Events
The following table lists the solutions to different abnormal events.
|
Error Information |
Description |
Solution |
|---|---|---|
|
Quota exceeded for resources: loadbalancer |
The load balancer fails to be automatically created for the Service due to insufficient quota of load balancers. |
|
|
Quota exceeded for resources: listener |
Listeners failed to be created for the load balancer associated with the Service due to insufficient quota of listeners. |
Apply for a higher quota. The default quota is 200. For details about how to obtain the quota details and increase the quota, see How Do I Increase My Quota? |
|
can't find backend pod with service |
The Service does not have any associated backend pods. You need to make sure that there are pods associated with the Service and that they are functioning properly. |
|
|
failed to ensure load balancer: Not Found |
The load balancer associated with the Service is not present. |
Log in to the ELB console. In the corresponding region, search for the load balancer based on the value of kubernetes.io/elb.id in annotations of the Service. If the needed load balancer is not present, create a Service as needed and choose another existing load balancer or create a new one automatically for the Service. |
|
Update loadbalancer of service([ServiceName]/[Namespace]) error: listener is empty |
The listener of the load balancer associated with the Service may be deleted by mistake. |
Log in to the ELB console. In the corresponding region, search for the load balancer based on the value of kubernetes.io/elb.id in annotations of the service and check whether the listener is present based on the port. If the specified listener of the load balancer is not present, create a Service as needed. |
|
Failed to CreateListener : request failed: {"error_msg":"Load Balancer [ELB id] already has a listener with protocol_port of [port number].","error_code":"ELB.8907","request_id":"xxx"}, status code: 409 |
A listener port conflict occurs. Another listener is already using that port on ELB. |
Update the Service and configure another available port. |
|
Failed to create member : {"error_msg":"Vpc [cluster-VPC-ID] of member's subnet_cidr [subnet-to-which-the-listener-belongs] and vpc [ELB-VPC-ID] of loadbalancer [ELB ID] mismatch","error_code":"ELB.8902","request_id":"xxx"}, status code: 400 |
When a Service is created, it has been associated with an existing load balancer. However, the load balancer is in a different VPC from the cluster. |
Ensure that the selected load balancer is in the same VPC as the cluster. |
|
Failed to CreateListener : request failed: {"error_msg":"Loadbalancer [ELB id] has no flavor of type L7_elastic and cannot create listeners of type L7.","error_code":"ELB.8907","request_id":"xxx"}, status code: 409 |
Layer-7 listeners cannot be created for the network load balancers. |
Select a load balancer of the required specifications. To create an HTTP/HTTPS Layer-7 listener, use a load balancer with application load balancing. |
|
Failed to CreateListener : request failed: {"error_msg":"Loadbalancer [ELB id] has only flavor of type l7 and cannot create listeners of type l4.","error_code":"ELB.8907","request_id":"xxx"}, status code: 409 |
Layer 4 listeners cannot be created for the application load balancers. |
Select a load balancer of the required specifications. To create a TCP/UDP Layer 4 listener, use a load balancer with network load balancing. |
Troubleshooting Based on the Common Problems
If there are no abnormal events, resolve the problems by referring to the following table.
|
Category |
Symptom |
Reference |
|---|---|---|
|
ELB access |
The load balancer is experiencing an imbalance in its load. |
This symptom is due to the improper configuration of the backend routing rules. For details, see Imbalance Load of a Load Balancer. |
|
A load balancer is inaccessible from within a cluster. |
If the service affinity of a Service is set to the node level, that is, the value of externalTrafficPolicy is Local, the Service may fail to be accessed from within the cluster (specifically, nodes or containers). For details, see Why Can't the ELB Address Be Used to Access Workloads in a Cluster? |
|
|
A load balancer is inaccessible from outside a cluster. |
The load balancer or its backend servers are configured improperly. For details, see An Inaccessible Load Balancer from Outside a Cluster. |
|
|
ELB configuration |
The configuration of a load balancer is changed. |
|
|
The backend of a load balancer is automatically deleted. |
||
|
ELB deletion issue |
The load balancer associated with the Service is not deleted when the Service is deleted. |
ELB uses different automatic deletion policies in different scenarios. For details, see Load Balancer Deletion Policies. |
Imbalance Load of a Load Balancer
Possible Cause
The backend routing rules of the load balancer are not properly configured.
Solution
- For a Service whose externalTrafficPolicy is Local, set the backend routing rules of the load balancer to the weighted round robin, which means, add the kubernetes.io/elb.lb-algorithm: ROUND_ROBIN annotation to the Service.
- For a service that uses a persistent connection, set the backend routing rules of the load balancer to the weighted least connections, which means, add the kubernetes.io/elb.lb-algorithm: LEAST_CONNECTIONS annotation to the Service.
An Inaccessible Load Balancer from Outside a Cluster
Possible Cause
The ELB configuration is incorrect or there is an issue with the backend servers.
Procedure
- View the Service events and handle the abnormal events: (For details, see Troubleshooting Based on the Abnormal Events.)
kubectl -n {your-namespace} describe svc {your-svc-name}
- Check whether the backend server group of the port or URL meets the expectation.
If there is no backend server group, check whether the service pods are functioning properly. If the associated pods are not functioning properly, rectify the pod fault. For details, see How Can I Locate the Root Cause If a Workload Is Abnormal?
Why Was the Configuration of a Load Balancer Changed?
Possible Cause
Under certain conditions, CCE automatically updates the load balancer configuration based on the configuration of the Service with which the load balancer associated. Any changes made to the load balancer configuration on the ELB console may be overwritten.
Solution
Configure load balancers associated with the LoadBalancer Services using annotations. For details about how to configure annotations, see Configuring LoadBalancer Services Using Annotations.
Do not manually change any configurations of a load balancer created and managed by CCE clusters on the ELB console. Doing so may result in the loss of configuration and render the Service inaccessible.
Load Balancer Deletion Policies
The automatic deletion policies of the load balancers when their associated Services are deleted are as follows:
- For a load balancer that is automatically created by its associated Service:
- If the load balancer is not being used by any other Services, it will be automatically deleted when the original Service is deleted.
- If there are other Services associated with the load balancer, it will not be deleted when the original Service is deleted.
- If an existing load balancer is selected for a Service, the original load balancer will not be deleted when the Service is deleted.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
