How Do I Troubleshoot an Unhealthy Backend Server of a Shared Load Balancer?
Symptom
If a client cannot access a backend server through a load balancer, the backend server is declared unhealthy. You can view the health check results for a backend server on the ELB console.
If a backend server is considered unhealthy, ELB will not route traffic to it until it is declared healthy again.
Background
To check the health of backend servers, shared load balancers use the IP addresses in the backend subnet to send heartbeat requests to the backend servers. For details about how a health check works, see Health Check.
- If health checks are disabled, the load balancer will consider the backend servers healthy by default and still route requests to it.
- When the weight of a backend server is set to 0:
- Dedicated load balancer: Requests are not routed to the backend server, even if the backend server is considered healthy.
- Shared load balancer: Requests are not routed to the backend server, and the health check result is unhealthy.
Troubleshooting
You can use ELB self-service troubleshooting to locate and fix unhealthy backend servers. If the faults persist, you can perform further checks based on Table 1.
If you need to change the health check configuration, it takes a while for the changes to be applied. The required time depends on the health check interval and timeout duration.
You can find the health check results in the backend server list of the load balancer.
Method |
Items |
---|---|
Self-service troubleshooting |
|
Other check items |
Checking Whether the Backend Server Group Is Associated with a Listener |
Checking Whether an EIP or Private IP Address Is Bound to a Load Balancer |
|
Checking Security Group Rules
The security group rules of backend servers must allow traffic from the IP addresses in 100.125.0.0/16 to the backend server using the health check protocol and over the health check port.
You can view the health check settings on the summary page of the backend server group, and configure health checks on the summary page of the listener.
You can use ELB self-service troubleshooting to check the security group rules configured for backend servers based on Table 2.
Check Item |
Solution |
---|---|
The source IP address configured for the inbound rule |
Ensure that the inbound rules of the security group allow traffic from the IP addresses in 100.125.0.0/16 to the backend server using the health check protocol and over the health check port. For details, see configuring security group rules (shared load balancers). |
The port configured for the inbound rule |
|
The protocol configured for the inbound rule |
|
The source IP address configured for the outbound rule |
Ensure that the outbound rules of the security group allow traffic from the IP addresses in 100.125.0.0/16 to the backend server using the health check protocol and over the health check port. For details, see configuring security group rules (shared load balancers). |
The port configured for the outbound rule |
|
The protocol configured for the outbound rule |
If Transfer Client IP Address is enabled for Layer 4 listeners, network ACL rules and security group rules will not work. You can use access control to limit which IP addresses are allowed to access the listener. Learn how to configure access control.
Checking Network ACL Rules
Network ACL rules are optional for subnets. If network ACL rules are configured for the subnets where the backend servers are deployed, the network ACL rules must allow traffic from the backend subnet of the dedicated load balancer to the backend server subnets using the health check protocol and over the health check port.
Default network ACL rules deny all inbound and outbound traffic. You can configure an inbound rule to allow traffic from the backend subnet of the load balancer through the port of the backend server.
You can use ELB self-service troubleshooting to check the network ACL rules configured for backend servers based on Table 3.
Check Item |
Solution |
---|---|
The protocol configured for the inbound rule |
Ensure that the inbound rules of the network ACL allow traffic from IP addresses in 100.125.0.0/16 to the backend server subnets using the health check protocol and over the health check port. For details, see Configuring Network ACL Rules (Shared Load Balancer). |
The source IP address configured for the inbound rule |
|
The source port configured for the inbound rule |
|
The destination address configured for the inbound rule |
|
The destination port configured for the inbound rule |
|
The protocol configured for the outbound rule |
Ensure that the outbound rules of the network ACL allow traffic from the IP addresses in 100.125.0.0/16 to the backend server subnets using the health check protocol and over the health check port. For details, see Configuring Network ACL Rules (Shared Load Balancer). |
The source IP address configured for the outbound rule |
|
The source port configured for the outbound rule |
|
The destination address configured for the outbound rule |
|
The destination port configured for the outbound rule |
If Transfer Client IP Address is enabled for Layer 4 listeners, network ACL rules and security group rules will not work. You can use access control to limit which IP addresses are allowed or denied to access the listener. Learn how to configure access control.
Checking the Health Check Configuration
- In the navigation pane on the left, choose Elastic Load Balance > Backend Server Groups.
- On the Backend Server Groups page, locate the backend server group and click its name.
- On the Summary page, click Health Check on the right.
On the Configure Health Check page, view the parameters in the following table. For details about how to set health check parameters, see Modifying Health Check Settings.
Table 4 Parameters for configuring a health check Parameter
Description
Domain Name
If you use HTTP for health checks and the backend server is configured to verify the Host header, enter the domain name configured for the backend server.
Protocol
The outbound rules of the network ACL do not allow traffic over health check protocol.
Port
Configure the backend port as the health check port by referring to Modifying Health Check Settings.
Path
If HTTP is used for health checks, you must check this parameter. A simple static HTML file is recommended.
- If health check protocol is HTTP and the health check port is normal, change the path or change the health check protocol to TCP.
- Enter an absolute path. The following is an example:
- If the URL is http://www.example.com or http://192.168.63.187:9096, enter / as the health check path.
- If the URL is http://www.example.com/chat/try/, enter /chat/try/ as the health check path.
- If the URL is http://192.168.63.187:9096/chat/index.html, enter /chat/index.html as the health check path.
Checking Whether the Backend Server Group Is Associated with a Listener
Check whether the backend server group that has unhealthy backend servers is associated with a listener.
If the backend server group is not associated with a listener, health checks may fail.
If the backend server group has been associated with a listener, proceed with the following possible reasons.
Checking Whether an EIP or Private IP Address Is Bound to a Load Balancer
- Check this only when you add a TCP or UDP listener to the load balancer.
If you add a TCP or UDP listener to the load balancer, check whether the load balancer has a private IP address or an EIP bound.
When you create a load balancer for the first time, if no EIP or private IP address is bound to the load balancer, the health check result of backend servers associated with a TCP or UDP listener is Unhealthy.
Checking the Backend Server
If the backend server runs on Windows, use a browser to access https://{Backend server IP address}:{Health check port}. If a 2xx or 3xx code is returned, the backend server is running normally.
- Run the following command on the backend server to check whether the health check port is listened on:
netstat -anlp | grep port
If the health check port and LISTEN are displayed, the health check port is in the listening state. As shown in Figure 1, TCP port 880 is listened on.
If you do not specify a health check port, backend ports are used by default.Figure 2 Backend server port not listened on
If the health check port is not in the listening state, the backend server is not listened on. You need to start the application on the backend server and check whether the health check port is listened on.
- For HTTP health checks, run the following command on the backend server to check the status code:
curl Private IP address of the backend server:Health check port/Health check path -iv
To perform an HTTP health check, the load balancer initiates a GET request to the backend server. If the following response status codes are displayed, the backend server is considered healthy:
TCP listeners: 200
200, 202, or 401 will be returned if the backend server is healthy.
Figure 3 Unhealthy backend server
Figure 4 Healthy backend server
- If the HTTP health check is abnormal, configure a TCP health check. The procedure is described as follows:
On the Listeners tab page, modify the target listener, select the backend server group for which TCP health check has been configured, or add a backend server group and select TCP as the health check protocol. After you complete the configuration, wait for a while and check the health check result.
Checking the Firewall on the Backend Server
If the firewall or other security software is enabled for the backend server, the software may block the IP addresses from the subnet where the load balancer works.
Configure inbound firewall rules to allow traffic from IP addresses in 100.125.0.0/16 to backend servers.
Checking the Backend Server Route
Check whether the default route configured for the primary NIC (for example, eth0) has been manually modified. If the default route is changed, health check packets may fail to reach the backend server.
ip route
Alternatively, run the following command:
route -n
Figure 5 shows the command output when the backend server route is normal.
If the command output does not contain the first route, or the route does not point to the gateway, configure or modify the default route to point to the gateway.
Checking the Backend Server Load
View the vCPU usage, memory usage, network connections of the backend server on the Cloud Eye console to check whether the backend server is overloaded.
If the workloads are high, connections or requests for health checks may time out.
Checking the hosts.deny File
Verify that IP addresses in backend subnet where the load balancer works are not written to the /etc/hosts.deny file on the backend servers.
Submitting a Service Ticket
If the problem persists, submit a service ticket.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot