How Do I Troubleshoot an Unhealthy Backend Server of a Shared Load Balancer?

Symptom

If a client cannot access a backend server through a load balancer, the backend server is declared unhealthy. You can view the health check results for a backend server on the ELB console.

If a backend server is considered unhealthy, ELB will not route traffic to it until it is declared healthy again.

Background

To check the health of backend servers, shared load balancers use the IP addresses in the backend subnet to send heartbeat requests to the backend servers. For details about how a health check works, see Health Check.

• If health checks are disabled, the load balancer will consider the backend servers healthy by default and still route requests to it.
• When the weight of a backend server is set to 0:
  • Dedicated load balancer: Requests are not routed to the backend server, even if the backend server is considered healthy.
  • Shared load balancer: Requests are not routed to the backend server, and the health check result is unhealthy.

Troubleshooting

You can use ELB self-service troubleshooting to locate and fix unhealthy backend servers. If the faults persist, you can perform further checks based on Table 1.

If you need to change the health check configuration, it takes a while for the changes to be applied. The required time depends on the health check interval and timeout duration.

You can find the health check results in the backend server list of the load balancer.

Checking Security Group Rules

The security group rules of backend servers must allow traffic from the IP addresses in 100.125.0.0/16 to the backend server using the health check protocol and over the health check port.

You can view the health check settings on the summary page of the backend server group, and configure health checks on the summary page of the listener.

You can use ELB self-service troubleshooting to check the security group rules configured for backend servers based on Table 2.

If Transfer Client IP Address is enabled for Layer 4 listeners, network ACL rules and security group rules will not work. You can use access control to limit which IP addresses are allowed to access the listener. Learn how to configure access control.

Checking Network ACL Rules

Network ACL rules are optional for subnets. If network ACL rules are configured for the subnets where the backend servers are deployed, the network ACL rules must allow traffic from the backend subnet of the dedicated load balancer to the backend server subnets using the health check protocol and over the health check port.

Default network ACL rules deny all inbound and outbound traffic. You can configure an inbound rule to allow traffic from the backend subnet of the load balancer through the port of the backend server.

You can use ELB self-service troubleshooting to check the network ACL rules configured for backend servers based on Table 3.

If Transfer Client IP Address is enabled for Layer 4 listeners, network ACL rules and security group rules will not work. You can use access control to limit which IP addresses are allowed or denied to access the listener. Learn how to configure access control.

Checking the Health Check Configuration

1. In the navigation pane on the left, choose Elastic Load Balance > Backend Server Groups.
2. On the Backend Server Groups page, locate the backend server group and click its name.
3. On the Summary page, click Health Check on the right.
   On the Configure Health Check page, view the parameters in the following table. For details about how to set health check parameters, see Modifying Health Check Settings.

   Table 4 Parameters for configuring a health check

   Parameter	Description
   Domain Name	If you use HTTP for health checks and the backend server is configured to verify the Host header, enter the domain name configured for the backend server.
   Protocol	The outbound rules of the network ACL do not allow traffic over health check protocol.
   Port	Configure the backend port as the health check port by referring to Modifying Health Check Settings.
   Path	If HTTP is used for health checks, you must check this parameter. A simple static HTML file is recommended.

   

   • If health check protocol is HTTP and the health check port is normal, change the path or change the health check protocol to TCP.
   • Enter an absolute path. The following is an example:
     • If the URL is http://www.example.com or http://192.168.63.187:9096, enter / as the health check path.
     • If the URL is http://www.example.com/chat/try/, enter /chat/try/ as the health check path.
     • If the URL is http://192.168.63.187:9096/chat/index.html, enter /chat/index.html as the health check path.

Checking Whether the Backend Server Group Is Associated with a Listener

Check whether the backend server group that has unhealthy backend servers is associated with a listener.

If the backend server group is not associated with a listener, health checks may fail.

If the backend server group has been associated with a listener, proceed with the following possible reasons.

Checking Whether an EIP or Private IP Address Is Bound to a Load Balancer

• Check this only when you add a TCP or UDP listener to the load balancer.

If you add a TCP or UDP listener to the load balancer, check whether the load balancer has a private IP address or an EIP bound.

When you create a load balancer for the first time, if no EIP or private IP address is bound to the load balancer, the health check result of backend servers associated with a TCP or UDP listener is Unhealthy.

Checking the Backend Server

If the backend server runs on Windows, use a browser to access https://{Backend server IP address}:{Health check port}. If a 2xx or 3xx code is returned, the backend server is running normally.

• Run the following command on the backend server to check whether the health check port is listened on:

  netstat -anlp | grep port

  If the health check port and LISTEN are displayed, the health check port is in the listening state. As shown in Figure 1, TCP port 880 is listened on.

  If you do not specify a health check port, backend ports are used by default.

  Figure 1 Backend server port listened on
  
  Figure 2 Backend server port not listened on
  

  If the health check port is not in the listening state, the backend server is not listened on. You need to start the application on the backend server and check whether the health check port is listened on.

• For HTTP health checks, run the following command on the backend server to check the status code:

  curl Private IP address of the backend server:Health check port/Health check path -iv

  To perform an HTTP health check, the load balancer initiates a GET request to the backend server. If the following response status codes are displayed, the backend server is considered healthy:

  TCP listeners: 200

  200, 202, or 401 will be returned if the backend server is healthy.

  Figure 3 Unhealthy backend server
  
  Figure 4 Healthy backend server
  
• If the HTTP health check is abnormal, configure a TCP health check. The procedure is described as follows:

  On the Listeners tab page, modify the target listener, select the backend server group for which TCP health check has been configured, or add a backend server group and select TCP as the health check protocol. After you complete the configuration, wait for a while and check the health check result.

Checking the Firewall on the Backend Server

If the firewall or other security software is enabled for the backend server, the software may block the IP addresses from the subnet where the load balancer works.

Configure inbound firewall rules to allow traffic from IP addresses in 100.125.0.0/16 to backend servers.

Checking the Backend Server Route

Check whether the default route configured for the primary NIC (for example, eth0) has been manually modified. If the default route is changed, health check packets may fail to reach the backend server.

ip route

Alternatively, run the following command:

route -n

Figure 5 shows the command output when the backend server route is normal.

Figure 5 Example default route pointing to the gateway

Figure 6 Example default route not pointing to the gateway

If the command output does not contain the first route, or the route does not point to the gateway, configure or modify the default route to point to the gateway.

Checking the Backend Server Load

View the vCPU usage, memory usage, network connections of the backend server on the Cloud Eye console to check whether the backend server is overloaded.

If the workloads are high, connections or requests for health checks may time out.

Checking the hosts.deny File

Verify that IP addresses in backend subnet where the load balancer works are not written to the /etc/hosts.deny file on the backend servers.

Submitting a Service Ticket

If the problem persists, submit a service ticket.