How Do I Troubleshoot an Unhealthy Backend Server of a Dedicated Load Balancer?

Symptom

If a client cannot access a backend server through a load balancer, the backend server is declared unhealthy. You can view the health check results for a backend server on the ELB console.

If a backend server is unhealthy, ELB will not route traffic to it until it is declared healthy again.

Background

To check the health of backend servers, dedicated load balancers use the IP addresses in the backend subnet to send heartbeat requests to the backend servers. For details about how a health check works, see Health Check.

• If health checks are disabled, the load balancer will consider the backend servers healthy by default and still route requests to them.
• When the weight of a backend server is set to 0:
  • Dedicated load balancer: Requests are not routed to this backend server, even if the backend server is healthy.
  • Shared load balancer: Requests are not routed to this backend server, and the health check result is unhealthy.

Troubleshooting

You can use ELB self-service troubleshooting to locate and fix unhealthy backend servers. If the faults persist, you can perform further checks based on Table 1.

Changes to the health check configuration will not be applied immediately, and the required time depends on the health check interval and timeout duration.

You can view the health check results in the backend server list of the load balancer.

Checking Security Group Rules

The security group rules for backend servers must allow traffic from the backend subnet where the load balancer resides to the backend server through the health check protocol and port.

You can view the health check settings on the summary page of the backend server group and configure health checks on the summary page of the listener.

You can use ELB self-service troubleshooting to check the security group rules configured for backend servers based on Table 2.

If the load balancer has Layer 4 listeners and IP as a Backend is disabled, network ACL rules and security group rules will not work. You can use access control to specify which IP addresses are allowed to access the listeners. Learn how to configure access control.

Checking Network ACL Rules

Network ACL rules are optional for subnets. If network ACL rules are configured for the subnets where the backend servers are deployed, the network ACL rules must allow traffic from the backend subnet of the dedicated load balancer to the backend server subnets through the health check protocol and port.

Default network ACL rules deny all inbound and outbound traffic. You can configure an inbound rule to allow traffic from the backend subnet of the load balancer through the port of the backend server.

You can use ELB self-service troubleshooting to check the network ACL rules configured for backend server subnets based on Table 3.

If the load balancer has Layer 4 listeners and IP as a Backend is disabled, network ACL rules and security group rules will not work. You can use access control to specify which IP addresses are allowed to access the listeners. Learn how to configure access control.

Checking the Health Check Configuration

1. Log in to the management console.
2. In the upper left corner of the page, click and select the desired region and project.
3. Click in the upper left corner to display Service List and choose Networking > Elastic Load Balance.
4. In the navigation pane on the left, choose Elastic Load Balance > Backend Server Groups.
5. On the Backend Server Groups page, locate the backend server group and click its name.
6. On the Summary page, click Health Check on the right.
   On the Configure Health Check page, view the parameters in the following table. For details about how to set health check parameters, see Modifying Health Check Settings.

   Table 4 Parameters for configuring a health check

   Parameter	Description
   Domain Name	If you use HTTP for health checks and the backend server is configured to verify the Host header, enter the domain name configured for the backend server.
   Protocol	The outbound rules of the network ACL do not allow traffic over the health check protocol.
   Port	Configure the backend port as the health check port by referring to Modifying Health Check Settings.
   Path	If HTTP is used for health checks, you must check this parameter. A simple static HTML file is recommended.

   

   • If the health check protocol is HTTP and the health check port is normal, change the path or change the health check protocol to TCP.
   • Enter an absolute path. The following are example paths:
     • If the URL is http://www.example.com or http://192.168.63.187:9096, enter / as the health check path.
     • If the URL is http://www.example.com/chat/try/, enter /chat/try/ as the health check path.
     • If the URL is http://192.168.63.187:9096/chat/index.html, enter /chat/index.html as the health check path.

Checking Whether the Backend Server Group Is Associated with a Listener

Check whether the backend server group that has unhealthy backend servers is associated with a listener.

If the backend server group is not associated with a listener, health checks will fail.

If the backend server group has been associated with a listener, proceed with the following possible reasons.

Checking Whether an EIP or Private IP Address Is Bound to the Load Balancer

Check this only when you add a TCP or UDP listener to the load balancer.

If you add a TCP or UDP listener to the load balancer, check whether the load balancer has a private IP address or an EIP bound.

When you create a load balancer for the first time, if no EIP or private IP address is bound to the load balancer, the health check result of backend servers associated with a TCP or UDP listener is Unhealthy.

Checking the Backend Server

If the backend server runs on Windows, use a browser to access https://{Backend server IP address}:{Health check port}. If a 2xx or 3xx code is returned, the backend server is running normally.

• Run the following command on the backend server to check whether the health check port is listened on:

  netstat -anlp | grep port

  If the health check port and LISTEN are displayed, the health check port is in the listening state. As shown in Figure 1, TCP port 880 is listened on.

  If you do not specify a health check port, backend ports are used as the health check port by default.

  Figure 1 Backend server port listened on
  
  Figure 2 Backend server port not listened on
  

  If the health check port is not in the listening state, the backend server is not listened on. You need to start the application on the backend server and check whether the health check port is listened on.

• For HTTP health checks, run the following command on the backend server to check the status code:

  curl Private IP address of the backend server:Health check port/Health check path -iv

  To perform an HTTP health check, the load balancer initiates a GET request to the backend server. If the following response status codes are displayed, the backend server is considered healthy:

  TCP listeners: 200

  200 for HTTP/HTTPS health checks

  Figure 3 Unhealthy backend server
  
  Figure 4 Healthy backend server
  
• If HTTP is used for health checks and the backend server is detected unhealthy, take the following step to configure a TCP health check:

  On the Listeners tab, modify the target listener, select the backend server group for which TCP health check has been configured, or add a backend server group and select TCP as the health check protocol. After you complete the configuration, wait for a while and check the health check result.

Checking the Firewall on the Backend Server

If the firewall or other security software is enabled for the backend server, the software may block the IP addresses from the backend subnet where the load balancer works.

Configure inbound firewall rules to allow traffic from the backend subnet where the load balancer works to backend servers.

Checking the Backend Server Route

Check whether the default route configured for the primary NIC (for example, eth0) has been manually modified. If the default route is changed, health check packets may fail to reach the backend server.

ip route

Alternatively, run the following command:

route -n

Figure 5 shows the command output when the backend server route is normal.

Figure 5 Example default route pointing to the gateway

Figure 6 Example default route not pointing to the gateway

If the command output does not contain the first route, or the route does not point to the gateway, configure or modify the default route to point to the gateway.

Checking the Backend Server Load

View the vCPU usage, memory usage, network connections of the backend server on the Cloud Eye console to check whether the backend server is overloaded.

If the workload is high, connections or requests for health checks may time out.

Checking the hosts.deny File

Verify that IP addresses in backend subnet where the load balancer works are not written to the /etc/hosts.deny file on the backend servers.

Submitting a Service Ticket

If the problem persists, submit a service ticket.