Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
Help Center/ Elastic Load Balance/ FAQs/ Health Checks/ How Do I Troubleshoot an Unhealthy Backend Server of a Dedicated Load Balancer?

How Do I Troubleshoot an Unhealthy Backend Server of a Dedicated Load Balancer?

Updated on 2024-09-24 GMT+08:00

Symptom

If a client cannot access a backend server through a load balancer, the backend server is declared unhealthy. You can view the health check results for a backend server on the ELB console.

If a backend server is considered unhealthy, ELB will not route traffic to it until it is declared healthy again.

Background

To check the health of backend servers, dedicated load balancers use the IP addresses in the backend subnet to send heartbeat requests to the backend servers. For details about how a health check works, see Health Check.

  • If health checks are disabled, the load balancer will consider the backend servers healthy by default and route requests to them.
  • When the weight of a backend server is set to 0:
    • Dedicated load balancer: Requests are not routed to the backend server, even if the backend server is considered healthy.
    • Shared load balancer: Requests are not routed to the backend server, and the health check result is unhealthy.

Troubleshooting

You can use ELB self-service troubleshooting to locate and fix unhealthy backend servers. If the faults persist, you can perform further checks based on Table 1.

If you need to change the health check configuration, it takes a while for the changes to be applied. The required time depends on the health check interval and timeout duration.

You can find the health check results in the backend server list of the load balancer.

Checking Security Group Rules

The security group rules of backend servers must allow traffic from the backend subnet where the load balancer resides to the backend server using the health check protocol and over the health check port.

You can view the health check settings on the summary page of the backend server group, and configure health checks on the summary page of the listener.

You can use ELB self-service troubleshooting to check the security group rules configured for backend servers based on Table 2.

Table 2 Checking security group rules

Check Item

Solution

The source IP address configured for the inbound rule

Ensure that the inbound rules of the security group allow traffic from the backend subnet where the load balancer resides to the backend server using the health check protocol and over the health check port.

For details, see configuring security group rules (dedicated load balancers).

The port configured for the inbound rule

The protocol configured for the inbound rule

The source IP address configured for the outbound rule

Ensure that the outbound rules of the security group allow traffic from the backend subnet where the dedicated load balancer resides to the backend server using the health check protocol and over the health check port.

For details, see configuring security group rules (dedicated load balancers).

The port configured for the outbound rule

The protocol configured for the outbound rule

NOTE:

If the load balancer has Layer 4 listeners and IP as a Backend is disabled, network ACL rules and security group rules will not take effect. You can use access control to limit which IP addresses are allowed to access the listener. Learn how to configure access control.

Checking Network ACL Rules

Network ACL rules are optional for subnets. If network ACL rules are configured for the subnets where the backend servers are deployed, the network ACL rules must allow traffic from the backend subnet of the dedicated load balancer to the backend server subnets using the health check protocol and over the health check port.

Default network ACL rules deny all inbound and outbound traffic. You can configure an inbound rule to allow traffic from the backend subnet of the load balancer through the port of the backend server.

You can use ELB self-service troubleshooting to check the network ACL rules configured for backend servers based on Table 3.

Table 3 Checking network ACL rules

Check Item

Solution

The protocol configured for the inbound rule

Ensure that the inbound rules of the network ACL allow traffic from the backend subnet of the dedicated load balancer to the backend server subnets using the health check protocol and over the health check port.

For details, see Network ACL Rules (Dedicated Load Balancer).

The source IP address configured for the inbound rule

The source port configured for the inbound rule

The destination address configured for the inbound rule

The destination port configured for the inbound rule

The protocol configured for the outbound rule

Ensure that the outbound rules of the network ACL allow traffic from the backend subnet of the dedicated load balancer to the backend server subnets using the health check protocol and over the health check port.

For details, see Network ACL Rules (Dedicated Load Balancer).

The source IP address configured for the outbound rule

The source port configured for the outbound rule

The destination address configured for the outbound rule

The destination port configured for the outbound rule

NOTE:

If the load balancer has Layer 4 listeners and IP as a Backend is disabled, network ACL rules and security group rules will not take effect. You can use access control to limit which IP addresses are allowed to access the listener. Learn how to configure access control.

Checking the Health Check Configuration

  1. Log in to the management console.
  2. In the upper left corner of the page, click and select the desired region and project.
  3. Click in the upper left corner to display Service List and choose Networking > Elastic Load Balance.
  4. In the navigation pane on the left, choose Elastic Load Balance > Backend Server Groups.
  5. On the Backend Server Groups page, locate the backend server group and click its name.
  6. On the Summary page, click Health Check on the right.
    On the Configure Health Check page, view the parameters in the following table. For details about how to set health check parameters, see Modifying Health Check Settings.
    Table 4 Parameters for configuring a health check

    Parameter

    Description

    Domain Name

    If you use HTTP for health checks and the backend server is configured to verify the Host header, enter the domain name configured for the backend server.

    Protocol

    The outbound rules of the network ACL do not allow traffic over health check protocol.

    Port

    Configure the backend port as the health check port by referring to Modifying Health Check Settings.

    Path

    If HTTP is used for health checks, you must check this parameter. A simple static HTML file is recommended.

    NOTE:
    • If health check protocol is HTTP and the health check port is normal, change the path or change the health check protocol to TCP.
    • Enter an absolute path. The following is an example:
      • If the URL is http://www.example.com or http://192.168.63.187:9096, enter / as the health check path.
      • If the URL is http://www.example.com/chat/try/, enter /chat/try/ as the health check path.
      • If the URL is http://192.168.63.187:9096/chat/index.html, enter /chat/index.html as the health check path.

Checking Whether the Backend Server Group Is Associated with a Listener

Check whether the backend server group that contains unhealthy backend servers is associated with a listener by referring to Viewing a Backend Server Group.

If the backend server group is not associated with a listener, health checks may fail.

If the backend server group has been associated with a listener, proceed with the following possible reasons.

Checking Whether an EIP or Private IP Address Is Bound to a Load Balancer

NOTE:
  • Check this only when you add a TCP or UDP listener to the load balancer.

If you add a TCP or UDP listener to the load balancer, check whether the load balancer has a private IP address or an EIP bound.

When you create a load balancer for the first time, if no EIP or private IP address is bound to the load balancer, the health check result of backend servers associated with a TCP or UDP listener is Unhealthy.

Checking the Backend Server

NOTE:

If the backend server runs on Windows, use a browser to access https://{Backend server IP address}:{Health check port}. If a 2xx or 3xx code is returned, the backend server is running normally.

  • Run the following command on the backend server to check whether the health check port is listened on:
    netstat -anlp | grep port

    If the health check port and LISTEN are displayed, the health check port is in the listening state. As shown in Figure 1, TCP port 880 is listened on.

    If you do not specify a health check port, backend ports are used by default.
    Figure 1 Backend server port listened on
    Figure 2 Backend server port not listened on

    If the health check port is not in the listening state, the backend server is not listened on. You need to start the application on the backend server and check whether the health check port is listened on.

  • For HTTP health checks, run the following command on the backend server to check the status code:
    curl Private IP address of the backend server:Health check port/Health check path -iv

    To perform an HTTP health check, the load balancer initiates a GET request to the backend server. If the following response status codes are displayed, the backend server is considered healthy:

    TCP listeners: 200

    Dedicated load balancers: 200 for HTTP/HTTPS health checks

    Figure 3 Unhealthy backend server
    Figure 4 Healthy backend server
  • If the HTTP health check is abnormal, configure a TCP health check. The procedure is described as follows:

    On the Listeners tab page, modify the target listener, select the backend server group for which TCP health check has been configured, or add a backend server group and select TCP as the health check protocol. After you complete the configuration, wait for a while and check the health check result.

Checking the Firewall on the Backend Server

If the firewall or other security software is enabled for the backend server, the software may block the IP addresses from the subnet where the load balancer works.

Configure inbound firewall rules to allow traffic from the backend subnet where the load balancer works to backend servers.

Checking the Backend Server Route

Check whether the default route configured for the primary NIC (for example, eth0) has been manually modified. If the default route is changed, health check packets may fail to reach the backend server.

Run the following command on the backend server to check whether the default route points to the gateway (For Layer 3 communications, the default route must be configured to point to the gateway of the VPC subnet where the backend server resides):
ip route

Alternatively, run the following command:

route -n

Figure 5 shows the command output when the backend server route is normal.

Figure 5 Example default route pointing to the gateway
Figure 6 Example default route not pointing to the gateway

If the command output does not contain the first route, or the route does not point to the gateway, configure or modify the default route to point to the gateway.

Checking the Backend Server Load

View the vCPU usage, memory usage, network connections of the backend server on the Cloud Eye console to check whether the backend server is overloaded.

If the workloads are high, connections or requests for health checks may time out.

Checking the hosts.deny File

Verify that IP addresses in backend subnet where the load balancer works are not written to the /etc/hosts.deny file on the backend servers.

Submitting a Service Ticket

If the problem persists, submit a service ticket.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback