How Does ELB Perform UDP Health Checks? What Are the Precautions for UDP Health Checks?

• The health check node sends an ICMP request to the backend server based on the health check configuration.
  • If the health check node receives an ICMP reply from the backend server, it considers the backend server healthy and continues the health check.
  • If the health check node does not receive an ICMP reply from the backend server, it considers the backend server unhealthy.
• After receiving the ICMP reply, the health check node sends a UDP probe packet to the backend server.
  • If the health check node receives an ICMP Port Unreachable message from the backend server within the timeout duration, the backend server is considered unhealthy.
  • If the health check node does not receive an ICMP Port Unreachable message from the backend server within the timeout duration, the backend server is considered healthy.

When you use UDP for health checks, retain default parameter settings.

• Check whether the timeout duration is too short.

  One possible cause is that the ICMP Echo Reply or ICMP Port Unreachable message returned by the backend server does not reach the health check node within the timeout duration. As a result, the health check result is inaccurate.

  It is recommended that you change the timeout duration to a larger value.

  UDP health checks are different from other health checks. If the health check timeout duration is too short, the health check result of the backend server frequently toggles back and forth between Healthy and Unhealthy.

• Check whether the backend server restricts the rate at which ICMP messages are generated.

For Linux servers, run the following commands to query the rate limit and rate mask:

sysctl -q net.ipv4.icmp_ratelimit

The default rate limit is 1000.

sysctl -q net.ipv4.icmp_ratemask

The default rate mask is 6168.

If the returned value of the first command is the default value or 0, run the following command to remove the rate limit of Port Unreachable messages:

sysctl -w net.ipv4.icmp_ratemask=6160

For more information, see the Linux Programmer's Manual. On the Linux CLI, run the following command to display the manual:

man 7 icmp

Alternatively, visit http://man7.org/linux/man-pages/man7/icmp.7.html.

Once the rate limit is lifted, the number of ICMP Port Unreachable messages on the backend server will not be limited.

• UDP health checks use ping packets to check the health of the backend server. To ensure smooth transmission of these packets, ensure that ICMP is enabled on the backend server by performing the following:

  Log in to the server and run the following command as user root:

  cat /proc/sys/net/ipv4/icmp_echo_ignore_all

  • If the returned value is 1, ICMP is disabled.
  • If the returned value is 0, ICMP is enabled.
• The health check result may be different from the actual health of the backend server.

  If the backend server runs Linux, the rate of ICMP packets may be limited due to Linux's defense against ping flood attacks when there is a large number of concurrent requests. In this case, if a service exception occurs, the load balancer will not receive error message port XX unreachable and will consider the health check to be successful. As a result, there is an inconsistency between the health check result and the actual server health.