Troubleshooting Request Timeout and Connection Interruption After WAF Is Connected

Requests cannot reach the origin server, or the origin server cannot return responses. There are occasional request interruptions and timeouts, an unstable link, packet loss, or jitter.

The network link between WAF and the origin server is abnormal. Issues such as packet loss, route errors, link interruption, and cross-region/VPC private line jitter may occur. As a result, request forwarding and response backhaul are abnormal.

• Capture packets on the client and WAF, respectively, at the same time to check whether requests reach the origin server and whether the origin server returns responses.
• Run the curl http://Origin-server-IP-address:port-kvv command to access the origin server without passing through WAF. Check whether the connection is established and whether the response is returned properly.
• Check whether packet loss, jitter, or interruption occurs on the network link (such as a cross-region Direct Connect connection or VPC peering connection) between WAF and the origin server. If the network link is abnormal, contact the network side or link provider to locate and rectify the fault. If the deployment is cross-region or cross-VPC, you are advised to evaluate the link quality and optimize the network path or deploy the origin server nearby if necessary.

The API response is slow, and a request timeout error is reported. Error 504 or connection interruption may occur in complex service, large file transfer, and persistent connection scenarios.

The time required for the origin server to process services exceeds the normal threshold in some scenarios, including complex database queries, large file uploads and downloads, and complex service logic operations. In addition, the CPU, memory, and number of connections of the origin server are fully occupied, causing performance bottlenecks. If no data is exchanged over a persistent connection for a long time, a timeout may occur.

• View the access logs in WAF to check the origin server response time. Compare the time required for WAF to forward requests with that required for directly connecting to the origin server to determine whether the fault is caused by WAF.
• Identify the cause based on the service type. Scenarios such as database queries, large file uploads and downloads, idle waiting for pushes over persistent connections, and complex service logic operations typically take a long time.
  • Long database query time: Optimize the query statements and indexes to shorten the processing time.
  • If the upload or download of large files takes a long time, optimize the transmission mode (for example, upload files in chunks).
  • Persistent connection services: Ensure that the connection maintains valid data exchange (such as heartbeat and keep-alive packets) to prevent timeouts caused by no data exchange for a long time.
  • If the origin server has performance bottlenecks (for example, the CPU, memory, or number of connections is used up), contact the origin server administrator to optimize the performance.

Normal services and persistent connection services are interrupted, and 504 Gateway Timeout errors or connection interruption occur. However, short request services are normal, and only complex and time-consuming services report errors.

If the WAF back-to-server connection, read, and write timeouts are set to small values, WAF cannot accommodate time-consuming scenarios such as large file transfers, complex queries, and persistent connection idle periods. In this case, the system determines that the request has timed out and interrupts the connection. If these parameters are set to large values, abnormal connections will occupy resources for a long time and cannot be released quickly.

• Adjust the connection timeout, read timeout, and write timeout between WAF and the origin server based on the actual service scenario. For details, see Configuring a Timeout.
• Simply increasing the timeouts may mask other potential faults. Before making any adjustments, use the following methods to check whether the issue is caused by timeout settings rather than network or origin server issues.
  • Short connections and small request body services: The default settings (connection timeout: 30 seconds; read/write timeout: 180 seconds) are usually sufficient.
  • Time-consuming services such as large file uploads and complex queries: You may need to increase the read and write timeouts.
  • Persistent connection services: The connection lifecycle is long, and the data exchange frequency is not fixed. Long idle intervals may occur due to extended processing times on the origin server. Under the default settings, the connection may be prematurely considered timed out.
• After increasing the timeouts, evaluate the impact on the usage of concurrent connection resources to prevent abnormal connections from occupying resources for a long time and causing resource stacking.

Requests are intermittently interrupted or time out. WAF cannot forward requests to the origin server, and no service response is returned. The persistent connection fails to be established or is frequently disconnected.

The origin server security group, firewall configured in Cloud Firewall (CFW), or third-party security software may not allow the WAF back-to-origin IP addresses and service port. As a result, WAF back-to-origin traffic may be identified as abnormal and blocked or discarded.

• Check the security group configuration of the origin server to see whether WAF back-to-origin IP addresses and corresponding ports are allowed. For details about how to allow WAF back-to-origin IP addresses to access the origin server, see How Do I Whitelist the Back-to-Origin IP Addresses of Cloud WAF?
• Check whether a firewall (such as CFW) or security software is deployed on the origin server and whether it blocks traffic from WAF back-to-origin IP addresses. If a firewall (such as CFW) is deployed on the origin server, add WAF back-to-origin IP addresses to the allowlist on the firewall. For third-party security software on the origin server, adjust the policies to ensure they do not block WAF back-to-origin IP addresses.
• Check the blocking logs on the origin server for packet loss or blocking records involving WAF back-to-origin IP addresses.