Help Center/ Web Application Firewall/ User Guide (Kuala Lumpur Region)/ FAQs/ Service Interruption Check/ How Do I Troubleshoot 404/502/504 Errors?
Updated on 2024-03-14 GMT+08:00
View PDF
Share

How Do I Troubleshoot 404/502/504 Errors?

If an error, such as 404 Not Found, 502 Bad Gateway, or 504 Gateway Timeout, occurs after a website is connected to WAF, use the following methods to locate the cause and remove the error:

404 Not Found Troubleshooting Process and Suggestions

Refer to Figure 1 to fix the 404 Not Found error occurred after your website is connected to WAF.

Figure 1 Troubleshooting for 404 Not Found error
  • If the page shown in Figure 2 is displayed, the possible causes and solutions are as follows:
    Figure 2 404 page

    Cause 1: A non-standard port is configured when you add the domain name to WAF, but the visitors use the domain name and standard port or use only the domain name to access the website. For example, a non-standard port is configured as shown in Figure 3. A visitor uses https://www.example.com or https://www.example.com:80 to access the website. As a result, 404 error page is displayed.
    Figure 3 Configuration of a non-standard port

    Solution: Add the non-standard port to the URL and access the origin server again, for example, https://www.example.com:8080.

    Cause 2: No non-standard port is configured when the domain name is added to WAF. The visitors use the domain name and a non-standard port or the non-standard port configured for origin server port to access the website. For example, access https://www.example.com:8080 when the protection service shown in Figure 4 is configured.
    Figure 4 Non-standard port not configured

    If no non-standard port is configured, WAF protects services on port 80/443 by default. To protect services on other ports, re-configure domain settings.

    Solution: Use only the domain name to access the website. For example, https://www.example.com.

    Cause 3: The domain name is incorrectly resolved.

    Solution:
    • If the domain name has been added to WAF, resolve the domain name to WAF by referring to Step 4: Modify the DNS Records of the Domain Name.
    • If the domain name is no longer protected by WAF, resolve it to the origin server IP address on the DNS hosting platform.

    Cause 4: If a WAF cluster pointed multiple domain names through HTTPS to an origin server over the same port, origin servers cannot tell which domain name a request originated from. This is because WAF uses persistent connections to forward requests to origin servers and Nginx identifies domain names based on Host and SNI. So, there might be a probability that requests destined for domain name A was mistakenly forwarded to domain name B, which causes 404 not found errors.

    Solution: Modify the server configuration in WAF to route different domain names over different origin server ports.

  • If the response page is not similar the one shown in Figure 2, the possible causes and solutions are as follows:

    Cause: The website does not exist or has been deleted.

    Solution: Check the website.

502 Bad Gateway Troubleshooting Process and Solutions

Your website can be accessed normally after it is connected to WAF. However, after a period of time, the error code 502 is reported frequently. Refer to Figure 5 to fix the issue.

Figure 5 Troubleshooting process for 502 Bad Gateway error
Table 1 Troubleshooting 502 Bad Gateway error

Possible Cause

Solution

Cause 1: Your website is using another security protection software. Such software considers WAF back-to-source IP addresses as malicious and blocks the requests forwarded by WAF.

Configure an access control policy on the origin server to whitelist the WAF IP addresses.

Cause 2: Multiple backend servers are configured for the website. However, one backend server is inaccessible.

Repeat Step 1 to Step 8 to ensure that all origin servers can be accessed.

Cause 3: Your website server may have performance issues.

Contact your website administrator to rectify the fault.

If one of your backend website servers is unreachable, perform the following steps to ensure that the website server configuration is correct.

It takes about two minutes for server information modification to take effect.

  1. Log in to the management console.
  2. Click in the upper left corner of the management console and select a region or project.
  3. Click in the upper left corner and choose Security > Web Application Firewall to go to the Dashboard page.
  4. In the navigation pane on the left, choose Website Settings.
  5. In the Protected Website column, click the target domain name to go to the Basic Information page.
  6. In the Server Information area, click . On the displayed page, check whether the client protocol, server protocol, origin server address, and port used by the origin server are correct.
  7. Check whether each origin server can be accessed properly.

    • Run the following command on the server: 
      curl http://xx.xx.xx.xx:yy -kvv
      • xx.xx.xx.xx indicates the IP address of the origin server. yy indicates the port of the origin server. xx.xx.xx.xx and yy must belong to the same origin server.
      • The host where the curl command can be run must meet the following requirements:
        • The network communication is normal.
        • The curl command has been installed. curl must be manually installed on the host running a Windows operating system. curl is installed along with other operating systems.
      Figure 6 Command output for checking origin server
      • If the command output indicates that the connection is normal, the website can be accessed.
      • If the command output returns connection refused, the origin server is unreachable and website cannot be accessed. Go to Step 8.
    • Enter http://origin server address: origin server port in the address box of the browser and press Enter.
      • If the website can be accessed, the website access is normal.
      • If the website cannot be accessed, the origin server is unreachable and the website cannot be accessed. Go to Step 8.

  8. Check whether the origin server runs properly.

    If not, restart it.

504 Gateway Timeout Troubleshooting Process and Solutions

After you connect your website to WAF, the possibility of 504 gateway timeout errors rises as your website traffic increases. In some other cases, there might be a possibility of 504 gateway timeout error if the visitors access your website through origin server IP addresses. Refer to Figure 7 to fix 504 gateway timeout errors.

Figure 7 Troubleshooting process for 504 Gateway Timeout errors
Table 2 Troubleshooting 504 Gateway Timeout errors

Possible Cause

Troubleshooting

Solution

Cause 1: Backend server performance issues (such as too many connections or high CPU usage)

If the origin server performance is insufficient, check the origin server access logs and access traffic to analyze issues.
  • Optimize the server configurations, including TCP network parameters and ulimit parameters.
  • You are advised to add backend server groups or create new load balancers to support the increasing service workloads, if your website is connected to WAF in cloud mode.
  • If you configure Client Protocol to HTTPS, to relieve burden on backend servers, configure HTTP for Server Protocol for WAF forwarding traffic to backend servers.

    For details, see Editing Server Information.

  • Use CC attack protection rules to block malicious traffic.

Cause 2

  • The WAF back-to-source IP addresses are not whitelisted or service port is not enabled in the security group.
  • WAF back-to-source IP addresses are blocked by the firewall on the origin server.

Follow the solutions below for troubleshooting:

  • Check whether your origin server has security groups, firewalls, and security software deployed.
  • Capture packets on the client and WAF, respectively, at the same time to check whether the origin server firewall proactively discards packets of the persistent connection to WAF.
  • Configure an access control policy on the origin server to whitelist WAF IP addresses.
  • Disable other firewalls and security software on origin servers.

Cause 3: Connection timeout and read timeout

NOTE:
  • A 504 error occurs if the origin server is too slow to respond, for example, a slow response to database queries, a long upload time for a large file, or a faulty origin server.
  • The timeout for WAF to forward traffic to an origin server is 60s or 180s. A 504 error occurs if WAF fails to forward traffic within the configured timeout.

Troubleshooting methods:

  • Bypass WAF and directly access the origin server and then check the response time.
  • View the origin server response time in access logs stored in Log Tank Service (LTS).
  • Bypass WAF, test the file upload function, and check the file size.
  • Database queries are slow.
    • Tune services to shorten the query duration and improve user experience.
    • Modify the request interaction mode so that the persistent connection can have some data transmitted within 60 seconds, such as ACK packets, heartbeat packets, keep-alive packets, and other packets that can keep the session alive.
  • It takes a long time to upload large files.
    • Tune services to shorten the file upload time.
    • An FTP server is recommended for file upload.
    • Upload the file through an IP address or a domain name that is not protected by WAF.
  • The origin server is faulty.

    Check whether the origin server works properly.

Cause 4: The bandwidth of the origin server is insufficient. When the access traffic is heavy, the origin server cannot handle all the traffic with its current bandwidth.

Troubleshooting methods:

  • If you have a layer-7 load balancer deployed in the rear of WAF, you can query 504 logs on the load balancer.
  • If you have a layer-4 load balancer deployed in the rear of WAF, you can query logs in the Traffic exceeded the bandwidth threshold field on the load balancer.
  • If you have an EIP bound to the backend WAF instances, check the EIP traffic monitoring when 504 errors rise to the peak volume.

Increase the bandwidth of the origin server.

Create a load balancer. Use the EIP of the load balancer as the IP address of the origin server and connect the EIP to WAF.

It takes about two minutes for server information modification to take effect.

  1. Create a shared load balancer.
  2. Log in to the management console.
  3. Click in the upper left corner and choose Web Application Firewall under Security.
  4. In the navigation pane on the left, choose Website Settings.
  5. In the Domain Name column, click the domain name. Its information is displayed.
  6. In the Server Information area, click . On the displayed page, click Add.
  7. Set the Server Address to the EIP bound to the load balancer.
  8. Click OK.