Using WAF to Block Crawler Attacks

Application Scenarios

Web crawlers make network information collection and query easy, but they also introduce the following negative impacts:

• Web crawlers always consume too much server bandwidth and increase server load as they use specific policies to browser as much information of high value on a website as possible.
• Bad actors may use web crawlers to launch DoS attacks against websites. As a result, websites may fail to provide normal services due to resource exhaustion.
• Bad actors may use web crawlers to steal mission-critical data on your websites, which will damage your economic interests.

To comprehensively mitigate crawler attacks against websites, WAF provides three anti-crawler policies, general check and web shell detection by identifying User-Agent, website anti-crawler by checking browser validity, and CC attack protection by limiting the access traffic rate.

Overview

Figure 1 shows how WAF detects crawlers, where step 1 and step 2 are called JS challenges and step 3 is called JS authentication.

Figure 1 JavaScript Anti-Crawler protection process

By collecting statistics on the number of JavaScript challenges and authentication responses, the system calculates how many requests the JavaScript anti-crawler defends. In Figure 2, the JavaScript anti-crawler has logged 18 events, 16 of which are JavaScript challenge responses, and 2 of which are JavaScript authentication responses. Other indicates the number of WAF authentication requests fabricated by the crawler.

Figure 2 Parameters of a JavaScript anti-crawler protection rule

Limitations and Constraints

• Cookies must be enabled and JavaScript supported by any browser used to access a website protected by anti-crawler protection rules.
• If your service is connected to CDN, exercise caution when using the JS anti-crawler function.

  CDN caching may impact JS anti-crawler performance and page accessibility.

Resource and Cost Planning

Step 1: Buy the Professional Edition Cloud WAF

The following describes how to buy the standard edition cloud WAF.

1. Log in to the WAF console.
2. In the upper right corner of the page, click Buy WAF. On the purchase page displayed, select Cloud Mode for WAF Mode.
   • Region: Select the region nearest to your services WAF will protect.
   • Edition: Select Professional.
   • Expansion Package and Required Duration: Set them based on site requirements.

3. Confirm the product details and click Buy Now in the lower right corner of the page.
4. Check the order details and read the WAF Disclaimer. Then, select the box and click Pay Now.
5. On the payment page, select a payment method and pay for your order.

   After the order is paid, click Access Console to go to the Dashboard page. Hover over the Product Details area to view the edition of the purchased instance and its specifications.

Step 2: Add Website Information to WAF

The following example shows how to add a website to WAF in cloud CNAME access mode.

1. In the navigation pane on the left, choose Website Settings.
2. In the upper left corner of the website list, click Add Website.
3. Select Cloud - CNAME and click Configure Now.
4. Configure website information as prompted.
   Figure 3 Configuring basic information
   

   Table 2 Key parameters

   Parameter	Description	Example Value
   Domain Name	Domain name you want to add to WAF for protection. The domain name has an ICP license. You can enter a single domain name (for example, top-level domain name example.com or level-2 domain name www.example.com) or a wildcard domain name (*.example.com).	www.example.com
   Protected Port	The port over which the website traffic goes	Standard ports
   Server Configuration	Web server address settings. You need to configure the client protocol, server protocol, server weights, server address, and server port. Client Protocol: protocol used by a client to access a server. The options are HTTP and HTTPS. Server Protocol: protocol used by WAF to forward client requests. The options are HTTP and HTTPS. Server Address: public IP address (generally corresponding to the A record of the domain name configured on the DNS) or domain name (generally corresponding to the CNAME record of the domain name configured on the DNS) of the web server that a client accesses. Server Port: service port over which the WAF instance forwards client requests to the origin server. Weight: Requests are distributed across backend origin servers based on the load balancing algorithm you select and the weight you assign to each server.	Client Protocol: Select HTTP. Server Protocol: HTTP Server Address: IPv4 XXX.XXX.1.1 Server Port: 80
   Proxy Your Website Uses	You need to configure whether you deploy other proxies in front of WAF. Set this parameter based on your website deployment.	Layer-7 proxy

5. Click Next and complete the basic information about the website to be protected. Perform the following operations as prompted on the Add Website page:
   Figure 4 Domain name added to WAF
   
   1. .
   2. .

   After the preceding steps are complete, you can check the Access Status of the added domain name in the domain name list. The Access Status of the domain name is Inaccessible at first. You need to modify the DNS record.

Step 3: Enable General Check and Web Shell Detection (Identifying User-Agent)

General check and web shell detection in WAF can help detect and block threats such as malicious crawlers and web shells.

1. Log in to the WAF console.
2. Click in the upper left corner and select a region or project.
3. In the navigation pane on the left, choose Website Settings.
4. In the Policy column of the row containing the domain name, click the number to go to the Policies page.
5. Ensure that Basic Web Protection is enabled (status: ).
   Figure 5 Basic Web Protection configuration area
   

6. On the Protection Status page, enable General Check and Webshell Detection.
   Figure 6 Protection configuration
   

   After the preceding configurations are complete, you can simulate a malicious crawler attack to check whether the request is blocked. If WAF detects that a malicious crawler is crawling your website, WAF immediately blocks it and logs the event. You can view the crawler protection logs on the Events page.

   

Step 4: Enable Anti-Crawler Protection to Verify Browser Validity

If you enable anti-crawler protection, WAF dynamically analyzes website service models and accurately identifies crawler behavior based on data risk control and bot identification approaches.

1. Click the Anti-Crawler configuration area and toggle it on.
   • : enabled.
   • : disabled.

2. On the Feature Library page, enable Scanner and set other parameters based on your service requirements.
   Figure 7 Feature Library
   

If you enable anti-crawler, web visitors can only access web pages through a browser.

Step 5: Configure CC Attack Protection to Limit Access Frequency

A CC attack protection rule uses a specific IP address, cookie, or referer to limit the access to a specific path (URL), mitigating the impact of CC attacks on web services.

1. Ensure that the Status of CC Attack Protection is enabled ().
   Figure 8 CC Attack Protection configuration area
   

2. In the upper left corner above the CC Attack Protection rule list, click Add Rule. The following uses IP address-based rate limiting and human-machine verification as examples to describe how to add an IP address-based rate limiting rule, as shown in Figure 9.
   Figure 9 Per IP address
   

   If the number of access requests exceeds the configured rate limit, the visitors are required to enter a verification code to continue the access.