Using WAF to Block Crawler Attacks

Application Scenarios

Web crawlers make network information collection and query easy, but they also introduce the following negative impacts:

• Web crawlers always consume too much server bandwidth and increase server load as they use specific policies to browser as much information of high value on a website as possible.
• Bad actors may use web crawlers to launch DoS attacks against websites. As a result, websites may fail to provide normal services due to resource exhaustion.
• Bad actors may use web crawlers to steal mission-critical data on your websites, which will damage your economic interests.

To comprehensively mitigate crawler attacks against websites, WAF provides three anti-crawler policies, general check and web shell detection by identifying User-Agent, website anti-crawler by checking browser validity, and CC attack protection by limiting the access traffic rate.

Overview

Figure 1 shows how WAF detects crawlers, where step 1 and step 2 are called JS challenges and step 3 is called JS authentication.

Figure 1 JavaScript Anti-Crawler protection process

By collecting statistics on the number of JavaScript challenges and authentication responses, the system calculates how many requests the JavaScript anti-crawler defends. In Figure 2, the JavaScript anti-crawler has logged 18 events, 16 of which are JavaScript challenge responses, and 2 of which are JavaScript authentication responses. Other indicates the number of WAF authentication requests fabricated by the crawler.

Figure 2 Parameters of a JavaScript anti-crawler protection rule

Limitations and Constraints

• Cookies must be enabled and JavaScript supported by any browser used to access a website protected by anti-crawler protection rules.
• If your service is connected to CDN, exercise caution when using the JS anti-crawler function.

  CDN caching may impact JS anti-crawler performance and page accessibility.

Resource and Cost Planning

Step 1: Buy the Professional Edition Cloud WAF

The following describes how to buy the standard edition cloud WAF.

1. Log in to Huawei Cloud management console.
2. On the management console page, choose Security > Web Application Firewall.
3. In the upper right corner of the page, click Buy WAF. On the purchase page displayed, select Cloud Mode for WAF Mode.
   • Region: Select the region nearest to your services WAF will protect.
   • Edition: Select Profesional.
   • Expansion Package and Required Duration: Set them based on site requirements.

4. Confirm the product details and click Buy Now in the lower right corner of the page.
5. Check the order details and read the WAF Disclaimer. Then, select the box and click Pay Now.
6. On the payment page, select a payment method and pay for your order.

Step 2: Add Website Information to WAF

The following example shows how to add a website information to WAF in cloud CNAME access mode.

1. In the navigation pane on the left, choose Website Settings.
2. In the upper left corner of the website list, click Add Website.
3. Select Cloud - CNAME and click Configure Now.
4. Configure website information as prompted.
   Figure 3 Configuring basic information
   

   Table 2 Key parameters

   Parameter	Description	Example Value
   Domain Name	Domain name you want to add to WAF for protection. The domain name has an ICP license. You can enter a single domain name (for example, top-level domain name example.com or level-2 domain name www.example.com) or a wildcard domain name (*.example.com).	www.example.com
   Protected Port	The port over which the website traffic goes	Standard ports
   Server Configuration	Web server address settings. You need to configure the client protocol, server protocol, server weights, server address, and server port. Client Protocol: protocol used by a client to access a server. The options are HTTP and HTTPS. Server Protocol: protocol used by WAF to forward client requests. The options are HTTP and HTTPS. Server Address: public IP address (generally corresponding to the A record of the domain name configured on the DNS) or domain name (generally corresponding to the CNAME record of the domain name configured on the DNS) of the web server that a client accesses. Server Port: service port over which the WAF instance forwards client requests to the origin server. Weight: Requests are distributed across backend origin servers based on the load balancing algorithm you select and the weight you assign to each server.	Client Protocol: Select HTTP. Server Protocol: HTTP Server Address: IPv4 XXX.XXX.1.1 Server Port: 80
   Proxy Your Website Uses	You need to configure whether you deploy other proxies in front of WAF. Set this parameter based on your website deployment.	Layer-7 proxy

5. Click Next. Then, , , and as prompted.
   Figure 4 Domain name added to WAF
   

Step 3: Enable General Check and Web Shell Detection (Identifying User-Agent)

General check and web shell detection in WAF can help detect and block threats such as malicious crawlers and web shells.

1. Log in to the management console.
2. Choose Security > Web Application Firewall to go to the Dashboard page.
3. In the navigation pane on the left, choose Website Settings.
4. In the Policy column of the row containing the domain name, click the number to go to the Policies page.
5. Ensure that Basic Web Protection is enabled (status: ).
   Figure 5 Basic Web Protection configuration area
   

6. On the Protection Status page, enable General Check and Webshell Detection.
   Figure 6 Protection configuration
   

   If WAF detects that a malicious crawler is crawling your website, WAF immediately blocks it and logs the event. You can view the crawler protection logs on the Events page.

   

Step 4: Enable Anti-Crawler Protection to Verify Browser Validity

If you enable anti-crawler protection, WAF dynamically analyzes website service models and accurately identifies crawler behavior based on data risk control and bot identification approaches.

1. Click the Anti-Crawler configuration area and toggle it on.
   • : enabled.
   • : disabled.

2. On the Feature Library page, enable Scanner and set other parameters based on your service requirements.
   Figure 7 Feature Library
   

3. Select the JavaScript tab and change Status if needed.

   JavaScript anti-crawler is disabled by default. To enable it, click and then click OK in the displayed dialog box to toggle on .

   

   • Cookies must be enabled and JavaScript supported by any browser used to access a website protected by anti-crawler protection rules.
   • If your service is connected to CDN, exercise caution when using the JS anti-crawler function.

     CDN caching may impact JS anti-crawler performance and page accessibility.

4. Configure a JavaScript-based anti-crawler rule by referring to Table 3.

   Two protective actions are provided: Protect all requests and Protect specified requests.

   • To protect all paths except a specified path

     Set Protection Mode to Protect all paths. Then, click Exclude Path, configure protected paths, and click Confirm.

     Figure 8 Exclude Rule
     
   • To protect a specified path only

     Set Protection Mode to Protect specified requests, click Add Rule, configure the request rule, and click Confirm.

     Figure 9 Add Rule
     

   Table 3 Parameters of a JavaScript-based anti-crawler protection rule

   Parameter	Description	Example Value
   Rule Name	Name of the rule	wafjs
   Path	A part of the URL, not including the domain name A URL is used to define the address of a web page. The basic URL format is as follows: Protocol name://Domain name or IP address[:Port]/[Path/.../File name]. For example, if the URL is http://www.example.com/admin, set Path to /admin. NOTE: The path does not support regular expressions. The path cannot contain two or more consecutive slashes. For example, ///admin. If you enter ///admin, WAF converts /// to /.	/admin
   Logic	Select a logical relationship from the drop-down list.	Include
   Rule Description	A brief description of the rule.	None
   Effective Date	Immediate	Immediate

If you enable anti-crawler, web visitors can only access web pages through a browser.

Step 5: Configure CC Attack Protection to Limit Access Frequency

A CC attack protection rule uses a specific IP address, cookie, or referer to limit the access to a specific path (URL), mitigating the impact of CC attacks on web services.

1. Ensure that the Status of CC Attack Protection is enabled ().
   Figure 10 CC Attack Protection configuration area
   

2. In the upper left corner above the CC Attack Protection rule list, click Add Rule. The following uses IP address-based rate limiting and human-machine verification as examples to describe how to add an IP address-based rate limiting rule, as shown in Figure 11.
   Figure 11 Per IP address
   

   If the number of access requests exceeds the configured rate limit, the visitors are required to enter a verification code to continue the access.