How Does JavaScript Anti-Crawler Detection Work?
Figure 1 shows how JavaScript anti-crawler detection works, which includes JavaScript challenges (step 1 and step 2) and JavaScript authentication (step 3).
After JavaScript anti-crawler is enabled, WAF returns a piece of JavaScript code to the client when the client sends a request.
- If the client sends a normal request to the website, triggered by the received JavaScript code, the client will automatically send the request to WAF again. WAF then forwards the request to the origin server. This process is called JavaScript verification.
- If the client is a crawler, it cannot be triggered by the received JavaScript code and will not send a request to WAF again. The client fails JavaScript authentication.
- If a client crawler fabricates a WAF authentication request and sends the request to WAF, the WAF will block the request. The client fails JavaScript authentication.
By collecting statistics on the number of JavaScript challenge and authentication responses, the system calculates how many requests the JavaScript anti-crawler defends. As shown in Figure 2, the JavaScript anti-crawler logs 18 events, 16 of which are JavaScript challenge responses, 2 of which are JavaScript authentication responses. The number of Other is the WAF authentication requests fabricated by the crawler.
![](https://support.huaweicloud.com/intl/en-us/waf_faq/public_sys-resources/notice_3.0-en-us.png)
The protective action for website anti-crawler JavaScript challenge is Log only, and that for JavaScript authentication is Verification code. If a visitor fails the JavaScript authentication, a verification code is required for access. Requests will be forwarded as long as the visitor enters a valid verification code.
Protection Rules FAQs
- How Do I Switch the Mode of Basic Web Protection from Log Only to Block?
- Which Protection Levels Can Be Set for Basic Web Protection?
- What Is the Peak Rate of CC Attack Protection?
- When Is Cookie Used to Identify Users?
- What Are the Differences Between Rate Limit and Allowable Frequency in a CC Rule?
- Why Cannot the Verification Code Be Refreshed When Verification Code Is Configured in a CC Attack Protection Rule?
- Can a Precise Protection Rule Take Effect in a Specified Period?
- Can a Path Containing # Be Matched in a Precise Protection Rule?
- How Can I Allow Access from .js Files?
- Can I Batch Add IP Addresses to a Blacklist or Whitelist Rule?
- Can I Import or Export a Blacklist or Whitelist into or from WAF?
- How Do I Block Abnormal IP Addresses?
- Why Does a Requested Page Fail to Respond to the Client After the JavaScript-based Anti-Crawler Is Enabled?
- Is There Any Impact on Website Loading Speed If Other Crawler Check in Anti-Crawler Is Enabled?
- How Does JavaScript Anti-Crawler Detection Work?
- In Which Situations Will the WAF Policies Fail?
- How Do I Allow Requests from Only IP Addresses in a Specified Geographical Region?
- Which of the WAF Protection Rules Support the Log-Only Protective Action?
- How Do I Allow Only Specified IP Addresses to Access Protected Websites?
- Which Protection Rules Are Included in the System-Generated Policy?
- Why Does the Page Fail to Be Refreshed After WTP Is Enabled?
- What Are the Differences Between Blacklist/Whitelist Rules and Precise Protection Rules on Blocking Access Requests from Specified IP Addresses?
- What Do I Do If a Scanner, such as AppScan, Detects that the Cookie Is Missing Secure or HttpOnly?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore