Configure Bot Mitigation

You can enable the detection function based on your service scenarios. The following describes three detection types: known BOT detection, request feature detection, and BOT behavior detection.

• Known BOT detection: compares visitor requests with the built-in signature database. The matched request is processed based on the configured protection action.

  Table 1 Known BOT detection items

  Detection Item	Description
  Search engine bots	Search engines use web crawlers to aggregate and index online content (such as web pages, images, and other file types). They provide search results in real time.
  Online scanners	Virus/Vulnerability scanners detect viruses or vulnerabilities caused by configuration errors or programming defects in network assets. Typical scanners include Nmap, sqlmap, and WPSec.
  Web crawlers	Popular crawler tools or services on the Internet. They are often used to capture any web page and extract content to meet user requirements. Scrapy, Pyspider, and Prerender are typical ones.
  Website development and monitoring bots	These bots help web developers monitor the performance of their sites. These bots can check the availability of links and domain names, connections and web page loading time for requests from different geographical locations, DNS resolution issues, and other functions.
  Business analysis and marketing bots	SEO helps websites or web pages rank higher in search engine results. SEO companies often use bots to analyze website content, audience, and competitiveness for online advertising and marketing.
  News and social media bots	News and social media platforms allow users to browse hot news, share ideas, and interact with each other online. Many enterprises' marketing strategies include operating pages on these websites and interacting with consumers about products or services. Some companies use robots to collect data from these platforms for insights into media trends and products, enriching network experience.
  Screenshot bots	Some companies use bots to provide website screenshot services. It can take complete long-screen screenshots of online content such as posts on websites and social networks, news, and posts on forums and blogs.
  Academic and research bots	Some universities and companies use bots to collect data from various websites for academic or research purposes, including reference search, semantic analysis, and specific types of search engines.
  RSS feed reader	Some universities and companies use bots to collect data from various websites for academic or research purposes, including reference search, semantic analysis, and specific types of search engines.
  Web archiving bots	Some organizations such as Wikipedia use bots to periodically crawl and archive valuable online information and content copies. These web archiving services are very similar to search engines, but the data provided is not up-to-date. They are mainly used for research.

• Request feature detection: Automatically identifies HTTP header features in visitor requests, checks the validity of HTTP headers, matches mainstream development frameworks and HTTP libraries, spoofs known BOTs, and processes requests based on the configured protection actions when a match is found.

  Table 2 Request feature detection items

  Type	Description
  HTTP request header detection	Abnormal request header
  Development framework and HTTP library	Popular development frameworks and HTTP libraries include Apache HttpComponents, OKHttp, Python requests, and Go HTTP client.
  Other	Spoofing BOT: If this function is enabled, the system checks whether the source IP address of a known bot request is its valid client IP address to prevent spoofing. Automated program: The service can detect automation programs with crawler behavior characteristics but unclear purposes.

• BOT behavior detection: The AI protection engine analyzes and automatically learns visitor requests and provides behavior detection scores. You can set protection actions corresponding to behavior detection scores.

  Table 3 BOT behavior detection scenarios

  No.	Behavior Detection Model	Scenario
  1	Random UA detection model	Bot with the random UA function enabled
  2	Repeated URL detection model	Data theft
  3	Abnormal Referer model	The Referer field is not correctly set.
  4	Abnormal short session quantity model	Low-frequency crawler
  5	Abnormal short session duration model	High-frequency crawler
  6	Repeated URL in long session active requests	The model is more accurate than the preceding duplicate URL detection model for data theft detection.
  7	Long session request interval CCE model	Crawler with regular request time