Using LTS to Query and Analyze WAF Access Logs

Application Scenarios

After you authorize WAF to access Log Tank Service (LTS), you can use the WAF logs recorded by LTS for quick and efficient real-time analysis, device O&M management, and analysis of service trends.

This practice uses the access log stream lts-waf-access of log group lts-waf as an example to describe how to use LTS to quickly query and analyze logs.

Resource and Cost Planning

Step 1: Buy the Standard Edition Cloud WAF

The following describes how to buy the standard edition cloud WAF.

1. Log in to the WAF console.
2. In the upper right corner of the page, click Buy WAF. On the purchase page displayed, select Cloud Mode for WAF Mode.
   • Region: Select the region nearest to your services WAF will protect.
   • Edition: Select Standard.
   • Expansion Package and Required Duration: Set them based on site requirements.

3. Confirm the product details and click Buy Now in the lower right corner of the page.
4. Check the order details and read the WAF Disclaimer. Then, select the box and click Pay Now.
5. On the payment page, select a payment method and pay for your order.

   After the order is paid, click Access Console to go to the Dashboard page. Hover over the Product Details area to view the purchased instance edition and its specifications.

Step 2: Add Website Information to WAF

The following example shows how to add a website to WAF in cloud CNAME access mode.

• For details about the cloud load balancer access mode, see Connecting a Website to WAF (Cloud Mode - ELB Access).
• For details about the dedicated mode, see Connecting a Website to WAF (Dedicated Mode).

1. In the navigation pane on the left, choose Access Management.

   Before adding a website to WAF, you can click Usage Guide in the upper right corner of the page to learn about the access process.

2. On the Access Management page, click the Cloud CNAME tab and click Add Website.

   If only one access mode is enabled, click Add Website.

3. Configure website information as prompted.
   Figure 1 Configuring basic information
   

   Table 2 Key parameters

   Parameter	Description	Example Value
   Domain Name	Domain name you want to add to WAF for protection. The domain name has an ICP license. You can enter a single domain name (for example, top-level domain name example.com or level-2 domain name www.example.com) or a wildcard domain name (*.example.com).	www.example.com
   Protected Port	The port over which the website traffic goes	Standard ports
   Server Configuration	Web server address settings. You need to configure the client protocol, server protocol, server weights, server address, and server port. Client Protocol: protocol used by a client to access a server. The options are HTTP and HTTPS. Server Protocol: protocol used by WAF to forward client requests. The options are HTTP and HTTPS. Server Address: public IP address (generally corresponding to the A record of the domain name configured on the DNS) or domain name (generally corresponding to the CNAME record of the domain name configured on the DNS) of the web server that a client accesses. Server Port: service port over which the WAF instance forwards client requests to the origin server. Weight: Requests are distributed across backend origin servers based on the load balancing algorithm you select and the weight you assign to each server.	Client Protocol: Select HTTP. Server Protocol: HTTP Server Address: IPv4 XXX.XXX.1.1 Server Port: 80
   IP Tag	After a website is connected to WAF, Use Layer-7 Proxy is set to Yes by default. In this scenario, WAF is unable to directly obtain the real IP address of the client. This makes IP address-based protection rules become invalid. You can select an IP tag to specify how WAF identifies client IP addresses. This enables WAF to obtain real client IP addresses. $remote_sockaddr: If there is no other proxy in front of WAF, the packet contains the TCP Option Address (TOA), but you do not want WAF to use the TOA as the client IP address, select this option. With this tag configured, WAF uses the Layer 3 source IP address of the packet as the real client IP address. $remote_addr: If there is a layer-4 proxy in front of WAF, WAF uses the TCP connection IP address as the real client IP address. x-forwarded-for: If there is a layer-7 proxy in front of WAF, WAF uses the first IP address in the X-Forwarded-For (XFF) header as the real client IP address. Custom: If there is a layer-7 proxy in front of WAF, WAF preferentially obtains the real client IP address from the configured field. If multiple fields are configured, WAF reads the real client IP address from left to right. If WAF does not obtain the real client IP address from the custom field, it reads the CDN-Src-Ip, X-Real-Ip, X-Forwarded-For, and $remote_addr fields in sequence to obtain the real client IP address.	x-forwarded-for

4. Click Next and complete the basic information about the website to be protected. Perform the following operations as prompted on the Add Website page:
   Figure 2 Domain name added to WAF
   
   1. Whitelist back-to-source IP addresses.
   2. Test WAF.

   After the preceding steps are complete, you can check the Access Status of the added domain name in the domain name list. The Access Status of the domain name is Inaccessible at first. You need to modify the DNS record.

Step 3: Enable LTS for Protection Logs

1. Log in to the WAF console.
2. Click in the upper left corner and select a region or project.
3. (Optional) If you have enabled the enterprise project function, in the upper part of the navigation pane on the left, select your enterprise project from the Filter by enterprise project drop-down list. Then, WAF will display the related security data in the enterprise project on the page.
4. In the navigation pane on the left, choose Security Operations > Events.
5. Click Connect to LTS on the Log Settings tab if needed.

   Table 3 LTS configuration parameters

   Parameter	Description	Example Value
   Log Types	Select the log types you want to transfer to LTS. You can transfer WAF access logs and WAF attack logs. Attack logs and access logs are in different formats. If you select both log types, you need to configure two different log streams.	WAF access logs and WAF attack logs
   Log Group	Select the log group for log transfer. You can also click Create Log Group to create a log group. A log group is the basic unit for LTS to manage logs. It comprises log streams and categorizes them. A log group does not store any log data. It only helps with log stream management. You can create up to 100 log groups for each account. For more details, see Managing Log Groups.	lts-group-waf
   WAF Access Log Stream	If you select WAF access logs for Log Types, you need to configure a WAF access log stream. You can also click Create Log Stream to create a WAF access log stream. This stream logs key information about each HTTP access, including the access time, client IP address, and requested resource URL.	lts-topic-waf-access
   WAF Attack Log Stream	If you select WAF attack logs for Log Types, you need to configure a WAF attack log stream. You can also click Create Log Stream to create a WAF attack log stream. This stream logs key information about each attack, including the attack type, protective action, and attack source IP address.	lts-topic-waf-attack

   The configuration takes about 10 minutes to take effect. After the configuration takes effect, LTS is billed by traffic. For details about LTS pricing, see LTS Pricing Details.

6. Check or analyze logs.

   After WAF is connected to LTS, log groups (① in Figure 3) and log streams (② in Figure 3) created for attack and access logs will be automatically displayed on the Log Settings tab. You can click WAF access log stream or WAF attack log stream to check, search, or analyze WAF logs. For more details, see Searching and Analyzing Logs.

   Figure 3 Log Settings
   

7. After selecting a log stream, on the Log Search tab (③ in Figure 3), choose > Download Logs (④ in Figure 3) to download the reported logs in the log stream.
   • Frontend download: You can directly save log query results to a local PC. Download records will not appear in your log download history. Each time you can download up to 5,000 log records. You can download logs in .csv or .txt format.
   • Offline backend download: You can download log files to a temporary OBS bucket via a backend task. Your browser must have public network access to download these files from your log download history. Each time you can download up to 20 million log records. You can download logs in .csv, .txt, or .json format.

   You can also download log files through an OBS transfer task. For details, see Transferring Logs to OBS.

Step 4: Query and Analyze WAF Access Logs in LTS

1. Log in to the LTS console.
2. In the Log Group Name column, click the name of the target log group (for example, lts-waf) to go the log stream page.
3. In the Log Stream Name column, click the name of the log stream used for WAF access logs (for example, lts-waf-access). Figure 4 shows an example. Then, select the Log Stream tab.
   Figure 4 Accessing the log stream page
   

4. On the log stream details page, click in the upper right corner. On the page displayed, click the Cloud Structured Parsing tab.
5. Select JSON as the log structure, as shown in Figure 5.
   Figure 5 JSON
   

6. In the Step 1 Select a sample log event. area, click Select from existing log events. In the displayed Select Log Event dialog box, select a log and click OK.
   Figure 6 Select Log Event
   

7. In the Step 2 Extract fields area, click Intelligent Extraction and enable quick analysis for the log field you want to analyze (for example, remote_ip). Figure 7 shows an example.

   remote_ip: IP address of a client from which the request originates.

   Figure 7 Selecting log fields for quick analysis
   

8. Click Save. Then, LTS will start a quick analysis and do statistics for logs collected in a certain period. Figure 8 shows an example.
   Figure 8 Quick analysis of access logs
   

9. In the navigation pane, choose Visualization. On the right pane, select a log query time range, enter an SQL statement in the search box, and click Query to query the specified log.

   You can enter either of the following SQL statements in the search box to query logs of a specified IP address:

   select * where remote_ip = 'xx.xx.xx.xx' or select * where remote_ip like 'xx.xx.xx%'