Transferring WAF Logs to OBS or DIS for Long-Term Storage

Application Scenarios

WAF generates a large number of access and attack logs when protecting your services. If you enable LTS logging for WAF, LTS can store logs for up to 365 days. Log data that exceeds the storage period will be automatically deleted. This is suitable for frequent retrieval and real-time troubleshooting of short-term logs.

You can enable log transfer in LTS. Together with OBS or DIS, WAF can enable tiered storage, separating hot and cold data for better management. During routine O&M, O&M personnel can query recent logs in LTS to meet immediate requirements such as troubleshooting and attack source tracing. If the real-time log transfer function is also configured in LTS, all logs can be synchronized to OBS or DIS for long-term storage. These logs can be retrieved from OBS or DIS on demand for compliance audits or historical security event investigations. Beyond simple initial configurations, the entire solution requires no manual intervention. Logs are automatically transferred based on their storage tiers, helping to meet data retention requirements while balancing efficiency and cost control. This solution is well-suited for the long-term, stable operation of medium- to high-traffic services.

Solution Architecture

This solution uses a lightweight architecture that combines Web Application Firewall (WAF), Log Tank Service (LTS), and Object Storage Service (OBS) or Data Ingestion Service (DIS). It streamlines WAF log management, including log collection, short-term storage for O&M, long-term archiving for disaster recovery, and cost optimization, in multi-IAM account scenarios. Figure 1 shows the solution architecture.

Figure 1 Transferring WAF logs to OBS or DIS

• Log source: WAF

  WAF generates structured logs in real time when forwarding traffic, blocking attacks, and controlling access. The logs contain core fields such as the log category, attack type, protective action, request IP address, and response status code returned by the origin server. With LTS logging enabled for WAF, WAF proactively pushes logs to the associated LTS log stream based on rules. This is the data entry of the entire architecture.

• Hot data processing layer: Log Tank Service (LTS)

  LTS works as the main carrier of recent logs. It receives and stores WAF raw logs. It supports structured log parsing, keyword search, field filtering, and anomaly alarms. LTS is well-suited for high-frequency query scenarios, meeting O&M requirements for routine troubleshooting, real-time attack source tracing, and short-term security auditing. It also provides log transfer capabilities to continuously synchronize logs to OBS, ensuring data continuity and preventing data loss.

• Cold data archiving layer: Object Storage Service (OBS) or Data Ingestion Service (DIS)

  It stores all logs dumped from LTS and serves as the long-term disaster recovery backup and archiving carrier. Old logs are migrated from the standard storage class to the infrequent access and then archive storage classes in sequence to separate hot and cold data. This meets the mandatory requirements of DJCP (MLPS) and industry regulations for long-term log retention, and greatly reduces the storage costs of massive historical logs. Archived data is read only when needed for historical event backtracking and compliance spot checks.

  Table 1 Differences between OBS and DIS for the cold data archiving layer

  Item	OBS for Cold Data Archiving Layer	DIS for Cold Data Archiving Layer
  Data type	Formatted log file	Real-time stream data
  Latency	High latency, non-real-time (minute-level)	Low latency (second-level)
  Application scenario	Long-term storage and compliance audit	Real-time analysis, alarm reporting, and monitoring
  Integration with LTS	Bidirectional integration (dumping and archiving, and importing and searching)	Unidirectional integration (dumping and archiving)

Resource and Cost Planning

Prerequisites

• You have connected a website to WAF.
• If you want to transfer logs to OBS, ensure that you have created an OBS bucket. For details, see Creating a Bucket.
• If you want to transfer logs to DIS, ensure that DIS has been enabled. For details, see Creating a DIS Stream. Transferring logs to DIS is supported only in the CN East-Shanghai1, CN North-Beijing4, CN North-Beijing1, CN East-Shanghai2, CN South-Guangzhou, CN-Hong Kong, and AP-Singapore regions. For other regions, you need to submit a service ticket to DIS to enable this function. submit a service ticket.

Step 1: Buy the Standard Edition Cloud WAF

The following describes how to buy the standard edition cloud WAF.

1. Log in to the WAF console.
2. In the upper right corner of the page, click Buy WAF. On the purchase page displayed, select Cloud Mode for WAF Mode.
   • Region: Select the region nearest to your services WAF will protect.
   • Edition: Select Standard.
   • Expansion Package and Required Duration: Set them based on site requirements.

3. Confirm the product details and click Buy Now in the lower right corner of the page.
4. Check the order details and read the WAF Disclaimer. Then, select the box and click Pay Now.
5. On the payment page, select a payment method and pay for your order.

   After the order is paid, click Access Console to go to the Dashboard page. Hover over the Product Details area to view the purchased instance edition and its specifications.

Step 2: Add Website Information to WAF

The following example shows how to add a website to WAF in cloud CNAME access mode.

• For details about the cloud load balancer access mode, see Connecting a Website to WAF (Cloud Mode - ELB Access).
• For details about the dedicated mode, see Connecting a Website to WAF (Dedicated Mode).

1. In the navigation pane on the left, choose Access Management.

   Before adding a website to WAF, you can click Usage Guide in the upper right corner of the page to learn about the access process.

2. On the Access Management page, click the Cloud CNAME tab and click Add Website.

   If only one access mode is enabled, click Add Website.

3. Configure website information as prompted.
   Figure 2 Configuring basic information
   

   Table 3 Key parameters

   Parameter	Description	Example Value
   Domain Name	Domain name you want to add to WAF for protection. The domain name has an ICP license. You can enter a single domain name (for example, top-level domain name example.com or level-2 domain name www.example.com) or a wildcard domain name (*.example.com).	www.example.com
   Protected Port	The port over which the website traffic goes	Standard ports
   Server Configuration	Web server address settings. You need to configure the client protocol, server protocol, server weights, server address, and server port. Client Protocol: protocol used by a client to access a server. The options are HTTP and HTTPS. Server Protocol: protocol used by WAF to forward client requests. The options are HTTP and HTTPS. Server Address: public IP address (generally corresponding to the A record of the domain name configured on the DNS) or domain name (generally corresponding to the CNAME record of the domain name configured on the DNS) of the web server that a client accesses. Server Port: service port over which the WAF instance forwards client requests to the origin server. Weight: Requests are distributed across backend origin servers based on the load balancing algorithm you select and the weight you assign to each server.	Client Protocol: Select HTTP. Server Protocol: HTTP Server Address: IPv4 XXX.XXX.1.1 Server Port: 80
   IP Tag	After a website is connected to WAF, Use Layer-7 Proxy is set to Yes by default. In this scenario, WAF is unable to directly obtain the real IP address of the client. This makes IP address-based protection rules become invalid. You can select an IP tag to specify how WAF identifies client IP addresses. This enables WAF to obtain real client IP addresses. $remote_sockaddr: If there is no other proxy in front of WAF, the packet contains the TCP Option Address (TOA), but you do not want WAF to use the TOA as the client IP address, select this option. With this tag configured, WAF uses the Layer 3 source IP address of the packet as the real client IP address. $remote_addr: If there is a layer-4 proxy in front of WAF, WAF uses the TCP connection IP address as the real client IP address. x-forwarded-for: If there is a layer-7 proxy in front of WAF, WAF uses the first IP address in the X-Forwarded-For (XFF) header as the real client IP address. Custom: If there is a layer-7 proxy in front of WAF, WAF preferentially obtains the real client IP address from the configured field. If multiple fields are configured, WAF reads the real client IP address from left to right. If WAF does not obtain the real client IP address from the custom field, it reads the CDN-Src-Ip, X-Real-Ip, X-Forwarded-For, and $remote_addr fields in sequence to obtain the real client IP address.	x-forwarded-for

4. Click Next and complete the basic information about the website to be protected. Perform the following operations as prompted on the Add Website page:
   Figure 3 Domain name added to WAF
   
   1. Whitelist back-to-source IP addresses.
   2. Test WAF.

   After the preceding steps are complete, you can check the Access Status of the added domain name in the domain name list. The Access Status of the domain name is Inaccessible at first. You need to modify the DNS record.

Step 3: Enable LTS for Protection Logs

1. Log in to the WAF console.
2. Click in the upper left corner and select a region or project.
3. (Optional) If you have enabled the enterprise project function, in the upper part of the navigation pane on the left, select your enterprise project from the Filter by enterprise project drop-down list. Then, WAF will display the related security data in the enterprise project on the page.
4. In the navigation pane on the left, choose Events.
5. Click Connect to LTS on the Log Settings tab if needed.

   Table 4 LTS configuration parameters

   Parameter	Description	Example Value
   Log Type	Select the log types you want to transfer to LTS. You can transfer WAF access logs and WAF attack logs. Attack logs and access logs are in different formats. If you select both log types, you need to configure two different log streams.	WAF access logs and WAF attack logs
   Log Group	Select the log group for log transfer. You can also click Create Log Group to create a log group. A log group is the basic unit for LTS to manage logs. It comprises log streams and categorizes them. A log group does not store any log data. It only helps with log stream management. You can create up to 100 log groups for each account. For more details, see Managing Log Groups.	lts-group-waf
   WAF Access Log Stream	If you select WAF access logs for Log Types, you need to configure a WAF access log stream. You can also click Create Log Stream to create a WAF access log stream. This stream logs key information about each HTTP access, including the access time, client IP address, and requested resource URL.	lts-topic-waf-access
   WAF Attack Log Stream	If you select WAF attack logs for Log Types, you need to configure a WAF attack log stream. You can also click Create Log Stream to create a WAF attack log stream. This stream logs key information about each attack, including the attack type, protective action, and attack source IP address.	lts-topic-waf-attack

   The configuration takes about 10 minutes to take effect. After the configuration takes effect, LTS is billed by traffic. For details about LTS pricing, see LTS Pricing Details.

6. Check or analyze logs.

   After WAF is connected to LTS, log groups (① in Figure 4) and log streams (② in Figure 4) created for attack and access logs will be automatically displayed on the Log Settings tab. You can click WAF access log stream or WAF attack log stream to check, search, or analyze WAF logs. For more details, see Searching and Analyzing Logs.

   Figure 4 Log Settings
   

7. After selecting a log stream, on the Log Search tab (③ in Figure 4), choose > Download Logs (④ in Figure 4) to download the reported logs in the log stream.
   • Frontend download: You can directly save log query results to a local PC. Download records will not appear in your log download history. Each time you can download up to 5,000 log records. You can download logs in .csv or .txt format.
   • Offline backend download: You can download log files to a temporary OBS bucket via a backend task. Your browser must have public network access to download these files from your log download history. Each time you can download up to 20 million log records. You can download logs in .csv, .txt, or .json format.

Step 4 (Method 1): Transferring Logs to OBS

OBS provides mass, secure, and cost-effective data storage for you to store data of any type and size. It is suitable for scenarios such as enterprise backup/archiving, video on demand (VOD), and video surveillance. You can choose scheduled or one-time transfers to OBS. This section describes how to configure one-time log transfer.

1. Log in to the LTS console.
2. Choose Log Transfer in the navigation pane. In the upper right corner of the Log Transfer page, click Configure Log Transfer.
3. On the displayed page, configure the log transfer parameters.
   Figure 5 One-time transfer
   

   Table 5 Transfer parameters

   Parameter	Description	Example
   Transfer Mode	One-time: Logs are transferred to OBS for long-term storage in a one-time manner.	One-time
   Transfer Destination	Select a cloud service to which logs are transferred.	OBS
   Log Group Name	Select a log group.	-
   Log Stream Name	Select a log stream. Log streams already configured with OBS transfer settings cannot be selected again. To avoid transferring empty files, ensure that raw logs have been uploaded to the selected log stream.	-
   Filter By	Keyword is selected by default. Enter the keyword to be filtered in the text box.	-
   Log Time Range	There are three types of time range: relative time from now, relative time from last, and specified time. Select a time range as required. From now: queries log data generated in a time range that ends with the current time, such as the previous 1, 5, or 15 minutes. For example, if the current time is 19:20:31 and 1 hour is selected as the relative time from now, the charts on the dashboard display the log data that is generated from 18:20:31 to 19:20:31. From last: queries log data generated in a time range that ends with the current time, such as the previous 1 or 15 minutes. For example, if the current time is 19:20:31 and 1 hour is selected as the relative time from last, the charts on the dashboard display the log data that is generated from 18:00:00 to 19:00:00. Specified: queries log data that is generated in a specified time range.	-
   Total Log Events	Total number of logs in the selected transfer time range. Click Search next to this parameter. LTS automatically calculates and displays the total number of logs within the selected transfer time range. If there is no log, change the transfer time range.	-
   Log Files	Max. log events for each transfer: 20 million. Max. transfer files: 200.	-
   OBS Bucket	Select an OBS bucket. If no OBS buckets are available, click View OBS Bucket to access OBS Console and create an OBS bucket. Currently, LTS supports only Standard OBS buckets. Data cannot be transferred to an OBS bucket whose storage class is Archive or for which cross-region replication has been configured. If you select an unauthorized OBS bucket, LTS will take 15 minutes to authorize the ACL for the bucket. After you configure a one-off transfer task, if the task fails for the first time, it will be automatically retried 15 minutes later. To prevent log transfer failures, exercise caution when modifying the bucket policy.	-
   Bucket Directory	OBS bucket directory. The value cannot start or end with a slash (/). It can contain only letters, digits, hyphens (-), underscores (_), and periods (.). You can obtain the bucket directory as follows: In the bucket list on OBS Console, click the desired bucket to go to the Objects page. On the Objects page, locate the OBS folder to store LTS logs and click More > Copy Path in the Operation column. Figure 6 shows an example. (The example is for reference only.) If you want to save logs to the test/yicixing folder, test/yicixing will be copied. The copied path is the bucket directory.	test/yicixing
   Transfer File Name	Custom transfer file name. Only letters, digits, hyphens (-), underscores (_), and periods (.) are allowed.	-
   Format	Storage format of logs. The value can be Raw log format, JSON, or CSV. Example of the raw log format: (Logs displayed on the LTS console are in the raw format.) Sep 30 07:30:01 ecs-bd70 CRON[3459]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1) Example of the JSON format: {"host_name":"ecs-bd70","ip":"192.168.0.54","line_no":249,"message":"Sep 30 14:40:01 ecs-bd70 CRON[4363]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)\n","path":"/var/log/syslog","time":1569825602303} CSV: Log content is displayed in a table.	Json

   Figure 6 Copying a path
   

4. Click OK. When the log transfer status changes to Normal, the transfer task has been created.

   After a log transfer task is created, the parameters cannot be modified. Ensure that the parameters are correctly set.

5. Click the OBS bucket name in the Transfer Destination column to switch to OBS Console and view the transferred log files.

   Transferred logs can be downloaded from OBS to your local PC for viewing.

   Figure 7 One-time transfer to OBS
   

   For details about how to modify a log transfer task, view transfer details, delete a transfer task, view transfer status, and configure periodic log transfer, see Transferring Logs to OBS.

Step 4 (Method 2): Transferring Logs to DIS

DIS provides both log storage and big data analysis. The service can perform offline analysis, and transmit a large number of log files to the cloud for backup, query, and machine learning. You can also use it for data recovery and fault analysis after data loss or exceptions. A large number of small text files can be merged into a large file for transfer to improve data processing performance. You can use DIS for log transfer based on your service scenario.

1. Log in to the LTS console. The Log Management page is displayed by default.
2. In the navigation pane on the left, choose Log Transfer.
3. In the upper right corner of the Log Transfer page, click Configure Log Transfer.
4. On the displayed page, configure the log transfer parameters.

   Table 6 Transfer parameters

   Parameter	Description	Example
   Source Account	Current: Logs of the current account will be transferred. Other: Logs of the delegator account will be transferred. Ensure that the delegator has created an agency for log transfer delegation. For details, see Creating an Agency. Record the names of the created agency and the delegator account.	Current
   Agency Name	This parameter is required when Log Source Account is set to Other. Enter the name of the IAM agency created by the delegator.	-
   Delegator Account Name	This parameter is required when Log Source Account is set to Other. Enter the account name of the delegator.	-
   Enable Transfer	Enable log transfer.	Enabled
   Transfer Destination	Select a cloud service to which logs are transferred.	DIS
   Log Group Name	Select a log group.	-
   Log Stream Name	Select a log stream. Log streams already configured with DIS transfer settings cannot be selected again.	-
   DIS Stream Name	Select a DIS stream. If no streams are available, click View DIS Streams to access the DIS console and create a stream.	-
   Format	The storage format of logs. The value can be Raw log format or JSON. The raw log format is as follows: (Logs displayed on the LTS console are in the raw format.) Sep 30 07:30:01 ecs-bd70 CRON[3459]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1) Example of the JSON format: {"host_name":"ecs-bd70","ip":"192.168.0.54","line_no":249,"message":"Sep 30 14:40:01 ecs-bd70 CRON[4363]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)\n","path":"/var/log/syslog","time":1569825602303}	JSON
   Transfer Interval	Logs are transferred to the DIS stream in real time.	Real time
   Filter by Tag Fields	During transfer, logs will be filtered by tag fields collected by ICAgent. Disabled: Logs will not be filtered by tag fields. Enabled: The default tag fields include those for hosts (hostIP, hostId, hostName, pathFile, and collectTime) and for Kubernetes (clusterName, clusterId, nameSpace, podName, containerName, and appName). Optional public tag fields include regionName, logStreamName, logGroupName, and projectId. When Filter by Tag Fields is enabled, Format must be JSON. Transfer Tag: After this function is enabled, log stream tags are also transferred.	Enabled

5. Click OK. When the log transfer status changes to Normal, the transfer task has been created.
6. Click the DIS stream name in the Transfer Destination column to access the DIS console and view transferred log files.

   Transferred logs can be downloaded from DIS to your local PC for viewing.

   

   Deleting a transfer task will stop log transfer, and the deleted task cannot be restored. Exercise caution when performing this operation.

   For details about how to transfer logs to DIS, see Transferring Logs to DIS.

FAQ

Does LTS Delete Logs That Have Been Transferred to OBS Buckets?

What Are the Common Causes of LTS Log Transfer Abnormalities?

What Do I Do If I Cannot View Historical Data in an OBS Bucket After Transferring Data from LTS to OBS?