Overview

Background

Secondary processing is often required for logs collected into LTS. Existing function-based processing methods present several disadvantages:

1. Limited flexibility: A single function can only convert one log structure for one log stream, requiring a new function for each new log structure.
2. Increased costs: Function-based processing relies on FunctionGraph, incurring separate charges that raise overall costs.

Scenarios

• Extracting structured data for subsequent retrieval, analysis, and dashboard display.
• Downsizing logs to save costs. Discarding unnecessary log data to save storage and traffic costs.
• Anonymizing sensitive data such as user IDs, mobile numbers, and other personal data.
• Delivering logs by category (for example, ERROR, WARNING, INFO levels) to different log topics.

Solution

You can create a DSL processing task to process log data in a source log stream and output the processed data to a target log stream. The procedure is as follows:

1. Data is read from the source log stream via a consumer group.
2. Each piece of read data is processed according to defined DSL rules.
3. The processed data is then written to the target log stream. After data processing is complete, you can check the processed data in the target log stream.

Functions

LTS provides the following DSL capabilities to process data:

• Data normalization: extracts fields and converts logs from chaotic formats into structured data, preparing it for subsequent stream processing and data warehouse computing.
• Data enrichment: joins log fields (such as order logs) and dimension table fields (such as user information tables) to add more dimensions for deeper data analysis.
• Data anonymization: anonymizes sensitive information such as passwords, mobile numbers, and addresses in data.
• Data filtering: filters logs of key services for focused analysis.