Creating a DSL Processing Task
LTS provides DSL for one-stop log processing. With domain-defined script languages and more than 200 built-in functions, you can perform E2E log processing tasks directly on the LTS console, including normalization, enrichment, transfer, anonymization, and filtering. For details about processing syntax, see DSL Data Processing Syntax (Beta).
For successful DSL processing, your log content must be structured. If you have not configured ICAgent structuring parsing when configuring log ingestion to LTS, you can configure ICAgent or cloud configuring parsing for the target log stream separately. For details, see Configuring ICAgent Structuring Parsing and Setting Cloud Structuring Parsing.
Prerequisites
Logs have been collected.
Creating a DSL Processing Task
- Log in to the LTS console.
- In the navigation pane, choose Log Jobs. On the DSL Processing tab page displayed by default, click Create DSL Processing Task.
- On the Basic Information page, set the following parameters and click Next.
Table 1 Basic information parameters Parameter
Description
Task Name
Name of the DSL processing task. A name can contain 1 to 64 characters, including letters, digits, hyphens (-), underscores (_), and periods (.).
Status
Enabled by default.
Source Log Stream
- Current
Select a log group and a log stream. Logs in the source log group/stream will be processed by DSL and the processed result will be stored in the target log streams.
- Other
Enter an agency name and agency creator account name, and select a log group and log stream. For details about agencies, see Creating an Agency.
Target Log Stream
Click Add, select a log group and a log stream, and enter a log stream alias (used for compiling DSL statements). You can add up to 30 target log streams.
When distributing a source log stream to multiple target log streams using a DSL processing rule, you must use a specific function in your processing statement to specify the target log streams' aliases. If no log stream alias is specified, LTS will output the processing result to only the first target log stream by default.
Processing Scope
Time range for data processing. This parameter is available only to whitelisted users. You can select From a certain time or Specific time period to process historical log data.
- All: Data processing starts from the time when the first log is received from the source log stream and ends when the task is manually stopped.
- From a certain time: Specify a start time for the data processing task. Data processing starts at this time and ends when the task is manually stopped.
- Specific time period: Specify both a start and an end time for the data processing task. The task will automatically stop once the specified end time is reached.
Advanced Settings
Passwords (such as database connection passwords) required for processing statements can be stored in key-value pairs.
Click Add and enter a key and a value.
- Current
- On the processing statement configuration page, enter a statement by referring to the rule statement examples and then check the result preview.
Figure 1 Processing rules
- In the Rule Statement Examples area on the right, click the DSL Statements tab, select the function syntax that matches your service needs, and enter it into the Rule Statement text box. For details about processing syntax, see DSL Data Processing Syntax (Beta).
- In the Raw Logs tab, select a time range, select the log data to be processed, and click Add Test Data.
- Click Execution Preview. After the processing syntax is executed, check the result under Result Preview. To locate or rectify faults, you can use breakpoint debugging to improve development efficiency.
- If the result preview is normal, click OK. A record for the new task will then appear on the DSL processing page.
- Click Export > Export all data to an XLSX file to export all processing tasks to your local PC.
- Click the name of a DSL processing task to go to the details page and view its basic information and execution details.
- Click Modify in the Operation column of a DSL processing task. On the displayed page, modify related parameters by referring to Table 1.
- Click Stop in the Operation column of a DSL processing task to pause the task.
- Click Start in the Operation column of a DSL processing task to start the task.
- Click Delete in the Operation column of a DSL processing task to delete the task.
Deleted tasks cannot be restored.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot