Setting ICAgent Structuring Parsing Rules

Advantages

• Collecting log data in non-intrusive mode as log files: You do not need to modify application code, and log collection does not affect application running.
• Handling various exceptions during log collection: Security measures such as proactive retry and local cache are taken when a network or server exception occurs.
• Centralized management: After installing ICAgent, you only need to set configurations such as host groups and ICAgent collection on the LTS console.
• Comprehensive self-protection mechanisms: ICAgent is designed with strict limitations and protective measures regarding CPU, memory, and network usage, to minimize its impact on the performance of other services sharing the same server.

Single-Line - Full-Text Log

If you want to display each line of log data as a single log on the LTS page, select Single-Line.

1. Select Single-Line - Full-Text Log.
2. Enable Log Filtering (disabled by default) as required and add a maximum of 20 whitelist or blacklist rules.
   Figure 1 Log filtering rules
   
   • Add a whitelist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a matching criterion, collecting and reporting only logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). In single-line and multi-line full-text modes, content is used as the key name {key} of the full text by default. The relationship between multiple filtering rules is OR. For example, to collect log events containing hello from the log source file, set the filtering rule to .*hello.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
        Figure 2 Verification
        
   • Add a blacklist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a discarding criterion, discarding logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). In single-line and multi-line full-text modes, content is used as the key name {key} of the full text by default. The relationship between multiple filtering rules is OR. For example, if you do not want to collect log events containing test from the log source file, set the filtering rule to .*test.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.

Multi-Line - Full-Text Log

Multiple lines of exception log events can be displayed as a single log event. This is helpful when you check logs to locate problems.

1. Select Multi-Line - Full-Text Log.
2. Select a log example from existing logs or paste it from the clipboard.
   • Click Select from Existing Logs, select a log event, and click OK. You can select different time ranges to filter logs.
   • Click Paste from Clipboard to paste the copied log content to the Log Example box.

   

3. A regular expression can be automatically generated or manually entered under Regular Expression of the First Line. The regular expression of the first line must match the entire first line, not just the beginning of the first line.
   Figure 3 Regular expression of the first line
   
4. Enable Log Filtering (disabled by default) as required and add a maximum of 20 whitelist or blacklist rules.
   Figure 4 Log filtering rules
   
   • Add a whitelist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a matching criterion, collecting and reporting only logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). In single-line and multi-line full-text modes, content is used as the key name {key} of the full text by default. The relationship between multiple filtering rules is OR. For example, to collect log events containing hello from the log source file, set the filtering rule to .*hello.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
        Figure 5 Verification
        
   • Add a blacklist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a discarding criterion, discarding logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). In single-line and multi-line full-text modes, content is used as the key name {key} of the full text by default. The relationship between multiple filtering rules is OR. For example, if you do not want to collect log events containing test from the log source file, set the filtering rule to .*test.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.

JSON

This option is applicable to JSON logs and splits them into key-value pairs.

1. Choose JSON.
2. Enable Log Filtering (disabled by default) as required and add a maximum of 20 whitelist or blacklist rules.
   Figure 6 Log filtering rules
   
   • Add a whitelist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a matching criterion, collecting and reporting only logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). A key is a log field name. The relationship between multiple filtering rules is OR. For example, to collect log events containing hello from the log source file, set the filtering rule to .*hello.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
        Figure 7 Verification
        
   • Add a blacklist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a discarding criterion, discarding logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). A key is a log field name. The relationship between multiple filtering rules is OR. For example, if you do not want to collect log events containing test from the log source file, set the filtering rule to .*test.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
3. Raw Log Upload:

   After this function is enabled, raw logs are uploaded to LTS as the value of the content field.

4. Upload Parsing Failure Log:

   After this function is enabled, raw logs are uploaded to LTS as the value of the _content_parse_fail_ field.

5. Custom Time:

   Enabling this lets you specify a field as the log time. Otherwise, the time set during ingestion configuration is used.

6. JSON Parsing Layers: Add 1 to 4 JSON parsing layers. The value must be an integer and is 1 by default.

   This function expands the fields of a JSON log. For example, for raw log {"key1":{"key2":"value"}}, if you choose to parse it into 1 layer, the log will become {"key1":{"key2":"value"}}; if you choose to parse it into 2 layers, the log will become {"key1.key2":"value"}.

Delimiter

Logs can be parsed by delimiters, such as commas (,), spaces, or other special characters.

1. Select a delimiter.
2. Select or customize a delimiter.
3. Select a log example from existing logs or paste it from the clipboard, click Verify, and view the results under Extraction Results.
   • Click Select from Existing Logs, select a log event, and click OK. You can select different time ranges to filter logs.
   • Click Paste from Clipboard to paste the copied log content to the Log Example box.
   Figure 8 Delimiter
   
4. Enable Log Filtering (disabled by default) as required and add a maximum of 20 whitelist or blacklist rules.
   Figure 9 Log filtering rules
   
   • Add a whitelist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a matching criterion, collecting and reporting only logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). A key is a log field name. The relationship between multiple filtering rules is OR. For example, to collect log events containing hello from the log source file, set the filtering rule to .*hello.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
        Figure 10 Verification
        
   • Add a blacklist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a discarding criterion, discarding logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). A key is a log field name. The relationship between multiple filtering rules is OR. For example, if you do not want to collect log events containing test from the log source file, set the filtering rule to .*test.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
5. Raw Log Upload:

   After this function is enabled, raw logs are uploaded to LTS as the value of the content field.

6. Upload Parsing Failure Log:

   After this function is enabled, raw logs are uploaded to LTS as the value of the _content_parse_fail_ field.

7. Custom Time:

   Enabling this lets you specify a field as the log time. Otherwise, the time set during ingestion configuration is used.

Single-Line - Completely Regular

This option is applicable to single-line logs in any format and uses a regular expression to extract fields.

1. Select Single-Line - Completely Regular.
2. Select a log example from existing logs or paste it from the clipboard.
   • Click Select from Existing Logs, select a log event, and click OK. You can select different time ranges to filter logs.
   • Click Paste from Clipboard to paste the copied log content to the Log Example box.
3. Enter a regular expression for extracting the log under Extraction Regular Expression, click Verify, and view the results under Extraction Results.

   Alternatively, click automatic generation of regular expressions. In the displayed dialog box, extract fields based on the log example, enter the key, and click OK to automatically generate a regular expression. Then, click OK.

   Figure 11 Extracting a regular expression
   
4. Enable Log Filtering (disabled by default) as required and add a maximum of 20 whitelist or blacklist rules.
   Figure 12 Log filtering rules
   
   • Add a whitelist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a matching criterion, collecting and reporting only logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). A key is a log field name. The relationship between multiple filtering rules is OR. For example, to collect log events containing hello from the log source file, set the filtering rule to .*hello.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
        Figure 13 Verification
        
   • Add a blacklist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a discarding criterion, discarding logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). A key is a log field name. The relationship between multiple filtering rules is OR. For example, if you do not want to collect log events containing test from the log source file, set the filtering rule to .*test.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
5. Raw Log Upload:

   After this function is enabled, raw logs are uploaded to LTS as the value of the content field.

6. Upload Parsing Failure Log:

   After this function is enabled, raw logs are uploaded to LTS as the value of the _content_parse_fail_ field.

7. Custom Time:

   Enabling this lets you specify a field as the log time. Otherwise, the time set during ingestion configuration is used.

Multi-Line - Completely Regular

This option is applicable to multi-line logs in any format and uses a regular expression to extract fields.

1. Select Multi-Line - Completely Regular.
2. Select a log example from existing logs or paste it from the clipboard.
   • Click Select from Existing Logs, select a log event, and click OK. You can select different time ranges to filter logs.
   • Click Paste from Clipboard to paste the copied log content to the Log Example box.
3. A regular expression can be automatically generated or manually entered under Regular Expression of the First Line. The regular expression of the first line must match the entire first line, not just the beginning of the first line.
   

   The regular expression of the first line is used to identify the beginning of a multi-line log. Example:

   2024-10-11 10:59:07.000 a.log:1 level:warn
     no.1 log
   2024-10-11 10:59:17.000 a.log:2 level:warn
     no.2 log

   Complete lines:

   2024-10-11 10:59:07.000 a.log:1 level:warn
   no.1 log

   First line:

   2024-10-11 10:59:07.000 a.log:1 level:warn

   Example of the regular expression of the first line: ^(\d+-\d+-\d+\s+\d+:\d+:\d+\.\d+). The date in each first line is unique. Therefore, the regular expression in the first line can be generated based on the date.

4. Enter a regular expression for extracting the log under Extraction Regular Expression, click Verify, and view the results under Extraction Results.

   Alternatively, click automatic generation of regular expressions. In the displayed dialog box, extract fields based on the log example, enter the key, and click OK to automatically generate a regular expression. Then, click OK.

   

   The extraction result is the execution result of the extraction regular expression instead of the first line regular expression. To check the execution result of the first line regular expression, go to the target log stream.

   If you enter an incorrect regular expression for Regular Expression of the First Line, you cannot view the reported log stream data.

   Figure 14 Setting a regular expression
   
5. Enable Log Filtering (disabled by default) as required and add a maximum of 20 whitelist or blacklist rules.
   Figure 15 Log filtering rules
   
   • Add a whitelist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a matching criterion, collecting and reporting only logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). A key is a log field name. The relationship between multiple filtering rules is OR. For example, to collect log events containing hello from the log source file, set the filtering rule to .*hello.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
        Figure 16 Verification
        
   • Add a blacklist rule (available only when Log Filtering is enabled).

     You can add filtering rules to retain valuable log data by applying regular expressions to the values of specified keys. A filtering rule acts as a discarding criterion, discarding logs that match the specified regular expression.

     1. Click Add and enter a key and filtering rule (regular expression). A key is a log field name. The relationship between multiple filtering rules is OR. For example, if you do not want to collect log events containing test from the log source file, set the filtering rule to .*test.*.
     2. Click in the Operation column, enter a field value, and click Verify to verify the rule.
6. Raw Log Upload:

   After this function is enabled, raw logs are uploaded to LTS as the value of the content field.

7. Upload Parsing Failure Log:

   After this function is enabled, raw logs are uploaded to LTS as the value of the _content_parse_fail_ field.

8. Custom Time:

   Enabling this lets you specify a field as the log time. Otherwise, the time set during ingestion configuration is used.

Combined Parsing

This option is applicable to logs in multiple nested formats, for example, JSON logs with delimiters. You can customize parsing rules based on the syntax.

1. Select Combined Parsing.
2. Select a log example from existing logs or paste it from the clipboard and enter the configuration content under Plug-in Settings.
3. Customize the settings based on the log content by referring to the following plug-in syntaxes.
   • processor_regex

     Table 1 Regular expression extraction

     Parameter	Type	Description
     source_key	string	Original field name.
     regex	string	() in a regular expression indicates the field to be extracted.
     keys	string	Field name for the extracted content.
     keep_source	boolean	Whether to retain the original field.
     keep_source_if_parse	boolean	Whether to retain the original field when a parsing error occurs.

   • processor_split_string

     Table 2 Parsing using delimiters

     Parameter	Type	Description
     source_key	string	Original field name.
     split_sep	string	Delimiter string.
     keys	string	Field name for the extracted content.
     keep_source	boolean	Whether to retain the original field in the parsed log.
     split_type	char/special_char/string	Delimiter type. The options are char (single character), special_char (invisible character), and string.
     keep_source_if_parse_error	boolean	Whether to retain the original field when a parsing error occurs.

   • processor_split_key_value

     Table 3 Key-value pair segmentation

     Parameter	Type	Description
     source_key	string	Original field name.
     split_sep	string	Delimiter between key-value pairs. The default value is the tab character (\t).
     expand_connector	string	Delimiter between the key and value in a key-value pair. The default value is a colon (:).
     keep_source	boolean	Whether to retain the original field in the parsed log.

   • processor_add_fields

     Table 4 Adding a field

     Parameter	Type	Description
     fields	json/object	Name and value of the field to be added. The field is in key-value pair format. Multiple key-value pairs can be added.

   • processor_drop

     Table 5 Discarded fields

     Parameter	Type	Description
     drop_keys	string	List of discarded fields.

   • processor_rename

     Table 6 Renaming a field

     Parameter	Type	Description
     source_keys	string	Original name.
     destkeys	string	New name.

   • processor_json

     Table 7 JSON expansion and extraction

     Parameter	Type	Description
     source_key	string	Original field name.
     keep_source	string	Whether to retain the original field in the parsed log.
     expand_depth	int	JSON expansion depth. The default value 0 indicates that the depth is not limited. Other numbers, such as 1, indicate the current level.
     expand_connector	string	Connector for expanding JSON. The default value is an underscore (_).
     prefix	string	Prefix added to a field name when JSON is expanded.
     keep_source_if_parse_error	boolean	Whether to retain the original field when a parsing error occurs.

   • processor_filter_regex

     Table 8 Filters

     Parameter	Type	Description
     include	json/object	The key indicates the log field, and the value indicates the regular expression to be matched.
     exclude	json/object	The key indicates the log field, and the value indicates the regular expression to be matched.

   • processor_gotime

     Table 9 Extraction time

     Parameter	Type	Description
     source_key	string	Original field name.
     source_format	string	Original time format.
     source_location	int	Original time zone. If the value is empty, it indicates the time zone of the host or container where the logtail is located.
     dest_key	string	Target field after parsing.
     dest_format	string	Time format after parsing.
     dest_location	int	Time zone after parsing. If this parameter is left blank, the time zone of the local host is used.
     set_time	boolean	Whether to set the parsed time as the log time.
     keep_source	boolean	Whether to retain the original field in the parsed log.

4. Example:

   [
           {
               "type": "processor_regex",
               "detail": {
                   "source_key": "content",
                   "regex": "*",
                   "keys": [
                       "key1",
                       "key2"
                   ],
                   "multi_line_regex": "*",
                   "keep_source": true,
                   "keep_source_if_parse_error": true
               }
           },
           {
               "type": "processor_split_string",
               "detail": {
                   "split_sep": ".",
                   "split_type": ".",
                   "split_keys": [
                       "key1",
                       "key2"
                   ],
                   "source_key": "context",
                   "keep_source": true,
                   "keep_source_if_parse_error": true
               }
           },
           {
               "type": "processor_add_fields",
               "detail": {
                   "fields": [
                       {
                           "key1": "value1"
                       },
                       {
                           "key2": "value2"
                       }
                   ]
               }
           },
           {
               "type": "processor_drop",
               "detail": {
                   "drop_keys": [
                       "key1",
                       "key2"
                   ]
               }
           },
           {
               "type": "processor_rename",
               "detail": {
                   "source_key": [
                       "skey1",
                       "skey2"
                   ],
                   "dest_keys": [
                       "dkey1",
                       "dkey2"
                   ]
               }
           },
           {
               "type": "processor_json",
               "detail": {
                   "source_key": "context",
                   "expand_depth": 4,
                   "expand_connector": "_",
                   "prefix": "prefix",
                   "keep_source": true,
                   "keep_source_if_parse_error": true
               }
           },
           {
               "type": "processor_gotime",
               "detail": {
                   "source_key": "skey",
                   "source_format": "ydm",
                   "source_location": 8,
                   "dest_key": "dkey",
                   "dest_format": "ydm",
                   "dest_location": 8,
                   "set_time": true,
                   "keep_source": true,
                   "keep_source_if_parse_error": true
               }
           },
           {
               "type": "processor_filter_regex",
               "detail": {
                   "include": {
                       "ikey1": "*",
                       "ikey2": "*"
                   },
                   "exclude": {
                       "ekey1": "*",
                       "ekey1": "*"
                   }
               }
           }
       ]

Custom Time

Enable Custom Time and set parameters by referring to Table 10.

• If the time format is incorrect or the specified field does not exist, the log time is the time set during ingestion configuration.
• The time field needs to be verified again when operations such as field name modification, field deletion, and field type modification are performed on structuring parsing.

Common Log Time Formats

Table 11 lists common log time formats.

By default, log timestamps in LTS are accurate to seconds. You do not need to configure information such as milliseconds and microseconds.