Parser Rules
The tenant-side data collection uses custom Logstash collectors for data transmission. Parsers mainly work as codeless filters in Logstash. Currently, the following types of Logstash filter plugins are supported.
Parser | Plugin in Logstash | Description |
|---|---|---|
Key-Value filter | kv | Parses key-value pairs. For details about parsing rules, see Table 2. |
Mutate filter | mutate | Performs general mutations on fields. For details about parsing rules, see Table 3. |
Grok filter | grok | Parses regular expressions. For details about parsing rules, see Table 4. |
Date filter | date | Parses the date. For details about parsing rules, see Table 5. |
Drop filter | drop | Deletes packets. There is no specific rule. If you use this parser, logs received will be deleted. |
Prune filter | prune | Parses blacklists and whitelists. For details about parsing rules, see Table 6. |
CSV filter | csv | Parses the CSV data. For details about parsing rules, see Table 7. |
Function filter | ruby | Executes ruby code. For details about parsing rules, see Table 8. |
JSON filter | json | Converts the JSON data. For details about parsing rules, see Table 9. |
Split filter | split | Splits data. For details about parsing rules, see Table 10. |
Clone filter | clone | Duplicates data. For details about parsing rules, see Table 11. |
UUID filter | uuid | Parses UUIDs. For details about parsing rules, see Table 12. |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
Source | source | string | source | Yes | Defines the fields to be translated. |
Target | target | string | message | No | Defines the target fields. |
Field_split | field_split | string | , | No | Splits fields. |
Value_split | value_split | string | = | No | Splits fields. |
Trim_key | trim_key | string | -- | No | Removes spaces from the key. |
Trim_value | trim_value | string | -- | No | Removes spaces from the value. |
Allow_duplicate_values | allow_duplicate_values | boolean | true | No | Allows duplicate values. |
Default_keys | default_keys | array | -- | No | Adds keys. |
Exclude_keys | exclude_keys | array | -- | No | Excludes certain keys. |
Include_keys | include_keys | array | -- | No | Includes certain keys. |
Prefix | prefix | string | -- | No | Performs prefix matches. |
Recursive | recursive | boolean | true | No | Performs Recursive parsing. |
Transform_key | transform_key | string | -- | No | Transforms keys. |
Add_field | add_field | hash | -- | No | Adds fields. |
add_tag | add_tag | array | -- | No | Adds tags. |
Remove_field | remove_field | array | -- | No | Removes fields. |
Remove_tag | remove_tag | array | -- | No | Removes tags. |
Id | id | string | -- | No | ID. |
Whitespace | whitespace | string | strict/lenient | No | Allows whitespace characters. |
Remove_char_key | remove_char_key | string | <>[](), | No | Removes characters from the key. |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
Convert | convert | hash | -- | No | Converts a field's value into a different type. |
Join | join | hash | -- | No | Joins arrays. |
Lowercase | lowercase | array | -- | No | Converts characters into its lowercase equivalent. |
Coerce | coerce | hash | -- | No | Sets the default value of a field. |
Rename | rename | hash | -- | No | Renames fields. |
Replace | replace | hash | -- | No | Replaces the value of a field with a new value. |
Split | split | hash | -- | No | Split a field to an array. |
Strip | strip | array | -- | No | Strips spaces from fields. |
Update | update | hash | -- | No | Updates fields. |
Uppercase | uppercase | array | -- | No | Converts characters into their uppercase equivalent. |
Add_field | add_field | hash | -- | No | Adds fields. |
Add_tag | add_tag | array | -- | No | Adds tags. |
Remove_field | remove_field | array | -- | No | Removes fields. |
Remove_tag | remove_tag | array | -- | No | Removes tags. |
ID | id | string | -- | No | Id |
Copy | copy | hash | -- | No | Copies fields. |
Gsub | gsub | array | -- | No | Replaces the gsub value. |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
match | match | hash | -- | Yes | Performs regex matches. |
Break_on_match | break_on_match | boolean | true | No | Breaks on the first match. |
Overwrite | overwrite | array | message | No | Overwrites fields. |
Add_field | add_field | hash | -- | No | Adds fields. |
Add_tag | add_tag | array | -- | No | Adds tags. |
Remove_field | remove_field | array | -- | No | Removes fields. |
Remove_tag | remove_tag | array | -- | No | Removes tags. |
Id | id | string | -- | No | Id |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
Match | match | array | -- | Yes | Performs regex match. |
Target | target | string | timestamp | Yes | Target fields. |
Add_field | add_field | hash | -- | No | Adds fields. |
Add_tag | add_tag | array | -- | No | Adds tags. |
Remove_field | remove_field | array | -- | No | Removes fields. |
Remove_tag | remove_tag | array | -- | No | Removes tags. |
Id | id | string | test | No | Id |
Locale | locale | string | -- | No | Locale |
Timezone | Specifies the time zone. | string | +8:00 | No | Specifies the time zone. |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
Blacklist_names | blacklist_names | array | -- | No | Excludes fields whose names match specified regular expressions. |
Blacklist_values | blacklist_values | array | -- | No | Excludes specified fields if their values match one of the supplied regular expressions. |
Whitelist_names | whitelist_names | array | -- | No | Includes specified fields only if their names match specified regular expressions. |
Whitelist_values | whitelist_values | array | -- | No | Includes specified fields only if their values match one of the supplied regular expressions. |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
Source | source | string | message | No | Defines the fields to be parsed. |
Columns | columns | array | -- | No | Defines a list of column names. |
Separator | separator | string | , | No | Defines the column separator value. |
Skip_empty_columns | skip_empty_columns | boolean | true | No | Defines whether empty columns can be skipped. |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
Filter_length | filter_length | number | 10 | No | Controls the field length. |
Set_time | set_time | ruby_time | 123 | No | Sets a time. |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
Source | source | string | message | Yes | Defines source fields. |
Skip_on_invalid_json | skip_on_invalid_json | boolean | true | No | Skips invalid json fields. |
Add_field | add_field | hash | null | No | Adds fields. |
Add_tag | add_tag | array | null | No | Adds tags. |
Remove_field | remove_field | array | null | No | Removes fields. |
Remove_tag | remove_tag | array | null | No | Removes tags. |
Target | target | string | message | No | Defines target fields. |
Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
Field | field | string | message | Yes | Defines fields to be split. |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
