Parser Rules
The tenant-side data collection uses custom Logstash collectors for data transmission. Parsers mainly work as codeless filters in Logstash. Currently, the following types of Logstash filter plugins are supported.
| Parser | Plugin in Logstash | Description |
|---|---|---|
| Key-Value filter | kv | Parses key-value pairs. For details about parsing rules, see Table 2. |
| Mutate filter | mutate | Performs general mutations on fields. For details about parsing rules, see Table 3. |
| Grok filter | grok | Parses regular expressions. For details about parsing rules, see Table 4. |
| Date filter | date | Parses the date. For details about parsing rules, see Table 5. |
| Drop filter | drop | Deletes packets. There is no specific rule. If you use this parser, logs received will be deleted. |
| Prune filter | prune | Parses blacklists and whitelists. For details about parsing rules, see Table 6. |
| CSV filter | csv | Parses the CSV data. For details about parsing rules, see Table 7. |
| Function filter | ruby | Executes ruby code. For details about parsing rules, see Table 8. |
| JSON filter | json | Converts the JSON data. For details about parsing rules, see Table 9. |
| Split filter | split | Splits data. For details about parsing rules, see Table 10. |
| Clone filter | clone | Duplicates data. For details about parsing rules, see Table 11. |
| UUID filter | uuid | Parses UUIDs. For details about parsing rules, see Table 12. |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| Source | source | string | source | Yes | Defines the fields to be translated. |
| Target | target | string | message | No | Defines the target fields. |
| Field_split | field_split | string | , | No | Splits fields. |
| Value_split | value_split | string | = | No | Splits fields. |
| Trim_key | trim_key | string | -- | No | Removes spaces from the key. |
| Trim_value | trim_value | string | -- | No | Removes spaces from the value. |
| Allow_duplicate_values | allow_duplicate_values | boolean | true | No | Allows duplicate values. |
| Default_keys | default_keys | array | -- | No | Adds keys. |
| Exclude_keys | exclude_keys | array | -- | No | Excludes certain keys. |
| Include_keys | include_keys | array | -- | No | Includes certain keys. |
| Prefix | prefix | string | -- | No | Performs prefix matches. |
| Recursive | recursive | boolean | true | No | Performs Recursive parsing. |
| Transform_key | transform_key | string | -- | No | Transforms keys. |
| Add_field | add_field | hash | -- | No | Adds fields. |
| add_tag | add_tag | array | -- | No | Adds tags. |
| Remove_field | remove_field | array | -- | No | Removes fields. |
| Remove_tag | remove_tag | array | -- | No | Removes tags. |
| Id | id | string | -- | No | ID. |
| Whitespace | whitespace | string | strict/lenient | No | Allows whitespace characters. |
| Remove_char_key | remove_char_key | string | <>[](), | No | Removes characters from the key. |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| Convert | convert | hash | -- | No | Converts a field's value into a different type. |
| Join | join | hash | -- | No | Joins arrays. |
| Lowercase | lowercase | array | -- | No | Converts characters into its lowercase equivalent. |
| Coerce | coerce | hash | -- | No | Sets the default value of a field. |
| Rename | rename | hash | -- | No | Renames fields. |
| Replace | replace | hash | -- | No | Replaces the value of a field with a new value. |
| Split | split | hash | -- | No | Split a field to an array. |
| Strip | strip | array | -- | No | Strips spaces from fields. |
| Update | update | hash | -- | No | Updates fields. |
| Uppercase | uppercase | array | -- | No | Converts characters into their uppercase equivalent. |
| Add_field | add_field | hash | -- | No | Adds fields. |
| Add_tag | add_tag | array | -- | No | Adds tags. |
| Remove_field | remove_field | array | -- | No | Removes fields. |
| Remove_tag | remove_tag | array | -- | No | Removes tags. |
| ID | id | string | -- | No | Id |
| Copy | copy | hash | -- | No | Copies fields. |
| Gsub | gsub | array | -- | No | Replaces the gsub value. |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| match | match | hash | -- | Yes | Performs regex matches. |
| Break_on_match | break_on_match | boolean | true | No | Breaks on the first match. |
| Overwrite | overwrite | array | message | No | Overwrites fields. |
| Add_field | add_field | hash | -- | No | Adds fields. |
| Add_tag | add_tag | array | -- | No | Adds tags. |
| Remove_field | remove_field | array | -- | No | Removes fields. |
| Remove_tag | remove_tag | array | -- | No | Removes tags. |
| Id | id | string | -- | No | Id |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| Match | match | array | -- | Yes | Performs regex match. |
| Target | target | string | timestamp | Yes | Target fields. |
| Add_field | add_field | hash | -- | No | Adds fields. |
| Add_tag | add_tag | array | -- | No | Adds tags. |
| Remove_field | remove_field | array | -- | No | Removes fields. |
| Remove_tag | remove_tag | array | -- | No | Removes tags. |
| Id | id | string | test | No | Id |
| Locale | locale | string | -- | No | Locale |
| Timezone | Specifies the time zone. | string | +8:00 | No | Specifies the time zone. |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| Blacklist_names | blacklist_names | array | -- | No | Excludes fields whose names match specified regular expressions. |
| Blacklist_values | blacklist_values | array | -- | No | Excludes specified fields if their values match one of the supplied regular expressions. |
| Whitelist_names | whitelist_names | array | -- | No | Includes specified fields only if their names match specified regular expressions. |
| Whitelist_values | whitelist_values | array | -- | No | Includes specified fields only if their values match one of the supplied regular expressions. |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| Source | source | string | message | No | Defines the fields to be parsed. |
| Columns | columns | array | -- | No | Defines a list of column names. |
| Separator | separator | string | , | No | Defines the column separator value. |
| Skip_empty_columns | skip_empty_columns | boolean | true | No | Defines whether empty columns can be skipped. |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| Filter_length | filter_length | number | 10 | No | Controls the field length. |
| Set_time | set_time | ruby_time | 123 | No | Sets a time. |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| Source | source | string | message | Yes | Defines source fields. |
| Skip_on_invalid_json | skip_on_invalid_json | boolean | true | No | Skips invalid json fields. |
| Add_field | add_field | hash | null | No | Adds fields. |
| Add_tag | add_tag | array | null | No | Adds tags. |
| Remove_field | remove_field | array | null | No | Removes fields. |
| Remove_tag | remove_tag | array | null | No | Removes tags. |
| Target | target | string | message | No | Defines target fields. |
| Parsing Rule | Logstash Configuration Item | Type | Default Value | Mandatory | Description |
|---|---|---|---|---|---|
| Field | field | string | message | Yes | Defines fields to be split. |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
