Parser Rules

The tenant-side data collection uses custom Logstash collectors for data transmission. Parsers mainly work as codeless filters in Logstash. Currently, the following types of Logstash filter plugins are supported.

**Table 1** Supported types
Parser	Plug-in in Logstash	Description
Key-Value filter	kv	Parses key-value pairs. For details about parsing rules, see Table 2.
Mutate filter	mutate	Performs general mutations on fields. For details about parsing rules, see Table 3.
Grok filter	grok	Parses regular expressions. For details about parsing rules, see Table 4.
Date filter	date	Parses the date. For details about parsing rules, see Table 5.
Drop filter	drop	Deletes packets. There is no specific rule. If you use this parser, logs received will be deleted.
Prune filter	prune	Parses blacklists and whitelists. For details about parsing rules, see Table 6.
CSV filter	csv	Parses the CSV data. For details about parsing rules, see Table 7.
Function filter	ruby	Executes ruby code. For details about parsing rules, see Table 8.
JSON filter	json	Converts the JSON data. For details about parsing rules, see Table 9.
Split filter	split	Splits data. For details about parsing rules, see Table 10.
Clone filter	clone	Duplicates data. For details about parsing rules, see Table 11.
UUID filter	uuid	Parses UUIDs. For details about parsing rules, see Table 12.

**Table 2** Kv filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Source	source	string	source	Yes	Defines the fields to be translated.
Target	target	string	message	No	Defines the target fields.
Field_split	field_split	string	,	No	Splits fields.
Value_split	value_split	string	=	No	Splits fields.
Trim_key	trim_key	string	--	No	Removes spaces from the key.
Trim_value	trim_value	string	--	No	Removes spaces from the value.
Allow_duplicate_values	allow_duplicate_values	boolean	true	No	Allows duplicate values.
Default_keys	default_keys	array	--	No	Adds keys.
Exclude_keys	exclude_keys	array	--	No	Excludes certain keys.
Include_keys	include_keys	array	--	No	Includes certain keys.
Prefix	prefix	string	--	No	Performs prefix matches.
Recursive	recursive	boolean	true	No	Performs Recursive parsing.
Transform_key	transform_key	string	--	No	Transforms keys.
Add_field	add_field	hash	--	No	Adds fields.
add_tag	add_tag	array	--	No	Adds tags.
Remove_field	remove_field	array	--	No	Removes fields.
Remove_tag	remove_tag	array	--	No	Removes tags.
Id	id	string	--	No	ID.
Whitespace	whitespace	string	strict/lenient	No	Allows whitespace characters.
Remove_char_key	remove_char_key	string	<>[](),	No	Removes characters from the key.

**Table 3** Mutate filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Convert	convert	hash	--	No	Converts a field's value into a different type.
Join	join	hash	--	No	Joins arrays.
Lowercase	lowercase	array	--	No	Converts characters into its lowercase equivalent.
Coerce	coerce	hash	--	No	Sets the default value of a field.
Rename	rename	hash	--	No	Renames fields.
Replace	replace	hash	--	No	Replaces the value of a field with a new value.
Split	split	hash	--	No	Split a field to an array.
Strip	strip	array	--	No	Strips spaces from fields.
Update	update	hash	--	No	Updates fields.
Uppercase	uppercase	array	--	No	Converts characters into its uppercase equivalent.
Add_field	add_field	hash	--	No	Adds fields.
Add_tag	add_tag	array	--	No	Adds tags.
Remove_field	remove_field	array	--	No	Removes fields.
Remove_tag	remove_tag	array	--	No	Removes tags.
ID	id	string	--	No	Id
Copy	copy	hash	--	No	Copies fields.
Gsub	gsub	array	--	No	Replaces the gsub value.

**Table 4** Grok filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
match	match	hash	--	Yes	Performs regex matches.
Break_on_match	break_on_match	boolean	true	No	Breaks on the first match.
Overwrite	overwrite	array	message	No	Overwrites fields.
Add_field	add_field	hash	--	No	Adds fields.
Add_tag	add_tag	array	--	No	Adds tags.
Remove_field	remove_field	array	--	No	Removes fields.
Remove_tag	remove_tag	array	--	No	Removes tags.
Id	id	string	--	No	Id

**Table 5** Date filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Match	match	array	--	Yes	Performs regex match.
Target	target	string	timestamp	Yes	Target fields.
Add_field	add_field	hash	--	No	Adds fields.
Add_tag	add_tag	array	--	No	Adds tags.
Remove_field	remove_field	array	--	No	Removes fields.
Remove_tag	remove_tag	array	--	No	Removes tags.
Id	id	string	test	No	Id
Locale	locale	string	--	No	Locale
Timezone	Specifies the time zone.	string	+8:00	No	Specifies the time zone.

**Table 6** Prune filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Blacklist_names	blacklist_names	array	--	No	Excludes fields whose names match specified regular expressions.
Blacklist_values	blacklist_values	array	--	No	Excludes specified fields if their values match one of the supplied regular expressions.
Whitelist_names	whitelist_names	array	--	No	Includes specified fields only if their names match specified regular expressions.
Whitelist_values	whitelist_values	array	--	No	Includes specified fields only if their values match one of the supplied regular expressions.

**Table 7** CSV filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Source	source	string	message	No	Defines the fields to be parsed.
Columns	columns	array	--	No	Defines a list of column names.
Separator	separator	string	,	No	Defines the column separator value.
Skip_empty_columns	skip_empty_columns	boolean	true	No	Defines whether empty columns can be skipped.

**Table 8** Function filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Filter_length	filter_length	number	10	No	Controls the field length.
Set_time	set_time	ruby_time	123	No	Sets a time.

**Table 9** JSON filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Source	source	string	message	Yes	Defines source fields.
Skip_on_invalid_json	skip_on_invalid_json	boolean	true	No	Skips invalid json fields.
Add_field	add_field	hash	null	No	Adds fields.
Add_tag	add_tag	array	null	No	Adds tags.
Remove_field	remove_field	array	null	No	Removes fields.
Remove_tag	remove_tag	array	null	No	Removes tags.
Target	target	string	message	No	Defines target fields.

**Table 10** Split filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Field	field	string	message	Yes	Defines fields to be split.

**Table 11** Clone filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Clone	clones	array	--	Yes	Defines the list of fields to be cloned.

**Table 12** UUID filter
Parsing Rule	Logstash Configuration Item	Type	Default Value	Mandatory	Description
Target	target	string	uuid	Yes	Target fields.
Overwrite	overwrite	boolean	true	Yes	Defines whether to overwrite.

Parent topic: Log Data Collection

Previous topic: Connector Rules

Next topic: Upgrading the Component Controller

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel