Help Center/ SecMaster/ User Guide/ Log Audit/ Log Data Collection/ Connector Rules

Updated on 2025-08-11 GMT+08:00

View PDF

Connector Rules

Source Connectors

SecMaster provides a wide range of source connectors for you to collect security data from your security products.

**Table 1** Source connector types
Connector Type	Logstash In Use	Description
TCP	tcp	This collector is used to receive TCP logs. For details about the configuration rules, see Table 2.
UDP	udp	This collector is used to receive UDP logs. For details about the configuration rules, see Table 3.
OBS	obs	This collector is used to obtain log data from an OBS bucket. For details about the configuration rules, see Table 4.
Kafka	kafka	This collector is used to obtain Kafka network log data. For details about the configuration rules, see Table 5.
SecMaster	pipe	This collector is used to transfer SecMaster data externally. For details about configuration rules, see Table 6.
Elasticsearch	elasticsearch	This collector is used to read data from the Elasticsearch cluster. For details about the configuration rules, see Table 7.

**Table 2** TCP connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the connector with custom sources. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Port	port	number	1025	Yes	Port for the collection node. Port range: 1025 to 65535 The port for each connector must be unique.
Codec	codec	string	plain	Yes	Encoding format. Plain: Reads the raw content. Json: Processes the content in JSON format.
SSL_enable	ssl_enable	boolean	false	No	Whether to enable SSL authentication.
SSL certificate	ssl_cert	file	null	No	Certificate.
SSL key	ssl_key	file	--	No	SSL key.
SSL key.	ssl_key_passphrase	string	--	No	SSL certificate key.

**Table 3** UDP connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Port	port	number	1025	Yes	Port for the collection node. The port ranges from 1025 to 65535. The port for each connector must be unique.
Codec	codec	string	plain	Yes	Decoding type Plain: Reads the raw content. Json: Processes the content in JSON format.
Type	type	string	udp	No	Packet label, which is used for subsequent processing. One data collection channel maps to multiple data connections. You are advised to add different packet labels to each data connection to distinguish packets of different data connections.
Queue_size	queue_size	number	1000000	No	Queue size.
Receive_buffer_bytes	receive_buffer_bytes	number	1000000	No	Number of bytes in the receiving buffer. Retain the default value.
Buffer_size	buffer_size	number	1000000	No	Buffer size. Retain the default value.
Workers	workers	number	1	No	Number of worker threads. Retain the default value.

**Table 4** OBS connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Region	Region	string	--	Yes	Region of the workspace where the data source is located.
Bucket	bucket	string	demo-obs-sec-mrd-datas	Yes	Bucket name, which is the name of the bucket where the data source is located.
Endpoint	Endpoint	string	https://obs.huawei.com	Yes	Endpoint address. Note that "https" must be added. You can obtain the endpoint address from the bucket details.
AK	ak	string	--	No	AK
SK	sk	string	--	No	SK
Prefix	prefix	string	/test	No	Prefix of the folder where logs are read. Enter the folder prefix corresponding to the storage path of the data source in the bucket.
Cache folder	temporary_directory	string	/temp	No	Cache folder for reading logs. When SecMaster log data is transferred out, the data needs to be cached in the cache folder first. The data is transferred out after the data volume reaches a certain threshold. You need to create the /opt/cloud/logstash/myobs directory on the ECS and use the directory as the cache folder path.
Packet label	type	string	--	No	Packet label, which is used for subsequent processing. One data collection channel maps to multiple data connections. You are advised to add different packet labels to each data connection to distinguish packets of different data connections.
Memory path	sincedb_path	string	/opt/cloud/logstash/pipeline/file_name	No	Log read position. This parameter is used to prevent full-text traversal caused by restart. You are advised to retain the default value.

Table 5 Kafka connector configuration rules

Rule

Logstash Settings

Type

Default Value

Mandatory

Description

Title

title

string

Yes

Name of the custom connector. It must meet the following requirements:

The name can contain 5 to 128 characters.
The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.

Description

description

string

Yes

Description of the custom connector. It must meet the following requirements:

The name can contain up to 256 characters.

Bootstrap_servers

bootstrap_servers

string

Yes

Kafka service address.

Topics

topics

array

logstash

Yes

Topics. Multiple topics can be consumed at the same time.

Consumer threads

consumer_threads

number

Yes

Consumer threads

Auto offset reset

auto_offset_reset

string

latest

Offset reset:

Earliest: Read the earliest message.
Latest: Read the latest messages.

SSL certificate

ssl_truststore_location

file

SSL certificate

This parameter is mandatory when SSL is selected.

SSL key

ssl_truststore_password

string

SSL key

This parameter is mandatory when SSL is selected.

Security_protocol

security_protocol

string

SASL_SSL

Security protocol.

PLAINTEXT: plaintext transmission scenario

SSL: SSL transmission scenario
SASL_PLAINTEXT: SASL plaintext transmission scenario
SASL_SSL: SASL ciphertext transmission scenario

Sasl_jaas_config

sasl_jaas_config

string

SASL connection configuration.

Is_pw_encrypted

is_pw_encrypted

string

false

Whether to encrypt the value.

Sasl_mechanism

sasl_mechanism

string

PLAIN

SASL mechanism.

Group_id

group_id

string

Consumer group ID.

Set sasl_jaas_config based on the Kafka specifications. Example:

Plaintext connection configuration

           
                org.apache.kafka.common.security.plain.PlainLoginModule required username='kafka user'password='kafka password';

Ciphertext connection configuration

           
                org.apache.kafka.common.security.scram.ScramLoginModule required username='kafka user name'password='kafka password';

**Table 6** SecMaster connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Type	type	string	Tenant	Yes	The options are Tenant, Platform, and ECS Agency. You are advised to set this parameter to ECS Agency. Tenant: Select Tenant in the IAM authentication scenario. You also need to specify Domain_name, User_name, and User_password. Platform: Select Platform in the AK/SK authentication scenario. You also need to specify AK and SK. ECS Agency: Select this option if you want to obtain tokens through ECS agencies. In doing this, the data collection function is not affected even if the account or password changes. Before using an ECS agency, create an iasp-agent agency and grant permissions to it on IAM, and configure the agency on ECS. For details, see 10.a and 10.b.
StreamTable	pipe_id	string	--	Yes	Select a stream table from the drop-down list. This table stores the source data you want to transfer out from SecMaster.
AK	ak	string	--	Yes	You need to specify AK only when you set Type to Platform in the AK/SK authentication scenario.
SK	sk	string	--	Yes	You need to specify SK only when you set Type to Platform in the AK/SK authentication scenario.
Domain_name	domain_name	string	domain_name	Yes	Domain name of the IAM user. You need to specify Domain_name only when you set Type to Tenant in IAM authentication scenario.
User_name	user_name	string	user_name	Yes	Username of the IAM user. You need to specify User_name only when you set Type to Tenant in the IAM authentication scenario.
User_password	user_password	string	--	Yes	Password of the IAM user. You need to specify User_password only when you set Type to Tenant in the IAM authentication scenario.
Subscription_type	subscription_type	string	Shared	No	Subscription type. Retain the default value. Shared: shared mode Exclusive: exclusive mode Failover: disaster recovery mode
Subscription_initial_position	subscription_initial_position	string	Latest	No	Initial subscription position. Retain the default value. Latest: The data is consumed from the latest position of the topic corresponding to the stream table. Earliest: The data is consumed from the earliest position of the topic corresponding to the stream table.

**Table 7** Elasticsearch connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Hosts	hosts	array	--	Yes	Address of the host where the data source is located.
Index	index	string	--	Yes	Index name.
Query	query	string	--	Yes	Retrieval statement for reading data.
User	user	string	--	Yes	Username for logging in to Elasticsearch.
User_password	user_password	string	--	Yes	Password for logging in to Elasticsearch.
Size	size	number	20	Yes	Queries
Scroll	scroll	string	5m	Yes	Volume
Docinfo	docinfo	boolean	Yes	Yes	Document
Is_pw_encrypted	is_pw_encrypted	boolean	Yes	Yes	Whether to enable encryption
Ssl	ssl	boolean	Yes	No	Whether to enable SSL.
Ca_file	ca_file	file	--	No	Certificate file.
SsL_certificate_verification	ssl_certificate_verification	boolean	Yes	No	Whether to enable SSL certificate verification.

Destination Connectors

SecMaster provides a wide range of destination connectors for you to collect security data from your security products.

**Table 8** Destination connectors
Connector Type	In-use Logstash	Description
TCP	tcp	This collector is used to send TCP logs. For details about the configuration rules, see Table 9.
UDP	udp	This collector is used to send UD logs. For details about the configuration rules, see Table 10.
Kafka	kafka	This collector is used to write logs to Kafka message queues. For details about the configuration rules, see Table 11.
OBS	obs	This collector is used to write logs to OBS buckets. For details about the configuration rules, see Table 12.
SecMaster pipeline	pipe	This collector is used to write logs to the SecMaster pipeline. For details about the configuration rules, see Table 13.

**Table 9** TCP connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Port	port	number	1025	Yes	Port
Codec	codec	string	plain	Yes	Decoding type, which can be Json_lines or Plain. Plain: Reads the original content. Json_lines: Processes the content in JSON format.
Hosts	host	string	192.168.0.66	Yes	Host address Note: The network between the host and the node is normal.
Ssl_enable	ssl_enable	boolean	false	No	Whether to enable SSL authentication.
SSL certificate	ssl_cert	file	--	No	SSL certificates
SSL key	ssl_key	file	--	No	SSL certificate file
SSL key	ssl_key_passphrase	string	--	No	SSL certificate key

**Table 10** UDP connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Hosts	host	string	--	Yes	Host IP address. Note: The network between the host and the node is normal.
Port	port	number	1025	Yes	Port
Decoding type	codec	string	json_lines	Yes	Decoding type, which can be Json_lines or Plain. Plain: Reads the original content. Json_lines: Processes the content in JSON format.
Retry count	retry_count	number	3	No	Time of retry attempts
Retry backoff (ms)	retry_backoff_ms	number	200	No	Retry backoff (ms)

**Table 11** Kafka connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Service address	bootstrap_servers	string	--	Yes	Service address, for example, 192.168.21.21:9092,192.168.21.24:9999.
Topics	topic_id	string	logstash	Yes	Topics
Decoding type	codec	string	plain	Yes	Decoding type, which can be Json or Plain.
Maximum length of the request	max_request_size	number	10485760	Yes	Maximum length of the request
Security_protocol	security_protocol	string	PLAINTEXT	No	Security protocol
SASL connection configuration	sasl_jaas_config	string	--	No	SASL connection configuration
Encrypted	is_pw_encrypted	string	true	No	Encrypted
SASL mechanism	sasl_mechanism	string	PLAIN	No	sasl_mechanism
Set sasl_jaas_config based on the Kafka specifications. The following is an example: Plaintext connection configuration org.apache.kafka.common.security.plain.PlainLoginModule required username='kafka user'password='kafka password'; Ciphertext connection configuration org.apache.kafka.common.security.scram.ScramLoginModule required username='kafka user name'password='kafka password';

**Table 12** OBS connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Region	Region	string	--	Yes	Region
Bucket	bucket	string	demo-obs-sec-mrd-datas	Yes	Bucket name
Endpoint	Endpoint	string	https://obs.huawei.com	Yes	Endpoint
Cache folder	temporary_directory	string	/temp/logstash/	Yes	Cache path
Encoding type	codec	string	plain	No	Encoding format: plain or JSON
AK	ak	string	--	No	AK
SK	sk	string	--	No	SK
Prefix	prefix	string	test	No	Path prefix.
Encoding format	encoding	string	gzip	No	Encoding format: gzip or pure file

**Table 13** SecMaster connector configuration rules
Rule	Logstash Settings	Type	Default Value	Mandatory	Description
Title	title	string	--	Yes	Name of the custom connector. It must meet the following requirements: The name can contain 5 to 128 characters. The name must be unique in the workspace and cannot be the same as the name of any other connector in the workspace.
Description	description	string	--	Yes	Description of the custom connector. It must meet the following requirements: The name can contain up to 256 characters.
Type	type	string	Tenant	Yes	The options are Tenant, Platform, and ECS Agency. Tenant: Select Tenant in the IAM authentication scenario. You also need to specify Domain_name, User_name, and User_password. Platform: Select Platform in the AK/SK authentication scenario. You also need to specify AK and SK. ECS Agency: Select this option if you want to obtain tokens through ECS agencies. The data collection function is not affected even if the tenant account or password changes. Before selecting the ECS agency, you need to configure an agency for ECS. For details, see Configuring an ECS Agency.
StreamTable	pipe_id	string	--	Yes	Select a stream table from the drop-down list. This table stores the source data you want to transfer into SecMaster.
AK	ak	string	--	Yes	You need to specify AK only when you set Type to Platform in the AK/SK authentication scenario.
SK	sk	string	--	Yes	You need to specify SK only when you set Type to Platform in the AK/SK authentication scenario.
Domain_name	domain_name	string	domain_name	Yes	Domain name of the IAM user. You need to specify Domain_name only when you set Type to Tenant in the IAM authentication scenario.
User_name	user_name	string	user_name	Yes	Username of the IAM user. You need to specify User_name only when you set Type to Tenant in the IAM authentication scenario.
User_password	user_password	string	--	Yes	Password of the IAM user. You need to specify User_password only when you set Type to Tenant in the IAM authentication scenario.
Compression_type	compression_type	string	NONE	No	Packet compression type. You are advised to retain the default value.
Block_if_queue_full	block_if_queue_full	boolean	true	Yes	If the queue is full, new requests are rejected. You are advised to retain the default value.
Enable_batching	enable_batching	boolean	true	Yes	Whether to enable batch processing. You are advised to retain the default value.