Logstash Configuration File Templates

Introduction to the Templates

Table 1 lists commonly used Logstash configuration file templates.

**Table 1** Logstash configuration file templates
Name	Description	Details
redis	Imports data from a Redis database to an Elasticsearch cluster.	Redis Template
elasticsearch	Migrates data between Elasticsearch clusters.	Elasticsearch Template
jdbc	Imports data from JDBC to an Elasticsearch cluster.	JDBC Template
kafka	Imports data from Kafka to an Elasticsearch cluster.	Kafka Template
beats	Imports data from Beats to an Elasticsearch cluster.	DIS Template
dis	Imports data from DIS to an Elasticsearch cluster.	Beats Template

Redis Template

Imports data from a Redis database to an Elasticsearch cluster.

input { 
    redis {
        data_type => "pattern_channel" # data_type, one of ["list", "channel", "pattern_channel"]
        key => "lgs-*" # The name of a Redis list or channel.
        host => "xxx.xxx.xxx.xxxx"
        port => 6379
    }
}

filter {
    # Delete some fields added by Logstash. 
    mutate {
        remove_field => ["@timestamp", "@version"] 
    } 
} 

output { 
    elasticsearch { 
        hosts => ["http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200"] # host 
        # user => "xxxx" # user name, just for security cluster 
        # password => "xxxx" # password, just for security cluster 
        index => "xxxxxx" # destination _index 
    } 
}

**Table 2** Configuration items
Configuration Item	Mandatory	Description
data_type	Yes	Data source type. The options are list, channel, and pattern_channel. If the parameter is set to list, the BLPOP key is used. If the parameter is set to channel, the SUBSCRIBE key is used. If the parameter is set to pattern_channel, the PSUBSCRIBE key is used.
key	Yes	Redis list or channel name
host	Yes	IP address of the Redis server
port	No	Number of the port to be connected. Default value: 6379.
hosts	Yes	Address for accessing the Elasticsearch cluster.
user	No	Username for accessing the Elasticsearch cluster. Generally, the value is admin. This parameter is required only for a security cluster.
password	No	Password for accessing the Elasticsearch cluster. The password is set when the cluster is created. This parameter is required only for a security cluster.
index	Yes	Index to which the data is to be migrated. Only one index can be configured.

For more information, see the Logstash document Redis input plugin.

Elasticsearch Template

Migrates data between Elasticsearch clusters.

input { 
    elasticsearch { 
		# Source ES cluster IP address. When SSL is enabled, please use 7.10.0 and you donot need to add a protocol, otherwise an error is reported.
		hosts => ["http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200"] 
		# user => "xxxx" # user name, just for security cluster
		# password => "xxxx" # password, just for security cluster
		index => "xxxx,xxx,xxx" # List of indexes to be migrated, separate with commas (,) 
		docinfo => true
		# source ES certificate, the cluster on the cloud this value remains the same. Enter the corresponding path when using a custom certificate. Self-built Logstash cluster, You can download the certs file from the ES cluster details page, enter the corresponding path here. 
		# ca_file => "/rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/certs"  # for 7.10.0
		# ssl => true # Set to true when SSL is enabled. 
    }
}

filter {
    # Delete some fields added by Logstash. 
    mutate {
		remove_field => ["@timestamp", "@version"] 
	} 
} 

output { 
    elasticsearch { 
		# Destination ES cluster IP address. When SSL is enabled, please use 7.10.0 and you donot need to add a protocol.
		hosts => ["http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200"]  
		# user => "xxx" # user name, just for security cluster 
		# password => "xxxx" # password, just for security cluster 
		index => "%{[@metadata][_index]}" # destination _index, this config ensure that the index is consistent with that on the source. You can also specify the index name.
		# document_type => "%{[@metadata][_type]}" # destination _type, this config ensure that the document_type is consistent with that on the source. 
		# document_id => "%{[@metadata][_id]}" # destination _id, if you do not need to retain the original _id, delete this line to improve performance. 
		# source ES certificate, the cluster on the cloud this value remains the same. Enter the corresponding path when using a custom certificate. Self-built Logstash cluster, You can download the certs file from the ES cluster details page, enter the corresponding path here. 
		# cacert => "/rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/certs"  # for 7.10.0
		# ssl => true # Set to true when SSL is enabled. 
		# ssl_certificate_verification => false # Set to false to ignore the validation server certificate when SSL is enabled.    
	} 
}

**Table 3** Configuration items
Configuration Item	Mandatory	Description
hosts	Yes	Address for accessing the destination Elasticsearch cluster to which data is imported.
user	No	Username for accessing the Elasticsearch cluster. Generally, the value is admin. This parameter is required only for a security cluster.
password	No	Password for accessing the Elasticsearch cluster. The password is set when the cluster is created. This parameter is required only for a security cluster.
index	Yes	Index from which data is to be migrated.
docinfo	No	Document information. Value: true or false. If this parameter is specified, include Elasticsearch document information in the event, such as the index, type, and ID.
ca_file	No	Default value: /rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/certs. For a cloud-based Logstash cluster, retain the default value, or when using a custom certificate, enter the path of the custom certificate. For an in-house built Logstash cluster, you can download the certificate on the details page of the SSL-enabled Elasticsearch cluster. Enter the path of the certificate here.
ssl	No	Set this parameter to true if SSL is enabled for the source Elasticsearch cluster.
hosts	Yes	Address for accessing the source Elasticsearch cluster where data comes from.
user	No	Username for accessing the Elasticsearch cluster. Generally, the value is admin. This parameter is required only for a security cluster.
password	No	Password for accessing the Elasticsearch cluster. The password is set when the cluster is created. This parameter is required only for a security cluster.
index	Yes	Index to which the data is to be migrated. Only one index can be configured.
document_type	No	This parameter is valid only when docinfo is set to true. If docinfo is set to false, delete this parameter from the configuration file.
document_id	No	This parameter is valid only when docinfo is set to true. If docinfo is set to false, delete this parameter from the configuration file.
cacert	No	Default value: /rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/certs. For a cloud-based Logstash cluster, retain the default value, or when using a custom certificate, enter the path of the custom certificate. For an in-house built Logstash cluster, you can download the certificate on the details page of the SSL-enabled Elasticsearch cluster. Enter the path of the certificate here.
ssl	No	Set this parameter to true if SSL is enabled for the destination Elasticsearch cluster.
ssl_certificate_verification	No	Set this parameter to false to enable SSL and ignore the server certificate verification.

For more information, see the Logstash document Elasticsearch input plugin.

JDBC Template

Imports data from JDBC to an Elasticsearch cluster.

input {
    jdbc {
        # for 7.10.0, jdbc_driver_library => "/rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/jars/mariadb-java-client-2.7.0.jar"
        jdbc_driver_library => "xxxxxxxxxxx"
        jdbc_driver_class => "org.mariadb.jdbc.Driver"
        jdbc_connection_string => "jdbc:mariadb://xxx.xxx.xxx.xxx:xxx/data_base_name" 
        jdbc_user => "xxxx"
        jdbc_password => "xxxx"
        statement => "SELECT * from table_name" # This SQL statement determines the data to be input
    }
}

filter {
    # Delete some fields added by Logstash. 
    mutate {
        remove_field => ["@timestamp", "@version"] 
    } 
} 

output { 
    elasticsearch { 
        hosts => ["http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200"] # host 
        # user => "xxxx" # user name, just for security cluster 
        # password => "xxxx" # password, just for security cluster 
        index => "xxxxxx" # destination _index 
    } 
}

**Table 4** Configuration items
Configuration Item	Mandatory	Description
jdbc_driver_library	Yes	Path of the JDBC driver library. If the Logstash cluster version is 7.10.0, set this parameter to /rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/jars/mariadb-java-client-2.7.0.jar. Currently, only these existing drivers are supported. User-defined upload is not supported.
jdbc_driver_class	Yes	JDBC driver class to be loaded, for example, org.mariadb.jdbc.Driver.
jdbc_connection_string	Yes	JDBC connection string
jdbc_user	Yes	JDBC username
jdbc_password	Yes	JDBC password
statement	Yes	SQL statement of the input data.
hosts	Yes	Address for accessing the source Elasticsearch cluster where data comes from.
user	No	Username for accessing the Elasticsearch cluster. Generally, the value is admin. This parameter is required only for a security cluster.
password	No	Password for accessing the Elasticsearch cluster. The password is set when the cluster is created. This parameter is required only for a security cluster.
index	Yes	Index to which the data is to be migrated. Only one index can be configured.

For more information, see the Logstash document Jdbc input plugin.

Kafka Template

Imports data from Kafka to an Elasticsearch cluster.

input {
    kafka{
        bootstrap_servers=>"xxx.xxx.xxx.xxx:xxxx"
        topics=>["xxxxxxxx"]
        group_id=>"kafka_es_test"
        auto_offset_reset=>"earliest"
    }
}

filter {
    # Delete some fields added by Logstash. 
    mutate {
        remove_field => ["@timestamp", "@version"] 
    } 
} 

output { 
    elasticsearch { 
        hosts => ["http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200"] # host 
        # user => "xxxx" # user name, just for security cluster 
        # password => "xxxx" # password, just for security cluster 
        index => "xxxxxx" # destination _index 
    } 
}

**Table 5** Configuration items
Configuration Item	Mandatory	Description
bootstrap_servers	Yes	IP address and port number of the Kafka instance
topics	Yes	List of topics to be subscribed to
group_id	Yes	Identifier of the group to which the consumer belongs.
auto_offset_reset	Yes	Initial offset in Kafka. earliest: Automatically reset the offset to the earliest offset. latest: Automatically reset the offset to the latest offset. none: Report an exception to the consumer if the previous offset for the consumer group is not found. anything else: The system throws an exception to the consumer.
hosts	Yes	Address for accessing the source Elasticsearch cluster where data comes from.
user	No	Username for accessing the Elasticsearch cluster. Generally, the value is admin. This parameter is required only for a security cluster.
password	No	Password for accessing the Elasticsearch cluster. The password is set when the cluster is created. This parameter is required only for a security cluster.
index	Yes	Index to which the data is to be migrated. Only one index can be configured.

For more information, see the Logstash document Kafka input plugin.

DIS Template

Imports data from DIS to an Elasticsearch cluster.

input {
    dis {
        # For details, see https://support.huaweicloud.com/usermanual-dis/dis_01_0099.html.
        streams => ["YOUR_DIS_STREAM_NAME"]
        endpoint => "https://dis.xxxxx.myhuaweicloud.com" # xxxxx should be replaced by your region name
        ak => "YOUR_ACCESS_KEY_ID"
        sk => "YOUR_SECRET_KEY_ID"
        region => "YOUR_Region"
        project_id => "YOUR_PROJECT_ID"
        group_id => "YOUR_APP_ID"
        client_id => "YOUR_CLIENT_ID"
        auto_offset_reset => "earliest"
    }
}

filter {
    # Delete some fields added by Logstash. 
    mutate {
        remove_field => ["@timestamp", "@version"] 
    } 
} 

output { 
    elasticsearch { 
        hosts => ["http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200"] # host 
        # user => "xxxx" # user name, just for security cluster 
        # password => "xxxx" # password, just for security cluster 
        index => "xxxxxx" # destination _index 
    } 
}

**Table 6** Configuration items
Configuration Item	Mandatory	Description
streams	Yes	Name of the DIS stream. The entered DIS stream name must be the same as the stream name specified when you are creating a DIS stream on the DIS console.
endpoint	Yes	Data API address of the region where DIS resides.
ak	Yes	User's access key (AK). For details, see Checking Authentication Information.
sk	Yes	User's secret key (SK). For details, see Checking Authentication Information.
region	Yes	Region where DIS is supported.
project_id	Yes	Project ID of the region. For details, see Checking Authentication Information.
group_id	Yes	DIS App name, used to identify a consumer group. The value can be any character string.
client_id	No	Client ID, which identifies a consumer in a consumer group. If multiple pipelines or Logstash instances are started for consumption, set this parameter to different values. For example, the value of instance 1 is client1, and the value of instance 2 is client2.
auto_offset_reset	No	Position where data starts to be consumed from the stream. The options are as follows: earliest: Data is consumed from the earliest one. latest: Data is consumed from the latest one.
hosts	Yes	Address for accessing the source Elasticsearch cluster where data comes from.
user	No	Username for accessing the Elasticsearch cluster. Generally, the value is admin. This parameter is required only for a security cluster.
password	No	Password for accessing the Elasticsearch cluster. The password is set when the cluster is created. This parameter is required only for a security cluster.
index	Yes	Index to which the data is to be migrated. Only one index can be configured.

Beats Template

Imports data from Beats to an Elasticsearch cluster.

input {
    beats {
        port => 5044 # port
    }
}

filter {
    # Delete some fields added by Logstash. 
    mutate {
        remove_field => ["@timestamp", "@version"] 
    } 
} 

output { 
    elasticsearch { 
        hosts => ["http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200", "http://xxx.xxx.xxx.xxx:9200"] # host 
        # user => "xxxx" # user name, just for security cluster 
        # password => "xxxx" # password, just for security cluster 
        index => "xxxxxx" # destination _index 
    } 
}

**Table 7** Configuration items
Configuration Item	Mandatory	Description
port	Yes	The port number 5044 is used for connecting and indexing Elasticsearch through Beats.
hosts	Yes	Address for accessing the source Elasticsearch cluster where data comes from.
user	No	Username for accessing the Elasticsearch cluster. Generally, the value is admin. This parameter is required only for a security cluster.
password	No	Password for accessing the Elasticsearch cluster. The password is set when the cluster is created. This parameter is required only for a security cluster.
index	Yes	Index to which the data is to be migrated. Only one index can be configured.