Example Logstash Configuration File

In the following example, the access types of the Elasticsearch clusters on the source and destination ends are the same. This means the source and destination ends both use security clusters or both use non-security clusters with HTTPS not enabled.

If the access types of the Elasticsearch clusters on the source and destination ends are different, you can combine the input and output parts of the following three sample files to obtain the required configuration file.

Non-security Cluster

If the security mode is not enabled for an Elasticsearch cluster, the access example is as follows:

input {
    elasticsearch {
        # Source Elasticsearch address
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # List of indexes to be migrated, separated by commas (,).
        index => "xxx,xxx,xxx"
        # Retain the default values.
        docinfo => true
    }
}
 
filter {
    # Delete fields added by Logstash.
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}
 
output {
    elasticsearch {
        # Destination Elasticsearch cluster address
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # Index name of the destination cluster. The following configurations must be the same as that of the source cluster.
        index => "%{[@metadata][_index]}"
        # ID of the destination data. If you do not need to retain the original ID, delete the following line to improve the performance.
        document_id => "%{[@metadata][_id]}"
        # Retain the default values.
        manage_template => false
        ilm_enabled => false
    }
}

Security Cluster (HTTPS Access Disabled)

If the security mode is enabled for the created cluster but HTTPS access is disabled, the access example is as follows:

input {
    elasticsearch {
        # Username of the source end
        user => "xxx"
        # Password of the source end
        password => "xxx"
        # IP address of the source Elasticsearch
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # List of indexes to be migrated, separated by commas (,).
        index => "xxx,xxx,xxx"
        # Retain the default values.
        docinfo => true
    }
}
 
filter {
    # Delete fields added by Logstash.
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}
 
output {
    elasticsearch {
        # Username of the destination end
        user => "xxx"
        # Password of the destination end
        password => "xxx"
        # Destination Elasticsearch cluster address
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # Index name of the target cluster. The following configurations must be the same as that of the source cluster.
        index => "%{[@metadata][_index]}"
        # ID of the destination data. If you do not need to retain the original ID, delete the following line to improve the performance.
        document_id => "%{[@metadata][_id]}"
        # Retain the default values.
        manage_template => false
        ilm_enabled => false
    }
}

Security Cluster (HTTPS Access Enabled)

If the security mode and HTTPS access are enabled for the created cluster, the access example is as follows:

input {
    elasticsearch {
        # Username of the source end
        user => "xxx"
        # Password of the source end
        password => "xxx"
        # IP address of the source Elasticsearch. Do not add protocols. If you add the HTTPS protocol, an error will be reported.
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # List of indexes to be migrated, separated by commas (,).
        index => "xxx,xxx,xxx"
        # Source Elasticsearch  certificate. For clusters on the cloud, retain the following information. For Logstash clusters built by yourself, download the certificate from the cluster details page. Enter the corresponding path.
        ca_file => "/rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/certs"
        # Retain the default values.
        docinfo => true
        ssl => true
    }
}
 
filter {
    # Delete fields added by Logstash.
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}
 
output {
    elasticsearch {
        # Username of the destination end
        user => "xxx"
        # Password of the destination end
        password => "xxx"
        # Destination Elasticsearch address. Do not add protocols.
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # Index name of the target cluster. The following configurations must be the same as that of the source cluster.
        index => "%{[@metadata][_index]}"
        # ID of the destination data. If you do not need to retain the original ID, delete the following line to improve the performance.
        document_id => "%{[@metadata][_id]}"
        # Source Elasticsearch certificate. For clusters on the cloud, retain the following information. For Logstash clusters built by yourself, download the certificate to the node on the cluster details page. Enter the corresponding path.
        cacert => "/rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/certs"
        # Retain the default values.
        manage_template => false
        ilm_enabled => false
        ssl => true
        ssl_certificate_verification => false
    }
}