Examples of Logstash Configuration Files for Migrating Elasticsearch Clusters Using Huawei Cloud Logstash

Checking the Cluster Type

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch to go to the Elasticsearch cluster list.
3. Select the source or destination Elasticsearch cluster, and click the cluster name to go to the cluster details page.
4. Check whether the security mode and HTTPS are enabled for the cluster. Figure 1 shows an Elasticsearch cluster with the security mode and HTTPS protocol both enabled.
   Figure 1 Checking cluster settings
   

   Table 1 Examples of Logstash configuration files for different types of clusters

   Scenario	Example Logstash Configuration File
   Migrating data between non-security mode clusters	Example of a Logstash Configuration File for Non-Security Mode Clusters
   Migrating data between security-mode clusters that use HTTP	Example of a Logstash Configuration File for Security-Mode Clusters That Use HTTP
   Migrating data between security-mode clusters that use HTTPS	Example of a Logstash Configuration File for Security-Mode Clusters That Use HTTPS

Example of a Logstash Configuration File for Non-Security Mode Clusters

The following is an example of a Logstash configuration file when security mode is disabled for both the source and destination Elasticsearch clusters.

input {
    elasticsearch {
        # Address of the source Elasticsearch cluster
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # List of indexes to be migrated, separated by commas (,).
        index => "xxx,xxx,xxx"
        # Retain the default values.
        docinfo => true
    }
}
 
filter {
    # Delete fields added by Logstash.
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}
 
output {
    elasticsearch {
        # Address of the destination Elasticsearch cluster
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # Index name of the destination cluster. The following configurations must be the same as that of the source cluster.
        index => "%{[@metadata][_index]}"
        # ID of the destination data. If you do not need to retain the original ID, delete the following line for better performance.
        document_id => "%{[@metadata][_id]}"
        # Retain the default values.
        manage_template => false
        ilm_enabled => false
    }
}

Example of a Logstash Configuration File for Security-Mode Clusters That Use HTTP

The following is an example of a Logstash configuration file when security mode is enabled for both the source and destination Elasticsearch clusters but HTTPS is disabled for them.

input {
    elasticsearch {
        # Username at the source end
        user => "xxx"
        # Password at the source end
        password => "xxx"
        # Address of the source Elasticsearch cluster
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # List of indexes to be migrated, separated by commas (,).
        index => "xxx,xxx,xxx"
        # Retain the default values.
        docinfo => true
    }
}
 
filter {
    # Delete fields added by Logstash.
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}
 
output {
    elasticsearch {
        # Username at the destination end
        user => "xxx"
        # Password at the destination end
        password => "xxx"
        # Address of the destination Elasticsearch cluster
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # Index name of the destination cluster. The following configurations must be the same as that of the source cluster.
        index => "%{[@metadata][_index]}"
        # ID of the destination data. If you do not need to retain the original ID, delete the following line for better performance.
        document_id => "%{[@metadata][_id]}"
        # Retain the default values.
        manage_template => false
        ilm_enabled => false
    }
}

Example of a Logstash Configuration File for Security-Mode Clusters That Use HTTPS

The following is an example of a Logstash configuration file when security mode and HTTPS are enabled for both the source and destination Elasticsearch clusters.

input {
    elasticsearch {
        # Username at the source end
        user => "xxx"
        # Password at the source end
        password => "xxx"
        # Address of the source Elasticsearch cluster
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # List of indexes to be migrated, separated by commas (,).
        index => "xxx,xxx,xxx"
        # Certificate of the source Elasticsearch cluster. For clusters on the cloud, retain the following information. For user-built Logstash clusters, download the certificate from the cluster details page. Enter the certificate path plus certificate name here.
        ca_file => "/rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/certs"
        # Retain the default values.
        docinfo => true
        ssl => true
    }
}
 
filter {
    # Delete fields added by Logstash.
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}
 
output {
    elasticsearch {
        # Username at the destination end
        user => "xxx"
        # Password at the destination end
        password => "xxx"
        # Address of the destination Elasticsearch cluster
        hosts => ["xx.xx.xx.xx:9200", "xx.xx.xx.xx:9200"]
        # Index name of the destination cluster. The following configurations must be the same as that of the source cluster.
        index => "%{[@metadata][_index]}"
        # ID of the destination data. If you do not need to retain the original ID, delete the following line for better performance.
        document_id => "%{[@metadata][_id]}"
        # Certificate of the destination Elasticsearch cluster. For clusters on the cloud, retain the following information. For user-built Logstash clusters, download the certificate to the node from the cluster details page. Enter the certificate path plus certificate name here.
        cacert => "/rds/datastore/logstash/v7.10.0/package/logstash-7.10.0/extend/certs"
        # Retain the default values.
        manage_template => false
        ilm_enabled => false
        ssl => true
        ssl_certificate_verification => false
    }
}