Example Configuration Files for Migrating Elasticsearch/OpenSearch Clusters Using CSS Logstash

Checking the Cluster Type

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the source or destination Elasticsearch cluster, and click the cluster name to go to the cluster information page.
4. Click the Overview tab. In the Network Information area, check whether Security Mode and HTTPS Access are enabled. Figure 1 shows an Elasticsearch cluster with the security mode and HTTPS both enabled.
   Figure 1 Checking a cluster's security settings
   

   Table 1 Example Logstash configuration files for different types of clusters

   Scenario	Example Logstash Configuration File
   Migrating data between non-security mode clusters	Example Logstash Configuration File for Non-Security Mode Clusters
   Migrating data between security-mode clusters that use HTTP	Example Logstash Configuration File for Security-Mode Clusters That Use HTTP
   Migrating data between security-mode clusters that use HTTPS	Example Logstash Configuration File for Security-Mode Clusters That Use HTTPS

   Depending on the security-mode settings of the source and destination clusters, modify the input and output modules provided in our examples to obtain the configuration file you need.

Example Logstash Configuration File for Non-Security Mode Clusters

The following is an example Logstash configuration file when security mode is disabled for both the source and destination Elasticsearch or OpenSearch clusters.

input {
    elasticsearch {
        # Source cluster node addresses. No need to include the protocol.
        hosts => ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200"]
        # Name of the source index to be migrated
        index => "xxx,xxx"
        docinfo => true
    }
}

filter {
    # Removes some metadata fields automatically added by Logstash
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}

output {
    elasticsearch {
        # Destination cluster node addresses. No need to include the protocol.
        hosts => ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200"]
        # Target index for writing events. The following configuration retains the source index name, document type, and document IDs.
        index => "%{[@metadata][_index]}"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"
    }
}

Example Logstash Configuration File for Security-Mode Clusters That Use HTTP

The following is an example Logstash configuration file for when security mode is enabled for both the source and destination Elasticsearch or OpenSearch clusters but HTTPS is disabled for them.

input {
    elasticsearch {
        # Source cluster node addresses. No need to include the protocol.
        hosts => ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200"]
        # Name of the source index to be migrated
        index => "xxx,xxx"
        docinfo => true
        user => "xxx"           # Username for accessing the cluster
        password => "xxx"       # Password corresponding to the username
    }
}

filter {
    # Removes some metadata fields automatically added by Logstash
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}

output {
    elasticsearch {
        # Destination cluster node addresses. No need to include the protocol.
        hosts => ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200"]
        # Target index for writing events. The following configuration retains the source index name, document type, and document IDs.
        index => "%{[@metadata][_index]}"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"
        user => "xxx"           # Username for accessing the cluster
        password => "xxx"       # Password corresponding to the username
    }
}

Example Logstash Configuration File for Security-Mode Clusters That Use HTTPS

The following is an example Logstash configuration file for when security mode and HTTPS are enabled for both the source and destination Elasticsearch or OpenSearch clusters.

input {
    elasticsearch {
        # Source cluster node addresses. No need to include the protocol.
        hosts => ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200"]
        # Name of the source index to be migrated
        index => "xxx,xxx"
        docinfo => true
        user => "xxx"           # Username for accessing the cluster
        password => "xxx"       # Password corresponding to the username
        ssl => true             # Enable SSL connections.
        ca_file => "/opt/logstash/extend/certs"     # Path of the CA certificate used to verify the source cluster
    }
}

filter {
    # Removes some metadata fields automatically added by Logstash
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}

output {
    elasticsearch {
        # Destination cluster node addresses. No need to include the protocol.
        hosts => ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200"]
        # Target index for writing events. The following configuration retains the source index name, document type, and document IDs.
        index => "%{[@metadata][_index]}"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"
        user => "xxx"           # Username for accessing the cluster
        password => "xxx"       # Password corresponding to the username
        ssl => true             # Enable SSL connections.
        cacert => "/opt/logstash/extend/certs"       # Path of the CA certificate used to verify the destination cluster
        ssl_certificate_verification => false        # Whether to enable SSL certificate verification for the destination cluster.
    }
}