Implementing Cross-Cluster Search Across CSS Elasticsearch Clusters

Scenarios

When your business data is distributed across multiple Elasticsearch clusters in different regions—for example, when a corporate headquarters needs to search log data from branch offices—the traditional approach is to physically migrate all data into a single cluster. This approach is costly and may disrupt business continuity. With cross-cluster search, you simply configure remote cluster connections on the local cluster. The local cluster can then search indexes stored in remote clusters directly, eliminating the need for data replication while enabling unified searches across multiple clusters.

Typical scenarios include:

• Cross-region data search: Search data stored across Elasticsearch clusters in different regions through a single entry point without physically consolidating the data.
• Unified search across multiple clusters: Aggregate search results from multiple business clusters without physically migrating the data.
• Multi-tenant isolation: Each tenant uses an independent cluster, while administrators perform unified searches across clusters.
• Disaster recovery: When the primary cluster becomes unavailable, quickly switch searches to the standby cluster by using cross-cluster search.

Solution Architecture

Figure 1 Cross-cluster search workflow

Cross-cluster search works as follows:

1. Configure network connectivity. Ensure that the local cluster can access the remote clusters through the Transport port (9300).
2. Configure remote cluster connections. Configure one or more remote cluster connections (using Transport addresses) on the local cluster.
3. Perform a cross-cluster search: The client sends a search request to the local cluster. In the request, remote indexes are specified using the syntax remote cluster name:index name.
4. Coordinate and aggregate results. Acting as the coordinating node, the local cluster forwards the search request to remote clusters, aggregates the returned results, and returns the combined response to the client.

Advantages

• Zero data replication: Cross-cluster search queries remote clusters directly over Transport connections, allowing newly indexed data to become searchable immediately without replication.
• Elastic scalability: Read and write clusters can be scaled independently and deployed across multiple regions.
• Unified query syntax: The standard Elasticsearch query syntax is used. You can specify remote indexes using the remote cluster name:index name format. This makes cross-cluster search easy to learn and use.

Performance Impact

• Impact on the local cluster: As the coordinating node, the local cluster uses CPU and memory resources to aggregate search results. If query concurrency is high, consider scaling out the local cluster.
• Impact of network latency: Query latency depends on the network latency between the local and remote clusters. Latency is typically negligible when clusters are in the same region. For cross-region deployments, enable transport.compress to reduce network overhead by compressing data transmitted between clusters.

Constraints

• The local cluster must be able to access the remote cluster through the Transport port (9300). By default, CSS does not expose port 9300. Configure VPC peering connections and cluster routes to allow traffic over this port.
• Cross-cluster search supports read operations only. You cannot use it to write data to remote clusters.
• The index mappings must be consistent across different clusters. Otherwise, unexpected search results may occur.
• The Elasticsearch image version must be x.x.x_25.3.0_x.x.x or later. Otherwise, cluster routes cannot be configured.
• The local and remote clusters should run the same Elasticsearch version to avoid compatibility issues.

Resource and Cost Planning

Prerequisites

• The local and remote clusters have been created and are running the same Elasticsearch version.
• The local and remote clusters are in the Available state.
• You have obtained the access addresses of the remote clusters.

Step 1: Configure Network Connectivity Between the Local and Remote Clusters

Ensure that the local cluster can access the remote clusters through the Transport port (9300). Choose the appropriate networking method based on whether the clusters are deployed in the same VPC.

• Same VPC

  If the clusters are deployed in the same VPC, they can communicate through private IP addresses. No additional configuration is required. Skip this step.

• Different VPCs

  If the clusters are deployed in different VPCs, perform the following in sequence: Create VPC peering connections; and configure routes for the clusters. For details, see Configuring Routes for a Cluster.

Step 2: Configure Remote Cluster Connections

Configure remote cluster connections on the local cluster so that it can access the remote clusters.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the local cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. In the navigation pane on the left, choose Dev Tools.
5. Run the following command to configure remote cluster connections in the local cluster.

   PUT _cluster/settings
   {
     "persistent": {
       "cluster": {
         "remote": {
           "<Name of remote cluster 1>": {
             "seeds": [
               "<Private IP address of node 1 in remote cluster 1>:9300,"
               "<Private IP address of node 2 in remote cluster 1>:9300,"
                "<Private IP address of node 3 in remote cluster 1>:9300,"
             ],
             "skip_unavailable": true
           },
           "<Name of remote cluster 2>": {
             "seeds": [
               "<Private IP address of node 1 in remote cluster 2>:9300,"
               "<Private IP address of node 2 in remote cluster 2>:9300,"
               "<Private IP address of node 3 in remote cluster 2>:9300,"
             ],
             "skip_unavailable": true
           }
         }
       }
     }
   }

   Table 2 Remote cluster connection parameters

   Parameter	Description
   <Remote cluster name>	Custom name of the remote cluster. This name is used to identify the remote cluster when performing cross-cluster searches. Examples: cluster_remote1 and cluster_remote2.
   seeds	List of Transport addresses of the remote cluster, in the format <IP address>:9300. You are advised to enter the addresses of all nodes in the remote cluster to improve connection reliability.
   skip_unavailable	Specifies whether searches should continue when a remote cluster is unavailable. true: Skips unavailable remote clusters. Results are returned only from available clusters, so a single remote cluster failure does not cause the entire search request to fail. false: Does not skip unavailable remote clusters. If any remote cluster is unavailable, the entire search request fails and an error is returned. You are advised to set this parameter to true.

6. Run the following command to verify connectivity between the local and remote clusters.

   GET _remote/info

   If the value connected is true for each remote cluster in the response, the connection has been established successfully.

Step 3: Prepare Test Data

Create indexes and import test data into each cluster to verify cross-cluster search. If your clusters already contain business data, skip this step and search the existing indexes directly.

The index mappings must be consistent across all clusters. Otherwise, cross-cluster search may return unexpected results.

1. In Kibana for the local cluster, run the following commands to create an index and import test data.

   # Create an index.
   PUT logs-2026
   {
     "mappings": {
       "properties": {
         "message": { "type": "text" }
       }
     }
   }
   
   # Import test data.
   POST logs-2026/_doc
   {"message": "info: local node healthy"}

2. In Kibana for remote cluster cluster_remote1, run the following commands to create an index and import test data.

   # Create an index.
   PUT logs-202601
   {
     "mappings": {
       "properties": {
         "message": { "type": "text" }
       }
     }
   }
   
   # Import test data.
   POST logs-202601/_doc
   {"message": "info: remote1 node healthy"}

3. In Kibana for remote cluster cluster_remote2, run the following commands to create an index and import test data.

   # Create an index.
   PUT logs-202602
   {
     "mappings": {
       "properties": {
         "message": { "type": "text" }
       }
     }
   }
   
   # Import test data.
   POST logs-202602/_doc
   {"message": "info: remote2 node healthy"}

Step 4: Perform Cross-Cluster Searches

In the local cluster, perform cross-cluster searches by specifying remote clusters and target indexes using the remote cluster name:index name format.

1. Search a single remote cluster.

   In the Kibana console for the local cluster, run the following command to search the logs-202601 index in the remote cluster cluster_remote1 for documents containing info.

   GET /cluster_remote1:logs-202601/_search
   {
     "query": {
       "match": {
         "message": "info"
       }
     }
   }

   Expected result: One document is returned, and the value of message is info: remote1 node healthy.

2. Search across multiple clusters.

   In the Kibana console for the local cluster, run the following command to search the logs-2026 index in the local cluster plus two remote clusters for documents containing info. For the local cluster, specify only the index name. For remote clusters, specify indexes using the remote cluster name:index name format.

   GET /logs-2026,cluster_remote1:logs-202601,cluster_remote2:logs-202602/_search
   {
     "query": {
       "match": {
         "message": "info"
       }
     }
   }

   Expected result: Three documents are returned, one from the local cluster and one from each remote cluster. The _index field displays the following values: logs-2026, cluster_remote1:logs-202601, and cluster_remote2:logs-202602. This confirms that the search results have been aggregated across all three clusters.

FAQ: What should I do if _remote/info shows connected: false after I configure a remote cluster connection?

Possible causes and solutions:

• Port 9300 is not allowed by the security group: Check the security group rules for both the local and remote clusters and ensure that port 9300 is allowed.
• The VPC peering connection is missing or incorrectly configured: If the clusters are deployed in different VPCs, verify that the required VPC peering connections and routing are configured correctly.
• Cluster routes or return routes are not configured: Ensure that routes have been configured in both the local cluster and remote clusters to enable them to connect to each other. The local cluster must include routes to the remote cluster's CIDR block, and the remote cluster must include return routes to the local cluster's CIDR block.
• The VPC CIDR blocks overlap: Verify that the VPC CIDR blocks used by the local and remote clusters do not overlap. If they do, reallocate addresses before configuring cross-cluster search.

Related Documents

• Configuring Routes for a Cluster: Learn how to configure routes for an Elasticsearch cluster.
• VPC Peering Connection: Learn how to create a VPC peering connection.
• Cross-Cluster Search: Learn more about cross-cluster search in the official Elastic documentation.