Configuring Read/Write Splitting Between Two Elasticsearch Clusters

Scenario

How read/write splitting between Elasticsearch clusters works:

1. Data writes: Users send write requests, and the leader cluster handles these requests.
2. Data synchronization.
   The leader synchronizes data changes to the follower through a REST API. Two synchronization methods are supported:

   • Exact match: synchronization of a specified index
   • Match by wildcard: batch synchronization of indexes matching a wildcard expression
3. Query processing: Users send query requests. The follower cluster handles these requests and returns the results.

Figure 1 illustrates how read/write splitting works.

Figure 1 How read/write splitting works

• Read and write split when both clusters are available (left): The leader handles writes, and the follower handles queries.
• Leader-to-follower switchover when the leader fails (right): If the leader becomes unavailable, the follower automatically upgrades to leader to ensure service continuity. For details, see Switching the Roles of the Leader and Follower Clusters.

Constraints

• Only Elasticsearch 7.6.2 and Elasticsearch 7.10.2 clusters support read/write splitting.
• The versions of the leader and follower clusters must be kept consistent, or errors may occur.

Prerequisites

Two clusters of the same version have been created. One functions as the leader cluster, and the other the follower cluster. The follower cluster must be able to access the REST API (default port: 9200) of the leader cluster.

Logging In to Kibana

Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. In the left navigation pane, choose Dev Tools.

   The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

Connecting the Leader and Follower Clusters

1. Run the following command to configure information about the leader cluster in the follower cluster:

   PUT /_cluster/settings
   {
     "persistent" : {
       "cluster" : {
         "remote.rest" : {
           "leader1" : {
             "seeds" : [
               "http://10.0.0.1:9200",
               "http://10.0.0.2:9200",
               "http://10.0.0.3:9200"
             ] ,
               "username": "elastic",
               "password": "*****"
           }
         }
       }
     }
   }

   Table 1 Request body parameters

   Parameter	Description
   leader1	Name of the leader cluster configuration task, which is user-defined and will be used for configuring read/write splitting later.
   seeds	Address for accessing the leader cluster. When HTTPS is enabled for the cluster, the URL schema must use HTTPS.
   username	Username of the leader cluster. This parameter is required only when security mode is enabled for the leader cluster.
   password	Password of the leader cluster. This parameter is required only when security mode is enabled for the leader cluster.

   Example response:

   {
     "acknowledged" : true,  //Whether the operation is successful
     "persistent" : {
       "cluster" : {
         "remote" : {
           "rest" : {
             "leader1" : {
               "seeds" : [
               "http://10.0.0.1:9200",
               "http://10.0.0.2:9200",
               "http://10.0.0.3:9200"
               ] ,
               "username": "elastic", 
                "password": "*****"
             }
           }
         }
       }
     },
     "transient" : { }
   }

2. After the configuration is complete, run the following command in the follower cluster to check the connection between the follower and leader clusters:

   GET _remote/rest/info

   Example response:

   {
     "leader1" : {
       "connected" : true  //The two clusters are connected.
     }
   }

Index Synchronization

There are two ways to synchronize indexes: specified index synchronization and matching index synchronization.

During synchronization, indexes in the follower cluster become read-only. The synchronization is performed periodically. The default synchronization interval is 30 seconds. For how to change it, see Changing the Synchronization Interval.

Stopping Index Synchronization

Run the following command in the follower cluster to stop synchronization tasks for specified indexes. Subsequent changes to the indexes in the leader cluster will not be synchronized to the follower cluster. The read-only state of the indexes in the follower cluster will be cancelled, so that new data can be written into them.

PUT log*/stop_remote_sync

In this command, log* indicates the index name. You can specify multiple index names (separated by commas) or use a wildcard. In this example, synchronization tasks for all indexes that start with log are stopped.

Querying and Deleting Created Patterns

1. Run the following command in the follower cluster to query created patterns:
   • Query the list of patterns.

     GET auto_sync/pattern

   • Query a specified pattern by name.

     GET auto_sync/pattern/{PATTERN}

   The following is an example of the response:

   {
     "patterns" : [
       {
         "name" : "pattern1",
         "pattern" : {
           "remote_cluster" : "leader",
           "remote_index_patterns" : [
             "log*"
           ],
           "local_index_pattern" : "{{remote_index}}-sync",
           "settings" : { }
         }
       }
     ]
   }

2. Run the following command in the follower cluster to delete a specified pattern by name:

   DELETE auto_sync/pattern/{PATTERN}

Enabling Forcible Synchronization

By default, the plug-in determines whether to synchronize metadata based on whether the number of documents in the index of the leader cluster changes. If the leader cluster only updates documents and the number of documents remains unchanged, the plug-in does not synchronize the updates to the follower cluster. The configuration can be modified. After forcible synchronization is enabled, the index metadata of the leader cluster is forcibly synchronized to the follower cluster in each synchronization cycle.

The following is an example of enabling forcible synchronization:

PUT _cluster/settings
{
  "persistent": {
    "remote_sync.force_synchronize": true
  }
}

Changing the Synchronization Interval

The default synchronization interval between the leader and follower clusters is 30 seconds and can be changed.

The example request below changes the synchronization interval to 2 seconds:

PUT {index_name}/_settings
{
  "index.remote_sync.sync_interval": "2s"
}

Changing the Synchronization Speed

You can change the data synchronization speed between the leader and follower clusters by configuring cluster-level settings.

The following is an example request: set the block size to 2 MB, the number of blocks to 20, and the maximum synchronization traffic to 100 MB per second.

PUT _cluster/settings
{
  "persistent": {
    "remote_sync.chunk_size": "2MB",
    "remote_sync.max_concurrent_file_chunks": 20,
    "remote_sync.max_bytes_per_sec": "100MB"
  }
}

Querying Index Synchronization Status

Obtain the auto synchronization status of a specified index.

An example request is as follows:

GET {index_name}/sync_stats

The following is an example of the response:

{
  "indices" : {
    "data1_follower" : {
      "shards" : {
        "0" : [
          {
            "primary" : false,             // Whether it is a primary shard
            "total_synced_times" : 27,    // Total synchronization times
            "total_empty_times" : 25,     // Total number of times when no data is synchronized between the leader and follower clusters because they have identical shards and data
            "total_synced_files" : 4,    // Number of synchronized files
            "total_synced_bytes" : 3580, // Total size of synchronized files
            "total_paused_nanos" : 0,   //Duration of synchronization pauses due to traffic throttling
            "total_paused_times" : 0,   //Number of synchronization pauses due to traffic throttling
            "current" : {
              "files_count" : 0, //Number of files that are being synchronized
              "finished_files_count" : 0, //Number of files that have been synchronized
              "bytes" : 0, //Size of files that are being synchronized
              "finished_bytes" : 0 //Size of files that have been synchronized
            }
          },
          {
            "primary" : true,             // Whether it is a primary shard
            "total_synced_times" : 28,    // Total synchronization times
            "total_empty_times" : 26,     // Total number of times when no data is synchronized between the leader and follower clusters because they have identical shards and data
            "total_synced_files": 20,    // Number of synchronized files
            "total_synced_bytes": 17547, // Total size of synchronized files
            "total_paused_nanos" : 0,     //Duration of synchronization pauses due to traffic throttling
            "total_paused_times" : 0,     //Number of synchronization pauses due to traffic throttling
            "current" : {
              "files_count" : 0,          //Number of files that are being synchronized
              "finished_files_count" : 0, //Number of files that have been synchronized
              "bytes" : 0,                //Size of files that are being synchronized
              "finished_bytes" : 0        //Size of files that have been synchronized
            }
          }
        ]
      }
    }
  }
}

Switching the Roles of the Leader and Follower Clusters

When the leader cluster becomes faulty, perform a leader/follower switchover to have the follower cluster take over services. The steps are as follows:

Leader/follower switchover won't work unless the leader and follower clusters use identical index names.

1. Determine the index synchronization method between the leader and follower clusters. Check whether pattern-matching index synchronization policies have been configured in the follower cluster. For the command to use, see Querying and Deleting Created Patterns.
   • If there are no such policies, synchronization is performed for specified indexes between the leader and follower clusters. In this case, go to 3.
   • If there are such policies, index synchronization between the leader and follower clusters is based on index patterns. In this case, go to 2.
2. Delete pattern-matching index synchronization policies in the follower cluster. For the command to use, see Querying and Deleting Created Patterns.
3. Perform Stopping Index Synchronization in the follower cluster. Then redirect read and write traffic to it. If the leader and follower clusters synchronize indexes based on index patterns, use a wildcard to match indexes when running the command that stops index synchronization.
4. After the leader cluster recovers, configure information about the follower cluster in the leader cluster, and connect the leader and follower clusters again. For details, see Connecting the Leader and Follower Clusters.
5. Under the leader cluster, perform Index Synchronization to synchronize data from the follower cluster to the leader cluster, and then perform a leader/follower switchover to switch back.