Configuring Read/Write Splitting Between Two OpenSearch Clusters

How the Feature Works

Read/write splitting between a pair of leader and follower clusters works as follows:

• Data synchronization: The leader synchronizes data changes to the follower through a REST API. Two synchronization methods are supported:
  • Exact match: synchronization of a specified index
  • Match by wildcard: batch synchronization of indexes matching a wildcard expression
• Leader/follower switchover: If the leader becomes unavailable, the follower automatically upgrades to leader to take over write and query loads.

Figure 1 How read/write splitting works

• Read and write split when both clusters are available (left): The leader handles writes, and the follower handles queries.
• Automatic failover when the leader fails (right): If the leader becomes unavailable, the follower automatically upgrades to leader to ensure service continuity.

Constraints

• Only OpenSearch 2.19.0 clusters support read/write splitting.
• The leader and follower clusters must use the same software version.

Prerequisites

Two clusters of the same version have been created. One functions as the leader cluster, and the other the follower cluster. The follower cluster must be able to access the REST API (default port: 9200) of the leader cluster.

Logging In to OpenSearch Dashboards

Log in to OpenSearch Dashboards and go to the command execution page. OpenSearch clusters support multiple access methods. This topic uses OpenSearch Dashboards as an example to describe the operation procedures.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > OpenSearch.
3. In the cluster list, find the target cluster, and click Dashboards in the Operation column to log in to OpenSearch Dashboards.
4. In the left navigation pane, choose Dev Tools.

Connecting the Leader and Follower Clusters

1. Run the following command to configure information about the leader cluster in the follower cluster:

   PUT /_cluster/settings
   {
     "persistent" : {
       "cluster" : {
         "remote.rest" : {
           "leader1" : {
             "seeds" : [
               "http://10.0.0.1:9200",
               "http://10.0.0.2:9200",
               "http://10.0.0.3:9200"
             ] ,
               "username": "elastic",
               "password": "*****"
           }
         }
       }
     }
   }

   Table 1 Request body parameters

   Parameter	Description
   leader1	Name of the leader cluster configuration task, which is user-defined and will be used for configuring read/write splitting later.
   seeds	Address for accessing the leader cluster. When HTTPS is enabled for the cluster, the URL schema must use HTTPS.
   username	Username of the leader cluster. This parameter is required only when security mode is enabled for the leader cluster.
   password	Password of the leader cluster. This parameter is required only when security mode is enabled for the leader cluster.

   Example response:

   {
     "acknowledged" : true,  //Whether the operation is successful
     "persistent" : {
       "cluster" : {
         "remote" : {
           "rest" : {
             "leader1" : {
               "seeds" : [
               "http://10.0.0.1:9200",
               "http://10.0.0.2:9200",
               "http://10.0.0.3:9200"
               ] ,
               "username": "elastic", 
                "password": "*****"
             }
           }
         }
       }
     },
     "transient" : { }
   }

2. After the configuration is complete, run the following command in the follower cluster to check the connection between the follower and leader clusters:

   GET _remote/rest/info

   Example response:

   {
     "leader1" : {
       "connected" : true  //The two clusters are connected.
     }
   }

Index Synchronization

There are two ways to synchronize indexes: synchronization of a specified index and synchronization of indexes matching a wildcard expression.

During synchronization, indexes in the follower cluster become read-only. The synchronization is performed periodically. The default synchronization interval is 30 seconds. For how to change it, see Changing the Synchronization Interval.

The following index configuration items cannot be modified: number_of_shards, version.created, uuid, creation_date, and soft_deletes.enabled.

Stopping Index Synchronization

Run the following command in the follower cluster to stop synchronization tasks for specified indexes. Subsequent changes to the indexes in the leader cluster will not be synchronized to the follower cluster. The read-only state of the indexes in the follower cluster will be cancelled, so that new data can be written into them.

PUT log*/stop_remote_sync

In this command, log* indicates the index name. You can specify multiple index names (separated by commas) or use a wildcard. In this example, synchronization tasks for all indexes that start with log are stopped.

Querying and Deleting Created Patterns

1. Run the following command in the follower cluster to query created patterns:
   • Query the list of patterns.

     GET auto_sync/pattern

   • Query a specified pattern by name.

     GET auto_sync/pattern/{PATTERN}

   The following is an example of the response:

   {
     "patterns" : [
       {
         "name" : "pattern1",
         "pattern" : {
           "remote_cluster" : "leader",
           "remote_index_patterns" : [
             "log*"
           ],
           "local_index_pattern" : "{{remote_index}}-sync",
           "settings" : { }
         }
       }
     ]
   }

2. Run the following command in the follower cluster to delete a specified pattern by name:

   DELETE auto_sync/pattern/{PATTERN}

Enabling Forcible Synchronization

By default, the plug-in determines whether to synchronize metadata based on whether the number of documents in the index of the leader cluster changes. If the leader cluster only updates documents and the number of documents remains unchanged, the plug-in does not synchronize the updates to the follower cluster. The configuration can be modified. After forcible synchronization is enabled, the index metadata of the leader cluster is forcibly synchronized to the follower cluster in each synchronization cycle.

The following is an example of enabling forcible synchronization:

PUT _cluster/settings
{
  "persistent": {
    "remote_sync.force_synchronize": true
  }
}

Changing the Synchronization Interval

The default synchronization interval between the leader and follower clusters is 30 seconds and can be changed.

The example request below changes the synchronization interval to 2 seconds:

PUT {index_name}/_settings
{
  "index.remote_sync.sync_interval": "2s"
}

Changing the Synchronization Speed

You can change the data synchronization speed between the leader and follower clusters by configuring cluster-level settings.

The following is an example request: set the block size to 2 MB, the number of blocks to 20, and the maximum synchronization traffic to 100 MB per second.

PUT _cluster/settings
{
  "persistent": {
    "remote_sync.chunk_size": "2MB",
    "remote_sync.max_concurrent_file_chunks": 20,
    "remote_sync.max_bytes_per_sec": "100MB"
  }
}

Querying Index Synchronization Status

Obtain the auto synchronization status of a specified index.

An example request is as follows:

GET {index_name}/sync_stats

The following is an example of the response:

{
  "indices" : {
    "data1_follower" : {
      "shards" : {
        "0" : [
          {
            "primary" : false,
            "total_synced_times" : 27,
            "total_empty_times" : 25,
            "total_synced_files" : 4,
            "total_synced_bytes" : 3580,
            "total_paused_nanos" : 0,
            "total_paused_times" : 0,
            "current" : {
              "files_count" : 0,
              "finished_files_count" : 0,
              "bytes" : 0,
              "finished_bytes" : 0
            }
          },
          {
            "primary" : true,
            "total_synced_times" : 28,
            "total_empty_times" : 26,
            "total_synced_files" : 20,
            "total_synced_bytes" : 17547,
            "total_paused_nanos" : 0,
            "total_paused_times" : 0,
            "current" : {
              "files_count" : 0,
              "finished_files_count" : 0,
              "bytes" : 0,
              "finished_bytes" : 0
            }
          }
        ]
      }
    }
  }
}

Switching the Roles of the Leader and Follower Clusters

When the leader cluster becomes faulty, perform a leader/follower switchover to have the follower cluster take over services. The steps are as follows:

1. Determine the index synchronization method between the leader and follower clusters. Check whether pattern-matching index synchronization policies have been configured in the follower cluster. For the command to use, see Querying and Deleting Created Patterns.
   • If there are no such policies, synchronization is performed for specified indexes between the leader and follower clusters. In this case, go to 3.
   • If there are such policies, index synchronization between the leader and follower clusters is based on index patterns. In this case, go to 2.
2. Delete pattern-matching index synchronization policies in the follower cluster. For the command to use, see Querying and Deleting Created Patterns.
3. Perform Stopping Index Synchronization in the follower cluster. Then redirect read and write traffic to it. If the leader and follower clusters synchronize indexes based on index patterns, use a wildcard to match indexes when running the command that stops index synchronization.
4. After the leader cluster recovers, configure information about the follower cluster in the leader cluster, and connect the leader and follower clusters again. For details, see Connecting the Leader and Follower Clusters.
5. Under the leader cluster, perform Index Synchronization to synchronize data from the follower cluster to the leader cluster, and then perform a leader/follower switchover to switch back.