Migrating an Elasticsearch Cluster Across Accounts or Regions Using Snapshot Backup and Restore

Scenarios

When Elasticsearch cluster data needs to be migrated across different accounts or regions, such as resource migration after enterprise organizational changes or data backup for disaster recovery purposes in the case of cross-region deployment, directly copying data is time-consuming and can disrupt ongoing services. By combining CSS snapshot backup and restore capabilities with Object Storage Service (OBS) and Object Storage Migration Service (OMS), you can efficiently migrate Elasticsearch cluster data across accounts or regions.

Typical scenarios include:

• Cross-region migration: Migrates data from an Elasticsearch cluster in region A to another Elasticsearch cluster in region B to support cross-region service deployment, disaster recovery backup, and nearby access.
• Cross-account migration: Migrates data from an Elasticsearch cluster under account A to another Elasticsearch cluster under account B. This is useful for scenarios such as enterprise organizational restructuring, resource transfer between projects, and centralized multi-account management.
• Cross-version migration: Migrates data from an earlier-version Elasticsearch cluster to a later-version cluster.
• Cluster merging: Merges index data from multiple Elasticsearch clusters into a single cluster for centralized data management.
• Disaster recovery: Backs up Elasticsearch cluster data from a production environment to a remote location using snapshots, enabling quick recovery when a disaster occurs.

Solution Architecture

Figure 1 Migration procedure

The migration procedure is as follows:

1. Enable snapshots for the source cluster: Enable snapshots for the source Elasticsearch cluster and configure an OBS bucket for storing snapshots.
2. Create a manual snapshot: Create a manual snapshot of the source Elasticsearch cluster to back up index data to the source OBS bucket.
3. Migrate OBS data: Migrate snapshot data from the source OBS bucket to the destination OBS bucket using OBS cross-region replication rules or OMS.
   • Same-account cross-region replication: Use cross-region replication rules to replicate snapshot data from the source OBS bucket in region A to the destination OBS bucket in region B under the same account.
   • Cross-account cross-region replication: Use OMS to migrate snapshot data from the source OBS bucket in region A under account A to the destination OBS bucket in region B under account B.
   • Same-account same-region replication: Use OMS to migrate snapshot data from the source OBS bucket in region A under account A to the destination OBS bucket also in region A under account A.
   • Cross-account same-region replication: Use OMS to migrate snapshot data from the source OBS bucket in region A under account A to the destination OBS bucket in region A under account B.
4. Configure a snapshot repository in the destination cluster (account B in region B) and associate it with the destination OBS bucket above.
5. Restore the snapshots stored in the destination OBS bucket to the destination Elasticsearch cluster.

Advantages

• Simplified operations and management: Use the CSS management console to back up and restore data using snapshots. No coding is required.
• Suitable for large-scale data migration: Snapshots are well suited for migrating large volumes of data, ranging from GBs to PBs.
• Supports cross-region and cross-account migration: Combined with the cross-region, cross-account data migration capabilities of OMS, cluster data can be migrated across different regions and accounts.
• Controllable restore process: During data restoration, you can choose to restore specific or all indexes. Indexes can be renamed to avoid index name conflicts.
• Controllable migration duration: Migration duration can be estimated using an established formula. You can control the migration speed by configuring the maximum backup and restore rates, reducing the impact on services.

Performance Impact

• Impact on the source cluster: Snapshots back up index files directly from the storage layer instead of relying on Elasticsearch read/write APIs. For latency-insensitive workloads, the impact on cluster performance is negligible.
• Impact on the destination cluster: Restoring a snapshot consumes CPU, memory, and disk I/O resources in the destination cluster. You are advised to perform restoration during off-peak hours or set Maximum Recovery Rate (/s) to limit the restore rate and reduce the impact on services.

Migration Duration Evaluation

The migration process consists of three phases: backup, OMS data migration, and restoration. The duration of the backup phase depends on the source cluster, the duration of the restore phase depends on the destination cluster, and the duration of OMS data migration depends on the data volume and network bandwidth.

The formula for calculating the total migration duration is as follows:

• When the number of index shards is greater than the number of nodes:

  Total duration (s) = Data volume to be migrated (GB) ÷ 40 MB ÷ (Number of nodes in the source cluster + Number of nodes in the destination cluster) x Number of indexes + Data volume to be migrated (GB) ÷ OMS migration rate (MB/s)

• When the number of index shards is smaller than the number of nodes:

  Total duration (s) = Data volume to be migrated (GB) ÷ 40 MB ÷ (Number of index shards in the source cluster + Number of index shards in the destination cluster) x Number of indexes + Data volume to be migrated (GB) ÷ OMS migration rate (MB/s)

The preceding formulas estimate migration duration under ideal conditions, assuming each node transfers data at the maximum rate of 40 MB/s. The actual duration also depends on factors such as network conditions, OBS performance, and OMS migration rate. The OMS migration rate can range from tens of MB/s to hundreds of MB/s. Rate limiting is supported, allowing you to limit the migration rate during peak hours.

Constraints

• The destination cluster must be running the same or a later version than the source cluster. For version compatibility, see Snapshot version compatibility.
• The number of nodes in the destination cluster should be greater than the maximum number of replicas per shard in the source cluster. This ensures that both primary and replica shards can be properly allocated. Otherwise, the cluster health status will turn Yellow.
• The number of nodes in the destination cluster should be greater than half the number in the source cluster. Otherwise, individual node CPU and I/O load in the destination cluster may become significantly higher than in the source cluster.
• The CPU, memory, and disk capacities of the destination cluster should be greater than or at least equal to those of the source cluster. This helps prevent performance degradation, such as service instability or slow responses, after the migration.
• Snapshot backup and restore does not support incremental synchronization of data changes. For data consistency, perform the snapshot after data updates have stopped or during off-peak hours.
• For cross-region migration, the source and destination OBS buckets are located in different regions. In this case, snapshot data must be migrated using OMS. When configuring the OMS migration task, set Object Metadata to Migrate. Otherwise, snapshot restoration may fail.
• For cross-account migration, ensure that the destination account has permission to access the destination OBS bucket and that the bucket is located in the same region as the destination cluster.
• For both the source and destination OBS buckets, the Storage Class must be Standard.
• You should not back up and restore system indexes whose name starts with a period (.), as doing so may cause errors.

Resource and Cost Planning

Step 1: Create a Manual Snapshot of the Source Cluster

Enable snapshots for the source Elasticsearch cluster, configure an OBS bucket, and create a manual snapshot to back up index data to the source OBS bucket.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
4. Click the Cluster Snapshots tab, and check whether cluster snapshots are enabled.
   • If snapshots are enabled, go to 7.
   • If snapshots are not enabled, continue with 5.
5. On the Cluster Snapshots tab, click Enable Snapshot. In the displayed dialog box, configure the necessary settings.
   Figure 2 Enable Snapshot
   

   Table 2 Enabling snapshots

   Parameter	Description
   OBS Bucket	Select the OBS bucket used to store snapshot data.
   Backup Path	Snapshot storage path in the OBS bucket, for example, css_repository/es-source.
   Maximum Backup Rate (/s)	Maximum backup rate per node. Keep the default value or adjust it based on performance requirements.
   Maximum Recovery Rate (/s)	Maximum restore rate per node. Keep the default value or adjust it based on service requirements.
   IAM Agency	Select an IAM agency to grant the current account the permission to access and use OBS. You can use the automatically created agency css_obs_agency or specify a custom agency.
   Automatic Snapshot Creation	Keep it unselected. Automatic snapshot creation is not required for cluster migration.

6. Click OK to enable cluster snapshots.

   The configuration information will be displayed on the Cluster Snapshots tab.

7. Under Cluster Snapshot Tasks, click Manually Create Snapshot. In the displayed dialog box, configure the snapshot policy.
   Figure 3 Manually creating a snapshot
   

   Table 3 Parameters for manually creating a snapshot

   Parameter	Description
   Snapshot Name	Set the snapshot name. You can retain the default value or use a custom value. Example: snapshot-source.
   Index	Specifies the indexes to be backed up. To specify multiple indexes, use commas (,) to separate them, for example, index1,index2,index3. You can use an asterisk (*) to match multiple indexes. For example, index* indicates that all indexes with the prefix index will be backed up. If you do not specify this parameter, all indexes in the cluster will be backed up by default.
   Snapshot Description	Add a snapshot description.

8. Click OK to start creating snapshots.

   When the snapshot task status changes to Available, the snapshot is created.

9. Check whether data is successfully backed up.

   In the cluster snapshot task list, click in front of the snapshot name to view the snapshot details. Check the shards and indexes that have been backed up to see if the backup is successful.

   Figure 4 Viewing snapshot details
   

Step 2: Migrate Snapshot Data Between OBS Buckets

Migrate snapshot data from the source OBS bucket to the destination OBS bucket.

• Same-account cross-region replication: Use cross-region replication rules to replicate snapshot data from the source OBS bucket in region A to the destination OBS bucket in region B under the same account.
• Cross-account cross-region replication: Use OMS to migrate snapshot data from the source OBS bucket in region A under account A to the destination OBS bucket in region B under account B.
• Same-account same-region replication: Use OMS to migrate snapshot data from the source OBS bucket in region A under account A to the destination OBS bucket also in region A under account A.
• Cross-account same-region replication: Use OMS to migrate snapshot data from the source OBS bucket in region A under account A to the destination OBS bucket in region A under account B.

For details, see OBS Data Replication.

When you use OMS to migrate snapshot data between OBS buckets, set Object Metadata to Migrate in the migration task. Otherwise, snapshot restoration may fail.

Wait until the OMS migration task is complete and verify that the migration is successful.

Step 3: Configure a Snapshot Repository in the Destination Cluster

In Kibana, create a snapshot repository in the destination Elasticsearch cluster and associate it with the destination OBS bucket so that the destination cluster can access the migrated snapshot data.

1. Log in to Kibana and go to the command execution page.
   1. Log in to the CSS management console.
   2. In the navigation pane on the left, choose Clusters > Elasticsearch.
   3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
   4. In the left navigation pane, choose Dev Tools.

      The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

2. Run the following command to create a snapshot repository, for example, my_backup_all, and associate it with the destination OBS bucket.

   PUT _snapshot/my_backup_all/
   {
     "type" : "obs",
     "settings" : {
       "endpoint" : "obs.<destination_region>.example.com",
       "region" : "<destination_region>",
       "access_key": "<destination_account_AK>",
       "secret_key": "<destination_account_SK>",
       "bucket" : "<destination_OBS_bucket_name>",
       "compress" : "false",
       "chunk_size" : "1g",
       "base_path" : "<backup_path>",
       "max_restore_bytes_per_sec": "100mb",
       "max_snapshot_bytes_per_sec": "100mb"
     }
   }

   Table 4 Snapshot repository parameters

   Parameter	Description
   endpoint	OBS endpoint in the format obs.<destination_region>.example.com.
   region	Region of the destination cluster.
   access_key	AK of the destination account. For details about how to obtain the AK, see How Do I Obtain an Access Key (AK/SK)?.
   secret_key	SK of the destination account. For details about how to obtain the AK, see How Do I Obtain an Access Key (AK/SK)?.
   bucket	Name of the destination OBS bucket used to store snapshots.
   base_path	Snapshot storage path in the OBS bucket. This value must be the same as that used by the source cluster. For example, if the source cluster uses css_repository/es-source, specify the same path here.
   max_restore_bytes_per_sec	Maximum restore rate per node. Adjust this value based on service requirements.
   max_snapshot_bytes_per_sec	Maximum backup rate per node. Adjust this value based on service requirements.

3. Run the following command to verify that the snapshot repository has been created successfully.

   GET _snapshot 

   If the response contains the my_backup_all repository, the repository has been created successfully.

4. Run the following command to view all snapshots in the repository and verify that the destination cluster has access to the snapshot data in the destination OBS bucket.

   GET _snapshot/my_backup_all/_all 

   If the response contains the snapshot snapshot-source created in Step 1: Create a Manual Snapshot of the Source Cluster, the destination cluster can access the migrated snapshot.

Step 4: Restore the Snapshot to the Destination Cluster

Restore the snapshot to the destination Elasticsearch cluster.

1. Choose one of the following restore methods as needed.
   • Restore all indexes in a snapshot:

     POST _snapshot/my_backup_all/<snapshot_name>/_restore?wait_for_completion=true

   • Restore selected indexes (excluding system indexes) in a snapshot:

     POST _snapshot/my_backup_all/<snapshot_name>/_restore
     {
       "indices": "*,-.monitoring*,-.security*,-.kibana*",
       "ignore_unavailable": "true"
     }

   • Restore selected indexes and rename them:

     POST _snapshot/my_backup_all/<snapshot_name>/_restore
     {
       "indices": "index_1,index_2",
       "rename_pattern": "index_(.+)",
       "rename_replacement": "restored_index_$1"
     }

2. View the progress of snapshot restoration.

   # View the restore status of all indexes:
   GET /_recovery
   
   # View the restore status of a specified index:
   GET <index_name>/_recovery

Step 5: Verify Data Integrity

Verify that the data in the destination cluster is consistent with that in the source cluster.

1. View the restored indexes in the destination cluster.

   GET _cat/indices?v 

2. Compare the document counts between the source and destination clusters.

   # Source cluster
   GET _cat/count/<index_name>?v
   
   # Destination cluster
   GET _cat/count/<index_name>?v

3. Perform sample queries on critical business data to verify that query results are consistent between the source and destination clusters.

Step 6: Clean Up Resources (Optional)

To reduce costs, after the migration is complete, delete resources that are no longer needed.

1. Delete snapshot data from the OBS buckets to reclaim storage space.
2. Delete the source Elasticsearch cluster to release compute and storage resources.

Related Documents

• Creating Snapshots to Back Up Data: Learn how to use snapshots in CSS.
• OBS Data Replication: Learn how to replicate data to OBS.
• Migrating Data Between Huawei Cloud OBS Buckets: Learn how to use OMS to migrate data between OBS buckets across accounts, across regions, or within the same region.