Migrating a Cluster Using Backup and Restoration

Overview

Data can be migrated between CSS Elasticsearch clusters by backing up and restoring cluster snapshots.

Application scenarios:

• Cluster upgrade: Migrate data from a cluster of an earlier version to a cluster of a later version.
• Cluster merge: Merge the index data of two clusters.

This section describes how to take a snapshot of a cluster and restore it to another cluster. Take Elasticsearch cluster Es-1 and Es-2 as an example.

Migration Duration

The number of nodes or index shards in the source and destination clusters determines how long the data migration will take. Data migration consists of two phases: data backup and restoration. The backup duration is determined by the source cluster and the restoration duration is determined by the destination cluster. The formula for calculating the total migration duration is as follows:

• If the number of index shards is greater than the number of nodes:

  

• If the number of index shards is smaller than the number of nodes:

  

The migration duration estimated using the formula is the minimal duration possible (if each node transmits data at the fastest speed, 40 MB/s). The actual duration also depends on factors such as the network and resources condition.

Prerequisites

• The destination cluster (Es-2) and source cluster (Es-1) are available. You are advised to migrate a cluster during off-peak hours.
• Ensure that the destination cluster (Es-2) and source cluster (Es-1) are in the same region.
• Ensure that the version of the destination cluster (Es-2) is later than or same as that of the source cluster (Es-1).
• Ensure that the number of nodes in the destination cluster (Es-2) is greater than half of the number of nodes in the source cluster (Es-1).
• Ensure that the number of nodes in the destination cluster (Es-2) is greater than or equal to the number of shards in the source cluster (Es-1).
• Ensure that the CPU, memory, and disk configurations of the target cluster (Es-2) are greater than or equal to those of the source cluster (Es-1).

Procedure

1. Log in to the Cloud Search Service management console.
2. Choose Clusters > Elasticsearch. On the displayed page, click the source cluster name Es-1 to go to the basic information page.
3. In the navigation pane, choose Cluster Snapshots, and set basic snapshot configurations.

   Table 1 Basic configurations for a cluster snapshot

   Parameter	Description
   OBS Bucket	Select an OBS bucket for storing cluster snapshots.
   Backup Path	Storage path of the cluster snapshot in the OBS bucket. You can retain the default value.
   IAM Agency	Select an IAM agency to authorize CSS to access or maintain data stored in OBS. The IAM agency must have the OBS Administrator permission for project OBS in region Global service.

4. Click Create. In the dialog box that is displayed, configure the parameters and click OK to manually create a snapshot.

   Table 2 Snapshot creation parameters

   Parameter	Description
   Snapshot Name	User-defined snapshot name. You can retain the default value.
   Index	Enter the name of the index to be backed up. Use commas (,) to separate multiple indexes. Uppercase letters, spaces, and the following special characters are not allowed: "\<|>/? If you do not specify this parameter, data of all indexes in the cluster is backed up by default. You can use the asterisk (*) to back up data of certain indexes. For example, if you enter index*, then data of indexes with the name prefix of index will be backed up.
   Description	Snapshot description.

   In the snapshot management list, if the snapshot status is Available, the snapshot has been created.

5. In the snapshot management list, click Restore in the Operation column of the snapshot and configure restoration parameters to restore data to destination cluster Es-2.

   Table 3 Snapshot restoration parameters

   Parameter	Description
   Index	Enter the name of the index to be restored. If this parameter is not specified, all index data will be restored. You can use the asterisk (*) to match multiple indexes. For example, index* indicates that all indexes with the prefix index in snapshots are restored.
   Rename Pattern	Index name matching rule. The Rename Pattern and Rename Replacement take effect only when they are configured at the same time. You can configure them to rename matched indexes in snapshots.
   Rename Replacement	Rule for renaming an index name. The Rename Pattern and Rename Replacement take effect only when they are configured at the same time. The default value restored_index_$1 indicates that restored_ is added in front of the names of all restored indexes.
   Cluster	Select a destination cluster, for example, Es-2. NOTICE: If the source and destination clusters have indexes with the same names, the indexes in the destination cluster will be overwritten by those in the source cluster after the restoration.

   In the snapshot management list, if Task Status changes to Restoration succeeded, data in source cluster Es-1 is successfully migrated to destination cluster Es-2.