Migrating Elasticsearch Data Using Snapshots

Scenario

HDFS is used as the snapshot repository to migrate Elasticsearch data from the source cluster to the target cluster.

To improve the migration speed, you are advised to use the normal mode for the cluster where the HDFS service is located.

Data of open-source Elasticsearch 6.8.0 or later can be migrated to the cluster of the current version.

FusionInsight 6.5.1 can be migrated to the cluster of the current version.

Prerequisites

The HDFS component has been installed. If the HDFS component and the Elasticsearch cluster are in security mode and are not in the same cluster, you need to configure cross-cluster mutual trust.
The Elasticsearch services of the source and target clusters are running properly.

HDFS can be installed in the same cluster with the source or target Elasticsearch. It can also be installed in another cluster.

Procedure

Check whether the source and target Elasticsearch services are in security mode.

On Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the Elasticsearch overview page, click Configurations. Search for the ELASTICSEARCH_SECURITY_ENABLE parameter in the upper right corner of the configuration page, check whether the parameter is searchable, and if it is, check whether the value is true.
- If yes, go to 2.
- If no, go to 3.
Configure using of the curl command in security mode. For details, see "Environment Preparations" in Running curl Commands in Linux.
Log in to any EsNode where the Elasticsearch client has been installed as user root.
On Manager of the HDFS cluster, choose Cluster > Name of the desired cluster > Cluster Properties. On the Cluster Properties page that is displayed, check whether the authentication mode is security mode.
- If yes, go to 5.
- If no, go to 9.
Create a machine-machine user for accessing the HDFS service.

On the Manager of HDFS, choose System > Permission > User > Create. Enter a username, for example, test, set User Type to Machine-Machine, add the user to user group supergroup, set Primary Group to supergroup.
Prepare the security authentication file.

On Manager of the HDFS component, choose System > Permission > User. Select the machine-machine user test from the username drop-down list, choose More > Download Authentication Credential, save the file, decompress it to obtain the user.keytab file, and rename the file krb5.keytab.
Use WinSCP to log in to the active OMS node of the source cluster and the target cluster as user omm and upload the krb5.keytab file to the specified directory. For example, /home/omm.
Configure the elasticsearch.install.path/cluster/Instance directory/config/repository-hdfs path for each instance of the source cluster and target Elasticsearch service to upload the krb5.keytab file.

To upload the krb5.keytab file to the instance directory, run the following commands:

./clusterscp.sh put /home/omm/krb5.keytab elasticsearch.install.path/cluster/EsMaster/config/repository-hdfs

./clusterscp.sh put /home/omm/krb5.keytab elasticsearch.install.path/cluster/EsNode1/config/repository-hdfs
- elasticsearch.install.path indicates the Elasticsearch component installation directory.
- If the EsNode2-9 instance is deployed, run the preceding command for all instances.
Prepare the HDFS client configuration files.
1. On Manager of the HDFS component, choose Cluster > Name of the desired cluster > Services > HDFS. In the upper right corner, choose More > Download Client, set Select Client Type to Configuration Files Only, and click OK to download the configuration file package.
2. Decompress the configuration file package to obtain the following configuration files for 10.
  core-site.xml
  
  hdfs-site.xml

Log in to the EsNode where the Elasticsearch client of the source cluster is located and run the curl command to create a snapshot repository.

curl -XPUT --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
{
"type": "hdfs",
"settings": {
"uri": "hdfs://hacluster/",        //core-site.xml fs.defaultFS
"path": "/test/backup",          //Specifies the HDFS storage path, which does not need to be created in advance.
"security.principal": "test@HADOOP.COM",    //Specifies the machine-machine user of the HDFS component. Set this parameter when the HDFS cluster is in security mode.
"hadoop.security.authentication":"kerberos",    //The authentication mode must be set to Kerberos. Set this parameter when the HDFS cluster is in security mode.
"load_defaults": true,
"concurrent_streams": 5,
"compress": "true",
"conf.hadoop.rpc.protection":"privacy",
"conf.dfs.nameservices":"hacluster",       //Set this parameter based on the value of dfs.nameservices in hdfs-site.xml.
"conf.dfs.ha.namenodes.hacluster": "19,20",//For details, see configurations of dfs.ha.namenodes.hacluster in hdfs-site.xml.
"conf.dfs.namenode.rpc-address.hacluster.19":"192.168.168.11:8020",//Set this parameter based on the dfs.namenode.rpc-address.hacluster.19 in hdfs-site.xml for configuration and replaces the host name with the corresponding IP address.
"conf.dfs.namenode.rpc-address.hacluster.20":"192.168.168.22:8020",//Set this parameter based on the dfs.namenode.rpc-address.hacluster.20 in hdfs-site.xml for configuration and replaces the host name with the corresponding IP address.
"conf.dfs.client.failover.proxy.provider.hacluster":"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"max_snapshot_bytes_per_sec":"500mb",      //Specifies the snapshot rate limit of each node. The default value is 40 MB/s.
"max_restore_bytes_per_sec":"500mb"        //Specifies the throttle recovery rate of each node. The default value is 40 MB/s.
}
}'

The following result is displayed:

{"acknowledged":true}

The values of conf.dfs.namenode.rpc-address.hacluster.19 and conf.dfs.namenode.rpc-address.hacluster.20 parameters must be set based on the actual value of dfs.ha.namenodes.hacluster.
If the HDFS cluster is in non-security mode, delete the security.principal and hadoop.security.authentication parameters.
--tlsv1.2 indicates that the TLSv1.2 protocol is used. This parameter can be omitted when you run the curl command. If the system displays a message indicating that TLSv1.2 is unavailable when this parameter is used, the curl client may not support TLSv1.2. You can run the curl --help command to query the TLS version. For example, if the TLSv1 protocol is used, set this parameter to --tlsv1.

After the snapshot repository is created, you can run the following command to query the repository information:

curl -XGET --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/_snapshot/my_backup?pretty"

Create a snapshot. A repository can contain multiple snapshots of the same cluster. The snapshot name must be unique in the repository.
1. Create a snapshot and back up data.
  - Run the following command to create a snapshot named snapshot_1 in the my_backup repository and back up the data of the specified indexes:
    curl -XPUT --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/_snapshot/my_backup/snapshot_1?wait_for_completion=true&pretty" -H 'Content-Type: application/json' -d' { "indices": "index_1,index_2" }'
    
    The following result is displayed:
```
{
  "snapshot" : {
    "snapshot" : "snapshot_1",
    "uuid" : "_02NT1ZJTLCrSJ5uJwqOsA",
    "version_id" : 6070199,
    "version" : "7.10.2",
    "indices" : [
      "index_1",
      "index_2"
    ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2020-03-10T07:32:21.458Z",
    "start_time_in_millis" : 1583825541458,
    "end_time" : "2020-03-10T07:32:25.283Z",
    "end_time_in_millis" : 1583825545283,
    "duration_in_millis" : 3825,
    "failures" : [ ],
    "shards" : {
      "total" : 20,
      "failed" : 0,
      "successful" : 20
    }
  }
}
```
  wait_for_completion: indicates whether the request needs to wait for the snapshot creation to complete. By default, the command output is returned immediately, and you do not need to wait for the snapshot creation to complete.
  - Run the following command to create a snapshot named snapshot_1 in the my_backup repository and back up the data of all open indexes:
    curl -XPUT --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/_snapshot/my_backup/snapshot_1?wait_for_completion=true&pretty"
2. After the command is executed successfully, run the following command to view the backup data:
  curl -XGET --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/_snapshot/my_backup/snapshot_1?pretty"
  
  Run the following command to view all the backup data:
  
  curl -XGET --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/_snapshot/my_backup/_all?pretty"
Create a snapshot repository in the target cluster Elasticsearch. For details, see 10.
Close the existing indexes to be restored in the target Elasticsearch cluster.

For example, if index_1 already exists in the target cluster and is enabled, run the following command to close it first.

curl -XPOST --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/index_1/_close?pretty"

The following result is displayed:
```
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "indices" : {
    "index_1" : {
      "closed" : true
    }
  }
}
```
Restore the snapshot of the target Elasticsearch cluster to restore the backed-up indexes. Run the following command:

curl -XPOST --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/_snapshot/my_backup/snapshot_1/_restore"

The following result is displayed:
```
{
  "snapshot" : {
    "snapshot" : "snapshot_1",
    "indices" : [
      "index_1",
      "index_2"
    ],
    "shards" : {
      "total" : 20,
      "failed" : 0,
      "successful" : 20
    }
  }
}
```
Repeat 11 and 14 until all index data in the source cluster is migrated to the target cluster.
Check whether Elasticsearch in the target cluster is in security mode.
- If yes, go to 17.
- If no, go to 18.
Log in to FusionInsight Manager of the target cluster, choose Cluster > Name of the desired cluster > Services > Elasticsearch > More, and check whether Enable Ranger is selected.
- If yes, create a Ranger policy for the target Elasticsearch cluster by referring to the Ranger policy of the source Elasticsearch cluster. For details about how to create a Ranger policy, see Adding a Ranger Access Permission Policy for Elasticsearch.
- If no, manually restore the index permission information. For details, see Synchronizing Index Owner Group.
Run the following command to delete the snapshot after the Elasticsearch data migration is complete:

curl -XDELETE --tlsv1.2 --negotiate -k -u : "https://10.131.112.121:24100/_snapshot/my_backup/snapshot_1?pretty"
- The snapshot and restoration mechanism allows only one snapshot and restoration task to be executed at the same time point.
- You can delete a running snapshot. After you perform the delete operation, the snapshot is stopped and the previous snapshot data is deleted from the repository.