Migrating Data Between Elasticsearch Clusters Using Elasticdump

Scenarios

Data migration between Elasticsearch clusters is a common task in enterprise data management. Elasticdump is an open-source tool capable of migrating data between Elasticsearch clusters of different versions. Below are typical scenarios where you might use Elasticdump for data migration.

• Cross-version migration: Use Elasticdump to seamlessly migrate data when upgrading an Elasticsearch cluster to a new version, ensuring data integrity and availability during and after the upgrade.
• Cluster merging: Use Elasticdump to merge indexes scattered across multiple Elasticsearch clusters into a single cluster for centralized data management and analysis.
• Cloud migration: Migrate a self-managed Elasticsearch service to the cloud to enjoy the benefits of cloud services, such as scalability, ease-of-maintenance, and cost-effectiveness.
• Changing the service provider: Switch from a third-party Elasticsearch service to Huawei Cloud for reasons related to cost, performance, or other strategic considerations.

Elasticdump is best suited for migrating small volumes of data and a limited number of indexes. For large-scale migrations, consider other more suitable methods. For details, see the official elasticsearch-dump documentation.

Solution Architecture

Figure 1 Migration procedure

Figure 1 shows how to migrate data between Elasticsearch clusters using Elasticdump.

1. Before you start, make sure the source and destination clusters are compatible and back up the source cluster's data to prevent data loss.
2. Install Elasticdump on a Linux ECS.
3. Run Elasticdump commands to migrate indexes from the source cluster to the destination cluster.

Advantages

• Cross-version data migration: You can use Elasticdump to migrate data between Elasticsearch clusters of different versions, including from an earlier version to a later version.
• Flexible migration solutions: Elasticdump can migrate index settings, mappings, and data separately, meeting custom migration needs.
• Free: As an open-source tool, the Elasticdump code is hosted on GitHub and accessible to all users for free.

Impact on Performance

Elasticdump reads data from source indexes through the Scroll API and writes it to indexes in the destination cluster. This process consumes resources on the source cluster, potentially impacting its performance. This is why you should perform the migration during off-peak hours.

Constraints

To ensure data consistency between the source and destination clusters, avoid any write operations (add, delete, or modify) on the source indexes while the migration is ongoing. Changes made after the migration begins will not be synchronized to the destination cluster.

Prerequisites

• The source and destination Elasticsearch clusters are both available.
• The network between the two clusters is connected.
  • If the source and destination clusters are in different VPCs, establish a VPC peering connection between them. For details, see VPC Peering Connection Overview.
  • To migrate an in-house built Elasticsearch cluster to Huawei Cloud, you need to enable public network access for this cluster.
  • To migrate a third-party Elasticsearch cluster to Huawei Cloud, you need to establish a VPN or Direct Connect connection between the third party's internal data center and Huawei Cloud.
• Ensure that _source has been enabled for indexes in the cluster.

  By default, _source is enabled. You can run the GET {index_name}/_search command to check whether it is enabled. If the returned index information contains _source, it is enabled.

Step 1: Obtaining Elasticsearch Cluster Information

Before data migration, obtain necessary information about the source and destination clusters for configuring a migration task.

The method of obtaining the cluster information varies depending on the source cluster. This section describes how to obtain information about a Huawei Cloud Elasticsearch cluster.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the destination cluster and obtain the cluster name and address.
   Figure 2 Obtaining cluster information
   

Step 2: Installing Elasticdump

1. Create an ECS for installing Elasticdump.
   1. Buy a Linux ECS, select the CentOS 8 image, set the VPC to that of the destination cluster, and enable public network access. For details, see Purchasing and Using a Linux ECS.
      

      If your environment has no network connectivity, you cannot enable public network access. Instead, install Elasticdump offline. (To do that, you need to download the Elasticdump tool and Node.js installation package and transfer them to the ECS in advance.) For details, see FAQ: How Do I Install Elasticdump Offline?.

   2. Test the connectivity between the ECS and the source and destination clusters.

      Run the following command on the ECS to test connectivity. If the correct cluster information is returned, the connection is ready.

      # For a cluster with the security mode disabled:
      curl http://[ip]:9200
      
      # For a security-mode cluster that uses HTTP:
      curl -ik http://[ip]:9200 -u [username]:[password]
      
      # For a security-mode cluster that uses HTTPS:
      curl -ik https://[ip]:9200 -u [username]:[password]

2. Install and configure Node.js on the ECS.
   1. Download the Node.js installation package.

      wget https://nodejs.org/dist/v20.10.0/node-v20.10.0-linux-x64.tar.gz

   2. Decompress the Node.js installation package.

      tar -xvf node-v20.10.0-linux-x64.tar.gz

   3. Import Node.js environment variables through the bash_profile file.

      vim ~/.bash_profile 

   4. Set environment variables. Run the vi command to open the bash_profile file and add the following line:

      export PATH=$PATH:/root/node-v20.10.0-linux-x64/bin/

   5. Run the source command to apply the change:

      source ~/.bash_profile

   6. Verify successful installation of Node.js.

      node -v

3. Install Elasticdump on the ECS.

   npm install elasticdump -g

4. Verify Elasticdump installation.

   elasticdump --help

Step 3: Migrating Data Using Elasticdump

1. Use Elasticdump to migrate data from a source cluster to a designated destination cluster.

   Elasticdump reads data from input and sends the data to output. Therefore, configure the information about the source Elasticsearch cluster at input, and the destination Elasticsearch cluster at output. Table 2 describes the parameter settings.

   • Migrate the settings of a specified index.

     elasticdump --input={protocol}://{host}:{port}/{index_name} --output={protocol}://{host}:{port}/{index_name} --type=settings
     
     # Example: Migrate data from a non-security mode cluster to a security-mode cluster that uses HTTPS.
     elasticdump --input=http://xx.xx.xx.xx:9200/my_index --output=https://username:password@xx.xx.xx.xx:9200/my_index --type=settings

   • Migrate the mapping of a specified index.

     elasticdump --input={protocol}://{host}:{port}/{index_name} --output={protocol}://{host}:{port}/{index_name} --type=mapping
     
     # Example: Migrate data from a non-security mode cluster to a security-mode cluster that uses HTTPS.
     elasticdump --input=http://xx.xx.xx.xx:9200/my_index --output=https://username:password@xx.xx.xx.xx:9200/my_index --type=mapping

   • Migrate the data of a specified index.

     elasticdump --input={protocol}://{host}:{port}/{index_name} --output={protocol}://{host}:{port}/{index_name} --type=data
     
     # Example: Migrate data from a non-security mode cluster to a security-mode cluster that uses HTTPS.
     elasticdump --input=http://xx.xx.xx.xx:9200/my_index --output=https://username:password@xx.xx.xx.xx:9200/my_index --type=data

   • Migrate data by specifying a search query as a filter.

     elasticdump --input={protocol}://{host}:{port}/{index_name} --output={protocol}://{host}:{port}/{index_name} --searchBody="QueryString"
     
     # Example: Migrate data from a non-security mode cluster to a security-mode cluster that uses HTTPS.
     elasticdump --input=http://xx.xx.xx.xx:9200/my_index --output=https://username:password@xx.xx.xx.xx:9200/my_index --searchBody="{\"query\":{\"term\":{\"username\": \"admin\"}}}"

   Table 2 Parameter description

   Parameter	Description
   protocol	Connection protocol. For a cluster with the security mode disabled, set this parameter to http. For a security-mode cluster that uses HTTP, set this parameter to http. For a security-mode cluster that uses HTTPS, set this parameter to https.
   host	Cluster access address. For a cluster with the security mode disabled, set this parameter to Cluster IP address, such as xx.xx.xx.xx. For a security-mode cluster, set this parameter to User name:password@Cluster IP address, such as username:password@xx.xx.xx.xx. If the cluster has multiple IP addresses, separate them using a comma (,), such as xx.xx.xx.xx,xx.xx.xx.xx. For details about how to obtain the cluster IP address, see Step 1: Obtaining Elasticsearch Cluster Information.
   port	Access port of the cluster. The default value is 9200.
   index_name	The name of the index to be migrated.
   QueryString	Search query that defines the data range for migration. A JSON string is supported. For example, {\"query\":{\"term\":{\"username\": \"admin\"}}} indicates that only documents whose username is admin will be migrated.

   For more information, see the official elasticsearch-dump documentation.

2. After the data migration is complete, run the following commands to obtain the number of documents in the source and destination clusters, respectively, and compare the results to verify data consistency.

   # For a cluster with the security mode disabled:
   curl http://[ip]:9200/[index_name]/_count 
     
   # For a security-mode cluster that uses HTTP:
   curl -ik -u [username]:[password] http://[ip]:9200/[index_name]/_count
   
   # For a security-mode cluster that uses HTTPS:
   curl -ik -u [username]:[password] https://[ip]:9200/[index_name]/_count

FAQ: How Do I Use Elasticdump to Export Data?

Elasticdump can be used to export data to a local directory.

• Export a specified index to a local JSON file.

  # Backup index data to a file
  elasticdump \
    --input={protocol}://{host}:{port}/{index_name} \
    --output=/{local_file}/my_index.json \
    --type=data

• Export a specified index to a local directory and compress it into a ZIP file.

  # Backup an index to a gzip using stdout
  elasticdump \
    --input={protocol}://{host}:{port}/{index_name} \
    --output=$ \
    | gzip > /{local_file}/my_index.json.gz

• Filter a specified index using a search query and export the results to a local file.

  # Backup the results of a query to a file
  elasticdump \
    --input={protocol}://{host}:{port}/{index_name} \
    --output=/{local_file}/query.json \
    --searchBody="QueryString"

FAQ: How Do I Disable Security Certificate Verification for an HTTPS-Enabled Cluster?

Elasticdump allows you to skip security certificate verification by setting the environment variable NODE_TLS_REJECT_UNAUTHORIZED=0.

# An alternative method of passing environment variables before execution
# NB : This only works with linux shells
NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --input={protocol}://{host}:{port}/{index_name} --output={protocol}://{host}:{port}/{index_name}

FAQ: How Do I Install Elasticdump Offline?

If Node.js is already installed, install Elasticdump offline by performing the following steps:

1. To install Elasticdump in an environment inaccessible from the public network, first prepare the Elasticdump installation package in a networked environment beforehand.

   1. Log in to an ECS that has public network access.
   2. Install and configure Node.js on the ECS. For details, see 2.
   3. Install Elasticdump.

      npm install elasticdump -g

   4. Install npm (Node Package Manager).

      npm install npm-pack-all -g

   5. Go to the node_modules directory. (Replace it with the path you actually use.)

      cd /root/node-v20.10.0-linux-x64/lib/node_modules/elasticdump/

   6. Create the Elasticdump installation package.

      npm-pack-all

   7. Run the ll command to check whether the Elasticdump installation package is generated in the current directory.

      If the result similar to that shown in Figure 3 is returned, the packaging is successful, as the elasticdump-6.x.x.tgz installation package has been generated. The name of the installation package may vary depending on the version.

      Figure 3 Elasticdump installation package
      
   8. Use SFTP to download the Elasticdump installation package.
2. Download the Node.js installation package in a networked environment.

   Download address: https://nodejs.org/dist/v20.10.0/node-v20.10.0-linux-x64.tar.gz.

3. Install Elasticdump in an environment without public network access.
   1. Upload the Elasticdump and Node.js installation packages to the root directory on your ECS.
   2. Install and configure Node.js on the ECS.
      1. Decompress the Node.js installation package.

         tar -xvf node-v20.10.0-linux-x64.tar.gz

      2. Import Node.js environment variables through the bash_profile file.

         vim ~/.bash_profile 

      3. Set environment variables. Run the vi command to open the bash_profile file and add the following line:

         export PATH=$PATH:/root/node-v20.10.0-linux-x64/bin/

      4. Run the source command to apply the change:

         source ~/.bash_profile

      5. Verify successful installation of Node.js.

         node -v

   3. Install Elasticdump.

      npm install elasticdump-6.x.x.tgz  -g

   4. Set a symbolic link for Elasticdump. (Replace it with the path you actually use.)

      ln -s /root/node-v20.10.0-linux-x64/lib/node_modules/elasticdump/bin/elasticdump  /usr/local/bin

4. Verify Elasticdump installation.

   elasticdump --help