Help Center/ Cloud Search Service/ Best Practices/ Elasticsearch Data Migration/ About Elasticsearch Cluster Migration Solutions
Updated on 2024-11-22 GMT+08:00

About Elasticsearch Cluster Migration Solutions

Table 1 Elasticsearch cluster migration solutions

Migration Scenario

Migration Tool/Method

Applicable Scenario

Constraints

Example

Migrating data between Huawei Cloud Elasticsearch clusters

Huawei Cloud Logstash

  • Migrate data from a Huawei Cloud Elasticsearch cluster of an earlier version to one of a later version.
  • Migrate data from multiple Huawei Cloud Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.

During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.

Migrating Data Between Elasticsearch Clusters Using Huawei Cloud Logstash

Backup and restoration

  • Migrate data between Huawei Cloud Elasticsearch clusters in the same region or across regions, and under the same or different accounts.
  • Migrate data from a Huawei Cloud Elasticsearch cluster of an earlier version to one of a later version.
  • Migrate data from multiple Huawei Cloud Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • The version of the destination cluster must not be earlier than that of the source cluster. For details, see Snapshot version compatibility.
  • The number of nodes in the destination cluster must be greater than half of that in the source cluster, and cannot be less than the number of shard replicas in the source cluster.
  • The CPU, memory, and disk capacities of the destination cluster must not be lower than those of the source cluster.

Migrating Data Between Huawei Cloud Elasticsearch Clusters Using Backup and Restoration

Read/write splitting plugin

  • Migrate data between Huawei Cloud Elasticsearch clusters in the same region or across regions, and under the same or different accounts.
  • Migrate data from multiple Huawei Cloud Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.

The versions of the source and destination clusters must both be 7.6.2 or 7.10.2.

Migrating Data Between Huawei Cloud Elasticsearch Clusters Using the Read/Write Splitting Plugin

Reindex API

Migrate data from multiple Huawei Cloud Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.

During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.

Migrating Data Between Elasticsearch Clusters Using the Reindex API

ESM

  • Migrate data from a Huawei Cloud Elasticsearch cluster of an earlier version to one of a later version.
  • Migrate data from multiple Huawei Cloud Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.

During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.

Migrating Data Between Elasticsearch Clusters Using ESM

Migrating data from an in-house built Elasticsearch cluster to Huawei Cloud

Huawei Cloud Logstash

  • Migrate data from an on-premises Elasticsearch cluster of an earlier version to a Huawei Cloud Elasticsearch cluster of a later version.
  • Migrate data from multiple on-premises Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • Migrate in-house built Elasticsearch workloads to Huawei Cloud.
  • During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.
  • Ensure that the network between the clusters is connected.

    Configure public network connectivity for in-house built Elasticsearch cluster.

Migrating Data Between Elasticsearch Clusters Using Huawei Cloud Logstash

Backup and restoration

  • Migrate data from an on-premises Elasticsearch cluster of an earlier version to a Huawei Cloud Elasticsearch cluster of a later version.
  • Migrate data from multiple on-premises Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • Migrate in-house built Elasticsearch workloads to Huawei Cloud.
  • The version of the destination cluster must not be earlier than that of the source cluster. For details, see Snapshot version compatibility.
  • This method does not support incremental data synchronization. You need to pause data update before starting to back up data.
  • An on-premises Elasticsearch cluster needs public network access to back up snapshots to OBS.

Migrating Data from an On-premises Elasticsearch Cluster to Huawei Cloud Using the S3 Plugin

Reindex API

  • Migrate data from multiple on-premises Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • Migrate in-house built Elasticsearch workloads to Huawei Cloud.

During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.

Migrating Data Between Elasticsearch Clusters Using the Reindex API

ESM

  • Migrate data from an on-premises Elasticsearch cluster of an earlier version to a Huawei Cloud Elasticsearch cluster of a later version.
  • Migrate data from multiple on-premises Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • Migrate in-house built Elasticsearch workloads to Huawei Cloud.

During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.

Migrating Data Between Elasticsearch Clusters Using ESM

CDM

This is a cloud migration tool provided by Huawei Cloud to migrate data between clusters that belong to different cloud services.

  • A VPN or Direct Connect connection needs to be established between the customer's IDC and Huawei Cloud.
  • During cluster migration, do not delete any of the source cluster's indexes, or something might go wrong.

Migrating an Entire Elasticsearch Database to CSS

Migrating data from a third-party Elasticsearch cluster to Huawei Cloud

Huawei Cloud Logstash

  • Migrate Elasticsearch clusters across major versions, for example, from 6.X to 7.X.
  • Migrate data from multiple third-party Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • Migrate third-party Elasticsearch workloads to Huawei Cloud.
  • During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.
  • Ensure that the network between the clusters is connected.

    A VPN or Direct Connect connection needs to be established between the customer's IDC and Huawei Cloud.

Migrating Data Between Elasticsearch Clusters Using Huawei Cloud Logstash

Backup and restoration

  • Migrate data from a third-party Elasticsearch cluster of an earlier version to a Huawei Cloud Elasticsearch cluster of a later version.
  • Migrate data from multiple third-party Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • Migrate third-party Elasticsearch workloads to Huawei Cloud.
  • The version of the destination cluster must not be earlier than that of the source cluster. For details, see Snapshot version compatibility.
  • This method does not support incremental data synchronization. You need to pause data update before starting to back up data.
  • Snapshots can be migrated only when public access is enabled for third-party storage repositories.

Migrating Data from a Third-Party Elasticsearch Cluster to Huawei Cloud Using Backup and Restoration

Reindex API

  • Migrate data from multiple third-party Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • Migrate third-party Elasticsearch workloads to Huawei Cloud.

During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.

Migrating Data Between Elasticsearch Clusters Using the Reindex API

ESM

  • Migrate data from a third-party Elasticsearch cluster of an earlier version to a Huawei Cloud Elasticsearch cluster of a later version.
  • Migrate data from multiple third-party Elasticsearch clusters to a single Huawei Cloud Elasticsearch cluster to form a unified data platform.
  • Migrate third-party Elasticsearch workloads to Huawei Cloud.

During cluster migration, do not add, delete, or modify indexes in the source cluster, or the source and destination clusters will have inconsistent data after the migration.

Migrating Data Between Elasticsearch Clusters Using ESM

CDM

This is a cloud migration tool provided by Huawei Cloud to migrate data between clusters that belong to different cloud services.

  • A VPN or Direct Connect connection needs to be established between the customer's IDC and Huawei Cloud.
  • During cluster migration, do not delete any of the source cluster's indexes, or something might go wrong.

Migrating an Entire Elasticsearch Database to CSS

Migrating data from a Huawei Cloud Elasticsearch Cluster to an OpenSearch cluster

Cross-engine upgrade

Upgrade an Elasticsearch 7.10.2 cluster to an OpenSearch 1.3.6 cluster in CSS.

  • You are advised to perform the upgrade during off-peak hours.
  • Make sure the Elasticsearch cluster does not have ongoing tasks during the upgrade.

Upgrading the Elasticsearch Cluster Version