Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
Help Center/ Cloud Search Service/ Best Practices/ Elasticsearch Data Migration/ Migrating Data Between Huawei Cloud Elasticsearch Clusters Using Backup and Restoration

Migrating Data Between Huawei Cloud Elasticsearch Clusters Using Backup and Restoration

Updated on 2025-01-06 GMT+08:00

Data can be migrated between CSS Elasticsearch clusters by backing up and restoring cluster snapshots.

Scenario

Data migration between Huawei Cloud Elasticsearch clusters via backup and restoration is applicable solely to scenarios where both the source and destination clusters are CSS clusters and rely on OBS. Typical application scenarios include:

  • Cross-region or cross-account migration: Migrate the data of an Elasticsearch cluster in another region or under another account to the current cluster.
  • Cross-version migration: Migrate data from a self-built Elasticsearch cluster of an earlier version to a cluster of a later version.
  • Cluster merge: Merge the index data of two Elasticsearch clusters.

Overview

Figure 1 Process

Figure 1 shows the process of migrating data between Huawei Cloud Elasticsearch clusters using backup and restoration.

  1. Create a snapshot for the source Elasticsearch cluster and store the snapshot in an OBS bucket.
  2. Restore the cluster to a destination cluster using the snapshot store in the OBS bucket.

Advantages

  • Easy operation and management: The cluster snapshot function on the CSS console allows for simple, easy-to-manage, and automatic data backup and restoration.
  • Applicable to large-scale data migration: Snapshot backup is suitable for scenarios involving large amounts of data, especially when the data volume reaches GB, TB, or even PB levels.
  • Cross-region and cross-account migration: With the cross-region replication function of OBS, data can be migrated across different regions and accounts.
  • Controllable restoration process: During data restoration, you can restore specific indexes or all indexes and specify the cluster status to be restored.
  • Controllable migration duration: The data migration rate can be configured based on the migration duration evaluation formula. Ideally, the data migration rate matches the file replication rate.

Impact on Performance

The essence of migrating data between clusters using backup and restoration is copying files at the data storage layer to back up data. This solution does not rely on any external Elasticsearch APIs. Hence it significantly reduces any impact on the performance of the source cluster. For clusters that are not particularly latency-sensitive, the performance impact of this method can be ignored.

Constraints

  • The version of the destination cluster must not be earlier than that of the source cluster. For details, see Snapshot version compatibility.
  • The number of nodes in the destination cluster must be greater than half of that in the source cluster, and cannot be less than the number of shard replicas in the source cluster.
  • The CPU, memory, and disk capacities of the destination cluster must not be lower than those of the source cluster.

Migration Duration

The number of nodes or index shards in the source and destination clusters determines how long the data migration will take. Data migration consists of two phases: data backup and restoration. The backup duration is determined by the source cluster and the restoration duration is determined by the destination cluster. The formula for calculating the total migration duration is as follows:

  • If the number of index shards is greater than the number of nodes:

    Total duration (in seconds) = (800 GB ÷ 40 MB ÷ Number of nodes in the source cluster + 800 GB ÷ 40 MB ÷ Number of nodes in the destination cluster) x Number of indexes

  • If the number of index shards is smaller than the number of nodes:

    Total duration (in seconds) = (800 GB ÷ 40 MB ÷ Number of index shards in the source cluster + 800 GB ÷ 40 MB ÷ Number of index shards in the destination cluster) x Number of indexes

NOTE:

The migration duration estimated using the formula is the minimal duration possible (if each node transmits data at the fastest speed, 40 MB/s). The actual duration also depends on factors such as the network and resources condition.

Prerequisites

  • The destination cluster (Es-2) and source cluster (Es-1) are available. You are advised to migrate a cluster during off-peak hours.
  • Ensure that the destination cluster (Es-2) and source cluster (Es-1) are in the same region.

    If the cluster is deployed across regions or accounts, refer to the Configuring Cross-Region Replication section to copy the OBS bucket that stores snapshots in the source cluster to the OBS bucket in the destination cluster. During the migration, restore the data in the destination cluster.

Procedure

  1. Log in to the Cloud Search Service management console.
  2. Choose Clusters > Elasticsearch. On the displayed page, click the source cluster name Es-1 to go to the basic information page.
  3. In the navigation pane, choose Cluster Snapshots, and set basic snapshot configurations.
    Table 1 Basic configurations for a cluster snapshot

    Parameter

    Description

    OBS Bucket

    Select an OBS bucket for storing cluster snapshots.

    Backup Path

    Storage path of the cluster snapshot in the OBS bucket. You can retain the default value.

    IAM Agency

    To store snapshot data to an OBS bucket, you must have the required OBS access permissions. Select an IAM agency to grant the current account the permission to access and use OBS.
    • If you are configuring an agency for the first time, click Automatically Create IAM Agency to create css-obs-agency.
    • If there is an IAM agency automatically created earlier, you can click One-click authorization to have the OBS Administrator permissions deleted automatically, and have the following custom policies added automatically instead to implement more refined permissions control.
      "obs:bucket:GetBucketLocation",
      "obs:object:GetObjectVersion",
      "obs:object:GetObject",
      "obs:object:DeleteObject",
      "obs:bucket:HeadBucket",
      "obs:bucket:GetBucketStoragePolicy",
      "obs:object:DeleteObjectVersion",
      "obs:bucket:ListBucketVersions",
      "obs:bucket:ListBucket",
      "obs:object:PutObject"
    • When OBS buckets use SSE-KMS encryption, the IAM agency must be granted KMS permissions. You can click Automatically Create IAM Agency and One-click authorization to have the following custom policies created automatically.
      "kms:cmk:create",
      "kms:dek:create",
      "kms:cmk:get",
      "kms:dek:decrypt",
      "kms:cmk:list"
    • To use Automatically Create IAM Agency and One-click authorization, the following minimum permissions are required:
      "iam:agencies:listAgencies",
      "iam:roles:listRoles",
      "iam:agencies:getAgency",
      "iam:agencies:createAgency",
      "iam:permissions:listRolesForAgency",
      "iam:permissions:grantRoleToAgency",
      "iam:permissions:listRolesForAgencyOnProject",
      "iam:permissions:revokeRoleFromAgency",
      "iam:roles:createRole"
    • To use an IAM agency, the following minimum permissions are required:
      "iam:agencies:listAgencies",
      "iam:agencies:getAgency",
      "iam:permissions:listRolesForAgencyOnProject",
      "iam:permissions:listRolesForAgency"
  4. Click Create. In the dialog box that is displayed, configure the parameters and click OK to manually create a snapshot.
    Table 2 Snapshot creation parameters

    Parameter

    Description

    Snapshot Name

    User-defined snapshot name. You can retain the default value.

    Index

    Indexes that you want to back up using snapshots. The index names cannot contain spaces or uppercase letters, and cannot contain "\<|>/?. Use commas (,) to separate different index names. If you do not specify this parameter, all indexes in the cluster are backed up by default. You can use the asterisk (*) to match multiple indexes. For example, if you enter index*, then the data of all indexes whose names are prefixed with index will be backed up.

    NOTE:

    Run the GET /_cat/indices command in Kibana to query the names of all indexes in the cluster.

    Description

    Snapshot description.

    In the snapshot management list, if the snapshot status is Available, the snapshot has been created.

  5. In the snapshot management list, click Restore in the Operation column of the snapshot and configure restoration parameters to restore data to destination cluster Es-2.
    Table 3 Snapshot restoration parameters

    Parameter

    Description

    Index

    Enter the name of the index to be restored. If this parameter is not specified, all index data will be restored. You can use the asterisk (*) to match multiple indexes. For example, index* indicates that all indexes with the prefix index in snapshots are restored.

    Rename Pattern

    Index name matching rule. The Rename Pattern and Rename Replacement take effect only when they are configured at the same time. You can configure them to rename matched indexes in snapshots.

    Rename Replacement

    Rule for renaming an index name. The Rename Pattern and Rename Replacement take effect only when they are configured at the same time.

    The default value restored_index_$1 indicates that restored_ is added in front of the names of all restored indexes.

    Cluster

    Select a destination cluster, for example, Es-2.

    Select or deselect Overwrite same-name indexes in the destination cluster. By default, it is deselected. Data restoration using snapshots works by overwriting existing snapshot files. When there are same-name indexes in the destination cluster, you need to select this option in order to restore same-name, same-shard structure indexes. Indexes with a different shard structure cannot be restored. Exercise caution when performing this operation.

    In the snapshot list, when the Task Status changes to Restoration succeeded, the data migration is complete.

  6. After the data migration is complete, check the data consistency between the destination Elasticsearch cluster Es-2 and source Elasticsearch cluster Es-1. For example, run the _cat/indices command in the source and destination clusters, separately, to check whether their indexes are consistent.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback