Help Center/ GeminiDB/ GeminiDB Cassandra API/ GeminiDB DynamoDB Instances/ Migrating Data from a DynamoDB Instance to a GeminiDB DynamoDB Instance
Updated on 2024-10-30 GMT+08:00
View PDF
Share

Migrating Data from a DynamoDB Instance to a GeminiDB DynamoDB Instance

GeminiX is a Huawei-developed tool that can seamlessly migrate data from a DynamoDB instance to a GeminiDB DynamoDB instance based on DynamoDB semantics. This section describes how to migrate data from an AWS DynamoDB instance to a GeminiDB DynamoDB instance.

Migration Principles

This solution consists of two modules: full migration and incremental migration. You need to create an ECS for data forwarding. GeminiX supports distributed migration and concurrent migration of multiple service tables on multiple servers.

  • During the full migration, DynamoDB scan interfaces read all data by segment and write the data to the GeminiDB DynamoDB instance.
  • In the incremental migration, DynamoDB stream interfaces synchronize incremental data to the GeminiDB DynamoDB instance.

Prerequisites

  • A GeminiDB DynamoDB instance has been created and is running normally.
  • An ECS with 16 vCPUs, 64 GB of memory, and 100 or more GB of disk space has been created. For details about how to create an ECS, see Purchasing an ECS" in Getting Started with Elastic Cloud Server.
  • Data is transmitted between the AWS DynamoDB instance and ECS through private lines. Ensure that the networks between the ECS and AWS DynamoDB instance and between the ECS and GeminiDB DynamoDB instance are normal.
  • The storage space of the GeminiDB DynamoDB instance cannot be less than that of the AWS DynamoDB instance.
  • AWS DynamoDB stream interfaces have been enabled.

Procedure

  1. Deploy the required migration tool.

    1. Obtain GeminiX.
    2. Modify the conf.yaml configuration file and configure the following items: 
      source:
  endpoint: 'https://dynamodb.us-west-2.amazonaws.com' #Domain name of the AWS DynamoDB instance
  access_key_id: 'xx' #Access key ID of the AWS DynamoDB instance
secret_access_key: 'xx' #Secret access key of the AWS DynamoDB instance
target:
  endpoint: ***.***.***.*** #Domain name of the GeminiDB DynamoDB instance
  access_key_id: 'xx' #Access key ID of the GeminiDB DynamoDB instance
  secret_access_key: 'xx' #Secret access key of the GeminiDB DynamoDB instance

  2. Perform a full data migration.

    Run the following command:

    nohup ./bin/migration.sh fullsync -c conf/conf.yaml >run.log 2>&1 &
    If information similar to the following is displayed, synchronizing full data is complete. 
    finish syncing all tables and indexes

  3. Migrate incremental data.

    Before running the incremental migration command, you need to configure the connection information of the DynamoDB instance and GeminiDB DynamoDB instance and the name of the incremental migration table in GeminiX. For details, see 1.b.

    Run the following command: 
    nohup ./bin/migration.sh incresync -c conf/conf.yaml >run.log 2>&1 &

  4. After the migration is complete, switch the service workloads to the GeminiDB DynamoDB instance.