Migration from DynamoDB to GeminiDB DynamoDB-Compatible API

How Data Is Migrated

This solution supports full migration and incremental migration. You need to create an ECS. GeminiX supports distributed migration and concurrent migration of multiple service tables on multiple servers.

• During full migration, DynamoDB scan APIs read all data by segment and write the data to the GeminiDB DynamoDB-Compatible instance.
• During incremental migration, DynamoDB stream APIs synchronize incremental data to the GeminiDB DynamoDB-Compatible instance.

Prerequisites

• A GeminiDB DynamoDB-Compatible instance has been created and is running properly.
• An ECS with 16 vCPUs, 64 GB of memory, and 100 or more GB of disk space has been created. For details about how to create an ECS, see Purchasing an ECS" in Getting Started with Elastic Cloud Server.
• Data is transmitted between the AWS DynamoDB instance and ECS via private connections. Ensure that the networks between the ECS and AWS DynamoDB instance and between the ECS and GeminiDB DynamoDB-Compatible instance are normal.
• The storage space of the GeminiDB DynamoDB-Compatible instance cannot be less than that of the AWS DynamoDB instance.
• AWS DynamoDB stream interfaces have been enabled.

Procedure

1. Deploy the required migration tool.
   1. Obtain GeminiX.
   2. Modify the conf.yaml configuration file and configure the following items:

      source:
        endpoint: 'https://dynamodb.us-west-2.amazonaws.com' #Domain name of a DynamoDB instance
        access_key_id: 'xx' #AK ID of the DynamoDB instance
      secret_access_key: 'xx' #SK of the DynamoDB instance
      target:
        endpoint: ***.***.***.*** #Domain name of a GeminiDB DynamoDB-Compatible instance
        access_key_id: 'xx' #AK of the GeminiDB DynamoDB-Compatible instance
        secret_access_key: 'xx' #SK of the GeminiDB DynamoDB-Compatible instance

2. Perform full data migration.

   Run the following command:

   nohup ./bin/migration.sh fullsync -c conf/conf.yaml >run.log 2>&1 &

   If the following information is displayed in execution logs, full synchronization is complete.

   finish syncing all tables and indexes

3. Perform incremental migration.

   Before running the incremental migration command, configure connection information of the DynamoDB instance and GeminiDB DynamoDB-Compatible instance and the name of the incremental migration table in GeminiX. For details, see 1.b.

   Run the following command:

   nohup ./bin/migration.sh incresync -c conf/conf.yaml >run.log 2>&1 &

4. After the migration is complete, switch workloads to the GeminiDB DynamoDB-Compatible instance.