Help Center/ GeminiDB/ GeminiDB Redis API/ Data Migration/ From On-Premises Redis to GeminiDB Redis API
Updated on 2024-08-12 GMT+08:00
View PDF
Share

From On-Premises Redis to GeminiDB Redis API

You can use DRS or Redis-Shake to migrate data from self-built Redis to GeminiDB Redis. For details about migration using DRS, see From Redis to GeminiDB Redis and From Cluster Redis to GeminiDB Redis. This section describes how to use Redis-Shake to migrate data from an on-premises Redis database to a GeminiDB Redis instance.

Migration Principles

Use Redis-Shake to migrate data from an on-premises Redis instance (source) to a GeminiDB Redis instance (destination). Full and incremental migrations are both supported. The source can be a single-node, primary/standby, or cluster instance, or an RDB file.

  • Full migration: Redis-Shake works as a slave node for the source, obtains data of an RDB file generated by the source, and then parses the data and sends it to the destination by running commands. You can also use an RDB file as the source to import snapshot data generated at a specific time point.
  • Incremental migration: After full migration is complete, Redis-Shake continues sending incremental data to the destination by running commands until you stop Redis-Shake.

Precautions

  • If data synchronization between master and slave Redis nodes is disconnected, stop Redis-Shake, clear all data in the destination, and retry a migration. To ensure a smooth synchronization, migrate data during off-peak hours and set a large value for parameter client-output-buffer-limit to increase the ring buffer size for incremental synchronization.
  • Redis-Shake does not write data into the source, but may have a temporary impact on the source performance.
  • If the migration involves multiple databases, ensure that source databases are correctly mapped to destination databases to prevent unexpected data overwriting.
  • Streaming data cannot be migrated.
  • Ensure that network communication among Redis-Shake, the source instance, and the destination instance is normal.
  • To get technical support, choose Service Tickets > Create Service Ticket in the upper right corner of the console and contact customer service.

Migrating Data from an Open-Source Redis Single-Node or Primary/Standby Instance to a GeminiDB Redis Instance

You can import a file similar to the above or perform the following operations to migrate data from an open-source single-node or primary/standby Redis instance to a GeminiDB Redis instance.

  1. Deploy the required migration tool.

    1. Obtain the Redis-Shake package.

      Download the Redis-Shake release package and decompress it.

    2. Modify the Redis-Shake.conf configuration file and configuring the following items:

      log.level = info #Default log level. A printed INFO log contains migration progress information, based on which you can judge whether the migration is complete.

      source.address = <host>:<port> # IP address and port of the host where the open-source Redis instance is deployed.

      source.password_raw = ***** # Password for logging in to the source instance.

      source.type = standalone # Type of the source instance.

      target.address = <host>:8635 # Address for logging in to the destination instance.

      target.password_raw = ***** # Password for logging in to the destination instance.

      target.version = 5.0 # Version of the destination Redis instance.

      target.type = standalone # Type of the destination instance.

      target.db = -1 # Specific database on the destination that all data will be migrated to. If this parameter is set to -1, a mapping relationship is established between migrated databases and databases in the source instance.

    3. Specify whether data of the destination is overwritten.

      key_exists = none

      If there are duplicate keys on the source and destination, specify whether data of the destination is overwritten. The options are as follows:

      • rewrite indicates that the source overwrites the destination.
      • none indicates that the migration process exists once duplicate keys are detected.
      • ignore indicates that keys in the source are retained and keys in the destination are ignored. This value does not take effect in rump mode.

      none is recommended. There will be no duplicate data because the source is an RDB file. If the migration process exits unexpectedly, contact customer service.

  2. Migrate data.

    Migration starting command:

    ./redis-shake.linux -conf=redis-shake.conf -type=sync

    • If the following information is displayed, the full synchronization is completed and incremental synchronization begins. 
      sync rdb done
    • If the following information is displayed, no new data is incremented. You can stop the migration process to disconnect incremental synchronization: 
      sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0

  3. Verify data.

    Download and decompress RedisFullCheck and use it to verify data by referring to Migrating Data from an Open-Source Redis Single-Node or Primary/Standby Instance to a GeminiDB Redis Instance.

    ./redis-full-check -s SOURCE_IP:SOURCE_PORT -p SOURCE_PWD -t TARGET_IP:8635 -a TARGET_PWD

    If the following information is displayed, the migration is successful, and data is consistent between the source and destination:

    all finish successfully, totally 0 key(s) and 0 field(s) conflict

Migrating Data from an Open-Source Redis Cluster Instance to a GeminiDB Redis Instance

Configure the following items in the configuration file:

source.address = <host1>:<port1>,<host2>:<port2>,<host2>:<port2> # IP addresses and ports of hosts at the source.

source.type = cluster # Cluster type of the source.

For other steps, see Migrating Data from an Open-Source Redis Single-Node or Primary/Standby Instance to a GeminiDB Redis Instance.

Migrating Data from an Open-Source Codis Cluster Instance to a GeminiDB Redis Instance

Obtain host IP addresses and ports of all shards of the Codis cluster instance and configure the configuration file as follows:

source.address = <host1>:<port1>,<host2>:<port2>,<host2>:<port2> # IP addresses and ports of hosts at the source.

source.type = cluster # Cluster type of the source.

For other steps, see Migrating Data from an Open-Source Redis Single-Node or Primary/Standby Instance to a GeminiDB Redis Instance.

Fully Scanning Data on and Migrating It from an Open-Source Redis Instance to a GeminiDB Redis Instance

If data cannot be migrated with any of the above methods, try rump of Redis-Shake to scan databases one by one and migrate them.

  1. Deploy the required migration tool.

    1. Obtain the Redis-Shake package.

      Download the Redis-Shake release package and decompress it.

    2. Modify the Redis-Shake.conf configuration file and configuring the following items:

      log.level = info #Default log level. A printed INFO log contains migration progress information, based on which you can judge whether the migration is complete.

      source.address = <host>:<port> # IP address and port of the host where the open-source Redis instance is deployed.

      source.password_raw = ***** # Password for logging in to the source instance.

      source.type = standalone # Type of the source instance.

      target.address = <host>:8635 # Address for logging in to the destination instance.

      target.password_raw = ***** # Password for logging in to the destination instance.

      target.version = 5.0 # Version of the destination Redis instance.

      target.type = standalone # Type of the destination instance.

      target.db = -1 # Specific database on the destination that all data will be migrated to. If this parameter is set to -1, a mapping relationship is established between migrated databases and databases in the source instance.

    3. Specify whether data of the destination is overwritten.

      key_exists = none

      If there are duplicate keys on the source and destination, specify whether data of the destination is overwritten. The options are as follows:

      • rewrite indicates that the source overwrites the destination.
      • none indicates that the migration process exists once duplicate keys are detected.
      • ignore indicates that keys in the source are retained and keys in the destination are ignored. This value does not take effect in rump mode.

      none is recommended. There will be no duplicate data because the source is an RDB file. If the migration process exits unexpectedly, contact customer service.

  2. Migrate data.

    Migration starting command:

    ./redis-shake.linux -conf=redis-shake.conf -type=rump

    • If information similar to the following is displayed, synchronizing full data is complete. 
      dbRumper[0] executor[0] finish

  3. Verify data.

    Download and decompress RedisFullCheck and use it to verify data.

    ./redis-full-check -s SOURCE_IP:SOURCE_PORT -p SOURCE_PWD -t TARGET_IP:8635 -a TARGET_PWD

    If the following information is displayed, the migration is successful, and data is consistent between the source and destination:

    all finish successfully, totally 0 key(s) and 0 field(s) conflict

Parent topic: Data Migration