Migrating Data from Open-source Redis to GeminiDB Redis API Using Redis-Shake

You can use DRS or Redis-Shake to migrate data from self-built Redis to GeminiDB Redis. For details about migration using DRS, see From Redis to GeminiDB Redis and From Cluster Redis to GeminiDB Redis. This section describes how to use Redis-Shake to migrate data from an on-premises Redis database to a GeminiDB Redis instance.

Migration Principles

Use Redis-Shake to migrate data from an on-premises Redis instance (source) to a GeminiDB Redis instance (destination). Full and incremental migrations are both supported. The source can be a single-node, primary/standby, or cluster instance, or an RDB file.

Full migration: Redis-Shake works as a slave node for the source, obtains data of an RDB file generated by the source, and then parses the data and sends it to the destination by running commands. You can also use an RDB file as the source to import snapshot data generated at a specific time point.
Incremental migration: After full migration is complete, Redis-Shake continues sending incremental data to the destination by running commands until you stop Redis-Shake.

Precautions

If data synchronization between master and slave Redis nodes is disconnected, stop Redis-Shake, clear all data in the destination, and retry a migration. To ensure a smooth synchronization, migrate data during off-peak hours and set a large value for parameter client-output-buffer-limit to increase the ring buffer size for incremental synchronization.
Redis-Shake does not write data into the source, but may have a temporary impact on the source performance.
If the migration involves multiple databases, ensure that source databases are correctly mapped to destination databases to prevent unexpected data overwriting.
Streaming data cannot be migrated.
Ensure that network communication among Redis-Shake, the source instance, and the destination instance is normal.
To get technical support, choose Service Tickets > Create Service Ticket in the upper right corner of the console and contact customer service.

Migrating Data from an Open-Source Redis Single-Node or Primary/Standby Instance to a GeminiDB Redis Instance

You can import a file similar to the above or perform the following operations to migrate data from an open-source single-node or primary/standby Redis instance to a GeminiDB Redis instance.

Deploy the required migration tool.
1. Obtain the Redis-Shake package.
  
  Download the Redis-Shake release package and decompress it.
2. Modify the Redis-Shake.conf configuration file and configuring the following items:
  log.level = info #Default log level. A printed INFO log contains migration progress information, based on which you can judge whether the migration is complete.
  
  source.address = <host>:<port> # IP address and port of the host where the open-source Redis instance is deployed.
  
  source.password_raw = ***** # Password for logging in to the source instance.
  
  source.type = standalone # Type of the source instance.
  
  target.address = <host>:8635 # Address for logging in to the destination instance.
  
  target.password_raw = ***** # Password for logging in to the destination instance.
  
  target.version = 5.0 # Version of the destination Redis instance.
  
  target.type = standalone # Type of the destination instance.
  
  target.db = -1 # Specific database on the destination that all data will be migrated to. If this parameter is set to -1, a mapping relationship is established between migrated databases and databases in the source instance.
3. Specify whether data of the destination is overwritten.
  key_exists = none
  If there are duplicate keys on the source and destination, specify whether data of the destination is overwritten. The options are as follows:
  - rewrite indicates that the source overwrites the destination.
  - none indicates that the migration process exists once duplicate keys are detected.
  - ignore indicates that keys in the source are retained and keys in the destination are ignored. This value does not take effect in rump mode.
  none is recommended. There will be no duplicate data because the source is an RDB file. If the migration process exits unexpectedly, contact customer service.
Migrate data.

Migration starting command:

./redis-shake.linux -conf=redis-shake.conf -type=sync
- If the following information is displayed, the full synchronization is completed and incremental synchronization begins.
```
sync rdb done
```
- If the following information is displayed, no new data is incremented. You can stop the migration process to disconnect incremental synchronization:
```
sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
```
Verify data.

Download and decompress RedisFullCheck and use it to verify data by referring to Migrating Data from an Open-Source Redis Single-Node or Primary/Standby Instance to a GeminiDB Redis Instance.

./redis-full-check -s SOURCE_IP:SOURCE_PORT -p SOURCE_PWD -t TARGET_IP:8635 -a TARGET_PWD

If the following information is displayed, the migration is successful, and data is consistent between the source and destination:
```
all finish successfully, totally 0 key(s) and 0 field(s) conflict
```

Migrating Data from Redis Cluster to GeminiDB Redis API

Configure the following items in the configuration file:

source.address = <host1>:<port1>,<host2>:<port2>,<host2>:<port2> # IP addresses and ports of hosts at the source.

source.type = cluster # Cluster type of the source.

For other steps, see Migrating Data from an Open-Source Redis Single-Node or Primary/Standby Instance to a GeminiDB Redis Instance.

Migrating Data from an Open-Source Codis Cluster Instance to a GeminiDB Redis Instance

Obtain host IP addresses and ports of all shards of the Codis cluster instance and configure the configuration file as follows:

source.address = <host1>:<port1>,<host2>:<port2>,<host2>:<port2> # IP addresses and ports of hosts at the source.

source.type = cluster # Cluster type of the source.

For other steps, see Migrating Data from an Open-Source Redis Single-Node or Primary/Standby Instance to a GeminiDB Redis Instance.

Fully Scanning Data on and Migrating It from an Open-Source Redis Instance to a GeminiDB Redis Instance

If data cannot be migrated with any of the above methods, try rump of Redis-Shake to scan databases one by one and migrate them.

Deploy the required migration tool.
1. Obtain the Redis-Shake package.
  
  Download the Redis-Shake release package and decompress it.
2. Modify the Redis-Shake.conf configuration file and configuring the following items:
  log.level = info #Default log level. A printed INFO log contains migration progress information, based on which you can judge whether the migration is complete.
  
  source.address = <host>:<port> # IP address and port of the host where the open-source Redis instance is deployed.
  
  source.password_raw = ***** # Password for logging in to the source instance.
  
  source.type = standalone # Type of the source instance.
  
  target.address = <host>:8635 # Address for logging in to the destination instance.
  
  target.password_raw = ***** # Password for logging in to the destination instance.
  
  target.version = 5.0 # Version of the destination Redis instance.
  
  target.type = standalone # Type of the destination instance.
  
  target.db = -1 # Specific database on the destination that all data will be migrated to. If this parameter is set to -1, a mapping relationship is established between migrated databases and databases in the source instance.
3. Specify whether data of the destination is overwritten.
  key_exists = none
  If there are duplicate keys on the source and destination, specify whether data of the destination is overwritten. The options are as follows:
  - rewrite indicates that the source overwrites the destination.
  - none indicates that the migration process exists once duplicate keys are detected.
  - ignore indicates that keys in the source are retained and keys in the destination are ignored. This value does not take effect in rump mode.
  none is recommended. There will be no duplicate data because the source is an RDB file. If the migration process exits unexpectedly, contact customer service.
Migrate data.

Migration starting command:

./redis-shake.linux -conf=redis-shake.conf -type=rump
- If information similar to the following is displayed, synchronizing full data is complete.
```
dbRumper[0] executor[0] finish
```
Verify data.

Download and decompress RedisFullCheck and use it to verify data.

./redis-full-check -s SOURCE_IP:SOURCE_PORT -p SOURCE_PWD -t TARGET_IP:8635 -a TARGET_PWD

If the following information is displayed, the migration is successful, and data is consistent between the source and destination:
```
all finish successfully, totally 0 key(s) and 0 field(s) conflict
```

Verifying Data Consistency After Migration

After the migration is complete, you can check data consistency.

Data has been migrated from Redis, or incremental migration has started.
Redis-full-check must be deployed on the ECS, and the ECS is connected to the source and destination databases.
During incremental migration, data may be inconsistent due to network latency between the source and destination databases. You are advised to stop writing data to the source and then verify data consistency.
When Redis is used, an expiration time is usually set for keys. During migration, setting a key expiration time affects data consistency. Data may be inconsistent due to inconsistent expiration time.

During migration, DTS writes temporary probing keys to Redis on the destination database. Non-service data may be detected during data verification, which is normal.

Procedure

Log in to the ECS and ensure it is connected to the source and destination Redis databases.
Deploy redis-full-check.

Verify data.

/redis-full-check -s {Source IP address}:{Source port} -p {Source password} -t {Destination IP address}:{Destination port} -a {Destination password} -m 1

**Table 1** Parameter description
Parameter	Description	Example Value
-s	Source Redis database address and port number	-s 10.0.0.1:6379
-p	Password of the source Redis database	-
-t	Destination GeminiDB Redis database address and port number	-t 10.0.0.2:6379
-a	Password of the destination GeminiDB Redis database	-
-m	Verification mode: All key-value pairs Value length only Key integrity only All key values are verified, but only the length of big keys is verified. By default, the second verification mode is used.	-m 1
-q	Maximum QPS. The default value is 15000.	-q 5000
-d	Name of the file for saving the verification result. The default value is result.db.	-d result.db

View the verification result file.

By default, three rounds of verification are performed and three verification result files are generated. Generally, you only need to view the last verification result file.
- Run the sqlite3 result.db.3 command.
- Run the select * from key command.
- Check whether there are abnormal keys.