Help Center> GeminiDB> GeminiDB Redis API> Data Migration> From LevelDB to GeminiDB Redis API
Updated on 2024-02-08 GMT+08:00
View PDF

From LevelDB to GeminiDB Redis API

LevelDB is an open-source, persistent, and single-node KV database engine. It provides high random write performance and sequential read/write performance, and applies to write intensive scenarios. LevelDB does not provide the C/S network structure and must be deployed on the same server as your services. Compared with RocksDB developed based on LevelDB, LevelDB has many disadvantages. For example, it cannot make the most out of multi-core servers, and does not support TB-level data storage, and cannot read data from HDFS.

GeminiDB Redis API uses RocksDB as the storage engine. It is compatible with the Redis protocol and provides various data types to meet LevelDB requirements. In addition, RocksDB has been customized to allow storage to be scaled in seconds, making it easy to migrate LevelDB workloads to the Redis ecosystem. You do not need to migrate data during scaling.

This section describes how to migrate data from LevelDB to GeminiDB Redis API.

Migration Principles

  • Use the self-developed migration tool leveldb-port to deploy LevelDB on the same server as your services, prepare the configuration file, and start the migration task to automatically complete full and incremental migration.
  • The full migration process is efficient. It takes a snapshot of the LevelDB data, scans the entire database, packs the data into a format that can be identified by GeminiDB Redis API, and then sends the data to GeminiDB Redis API.
  • During incremental migration, the WAL file of LevelDB and the LevelDB operations are parsed, and the keys in the WAL file are sharded and sent by multiple threads.

Precautions

  • The migration tool needs to be deployed on the source DB, which consumes certain performance. You can modify the configuration file to control the performance.
  • During the migration, the source data file of LevelDB is read-only. There is no risk of data damage.
  • Services do not need to be stopped during the migration.
  • If a fault occurs during the migration, clear the GeminiDB Redis instance and restart the migration.

Procedure

To migrate data from LevelDB to GeminiDB Redis API, choose Service Tickets > Create Service Ticket in the upper right corner of the management console and contact technical support.

Migration Performance Reference

  • Environment: The source LevelDB and leveldb-port are deployed on a Huawei Cloud ECS with 4 vCPUs and 16 GB memory. The target DB is a three-node GeminiDB Redis instance with 2 vCPUs and 8 GB memory.
  • Full migration: 10 GB data is preconfigured, and the migration speed is about 8 MB/s.
  • Incremental migration: Set the value to 1 KB and the migration speed to 7,000 QPS.
Parent topic: Data Migration