Help Center/ Data Replication Service/ Real-Time Migration/ To the Cloud/ From MongoDB to GaussDB(for Mongo)
Updated on 2022-09-21 GMT+08:00

From MongoDB to GaussDB(for Mongo)

Supported Source and Destination Databases

Table 1 Supported Databases

Source DB

Destination DB

  • On-premises MongoDB databases
  • MongoDB databases on an ECS
  • MongoDB database on other clouds
  • GaussDB(for Mongo) instances

Prerequisites

  • You have logged in to the DRS console.
  • Your account balance is greater than or equal to $0 USD.
  • For details about the DB types and versions supported by real-time migration, see Supported Databases.
  • If a subaccount is used to create a DRS task, ensure that an agency has been added. To create an agency, see Agency Management.

Suggestions

  • The success of database migration depends on environment and manual operations. To ensure a smooth migration, perform a migration trial before you start the migration to help you detect and resolve problems in advance.
  • In the migration, ensure that no data is written to the destination database to ensure data consistency before and after the migration.
  • Start your migration task during off-peak hours. A less active database is easier to migrate successfully. If the data is fairly static, there is less likely to be any severe performance impacts during the migration.
    • If network bandwidth is not limited, the query rate of the source database increases by about 20 MB/s during full migration, and two to four CPUs are occupied.
  • Data-Level Comparison

    To obtain accurate comparison results, start data comparison at a specified time point during off-peak hours. If it is needed, select Start at a specified time for Comparison Time. Due to slight time difference and continuous operations on data, data inconsistency may occur, reducing the reliability and validity of the comparison results.

Precautions

Before creating a migration task, read the following notes:

Table 2 Precautions

Type

Restrictions

Database permissions

Source database (minimum permissions):

  • Full migration:
    • Replica set: The source database user must have the readAnyDatabase permission for the admin database.
    • Single node: The source database user must have the readAnyDatabase permission for the admin database.
    • Cluster: The source database user must have the readAnyDatabase permission for the admin database and the read permission for the config database.
    • To migrate accounts and roles of the source database, the source and destination database users must have the read permission for the system.users and system.roles system tables of the admin database.
  • Full+incremental migration:
    • Replica set: The source database user must have the readAnyDatabase permission for the admin database and the read permission for the local database.
    • Single node: The source database user must have the readAnyDatabase permission for the admin database and the read permission for the local database.
    • Cluster: The source mongos node user must have the readAnyDatabase permission for the admin database and the read permission for the config database. The source shard node user must have the readAnyDatabase permission for the admin database and the read permission for the local database.
    • To migrate accounts and roles of the source database, the source and destination database users must have the read permission for the system.users and system.roles system tables of the admin database.

Minimum permission requirements: The destination database user must have the dbAdminAnyDatabase permission for the admin database and the readWrite permission for the destination database. If the destination database is a cluster instance, the database user must have the clusterManager permission for the admin database.

Migration objects

  • Replica set: Only collections (including validator and capped collections), indexes, and views can be migrated.
  • Cluster: Only collections (including validator and capped collections), shard keys, indexes, and views can be migrated.
  • Single node: Only collections (including validator and capped collections), indexes, and views can be migrated.
  • The statement for creating a view cannot contain a regular expression.
  • Collections that contain the _id field without indexes are not supported.
  • The first parameter of BinData() cannot be 2.

Source database

  • The source database name cannot contain /\."$ or spaces. The collection name and view name cannot start with system. or contain the dollar sign ($).
  • If the incremental source data cluster is migrated, the source database balancer must be disabled.
  • The source cannot be a GaussDB(for Mongo) instance.

Destination database

  • The destination DB instance is running properly.
  • The destination DB instance must have sufficient storage space.
  • When multiple source databases are migrated to the same destination database, the name of the database to be migrated must be unique.
  • DRS supports full migration between cluster instances. If the source cluster instance is not sharded, ensure that the size of the primary shard on the destination database is greater than that of the source database.
  • Data cannot be migrated from a newer version database to an older version database.

Precautions

  • Objects that have dependencies must be migrated at the same time to avoid migration failure. Common dependencies: collections referenced by views, and views referenced by views
  • Replica set: The MongoDB replica set instance must be available and have primary nodes.
  • If the source is not a cluster instance, you can perform the following operations during incremental migration:
    • Creating and deleting databases
    • Adding, deleting, and updating documents
    • Creating and deleting collections
    • Creating and deleting indexes
    • Creating and deleting views
    • The convertToCapped, collMod, and renameCollection commands are supported.
  • During a full plus incremental migration from a replica set to a cluster or between clusters, the objects to be migrated cannot be deleted. Otherwise, the migration task will fail.
  • If a Time-to-Live (TTL) index already exists in the collection of the source database or is created during an incremental migration, data consistency cannot be ensured when source and destination databases are in different time zone.
  • If the destination database does not support compression parameter settings, DRS does not migrate compression parameters, and no error is reported during the migration.
  • If the MongoDB service of the source database is deployed with other services on the same server, set the value of the cacheSizeGB parameter to the half of the minimum idle cache for the WiredTiger engine of the source database.
  • If the source is a replica set instance, enter information about all primary and secondary nodes to reduce the impact of a primary/secondary switchover on the migration task. If you enter information about multiple primary and secondary nodes, ensure that all nodes belong to the same replica set instance.
  • If the source is a cluster instance, enter information about multiple mongos nodes to reduce the impact of single-node failure on the migration task. In addition, ensure that all mongos nodes belong to the same cluster instance. For an incremental migration of a cluster instance, enter the information about all primary and secondary shard nodes in the same cluster to reduce the impact of a primary/secondary switchover on the migration task.
  • The maximum number of collections that can be created in a GaussDB(for Mongo) cluster is calculated as follows: Maximum number of chunks x 4 x Number of shards

    The following table lists the maximum number of chunks of different GaussDB(for Mongo) instance specifications.

    1 vCPUs, 4 GB -> 50

    2 vCPUs, 8 GB -> 100

    4 vCPUs, 16 GB -> 200

    8 vCPUs, 32 GB -> 400

    16 vCPUs, 64 GB -> 800

    32 vCPUs, 128 GB -> 1600

    Check whether the destination database meets the requirements based on the number of source database collections.

  • In some migration scenarios, to prevent the drop database operation from deleting the existing collections in the destination database, the drop database operation will not be synchronized to the destination database.
    • If the source database version is earlier than MongoDB 3.6, running the drop database command will delete the collections only from the source database. The collections in destination database will not be deleted.
    • If the source database version is MongoDB 3.6 or later, the drop database operation is represented by the drop database and drop collection operations in oplog. Running the drop database command will delete the collections from both the source and destination databases.
  • To ensure data consistency, you are not allowed to modify the destination database (including but not limited to DDL and DML operations) during migration.
  • During the migration, do not modify or delete the usernames, passwords, permissions, or ports of the source and destination databases.
  • During task startup or full migration, you are not advised to perform DDL operations on the source database, such as deleting databases, collections, indexes, documents, or views. Otherwise, the migration may fail.
  • During migration, data rollback caused by a primary/standby switchover of the source database is not supported.
  • During an incremental migration of collections, you are advised not to rename the collections.
  • Files larger than 16 MB cannot be inserted to or updated in the source database during full or incremental migration.
  • To accelerate the migration, delete unnecessary indexes from the source database and retain only necessary indexes before the migration. You are advised not to create indexes for the source database during the migration. If indexes must be created, create them in the background.