Migrating Data Using mongodump and mongorestore

Precautions

• The mongodump and mongorestore tools support only full migration. To ensure data consistency, stop services on the source database and stop writing data to the source database before the migration.
• You are advised to perform the migration during off-peak hours to avoid impacting services.
• The admin and local system databases cannot be migrated.
• Make sure that no service set has been created in the system databases admin and local in the source database. If there is already a service set, migrate them out of the system databases admin and local before migration.
• Before importing data, ensure that the necessary indexes are there on the source database. Delete any unnecessary indexes and create any necessary indexes before migration.
• If you choose to migrate a sharded cluster, you must create a set of shards in the destination database and configure sharding. In addition, indexes must be created before migration.
• If the backup using the mongodump tool fails (for example, an error is reported when the backup progress reaches 97%), you are advised to increase the VM storage space and reserve some redundant space before performing the backup again.
• User rwuser can only operate service database tables. You are advised to specify databases and tables to import and export only service data. Otherwise, the insufficient permission problem may occur during full import and export.

Prerequisites

1. Prepare an ECS or a device that can access DDS.
   • To bind an EIP to a DB instance:
     1. Bind an EIP to a node in the DB instance. For details about how to bind an EIP to a node, see "Binding an EIP" in the Getting Started with Document Database Service .
     2. Ensure that your local device can access the EIP that has been bound to the DB instance.
2. A migration tool has been installed on the prepared ECS.

   For details on how to install the migration tool, see How Can I Install a MongoDB Client?

   

   • The mongodump and mongorestore tools are part of the MongoDB client installation package.
   • The MongoDB client version must match the instance version. Otherwise, compatibility issues may occur.

Exporting Data

1. Log in to the ECS or the device that can access DDS.
2. Back up the source database data using the mongodump tool.

   An SSL connection is used in this example. If you select an unencrypted connection, delete --ssl --sslCAFile <FILE_PATH> --sslAllowInvalidCertificates from the following command.

   ./mongodump --host <DB_HOST> --port <DB_PORT> --authenticationDatabase <AUTH_DB> -u <DB_USER> --ssl --sslCAFile <FILE_PATH> --sslAllowInvalidCertificates --db <DB_NAME> --collection <DB_COLLECTION> --gzip --archive=<Name of the backup file that contains the file path>

   Table 1 Parameter description

   Parameter	Description
   <DB_HOST>	Database address
   <DB_PORT>	Database port
   <DB_USER>	Database username
   <AUTH_DB>	Database that stores <DB_USER> information. Generally, the value is admin.
   <FILE_PATH>	Path for storing the root certificate
   <DB_NAME>	The name of the database to be migrated.
   <DB_COLLECTION>	Collection in the database to be migrated

   Enter the database administrator password when prompted:

   Enter password:

   After the command is executed, the file specified by archive is the final backup file. The following command uses backup.tar.gz as an example.

   ./mongodump --host 192.168.xx.xx --port 8635 --authenticationDatabase admin -u rwuser --ssl --sslCAFile/tmp/ca.crt --sslAllowInvalidCertificates --db test --collection usertable --gzip --archive=backup.tar.gz

   2019-03-04T18:42:10.687+0800    writing admin.system.users to
   2019-03-04T18:42:10.688+0800    done dumping admin.system.users (1 document)
   2019-03-04T18:42:10.688+0800    writing admin.system.roles to
   2019-03-04T18:42:10.690+0800    done dumping admin.system.roles (0 documents)
   2019-03-04T18:42:10.690+0800    writing admin.system.version to
   2019-03-04T18:42:10.691+0800    done dumping admin.system.version (2 documents)
   2019-03-04T18:42:10.691+0800    writing test.test_collection to
   2019-03-04T18:42:10.691+0800    writing admin.system.profile to
   2019-03-04T18:42:10.692+0800    done dumping admin.system.profile (4 documents)
   2019-03-04T18:42:10.695+0800    done dumping test.test_collection (198 documents)

Importing Data

1. Log in to the ECS or whichever device you will be using to access DDS.
2. Upload the data to be imported to the ECS or the device.

   Select an uploading method based on the OS you are using.

   • In Linux, for example, you can use secure copy protocol (SCP):

     scp -r <IDENTITY_DIR> <REMOTE_USER>@<REMOTE_ADDRESS>:<REMOTE_DIR>

     Table 2 Parameter description

     Parameter	Description
     <IDENTITY_DIR>	Directory where the backup folder is located.
     <REMOTE_USER>	User of ECS OS in 1
     <REMOTE_ADDRESS>	IP address of the ECS in 1
     <REMOTE_DIR>	Directory of the ECS to be imported

   • In Windows, upload the backup directory to the ECS using a file transfer tool.

3. Import the backup data to DDS.

   An SSL connection is used in this example. If you use an unencrypted connection, delete --ssl --sslCAFile <FILE_PATH> --sslAllowInvalidCertificates from the following command.

   ./mongorestore --host <DB_HOST> --port <DB_PORT> --authenticationDatabase <AUTH_DB> -u<DB_USER> --ssl --sslCAFile <FILE_PATH> --sslAllowInvalidCertificates --db <DB_NAME> --collection <DB_COLLECTION> --gzip --archive=<Name of the backup file that contains the file path>

   Table 3 Parameter description

   Parameter	Description
   <DB_HOST>	DDS database address
   <DB_PORT>	Database port
   <AUTH_DB>	The database that authenticates DB_USER. Generally, the value is admin.
   <DB_USER>	Account name of the database administrator. The default value is rwuser.
   <FILE_PATH>	Path for storing the root certificate
   <DB_NAME>	The name of the database to be migrated.
   <DB_COLLECTION>	Collection in the database to be migrated

   Enter the database administrator password when prompted:

   Enter password:

   The following is an example:

   ./mongorestore --host 192.168.xx.xx --port 8635 --authenticationDatabase admin -u rwuser --ssl --sslCAFile/tmp/ca.crt --sslAllowInvalidCertificates --db test --collection usertable --gzip --archive=backup.tar.gz

   2019-03-05T14:19:43.240+0800    preparing collections to restore from
   2019-03-05T14:19:43.243+0800    reading metadata for test.test_collection from dump/test/test_collection.metadata.json
   2019-03-05T14:19:43.263+0800    restoring test.test_collection from dump/test/test_collection.bson
   2019-03-05T14:19:43.271+0800    restoring indexes for collection test.test_collection from metadata
   2019-03-05T14:19:43.273+0800    finished restoring test.test_collection (198 documents)
   2019-03-05T14:19:43.273+0800    restoring users from dump/admin/system.users.bson
   2019-03-05T14:19:43.305+0800    roles file 'dump/admin/system.roles.bson' is empty; skipping roles restoration
   2019-03-05T14:19:43.305+0800    restoring roles from dump/admin/system.roles.bson
   2019-03-05T14:19:43.333+0800    done

Related Issues

When you back up the entire instance using mongodump and mongorestore, the permission verification fails.

• Cause

  The rwuser user has limited permissions on the admin and config databases of the instance. As a result, the permission verification fails.

• Solution

  Grant permissions on certain databases and tables to the user.