MongoDB

Preparation and Constraints

• Network requirements

  The MongoDB data source can communicate with CDM. This ensures smooth data transmission. For details, see Enabling Network Connectivity.

• Required permissions
  • Read permission: Assign the read built-in role so that DataArts Migration can read data from a destination MongoDB. This role allows users to perform read operations, such as find, aggregate, listCollections, and listIndexes.
  • Write permission: Assign the readWrite built-in role so that DataArts Migration can write data to a destination MongoDB. This role has all the permissions of the read role as well as the permissions to perform write and structure change operations, such as insert, update, remove, createCollection, and createIndex.
• Enabling ports

  MongoDB port (27017): TCP 27017 and the listening ports of the replica set or sharded cluster must be enabled so that DataArts Migration can access MongoDB.

Supported Data Types

Supported Migration Scenarios

DataArts Migration supports the following modes for synchronizing on-premises data:

• Single table synchronization

  DataArts Migration supports table/file synchronization in data ingestion into a data lake or data migration to the cloud.

• Database and table shard synchronization

  DataArts Migration supports synchronization of data from multiple databases and tables in data ingestion into a data lake or data migration to the cloud.

• Entire DB migration

  DataArts Migrations supports synchronization of data from an on-premises database in data ingestion into a data lake or data migration to the cloud.

Database and table shard synchronization and entire DB migration are not supported in all regions. The following table lists the supported MongoDB migration scenarios.

Core Capabilities

• Connection configuration

  Configuration Item	Supported	Description
  Support for Secure Shell (SSL)	x	SSL encryption ensures secure data transmission. Currently, this function is not supported.

• Read capabilities

  Configuration Item	Supported	Description
  Incremental read	√	The find condition can be configured to enable incremental read.
  Shard concurrency	x	Concurrent read can fully utilize resources and improve read performance. MongoDB links do not support this function.
  Nested document parsing	√	Nested document data structures can be parsed. Document data with multiple layers of nested fields can be correctly processed, ensuring data integrity and accuracy.
  Custom fields	√	You can add computed columns, constant columns, or masking functions for tasks to meet personalized service requirements.
  Dirty data processing	√	Abnormal data can be written to the dirty data bucket to prevent job failures caused by a small amount of abnormal data.

• Write capabilities

  Configuration Item	Supported	Description
  Data clearance before import	√	Operations such as preSql can clean and process data before data import.
  Conflict resolution	√	Multiple write modes, such as addition and replacement, can flexibly handle data conflicts.
  Concurrent write	√	Concurrent write improves efficiency.
  Dirty data processing	√	Abnormal data can be written to the dirty data bucket to prevent job failures caused by a small amount of abnormal data.

Creating a Data Source

Create a data source in Management Center. For details, see Configuring Data Connection Parameters.

Creating an Offline Data Migration Job

Create a MongoDB migration job in DataArts Factory. For details, see Creating an Offline Processing Migration Job.