Help Center/ DataArts Studio/ User Guide/ DataArts Migration (Real-Time Jobs)/ Tutorials/ Configuring a Job for Synchronizing Data from MongoDB to GaussDB(DWS)
Updated on 2025-08-05 GMT+08:00
View PDF

Configuring a Job for Synchronizing Data from MongoDB to GaussDB(DWS)

Supported Source and Destination Database Versions

Table 1 Supported database versions

Source Database

Destination Database

MongoDB database (version 4.0.0 and later)

GaussDB(DWS) cluster of version 8.1.3, 8.2.0, or a later version, except GaussDB(DWS) 3.0

Database Account Permissions

Before you use DataArts Migration for data synchronization, ensure that the source and destination database accounts meet the requirements in the following table. The required account permissions vary depending on the synchronization task type.

Table 2 Database account permissions

Type

Required Permissions

Source database account

The read and readwrite role of the destination database user has the permissions to grant the changeStream and find permissions to the destination collection.

Destination database account

The destination database account must have the following permissions for each table in the database: INSERT, SELECT, UPDATE, DELETE, CONNECT, and CREATE.

Supported Synchronization Objects

The following table lists the objects that can be synchronized using different links in DataArts Migration.

Table 3 Synchronization objects

Type

Note

Synchronization objects
  • The following DML operations can be synchronized: INSERT, UPDATE, and DELETE.
  • For the DDL operations that are not involved and cannot be synchronized, field mapping must be specified during synchronization.
  • Only primary key tables can be synchronized. The default primary key of MongoDB is _id.
  • Views, foreign keys, stored procedures, triggers, functions, events, virtual columns, unique constraints, unique indexes, foreign key indexes, and check constraints are not involved and cannot be synchronized.
  • Automatic table creation is not supported. You need to manually create tables at the destination.
  • Time to live (TTL) indexes cannot be synchronized.
  • Data in the admin and local databases cannot be synchronized or migrated.

Important Notes

In addition to the constraints on supported data sources and versions, connection account permissions, and synchronization objects, you also need to pay attention to the notes in the following table.

Table 4 Important notes

Type

Restriction

Database
  • The source database name must comply with the naming rules of open-source MongoDB.
    • Database name constraints:

      Do not distinguish databases by letter case. For example, you cannot name a database salesData and another database SalesData.

      When referencing a MongoDB database, you must use the same letter case. For example, if you want to reference the salesData database, do not use salesdata or SalesData.

      The name of a database running on Windows cannot contain any of the following characters: /\."$*<>:|?

      The name of a database running on Unix or Linux cannot contain any of the following characters: /\."$

      The database name can contain a maximum of 63 bytes.

    • Collection name constraints:

      A collection name must start with an underscore (_) or a letter.

      A collection name cannot contain null or a dollar sign ($), and cannot be an empty string (for example, "").

      A collection name cannot start with system.

      The name of an unsharded collection or view can contain a maximum of 255 bytes, and that of a sharded collection can contain a maximum of 235 bytes.

    • Field name constraints:

      A field name can contain a maximum of 255 bytes, and cannot contain null, periods (.), or dollar sign ($).

  • The name of an object in the destination database must contain 1 to 63 characters, start with a letter or underscore (_), and can contain letters, digits, underscores (_), and dollar signs ($).

Usage

General:

  • During real-time synchronization, the IP addresses, ports, accounts, and passwords cannot be changed.
  • Real-time MongoDB data synchronization does not support data sources with a single backup.
  • The MongoDB database name and collection name cannot be changed during migration.
  • Automatic table creation is not supported. You need to manually create a receiving table in GaussDB(DWS).
  • When setting field mapping, you can select the default source field extraColumn to determine a specified destination field and receive all the source MongoDB fields that have not been mapped to the destination in the job.
  • Field mapping can be customized in advance. During data synchronization, source fields are transferred if destination fields with the same names are detected and are not transferred if no destination fields with the same names are detected.
  • DML operations INSERT, UPDATE, and DELETE can be synchronized.

Full synchronization phase:

DataArts Migration traverses all chunks in the table. If the source table contains too many preset chunks, the full synchronization takes a long time.

Troubleshooting:

If any problem occurs during task creation, startup, full synchronization, incremental synchronization, or completion, rectify the fault by referring to .

Other

N/A

Procedure

This section uses real-time synchronization from MongoDB to GaussDB(DWS) as an example to describe how to configure a real-time data migration job. Before that, ensure that you have read the instructions described in Check Before Use and completed all the preparations.

  1. Create a real-time migration job by following the instructions in Creating a Real-Time Migration Job and go to the job configuration page.
  2. Select the data connection type. Select MongoDB for Source and DWS for Destination.

    Figure 1 Selecting the data connection type

  3. Select a job type. The default migration type is Real-time. The migration scenarios include Entire DB and Database/Table partition.

    Figure 2 Setting the migration job type

    For details about synchronization scenarios, see Synchronization Scenarios.

  4. Configure network resources. Select the created MongoDB and GaussDB(DWS) data connections and the migration resource group for which the network connection has been configured.

    Figure 3 Selecting data connections and a migration resource group

    If no data connection is available, click Create to go to the Manage Data Connections page of the Management Center console and click Create Data Connection to create a connection. For details, see Configuring DataArts Studio Data Connection Parameters.

  5. Configure source parameters.

    Select the databases and tables to be synchronized based on the following table.

    Table 5 Selecting the Kafka topics to be synchronized

    Synchronization Scenario

    Configuration Method

    Single table
    Select the MongoDB databases and tables to be migrated.
    Figure 4 Selecting databases and tables

  6. Configure destination parameters.

    • Set Database and Table Matching Policy.

      For details about the matching policy between source and destination databases and tables in each synchronization scenario, see the following table.

      Table 6 Database and table matching policy

      Synchronization Scenario

      Configuration Method

      Single table
      • Schema Matching Policy
        • Same name as the source database: Data will be synchronized to the GaussDB(DWS) schema with the same name as the source MongoDB database.
        • Custom: Data will be synchronized to the GaussDB(DWS) schema you specify.
      • Table Matching Policy
        • Same name as the source table: Data will be synchronized to the GaussDB(DWS) table with the same name as the source PostgreSQL table.
        • Custom: Data will be synchronized to the GaussDB(DWS) table you specify.
          Figure 5 Database and table matching policy in the entire database migration scenario
          NOTE:

          When you customize a matching policy, you can use built-in variables #{source_db_name} and #{source_table_name} to identify the source MySQL database name and table name. The table matching policy must contain #{source_table_name}.
    • Configure GaussDB(DWS) parameters.

      For details, see the following table.

      Figure 6 GaussDB(DWS) parameters
      Table 7 GaussDB(DWS) parameters

      Parameter

      Default Value

      Unit

      Description

      Write Mode

      UPSERT MODE

      N/A
      • UPSERT MODE: batch update
      • COPY MODE: DWS-dedicated high-performance batch import

      Maximum Data Volume for Batch Write

      50000

      Count

      Number of data records written to GaussDB(DWS) in a batch. You can adjust the value based on the table data size and job memory usage.

      Scheduled Batch Write Interval

      3

      Second

      Interval at which data is written to GaussDB(DWS)

      Advanced Settings

      N/A

      N/A

      Some advanced functions can be configured using parameters. For details, see Table 8.
      Table 8 GaussDB(DWS) advanced parameters

      Parameter

      Type

      Default Value

      Unit

      Description

      sink.buffer-flush.max-size

      int

      512

      MB

      Maximum number of bytes in each batch of data written to GaussDB(DWS). You can adjust the value based on the memory and data size configured for the job.

      sink.keyby.enable

      boolean

      true

      N/A

      Whether to enable data distribution. If this function is enabled in multi-concurrency scenarios, data can be distributed to different processes based on specific rules and written to the destination, which improves the write performance.

      sink.keyby.mode

      string

      table

      N/A

      Data distribution mode. The following modes are available:

      • pk: Data is distributed by primary key value.
      • table: Data is distributed by table name.
        NOTE:
        • In multi-concurrency scenarios, if DDL is enabled, data can be distributed only by table name. Otherwise, data may be inconsistent.
        • If there is no DDL, you can select pk, which improves the write performance in multi-concurrency scenarios.

      sink.field.name.case-sensitive

      boolean

      true

      N/A

      Whether to enable case sensitivity for data synchronization. If this function is enabled, the database names, table names, and field names are case sensitive during data synchronization.

      sink.verify.column-number

      boolean

      false

      N/A

      Whether to verify the number of data columns. By default, data is synchronized from MySQL to GaussDB(DWS) in the same-name mapping mode. The system does not check whether all columns are synchronized.

      If this function is enabled and the number of columns at the source is different from that at the destination, the system determines that data is inconsistent. As a result, the job is abnormal.

      sink.server.timezone

      string

      Local time zone

      N/A

      Session time zone specified for connecting to the destination database. The standard time zone format is supported, for example, UTC+08:00.

  7. Refresh and check the mapping between the source and destination tables. In addition, you can modify table attributes, add additional fields, and use the automatic table creation capability to create tables in the destination GaussDB(DWS) database.

    Figure 7 Mapping between source and destination tables
    • Assign values to fields: Click Assign Value to Destination Field in the Operation column to customize the field mapping from MongoDB to GaussDB(DWS). In addition, you can set the source table fields to be mapped to all the fields in the destination GaussDB(DWS) table, or set the values to be manually assigned to the fields.
      • Column Name: name of a field in the destination GaussDB(DWS) table
      • Type: type of the field in the destination GaussDB(DWS) table
      • Field Value: Value source of the field in the destination GaussDB(DWS) table
        Table 9 Method of obtaining a field value

        Type

        Example

        Manually assigned value

        Any character

        Source table field

        Preset source table field: name of a field that is obtained from the drop-down list or manually entered, and complies with the MongoDB field restrictions. For details, see the database restrictions in Table 4.

        extraColumns: special field name. If this field is used, all MongoDB source fields that have not been mapped will be written to this field and transmitted to GaussDB(DWS).
        • Automatic table creation is not supported if the source is MongoDB.
        • For details about the field type mapping for automatic table creation, see Field Type Mapping.

  8. Configure task parameters.

    Table 10 Task parameters

    Parameter

    Description

    Default Value

    Execution Memory

    Memory allocated for job execution, which automatically changes with the number of CPU cores

    8GB

    CPU Cores

    Value range: 2 to 32

    For each CPU core added, 4 GB execution memory and one concurrency are automatically added.

    2

    Maximum Concurrent Requests

    Maximum number of jobs that can be concurrently executed. This parameter does not need to be configured and automatically changes with the number of CPU cores.

    1

    Custom attributes

    You can add custom attributes to modify some job parameters and enable some advanced functions. For details, see Job Performance Optimization.

    -

  9. Submit and run the job.

    After configuring the job, click Submit in the upper left corner to submit the job.

    Figure 8 Submitting the job

    After submitting the job, click Start on the job development page. In the displayed dialog box, set required parameters and click OK.

    Figure 9 Starting the job
    Table 11 Parameters for starting the job

    Parameter

    Description

    Offset Parameter
    • Incremental synchronization: Incremental data synchronization starts from a specified time point.
    • Full and incremental synchronization: All data is synchronized first, and then incremental data is synchronized in real time.

    Time

    This parameter must be set for incremental synchronization, and it specifies the start time of incremental synchronization.

    NOTE:

    If you set a time that is earlier than the earliest binlog time, the latest log time is used.

  10. Pause and modify the migration job.

    You can pause a migration job, modify the job, and submit a job version again to resume synchronization.

    The following modification operations are supported:

    • Add or delete field mapping rules for destination field values.
    • Modify global parameters, such as the number of vCPUs and whether to enable automatic retry.
    • Modify the advanced properties of the data source.

    The following modification operations are not supported:

    • Change the existing field mapping rule for destination field values. For example, you can change the rule from source field assignment to manual assignment.
    • Change the source collection or destination table.

Parent topic: Tutorials