Configuring a Job for Synchronizing Data from MongoDB to GaussDB(DWS)

• Click Source Configuration. The system will test the connectivity of the entire migration job.
• Click Test in the source and destination and resource group.
  NOTE:

  If the network connectivity is abnormal, see How Do I Troubleshoot the Disconnectivity Between a Data Source and Resource Group?

Select the databases and tables to be synchronized based on the following table.

Table 5 Selecting the Kafka topics to be synchronized

Synchronization Scenario	Configuration Method
Single table	Select the MongoDB databases and tables to be migrated. Figure 4 Selecting databases and tables

• Set Database and Table Matching Policy.

  For details about the matching policy between source and destination databases and tables in each synchronization scenario, see the following table.

  Table 6 Database and table matching policy

  Synchronization Scenario

  Configuration Method

  Single table

  Schema Matching Policy Same name as the source database: Data will be synchronized to the GaussDB(DWS) schema with the same name as the source MongoDB database. Custom: Data will be synchronized to the GaussDB(DWS) schema you specify. Table Matching Policy Same name as the source table: Data will be synchronized to the GaussDB(DWS) table with the same name as the source PostgreSQL table. Custom: Data will be synchronized to the GaussDB(DWS) table you specify. Figure 5 Database and table matching policy in the entire database migration scenario NOTE: When you customize a matching policy, you can use built-in variables #{source_db_name} and #{source_table_name} to identify the source MySQL database name and table name. The table matching policy must contain #{source_table_name}.

• Configure GaussDB(DWS) parameters.

  For details, see the following table.

  Figure 6 GaussDB(DWS) parameters
  

  Table 7 GaussDB(DWS) parameters

  Parameter	Default Value	Unit	Description
  Write Mode	UPSERT MODE	N/A	UPSERT MODE: batch update COPY MODE: DWS-dedicated high-performance batch import
  Maximum Data Volume for Batch Write	50000	Count	Number of data records written to GaussDB(DWS) in a batch. You can adjust the value based on the table data size and job memory usage.
  Scheduled Batch Write Interval	3	Second	Interval at which data is written to GaussDB(DWS)
  Advanced Settings	N/A	N/A	Some advanced functions can be configured using parameters. For details, see Table 8.

  Table 8 GaussDB(DWS) advanced parameters

  Parameter	Type	Default Value	Unit	Description
  sink.buffer-flush.max-size	int	512	MB	Maximum number of bytes in each batch of data written to GaussDB(DWS). You can adjust the value based on the memory and data size configured for the job.
  sink.keyby.enable	boolean	true	N/A	Whether to enable data distribution. If this function is enabled in multi-concurrency scenarios, data can be distributed to different processes based on specific rules and written to the destination, which improves the write performance.
  sink.keyby.mode	string	table	N/A	Data distribution mode. The following modes are available: pk: Data is distributed by primary key value. table: Data is distributed by table name. NOTE: In multi-concurrency scenarios, if DDL is enabled, data can be distributed only by table name. Otherwise, data may be inconsistent. If there is no DDL, you can select pk, which improves the write performance in multi-concurrency scenarios.
  sink.field.name.case-sensitive	boolean	true	N/A	Whether to enable case sensitivity for data synchronization. If this function is enabled, the database names, table names, and field names are case sensitive during data synchronization.
  sink.verify.column-number	boolean	false	N/A	Whether to verify the number of data columns. By default, data is synchronized from MySQL to GaussDB(DWS) in the same-name mapping mode. The system does not check whether all columns are synchronized. If this function is enabled and the number of columns at the source is different from that at the destination, the system determines that data is inconsistent. As a result, the job is abnormal.
  sink.server.timezone	string	Local time zone	N/A	Session time zone specified for connecting to the destination database. The standard time zone format is supported, for example, UTC+08:00.

Figure 7 Mapping between source and destination tables

• Assign values to fields: Click Assign Value to Destination Field in the Operation column to customize the field mapping from MongoDB to GaussDB(DWS). In addition, you can set the source table fields to be mapped to all the fields in the destination GaussDB(DWS) table, or set the values to be manually assigned to the fields.
  • Column Name: name of a field in the destination GaussDB(DWS) table
  • Type: type of the field in the destination GaussDB(DWS) table
  • Field Value: Value source of the field in the destination GaussDB(DWS) table

    Table 9 Method of obtaining a field value

    Type	Example
    Manually assigned value	Any character
    Source table field	Preset source table field: name of a field that is obtained from the drop-down list or manually entered, and complies with the MongoDB field restrictions. For details, see the database restrictions in Table 4. extraColumns: special field name. If this field is used, all MongoDB source fields that have not been mapped will be written to this field and transmitted to GaussDB(DWS).

    NOTE:

    • Automatic table creation is not supported if the source is MongoDB.
    • For details about the field type mapping for automatic table creation, see Field Type Mapping.