Configuring a Job for Synchronizing Data from PostgreSQL to Doris (Internal Beta)

• Click Source Configuration. The system will test the connectivity of the entire migration job.
• Click Test in the source and destination and migration resource group.
  

  If the network connectivity is abnormal, see How Do I Troubleshoot a Network Disconnection Between the Data Source and Resource Group?

Select the PostgreSQL databases and tables to be migrated.

Figure 4 Selecting databases and tables

Both databases and tables can be customized. You can select one database and one table, or multiple databases and tables.

• Set Database and Table Matching Policy.
  • Database Matching Policy
    • Same name as the source database: Data will be synchronized to the Doris database with the same name as the source PostgreSQL database.
    • Custom: Data will be synchronized to the Doris database you specify.
  • Table Matching Policy
    • Same name as the source table: Data will be synchronized to the Doris table with the same name as the source PostgreSQL table.
    • Custom: Data will be synchronized to the Doris table you specify.
      Figure 5 Database and table matching policy in the entire database migration scenario
      
      

      When you customize a matching policy, you can use built-in variables #{source_db_name} and #{source_table_name} to identify the source database name and table name. The table matching policy must contain #{source_table_name}.

• Set Doris parameters.
  You can configure the advanced parameters in the following table to enable some advanced functions.

  Table 5 Doris advanced parameters

  Parameter	Type	Default Value	Unit	Description
  doris.request.connect.timeout.ms	int	30000	ms	Doris connection timeout interval
  doris.request.read.timeout.ms	int	30000	ms	Doris read timeout interval
  doris.request.retries	int	3	-	Number of retries upon a Doris request failure
  sink.max-retries	int	3	-	Maximum number of retries upon a data writing failure
  sink.batch.interval	string	1s	h/min/s	Interval at which an asynchronous thread writes data
  sink.enable-delete	boolean	true	-	Whether to enable the deletion function. If this function is disabled, data deleted from the source will not be deleted from the destination.
  sink.batch.size	int	20000	-	Maximum number of rows that can be written (inserted, updated, or deleted) at a time
  sink.batch.bytes	long	10485760	bytes	Maximum number of bytes that can be written (inserted, updated, or deleted) at a time
  logical.delete.enabled	boolean	false	-	Whether to enable logical deletion. If this function is enabled, the destination must contain the deletion flag column. When data is deleted from the source database, the corresponding data in the destination database will not be deleted. Instead, the deletion flag column is set to true, indicating that the data is not contained at the source.
  logical.delete.column	string	logical_is_deleted	-	Name of the logical deletion column. The default value is logical_is_deleted. You can customize the value.
  sink.keyby.mode	string	pk	-	Partitioning mode when concurrent writes are performed on Doris. Default value: pk (primary key). If the source is Kafka, and it does not have a primary key, select table to partition data by table name.
  doris.sink.flush.tasks	int	1	-	Number of concurrent flushes of a single TaskManager
  sink.properties.format	string	json	-	Data format used by Stream Load. The value can be json or csv.
  sink.properties.Content-Encoding	string	-	-	Compression format of the HTTP header message body. Currently, only CSV files can be compressed, and the .gzip format is supported.
  sink.properties.compress_type	string	-	-	File compression format. Currently, only CSV files can be compressed. The .gz, .lzo, .bz2, .lz4, .lzop, and .deflate compression formats are supported.

Figure 6 Mapping between source and destination tables

• Edit additional fields: Click Additional Field in the Operation column to add custom fields to the destination Doris table. For a new table, you can add additional fields to the existing fields in the source table. You can customize the field name, select the field type, and enter the field value.
  • Field Name: name of the new field in the destination Doris table
  • Field Type: type of the new field in the destination Doris table
  • (Optional) Field Type Length: length of the new field type in the destination Doris table
  • Field Value: value source of the new field in the destination Doris table

    Table 6 Additional field value obtaining mode

    Type	Example
    Constant	Digits, letters, and special characters are supported. Color emoticons may cause a job submission failure.

• Automatic table creation: Click Auto Table Creation to automatically create tables in the destination database based on the configured mapping policy. After the tables are created, Existing table is displayed for them.
  

  • DataArts Migration supports only automatic table creation. You need to manually create databases and schemas at the destination before using this function.
  • For details about the field type mapping for automatic table creation, see Field Type Mapping.

After configuring the job, click Submit in the upper left corner to submit the job.

Figure 7 Submitting the job

After submitting the job, click Start on the job development page. In the displayed dialog box, set required parameters and click OK.

Figure 8 Starting the job

On the job development page, click Monitor to go to the Job Monitoring page. You can view the status and log of the job, and configure alarm rules for the job. For details, see Real-Time Migration Job O&M.

Figure 9 Monitoring the job