Help Center/ DataArts Studio/ User Guide/ DataArts Migration (Real-Time Jobs)/ Job Performance Optimization/ Optimizing the Parameters of a Job for Migrating Data from SQL Server to Doris
Updated on 2026-09-14 GMT+08:00
View PDF
Share

Optimizing the Parameters of a Job for Migrating Data from SQL Server to Doris

Optimizing Source Parameters

Optimize data extraction from SQL Server.

You can click Add Custom Attribute in the Configure Task area and add SQL Server synchronization parameters.

Figure 1 Adding custom attributes

The following tuning parameters are available.

Table 1 Tuning parameters for full data synchronization

Parameter

Type

Default Value

Description

scan.incremental.snapshot.backfill.skip

boolean

true

Whether to skip reading Binlog data. The default value is true. Skipping reading Binlog data can effectively reduce memory usage. Note that skipping reading Binlog data provides only at-least-once guarantee.
Table 2 Tuning parameters for incremental data synchronization

Parameter

Type

Default Value

Description

debezium.max.iteration.transactions

int

1000

Number of data records extracted from each table at a time during data replay. If the value of this parameter is large, the memory usage increases and incremental synchronization tasks are blocked.

Optimizing Destination Parameters

You can modify writing parameters in the Doris destination configuration or click View and Edit in the advanced configuration to add advanced attributes.

Figure 2 Adding advanced attributes
Table 3 Parameters for optimizing Doris writing

Parameter

Type

Default Value

Unit

Description

sink.properties.format

string

json

-

Data format used by Stream Load. The value can be json or csv. Using the CSV format and compression parameters can improve the write rate. However, the CSV format is not recommended for the following Doris versions: 1.2, 2.0.x (x < 14), 2.1.x (x < 6), and 3.0.x (x < 1). Open-source issues may cause write exceptions for special characters if the CSV format is used.

sink.properties.Content-Encoding

string

-

-

Compression format of the HTTP header message body. Currently, only CSV files can be compressed, and the .gzip format is supported.

sink.properties.compress_type

string

-

-

File compression format. Currently, only CSV files can be compressed. The .gz, .lzo, .bz2, .lz4, .lzop, and .deflate compression formats are supported.

doris.sink.flush.tasks

int

1

-

Number of concurrent flushes of a single TaskManager. You can increase the value of this parameter to improve the write rate when there are sufficient resources.

sink.batch.interval

string

1s

h/min/s

Interval at which an asynchronous thread writes data. You can increase the value of this parameter to reduce the database I/O if there is a large amount of data at the source. For example, you can increase the value to 30s.

sink.batch.size

int

20000

-

Maximum number of rows that can be written (inserted, updated, or deleted) at a time. You can increase the value of this parameter to reduce the database I/O if there is a large amount of data at the source. For example, you can increase the value to 50000.

sink.batch.bytes

int

10485760

bytes

Maximum number of bytes that can be written (inserted, updated, or deleted) at a time. You can increase the value of this parameter to reduce the database I/O if there is a large amount of data at the source. For example, you can increase the value to 50485760.