Help Center/ DataArts Studio/ User Guide/ DataArts Migration (Offline Jobs)/ Configuring Connection Properties/ Configuring Kafka Connection Properties
Updated on 2026-05-20 GMT+08:00
View PDF
Share

Configuring Kafka Connection Properties

When using DataArts Migration, you can configure properties of the JDBC connection to the Kafka data source to optimize the data migration process and resolve possible problems.

This section describes the connection properties. You can click Add to add client properties of the connection to a specified data source and configure the properties by referring to the documentation of the corresponding Kafka client.

Common Connection Properties

Table 1 Producer client configuration

Property

Type

Description

batch.size

int (unit: byte)

Size of the batch sent by the producer. Increasing the batch size improves the throughput but increases the delay. Example: batch.size=32768.

buffer.memory

long (unit: byte)

Producer buffer size. Increasing the buffer size improves the throughput but requires more memory. Example: buffer.memory=67108864.

compression.type

String

Message compression type. The value can be none, gzip, snappy, lz4, or zstd. Compression reduces network transmission and storage costs but increases CPU usage. Example: compression.type=gzip.

retries

int

Number of retries allowed when a producer encounters an error. Increasing the number of retries improves reliability, but may cause duplicate messages. Example: retries=3.
Table 2 Consumer client configuration

Property

Type

Description

session.timeout.ms

int (unit: ms)

Heartbeat timeout interval between the consumer and coordinator. Example: session.timeout.ms=30000.

max.poll.records

int

Maximum number of records returned each time the poll() method is called. Increasing this value improves the throughput but increases memory usage. Example: max.poll.records=1000.

fetch.min.bytes

int (unit: byte)

Minimum amount of data pulled from the server each time. Increasing this value reduces the number of network requests but may increase the delay. Example: fetch.min.bytes=1024.

fetch.max.wait.ms

int (unit: ms)

Maximum time that the server waits for data before returning a response. Example: fetch.max.wait.ms=1000.

Configuration Method

  1. On the Management Center page, locate the Kafka data source configuration.
  2. In the connection property configuration area, click Add to add properties for the connection.
  3. Enter property names and values as needed based on the descriptions in the preceding table.
  4. Save the change and test the connection to ensure that the configuration takes effect and data can be migrated properly.

Precautions

  • When configuring connection properties, ensure that you understand the meaning and function of each property to avoid data migration failures or other issues due to incorrect configurations.
  • If you have any questions during the configuration, refer to the official Kafka documentation for more information and instructions.
  • You are advised to adjust and optimize connection properties based on the data migration scenarios and requirements to achieve optimal migration performance.

By properly configuring Kafka connection properties, you can improve data migration stability and efficiency for smooth data integration.