From Kafka/DMS Kafka

Basic parameters

Topics

One or more topics can be entered.

est1,est2

Data Format

Format used for parsing data. The options are as follows:

• Binary: Data is transferred directly. It is not converted to another format. This setting is suitable for file migration.
• CSV: Source data will be migrated after being converted in CSV format.
• JSON: Source data will be migrated after being converted in JSON format.
• CDC (DRS): Source data will be migrated after being converted in DRS format.
• CDC (JSON): Source data will be migrated after being converted in JSON format.
• CDC (DRS_AVRO): Source data will be migrated after being converted in DRS_AVRO format.
• CDC (DRS_JSON): Source data will be migrated after being converted in DRS_JSON format.

Binary

Offset

Initial offset parameter

• Latest: Maximum offset, indicating that the latest data will be extracted.
• Earliest: Minimum offset, indicating that the earliest data will be extracted.
• Submitted: data that has been submitted
• Time Range: data within a specified time range

Latest

Data Extraction Timeout Duration

Maximum duration (minutes) of data extraction. For example, a job scheduled daily needs a sufficient duration to extract the data generated by the topic every day.

60

Suspension Period

If the value is set to 60 and no data is returned within 60s after the consumer requests data extraction from Kafka (generally because all the data in the topic has been read or the network or Kafka cluster is unavailable), the task will stop immediately. Otherwise, the system will retry reading data.

60

Consumer Group ID

Consumer group ID

If you export data from DMS Kafka, enter any value for Kafka Platinum but a valid consumer group ID for Kafka Basic.

sumer-group

Start Time

This parameter is required when Offset is set to Time Range. It specifies the start time for pulling data, including the data at the specified time point.

2020-12-20 12:00:00

End Time

This parameter is required when Offset is set to Time Range. It specifies the end time for pulling data, excluding the data at the specified time point.

2020-12-20 20:00:00

Field Delimiter

This parameter is required when Data Format is set to CSV. The default value is space. To set the Tab key as the delimiter, set this parameter to \t.

,

Record Delimiter

This parameter is required when Data Format is set to CSV or JSON. The default value is space. To set the Tab key as the delimiter, set this parameter to \t.

,

Advanced parameters

UseConfigFile

This parameter is required when Data Format is set to CDC. It is used to configure OBS files.

No

OBS Link

Select an OBS link.

obs_link

OBS Bucket

Select an OBS bucket.

obs_test

Config File

Select the OBS configuration file.

/obs/config.csv

Max. Poll Records

(Optional) Maximum number of records per poll

100

Max. Poll Interval

(Optional) Maximum interval between polls (seconds)

100

Notice Topic

Topic for sending notification data. If the data format is CDC, the notification content is the names of the generated files.

notice