Updated on 2023-12-18 GMT+08:00

From DIS

The data in the message body is a record in CSV format that supports multiple delimiters. Messages cannot be parsed in binary or other formats.

If the source link of a job is a DIS link, configure the source job parameters based on Table 1.

Table 1 Parameter description

Category

Parameter

Description

Example Value

Basic parameters

DIS Stream

DIS stream name

dis

Permanent Running

Whether a job runs permanently. If a job is set to run for a long time, the job will fail if the DIS system is interrupted.

Yes

DIS Partition ID

ID of the DIS partition. You can enter multiple partition IDs separated by commas (,).

0,1,2

Offset

Initial offset when data is pulled from DIS
  • Latest: Maximum offset, indicating that the latest data will be extracted.
  • From last stop: Data read will start from which the last read ended.
  • Earliest: Minimum offset, indicating that the earliest data will be extracted.

Latest

Application Name

Unique identifier of the consumer application to be used. If no application exists, CDM creates one automatically.

cdm

Data Format

Format used for parsing data. The options are as follows:

  • Binary: Data is transferred directly. It is not converted to another format. This setting is suitable for file migration.
  • CSV: Source data will be migrated after being converted in CSV format.
  • JSON: Source data will be migrated after being converted in JSON format.

Binary

Field Delimiter

This parameter is displayed when Data Format is set to CSV. The default value is comma (,). To set the Tab key as the delimiter, set this parameter to \t.

,

Record Delimiter

This parameter is displayed when Data Format is set to CSV or JSON. It is used to separate each two records.

,

Advanced attributes

Max. Poll Records

(Optional) Maximum number of records per poll

100