From Kafka/DMS Kafka
If the source link of a job is a Kafka link or DMS Kafka link, configure the source job parameters based on Table 1.
Type |
Parameter |
Description |
Example Value |
---|---|---|---|
Basic parameters |
Topics |
One or more topics can be entered. |
est1,est2 |
Data Format |
Format used for parsing data. The options are as follows:
|
Binary |
|
Offset |
Initial offset parameter
|
Latest |
|
Data Extraction Timeout Duration |
Maximum duration (minutes) of data extraction. For example, a job scheduled daily needs a sufficient duration to extract the data generated by the topic every day. |
60 |
|
Suspension Period |
If the value is set to 60 and no data is returned within 60s after the consumer requests data extraction from Kafka (generally because all the data in the topic has been read or the network or Kafka cluster is unavailable), the task will stop immediately. Otherwise, the system will retry reading data. |
60 |
|
Consumer Group ID |
Consumer group ID If you export data from DMS Kafka, enter any value for Kafka Platinum but a valid consumer group ID for Kafka Basic. |
sumer-group |
|
Start Time |
This parameter is required when Offset is set to Time Range. It specifies the start time for pulling data, including the data at the specified time point. |
2020-12-20 12:00:00 |
|
End Time |
This parameter is required when Offset is set to Time Range. It specifies the end time for pulling data, excluding the data at the specified time point. |
2020-12-20 20:00:00 |
|
Field Delimiter |
This parameter is required when Data Format is set to CSV. The default value is space. To set the Tab key as the delimiter, set this parameter to \t. |
, |
|
Record Delimiter |
This parameter is required when Data Format is set to CSV or JSON. The default value is space. To set the Tab key as the delimiter, set this parameter to \t. |
, |
|
Advanced parameters |
UseConfigFile |
This parameter is required when Data Format is set to CDC. It is used to configure OBS files. |
No |
OBS Link |
Select an OBS link. |
obs_link |
|
OBS Bucket |
Select an OBS bucket. |
obs_test |
|
Config File |
Select the OBS configuration file. |
/obs/config.csv |
|
Max. Poll Records |
(Optional) Maximum number of records per poll |
100 |
|
Max. Poll Interval |
(Optional) Maximum interval between polls (seconds) |
100 |
|
Notice Topic |
Topic for sending notification data. If the data format is CDC, the notification content is the names of the generated files. |
notice |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot