To MRS Kafka

Basic parameters

Topic

Topic database name

default

Data Format

Format of the data written to the database

CSV: Columns are concatenated based on field separators.

JSON: All columns are concatenated into a JSON string based on specified field names.

JSON

Field Delimiter

This parameter is available when Data Format is set to CSV.

Field delimiter between data written to the destination. The default value is a space.

,

keyIndex

This parameter is available when Data Format is set to CSV.

Key column in Kafka Writer. After this parameter is set, this column is not recorded in the value. For example, if the subscripts of a field column are 0, 1, and 2, and the value of keyIndex is 0, the values of valueIndex are 1 and 2. The subscript of keyIndex must be a positive integer starting from 0. Otherwise, an error is reported during task execution.

-

Additional Configuration

This parameter is available when Data Format is set to JSON.

This parameter specifies different types of formats of the data to be written and configuration parameters.

To use this function, you must set the configType parameter to COMBINE_DATA first.

When configType is set to COMBINE_DATA, the following parameters are supported:

• batchnum: combines multiple pieces of data into one. The default value is 1.
• featureTag: adds a tag to each piece of data.
• startEndMark: The default value is false. If this parameter is set to true, a start message and an end message are synchronized before a message is written.
• columnAsKey: key value of the data to be written. You can also specify a field value as the key by configuring @{column1}--@{column2}.

  For example, if the destination fields are id and name, set this parameter to @{id}--@{name}.

• schema: This parameter is displayed in the message body of the written data. If this parameter is set, the configured value is displayed. If this parameter is not set, the schema value of the source table is used by default.
• table: This parameter is displayed in the message body of the written data. If this parameter is set, the configured value is displayed. If this parameter is not set, the name of the source table is used by default.
• acks: The value can be 0, 1, or all.
• jobId: The default value is 0. If this parameter is set, its value is generated in the message.

For example, to tag data, you need to set configType to COMBINE_DATA and featureTag to group.