Kafka

Updated on 2024-04-23 GMT+08:00

View PDF

Kafka is a distributed streaming platform developed by LinkedIn. It features high throughput and low latency for massive amounts of real-time data streaming. Kafka consists of producers, consumers, and brokers. Producers send data to the Kafka cluster; consumers subscribe to the cluster and process the data; brokers are the core component of a Kafka cluster as they are responsible for storing and forwarding messages. Featuring scalability, high throughput, low latency, reliability, and durability, Kafka is widely used for big data processing, real-time data streaming, and log collection.

Creating a Kafka Connection

Log in to the new ROMA Connect console.
In the navigation pane on the left, choose Connector. On the page displayed, click New Connection.
Select the Kafka connector.

In the dialog box displayed, configure the connector and click OK.

Parameter	Description
Name	Enter the connector instance name.
Broker Address	Enter the Kafka broker address.
Authentication Mode	Select the authentication mode of Kafka. SSL None
SASL	Mandatory when Authentication Mode is set to SSL. Select the SASL mechanism. PLAIN SCRAM-SHA-512
SSL Username	Mandatory when Authentication Mode is set to SSL. Enter the SSL username or application key.
SSL Password	Mandatory when Authentication Mode is set to SSL. Enter the SSL password.
SSL Certificate Format	Mandatory when Authentication Mode is set to SSL. Select the SSL certificate format. PEM JKS
SSL Certificate	Mandatory when Authentication Mode is set to SSL. Enter the Base64-encoded certificate content.
SSL Certificate Password	Mandatory when Authentication Mode is set to SSL. Enter the SSL certificate password.
Description	Enter the description of the connector to identify it.

Action

Consuming messages

Configuration Parameters

Parameter	Description
Topic	Topic to listen in on.
GroupId	Unique string to identify the group of consumer processes. Processes with the same group ID belong to the same consumer group.
KeyDeserializer	Deserialization mode of a key. Default: org.apache.kafka.common.serialization.StringDeserializer
ValueDeserializer	Deserialization mode of a value. Default: org.apache.kafka.common.serialization.StringDeserializer
AutoOffsetReset	Where to start consumption when there is no initial offset or the offset is out of range. latest: Start consumption from the latest offset. earliest: Start consumption from the earliest offset. none: Report an error.