Kafka

Kafka is a distributed streaming platform developed by LinkedIn. It features high throughput and low latency for massive amounts of real-time data streaming. Kafka consists of producers, consumers, and brokers. Producers send data to the Kafka cluster; consumers subscribe to the cluster and process the data; brokers are the core component of a Kafka cluster as they are responsible for storing and forwarding messages. Featuring scalability, high throughput, low latency, reliability, and durability, Kafka is widely used for big data processing, real-time data streaming, and log collection.

Creating a Kafka Connection

Log in to the new ROMA Connect console.
In the navigation pane on the left, choose Connector. On the page displayed, click New Connection.
Select the Kafka connector.

In the dialog box displayed, configure the connector and click OK.

Parameter	Description
Name	Enter the connector instance name.
Broker Address	Enter the Kafka broker address.
Authentication Mode	Select the authentication mode of Kafka. SSL None
SASL	Mandatory when Authentication Mode is set to SSL. Select the SASL mechanism. PLAIN SCRAM-SHA-512
SSL Username	Mandatory when Authentication Mode is set to SSL. Enter the SSL username or application key.
SSL Password	Mandatory when Authentication Mode is set to SSL. Enter the SSL password.
SSL Certificate Format	Mandatory when Authentication Mode is set to SSL. Select the SSL certificate format. PEM JKS
SSL Certificate	Mandatory when Authentication Mode is set to SSL. Enter the Base64-encoded certificate content.
SSL Certificate Password	Mandatory when Authentication Mode is set to SSL. Enter the SSL certificate password.
Description	Enter the description of the connector to identify it.

Action

Writing messages

Configuration Parameters

**Table 1** Sending messages
Parameter	Description
Topic	Enter the topic to be listened.
Partition	Partition to which the message will be written.
Key	Key value of the message.
Message	Message content.
Acks	Producer acknowledgement. all -1 0 1
KeySerializer	Method of turning the key into bytes. The default value is org.apache.kafka.common.serialization.StringSerializer.
ValueSerializer	Method of turning the value into bytes. The default value is org.apache.kafka.common.serialization.StringSerializer.
BufferMemorySize	Total bytes of memory the producer can use to buffer records waiting to be sent to the server. The default value is 33554432.
Retries	Number of retries.