Updated on 2024-04-23 GMT+08:00

Kafka

Kafka is a distributed streaming platform developed by LinkedIn. It features high throughput and low latency for massive amounts of real-time data streaming. Kafka consists of producers, consumers, and brokers. Producers send data to the Kafka cluster; consumers subscribe to the cluster and process the data; brokers are the core component of a Kafka cluster as they are responsible for storing and forwarding messages. Featuring scalability, high throughput, low latency, reliability, and durability, Kafka is widely used for big data processing, real-time data streaming, and log collection.

Creating a Kafka Connection

  1. Log in to the new ROMA Connect console.
  2. In the navigation pane on the left, choose Connector. On the page displayed, click New Connection.
  3. Select the Kafka connector.
  4. In the dialog box displayed, configure the connector and click OK.

    Parameter

    Description

    Name

    Enter the connector instance name.

    Broker Address

    Enter the Kafka broker address.

    Authentication Mode

    Select the authentication mode of Kafka.

    • SSL
    • None

    SASL

    Mandatory when Authentication Mode is set to SSL.

    Select the SASL mechanism.

    • PLAIN
    • SCRAM-SHA-512

    SSL Username

    Mandatory when Authentication Mode is set to SSL.

    Enter the SSL username or application key.

    SSL Password

    Mandatory when Authentication Mode is set to SSL.

    Enter the SSL password.

    SSL Certificate Format

    Mandatory when Authentication Mode is set to SSL.

    Select the SSL certificate format.

    • PEM
    • JKS

    SSL Certificate

    Mandatory when Authentication Mode is set to SSL.

    Enter the Base64-encoded certificate content.

    SSL Certificate Password

    Mandatory when Authentication Mode is set to SSL.

    Enter the SSL certificate password.

    Description

    Enter the description of the connector to identify it.

Action

Consuming messages

Configuration Parameters

Parameter

Description

Topic

Topic to listen in on.

GroupId

Unique string to identify the group of consumer processes. Processes with the same group ID belong to the same consumer group.

KeyDeserializer

Deserialization mode of a key. Default: org.apache.kafka.common.serialization.StringDeserializer

ValueDeserializer

Deserialization mode of a value. Default: org.apache.kafka.common.serialization.StringDeserializer

AutoOffsetReset

Where to start consumption when there is no initial offset or the offset is out of range.

  • latest: Start consumption from the latest offset.
  • earliest: Start consumption from the earliest offset.
  • none: Report an error.