Basic Concepts

Topic

A topic is a category for messages. Messages are created, retrieved, and managed in the form of topics.

Topics adopt the publish-subscribe pattern. Producers publish messages into topics. One or more consumers subscribe to the messages in the topics. The producers and consumers are not directly linked to each other.

Producer

A producer publishes messages into topics. The messages are then delivered to other systems or modules for processing as agreed.

Consumer

A consumer subscribes to messages in topics and processes the messages. For example, a monitoring and alarm platform (a consumer) subscribing to log messages in certain topics can identify alarm logs and then send SMS or email alarm notifications.

Consumer Group

A consumer group is a logical collection of consumers. Kafka uses consumer groups to balance messages and manage the consumer offset.

Broker

A broker is a Kafka process in a Kafka cluster. Each process runs on a server, so a broker includes the storage, bandwidth, and other server resources.

Partition

A topic is divided into partitions. Messages are distributed to multiple partitions to achieve scalability and fault tolerance.

Replica

A replica is a redundant copy of a partition in a topic. Each partition can have one or more replicas, enabling message reliability.

Messages in each partition are fully replicated and synchronized, preventing data loss if one replica fails.

Each partition has one replica as the leader which handles the creation and retrievals of all messages. The rest replicas are followers which replicate the leader.

Aging Time

The period that messages are retained for. Consumers must retrieve messages before this period ends. Otherwise, the messages will be deleted and can no longer be retrieved.

Cold Read

In Kafka, cold read refers to the scenario where the target data is not in the OS page cache when consumers pull messages or synchronize replicas. As a result, physical disk I/O is triggered. Fundamentally, the cache miss makes read requests read from the bottom-layer disk. In cold read scenarios, the read latency increases due to disk I/O operations. Moreover, the increase easily triggers disk bandwidth or IOPS bottlenecks, causing overload. As a result, the instance throughput is limited and the latency increases.

Cold read usually occurs in the following scenarios: Messages are stacked due to a consumer, services backtrack historical data, new consumers read partition messages for the first time, the cache is invalidated due to instance restart, and historical data is retrieved when replica synchronization is delayed.