Help Center/ Distributed Message Service for Kafka/ Best Practices/ Configuring Kafka Message Compression
Updated on 2025-11-18 GMT+08:00
View PDF
Share

Configuring Kafka Message Compression

Overview

Compressing Kafka messages during storage and transmission optimizes performance and reduces resource overhead. Kafka supports compression algorithms such as GZIP, SNAPPY, and LZ4. These algorithms compress messages before they are written to disks, reducing storage space usage and bandwidth consumption.

Compression principle:

Both disk storage and network transmission can be optimized and the throughput can be improved when the Kafka compression algorithms are configured on the producer client. When they are configured on the server, its CPU and network bandwidth consumption increase, causing performance bottlenecks and reducing throughput.

To configure the compression algorithms on the producer client:

  1. The producer client compresses messages: Multiple messages are compressed into one data packet and then sent to the server over a network.
  2. The server stores messages: Data packets are stored to disks without being decompressed.
  3. The consumer client decompresses messages: Data packets are pulled from the broker and decompressed based on their compression algorithm identifiers. The original messages are obtained.

Comparing compression algorithms:

Kafka compression algorithms differ greatly in core metrics, as described in Table 1.

Table 1 Compression algorithm core metrics

Algorithm

Compression Ratio

Compression/Decompression Speed

CPU Consumption

Applicable Scenario

GZIP

High

Slow

High

Highest compression ratio, tolerant to high latency, CPU consumption, and network transmission costs.

SNAPPY

Medium

Quick

Low

General preferred. Compression ratio and compression/decompression speed are well balanced. Low CPU overhead and stable performance.

LZ4

Relatively low

Very quick

Very low

Highest compression/decompression speed and minimum CPU overhead.

Implementation

Configure the compression.type parameter on producer clients to specify a compression algorithm.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
// Enable GZIP.
props.put("compression.type", "gzip");
 
Producer<String, String> producer = new KafkaProducer<>(props);