Updated on 2025-08-21 GMT+08:00

Kafka Production Rate and CPU Usage

This section describes performance tests on Distributed Message Service (DMS) for Kafka. The performance is measured by the message production rate on the client side and CPU usage on the server side. The tests cover the following scenarios:

  • Scenario 1 (batch size): same Kafka instance, same topics, different message size settings
  • Scenario 2 (cross-AZ or intra-AZ production): same Kafka instance, same topics, different AZ settings for the client and server
  • Scenario 3 (number of replicas): same Kafka instance, different numbers of replicas
  • Scenario 4 (synchronous or asynchronous replication): same Kafka instance, topics with different replication settings
Table 1 Test parameters

Partitions

Replicas

Synchronous Replication

batch.size

Cross-AZ Production

3

1

No

1 KB

No

3

1

No

16 KB

No

3

1

No

1 KB

Yes

3

3

Yes

1 KB

No

3

3

No

1 KB

No

Environment

Perform the following steps to set up the test environment.

  1. Purchase a Kafka instance with parameters specified as follows and retain the default settings for other ones. For details about how to purchase one, see Buying a Kafka Instance.
    Table 2

    Parameter

    Example Value

    Region

    CN-Hong Kong

    AZ

    AZ1

    Version

    2.7

    Architecture

    Cluster

    Broker Flavor

    kafka.2u4g.cluster

    Brokers

    3

    Storage Space per Broker

    Ultra-high I/O, 200 GB

    VPC

    Select a VPC.

    Subnet

    Select a subnet.

    Security Group

    Select a security group.

    Access Mode

    Default value

    Instance Name

    kafka-test

    Enterprise Project

    default

    After the purchase, obtain Address (Private Network, Plaintext) on the instance details page.

  2. Create three topics with parameters specified as follows for the purchased Kafka instance. For details, see Creating a Kafka Topic.
    • Topic-01: 3 partitions, 1 replica, asynchronous replication
    • Topic-02: 3 partitions, 3 replicas, asynchronous replication
    • Topic-03: 3 partitions, 3 replicas, synchronous replication
  3. Obtain the test tool.

    Obtain Kafka CLI 2.7.2.

  4. Purchase a server for the client.
    Buy two ECSs with the following configurations. For details about how to purchase an ECS, see Purchasing an ECS.
    • One ECS is 4 vCPUs | 8 GB, runs Linux, and is configured with the same region, AZ, VPC, subnet, and security group as the Kafka instance.
    • The other ECS is 4 vCPUs | 8 GB, runs Linux, and is configured with the same region, VPC, subnet, and security group but a different AZ from the Kafka instance.

    Perform the following operations on the ECSs:

    • Install Java JDK and configure the environment variables JAVA_HOME and PATH.
      export JAVA_HOME=/root/jdk1.8.0_231 
      export PATH=$JAVA_HOME/bin:$PATH
    • Download Kafka CLI 2.7.2 and decompress it.
      tar -zxf kafka_2.12-2.7.2.tgz

Script

./kafka-producer-perf-test.sh --producer-props bootstrap.servers={Connection address} acks=1 batch.size={batch.size} linger.ms=0 --topic {Topic name} --num-records {num-records} --record-size 1024 --throughput 102400
Table 3 Script parameters

Parameter

Description

bootstrap.servers

Address of the Kafka instance obtained in 1.

acks

Message synchronization policy. acks=1 indicates asynchronous replication, and acks=-1 indicates synchronous replication.

batch.size

Size of messages sent in each batch, in bytes.

linger.ms

Interval between two batches.

topic

Topic name set in 2.

num-records

Total number of messages to be sent.

record-size

Size of each message.

throughput

Number of messages sent per second.

Procedure

Result

Table 4 Testing results

Partitions

Replicas

Synchronous Replication

batch.size

Cross-AZ Production

Message Production Rate on the Client Side (Records/Second)

CPU Usage on the Server Side (broker-0)

CPU Usage on the Server Side (broker-1)

CPU Usage on the Server Side (broker-2)

3

1

No

1 KB

No

34,128

58.10%

56.70%

53.30%

3

1

No

16 KB

No

102,399

24.10%

25.00%

23.30%

3

1

No

1 KB

Yes

8,523

17.20%

16.70%

18.80%

3

3

Yes

1 KB

No

3981

60.00%

55.20%

50.00%

3

3

No

1 KB

No

14,468

86.70%

80.60%

86.20%

Based on the test results, the following conclusions are drawn (for reference only):

  • When the batch.size of production requests is 16 times larger, the message production rate increases, and the CPU usage decreases.
  • Compared with cross-AZ production, intra-AZ production significantly increases message production rate and CPU usage.
  • When the number of replicas changes from 1 to 3, the message production rate decreases significantly, and the CPU usage increases.
  • Compared with synchronous replication, asynchronous replication increases the message production rate and the CPU usage.