Help Center/ Distributed Message Service for Kafka/ Performance Whitepaper/ Test Procedure
Updated on 2023-03-30 GMT+08:00
View PDF

Test Procedure

Test the message production rate on the client side and the CPU usage on the server side in the following scenarios:

  • Test scenario 1 (batch size): same Kafka instance, same topics, different batch.size settings
  • Test scenario 2 (cross-AZ or intra-AZ production): same Kafka instance, same topics, different AZ settings for the client and server
  • Test scenario 3 (number of replicas): same Kafka instance, different numbers of replicas
  • Test scenario 4 (synchronous or asynchronous replication): same Kafka instance, topics with different replication settings
Table 1 Test parameters

Partitions

Replicas

Synchronous Replication Enabled

batch.size

Cross-AZ Production

3

1

No

1 KB

No

3

1

No

16 KB

No

3

1

No

1 KB

Yes

3

3

Yes

1 KB

No

3

3

No

1 KB

No

Test script:

./kafka-producer-perf-test.sh --producer-props bootstrap.servers=${connection address} acks=1 batch.size=${batch.size} linger.ms=0 --topic ${topic name} --num-records ${num-records} --record-size 1024 --throughput -102400
  • bootstrap.servers: address of the Kafka instance obtained in Step 1: Buy a Kafka Instance.
  • acks: the message synchronization policy. acks=1 indicates asynchronous replication, and acks=-1 indicates synchronous replication.
  • batch.size: size of messages sent in each batch, in bytes.
  • linger.ms: interval between two batches.
  • topic: topic name set in Step 2: Create Topics.
  • num-records: total number of messages to be sent.
  • record-size: size of each message.
  • throughput: number of messages sent per second.

Test Scenario 1: Varied Batch Sizes

  1. Log in to the client server, go to the kafka_2.11-2.3.0/bin directory, and run the following scripts.

    Set batch.size to 1 KB, and run the following script: 
    ./kafka-producer-perf-test.sh --producer-props bootstrap.servers=192.168.0.69:9092,192.168.0.42:9092,192.168.0.66:9092 acks=1 batch.size=1024 linger.ms=0 --topic Topic-01 --num-records 8000000 --record-size 1024 --throughput 102400

    The result is as follows:

    8000000 records sent, 27417.353814 records/sec (26.77 MB/sec), 1095.79 ms avg latency, 5989.00 ms max latency, 679 ms 50th, 2957 ms 95th, 3505 ms 99th, 5951 ms 99.9th.

    Message production rate: 27,417 records/second

    Set batch.size to 16 KB, and run the following script:

    ./kafka-producer-perf-test.sh --producer-props bootstrap.servers=192.168.0.69:9092,192.168.0.42:9092,192.168.0.66:9092 acks=1 batch.size=16384 linger.ms=0 --topic Topic-01 --num-records 100000000 --record-size 1024 --throughput 102400

    The result is as follows:

    100000000 records sent, 102399.213574 records/sec (100.00 MB/sec), 11.27 ms avg latency, 876 ms max latency, 1 ms 50th, 15 ms 95th, 384 ms 99th, 724 ms 99.9th.

    Message production rate: 102,399 records/second

  2. Log in to the Kafka console, locate the row that contains the test instance, and click to go to the Cloud Eye console.
  3. On the Brokers tab page, view the CPU usage of the server nodes.

    Figure 1 broker-0 CPU usage (batch.size = 1 KB)

    CPU usage: 57%

    Figure 2 broker-0 CPU usage (batch.size = 16 KB)

    CPU usage: 31%

    Figure 3 broker-1 CPU usage (batch.size = 1 KB)

    CPU usage: 58%

    Figure 4 broker-1 CPU usage (batch.size = 16 KB)

    CPU usage: 33%

    Figure 5 broker-2 CPU usage (batch.size = 1 KB)

    CPU usage: 58%

    Figure 6 broker-2 CPU usage (batch.size = 16 KB)

    CPU usage: 31%

Test Scenario 2: Cross-AZ or Intra-AZ Production

  1. Log in to the client server, go to the kafka_2.11-2.3.0/bin directory, and run the following scripts.

    Configure the same AZ for the client and the instance, and run the following script:

    ./kafka-producer-perf-test.sh --producer-props bootstrap.servers=192.168.0.69:9092,192.168.0.42:9092,192.168.0.66:9092 acks=1 batch.size=1024 linger.ms=0 --topic Topic-01 --num-records 8000000 --record-size 1024 --throughput 102400

    The result is as follows:

    8000000 records sent, 27417.353814 records/sec (26.77 MB/sec), 1095.79 ms avg latency, 5989.00 ms max latency, 679 ms 50th, 2957 ms 95th, 3505 ms 99th, 5951 ms 99.9th.

    Message production rate: 27,417 records/second

    Configure different AZs for the client and the instance, and run the following script:

    ./kafka-producer-perf-test.sh --producer-props bootstrap.servers=192.168.0.69:9092,192.168.0.42:9092,192.168.0.66:9092 acks=1 batch.size=1024 linger.ms=0 --topic Topic-01 --num-records 4000000 --record-size 1024 --throughput 102400

    The result is as follows:

    4000000 records sent, 15351.079181 records/sec (14.99 MB/sec), 1927.26 ms avg latency, 8974.00 ms max latency, 15 ms 50th, 5925 ms 95th, 6838 ms 99th, 8925 ms 99.9th.

    Message production rate: 15,351 records/second

  2. Log in to the Kafka console, locate the row that contains the test instance, and click to go to the Cloud Eye console.
  3. On the Brokers tab page, view the CPU usage of the server nodes.

    Figure 7 broker-0 CPU usage (same AZ)

    CPU usage: 57%

    Figure 8 broker-0 CPU usage (different AZs)

    CPU usage: 37%

    Figure 9 broker-1 CPU usage (same AZ)

    CPU usage: 58%

    Figure 10 broker-1 CPU usage (different AZs)

    CPU usage: 39%

    Figure 11 broker-2 CPU usage (same AZ)

    CPU usage: 58%

    Figure 12 broker-2 CPU usage (different AZs)

    CPU usage: 36%

Test Scenario 3: Varied Numbers of Replicas

  1. Log in to the client server, go to the kafka_2.11-2.3.0/bin directory, and run the following scripts.

    For the one-replica topic, run the following script:

    ./kafka-producer-perf-test.sh --producer-props bootstrap.servers=192.168.0.69:9092,192.168.0.42:9092,192.168.0.66:9092 acks=1 batch.size=1024 linger.ms=0 --topic Topic-01 --num-records 8000000 --record-size 1024 --throughput 102400

    The result is as follows:

    8000000 records sent, 27417.353814 records/sec (26.77 MB/sec), 1095.79 ms avg latency, 5989.00 ms max latency, 679 ms 50th, 2957 ms 95th, 3505 ms 99th, 5951 ms 99.9th.

    Message production rate: 27,417 records/second

    For the three-replica topic, run the following script:

    ./kafka-producer-perf-test.sh --producer-props bootstrap.servers=192.168.0.69:9092,192.168.0.42:9092,192.168.0.66:9092 acks=1 batch.size=1024 linger.ms=0 --topic Topic-02 --num-records 4000000 --record-size 1024 --throughput 102400

    The result is as follows:

    4000000 records sent, 11557.954473 records/sec (11.29 MB/sec), 2591.66 ms avg latency, 8071.00 ms max latency, 2566 ms 50th, 5396 ms 95th, 6276 ms 99th, 8003 ms 99.9th.

    Message production rate: 11,558 records/second

  2. Log in to the Kafka console, locate the row that contains the test instance, and click to go to the Cloud Eye console.
  3. On the Brokers tab page, view the CPU usage of the server nodes.

    Figure 13 broker-0 CPU usage (one replica)

    CPU usage: 57%

    Figure 14 broker-0 CPU usage (three replicas)

    CPU usage: 89%

    Figure 15 broker-1 CPU usage (one replica)

    CPU usage: 58%

    Figure 16 broker-1 CPU usage (three replicas)

    CPU usage: 88%

    Figure 17 broker-2 CPU usage (one replica)

    CPU usage: 58%

    Figure 18 broker-2 CPU usage (three replicas)

    CPU usage: 87%

Test Scenario 4: Synchronous/Asynchronous Replication

  1. Log in to the client server, go to the kafka_2.11-2.3.0/bin directory, and run the following scripts.

    For asynchronous replication, run the following script:

    ./kafka-producer-perf-test.sh --producer-props bootstrap.servers=192.168.0.69:9092,192.168.0.42:9092,192.168.0.66:9092 acks=1 batch.size=1024 linger.ms=0 --topic Topic-02 --num-records 4000000 --record-size 1024 --throughput 102400

    The result is as follows:

    4000000 records sent, 11557.954473 records/sec (11.29 MB/sec), 2591.66 ms avg latency, 8071.00 ms max latency, 2566 ms 50th, 5396 ms 95th, 6276 ms 99th, 8003 ms 99.9th.

    Message production rate: 11,558 records/second

    For synchronous replication, run the following script:

    ./kafka-producer-perf-test.sh --producer-props bootstrap.servers=192.168.0.69:9092,192.168.0.42:9092,192.168.0.66:9092 acks=-1 batch.size=1024 linger.ms=0 --topic Topic-03 --num-records 1000000 --record-size 1024 --throughput 102400

    The result is as follows:

    1000000 records sent, 3859.960628 records/sec (3.77 MB/sec), 7675.00 ms avg latency, 13852.00 ms max latency, 7695 ms 50th, 11056 ms 95th, 12907 ms 99th, 13809 ms 99.9th.

    Message production rate: 3859 records/second

  2. Log in to the Kafka console, locate the row that contains the test instance, and click to go to the Cloud Eye console.
  3. On the Brokers tab page, view the CPU usage of the server nodes.

    Figure 19 broker-0 CPU usage (asynchronous replication)

    CPU usage: 89%

    Figure 20 broker-0 CPU usage (synchronous replication)

    CPU usage: 62%

    Figure 21 broker-1 CPU usage (asynchronous replication)

    CPU usage: 88%

    Figure 22 broker-1 CPU usage (synchronous replication)

    CPU usage: 62%

    Figure 23 broker-2 CPU usage (asynchronous replication)

    CPU usage: 87%

    Figure 24 broker-2 CPU usage (synchronous replication)

    CPU usage: 60%