Kafka Instance TPS

Environment

Perform the following steps to set up the test environment.

1. Purchase Kafka instances with parameters specified in Table 1. For more information, see Buying a Kafka Instance.

   Table 1 Instance parameters

   Instance Name	Brokers	Broker Flavor	SASL	Storage space per broker
   kafka-01	3	kafka.2u4g.cluster	Yes	Ultra-high I/O
   kafka-02	3	kafka.4u8g.cluster	Yes	Ultra-high I/O
   kafka-03	3	kafka.8u16g.cluster	Yes	Ultra-high I/O
   kafka-04	3	kafka.12u24g.cluster	Yes	Ultra-high I/O
   kafka-05	3	kafka.16u32g.cluster	Yes	Ultra-high I/O
   kafka-06	3	kafka.2u4g.cluster	No	Ultra-high I/O
   kafka-07	3	kafka.4u8g.cluster	No	Ultra-high I/O
   kafka-08	3	kafka.8u16g.cluster	No	Ultra-high I/O
   kafka-09	3	kafka.12u24g.cluster	No	Ultra-high I/O
   kafka-10	3	kafka.16u32g.cluster	No	Ultra-high I/O
   kafka-11	3	kafka.2u4g.cluster	No	High I/O
   kafka-12	3	kafka.4u8g.cluster	No	High I/O
   kafka-13	3	kafka.8u16g.cluster	No	High I/O
   kafka-14	3	kafka.12u24g.cluster	No	High I/O
   kafka-15	3	kafka.16u32g.cluster	No	High I/O

   After the purchase, obtain Address (Private Network, Plaintext) on the instance details page.

   

2. Create topics with parameters specified in Table 2 for each instance purchased above. For details about how to create topics, see Creating a Kafka Topic.

   Table 2 Topic parameters

   Topic Name	Synchronous Replication	Synchronous Flushing	Replicas	Partitions
   topic-01	No	No	3	30
   topic-02	Yes	No	3	30
   topic-03	No	Yes	3	30
   topic-04	No	No	3	3
   topic-05	No	No	3	12
   topic-06	No	No	3	100

3. Obtain the test tool.

   Obtain Kafka CLI v2.7.2.

4. Purchase a server for the client.

   Buy a Linux ECS (with the same region, AZ, VPC, subnet, and security group as the Kafka instance). For details about how to purchase an ECS, see Purchasing a Custom ECS.

   Perform the following operations on the ECSs:

   • Install Java JDK and configure the environment variables JAVA_HOME and PATH.

     export JAVA_HOME=/root/jdk1.8.0_231 
     export PATH=$JAVA_HOME/bin:$PATH

   • Download Kafka CLI v2.7.2 and decompress it.

     tar -zxf kafka_2.12-2.7.2.tgz

Script

./kafka-producer-perf-test.sh --producer-props bootstrap.servers=${connection address} acks=1 batch.size=16384 linger.ms=10 --topic ${topic name} --num-records 10000000 --record-size 1024 --throughput -1 --producer.config ../config/producer.properties

• bootstrap.servers: address of the Kafka instance obtained in 1.
• acks: message synchronization policy. acks=1 indicates asynchronous replication, and acks=-1 indicates synchronous replication.
• batch.size: size of messages sent in each batch, in bytes.
• linger.ms: interval between two batches.
• topic: topic name set in 2.
• num-records: total number of messages to be sent.
• record-size: size of each message.
• throughput: number of messages sent per second.

Result

Scenario 1 (whether SASL is enabled): same topic (30 partitions, 3 replicas, asynchronous replication, and asynchronous flushing), instances with SASL enabled or disabled. The test result is as follows.

Scenario 2 (synchronous/asynchronous replication): same instance (ultra-high I/O, three brokers, SASL disabled), topics with different replication settings, and number of producer processes is three. The test result is as follows.

Scenario 3 (synchronous/asynchronous replication flushing): same instance (ultra-high I/O, three brokers, SASL disabled), topics with different flushing settings. The test result is as follows.

Scenario 4 (different disk types): same topic (30 partitions, 3 replicas, asynchronous replication, and asynchronous flushing) with different disk types. The test result is as follows.

Scenario 5 (different numbers of partitions): same instance (ultra-high I/O, three brokers, SASL disabled), topics with different number of partitions. The test result is as follows.