Kafka Production Rate and CPU Usage
This section describes performance tests on Distributed Message Service (DMS) for Kafka. The performance is measured by the message production rate on the client side and CPU usage on the server side. The tests cover the following scenarios:
- Scenario 1 (batch size): same Kafka instance, same topics, different message size settings
- Scenario 2 (cross-AZ or intra-AZ production): same Kafka instance, same topics, different AZ settings for the client and server
- Scenario 3 (number of replicas): same Kafka instance, different numbers of replicas
- Scenario 4 (synchronous or asynchronous replication): same Kafka instance, topics with different replication settings
Partitions |
Replicas |
Synchronous Replication |
batch.size |
Cross-AZ Production |
---|---|---|---|---|
3 |
1 |
No |
1 KB |
No |
3 |
1 |
No |
16 KB |
No |
3 |
1 |
No |
1 KB |
Yes |
3 |
3 |
Yes |
1 KB |
No |
3 |
3 |
No |
1 KB |
No |
Environment
Perform the following steps to set up the test environment.
- Purchase a Kafka instance with parameters specified as follows and retain the default settings for other ones. For details about how to purchase one, see Buying a Kafka Instance.
Table 2 Parameter
Example Value
Region
CN-Hong Kong
AZ
AZ1
Version
2.7
Architecture
Cluster
Broker Flavor
kafka.2u4g.cluster
Brokers
3
Storage Space per Broker
Ultra-high I/O, 200 GB
VPC
Select a VPC.
Subnet
Select a subnet.
Security Group
Select a security group.
Access Mode
Default value
Instance Name
kafka-test
Enterprise Project
default
After the purchase, obtain Address (Private Network, Plaintext) on the instance details page.
- Create three topics with parameters specified as follows for the purchased Kafka instance. For details, see Creating a Kafka Topic.
- Topic-01: 3 partitions, 1 replica, asynchronous replication
- Topic-02: 3 partitions, 3 replicas, asynchronous replication
- Topic-03: 3 partitions, 3 replicas, synchronous replication
- Obtain the test tool.
Obtain Kafka CLI 2.7.2.
- Purchase a server for the client.
Buy two ECSs with the following configurations. For details about how to purchase an ECS, see Purchasing an ECS.
- One ECS is 4 vCPUs | 8 GB, runs Linux, and is configured with the same region, AZ, VPC, subnet, and security group as the Kafka instance.
- The other ECS is 4 vCPUs | 8 GB, runs Linux, and is configured with the same region, VPC, subnet, and security group but a different AZ from the Kafka instance.
Perform the following operations on the ECSs:
- Install Java JDK and configure the environment variables JAVA_HOME and PATH.
export JAVA_HOME=/root/jdk1.8.0_231 export PATH=$JAVA_HOME/bin:$PATH
- Download Kafka CLI 2.7.2 and decompress it.
tar -zxf kafka_2.12-2.7.2.tgz
Script
./kafka-producer-perf-test.sh --producer-props bootstrap.servers={Connection address} acks=1 batch.size={batch.size} linger.ms=0 --topic {Topic name} --num-records {num-records} --record-size 1024 --throughput 102400
Parameter |
Description |
---|---|
bootstrap.servers |
Address of the Kafka instance obtained in 1. |
acks |
Message synchronization policy. acks=1 indicates asynchronous replication, and acks=-1 indicates synchronous replication. |
batch.size |
Size of messages sent in each batch, in bytes. |
linger.ms |
Interval between two batches. |
topic |
Topic name set in 2. |
num-records |
Total number of messages to be sent. |
record-size |
Size of each message. |
throughput |
Number of messages sent per second. |
Procedure
Result
Partitions |
Replicas |
Synchronous Replication |
batch.size |
Cross-AZ Production |
Message Production Rate on the Client Side (Records/Second) |
CPU Usage on the Server Side (broker-0) |
CPU Usage on the Server Side (broker-1) |
CPU Usage on the Server Side (broker-2) |
---|---|---|---|---|---|---|---|---|
3 |
1 |
No |
1 KB |
No |
34,128 |
58.10% |
56.70% |
53.30% |
3 |
1 |
No |
16 KB |
No |
102,399 |
24.10% |
25.00% |
23.30% |
3 |
1 |
No |
1 KB |
Yes |
8,523 |
17.20% |
16.70% |
18.80% |
3 |
3 |
Yes |
1 KB |
No |
3981 |
60.00% |
55.20% |
50.00% |
3 |
3 |
No |
1 KB |
No |
14,468 |
86.70% |
80.60% |
86.20% |
Based on the test results, the following conclusions are drawn (for reference only):
- When the batch.size of production requests is 16 times larger, the message production rate increases, and the CPU usage decreases.
- Compared with cross-AZ production, intra-AZ production significantly increases message production rate and CPU usage.
- When the number of replicas changes from 1 to 3, the message production rate decreases significantly, and the CPU usage increases.
- Compared with synchronous replication, asynchronous replication increases the message production rate and the CPU usage.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot