- What's New
- Function Overview
- Product Bulletin
-
Service Overview
- Infographics
- What Is DMS for Kafka?
- Product Advantages
- Application Scenarios
- Kafka Instance Specifications
- Comparing Single-node and Cluster Kafka Instances
- Comparing Kafka, RabbitMQ, and RocketMQ
- Comparing DMS for Kafka and Open-Source Kafka
- Security
- Notes and Constraints
- Related Services
- Basic Concepts
- Permissions
- Billing
- Getting Started
-
User Guide
- Process of Using Kafka
- Permissions Management
- Buying a Kafka Instance
- Configuring Topics
- Connecting to an Instance
- Managing Messages
- Managing Consumer Groups
- Managing Quotas
-
Managing Instances
- Viewing and Modifying Basic Information of a Kafka Instance
- Viewing Kafka Disk Usage
- Viewing Kafka Background Tasks
- Viewing Sample Code of Kafka Production and Consumption
- Modifying Kafka Instance Configuration Parameters
- Configuring Kafka Instance Tags
- Configuring Kafka Recycling Policies
- Upgrading the Kafka Instance Kernel
- Exporting the Kafka Instance List
- Restarting a Kafka Instance
- Deleting Kafka Instances
- Using Kafka Manager
- Modifying Instance Specifications
- Migrating Data
- Testing Instance Performance
- Applying for Increasing Kafka Quotas
- Monitoring and Alarms
- Viewing Kafka Audit Logs
-
Best Practices
- Kafka Best Practices
- Improving Kafka Message Processing Efficiency
- Optimizing Consumer Polling
- Interconnecting Logstash to Kafka to Produce and Consume Messages
- Using MirrorMaker to Synchronize Data Across Clusters
- Handling Message Accumulation
- Handling Service Overload
- Handling Uneven Service Data
- Configuring Message Accumulation Monitoring
- Suggestions on Using DMS for Kafka Securely
- Developer Guide
-
API Reference
- Before You Start
- API Overview
- Calling APIs
- Getting Started
-
APIs V2 (Recommended)
- Managing Lifecycle
-
Instance Management
- Resetting the Password
- Resetting Kafka Manager Password
- Restarting Kafka Manager
- Configuring Automatic Topic Creation
- Modifying the Private IP Address for Cross-VPC Access
- Querying Kafka Cluster Metadata
- Querying Coordinator Details of a Kafka Instance
- Reassigning Replicas of a Topic for a Kafka Instance
- Modifying Kafka Access Modes
- Querying the Disk Usage Status of Topics
- Initiating Partition Reassigning for a Kafka Instance
- Disabling Kafka Manager
- Deleting a User or Client Quota
- Querying User or Client Quotas
- Creating User or Client Quotas
- Modifying User or Client Quotas
-
Smart Connect
- Enabling Smart Connect (Pay-per-Use Instance)
- Disabling Smart Connect (Pay-per-Use Instance)
- Creating a Smart Connect Task
- Querying Smart Connect Tasks
- Querying Smart Connect Task Details
- Deleting a Smart Connect Task
- Pausing a Smart Connect Task
- Restarting a Smart Connect Task
- Starting a Smart Connect Task or Restarting a Paused or Running Smart Connect Task
- Specification Modification Management
- Topic Management
-
Managing Consumer Groups
- Querying Consumer Group Details
- Querying All Consumer Groups
- Deleting Consumer Groups of a Kafka Instance in Batches
- Creating a Consumer Group
- Resetting Consumer Group Offset to the Specified Position
- Querying the Offset of a Consumer Group
- Modifying All Consumer Groups
- Querying a Specified Consumer Group
- Deleting a Specified Consumer Group
- Modifying a Specified Consumer Group
- Querying Topics of a Specified Consumer Group
- Querying Consumers in a Specified Consumer Group
- User Management
- Managing Messages
- Background Task Management
- Tag Management
- Diagnosis Management
- Other APIs
- Permissions and Supported Actions
- Out-of-Date APIs
- Appendix
- Change History
- SDK Reference
-
FAQs
-
Instances
- Why Can't I Select Two AZs?
- Why Can't I View the Subnet and Security Group Information When Creating a DMS Instance?
- How Do I Select Storage Space for a Kafka Instance?
- How Do I Choose Between High I/O and Ultra-high I/O?
- Which Capacity Threshold Policy Should I Use?
- Which Kafka Versions Are Supported?
- What Is the ZooKeeper Address of a Kafka Instance?
- Are Kafka Instances in Cluster Mode?
- Can I Modify the Port for Accessing a Kafka Instance?
- How Long Are Kafka SSL Certificates Valid for?
- How Do I Synchronize Data from One Kafka Instance to Another?
- How Do I Change the SASL_SSL Setting of a Kafka Instance?
- How Do I Modify the SASL Mechanism?
- How Do I Change the Security Protocol?
- Will a Kafka Instance Be Restarted After Its Enterprise Project Is Modified?
- Are Kafka Brokers and ZooKeeper Deployed on the Same VM or on Different VMs?
- Can I Delete a Kafka Instance Billed in Yearly/Monthly Mode?
- Which Cipher Suites Are Supported by Kafka?
- Can I Change Single-AZ Deployment to Multi-AZ Deployment for an Instance?
- Does DMS for Kafka Support Cross-AZ Disaster Recovery? Where Can I Check Whether an Existing Instance is Across-AZs?
- Do Kafka Instances Support Disk Encryption?
- Can I Change the VPC and Subnet After a Kafka Instance Is Created?
- Where Can I Find Kafka Streams Use Cases?
- Can I Upgrade Kafka Instances?
- How Do I Bind an EIP Again?
- Specification Modification
-
Connections
- How Do I Select and Configure a Security Group?
- Can I Access a Kafka Instance Over a Public Network?
- How Many Connection Addresses Does a Kafka Instance Have by Default?
- Do Kafka Instances Support Cross-Region Access?
- Do Kafka Instances Support Cross-VPC Access?
- Do Kafka Instances Support Cross-Subnet Access?
- Does DMS for Kafka Support Authentication with Kerberos?
- Does DMS for Kafka Support Password-Free Access?
- How Do I Obtain the Public Access Address After Public Access Is Enabled?
- Does DMS for Kafka Support Authentication on Clients by the Server?
- Can I Use PEM SSL Truststore When Connecting to a Kafka Instance with SASL_SSL Enabled?
- What Are the Differences Between JKS and CRT Certificates?
- Which TLS Version Does DMS for Kafka Support?
- Is There a Limit on the Number of Client Connections to a Kafka Instance?
- How Many Connections Are Allowed from Each IP Address?
- Can I Change the Private Network Addresses of a Kafka Instance?
- Is the Same SSL Certificate Used for Different Instances?
- Why Is It Not Recommended to Use a Sarama Client for Messaging?
-
Topics and Partitions
- Is There a Limit on the Number of Topics in a Kafka Instance?
- Why Is Partition Quantity Limited?
- Can I Reduce the Partition Quantity?
- Why Do I Fail to Create Topics?
- Do Kafka Instances Support Batch Importing Topics or Automatic Topic Creation?
- Why Do Deleted Topics Still Exist?
- Can I View the Disk Space Used by a Topic?
- Can I Add ACL Permissions for Topics?
- What Should I Do If Kafka Storage Space Is Used Up Because Retrieved Messages Are Not Deleted?
- How Do I Increase the Partition Quantity?
- Will a Kafka Instance Be Restarted After Its Automatic Topic Creation Setting Is Modified?
- What Can I Do If a Consumer Fails to Retrieve Messages from a Topic Due to Insufficient Permissions?
-
Consumer Groups
- Do I Need to Create Consumer Groups, Producers, and Consumers for Kafka Instances?
- Will a Consumer Group Without Active Consumers Be Automatically Deleted in 14 Days?
- Why Does a Deleted Consumer Group Still Exist?
- Why Can't I View Consumers When Instance Consumption Is Normal?
- Can I Delete Unnecessary Topics in a Consumer Group?
-
Messages
- What Is the Maximum Size of a Message that Can be Created?
- Why Does Message Poll Often Fail During Rebalancing?
- Why Can't I Query Messages on the Console?
- What Can I Do If Kafka Messages Are Accumulated?
- Why Do Messages Still Exist After the Retention Period Elapses?
- Do Kafka Instances Support Delayed Message Delivery?
- How Do I View the Number of Accumulated Messages?
- Why Is the Message Creation Time Displayed as Year 1970?
- How Do I Modify message.max.bytes?
- Why Are Offsets Not Continuous?
-
Kafka Manager
- Can I Configure a Kafka Manager Account to Be Read-Only?
- Why Can't I See Broker Information After Logging In to Kafka Manager?
- Yikes! Insufficient partition balance when creating topic : projectman_project_enterprise_project Try again.
- Can I Query the Body of a Message by Using Kafka Manager?
- Can I Change the Port of the Kafka Manager Web UI?
- Which Topic Configurations Can Be Modified on Kafka Manager?
- Why Is Information Displayed on Kafka Manager Inconsistent with Cloud Eye Monitoring Data?
- How Do I Change a Partition Leader for a Topic in Kafka Manager?
- Why Is the Version on the Console Different from That in Kafka Manager?
- Why Does an Instance Contain Default Topics __trace and __consumer_offsets?
- Why Do I See a Deleted Consumer Group on Kafka Manager?
-
Monitoring & Alarm
- Why Can't I View the Monitoring Data?
- Why Is the Monitored Number of Accumulated Messages Inconsistent with the Message Quantity Displayed on the Kafka Console?
- Why Is a Consumer Group Still on the Monitoring Page After Being Deleted?
- Why Do Metrics Fluctuate Significantly (Disk Read/Write Speed, Average Disk Read/Write Time, and CPU Usage)?
- Why Does JVM Heap Memory Usage of JVM Fluctuate Significantly?
- Why Is Production Rate Still 0 When There Are Produced Messages?
-
Instances
-
Troubleshooting
- Troubleshooting Kafka Connection Exceptions
- Troubleshooting 6-Min Latency Between Message Creation and Retrieval
- Troubleshooting Message Creation Failures
- Troubleshooting Topic Deletion Failures
- Troubleshooting Failure to Log In to Kafka Manager in Windows
- Troubleshooting Error "Topic {{topic_name}} not present in metadata after 60000 ms" During Message Production or Consumption
- Flink 1.15 Consumption Progress Submission Failure
- Videos
-
More Documents
-
User Guide (ME-Abu Dhabi Region)
- Service Overview
- Getting Started
- Permissions Management
- Preparing Required Resources
- Buying an Instance
- Accessing a Kafka Instance
- Managing Instances
- Managing Topics
- Managing Messages
- Managing Users
- Managing Consumer Groups
- Managing Kafka Quotas
- Modifying Kafka Parameters
- Quotas
- Monitoring
- Auditing
-
FAQs
-
Instances
- Why Can't I Select Two AZs?
- Why Can't I View the Subnet and Security Group Information When Creating a DMS Instance?
- How Do I Select Storage Space for a Kafka Instance?
- How Do I Choose Between High I/O and Ultra-high I/O?
- Which Capacity Threshold Policy Should I Use?
- Which Kafka Versions Are Supported?
- What Is the ZooKeeper Address of a Kafka Instance?
- Are Kafka Instances in Cluster Mode?
- Can I Modify the Port for Accessing a Kafka Instance?
- How Long Are Kafka SSL Certificates Valid for?
- How Do I Synchronize Data from One Kafka Instance to Another?
- How Do I Change the SASL_SSL Setting of a Kafka Instance?
- How Do I Modify the SASL Mechanism?
- Will a Kafka Instance Be Restarted After Its Enterprise Project Is Modified?
- Are Kafka Brokers and ZooKeeper Deployed on the Same VM or on Different VMs?
- Which Cipher Suites Are Supported by Kafka?
- Can I Change an Instance from Single-AZ Deployment to Multi-AZ Deployment?
- Does DMS for Kafka Support Cross-AZ Disaster Recovery? Where Can I View the AZs Configured for an Existing Instance?
- Do Kafka Instances Support Disk Encryption?
- Does Specification Modification Affect Services?
- Can I Change the VPC and Subnet After a Kafka Instance Is Created?
- Where Can I Find Kafka Streams Use Cases?
- Can I Upgrade Kafka Instances?
- Why Is the Version on the Console Different from That in Kafka Manager?
- How Do I Bind an EIP Again?
-
Connections
- How Do I Select and Configure a Security Group?
- Can I Access a Kafka Instance Over a Public Network?
- How Many Connection Addresses Does a Kafka Instance Have by Default?
- Do Kafka Instances Support Cross-Region Access?
- Do Kafka Instances Support Cross-VPC Access?
- Do Kafka Instances Support Cross-Subnet Access?
- Does DMS for Kafka Support Authentication with Kerberos?
- Does DMS for Kafka Support Password-Free Access?
- How Do I Obtain the Public Access Address After Public Access Is Enabled?
- Does DMS for Kafka Support Authentication on Clients by the Server?
- Can I Use PEM SSL Truststore When Connecting to a Kafka Instance with SASL_SSL Enabled?
- What Are the Differences Between JKS and CRT Certificates?
- Which TLS Version Does DMS for Kafka Support?
- Is There a Limit on the Number of Client Connections to a Kafka Instance?
- How Many Connections Are Allowed from Each IP Address?
- Can I Change the Private Network Addresses of a Kafka Instance?
- Is the Same SSL Certificate Used for Different Instances?
-
Topics and Partitions
- Is There a Limit on the Number of Topics in a Kafka Instance?
- Why Is Partition Quantity Limited?
- Can I Change the Partition Quantity?
- Why Do I Fail to Create Topics?
- Do Kafka Instances Support Batch Importing Topics or Automatic Topic Creation?
- Why Do Deleted Topics Still Exist?
- Can I View the Disk Space Used by a Topic?
- Can I Add ACL Permissions for Topics?
- What Should I Do If Kafka Storage Space Is Used Up Because Retrieved Messages Are Not Deleted?
- How Do I Increase the Partition Quantity?
- Will a Kafka Instance Be Restarted After Its Automatic Topic Creation Setting Is Modified?
- How Do I Disable Automatic Topic Creation?
- Can I Delete Unnecessary Topics in a Consumer Group?
- What Can I Do If a Consumer Fails to Retrieve Messages from a Topic Due to Insufficient Permissions?
- Why Does an Instance Contain Default Topics __trace and __consumer_offsets?
- Consumer Groups
-
Messages
- What Is the Maximum Size of a Message that Can be Created?
- Why Does Message Poll Often Fail During Rebalancing?
- Why Can't I Query Messages on the Console?
- What Can I Do If Kafka Messages Are Accumulated?
- Why Do Messages Still Exist After the Retention Period Elapses?
- Do Kafka Instances Support Delayed Message Delivery?
- How Do I View the Number of Accumulated Messages?
- Why Is the Message Creation Time Displayed as Year 1970?
-
Kafka Manager
- Can I Configure a Kafka Manager Account to Be Read-Only?
- Why Can't I See Broker Information After Logging In to Kafka Manager?
- Yikes! Insufficient partition balance when creating topic : projectman_project_enterprise_project Try again.
- Can I Query the Body of a Message by Using Kafka Manager?
- Can I Change the Port of the Kafka Manager Web UI?
- Which Topic Configurations Can Be Modified on Kafka Manager?
- Why Is Information Displayed on Kafka Manager Inconsistent with Cloud Eye Monitoring Data?
- How Do I Change a Partition Leader for a Topic in Kafka Manager?
- Monitoring & Alarm
-
Instances
-
Troubleshooting
- Troubleshooting Kafka Connection Exceptions
- Troubleshooting 6-Min Latency Between Message Creation and Retrieval
- Troubleshooting Message Creation Failures
- Troubleshooting Topic Deletion Failures
- Troubleshooting Failure to Log In to Kafka Manager in Windows
- Troubleshooting Error "Topic {{topic_name}} not present in metadata after 60000 ms" During Message Production or Consumption
- Change History
- API Reference (ME-Abu Dhabi Region)
-
User Guide (Paris Region)
- Service Overview
- Getting Started
- Permissions Management
- Preparing Required Resources
- Creating an Instance
- Accessing a Kafka Instance
- Managing Instances
- Managing Topics
- Managing Messages
- Managing Users
- Managing Consumer Groups
- Managing Kafka Quotas
- Modifying Kafka Parameters
- Quotas
- Monitoring
- Auditing
-
FAQs
-
Instances
- Why Can't I Select Two AZs?
- Why Can't I View the Subnet and Security Group Information When Creating a DMS Instance?
- How Do I Select Storage Space for a Kafka Instance?
- How Do I Choose Between High I/O and Ultra-high I/O?
- Which Capacity Threshold Policy Should I Use?
- Which Kafka Versions Are Supported?
- What Is the ZooKeeper Address of a Kafka Instance?
- Are Kafka Instances in Cluster Mode?
- Can I Modify the Port for Accessing a Kafka Instance?
- How Long Are Kafka SSL Certificates Valid for?
- How Do I Synchronize Data from One Kafka Instance to Another?
- How Do I Change the SASL_SSL Setting of a Kafka Instance?
- How Do I Modify the SASL Mechanism?
- Will a Kafka Instance Be Restarted After Its Enterprise Project Is Modified?
- Are Kafka Brokers and ZooKeeper Deployed on the Same VM or on Different VMs?
- Which Cipher Suites Are Supported by Kafka?
- Can I Change an Instance from Single-AZ Deployment to Multi-AZ Deployment?
- Does DMS for Kafka Support Cross-AZ Disaster Recovery? Where Can I View the AZs Configured for an Existing Instance?
- Do Kafka Instances Support Disk Encryption?
- Can I Change the VPC and Subnet After a Kafka Instance Is Created?
- Where Can I Find Kafka Streams Use Cases?
- Can I Upgrade Kafka Instances?
- Why Is the Version on the Console Different from That in Kafka Manager?
- How Do I Bind an EIP Again?
- Specification Modification
-
Connections
- How Do I Select and Configure a Security Group?
- Can I Access a Kafka Instance Over a Public Network?
- How Many Connection Addresses Does a Kafka Instance Have by Default?
- Do Kafka Instances Support Cross-Region Access?
- Do Kafka Instances Support Cross-VPC Access?
- Do Kafka Instances Support Cross-Subnet Access?
- Does DMS for Kafka Support Authentication with Kerberos?
- Does DMS for Kafka Support Password-Free Access?
- How Do I Obtain the Public Access Address After Public Access Is Enabled?
- Does DMS for Kafka Support Authentication on Clients by the Server?
- Can I Use PEM SSL Truststore When Connecting to a Kafka Instance with SASL_SSL Enabled?
- What Are the Differences Between JKS and CRT Certificates?
- Which TLS Version Does DMS for Kafka Support?
- Is There a Limit on the Number of Client Connections to a Kafka Instance?
- How Many Connections Are Allowed from Each IP Address?
- Can I Change the Private Network Addresses of a Kafka Instance?
- Is the Same SSL Certificate Used for Different Instances?
- Why Is It Not Recommended to Use a Sarama Client for Messaging?
-
Topics and Partitions
- Is There a Limit on the Number of Topics in a Kafka Instance?
- Why Is Partition Quantity Limited?
- Can I Reduce the Partition Quantity?
- Why Do I Fail to Create Topics?
- Do Kafka Instances Support Batch Importing Topics or Automatic Topic Creation?
- Why Do Deleted Topics Still Exist?
- Can I View the Disk Space Used by a Topic?
- Can I Add ACL Permissions for Topics?
- What Should I Do If Kafka Storage Space Is Used Up Because Retrieved Messages Are Not Deleted?
- How Do I Increase the Partition Quantity?
- Will a Kafka Instance Be Restarted After Its Automatic Topic Creation Setting Is Modified?
- How Do I Disable Automatic Topic Creation?
- Can I Delete Unnecessary Topics in a Consumer Group?
- What Can I Do If a Consumer Fails to Retrieve Messages from a Topic Due to Insufficient Permissions?
- Why Does an Instance Contain Default Topics __trace and __consumer_offsets?
- Consumer Groups
-
Messages
- What Is the Maximum Size of a Message that Can be Created?
- Why Does Message Poll Often Fail During Rebalancing?
- Why Can't I Query Messages on the Console?
- What Can I Do If Kafka Messages Are Accumulated?
- Why Do Messages Still Exist After the Retention Period Elapses?
- Do Kafka Instances Support Delayed Message Delivery?
- How Do I View the Number of Accumulated Messages?
- Why Is the Message Creation Time Displayed as Year 1970?
-
Kafka Manager
- Can I Configure a Kafka Manager Account to Be Read-Only?
- Why Can't I See Broker Information After Logging In to Kafka Manager?
- Yikes! Insufficient partition balance when creating topic : projectman_project_enterprise_project Try again.
- Can I Query the Body of a Message by Using Kafka Manager?
- Can I Change the Port of the Kafka Manager Web UI?
- Which Topic Configurations Can Be Modified on Kafka Manager?
- How Do I Change a Partition Leader for a Topic in Kafka Manager?
- Monitoring & Alarm
-
Instances
-
Troubleshooting
- Troubleshooting Kafka Connection Exceptions
- Troubleshooting 6-Min Latency Between Message Creation and Retrieval
- Troubleshooting Message Creation Failures
- Troubleshooting Topic Deletion Failures
- Troubleshooting Failure to Log In to Kafka Manager in Windows
- Troubleshooting Error "Topic {{topic_name}} not present in metadata after 60000 ms" During Message Production or Consumption
- Change History
-
API Reference (Paris Region)
- Before You Start
- API Overview
- Calling APIs
- Getting Started
-
APIs V2 (Recommended)
- Lifecycle Management
-
Instance Management
- Resetting the Password
- Resetting Kafka Manager Password
- Configuring Automatic Topic Creation
- Modifying the Private IP Address for Cross-VPC Access
- Querying Kafka Cluster Metadata
- Querying Consumer Group Details
- Resetting Consumer Group Offset to the Specified Position
- Querying Coordinator Details of a Kafka Instance
- Adding Partitions to a Topic for a Kafka Instance
- Reassigning Replicas of a Topic for a Kafka Instance
- Querying the Disk Usage Status of Topics
- Querying All Consumer Groups
- Querying a Specific Consumer Group
- Deleting a Consumer Group from a Kafka Instance
- Batch Deleting Consumer Groups of a Kafka Instance
- Initiating Partition Reassigning for a Kafka Instance
- Specification Modification Management
- Topic Management
- User Management
- Message Query
- Background Task Management
- Tag Management
- Other APIs
- Permissions and Supported Actions
- Out-of-Date APIs
- Appendix
- Change History
- Developer Guide (Paris Region)
-
User Guide (Kuala Lumpur Region)
- Service Overview
- Getting Started
- Process of Using Kafka
- Permissions Management
- Buying a Kafka Instance
- Configuring Topics
- Connecting to an Instance
- Managing Messages
- Managing Consumer Groups
- Managing Quotas
-
Managing Instances
- Viewing and Modifying Basic Information of a Kafka Instance
- Viewing Kafka Disk Usage
- Viewing Kafka Background Tasks
- Viewing Sample Code of Kafka Production and Consumption
- Modifying Kafka Instance Configuration Parameters
- Configuring Kafka Instance Tags
- Exporting the Kafka Instance List
- Restarting a Kafka Instance
- Deleting Kafka Instances
- Using Kafka Manager
- Modifying Kafka Instance Specifications
- Migrating Data
- Applying for Increasing Kafka Quotas
- Monitoring and Alarms
- Viewing Kafka Audit Logs
-
FAQs
-
Instances
- Why Can't I Select Two AZs?
- Why Can't I View the Subnet and Security Group Information When Creating a DMS Instance?
- How Do I Select Storage Space for a Kafka Instance?
- How Do I Choose Between High I/O and Ultra-high I/O?
- Which Capacity Threshold Policy Should I Use?
- Which Kafka Versions Are Supported?
- What Is the ZooKeeper Address of a Kafka Instance?
- Are Kafka Instances in Cluster Mode?
- Can I Modify the Port for Accessing a Kafka Instance?
- How Long Are Kafka SSL Certificates Valid for?
- How Do I Synchronize Data from One Kafka Instance to Another?
- How Do I Change the SASL_SSL Setting of a Kafka Instance?
- How Do I Modify the SASL Mechanism?
- Will a Kafka Instance Be Restarted After Its Enterprise Project Is Modified?
- Are Kafka Brokers and ZooKeeper Deployed on the Same VM or on Different VMs?
- Which Cipher Suites Are Supported by Kafka?
- Can I Change an Instance from Single-AZ Deployment to Multi-AZ Deployment?
- Does DMS for Kafka Support Cross-AZ Disaster Recovery? Where Can I View the AZs Configured for an Existing Instance?
- Do Kafka Instances Support Disk Encryption?
- Can I Change the VPC and Subnet After a Kafka Instance Is Created?
- Where Can I Find Kafka Streams Use Cases?
- Can I Upgrade Kafka Instances?
- Why Is the Version on the Console Different from That in Kafka Manager?
- Specification Modification
-
Connections
- How Do I Select and Configure a Security Group?
- Can I Access a Kafka Instance Over a Public Network?
- How Many Connection Addresses Does a Kafka Instance Have by Default?
- Do Kafka Instances Support Cross-Region Access?
- Do Kafka Instances Support Cross-VPC Access?
- Do Kafka Instances Support Cross-Subnet Access?
- Does DMS for Kafka Support Authentication with Kerberos?
- Does DMS for Kafka Support Password-Free Access?
- Does DMS for Kafka Support Authentication on Clients by the Server?
- Can I Use PEM SSL Truststore When Connecting to a Kafka Instance with SASL_SSL Enabled?
- What Are the Differences Between JKS and CRT Certificates?
- Which TLS Version Does DMS for Kafka Support?
- Is There a Limit on the Number of Client Connections to a Kafka Instance?
- How Many Connections Are Allowed from Each IP Address?
- Can I Change the Private Network Addresses of a Kafka Instance?
- Is the Same SSL Certificate Used for Different Instances?
- Why Is It Not Recommended to Use a Sarama Client for Messaging?
-
Topics and Partitions
- Is There a Limit on the Number of Topics in a Kafka Instance?
- Why Is Partition Quantity Limited?
- Can I Reduce the Partition Quantity?
- Why Do I Fail to Create Topics?
- Do Kafka Instances Support Batch Importing Topics or Automatic Topic Creation?
- Why Do Deleted Topics Still Exist?
- Can I View the Disk Space Used by a Topic?
- Can I Add ACL Permissions for Topics?
- What Should I Do If Kafka Storage Space Is Used Up Because Retrieved Messages Are Not Deleted?
- How Do I Increase the Partition Quantity?
- Will a Kafka Instance Be Restarted After Its Automatic Topic Creation Setting Is Modified?
- Can I Delete Unnecessary Topics in a Consumer Group?
- What Can I Do If a Consumer Fails to Retrieve Messages from a Topic Due to Insufficient Permissions?
- Why Does an Instance Contain Default Topics __trace and __consumer_offsets?
- Consumer Groups
-
Messages
- What Is the Maximum Size of a Message that Can be Created?
- Why Does Message Poll Often Fail During Rebalancing?
- Why Can't I Query Messages on the Console?
- What Can I Do If Kafka Messages Are Accumulated?
- Why Do Messages Still Exist After the Retention Period Elapses?
- Do Kafka Instances Support Delayed Message Delivery?
- How Do I View the Number of Accumulated Messages?
- Why Is the Message Creation Time Displayed as Year 1970?
-
Kafka Manager
- Can I Configure a Kafka Manager Account to Be Read-Only?
- Why Can't I See Broker Information After Logging In to Kafka Manager?
- Yikes! Insufficient partition balance when creating topic : projectman_project_enterprise_project Try again.
- Can I Query the Body of a Message by Using Kafka Manager?
- Can I Change the Port of the Kafka Manager Web UI?
- Which Topic Configurations Can Be Modified on Kafka Manager?
- Why Is Information Displayed on Kafka Manager Inconsistent with Cloud Eye Monitoring Data?
- How Do I Change a Partition Leader for a Topic in Kafka Manager?
- Monitoring & Alarm
-
Instances
-
Troubleshooting
- Troubleshooting Kafka Connection Exceptions
- Troubleshooting 6-Min Latency Between Message Creation and Retrieval
- Troubleshooting Message Creation Failures
- Troubleshooting Topic Deletion Failures
- Troubleshooting Failure to Log In to Kafka Manager in Windows
- Troubleshooting Error "Topic {{topic_name}} not present in metadata after 60000 ms" During Message Production or Consumption
- Change History
- API Reference (Kuala Lumpur Region)
-
User Guide (ME-Abu Dhabi Region)
- General Reference
Show all
Copied.
Kafka Instance TPS
TPS tests can be performed in the following scenarios:
- Scenario 1 (whether SASL is enabled): same topic, different SASL settings
- Scenario 2 (synchronous or asynchronous replication): same instance, topics with different replication settings
- Scenario 3 (synchronous or asynchronous flushing): same instance, topics with different flushing settings
- Scenario 4 (disk type): same topic, instances with different disk types
- Scenario 5 (number of partitions): same instance, topics with different number of partitions
Environment
Perform the following steps to set up the test environment.
- Purchase Kafka instances with parameters specified in Table 1. For more information, see Buying a Kafka Instance.
Table 1 Instance parameters Instance Name
Brokers
Broker Flavor
SASL
Storage space per broker
kafka-01
3
kafka.2u4g.cluster
Yes
Ultra-high I/O
kafka-02
3
kafka.4u8g.cluster
Yes
Ultra-high I/O
kafka-03
3
kafka.8u16g.cluster
Yes
Ultra-high I/O
kafka-04
3
kafka.12u24g.cluster
Yes
Ultra-high I/O
kafka-05
3
kafka.16u32g.cluster
Yes
Ultra-high I/O
kafka-06
3
kafka.2u4g.cluster
No
Ultra-high I/O
kafka-07
3
kafka.4u8g.cluster
No
Ultra-high I/O
kafka-08
3
kafka.8u16g.cluster
No
Ultra-high I/O
kafka-09
3
kafka.12u24g.cluster
No
Ultra-high I/O
kafka-10
3
kafka.16u32g.cluster
No
Ultra-high I/O
kafka-11
3
kafka.2u4g.cluster
No
High I/O
kafka-12
3
kafka.4u8g.cluster
No
High I/O
kafka-13
3
kafka.8u16g.cluster
No
High I/O
kafka-14
3
kafka.12u24g.cluster
No
High I/O
kafka-15
3
kafka.16u32g.cluster
No
High I/O
After the purchase, obtain Address (Private Network, Plaintext) on the instance details page.
- Create topics with parameters specified in Table 2 for each instance purchased above. For details about how to create topics, see Creating a Kafka Topic.
- Obtain the test tool.
Obtain Kafka CLI v2.7.2.
- Purchase a server for the client.
Buy a Linux ECS (with the same region, AZ, VPC, subnet, and security group as the Kafka instance). For details about how to purchase an ECS, see Purchasing a Custom ECS.
Perform the following operations on the ECSs:
- Install Java JDK and configure the environment variables JAVA_HOME and PATH.
export JAVA_HOME=/root/jdk1.8.0_231 export PATH=$JAVA_HOME/bin:$PATH
- Download Kafka CLI v2.7.2 and decompress it.
tar -zxf kafka_2.12-2.7.2.tgz
- Install Java JDK and configure the environment variables JAVA_HOME and PATH.
Script
./kafka-producer-perf-test.sh --producer-props bootstrap.servers=${connection address} acks=1 batch.size=16384 linger.ms=10 --topic ${topic name} --num-records 10000000 --record-size 1024 --throughput -1 --producer.config ../config/producer.properties
- bootstrap.servers: address of the Kafka instance obtained in 1.
- acks: message synchronization policy. acks=1 indicates asynchronous replication, and acks=-1 indicates synchronous replication.
- batch.size: size of messages sent in each batch, in bytes.
- linger.ms: interval between two batches.
- topic: topic name set in 2.
- num-records: total number of messages to be sent.
- record-size: size of each message.
- throughput: number of messages sent per second.
Result
Scenario 1 (whether SASL is enabled): same topic (30 partitions, 3 replicas, asynchronous replication, and asynchronous flushing), instances with SASL enabled or disabled. The test result is as follows.
Instance Flavor |
Storage space per broker |
Brokers |
TPS (SASL Enabled) |
TPS (SASL Disabled) |
---|---|---|---|---|
kafka.2u4g.cluster |
Ultra-high I/O |
3 |
100,000 |
280,000 |
kafka.4u8g.cluster |
Ultra-high I/O |
3 |
170,000 |
496,000 |
kafka.8u16g.cluster |
Ultra-high I/O |
3 |
200,000 |
730,000 |
kafka.12u24g.cluster |
Ultra-high I/O |
3 |
320,000 |
790,000 |
kafka.16u32g.cluster |
Ultra-high I/O |
3 |
360,000 |
1,000,000 |
Conclusion: When messages are produced to Kafka instances with the same flavor and topic but different access modes, instances without SASL show higher TPS than those with SASL.
Scenario 2 (synchronous/asynchronous replication): same instance (ultra-high I/O, three brokers, SASL disabled), topics with different replication settings, and number of producer processes is three. The test result is as follows.
Instance Flavor |
Synchronous Flushing |
Replicas |
Partitions |
TPS (Synchronous Replication) |
TPS (Asynchronous Replication) |
---|---|---|---|---|---|
kafka.2u4g.cluster |
No |
3 |
30 |
100,000 |
280,000 |
kafka.4u8g.cluster |
No |
3 |
30 |
230,000 |
496,000 |
kafka.8u16g.cluster |
No |
3 |
30 |
342,000 |
730,000 |
kafka.12u24g.cluster |
No |
3 |
30 |
383,000 |
790,000 |
kafka.16u32g.cluster |
No |
3 |
30 |
485,000 |
1,000,000 |
Conclusion: When messages are produced to different topics of a Kafka instance, topics with asynchronous replication show higher TPS than those with synchronous replication when other topic parameters are the same.
Scenario 3 (synchronous/asynchronous replication flushing): same instance (ultra-high I/O, three brokers, SASL disabled), topics with different flushing settings. The test result is as follows.
Instance Flavor |
Synchronous Replication |
Replicas |
Partitions |
TPS (Synchronous Flushing) |
TPS (Asynchronous Flushing) |
---|---|---|---|---|---|
kafka.2u4g.cluster |
No |
3 |
30 |
30,000 |
280,000 |
kafka.4u8g.cluster |
No |
3 |
30 |
32,500 |
496,000 |
kafka.8u16g.cluster |
No |
3 |
30 |
36,100 |
730,000 |
kafka.12u24g.cluster |
No |
3 |
30 |
37,400 |
790,000 |
kafka.16u32g.cluster |
No |
3 |
30 |
40,400 |
1,000,000 |
Conclusion: When messages are produced to different topics of a Kafka instance, topics with asynchronous flushing show significantly higher TPS than those with synchronous flushing when other topic parameters are the same.
Scenario 4 (different disk types): same topic (30 partitions, 3 replicas, asynchronous replication, and asynchronous flushing) with different disk types. The test result is as follows.
Instance Flavor |
Brokers |
SASL |
TPS (High I/O) |
TPS (Ultra-High I/O) |
---|---|---|---|---|
kafka.2u4g.cluster |
3 |
No |
110,000 |
250,000 |
kafka.4u8g.cluster |
3 |
No |
135,000 |
380,000 |
kafka.8u16g.cluster |
3 |
No |
213,000 |
480,000 |
kafka.12u24g.cluster |
3 |
No |
240,000 |
577,000 |
kafka.16u32g.cluster |
3 |
No |
280,000 |
840,000 |
Conclusion: When messages are produced to the same topics of Kafka instances with the same flavor but different disk types, instances with ultra-high I/O disks show higher TPS than those with high I/O disks.
Scenario 5 (different numbers of partitions): same instance (ultra-high I/O, three brokers, SASL disabled), topics with different number of partitions. The test result is as follows.
Instance Flavor |
Synchronous Flushing |
Synchronous Replication |
Replicas |
TPS (3 Partitions) |
TPS (12 Partitions) |
TPS (100 Partitions) |
---|---|---|---|---|---|---|
kafka.2u4g.cluster |
No |
No |
3 |
250,000 |
260,000 |
250,000 |
kafka.4u8g.cluster |
No |
No |
3 |
330,000 |
280,000 |
260,000 |
kafka.8u16g.cluster |
No |
No |
3 |
480,000 |
410,000 |
340,000 |
kafka.12u24g.cluster |
No |
No |
3 |
570,000 |
750,000 |
520,000 |
kafka.16u32g.cluster |
No |
No |
3 |
840,000 |
1,000,000 |
630,000 |
Conclusion: When messages are produced to topics with different partition quantities of a Kafka instance, instances with more partitions show higher performance when other parameters are the same. However, performance reaches a peak and then deteriorates when partitions continue to increase.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot