Using the Kafka Balancing Tool to Limit the Production and Consumption Speed

Updated on 2024-12-13 GMT+08:00

View PDF

Scenario

Use the kafka-configs.sh command-line tool on the client side to adjust Kafka cluster settings in line with service demands. This tool also enables the regulation of Kafka message throughput at various scopes: topic, user, and client levels.

NOTE:

This function is available in MRS 3.3.1 and later versions.

Prerequisites

The MRS cluster administrator has learned service requirements. You have created a Kafka component service user that belongs to the kafkaadmin user group (this is not required for normal clusters).
The Kafka client has been installed in a directory, for example, /opt/client.

Procedure

Log in as a client installation user to the node on which the Kafka client is installed.
Switch to the Kafka client installation directory, for example, /opt/client.

cd /opt/client
Configure environment variables.

source bigdata_env
Authenticate the user (skip this step for normal clusters).

kinit Component service user
Switch to the Kafka client installation directory.

cd Kafka/kafka
Use kafka-configs.sh to manage Kafka traffic. Below are the commonly used commands.
NOTE:
The subsequent section details the retrieval process for certain command parameters. The actual parameter values should be used when available.
- Service IP address of any ZooKeeper node: Log in to FusionInsight Manager, choose Cluster > Services > ZooKeeper and click Instance. You can choose the service IP address of any ZooKeeper instance, for example, 192.168.20.36.
- clientPort: Log in to FusionInsight Manager, choose Cluster > Services > ZooKeeper, click Configurations, and click All Configurations. On the displayed page, search for and record the value of clientPort, for example, 24002.
- Kafka cluster IP address: Log in to FusionInsight Manager, click Services, click Kafka, and click Instance. You can choose the service IP address of any Broker instance, for example, 192.168.20.36.
- The IP port ID is 21007 for Kafka clusters with Kerberos authentication enabled. For normal Kafka clusters, the IP port ID is 9092.
- Client ID: Log in to the Kafka client, run the following command to obtain the value of CLIENT-ID. For example, the obtained client ID is clientA.
  bin/kafka-consumer-groups.sh --describe --bootstrap-server Kafka cluster IP address:Port --all-groups --command-config config/consumer.properties
- Topic-level production traffic limiting
  bin/kafka-configs.sh --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --alter --add-config 'producer_byte_rate=Production traffic limiting speed' --entity-type topics_limit --entity-name Topic name
  
  The following is an example:
  
  bin/kafka-configs.sh --zookeeper 192.168.20.36:24002/kafka --alter --add-config 'producer_byte_rate=10485760' --entity-type topics_limit --entity-name testTopic-01
- Topic-level consumption rate limiting
  bin/kafka-configs.sh --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --alter --add-config 'consumer_byte_rate=Consumption traffic limiting speed' --entity-type topics_limit --entity-name Topic name
  
  The following is an example:
  
  bin/kafka-configs.sh --zookeeper 192.168.20.36:24002/kafka --alter --add-config 'consumer_byte_rate=1048576' --entity-type topics_limit --entity-name testTopic-01
- User-level production traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'producer_byte_rate=Production traffic limiting speed' --entity-type users --entity-name Component service user --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'producer_byte_rate=10485760' --entity-type users --entity-name kafkauser --command-config config/client.properties
- User-level consumption traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'consumer_byte_rate=Consumption traffic limiting speed' --entity-type users --entity-name Component service user --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'consumer_byte_rate=1048576' --entity-type users --entity-name kafkauser --command-config config/client.properties
- Client-level production traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'producer_byte_rate=Production traffic limiting speed' --entity-type clients --entity-name Client ID --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'producer_byte_rate=10485760' --entity-type clients --entity-name clientA --command-config config/client.properties
- Client-level consumption traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'consumer_byte_rate=Consumption traffic limiting speed' --entity-type clients --entity-name Client ID --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'consumer_byte_rate=1048576' --entity-type clients --entity-name clientA --command-config config/client.properties
- Combined user and client production and consumption traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'producer_byte_rate=Production traffic limiting speed,consumer_byte_rate=Consumption traffic limiting speed' --entity-type users --entity-name Component service user --entity-type clients --entity-name Client ID --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'producer_byte_rate=1048576,consumer_byte_rate=1048576' --entity-type users --entity-name kafkauser --entity-type clients --entity-name clientD --command-config config/client.properties
- Viewing traffic limiting information
  bin/kafka-configs.sh --describe --bootstrap-server Kafka cluster IP address:Port number --entity-type myType --entity-name myName --command-config config/client.properties
  NOTE:
  - The permissible values for myType are topics_limit, users, and clients, and those for myName are Topic name, Component service user, and Client ID.
  - producer_byte_rate and consumer_byte_rate indicate the production rate and consumption rate, respectively. The unit is byte/s.
  - If myType is set to topics_limit, change the command to the following:
    bin/kafka-configs.sh --describe --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --entity-type topics_limit --entity-name Topic name
  The following is an example:
  
  bin/kafka-configs.sh --describe --bootstrap-server 192.168.20.36:21007 --entity-type users --entity-name kafkauser --entity-type clients --entity-name clientD --command-config config/client.properties
- Canceling traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --delete-config 'producer_byte_rate,consumer_byte_rate' --entity-type myType --entity-name myName --command-config config/client.properties
  NOTE:
  - The permissible values for myType are topics_limit, users, and clients, and those for myName are Topic name, Component service user, and Client ID.
  - If myType is set to topics_limit, change the command to the following:
    bin/kafka-configs.sh --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --alter --delete-config 'producer_byte_rate,consumer_byte_rate' --entity-type topics_limit --entity-name Topic name
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --delete-config 'producer_byte_rate,consumer_byte_rate' --entity-type users --entity-name kafkauser --entity-type clients --entity-name client --command-config config/client.properties

Parent topic: Kafka O&M Management

Previous topic: Migrating Data Between Kafka Nodes

Next topic: Configure Lag Alarm Rules

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot