Using the Kafka Balancing Tool to Limit the Production and Consumption Speed

Scenario

Use the kafka-configs.sh command-line tool on the client side to adjust Kafka cluster settings in line with service demands. This tool also enables the regulation of Kafka message throughput at various scopes: topic, user, and client levels.

This function is available in MRS 3.3.1 and later versions.

Prerequisites

The MRS cluster administrator has learned service requirements. You have created a Kafka component service user that belongs to the kafkaadmin user group (this is not required for normal clusters).
The Kafka client has been installed in a directory, for example, /opt/client.

Procedure

Log in as a client installation user to the node on which the Kafka client is installed.
Switch to the Kafka client installation directory, for example, /opt/client.

cd /opt/client
Configure environment variables.

source bigdata_env
Authenticate the user (skip this step for normal clusters).

kinit Component service user
Switch to the Kafka client installation directory.

cd Kafka/kafka
Use kafka-configs.sh to manage Kafka traffic. Below are the commonly used commands.
The subsequent section details the retrieval process for certain command parameters. The actual parameter values should be used when available.
- Service IP address of any ZooKeeper node: Log in to FusionInsight Manager, choose Cluster > Services > ZooKeeper and click Instance. You can choose the service IP address of any ZooKeeper instance, for example, 192.168.20.36.
- clientPort: Log in to FusionInsight Manager, choose Cluster > Services > ZooKeeper, click Configurations, and click All Configurations. On the displayed page, search for and record the value of clientPort, for example, 24002.
- Kafka cluster IP address: Log in to FusionInsight Manager, click Services, click Kafka, and click Instance. You can choose the service IP address of any Broker instance, for example, 192.168.20.36.
- The IP port ID is 21007 for Kafka clusters with Kerberos authentication enabled. For normal Kafka clusters, the IP port ID is 9092.
- Client ID: Log in to the Kafka client, run the following command to obtain the value of CLIENT-ID. For example, the obtained client ID is clientA.
  bin/kafka-consumer-groups.sh --describe --bootstrap-server Kafka cluster IP address:Port --all-groups --command-config config/consumer.properties
- Topic-level production traffic limiting
  bin/kafka-configs.sh --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --alter --add-config 'producer_byte_rate=Production traffic limiting speed' --entity-type topics_limit --entity-name Topic name
  
  The following is an example:
  
  bin/kafka-configs.sh --zookeeper 192.168.20.36:24002/kafka --alter --add-config 'producer_byte_rate=10485760' --entity-type topics_limit --entity-name testTopic-01
- Topic-level consumption rate limiting
  bin/kafka-configs.sh --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --alter --add-config 'consumer_byte_rate=Consumption traffic limiting speed' --entity-type topics_limit --entity-name Topic name
  
  The following is an example:
  
  bin/kafka-configs.sh --zookeeper 192.168.20.36:24002/kafka --alter --add-config 'consumer_byte_rate=1048576' --entity-type topics_limit --entity-name testTopic-01
- User-level production traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'producer_byte_rate=Production traffic limiting speed' --entity-type users --entity-name Component service user --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'producer_byte_rate=10485760' --entity-type users --entity-name kafkauser --command-config config/client.properties
- User-level consumption traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'consumer_byte_rate=Consumption traffic limiting speed' --entity-type users --entity-name Component service user --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'consumer_byte_rate=1048576' --entity-type users --entity-name kafkauser --command-config config/client.properties
- Client-level production traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'producer_byte_rate=Production traffic limiting speed' --entity-type clients --entity-name Client ID --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'producer_byte_rate=10485760' --entity-type clients --entity-name clientA --command-config config/client.properties
- Client-level consumption traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'consumer_byte_rate=Consumption traffic limiting speed' --entity-type clients --entity-name Client ID --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'consumer_byte_rate=1048576' --entity-type clients --entity-name clientA --command-config config/client.properties
- Combined user and client production and consumption traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --add-config 'producer_byte_rate=Production traffic limiting speed,consumer_byte_rate=Consumption traffic limiting speed' --entity-type users --entity-name Component service user --entity-type clients --entity-name Client ID --command-config config/client.properties
  
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --add-config 'producer_byte_rate=1048576,consumer_byte_rate=1048576' --entity-type users --entity-name kafkauser --entity-type clients --entity-name clientD --command-config config/client.properties
- Viewing traffic limiting information
  bin/kafka-configs.sh --describe --bootstrap-server Kafka cluster IP address:Port number --entity-type myType --entity-name myName --command-config config/client.properties
  - The permissible values for myType are topics_limit, users, and clients, and those for myName are Topic name, Component service user, and Client ID.
  - producer_byte_rate and consumer_byte_rate indicate the production rate and consumption rate, respectively. The unit is byte/s.
  - If myType is set to topics_limit, change the command to the following:
    bin/kafka-configs.sh --describe --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --entity-type topics_limit --entity-name Topic name
  The following is an example:
  
  bin/kafka-configs.sh --describe --bootstrap-server 192.168.20.36:21007 --entity-type users --entity-name kafkauser --entity-type clients --entity-name clientD --command-config config/client.properties
- Canceling traffic limiting
  bin/kafka-configs.sh --bootstrap-server Kafka cluster IP address:Port number --alter --delete-config 'producer_byte_rate,consumer_byte_rate' --entity-type myType --entity-name myName --command-config config/client.properties
  - The permissible values for myType are topics_limit, users, and clients, and those for myName are Topic name, Component service user, and Client ID.
  - If myType is set to topics_limit, change the command to the following:
    bin/kafka-configs.sh --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --alter --delete-config 'producer_byte_rate,consumer_byte_rate' --entity-type topics_limit --entity-name Topic name
  The following is an example:
  
  bin/kafka-configs.sh --bootstrap-server 192.168.20.36:21007 --alter --delete-config 'producer_byte_rate,consumer_byte_rate' --entity-type users --entity-name kafkauser --entity-type clients --entity-name client --command-config config/client.properties