Viewing Kafka Rebalancing Logs

Constraints

• Rebalancing logging is not available for instances created before April 6, 2023.
• Rebalancing logs are stored for seven days by default. To retain them longer, see Modifying a Log Group
• Enabling rebalancing logging will create a log group, log stream, and dashboard in LTS. Fees are generated based on the log volume. For details, see LTS pricing details.

Prerequisites

• Ensure that you have permissions to create log groups and log streams in LTS.
• Rebalancing logging can be enabled or disabled only when the Kafka instance is in the Running state.

Enabling Rebalancing Logging

1. Log in to the console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is.

3. Click and choose Middleware > Distributed Message Service (for Kafka) to open the console of DMS for Kafka.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose Rebalancing Logs.
6. Click Enable Logging. If the message "Rebalancing logging enabled" is displayed in the upper right corner of the page, the rebalancing log function is enabled successfully.

   Enabling rebalancing logging will create a log group and log stream in LTS.

Viewing Rebalancing Logs

1. Log in to the console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is.

3. Click and choose Middleware > Distributed Message Service (for Kafka) to open the console of DMS for Kafka.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose Rebalancing Logs.
6. On the Dashboard tab page, view the number of consumer group rebalancing times and reasons. On the Logs tab page, view rebalancing logs.

   To search for logs, see Log Search.

   An example rebalancing log:

   {
       "level":"INFO",
       "timestamp":"2023-03-23 17:23:22,906",
       "message":{
           "leaderId":"consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635",
           "generationId":"1",
           "reason":"Assignment received from leader for group KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0 for generation 1",
           "groupId":"KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0",
           "coordinatorId":"0",
           "type":"END_REBALANCE",
           "group":"GroupMetadata(groupId=KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0, generation=1, protocolType=Some(consumer), currentState=CompletingRebalance, members=Map(consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635 -> MemberMetadata(memberId=consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635, clientId=consumer-1, clientHost=/172.31.2.168, sessionTimeoutMs=10000, rebalanceTimeoutMs=300000, supportedProtocols=List(range), )))"
       }
   }

   Table 1 describes the parameters.

   Table 1 Rebalancing parameters

   Parameter	Description
   level	Level of the rebalancing logs. The only value is INFO.
   timestamp	Time of rebalancing.
   leaderId	Leader consumer ID.
   generationId	Generation ID of the consumer group. Generation is the number of times that a consumer group performs rebalancing. It is incremented by 1 each time a rebalancing is complete.
   reason	Reason for triggering rebalancing.
   groupId	Consumer group ID.
   coordinatorId	Broker where the Coordinator component is.
   type	Operation that triggered rebalancing. Values: JOIN_GROUP: A new consumer is added to a consumer group. OVER_CAPACITY: The group limit is exceeded. UPDATE_MEMBER: The consumer metadata is updated. PROTOCOL_CHANGE: The protocol is changed. HEARTBEAT_EXPIRED: The consumer heartbeat timed out. SYNC_GROUP: The reassignment plan is synchronized. END_REBALANCE: Rebalancing ended. LEAVE_GROUP: A consumer left a consumer group.
   group	Information about consumers in the consumer group.

Disabling Rebalancing Logging

1. Log in to the console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is.

3. Click and choose Middleware > Distributed Message Service (for Kafka) to open the console of DMS for Kafka.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose Rebalancing Logs.
6. In the upper right corner of the page, click Disable Logging. In the dialog box that is displayed, click OK.
   

   This only disables the rebalancing logging function. The log groups and log streams on LTS are retained and still generate fees. If you no longer need the logs, delete the log groups and log streams on LTS.