Help Center/ Distributed Message Service for Kafka/ User Guide/ Managing Consumer Groups/ Viewing Kafka Rebalancing Logs
Updated on 2025-02-14 GMT+08:00
View PDF
Share

Viewing Kafka Rebalancing Logs

Rebalancing is to reallocate subscription relationships between consumers and topic partitions in a consumer group. During rebalancing, all consumers in the consumer group stop consuming messages until rebalancing completes.

Possible causes of rebalancing:

  • The number of consumer group members changes. For example, a new consumer joins the group or a consumer quits the group.
  • The number of topics subscribed to by a consumer group changes.
  • The number of topic partitions subscribed to by a consumer group changes.

Rebalancing logs record rebalancing details, including the time, reason, and triggering client of rebalancing. This section describes how to view rebalancing logs on the console.

Rebalancing logs are stored and can be queried in Log Tank Service (LTS).

Notes and Constraints

  • Rebalancing logging is not available for instances created before April 6, 2023.
  • Unavailable for single-node instances.
  • Rebalancing logs are stored for seven days by default. To retain them longer, see Modifying a Log Group.
  • Enabling rebalancing logging will create a log group, log stream, and dashboard in LTS. Fees are generated based on the log volume. For details, see LTS pricing details.

Prerequisites

  • Ensure that you have permissions to create log groups and log streams in LTS.
  • Rebalancing logging can be enabled or disabled only when the Kafka instance is in the Running state.

Enabling Rebalancing Logging

  1. Log in to the console.
  2. Click in the upper left corner to select the region where your instance is located.
  3. Click and choose Middleware > Distributed Message Service (for Kafka) to open the console of DMS for Kafka.
  4. In the navigation pane, choose Kafka Instances.
  5. Click the desired instance to go to the instance details page.
  6. In the navigation pane, choose Rebalancing Logs.
  7. Click Enable Logging. The Enable Logging dialog box is displayed.
  8. Click OK. The Background Tasks page is displayed. The rebalancing log function is enabled when the rebalancing logging task is in the Successful state.

    Enabling rebalancing logging will create a log group and log stream in LTS.

Viewing Rebalancing Logs

  1. Log in to the console.
  2. Click in the upper left corner to select the region where your instance is located.
  3. Click and choose Middleware > Distributed Message Service (for Kafka) to open the console of DMS for Kafka.
  4. In the navigation pane, choose Kafka Instances.
  5. Click the desired instance to go to the instance details page.
  6. In the navigation pane, choose Rebalancing Logs.
  7. On the Dashboard tab page, view the number of consumer group rebalancing times and reasons. On the Logs tab page, view rebalancing logs.

    To search for logs, see Log Search.

    An example rebalancing log:

    {
    "level":"INFO",
    "timestamp":"2023-03-23 17:23:22,906",
    "message":{
        "leaderId":"consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635",
        "generationId":"1",
        "reason":"Assignment received from leader for group KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0 for generation 1",
        "groupId":"KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0",
        "coordinatorId":"0",
        "type":"END_REBALANCE",
        "group":"GroupMetadata(groupId=KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0, generation=1, protocolType=Some(consumer), currentState=CompletingRebalance, members=Map(consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635 -> MemberMetadata(memberId=consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635, clientId=consumer-1, clientHost=/172.31.2.168, sessionTimeoutMs=10000, rebalanceTimeoutMs=300000, supportedProtocols=List(range), )))"
    }
}

    Table 1 describes the parameters.

    Table 1 Rebalancing parameters

    Parameter

    Description

    level

    Level of the rebalancing logs. The only value is INFO.

    timestamp

    Time of rebalancing.

    leaderId

    Leader consumer ID.

    generationId

    Generation ID of the consumer group. Generation is the number of times that a consumer group performs rebalancing. It is incremented by 1 each time a rebalancing is complete.

    reason

    Reason for triggering rebalancing.

    groupId

    Consumer group ID.

    coordinatorId

    Broker where the Coordinator component is.

    type

    Operation that triggered rebalancing. Values:

    • JOIN_GROUP: A new consumer is added to a consumer group.
    • OVER_CAPACITY: The group limit is exceeded.
    • UPDATE_MEMBER: The consumer metadata is updated.
    • PROTOCOL_CHANGE: The protocol is changed.
    • HEARTBEAT_EXPIRED: The consumer heartbeat timed out.
    • SYNC_GROUP: The reassignment plan is synchronized.
    • END_REBALANCE: Rebalancing ended.
    • LEAVE_GROUP: A consumer left a consumer group.

    group

    Information about consumers in the consumer group.

Disabling Rebalancing Logging

  1. Log in to the console.
  2. Click in the upper left corner to select the region where your instance is located.
  3. Click and choose Middleware > Distributed Message Service (for Kafka) to open the console of DMS for Kafka.
  4. In the navigation pane, choose Kafka Instances.
  5. Click the desired instance to go to the instance details page.
  6. In the navigation pane, choose Rebalancing Logs.
  7. Click Disable Logging in the upper right corner. The Disable Logging dialog box is displayed.
  8. Click OK. The Background Tasks page is displayed. The rebalancing log function is disabled when the rebalancing logging task is in the Successful state.

    This only disables the rebalancing logging function. The log groups and log streams on LTS are retained and still generate fees. If you no longer need the logs, delete the log groups and log streams on LTS.

Parent topic: Managing Consumer Groups