Updated on 2024-05-30 GMT+08:00

Kafka Metrics

Introduction

This section describes metrics reported by DMS for Kafka to Cloud Eye as well as their namespaces and dimensions. You can use the Cloud Eye console or APIs to query the Kafka metrics and alarms, or view Kafka instance metrics on the Monitoring page of the DMS for Kafka console.

For example, you can call the API to query the monitoring data of the Disk Capacity Usage metric.

Namespace

SYS.DMS

Instance Metrics

Table 1 Instance metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

current_partitions

Partitions

Number of used partitions in the instance

Unit: count

0–1800

Kafka instance

1 minute

current_topics

Topics

Number of created topics in the instance

Unit: count

0–1800

Kafka instance

1 minute

group_msgs

Accumulated Messages

Total number of accumulated messages in all consumer groups of the instance

Unit: count

0–1,000,000,000

Kafka instance

1 minute

Broker Metrics

Table 2 Broker metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

broker_data_size

Message Size

Total size of messages in the broker

Unit: byte, KB, MB, GB, TB or PB

0–5,000,000,000,000

Kafka instance broker

1 minute

broker_messages_in_rate

Message Creation Rate

Number of messages created per second

Unit: count/s

0–500,000

Kafka instance broker

1 minute

broker_bytes_out_rate

Message Retrieval

Number of bytes retrieved per second

Unit: byte/s, KB/s, MB/s, or GB/s

0–500,000,000

Kafka instance broker

1 minute

broker_bytes_in_rate

Message Creation

Number of bytes created per second

Unit: byte/s, KB/s, MB/s, or GB/s

0–500,000,000

Kafka instance broker

1 minute

broker_public_bytes_in_rate

Public Inbound Traffic

Inbound traffic over public networks per second

Unit: byte/s, KB/s, MB/s, or GB/s

0–500,000,000

Kafka instance broker

1 minute

broker_public_bytes_out_rate

Public Outbound Traffic

Outbound traffic over public networks per second

Unit: byte/s, KB/s, MB/s, or GB/s

0–500,000,000

Kafka instance broker

1 minute

broker_fetch_mean

Average Message Retrieval Processing Duration

Average time that the broker spends processing message retrieval requests

Unit: ms

0–10,000

Kafka instance broker

1 minute

broker_produce_mean

Average Message Creation Processing Duration

Average time that the broker spends processing message creation requests

Unit: ms

0–10,000

Kafka instance broker

1 minute

broker_cpu_core_load

Average Load per CPU Core

Average load of each CPU core of the Kafka VM

Unit: %

0–20

Kafka instance broker

1 minute

broker_disk_usage

Disk Capacity Usage

Disk usage of the Kafka VM

Unit: %

0–100

Kafka instance broker

1 minute

broker_memory_usage

Memory Usage

Memory usage of the Kafka VM

Unit: %

0–100

Kafka instance broker

1 minute

broker_heap_usage

JVM Heap Memory Usage of Kafka

Heap memory usage of the Kafka JVM

Unit: %

0–100

Kafka instance broker

1 minute

broker_alive

Broker Alive

Whether the Kafka broker is alive

  • 1: alive
  • 0: not alive

Kafka instance broker

1 minute

broker_connections

Connections

Total number of TCP connections on the Kafka broker

Unit: count

0–65,535

Kafka instance broker

1 minute

broker_cpu_usage

CPU Usage

CPU usage of the Kafka VM

Unit: %

0–100

Kafka instance broker

1 minute

broker_disk_read_await

Average Disk Read Time

Average time for each disk I/O read in the monitoring period

Unit: ms

> 0

Kafka instance broker

1 minute

broker_disk_write_await

Average Disk Write Time

Average time for each disk I/O write in the monitoring period

Unit: ms

> 0

Kafka instance broker

1 minute

broker_total_bytes_in_rate

Inbound Traffic

Inbound traffic per second

Unit: byte/s

0–1,000,000,000

Kafka instance broker

1 minute

broker_total_bytes_out_rate

Outbound Traffic

Outbound traffic per second

Unit: byte/s

0–1,000,000,000

Kafka instance broker

1 minute

broker_disk_read_rate

Disk Read Speed

Read traffic on the disk

Unit: byte/s, KB/s, MB/s, or GB/s

≥ 0

Kafka instance broker

1 minute

broker_disk_write_rate

Disk Write Speed

Write traffic on the disk

Unit: byte/s, KB/s, MB/s, or GB/s

≥ 0

Kafka instance broker

1 minute

network_bandwidth_usage

Network Bandwidth Usage

Network bandwidth usage

Unit: %

0–100

Kafka instance broker

1 minute

Topic Metrics

Table 3 Topic metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

topic_bytes_in_rate

Message Creation

Number of bytes created per second

Unit: byte/s, KB/s, MB/s, or GB/s

NOTE:

This metric is available only when Scope is set to Basic monitoring on the By Topic tab page.

0–500,000,000

Topic in a Kafka instance

1 minute

topic_bytes_out_rate

Message Retrieval

Number of bytes retrieved per second

Unit: byte/s, KB/s, MB/s, or GB/s

NOTE:

This metric is available only when Scope is set to Basic monitoring on the By Topic tab page.

0–500,000,000

Topic in a Kafka instance

1 minute

topic_data_size

Message Size

Total size of messages in the queue

Unit: byte, KB, MB, GB, TB or PB

NOTE:

This metric is available only when Scope is set to Basic monitoring on the By Topic tab page.

0–5,000,000,000,000

Topic in a Kafka instance

1 minute

topic_messages

Total Messages

Total number of messages in the queue

Unit: count

NOTE:

This metric is available only when Scope is set to Basic monitoring on the By Topic tab page.

≥ 0

Topic in a Kafka instance

1 minute

topic_messages_in_rate

Message Creation Rate

Number of messages created per second

Unit: count/s

NOTE:

This metric is available only when Scope is set to Basic monitoring on the By Topic tab page.

0–500,000

Topic in a Kafka instance

1 minute

partition_messages

Partition Messages

Total number of messages in the partition

Unit: count

NOTE:

This metric is available only when Scope is set to Partition monitoring on the By Topic tab page.

≥ 0

Topic in a Kafka instance

1 minute

produced_messages

Created Messages

Number of messages that have been created

Unit: count

NOTE:

This metric is available only when Scope is set to Partition monitoring on the By Topic tab page.

≥ 0

Topic in a Kafka instance

1 minute

Consumer Group Metrics

Table 4 Consumer group metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

messages_consumed

Retrieved Messages

Number of messages that have been retrieved in the consumer group

Unit: count

NOTE:

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

≥ 0

Consumer group of a Kafka instance

1 minute

messages_remained

Available Messages

Number of messages that can be retrieved in the consumer group

Unit: count

NOTE:

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

≥ 0

Consumer group of a Kafka instance

1 minute

topic_messages_remained

Topic Available Messages

Number of remaining messages that can be retrieved from the specified topic in the consumer group

Unit: Count

NOTE:

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

topic_messages_consumed

Topic Retrieved Messages

Number of messages that have been retrieved from the specified topic in the consumer group

Unit: Count

NOTE:

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

consumer_messages_remained

Accumulated Messages (Consumer Available Messages)

Number of remaining messages that can be retrieved in the consumer group

Unit: Count

NOTE:
  • On the Cloud Eye console, this metric is available only when Queues is set to All queues on the Consumer Groups tab page.
  • On the Monitoring page of the DMS console, this metric is available only when Topic is set to All topics on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

consumer_messages_consumed

Consumer Retrieved Messages

Number of messages that have been retrieved in the consumer group

Unit: Count

NOTE:
  • On the Cloud Eye console, this metric is available only when Queues is set to All queues on the Consumer Groups tab page.
  • On the Monitoring page of the DMS console, this metric is available only when Topic is set to All topics on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

messages_consumed_per_min

Partition Consumption Rate

Number of messages consumed from the specified queue partition in the consumer group every minute

Unit: count/minute

NOTE:
  • This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.
  • Some instances do not support this metric. Check whether your instance supports it on the console.

0–30,000,000

Consumer group of a Kafka instance

1 minute

topic_messages_consumed_per_min

Queue Consumption Rate

Number of messages consumed from the specified queue in the consumer group every minute

Unit: count/minute

NOTE:
  • This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.
  • Some instances do not support this metric. Check whether your instance supports it on the console.

0–30,000,000

Consumer group of a Kafka instance

1 minute

consumer_messages_consumed_per_min

Consumer Group Consumption Rate

Number of messages consumed from the consumer group every minute

Unit: count/minute

NOTE:
  • This metric is available only when Topic is set to All topics on the By Consumer Group tab page.
  • Some instances do not support this metric. Check whether your instance supports it on the console.

0–30,000,000

Consumer group of a Kafka instance

1 minute

Dimension

Key

Value

kafka_instance_id

Kafka instance

kafka_broker

Kafka instance broker

kafka_topics

Kafka instance topic

kafka_partitions

Partition in a Kafka instance

kafka_groups-partitions

Partition consumer group in a Kafka instance

kafka_groups_topics

Topic consumer group in a Kafka instance

kafka_groups

Consumer group of a Kafka instance