Kafka Metrics
Introduction
This section describes metrics reported by DMS for Kafka to Cloud Eye as well as their namespaces and dimensions. You can use the Cloud Eye console or APIs to query the Kafka metrics and alarms, or view Kafka instance metrics on the Monitoring page of the DMS for Kafka console.
For example, you can call the API to query the monitoring data of the Disk Capacity Usage metric.
Namespace
SYS.DMS
Instance Metrics
Metric ID |
Metric Name |
Description |
Value Range |
Monitored Object |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|
current_partitions |
Partitions |
Number of used partitions in the instance Unit: Count |
0–100,000 |
Kafka instance |
1 minute |
current_topics |
Topics |
Number of created topics in the instance Unit: Count |
0–100,000 |
Kafka instance |
1 minute |
group_msgs |
Accumulated Messages |
Total number of accumulated messages in all consumer groups of the instance Unit: Count |
0–1,000,000,000 |
Kafka instance |
1 minute |
instance_bytes_in_rate |
Message Creation |
Number of bytes produced in the instance per second Unit: Byte/s, KiB/s, MiB/s, GiB/s, TiB/s, PiB/s
NOTE:
Some instances do not support this metric. Check whether your instance supports it on the console. |
0–1,000,000 |
Kafka instance |
1 minute |
instance_bytes_out_rate |
Message Retrieval |
Number of bytes consumed from the instance per second Unit: Byte/s, KiB/s, MiB/s, GiB/s, TiB/s, PiB/s
NOTE:
Some instances do not support this metric. Check whether your instance supports it on the console. |
0–1,000,000 |
Kafka instance |
1 minute |
Broker Metrics
Enabling Smart Connect for a Kafka instance creates two or more brokers. On the By Broker tab page, select "connector" for Node Type for Smart Connect broker metrics, or select "broker" for Node Type for Kafka instance broker metrics.
Metrics of Smart Connect brokers: disk capacity usage, memory usage, JVM heap memory usage, node alive status, and connections.
Metric ID |
Metric Name |
Description |
Value Range |
Monitored Object |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|
broker_data_size |
Message Size |
Total size of messages in the broker Unit: byte, KB, MB, GB, TB or PB |
0–5,000,000,000,000 |
Kafka instance broker |
1 minute |
broker_messages_in_rate |
Message Creation Rate |
Number of messages created per second Unit: count/s |
0–500,000 |
Kafka instance broker |
1 minute |
broker_bytes_out_rate |
Message Retrieval |
Number of bytes retrieved per second Unit: byte/s, KB/s, MB/s, or GB/s |
0–500,000,000 |
Kafka instance broker |
1 minute |
broker_bytes_in_rate |
Message Creation |
Number of bytes created per second Unit: byte/s, KB/s, MB/s, or GB/s |
0–500,000,000 |
Kafka instance broker |
1 minute |
broker_public_bytes_in_rate |
Public Inbound Traffic |
Inbound traffic over public networks per second Unit: byte/s, KB/s, MB/s, or GB/s
NOTE:
You can view this metric if public access has been enabled and EIPs have been assigned to the instance. |
0–500,000,000 |
Kafka instance broker |
1 minute |
broker_public_bytes_out_rate |
Public Outbound Traffic |
Outbound traffic over public networks per second Unit: byte/s, KB/s, MB/s, or GB/s
NOTE:
You can view this metric if public access has been enabled and EIPs have been assigned to the instance. |
0–500,000,000 |
Kafka instance broker |
1 minute |
broker_fetch_mean |
Average Message Retrieval Processing Duration |
Average time that the broker spends processing message retrieval requests Unit: ms |
0–10,000 |
Kafka instance broker |
1 minute |
broker_produce_mean |
Average Message Creation Processing Duration |
Average time that the broker spends processing message creation requests Unit: ms |
0–10,000 |
Kafka instance broker |
1 minute |
broker_cpu_core_load |
Average Load per CPU Core |
Average load of each CPU core of the Kafka VM Unit: % |
0–20 |
Kafka instance broker |
1 minute |
broker_disk_usage |
Disk Capacity Usage |
Disk usage of the Kafka VM Unit: % |
0–100 |
Kafka instance broker |
1 minute |
broker_memory_usage |
Memory Usage |
Memory usage of the Kafka VM Unit: % |
0–100 |
Kafka instance broker |
1 minute |
broker_heap_usage |
JVM Heap Memory Usage of Kafka |
Heap memory usage of the Kafka JVM Unit: % |
0–100 |
Kafka instance broker |
1 minute |
broker_alive |
Broker Alive |
Whether the Kafka broker is alive
NOTE:
This metric is supported by instances purchased in April 2020 or later. |
|
Kafka instance broker |
1 minute |
broker_connections |
Connections |
Total number of TCP connections on the Kafka broker Unit: count
NOTE:
This metric is supported by instances purchased in April 2020 or later. |
0–65,535 |
Kafka instance broker |
1 minute |
broker_cpu_usage |
CPU Usage |
CPU usage of the Kafka VM Unit: %
NOTE:
This metric is supported by instances purchased in April 2020 or later. |
0–100 |
Kafka instance broker |
1 minute |
broker_disk_read_await |
Average Disk Read Time |
Average time for each disk I/O read in the monitoring period Unit: ms
NOTE:
This metric is supported for instances purchased in June 2020 or later. |
> 0 |
Kafka instance broker |
1 minute |
broker_disk_write_await |
Average Disk Write Time |
Average time for each disk I/O write in the monitoring period Unit: ms
NOTE:
This metric is supported for instances purchased in June 2020 or later. |
> 0 |
Kafka instance broker |
1 minute |
broker_total_bytes_in_rate |
Inbound Traffic |
Inbound traffic per second Unit: byte/s
NOTE:
This metric is supported for instances purchased in June 2020 or later. |
0–1,000,000,000 |
Kafka instance broker |
1 minute |
broker_total_bytes_out_rate |
Outbound Traffic |
Outbound traffic per second Unit: byte/s
NOTE:
This metric is supported for instances purchased in June 2020 or later. |
0–1,000,000,000 |
Kafka instance broker |
1 minute |
broker_disk_read_rate |
Disk Read Speed |
Read traffic on the disk Unit: byte/s, KB/s, MB/s, or GB/s
NOTE:
This metric is supported for instances purchased on or after May 16, 2022. |
≥ 0 |
Kafka instance broker |
1 minute |
broker_disk_write_rate |
Disk Write Speed |
Write traffic on the disk Unit: byte/s, KB/s, MB/s, or GB/s
NOTE:
This metric is supported for instances purchased on or after May 16, 2022. |
≥ 0 |
Kafka instance broker |
1 minute |
network_bandwidth_usage |
Network Bandwidth Usage |
Network bandwidth usage Unit: %
NOTE:
|
0–100 |
Kafka instance broker |
1 minute |
Topic Metrics
Metric ID |
Metric Name |
Description |
Value Range |
Monitored Object |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|
topic_bytes_in_rate |
Message Creation |
Number of bytes created per second Unit: byte/s, KB/s, MB/s, or GB/s
NOTE:
This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
0–500,000,000 |
Topic in a Kafka instance |
1 minute |
topic_bytes_out_rate |
Message Retrieval |
Number of bytes retrieved per second Unit: byte/s, KB/s, MB/s, or GB/s
NOTE:
This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
0–500,000,000 |
Topic in a Kafka instance |
1 minute |
topic_data_size |
Message Size |
Total size of messages in the queue Unit: byte, KB, MB, GB, TB or PB
NOTE:
This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
0–5,000,000,000,000 |
Topic in a Kafka instance |
1 minute |
topic_messages |
Total Messages |
Total number of messages in the queue Unit: count
NOTE:
This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
≥ 0 |
Topic in a Kafka instance |
1 minute |
topic_messages_in_rate |
Message Creation Rate |
Number of messages created per second Unit: count/s
NOTE:
This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
0–500,000 |
Topic in a Kafka instance |
1 minute |
partition_messages |
Partition Messages |
Total number of messages in the partition Unit: count
NOTE:
This metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page. |
≥ 0 |
Topic in a Kafka instance |
1 minute |
produced_messages |
Created Messages |
Number of messages that have been created Unit: count
NOTE:
This metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page. |
≥ 0 |
Topic in a Kafka instance |
1 minute |
Consumer Group Metrics
Metric ID |
Metric Name |
Description |
Value Range |
Monitored Object |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|
messages_consumed |
Retrieved Messages |
Number of messages that have been retrieved in the consumer group Unit: count
NOTE:
This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page. |
≥ 0 |
Consumer group of a Kafka instance |
1 minute |
messages_remained |
Available Messages |
Number of messages that can be retrieved in the consumer group Unit: count
NOTE:
This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page. |
≥ 0 |
Consumer group of a Kafka instance |
1 minute |
topic_messages_remained |
Topic Available Messages |
Number of remaining messages that can be retrieved from the specified topic in the consumer group Unit: Count
NOTE:
This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page. |
0 to 263–1 |
Consumer group of a Kafka instance |
1 minute |
topic_messages_consumed |
Topic Retrieved Messages |
Number of messages that have been retrieved from the specified topic in the consumer group Unit: Count
NOTE:
This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page. |
0 to 263–1 |
Consumer group of a Kafka instance |
1 minute |
consumer_messages_remained |
Accumulated Messages (Consumer Available Messages) |
Number of remaining messages that can be retrieved in the consumer group Unit: Count
NOTE:
This metric is available only when Topic is set to All topics on the By Consumer Group tab page. |
0 to 263–1 |
Consumer group of a Kafka instance |
1 minute |
consumer_messages_consumed |
Consumer Retrieved Messages |
Number of messages that have been retrieved in the consumer group Unit: Count
NOTE:
This metric is available only when Topic is set to All topics on the By Consumer Group tab page. |
0 to 263–1 |
Consumer group of a Kafka instance |
1 minute |
messages_consumed_per_min |
Partition Consumption Rate |
Number of messages consumed from the specified queue partition in the consumer group every minute Unit: count/minute
NOTE:
|
0–30,000,000 |
Consumer group of a Kafka instance |
1 minute |
topic_messages_consumed_per_min |
Queue Consumption Rate |
Number of messages consumed from the specified queue in the consumer group every minute Unit: count/minute
NOTE:
|
0–30,000,000 |
Consumer group of a Kafka instance |
1 minute |
consumer_messages_consumed_per_min |
Consumer Group Consumption Rate |
Number of messages consumed from the consumer group every minute Unit: count/minute
NOTE:
|
0–30,000,000 |
Consumer group of a Kafka instance |
1 minute |
Smart Connect Metrics
Metric ID |
Metric Name |
Description |
Value Range |
Monitored Object |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|
kafka_wait_synchronize_data |
Kafka Data to Sync |
Data to synchronize in the Kafka migration task Unit: count |
≥ 0 |
Smart Connect task of a Kafka instance |
1 minute |
kafka_synchronize_rate |
Kafka Data Synced per Minute |
Data synchronized per minute in the Kafka migration task Unit: count |
≥ 0 |
Smart Connect task of a Kafka instance |
1 minute |
task_status |
Task Status |
Status of the current task |
|
Smart Connect task of a Kafka instance |
1 minute |
message_delay |
Message Delay |
Time elapsed between when a message is sent from the source and received by the target Unit: ms |
≥ 0 |
Smart Connect task of a Kafka instance |
1 minute |
- A Smart Connect task that bidirectionally copies Kafka data is split into two tasks for monitoring: Smart Connect task name_source_0 and Smart Connect task name_source_1.
- If all messages in a topic have aged before the next synchronization, there is no Kafka data to be synchronized. However, since the Kafka data synchronization metric uses the offset value that contains aged data, Kafka Data Synced per Minute will display the number of aged messages.
Dimension
Key |
Value |
---|---|
kafka_instance_id |
Kafka instance |
kafka_broker |
Kafka instance broker |
kafka_topics |
Kafka instance topic |
kafka_partitions |
Partition in a Kafka instance |
kafka_groups-partitions |
Partition consumer group in a Kafka instance |
kafka_groups_topics |
Topic consumer group in a Kafka instance |
kafka_groups |
Consumer group of a Kafka instance |
connector_task |
Smart Connect task of a Kafka instance |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot