Kafka Metrics
Introduction
This section describes metrics reported by DMS for Kafka to Cloud Eye as well as their namespaces and dimensions. You can use the Cloud Eye console or APIs to query the Kafka metrics and alarms, or view Kafka instance metrics on the Monitoring Details page of the DMS for Kafka console.
Namespace
SYS.DMS
Instance Metrics
Metric ID |
Metric Name |
Description |
Value Range |
Unit |
Conversion Rule |
Monitored Object (Dimension) |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|---|---|
current_partitions |
Partitions |
Number of used partitions in the instance |
0–100,000 |
Count |
N/A |
Kafka instance |
1 minute |
current_topics |
Topics |
Number of created topics in the instance |
0–100,000 |
Count |
N/A |
Kafka instance |
1 minute |
group_msgs |
Accumulated Messages |
Total number of accumulated messages in all consumer groups of the instance |
0–1,000,000,000 |
Count |
N/A |
Kafka instance |
1 minute |
instance_bytes_in_rate |
Message Production |
Number of bytes produced in the instance per second Some instances do not support this metric. Check whether your instance supports it on the console. |
0–1,000,000 |
Byte/s |
1024(IEC) |
Kafka instance |
1 minute |
instance_bytes_out_rate |
Message Consumption |
Number of bytes consumed from the instance per second Some instances do not support this metric. Check whether your instance supports it on the console. |
0–1,000,000 |
Byte/s |
1024(IEC) |
Kafka instance |
1 minute |
current_partitions_usage |
Current Partitions Usage |
Current Partitions Usage Some instances do not support this metric. Check whether your instance supports it on the console. |
0–100 |
% |
N/A |
Kafka instance |
1 minute |
Broker Metrics
Enabling Smart Connect for a Kafka instance creates two or more brokers. On the By Broker tab page, select "connector" for Node Type for Smart Connect broker metrics, or select "broker" for Node Type for Kafka instance broker metrics.
Metrics of Smart Connect brokers: disk capacity usage, memory usage, JVM heap memory usage, node alive status, and connections.
Metric ID |
Metric Name |
Description |
Value Range |
Unit |
Conversion Rule |
Monitored Object (Dimension) |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|---|---|
broker_data_size |
Message Size |
Total size of messages in the broker |
0–5,000,000,000,000 |
Byte |
1024(IEC) |
Kafka instance broker |
1 minute |
broker_messages_in_rate |
Message Creation Rate |
Number of messages created per second |
0–500,000 |
Count/s |
N/A |
Kafka instance broker |
1 minute |
broker_bytes_out_rate |
Message Retrieval |
Number of bytes retrieved per second |
0–500,000,000 |
Byte/s |
1024(IEC) |
Kafka instance broker |
1 minute |
broker_bytes_in_rate |
Message Creation |
Number of bytes created per second |
0–500,000,000 |
Byte/s |
1024(IEC) |
Kafka instance broker |
1 minute |
broker_fetch_mean |
Average Message Retrieval Processing Duration |
Average time that the broker spends processing message retrieval requests |
0–10,000 |
ms |
N/A |
Kafka instance broker |
1 minute |
broker_produce_mean |
Average Message Creation Processing Duration |
Average time that the broker spends processing message creation requests |
0–10,000 |
ms |
N/A |
Kafka instance broker |
1 minute |
broker_cpu_core_load |
Average Load per CPU Core |
Average load of each CPU core of the Kafka VM |
0–20 |
N/A |
N/A |
Kafka instance broker |
1 minute |
broker_disk_usage |
Disk Capacity Usage |
Disk usage of the Kafka VM |
0–100 |
% |
N/A |
Kafka instance broker |
1 minute |
broker_memory_usage |
Memory usage of the broker VM |
Memory usage of the Kafka VM |
0–100 |
% |
N/A |
Kafka instance broker |
1 minute |
broker_heap_usage |
JVM Heap Memory Usage of JVM |
Heap memory usage of the Kafka JVM |
0–100 |
% |
N/A |
Kafka instance broker |
1 minute |
broker_alive |
Broker Alive |
Whether the Kafka broker is alive This metric is supported by instances purchased in April 2020 or later. |
|
N/A |
N/A |
Kafka instance broker |
1 minute |
broker_connections |
Connections |
Total number of TCP connections on the Kafka broker This metric is supported by instances purchased in April 2020 or later. |
0–65,535 |
Count |
N/A |
Kafka instance broker |
1 minute |
broker_cpu_usage |
CPU Usage |
CPU usage of the Kafka VM This metric is supported by instances purchased in April 2020 or later. |
0–100 |
% |
N/A |
Kafka instance broker |
1 minute |
broker_disk_read_await |
Average Disk Read Time |
Average time for each disk I/O read in the monitoring period This metric is supported for instances purchased in June 2020 or later. |
> 0 |
ms |
N/A |
Kafka instance broker |
1 minute |
broker_disk_write_await |
Average Disk Write Time |
Average time for each disk I/O write in the monitoring period This metric is supported for instances purchased in June 2020 or later. |
> 0 |
ms |
N/A |
Kafka instance broker |
1 minute |
broker_total_bytes_in_rate |
Inbound Traffic |
Inbound traffic per second This metric is supported for instances purchased in June 2020 or later. |
0–1,000,000,000 |
Byte/s |
1024(IEC) |
Kafka instance broker |
1 minute |
broker_total_bytes_out_rate |
Outbound Traffic |
Outbound traffic per second This metric is supported for instances purchased in June 2020 or later. |
0–1,000,000,000 |
Byte/s |
1024(IEC) |
Kafka instance broker |
1 minute |
broker_disk_read_rate |
Disk Read Speed |
Read traffic on the disk This metric is supported for instances purchased on or after May 16, 2022. |
≥ 0 |
Byte/s |
1024(IEC) |
Kafka instance broker |
1 minute |
broker_disk_write_rate |
Disk Write Speed |
Write traffic on the disk This metric is supported for instances purchased on or after May 16, 2022. |
≥ 0 |
Byte/s |
1024(IEC) |
Kafka instance broker |
1 minute |
network_bandwidth_usage |
Network Bandwidth Usage |
Network bandwidth usage This metric is supported only:
|
0–100 |
% |
N/A |
Kafka instance broker |
1 minute |
Topic Metrics
Metric ID |
Metric Name |
Description |
Value Range |
Unit |
Conversion Rule |
Monitored Object (Dimension) |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|---|---|
topic_bytes_in_rate |
Message Creation |
Number of bytes created per second This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
0–500,000,000 |
Byte/s |
1024(IEC) |
Topic in a Kafka instance |
1 minute |
topic_bytes_out_rate |
Message Retrieval |
Number of bytes retrieved per second This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
0–500,000,000 |
Byte/s |
1024(IEC) |
Topic in a Kafka instance |
1 minute |
topic_data_size |
Message Size |
Total size of messages in the queue This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
0–5,000,000,000,000 |
Byte |
1024(IEC) |
Topic in a Kafka instance |
1 minute |
topic_messages |
Total Messages |
Total number of messages in the queue This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
≥ 0 |
Count |
N/A |
Topic in a Kafka instance |
1 minute |
topic_messages_in_rate |
Message Creation Rate |
Number of messages created per second This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. |
0–500,000 |
Count/s |
N/A |
Topic in a Kafka instance |
1 minute |
partition_messages |
Partition Messages |
Total number of messages in the partition This metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page. |
≥ 0 |
Count |
N/A |
Topic in a Kafka instance |
1 minute |
produced_messages |
Created Messages |
Number of messages that have been created This metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page. |
≥ 0 |
Count |
N/A |
Topic in a Kafka instance |
1 minute |
Consumer Group Metrics
Metric ID |
Metric Name |
Description |
Value Range |
Unit |
Conversion Rule |
Monitored Object (Dimension) |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|---|---|
messages_consumed |
Retrieved Messages |
Number of messages that have been retrieved in the consumer group This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page. |
≥ 0 |
Count |
N/A |
Consumer group of a Kafka instance |
1 minute |
messages_remained |
Number of accumulations Available Messages |
Number of messages that can be retrieved in the consumer group This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page. |
≥ 0 |
Count |
N/A |
Consumer group of a Kafka instance |
1 minute |
topic_messages_remained |
Topic Available Messages |
Number of remaining messages that can be retrieved from the specified topic in the consumer group This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page. |
0 to 263–1 |
Count |
N/A |
Consumer group of a Kafka instance |
1 minute |
topic_messages_consumed |
Topic Retrieved Messages |
Number of messages that have been retrieved from the specified topic in the consumer group This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page. |
0 to 263–1 |
Count |
N/A |
Consumer group of a Kafka instance |
1 minute |
consumer_messages_remained |
Consumer Available Messages |
Number of remaining messages that can be retrieved in the consumer group This metric is available only when Topic is set to All topics on the By Consumer Group tab page. |
0 to 263–1 |
Count |
N/A |
Consumer group of a Kafka instance |
1 minute |
consumer_messages_consumed |
Consumer Retrieved Messages |
Number of messages that have been retrieved in the consumer group This metric is available only when Topic is set to All topics on the By Consumer Group tab page. |
0 to 263–1 |
Count |
N/A |
Consumer group of a Kafka instance |
1 minute |
messages_consumed_per_min |
Partition Consumption Rate |
Number of messages consumed from the specified queue partition in the consumer group every minute This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page. Some instances do not support this metric. Check whether your instance supports it on the console. |
0–30,000,000 |
Count/min |
N/A |
Consumer group of a Kafka instance |
1 minute |
topic_messages_consumed_per_min |
Queue Consumption Rate |
Number of messages consumed from the specified queue in the consumer group every minute This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page. Some instances do not support this metric. Check whether your instance supports it on the console. |
0–30,000,000 |
Count/min |
N/A |
Consumer group of a Kafka instance |
1 minute |
consumer_messages_consumed_per_min |
Consumer Group Consumption Rate |
Number of messages consumed from the consumer group every minute This metric is available only when Topic is set to All topics on the By Consumer Group tab page. Some instances do not support this metric. Check whether your instance supports it on the console. |
0–30,000,000 |
Count/min |
N/A |
Consumer group of a Kafka instance |
1 minute |
Smart Connect Metrics
Smart Connect metrics are only available for cluster instances.
Metric ID |
Metric Name |
Description |
Value Range |
Unit |
Conversion Rule |
Monitored Object (Dimension) |
Monitoring Period (Raw Data) |
---|---|---|---|---|---|---|---|
kafka_wait_synchronize_data |
Kafka Data to Sync |
Data to synchronize in the Kafka migration task |
≥ 0 |
Count |
N/A |
Smart Connect task of a Kafka instance |
1 minute |
kafka_synchronize_rate |
Kafka Data Synced per Minute |
Data synchronized per minute in the Kafka migration task |
≥ 0 |
Count |
N/A |
Smart Connect task of a Kafka instance |
1 minute |
task_status |
Task Status |
Status of the current task |
|
N/A |
N/A |
Smart Connect task of a Kafka instance |
1 minute |
message_delay |
Message Delay |
Time elapsed between when a message is sent from the source and received by the target |
≥ 0 |
ms |
N/A |
Smart Connect task of a Kafka instance |
1 minute |
Precautions:
- A Smart Connect task that bidirectionally copies Kafka data is split into two tasks for monitoring: Smart Connect task name_source_0 and Smart Connect task name_source_1.
- If all messages in a topic have aged before the next synchronization, there is no Kafka data to be synchronized. However, since the Kafka data synchronization metric uses the offset value that contains aged data, Kafka Data Synced per Minute will display the number of aged messages.
Dimension
Key |
Value |
---|---|
kafka_instance_id |
Kafka instance |
kafka_broker |
Kafka instance broker |
kafka_topics |
Kafka instance topic |
kafka_partitions |
Partition in a Kafka instance |
kafka_groups-partitions |
Partition consumer group in a Kafka instance |
kafka_groups_topics |
Topic consumer group in a Kafka instance |
kafka_groups |
Consumer group of a Kafka instance |
connector_task |
Smart Connect task of a Kafka instance |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot