Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Kafka Metrics

Updated on 2025-02-14 GMT+08:00

Introduction

This section describes metrics reported by DMS for Kafka to Cloud Eye as well as their namespaces and dimensions. You can use the Cloud Eye console or APIs to query the Kafka metrics and alarms, or view Kafka instance metrics on the Monitoring page of the DMS for Kafka console.

For example, you can call the API to query the monitoring data of the Disk Capacity Usage metric.

Namespace

SYS.DMS

Instance Metrics

Table 1 Instance metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

current_partitions

Partitions

Number of used partitions in the instance

Unit: Count

0–100,000

Kafka instance

1 minute

current_topics

Topics

Number of created topics in the instance

Unit: Count

0–100,000

Kafka instance

1 minute

group_msgs

Accumulated Messages

Total number of accumulated messages in all consumer groups of the instance

Unit: Count

0–1,000,000,000

Kafka instance

1 minute

instance_bytes_in_rate

Message Production

Number of bytes produced in the instance per second

Unit: Byte/s, KiB/s, MiB/s, GiB/s, TiB/s, PiB/s

Some instances do not support this metric. Check whether your instance supports it on the console.

0–1,000,000

Kafka instance

1 minute

instance_bytes_out_rate

Message Consumption

Number of bytes consumed from the instance per second

Unit: Byte/s, KiB/s, MiB/s, GiB/s, TiB/s, PiB/s

Some instances do not support this metric. Check whether your instance supports it on the console.

0–1,000,000

Kafka instance

1 minute

current_partitions_usage

Partition Usage

Partition usage

Unit: %

Some instances do not support this metric. Check whether your instance supports it on the console.

0–100

Kafka instance

1 minute

Broker Metrics

Enabling Smart Connect for a Kafka instance creates two or more brokers. On the By Broker tab page, select "connector" for Node Type for Smart Connect broker metrics, or select "broker" for Node Type for Kafka instance broker metrics.

Metrics of Smart Connect brokers: disk capacity usage, memory usage, JVM heap memory usage, node alive status, and connections.

Table 2 Broker metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

broker_data_size

Message Size

Total size of messages in the broker

Unit: byte, KB, MB, GB, TB or PB

0–5,000,000,000,000

Kafka instance broker

1 minute

broker_messages_in_rate

Message Creation Rate

Number of messages created per second

Unit: count/s

0–500,000

Kafka instance broker

1 minute

broker_bytes_out_rate

Message Retrieval

Number of bytes retrieved per second

Unit: byte/s, KB/s, MB/s, or GB/s

0–500,000,000

Kafka instance broker

1 minute

broker_bytes_in_rate

Message Creation

Number of bytes created per second

Unit: byte/s, KB/s, MB/s, or GB/s

0–500,000,000

Kafka instance broker

1 minute

broker_fetch_mean

Average Message Retrieval Processing Duration

Average time that the broker spends processing message retrieval requests

Unit: ms

0–10,000

Kafka instance broker

1 minute

broker_produce_mean

Average Message Creation Processing Duration

Average time that the broker spends processing message creation requests

Unit: ms

0–10,000

Kafka instance broker

1 minute

broker_cpu_core_load

Average Load per CPU Core

Average load of each CPU core of the Kafka VM

Unit: %

0–20

Kafka instance broker

1 minute

broker_disk_usage

Disk Capacity Usage

Disk usage of the Kafka VM

Unit: %

0–100

Kafka instance broker

1 minute

broker_memory_usage

Memory Usage

Memory usage of the Kafka VM

Unit: %

0–100

Kafka instance broker

1 minute

broker_heap_usage

JVM Heap Memory Usage of Kafka

Heap memory usage of the Kafka JVM

Unit: %

0–100

Kafka instance broker

1 minute

broker_alive

Broker Alive

Whether the Kafka broker is alive

This metric is supported by instances purchased in April 2020 or later.

  • 1: alive
  • 0: not alive

Kafka instance broker

1 minute

broker_connections

Connections

Total number of TCP connections on the Kafka broker

Unit: count

This metric is supported by instances purchased in April 2020 or later.

0–65,535

Kafka instance broker

1 minute

broker_cpu_usage

CPU Usage

CPU usage of the Kafka VM

Unit: %

This metric is supported by instances purchased in April 2020 or later.

0–100

Kafka instance broker

1 minute

broker_disk_read_await

Average Disk Read Time

Average time for each disk I/O read in the monitoring period

Unit: ms

This metric is supported for instances purchased in June 2020 or later.

> 0

Kafka instance broker

1 minute

broker_disk_write_await

Average Disk Write Time

Average time for each disk I/O write in the monitoring period

Unit: ms

This metric is supported for instances purchased in June 2020 or later.

> 0

Kafka instance broker

1 minute

broker_total_bytes_in_rate

Inbound Traffic

Inbound traffic per second

Unit: byte/s

This metric is supported for instances purchased in June 2020 or later.

0–1,000,000,000

Kafka instance broker

1 minute

broker_total_bytes_out_rate

Outbound Traffic

Outbound traffic per second

Unit: byte/s

This metric is supported for instances purchased in June 2020 or later.

0–1,000,000,000

Kafka instance broker

1 minute

broker_disk_read_rate

Disk Read Speed

Read traffic on the disk

Unit: byte/s, KB/s, MB/s, or GB/s

This metric is supported for instances purchased on or after May 16, 2022.

≥ 0

Kafka instance broker

1 minute

broker_disk_write_rate

Disk Write Speed

Write traffic on the disk

Unit: byte/s, KB/s, MB/s, or GB/s

This metric is supported for instances purchased on or after May 16, 2022.

≥ 0

Kafka instance broker

1 minute

network_bandwidth_usage

Network Bandwidth Usage

Network bandwidth usage

Unit: %

This metric is supported only:

  • For instances purchased since July 9 2023.
  • For instances purchased before July 9 2023, this metric is supported for brokers if they are added since July 9 2023.

0–100

Kafka instance broker

1 minute

Topic Metrics

Table 3 Topic metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

topic_bytes_in_rate

Message Creation

Number of bytes created per second

Unit: byte/s, KB/s, MB/s, or GB/s

This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

0–500,000,000

Topic in a Kafka instance

1 minute

topic_bytes_out_rate

Message Retrieval

Number of bytes retrieved per second

Unit: byte/s, KB/s, MB/s, or GB/s

This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

0–500,000,000

Topic in a Kafka instance

1 minute

topic_data_size

Message Size

Total size of messages in the queue

Unit: byte, KB, MB, GB, TB or PB

This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

0–5,000,000,000,000

Topic in a Kafka instance

1 minute

topic_messages

Total Messages

Total number of messages in the queue

Unit: count

This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

≥ 0

Topic in a Kafka instance

1 minute

topic_messages_in_rate

Message Creation Rate

Number of messages created per second

Unit: count/s

This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

0–500,000

Topic in a Kafka instance

1 minute

partition_messages

Partition Messages

Total number of messages in the partition

Unit: count

This metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page.

≥ 0

Topic in a Kafka instance

1 minute

produced_messages

Created Messages

Number of messages that have been created

Unit: count

This metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page.

≥ 0

Topic in a Kafka instance

1 minute

Consumer Group Metrics

Table 4 Consumer group metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

messages_consumed

Retrieved Messages

Number of messages that have been retrieved in the consumer group

Unit: count

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

≥ 0

Consumer group of a Kafka instance

1 minute

messages_remained

Available Messages

Number of messages that can be retrieved in the consumer group

Unit: count

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

≥ 0

Consumer group of a Kafka instance

1 minute

topic_messages_remained

Topic Available Messages

Number of remaining messages that can be retrieved from the specified topic in the consumer group

Unit: Count

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

topic_messages_consumed

Topic Retrieved Messages

Number of messages that have been retrieved from the specified topic in the consumer group

Unit: Count

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

consumer_messages_remained

Accumulated Messages (Consumer Available Messages)

Number of remaining messages that can be retrieved in the consumer group

Unit: Count

This metric is available only when Topic is set to All topics on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

consumer_messages_consumed

Consumer Retrieved Messages

Number of messages that have been retrieved in the consumer group

Unit: Count

This metric is available only when Topic is set to All topics on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

messages_consumed_per_min

Partition Consumption Rate

Number of messages consumed from the specified queue partition in the consumer group every minute

Unit: count/minute

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

Some instances do not support this metric. Check whether your instance supports it on the console.

0–30,000,000

Consumer group of a Kafka instance

1 minute

topic_messages_consumed_per_min

Queue Consumption Rate

Number of messages consumed from the specified queue in the consumer group every minute

Unit: count/minute

This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

Some instances do not support this metric. Check whether your instance supports it on the console.

0–30,000,000

Consumer group of a Kafka instance

1 minute

consumer_messages_consumed_per_min

Consumer Group Consumption Rate

Number of messages consumed from the consumer group every minute

Unit: count/minute

This metric is available only when Topic is set to All topics on the By Consumer Group tab page.

Some instances do not support this metric. Check whether your instance supports it on the console.

0–30,000,000

Consumer group of a Kafka instance

1 minute

Smart Connect Metrics

Smart Connect metrics are only available for cluster instances.

Table 5 Smart Connect metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

kafka_wait_synchronize_data

Kafka Data to Sync

Data to synchronize in the Kafka migration task

Unit: count

≥ 0

Smart Connect task of a Kafka instance

1 minute

kafka_synchronize_rate

Kafka Data Synced per Minute

Data synchronized per minute in the Kafka migration task

Unit: count

≥ 0

Smart Connect task of a Kafka instance

1 minute

task_status

Task Status

Status of the current task

  • 0: abnormal
  • 1: normal

Smart Connect task of a Kafka instance

1 minute

message_delay

Message Delay

Time elapsed between when a message is sent from the source and received by the target

Unit: ms

≥ 0

Smart Connect task of a Kafka instance

1 minute

Precautions:

  • A Smart Connect task that bidirectionally copies Kafka data is split into two tasks for monitoring: Smart Connect task name_source_0 and Smart Connect task name_source_1.
  • If all messages in a topic have aged before the next synchronization, there is no Kafka data to be synchronized. However, since the Kafka data synchronization metric uses the offset value that contains aged data, Kafka Data Synced per Minute will display the number of aged messages.

Dimension

Key

Value

kafka_instance_id

Kafka instance

kafka_broker

Kafka instance broker

kafka_topics

Kafka instance topic

kafka_partitions

Partition in a Kafka instance

kafka_groups-partitions

Partition consumer group in a Kafka instance

kafka_groups_topics

Topic consumer group in a Kafka instance

kafka_groups

Consumer group of a Kafka instance

connector_task

Smart Connect task of a Kafka instance

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback