Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Kafka Metrics

Updated on 2023-05-06 GMT+08:00

Introduction

This section describes metrics reported by DMS for Kafka to Cloud Eye as well as their namespaces and dimensions. You can use the Cloud Eye console or APIs to query the Kafka metrics and alarms.

For example, you can call the APIto query the monitoring data of the Disk Capacity Usage metric.

Namespace

SYS.DMS

Instance Metrics

Table 1 Instance metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

current_partitions

Partitions

Number of used partitions in the instance

Unit: count

0–1800

Kafka instance

1 minute

current_topics

Topics

Number of created topics in the instance

Unit: count

0–1800

Kafka instance

1 minute

group_msgs

Accumulated Messages

Total number of accumulated messages in all consumer groups of the instance

Unit: count

0–1,000,000,000

Kafka instance

1 minute

Broker Metrics

Table 2 Broker metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

broker_data_size

Message Size

Total size of messages in the broker

Unit: byte, KB, MB, GB, TB or PB

0–5,000,000,000,000

Kafka instance broker

1 minute

broker_messages_in_rate

Message Creation Rate

Number of messages created per second

Unit: count/s

0–500,000

Kafka instance broker

1 minute

broker_bytes_out_rate

Message Retrieval

Number of bytes retrieved per second

Unit: byte/s, KB/s, MB/s, or GB/s

0–500,000,000

Kafka instance broker

1 minute

broker_bytes_in_rate

Message Creation

Number of bytes created per second

Unit: byte/s, KB/s, MB/s, or GB/s

0–500,000,000

Kafka instance broker

1 minute

broker_public_bytes_in_rate

Public Inbound Traffic

Inbound traffic over public networks per second

Unit: byte/s, KB/s, MB/s, or GB/s

NOTE:

You can view this metric on the EIP console if public access has been enabled and EIPs have been assigned to the instance.

0–500,000,000

Kafka instance broker

1 minute

broker_public_bytes_out_rate

Public Outbound Traffic

Outbound traffic over public networks per second

Unit: byte/s, KB/s, MB/s, or GB/s

NOTE:

You can view this metric on the EIP console if public access has been enabled and EIPs have been assigned to the instance.

0–500,000,000

Kafka instance broker

1 minute

broker_fetch_mean

Average Message Retrieval Processing Duration

Average time that the broker spends processing message retrieval requests

Unit: ms

0–10,000

Kafka instance broker

1 minute

broker_produce_mean

Average Message Creation Processing Duration

Average time that the broker spends processing message creation requests

Unit: ms

0–10,000

Kafka instance broker

1 minute

broker_cpu_core_load

Average Load per CPU Core

Average load of each CPU core of the Kafka VM

Unit: %

0–20

Kafka instance broker

1 minute

broker_disk_usage

Disk Capacity Usage

Disk usage of the Kafka VM

Unit: %

0–100

Kafka instance broker

1 minute

broker_memory_usage

Memory Usage

Memory usage of the Kafka VM

Unit: %

0–100

Kafka instance broker

1 minute

broker_heap_usage

JVM Heap Memory Usage of Kafka

Heap memory usage of the Kafka JVM

Unit: %

0–100

Kafka instance broker

1 minute

broker_alive

Broker Alive

Whether the Kafka broker is alive

  • 1: alive
  • 0: not alive

Kafka instance broker

1 minute

broker_connections

Connections

Total number of TCP connections on the Kafka broker

Unit: count

0–65,535

Kafka instance broker

1 minute

broker_cpu_usage

CPU Usage

CPU usage of the Kafka VM

Unit: %

0–100

Kafka instance broker

1 minute

broker_disk_read_await

Average Disk Read Time

Average time for each disk I/O read in the monitoring period

Unit: ms

> 0

Kafka instance broker

1 minute

broker_disk_write_await

Average Disk Write Time

Average time for each disk I/O write in the monitoring period

Unit: ms

> 0

Kafka instance broker

1 minute

broker_total_bytes_in_rate

Inbound Traffic

Inbound traffic per second

Unit: byte/s

0–1,000,000,000

Kafka instance broker

1 minute

broker_total_bytes_out_rate

Outbound Traffic

Outbound traffic per second

Unit: byte/s

0–1,000,000,000

Kafka instance broker

1 minute

broker_disk_read_rate

Disk Read Speed

Read traffic on the disk

Unit: byte/s, KB/s, MB/s, or GB/s

≥ 0

Kafka instance broker

1 minute

broker_disk_write_rate

Disk Write Speed

Write traffic on the disk

Unit: byte/s, KB/s, MB/s, or GB/s

≥ 0

Kafka instance broker

1 minute

Topic Metrics

Table 3 Topic metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

topic_bytes_in_rate

Message Creation

Number of bytes created per second

Unit: byte/s, KB/s, MB/s, or GB/s

NOTE:

This metric is available only when Scope is set to Basic monitoring on the Queues tab page.

0–500,000,000

Topic in a Kafka instance

1 minute

topic_bytes_out_rate

Message Retrieval

Number of bytes retrieved per second

Unit: byte/s, KB/s, MB/s, or GB/s

NOTE:

This metric is available only when Scope is set to Basic monitoring on the Queues tab page.

0–500,000,000

Topic in a Kafka instance

1 minute

topic_data_size

Message Size

Total size of messages in the queue

Unit: byte, KB, MB, GB, TB or PB

NOTE:

This metric is available only when Scope is set to Basic monitoring on the Queues tab page.

0–5,000,000,000,000

Topic in a Kafka instance

1 minute

topic_messages

Total Messages

Total number of messages in the queue

Unit: count

NOTE:

This metric is available only when Scope is set to Basic monitoring on the Queues tab page.

≥ 0

Topic in a Kafka instance

1 minute

topic_messages_in_rate

Message Creation Rate

Number of messages created per second

Unit: count/s

NOTE:

This metric is available only when Scope is set to Basic monitoring on the Queues tab page.

0–500,000

Topic in a Kafka instance

1 minute

partition_messages

Partition Messages

Total number of messages in the partition

Unit: count

NOTE:

This metric is available only when Scope is set to Partition monitoring on the Queues tab page.

≥ 0

Topic in a Kafka instance

1 minute

produced_messages

Created Messages

Number of messages that have been created

Unit: count

NOTE:

This metric is available only when Scope is set to Partition monitoring on the Queues tab page.

≥ 0

Topic in a Kafka instance

1 minute

Consumer Group Metrics

Table 4 Consumer group metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

messages_consumed

Retrieved Messages

Number of messages that have been retrieved in the consumer group

Unit: count

NOTE:

This metric is available only when Queue is set to a specified topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

≥ 0

Consumer group of a Kafka instance

1 minute

messages_remained

Available Messages

Number of messages that can be retrieved in the consumer group

Unit: count

NOTE:

This metric is available only when Queue is set to a specified topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

≥ 0

Consumer group of a Kafka instance

1 minute

topic_messages_remained

Topic Available Messages

Number of remaining messages that can be retrieved from the specified topic in the consumer group

Unit: Count

NOTE:

This metric is available only when Queue is set to a specified topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

topic_messages_consumed

Topic Retrieved Messages

Number of messages that have been retrieved from the specified topic in the consumer group

Unit: Count

NOTE:

This metric is available only when Queue is set to a specified topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

consumer_messages_remained

Consumer Available Messages

Number of remaining messages that can be retrieved in the consumer group

Unit: Count

NOTE:

This metric is available only when Queue is set to All queues on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

consumer_messages_consumed

Consumer Retrieved Messages

Number of messages that have been retrieved in the consumer group

Unit: Count

NOTE:

This metric is available only when Queue is set to All queues on the By Consumer Group tab page.

0 to 263–1

Consumer group of a Kafka instance

1 minute

Dimension

Key

Value

kafka_instance_id

Kafka instance

kafka_broker

Kafka instance broker

kafka_topics

Kafka instance topic

kafka_partitions

Partition in a Kafka instance

kafka_groups-partitions

Partition consumer group in a Kafka instance

kafka_groups_topics

Topic consumer group in a Kafka instance

kafka_groups

Consumer group of a Kafka instance

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback