Help Center/ MapReduce Service/ User Guide (Ankara Region)/ Alarm Reference/ ALM-38006 Percentage of Kafka Partitions That Are Not Completely Synchronized Exceeds the Threshold

Updated on 2024-11-29 GMT+08:00

View PDF

ALM-38006 Percentage of Kafka Partitions That Are Not Completely Synchronized Exceeds the Threshold

Alarm Description

The system checks the percentage of Kafka partitions that are not completely synchronized to the total number of partitions every 60 seconds. This alarm is generated when the percentage exceeds the threshold for 3 consecutive times.

When the Trigger Count is 1, this alarm is cleared when the percentage is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the percentage is less than or equal to 90% of the threshold.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
38006	Critical (default threshold: 60%) Major (default threshold: 50%)	Quality of service	Kafka	Yes

Alarm ID

Alarm Severity

Alarm Type

Service Type

Auto Cleared

38006

Critical (default threshold: 60%)

Major (default threshold: 50%)

Quality of service

Kafka

Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
Additional Information	Trigger Condition	Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

Too many Kafka partitions that are not completely synchronized affect service reliability. In addition, data may be lost when leaders are switched.

Possible Causes

Some nodes where the Broker instance resides are abnormal or stop running. As a result, replicas of some partitions in Kafka are out of the in-sync replicas (ISR) set.

Handling Procedure

Check Broker instances.

On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Kafka > Instance. The Kafka instances page is displayed.
Check whether faulty nodes exist among all Broker nodes.
- If yes, record the host name of the node and go to 3.
- If no, go to 5.
On the FusionInsight Manager portal, click O&M > Alarm > Alarms to check whether the fault described in 2 exists in the alarm information and handle the alarm based on corresponding methods.
On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Kafka > Instance. The Kafka instances page is displayed.
Check whether stopped nodes exist among all Broker instance.
- If yes, go to 6.
- If no, go to 7.
Select all stopped Broker instances and click Start Instance.
Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.

Collect fault information.

On the FusionInsight Manager portal, choose O&M > Log > Download.
Select Kafka in the required cluster from the Service drop-down list.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M engineers and send the collected logs.