ALM-38005 GC Duration of the Broker Process Exceeds the Threshold

Alarm Description

The system checks the garbage collection (GC) duration of the Broker process every 60 seconds. This alarm is generated when the GC duration exceeds the threshold (12 seconds by default) for 3 consecutive times.

When the Trigger Count is 1, this alarm is cleared when the GC duration is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the GC duration is less than or equal to 90% of the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

A long GC duration of the Broker process may interrupt the services.

Possible Causes

The GC duration of the Kafka instance process on the node is too long, or the heap memory is not properly configured. As a result, GCs occur frequently.

Handling Procedure

Check the GC duration of the Broker process.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms > GC Duration of the Broker Process Exceeds the Threshold > Location. Check the name of the host for which the alarm is generated.
2. On the FusionInsight Manager, choose Cluster > Services > Kafka > Instances. Click the role of the instance for which this alarm is generated. Click the drop-down list in the upper right corner of the chart area, choose Customize > Process > Broker GC Duration per Minute, and click OK.
   Figure 1 Broker GC Duration per Minute
   

3. Check whether the GC duration of the Broker process collected every minute exceeds the threshold (12 seconds by default).
   • If yes, go to Step 4.
   • If no, go to Step 7.

Check the heap memory size configured for Kafka.

4. On the FusionInsight Manager, choose Cluster > Services > Kafka. On the displayed page, click Configurations and then choose All Configurations > Broker(Role) > Environment. Increase the value of -Xmx in the KAFKA_HEAP_OPTS parameter.
   

   • It is recommended that -Xmx and -Xms be set to the same value.
   • It is recommended that you set the value of KAFKA_HEAP_OPTS to twice the value of Heap Memory Usage of Kafka based on Kafka heap memory data. To check the Kafka heap memory data, choose Cluster > Services > Kafka on the FusionInsight Manager homepage, click Instances, and select the role corresponding to the host name of the instance for which the alarm is generated. Click the drop-down list in the upper right corner of the chart area and choose Customize. In the displayed window, choose Process, and select Heap Memory Utilization of Kafka to check its data.

5. Save the configuration and restart the Kafka service.
   

   If rolling restart is performed and the current Topic has multiple copies, services are not affected. Otherwise, the Kafka service will be unavailable during the restart, and upper-layer services that depend on the service will be affected.

6. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

Collect fault information.

7. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
8. Expand the Service drop-down list, and select Kafka for the target cluster.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively. Then, click Download.
10. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None