On this page

Show all

Help Center/ MapReduce Service/ User Guide (Ankara Region)/ Alarm Reference/ ALM-14015 DataNode GC Time Exceeds the Threshold

ALM-14015 DataNode GC Time Exceeds the Threshold

Updated on 2024-11-29 GMT+08:00

View PDF

Alarm Description

The system checks the garbage collection (GC) duration of the DataNode process every 60 seconds. This alarm is generated when the GC duration exceeds the threshold.

This alarm is cleared when the GC duration is less than the threshold.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
14015	Critical (default threshold: 20000ms) Major (default threshold: 12000ms)	Quality of service	HDFS	Yes

Alarm ID

Alarm Severity

Alarm Type

Service Type

Auto Cleared

14015

Critical (default threshold: 20000ms)

Major (default threshold: 12000ms)

Quality of service

HDFS

Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.
Additional Information	Trigger Condition	Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

A long GC duration of the DataNode process may interrupt the services and users cannot read or write files.

Possible Causes

The heap memory of the DataNode instance is overused or the heap memory is inappropriately allocated. As a result, GCs occur frequently.

Handling Procedure

Check the GC duration.

On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. On the displayed interface, click the drop-down button of ALM-14015 DataNode GC Time Exceeds the Threshold. Then check the role name in Location and confirm the IP adress of the instance.
On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS > Instance > DataNode (IP address for which the alarm is generated). Click the drop-down menu in the upper right corner of Chart, choose Customize > Garbage Collection, and select DataNode Garbage Collection (GC) to check the GC duration statistics of the DataNode process collected every minute.
Check whether the GC duration of the DataNode process collected every minute exceeds the threshold.
- If yes, go to 4.
- If no, go to 7.
On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS > Configurations > All Configurations > DataNode > System to increase the value of GC_OPTS parameter as required.
NOTE:
The mapping between the average number of blocks of a DataNode instance and the DataNode memory is as follows:
- If the average number of blocks of a DataNode instance reaches 2,000,000, the reference values of the JVM parameters of the DataNode are as follows: -Xms6G -Xmx6G -XX:NewSize=512M -XX:MaxNewSize=512M
- If the average number of blocks of a DataNode instance reaches 5,000,000, the reference values of the JVM parameters of the DataNode are as follows: -Xms12G -Xmx12G -XX:NewSize=1G -XX:MaxNewSize=1G
Save the configuration and restart the DataNode instance.
Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.

Collect fault information.

On the FusionInsight Manager portal, choose O&M > Log > Download.
Select DataNode in the required cluster from the Service.
Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.

Parent topic: Alarm Reference

Previous topic: ALM-14014 NameNode GC Time Exceeds the Threshold

Next topic: ALM-14016 DataNode Direct Memory Usage Exceeds the Threshold

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot