ALM-14026 Blocks on DataNode Exceed the Threshold

Alarm Description

The system checks the number of blocks on each DataNode every 30 seconds. This alarm is generated when the number of blocks on the DataNode exceeds the threshold.

If Trigger Count is 1 and the number of blocks on the DataNode is less than or equal to the threshold, this alarm is cleared. If Trigger Count is greater than 1 and the number of blocks on the DataNode is less than or equal to 90% of the threshold, this alarm is cleared.

Alarm Attributes

Alarm Parameters

Impact on the System

If this alarm is reported, there are too many blocks on the DataNode. In this case, data writing into the HDFS may fail due to insufficient disk space.

Possible Causes

• The alarm threshold is improperly configured.

• Data skew occurs among DataNodes.
• The disk space configured for the HDFS cluster is insufficient.

Handling Procedure

Change the threshold.

1. On FusionInsight Manager, choose Cluster, click the name of the desired cluster, and choose HDFS. Then choose Configurations > All Configurations. On the displayed page, find the GC_OPTS parameter under HDFS->DataNode.
2. Set the threshold of the DataNode blocks. Specifically, change the value of Xmx of the GC_OPTS parameter. Xmx specifies the memory, and each GB memory supports a maximum of 500,000 DataNode blocks. Set the memory as required. Confirm that GC_PROFILE is set to custom and save the configuration.
3. Choose Cluster, click the name of the desired cluster, and choose HDFS > Instance. Select the DataNode instance whose status is Expired, click More, and select Restart Instance to make the GC_OPTS configuration take effect.
   

   Services may be affected or interrupted during the restart. You are advised to perform the restart during off-peak hours.

4. Check whether the alarm is cleared 5 minutes later.
   • If yes, no further action is required.
   • If no, go to 5.

Check whether associated alarms are reported.

5. On FusionInsight Manager, choose O&M > Alarm > Alarms, and check whether the ALM-14002 DataNode Disk Usage Exceeds the Threshold alarm exists.
   • If yes, go to 6.
   • If no, go to 8.

6. Handle the alarm by following the instructions in ALM-14002 DataNode Disk Usage Exceeds the Threshold and check whether the alarm is cleared.
   • If yes, go to 7.
   • If no, go to 8.

7. Check whether the alarm is cleared 5 minutes later.
   • If yes, no further action is required.
   • If no, go to 8.

Expand the DataNode capacity.

8. Expand the DataNode capacity.
9. On FusionInsight Manager, wait for 5 minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 10.

Collect fault information.

10. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
11. Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
12. Click in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
13. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

Configuration rules of the DataNode JVM parameter.

Default value of the DataNode JVM parameter GC_OPTS:

-Xms2G -Xmx4G -XX:NewSize=128M -XX:MaxNewSize=256M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=128M -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=65 -XX:+PrintGCDetails -Dsun.rmi.dgc.client.gcInterval=0x7FFFFFFFFFFFFFE -Dsun.rmi.dgc.server.gcInterval=0x7FFFFFFFFFFFFFE -XX:-OmitStackTraceInFastThrow -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -Djdk.tls.ephemeralDHKeySize=2048

Average number of saved blocks = Number of HDFS blocks x 3/Number of DataNodes

To obtain the number of HDFS blocks, log in to FusionInsight Manager, choose Cluster > Services > HDFS, click NameNode(xxx,Active) to the right of NameNode Web UI to go to the native HDFS web UI, and then view the information in the Summary area.

If the average number of blocks on a single DataNode has changed, modify -Xms2G -Xmx4G -XX:NewSize=128M -XX:MaxNewSize=256M in the default value. The following table lists the reference values.

Xmx specifies memory which corresponds to the threshold of the number of DataNode blocks, and each GB memory supports a maximum of 500,000 DataNode blocks. Set the memory as required.