OS Disk Space Is Insufficient Due to Oversized HDFS Log Files

Issue

The space of the /var/log partition on the system disk is insufficient.

Symptom

The /var/log/Bigdata/hdfs/*/hdfs-omm-*.out log file is too large, causing insufficient space of the /var/log partition on the system disk.

Cause Analysis

During the long-term running of HDFS, the OS periodically deletes the /tmp/.java_pid* files created by the JVM. The HDFS memory monitoring uses the jinfo command, which depends on the /tmp/.java_pid* file. If the file does not exist, the jinfo command runs kill -3 to print the jstack information to the .out log file. As a result, the .out log file becomes oversize as time goes by.

Procedure

On each node hosting the HDFS instance, deploy a scheduled task to periodically clear the .out log file. For example, log in to the HDFS instance node and run the crontab -e command to add a scheduled task to clear the .out log file at 00:00:00 every day.

crontab -e

00 00 * * * for file in `ls /var/log/Bigdata/hdfs/*/hdfs-omm-*.out`; do echo "" > $file; done

If large .out files are generated frequently, you can clear the files multiple times every day or adjust the automatic clearing policy of the OS.