HDFS Troubleshooting

This section describes common HDFS faults and troubleshooting methods.

• HDFS Component Exceptions:
  • HDFS Capacity Reaches 100%, Causing Unavailable Upper-Layer Services Such as HBase and Spark
  • Error Message "Permission denied" Is Displayed When HDFS and Yarn Are Started
  • An Error Is Reported When a Node Outside the Cluster Accesses MRS HDFS
• NameNode Instance Exceptions:
  • HDFS NameNode Instances Become Standby After the RPC Port Is Changed
  • HDFS NameNode Failed to Start Due to Insufficient Memory
  • Manually Performing Checkpoints When a NameNode Is Faulty for a Long Time
• DataNode Instance Exceptions:
  • A DataNode of HDFS Is Always in the Decommissioning State
  • CPU Usage of DataNodes Is Close to 100% Occasionally, Causing Node Loss
  • Failed to Decommission a DataNode Due to HDFS Block Loss
• File Read/Write Exceptions:
  • Failed to Write Files to HDFS, and Error Message "item limit of xxx is exceeded" Is Displayed
  • Error "Failed to place enough replicas" Is Reported When HDFS Reads or Writes Files
  • HDFS File Fails to Be Read, and Error Message "FileNotFoundException" Is Displayed
  • HDFS File Read Fails, and Error Message "No common protection layer" Is Displayed
  • Insufficient Number of Replicas Is Reported During High Concurrent HDFS Writes
  • HDFS Client File Fails to Be Closed After Data Writing
  • Failed to Write Files Because the HDFS Directory Quota Is Insufficient
  • File Fails to Be Uploaded to HDFS Due to File Errors
  • HDFS Files Fail to Be Uploaded When the Client Is Installed on a Node Outside the Cluster
  • Failed to Query or Delete HDFS Files
  • Maximum Number of File Handles Is Set to a Too Small Value, Causing File Reading and Writing Exceptions
• HDFS Client Usage Exceptions:
  • Adjusting the Log Level of the HDFS Shell Client
  • Error Message "error creating DomainSocket" Is Displayed When the HDFS Client Installed in a Normal Cluster Is Used
  • Error Message "Source and target differ in block-size" Is Displayed When the distcp Command Is Executed to Copy Files Across Clusters
  • An Error Is Reported When DistCP Is Used to Copy an Empty Folder
  • HDFS Client Failed to Delete Overlong Directories
• Data Block Exceptions:
  • A Large Number of Blocks Are Lost in HDFS due to the Time Change Using ntpdate
  • After dfs.blocksize Is Configured on the UI and Data Is Uploaded, the Block Size Does Not Change
• Network Connection Exceptions:
  • An Error Is Reported When the HDFS Client Is Connected Through a Public IP Address
  • Failed to Use Python to Remotely Connect to the Port of HDFS
• Data Balancing/Disk Balancing Exceptions:
  • Locating Common Balance Problems
  • Data Distribution Is Unbalanced Due to Residual Non-HDFS Data
  • Uneven Data Distribution Due to Non-HDFS Data Residuals
  • Unbalanced DataNode Disk Usages of a Node
  • HDFS Displays Insufficient Disk Space But 10% Disk Space Remains
• Alarms:
  • "ALM-12027 Host PID Usage Exceeds the Threshold" Is Generated for a NameNode
  • ALM-14012 JournalNode Is Out of Synchronization Is Generated in the Cluster