RegionServer Instance Is in the Restoring State

Symptom

HBase fails to start, and the RegionServer stays in the Restoring state.

Cause Analysis

Check the running log (/var/log/Bigdata/hbase/rs/hbase-omm-XXX.log) of the abnormal RegionServer instance. It is found that the following information is displayed: ClockOutOfSyncException..., Reported time is too far out of sync with master.

2017-09-18 11:16:23,636 | FATAL | regionserver21302 | Master rejected startup because clock is out of sync | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2059)
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server nl-bi-fi-datanode-24-65,21302,1505726180086 has been rejected; Reported time is too far out of sync with master.  Time difference of 152109ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:354)
...

...
2017-09-18 11:16:23,858 | ERROR | main | Region server exiting | org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:70)
java.lang.RuntimeException: HRegionServer Aborted

This log indicates that the time difference between the abnormal RegionServer instance and the HMaster instance is greater than the allowed time difference 30s (specified by the hbase.regionserver.maxclockskew parameter and the default value is 30000 ms). As a result, the RegionServer instance is abnormal.