Failed to Start HRegionServer on the Node Newly Added to the Cluster

Symptom

• After the cluster is scaled out, HRegionserver on the new node fails to be started and remains abnormal.
• Log in to the node where the faulty RegionServer resides and run the jps command. The command output shows that the RegionServer process is not started. Then manually restart the faulty RegionServer instance on Manager, but the restart fails. Check the /var/log/Bigdata/hbase/rs/hbase-omm-regionserver-node-ana-coreqRvt.log file of the RegionServer node, and error message "ClassNotFound: org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec" is displayed.
• The following custom configuration of RegionServer is available on Manager.

  

Cause Analysis

The indexing function of Phoenix is configured, but the new node does not have the JAR package of Phoenix. As a result, the related class cannot be found and the startup fails.

Procedure

1. Log in to a normal RegionServer node and run the following command:

   grep -Rn 'org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec' /opt/Bigdata/MRS_Current/1_16_RegionServer/

   Two Phoenix packages (phoenix-4.14.1-server.jar and phoenix-core-4.14.1.jar) in the /opt/Bigdata/MRS_Current/1_16_RegionServer/install/hbase/lib directory contain this class.

2. Run the scp command to copy the preceding two packages to the same directory on the faulty RegionServer node and restart the faulty node.