HDFS NameNode Instances Become Standby After the RPC Port Is Changed

Issue

After the NameNode RPC port is changed on the page and HDFS is restarted, all NameNodes are in the standby state, causing a cluster exception.

Symptom

All NameNodes are in the standby state, causing a cluster exception.

Cause Analysis

After the cluster is installed and started, if the NameNode RPC port is changed, the Zkfc service must be formatted to update node information on ZooKeeper.

Procedure

1. Log in to Manager and stop the HDFS service.
   

   Do not stop related services when stopping HDFS.

2. After the services are stopped, log in to the Master node whose RPC port is changed.
   

   If the RPC port is changed on both Master nodes, you can log in to either of the Master nodes.

3. Run the su - omm command to switch to user omm.
   

   For a security cluster, run the kinit hdfs command for authentication.

4. Run the following command to load the environment variable script to the environment:

   cd ${BIGDATA_HOME}/MRS_X.X.X/1_8_Zkfc/etc

   source ${BIGDATA_HOME}/MRS_X.X.X/install/FusionInsight-Hadoop-3.1.1/hadoop/sbin/exportENV_VARS.sh

   

   In the preceding command, MRS_X.X.X and 1_8 vary depending on the actual version.

5. After the loading is complete, run the following command to format the Zkfc:

   cd ${HADOOP_HOME}/bin

   ./hdfs zkfc -formatZK

6. After the formatting is successful, restart HDFS on Manager.
   

   If the RPC port of the NameNode is changed, the configuration file must be updated for all clients that have been installed.