An Error Is Reported When a Node Outside the Cluster Accesses MRS HDFS

Symptom

When a node outside the MRS cluster accesses HDFS in the cluster, the following error message is reported: Class org.apache.hadoop.hdfs.server.namenode.ha.AdaptiveFailoverProxyProvider not found

Cause Analysis

The possible causes are as follows:

• An error is reported when an open-source HDFS client accesses HDFS of an MRS cluster.
• An error is reported when the JAR package is used to connect to HDFS of the MRS cluster (including connection to HDFS during task submission).

Procedure

Method 1:

1. Locate the HDFS configuration file hdfs-site.xml used by the command or JAR package.
2. Modify the dfs.client.failover.proxy.provider.hacluster configuration as follows:

   <property>
   <name>dfs.client.failover.proxy.provider.hacluster</name>
   <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
   </property>

   

   You can also delete the preceding configuration items.

3. Save the file and access MRS HDFS again.

Method 2:

1. Download the hadoop-plugins matching the MRS cluster version from the Maven repository.

   Download path: https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/com/huawei/mrs/hadoop-plugins/

2. Add the downloaded JAR package to the dependency of the command or JAR package.