HDFS

hadoop-plugins-*.jar

HDFS can directly use the open-source Hadoop JAR file of the same version to run the sample code. However, in versions later than MRS 3.x, the default active/standby switchover class is dfs.client.failover.proxy.provider.hacluster=org.apache.hadoop.hdfs.server.namenode.ha.AdaptiveFailoverProxyProvider.

The default HDFS LZC compression format class is io.compression.codec.lzc.class=com.huawei.hadoop.datasight.io.compress.lzc.ZCodec.

• Method 1: Add the following configuration to the pom.xml file in the sample code:

  <properties>
      <hadoop.ext.version>8.0.2-302002</hadoop.ext.version>
  </properties>
  ...
  <dependency>
      <groupId>com.huawei.mrs</groupId>
      <artifactId>hadoop-plugins</artifactId>
      <version>${hadoop.ext.version}</version>
  </dependency>

• Method 2:
  1. Change the value of the dfs.client.failover.proxy.provider.hacluster parameter in the hdfs-site.xml configuration file to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider, which is used in the open source community.
  2. Do not use the LZC compression format.