Updated on 2022-11-18 GMT+08:00

HDFS

HDFS JAR File Conflict List

JAR File

Description

Solution

hadoop-plugins-*.jar

HDFS can directly use the open-source Hadoop JAR file of the same version to run the sample code. However, in versions later than MRS 3.x, the default active/standby switchover class is dfs.client.failover.proxy.provider.hacluster=org.apache.hadoop.hdfs.server.namenode.ha.AdaptiveFailoverProxyProvider.

The default HDFS LZC compression format class is io.compression.codec.lzc.class=com.huawei.hadoop.datasight.io.compress.lzc.ZCodec.

  • Method 1: Add the following configuration to the pom.xml file in the sample code:
    <properties>
        <hadoop.ext.version>8.0.2-302002</hadoop.ext.version>
    </properties>
    ...
    <dependency>
        <groupId>com.huawei.mrs</groupId>
        <artifactId>hadoop-plugins</artifactId>
        <version>${hadoop.ext.version}</version>
    </dependency>
  • Method 2:
    1. Change the value of the dfs.client.failover.proxy.provider.hacluster parameter in the hdfs-site.xml configuration file to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider, which is used in the open source community.
    2. Do not use the LZC compression format.