Help Center/ MapReduce Service/ Developer Guide (Normal_3.x)/ Using Open-source JAR File Conflict Lists/ HDFS
Updated on 2022-08-16 GMT+08:00
View PDF
Share

HDFS

HDFS JAR File Conflict List

JAR File

Description

Solution

hadoop-plugins-*.jar

HDFS can directly use the open-source Hadoop JAR file of the same version to run the sample code. However, in versions later than MRS 3.x, the default active/standby switchover class is dfs.client.failover.proxy.provider.hacluster=org.apache.hadoop.hdfs.server.namenode.ha.AdaptiveFailoverProxyProvider.

The default HDFS LZC compression format class is io.compression.codec.lzc.class=com.huawei.hadoop.datasight.io.compress.lzc.ZCodec.
  • Method 1: Add the following configuration to the pom.xml file in the sample code: 
    <properties>
    <hadoop.ext.version>8.0.2-302002</hadoop.ext.version>
</properties>
...
<dependency>
    <groupId>com.huawei.mrs</groupId>
    <artifactId>hadoop-plugins</artifactId>
    <version>${hadoop.ext.version}</version>
</dependency>
  • Method 2:
    1. Change the value of the dfs.client.failover.proxy.provider.hacluster parameter in the hdfs-site.xml configuration file to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider, which is used in the open source community.
    2. Do not use the LZC compression format.