HDFS

Updated on 2022-11-18 GMT+08:00

View PDF

HDFS JAR File Conflict List

JAR File	Description	Solution
hadoop-plugins-*.jar	HDFS can directly use the open-source Hadoop JAR file of the same version to run the sample code. However, in versions later than MRS 3.x, the default active/standby switchover class is dfs.client.failover.proxy.provider.hacluster=org.apache.hadoop.hdfs.server.namenode.ha.AdaptiveFailoverProxyProvider. The default HDFS LZC compression format class is io.compression.codec.lzc.class=com.huawei.hadoop.datasight.io.compress.lzc.ZCodec.	Method 1: Add the following configuration to the pom.xml file in the sample code: <properties> <hadoop.ext.version>8.0.2-302002</hadoop.ext.version> </properties> ... <dependency> <groupId>com.huawei.mrs</groupId> <artifactId>hadoop-plugins</artifactId> <version>${hadoop.ext.version}</version> </dependency> Method 2: Change the value of the dfs.client.failover.proxy.provider.hacluster parameter in the hdfs-site.xml configuration file to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider, which is used in the open source community. Do not use the LZC compression format.

JAR File

Description

Solution

hadoop-plugins-*.jar

HDFS can directly use the open-source Hadoop JAR file of the same version to run the sample code. However, in versions later than MRS 3.x, the default active/standby switchover class is dfs.client.failover.proxy.provider.hacluster=org.apache.hadoop.hdfs.server.namenode.ha.AdaptiveFailoverProxyProvider.

The default HDFS LZC compression format class is io.compression.codec.lzc.class=com.huawei.hadoop.datasight.io.compress.lzc.ZCodec.

Method 1: Add the following configuration to the pom.xml file in the sample code:

<properties>
    <hadoop.ext.version>8.0.2-302002</hadoop.ext.version>
</properties>
...
<dependency>
    <groupId>com.huawei.mrs</groupId>
    <artifactId>hadoop-plugins</artifactId>
    <version>${hadoop.ext.version}</version>
</dependency>

Method 2:
1. Change the value of the dfs.client.failover.proxy.provider.hacluster parameter in the hdfs-site.xml configuration file to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider, which is used in the open source community.
2. Do not use the LZC compression format.