Commissioning an HDFS Application on Linux

Scenario

HDFS applications can run in a Linux environment where an HDFS client is installed. After application code development is complete, you can upload a JAR file to the Linux client to run applications.

HDFS applications can run only on Linux, but not on Windows.

Prerequisites

• You have installed an HDFS client.
• If the host where the client is installed is not a node in the cluster, the mapping between the host name and the IP address must be set in the hosts file on the node where the client locates. The host names and IP addresses must be mapped one by one.

Procedure

1. Run the mvn package command to generate a JAR file, for example, hdfs-examples-1.0.jar, and obtain it from the target directory in the project directory.
2. Upload the exported JAR file to any directory in the Linux client operating environment, for example, /opt/client. Create the conf directory in the directory, and copy the user.keytab and krb5.conf files to the conf directory. For details, see 6.
3. Run the following command to set the environment variables:

   source /opt/client/bigdata_env

4. Run the following commands to run the JAR file:

   hadoop jar hdfs-examples-1.0.jar com.huawei.bigdata.hdfs.examples.HdfsMain

   

   Before you run the commands, ensure that the Kerberos information is consistent between Yarn/config/hdfs-site.xml and HDFS/hadoop/etc/hadoop/hdfs-site.xml. Modify mapred in hdfs-site.xml to hdfs. Figure 1 lists points that need to be modified.

   Figure 1 hdfs-site.xml