Help Center/ MapReduce Service/ Developer Guide (Normal_3.x)/ HDFS Development Guide (Security Mode)/ Commissioning the Application/ Commissioning an Application in the Linux Environment/ Compiling and Running an Application With the Client Not Installed
Updated on 2022-11-18 GMT+08:00
View PDF

Compiling and Running an Application With the Client Not Installed

Scenario

The Hadoop distributed file system (HDFS) application can run in the Linux operating system (OS) with the HDFS client not installed. After the application code has been developed, you can upload the jar packages to the Linux OS and run the application.

Prerequisite

  • A JDK has been installed in the Linux environment. The version of the JDK must be consistent with that of the JDK used by IDEA to export the JAR package.
  • When the host where the Linux OS runs is not a node of the cluster, you are required to set the mapping between the host name and IP address in the hosts file of the node where the Linux OS runs. The host name must be correctly mapped to the IP address.

Procedure

  1. Go to the local root directory of the project, copy the required configuration file to the conf folder of the local project and run the following command in Windows cmd to compress the package:

    mvn -s "{maven_setting_path}" clean package

    • In the preceding command, {maven_setting_path} is the path of the settings.xml file of the local Maven.
    • After the package is successfully packed, obtain the JAR package from the target subdirectory in the root directory of the project.

  2. Upload the exported jar packages to any directory in the running environment of the Linux OS, for example, /opt/hadoop_client.
  3. Upload the lib and conf folders of the project to the same directory that stores the jar packages (the lib Create a lib folder in the Linux operating environment directory (for example, /opt/client) and upload the required JAR packages. The lib folder contains all the JAR packages that the project depends on. For details, see section Preparing an Operating Environment.
  4. (Optional) To run the SmallFS example code, modify the value of the fs.defaultFS configuration item in the core-site.xml file from hdfs://hacluster to sfs://hacluster.
  5. Run the following commands to execute the jar packages.

    java -cp HDFSTest-XXX.jar:conf/:lib/* com.huawei.bigdata.hdfs.examples.HdfsExample

    java -cp HDFSTest-XXX.jar:conf/:lib/* com.huawei.bigdata.hdfs.examples.ColocationExample

    When com.huawei.bigdata.hdfs.examples.ColocationExample is run, the HDFSparameter fs.defaultFS cannot be set to viewfs://ClusterX.