Help Center/
MapReduce Service/
Developer Guide (LTS)/
HDFS Development Guide (Normal Mode)/
Commissioning an HDFS Application/
Commissioning an HDFS Application in the Linux Environment
Updated on 2025-04-14 GMT+08:00
Commissioning an HDFS Application in the Linux Environment
Scenario
The Hadoop distributed file system (HDFS) application can run in the Linux operating system (OS). After the application code has been developed, you can upload the jar packages to the prepared Linux and run the application.
After an HDFS application is run, you can learn the application running conditions by viewing the running result or HDFS logs.
Prerequisite
- When the client is installed:
- The HDFS client has been installed.
- When the host where the Linux OS runs is not a node of the cluster, you are required to set the mapping between the host name and IP address in the hosts file of the node where the client is installed. The host name must be correctly mapped to the IP address.
- When the client is not installed:
- A JDK has been installed in the Linux environment. The version of the JDK must be consistent with that of the JDK used by IDEA to export the JAR package.
- When the host where the Linux OS runs is not a node of the cluster, you are required to set the mapping between the host name and IP address in the hosts file of the node where the Linux OS runs. The host name must be correctly mapped to the IP address.
Compiling and Running the Program When the Client is Installed
- Go to the local root directory of the sample project, copy the required configuration file to the conf folder of the local project, and run the following command in Windows CLI to compress the package:
mvn -s "{maven_setting_path}" clean package
- In the preceding command, {maven_setting_path} is the path of the settings.xml file of the local Maven.
- After the package is successfully packed, obtain the JAR package, for example, HDFSTest-XXX.jar, from the target subdirectory in the root directory of the project. The name of the JAR package varies according to the actual package.
- Upload the exported jarpackages to any directory in the running environment of the client, for example, /opt/client.
- Configure the environment variables:
cd /opt/client
source bigdata_env
- Specify a user to run the example. There are two ways to specify the user: add the environment variable HADOOP_USER_NAME and modify the code. If the code cannot be modified, run the following statement to add the environment variable:
export HADOOP_USER_NAME=test
The test user here is an example. To run the example code related to the Colocation operation, the user must be a member of the supergroup group.
- Run the following commands to execute the jar packages.
hadoop jar HDFSTest-XXX.jar com.huawei.bigdata.hdfs.examples.HdfsExample
hadoop jar HDFSTest-XXX.jar com.huawei.bigdata.hdfs.examples.ColocationExample
When com.huawei.bigdata.hdfs.examples.ColocationExample is run, the HDFSparameter fs.defaultFS cannot be set to viewfs://ClusterX.
Compiling and Running an Application with the Client Not Installed
- Go to the local root directory of the project, copy the required configuration file to the conf folder of the local project and run the following command in Windows CLI to compress the package:
mvn -s "{maven_setting_path}" clean package
- In the preceding command, {maven_setting_path} is the path of the settings.xml file of the local Maven.
- After the package is successfully packed, obtain the JAR package from the target subdirectory in the root directory of the project.
- Upload the exported jar packages to any directory in the running environment of the Linux OS, for example, /opt/client.
- Create a lib folder in the Linux operating environment directory (for example, /opt/client) and upload the required JAR packages. The lib folder contains all the JAR packages that the project depends on. For details, see section Preparing an Operating Environment.
- Specify a user to run the example. There are two ways to specify the user: add the environment variable HADOOP_USER_NAME and modify the code. If the code cannot be modified, run the following statement to add the environment variable:
export HADOOP_USER_NAME=test
The test user here is an example. To run the example code related to the Colocation operation, the user must be a member of the supergroup group.
- Run the following commands to execute the jar packages.
java -cp HDFSTest-XXX.jar:lib/* com.huawei.bigdata.hdfs.examples.HdfsExample
java -cp HDFSTest-XXX.jar:lib/* com.huawei.bigdata.hdfs.examples.ColocationExample
When com.huawei.bigdata.hdfs.examples.ColocationExample is run, the HDFSparameter fs.defaultFS cannot be set to viewfs://ClusterX.
Checking the Commissioning Result
- Learn the application running conditions by viewing the running result.
- The running result of the HDFS example application is shown as follows:
[root@192-168-32-144 client]#hadoop jar HDFSTest-XXX.jar com.huawei.bigdata.hdfs.examples.HdfsExample WARNING: Use "yarn jar" to launch YARN applications. 17/10/26 19:11:44 INFO examples.HdfsExample: success to create path /user/hdfs-examples 17/10/26 19:11:44 INFO examples.HdfsExample: success to write. 17/10/26 19:11:45 INFO examples.HdfsExample: success to append. 17/10/26 19:11:45 INFO examples.HdfsExample: result is : hi, I am bigdata. It is successful if you can see me.I append this content. 17/10/26 19:11:45 INFO examples.HdfsExample: success to read. 17/10/26 19:11:45 INFO examples.HdfsExample: success to delete the file /user/hdfs-examples/test.txt 17/10/26 19:11:45 INFO examples.HdfsExample: success to delete path /user/hdfs-examples 17/10/26 19:11:45 INFO examples.HdfsExample: success to create path /user/hdfs-examples/hdfs_example_1 17/10/26 19:11:45 INFO examples.HdfsExample: success to create path /user/hdfs-examples/hdfs_example_0 17/10/26 19:11:45 INFO examples.HdfsExample: success to write. 17/10/26 19:11:45 INFO examples.HdfsExample: success to write. 17/10/26 19:11:46 INFO examples.HdfsExample: success to append. 17/10/26 19:11:46 INFO examples.HdfsExample: result is : hi, I am bigdata. It is successful if you can see me.I append this content. 17/10/26 19:11:46 INFO examples.HdfsExample: success to read. 17/10/26 19:11:46 INFO examples.HdfsExample: success to delete the file /user/hdfs-examples/hdfs_example_1/test.txt 17/10/26 19:11:46 INFO examples.HdfsExample: success to delete path /user/hdfs-examples/hdfs_example_1 17/10/26 19:11:46 INFO examples.HdfsExample: success to append. 17/10/26 19:11:46 INFO examples.HdfsExample: result is : hi, I am bigdata. It is successful if you can see me.I append this content. 17/10/26 19:11:46 INFO examples.HdfsExample: success to read. 17/10/26 19:11:46 INFO examples.HdfsExample: success to delete the file /user/hdfs-examples/hdfs_example_0/test.txt 17/10/26 19:11:46 INFO examples.HdfsExample: success to delete path /user/hdfs-examples/hdfs_example_0
- The running result of the Colocation example application is shown as follows:
[root@192-168-32-144 client]#hadoop jar HDFSTest-XXX.jar com.huawei.bigdata.hdfs.examples.ColocationExample WARNING: Use "yarn jar" to launch YARN applications. 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=V100R002C30, built on 10/19/2017 04:21 GMT 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:host.name=192-168-32-144 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_144 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:java.home=/opt/client/JDK/jdk1.8.0_144/jre 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/client/Yarn/config:/opt/client/HDFS/hadoop/etc/hadoop:/opt/client/HDFS/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/crypter-0.0.6.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/dynalogger-V100R002C30.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/json-smart-1.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/zookeeper-3.5.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/cas-client-core-hw-3.3.3.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/opensaml-2.6.5.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/wc2frm-v1r2c60-20160429.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/asm-3.2.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/hadoop-auth-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/apache-log4j-extras-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/hadoop-crypto-adapter-0.0.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jsch-0.1.54.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/hadoop-annotations-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/activation-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/junit-4.11.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/xz-1.0.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jcip-annotations-1.0.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/xmlsec-1.5.7.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/SSO-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/client/HDFS/hadoop/share/hadoop/common/datasight-hadoop-trace-ping-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/hadoop-common-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/common/hadoop-nfs-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/javaluator-3.0.1.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/hdfs-inode-provider-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/hadoop-hdfs-client-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/hadoop-hdfs-colocation-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/hadoop-hdfs-datamovement-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/superior-client-1.0.0.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/spark-3.1.1-yarn-shuffle.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/dynalogger-V100R002C30.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/zookeeper-3.5.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/javaluator-3.0.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/xercesImpl-2.9.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/htrace-core-3.1.0-incubating.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/spark-3.1.1-yarn-shuffle.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/apache-log4j-extras-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/xml-apis-1.3.04.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/superior-yarn-scheduler-1.0.0.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/netty-all-4.0.23.Final.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-daemon-1.0.13.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/xmlenc-0.52.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-datasight-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-client-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-registry-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-api-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.1.1.jar:/opt/client/HDFS/hadoop/contrib/capacity-scheduler/*.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/apacheds-i18n-2.0.0-M15.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-math3-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jackson-xc-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jaxb-impl-2.2.3-1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-openstack-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-io-2.4.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-digester-1.8.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jetty-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jackson-core-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/api-asn1-api-1.0.0-M20.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-datajoin-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/avro-1.7.4.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jsr305-3.0.0.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/dynalogger-V100R002C30.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jersey-server-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-codec-1.4.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/curator-client-2.7.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-gridmix-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jaxb-api-2.2.2.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/nimbus-jose-jwt-3.9.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/json-smart-1.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/zookeeper-3.5.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-rumen-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/htrace-core-3.1.0-incubating.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jettison-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-collections-3.2.2.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/asm-3.2.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jetty-util-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/servlet-api-2.5.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-auth-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/log4j-1.2.17.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jackson-databind-2.2.3.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/curator-recipes-2.7.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/gson-2.2.4.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-beanutils-1.7.0.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jets3t-0.9.0.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/mockito-all-1.8.5.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jersey-json-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/apache-log4j-extras-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jackson-mapper-asl-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/protobuf-java-2.5.0.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/api-util-1.0.0-M20.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jsch-0.1.54.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-archives-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hamcrest-core-1.3.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-logging-1.1.3.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/java-xmlbuilder-0.4.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/joda-time-2.9.7.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/netty-3.6.2.Final.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-distcp-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-cli-1.2.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/activation-1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jackson-core-2.2.3.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/junit-4.11.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jsp-api-2.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/httpcore-4.4.4.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-hdfs-restore-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/httpclient-4.5.2.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/metrics-core-3.0.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-beanutils-core-1.8.0.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-azure-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jackson-annotations-2.2.3.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-streaming-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/xz-1.0.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jcip-annotations-1.0.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-lang3-3.3.2.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/paranamer-2.3.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/curator-framework-2.7.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-sls-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-httpclient-3.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/stax-api-1.0-2.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-compress-1.4.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/xmlenc-0.52.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-lang-2.6.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/azure-storage-2.0.0.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jersey-core-1.9.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-extras-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-net-3.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/hadoop-ant-3.1.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jackson-jaxrs-1.9.13.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/jetty-sslengine-6.1.26.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/guava-11.0.2.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/snappy-java-1.0.4.1.jar:/opt/client/HDFS/hadoop/share/hadoop/tools/lib/commons-configuration-1.6.jar 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/client/HDFS/hadoop/lib/native 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-504.el6.x86_64 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:user.name=root 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:user.home=/root 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/client 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:os.memory.free=97MB 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:os.memory.max=123MB 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Client environment:os.memory.total=123MB 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192-168-32-144:2181,192-168-32-67:2181,192-168-33-190:2181 sessionTimeout=20000 watcher=com.huawei.hadoop.oi.colocation.ZooKeeperWatcher@60410cd 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: zookeeper.request.timeout is not configured. Using default value 120000. 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: zookeeper.client.bind.port.range is not configured. 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: zookeeper.client.bind.address is not configured. 17/10/26 19:12:38 INFO client.FourLetterWordMain: connecting to 192-168-32-67 2181 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Got server principal from the server and it is null 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Using server principal zookeeper/192-168-32-67 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Opening socket connection to server 192-168-32-67/192.168.32.67:2181. Will not attempt to authenticate using SASL (unknown error) 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.32.144:52882, server: 192-168-32-67/192.168.32.67:2181 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Session establishment complete on server 192-168-32-67/192.168.32.67:2181, sessionid = 0x13000074b7e4687f, negotiated timeout = 20000 17/10/26 19:12:38 INFO colocation.ZKUtil: ZooKeeper colocation znode : /hadoop/colocationDetails. Will publish colocation details under this znode hierarchy. Create Group is running... 17/10/26 19:12:38 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192-168-32-144:2181,192-168-32-67:2181,192-168-33-190:2181 sessionTimeout=20000 watcher=com.huawei.hadoop.oi.colocation.ZooKeeperWatcher@f80945f 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: zookeeper.request.timeout is not configured. Using default value 120000. 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: zookeeper.client.bind.port.range is not configured. 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: zookeeper.client.bind.address is not configured. 17/10/26 19:12:38 INFO client.FourLetterWordMain: connecting to 192-168-32-144 2181 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Got server principal from the server and it is null 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Using server principal zookeeper/192-168-32-144 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Opening socket connection to server 192-168-32-144/192.168.32.144:2181. Will not attempt to authenticate using SASL (unknown error) 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.32.144:40383, server: 192-168-32-144/192.168.32.144:2181 17/10/26 19:12:38 INFO zookeeper.ClientCnxn: Session establishment complete on server 192-168-32-144/192.168.32.144:2181, sessionid = 0x12000059699f69e1, negotiated timeout = 20000 17/10/26 19:12:38 INFO colocation.ZKUtil: ZooKeeper colocation znode : /hadoop/colocationDetails. Will publish colocation details under this znode hierarchy. Create Group has finished. Put file is running... Put file has finished. Delete file is running... Delete file has finished. Delete Group is running... Delete Group has finished. 17/10/26 19:12:39 INFO zookeeper.ZooKeeper: Session: 0x13000074b7e4687f closed 17/10/26 19:12:39 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x13000074b7e4687f 17/10/26 19:12:39 INFO zookeeper.ZooKeeper: Session: 0x12000059699f69e1 closed 17/10/26 19:12:39 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x12000059699f69e1
- The running result of the HDFS example application is shown as follows:
- Learn the application running conditions by viewing HDFS logs.
The NameNode logs of HDFS offer immediate visibility into application running conditions. You can adjust application programs based on the logs.
Parent topic: Commissioning an HDFS Application
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
The system is busy. Please try again later.
For any further questions, feel free to contact us through the chatbot.
Chatbot
