Commissioning an HDFS Application in the Linux Environment

Updated on 2024-10-23 GMT+08:00

View PDF

Scenario

The Hadoop distributed file system (HDFS) application can run in the Linux operating system (OS). After the application code has been developed, you can upload the jar packages to the Linux and run the application.

After an HDFS application is run, you can learn the application running conditions by viewing the running result or HDFS logs.

Prerequisite

When the client is installed:
- The HDFS client has been installed.
- When the host where the Linux OS runs is not a node of the cluster, you are required to set the mapping between the host name and IP address in the hosts file of the node where the client is installed. The host name must be correctly mapped to the IP address.
When the client is not installed:
- A JDK has been installed in the Linux environment. The version of the JDK must be consistent with that of the JDK used by IDEA to export the JAR package.
- When the host where the Linux OS runs is not a node of the cluster, you are required to set the mapping between the host name and IP address in the hosts file of the node where the Linux OS runs. The host name must be correctly mapped to the IP address.

Compiling and Running the Program When the Client is Installed

Go to the local root directory of the project, copy the required configuration file to the conf folder of the local project, and run the following command in Windows cmd to compress the package:

mvn -s "{maven_setting_path}" clean package
NOTE:
- In the preceding command, {maven_setting_path} is the path of the settings.xml file of the local Maven.
- After the package is successfully packed, obtain the JAR package, for example, HDFSTest-XXX.jar, from the target subdirectory in the root directory of the project. The name of the JAR package varies according to the actual package.
Upload the exported jarpackages to any directory in the running environment of the client, for example, /opt/client.
Configure the environment variables:

cd /opt/client

source bigdata_env
Run the following commands to execute the jar packages.

hadoop jar HDFSTest-XXX.jar com.huawei.bigdata.hdfs.examples.HdfsExample

hadoop jar HDFSTest-XXX.jar com.huawei.bigdata.hdfs.examples.ColocationExample

NOTE:

When com.huawei.bigdata.hdfs.examples.ColocationExample is run, the HDFSparameter fs.defaultFS cannot be set to viewfs://ClusterX.

Compiling and Running the Program When the Client is not Installed

Go to the local root directory of the project, copy the required configuration file to the conf folder of the local project and run the following command in Windows cmd to compress the package:

mvn -s "{maven_setting_path}" clean package
NOTE:
- In the preceding command, {maven_setting_path} is the path of the settings.xml file of the local Maven.
- After the package is successfully packed, obtain the JAR package from the target subdirectory in the root directory of the project.
Upload the exported jar packages to any directory in the running environment of the Linux OS, for example, /opt/hadoop_client.
Upload the lib and conf folders of the project to the same directory that stores the jar packages (the lib Create a lib folder in the Linux operating environment directory (for example, /opt/client) and upload the required JAR packages. The lib folder contains all the JAR packages that the project depends on. For details, see section Preparing an Operating Environment.
(Optional) To run the SmallFS example code, modify the value of the fs.defaultFS configuration item in the core-site.xml file from hdfs://hacluster to sfs://hacluster.
Run the following commands to execute the jar packages.

java -cp HDFSTest-XXX.jar:conf/:lib/* com.huawei.bigdata.hdfs.examples.HdfsExample

java -cp HDFSTest-XXX.jar:conf/:lib/* com.huawei.bigdata.hdfs.examples.ColocationExample

NOTE:

When com.huawei.bigdata.hdfs.examples.ColocationExample is run, the HDFSparameter fs.defaultFS cannot be set to viewfs://ClusterX.

Viewing Commissioning Results

Learn the application running conditions by viewing the running result.

The running result of the HDFS example application is shown as follows:

0    [main] INFO  org.apache.hadoop.security.UserGroupInformation  - Login successful for user hdfsDevelop using keytab file user.keytab
1    [main] INFO  com.huawei.hadoop.security.LoginUtil  - Login success!!!!!!!!!!!!!!
568  [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
582  [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
793  [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples
969  [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
1068 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
1191 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
1191 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
1202 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/test.txt
1210 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples
1223 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_0
1224 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_1
1261 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
1264 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
2807 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
2810 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
2861 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
2861 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
2866 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_0/test.txt
2874 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_0
2874 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
2874 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
2879 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_1/test.txt
2885 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_1

The running result of the Colocation example application is shown as follows:

0    [main] INFO  com.huawei.hadoop.security.LoginUtil  - JaasConfiguration loginContextName=Client principal=hdfsDevelop useTicketCache=false keytabFile=/opt/hdfsDemo/conf/user.keytab
817  [main] INFO  org.apache.hadoop.security.UserGroupInformation  - Login successful for user hdfsDevelop using keytab file user.keytab
817  [main] INFO  com.huawei.hadoop.security.LoginUtil  - Login success!!!!!!!!!!!!!!
1380 [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
...
Create Group has finished.
Put file is running...
Put file has finished.
Delete file is running...
Delete file has finished.
Delete Group is running...
Delete Group has finished.
...

Learn the application running conditions by viewing HDFS logs.
The namenode logs of HDFS offer immediate visibility into application running conditions. You can adjust application programs based on the logs.

Parent topic: Commissioning an HDFS Application

Previous topic: Commissioning an HDFS Application in the Windows Environment

Next topic: FAQs in HDFS Application Development

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot