Commissioning an Application in the Windows Environment
Scenario
After the code development is complete, you can run an application in the Windows development environment.
If the network between the local and the cluster service plane is normal, you can perform the commissioning on the local host.
After an HDFS application is run, you can learn the application running conditions by viewing the running result or HDFS logs.
Commissioning an Application in the Windows Environment
- In a development environment (such as IntelliJ IDEA), choose the following two projects separately and run the projects:
- Choose HdfsExample.java and right-click the project and choose from the shortcut menu to run the project.
- Choose ColocationExample.java and right-click the project and choose from the shortcut menu to run the project.
- Do not restart HDFS service while HDFS application is in running status, otherwise the application will fail.
- When the Colocation project is run, the HDFS parameter fs.defaultFS cannot be set to viewfs://ClusterX.
Note
During security authentication, Hadoop needs to obtain the domain name of the host where the client is located (Default Realm, which is obtained from environment variable USERDNSDOMAIN). If the host does not have a domain name, the following error message is displayed when you run the sample program:
You can set environment variable USERDNSDOMAIN of the system to prevent this problem. The details are as follows:
- Right-click Computer and choose Properties from the shortcut menu. The dialog box shown in the following figure is displayed. Click Advanced system settings > Advanced > Environment Variables.
- Set the system environment variable and click "New". The New System Variable dialog box is displayed. Enter USERDNSDOMAIN in Variable name and set the Variable value to a non-null character string, for example, test.
Click OK twice. The system environment variable is set.
- Close the sample project, open it again, and run it.
Checking the Commissioning Result
- Learn the application running conditions by viewing the running result.
- The running result of the HDFS windows example application is shown as follows:
1308 [main] INFO org.apache.hadoop.security.UserGroupInformation - Login successful for user hdfsDevelop using keytab file 1308 [main] INFO com.huawei.hadoop.security.LoginUtil - Login success!!!!!!!!!!!!!! 2040 [main] WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows. 3006 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to create path /user/hdfs-examples 3131 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 3598 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to write. 4408 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to append. 5015 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - result is : hi, I am bigdata. It is successful if you can see me.I append this content. 5015 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to read. 5077 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete the file /user/hdfs-examples\test.txt 5186 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete path /user/hdfs-examples 5311 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to create path /user/hdfs-examples/hdfs_example_0 5311 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to create path /user/hdfs-examples/hdfs_example_1 5669 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to write. 5669 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to write. 7258 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to append. 7741 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to append. 7896 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - result is : hi, I am bigdata. It is successful if you can see me.I append this content. 7896 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to read. 7959 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete the file /user/hdfs-examples/hdfs_example_1\test.txt 8068 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete path /user/hdfs-examples/hdfs_example_1 8364 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - result is : hi, I am bigdata. It is successful if you can see me.I append this content. 8364 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to read. 8426 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete the file /user/hdfs-examples/hdfs_example_0\test.txt 8535 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete path /user/hdfs-examples/hdfs_example_0
In the Windows environment, the following exception occurs but does not affect services.
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
- The running result of the Colocation windows example application is shown as follows:
... 945 [main] INFO org.apache.hadoop.security.UserGroupInformation - Login successful for user hdfsDeveloper using keytab file user.keytab 945 [main] INFO com.huawei.hadoop.security.LoginUtil - Login success!!!!!!!!!!!!!! 945 [main] INFO com.huawei.hadoop.security.LoginUtil - JaasConfiguration loginContextName=Client principal=hdfsDeveloper useTicketCache=false keytabFile=XXX\sample_project\src\hdfs-example-security\conf\user.keytab 946 [main] INFO com.huawei.hadoop.security.KerberosUtil - Get default realm successfully, the realm is : HADOOP.COM ... Create Group has finished. Put file is running... Put file has finished. Delete file is running... Delete file has finished. Delete Group is running... Delete Group has finished. 4946 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for connection: 0x440751cb41a4d415 4946 [main] INFO org.apache.zookeeper.ZooKeeper - Connection: 0x440751cb41a4d415 closed ...
- The running result of the HDFS windows example application is shown as follows:
- Learn the application running conditions by viewing HDFS logs.
The NameNode logs of HDFS offer immediate visibility into application running conditions. You can adjust application programs based on the logs.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot