更新时间：2024-10-31 GMT+08:00
在本地Windows中调测HDFS程序
操作场景
在代码完成开发后，您可以在Windows开发环境中运行应用。本地和集群业务平面网络互通时，您可以直接在本地进行调测。
HDFS应用程序运行完成后，可直接通过运行结果查看应用程序运行情况，也可以通过HDFS日志获取应用运行情况。
在本地Windows中调测HDFS程序
- （可选）在开发环境中（例如IntelliJ IDEA中），运行此样例代码需要设置运行用户，设置运行用户有两种方式，添加环境变量HADOOP_USER_NAME或者修改代码。设置环境变量方法具体如下：
选中需要运行的样例程序HdfsExample.java或者ColocationExample.java，右键工程，选择，在对话框中选择“JavaApplication > HdfsExample”进行运行参数设置。在IntelliJ IDEA的菜单栏依次选择“Run > Edit Configurations”，在弹出的会话框中设置运行用户。
-DHADOOP_USER_NAME=test
图1 参数配置
用户可向管理员咨询运行用户。test在这里只是举例，若需运行Colocation相关操作的样例代码，则此用户需属于supergroup用户组。
- 若已按照1设置环境变量，则直接单击Run，运行应用工程。否则分别选中以下两个工程运行程序：
- 选中HdfsExample.java，右键工程，选择运行应用工程。
- 选中ColocationExample.java，右键工程，选择运行应用工程。
- 在HDFS任务运行过程中禁止重启HDFS服务，否则可能会导致任务失败。
- 在运行Colocation工程时，HDFS的配置项fs.defaultFS不能配置为viewfs://ClusterX。
查看调测结果
- 查看运行结果获取应用运行情况
- HdfsExample Windows样例程序运行结果如下所示。
1654 [main] WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows. 2013 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to create path /user/hdfs-examples 2137 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2590 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to write. 3245 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to append. 4447 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - result is : hi, I am bigdata. It is successful if you can see me.I append this content. 4447 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to read. 4509 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete the file /user/hdfs-examples\test.txt 4618 [main] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete path /user/hdfs-examples 4743 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to create path /user/hdfs-examples/hdfs_example_1 4743 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to create path /user/hdfs-examples/hdfs_example_0 5087 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to write. 5087 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to write. 6507 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to append. 6553 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to append. 7505 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - result is : hi, I am bigdata. It is successful if you can see me.I append this content. 7505 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to read. 7568 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete the file /user/hdfs-examples/hdfs_example_1\test.txt 7583 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - result is : hi, I am bigdata. It is successful if you can see me.I append this content. 7583 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to read. 7630 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete the file /user/hdfs-examples/hdfs_example_0\test.txt 7677 [hdfs_example_1] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete path /user/hdfs-examples/hdfs_example_1 7739 [hdfs_example_0] INFO com.huawei.bigdata.hdfs.examples.HdfsExample - success to delete path /user/hdfs-examples/hdfs_example_0
在Windows环境运行样例代码时会出现下面的异常，但是不影响业务：
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
- ColocationExample Windows样例程序运行结果如下所示。
1623 [main] WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows. 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=***, built on 10/19/2017 04:21 GMT 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=siay7user1.china.huawei.com 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=*** 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=D:\Program Files\Java\jre1.8.0_131 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\bin;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-cli-1.2.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-codec-1.4.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-collections-3.2.2.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-configuration-1.6.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-io-2.4.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-lang-2.6.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-logging-1.1.3.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\dynalogger-V100R002C30.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\guava-11.0.2.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-annotations-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-auth-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-common-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-client-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-colocation-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-datamovement-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-nfs-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-restore-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-mapreduce-client-core-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-nfs-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\htrace-core-3.1.0-incubating.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\log4j-1.2.17.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\protobuf-java-2.5.0.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\slf4j-api-1.7.10.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\slf4j-log4j12-1.7.10.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\zookeeper-3.5.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\SmallFS\FusionInsight-SmallFS-1.0.0.tar.gz\smallfs\share\datasight\smallfs\smallfs-main-V100R002C30.jar 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=D:\Program Files\Java\jre1.8.0_131\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;D:/Program Files/Java/jre1.8.0_131/bin/server;D:/Program Files/Java/jre1.8.0_131/bin;D:/Program Files/Java/jre1.8.0_131/lib/amd64;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\Program Files\Java\jdk1.8.0_131\bin;D:\Program Files\Java\jdk1.8.0_131\jre\bin;D:\Program Files (x86)\GitExtensions\;D:\Program Files\Git\cmd;D:\soft\apache-maven-3.2.2\bin;D:\soft\gnubin;D:\soft\protoc-2.5.0-win32;;D:\soft\TMSS;D:\安装包\eclipse-jee-mars-R-win32-x86_64\eclipse;;. 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=C:\Users\L00430~1\AppData\Local\Temp\ 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Windows 7 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=6.1 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=user 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=C:\Users\user 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=D:\FIClient\nonSafety\FusionInsight_Cluster_<集群ID>_Services_ClientConfig\HDFS\hdfs-example-normal 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=107MB 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1819MB 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=123MB 1670 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192-168-32-144:2181,192-168-32-67:2181,192-168-33-190:2181 sessionTimeout=45000 watcher=com.huawei.hadoop.oi.colocation.ZooKeeperWatcher@5f9b2141 1794 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout is not configured. Using default value 120000. 1794 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.client.bind.port.range is not configured. 1794 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.client.bind.address is not configured. 1794 [main-SendThread(192-168-32-67:2181)] INFO org.apache.zookeeper.client.FourLetterWordMain - connecting to 192-168-32-67 2181 1904 [main-SendThread(192-168-32-67:2181)] INFO org.apache.zookeeper.ClientCnxn - Got server principal from the server and it is null 1904 [main-SendThread(192-168-32-67:2181)] INFO org.apache.zookeeper.ClientCnxn - Using server principal zookeeper/192-168-32-67 1904 [main-SendThread(192-168-32-67:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192-168-32-67/192.168.32.67:2181. Will not attempt to authenticate using SASL (unknown error) 1966 [main-SendThread(192-168-32-67:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.35.189:50954, server: 192-168-32-67/192.168.32.67:2181 2029 [main-SendThread(192-168-32-67:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192-168-32-67/192.168.32.67:2181, sessionid = 0x13000074b7e464b7, negotiated timeout = 45000 2169 [main] INFO com.huawei.hadoop.oi.colocation.ZKUtil - ZooKeeper colocation znode : /hadoop/colocationDetails. Will publish colocation details under this znode hierarchy. Create Group is running... 5212 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192-168-32-144:2181,192-168-32-67:2181,192-168-33-190:2181 sessionTimeout=45000 watcher=com.huawei.hadoop.oi.colocation.ZooKeeperWatcher@2438dcd 5212 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout is not configured. Using default value 120000. 5212 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.client.bind.port.range is not configured. 5212 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.client.bind.address is not configured. 5212 [main-SendThread(192-168-33-190:2181)] INFO org.apache.zookeeper.client.FourLetterWordMain - connecting to 192-168-33-190 2181 5321 [main-SendThread(192-168-33-190:2181)] INFO org.apache.zookeeper.ClientCnxn - Got server principal from the server and it is null 5321 [main-SendThread(192-168-33-190:2181)] INFO org.apache.zookeeper.ClientCnxn - Using server principal zookeeper/192-168-33-190 5321 [main-SendThread(192-168-33-190:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192-168-33-190/192.168.33.190:2181. Will not attempt to authenticate using SASL (unknown error) 5368 [main-SendThread(192-168-33-190:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.35.189:50962, server: 192-168-33-190/192.168.33.190:2181 5430 [main-SendThread(192-168-33-190:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192-168-33-190/192.168.33.190:2181, sessionid = 0x14000073f13b657b, negotiated timeout = 45000 5540 [main] INFO com.huawei.hadoop.oi.colocation.ZKUtil - ZooKeeper colocation znode : /hadoop/colocationDetails. Will publish colocation details under this znode hierarchy. Create Group has finished. Put file is running... 5930 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Put file has finished. Delete file is running... Delete file has finished. Delete Group is running... Delete Group has finished. 6866 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x13000074b7e464b7 closed 6866 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x13000074b7e464b7 6928 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x14000073f13b657b 6928 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x14000073f13b657b closed
- HdfsExample Windows样例程序运行结果如下所示。
- 查看HDFS日志获取应用运行情况
可以查看HDFS的NameNode日志了解应用运行情况，并根据日志信息调整应用程序。
父主题： 调测HDFS应用
