在本地Windows中调测HDFS程序

操作场景

在代码完成开发后，您可以在Windows开发环境中运行应用。本地和集群业务平面网络互通时，您可以直接在本地进行调测。

HDFS应用程序运行完成后，可直接通过运行结果查看应用程序运行情况，也可以通过HDFS日志获取应用运行情况。

在本地Windows中调测HDFS程序

1. （可选）在开发环境中（例如IntelliJ IDEA中），运行此样例代码需要设置运行用户，设置运行用户有两种方式，添加环境变量HADOOP_USER_NAME或者修改代码。设置环境变量方法具体如下：
   选中需要运行的样例程序HdfsExample.java或者ColocationExample.java，右键工程，选择“Run Configurations”，在对话框中选择“JavaApplication > HdfsExample”进行运行参数设置。在IntelliJ IDEA的菜单栏依次选择“Run > Edit Configurations”，在弹出的会话框中设置运行用户。

   -DHADOOP_USER_NAME=test

   

   

   用户可向管理员咨询运行用户。test在这里只是举例，若需运行Colocation相关操作的样例代码，则此用户需属于supergroup用户组。

2. 若已按照1设置环境变量，则直接单击Run，运行应用工程。否则分别选中以下两个工程运行程序：
   • 选中HdfsExample.java，右键工程，选择“Run 'HdfsExample.main()'"”运行应用工程。
   • 选中ColocationExample.java，右键工程，选择“Run 'ColocationExample.main()'”运行应用工程。
   

   • 在HDFS任务运行过程中禁止重启HDFS服务，否则可能会导致任务失败。

   • 在运行Colocation工程时，HDFS的配置项fs.defaultFS不能配置为viewfs://ClusterX。

查看调测结果

• 查看运行结果获取应用运行情况
  • HdfsExample Windows样例程序运行结果如下所示。

    1654 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
    2013 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples
    2137 [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    2590 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
    3245 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
    4447 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
    4447 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
    4509 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples\test.txt
    4618 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples
    4743 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_1
    4743 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_0
    5087 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
    5087 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
    6507 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
    6553 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
    7505 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
    7505 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
    7568 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_1\test.txt
    7583 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
    7583 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
    7630 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_0\test.txt
    7677 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_1
    7739 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_0

    

    在Windows环境运行样例代码时会出现下面的异常，但是不影响业务：

    java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

  • ColocationExample Windows样例程序运行结果如下所示。

    1623 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
    1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:zookeeper.version=***, built on 10/19/2017 04:21 GMT
    1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:host.name=siay7user1.china.huawei.com
    1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.version=***
    1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.vendor=Oracle Corporation
    1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.home=D:\Program Files\Java\jre1.8.0_131
    ......
    Create Group has finished.
    Put file is running...
    5930 [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Put file has finished.
    Delete file is running...
    Delete file has finished.
    Delete Group is running...
    Delete Group has finished.
    6866 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session: 0x13000074b7e464b7 closed
    6866 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  - EventThread shut down for session: 0x13000074b7e464b7
    6928 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  - EventThread shut down for session: 0x14000073f13b657b
    6928 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session: 0x14000073f13b657b closed

• 查看HDFS日志获取应用运行情况

  可以查看HDFS的NameNode日志了解应用运行情况，并根据日志信息调整应用程序。