Updated on 2023-04-10 GMT+08:00

Checking the Commissioning Result

Scenario

After an HDFS application is run, you can learn the application running conditions by viewing the running result or HDFS logs.

Procedure

  • Learn the application running conditions by viewing the running result.
    • The running result of the HDFS windows example application is shown as follows:
      1654 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
      2013 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples
      2137 [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      2590 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
      3245 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
      4447 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
      4447 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
      4509 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples\test.txt
      4618 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples
      4743 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_1
      4743 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_0
      5087 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
      5087 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
      6507 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
      6553 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
      7505 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
      7505 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
      7568 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_1\test.txt
      7583 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
      7583 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
      7630 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_0\test.txt
      7677 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_1
      7739 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_0

      In the Windows environment, the following exception occurs but does not affect services.

      java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    • The running result of the Colocation windows example application is shown as follows:
      1623 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:zookeeper.version=V100R002C30, built on 10/19/2017 04:21 GMT
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:host.name=siay7user1.china.huawei.com
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.version=1.8.0_131
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.vendor=Oracle Corporation
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.home=D:\Program Files\Java\jre1.8.0_131
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.class.path=D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\bin;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-cli-1.2.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-codec-1.4.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-collections-3.2.2.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-configuration-1.6.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-io-2.4.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-lang-2.6.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\commons-logging-1.1.3.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\dynalogger-V100R002C30.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\guava-11.0.2.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-annotations-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-auth-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-common-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-client-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-colocation-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-datamovement-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-nfs-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-hdfs-restore-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-mapreduce-client-core-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\hadoop-nfs-3.1.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\htrace-core-3.1.0-incubating.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\log4j-1.2.17.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\protobuf-java-2.5.0.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\slf4j-api-1.7.10.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\slf4j-log4j12-1.7.10.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal\lib\zookeeper-3.5.1.jar;D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\SmallFS\FusionInsight-SmallFS-1.0.0.tar.gz\smallfs\share\datasight\smallfs\smallfs-main-V100R002C30.jar
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.library.path=D:\Program Files\Java\jre1.8.0_131\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;D:/Program Files/Java/jre1.8.0_131/bin/server;D:/Program Files/Java/jre1.8.0_131/bin;D:/Program Files/Java/jre1.8.0_131/lib/amd64;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\Program Files\Java\jdk1.8.0_131\bin;D:\Program Files\Java\jdk1.8.0_131\jre\bin;D:\Program Files (x86)\GitExtensions\;D:\Program Files\Git\cmd;D:\soft\apache-maven-3.2.2\bin;D:\soft\gnubin;D:\soft\protoc-2.5.0-win32;;D:\soft\TMSS;D:\installation package\eclipse-jee-mars-R-win32-x86_64\eclipse;;.
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.io.tmpdir=C:\Users\L00430~1\AppData\Local\Temp\
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.compiler=<NA>
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.name=Windows 7
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.arch=amd64
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.version=6.1
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:user.name=user
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:user.home=C:\Users\user
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:user.dir=D:\FIClient\nonSafety\FusionInsight_Cluster_<Cluster ID>_Services_ClientConfig\HDFS\hdfs-example-normal
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.memory.free=107MB
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.memory.max=1819MB
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.memory.total=123MB
      1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating client connection, connectString=192-168-32-144:2181,192-168-32-67:2181,192-168-33-190:2181 sessionTimeout=45000 watcher=com.huawei.hadoop.oi.colocation.ZooKeeperWatcher@5f9b2141
      1794 [main] INFO  org.apache.zookeeper.ClientCnxn  - zookeeper.request.timeout is not configured. Using default value 120000.
      1794 [main] INFO  org.apache.zookeeper.ClientCnxn  - zookeeper.client.bind.port.range is not configured.
      1794 [main] INFO  org.apache.zookeeper.ClientCnxn  - zookeeper.client.bind.address is not configured.
      1794 [main-SendThread(192-168-32-67:2181)] INFO  org.apache.zookeeper.client.FourLetterWordMain  - connecting to 192-168-32-67 2181
      1904 [main-SendThread(192-168-32-67:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Got server principal from the server and it is null
      1904 [main-SendThread(192-168-32-67:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Using server principal zookeeper/192-168-32-67
      1904 [main-SendThread(192-168-32-67:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 192-168-32-67/192.168.32.67:2181. Will not attempt to authenticate using SASL (unknown error)
      1966 [main-SendThread(192-168-32-67:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Socket connection established, initiating session, client: /192.168.35.189:50954, server: 192-168-32-67/192.168.32.67:2181
      2029 [main-SendThread(192-168-32-67:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Session establishment complete on server 192-168-32-67/192.168.32.67:2181, sessionid = 0x13000074b7e464b7, negotiated timeout = 45000
      2169 [main] INFO  com.huawei.hadoop.oi.colocation.ZKUtil  - ZooKeeper colocation znode : /hadoop/colocationDetails. Will publish colocation details under this znode hierarchy.
      Create Group is running...
      5212 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating client connection, connectString=192-168-32-144:2181,192-168-32-67:2181,192-168-33-190:2181 sessionTimeout=45000 watcher=com.huawei.hadoop.oi.colocation.ZooKeeperWatcher@2438dcd
      5212 [main] INFO  org.apache.zookeeper.ClientCnxn  - zookeeper.request.timeout is not configured. Using default value 120000.
      5212 [main] INFO  org.apache.zookeeper.ClientCnxn  - zookeeper.client.bind.port.range is not configured.
      5212 [main] INFO  org.apache.zookeeper.ClientCnxn  - zookeeper.client.bind.address is not configured.
      5212 [main-SendThread(192-168-33-190:2181)] INFO  org.apache.zookeeper.client.FourLetterWordMain  - connecting to 192-168-33-190 2181
      5321 [main-SendThread(192-168-33-190:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Got server principal from the server and it is null
      5321 [main-SendThread(192-168-33-190:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Using server principal zookeeper/192-168-33-190
      5321 [main-SendThread(192-168-33-190:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 192-168-33-190/192.168.33.190:2181. Will not attempt to authenticate using SASL (unknown error)
      5368 [main-SendThread(192-168-33-190:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Socket connection established, initiating session, client: /192.168.35.189:50962, server: 192-168-33-190/192.168.33.190:2181
      5430 [main-SendThread(192-168-33-190:2181)] INFO  org.apache.zookeeper.ClientCnxn  - Session establishment complete on server 192-168-33-190/192.168.33.190:2181, sessionid = 0x14000073f13b657b, negotiated timeout = 45000
      5540 [main] INFO  com.huawei.hadoop.oi.colocation.ZKUtil  - ZooKeeper colocation znode : /hadoop/colocationDetails. Will publish colocation details under this znode hierarchy.
      Create Group has finished.
      Put file is running...
      5930 [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Put file has finished.
      Delete file is running...
      Delete file has finished.
      Delete Group is running...
      Delete Group has finished.
      6866 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session: 0x13000074b7e464b7 closed
      6866 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  - EventThread shut down for session: 0x13000074b7e464b7
      6928 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  - EventThread shut down for session: 0x14000073f13b657b
      6928 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session: 0x14000073f13b657b closed
  • Learn the application running conditions by viewing HDFS logs.

    The NameNode logs of HDFS offer immediate visibility into application running conditions. You can adjust application programs based on the logs.