Help Center/ MapReduce Service/ Developer Guide (LTS)/ HDFS Development Guide (Security Mode)/ Commissioning HDFS Applications/ Commissioning an HDFS Application in the Windows Environment
Updated on 2026-06-29 GMT+08:00
View PDF
Share

Commissioning an HDFS Application in the Windows Environment

Scenario

After the code development is complete, you can run an application in the Windows development environment.

If the network between the local and the cluster service plane is normal, you can perform the commissioning on the local host.

After an HDFS application is run, you can learn the application running conditions by viewing the running result or HDFS logs.

Compiling and Running the Program

  • Compiling and Running the Example for Connecting Spring Boot to HDFS
    1. Table 1 lists the configuration files to be used during the login to the HDFS. These files have been imported into the src/main/resource directory of the springboot > hdfs-examples project.
      Table 1 Configuration files

      File

      Function

      core-site.xml

      HDFS configuration parameters

      hdfs-site.xml

      HDFS configuration parameters

      user.keytab

      HDFS user information for Kerberos security authentication

      krb5.conf

      Kerberos server configuration information

      application.properties

      Configuration information for Spring Boot. The following information needs to be configured:

      • Absolute path of the src/main/resource directory of the hdfs-examples project
      • Username needed for logging in to the HDFS
    2. Modify the parameters in the application.properties file.
      • spring.hdfs.config.dir: URL of the HDFS configuration file directory. The configuration files include hdfs-site.xml, core-site.xml, krb5.conf, and user.keytab.
      • hdfs.user: service user name for accessing the HDFS.
    3. In the development environment (for example, IntelliJ IDEA), choose SpringDataHDFSApplication.java and right-click the project. Select Run 'SpringDataHDFSApplication.main()' to run the project.
    4. Access the following links in sequence in the browser:
      • http://127.0.0.1:8080/save?fileName=test.txt: stores test.txt data in the /tmp/examples directory of the HDFS.
      • http://127.0.0.1:8080/read?fileName=test.txt: reads the test.txt file stored in the /tmp/examples directory of the HDFS.
      • http://127.0.0.1:8080/remove?fileName=test.txt: delete the test.txt file from the HDFS directory /tmp/examples.
  • Compiling and Running Sample Programs Except Spring Boot
    1. In a development environment (such as IntelliJ IDEA), choose the following two projects separately and run the projects:
      • Choose HdfsExample.java and right-click the project and choose Run 'HdfsExample.main()' from the shortcut menu to run the project.
      • Choose ColocationExample.java and right-click the project and choose Run 'ColocationExample.main()' from the shortcut menu to run the project.
      • Do not restart HDFS service while HDFS application is in running status, otherwise the application will fail.
      • When the Colocation project is run, the HDFS parameter fs.defaultFS cannot be set to viewfs://ClusterX.

Setting the Environment Variables of the System

During security authentication, Hadoop needs to obtain the domain name of the host where the client is located (Default Realm, which is obtained from environment variable USERDNSDOMAIN). If the host does not have a domain name, the following error message is displayed when you run the sample program:

You can set environment variable USERDNSDOMAIN of the system to prevent this problem. The details are as follows:

  1. Right-click Computer and choose Properties from the shortcut menu. The dialog box shown in the following figure is displayed. Click Advanced system settings > Advanced > Environment Variables.
  2. Set the system environment variable and click "New". The New System Variable dialog box is displayed. Enter USERDNSDOMAIN in Variable name and set the Variable value to a non-null character string, for example, test.

    Click OK twice. The system environment variable is set.

  3. Close the sample project, open it again, and run it.

Checking the Commissioning Result

  • Learn the application running conditions by viewing the running result.
    • The running result of the HDFS windows example application is shown as follows: 
      1308 [main] INFO  org.apache.hadoop.security.UserGroupInformation  - Login successful for user hdfsDevelop using keytab file 
1308 [main] INFO  com.huawei.hadoop.security.LoginUtil  - Login success!!!!!!!!!!!!!!
2040 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
3006 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples
3131 [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3598 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
4408 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
5015 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
5015 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
5077 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples\test.txt
5186 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples
5311 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_0
5311 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_1
5669 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
5669 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
7258 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
7741 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
7896 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
7896 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
7959 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_1\test.txt
8068 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_1
8364 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
8364 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
8426 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_0\test.txt
8535 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_0

      In the Windows environment, the following exception occurs but does not affect services.

      java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    • The running result of the Colocation windows example application is shown as follows: 
      ...
945  [main] INFO  org.apache.hadoop.security.UserGroupInformation  - Login successful for user hdfsDeveloper using keytab file user.keytab
945  [main] INFO  com.huawei.hadoop.security.LoginUtil  - Login success!!!!!!!!!!!!!!
945  [main] INFO  com.huawei.hadoop.security.LoginUtil  - JaasConfiguration loginContextName=Client principal=hdfsDeveloper useTicketCache=false keytabFile=XXX\sample_project\src\hdfs-example-security\conf\user.keytab
946  [main] INFO  com.huawei.hadoop.security.KerberosUtil  - Get default realm successfully, the realm is : HADOOP.COM
...
Create Group has finished.
Put file is running...
Put file has finished.
Delete file is running...
Delete file has finished.
Delete Group is running...
Delete Group has finished.
4946 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  - EventThread shut down for connection: 0x440751cb41a4d415
4946 [main] INFO  org.apache.zookeeper.ZooKeeper  - Connection: 0x440751cb41a4d415 closed
...
    • Sample running result of the interconnection between Spring Boot and HDFS.

      Spring Boot startup logs:

      SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/example/spring-boot-hdfs-1.0-SNAPSHOT.jar!/BOOT-INF/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/example/spring-boot-hdfs-1.0-SNAPSHOT.jar!/BOOT-INF/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.4.2)
2025-01-21 10:43:58.049  INFO 3628949 --- [           main] c.h.s.hdfs.SpringDataHDFSApplication     : Starting SpringDataHDFSApplication v1.0-SNAPSHOT using Java 1.8.0_422 on hdcore-common-001 with PID 3628949 (/opt/example/spring-boot-hdfs-1.0-SNAPSHOT.jar started by root in /opt/example)
2025-01-21 10:43:58.054  INFO 3628949 --- [           main] c.h.s.hdfs.SpringDataHDFSApplication     : No active profile set, falling back to default profiles: default
2025-01-21 10:43:58.928  WARN 3628949 --- [kground-preinit] o.s.h.c.j.Jackson2ObjectMapperBuilder    : For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
2025-01-21 10:43:59.695  INFO 3628949 --- [           main] org.eclipse.jetty.util.log               : Logging initialized @3808ms to org.eclipse.jetty.util.log.Slf4jLog
2025-01-21 10:43:59.845  INFO 3628949 --- [           main] o.s.b.w.e.j.JettyServletWebServerFactory : Server initialized with port: 8080
2025-01-21 10:43:59.848  INFO 3628949 --- [           main] org.eclipse.jetty.server.Server          : jetty-9.4.35.v20201120; built: 2020-11-20T21:17:03.964Z; git: bdc54f03a5e0a7e280fab27f55c3c75ee8da89fb; jvm 1.8.0_422-BiSheng_JDK8_Enterprise_202.1.0.420.B002-b05
2025-01-21 10:43:59.886  INFO 3628949 --- [           main] o.e.j.s.h.ContextHandler.application     : Initializing Spring embedded WebApplicationContext
2025-01-21 10:43:59.886  INFO 3628949 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1180 ms
2025-01-21 10:44:00.018  INFO 3628949 --- [           main] org.eclipse.jetty.server.session         : DefaultSessionIdManager workerName=node0
2025-01-21 10:44:00.018  INFO 3628949 --- [           main] org.eclipse.jetty.server.session         : No SessionScavenger set, using defaults
2025-01-21 10:44:00.021  INFO 3628949 --- [           main] org.eclipse.jetty.server.session         : node0 Scavenging every 660000ms
2025-01-21 10:44:00.035  INFO 3628949 --- [           main] o.e.jetty.server.handler.ContextHandler  : Started o.s.b.w.e.j.JettyEmbeddedWebAppContext@7ce3cb8e{application,/,[file:///tmp/jetty-docbase.8080.5072591134057219935/],AVAILABLE}
2025-01-21 10:44:00.036  INFO 3628949 --- [           main] org.eclipse.jetty.server.Server          : Started @4150ms
2025-01-21 10:44:00.643  INFO 3628949 --- [           main] o.a.h.security.UserGroupInformation      : Login successful for user admintest using keytab file user.keytab. Keytab auto renewal enabled : false
2025-01-21 10:44:00.643  INFO 3628949 --- [           main] com.huawei.spring.hdfs.LoginUtil         : Login success!!!!!!!!!!!!!!
2025-01-21 10:44:01.464  WARN 3628949 --- [           main] org.apache.hadoop.util.NativeCodeLoader  : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2025-01-21 10:44:01.471  WARN 3628949 --- [           main] o.a.h.h.s.DomainSocketFactory            : The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2025-01-21 10:44:01.711  INFO 3628949 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2025-01-21 10:44:01.975  INFO 3628949 --- [           main] o.e.j.s.h.ContextHandler.application     : Initializing Spring DispatcherServlet 'dispatcherServlet'
2025-01-21 10:44:01.975  INFO 3628949 --- [           main] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2025-01-21 10:44:01.976  INFO 3628949 --- [           main] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
2025-01-21 10:44:02.004  INFO 3628949 --- [           main] o.e.jetty.server.AbstractConnector       : Started ServerConnector@2133814f{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2025-01-21 10:44:02.005  INFO 3628949 --- [           main] o.s.b.web.embedded.jetty.JettyWebServer  : Jetty started on port(s) 8080 (http/1.1) with context path '/'
2025-01-21 10:44:02.023  INFO 3628949 --- [           main] c.h.s.hdfs.SpringDataHDFSApplication     : Started SpringDataHDFSApplication in 4.513 seconds (JVM running for 6.138)

      The access result is as follows:

  • Learn the application running conditions by viewing HDFS logs.

    The NameNode logs of HDFS offer immediate visibility into application running conditions. You can adjust application programs based on the logs.