MapReduce Service
MapReduce Service

    All results for "" in this service

    Compute
    Elastic Cloud Server
    Huawei Cloud Flexus
    Bare Metal Server
    Auto Scaling
    Image Management Service
    Dedicated Host
    FunctionGraph
    Cloud Phone Host
    Huawei Cloud EulerOS
    Management & Governance
    Cloud Eye
    Identity and Access Management
    Cloud Trace Service
    Resource Formation Service
    Tag Management Service
    Log Tank Service
    Config
    OneAccess
    Resource Access Manager
    Simple Message Notification
    Application Performance Management
    Application Operations Management
    Organizations
    Optimization Advisor
    IAM Identity Center
    Cloud Operations Center
    Resource Governance Center
    Cloud Ecosystem
    KooGallery
    Partner Center
    Analytics
    MapReduce Service
    Data Lake Insight
    CloudTable Service
    Cloud Search Service
    Data Lake Visualization
    Data Ingestion Service
    GaussDB(DWS)
    DataArts Studio
    Data Lake Factory
    DataArts Lake Formation
    Others
    Product Pricing Details
    System Permissions
    Console Quick Start
    Common FAQs
    Instructions for Associating with a HUAWEI CLOUD Partner
    Message Center
    Blockchain
    Blockchain Service
    Web3 Node Engine Service
    Storage
    Object Storage Service
    Elastic Volume Service
    Cloud Backup and Recovery
    Storage Disaster Recovery Service
    Scalable File Service Turbo
    Scalable File Service
    Volume Backup Service
    Cloud Server Backup Service
    Data Express Service
    Dedicated Distributed Storage Service
    Databases
    Relational Database Service
    Document Database Service
    Data Admin Service
    Data Replication Service
    GeminiDB
    GaussDB
    Distributed Database Middleware
    Database and Application Migration UGO
    TaurusDB
    Dedicated Cloud
    Dedicated Computing Cluster
    AI
    Face Recognition Service
    Graph Engine Service
    Content Moderation
    Image Recognition
    Optical Character Recognition
    ModelArts
    ImageSearch
    Conversational Bot Service
    Speech Interaction Service
    Huawei HiLens
    Video Intelligent Analysis Service
    Content Delivery & Edge Computing
    Content Delivery Network
    Intelligent EdgeFabric
    CloudPond
    Intelligent EdgeCloud
    Developer Services
    ServiceStage
    CodeArts
    CodeArts PerfTest
    CodeArts Req
    CodeArts Pipeline
    CodeArts Build
    CodeArts Deploy
    CodeArts Artifact
    CodeArts TestPlan
    CodeArts Check
    CodeArts Repo
    Cloud Application Engine
    Networking
    Virtual Private Cloud
    Elastic IP
    Elastic Load Balance
    NAT Gateway
    Direct Connect
    Virtual Private Network
    VPC Endpoint
    Cloud Connect
    Enterprise Router
    Enterprise Switch
    Global Accelerator
    Migration
    Server Migration Service
    Object Storage Migration Service
    Cloud Data Migration
    Migration Center
    User Support
    My Account
    Billing Center
    Cost Center
    Resource Center
    Enterprise Management
    Service Tickets
    HUAWEI CLOUD (International) FAQs
    ICP Filing
    Support Plans
    My Credentials
    Customer Operation Capabilities
    Partner Support Plans
    Professional Services
    IoT
    IoT Device Access
    Security & Compliance
    Security Technologies and Applications
    Web Application Firewall
    Host Security Service
    Cloud Firewall
    SecMaster
    Anti-DDoS Service
    Data Encryption Workshop
    Database Security Service
    Cloud Bastion Host
    Data Security Center
    Cloud Certificate Manager
    Edge Security
    Managed Threat Detection
    Media Services
    Media Processing Center
    Video On Demand
    Live
    SparkRTC
    MetaStudio
    Containers
    Cloud Container Engine
    SoftWare Repository for Container
    Application Service Mesh
    Ubiquitous Cloud Native Service
    Cloud Container Instance
    Middleware
    Distributed Cache Service
    API Gateway
    Distributed Message Service for Kafka
    Distributed Message Service for RabbitMQ
    Distributed Message Service for RocketMQ
    Cloud Service Engine
    Multi-Site High Availability Service
    EventGrid
    Business Applications
    Workspace
    ROMA Connect
    Message & SMS
    Domain Name Service
    Edge Data Center Management
    Meeting
    Developer Tools
    SDK Developer Guide
    API Request Signing Guide
    Terraform
    Koo Command Line Interface
    Solutions
    SAP Cloud
    High Performance Computing
    MacroVerse aPaaS
    KooMessage
    KooPhone
    KooDrive
    Compute
    Elastic Cloud Server
    Huawei Cloud Flexus
    Bare Metal Server
    Auto Scaling
    Image Management Service
    Dedicated Host
    FunctionGraph
    Cloud Phone Host
    Huawei Cloud EulerOS
    Networking
    Virtual Private Cloud
    Elastic IP
    Elastic Load Balance
    NAT Gateway
    Direct Connect
    Virtual Private Network
    VPC Endpoint
    Cloud Connect
    Enterprise Router
    Enterprise Switch
    Global Accelerator
    Management & Governance
    Cloud Eye
    Identity and Access Management
    Cloud Trace Service
    Resource Formation Service
    Tag Management Service
    Log Tank Service
    Config
    OneAccess
    Resource Access Manager
    Simple Message Notification
    Application Performance Management
    Application Operations Management
    Organizations
    Optimization Advisor
    IAM Identity Center
    Cloud Operations Center
    Resource Governance Center
    Migration
    Server Migration Service
    Object Storage Migration Service
    Cloud Data Migration
    Migration Center
    Cloud Ecosystem
    KooGallery
    Partner Center
    User Support
    My Account
    Billing Center
    Cost Center
    Resource Center
    Enterprise Management
    Service Tickets
    HUAWEI CLOUD (International) FAQs
    ICP Filing
    Support Plans
    My Credentials
    Customer Operation Capabilities
    Partner Support Plans
    Professional Services
    Analytics
    MapReduce Service
    Data Lake Insight
    CloudTable Service
    Cloud Search Service
    Data Lake Visualization
    Data Ingestion Service
    GaussDB(DWS)
    DataArts Studio
    Data Lake Factory
    DataArts Lake Formation
    IoT
    IoT Device Access
    Others
    Product Pricing Details
    System Permissions
    Console Quick Start
    Common FAQs
    Instructions for Associating with a HUAWEI CLOUD Partner
    Message Center
    Security & Compliance
    Security Technologies and Applications
    Web Application Firewall
    Host Security Service
    Cloud Firewall
    SecMaster
    Anti-DDoS Service
    Data Encryption Workshop
    Database Security Service
    Cloud Bastion Host
    Data Security Center
    Cloud Certificate Manager
    Edge Security
    Managed Threat Detection
    Blockchain
    Blockchain Service
    Web3 Node Engine Service
    Media Services
    Media Processing Center
    Video On Demand
    Live
    SparkRTC
    MetaStudio
    Storage
    Object Storage Service
    Elastic Volume Service
    Cloud Backup and Recovery
    Storage Disaster Recovery Service
    Scalable File Service Turbo
    Scalable File Service
    Volume Backup Service
    Cloud Server Backup Service
    Data Express Service
    Dedicated Distributed Storage Service
    Containers
    Cloud Container Engine
    SoftWare Repository for Container
    Application Service Mesh
    Ubiquitous Cloud Native Service
    Cloud Container Instance
    Databases
    Relational Database Service
    Document Database Service
    Data Admin Service
    Data Replication Service
    GeminiDB
    GaussDB
    Distributed Database Middleware
    Database and Application Migration UGO
    TaurusDB
    Middleware
    Distributed Cache Service
    API Gateway
    Distributed Message Service for Kafka
    Distributed Message Service for RabbitMQ
    Distributed Message Service for RocketMQ
    Cloud Service Engine
    Multi-Site High Availability Service
    EventGrid
    Dedicated Cloud
    Dedicated Computing Cluster
    Business Applications
    Workspace
    ROMA Connect
    Message & SMS
    Domain Name Service
    Edge Data Center Management
    Meeting
    AI
    Face Recognition Service
    Graph Engine Service
    Content Moderation
    Image Recognition
    Optical Character Recognition
    ModelArts
    ImageSearch
    Conversational Bot Service
    Speech Interaction Service
    Huawei HiLens
    Video Intelligent Analysis Service
    Developer Tools
    SDK Developer Guide
    API Request Signing Guide
    Terraform
    Koo Command Line Interface
    Content Delivery & Edge Computing
    Content Delivery Network
    Intelligent EdgeFabric
    CloudPond
    Intelligent EdgeCloud
    Solutions
    SAP Cloud
    High Performance Computing
    Developer Services
    ServiceStage
    CodeArts
    CodeArts PerfTest
    CodeArts Req
    CodeArts Pipeline
    CodeArts Build
    CodeArts Deploy
    CodeArts Artifact
    CodeArts TestPlan
    CodeArts Check
    CodeArts Repo
    Cloud Application Engine
    MacroVerse aPaaS
    KooMessage
    KooPhone
    KooDrive

      All results for "" in this service

      On this page

      Show all

      Help Center/ MapReduce Service/ Developer Guide (Normal_3.x)/ HDFS Development Guide (Normal Mode)/ Commissioning an HDFS Application/ Commissioning an HDFS Application in the Windows Environment

      Commissioning an HDFS Application in the Windows Environment

      Updated on 2024-10-23 GMT+08:00
      View PDF
      Share

      Scenario

      After the code development is complete, you can run an application in the Windows development environment. If the network between the local and the cluster service plane is normal, you can perform the commissioning on the local host.

      After an HDFS application is run, you can learn the application running conditions by viewing the running result or HDFS logs.

      Compiling and Running the Program

      1. (Optional) In a development environment (for example, IntelliJ IDEA), a user must be specified to run the example code. There are two ways to specify the user:

        Select the sample program HdfsExample.java or ColocationExample.java to be run, right-click the project, and choose Run Configurations from the shortcut menu. In the dialog box that is displayed, select JavaApplication > HdfsExample to set the running parameters. On the menu bar of the IntelliJ IDEA, choose Run > Edit Configurations. In the dialog box that is displayed, set the running user.

        -DHADOOP_USER_NAME=test

        NOTE:

        The test user here is an example. To run the example code related to the Colocation operation, the user must be a member of the supergroup group.

      2. If the environment variable is configured according to 1 click Run to run the application. If not, choose the following two projects separately and run the projects:

        • Choose HdfsExample.java, right-click the project and choose Run 'HdfsExample.main()' from the shortcut menu to run the project.
        • Choose ColocationExample.java, right-click the project and choose Run 'ColocationExample.main()' from the shortcut menu to run the project.
        NOTE:
        • It is forbidden to restart HDFS service while HDFS application is in running status, otherwise the application will fail.
        • When the Colocation project is run, the HDFS parameter fs.defaultFS cannot be set to viewfs://ClusterX.

      Checking the Commissioning Result

      • Learn the application running conditions by viewing the running result.
        • The running result of the HDFS windows example application is shown as follows: 
          1654 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
2013 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples
2137 [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2590 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
3245 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
4447 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
4447 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
4509 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples\test.txt
4618 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples
4743 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_1
4743 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_0
5087 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
5087 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
6507 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
6553 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
7505 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
7505 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
7568 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_1\test.txt
7583 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
7583 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
7630 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_0\test.txt
7677 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_1
7739 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_0
          NOTE:

          In the Windows environment, the following exception occurs but does not affect services.

          java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

        • The running result of the Colocation windows example application is shown as follows: 
          1623 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:zookeeper.version=***, built on 10/19/2017 04:21 GMT
1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:host.name=siay7user1.china.huawei.com
1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.version=***
1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.vendor=Oracle Corporation
1670 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:java.home=D:\Program Files\Java\jre1.8.0_131
......
Create Group has finished.
Put file is running...
5930 [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Put file has finished.
Delete file is running...
Delete file has finished.
Delete Group is running...
Delete Group has finished.
6866 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session: 0x13000074b7e464b7 closed
6866 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  - EventThread shut down for session: 0x13000074b7e464b7
6928 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  - EventThread shut down for session: 0x14000073f13b657b
6928 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session: 0x14000073f13b657b closed
      • Learn the application running conditions by viewing HDFS logs.

        The NameNode logs of HDFS offer immediate visibility into application running conditions. You can adjust application programs based on the logs.

      We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

      Feedback

      Feedback

      Feedback

      0/500

      Selected Content

      Submit selected content with the feedback