Compiling and Running Applications

Updated on 2024-10-23 GMT+08:00

View PDF

Scenario

You can run applications in the Linux environment after application code development is complete.

Prerequisites

You have stored the configuration file obtained in Preparing the Configuration Files for Connecting to the Cluster in the conf directory, for example, /opt/client/conf. This directory must be the same as the directory where the MRTest-XXX.jar package uploaded in 2 is stored.
You have prepared the data to be processed by referring to Planning MapReduce Accessing Multi-Component Sample Program Data.

Procedure

Go to the local root directory of the sample project and run the following command in the Windows CLI to package the sample project:

mvn -s "{maven_setting_path}" clean package
NOTE:
- In the preceding command, {maven_setting_path} indicates the path of the settings.xml file of the local Maven.
- After the packaging is successful, obtain the JAR package, for example, MRTest-XXX.jar, from the target subdirectory in the root directory of the project. The JAR package name is subject to the actual packaging result.
Upload the generated application package MRTest-XXX.jar to the Linux client, for example, /opt/client/conf, in the same directory as the configuration file.
Execute the sample project on Linux.
- For the MapReduce statistics sample project, run the following command to configure parameters and submit jobs.
  yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath>
  
  This command is used to set parameters and submit jobs. <inputPath> indicates the input path in HDFS and <outputPath> indicates the output path in HDFS.
  NOTE:
  - Before running the preceding command, ensure that the <outputPath> directory does not exist. Otherwise, an error is reported.
  - Do not restart the HDFS service during the running of MapReduce jobs. Otherwise, the jobs may fail.
- For the sample application about multi-component access from MapReduce, perform the following steps.
  1. Create the jaas_mr.conf file in the /opt/client/conf folder. The file content is as follows:
```
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="user.keytab"
principal="test@<System domain name>"
useTicketCache=false
storeKey=true
debug=true;
};
```
    NOTE:
    test: indicates the name of the user created in User Information for Cluster Authentication, for example, developuser.
    
    System domain name: Log in to FusionInsight Manager, choose System > Permission > Domain and Mutual Trust, and view the value of Local Domain, which is the current system domain name.
  2. Add the classpath required for running the sample project in the Linux environment. For example:
    export YARN_USER_CLASSPATH=/opt/client/conf:/opt/client/HBase/hbase/lib/*:/opt/client/HBase/hbase/lib/client-facing-thirdparty/*:/opt/client/Hive/Beeline/lib/*
  3. Submit the MapReduce job and run the following command to run the sample project.
    yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.MultiComponentExample
    NOTE:
    If ZooKeeper SSL is enabled in the cluster, check the mapred-site.xml configuration file before running the sample. (Obtained from the conf directory in the same directory as the JAR package to be compiled in the Preparing the Configuration Files of the Running Environment) Check whether the configuration items mapreduce.admin.map.child.java.opts and mapreduce.admin.reduce.child.java.opts of the contain the following information:
    
    -Dzookeeper.client.secure=true -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
    
    If no, add the preceding content to the end of the configuration item.

Parent topic: Commissioning an Application in the Linux Environment

Previous topic: Commissioning an Application in the Linux Environment

Next topic: Checking the Commissioning Result

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot