Compiling and Running the Application

Scenario

After the code development is complete, you can run an application in the Linux development environment.

Procedure

1. Go to the local root directory of the project and run the following command in Windows cmd to compress the package:

   mvn -s "{maven_setting_path}" clean package

   

   • In the preceding command, {maven_setting_path} is the path of the setting.xml file of the local maven.
   • After the package is successfully packed, obtain the JAR package, for example, MRTest-XXX.jar, from the target subdirectory in the root directory of the project. The name of the JAR package varies according to the actual package.

2. Upload MRTest-XXX.jar to the Linux client, such as /opt/client/conf, the same directory where the configuration files are located in.
3. Submit the MapReduce job in the Linux environment and execute the sample project.
   • Run the following command for MapReduce statistics sample project to configure parameters and submit jobs:

     yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath>

     <inputPath> indicates the input path and <outputPath> indicates the output path in HDFS.

     

     • Before running the preceding command, upload the log1.txt and log2.txt files to the <inputPath> directory in HDFS. See Typical Scenarios.
     • Before running the preceding command, ensure that the <outputPath> directory is deleted. Otherwise, an error will occur.
     • Do not restart the HDFS service during the running of MapReduce tasks. Otherwise, the tasks may fail.
   • In MapReduce accessing multi-components sample project, perform the following operations:
     1. Obtain the hbase-site.xml, hiveclient.properties and hive-site.xml configuration files, create a folder for example /opt/client/conf, and save the configurations files.
        

        The file hbase-site.xml is acquired from the HBase client, hiveclient.properties and hive-site.xml are acquired from the Hive client.

     2. Add the classpath required for sample projects in the Linux environment, Example of classpath:

        export YARN_USER_CLASSPATH=/opt/client/conf:/opt/client/HBase/hbase/lib/*:/opt/client/HBase/hbase/lib/client-facing-thirdparty/*:/opt/client/Hive/Beeline/lib/*

     3. Submit MapReduce jobs. Run the following command to run the sample project:

        yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.MultiComponentExample

        

        If ZooKeeper SSL is enabled in the cluster, check the mapred-site.xml configuration file before running the sample. (Obtained from the conf directory in the same directory as the JAR package to be compiled in the Preparing an Operating Environment) Check whether the configuration items mapreduce.admin.map.child.java.opts and mapreduce.admin.reduce.child.java.opts of the contain the following information:

         -Dzookeeper.client.secure=true -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty

        If no, add the preceding content to the end of the configuration item.