Help Center/ MapReduce Service/ Developer Guide (LTS)/ Spark2x Development Guide (Common Mode)/ Commissioning a Spark Application/ Commissioning a Spark Application in a Local Windows Environment
Updated on 2024-08-10 GMT+08:00
View PDF
Share

Commissioning a Spark Application in a Local Windows Environment

You can run an application in the Windows environment after application code development is complete. The procedures for running applications developed using Scala or Java are the same on IDEA.

  • In the Windows environment, only the sample code for accessing Spark SQL using JDBC is provided.
  • Ensure that the Maven image repository of the SDK in the Huawei image site has been configured for Maven. For details, see Configuring Huawei Open Source Image Repository.

Compiling and Running the Application

  1. Obtain the sample code.

    Download the Maven project source code and configuration files of the sample project. For details, see Obtaining the MRS Application Development Sample Project.

    Import the sample code to IDEA.

  1. Obtain the configuration files.

    Obtain the files from the cluster client. Download the hive-site.xml and spark-defaults.conf files from $SPARK_HOME/conf to a local directory.

  2. Upload data to HDFS.

    1. Create a data text file on Linux and save the following data to the data file: 
      Miranda,32 
Karlie,23 
Candice,27
    2. On the HDFS client running the Linux OS, run the hadoop fs -mkdir /data command (or the hdfs dfs command) to create a directory.
    3. On the HDFS client running the Linux OS, run the hadoop fs -put data /data command to upload the data file.

  3. Configure related parameters in the sample code.

    Change the SQL statement for loading data to LOAD DATA INPATH 'hdfs:/data/data' INTO TABLE CHILD.

  4. Add running parameters to the hive-site.xml and spark-defaults.conf files when the application is running.

  5. Run the application.

View Debugging Results

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/D:/mavenlocal/org/apache/logging/log4j/log4j-slf4j-impl/2.6.2/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/D:/mavenlocal/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
---- Begin executing sql: CREATE TABLE IF NOT EXISTS CHILD (NAME STRING, AGE INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ----
Result	
---- Done executing sql: CREATE TABLE IF NOT EXISTS CHILD (NAME STRING, AGE INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ----
---- Begin executing sql: LOAD DATA INPATH 'hdfs:/data/data' INTO TABLE CHILD ----
Result	
---- Done executing sql: LOAD DATA INPATH 'hdfs:/data/data' INTO TABLE CHILD ----
---- Begin executing sql: SELECT * FROM child ----
NAME	AGE	
Miranda	32	
Karlie	23	
Candice	27	
---- Done executing sql: SELECT * FROM child ----
---- Begin executing sql: DROP TABLE child ----
Result	
---- Done executing sql: DROP TABLE child ----

Process finished with exit code 0