Common Commands

For details about how to use Spark commands, visit http://spark.apache.org/docs/latest/quick-start.html.

• spark-shell

  It provides a simple commissioning tool that supports Scala language.

  Run the spark-shell command on the shell console to enter the Scala interactive interface. Obtain data from HDFS, perform calculation by using the RDD, and output and print the result.

  Example: A line of code can be used to collect statistics on the frequency of all words in a file.

  scala> sc.textFile("hdfs://hacluster/tmp/wordcount_data.txt").flatMap(line=> line.split(" ")).map(w => (w,1)).reduceByKey(_+_).collect()

• spark-submit

  This command is used to submit Spark applications to an MRS cluster for running and return the running result. The class, master, JAR file, and input parameters must be specified.

  Example: Run GroupByTest in the JAR file. There are four input parameters and the cluster running mode is the yarn-client mode.

  spark-submit --class org.apache.spark.examples.GroupByTest --master yarn --deploy-mode client ${SPARK_HOME}/examples/jars/spark-examples_2.11-2.3.2-mrs-2.0.jar 6 3000 3000 3

• spark-sql

  Start a Spark application and execute Spark SQL. You can specify local(--master local) or cluster mode (--master yarn).

  Example of starting an instance:

  spark-sql --master yarn

  Example of SQL:

  • SELECT key FROM src GROUP BY key;
  • EXPLAIN EXTENDED SHOW TABLES;
• spark-beeline

  Call Spark JDBCServer to execute Spark SQL to efficiently compute and analyze massive amounts of data. JDBCServer contains a long-term Spark task. All statements in the spark-beeline are submitted to the task for execution.

  Example of starting an instance:

  cd $SPARK_HOME/bin

  spark-beeline

  Example of SQL:

  1. CREATE TABLE info(id int, name string, company string);
  2. INSERT INTO TABLE info values(001,'jack','huawei');
  3. SELECT * FROM info;
• beeline

  Call Spark JDBCServer to execute Spark SQL to efficiently compute and analyze massive amounts of data. JDBCServer contains a long-term Spark task. All statements in the beeline are submitted to the task for execution.

  For security clusters with Kerberos authentication enabled

  cd $SPARK_HOME/bin

  ./beeline -u 'jdbc:hive2://ha-cluster/default;user.principal=spark/hadoop.COM;saslQop=auth-conf;auth=KERBEROS;principal=spark/hadoop.COM;'

  

  The spark/hadoop.COM character string is obtained from the principal character string displayed by running the klist -kt /opt/Bigdata/MRS_XXX/1_20_SparkResource/etc/spark.keytay command in the cluster. You can paste the character string to the beeline command.

  For normal clusters with Kerberos authentication disabled

  cd $SPARK_HOME/bin

  beeline

  Example of SQL:

  1. CREATE TABLE info(id int, name string, company string);
  2. INSERT INTO TABLE info values(001,'jack','huawei');
  3. SELECT * FROM info;

  You are advised to use spark-beeline because it is encapsulated based on beeline, which can be directly executed.

• run-example

  This command is used to run or debug the built-in sample code in the Spark open source community.

  Example: Running SparkPi

  run-example --master yarn --deploy-mode client SparkPi 100