Common Spark Commands

Updated on 2024-08-16 GMT+08:00

View PDF

For details about how to use Spark commands, visit http://spark.apache.org/docs/latest/quick-start.html.

Common Commands

Methods of running shell commands:

Go to the directory where the Spark client is installed.
Run the following command to initialize environment variables:

source /opt/client/bigdata_env
If the Kerberos authentication is enabled for the current cluster, run the following command to authenticate the user. If the Kerberos authentication is disabled for the current cluster, skip this step. The current user is the development user added in Preparing a Spark Application Development User.

kinit MRS cluster user

Example:
- If the development user is a machine-machine user, run kinit -kt user.keytab sparkuser.
- If the development user is a human-machine user, run kinit sparkuser.
Run the Spark shell command.

The common Spark commands are as follows:

spark-shell
It provides a simple commissioning tool that supports Scala language.

Run the spark-shell command on the shell console to enter the Scala interactive interface. Obtain data from HDFS, perform calculation by using the RDD, and output and print the result.

Example: A line of code can be used to collect statistics on the frequency of all words in a file.

scala> sc.textFile("hdfs://hacluster/tmp/wordcount_data.txt").flatMap(line=> line.split(" ")).map(w => (w,1)).reduceByKey(_+_).collect()
spark-submit
This command is used to submit Spark applications to an MRS cluster for running and return the running result. The class, master, JAR file, and input parameters must be specified.

Example: Run GroupByTest in the JAR file. There are four input parameters and the cluster running mode is the yarn-client mode.

spark-submit --class org.apache.spark.examples.GroupByTest --master yarn --deploy-mode client ${SPARK_HOME}/examples/jars/spark-examples_2.11-2.3.2-mrs-2.0.jar 6 3000 3000 3
spark-sql
Start a Spark application and execute Spark SQL. You can specify local(--master local) or cluster mode (--master yarn).

Example of starting an instance:

spark-sql --master yarn

Example of SQL:
- SELECT key FROM src GROUP BY key;
- EXPLAIN EXTENDED SHOW TABLES;
spark-beeline
Call Spark JDBCServer to execute Spark SQL to efficiently compute and analyze massive amounts of data. JDBCServer contains a long-term Spark task. All statements in the spark-beeline are submitted to the task for execution.

Example of starting an instance:

cd $SPARK_HOME/bin

spark-beeline

Example of SQL:
1. CREATE TABLE info(id int, name string, company string);
2. INSERT INTO TABLE info values(001,'jack','huawei');
3. SELECT * FROM info;
beeline
Call Spark JDBCServer to execute Spark SQL to efficiently compute and analyze massive amounts of data. JDBCServer contains a long-term Spark task. All statements in the beeline are submitted to the task for execution.

For security clusters with Kerberos authentication enabled

cd $SPARK_HOME/bin

./beeline -u 'jdbc:hive2://ha-cluster/default;user.principal=spark/hadoop.COM;saslQop=auth-conf;auth=KERBEROS;principal=spark/hadoop.COM;'

NOTE:

The spark/hadoop.COM character string is obtained from the principal character string displayed by running the klist -kt /opt/Bigdata/MRS_XXX/1_20_SparkResource/etc/spark.keytay command in the cluster. You can paste the character string to the beeline command.

For normal clusters with Kerberos authentication disabled

cd $SPARK_HOME/bin

beeline

Example of SQL:
1. CREATE TABLE info(id int, name string, company string);
2. INSERT INTO TABLE info values(001,'jack','huawei');
3. SELECT * FROM info;
You are advised to use spark-beeline because it is encapsulated based on beeline, which can be directly executed.
run-example
This command is used to run or debug the built-in sample code in the Spark open source community.

Example: Running SparkPi

run-example --master yarn --deploy-mode client SparkPi 100