Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ Spark Development Guide/ FAQs About Spark Application Development/ Spark APIs/ Common Spark Commands
Updated on 2024-08-16 GMT+08:00
View PDF
Share

Common Spark Commands

For details about how to use Spark commands, visit http://spark.apache.org/docs/latest/quick-start.html.

Common Commands

Methods of running shell commands:

  1. Go to the directory where the Spark client is installed.
  2. Run the following command to initialize environment variables:

    source /opt/client/bigdata_env

  3. If the Kerberos authentication is enabled for the current cluster, run the following command to authenticate the user. If the Kerberos authentication is disabled for the current cluster, skip this step. The current user is the development user added in Preparing a Spark Application Development User.

    kinit MRS cluster user

    Example:

    • If the development user is a machine-machine user, run kinit -kt user.keytab sparkuser.
    • If the development user is a human-machine user, run kinit sparkuser.

  4. Run the Spark shell command.

The common Spark commands are as follows:

  • spark-shell

    It provides a simple commissioning tool that supports Scala language.

    Run the spark-shell command on the shell console to enter the Scala interactive interface. Obtain data from HDFS, perform calculation by using the RDD, and output and print the result.

    Example: A line of code can be used to collect statistics on the frequency of all words in a file.

    scala> sc.textFile("hdfs://hacluster/tmp/wordcount_data.txt").flatMap(line=> line.split(" ")).map(w => (w,1)).reduceByKey(_+_).collect()

  • spark-submit

    This command is used to submit Spark applications to an MRS cluster for running and return the running result. The class, master, JAR file, and input parameters must be specified.

    Example: Run GroupByTest in the JAR file. There are four input parameters and the cluster running mode is the yarn-client mode.

    spark-submit --class org.apache.spark.examples.GroupByTest --master yarn --deploy-mode client ${SPARK_HOME}/examples/jars/spark-examples_2.11-2.3.2-mrs-2.0.jar 6 3000 3000 3

  • spark-sql

    Start a Spark application and execute Spark SQL. You can specify local(--master local) or cluster mode (--master yarn).

    Example of starting an instance:

    spark-sql --master yarn

    Example of SQL:

    • SELECT key FROM src GROUP BY key;
    • EXPLAIN EXTENDED SHOW TABLES;
  • spark-beeline

    Call Spark JDBCServer to execute Spark SQL to efficiently compute and analyze massive amounts of data. JDBCServer contains a long-term Spark task. All statements in the spark-beeline are submitted to the task for execution.

    Example of starting an instance:

    cd $SPARK_HOME/bin

    spark-beeline

    Example of SQL:

    1. CREATE TABLE info(id int, name string, company string);
    2. INSERT INTO TABLE info values(001,'jack','huawei');
    3. SELECT * FROM info;
  • beeline

    Call Spark JDBCServer to execute Spark SQL to efficiently compute and analyze massive amounts of data. JDBCServer contains a long-term Spark task. All statements in the beeline are submitted to the task for execution.

    For security clusters with Kerberos authentication enabled

    cd $SPARK_HOME/bin

    ./beeline -u 'jdbc:hive2://ha-cluster/default;user.principal=spark/hadoop.COM;saslQop=auth-conf;auth=KERBEROS;principal=spark/hadoop.COM;'

    The spark/hadoop.COM character string is obtained from the principal character string displayed by running the klist -kt /opt/Bigdata/MRS_XXX/1_20_SparkResource/etc/spark.keytay command in the cluster. You can paste the character string to the beeline command.

    For normal clusters with Kerberos authentication disabled

    cd $SPARK_HOME/bin

    beeline

    Example of SQL:

    1. CREATE TABLE info(id int, name string, company string);
    2. INSERT INTO TABLE info values(001,'jack','huawei');
    3. SELECT * FROM info;

    You are advised to use spark-beeline because it is encapsulated based on beeline, which can be directly executed.

  • run-example

    This command is used to run or debug the built-in sample code in the Spark open source community.

    Example: Running SparkPi

    run-example --master yarn --deploy-mode client SparkPi 100

Parent topic: Spark APIs