Help Center/ Data Lake Insight/ Developer Guide/ Spark Jar Jobs/ Using Spark-submit to Submit a Spark Jar Job
Updated on 2024-06-13 GMT+08:00
View PDF

Using Spark-submit to Submit a Spark Jar Job

Introduction to DLI Spark-submit

DLI Spark-submit is a command line tool used to submit Spark jobs to the DLI server. This tool provides command lines compatible with open-source Spark.

Preparations

  1. Getting authorized.

    DLI uses the Identity and Access Management (IAM) to implement fine-grained permissions for your enterprise-level tenants. IAM provides identity authentication, permissions management, and access control, helping you securely access your HUAWEI CLOUD resources.

    With IAM, you can use your HUAWEI CLOUD account to create IAM users for your employees, and assign permissions to the users to control their access to specific resource types.

    Currently, roles (coarse-grained authorization) and policies (fine-grained authorization) are supported. For details about permissions and authorization operations, see the Data Lake Insight User Guide.

  2. Create a queue. Choose Resources > Queue Management. On the page displayed, click Buy Queue in the upper right corner. On the Buy Queue page displayed, select For general purpose for Type, that is, the compute resources of the Spark job.

    If the user who creates the queue is not an administrator, the queue can be used only after being authorized by the administrator. For details about how to assign permissions, see Queue Permission Management.

Downloading the DLI Client Tool

You can download the DLI client tool from the DLI management console.

  1. Log in to the DLI management console.
  2. Obtain the SDK download address from the administrator.
  3. On the DLI SDK DOWNLOAD page, click dli-clientkit-<version> to download the DLI client tool.

    The Beeline client is named dli-clientkit-<version>-bin.tar.gz, which can be used in Linux and depends on JDK 1.8 or later.

Configuring DLI Spark-submit

Ensure that you have installed JDK 1.8 or later and configured environment variables on the computer where spark-submit is installed. You are advised to use spark-submit on the computer running Linux.

  1. Download and decompress dli-clientkit-<version>-bin.tar.gz. In this step, set version to the actual version.
  2. Go to the directory where dli-clientkit-<version>-bin.tar.gz is decompressed. In the directory, there are three subdirectories bin, conf, and lib, which respectively store the execution scripts, configuration files, and dependency packages related to Spark-submit.
  3. Go to the conf directory and modify the configuration items in the client.properties file. For details about the configuration items, see Table 1.

    Table 1 DLI client parameters

    Item

    Mandatory

    Default Value

    Description

    dliEndPont

    No

    -

    Domain name of DLI

    Obtain the domain name corresponding to the region of DLI at "Regions and Endpoints."

    If you lef this parameter empty, the program determines the domain name based on region.

    obsEndPoint

    Yes

    -

    OBS service domain name.

    Obtain the domain name corresponding to the region of OBS at "Regions and Endpoints."

    bucketName

    Yes

    -

    Name of a bucket on OBS. This bucket is used to store JAR files, Python program files, and configuration files used in Spark programs.

    obsPath

    Yes

    dli-spark-submit-resources

    Directory for storing JAR files, Python program files, and configuration files on OBS. The directory is in the bucket specified by Bucket Name. If the directory does not exist, the program automatically creates it.

    localFilePath

    Yes

    -

    The local directory for storing JAR files, Python program files, and configuration files used in Spark programs.

    The program automatically uploads the files on which Spark depends to the OBS path and loads them to the resource package on the DLI server.

    ak

    Yes

    -

    User's Access Key (AK)

    sk

    Yes

    -

    User's Secret Key (SK)

    projectId

    Yes

    -

    Project ID used by a user to access DLI.

    region

    Yes

    -

    Region of interconnected DLI.

    Modify the configuration items in the spark-defaults.conf file based on the Spark application requirements. The configuration items are compatible with the open-source Spark configuration items. For details, see the open-source Spark configuration item description.

Using Spark-submit to Submit a Spark Job

  1. Go to the bin directory of the tool file, run the spark-submit command, and carry related parameters.

    The command format is as follows: 
    spark-submit [options] <app jar | python file> [app arguments]
    Table 2 DLI Spark-submit parameters

    Parameter

    Value

    Description

    --class

    <CLASS_NAME>

    Name of the main class of the submitted Java or Scala application.

    --conf

    <PROP=VALUE>

    Spark program parameters can be configured in the spark-defaults.conf file in the conf directory. If both the command and the configuration file are configured, the parameter value specified in the command is preferentially used.

    NOTE:

    If there are multiple conf files, the format is --conf key1=value1 --conf key2=value2.

    --jars

    <JARS>

    Name of the JAR file on which the Spark application depends. Use commas (,) to separate multiple names. The JAR file must be stored in the local path specified by localFilePath in the client.properties file in advance.

    --name

    <NAME>

    Name of a Spark application.

    --queue

    <QUEUE_NAME>

    Name of the Spark queue on the DLI server. Jobs are submitted to the queue for execution.

    --py-files

    <PY_FILES>

    Name of the Python program file on which the Spark application depends. Use commas (,) to separate multiple file names. The Python program file must be saved in the local path specified by localFilePath in the client.properties file in advance.

    -s,--skip-upload-resources

    <all | app | deps>

    Specifies whether to skip. Upload the JAR file, Python program file, and configuration file to OBS and load them to the resource list on the DLI server. If related resource files have been loaded to the DLI resource list, skip this step.

    If this parameter is not specified, all resource files in the command are uploaded and loaded to DLI by default.

    • all: Skips the upload and loading all resource files.
    • app: Skips the upload and loading of Spark application files.
    • deps: skips the upload and loading of all dependent files.

    -h,--help

    -

    Displays command help information.
    Command example: 
    ./spark-submit --name <name> --queue <queue_name> --class org.apache.spark.examples.SparkPi spark-examples_2.11-2.1.0.luxor.jar 10
./spark-submit --name <name> --queue <queue_name> word_count.py

    To use the DLI queue rather than the existing Spark environment, use ./spark-submit instead of spark-submit.

Parent topic: Spark Jar Jobs