Help Center/ Data Lake Insight/ Developer Guide/ Spark Jar Jobs/ Submitting a Spark Jar Job Using Livy
Updated on 2024-01-09 GMT+08:00
View PDF

Submitting a Spark Jar Job Using Livy

Introduction to DLI Livy

DLI Livy is an Apache Livy-based client tool used to submit Spark jobs to DLI.

Preparations

  • Create a queue. Set Queue Usage to For general purpose, that is, the computing resources of the Spark job.
  • Prepare a Linux ECS for installing DLI Livy.
    • Enable ports 30000 to 32767 and port 8998 on the ECS. For details, see Adding a Security Group Rule.
    • Install JDK on the ECS. JDK 1.8 is recommended. Configure Java environment variable JAVA_HOME.
    • View the ECS details to obtain its private IP address.
  • Use an enhanced datasource connection to connect the DLI queue to the VPC where the Livy instance is located.

Downloading and Installing DLI Livy

The software package used in the following operations is apache-livy-0.7.2.0103-bin.tar.gz. Replace it with the latest one.

  1. Download the DLI Livy software package.
  2. Use WinSCP to upload the obtained software package to the prepared ECS directory.
  3. Log in to ECS as user root and perform the following steps to install DLI Livy:

    1. Run the following command to create an installation directory:

      mkdir Livy installation directory

      For example, to create the /opt/livy directory, run the mkdir /opt/livy command. The following operations use the /opt/livy installation directory as an example. Replace it as required.

    2. Run the following command to decompress the software package to the installation directory:

      tar --extract --file apache-livy-0.7.2.0103-bin.tar.gz --directory /opt/livy --strip-components 1 --no-same-owner

    3. Run the following commands to change the configuration file name:

      cd /opt/livy/conf

      mv livy-client.conf.template livy-client.conf

      mv livy.conf.template livy.conf

      mv livy-env.sh.template livy-env.sh

      mv log4j.properties.template log4j.properties

      mv spark-blacklist.conf.template spark-blacklist.conf

      touch spark-defaults.conf

Modifying the DLI Livy Configuration File

  1. Upload the specified DLI Livy JAR package to the OBS bucket directory.

    1. Log in to OBS console and create a directory for storing the DLI Livy JAR package in the specified OBS bucket, for example: obs://bucket/livy/jars/.
    2. Go to the installation directory of the ECS where the DLI-Livy tool has been installed in 3.a, obtain Livy JAR packages, and upload them to the OBS bucket directory created in 1.a:
      For example, if the installation path is /opt/livy, the JAR packages you need to upload are as follows: 
      /opt/livy/rsc-jars/livy-api-0.7.2.0103.jar
/opt/livy/rsc-jars/livy-rsc-0.7.2.0103.jar
/opt/livy/repl_2.11-jars/livy-core_2.11-0.7.2.0103.jar
/opt/livy/repl_2.11-jars/livy-repl_2.11-0.7.2.0103.jar

  2. Modify the DLI Livy configuration file.

    1. Run the following command to modify the /opt/livy/conf/livy-client.conf configuration file:

      vi /opt/livy/conf/livy-client.conf

      Add the following content to the file and modify the configuration items as required: 
      # Set the private IP address of the ECS, which can be obtained by running the ifconfig command.
livy.rsc.launcher.address = X.X.X.X
# Set the ports enabled on the ECS.
livy.rsc.launcher.port.range = 30000~32767
    2. Run the following command to modify the /opt/livy/conf/livy.conf configuration file:

      vi /opt/livy/conf/livy.conf

      Add the following content to the file and modify the configuration items as required: 
      livy.server.port = 8998
livy.spark.master = yarn
 
livy.server.contextLauncher.custom.class=org.apache.livy.rsc.DliContextLauncher
livy.server.batch.custom.class=org.apache.livy.server.batch.DliBatchSession
livy.server.interactive.custom.class=org.apache.livy.server.interactive.DliInteractiveSession
livy.server.sparkApp.custom.class=org.apache.livy.utils.SparkDliApp
 
livy.server.recovery.mode = recovery
livy.server.recovery.state-store = filesystem
# Change the following file directory of DLI Livy as needed:
livy.server.recovery.state-store.url = file:///opt/livy/store/
 
livy.server.session.timeout-check = true
livy.server.session.timeout = 1800s
livy.server.session.state-retain.sec = 1800s
 
livy.dli.spark.version = 2.3.2
livy.dli.spark.scala-version = 2.11
 
# Enter the OBS bucket directory created in 1.a for storing the Livy JAR package.
livy.repl.jars = obs://bucket/livy/jars/livy-core_2.11-0.7.2.0103.jar, obs://bucket/livy/jars/livy-repl_2.11-0.7.2.0103.jar
livy.rsc.jars = obs://bucket/livy/jars/livy-api-0.7.2.0103.jar, obs://bucket/livy/jars/livy-rsc-0.7.2.0103.jar
    3. Run the following command to modify the /opt/livy/conf/spark-defaults.conf configuration file:

      vi /opt/livy/conf/spark-defaults.conf

      Add the following parameters to the file. For details about the parameter configurations, see Table 1.

      # The following parameters can be overwritten when a job is submitted.
spark.yarn.isPython=true
spark.pyspark.python=python3
 
# Enter the production environment URL of DLI.
spark.dli.user.uiBaseAddress=https://console.huaweicloud.com/dli/web
# Set the region where the queue is located.
spark.dli.user.regionName=XXXX
 
# Set the DLI endpoint address.
spark.dli.user.dliEndPoint=XXXX
 
# Enter the name of the created DLI queue.
spark.dli.user.queueName=XXXX
 
# Set the AK used for submitting a job.
spark.dli.user.access.key=XXXX
# Set the SK used for submitting a job.
spark.dli.user.secret.key=XXXX
 
# Set the project ID used for submitting a job.
spark.dli.user.projectId=XXXX
      Table 1 Mandatory parameters in spark-defaults.conf

      Parameter

      Description

      spark.dli.user.regionName

      Name of the region where the DLI queue is located.

      you can obtain the region name from .

      spark.dli.user.dliEndPoint

      Endpoint where the DLI queue is located. You can obtain the endpoint from .

      spark.dli.user.queueName

      Queue name.

      spark.dli.user.access.key

      Access key of a user. The user must have the Spark job permissions. For details, see .

      For details about how to obtain the AK/SK, see .

      spark.dli.user.secret.key

      spark.dli.user.projectId

      Project ID, which can be obtained by refering to "Obtaining a Project ID".

      The following parameters are optional. Set them based on the parameter description and site requirements. For details about these parameters, see Spark Configuration.

      Table 2 Optional parameters in spark-defaults.conf

      Spark Job Parameter

      Spark Batch Processing Parameter

      Remarks

      spark.dli.user.file

      file

      Not required for connecting to the notebook tool

      spark.dli.user.className

      class_name

      Not required for connecting to the notebook tool

      spark.dli.user.scType

      sc_type

      Same as the native Livy configuration

      spark.dli.user.args

      args

      Same as the native Livy configuration

      spark.submit.pyFiles

      python_files

      Same as the native Livy configuration

      spark.files

      files

      Same as the native Livy configuration

      spark.dli.user.modules

      modules

      -

      spark.dli.user.image

      image

      Custom image used for submitting a job. This parameter is available for container clusters only and is not set by default.

      spark.dli.user.autoRecovery

      auto_recovery

      -

      spark.dli.user.maxRetryTimes

      max_retry_times

      -

      spark.dli.user.catalogName

      catalog_name

      To access metadata, set this parameter to dli.

Starting DLI Livy

  1. Run the following command to go to the DLI Livy installation directory:

    Example: cd /opt/livy

  2. Run the following command to start DLI Livy:

    ./bin/livy-server start

Submitting a Spark Job Using DLI Livy

The following demonstrates how to submit a Spark job to DLI using DLI Livy and by running the curl command.

  1. Upload the JAR file of the developed Spark job program to the OBS directory.

    For example, upload spark-examples_2.11-XXXX.jar to the obs://bucket/path directory.

  2. Log in to the ECS server where DLI Livy is installed as user root.
  3. Run the curl command to submit a Spark job request to DLI using DLI Livy.

    ECS_IP indicates the private IP address of the ECS where DLI Livy is installed.

    curl --location --request POST 'http://ECS_IP:8998/batches' \
--header 'Content-Type: application/json' \
--data-raw '{
    "driverMemory": "3G",
    "driverCores": 1,
    "executorMemory": "2G",
    "executorCores": 1,
    "numExecutors": 1,
    "args": [
        "1000"
    ],
    "file": "obs://bucket/path/spark-examples_2.11-XXXX.jar",
    "className": "org.apache.spark.examples.SparkPi",
    "conf": {
        "spark.dynamicAllocation.minExecutors": 1,
        "spark.executor.instances": 1,
        "spark.dynamicAllocation.initialExecutors": 1,
        "spark.dynamicAllocation.maxExecutors": 2
    }
}'

Parent topic: Spark Jar Jobs