使用ma-cli dli-job submit命令提交DLI Spark作业

执行ma-cli dli-job submit命令提交DLI Spark作业。

ma-cli dli-job submit命令需要指定一个位置参数YAML_FILE表示作业的配置文件路径，如果不指定该参数，则表示配置文件为空。配置文件是一个YAML格式的文件，里面的参数就是命令的option参数。此外，如果用户在命令行中同时指定YAML_FILE配置文件和option参数，命令行中指定的option参数的值将会覆盖配置文件相同的值。

命令参数预览

ma-cli dli-job submit -h
Usage: ma-cli dli-job submit [OPTIONS] [YAML_FILE]...

  Submit DLI Spark job.

  Example:

  ma-cli dli-job submit  --name test-spark-from-sdk
                          --file test/sub_dli_task.py
                          --obs-bucket dli-bucket
                          --queue dli_test
                          --spark-version 2.4.5
                          --driver-cores 1
                          --driver-memory 1G
                          --executor-cores 1
                          --executor-memory 1G
                          --num-executors 1

Options:
  --file TEXT                    Python file or app jar.
  -cn, --class-name TEXT         Your application's main class (for Java / Scala apps).
  --name TEXT                    Job name.
  --image TEXT                   Full swr custom image path.
  --queue TEXT                   Execute queue name.
  -obs, --obs-bucket TEXT        DLI obs bucket to save logs.
  -sv, --spark-version TEXT      Spark version.
  -st, --sc-type [A|B|C]         Compute resource type.
  --feature [basic|custom|ai]    Type of the Spark image used by a job (default: basic).
  -ec, --executor-cores INTEGER  Executor cores.
  -em, --executor-memory TEXT    Executor memory (eg. 2G/2048MB).
  -ne, --num-executors INTEGER   Executor number.
  -dc, --driver-cores INTEGER    Driver cores.
  -dm, --driver-memory TEXT      Driver memory (eg. 2G/2048MB).
  --conf TEXT                    Arbitrary Spark configuration property (eg. <PROP=VALUE>).
  --resources TEXT               Resources package path.
  --files TEXT                   Files to be placed in the working directory of each executor.
  --jars TEXT                    Jars to include on the driver and executor class paths.
  -pf, --py-files TEXT           Python files to place on the PYTHONPATH for Python apps.
  --groups TEXT                  User group resources.
  --args TEXT                    Spark batch job parameter args.
  -q, --quiet                    Exit without waiting after submit successfully.
  -C, --config-file PATH         Configure file path for authorization.
  -D, --debug                    Debug Mode. Shows full stack trace when error occurs.
  -P, --profile TEXT             CLI connection profile to use. The default profile is "DEFAULT".
  -H, -h, --help                 Show this message and exit.

yaml文件预览

# dli-demo.yaml
name: test-spark-from-sdk
file: test/sub_dli_task.py
obs-bucket: ${your_bucket}
queue: dli_notebook 
spark-version: 2.4.5
driver-cores: 1
driver-memory: 1G
executor-cores: 1
executor-memory: 1G
num-executors: 1

## [Optional] 
jars:
  - ./test.jar
  - obs://your-bucket/jars/test.jar
  - your_group/test.jar

## [Optional] 
files:
  - ./test.csv
  - obs://your-bucket/files/test.csv
  - your_group/test.csv

## [Optional] 
python-files:
  - ./test.py
  - obs://your-bucket/files/test.py
  - your_group/test.py

## [Optional] 
resources:
  - name: your_group/test.py
    type: pyFile
  - name: your_group/test.csv
    type: file
  - name: your_group/test.jar
    type: jar
  - name: ./test.py
    type: pyFile
  - name: obs://your-bucket/files/test.py
    type: pyFile

## [Optional]
groups:
  - group1
  - group2

指定options参数提交DLI Spark作业示例：

$ ma-cli dli-job submit --name test-spark-from-sdk \
                        --file test/sub_dli_task.py \
                        --obs-bucket ${your_bucket} \
                        --queue dli_test \
                        --spark-version 2.4.5 \
                        --driver-cores 1 \
                        --driver-memory 1G \
                        --executor-cores 1 \
                        --executor-memory 1G \
                        --num-executors 1