Esta página ainda não está disponível no idioma selecionado. Estamos trabalhando para adicionar mais opções de idiomas. Agradecemos sua compreensão.

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Creating and Submitting a Spark Jar Job

Updated on 2022-12-07 GMT+08:00

You can use DLI to submit Spark jobs for real-time computing. The general procedure is as follows:

Step 1: Logging in to the Cloud

Step 2: Uploading Data to OBS

Step 3: Logging In to the DLI Management Console

Step 4: Creating a Queue

Step 5: Creating a Package

Step 6: Submitting a Spark Job

Step 1: Logging in to the Cloud

To use DLI, you need to log in to the cloud.

  1. Open the DLI home page.
  2. On the login page, enter the username and password, and click Log In.

Step 2: Uploading Data to OBS

Write a Spark Jar job program , and compile and package it as spark-examples.jar. Perform the following steps to submit the job:

Before submitting Spark Jar jobs, upload data files to OBS.

  1. In the services displayed, click Object Storage Service (OBS) in Storage.
  2. The OBS console page is displayed.
  3. Create a bucket. The bucket name must be globally unique. In this example, assume that the bucket name is dli-test-obs01.
    1. Click Create Bucket.
    2. On the displayed Create Bucket page, enter the Bucket Name. Retain the default values for other parameters or set them as required.
    3. Click Create Now.
  4. Click dli-test-obs01 to switch to the Overview page.
  5. From the left navigation tree, click Objects. Click Upload Object. In the displayed dialog box, drag files or folders to upload or add file to the file upload box, for example, spark-examples.jar. Then, click Upload.

    After the file is uploaded successfully, the file path is obs://dli-test-obs01/spark-examples.jar.

    For more information about OBS operations, see the Object Storage Service Console Operation Guide.

    For more information about the tool, see the OBS Tool Guide.

    NOTE:

    You are advised to use an OBS tool, such as OBS Browser+, to upload large files because OBS Console has restrictions on the file size and quantity.

    • OBS Browser+ is a graphical tool that provides complete functions for managing your buckets and objects in OBS. You are advised to use this tool to create buckets or upload objects.

Step 3: Logging In to the DLI Management Console

To submit Spark jobs, you need to enter the Spark job creation page first.

  1. In the list, choose .
  2. The DLI management console page is displayed.
    NOTE:

    If you log in to the DLI management console for the first time, you need to be authorized to access OBS. For details, see "Service Authorization".

Step 4: Creating a Queue

If you submit a Spark job for the first time, you need to create a queue first. For example, create a queue named sparktest and set Queue Type to General Queue.

  1. In the navigation pane of the DLI management console, choose Queue Management.
  2. In the upper right corner of the Queue Management page, click Create Queue to create a queue.
  3. Create a queue, name it sparktest, and set the queue usage to for general purpose. For details, see Creating a Queue.

  4. Click Create Now to create a queue.

Step 5: Creating a Package

Before submitting a Spark job, you need to create a package, for example, spark-examples.jar.

  1. In the navigation pane on the left of the DLI console, choose Data Management > Package Management.
  2. On the Package Management page, click Create in the upper right corner to create a package.
  3. In the Create Package dialog box, set Type to JAR, OBS Path to the path of the spark-examples.jar package in Step 2: Uploading Data to OBS, and Group to Do not use.

  4. Click OK.

    You can view and select the package on the Package Management page.

For details about how to create a package, see "Creating a Package".

Step 6: Submitting a Spark Job

  1. On the DLI management console, choose Job Management > Spark Jobs in the navigation pane on the left. On the displayed page, click Create Job.
  2. On the Spark job editing page, set Queues to the queue created in Step 4: Creating a Queue and Application to the package created in Step 5: Creating a Package.

    For details about other parameters, see the description of the Spark job editing page in "Creating a Spark Job".

  3. Click Execute in the upper right corner of the Spark job editing window, read and agree to the privacy agreement, and click OK. Submit the job. A message is displayed, indicating that the job is submitted.

  4. (Optional) Switch to the Job Management > Spark Jobs page to view the status and logs of the submitted Spark job.
    NOTE:

    When you click Execute on the DLI management console for the first time, you need to read the privacy agreement. Once agreed to the agreement, you will not receive any privacy agreement messages for subsequent operations.

Usamos cookies para aprimorar nosso site e sua experiência. Ao continuar a navegar em nosso site, você aceita nossa política de cookies. Saiba mais

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback