このページは、お客様の言語ではご利用いただけません。Huawei Cloudは、より多くの言語バージョンを追加するために懸命に取り組んでいます。ご協力ありがとうございました。

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Creating a Flink Jar Job

Updated on 2022-12-07 GMT+08:00

This section describes how to create a Flink Jar job. You can perform secondary development based on Flink APIs, build your own JAR file, and submit the JAR file to DLI queues. DLI is fully compatible with open-source community APIs. To create a custom Flink job, you need to compile and build application JAR files. You must have a certain understanding of Flink secondary development and have high requirements related to stream computing complexity.

Prerequisites

  • Ensure that a dedicated queue has been created.
  • When creating a Flink Jar job to access other external data sources, such as OpenTSDB, HBase, Kafka, DWS, RDS, CSS, CloudTable, DCS Redis, and DDS MongoDB, you need to create a cross-source connection to connect the job running queue to the external data source.
  • When running a Flink Jar job, you need to build the secondary development application code into a Jar package and upload the JAR package to the created OBS bucket. Choose Data Management > Package Management to create a package. For details, see Creating a Package.
    NOTE:

    DLI does not support the download function. If you need to modify the uploaded data file, please edit the local file and upload it again.

  • Flink dependencies have been built in the DLI server and security hardening has been performed based on the open-source community version. To avoid dependency package compatibility issues or log output and dump issues, be careful to exclude the following files when packaging:
    • Built-in dependencies (or set the package dependency scope to provided in Maven or SBT)
    • Log configuration files (example, log4j.properties/logback.xml)
    • JAR package for log output implementation (example, log4j).

Creating a Flink Jar Job

  1. In the left navigation pane of the DLI management console, choose Job Management > Flink Jobs. The Flink Jobs page is displayed.
  2. In the upper right corner of the Flink Jobs page, click Create Job.
  3. Specify job parameters.

    Table 1 Job configuration information

    Parameter

    Description

    Type

    Select Flink Jar.

    Name

    Name of a job. Enter 1 to 57 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.

    NOTE:

    The job name must be globally unique.

    Description

    Description of a job. It can be up to 512 characters long.

  4. Click OK to enter the page for job information editing.
  5. Select a queue. Flink Jar jobs can run only on general queues.

    NOTE:
    • A Flink Jar job can run only on a pre-created dedicated queue.
    • If no dedicated queue is available in the Queue drop-down list, create a dedicated queue and bind it to the current user.

  6. Configure Flink Jar job parameters.

    Table 2 Parameter description

    Name

    Description

    Application

    User-defined package. Before selecting a JAR file to be inserted, upload the corresponding JAR file to the OBS bucket and choose Data Management > Package Management to create a package. For details, see Creating a Package. For details about the built-in dependencies, see Flink 1.7.2 Dependencies and Flink 1.10 Dependencies.

    Main Class

    The name of the JAR package to be loaded, for example, KafkaMessageStreaming.

    • Default: Specified based on the Manifest file in the JAR package.
    • Manually assign: You must enter the class name and confirm the class arguments (separate arguments with spaces).
    NOTE:

    When a class belongs to a package, the main class path must contain the complete package path, for example, packagePath.KafkaMessageStreaming.

    Class Arguments

    List of arguments of a specified class. The arguments are separated by spaces.

    Flink parameters support replacement of non-sensitive global variables. For example, if you add the global variable windowsize in Global Configuration > Global Variables, you can add the -windowsSize {{windowsize}} parameter for the Flink Jar job.

    JAR Package Dependencies

    Select a user-defined package dependency. The dependent program packages are stored in the classpath directory of the cluster.

    Before selecting a JAR file to be inserted, upload the corresponding JAR file to the OBS bucket and choose Data Management > Package Management to create a package. Select JAR as the package type. For details, see Creating a Package.

    For details about the built-in dependencies, see Flink 1.7.2 Dependencies and Flink 1.10 Dependencies.

    Other Dependencies

    User-defined dependency files. Other dependency files need to be referenced in the code.

    Before selecting a dependency file, upload the file to the OBS bucket and choose Data Management > Package Management to create a package. The package type is not limited. For details, see Creating a Package.

    You can add the following command to the application to access the corresponding dependency file. In the command, fileName indicates the name of the file to be accessed, and ClassName indicates the name of the class that needs to access the file.

    ClassName.class.getClassLoader().getResource("userData/fileName")

    Flink Version

    Before selecting a Flink version, you need to select the queue to which the Flink version belongs. Currently, the following versions are supported: 1.10.

    Runtime Configuration

    User-defined optimization parameters. The parameter format is key=value.

    Flink optimization parameters support replacement non-sensitive global variable. For example, if you create global variable phase in Global Configuration > Global Variables, optimization parameter table.optimizer.agg-phase.strategy={{phase}} can be added to the Flink Jar job.

  7. Configure job parameters.

    Table 3 Parameter description

    Name

    Description

    CUs

    One CU has one vCPU and 4-GB memory. The number of CUs ranges from 2 to 400.

    Job Manager CUs

    Set the number of CUs on a management unit. The value ranges from 1 to 4. The default value is 1.

    Parallelism

    Maximum number of parallel operators in a job.

    NOTE:
    • The value must be less than or equal to four times the number of compute units (CUs minus the number of job manager CUs).
    • You are advised to set this parameter to a value greater than that configured in the code. Otherwise, job submission may fail.

    Task Manager Configuration

    Whether to set Task Manager resource parameters

    If this option is selected, you need to set the following parameters:

    • CU(s) per TM: Number of resources occupied by each Task Manager.
    • Slot(s) per TM: Number of slots contained in each Task Manager.

    Save Job Log

    Whether to save the job running logs to the OBS bucket.

    CAUTION:

    You are advised to select this parameter. Otherwise, no run log is generated after the job is executed. If the job is abnormal, the run log cannot be obtained for fault locating.

    If this option is selected, you need to set the following parameters:

    OBS Bucket: Select an OBS bucket to store job logs. If the selected OBS bucket is not authorized, click Authorize.

    Alarm Generation upon Job Exception

    Whether to report job exceptions, for example, abnormal job running or exceptions due to an insufficient balance, to users via SMS or email

    If this option is selected, you need to set the following parameters:

    SMN Topic

    Select a user-defined SMN topic. For details about how to customize SMN topics, see Creating a Topic in the Simple Message Notification User Guide.

    Auto Restart upon Exception

    Whether to enable automatic restart. If this function is enabled, any job that has become abnormal will be automatically restarted.

    If this option is selected, you need to set the following parameters:

    • Max. Retry Attempts: maximum number of retry times upon an exception. The unit is Times/hour.
      • Unlimited: The number of retries is unlimited.
      • Limited: The number of retries is user-defined.
    • Restore Job from Checkpoint: Restore the job from the saved checkpoint.

      If you select this parameter, you also need to set Checkpoint Path.

      Checkpoint Path: Select the checkpoint saving path. The checkpoint path must be the same as that you set in the application package. Note that the checkpoint path for each job must be unique. Otherwise, the checkpoint cannot be obtained.

  8. Click Save on the upper right of the page.
  9. Click Start on the upper right side of the page. On the displayed Start Flink Job page, confirm the job specifications, and click Start Now to start the job.

    After the job is started, the system automatically switches to the Flink Jobs page, and the created job is displayed in the job list. You can view the job status in the Status column. After a job is successfully submitted, the job status will change from Submitting to Running. After the execution is complete, the message Completed is displayed.

    If the job status is Submission failed or Running exception, the job submission failed or the job did not execute successfully. In this case, you can move the cursor over the status icon in the Status column of the job list to view the error details. You can click to copy these details. After handling the fault based on the provided information, resubmit the job.

    NOTE:

    Other available buttons are as follows:

    Save As: Save the created job as a new job.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback