Configuring Dependency Management

Updated on 2024-11-29 GMT+08:00

View PDF

Flink allows you to run user-defined Flink jobs using third-party dependency packages. You can upload and manage dependency JAR packages on the Flink web UI and invoke required dependencies when running jobs. The dependency management function does not support semantic verification. A dependency JAR package name must start with a letter, digit, or underscore (_) and cannot exceed 32 characters. The following third-party dependencies are supported:

Custom connector dependency: After a custom connector JAR package is uploaded, Dependency Type is connector on the Flink web UI.
Non-custom connector dependency: After a non-custom connector JAR package, such as a job dependency package, is uploaded, Dependency Type is normal on the Flink web UI.

Prerequisites

Prepare dependency files. If you upload dependencies to the cluster by specifying a path, you need to create an HDFS path and upload the JAR package to HDFS.

Uploading Dependency Packages

Log in to FusionInsight Manager and access the Flink web UI. For details, see Accessing the Flink Web UI.
Click Dependency Management. The Dependency Management page is displayed.

Click Add Dependency.

**Table 1** Adding a dependency
Parameter	Description	Example
Customized connector or not	Whether the dependency is a custom connector. Set this parameter based on the site requirements. Yes: The file is a custom connector dependency. No: The file is a non-custom connector dependency.	Yes
Name	Name of the dependency, which must be the same as the connection name of connector in the uploaded dependency package Dependency packages with the same name cannot be uploaded.	kafka
Register jar	Upload method of a JAR package: Upload File: Upload a JAR package form the local host. Specify Path: Upload a prepared dependency file from an HDFS path.	Upload File
Upload File	If Register jar is set to Upload File, select a JAR file from the local host.	-
Specify Path	If Register jar is set to Specify Path, enter the HDFS path of the dependency file. (The JAR package must have been uploaded to the HDFS.)	/flink_upload_test/flink-connector-kafka-customization.jar
Description	Description of the dependency to be uploaded.	-

Click OK

Example

Custom connector dependencies

Upload a custom connector dependency by referring to Uploading Dependency Packages.
For example, the dependency name is kafka, and the name of the custom connector JAR package is flink-connector-kafka-customization.jar.

Create a SQL job by referring to Creating a Job. Set connector in the SQL statement to the dependency name, for example, 'connector'='kafka'.

CREATE TABLE KafkaSinkTable (`user_id` INT, `name` VARCHAR) WITH (
  'connector' = 'kafka',
  'topic' = 'test_sink6',
  'properties.bootstrap.servers' = '192.168.20.134:21005',
  'properties.group.id' = 'testGroup',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'csv'
);
CREATE TABLE datagen (`user_id` INT, `name` VARCHAR) WITH (
  'connector' = 'datagen',
  'rows-per-second' = '5',
  'fields.user_id.kind' = 'sequence',
  'fields.user_id.start' = '1',
  'fields.user_id.end' = '1000'
);
insert INTO
  KafkaSinkTable
select
  *
from
  datagen;

Non-Custom connector dependencies
Upload a non-custom connector dependency for a job. For details, see Uploading Dependency Packages.

Parent topic: Configuring and Developing a Flink Visualization Job

Previous topic: Restoring a Job

Next topic: Configuring and Managing UDFs

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot