Overview

When submitting Flink or Spark jobs through DLI to access external data sources (such as OBS and Kafka), there is a risk of plaintext exposure if AK/SK, usernames/passwords are directly embedded in the job code or parameter configurations.

To securely store access credentials for data sources, ensure the safety of data source authentication, and facilitate secure access by DLI, you are advised to use DEW to manage these credentials. This approach allows DLI to safely retrieve access credentials through an "agency + temporary credential" mechanism.

DEW is designed to address critical challenges such as data security, key security, and the complexities of key management.

This section describes how to use DEW to store data source authentication information across various job types.

Learn about DEW.

Notes and Constraints

You are advised to use DEW for storing data source authentication information exclusively when Spark 3.3.1 or later and Flink 1.15 or later jobs access data sources using datasource connections.

When SQL and Flink 1.12 jobs access data sources using datasource connections, use DLI's datasource authentication feature to manage data source access credentials. For details, see Using DLI Datasource Authentication to Manage Access Credentials for Data Sources.

Methods of Using DEW to Manage Data Source Access Credentials for Different Types of Jobs

**Table 1** Methods of using DEW to manage data source access credentials for different types of jobs
Job Type	Helpful Link	Description
Flink OpenSource SQL job	Flink OpenSource SQL Jobs Using DEW to Manage Access Credentials	Instructions on using DEW to manage access credentials for Flink OpenSource SQL jobs, along with instructions for setting properties such as account and password in connectors.
Flink Jar job	Flink Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS	Instructions on using DEW to acquire AK/SK for reading and writing data from and to OBS in Flink Jar jobs.
Flink Jar job	Obtaining Temporary Credentials from a Flink Job's Agency for Accessing Other Cloud Services	DLI provides a common interface to obtain temporary credentials for Flink job agencies set by users during job launch. The interface encapsulates the obtained temporary credentials for the job agency in the com.huaweicloud.sdk.core.auth.BasicCredentials class. Instructions on obtaining temporary credentials for Flink job agencies.
Spark Jar job	Spark Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS	Instructions on using DEW to acquire AK/SK for reading and writing data from and to OBS in Spark Jar jobs.
Spark Jar job	Obtaining Temporary Credentials from a Spark Job's Agency for Accessing Other Cloud Services	Instructions on obtaining temporary credentials for Spark Jar job agencies.