Overview

When submitting Flink or Spark jobs through DLI to access external data sources (such as OBS and Kafka), there is a risk of plaintext exposure if AK/SK, usernames/passwords are directly embedded in the job code or parameter configurations.

To securely store data source access credentials, ensure data source authentication safety, and facilitate secure access to data sources by DLI, you are advised to use DEW for managing data source access credentials. DLI employs "agency + temporary credentials" to safely retrieve data source access credentials.

DEW is a comprehensive cloud-based encryption service designed to address challenges related to data security, key security, and the complexities of key management.

This section describes how to use DEW to store data source authentication information across various job types.

For details, see DEW.

Notes and Constraints

You are advised to use DEW for storing data source authentication information exclusively when Spark 3.3.1 or later and Flink 1.15 or later jobs access data sources using datasoure connections.

When SQL and Flink 1.12 jobs access data sources using datasource connections, use DLI's datasource authentication feature to manage data source access credentials. For details, see Using DLI Datasource Authentication to Manage Access Credentials for Data Sources.

Methods of Using DEW to Manage Data Source Access Credentials for Different Types of Jobs

**Table 1** Methods of using DEW to manage data source access credentials for different types of jobs
Job Type	Helpful Link	Description
Flink OpenSource SQL job	Flink OpenSource SQL Jobs Using DEW to Manage Access Credentials	Instructions on using DEW to manage access credentials for Flink OpenSource SQL jobs, along with instructions for setting properties such as account and password in connectors.
Flink Jar job	Flink Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS	Instructions on using DEW to acquire AK/SK for reading and writing data from and to OBS in Flink Jar jobs.
Flink Jar job	Obtaining Temporary Credentials from a Flink Job's Agency for Accessing Other Cloud Services	DLI provides a common interface to obtain temporary credentials for Flink job agencies set by users during job launch. The interface encapsulates the obtained temporary credentials for the job agency in the com.huaweicloud.sdk.core.auth.BasicCredentials class. Instructions on obtaining temporary credentials for Flink job agencies.
Spark Jar job	Spark Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS	Instructions on using DEW to acquire AK/SK for reading and writing data from and to OBS in Spark Jar jobs.
Spark Jar job	Obtaining Temporary Credentials from a Spark Job's Agency for Accessing Other Cloud Services	Instructions on obtaining temporary credentials for Spark Jar job agencies.