Using an Agency to Obtain Access Credentials in DLI
When you use DLI to perform big data processing or cross-service data queries, authentication is a critical prerequisite to ensure legitimate access and data security. Embedding AK/SK or usernames and passwords directly in job code or configuration files introduces the risk of plaintext credential leakage.
To address diverse access scenarios and authentication protocol requirements, DLI provides two core authentication solutions.
- (Recommended) DLI Agency with Temporary Credentials
Temporary security credentials include a temporary AK/SK and a security token. These credentials are dynamically generated, short-lived, automatically expire, and do not require manual management. DLI automatically obtains temporary credentials through agencies and uses them to access services. Once the credentials expire, they become invalid automatically. This approach significantly reduces the risk of key leakage and eliminates key maintenance overhead. It is the preferred solution for large-scale automated processing and high-security workloads.
- DLI Agency with DEW Temporary Credentials and Permanent AK/SK
This solution is suitable for services that support only fine-grained authorization (version = 1.1) or workloads that require highly stable credentials without frequent rotation. It provides reliable authentication for these scenarios.
In this approach, DLI uses agencies and temporary credentials to access the DEW service. DEW then provides permanent AK/SK, which are used to securely access other cloud services.
Related guidance:
Use Cases
- Address security risks caused by hard-coded credentials.
- Avoid embedding sensitive information, such as data source usernames and passwords, in job code.
- Enable dynamic credential acquisition and periodic credential rotation.
Notes and Constraints
You are advised to use DEW for storing data source authentication information exclusively when Spark 3.3.1 or later and Flink 1.15 or later jobs access data sources using datasource connections.
When SQL and Flink 1.12 jobs access data sources using datasource connections, use DLI's datasource authentication feature to manage data source access credentials. For details, see Overview.
Methods of Using DEW to Manage Data Source Access Credentials for Different Types of Jobs
| Job Type | Helpful Link | Description |
|---|---|---|
| Flink OpenSource SQL job | Flink OpenSource SQL Jobs Using DEW to Manage Access Credentials | Instructions on using DEW to manage access credentials for Flink OpenSource SQL jobs, along with instructions for setting properties such as account and password in connectors. |
| Flink Jar job | Flink Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS | Instructions on using DEW to acquire AK/SK for reading and writing data from and to OBS in Flink Jar jobs. |
| Obtaining Temporary Credentials from a Flink Job's Agency for Accessing Other Cloud Services | DLI provides a common interface to obtain temporary credentials for Flink job agencies set by users during job launch. The interface encapsulates the obtained temporary credentials for the job agency in the com.huaweicloud.sdk.core.auth.BasicCredentials class. Instructions on obtaining temporary credentials for Flink job agencies. | |
| Spark Jar job | Spark Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS | Instructions on using DEW to acquire AK/SK for reading and writing data from and to OBS in Spark Jar jobs. |
| Obtaining Temporary Credentials from a Spark Job's Agency for Accessing Other Cloud Services | Instructions on obtaining temporary credentials for Spark Jar job agencies. |
Learn More: What Are Temporary Security Credentials?
A temporary security credential grants temporary access rights. It includes a temporary AK/SK and a security token, both of which must be used together.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
