DLI Agency Overview

Updated on 2024-09-30 GMT+08:00

View PDF

What Is an Agency?

Cloud services often interact with each other, with some of which dependent on other services. You can create an agency to delegate DLI to use other cloud services and perform resource O&M on your behalf.

For example, the AK/SK required by DLI Flink jobs is stored in DEW. To allow DLI to access DEW data during job execution, you need to provide an IAM agency to delegate the permissions to perform operations on DEW data to DLI.

Figure 1 DLI service agency

DLI Agencies

Before using DLI, you are advised to set up DLI agency permissions to ensure the proper functioning of DLI.

By default, DLI provides the following agencies: dli_admin_agency, dli_management_agency, and dli_data_clean_agency. The names of these agencies are fixed, but the permissions contained in them can be customized. In other scenarios, you need to create custom agencies. For details about the agencies, see Table 1.
DLI upgrades dli_admin_agency to dli_management_agency to meet the demand for fine-grained agency permissions management. The new agency has the necessary permissions for datasource operations, notifications, and user authorization operations. For details, see Configuring DLI Agency Permissions.
To use Flink 1.15, Spark 3.3.1 (Spark general queue scenario), or a later version to execute jobs, perform the following operations:
Create an agency on the IAM console and add the agency information to the job configuration. For details, see Creating a Custom DLI Agency.
- Common scenarios for creating an agency: DLI is allowed to read and write data from and to OBS, dump logs, and read and write Flink checkpoints. DLI is allowed to access DEW to obtain data access credentials and access catalogs to obtain metadata.
- You cannot use the default agency names dli_admin_agency, dli_management_agency, or dli_data_clean_agency. It must be unique.
If the engine version is earlier than Flink 1.15, dli_admin_agency is used by default during job execution. If the engine version is earlier than Spark 3.3.1, user authentication information (AK/SK and security token) is used during job execution.
This means that jobs whose engine versions are earlier than Flink 1.15 or Spark 3.3.1 are not affected by the update of agency permissions and do not require custom agencies.
To maintain compatibility with existing job agency permission requirements, dli_admin_agency will still be listed in the IAM agency list even after the update.

NOTE:

Only the tenant account or a member account of user group admin can authorize the service.
Do not delete the agency created by the system by default.

**Table 1** DLI agencies
Permission	Type	Description
dli_admin_agency	Default agency	This agency has been discarded and is not recommended. Upgrade the agency to dli_management_agency as soon as possible. For details about how to update an agency, see Configuring DLI Agency Permissions.
dli_management_agency	Default agency	DLI system agency, which is used to delegate operation permissions to DLI so that DLI can use other cloud services and perform resource O&M operations on your behalf. This agency grants permissions for datasource operations, message notifications, and user authorization operations. For details about the permissions of an agency, see Table 2.
dli_data_clean_agency	Default agency, which needs to be authorized by users	Data cleanup agency, which is used to clean up data according to the lifecycle of a table and clean up lakehouse table data. You need to create a DLI agency named dli_data_clean_agency on IAM and grant permissions to it. You need to create an agency and customize permissions for it. However, the agency name is fixed to dli_data_clean_agency. For details about the permission policies of an agency, see Agency Permission Policies in Common Scenarios.
Other custom agencies	Custom agency	When using Flink 1.15, Spark 3.3, or a later version to execute jobs, create an agency on the IAM console and add new agency information to the job configuration. For details, see Creating a Custom DLI Agency. Common scenarios for creating an agency: DLI is allowed to read and write data from and to OBS to transfer logs. DLI is allowed to access DEW to obtain data access credentials and access catalogs to obtain metadata. You cannot use the default agency names dli_admin_agency, dli_management_agency, or dli_data_clean_agency. It must be unique. For details about the permission policies of an agency, see Agency Permission Policies in Common Scenarios.

**Table 2** Permissions contained in the dli_management_agency agency
Policy	Description
IAM ReadOnlyAccess	To authorize IAM users who have not logged in to DLI, you need to obtain their information. So, the permissions contained in the IAM ReadOnlyAccess policy are required.
DLI Datasource Connections Agency Access	Permissions to access and use VPCs, subnets, routes, and VPC peering connections
DLI Notification Agency Access	Permissions to send notifications through SMN when a job fails to be executed

Parent topic: Configuring an Agency to Allow DLI to Access Other Cloud Services

Previous topic: Configuring an Agency to Allow DLI to Access Other Cloud Services

Next topic: Creating a Custom DLI Agency

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot