Updating the Permissions of a DLI Agency

Notes and Constraints

• Only the tenant account or a member account of user group admin can authorize the service.
• DLI authorization needs to be conducted by project. The permissions of required agencies must be updated separately in each project. This means you need to switch to the corresponding project and then update the agency by following the instructions provided in this section.
• Among the permissions contained in dli_management_agency:
  • The authorization scope of the IAM ReadOnlyAccess policy covers all global service resources in all regions.
    • If you select this policy when updating a DLI agency in any region, this policy's permissions apply to the projects in all regions.
    • If you do not select this policy when updating an agency in any project, this policy's permissions will be revoked from all regions. This means that all projects cannot obtain IAM user information.
      

      When updating agency permissions, you will need to select the IAM ReadOnlyAccess policy, as its authorization scope covers global service resources. If you deselect it and update the agency, all its permissions will become invalid in all regions and projects.

  • The authorization scope of the DLI Datasource Connections Agency Access and DLI Notification Agency Access policies covers the project resources in specified regions.

    These policies' permissions only apply to projects for which these policies are selected and the DLI agency permissions are updated. Projects for which these policies are not selected do not have the permissions required in datasource scenarios and the permission to send notifications using SMN.

  Example 1 and Example 2 demonstrate the agency permission differences caused by updating a DLI agency for different projects in a region.

Updating DLI Agency Permissions (dli_management_agency)

This part describes how to update dli_management_agency. If you still use dli_admin_agency, refer to the following:

1. In the navigation pane of the DLI console, choose Global Configuration > Service Authorization.
2. On the displayed page, select permissions for scenarios.

   Click on a permission card to view its detailed permission policies.

   Table 1 describes these agencies.

   Table 1 Permissions contained in the dli_management_agency agency

   Use Case	Agency	Description
   Basic usage	IAM ReadOnlyAccess	To authorize IAM users who have not logged in to DLI, you need to obtain their information. So, the permissions contained in the IAM ReadOnlyAccess policy are required. IAM ReadOnlyAccess is a global policy. Make sure you select this policy. If you do not select it, all its permissions will become invalid in all regions, and the system cannot obtain IAM user information.
   Datasource	DLI Datasource Connections Agency Access	Permissions to access and use VPCs, subnets, routes, and VPC peering connections
   O&M	DLI Notification Agency Access	Permissions to send notifications through SMN when a job fails to be executed

3. Select the policies to be included in dli_management_agency and click Update.
   Figure 1 Updating agency permissions
   
4. View and understand the notes for updating the agency, and click OK. The DLI agency permissions are updated.
   • Once the agency permissions are updated, your dli_admin_agency will be upgraded to dli_management_agency. This new agency will have the necessary permissions for datasource operations, notifications, and user authorization operations.
   • To use Flink 1.15, Spark 3.3.1 (Spark general queue scenario), or a later version to execute jobs, perform the following operations:
     Create an agency on the IAM console and add the agency information to the job configuration. For details, see Customizing DLI Agency Permissions.

     • Common scenarios for creating an agency: DLI is allowed to read and write data from and to OBS, dump logs, and read and write Flink checkpoints. DLI is allowed to access DEW to obtain data access credentials and access catalogs to obtain metadata.
     • You cannot use the default agency names dli_admin_agency, dli_management_agency, or dli_data_clean_agency. It must be unique.
   • If the engine version is earlier than Flink 1.15, dli_admin_agency is used by default during job execution. If the engine version is earlier than Spark 3.3.1, user authentication information (AK/SK and security token) is used during job execution.
   • To maintain compatibility with existing job agency permission requirements, dli_admin_agency will still be listed in the IAM agency list even after the update.
   • Do not delete the agency created by the system by default.

Follow-Up Operations

In addition to the permissions provided by dli_management_agency, you need to create an agency on the IAM console and add information about the new agency to the job configuration for scenarios like allowing DLI to read and write data from and to OBS to transfer logs, or allowing DLI to access DEW to obtain data access credentials. For details, see Customizing DLI Agency Permissions and Agency Permission Policies in Common Scenarios.

• When Flink 1.15, Spark 3.3.1 (Spark general queue scenario), or a later version is used to execute jobs, you need to create an agency on the IAM console.
• If the engine version is earlier than Flink 1.15, dli_admin_agency is used by default during job execution. If the engine version is earlier than Spark 3.3.1, user authentication information (AK/SK and security token) is used during job execution.

  This means that jobs whose engine versions are earlier than Flink 1.15 or Spark 3.3.1 are not affected by the update of agency permissions and do not require custom agencies.

Common service scenarios where you need to create an agency:

• Data cleanup agency required for clearing data according to the lifecycle of a table and clearing lakehouse table data. You need to create a DLI agency named dli_data_clean_agency on IAM and grant permissions to it. You need to create an agency and customize permissions for it. However, the agency name is fixed to dli_data_clean_agency.
• Tenant Administrator permissions are required to access data from OBS to execute Flink jobs on DLI, for example, obtaining OBS data sources, log dump (including bucket authorization), checkpointing enabling, and job import and export.
• The AK/SK required by DLI Flink jobs is stored in DEW. To allow DLI to access DEW data during job execution, you need to create an agency to delegate the permissions to operate on DEW data to DLI.
• To allow DLI to access DLI catalogs to retrieve metadata when executing jobs, you need to create a new agency that grants DLI catalog data operation permissions to DLI. This will enable DLI to access DLI catalogs on your behalf.
• Cloud data required by DLI Flink jobs is stored in LakeFormation. To allow DLI to access catalogs to retrieve metadata during job execution, you need to create an agency to delegate the permissions to operate on catalog data to DLI.

When creating an agency, you cannot use the default agency names dli_admin_agency, dli_management_agency, or dli_data_clean_agency. It must be unique.

For more information about custom agency operations, see Customizing DLI Agency Permissions and Agency Permission Policies in Common Scenarios.

Example 1

• Operation instruction: A DLI user upgrades dli_admin_agency to dli_management_agency for project A in CN North-Beijing4.
  1. On the DLI management console, switch to project A in the CN North-Beijing4 region and choose Global Configuration > Service Authorization.
  2. Select the policies under Basic Usage, Datasource, and O&M.
     Figure 2 Updating a DLI agency for project A in CN North-Beijing4
     
  3. Click Update.
• Permission description: The agency permissions are updated for project A in CN North-Beijing4.
  • The IAM ReadOnlyAccess policy's permissions are granted to global service resources, meaning that all regions and projects have these permissions.
  • The DLI Datasource Connections Agency Access and DLI Notification Agency Access policies contain only regional permissions, meaning that their permissions only apply to project A in CN North-Beijing4.

    Example of Agency Permissions for Project A in CN North-Beijing4	Example of Agency Permissions for Project B in CN North-Beijing4
    dli_management_agency The new agency contains the following policy permissions: IAM ReadOnlyAccess DLI Datasource Connections Agency Access DLI Notification Agency Access	dli_management_agency The new agency contains the following policy permissions: IAM ReadOnlyAccess

Example 2

Operation instruction: To assign the permissions of the DLI Datasource Connections Agency Access and DLI Notification Agency Access policies to project B in the CN North-Beijing4 region, perform the following operations to update the agency permissions of project B in CN North-Beijing4:

1. On the DLI management console, switch to project B in the CN North-Beijing4 region and choose Global Configuration > Service Authorization.
2. Select the policies under Basic Usage, Datasource, and O&M.
   

   When updating the agency permissions for project B, you will need to select the IAM ReadOnlyAccess policy, as its authorization scope covers global service resources. If you deselect it and update the agency, all its permissions will become invalid in all regions and projects.

   Figure 3 Updating a DLI agency for project B in CN North-Beijing4
   
3. Click Update.

Permission description: