Managing LakeFormation Data Sources

Application Scenarios

Compared with managing foreign table metadata directly in DWS, LakeFormation is more suitable for managing metadata in the following scenarios:

• Scenario 1: An enterprise already has a lakehouse data catalog created using LakeFormation and wants DWS to directly reuse the existing metadata to avoid repeated maintenance.
• Scenario 2: Multiple compute engines (such as DWS, Spark, and Flink) share the same set of metadata. LakeFormation is used for unified metadata management.
• Scenario 3: A unified permissions control system is required. LakeFormation is used to centrally manage data access permissions.

Notes and Constraints

• This feature is restricted for commercial use (contact technical support). It is available only for the following cluster versions:
  • Storage-compute decoupled clusters running version 9.0.1 or later.
  • Storage-compute coupled clusters running version 8.2.1.300 or later.
• After a LakeFormation data source is deleted, the foreign tables that depend on the data source cannot be used. The operation is irreversible. Exercise caution when performing this operation.
• During the update of data source configurations, related queries may fail temporarily. Perform the update during off-peak hours.

Prerequisites

• A LakeFormation instance is available. For details, see Creating a LakeFormation Instance.
• Access the LakeFormation console, click Clients, and create a client. For details, see Managing Clients.
• Create an agency with LakeFormation permissions (including the minimum permissions). For details, see Creating a LakeFormation Agency. If you do not create an agency, an error will be reported.
• For an IAM user to use DWS to call APIs on the LakeFormation management plane, the user must have LakeFormation permissions (at least lakeformation:instance:access, lakeformation:instance:describe, and lakeformation:credential:describe).

Creating a LakeFormation Agency

Before creating a LakeFormation data source, create an agency that grants DWS the LakeFormation FullAccess and LakeFormationObsAccess permissions.

1. Click your account in the upper right corner of the page and choose Identity and Access Management.
2. In the navigation pane on the left, choose Permissions > Policies/Roles. In the upper right corner of the displayed page, click Create Custom Policy.
3. Create the LakeFormationObsAccess policy (skip this step if the same policy already exists). Set the parameters as follows and click OK.
   • Policy Name: Enter LakeFormationObsAccess.
   • Policy View: Select JSON.
   • Policy Content: Enter the following content:

     {
         "Version": "1.1",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "obs:bucket:ListAllMyBuckets"
                 ]
             },
             {
                 "Effect": "Allow",
                 "Action": [
                     "obs:bucket:GetLifecycleConfiguration",
                     "obs:bucket:GetBucketLocation",
                     "obs:bucket:PutLifecycleConfiguration",
                     "obs:bucket:PutBucketAcl",
                     "obs:bucket:HeadBucket",
                     "obs:bucket:GetBucketAcl",
                     "obs:bucket:ListBucket"
                 ],
                 "Resource": [
                     "OBS:*:*:bucket:*"
                 ]
             },
             {
                 "Effect": "Allow",
                 "Action": [
                     "obs:object:GetAccessLabel",
                     "obs:object:PutObjectVersionAcl",
                     "obs:object:DeleteAccessLabel",
                     "obs:object:PutObjectAcl",
                     "obs:object:PutAccessLabel",
                     "obs:object:GetObjectVersion",
                     "obs:object:GetObject",
                     "obs:object:GetObjectVersionAcl",
                     "obs:object:DeleteObject",
                     "obs:object:DeleteObjectVersion",
                     "obs:object:GetObjectAcl",
                     "obs:object:PutObject"
                 ],
                 "Resource": [
                     "OBS:*:*:object:*"
                 ]
             }
         ]
     }

   • Policy description: Permissions required for LakeFormation to access OBS (permissions required for LakeFormation to access OBS)
   • Scope: Select Global services.

4. In the navigation pane on the left, choose Agencies. In the upper right corner, click Create Agency.
5. Select Cloud service for Agency Type and select Data Warehouse Service (DWS) for Cloud Service.
6. Click OK. In the displayed dialog box, click Authorize to grant the created LakeFormationObsAccess policy and LakeFormation FullAccess system policy to the agency.

   If "ERROR: LakeFormation respond error, Http Code: 401, Error Code: lakeformation.00000016, Error Message: Insufficient IAM permission." is reported for the DWS foreign table, permissions are insufficient. In this case, check the permissions of the LakeFormation agency.

   

7. Click Next. Select All resources or specific resources for Scope, confirm the information, and click OK.

Creating a LakeFormation Data Source

1. Log in to the DWS console.
2. In the navigation pane on the left, choose Cluster > Cluster List.
3. In the cluster list, click the name of the target cluster. The Cluster Information page is displayed.
4. In the navigation pane, choose Data Source and click the LakeFormation Data Sources tab.
5. Click Create LakeFormation Data Source Connection and configure parameters.
   Figure 1 Creating a LakeFormation data source connection
   

   Table 1 LakeFormation data source connection parameters

   Parameter	Description
   Data Source	Name of the LakeFormation data source connection to be created
   LakeFormation Instance	LakeFormation cluster instance to be bound Select an existing LakeFormation instance from the drop-down list.
   Database	Database where the LakeFormation data source connection is located. Select an existing database in the LakeFormation instance from the drop-down list.
   Agency	An agency authorized by LakeFormation. DWS interacts with LakeFormation through this agency token o obtain metadata. Select an existing agency from the drop-down list.
   Description	Description of the LakeFormation data source connection to be created, which makes it easy to manage and identify the connection.

6. Confirm the settings and click OK. In the displayed dialog box, confirm information about the VPC endpoint and click Yes. The creation takes about one minute.
7. After the creation is complete, view the created data source connection in the LakeFormation data source list. If the configuration status of the connection is available, the connection is created successfully.

Updating a LakeFormation Data Source

This part describes how to update a LakeFormation data source connection.

Deleting a LakeFormation Data Source

1. Log in to the DWS console.
2. In the navigation pane on the left, choose Cluster > Cluster List.
3. In the cluster list, click the name of the target cluster. The Cluster Information page is displayed.
4. In the navigation pane, choose Data Source and click the LakeFormation Data Sources tab.
5. In the Operation column of a LakeFormation data source, click Delete.
6. In the displayed dialog box, confirm the information, enter DELETE or click Auto Enter, and click OK. The deletion takes about 10 seconds.
7. After the deletion is complete, check whether the data source connection disappears from the LakeFormation data source list. If it does, the deletion is successful.

Using a LakeFormation Data Source

For details about how to use a LakeFormation data source, see Using the LakeFormation Data Source to Import Data.

Documentation

• Introduction to LakeFormation: What Is LakeFormation?