Preparing for a LakeFormation Data Connection

Updated on 2024-09-23 GMT+08:00

View PDF

Creating a LakeFormation Instance

Log in to the Huawei Cloud management console, click in the upper left corner, and choose Analytics > DataArts LakeFormation. The LakeFormation console is displayed.
Click Buy Instance in the upper right corner of the page and create a LakeFormation instance by referring to Creating a LakeFormation Instance.
Create the hive catalog and the default database. If the catalog and database already exist in the instance, skip this step. For details, see Managing Metadata.

MRS can interconnect with LakeFormation only when the catalog name of the LakeFormation instance is hive.
1. Ensure that the instance name displayed in the upper left corner is the name of the newly created instance. Then, choose Metadata > Catalog page. You only need one hive catalog per instance. If you already have one in the current instance, you can skip creating another one.
2. Click Create in the upper right corner. On the displayed page, set the following parameters, and click Submit.
  - Catalog Name: Enter hive. Do not use custom names.
  - Select Location: Click to select the OBS storage path corresponding to the catalog, for example, obs://lakeformation-test/hive (which must be an existing path), and click OK.
  - Set other parameters based on the site requirements.
3. In the navigation pane on the left, choose Metadata > Database, click Create Database, configure the following parameters, and click Submit. You only need one default database per instance. If you already have one in the current instance, you can skip creating another one.
  - Database Name: Enter default. Do not use custom names.
  - Catalog: hive
  - Select Location: Click to select a location in the Hive Catalog storage path, for example, obs://lakeformation-test/hive/default (which must be an existing path), and click OK.
  - Set other parameters based on the site requirements.
Choose Data Permissions > Data Authorization in the navigation pane on the left. On the displayed page, you can grant permissions to access Hive Catalog to users and user groups based on service requirements. For details, see Granting Permissions.
Choose Clients in the navigation pane on the left. Click Create to create a client for access management. The VPC and Subnet must be the same as those of the MRS cluster you want to interconnect. For details, see Managing Access Clients.

You can obtain the VPC and subnet of the MRS cluster on the Dashboard page of the cluster on the MRS management console.

Go to the client details page, record the access IP address of the client.

Creating an Agency for Interconnecting with LakeFormation

Log in to Huawei Cloud management console and go to the Identity and Access Management console page.
In the navigation pane on the left, choose Agencies. Click Create Agency in the upper right corner, set related parameters, and click Next.

Refer to the following descriptions to configure the parameters:
- Agency Name: For example, enter visit_lakeformation_agency.
- Agency Type: Select Account.
- Delegated Account: Enter the name of the delegated Huawei Cloud account.
- Validity Period: Set this parameter as you need.
  Figure 1 Creating an agency

In the upper right corner of the Select Policy/Role page, click Create Policy, configure the following information, and click Next.

Policy Name: For example, enter dev_visit_lakeformation.
Policy View: Select Visual editor or JSON.

Policy Content:

The policy must contain lakeformation:policy:export and lakeformation:role:describe. Set other parameters as you need.

If you are using the visual editor, set the Policy Content as follows: Select Data Arts LakeFormation for Cloud Service and select the required permissions in Actions. Set other parameters as you need.

If you are using the JSON view, configure the policy content by referring to the following content:

{
    "Version": "1.1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lakeformation:table:create",
                "lakeformation:database:alter",
                "lakeformation:table:alter",
                "lakeformation:database:drop",
                "lakeformation:database:create",
                "lakeformation:role:describe",
                "lakeformation:policy:create",
                "lakeformation:policy:export",
                "lakeformation:function:alter",
                "lakeformation:function:describe",
                "lakeformation:table:drop",
                "lakeformation:catalog:describe",
                "lakeformation:table:describe",
                "lakeformation:function:drop",
                "lakeformation:database:describe",
                "lakeformation:function:create",
                "lakeformation:transaction:operate",
                "lakeformation:policy:drop"
            ]
        }
    ]
}

Select the new policy name, for example, dev_visit_lakeformation, and click Next.
Select the desired scope requiring minimum authorization and click OK to create an agency.
On the Agencies page, move the cursor to the name of the newly created agency to obtain the ID of the agency that has the permission to access LakeFormation.

Figure 2 Viewing an agency ID

Creating an Agency for Interconnecting with OBS

Log in to Huawei Cloud management console and go to the Identity and Access Management console page.
In the navigation pane on the left, choose Agencies. Click Create Agency in the upper right corner, set related parameters, and click Next.

Refer to the following descriptions to set the parameters:
- Agency Name: For example, enter visit_obs_agency.
- Agency Type: Select Account.
- Delegated Account: Enter the name of the delegated Huawei Cloud account.
- Validity Period: Set this parameter as you need.

In the upper right corner of the Select Policy/Role page, click Create Policy, configure the following information, and click Next.

Policy Name: For example, enter dev_visit_obs.
Policy View: Select JSON.

Policy Content: Enter the following information.

{
    "Version": "1.1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "obs:bucket:GetBucketLocation",
                "obs:bucket:ListBucketMultipartUploads",
                "obs:object:GetObject",
                "obs:object:ModifyObjectMetaData",
                "obs:object:DeleteObject",
                "obs:object:ListMultipartUploadParts",
                "obs:bucket:HeadBucket",
                "obs:object:AbortMultipartUpload",
                "obs:bucket:ListBucket",
                "obs:object:PutObject"
            ],
            "Resource": [
                "OBS:*:*:bucket:*",
                "OBS:*:*:object:*"
            ]
        }
    ]
}

NOTE:

In the Resource parameter, bucket indicates the OBS bucket name, and object indicates the OBS object name. Specify the names as needed. If this parameter is set to *, the policy applies to all OBS buckets or objects.

Set other parameters as you need.

Select the new policy name, for example, dev_visit_obs, and click Next.
Select the desired scope requiring minimum authorization and click OK to create an agency.
On the Agencies page, move the cursor to the name of the newly created agency to obtain the ID of the agency that has the permission to access OBS.

Creating an ECS/BMS Agency

Log in to Huawei Cloud management console and go to the Identity and Access Management console page.
In the navigation pane on the left, choose Agencies. Click Create Agency in the upper right corner, set related parameters, and click Next.

Refer to the following descriptions to set the parameters:
- Agency Name: For example, enter lakeformation_test.
- Agency Type: Select Cloud service.
- Cloud Service: Select ECS BMS.
- Validity Period: Set this parameter as you need.

In the upper right corner of the Select Policy/Role page, click Create Policy, configure the following information, and click Next.

Policy Name: Enter a policy name.
Policy View: Select JSON.

Policy Content: Configure the parameter as follows.

{
    "Version": "1.1",
    "Statement": [
        {
            "Action": [
                "iam:agencies:assume"
            ],
            "Resource": {
                "uri": [
                    "/iam/agencies/ID of the agency that grants LakeFormation access permission to your account",
                    "/iam/agencies/ID of the agency that grants the OBS access permission to your account"
                ]
            },
            "Effect": "Allow"
        }
    ]
}

NOTE:

To obtain the ID of the agency that has the permission to access LakeFormation, refer to 6.
To obtain the ID of the agency that has the permission to access OBS, refer to 6.

Select the name of the newly created custom agency and click Next.
Select the desired scope requiring minimum authorization and click OK to create an agency.

Creating a LakeFormation Connection

NOTE:

You need to contact technical support to be added in the whitelist before you create a LakeFormation data connection.

Log in to the MRS management console, and choose Data Connections in the left navigation pane.
Click Create Data Connection.

Set parameters by referring to Table 1 and click OK.

**Table 1** LakeFormation data connection
Parameter	Description
Type	Select LakeFormation. Only MRS 3.3.0-LTS supports this connection type.
Name	The name of a data connection.
LakeFormation Instance	Select an instance name. This instance is created in the LakeFormation console before being referenced here. For details about how to create the instance, see Creating a LakeFormation Instance. Click View instances to view created instances.
VPC	It must be in the same VPC as the MRS cluster you want to interconnect.
Subnet	Subnet name
VPC Endpoint	Select a VPC endpoint or click Create VPC endpoints for LakeFormation Instance. After you select a VPC endpoint, you will be charged by the VPCEP service.
LakeFormation Agency	Select Available agencies and select the agency created in Creating an Agency for Interconnecting with LakeFormation, for example, visit_lakeformation_agency.

Figure 3 Creating a LakeFormation data connection

Record the ID of the created data connection on the Data Connections page.

Obtaining the Account ID

Log in to the management console as the user for interconnecting MRS with LakeFormation.
Hover over the username and choose My Credentials from the drop-down list.
On the API Credentials page, obtain the Account ID and view the Project ID in the project list.
Grant the current user the permission to use LakeFormation.
1. Click in the upper left corner and choose Analytics > LakeFormation.
2. Check whether the service authorization page is displayed, or go to the Service Authorization page to check whether the service has been authorized.
  - Select the checkbox to agree with the LakeFormation Service Statement and click Authorize.
  - If no, the current user has the permission to perform operations on LakeFormation.