Lakehouse: Enabling DWS to Read MRS Hive Data by Interconnecting with LakeFormation

Scenarios

Data lakes, data warehouses, and AI data are isolated. Although data is centrally stored in OBS, the metadata of different services and clusters is managed independently. As a result, data management is not streamlined.

DWS can access the MRS data stored in OBS only through foreign tables. The metadata of different services cannot be centrally configured or managed.

Figure 1 How DWS accesses MRS

LakeFormation is an enterprise-grade one-stop data lake construction service. It adopts a storage-compute decoupled architecture and provides unified metadata management, data permission management, and APIs for eliminating data silos and achieving data-AI convergence. It can connect to compute engines and big data cloud services, such as MRS, DWS, DLI, ModelArts, and DataArts Studio. This enables you to easily and efficiently build data lakes and run workloads to unlock the value of business data.

By connecting to LakeFormation, DWS can eliminate data silos. Without foreign tables, DWS can access LakeFormation to obtain the metadata of tables from other services and access the data of these services stored in OBS. In addition, DWS offers fine-grained permission control based on the permission management capabilities provided by LakeFormation.

Figure 2 Interconnecting DWS with LakeFormation

Procedure

This practice describes how to interconnect DWS with LakeFormation and read MRS Hive data. The entire process consists of the following operations. If MRS is available and is interconnected with LakeFormation, skip the corresponding steps.

This practice takes about 2 hours. The following cloud services are required: DWS, MRS, OBS, LakeFormation, VPC Endpoint, and IAM.

1. Step 1: Create a LakeFormation Instance
2. Step 2: Interconnect MRS with LakeFormation
3. Step 3: Interconnect DWS with LakeFormation
4. Step 4: Read MRS Hive Data Using LakeFormation

Constraints

• Ensure that the target MRS cluster runs version 3.3.0-LTS or later, and that Kerberos authentication is enabled for the cluster.
• The following table storage formats are supported: ORC, Parquet, TEXT, CSV, and Hudi.
• Only MRS tables in LakeFormation can be read.
• Only query operations are supported.
• Tables cannot contain fields of the following types: map, struct, array, binary, tinyint, and uniontype.
• Analysis is not supported.

Prerequisites

• A DWS cluster has been created.
• An MRS cluster has been created.
• An agency with LakeFormation permissions (including the minimum permissions) has been created.

Step 1: Create a LakeFormation Instance

Before creating a LakeFormation instance, catalog, and database, plan the OBS bucket for storing metadata.

1. Create a bucket. Set the OBS bucket name to lakeformation-obs-xx (where xx is a number), for example, lakeformation-obs-01. If a number is already in use, use a larger number.
2. Access the OBS bucket, create a folder named hive, and create a subfolder named default in the hive folder.
3. Log in to the LakeFormation console and click Buy Instance in the upper right corner.

   Set the following key parameters and retain default values for the other parameters. For details about the parameters, see Creating a LakeFormation Instance.

   Table 1 Creating a LakeFormation instance

   Parameter	Value
   Type	Shared
   Billing Mode	Pay-per-use
   Name	lakeformation-demo

4. Click Buy Now.
5. Return to the LakeFormation console homepage, refresh the page, select the created LakeFormation instance in the left navigation pane, and choose Metadata > Catalog.
6. Click Create Catalog. On the displayed page, configure the following parameters, and click Submit.
   • Catalog Name: Enter hive. Do not use a custom name.
   • Select Location: Select the path of the created OBS bucket, for example, obs://lakeformation-obs-01/hive/.

   

7. In the navigation pane on the left, choose Metadata > Database. Then, click Create Database.
8. Set the following parameters and click Submit.
   • Database Name: Enter default. Do not use a custom name.
   • Catalog: Select hive.
   • Select Location: Select a path under the hive catalog storage path, for example, obs://lakeformation-obs-01/hive/default.

   

9. Choose Clients in the navigation pane on the left. Click Create to create a client for access management. The VPC and subnet must be the same as those of the MRS cluster you want to interconnect.

   You can log in to the MRS console and obtain the VPC and subnet information of the cluster from the Dashboard page.

   Go to the client details page and record the access IP address of the client.

10. Create an agency for interconnecting with LakeFormation.
    1. Log in to the IAM console.
    2. In the navigation pane, choose Agencies. Click Create Agency in the upper right corner, set the parameters, and click Next.

       Set the parameters as follows:

       • Agency Name: For example, enter visit_lakeformation_agency.
       • Agency Type: Select Account.
       • Delegated Account: Enter the name of the delegated Huawei Cloud account.
       • Validity Period: Set it based on your requirements.
    3. In the upper right corner of the Select Policy/Role page, click Create Policy. Configure the following information and click Next.
       • Policy Name: For example, enter dev_visit_lakeformation.
       • Policy View: Select Visual editor or JSON.
       • Policy Content: If you select JSON for Policy View, enter the following policy content:

         {
             "Version": "1.1",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Action": [
                         "lakeformation:table:create",
                         "lakeformation:database:alter",
                         "lakeformation:table:alter",
                         "lakeformation:database:drop",
                         "lakeformation:database:create",
                         "lakeformation:role:describe",
                         "lakeformation:policy:create",
                         "lakeformation:policy:export",
                         "lakeformation:function:alter",
                         "lakeformation:function:describe",
                         "lakeformation:table:drop",
                         "lakeformation:catalog:describe",
                         "lakeformation:table:describe",
                         "lakeformation:function:drop",
                         "lakeformation:database:describe",
                         "lakeformation:function:create",
                         "lakeformation:transaction:operate",
                         "lakeformation:policy:drop",
                         "lakeformation:policy:describe",
                         "lakeformation:connection:describe"
                     ]
                 }
             ]
         }

    4. Confirm the information, return to the policy list, select the created dev_visit_lakeformation policy, and click Next.
    5. Retain the default settings for Scope and click OK.
    6. On the Agencies page, hover over the name of the created agency to obtain the agency ID. The agency has the permission to access LakeFormation.

11. Create an agency for interconnecting with OBS.
    1. Log in to the IAM console.
    2. In the navigation pane, choose Agencies. Click Create Agency in the upper right corner, set the parameters, and click Next.

       Set the parameters as follows:

       • Agency Name: For example, enter visit_obs_agency.
       • Agency Type: Select Account.
       • Delegated Account: Enter the name of the delegated Huawei Cloud account.
       • Validity Period: Set it based on your requirements.
    3. In the upper right corner of the Select Policy/Role page, click Create Policy. Configure the following information and click Next.
       • Policy Name: For example, enter dev_visit_obs.
       • Policy View: Select JSON.
       • Policy Content: Enter the following information:

         {
             "Version": "1.1",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Action": [
                         "obs:bucket:GetBucketLocation",
                         "obs:bucket:ListBucketMultipartUploads",
                         "obs:object:GetObject",
                         "obs:object:ModifyObjectMetaData",
                         "obs:object:DeleteObject",
                         "obs:object:ListMultipartUploadParts",
                         "obs:bucket:HeadBucket",
                         "obs:object:AbortMultipartUpload",
                         "obs:bucket:ListBucket",
                         "obs:object:PutObject"
                     ],
                     "Resource": [
                         "OBS:*:*:bucket:*",
                         "OBS:*:*:object:*"
                     ]
                 }
             ]
         }

    4. Confirm the information, return to the policy list, select the created dev_visit_obs policy, and click Next.
    5. Retain the default settings for Scope and click OK.
    6. On the Agencies page, hover over the name of the created agency to obtain the agency ID. The agency has the permission to access OBS.

12. Create an agency for interconnecting with ECS/BMS..
    1. Log in to the IAM console.
    2. In the navigation pane, choose Agencies. Click Create Agency in the upper right corner, set the parameters, and click Next.

       Set the parameters as follows:

       • Agency Name: For example, enter lakeformation_test.
       • Agency Type: Select Cloud service.
       • Cloud Service: Select ECS BMS.
       • Validity Period: Set it based on your requirements.
    3. In the upper right corner of the Select Policy/Role page, click Create Policy. Configure the following information and click Next.
       • Policy Name: Enter a policy name.
       • Policy View: Select JSON.
       • Policy Content: Enter the following information. Obtain the IDs of the agencies for accessing LakeFormation and OBS from 10.f and 11.f, respectively.

         {
             "Version": "1.1",
             "Statement": [
                 {
                     "Action": [
                         "iam:agencies:assume"
                     ],
                     "Resource": {
                         "uri": [
                             "/iam/agencies/ID of the agency that grants the LakeFormation access permission to your account",
                             "/iam/agencies/ID of the agency that grants the OBS access permission to your account"
                         ]
                     },
                     "Effect": "Allow"
                 }
             ]
         }

    4. Confirm the information, return to the policy list, select the created lakeformation_test policy, and click Next.
    5. Select All resources for Scope and click OK.

13. Create a LakeFormation data connection.
    1. Log in to the MRS console. In the navigation pane, choose Data Connections.
    2. Click Create Data Connection.
    3. Set the following parameters.

       Table 2 Creating a LakeFormation data connection

       Parameter	Example	Description
       Type	LakeFormation	Select LakeFormation. Only MRS 3.3.0-LTS and later versions support this connection type.
       Name	mrs_LakeFormation	Name of the data connection
       LakeFormation Instance	-	Select a LakeFormation instance.
       VPC	-	Select the same VPC as the MRS cluster to be interconnected with.
       Subnet	-	Subnet name
       VPC Endpoint	-	Select a VPC endpoint or click Create VPC Endpoint to create one. After you select a VPC endpoint, you will be billed by the VPCEP service.
       LakeFormation Agency	Available agencies	Select Available agencies and select the agency created in 10, for example, visit_lakeformation_agency.

    4. Record the ID of the created data connection on the Data Connections page.

Step 2: Interconnect MRS with LakeFormation

1. Ensure that an MRS cluster meeting the following requirements has been created. For details, see MRS Documentation.
   • The version must be MRS 3.3.0-LTS or later, and Kerberos authentication and storage-compute decoupling must be enabled.
   • The cluster must contain the following components: Hadoop, Ranger, Hive, and Guardian. Spark and Flink are optional.
   • When buying the cluster, you must select Topology Adjustment, select at least one PolicySync (PSC) instance under Ranger, and ensure that PolicySync and RangerAdmin instances are deployed on the same node. In addition, ensure that the Guardian component contains at least two TokenServer (TS) instances.
   • IAM users have been synchronized on the MRS console, and all cluster services are running properly.
   • Ranger authentication has been enabled for Hive in the MRS cluster. For details, see Enabling Ranger Authentication for Cluster Components.

2. Log in to the MRS console and choose Active Clusters.
3. Click the name of the target MRS cluster to enter the Dashboard page.
4. Click Select Agency next to Agency and select the agency created in 12.
5. Click Manage next to Data Connection. The Data Connection dialog box is displayed.
6. Click Configure LakeFormation Data Connection, select the LakeFormation data connection ID recorded in 13 from the drop-down list, and click OK.
7. Log in to FusionInsight Manager of the MRS cluster. For details, see Accessing MRS Manager.
8. Configure Guardian.
   1. Log in to the IAM console.
   2. Click the username and choose My Credentials from the drop-down list.
   3. On the API Credentials page, obtain the Account ID and Project ID from the project list.
   4. On FusionInsight Manager, choose Cluster > Services > Guardian, and click Configurations and then All Configurations. Search for and modify the following parameters, and click Save.

      Table 3 Configuring Guardian

      Parameter	Description	Value
      token.server.access.iam.domain.id	Account ID of the user accessing IAM Obtain the account ID from 8.c.	xxx
      token.server.access.iam.project.id	Project ID of the user accessing IAM Obtain the project ID from 8.c.	xxx
      token.server.access.label.agency.name	Name of an IAM agency. The agency must have the permission to access OBS. This is the name of the agency created in 11.	visit_obs_agency
      fs.obs.delegation.token.providers	Name of the class that generates delegation.token. The default value is empty. Select the following values: com.huawei.mrs.dt.MRSDelegationTokenProvider com.huawei.mrs.dt.GuardianDTProvider	com.huawei.mrs.dt.MRSDelegationTokenProvider,com.huawei.mrs.dt.GuardianDTProvider
      fs.obs.guardian.accesslabel.enabled	Whether to enable an access label on OBS, which allows Guardian to connect to OBS	true
      fs.obs.guardian.enabled	Whether to enable Guardian	true

   5. On the Dashboard page of the Guardian service, choose More > Restart Service.

9. Interconnect Hive with the OBS file system.
   1. On FusionInsight Manager, choose Cluster > Services > Hive. Click Configurations and then All Configurations.
   2. In the navigation pane, choose HiveServer > Customization. Add the following custom parameters.

      Table 4 Configuring interconnection between Hive and OBS

      Parameter	Description	Example Value
      hive.server.customized.configs	Add the hive.metastore.warehouse.dir parameter. Set the value to the storage path of the hive catalog in OBS, which can be obtained in 6.	Name: hive.metastore.warehouse.dir Value: obs://lakeformation-obs-01/hive
      hive.metastore.customized.configs	This parameter is required for clusters of MRS 3.3.1 or later. Add the hive.metastore.warehouse.dir parameter. Set the value to the storage path of the hive catalog in OBS, which can be obtained in 6.	Name: hive.metastore.warehouse.dir Value: obs://lakeformation-obs-01/hive

   3. Click Save.

10. On the Components tab page of the MRS cluster, check whether there are components whose configurations have expired. If there are, click Restart in the Operation column to restart these components.
11. Download and reinstall an MRS cluster client. For details, see Installing a Client.
12. Update the built-in client configuration file of the cluster to submit jobs on the management console.

    On the dashboard page of the MRS cluster, obtain the EIP, use it to log in to a Master node, and run the following commands to update the built-in client of the cluster:

    su - omm

    sh /opt/executor/bin/refresh-client-config.sh

13. Log in to the node where the client is installed and check the database on the Hive client to verify that the interconnection is successful.

    source Client installation path/bigdata_env

    kinit Component service user

    beeline

    show databases;

    desc database default;

    !q

    

Step 3: Interconnect DWS with LakeFormation

1. Log in to the DWS console and choose Cluster > Cluster List in the navigation pane.
2. Click the name of the cluster that has been created to go to the cluster details page. Choose Data Sources > LakeFormation Data Sources.
3. Click Create LakeFormation Data Source Connection and configure parameters.

   

   Table 5 Creating a LakeFormation data source connection

   Parameter	Description
   Data Source	lakeformation-dws
   LakeFormation Instance	Select the LakeFormation instance created in Step 1: Create a LakeFormation Instance from the drop-down list.
   Database	Database where the LakeFormation data source connection is to be created
   Agency	Select the agency created in 10.
   Description	-

4. Click OK.

Step 4: Read MRS Hive Data Using LakeFormation

1. Create an external schema to access the metadata of MRS tables stored in LakeFormation and then access the table data stored in OBS.
   For more syntax, see CREATE EXTERNAL SCHEMA.

   CREATE EXTERNAL SCHEMA ex_lf    --
       WITH SOURCE lakeformation   --Type of the external metadata storage engine. Set this parameter to lakeformation.
            DATABASE 'default'   --Name of the LakeFormation database to be accessed
            SERVER lakeformation-dws    --Name of the created LakeFormation data source
            CATALOG 'hive';   --Name of the LakeFormation catalog to be accessed. Set this parameter to hive.

2. View the current DWS user, for example, dbadmin.

   1	SELECT current_user;

3. Create a role with the same name as the user in LakeFormation and grant permissions to the role.
   1. Log in to the LakeFormation console.
   2. Select the LakeFormation instance to be operated from the drop-down list on the left and choose Data Permissions > Role.
   3. Click Create. In the displayed dialog box, enter dbadmin for Role Name and click OK.
   4. Choose Data Permissions > Data Authorization Click Authorize. In the displayed dialog box, set parameters by referring to the table below and click OK.

      Table 6 Authorizing a role in LakeFormation

      Parameter	Description
      Entity Type	Role
      Role	dbadmin
      Granted To	Resources
      Resource Type	Select the default database under hive for Catalog.
      Permission	ALL

   5. Click OK.

4. Return to the page for connecting to the DWS database and run the following SQL statement to query data in the Hive table:

   1	SELECT * FROM ex_lf.test;