Configuring a LakeFormation Data Connection During Cluster Creation

Updated on 2024-09-23 GMT+08:00

View PDF

You can configure a LakeFormation data connection when you create an MRS 3.3.0-LTS cluster. You need to configure MRS cluster parameters to interconnect with LakeFormation after the cluster is created.

Configuring a LakeFormation Data Connection During Cluster Creation

Go to the Buy Cluster page.
Click Buy Cluster. The page for buying a cluster is displayed.
On the page for buying a cluster, click the Custom Config tab.

Create a cluster by referring to Configuring Custom Topology. The cluster must meet the requirements described in Table 1.

**Table 1** Data connection parameters
Parameter	Description
Version Type	LTS
Cluster Version	Version of the MRS cluster to be interconnected Currently, LakeFormation data connections can be configured during creation of MRS 3.3.0-LTS clusters and later versions only.
Component	Components such as Hadoop, Ranger, Hive, Guardian, Spark (optional), and Flink (optional) must be included. For example, the following figure shows an example configuration. The configuration may vary depending on the cluster version.
Metadata	Select External data connection and set the following parameters: LakeFormation Metadata: Enable this function. LakeFormation Instance: Select the LakeFormation data connection created in Creating a LakeFormation Connection. Data Connection Type: Retain the default value. For example, the following figure shows an example configuration. The configuration may vary depending on the cluster version.
VPC	VPC where the LakeFormation data connection belongs.
Subnet	Subnet name
Topology Adjustment	Cluster node topology. Select Enable and ensure that at least one PolicySync (PSC) instance is on a Ranger node which also has the RangerAdmin (RA) instance and at least two TokenSever (TS) instances are added to the Guardian component. For example, the following figure shows an example configuration. The configuration may vary depending on the cluster version.
Kerberos Authentication	Enable this function.
Agency	Select Configure next to Set Advanced Options, select Available agencies for Agency, and select the agency created in Creating an ECS/BMS Agency. For example, the following figure shows an example configuration. The configuration may vary depending on the cluster version.

Click the name of the created MRS cluster on the Active Clusters page. In the displayed tab, click Synchronize next to IAM User Sync to synchronize IAM users.
Refer to Configuring the MRS 3.3.0-LTS Cluster to configure decoupled storage and compute and download the client.

Configuring the MRS 3.3.0-LTS Cluster

Configure the Guardian component.

On FusionInsight Manager, click Cluster, choose Services > Guardian, click Configurations, and then All Configurations. Search for and modify the following parameters, and click Save.

**Table 2** Guardian parameters
Parameter	Description	Value
token.server.access.iam.domain.id	Account ID of the user who accesses IAM Obtain the account ID by referring to Obtaining the Account ID.	xxx
token.server.access.iam.project.id	Project ID of the user who accesses IAM Obtain the project ID by referring to Obtaining the Account ID.	xxx
token.server.access.label.agency.name	Name of an IAM agency. The agency must have the permission to access OBS. Set the value to the name of the agency created in Creating an Agency for Interconnecting with OBS.	visit_obs_agency
fs.obs.delegation.token.providers	Name of the class that generates delegation.token. The default value is empty. Select both the following parameters: com.huawei.mrs.dt.MRSDelegationTokenProvider com.huawei.mrs.dt.GuardianDTProvider	com.huawei.mrs.dt.MRSDelegationTokenProvider,com.huawei.mrs.dt.GuardianDTProvider
fs.obs.guardian.accesslabel.enabled	Whether to enable access label for using Guardian to connect to OBS.	true
fs.obs.guardian.enabled	Whether to enable Guardian.	true

In the Dashboard tab of the Guardian service, choose More > Restart Service.

Interconnecting Hive with OBS

Log in to FusionInsight Manager, click Cluster, choose Services > Hive, click Configurations, and then All Configurations.

In the navigation pane on the left, choose HiveServer > Customization. Configure the following parameters.

**Table 3** Custom HiveServer parameters
Parameter	Description	Example Value
hive.server.customized.configs	Add the hive.metastore.warehouse.dir parameter. Set this parameter to the OBS storage path of the hive Catalog, which is obtained in Creating a LakeFormation Instance.	Name: hive.metastore.warehouse.dir Value: obs://lakeformation-test/hive

Figure 1 Configuring hive.metastore.warehouse.dir

Click Save.

Interconnecting Spark with OBS If your cluster does not have the Spark component, skip this step.

Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations then All Configurations.

In the navigation pane on the left, choose JDBCServer > Customization. Add custom parameters and set them by referring to the following table.

**Table 4** Spark parameters
Custom Item	Value
spark.hive-site.customized.configs	Parameter: hive.metastore.warehouse.dir Value: Set this parameter to the OBS storage path of the hive Catalog, which is obtained in Creating a LakeFormation Instance. For example, obs://lakeformation-test/hive.

In the navigation pane on the left, choose SparkResource > Customization. Set parameters by referring to Table 4.
Click Save.

In the Components tab of the MRS cluster, check whether there are expired components. If there are, click Restart in the Operation column to restart these components.
Download and reinstall the complete MRS cluster client. For details, see Installing a Client (MRS 3.x or Later).
Update the built-in client configuration file of the cluster to submit jobs on the management console.

On the dashboard page of the MRS cluster, obtain the EIP, use this IP address to log in to a Master node, and run the following commands to refresh the built-in client of the cluster:

su - omm

sh /opt/executor/bin/refresh-client-config.sh
Log in to the node where the client is installed and check the database on the Hive client to verify that the interconnection is successful.

source Client installation path//bigdata_env

kinit Component service user

beeline

show databases;desc database default;

!q
On the Spark client, check the database to ensure that the interconnection is successful. If your cluster does not have the Spark component, skip this step.

source Client installation path//Spark/component_env

spark-sql

show databases;desc database default;