Help Center/ MapReduce Service/ User Guide/ Managing Clusters/ Managing MRS Cluster Metadata/ Configuring a LakeFormation Data Connection/ Configuring a LakeFormation Data Connection During Cluster Creation
Updated on 2024-09-23 GMT+08:00
View PDF
Share

Configuring a LakeFormation Data Connection During Cluster Creation

You can configure a LakeFormation data connection when you create an MRS 3.3.0-LTS cluster. You need to configure MRS cluster parameters to interconnect with LakeFormation after the cluster is created.

Configuring a LakeFormation Data Connection During Cluster Creation

  1. Go to the Buy Cluster page.
  2. Click Buy Cluster. The page for buying a cluster is displayed.
  3. On the page for buying a cluster, click the Custom Config tab.
  4. Create a cluster by referring to Configuring Custom Topology. The cluster must meet the requirements described in Table 1.

    Table 1 Data connection parameters

    Parameter

    Description

    Version Type

    LTS

    Cluster Version

    Version of the MRS cluster to be interconnected

    Currently, LakeFormation data connections can be configured during creation of MRS 3.3.0-LTS clusters and later versions only.

    Component

    Components such as Hadoop, Ranger, Hive, Guardian, Spark (optional), and Flink (optional) must be included.

    For example, the following figure shows an example configuration. The configuration may vary depending on the cluster version.

    Metadata

    Select External data connection and set the following parameters:

    1. LakeFormation Metadata: Enable this function.
    2. LakeFormation Instance: Select the LakeFormation data connection created in Creating a LakeFormation Connection.
    3. Data Connection Type: Retain the default value.

    For example, the following figure shows an example configuration. The configuration may vary depending on the cluster version.

    VPC

    VPC where the LakeFormation data connection belongs.

    Subnet

    Subnet name

    Topology Adjustment

    Cluster node topology. Select Enable and ensure that at least one PolicySync (PSC) instance is on a Ranger node which also has the RangerAdmin (RA) instance and at least two TokenSever (TS) instances are added to the Guardian component.

    For example, the following figure shows an example configuration. The configuration may vary depending on the cluster version.

    Kerberos Authentication

    Enable this function.

    Agency

    Select Configure next to Set Advanced Options, select Available agencies for Agency, and select the agency created in Creating an ECS/BMS Agency.

    For example, the following figure shows an example configuration. The configuration may vary depending on the cluster version.

  5. Click the name of the created MRS cluster on the Active Clusters page. In the displayed tab, click Synchronize next to IAM User Sync to synchronize IAM users.

  6. Refer to Configuring the MRS 3.3.0-LTS Cluster to configure decoupled storage and compute and download the client.

Configuring the MRS 3.3.0-LTS Cluster

  1. Log in to FusionInsight Manager of the MRS cluster. For details, see Accessing FusionInsight Manager (MRS 3.x or Later).
  2. Configure the Guardian component.

    1. On FusionInsight Manager, click Cluster, choose Services > Guardian, click Configurations, and then All Configurations. Search for and modify the following parameters, and click Save.
      Table 2 Guardian parameters

      Parameter

      Description

      Value

      token.server.access.iam.domain.id

      Account ID of the user who accesses IAM

      Obtain the account ID by referring to Obtaining the Account ID.

      xxx

      token.server.access.iam.project.id

      Project ID of the user who accesses IAM

      Obtain the project ID by referring to Obtaining the Account ID.

      xxx

      token.server.access.label.agency.name

      Name of an IAM agency. The agency must have the permission to access OBS.

      Set the value to the name of the agency created in Creating an Agency for Interconnecting with OBS.

      visit_obs_agency

      fs.obs.delegation.token.providers

      Name of the class that generates delegation.token. The default value is empty.

      Select both the following parameters:

      • com.huawei.mrs.dt.MRSDelegationTokenProvider
      • com.huawei.mrs.dt.GuardianDTProvider

      com.huawei.mrs.dt.MRSDelegationTokenProvider,com.huawei.mrs.dt.GuardianDTProvider

      fs.obs.guardian.accesslabel.enabled

      Whether to enable access label for using Guardian to connect to OBS.

      true

      fs.obs.guardian.enabled

      Whether to enable Guardian.

      true
    2. In the Dashboard tab of the Guardian service, choose More > Restart Service.

  3. Interconnecting Hive with OBS

    1. Log in to FusionInsight Manager, click Cluster, choose Services > Hive, click Configurations, and then All Configurations.
    2. In the navigation pane on the left, choose HiveServer > Customization. Configure the following parameters.
      Table 3 Custom HiveServer parameters

      Parameter

      Description

      Example Value

      hive.server.customized.configs
      • Add the hive.metastore.warehouse.dir parameter.
      • Set this parameter to the OBS storage path of the hive Catalog, which is obtained in Creating a LakeFormation Instance.
      • Name: hive.metastore.warehouse.dir
      • Value: obs://lakeformation-test/hive
      Figure 1 Configuring hive.metastore.warehouse.dir
    3. Click Save.

  4. Interconnecting Spark with OBS If your cluster does not have the Spark component, skip this step.

    1. Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations then All Configurations.
    2. In the navigation pane on the left, choose JDBCServer > Customization. Add custom parameters and set them by referring to the following table.
      Table 4 Spark parameters

      Custom Item

      Value

      spark.hive-site.customized.configs
      • Parameter: hive.metastore.warehouse.dir
      • Value: Set this parameter to the OBS storage path of the hive Catalog, which is obtained in Creating a LakeFormation Instance. For example, obs://lakeformation-test/hive.
    3. In the navigation pane on the left, choose SparkResource > Customization. Set parameters by referring to Table 4.
    4. Click Save.

  5. In the Components tab of the MRS cluster, check whether there are expired components. If there are, click Restart in the Operation column to restart these components.
  6. Download and reinstall the complete MRS cluster client. For details, see Installing a Client (MRS 3.x or Later).
  7. Update the built-in client configuration file of the cluster to submit jobs on the management console.

    On the dashboard page of the MRS cluster, obtain the EIP, use this IP address to log in to a Master node, and run the following commands to refresh the built-in client of the cluster:

    su - omm

    sh /opt/executor/bin/refresh-client-config.sh

  8. Log in to the node where the client is installed and check the database on the Hive client to verify that the interconnection is successful.

    source Client installation path//bigdata_env

    kinit Component service user

    beeline

    show databases;desc database default;

    !q

  9. On the Spark client, check the database to ensure that the interconnection is successful. If your cluster does not have the Spark component, skip this step.

    source Client installation path//Spark/component_env

    spark-sql

    show databases;desc database default;