Disabling Ranger OBS Path Authentication for Guardian

Scenario

This section describes how to enable the storage-compute decoupling function for Guardian without configuring the permission policies of OBS paths through Ranger. That is, you do not need to check whether the AccessLabel function is enabled for OBS when connecting Guardian to OBS. Once enabled, Guardian can provide temporary authentication credentials for components such as HDFS, Hive, Spark, Loader, and HetuEngine to access OBS in storage-compute decoupling scenarios.

To connect Guardian to OBS, do as follows:

1. Creating an OBS Parallel File System
2. Creating a Cloud Service Agency and Binding It to a Cluster
3. Granting Guardian the Permission to Access OBS
4. Configuring a Recycle Bin Cleanup Policy

Prerequisites

• Components such as Guardian, Ranger, and Hadoop have been installed in the cluster.
• If components such as Hadoop, HetuEngine, Hive, and Spark have been installed in an environment before installing Guardian, you need to download these component clients again and refresh the default clients for job submission on the MRS console.

Impact on the System

Once you finish the configuration, you will need to either refresh the original client's configuration or reinstall the client.

Creating an OBS Parallel File System

1. Log in to the OBS console.
2. Choose Parallel File Systems > Create Parallel File System.
3. Enter a file system name, for example, guardian-obs.

   Use the enterprise project selected during MRS cluster creation and set other parameters as needed.

4. Click Create Now.

Creating a Cloud Service Agency and Binding It to a Cluster

• MRS presets MRS_ECS_DEFAULT_AGENCY in the IAM agency list by default, allowing you to choose this agency when creating a cluster. This agency has OBS OperateAccess permission and, for users with fine-grained policies enabled, CES FullAccess, CES Administrator, and KMS Administrator permissions in the region where the cluster is located. Do not modify MRS_ECS_DEFAULT_AGENCY on IAM.
• If you want to use the preset agency, skip the step for creating an agency. If you want to use a custom agency, perform the following steps to create an agency. (To create or modify an agency, you must have the Security Administrator permission.) If you need to have more fine-grained control over the permissions of a specific path in the OBS file system, you can refer to Configuring Fine-Grained OBS Access Permissions for MRS Cluster Users to create a custom role policy.

1. Log in to the Huawei Cloud management console.
2. In the service list, choose Management & Governance > Identity and Access Management.
3. Choose Agencies. On the displayed page, click Create Agency.
4. On the Create Agency page, set the following parameters and click Done:
   • Agency Name: Enter an agency name, for example, mrs_ecs_obs.
   • Agency Type: Select Cloud service.
   • Cloud Service: Select Elastic Cloud Server (ECS) and Bare Metal Server (BMS).
   • Validity Period: Select Unlimited.
5. In the displayed dialog box, click Authorize. On the displayed page, click Create Policy.
   On the Create Policy page, set the following parameters and click Next:

   • Policy Name: Enter a policy name, for example, guardian-policy.
   • Policy View: Select JSON.
   • Policy Content: Configure the parameter as follows:

     {
         "Version": "1.1",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
     			"obs:bucket:GetBucketLocation",
     			"obs:bucket:ListBucketMultipartUploads",
     			"obs:object:GetObject",
     			"obs:object:ModifyObjectMetaData",
     			"obs:object:DeleteObject",
     			"obs:object:ListMultipartUploadParts",
     			"obs:bucket:HeadBucket",
     			"obs:object:AbortMultipartUpload",
     			"obs:bucket:ListBucket",
     			"obs:object:PutObject",
     		        "obs:bucket:ListAllMyBuckets"
                 ],
                 "Resource": [
                     "OBS:*:*:bucket:guardian-obs",
                     "OBS:*:*:bucket:guardian-obs2",
                     "OBS:*:*:object:*"
                 ]
             }
         ]
     }

     In the preceding configuration, Resource indicates that all resources of the configured parallel file system can be accessed. guardian-obs indicates the name of the OBS parallel file system created in Creating an OBS Parallel File System.

6. Click Next. On the Select Policy/Role page, select the policy created in 5.
7. Click Next, select All resources, click Show More, select Global resources, and click OK.
8. In the displayed dialog box, click OK to start authorization. After the message "Authorization successful." is displayed, click Finish. The agency is successfully created.
9. Log in to the MRS console. In the navigation pane on the left, choose Active Clusters.
10. Click the name of the target cluster to access its details page.
11. On the Dashboard page, click Synchronize next to IAM User Sync to synchronize IAM users.
12. On the Dashboard tab, click Select Agency next to Agency. In the displayed dialog box, select the agency you created, for example, mrs_ecs_obs, and click OK.
    Figure 1 Binding an agency
    

Granting Guardian the Permission to Access OBS

1. Log in to MRS Manager by referring to Accessing MRS Manager. Choose Cluster > Services > Guardian, click Configurations then All Configurations, search for and configure the following parameters:

   Parameter	Description	Example Value
   fs.obs.guardian.accesslabel.enabled	Whether to enable AccessLabel on OBS, which allows Guardian to connect to OBS. true: indicates that after OBS is connected, you can use Ranger to configure the permission policies of OBS paths. Ensure that the AccessLabel function has been enabled for OBS. If the function is not enabled, manually enable it. For details, contact OBS O&M personnel. false: indicates After OBS is interconnected, the permission policies of OBS paths cannot be configured using Ranger. You do not need to check whether the AccessLabel function is enabled for OBS. Set this parameter to false.	false
   fs.obs.guardian.enabled	Whether to enable Guardian. true: enables the Guardian. false: (default value) disables the Guardian.	true
   fs.obs.delegation.token.providers	Delegation token generator. When fs.obs.guardian.enabled is set to true, com.huawei.mrs.dt.MRSDelegationTokenProvider and com.huawei.mrs.dt.GuardianDTProvider must be configured.	com.huawei.mrs.dt.MRSDelegationTokenProvider and com.huawei.mrs.dt.GuardianDTProvider

2. Click Save to save the service configuration. On the FusionInsight Manager, choose More > Restart Configuration-Expired Instances and restart all service instances with expired configurations as prompted.
3. To submit jobs on the MRS console, log in to the active OMS node as user omm and run the following command to refresh the built-in client configuration:

   sh /opt/executor/bin/refresh-client-config.sh

4. (Optional) If you need to update the client configurations of other installed cluster components, see Updating Client Configurations.

Configuring a Recycle Bin Cleanup Policy

1. Log in to the OBS console.
2. In the navigation pane on the left, choose Resources > Parallel File Systems. On the displayed page, click the name of the file system created in Creating an OBS Parallel File System.
3. Choose Data Management > Lifecycle Rules. On the displayed page, click Create to create a lifecycle rule for the /user/.Trash directory.
   

   After the decoupled storage-compute solution is used, you must configure lifecycle rules for related directories. Otherwise, there is a risk of running out of storage space and incurring additional storage costs. For details about OBS billing, see OBS Billing Overview.

   Table 1 Parameters for creating a lifecycle rule

   Parameter	Description	Example Value
   Status	Whether to enable the lifecycle rule.	Enabled
   Rule Name	Rule name, used to identify different lifecycle configurations.	rule-test
   Prefix	Prefix of the objects to which the lifecycle rule applies. Typically, the recycle bin directory of MRS components is prefixed with /user/.Trash.	user/.Trash
   Transition to Infrequent Access After (Days)	Number of days after the last update of an object that it will be transitioned to infrequent access storage based on the rule. The minimum value is 30.	30
   Transition to Archive After (Days)	Number of days after the last update of an object that it will be transitioned to archive based on the rule. If you are setting both this parameter and Transition to Infrequent Access After (Days), make sure this parameter value is at least 30 days greater than the value of Transition to Infrequent Access After (Days). If you are only setting this parameter, assign any value to it as needed.	31
   Delete Files After (Days)	Number of days after the last update of an object that it will expire and be automatically deleted by OBS based on the rule. The value of this parameter must be greater than the values of the parameters Transition to Infrequent Access After (Days) and Transition to Archive After (Days).	32
   Delete Fragments Upon Expiration	Number of days of a fragment that it will expire and be automatically deleted by OBS based on the rule.	30

4. Click OK.

   To modify the lifecycle configuration, locate the target lifecycle rule, click Edit or Disable on the right. To enable a lifecycle rule, click Enable.