Disabling Ranger OBS Path Authentication for Guardian

Scenarios

This section describes how to enable storage–compute decoupling for the Guardian component when Kerberos authentication is disabled for the cluster (in normal mode). In this scenario, the Ranger component cannot be used to configure permission policies for OBS paths—that is, the AccessLabel function used by Guardian to interconnect with OBS is disabled, and you do not need to consider whether AccessLabel is enabled on OBS. Once enabled, Guardian can provide temporary authentication credentials for components such as HDFS, Hive, Spark, Loader, and HetuEngine to access OBS in storage-compute decoupling scenarios.

To interconnect Guardian with OBS, do as follows:

1. Creating an OBS Parallel File System
2. Creating a Cloud Service Agency and Binding It to a Cluster
3. Granting Guardian Permissions to Access OBS
4. Configuring a Recycle Bin Cleanup Policy
5. Configuring a Lifecycle Rule for OBS Fragments

Prerequisites

• Components such as Guardian, Ranger, and Hadoop have been installed in the cluster.
• If components such as Hadoop, HetuEngine, Hive, and Spark have been installed in an environment before installing Guardian, you need to download these component clients again and refresh the default clients for job submission on the MRS console.

Impact on the System

Once you finish the configuration, you will need to either refresh the original client's configuration or reinstall the client.

Creating an OBS Parallel File System

1. Log in to the OBS console.
2. Choose Parallel File Systems > Create Parallel File System.
3. Enter a file system name, for example, guardian-obs.

   Use the enterprise project selected during MRS cluster creation and set other parameters as needed.

4. Click Create Now.

Creating a Cloud Service Agency and Binding It to a Cluster

• MRS provides the default agency MRS_ECS_DEFAULT_AGENCY in IAM. You can select this agency when creating a cluster. This agency includes the MRS Agency KMSAccess permissions in the cluster's region and the OBS OperateAccess permissions. Do not modify MRS_ECS_DEFAULT_AGENCY in IAM.
• If you want to use the preset agency, skip the step for creating an agency. If you want to use a custom agency, perform the following steps to create an agency. (To create or modify an agency, you must have the Security Administrator permission.) If you need to have more fine-grained control over the permissions of a specific path in the OBS file system, you can refer to Configuring Fine-Grained OBS Access Permissions for MRS Cluster Users to create a custom role policy.

1. Log in to the IAM console.
2. Choose Agencies. On the displayed page, click Create Agency.
3. On the Create Agency page, set the following parameters and click Done:
   • Agency Name: Enter an agency name, for example, mrs_ecs_obs.
   • Agency Type: Select Cloud service.
   • Cloud Service: Select Elastic Cloud Server (ECS) and Bare Metal Server (BMS).
   • Validity Period: Select Unlimited.
4. In the displayed dialog box, click Authorize. On the displayed page, click Create Policy.
   On the Create Policy page, set the following parameters and click Next:

   • Policy Name: Enter a policy name, for example, guardian-policy.
   • Policy View: Select JSON.
   • Policy Content: Configure the parameter as follows:

     {
         "Version": "1.1",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
     			"obs:bucket:GetBucketLocation",
     			"obs:bucket:ListBucketMultipartUploads",
     			"obs:object:GetObject",
     			"obs:object:ModifyObjectMetaData",
     			"obs:object:DeleteObject",
     			"obs:object:ListMultipartUploadParts",
     			"obs:bucket:HeadBucket",
     			"obs:object:AbortMultipartUpload",
     			"obs:bucket:ListBucket",
     			"obs:object:PutObject",
     		        "obs:bucket:ListAllMyBuckets"
                 ],
                 "Resource": [
                     "OBS:*:*:bucket:guardian-obs",
                     "OBS:*:*:bucket:guardian-obs2",
                     "OBS:*:*:object:*"
                 ]
             }
         ]
     }

     In the preceding configuration, Resource indicates all resources that can access the configured parallel file system. guardian-obs indicates the name of the OBS parallel file system created in Creating an OBS Parallel File System.

5. Click Next. On the Select Policy/Role page, select the policy created in 4.
6. Click Next, select All resources, click Show More, select Global resources, and click OK.
7. In the displayed dialog box, click OK to start authorization. After the message "Authorization successful." is displayed, click Finish. The agency is successfully created.
8. Log in to the MRS console.
9. In the navigation pane, choose Active Clusters.
10. Click the name of the target cluster to access its details page.
11. On the Dashboard page, click Synchronize next to IAM User Sync to synchronize IAM users.
12. On the Dashboard tab, click Select Agency next to Agency. In the displayed dialog box, select the agency you created, for example, mrs_ecs_obs, and click OK.

Granting Guardian Permissions to Access OBS

1. Log in to MRS Manager by referring to Accessing MRS Manager. Choose Cluster > Services > Guardian, click Configurations and then All Configurations, search for and configure the following parameters.

   Parameter	Description	Example Value
   fs.obs.guardian.accesslabel.enabled	Whether to enable AccessLabel on OBS, which allows Guardian to connect to OBS. true: indicates that after OBS is connected, you can use Ranger to configure the permission policies of OBS paths. Ensure that the AccessLabel function has been enabled for OBS. If the function is not enabled, manually enable it. For details, contact OBS O&M personnel. false: indicates that after OBS is interconnected, Ranger cannot be used to configure permission policies for OBS paths. You do not need to check whether AccessLabel is enabled for OBS. Set this parameter to false.	false
   fs.obs.guardian.enabled	Whether to enable Guardian. true: enables the Guardian. false: (default value) disables the Guardian.	true
   fs.obs.delegation.token.providers	Delegation token generator. When fs.obs.guardian.enabled is set to true, com.huawei.mrs.dt.MRSDelegationTokenProvider and com.huawei.mrs.dt.GuardianDTProvider must be configured.	com.huawei.mrs.dt.MRSDelegationTokenProvider and com.huawei.mrs.dt.GuardianDTProvider

2. Click Save to save the service configuration. On the FusionInsight Manager, choose More > Restart Configuration-Expired Instances and restart all service instances with expired configurations as prompted.
3. To submit jobs on the MRS console, log in to the active OMS node as user omm and run the following command to refresh the built-in client configuration:

   sh /opt/executor/bin/refresh-client-config.sh

4. (Optional) If you need to update the client configurations of other installed cluster components, see Updating Client Configurations.

Configuring a Recycle Bin Cleanup Policy

1. Log in to the OBS console.
2. In the navigation pane on the left, choose Resources > Parallel File Systems. On the displayed page, click the name of the file system created in Creating an OBS Parallel File System.
3. Choose Data Management > Lifecycle Rules. On the displayed page, click Create to create a lifecycle rule for the /user/.Trash directory.
   

   When using the decoupled storage-compute solution, configure lifecycle rules for the relevant directories. Without them, you risk running out of storage space and paying higher storage fees.

   Table 1 Parameters for creating a lifecycle rule

   Parameter	Description	Example Value
   Status	Whether to enable the lifecycle rule.	Enabled
   Rule Name	Rule name that identifies different lifecycle configurations. You can customize the name.	rule-test
   Prefix	Prefix of the objects to which the lifecycle rule applies. Typically, the recycle bin directory of MRS components is prefixed with /user/.Trash.	user/.Trash
   Transition to Infrequent Access After (Days)	Number of days after the last update of an object that it will be transitioned to infrequent access storage based on the rule. The minimum value is 30.	30
   Transition to Archive After (Days)	Number of days after an object's last update before it transitions to the Archive storage class based on the lifecycle rule. If you are setting both this parameter and Transition to Infrequent Access After (Days), make sure this parameter value is at least 30 days greater than the value of Transition to Infrequent Access After (Days). If you are only setting this parameter, assign any value to it as needed.	31
   Delete Files After (Days)	Number of days after an object's last update before it expires and is automatically deleted by OBS based on the lifecycle rule. The value of this parameter must be greater than the values of the parameters Transition to Infrequent Access After (Days) and Transition to Archive After (Days).	32
   Delete Fragments Upon Expiration	Number of days before fragments expire and are automatically deleted by OBS.	30

4. Click OK.

   To modify the lifecycle configuration, locate the target lifecycle rule, click Edit or Disable on the right. To enable a lifecycle rule, click Enable.

Configuring a Lifecycle Rule for OBS Fragments

1. Log in to the OBS console.
2. In the navigation pane on the left, choose Resources > Parallel File Systems. On the displayed page, click the name of the file system created in Creating an OBS Parallel File System.
3. Choose Data Management > Lifecycle Rules. On the displayed page, click Create to create a lifecycle rule for OBS fragments.
   

   • OBS uses multipart upload. If a data write task fails or is interrupted, file fragments remain in OBS and generate storage charges. To avoid extra costs, set a lifecycle policy for the required directory.
   • It is recommended that you add a lifecycle rule with Scope set to All files for OBS fragments. This rule should be separate from the lifecycle rule you configure for the component recycle bin directory.

   Table 2 Parameters for creating a lifecycle rule

   Category	Parameter	Description	Example Value
   Basic information	Status	Whether to enable the lifecycle rule.	Enabled
   	Rule Name	Rule name that identifies different lifecycle configurations. You can customize the name.	rule-test1
   	Scope	Scope of objects the lifecycle rule applies to. Configure the scope based on your actual requirements. For example, set it to All files.	All files
   Current Version	Status	Whether to enable the current version. Disable this option.	Disabled
   Delete Fragments Upon Expiration	Status	Whether to enable the Delete Fragments Upon Expiration function. Enable this option.	Enabled
   	Days	The number of days after which a fragment expires and is automatically deleted by OBS. Set this parameter as required. For example, set it to 7 days.	7 days

4. Select the confirmation information and click OK. The lifecycle rule is configured.

   To modify, disable, or enable a lifecycle rule, locate the rule and click Edit, Disable, or Enable in the Operation column.