Interconnecting an MRS Cluster with OBS Using an IAM Agency

Creating an ECS Agency with OBS Access Permissions

• MRS presets MRS_ECS_DEFAULT_AGENCY in the IAM agency list by default, allowing you to choose this agency when creating a cluster. This agency has OBS OperateAccess permission and, for users with fine-grained policies enabled, CES FullAccess, CES Administrator, and KMS Administrator permissions in the region where the cluster is located. Do not modify MRS_ECS_DEFAULT_AGENCY on IAM.
• If you want to use the preset agency, skip the step for creating an agency. If you want to use a custom agency, perform the following steps to create an agency. (To create or modify an agency, you must have the Security Administrator permission.) If you need to have more fine-grained control over the permissions of a specific path in the OBS file system, you can refer to Configuring Fine-Grained OBS Access Permissions for MRS Cluster Users to create a custom role policy.

1. Log in to the Huawei Cloud management console.
2. In the service list, choose Management & Governance > Identity and Access Management.
3. Choose Agencies. On the displayed page, click Create Agency.
4. Set Agency Name. For example, enter mrs_ecs_obs.
5. Set Agency Type to Cloud service and select ECS BMS to authorize ECS or BMS to invoke OBS. See Figure 1.
6. Set Validity Period to Unlimited and click Done.
   Figure 1 Creating an agency
   
7. In the displayed dialog box shown in Figure 2, click Authorize. Search for OBS OperateAccess and select it.

   If KMS encryption is configured for an OBS parallel file system, the KMS Administrator policy must be selected.

   Figure 2 Configuring permissions
   
8. Click Next. On the page that is displayed, select the desired scope for the permissions you selected. By default, All resources is selected. Click Show More, select Global resources, and click OK.
9. In the dialog box that is displayed, click OK to start authorization. After the message "Authorization successful." is displayed, click Finish. The agency is successfully created.

Creating a Decoupled Storage and Compute Cluster

You can configure an agency when creating a cluster or bind an agency to an existing cluster to decouple storage and compute. This section uses a cluster with Kerberos authentication enabled as an example.

Configuring an agency when creating a cluster:

1. Go to the Buy Cluster page.
2. Click Buy Cluster. The page for buying a cluster is displayed.
3. Click the Custom Config tab.
4. On the Custom Config page, set cluster parameters based on your service requirements. (The following operations show how to create a pay-per-use MRS 3.2.0-LTS.1 cluster in custom config mode. For details about more parameter settings, see Buying a Custom MRS Cluster).

   Table 1 Parameters for a custom MRS cluster

   Parameter	Description	Example Value
   Billing Mode	Billing mode of a cluster. MRS provides two billing modes: yearly/monthly and pay-per-use. If you select Pay-per-use, a certain deposit will be frozen. For details, see Billing Overview.	Pay-per-use
   Region	Region where the resource to be created is located. Resources in different regions cannot communicate with each other over the intranet. To improve access speed and reduce network latency, select the region nearest to you.	-
   Cluster Name	MRS cluster name. You can use the default name. However, you are advised to include a project name abbreviation or date in the cluster name for remembering and distinguishing. After a cluster is created, you can change the cluster name in the cluster list.	mrs-test
   Cluster Type	Select a required MRS cluster type based on service requirements. Analysis cluster: It is well-suited for analyzing and processing large amounts of offline data to obtain results. Recommended components include Hadoop, Spark, HBase, Hive, Flink, Oozie, and Tez. Streaming cluster: It is ideal for streaming data processing and quick analysis of real-time data sources. Recommended components include Kafka and Flume. Hybrid cluster: It is suitable for both offline data analysis and stream processing. Custom: A wide range of components are available. You can select supported components as required based on your MRS cluster version.	Custom
   Version Type	MRS provides two cluster version types: LTS and Normal. Different version types provide different components. You can select a version type as required. LTS: employs MRS's own components to provide highly reliable clusters with strong DR capabilities, making long-term support and evolution possible. Normal: integrates MRS's mature and stable features and functions with open-source capabilities, offering high performance and stability.	LTS
   Cluster Version	Version of the MRS cluster. Different versions may contain different open-source component versions and functions. You are advised to select the latest version. For details about the lifecycle of each cluster version, see MRS Cluster Version Lifecycle.	MRS 3.2.0-LTS.1
   Component	Cluster templates containing preset open-source components you will need for your business. Components that can connect to OBS through IAM agencies include Hadoop, Hive, Spark, Presto, Sqoop, Flink, and Flume. Select the components based on your requirements. NOTE: Hadoop components include HDFS, YARN, and MapReduce. Components used within a cluster, such as DBService, KrbServer, and LdapServer, are not displayed in the component list during cluster creation.	Hadoop and Hive
   Metadata	Whether to use external data sources to store Hive and Ranger metadata of the cluster. Local: Metadata is stored in the local cluster. External data connection: Metadata of external data sources is used. If the cluster is abnormal or deleted, metadata is not affected. This mode applies to scenarios where storage and compute are decoupled.	Local
   AZ	AZ to which resources to be created belong in the current region. An AZ is a physical area where resources use independent power supply and networks.	AZ1
   VPC	VPC to which the MRS cluster node belongs. If no VPC is available, click View VPC to access the network console and create a VPC. For more information about VPCs, see What Is Virtual Private Cloud?	-
   Subnet	Subnet information in the VPC. If no subnet is available, click View Subnet to access the network console and create a subnet. For more information about subnets, see Subnet.	-
   Security Group	A security group is a set of ECS access rules. It provides access policies for ECSs that have the same security protection requirements and are mutually trusted in a VPC. When you create an MRS cluster, a security group is automatically created by default. You can also select an existing security group from the drop-down list.	Auto create
   CPU Architecture	CPU architecture of an MRS cluster node. The value can be x86 or Kunpeng. This parameter is not available for MRS 3.1.0 and 3.1.5.	x86
   Common Node Configurations	This parameter is available only when Cluster Type is set to Custom. Common Node Configurations can be Compact, Full-size, or OMS-separate. Select a required one by referring to MRS Cluster Deployment Types.	Compact
   Cluster Nodes	Specifications and quantity of nodes in an MRS cluster. For MRS 3.x or later, the memory of a Master node must be greater than 64 GB.	Set the number of cluster nodes and node specifications as required.
   Kerberos Authentication	Whether to enable Kerberos authentication for each component in the MRS cluster. If Kerberos authentication is enabled, users can access component resources only after being authenticated. This option cannot be changed after you buy a cluster.	Kerberos authentication enabled
   Username	Default user for logging in to MRS Manager and nodes in the MRS cluster. User admin is used to log in to MRS Manager, while user root is the OS user used to log in to the nodes in the cluster.	-
   Password/Confirm Password	Password of the MRS Manager administrator admin. Keep the password secure. Must contain 8 to 26 characters. Must contain every type of the following: Lowercase letters Uppercase letters Digits Special characters: `~!@#$%^&*()-_=+|[{}];:',<.>/? Cannot be the same as the username or the username spelled backwards.	-
   Login Mode	Mode for logging in to a node in the MRS cluster. Password Use a password to log in to the node as user root. You need to set the password of user root. Key Pair Use a key pair to log in to the node. Select a key pair from the drop-down list. If you have obtained the private key file, select I acknowledge that I have obtained private key file SSHkey-xxx and that without this file I will not be able to log in to my ECS. If no key pair is available, click View Key Pair to create or import a key pair, and then obtain the private key file.	Password
   Advanced Configuration	Enable Advanced Configuration and set Agency. After an agency is bound, the ECS or BMS services will have the permission to manage some resources in the cluster. Click Available agencies and select the agency created in Creating an ECS Agency with OBS Access Permissions from the drop-down list. Select the MRS_ECS_DEFAULT_AGENCY agency preset by MRS in IAM.	MRS_ECS_DEFAULT_AGENCY
   Enterprise Project	An enterprise project facilitates project-level management and grouping of cloud resources and users. You can select the system-defined enterprise project default or create a new one.	default
   Secure Communications	To allow the MRS console to access big data components in your VPC, you need to enable relevant security group rules to allow traffic to pass. For details, see Configuring Secure Communication Authorization for an MRS Cluster.	Select the check box.

5. Click Buy Now and wait until the cluster is created.

Configuring an agency for an existing cluster:

1. Log in to the MRS console. In the navigation pane on the left, choose Active Clusters.
2. Click the name of the cluster to enter its details page.
3. On the Dashboard page, click Synchronize on the right of IAM User Sync to synchronize IAM users.
4. On the Dashboard tab page, click Select Agency on the right side of Agency to select an agency and click OK to bind it. Alternatively, click Create New to go to the IAM console to create an agency and select it.

Creating an OBS Parallel File System for Storing Service Data

In decoupled storage-compute scenarios, make sure to use an OBS parallel file system. (For details, see Parallel File System). Using an OBS bucket can significantly affect the performance of the cluster.

1. Log in to the OBS Console.
2. Choose Parallel File Systems > Create Parallel File System.
3. Enter the file system name, for example, mrs-word001.

   Set other parameters as required.

   Figure 3 Creating an OBS parallel file system
   
4. Click Create Now.
5. In the parallel file system list on the OBS console, click the file system name to go to the details page.
6. In the navigation pane, choose Files and create the program and input folders.
   • program: Upload the program package to this folder.
   • input: Upload the input data to this folder.

Creating a Lifecycle Rule

In MRS 3.2.0-LTS.1 and later versions, components prevent mis-deletion by default. That is, file data deleted by component users is not directly deleted but stored in the recycle bin directory in the OBS file system.

To save OBS space, you need to enable periodical deletion of file data from the OBS recycle bin by referring to Configuring the Policy for Clearing Recycle Bin Directories of MRS Cluster Components.