This section describes how to create an MRS cluster on the MRS console.
MRS consists of multiple big data components, and you can select the cluster type that best fits your service requirements, data types, reliability expectations, and resource budget.
You can quickly buy a cluster using the preset cluster template or select the component list and advanced settings to manually buy a cluster.
On the MRS console, a maximum of 10 clusters can be created concurrently, and a maximum of 100 clusters can be managed.
Video Tutorial
This tutorial demonstrates how to create an MRS cluster on the management console and configure software, hardware, and advanced settings.
The UI may vary depending on the version. This tutorial is for reference only.
Buying a Custom MRS Cluster
- Go to the Buy Cluster page.
- Click the Custom Config tab.
Pay attention to quota reminders when creating a cluster. If the resource quota is insufficient, request a quota increase as prompted before creating the cluster.
- In the Basic Configuration area, set basic information about the MRS cluster.
- Billing Mode: Billing mode of a cluster. MRS provides two billing modes: yearly/monthly and pay-per-use. For details about billing, see Billing Modes.
- Region: Region where the resource to be created is located. Resources in different regions cannot communicate with each other over the intranet. To improve access speed and reduce network latency, select the region nearest to you.
- Configure the MRS cluster information as prompted.
When creating a cluster, you need to configure the cluster version, network, node specifications and quantity, and other advanced settings.
- After the cluster information is configured, click Buy Now.
- For any doubt about the pricing, click Pricing details in the lower left corner.
- If you select the pay-per-use billing mode, the order may fail to be placed because the account may be in risk. In this case, contact technical support.
- Click Back to MRS Cluster List to view the cluster status.
Wait for the cluster creation to complete. The initial status of the cluster is Starting. After the cluster is created, the cluster status becomes Running.
Figure 1 MRS cluster running status
For details about cluster status during creation, see the description of the status parameters in Table 1 of Checking the Running Status of an MRS Cluster.
MRS Cluster Configuration
Table 1 MRS cluster parameters
Parameter |
Description |
Example Value |
Cluster Name |
The cluster name must be unique. It can contain 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed. |
mrs_test |
Cluster Type |
Select a proper MRS cluster type based on service requirements.
- Analysis cluster: It is used for offline data analysis and comprises data analysis tools like Hadoop, Spark, HBase, Hive, Flink, Oozie, and Tez.
- Streaming cluster: It processes streaming data to quickly analyze real-time data sources, and it mainly includes streaming data processing tools such as Kafka and Flume.
- Hybrid cluster: It is suitable for both offline data analysis and stream processing.
- Custom: You can select from a variety of components that are supported by the corresponding version of the MRS cluster.
|
Custom |
Version Type |
MRS provides two cluster version types: LTS and Normal. Different version types provide different components. You can select a version type as required.
- LTS: employs MRS's own components to provide highly reliable clusters with strong DR capabilities, making long-term support and evolution possible.
- Normal: integrates MRS's mature and stable features and functions with open-source capabilities, offering high performance and stability. Currently, MRS clusters of the normal version type cannot be purchased directly. If you want to use clusters of this type, submit a service ticket.
|
LTS |
Cluster Version |
Version of the MRS cluster. Different versions may contain different open-source component versions and functions. You are advised to select the latest version. For details about the lifecycle of each cluster version, see MRS Cluster Version Lifecycle. |
MRS 3.2.0-LTS.1 |
Component |
Select the components to be deployed in the MRS cluster.
You can change components based on your needs. For some clusters, components cannot be added after creation.
NOTE:
Hadoop components include HDFS, YARN, and MapReduce. Components used within a cluster, such as DBService, KrbServer, and LdapServer, are not displayed in the component list during cluster creation.
|
- |
Metadata |
Whether to use external data sources to store Hive and Ranger metadata of the cluster.
- Local: Metadata is stored in the local cluster.
- External data connection: Metadata of external data sources is used. If the cluster is abnormal or deleted, metadata is not affected. This mode applies to scenarios where storage and compute are decoupled.
When creating an MRS cluster, you can connect to LakeFormation instances to store metadata of components such as Hive and Spark. For details, see Configuring a LakeFormation Data Connection.
After a cluster is created, you can manually store the component metadata to external data sources. For details, see Managing MRS Cluster Metadata.
|
Local |
Component Port |
Default communication port policy of each component in the MRS cluster. The LTS cluster supports the configuration.
- Open source: Use the port provided by the open source component.
- Custom: Customize a port for the component.
For details about the differences between default open source port and default custom port, see Common Ports for MRS Cluster Services. |
Open source |
MRS Cluster Network Configuration
Table 2 Network configuration parameters
Parameter |
Description |
Example Value |
AZ |
AZ to which resources to be created belong in the current region. An AZ is a physical area where resources use independent power supply and networks. |
AZ1 |
VPC |
VPC to which the MRS cluster node belongs. If no VPC is available, click View VPC to access the network console and create a VPC.
For more information about VPCs, see What Is Virtual Private Cloud? |
- |
Subnet |
Subnet information in the VPC. If no subnet is available, click View Subnet to access the network console and create a subnet.
- A subnet provides dedicated network resources that are logically isolated from other networks for network security. For details about how to configure network ACL outbound rules, see How Do I Configure a Network ACL Outbound Rule?.
- The number of IP addresses required by creating an MRS cluster depends on the number of cluster nodes and selected components, but not the cluster type.
- In MRS, IP addresses are automatically assigned to clusters during cluster creation basically based on the following formula: Quantity of IP addresses = Number of cluster nodes + 2 (one for Manager; one for the DB). In addition, if the Hadoop, Hue, Sqoop, and Presto or Loader and Presto components are selected during cluster deployment, one IP address is added for each component. To buy a ClickHouse cluster separately, the number of IP addresses required is calculated as follows: Number of IP addresses = Number of cluster nodes + 1 (for FusionInsight Manager).
- For more information about subnets, see Subnet.
|
- |
Security Group |
A security group is a set of ECS access rules. It provides access policies for ECSs that have the same security protection requirements and are mutually trusted in a VPC.
When you create an MRS cluster, a security group is automatically created by default. You can also select an existing security group from the drop-down list.
- When you select a security group created by yourself, ensure that there is an inbound rule that allows traffic from all protocols, all ports, and the trusted accessible IP address range. Do not use 0.0.0.0/0 as a source address. Otherwise, security risks may occur. If you do not know the trusted accessible IP address range, select Auto create.
- For more information about security groups, see Security Group.
|
Auto create |
EIP |
After binding an EIP to an MRS cluster, you can use the EIP to access the Manager web UI of the cluster.
- When creating a cluster, you can select an available EIP from the drop-down list and bind it. If no EIP is available in the drop-down list, click Create EIPs to buy an EIP.
- The EIP must be in the same region as the cluster.
- For more information about EIPs, see What Is Elastic IP?
|
Bind later |
MRS Cluster Node Configuration
Table 3 Cluster node information
Parameter |
Description |
Example Value |
CPU Architecture |
CPU architecture of an MRS cluster node. The value can be x86 or Kunpeng. This parameter is not available for MRS 3.1.0 and 3.1.5. |
x86 |
Common Node Configurations |
This parameter is available only when Cluster Type is set to Custom. Value options include Compact, Full-size, and OMS-separate. For details, see MRS Cluster Deployment Types. |
Compact |
Node Group |
Name of the node group in the cluster.
- The name of the Master node group is fixed to master_node_default_group.
- The system automatically creates a Core node group based on the components contained in the cluster. For example, if you select the ClickHouse component, the system adds the ClickHouse node group and deploys the ClickHouseServer role in the node group by default.
|
node_group_1 |
Node Type |
If Cluster Type is set to Custom, you can select the node type of a non-Master node group. If the node group type is set to Task, only the NodeManager role, and certain mandatory node roles, can be deployed in the node group.
Nodes in a cluster can be classified into Master, Core, and Task nodes.
- Master node: MRS cluster management node. OMSServer is deployed on the node to manage and monitor the cluster.
- Core node: MRS cluster work node, which processes and analyzes data and stores process data.
- Task node: Compute node. When the compute resources in a cluster are insufficient, you can configure elastic scaling policies to increase nodes automatically.
For details about node types, see MRS Cluster Node Types. |
Core |
Payment Type |
Billing mode of nodes in a cluster.
- The billing mode of the Master and Core node groups is the same as that of the cluster.
- The billing mode of the Task node group is fixed to pay-per-use.
|
Pay-per-use |
Node Count |
Configure node quantity in each node group.
- The number of nodes in a master node group ranges from 3 to 9.
- There must be at least one Core node and the total number of Core and Task nodes must not exceed 10,000.
- If Cluster Type is set to Custom, you can click Add Node Group to add node groups.
A small number of nodes may cause clusters to run slowly while a large number of nodes may be unnecessarily costly. Set an appropriate value based on data to be processed. |
- |
Instance Specifications |
Select the instance specifications of the MRS cluster node. You can click to adjust the specifications.
- For details about the MRS cluster node specifications, see MRS Cluster Node Specifications.
- More advanced instance specifications provide better data processing. However, they require higher cluster cost.
- Instance specifications may vary in different AZs. If no instance specifications in the current AZ can meet your requirements, switch to another AZ.
- If you select HDDs for Core nodes, there is no billing information for data disks. The fees are charged with ECSs.
- If you select non-HDD disks for Core nodes, the disk types of Master and Core nodes are determined by Data Disk.
- If Sold out appears next to an instance specification of a node, the node of this specification cannot be bought. You can only buy nodes of other specifications.
- The Master node specification (4 vCPUs and 8 GB memory) is not within the after-sales SLA. It is applicable only to the test environment and is not recommended for the production environment.
- For MRS 3.x or later, the memory of the Master node must be greater than 64 GB.
|
- |
System Disk |
You can adjust the storage type and space of the system disk on a node as required. For details about the MRS cluster storage, see Cluster Node Disk Types. |
- |
Data Disk |
Storage type and space of data disks on a node. To increase the data storage capacity, you can add disks during cluster creation. A maximum of 10 disks can be added to each Core or Task node. For more information about MRS cluster storage, see Cluster Node Disk Types. |
- |
LVM |
This parameter is valid when a streaming Core node is created only. Click this parameter to enable or disable the disk LVM management function. This parameter is not available in MRS 3.x and later versions.
If LVM is enabled, all disks on a node are mounted as logical volumes. This delivers more proper disk planning to avoid data skew, thereby improving system stability. |
Disable this function. |
Topology Adjustment |
If Cluster Type is set to Custom, you can adjust the deployment of each component in the cluster in the node group.
Set Topology Adjustment to Enable and adjust the instance deployment mode based on service requirements. For details, see Role Deployment Rules for MRS Clusters. |
Disable this function. |
Other MRS Cluster Configuration Parameters
Table 4 Other configuration parameters
Parameter |
Description |
Example Value |
Kerberos Authentication |
Whether to enable Kerberos authentication for each component in the MRS cluster. If Kerberos authentication is enabled, users can access component resources only after being authenticated.
This option cannot be changed after you buy a cluster. |
Kerberos authentication enabled |
Username |
Name of the administrator of Manager. admin is used by default. |
admin |
Password/Confirm Password |
Password of the Manager administrator admin. Keep the password secure.
- Must contain 8 to 26 characters.
- Must contain at least four of the following:
- Lowercase letters
- Uppercase letters
- Digits
- At least one of the following special characters: `~!@#$%^&*()-_=+|[{}];:',<.>/?
- Cannot be the same as the username or the username spelled backwards.
|
- |
Login Mode |
Method for logging in to a node in the MRS cluster.
- Password
You can log in to the node as user root using a password. You need to customize the password of user root.
- Key Pair
Select a key pair from the drop-down list. Select "I acknowledge that I have obtained private key file SSHkey-xxx and that without this file I will not be able to log in to my ECS." Click View Key Pair to create or import a key pair, and then obtain the private key file.
|
Password |
Kerberos Encryption Type |
Encryption algorithm and method used by Kerberos. (This parameter is supported in MRS 3.3.1-LTS and later versions. In earlier versions, the default Kerberos encryption type is aes256-sha1,aes128-sha1.)
- aes256-sha1,aes128-sha1: indicates that the encryption algorithm and mode are AES256-CTS-HMAC-SHA1-96 AES128-CTS-HMAC-SHA1-96.
- aes256-sha2,aes128-sha2: indicates that the encryption algorithm and mode are AES256-CTS-HMAC-SHA384-192 AES128-CTS-HMAC-SHA256-128.
When setting this parameter, note the following:
- If the encryption types of two clusters are different, mutual trust cannot be set up across the FusionInsight Manager clusters.
- If you are using an external Java program to connect to the MRS cluster for Kerberos authentication and the Kerberos encryption type is aes256-sha2,aes128-sha2, the external Java program must use JDK 11 (or BiSheng JDK 1.8.0_392) or later.
- To use an instance created in DataArts Studio to connect to the MRS cluster, set the Kerberos encryption type to aes256-sha1,aes128-sha1.
|
aes256-sha2,aes128-sha2 |
Set Advanced Options |
Advanced function parameters of the MRS cluster. For details, see Table 5. |
- |
Enterprise Project |
Select the enterprise project to which the cluster belongs. To use an enterprise project, create one on the Enterprise > Project Management page.
The Enterprise Management console is designed for resource management. It helps you manage cloud-based personnel, resources, permissions, and finance in a hierarchical manner, such as management of companies, departments, and projects. |
default |
Required Duration |
This parameter is valid in Yearly/Monthly billing mode and indicates a cluster subscription duration. The minimum cluster duration is 1 month and the maximum available cluster duration is 1 year.
If Auto-renew is selected, monthly subscriptions are automatically renewed every month and yearly subscriptions are automatically renewed every year. |
- |
Secure Communications |
To allow the MRS console to access big data components in your VPC, you need to enable relevant security group rules to allow traffic to pass. For details, see Configuring Secure Communication Authorization for an MRS Cluster. |
Select the check box. |
Table 5 MRS cluster advanced configuration
Parameter |
Description |
Example Value |
Hostname Prefix |
Enter the prefix for the computer hostname of an ECS or BMS in the cluster. |
- |
Cryptographic Algorithm |
Algorithm used for encrypting and decrypting passwords in the cluster system.
- International: general cryptographic algorithm
- Chinese: SM series cryptographic algorithms are compatible with general cryptographic algorithms.
|
International |
Tag |
It is recommended that you use the tag function of TMS to add the same tag to different cloud resources. For details, see Adding a Tag to an MRS Cluster/Node.
For more information about TMS, see What Is Tag Management Service? |
- |
Auto Scaling |
Configure an auto scaling policy for the Task node group after the cluster is created. For details, see MRS Task Node Auto Scaling. |
- |
Bootstrap Action |
Bootstrap actions run scripts on specified nodes in a cluster to install third-party software and modify the cluster's running environment. For details, see Adding MRS Node Bootstrap Actions and Installing Third-Party Software. |
- |
Agency |
When an agency is bound to an ECS or BMS, it can manage designated resources in the cluster. It is essential to consider service requirements before deciding to configure an agency.
For example, you can configure an ECS agency to automatically obtain the AK/SK to access OBS. For details, see Interconnecting an MRS Cluster with OBS Using an IAM Agency.
To bind an agency to an MRS cluster, you need to create an IAM agency with required permissions in advance. By default, the system generates an MRS_ECS_DEFAULT_AGENCY agency. This agency has the OBSOperateAccess permission and the CESFullAccess (for users who have enabled fine-grained policies), CES Administrator, and KMS Administrator permissions in the region where the cluster is located. |
Bind later |
Metric Sharing |
Monitoring metrics of big data components are collected. If a fault occurs when you use a cluster, share the monitoring metrics with Huawei Cloud technical support for troubleshooting. |
Disable |
System Disk Encryption |
Whether to encrypt data in the system disk mounted to the MRS cluster node. This function is disabled by default. This parameter is available only in MRS 3.3.0-LTS or later.
Keys used by encrypted system disks are provided by Key Management Service (KMS) in Data Encryption Workshop (DEW). You do not need to build and maintain the key management infrastructure. To enable this function, you must have the Security Administrator and KMS Administrator permissions.
For more information about disk encryption, see Managing Encrypted EVS Disks.
For more information about DEW, see What Is DEW?
Configure the following parameters to enable this function:
- Data Disk Key ID: Key ID of the selected key name.
- Data Disk Key Name: Select the name of the key used to encrypt the data disk. By default, the default master key named evs/default is selected. You can select another master key from the drop-down list.
If cloud disks are encrypted using a master key and the key is then disabled or scheduled for deletion, the disks can no longer be read from or written to, and data on these disks may never be recovered. Exercise caution when performing relevant operations.
Click View the Key List to enter a page where you can create and manage keys.
|
Disable |
Data Disk Encryption |
Whether to encrypt data in the data disk mounted to the MRS cluster node. This function is disabled by default.
For more information about disk encryption, see Managing Encrypted EVS Disks.
Keys used by encrypted system disks are provided by Key Management Service (KMS) in Data Encryption Workshop (DEW). You do not need to build and maintain the key management infrastructure. To enable this function, you must have the Security Administrator and KMS Administrator permissions.
Configure the following parameters to enable this function:
- Data Disk Key ID: Key ID of the selected key name.
- Data Disk Key Name: Select the name of the key used to encrypt the data disk. By default, the default master key named evs/default is selected. You can select another master key from the drop-down list.
If cloud disks are encrypted using a master key and the key is then disabled or scheduled for deletion, the disks can no longer be read from or written to, and data on these disks may never be restored. Exercise caution when performing this operation.
Click View the Key List to enter a page where you can create and manage keys.
|
Disable |
Alarm |
If the alarm function is enabled, the cluster maintenance personnel can be notified in a timely manner to locate faults when the cluster runs abnormally or the system is faulty. To send alarm messages, you need to enable the Simple Message Notification (SMN) service, set a notification rule, and bind the rule to an SMN topic.
|
Enable |
Install UniAgent |
Install UniAgent on MRS cluster nodes to simplify plug-in management, provide AOM instructions, and enable script delivery and execution. |
No |
Logging |
Whether to collect logs when cluster creation fails.
After the logging function is enabled, system logs and component run logs are automatically collected and saved to the OBS file system in scenarios such as cluster creation failures and scale-out or scale-in failures for O&M personnel to quickly locate faults.
The log information is retained for a maximum of seven days. |
Disable |
FAQs
The following lists the questions frequently asked during MRS cluster creation and their possible causes:
How Do I Manually Add a Task Node Group During MRS Cluster Creation
To add a Task node group in the current MRS cluster for auto scaling, do as follows:
Add an analysis Task node group.
- For a cluster whose Cluster Type is Analysis cluster or Hybrid cluster:
- In the cluster node configuration area, click Add Analysis Task Node Group. The node group name is task_node_analysis_group by default.
- Set the number of nodes, instance specifications, and node disk configuration as required.
- For a cluster whose Cluster Type is Custom:
- In the cluster node configuration area, click Add Node Group to manually add a node group.
- Set the node type to Task.
- Set the number of nodes, instance specifications, and node disk configuration as required.
- Set Topology Adjustment to Enable.
- Select the role topology so that the node group contains only the NodeManager (NM) role.
Adding streaming Task node group
For a streaming or hybrid cluster that contains the Storm component:
The system automatically adds the streaming Task node group task_node_streaming_group. Set the number of nodes, instance specifications, and node disk configuration as required.
If the streaming Task node group is not required, you can manually delete it.
How Do I View Details About Failed MRS Cluster Creation Tasks
If a cluster fails to be created, the failed task will be managed on the Manage Failed Tasks page.
On the cluster list page, click
to switch to the Manage Failed Tasks page. In the Task Status column, hover the cursor over the task status to view the failure cause.
Table 6 lists the error codes of MRS cluster creation failures.
Table 6 Error codes
Error Code |
Description |
MRS.101 |
Insufficient quota to meet your request. Contact customer service to increase the quota. |
MRS.102 |
The token cannot be null or invalid. Try again later or contact customer service. |
MRS.103 |
Invalid request. Try again later or contact customer service. |
MRS.104 |
Insufficient resources. Try again later or contact customer service. |
MRS.105 |
Insufficient IP addresses in the existing subnet. Try again later or contact customer service. |
MRS.201 |
Failed due to an ECS error. Try again later or contact customer service. |
MRS.202 |
Failed due to an IAM error. Try again later or contact customer service. |
MRS.203 |
Failed due to a VPC error. Try again later or contact customer service. |
MRS.400 |
MRS system error. Try again later or contact customer service. |
Helpful Links
- After an MRS cluster is created, you can install the MRS cluster client to connect and use the components in the cluster. For details, see Installing an MRS Cluster Client.
- For details about how to submit and run jobs in an MRS cluster, see Submitting an MRS Job.
- You can log in to FusionInsight Manager of a cluster to view cluster resource status, alarms, and other information. For details, see Accessing MRS Manager.