Help Center > > User Guide> Configuring a Cluster> Customizing a Topology Cluster

Customizing a Topology Cluster

Updated at: Mar 25, 2021 GMT+08:00

The analysis cluster, streaming cluster, and hybrid cluster provided by MRS use fixed templates to deploy cluster processes. Therefore, you cannot customize service processes on management nodes and control nodes. If you want to customize the cluster deployment, set Cluster Type to Custom when creating a cluster. In this way, you can customize the deployment mode of process instances on the management nodes and control nodes in the cluster.

A custom cluster provides the following functions:

  • Separated deployment of the management and control roles: The management role and control role are deployed on different Master nodes.
  • Co-deployment of the management and control roles: The management and control roles are co-deployed on the Master node.
  • ZooKeeper is deployed on an independent node to improve reliability.
  • Components are deployed separately to avoid resource contention.

Roles in an MRS cluster:

  • Management Node (MN): is the node to install Manager (the management system of the MRS cluster). It provides a unified access entry. Manager centrally manages nodes and services deployed in the cluster.
  • Control Node (CN): controls and monitors how data nodes store and receive data, and send process status, and provides other public functions. Control nodes of MRS include HMaster, HiveServer, ResourceManager, NameNode, JournalNode, and SlapdServer.
  • Data Node (DN): A data node executes the instructions sent by the management node, reports task status, stores data, and provides other public functions. Data nodes of MRS include DataNode, RegionServer, and NodeManager.

Customizing a Cluster

  1. Log in to the MRS management console.
  2. Click Buy Cluster. The page for buying a cluster is displayed.
  3. On the page for buying a cluster, select Custom Config.
  4. Configure basic cluster information. For details about the parameters, see Software Configurations.

    • Region: Use the default value.
    • Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
    • Cluster Version: Currently, only MRS 1.9.3 and MRS 3.x are supported.
    • Cluster Type: Select Custom and select components as required.
    • Kerberos Authentication: Select whether to enable Kerberos authentication as required.
    • Username: name of the Manager administrator. admin is used by default.
    • Password: password of the Manager administrator.
    • Confirm Password: Enter the Manager administrator password again.

  5. Click Next. Configure hardware information.

    • Billing Mode: Use the default value.
    • AZ: Use the default value.
    • VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
    • Subnet: Use the default value.
    • Security Group: Select Auto create.
    • EIP: Select Bind later.
    • Enterprise Project: Use the default value.
    • CPU Architecture: Use the default value.
    • Common Node: For details, see Custom Cluster Template Description.
    • Instance Specifications: Click to configure the instance specifications, system disk and data disk storage types, and storage space.
    • Instance Count: Adjust the number of cluster instances based on the service volume.
    • Deploy Role: Click Deploy Role to display the components in the node group, nodes where the role is deployed, and the number of instances deployed on each node.

  6. Click Next and set advanced options.

    If the deployment mode in the Common Node does not meet the requirements, set Topology Adjustment to Enable and adjust the instance deployment mode based on service requirements. For details, see Topology Adjustment for a Custom Cluster. For more information about other parameters, see (Optional) Advanced Configuration.

  7. Click Buy Now.
  8. Click Back to Cluster List to view the cluster status.

    It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.

Custom Cluster Template Description

Table 1 Common templates for custom clusters

Common Node

Description

Node Range

Compact

The management role and control role are deployed on the Master node, and data instances are deployed in the same node group. This deployment mode applies to scenarios where the number of control nodes is less than 100, reducing costs.

  • The number of Master nodes is greater than or equal to 3 and less than or equal to 11.
  • The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.

OMS-separate

The management role and control role are deployed on different Master nodes, and data instances are deployed in the same node group. This deployment mode is applicable to a cluster with 100 to 500 nodes and delivers better performance in high-concurrency load scenarios.

  • The number of Master nodes is greater than or equal to 5 and less than or equal to 11.
  • The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.

Full-size

The management role and control role are deployed on different Master nodes, and data instances are deployed in different node groups. This deployment mode is applicable to a cluster with more than 500 nodes. Components can be deployed separately, which can be used for a larger cluster scale.

  • The number of Master nodes is greater than or equal to 9 and less than or equal to 11.
  • The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.

Topology Adjustment for a Custom Cluster

Table 2 Topology adjustment

Service

Dependency

Role

Role Deployment Suggestions

Description

OMSServer

-

OMSServer

This role can be deployed it on the Master node and cannot be modified.

-

ClickHouse

Depends on ZooKeeper.

ClickHouse

This role can be deployed on all nodes.

The number of role instances to be deployed must be an even number. The maximum number is 24.

A non-Master node group with this role assigned is considered as a Core node.

DBService

-

DBServer

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

-

ZooKeeper

-

QP(quorumpeer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1, 3, or 5

-

Hadoop

Depends on ZooKeeper.

NN(NameNode)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 or 2

The NameNode and ZKFC processes are deployed on the same server for cluster HA.

JN(JournalNode)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 or 3

-

DN(DataNode)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all node instances

A non-Master node group with this role assigned is considered as a Core node.

RM(ResourceManager)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 or 2

-

NM(NodeManager)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all node instances

-

JHS(JobHistoryServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 or 2

-

TLS(TimelineServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1

-

Spark2x

  • Depends on Hadoop.
  • Depends on Hive.

JS2x(JDBCServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2 to number of all Master nodes

-

JH2x(JobHistory)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

-

SR2x(SparkResource)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2 to number of all Master nodes

-

IS2X(IndexServer2x)

(Optional) This role can be deployed on the Master node only.

Number of role instances to be deployed: 0 to 2

-

HBase

Depends on Hadoop.

HM(HMaster)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

-

TS (ThriftServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to number of all Master nodes

-

RT(RESTServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to number of all Master nodes

-

RS(RegionServer)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all nodes

-

Hive

  • Depends on Hadoop.
  • Depends on DBService.

MS(MetaStore)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

-

WebHCat

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

-

HS(HiveServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to number of all Master nodes

-

Hue

Depends on DBService.

Hue

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

-

Loader

  • Depends on Hadoop.
  • Depends on DBService.

LS(Sqoop)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all nodes

-

Sqoop

Depends on Hadoop.

SS(SqoopClient)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all nodes

-

Kafka

Depends on ZooKeeper.

Broker

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all nodes

-

MirrorMaker

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all nodes

-

Storm

Depends on ZooKeeper.

Supervisor

This role can be deployed on all nodes.

Number of role instances to be deployed: 0 to number of all nodes

Logviewer is deployed on the same node as this role.

Nimbus

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

Web UI is deployed on the same node as this role.

Flume

-

MonitorServer

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

-

Flume

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all nodes

A non-Master node group with this role assigned is considered as a Core node.

Tez

Depends on Hive.

TezUI

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

-

Flink

-

FlinkResource

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to number of all nodes

-

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel