Overview

Updated on 2024-09-23 GMT+08:00

View PDF

The analysis cluster, streaming cluster, and hybrid cluster provided by MRS use fixed templates to deploy cluster processes. Therefore, you cannot customize service processes on management nodes and control nodes.

If you want to customize the cluster deployment, set Cluster Type to Custom when creating a cluster. In this way, you can customize the deployment mode of process instances on the management nodes and control nodes in the cluster.

NOTE:

Only MRS 3.x and later versions support the creation of clusters in a custom topology.

A custom cluster provides the following functions:

Separated deployment of the management and control roles: The management role and control role are deployed on different Master nodes.
Co-deployment of the management and control roles: The management and control roles are co-deployed on the Master node.
Components are deployed separately to avoid resource contention.

MRS Cluster Deployment Types

**Table 1** MRS cluster deployment types
Common Node	Description	Node Range
Compact	The management role and control role are deployed on the Master node, and data instances are deployed in the same node group. This deployment mode applies to scenarios where the number of control nodes is less than 100, reducing costs.	The number of Master nodes is greater than or equal to 3 and less than or equal to 11. The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.
Co-deployment of the management role, control role, and data instances	This template is not recommended in the production environment or commercial environment. If management, control, and data nodes are deployed together, cluster performance and reliability may deteriorate. If there are enough nodes, deploy data nodes separately.	There must be at least 3 master nodes and 100 at most. The total node groups are no more than 10.
OMS-separate	The management role and control role are deployed on different Master nodes, and data instances are deployed in the same node group. This deployment mode is applicable to a cluster with 100 to 500 nodes and delivers better performance in high-concurrency load scenarios.	The number of Master nodes is greater than or equal to 5 and less than or equal to 11. The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.
Full-size	The management role and control role are deployed on different Master nodes, and data instances are deployed in different node groups. This deployment mode is applicable to a cluster with more than 500 nodes. Components can be deployed separately, which can be used for a larger cluster scale.	The number of Master nodes is greater than or equal to 9 and less than or equal to 11. The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.

Role Deployment Rules for MRS Clusters

The MRS system consists of multiple services in logical architecture. Each service contains one or more roles. Each role can deploy one or more instances.

Service: A service is a kind of service capabilities provided by a component in a cluster. Each component in the cluster provides a service and is named by the service.
Role: Roles are components of a service. Each service consists of one or more roles. Services are deployed on nodes (servers) through roles to ensure normal service running.
Instance: An instance is formed when a service role is installed on a node. A service has one or more role instances.

For details about the deployment principles of each service, see Overview.

Service A depending on service B: If service A is deployed in a cluster, service B must be deployed in advance. Service B provides basic capabilities for service A.
Service A associated with service B: Service A exchanges data with service B during service running but does not depend on service B during deployment.
Role A and role B deployed on the same server: If role A is deployed in a cluster, role B must also be deployed, and role A and role B must be deployed on the same node.

**Table 2** MRS role deployment rules
Service	Dependency	Role	Role Deployment Suggestions
OMSServer	-	OMSServer	This role can be deployed it on the Master node and cannot be modified.
ClickHouse	Depends on ZooKeeper.	CHS (ClickHouseServer)	This role can be deployed on all nodes. You can deploy 2 to 256 role instances, but the number must be even.
ClickHouse	Depends on ZooKeeper.	CLB (ClickHouseBalancer)	This role can be deployed on all nodes. Number of role instances to be deployed: 2 to 256
Flink	Depends on ZooKeeper. Depends on KrbServer. Depends on DBService. Depends on Hadoop.	FR(FlinkResource)	This role can be deployed on all nodes. Number of role instances to be deployed: 1 to 10,000
Flink		FS(FlinkServer)	This role can be deployed on all nodes. Number of role instances to be deployed: 0 to 2
Flume	-	MS(MonitorServer)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1 to 2
Flume	-	F(Flume)	This role can be deployed on all nodes. Number of role instances to be deployed: 1 to 10,000
Hadoop	Depends on ZooKeeper.	NN(NameNode)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2
		HFS (HttpFS)	This role can be deployed on the Master node only. Number of role instances to be deployed: 0 to 10
		JN(JournalNode)	This role can be deployed on the Master node only. Number of role instances to be deployed: 3 to 60, with the step size of 2
		DN(DataNode)	This role can be deployed on all nodes. Number of role instances to be deployed: 3 to 10,000
		RM(ResourceManager)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2
		NM(NodeManager)	This role can be deployed on all nodes. Number of role instances to be deployed: 3 to 10,000
		JHS(JobHistoryServer)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1 to 2
		TLS(TimelineServer)	This role can be deployed on the Master node only. Number of role instances to be deployed: 0 to 1
HBase	Depends on Hadoop.	HM(HMaster)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2
		TS(ThriftServer)	This role can be deployed on all nodes. Number of role instances to be deployed: 0 to 10,000
		RT(RESTServer)	This role can be deployed on all nodes. Number of role instances to be deployed: 0 to 10,000
		RS(RegionServer)	This role can be deployed on all nodes. Number of role instances to be deployed: 3 to 10,000
		TS1(Thrift1Server)	This role can be deployed on all nodes. Number of role instances to be deployed: 0 to 10,000 If the Hue service is installed in a cluster and HBase needs to be used on the Hue web UI, install this instance for HBase.
HetuEngine	Depends on Hadoop. Depends on DBService. Depends on Hive. Depends on ZooKeeper. Depends on KrbServer. Depends on Yarn. Depends on HDFS.	HSB(HSBroker)	This role can be deployed on all nodes. Number of role instances to be deployed: 2 to 50
		HSC(HSConsole)	This role can be deployed on all nodes. Number of role instances to be deployed: 2
		HSF(HSFabric)	This role can be deployed on all nodes. Number of role instances to be deployed: 0 to 50
		QAS (available for MRS 3.2.0-LTS.1 and later versions only)	This role can be deployed on all nodes. Number of role instances to be deployed: 0 to 2
Hive	Depends on Hadoop. Depends on DBService.	MS(MetaStore)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2 to 10
		WH (WebHCat)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1 to 10
		HS(HiveServer)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2 to 80
Hue	Depends on DBService.	H(Hue)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2
Impala	Depends on Hadoop. Depends on Hive. Depends on DBService. Depends on ZooKeeper.	StateStore	This role can be deployed on the Master node only. Number of role instances to be deployed: 1
		Catalog	This role can be deployed on the Master node only. Number of role instances to be deployed: 1
		Impalad	This role can be deployed on all nodes. Number of role instances to be deployed: 1 to 10,000
IoTDB	Depends on KrbServer.	ConfigNode (CN)	This role can be deployed on Master nodes only. Number of role instances to be deployed: 3 to 9, with the step size of 2
IoTDB	Depends on KrbServer.	IoTDBServer (IoTDBS)	This role can be deployed on all nodes. Number of role instances to be deployed: 3 to 256
Kafka	Depends on ZooKeeper.	B(Broker)	This role can be deployed on all nodes. Number of role instances to be deployed: 3 to 10,000
Kudu	-	KuduMaster	This role can be deployed on the Master node only. Number of role instances to be deployed: 3 or 5
Kudu	-	KuduTserver	This role can be deployed on all nodes. Number of role instances to be deployed: 3 to 10,000
Loader	Depends on Hadoop. Depends on DBService.	LS(LoaderServer)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2
Oozie	Depends on Hadoop. Depends on DBService. Depends on ZooKeeper.	O(oozie)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2
Presto	Depends on Hive.	PCD(Coordinator)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2
Presto	Depends on Hive.	PWK(Worker)	This role can be deployed on all nodes. Number of role instances to be deployed: 1 to 10,000
Ranger	Depends on DBService.	RA(RangerAdmin)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1 to 2
		USC(UserSync)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1
		TSC (TagSync)	This role can be deployed on all nodes. Number of role instances to be deployed: 0 to 1
Spark	Depends on Hadoop. Depends on Hive. Depends on ZooKeeper.	JS(JDBCServer)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1 to 2
		JH(JobHistory)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1 to 2
		SR(SparkResource)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1 to 2
Spark2x	Depends on Hadoop. Depends on Hive. Depends on ZooKeeper.	JS2X(JDBCServer2x)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2 to 10
		JH2X(JobHistory2x)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2
		SR2X(SparkResource2x)	This role can be deployed on the Master node only. Number of role instances to be deployed: 2 to 50
		IS2X(IndexServer2x)	(Optional) This role can be deployed on the Master node only. Number of role instances to be deployed: 0 to 2, with the step size of 2
Sqoop	Depends on Hadoop.	SC(SqoopClient)	This role can be deployed on all nodes. Number of role instances to be deployed: 1 to 10,000
Tez	Depends on Hadoop. Depends on DBService. Depends on ZooKeeper.	TUI(TezUI)	This role can be deployed on the Master node only. Number of role instances to be deployed: 1 to 2
ZooKeeper	-	QP(quorumpeer)	This role can be deployed on the Master node only. Number of role instances to be deployed: 3 to 9, with the step size of 2