Help Center/ MapReduce Service/ User Guide/ MRS Cluster Planning/ MRS Cluster Deployment/ Overview
Updated on 2024-09-23 GMT+08:00
View PDF
Share

Overview

The analysis cluster, streaming cluster, and hybrid cluster provided by MRS use fixed templates to deploy cluster processes. Therefore, you cannot customize service processes on management nodes and control nodes.

If you want to customize the cluster deployment, set Cluster Type to Custom when creating a cluster. In this way, you can customize the deployment mode of process instances on the management nodes and control nodes in the cluster.

Only MRS 3.x and later versions support the creation of clusters in a custom topology.

A custom cluster provides the following functions:

  • Separated deployment of the management and control roles: The management role and control role are deployed on different Master nodes.
  • Co-deployment of the management and control roles: The management and control roles are co-deployed on the Master node.
  • Components are deployed separately to avoid resource contention.

MRS Cluster Deployment Types

Table 1 MRS cluster deployment types

Common Node

Description

Node Range

Compact

The management role and control role are deployed on the Master node, and data instances are deployed in the same node group. This deployment mode applies to scenarios where the number of control nodes is less than 100, reducing costs.
  • The number of Master nodes is greater than or equal to 3 and less than or equal to 11.
  • The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.

Co-deployment of the management role, control role, and data instances

This template is not recommended in the production environment or commercial environment.

If management, control, and data nodes are deployed together, cluster performance and reliability may deteriorate.

If there are enough nodes, deploy data nodes separately.
  • There must be at least 3 master nodes and 100 at most.
  • The total node groups are no more than 10.

OMS-separate

The management role and control role are deployed on different Master nodes, and data instances are deployed in the same node group. This deployment mode is applicable to a cluster with 100 to 500 nodes and delivers better performance in high-concurrency load scenarios.
  • The number of Master nodes is greater than or equal to 5 and less than or equal to 11.
  • The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.

Full-size

The management role and control role are deployed on different Master nodes, and data instances are deployed in different node groups. This deployment mode is applicable to a cluster with more than 500 nodes. Components can be deployed separately, which can be used for a larger cluster scale.
  • The number of Master nodes is greater than or equal to 9 and less than or equal to 11.
  • The total number of node groups is less than or equal to 10, and the total number of nodes in non-Master node groups is less than or equal to 10,000.

Role Deployment Rules for MRS Clusters

The MRS system consists of multiple services in logical architecture. Each service contains one or more roles. Each role can deploy one or more instances.

  • Service: A service is a kind of service capabilities provided by a component in a cluster. Each component in the cluster provides a service and is named by the service.
  • Role: Roles are components of a service. Each service consists of one or more roles. Services are deployed on nodes (servers) through roles to ensure normal service running.
  • Instance: An instance is formed when a service role is installed on a node. A service has one or more role instances.

For details about the deployment principles of each service, see Overview.

  • Service A depending on service B: If service A is deployed in a cluster, service B must be deployed in advance. Service B provides basic capabilities for service A.
  • Service A associated with service B: Service A exchanges data with service B during service running but does not depend on service B during deployment.
  • Role A and role B deployed on the same server: If role A is deployed in a cluster, role B must also be deployed, and role A and role B must be deployed on the same node.
Table 2 MRS role deployment rules

Service

Dependency

Role

Role Deployment Suggestions

OMSServer

-

OMSServer

This role can be deployed it on the Master node and cannot be modified.

ClickHouse

Depends on ZooKeeper.

CHS (ClickHouseServer)

This role can be deployed on all nodes.

You can deploy 2 to 256 role instances, but the number must be even.

CLB (ClickHouseBalancer)

This role can be deployed on all nodes.

Number of role instances to be deployed: 2 to 256

Flink
  • Depends on ZooKeeper.
  • Depends on KrbServer.
  • Depends on DBService.
  • Depends on Hadoop.

FR(FlinkResource)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to 10,000

FS(FlinkServer)

This role can be deployed on all nodes.

Number of role instances to be deployed: 0 to 2

Flume

-

MS(MonitorServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

F(Flume)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to 10,000

Hadoop

Depends on ZooKeeper.

NN(NameNode)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2

HFS (HttpFS)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 0 to 10

JN(JournalNode)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 3 to 60, with the step size of 2

DN(DataNode)

This role can be deployed on all nodes.

Number of role instances to be deployed: 3 to 10,000

RM(ResourceManager)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2

NM(NodeManager)

This role can be deployed on all nodes.

Number of role instances to be deployed: 3 to 10,000

JHS(JobHistoryServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

TLS(TimelineServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 0 to 1

HBase

Depends on Hadoop.

HM(HMaster)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2

TS(ThriftServer)

This role can be deployed on all nodes.

Number of role instances to be deployed: 0 to 10,000

RT(RESTServer)

This role can be deployed on all nodes.

Number of role instances to be deployed: 0 to 10,000

RS(RegionServer)

This role can be deployed on all nodes.

Number of role instances to be deployed: 3 to 10,000

TS1(Thrift1Server)

This role can be deployed on all nodes.

Number of role instances to be deployed: 0 to 10,000

If the Hue service is installed in a cluster and HBase needs to be used on the Hue web UI, install this instance for HBase.

HetuEngine
  • Depends on Hadoop.
  • Depends on DBService.
  • Depends on Hive.
  • Depends on ZooKeeper.
  • Depends on KrbServer.
  • Depends on Yarn.
  • Depends on HDFS.

HSB(HSBroker)

This role can be deployed on all nodes.

Number of role instances to be deployed: 2 to 50

HSC(HSConsole)

This role can be deployed on all nodes.

Number of role instances to be deployed: 2

HSF(HSFabric)

This role can be deployed on all nodes.

Number of role instances to be deployed: 0 to 50

QAS (available for MRS 3.2.0-LTS.1 and later versions only)

This role can be deployed on all nodes.

Number of role instances to be deployed: 0 to 2

Hive
  • Depends on Hadoop.
  • Depends on DBService.

MS(MetaStore)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2 to 10

WH (WebHCat)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 10

HS(HiveServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2 to 80

Hue

Depends on DBService.

H(Hue)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2

Impala
  • Depends on Hadoop.
  • Depends on Hive.
  • Depends on DBService.
  • Depends on ZooKeeper.

StateStore

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1

Catalog

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1

Impalad

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to 10,000

IoTDB

Depends on KrbServer.

ConfigNode (CN)

This role can be deployed on Master nodes only.

Number of role instances to be deployed: 3 to 9, with the step size of 2

IoTDBServer (IoTDBS)

This role can be deployed on all nodes.

Number of role instances to be deployed: 3 to 256

Kafka

Depends on ZooKeeper.

B(Broker)

This role can be deployed on all nodes.

Number of role instances to be deployed: 3 to 10,000

Kudu

-

KuduMaster

This role can be deployed on the Master node only.

Number of role instances to be deployed: 3 or 5

KuduTserver

This role can be deployed on all nodes.

Number of role instances to be deployed: 3 to 10,000

Loader
  • Depends on Hadoop.
  • Depends on DBService.

LS(LoaderServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2

Oozie
  • Depends on Hadoop.
  • Depends on DBService.
  • Depends on ZooKeeper.

O(oozie)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2

Presto

Depends on Hive.

PCD(Coordinator)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2

PWK(Worker)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to 10,000

Ranger

Depends on DBService.

RA(RangerAdmin)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

USC(UserSync)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1

TSC (TagSync)

This role can be deployed on all nodes.

Number of role instances to be deployed: 0 to 1

Spark
  • Depends on Hadoop.
  • Depends on Hive.
  • Depends on ZooKeeper.

JS(JDBCServer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

JH(JobHistory)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

SR(SparkResource)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

Spark2x
  • Depends on Hadoop.
  • Depends on Hive.
  • Depends on ZooKeeper.

JS2X(JDBCServer2x)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2 to 10

JH2X(JobHistory2x)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2

SR2X(SparkResource2x)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 2 to 50

IS2X(IndexServer2x)

(Optional) This role can be deployed on the Master node only.

Number of role instances to be deployed: 0 to 2, with the step size of 2

Sqoop

Depends on Hadoop.

SC(SqoopClient)

This role can be deployed on all nodes.

Number of role instances to be deployed: 1 to 10,000

Tez
  • Depends on Hadoop.
  • Depends on DBService.
  • Depends on ZooKeeper.

TUI(TezUI)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 1 to 2

ZooKeeper

-

QP(quorumpeer)

This role can be deployed on the Master node only.

Number of role instances to be deployed: 3 to 9, with the step size of 2
Parent topic: MRS Cluster Deployment