Restarting an MRS Cluster

During MRS cluster running, restart the components in the cluster if you have modified component settings, encountered infrastructure resource faults, or detected service process errors.

Components within an MRS cluster support both the standard restart and rolling restart.

Standard restart: concurrently restarts all components or instances in the cluster, which may interrupt services.
Rolling restart: restarts required components or instances without interrupting services as much as possible. Compared with a standard restart, a rolling restart takes a longer time and may affect service throughput and performance. To minimize or eliminate the impact on services during a component restart, you can perform rolling restarts to restart components or instances in batches. For instances in active/standby mode, the standby instance is restarted first, followed by the active instance.
Table 2 describes the impact on services when a rolling restart is performed on components.

Restarting a cluster will stop the cluster components from providing services, which adversely affects the running of upper-layer applications or jobs. You are advised to perform rolling restarts during off-peak hours.

Notes and Constraints

Not all components in an MRS cluster support rolling restarts. When you perform a rolling restart on a cluster, components that do not support rolling restarts will be restarted in standard restart mode. For details, see Component Restart Reference Information.
For configurations that must take effect immediately, for example, server port configurations, a standard restart is generally recommended over a rolling restart.

Impact on the System

Table 2 describes the impact of the rolling restart on each component during the rolling restart of a cluster.

Restarting an MRS Cluster

Restart an MRS cluster on the MRS console or on FusionInsight Manager of the cluster.

Log in to the MRS console.
Choose Active Clusters and click a cluster name to go to the cluster details page.
In the upper right corner of the page, choose Management Operations > Perform Rolling Cluster Restart.

Figure 1 Performing rolling cluster restart
The Rolling-restart Cluster page is displayed. Select Only restart instances whose configurations have expired and click OK to perform rolling restart for the target cluster.
After the rolling restart task is complete, click Finish.

Log in to FusionInsight Manager of the MRS cluster.

For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.
Choose Cluster > Dashboard. In the upper right corner, click More > Restart.
- For MRS 3.3.0 or later, the Cluster > Dashboard page has been removed from FusionInsight Manager. You can choose More in the upper right corner of Homepage to access cluster maintenance and management functions.
- When restarting a cluster, you can choose between a standard restart or rolling restart. A rolling restart takes a longer time but has less impact on services.
In the displayed dialog box, enter the password of the current login user and click OK.

If you select the rolling restart, adjust related parameters based on the site requirements.

Figure 2 Rolling restart
Click to enlarge

**Table 1** Rolling restart parameters
Parameter	Example Value	Description
Restart only instances with expired configurations in the cluster	-	Whether to restart only the modified instances in a cluster.
Enable rack strategy	-	Whether to enable the concurrent rack rolling restart strategy. This parameter takes effect only for roles that meet the rack rolling restart strategy. (The roles support rack awareness, and instances of the roles belong to two or more racks.) This parameter is available only when a rolling restart is performed on HDFS or YARN.
Data Nodes to Be Batch Restarted	1	Number of instances that are restarted in each batch when the batch rolling restart strategy is used. The default value is 1. This parameter is valid only when the batch rolling restart strategy is used and the instance type is DataNode. This parameter is invalid when the rack strategy is enabled. In this case, the cluster uses the maximum number of instances (20 by default) configured in the rack strategy as the maximum number of instances that are concurrently restarted in a rack. This parameter is configurable only when a rolling restart is performed on HDFS, HBase, YARN, Kafka, Storm, or Flume. This parameter for the RegionServer of HBase cannot be manually configured. Instead, it is automatically adjusted based on the number of RegionServer nodes. Specifically, if the number of RegionServer nodes is less than 30, the parameter value is 1. If the number is greater than or equal to 30 and less than 300, the parameter value is 2. If the number is greater than or equal to 300, the parameter value is 1% of the number (rounded-down).
Batch Interval	0	Interval between two batches of instances to be roll-restarted. The default value is 0.
Decommissioning Timeout Interval	1800	Decommissioning interval for role instances during a rolling restart. The default value is 1800s. Some roles (such as HiveServer and JDBCServer) stop providing services before the rolling restart. Stopped instances cannot be connected to new clients. Existing connections will be completed after a period of time. An appropriate timeout interval can ensure service continuity. This parameter is configurable only when a rolling restart is performed on Hive, Spark, or Spark2x.
Batch Fault Tolerance Threshold	0	Tolerance times when the rolling restart of instances fails to be batch executed. The default value is 0, which indicates that the rolling restart task ends after any batch of instances fails to restart.

Advanced parameters, such as Data Nodes to Be Batch Restarted, Batch Interval, and Batch Fault Tolerance Threshold, should be properly configured based on site requirements. Otherwise, services may be interrupted or cluster performance may be severely affected.

Example:

If Data Nodes to Be Batch Restarted is set to an unnecessarily large value, a large number of instances are restarted concurrently. As a result, services are interrupted or cluster performance is severely affected due to too few working instances.
If Batch Fault Tolerance Threshold is set to an unnecessarily large value, services will be interrupted because a next batch of instances will be restarted after a batch of instances fails to restart.

Click OK.

Reference

Table 2 describes the impact on the system during the rolling restart of components and instances.

**Table 2** Impact on the system
Component	Service Interruption	Impact on System
ClickHouse	During the rolling restart, if the submitted workloads can be complete within the timeout period (30 minutes by default), there is no impact.	Nodes undergoing a rolling restart reject all new requests, which affects single-replica services, the ON CLUSTER operation, and workloads dependent on the instances being rolling-restarted. If a request that is being executed is not complete within the timeout period (30 minutes by default), the request fails.
DBService	All services are normal during the rolling restart.	During the rolling restart, alarms indicating a heartbeat interruption between the active and standby DBService nodes may be reported.
Doris	Doris services will not be interrupted during the rolling restart only when the following conditions are met: ELB or DBalancer is used to connect to Doris for job submission. Tables being read or written must have multiple replicas (three replicas are recommended). If the submitted workloads are complete within the timeout period (30 minutes by default), the workloads are not affected. Otherwise, the workloads will fail.	During the rolling restart, the total resources decrease, affecting the maximum memory and CPU resources that can be used by jobs. In extreme cases, the jobs may fail due to insufficient resources. If a job times out (30 minutes by default), retry the job.
Flink	All services are normal during the rolling restart.	The FlinkServer UI cannot be accessed during the rolling restart.
Flume	To prevent service interruptions and data loss, the following conditions must be met: Persistent cache, for example File Channel, is used. Flume cascading is configured. Flume client sink supports fault failover or load balancing.	Data loss may occur if channels do not have persistent cache. The performance deteriorates for a short period of time during the fault failover.
Guardian	All services are normal during the rolling restart.	None
HBase	HBase read and write services are normal during the rolling restart.	Real-time read and write performance may deteriorate during RegionServer rolling restart. During the HMaster rolling restart, services other than the real-time read and write services (excluding BulkLoad) are affected. Creating a table (create) Creating a namespace (create_namespace) Disabling a table (disable, disable_all) Re-creating a table (truncate and truncate_preserve) Moving a region (move) Taking a region offline (unassign) Merging regions (merge_region) Splitting a region (split) Enabling balance (balance_switch) DR operations (add_peer, remove_peer, enable_table_replication, disable_peer, show_peer_tableCFs, set_peer_tableCFs, enable_peer, disable_table_replication, set_clusterState_active, and set_clusterState_standby) Querying the cluster status (status)
HDFS	An active/standby switchover is triggered for NameNodes. During the switchover, there is temporarily no active NameNode. As a result, the HDFS unavailable alarm may be reported, and running read/write tasks may encounter errors. However, HDFS services are not interrupted. If a rolling restart is being performed on DataNodes, errors may be reported for some data that is being read or written. Read/write speed is affected when the client retries. During the rolling restart of ZKFC, an active/standby NameNode switchover occurs. During the rolling restart of HDFS, enabling the rack policy may affect service read and write operations.	If a third-party client is used, the reliability of the third-party client during the rolling restart cannot be guaranteed.
HetuEngine	There are at least two HSFabric instances, and at least two HSFabric instances are used for interconnection. Cross-domain services are not interrupted during the rolling restart. HetuEngine services are not interrupted during the rolling restart of HSBroker, HSConsole, and QAS. The rolling restart of HetuEngine compute instances can be performed only when there are at least two Coordinator and Worker nodes, respectively. The rolling restart of compute instances does not interrupt services.	HSFabric nodes undergoing the rolling restart reject all new requests. SQL requests that are being executed will fail if they are not complete within the timeout period (30 minutes by default). During the rolling restart, you cannot perform O&M operations on the HSConsole page. During the rolling restart of HetuEngine compute instances, the performance decreases by approximately 20%. If the memory consumed by SQL execution exceeds 80% of query_max_total_memory, SQL tasks fail.
Hive	During the rolling restart, services with execution time longer than the decommissioning timeout period may fail.	If the execution time of an existing task exceeds the timeout interval of the rolling restart, the task may fail during the restart. You can retry the task if it fails. Nodes undergoing a rolling restart do not receive new requests. The number of requests processed by other instance nodes increases, and these nodes occupy more resources.
Kafka	During the rolling restart, the read and write of Kafka topics with multiple replicas are normal, but operations on Kafka topics with only a single replica are interrupted.	Topics and partitions cannot be added, deleted, or modified. When acks is set to 1 or 0 in Producer, the next Broker will be forcibly restarted if the replica data is not synchronized within 30 minutes during the rolling restart. For a dual-replica Partition whose replicas are in the two Brokers, when unclean.leader.election.enable is set to true in the server configuration, data may be lost; when unclean.leader.election.enable is set to false, the Partition may not have a leader for a period of time until the Broker starts.
KrbServer	All services are normal during the rolling restart.	During the rolling restart, Kerberos authentication of a cluster may take a longer time.
LdapServer	All services are normal during the rolling restart.	During the rolling restart, Kerberos authentication of a cluster may take a longer time.
MapReduce	None	You cannot view the task logs during the rolling restart. In the scenario where only one JobHistoryServer instance is deployed, stopping JobHistoryServer will make all upper-layer components that depend on it become faulty.
Ranger	All services are normal during the rolling restart.	The RangerAdmin, RangerKMS, and PolicySync instances of Ranger are configured in active-active mode, and these instances can provide services in turn during the rolling restart. While UserSync supports only one-instance configuration and users cannot be synchronized during the restart. The user synchronization period is 5 minutes, and UserSync takes a short time to restart. Therefore, the UserSync restart has little impact on user synchronization.
Spark	Except the listed items, other services are not affected.	When performing a rolling restart on HBase, you cannot create or delete Spark on HBase tables in Spark. When you perform a rolling restart on HBase, an active/standby switchover is triggered for HMaster. During the switchover, the Spark on HBase function is unavailable. If you have used the advanced Kafka APIs, interruptions and data loss may occur when Spark reads data from or writes data to Kafka during the rolling restart. During the rolling restart of ZooKeeper, spark-beeline fails to start. During the rolling restart of YARN, Spark jobs may trigger retries at the task, stage, and application levels. During the rolling restart of JDBCServer of Spark, long-running tasks will be terminated.
YARN	An active/standby switchover is triggered for ResourceManager nodes during the rolling restart. Running tasks will encounter errors, but services are not interrupted. During the rolling restart of NodeManager, containers submitted to the node may be retried on other nodes.	During the rolling restart of YARN, tasks running on YARN may experience exceptions due to excessive retries.
ZooKeeper	ZooKeeper read and write operations are normal during the rolling restart.	The rolling restart affects the ClickHouse service. During the instance restart, ClickHouse tables become read-only on each QuorumPeer instance for approximately 10 seconds. The rolling restart causes ZooKeeper disconnection in Loader HA. Loader automatically retries for three times, each for 10 seconds. If ZooKeeper still cannot be connected, the active LoaderServer becomes the standby one and then a new active LoaderServer will be elected.