Restarting an MRS Cluster Component

Restrictions

• Perform a rolling restart during off-peak hours.
  • If the service throughput of the Kafka service is high (over 100 MB/s) during a rolling restart, the restart will fail.
  • To avoid RegionServer restart failures caused by heavy loads during an HBase rolling restart, increase the number of handles if the requests per second of each RegionServer on the native interface exceed 10,000.
• Before restarting, check the current number of requests in HBase. If the number of requests on the native interface for each RegionServer is over 10,000, increase the number of handles to prevent overloading.
• If the number of Core nodes in a cluster is less than six, services may be affected for a short period of time.
• Preferentially perform a rolling instance or service restart and select Only restart instances whose configurations have expired.

Prerequisites

• The IAM users have been synchronized in advance. You can do this by clicking Synchronize next to IAM User Sync on the Dashboard page of the cluster details.
• You have logged in to MRS Manager. For how to log in, see Accessing MRS Manager.

Restarting Cluster Components

1. Access the MRS cluster component management page.
   • Log in to the MRS console and click the cluster name to go to the cluster details page. Click Components.
   • If you are using the Manager of MRS 3.x and later versions, log in to Manager and choose Cluster > Services.
   • If you are using the Manager of MRS 2.x and earlier versions, log in to Manager and click Services.

2. Click the name of the target component to go to the details page.
3. On the service details page, expand the More drop-down list and select Restart Service or Service Rolling Restart.
4. Enter the user password (required when you perform operations on Manager), confirm the operation impact, and click OK to restart the system.

   If you select rolling restart, set parameters listed in Table 1. (Required parameters may vary by version, set parameters based on the actual GUI.)

   Figure 1 Performing a rolling restart on Manager
   

   Table 1 Rolling restart configuration parameters

   Parameter	Description
   Restart only instances with expired configurations	Whether to restart only the modified instances in a cluster. The name of this parameter may be different in other versions.
   Enable rack strategy	Whether to enable the concurrent rack rolling restart strategy. This parameter takes effect only for roles that meet the rack rolling restart strategy. (The roles support rack awareness, and instances of the roles belong to two or more racks.) This parameter can be set only when a rolling restart is performed on HDFS or YARN.
   Data Nodes to Be Batch Restarted	Number of instances that are restarted in each batch when the batch rolling restart strategy is used. The default value is 1. NOTE: This parameter is valid only when the batch rolling restart strategy is used and the instance type is DataNode. This parameter is invalid when the rack strategy is enabled. In this case, the cluster uses the maximum number of instances (20 by default) configured in the rack strategy as the maximum number of instances that are concurrently restarted in a rack. This parameter can be set only when a rolling restart is performed on some components, such as HDFS, HBase, YARN, Kafka, Storm, and Flume. The actual value displayed on the GUI prevails. The number of concurrent RegionServer rolling restarts of HBase cannot be manually configured. It is automatically adjusted based on the number of RegionServer nodes. The adjustment rules are as follows: If the number of nodes is less than 30, one node will be added in each batch. For node counts less than 300, two nodes will be added in each batch. If the node count exceeds 300 (including 300 nodes), each batch will add 1% (rounded down) of the total nodes.
   Batch Interval	Interval between two batches of instances to be roll-restarted. The default value is 0. Setting the batch interval parameter can increase the stability of the big data component process during the rolling restart. You are advised to set this parameter to a non-default value, for example, 10.
   Decommissioning Timeout Interval	Decommissioning waiting time of a role instance during a rolling restart. This parameter can be set only when a rolling restart is performed on Hive or Spark. Some roles (such as HiveServer and JDBCServer) stop providing services before the rolling restart. Stopped instances cannot be connected to new clients. Existing connections will be completed after a period of time. An appropriate timeout interval can ensure service continuity.
   Batch Fault Tolerance Threshold	Tolerance times when the rolling restart of instances fails to be batch executed. The default value is 0, which indicates that the rolling restart task ends after any batch of instances fails to restart.

Component Restart Reference Information

Table 2 provides services and instances that support or do not support rolling restart in the MRS cluster.

Table 3 lists the instance startup duration.