Isolating an MRS Cluster Node

Updated on 2024-09-23 GMT+08:00

View PDF

If a host is found to be abnormal or faulty, affecting cluster performance or preventing services from being provided, you can temporarily exclude that host from the available nodes in the cluster. In this way, the client can access other available nodes. In scenarios where patches are to be installed in a cluster, you can also exclude a specified node from patch installation.

Only non-management nodes can be isolated.

You can isolate a host manually on MRS based on the actual service requirements or O&M plan.

Impact on the System

After a host is isolated, all role instances on the host will be stopped. You cannot start, stop, or configure the host and any instances on the host.
After a host is isolated, statistics of the monitoring status and indicator data of the host hardware and instances cannot be collected or displayed.
For some services, after a host is isolated, some instances on other nodes do not work, and the service configuration status may expire.
Retain the default SSH port (22) of the target node. Otherwise, the task described in this section will fail.

Prerequisites

The IAM users have been synchronized in advance. You can do this by clicking Synchronize next to IAM User Sync on the Dashboard page of the cluster details.
You have logged in to MRS Manager. For how to log in, see Accessing MRS Manager.

Isolating a Host on the Console

Log in to the MRS console.
On the Active Clusters page, select a running cluster and click its name to switch to the cluster details page.
On the MRS details page, click Nodes.
Unfold the node group information and select the check box of the target host.
Choose Node Operation > Isolate Host.

Figure 1 Isolating a host
Confirm the information about the host to be isolated and click OK.

When Operation successful is displayed, click Finish. The host is isolated successfully, and the value of Operating Status becomes Isolated.

NOTE:

The isolation of a host can be canceled and the host can be added to the cluster again. After the exception or fault of a host is handled, you must cancel the isolation of the host for proper usage.
Cancel the isolation status of the host before using the host if you have rectified the host exception or fault.
1. On the Nodes page, expand the node group information.
2. Select the check box of the host to be de-isolated and choose Node Operation > Cancel Host Isolation.
3. Confirm the information about the host for which the isolation is to be cancelled and click OK.
  When Operation successful is displayed, click Finish. The host is de-isolated, and the value of Operating Status becomes Normal.

Isolating a Host on Manager

For MRS 3.x or later:

Log in to FusionInsight Manager and click Hosts.
Select the check box of the host to be isolated, choose More > Isolate Host, enter the password for identity authentication, and click OK.
Confirm the information about the host to be isolated and click OK.

After the system displays a message indicating that the operation is successful, click Finish. If the host is isolated, its Running Status is displayed as Isolated.
Log in to the isolated host as user root and run the pkill -9 -u omm command to stop the processes of user omm on the node. Then run the ps -ef | grep 'container' | grep '${BIGDATA_HOME}' | awk '{print $2}' | xargs -I '{}' kill -9 '{}' command to find and stop the container process.
Cancel the isolation status of the host before using the host if you have rectified the host exception or fault.

On the Hosts page, select the isolated host and choose More > Cancel Isolation.

NOTE:

After the isolation is canceled, all role instances on the host are not started by default. To start role instances on the host, select the target host on the Hosts page and choose More > Start All Instances.

For MRS 2.x or earlier:

Log in to MRS Manager, and choose Hosts.
Select the check box of the host to be isolated, choose More > Isolate Host, enter the password for identity authentication, and click OK.
Confirm the information about the host to be isolated and click OK.

After the system displays a message indicating that the operation is successful, click Finish. If the host is isolated, its Operating Status is displayed as Isolated.
Cancel the isolation status of the host before using the host if you have rectified the host exception or fault.
1. On MRS Manager, click Hosts.
2. Select the check box of the host to be de-isolated and choose More > Cancel Host Isolation.
3. Click OK in the displayed dialog box.
  When Operation successful is displayed, click Finish. The host is de-isolated successfully, and the value of Operating Status becomes Normal.
4. Click the name of the de-isolated host to show its status, and click Start All Roles.