Isolating an MRS Cluster Node
If a host is found to be abnormal or faulty, affecting cluster performance or preventing services from being provided, you can temporarily exclude that host from the available nodes in the cluster. In this way, the client can access other available nodes. In scenarios where patches are to be installed in a cluster, you can also exclude a specified node from patch installation.
You can isolate a host manually on MRS based on the actual service requirements or O&M plan.
Notes and Constraints
Only non-management nodes can be isolated.
Impact on the System
- After a host is isolated, all role instances on the host will be stopped. You cannot start, stop, or configure the host and any instances on the host.
- After a host is isolated, statistics of the monitoring status and indicator data of the host hardware and instances cannot be collected or displayed.
- For some services, after a host is isolated, some instances on other nodes do not work, and the service configuration status may expire.
- Retain the default SSH port (22) of the target node. Otherwise, the task described in this section will fail.
Prerequisites
- The IAM users have been synchronized in advance. You can do this by clicking Synchronize next to IAM User Sync on the Dashboard page of the cluster details.
- You have logged in to MRS Manager. For how to log in, see Accessing MRS Manager.
- Log in to the MRS console.
- On the Active Clusters page, select a running cluster and click its name to switch to the cluster details page.
- On the cluster details page, click Nodes.
- Unfold the node group information and select the check box of the target host.
- Choose Node Operation > Isolate Host.
- Confirm the information about the host to be isolated and click OK.
When Operation successful is displayed, click Finish. The host is isolated successfully, and the value of Operating Status becomes Isolated.
- Log in to the isolated host as user root, run the pkill -9 -u omm command to stop the processes of user omm on the node, and run the ps -ef | grep 'container' | grep '${BIGDATA_HOME}' | awk '{print $2}' | xargs -I '{}' kill -9 '{}' command to search for and stop the container process.
- Host isolation can be canceled, after which the host can be added back to the cluster. If the host exception or fault has been rectified, cancel the host's isolation status before reusing the host.
- On the Nodes page, expand the node group information.
- Select the check box of the host to be de-isolated and choose Node Operation > Cancel Host Isolation.
- Confirm the information about the host for which the isolation is to be cancelled and click OK.
When Operation successful is displayed, click Finish. The host is de-isolated, and the value of Operating Status becomes Normal.
For MRS 3.x or later:
- Log in to FusionInsight Manager of the MRS cluster.
For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.
- Click Hosts.
- Select the check box of the host to be isolated, choose More > Isolate Host, enter the password for identity authentication, and click OK.
- Confirm the information about the host to be isolated and click OK.
After the system displays a message indicating that the operation is successful, click Finish. If the host is isolated, its Running Status is displayed as Isolated.
- Log in to the isolated host as user root and run the pkill -9 -u omm command to stop the processes of user omm on the node. Then run the ps -ef | grep 'container' | grep '${BIGDATA_HOME}' | awk '{print $2}' | xargs -I '{}' kill -9 '{}' command to find and stop the container process.
- Cancel the isolation status of the host before using the host if you have rectified the host exception or fault.
On the Hosts page, select the isolated host and choose More > Cancel Isolation.
When the isolation of a host is canceled, its role instances remain stopped by default. To start role instances on the host, select the target host on the Hosts page and choose More > Start All Instances.
For MRS 2.x or earlier:
- Log in to FusionInsight Manager of the MRS cluster.
For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.
- Click Hosts.
- Select the check box of the host to be isolated, choose More > Isolate Host, enter the password for identity authentication, and click OK.
- Confirm the information about the host to be isolated and click OK.
After the system displays a message indicating that the operation is successful, click Finish. If the host is isolated, its Operating Status is displayed as Isolated.
- Cancel the isolation status of the host before using the host if you have rectified the host exception or fault.
- On MRS Manager, click Hosts.
- Select the check box of the host to be de-isolated and choose More > Cancel Host Isolation.
- Click OK in the displayed dialog box.
When Operation successful is displayed, click Finish. The host is de-isolated successfully, and the value of Operating Status becomes Normal.
- Click the name of the de-isolated host to show its status, and click Start All Roles.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.