MRS Cluster Becomes Unavailable After the VPC Is Changed

Updated on 2024-12-18 GMT+08:00

View PDF

Symptom

In an MRS cluster, after the VPCs of all nodes are changed on the ECS console, the cluster status becomes abnormal.

All services are unavailable. The Hive Beeline reports the following error.

Cause Analysis

MRS does not support VPC change. After the VPC is changed, the internal IP address of the node changes, but the configuration file and database still use the original IP address. As a result, functions such as cluster communication are abnormal, and the cluster status is also abnormal. Therefore, to restore the cluster, you need to change the VPC back and ensure that the IP address maps that in the hosts file.

Procedure

Log in to the Master1 node and run the ifconfig command to view the new VPC. Run the cat /etc/hosts command to check the IP address recorded in the hosts file before the VPC change.
Log in to the MRS console and view the cluster ID and VPC on the Dashboard page of the cluster.
Log in to the ECS console, select Name in the search box, and enter the MRS cluster ID to search for all nodes in the MRS cluster.
In the Operation column of the MRS cluster node, click More and choose Manage Network > Change VPC.
NOTE:
- You need to change the VPC for each node.
- When changing the VPC, ensure that the VPC, subnet, and security group are the same as those in the initial cluster configuration.
- Set Private IP Address to Assign new and enter the IP address of the node queried in 1.
After the change is successful, click the node name, switch to the Network Interfaces tab, and enable Source/Destination Check again.
Perform the following steps to rebind the virtual IP address to the master node of the cluster:
1. Log in to the MRS console and access the MRS cluster. On the Dashboard page, click next to Access Manager, set Access Mode to Direct Connect, and record the floating IP address of the cluster. View and take note of the subnet in Default Subnet.
2. Log in to the VPC console, choose Virtual Private Cloud > Subnets, and search for the subnet of the MRS cluster.
3. Click the subnet name, click the IP Addresses tab, and search for the floating IP address of the MRS cluster.
4. Click Bind to Server in the Operation column of the floating IP address. On the Bind to Server page, select the master node of the MRS cluster. After the binding is successful, the following figure is displayed.
Wait for the cluster to restore.