Replacing a Disk in an MRS Cluster (Applicable to 3.x)
Issue
A disk is not accessible.
Symptom
A user created an MRS cluster with local disks. A disk of a core node in this cluster is damaged, resulting in file read failures.
Cause Analysis
The disk hardware is faulty.
Procedure
This procedure is applicable to troubleshooting disk hardware faults of core and task nodes in MRS clusters using local disks (ECSs of D, I, IR, and KI series).
Kafka does not support disk replacement. If the node that stores Kafka data is faulty, contact Huawei Cloud technical support.
- Log in to FusionInsight Manager.
- Choose Hosts and click the name of the faulty host. In the Instance area, click DataNode. Then on the page that is displayed, click More and select Decommission.
- If this host accommodates DataNode, NodeManager, RegionServer, and ClickHouseServer instances, decommission these instances by referring to this step.
- In versions later than MRS 3.1.2, the ClickHouseServer role instance can be decommissioned.
- Choose Hosts, select the faulty host, click More, and select Stop All Instances.
- Run the vim /etc/fstab command to comment out the mount point of the faulty disk.
Figure 1 Commenting out the mount point of the faulty disk
- If the old disk is still accessible, migrate user data on the old disk (for example, /srv/BigData/data1/).
cp -r Mount point of the old disk Temporary data storage directory
Example: cp -r /srv/BigData/data1 /tmp/
- Log in to the MRS console.
- On the cluster details page, click the Nodes tab.
- Click the node whose disk is to be replaced to go to the ECS console. Click Stop to stop the node.
- Contact Huawei Cloud technical support to replace the disk in the background.
- On the ECS console, click Start to start the node where the disk has been replaced.
- Initialize the Linux data disk.
For details, see Initializing EVS Data Disks.
- Run the lsblk command to view information about the new disk partition.
Figure 2 Viewing the new disk partition
- Run the df -TH command to obtain the file system type.
Figure 3 Obtaining the file system type
- Format the new disk partition based on the obtained file system type.
Example: mkfs.ext4 /dev/sdd1
- Run the following command to mount the new disk:
mount New disk Mount point
Example: mount /dev/sdd1 /srv/BigData/data1
If the disk cannot be mounted, run the following command to reload the configuration and mount it again:
systemctl daemon-reload
- Run the following command to grant the omm user permission to the new disk:
chown omm:wheel Mount point
Example: chown -R omm:wheel /srv/BigData/data1
- Migrate user data from the old disk (for example, /srv/BigData/data1/) to the new disk.
cp -rTemporary data storage directory Mount point of the new disk
Example: cp -r /tmp/data1/* /srv/BigData/data1/
- Add the UUID of the new disk to the fstab file.
- Run the blkid command to check the UUID of the new disk.
- Open the /etc/fstab file and add the following information:
UUID=UUID of the new disk /srv/BigData/data1 ext4 defaults,noatime,nodiratime,nodev 1 0
- Run the blkid command to check the UUID of the new disk.
- Log in to FusionInsight Manager.
- Choose Hosts and click the name of the host to be recommissioned. In the Instance area, click DataNode. Then on the page that is displayed, click More and select Recommission.
- If this host accommodates DataNode, NodeManager, RegionServer, and ClickHouseServer instances, recommission these instances by referring to this step.
- In versions later than MRS 3.1.2, the ClickHouseServer role instance can be recommissioned.
- Choose Hosts, select the faulty host, click More, and select Start All Instances.
- Choose Cluster > HDFS. In the Basic Information area on the Dashboard page, check whether Missing Blocks is 0.
- If Missing Blocks is 0, no further action is required.
- If Missing Blocks is not 0, contact Huawei Cloud technical support.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot