Replacing a Disk in an MRS Cluster (Applicable to 3.x)

Issue

A disk is not accessible.

Symptom

A user created an MRS cluster with local disks. A disk of a core node in this cluster is damaged, resulting in file read failures.

Cause Analysis

The disk hardware is faulty.

Procedure

This procedure is applicable to troubleshooting disk hardware faults of core and task nodes in MRS clusters using local disks (ECSs of D, I, IR, and KI series).

Kafka does not support disk replacement. If the node that stores Kafka data is faulty, contact Huawei Cloud technical support.

1. Log in to FusionInsight Manager.
2. Choose Hosts and click the name of the faulty host. In the Instance area, click DataNode. Then on the page that is displayed, click More and select Decommission.
   

   • If this host accommodates DataNode, NodeManager, RegionServer, and ClickHouseServer instances, decommission these instances by referring to this step.
   • In versions later than MRS 3.1.2, the ClickHouseServer role instance can be decommissioned.

3. Choose Hosts, select the faulty host, click More, and select Stop All Instances.
4. Run the vim /etc/fstab command to comment out the mount point of the faulty disk.
   Figure 1 Commenting out the mount point of the faulty disk
   

5. If the old disk is still accessible, migrate user data on the old disk (for example, /srv/BigData/data1/).

   cp -r Mount point of the old disk Temporary data storage directory

   Example: cp -r /srv/BigData/data1 /tmp/

6. Log in to the MRS console.
7. On the cluster details page, click the Nodes tab.
8. Click the node whose disk is to be replaced to go to the ECS console. Click Stop to stop the node.
9. Contact Huawei Cloud technical support to replace the disk in the background.
10. On the ECS console, click Start to start the node where the disk has been replaced.
11. Initialize the Linux data disk.

    For details, see Initialize an EVS Data Disk.

12. Run the lsblk command to view information about the new disk partition.
    Figure 2 Viewing the new disk partition
    

13. Run the df -TH command to obtain the file system type.
    Figure 3 Obtaining the file system type
    

14. Format the new disk partition based on the obtained file system type.

    Example: mkfs.ext4 /dev/sdd1

15. Run the following command to mount the new disk:

    mount New disk Mount point

    Example: mount /dev/sdd1 /srv/BigData/data1

    

    If the disk cannot be mounted, run the following command to reload the configuration and mount it again:

    systemctl daemon-reload

16. Run the following command to grant the omm user permission to the new disk:

    chown omm:wheel Mount point

    Example: chown -R omm:wheel /srv/BigData/data1

17. Migrate user data from the old disk (for example, /srv/BigData/data1/) to the new disk.

    cp -rTemporary data storage directory Mount point of the new disk

    Example: cp -r /tmp/data1/* /srv/BigData/data1/

18. Add the UUID of the new disk to the fstab file.
    1. Run the blkid command to check the UUID of the new disk.

       

    2. Open the /etc/fstab file and add the following information:

       UUID=UUID of the new disk /srv/BigData/data1 ext4 defaults,noatime,nodiratime,nodev 1 0

19. Log in to FusionInsight Manager.
20. Choose Hosts and click the name of the host to be recommissioned. In the Instance area, click DataNode. Then on the page that is displayed, click More and select Recommission.
    

    • If this host accommodates DataNode, NodeManager, RegionServer, and ClickHouseServer instances, recommission these instances by referring to this step.
    • In versions later than MRS 3.1.2, the ClickHouseServer role instance can be recommissioned.

21. Choose Hosts, select the faulty host, click More, and select Start All Instances.
22. Choose Cluster > HDFS. In the Basic Information area on the Dashboard page, check whether Missing Blocks is 0.
    • If Missing Blocks is 0, no further action is required.
    • If Missing Blocks is not 0, contact Huawei Cloud technical support.