# 误卸载存储池的磁盘后如何恢复 存储池是Everest创建的一种Custom Resource，资源为nodelocalvolumes，该资源在正常情况下不建议手动操作。Everest每分钟会扫描空闲磁盘，并检查已添加进存储池的磁盘是否正常。 Everest使用LVM进行存储池管理，本地持久卷和本地临时卷均为LVM中的一个卷组（Volume Group，VG）： - 本地持久卷使用的卷组名称为：vg-everest-localvolume-persistent - 本地临时卷使用的卷组名称为：vg-everest-localvolume-ephemeral 本文以本地持久卷举例，修复本地临时卷时，使用相应的卷组即可。 ![](https://support.huaweicloud.com/cce_faq/public_sys-resources/notice_3.0-zh-cn.png) 本文提供的指导仅能恢复误卸载磁盘后存储池页面显示"不可用"状态，修复后可重新导入持久卷或临时卷，但无法恢复原有数据。 #### 问题现象 当用户错误卸载存储池的磁盘时，会导致该节点存储池状态不可用。 ![](https://support.huaweicloud.com/cce_faq/zh-cn_image_0000002025575105.png "点击放大") #### 问题定位 使用kubectl命令查看nodelocalvolumes资源： ``` kubectl get nodelocalvolumes.localvolume.everest.io -n kube-system 192.168.1.137 -o yaml ``` 看到状态已列出错误原因为/dev/vde is lost： ``` ... status: lastUpdateTime: "2024-07-16T07:13:55Z" message: the device 6eef2886-f5ad-4b3f-a:/dev/vde is lost phase: Unavailable totalDisks: - capacity: 100Gi name: /dev/vdb uuid: 3511993c-61e6-4faa-a usedDisks: - totalSize: 102392Mi type: persistent usedSize: 10Gi volume-group: vg-everest-localvolume-persistent volumes: - capacity: 50Gi name: /dev/vdd pv-uuid: Jo9Uur-evVi-RLWM-yaln-J6rz-3QCo-aHpvwB uuid: 40b8b92b-5852-4a97-9 - capacity: 50Gi name: /dev/vde pv-uuid: ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo uuid: 6eef2886-f5ad-4b3f-a ``` 在节点上查看LVM资源均会产生告警。 - 运行**vgdisplay** 命令告警如下： ``` WARNING: Couldn't find device with uuid ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo. WARNING: VG vg-everest-localvolume-persistent is missing PV ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo (last written to /dev/vde). --- Volume group --- VG Name vg-everest-localvolume-persistent System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 2 Act PV 1 VG Size 99.99 GiB PE Size 4.00 MiB Total PE 25598 Alloc PE / Size 2560 / 10.00 GiB Free PE / Size 23038 / 89.99 GiB VG UUID 31LHdA-yZPV-M7JX-ttwK-aynz-IyxY-usp22p ``` - 运行**lvdisplay** 命令告警如下： ``` WARNING: Couldn't find device with uuid ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo. WARNING: VG vg-everest-localvolume-persistent is missing PV ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo (last written to /dev/vde). --- Logical volume --- LV Path /dev/vg-everest-localvolume-persistent/pvc-fad963ae-f168-4e6c-a060-088b68ee503e LV Name pvc-fad963ae-f168-4e6c-a060-088b68ee503e VG Name vg-everest-localvolume-persistent LV UUID NzZvao-peoH-q4Zr-YTs3-WgqX-vxt5-W20Og2 LV Write Access read/write LV Creation host, time , 2024-07-16 12:03:20 +0800 LV Status available # open 1 LV Size 10.00 GiB Current LE 2560 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:1 ``` - 运行**pvdisplay** 命令告警如下： ``` WARNING: Couldn't find device with uuid ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo. WARNING: VG vg-everest-localvolume-persistent is missing PV ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo (last written to /dev/vde). --- Physical volume --- PV Name /dev/vdd VG Name vg-everest-localvolume-persistent PV Size 50.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 12799 Free PE 10239 Allocated PE 2560 PV UUID Jo9Uur-evVi-RLWM-yaln-J6rz-3QCo-aHpvwB --- Physical volume --- PV Name [unknown] VG Name vg-everest-localvolume-persistent PV Size 50.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 12799 Free PE 12799 Allocated PE 0 PV UUID ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo ``` - 运行**pvs** 命令告警如下： ``` WARNING: Couldn't find device with uuid ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo. WARNING: VG vg-everest-localvolume-persistent is missing PV ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo (last written to /dev/vde). PV VG Fmt Attr PSize PFree /dev/vdc vgpaas lvm2 a-- <50.00g 0 /dev/vdd vg-everest-localvolume-persistent lvm2 a-- <50.00g <40.00g [unknown] vg-everest-localvolume-persistent lvm2 a-m <50.00g <50.00g ``` #### 解决方案 1. 恢复nodelocalvolumes资源。 ``` kubectl edit nodelocalvolumes.localvolume.everest.io -n kube-system 192.168.1.137 ``` 修改上述资源，删除 spec.volumes.type: persistent 中已丢失的磁盘，同时删除资源中整个status字段。 2. 移除VG中相应PV。 ![](https://support.huaweicloud.com/cce_faq/public_sys-resources/note_3.0-zh-cn.png) VG信息会存在于相应的磁盘上： - 当多块磁盘组成的VG缺少一部分时，会显示PV丢失。 - 当VG中仅有的磁盘被拔出时，vgdisplay将无法显示相应VG，通过vgdisplay无法查看到vg-everest-localvolume-persistent或vg-everest-localvolume-ephemeral时当前步骤可以跳过。 通过以下命令移除本地持久卷VG中所有已丢失的PV。其中，*vg-everest-localvolume-persistent* 为本地持久卷的VG名称，若修复本地临时卷则VG名称为*vg-everest-localvolume-ephemeral*。 ``` vgreduce --removemissing vg-everest-localvolume-persistent ``` 回显如下： ``` WARNING: Couldn't find device with uuid ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo. WARNING: VG vg-everest-localvolume-persistent is missing PV ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo (last written to /dev/vde). WARNING: Couldn't find device with uuid ZxA9kY-5C28-96Z9-ZjOE-dCrc-yTgp-DOhUHo. Wrote out consistent volume group vg-everest-localvolume-persistent. ``` 再次使用 vgdisplay 可以看到显示正常。 ``` vgdisplay vg-everest-localvolume-persistent ``` 回显如下： ``` --- Volume group --- VG Name vg-everest-localvolume-persistent System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 4 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size <50.00 GiB PE Size 4.00 MiB Total PE 12799 Alloc PE / Size 2560 / 10.00 GiB Free PE / Size 10239 / <40.00 GiB VG UUID 31LHdA-yZPV-M7JX-ttwK-aynz-IyxY-usp22p ``` 3. 重启相应节点的everest-csi-driver。 1. 查看everest-csi-driver的Pod名称。 ``` kubectl get pod -A -owide ``` 回显如下： ``` NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system everest-csi-driver-7clbg 1/1 Running 0 5d4h 192.168.1.38 192.168.1.38 kube-system everest-csi-driver-jvj9f 1/1 Running 0 5d4h 192.168.1.137 192.168.1.137 ``` 2. 删除192.168.1.137节点上的Pod。 ``` kubectl delete pod -n kube-system everest-csi-driver-jvj9f ``` 3. 此时查看 nodelocalvolumes 状态已正常，可继续导入持久卷。 ![](https://support.huaweicloud.com/cce_faq/zh-cn_image_0000002025576485.png "点击放大") 4. 重启节点。 某些情况下如NPD可能依然会认为节点存在指标异常导致节点不可用，因此通过重启节点解决此类问题。