Symptom
When the disk space of a thin pool on a node is about to be used up, the following exceptions occasionally occur:
Files or directories fail to be created in the container, the file system in the container is read-only, the node is tainted disk-pressure, or the node is unavailable.
You can run the docker info command on the node to view the used and remaining thin pool space to locate the fault. The following figure is an example.
Possible Cause
When Docker device mapper is used, although you can configure the basesize parameter to limit the size of the /home directory of a single container (to 10 GB by default), all containers on the node still share the thin pool of the node for storage. They are not completely isolated. When the sum of the thin pool space used by certain containers reaches the upper limit, other containers cannot run properly.
In addition, after a file is deleted in the /home directory of the container, the thin pool space occupied by the file is not released immediately. Therefore, even if basesize is set to 10 GB, the thin pool space occupied by files keeps increasing until 10 GB when files are created in the container. The space released after file deletion will be reused only after a while. If the number of service containers on the node multiplied by basesize is greater than the thin pool space size of the node, there is a possibility that the thin pool space has been used up.
Solution
When the thin pool space of a node is used up, some services can be migrated to other nodes to quickly recover services. But you are advised to use the following solutions to resolve the root cause:
Solution 1:
Properly plan the service distribution and data plane disk space to avoid the scenario where the number of service containers multiplied by basesize is greater than the thin pool size of the node. To expand the thin pool size, perform the following operations:
- Expand the capacity of a data disk on the EVS console. For details, see Expanding EVS Disk Capacity.
Only the storage capacity of EVS disks can be expanded. You need to perform the following operations to expand the capacity of logical volumes and file systems.
- Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Nodes. In the right pane, click the Nodes tab, locate the row containing the node, and choose More > Sync Server Data in the Operation column.
- Log in to the node.
- Run lsblk to view the block device information of the node.
A data disk is divided depending on the container storage Rootfs:
Overlayfs: No independent thin pool is allocated. Image data is stored in dockersys.
- Check the disk and partition capacity of the device.
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 150G 0 disk # The data disk has been expanded to 150 GiB, but 50 GiB space is free.
├─vgpaas-dockersys 253:0 0 90G 0 lvm /var/lib/containerd
└─vgpaas-kubernetes 253:1 0 10G 0 lvm /mnt/paas/kubernetes/kubelet
- Expand the disk capacity.
Add the new disk capacity to the dockersys logical volume used by the container runtime.
- Expand the PV capacity so that LVM can identify the new EVS capacity. /dev/sdb specifies the physical volume where dockersys is located.
pvresize /dev/sdb
Information similar to the following is displayed:
Physical volume "/dev/sdb" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized
- Expand 100% of the free capacity to the logical volume. vgpaas/dockersys specifies the logical volume used by the container runtime.
lvextend -l+100%FREE -n vgpaas/dockersys
Information similar to the following is displayed:
Size of logical volume vgpaas/dockersys changed from <90.00 GiB (23039 extents) to 140.00 GiB (35840 extents).
Logical volume vgpaas/dockersys successfully resized.
- Adjust the capacity of the file system. /dev/vgpaas/dockersys specifies the file system path of the container runtime.
resize2fs /dev/vgpaas/dockersys
Information similar to the following is displayed:
Filesystem at /dev/vgpaas/dockersys is mounted on /var/lib/containerd; on-line resizing required
old_desc_blocks = 12, new_desc_blocks = 18
The filesystem on /dev/vgpaas/dockersys is now 36700160 blocks long.
- Check whether the capacity has been expanded.
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 150G 0 disk
├─vgpaas-dockersys 253:0 0 140G 0 lvm /var/lib/containerd
└─vgpaas-kubernetes 253:1 0 10G 0 lvm /mnt/paas/kubernetes/kubelet
Device Mapper: A thin pool is allocated to store image data.
- Check the disk and partition capacity of the device.
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 8:0 0 50G 0 disk
└─vda1 8:1 0 50G 0 part /
vdb 8:16 0 200G 0 disk
├─vgpaas-dockersys 253:0 0 18G 0 lvm /var/lib/docker
├─vgpaas-thinpool_tmeta 253:1 0 3G 0 lvm
│ └─vgpaas-thinpool 253:3 0 67G 0 lvm # Space used by thin pool
│ ...
├─vgpaas-thinpool_tdata 253:2 0 67G 0 lvm
│ └─vgpaas-thinpool 253:3 0 67G 0 lvm
│ ...
└─vgpaas-kubernetes 253:4 0 10G 0 lvm /mnt/paas/kubernetes/kubelet
- Expand the disk capacity.
Option 1: Add the new disk capacity to the thin pool.
- Expand the PV capacity so that LVM can identify the new EVS capacity. /dev/vdb specifies the physical volume where thin pool is located.
pvresize /dev/vdb
Information similar to the following is displayed:
Physical volume "/dev/vdb" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized
- Expand 100% of the free capacity to the logical volume. vgpaas/thinpool specifies the logical volume used by the container runtime.
lvextend -l+100%FREE -n vgpaas/thinpool
Information similar to the following is displayed:
Size of logical volume vgpaas/thinpool changed from <67.00 GiB (23039 extents) to <167.00 GiB (48639 extents).
Logical volume vgpaas/thinpool successfully resized.
- Do not adjust the capacity of the file system because the thin pool is not mounted to any devices.
- Run the lsblk command to check the disk and partition capacity of the device and check whether the capacity has been expanded. If the new disk capacity is added to the thin pool, the capacity has been expanded.
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 8:0 0 50G 0 disk
└─vda1 8:1 0 50G 0 part /
vdb 8:16 0 200G 0 disk
├─vgpaas-dockersys 253:0 0 18G 0 lvm /var/lib/docker
├─vgpaas-thinpool_tmeta 253:1 0 3G 0 lvm
│ └─vgpaas-thinpool 253:3 0 167G 0 lvm # Thin pool space after capacity expansion
│ ...
├─vgpaas-thinpool_tdata 253:2 0 67G 0 lvm
│ └─vgpaas-thinpool 253:3 0 67G 0 lvm
│ ...
└─vgpaas-kubernetes 253:4 0 10G 0 lvm /mnt/paas/kubernetes/kubelet
Option 2: Add the new disk capacity to the dockersys disk.
- Expand the PV capacity so that LVM can identify the new EVS capacity. /dev/vdb specifies the physical volume where dockersys is located.
pvresize /dev/vdb
Information similar to the following is displayed:
Physical volume "/dev/vdb" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized
- Expand 100% of the free capacity to the logical volume. vgpaas/dockersys specifies the logical volume used by the container runtime.
lvextend -l+100%FREE -n vgpaas/dockersys
Information similar to the following is displayed:
Size of logical volume vgpaas/dockersys changed from <18.00 GiB (4607 extents) to <118.00 GiB (30208 extents).
Logical volume vgpaas/dockersys successfully resized.
- Adjust the capacity of the file system. /dev/vgpaas/dockersys specifies the file system path of the container runtime.
resize2fs /dev/vgpaas/dockersys
Information similar to the following is displayed:
Filesystem at /dev/vgpaas/dockersys is mounted on /var/lib/docker; on-line resizing required
old_desc_blocks = 3, new_desc_blocks = 15
The filesystem on /dev/vgpaas/dockersys is now 30932992 blocks long.
- Run the lsblk command to check the disk and partition capacity of the device and check whether the capacity has been expanded. If the new disk capacity was added to the dockersys, the capacity has been expanded.
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 8:0 0 50G 0 disk
└─vda1 8:1 0 50G 0 part /
vdb 8:16 0 200G 0 disk
├─vgpaas-dockersys 253:0 0 118G 0 lvm /var/lib/docker # dockersys after capacity expansion
├─vgpaas-thinpool_tmeta 253:1 0 3G 0 lvm
│ └─vgpaas-thinpool 253:3 0 67G 0 lvm
│ ...
├─vgpaas-thinpool_tdata 253:2 0 67G 0 lvm
│ └─vgpaas-thinpool 253:3 0 67G 0 lvm
│ ...
└─vgpaas-kubernetes 253:4 0 10G 0 lvm /mnt/paas/kubernetes/kubelet
Solution 2:
Create and delete files in service containers in the local storage (such as emptyDir and hostPath) or cloud storage directory mounted to the container. Such files do not occupy the thin pool space.
Solution 3:
If the OS uses OverlayFS, services can be deployed on such nodes to prevent the problem that the disk space occupied by files created or deleted in the container is not released immediately.
