How Do I Expand the Disk Capacity of the CIA Add-on in an On-Premises Cluster?

Symptom

When the disk in the PVC storage on which the CIA add-on (kube-prometheus-stack) depends is full, the no space left on device error message is reported in the standard log output of prometheus-server-0 Pod. In this case, Prometheus cannot work properly. You need to expand the disk capacity of the node where the PVC is located and restart prometheus-server-0 Pod.

Procedure

1. Check the PVC information of Prometheus and obtain the node and path where the PV bound to the PVC is located.

   kubectl describe pvc pvc-prometheus-server-0 -nmonitoring|grep volume.kubernetes.io/selected-node

   kubectl describe pvc pvc-prometheus-server-0 -nmonitoring|grep volume.kubernetes.io/targetPath

2. Log in to the storage node, run the df -TH command to query the disk configuration of the node, and expand the capacity according to the disk configuration. After the capacity expansion is complete, you need to partition the disk. For details, see Extending Disk Partitions and File Systems (Linux) in the Elastic Volume Service User Guide.
3. After the capacity expansion and partitioning are complete, restart Prometheus.

   kubectl delete pod prometheus-server-0 -nmonitoring

   

   If prometheus-server-0 is restarted, the CIA function of the container will be unavailable during the restart. Select an appropriate restart time.