How Can I Troubleshoot High Disk Usage on a Node?

Symptom

The disk usage on a node is high. After the execution of df -h, the command output shows that the disk usage exceeds 90%.

Possible Cause

The disk usage on a node may be high due to the following causes:

• There are large files on the disk.
• A large amount of pod data is written to the host node because the pods have the hostPath or emptyDir volumes mounted.
• The pods continuously generate logs, but without a configured log retention period, the logs accumulate over time and grow excessively large.

Fault Locating

• Cause 1: Check whether there are any large files on the disk.
  Run find to search for large files. For example, to search for all files larger than 1 GiB, run the following command:

  find / -type f -size +1G

  This command searches for all files (including hidden files) whose size exceeds 1 GiB from the root directory (/).

  Some examples are shown below.

  • Search for files larger than 10 KB in the current folder.

    

  • Search for files larger than 100 MiB in the root directory.

    

• Cause 2: Obtain the pod mount path.

  Obtain the mounting mode of a pod volume.

  kubectl describe pods | grep -Pz  Volumes

  Information similar to that shown in the figure below is displayed.

  

  If a pod has a hostPath or emptyDir volume mounted, check the file size in the mount path folder.

• Cause 3: Check the log storage settings.
  • Docker nodes:
    Run the following command:

    cat /etc/docker/daemon.json

    Example output:

    {
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "10m",
        "max-file": "3"
      }
    }

  • containerd nodes:

    Check the kubelet configuration file.

    cat /opt/cloud/cce/kubernetes/kubelet/kubelet_config.yaml

    Example output:

    containerLogMaxSize: 50Mi
    containerLogMaxFiles: 20

Suggestions

1. Delete unnecessary files.
   The following commands are for your reference.

   • List all files in the current directory.

     ls -la

   • Delete unnecessary files.

     rm -f <file_name>

2. Use cloud storage for pods where a large amount of data needs to be written. For details, see Storage Overview.
3. Configure maxSize and maxFile for Docker or containerd log files.
   

   Docker or kubelet needs to be restarted after the configuration is updated. Exercise caution when performing this operation.

   containerd node pools

   1. Log in to the CCE console.
   2. Click the cluster name to access the cluster console. In the navigation pane, choose Nodes. In the right pane, click the Node Pools tab.
   3. Locate the row containing the target node pool and choose More > Manage.
   4. In the window that slides out from the right, find the Maximum Number of Container Log Files (container-log-max-files) and Maximum Container Log File Size (container-log-max-size) in the kubelet area.
   5. Enter a new value in each text box.

      

   6. Save the changes.

   Docker node pools

   1. Log in to the CCE console.
   2. Click the cluster name to access the cluster console. In the navigation pane, choose Nodes. In the right pane, click the Node Pools tab.
   3. Locate the row containing the target node pool and choose More > Manage.
   4. In the window that slides out from the right, find the Maximum Container Log File Size (max-size) and Maximum Number of Container Log Files (max-file) parameters in the Docker area.
   5. Enter a new value in each text box.

      

   6. Save the changes.
4. Periodically delete unnecessary images from the node.
   • containerd nodes
     1. Obtain local images on the node.

        crictl images -v

     2. Delete the unnecessary images by image ID.

        crictl rmi {Image ID}

   • Docker nodes
     1. Obtain local images on the node.

        docker images

     2. Delete the unnecessary images by image ID.

        docker rmi {image-ID}