Why Does My Node Memory Usage Obtained by Running the kubelet top node Command Exceeds 100%?
Symptom
The memory usage of the node obtained on the CCE console is not high, but that obtained by running the kubelet top node command exceeded 100%.
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 192.168.0.243 79m 4% 2357Mi 109%
Possible Causes
kubectl top node calls the kubelet metrics API to obtain data, and the displayed information indicates the total number of used resources on the node divided by all allocatable resources.
For details, see https://github.com/kubernetes/kubernetes/issues/86499.
Example Scenarios
To obtain the parameters of a node, run kubectl describe node. The following is an example:
... Capacity: cpu: 2 ephemeral-storage: 51286496Ki hugepages-1Gi: 0 hugepages-2Mi: 0 localssd: 0 localvolume: 0 memory: 3494556Ki pods: 40 Allocatable: cpu: 1960m ephemeral-storage: 47265634636 hugepages-1Gi: 0 hugepages-2Mi: 0 localssd: 0 localvolume: 0 memory: 2213604Ki pods: 40 ...
- The Capacity.memory field whose value is 4030180Ki indicates the total memory of the node.
- The Allocatable.memory field whose value is 2213604Ki indicates the allocatable memory of the node.
- The value of the node's used memory in this example is 2413824Ki. To obtain the value, run the following command:
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/
Information similar to the following will be displayed:
{ "kind": "NodeMetricsList", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": {}, "items": [ { ... "timestamp": "2023-08-15T14:09:38Z", "window": "1m0.177s", "usage": { "cpu": "78528126n", "memory": "2413824Ki" } } ] }
To check the memory usage of the node, run kubelet top node.
Memory usage of a node = Used memory of the node/Allocatable memory of the node = 2413824Ki/2213604Ki = 109%
The actual memory usage of the node is calculated as follows:
Actual memory usage of the node = Used memory of the node/Total memory of the node = 2413824Ki/4030180Ki = 59.9%
Node Running FAQs
- What Should I Do If a Cluster Is Available But Some Nodes Are Unavailable?
- How Do I Troubleshoot the Failure to Remotely Log In to a Node in a CCE Cluster?
- How Do I Log In to a Node Using a Password and Reset the Password?
- How Do I Collect Logs of Nodes in a CCE Cluster?
- What Can I Do If the Container Network Becomes Unavailable After yum update Is Used to Upgrade the OS?
- What Should I Do If the vdb Disk of a Node Is Damaged and the Node Cannot Be Recovered After Reset?
- Which Ports Are Used to Install kubelet on CCE Cluster Nodes?
- How Do I Configure a Pod to Use the Acceleration Capability of a GPU Node?
- What Should I Do If I/O Suspension Occasionally Occurs When SCSI EVS Disks Are Used?
- What Should I Do If Excessive Docker Audit Logs Affect the Disk I/O?
- How Do I Fix an Abnormal Container or Node Due to No Thin Pool Disk Space?
- Which Ports Does a Node Listen On?
- How Do I Rectify Failures When the NVIDIA Driver Is Used to Start Containers on GPU Nodes?
- What Should I Do If a Node Does Not Synchronize with the NTP Clock Source?
- What Should I Do If the Data Disk Usage Is High Because a Large Volume of Data Is Written Into the Log File?
- Why Does My Node Memory Usage Obtained by Running the kubelet top node Command Exceeds 100%?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore