Why Does My Node Memory Usage Obtained by Running the kubelet top node Command Exceed 100%?

Symptom

The memory usage of the node obtained on the CCE console is not high, but that obtained by running the kubelet top node command exceeded 100%.

NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
192.168.0.243   79m          4%     2357Mi          109%

Possible Causes

kubectl top node calls the kubelet metrics API to obtain data, and the displayed information indicates the total number of used resources on the node divided by all allocatable resources.

For details, see https://github.com/kubernetes/kubernetes/issues/86499.

Example Scenarios

To obtain the parameters of a node, run kubectl describe node. The following is an example:

...
Capacity:
  cpu:                2
  ephemeral-storage:  51286496Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  localssd:           0
  localvolume:        0
  memory:             3494556Ki
  pods:               40
Allocatable:
  cpu:                1960m
  ephemeral-storage:  47265634636
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  localssd:           0
  localvolume:        0
  memory:             2213604Ki
  pods:               40
...

• The Capacity.memory field whose value is 4030180Ki indicates the total memory of the node.
• The Allocatable.memory field whose value is 2213604Ki indicates the allocatable memory of the node.
• The value of the node's used memory in this example is 2413824Ki. To obtain the value, run the following command:

  kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/

  Information similar to the following will be displayed:

  {
  	"kind": "NodeMetricsList",
  	"apiVersion": "metrics.k8s.io/v1beta1",
  	"metadata": {},
  	"items": [
  		{
  			...
  			"timestamp": "2023-08-15T14:09:38Z",
  			"window": "1m0.177s",
  			"usage": {
  				"cpu": "78528126n",
  				"memory": "2413824Ki"
  			}
  		}
  	]
  }

To check the memory usage of the node, run kubelet top node.

Memory usage of a node = Used memory of the node/Allocatable memory of the node = 2413824Ki/2213604Ki = 109%

The actual memory usage of the node is calculated as follows:

Actual memory usage of the node = Used memory of the node/Total memory of the node = 2413824Ki/4030180Ki = 59.9%