Why Is the Reported Container Memory Usage Inconsistent with the Auto Scaling Action?

Symptom

The memory usage of a container being monitored does not match the auto scaling requirements. For example, the GUI shows that memory usage of a container is around 40%, but the HPA scale-in threshold is set at 70%. The memory usage displayed on the GUI is below the HPA threshold, but no scale-in action has taken place.

Possible Causes

The way the container memory usage is calculated on the GUI differs from the method used for HPA auto scaling.

• The memory usage of a container displayed on the GUI: container_memory_rss/Memory limit of the container

  container_memory_rss specifies the resident set size (RSS). It includes some memory parts that may not be actively or effectively used.

• The memory usage of a container calculated for HPA auto scaling: container_memory_working_set_bytes/Memory request of the container

  container_memory_working_set_bytes specifies the working set size (WSS) and is calculated by doing as follows:

  Run cat /sys/fs/cgroup/memory/memory.stat in the pod to obtain the values of total_cache (cache memory), total_rss (memory used by the current application process), and total_inactive_file (memory used by inactive files).

  WSS = The value of total_cache + The value of total_rss - The value of total_inactive_file

If an application's memory usage displayed on the GUI is below the HPA scale-in threshold, but no scale-in action has been taken, or if the memory usage is not higher than the HPA scale-out threshold, but a scale-out happened anyways, the HPA scale-out behavior may not work as expected. This can occur when:

• The application cache usage is high, which can cause the WSS to be significantly greater than the RSS, resulting in the container memory usage displayed on the GUI being lower than the memory usage calculated by the HPA.
• The difference between the resource limit and request is large. The request may be significantly lower than the limit, causing the container memory usage shown on the GUI to be lower than the memory usage calculated by the HPA.