How Do I Handle OOM Issues in a Linux ECS?

Background

Out of Memory (OOM) occurs when all available memory is exhausted and the system is unable to allocate memory for processes, which will trigger a kernel panic or OOM killer. On Linux, OOM killer is a process that prevents other processes from collectively exhausting the host's memory. When the system is critically low on memory, the processes that use more memory than available will be killed to ensure the overall availability of the system.

Symptom

When an ECS is running, information similar to the following is recorded in the log:

[  327.199955] 5202 total pagecache pages
[  327.200965] 0 pages in swap cache
[  327.201950] Swap cache stats: add 0, delete 0, find 0/0
[  327.203656] Free swap  = 0kB
[  327.204580] Total swap = 0kB
[  327.205491] 2096926 pages RAM
[  327.206375] 0 pages HighMem/MovableOnly
[  327.207509] 94837 pages reserved
[  327.208746] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  327.210727] [  352]     0   352     9764      616      22        0             0 systemd-journal
[  327.212897] [  372]     0   372    11370      150      23        0         -1000 systemd-udevd
[  327.214662] [  410]     0   410    13883      102      27        0         -1000 auditd
[  327.216276] [  498]    81   498    14558      164      33        0          -900 dbus-daemon
[  327.217880] [  499]     0   499     5385       62      16        0             0 irqbalance
[  327.219369] [  500]   999   500   153060     1350      63        0             0 polkitd
[  327.221158] [  502]     0   502   243884     3537      36        0             0 uniagent
[  327.222916] [  503]     0   503   137033     1033      84        0             0 NetworkManager
[  327.225002] [  505]     0   505     6596       77      19        0             0 systemd-logind
[  327.226841] [  508]   998   508    29452      105      30        0             0 chronyd
[  327.228427] [  529]     0   529    26123      488      21        0             0 uniagentd
[  327.230120] [  532]     0   532   143572     2883      96        0             0 tuned
[  327.231779] [  563]     0   563    25753      514      51        0             0 dhclient
[  327.233396] [  608]     0   608    54632      688      42        0             0 rsyslogd
[  327.235004] [  839]     0   839    39781      400      42        0             0 hostwatch
[  327.236505] [  904]     0   904   335577     1654     112        0             0 [kworker/0:0H]
[  327.237967] [ 1059]     0  1059    22452      253      45        0             0 master
[  327.239371] [ 1074]    89  1074    22478      250      44        0             0 pickup
[  327.240758] [ 1075]    89  1075    22495      252      45        0             0 qmgr
[  327.242087] [ 1416]     0  1416    28251      259      57        0         -1000 sshd
[  327.243474] [ 1418]     0  1418     6477       51      18        0             0 atd
[  327.244773] [ 1420]     0  1420    31597      152      18        0             0 crond
[  327.246133] [ 1422]     0  1422    24842      166      51        0             0 login
[  327.247893] [ 1423]     0  1423    27552       32      10        0             0 agetty
[  327.249319] [ 2359]     0  2359   197550     4903      48        0             0 containerserver
[  327.250828] [ 7726]     0  7726    63736      535      22        0             0 uniagentd
[  327.252278] [ 7727]     0  7727    98075      305      26        0             0 uniagentd
[  327.253688] [ 7773]     0  7773    28886      101      13        0             0 bash
[  327.254906] [ 7840]     0  7840   225333     3183      34        0             0 telescope
[  327.256240] [ 7922]     0  7922    39390      374      83        0             0 sshd
[  327.257496] [ 7924]     0  7924    28887      102      13        0             0 bash
[  327.258769] [ 7943]     0  7943    18088      190      39        0             0 sftp-server
[  327.260041] [ 8076]     0  8076  1958387  1904796    3739        0             0 python3
[  327.261301] Out of memory: Kill process 8076 (python3) score 924 or sacrifice child
[  327.262504] Killed process 8076 (python3), UID 0, total-vm:7833548kB, anon-rss:7619112kB, file-rss:72kB, shmem-rss:0kB

Possible Causes

If OOM frequently occurs when you run programs on a Linux instance, the possible causes are as follows:

• The configuration of the Linux instance is too low to meet the memory required for program running, causing OOM.
• The application code uses the memory improperly, causing OOM.

Solution

To handle OOM issues, perform the following steps:

1. Run the following commands to view OOM-related logs:

   sudo less /var/log/messages    # CentOS 
   sudo less /var/log/syslog      # Ubuntu

   Search for the keyword oom or kill in the log to view the log information related to OOM issues.

   The log contains the standard output header line generated when the OOM Killer evaluates processes. You can sort the processes by the rss column (unit: 4 KB) to view the process that consumes the most memory.

   

   If the memory OOM is caused by high memory usage of service processes, you are advised to further locate the cause of high memory usage based on service application logs.

2. You can select a solution based on the check result.

   Possible Cause	Solution
   The configuration of the Linux instance is too low to meet the memory required for program running.	Upgrade the instance memory configuration by referring to Modifying ECS Specifications (vCPUs and Memory). CAUTION: During the modification, you need to stop and restart the ECS. You are advised to perform this operation during off-peak hours to reduce the impact on your services.
   The application code uses the memory improperly.	Optimize the application code, for example, paginating batch query requests.

Related Information

1. Table 1 describes the standard output header line generated when the OOM Killer evaluates processes.

   Table 1 Standard output header line generated when the OOM Killer evaluates processes

   Column Name	Full Name	Description
   pid	Process ID	The ID that the operating system assigns internally to each process.
   uid	User ID	The ID of the user who started the process. The value 0 indicates the root user.
   tgid	Thread Group ID	The thread group ID. For a single-threaded process, this is the same as pid. For a multi-thread program, all threads share the same tgid.
   total_vm	Total Virtual Memory	The total size of the virtual memory occupied by the process (measured in 4 KB pages). This value is large because it includes shared libraries, mapped files, and more. It is not the best metric for measuring memory pressure.
   rss	Resident Set Size	The size of the resident set (measured in 4 KB pages). This is the total non-swap memory currently occupied by the process in the physical memory. This is one of the most important factors considered by the OOM Killer, as it directly reflects the physical memory consumption of processes.
   nr_ptes	Number of Page Table Entries	The number of page table entries. The more virtual memory a process uses, the more page table entries it requires. A page table entry itself consumes the memory, so it is also a factor for OOM scoring.
   swapents	Swap Entries	The size of the memory swapped out to the swap partition (measured in 4 KB pages). A higher value indicates that some memory of the process is not currently in the physical RAM.
   oom_score_adj	OOM Score Adjust	OOM adjustment value (value ranges: –1000 to 1000). This is the most important parameter that can be set in user mode to actively influence the OOM Killer's decision-making.
   name	Process Name	The name of the executable file of a process.

2. Table 2 lists the OOM-related parameters.

   Table 2 OOM-related parameters

   Parameter	Description	Value	How to Change
   panic_on_oom	The panic_on_oom parameter controls what the system should do when it encounters OOM errors. When an OOM error occurs, the system has two choices: Trigger a kernel panic, during which the system may break down frequently. Trigger OOM Killer to terminate certain processes and release the memory so that the system can be used normally.	You can run either of the following commands to view the parameter value: cat /proc/sys/vm/panic_on_oom or sysctl -a | grep panic_on_oom If the value is set to 0, OOM Killer is triggered when the memory is insufficient. If the value is set to 1, either OOM Killer or kernel panic will be triggered. If the value is set to 2, a kernel panic will be forcibly triggered. As a result, the system restarts.	For example, to set the value to 0, use either of the following methods: Temporary configuration: The configuration takes effect immediately. However, after the system is restarted, the value changes to the default one. sysctl -w vm.panic_on_oom=0 Persistent configuration: The configuration still takes effect after the system is restarted. Run vim /etc/sysctl.conf, add vm.panic_on_oom =0 to the configuration file, and then run sysctl –p or restart the system for the configuration to take effect.
   oom_kill_allocating_task	Determines which processes are selected when the system triggers OOM Killer and attempts to end some processes. The options are as follows: Process that triggers OOM Process with the highest oom_score value	You can run either of the following commands to view the parameter value: cat /proc/sys/vm/oom_kill_allocating_task or sysctl -a | grep oom_kill_allocating_task If the value is set to 0, the process with the highest oom_score value is selected. If the value is a non-zero value, the process that triggers OOM is selected.	For example, to set the value to 1, use either of the following methods: Temporary configuration: The configuration takes effect immediately. However, after the system is restarted, the value changes to the default one. sysctl -w vm.oom_kill_allocating_task=1 Persistent configuration: The configuration still takes effect after the system is restarted. Run vim /etc/sysctl.conf, add vm.oom_kill_allocating_task=1 to the configuration file, and then run sysctl –p or restart the system for the configuration to take effect.
   oom_score	Indicates the score of a process, which consists of two parts: System score: The system automatically calculates the score based on the memory usage of the process. User score: This is the oom_score_adj score, which can be customized.	You can adjust the value of oom_score_adj to adjust the final score of a process. You can run the following command to view the parameter value: cat /proc/Process ID/oom_score_adj If the value is set to 0, the oom_score value is not adjusted. If the value is negative, the probability of the process getting picked and terminated by OOM Killer is reduced. If the value is positive, the probability of the process getting picked and terminated by OOM Killer is increased.	For example, to set oom_score_adj to 1000 for the process whose ID is 2939, run the following command: echo 1000 > /proc/2939/oom_score_adj
   oom_dump_tasks	The oom_dump_tasks parameter controls whether to record system process information and OOM Killer information when an OOM event occurs. For example, dump information about all user-space processes related to memory usage, including the process ID, memory usage of each process, and page table information of each process. This information helps you understand the cause of the OOM event.	You can run either of the following commands to view the parameter value: cat /proc/sys/vm/oom_dump_tasks or sysctl -a | grep oom_dump_tasks If the value is set to 0, related information is not printed when OOM occurs. If the value is not 0, a system-wide task dump is produced to print the memory usage of all tasks in the system in the following scenarios: A kernel panic is triggered due to OOM. The process to be terminated is not found. The process is found and terminated.	For example, to set the value to 0, use either of the following methods: Temporary configuration: The configuration takes effect immediately. However, after the system is restarted, the value changes to the default one. sysctl –w vm.oom_dump_tasks=0 Persistent configuration: The configuration still takes effect after the system is restarted. Run vim /etc/sysctl.conf, add vm.oom_dump_tasks=0 to the configuration file, and then run sysctl –p or restart the system for the configuration to take effect.