How Can I Handle an OOM Event in a Workload?

Symptom

If a process stops because an OOM event blocks the operation on a container, it may result in unexpected restart of the container.

Possible Causes

The container cannot use more memory than the limit. If the memory usage exceeds the limit, the container may be terminated and an OOM event is triggered. As a result, the container exits abnormally. For details about OOM events, see Assign Memory Resources to Containers and Pods.

Solution

1. Log in to the CCE console.
2. Click the cluster name to access the cluster console. In the navigation pane, choose Monitoring Center. If you have not enabled Monitoring Center, enable it as prompted. For details, see Enabling Monitoring Center.
3. Click the Pods tab on the right and click the name of the abnormal pod. On the displayed page, click the Monitoring tab. On this tab, you can view the monitoring information about the vCPU, memory, and network metrics of the pod, and container metrics.
4. View the memory increase curve to determine the abnormal container and the time when the exception occurred.
5. Check whether a memory leak occurred in the corresponding process of the pod based on the monitoring information, time of memory increase, logs, and process names.
   • If the OOM event is caused by a process memory leak, identify the root cause of the memory leak.
   • If the process is normal, adjust the memory limit of the pod based on service needs. It is advisable to ensure that the actual memory usage of a pod remains at or below 80% of the memory limit. The detailed operations are as follows:
     1. In the navigation pane, choose Workloads. On the right of the page, locate the target workload and click Upgrade in the Operation column.
        Figure 1 Workload list
        

     2. In the Container Settings area of the Upgrade Workload page, select a container, click Basic Info, and modify Memory Quota. The memory quota of a pod is the sum of the memory quotas of all containers. You can view Single-Pod Resource Flavor when confirming the specifications.
        Figure 2 Modifying the memory quota
        

     3. Click Upgrade Workload in the lower right corner. If the workload is in the Running state, the memory quota has been modified.