How Do I Handle Unexpected Restarts?

Possible Cause 1: Out of Memory (OOM)

• Symptom

  One or more processes used up all physical memory and swap space. To prevent a system hang, the kernel's OOM Killer forcibly terminates the process that occupies the most memory. This may cause a chain reaction and the system restarts.

• Check method

  Check whether the /var/log/messages file contains keywords like "Out of memory: Killed process" or "invoked oom-killer".

• Solution
  1. Check service processes to see if there is memory leak, which causes OOM.
  2. If more memory is required by services, upgrade the ECS memory specifications.

Possible Cause 2: Kernel Panic

• Symptom

  The kernel detects a severe, unrecoverable error (such as a driver bug, kernel bug, and hardware communication problem). The system will be frozen and may be automatically restarted.

• Check method

1. Check whether the /var/log/messages file contains keywords like "Kernel panic - not syncing" and "Oops".
2. Check the automatic restart settings.

   cat /proc/sys/kernel/panic           
   # If the output is 0, the system is suspended after a panic occurs and waits for manual processing.
   # If the output is N, the system automatically restarts N seconds after a panic occurs.

• Solution
  1. Roll back the kernel or drivers. If the problem occurs after a recent update, roll back the kernel or related drivers (such as graphics and network drivers) to the previous stable version.
  2. Capture onsite information. To preserve the scene if the fault occurs again, you can temporarily disable automatic restart.

     echo 0 > /proc/sys/kernel/panic # Temporarily disabling automatic restart

Possible Cause 3: Scheduled Restart

• Symptom

  The system is scheduled to restart.

• Check method

  Check scheduled tasks.

  sudo grep -r "reboot\|shutdown" /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/ /var/spool/cron/

• Solution

  Locate the scheduled restart task and delete or comment it out.