Changing Process ID Limits (kernel.pid_max)

Context

Process IDs (PIDs) are a fundamental resource on nodes. It is trivial to hit the task limit without hitting any other resource limits, which can then cause instability to a host machine.

You can adjust the PID limit (kernel.pid_max) according to service requirements.

kernel.pid_max Defaults

Starting from January 2022, CCE changes the default value of kernel.pid_max to 4194304 for EulerOS 2.5, CentOS 7.6, and Ubuntu 18.04 nodes in clusters of v1.17 or later. Specific conditions:

• Cluster version: v1.17.17 or later
• Node creation: after January 30, 2022

If the preceding two conditions are not met, kernel.pid_max on EulerOS 2.5, CentOS 7.6, and Ubuntu 18.04 nodes defaults to 32768.

Change Suggestion

• EulerOS 2.3: Change the default to 4194304 for all nodes. For details, see Changing kernel.pid_max of a Node. Use a pre-installation script to do so for new nodes and node pools. For details, see Configuring kernel.pid_max When Creating a Node Pool or Configuring kernel.pid_max When Creating a Node.
• EulerOS 2.5, CentOS 7.6, and Ubuntu 18.04:
  • Change the value of kernel.pid_max to 4194304 for nodes created on January 30, 2022 or earlier in clusters of v1.17.17 or later. For details, see Changing kernel.pid_max of a Node.
  • For clusters of 1.17.9 and earlier:
    • Change the value of kernel.pid_max to 4194304 for existing nodes. For details, see Changing kernel.pid_max of a Node.
    • Use a pre-installation script to do so for new nodes and node pools. For details, see Configuring kernel.pid_max When Creating a Node Pool or Configuring kernel.pid_max When Creating a Node.

Viewing kernel.pid_max

Log in to the node and run the following command to obtain the value of kernel.pid_max:

sysctl kernel.pid_max

# sysctl kernel.pid_max
kernel.pid_max = 32768

Change kernel.pid_max, if necessary, as instructed in Changing kernel.pid_max of a Node.

Checking Node PIDs

Log in to the node and run the following command to check how many PIDs are in use:

ps -eflL | wc -l

# ps -eflL | wc -l
691

Changing kernel.pid_max of a Node

Log in to the node and run the following command. 4194304 indicates the value of kernel.pid_max and is used as an example here.

echo kernel.pid_max = 4194304 >> /etc/sysctl.conf && sysctl -p

echo 4194304 > /sys/fs/cgroup/pids/kubepods/pids.max

Run the following commands to check whether the returned value is the same as that you configured:

# sysctl kernel.pid_max
kernel.pid_max = 4194304
# cat /sys/fs/cgroup/pids/kubepods/pids.max
4194304

Configuring kernel.pid_max When Creating a Node Pool

EulerOS 2.3: Configuration required.

EulerOS 2.5, CentOS 7.6, and Ubuntu 18.04: Configuration required for clusters of v1.17.9 and earlier. Configuration NOT required for clusters of v1.17.17 and later because the value has been changed.

You can configure kernel.pid_max in the pre-installation script to create a node from a node pool.

When creating a node pool, choose Advanced Settings > Post-installation Command and add the following command:

echo kernel.pid_max = 4194304 >> /etc/sysctl.conf && sysctl -p

Configuring kernel.pid_max When Creating a Node

EulerOS 2.3: Configuration required.

EulerOS 2.5, CentOS 7.6, and Ubuntu 18.04: Configuration required for clusters of v1.17.9 and earlier. Configuration NOT required for clusters of v1.17.17 and later because the value has been changed.

You can configure kernel.pid_max by using the pre-installation script when creating a node.

Choose Advanced Settings > Post-installation Command and add the following command:

echo kernel.pid_max = 4194304 >> /etc/sysctl.conf && sysctl -p