How Do I Troubleshoot "nf_conntrack:table full, dropping packet"?

Symptom

A timeout error occurred when you accessed a website. You got a lot of log messages kernel nf_conntrack: table full, dropping packet in /var/log/messages.

Figure 1 System logs

Scenarios

The operations in this section only apply to CentOS with firewalls enabled.

Constraints

The operations in this section involve modifying kernel parameters at runtime, which may render kernel unstable, requiring system reboot.

Possible Cause

The connection-tracking module within iptables stores connections in the conntrack table. table full, dropping packet indicates that the table is full and new entries cannot be created for new connections. As a result, packet dropping occurs. This problem can be solved by increasing the number of allowed entries for tracked connections.

Solution for CentOS 6

1. Run the following command to check the value of nf_conntrack_max:

   sysctl net.netfilter.nf_conntrack_max

2. Run the following command to check the number of tracked connections:

   cat /proc/sys/net/netfilter/nf_conntrack_count

   If the value of nf_conntrack_max is reached, packet dropping occurs.

3. Set a larger value for net.netfilter.nf_conntrack_max. The following uses an ECS with 64 GB of memory as an example and uses 2097152 as the value of net.netfilter.nf_conntrack_max.

   Run the following command for the configuration to take effect:

   sysctl -w net.netfilter.nf_conntrack_max=2097152

   Run the following command to ensure that the configurations are still valid after the ECS is restarted:

   echo "net.netfilter.nf_conntrack_max = 2097152" >> /etc/sysctl.conf

   

   • Set .net.netfilter.nf_conntrack_max based on the memory size of an ECS.
   • Use the following rule to calculate an appropriate value for nf_conntrack_max:

     CONNTRACK_MAX = RAMSIZE (in bytes)/16384/2

     For an ECS running a 64-bit OS with 64 GB of memory, the most appropriate value for .net.netfilter.nf_conntrack_max is 2097152.

     CONNTRACK_MAX = 64 x 1024 x 1024 x 1024/16384/2 = 2097152

4. If the number of entries in the conntrack table increases significantly, for example, by four times the number of tracked entries, increase the size of the hash table for storing conntrack entries.

   For CentOS 6 and later versions, calculate a new hash value using rule hashsize = conntrack_max/4.

5. Run the following command to set the size of the hash table to 131072:

   echo "options nf_conntrack expect_hashsize=524288 hashsize=524288" >/etc/modprobe.conf

6. Run the following command to restart iptables:

   service iptables restart

Solution for CentOS 7

1. Run the following command to change the size of the hash table for conntrack connections in /etc/modprobe.d/firewalld-sysctls.conf:

   For CentOS 6 and later versions, calculate a new hash value using rule hashsize = conntrack_max/4.

   echo "options nf_conntrack expect_hashsize=131072 hashsize=131072" >> /etc/modprobe.d/firewalld-sysctls.conf

2. Run the following command to restart firewalld:

   systemctl restart firewalld

3. Run the following command to check whether the proceeding configurations have taken effect:

   sysctl -a |grep nf_conntrack_max

For more information, see Red Hat Customer Portal.