Help Center/ Cloud Container Engine/ Best Practices/ Networking/ Pre-Binding Container ENI for CCE Turbo Clusters
Updated on 2024-05-31 GMT+08:00
View PDF
Share

Pre-Binding Container ENI for CCE Turbo Clusters

In the Cloud Native 2.0 network model, each pod is allocated an ENI or a sub-ENI (called container ENI). The speed of ENI creation and binding is slower than that of pod scaling, severely affecting the container startup speed in large-scale batch creation. Therefore, the Cloud Native Network 2.0 model provides the dynamic pre-binding of container ENIs to accelerate pod startup while improving IP resource utilization.

Constraints

  • CCE Turbo clusters of 1.19.16-r4, 1.21.7-r0, 1.23.5-r0, 1.25.1-r0, or later support ENI pre-binding, global configuration at the cluster level, and custom settings at the node pool level. Custom settings of nodes out of a node pool is not supported.
  • CCE Turbo clusters of 1.19.16-r2, 1.21.5-r0, 1.23.3-r0 to 1.19.16-r4, 1.21.7-r0, 1.23.5-r0 only support two parameters, nic-minimum-target and nic-warm-target, and do not support custom settings at the node pool level.
  • Modify the dynamic pre-binding parameters using the console or API instead of the node annotations in the background. Otherwise, the modified annotations will be overwritten by the original values after the cluster is upgraded.
  • CCE Turbo clusters of 1.19.16-r4, 1.21.7-r0, 1.23.5-r0, 1.25.1-r0, or earlier support high and low threshold for ENI buffers. If users have enabled this feature, the original high and low threshold for ENI pre-binding parameters is automatically converted to the dynamic pre-binding parameters of the container ENIs. If you want to modify the dynamic pre-binding parameters of the container ENIs on the console, change the original high and low threshold to 0:0 on the cluster configuration management console.
  • For BMS node pools in CCE Turbo clusters earlier than 1.19.16-r4, 1.21.7-r0, 1.23.5-r0, and 1.25.1-r0, the high and low thresholds for ENI pre-binding (0.3:0.6 by default) are used by default. After the cluster is upgraded, the original high and low threshold still takes effect. You are advised to convert the high and low threshold parameters to the dynamic pre-binding parameters of container ENIs on the configuration management console of the node pool and delete the high and low threshold configuration to enable the latest dynamic pre-binding parameters.
  • In the non-node pool BMS scenario of CCE Turbo, clusters of version earlier than 1.19.16-r4, 1.21.7-r0, 1.23.5-r0 and 1.25.1-r0 use the high and low threshold for ENI buffers (0.3:0.6 by default) by default. After the cluster is upgraded, the original high and low threshold still takes effect. If you want to enable the cluster-level global configuration, delete the annotation (node.yangtse.io/eni-warm-policy) of the node in the background.

How It Works

CCE Turbo provides four dynamic pre-binding parameters for container ENIs. You can properly configure the parameters based on your service requirements. (The node pool-level dynamic ENI pre-binding parameters take priority over the cluster-level dynamic ENI pre-binding parameters.)

Table 1 Parameters of the dynamic ENI pre-binding policy

Parameter

Default Value

Description

Suggestion

nic-minimum-target

10

Minimum number of container ENIs bound to a node.

The parameter value must be a positive integer. The value 10 indicates that there are at least 10 container ENIs bound to a node. If the number you entered exceeds the container ENI quota of the node, the ENI quota will be used.

Configure these parameters based on the number of pods.

nic-maximum-target

0

If the number of ENIs bound to a node exceeds the value of nic-maximum-target, the system does not proactively pre-bind ENIs.

If the value of this parameter is greater than or equal to the value of nic-minimum-target, the check on the maximum number of the pre-bound ENIs is enabled. Otherwise, the check is disabled.

The parameter value must be a positive integer. The value 0 indicates that the check on the upper limit of pre-bound container ENIs is disabled. If the number you entered exceeds the container ENI quota of the node, the ENI quota will be used.

Configure these parameters based on the number of pods.

nic-warm-target

2

Minimum number of pre-bound ENIs on a node. The value must be a number.

When the value of nic-warm-target + the number of bound ENIs is greater than the value of nic-maximum-target, the system will pre-bind ENIs based on the difference between the value of nic-maximum-target and the number of bound ENIs.

Set this parameter to the number of pods that can be scaled out instantaneously within 10 seconds.

nic-max-above-warm-target

2

Only when the number of idle ENIs on a node minus the value of nic-warm-target is greater than the threshold, the pre-bound ENIs will be unbound and reclaimed. The value can only be a number.

  • Setting a larger value of this parameter slows down the recycling of idle ENIs and accelerates pod startup. However, the IP address usage decreases, especially when IP addresses are insufficient. Therefore, exercise caution when increasing the value of this parameter.
  • Setting a smaller value of this parameter accelerates the recycling of idle ENIs and improves the IP address usage. However, when a large number of pods increase instantaneously, the startup of some pods slows down.

Set this parameter based on the difference between the number of pods that are frequently scaled on most nodes within minutes and the number of pods that are instantly scaled out on most nodes within 10 seconds.

Configuration Example

Level

Service Scenario

Configuration Example

Cluster

All nodes use the c7.4xlarge.2 model (sub-ENI quota: 128).

Most nodes run about 20 pods.

Most nodes can run a maximum of 60 pods.

Most nodes can scale out 10 pods within 10 seconds.

Most nodes frequently scale in or out 15 pods within minutes.

Cluster-level global configuration:

  • nic-minimum-target: 20
  • nic-maximum-target: 60
  • nic-warm-target: 10
  • nic-max-above-warm-target: 5

Node pool

A node pool that uses the c7.8xlarge.2 high-specification model is created in the cluster. (sub-ENI quota: 256)

Most nodes run about 100 pods.

Most nodes can run a maximum of 128 pods.

Most nodes can scale out 10 pods within 10 seconds.

Most nodes frequently scale in or out 12 pods within minutes.

Custom settings at the node pool level:

  • nic-minimum-target: 100
  • nic-maximum-target: 120
  • nic-warm-target: 10
  • nic-max-above-warm-target: 2

Pods using HostNetwork are excluded.

Cluster-level Global Configuration

  1. Log in to the CCE console. In the navigation pane, choose Clusters.
  2. Click next to the target cluster and choose Manage.

    Figure 1 Managing a cluster

  3. In the window that slides out from the right, click Networking Components. For details about the parameter configurations, see Configuration Example.

  4. After the configuration is complete, click OK. Wait for about 10 seconds for the configuration to take effect.

Custom Settings at the Node Pool Level

  1. Log in to the CCE console.
  2. Click the cluster name to access the cluster console, choose Nodes in the navigation pane, and click the Node Pools tab.
  3. Locate the row containing the target node pool and click Manage.
  4. In the window that slides out from the right, click Networking Components and enable node pool container ENI pre-binding. For details about the parameter configurations, see Configuration Example.

  5. After the configuration is complete, click OK. Wait for about 10 seconds for the configuration to take effect.
Parent Topic: Networking