Using NodeLocal DNSCache to Improve DNS Performance

Challenges

When the number of DNS requests in a cluster increases, the load of CoreDNS increases and the following issues may occur:

• Increased delay: CoreDNS needs to process more requests, which may slow down the DNS query and affect service performance.
• Increased resource usage: To ensure DNS performance, CoreDNS requires higher specifications.

Solution

To minimize the impact of DNS delay, deploy NodeLocal DNSCache in the cluster to improve the networking stability and performance. NodeLocal DNSCache runs a DNS cache proxy on cluster nodes. All pods with DNS configurations use the DNS cache proxy running on nodes instead of the CoreDNS service for domain name resolution. This reduces CoreDNS's load and improves the cluster DNS performance.

After NodeLocal DNSCache is enabled, a DNS query goes through the path as shown below.

Figure 1 NodeLocal DNSCache query path

Notes and Constraints

• Only clusters of version 1.19 or later support the NodeLocal DNSCache add-on.
• The node-local-dns-injection label is the system label used by NodeLocal DNSCache. Use this label only to prevent an automatic DNSConfig injection.

Installing the Add-on

CCE provides add-on NodeLocal DNSCache for you to install NodeLocal DNSCache.

NodeLocal DNSCache serves as a transparent caching proxy for CoreDNS and does not provide plug-ins such as hosts or rewrite. If you want to enable these plug-ins, modify the CoreDNS configurations.

1. (Optional) Modify the CoreDNS configuration so that the CoreDNS preferentially uses UDP to communicate with the upstream DNS server.

   The NodeLocal DNSCache uses TCP to communicate with the CoreDNS. The CoreDNS communicates with the upstream DNS server based on the protocol used by the request source. However, the cloud server does not support TCP. To use NodeLocal DNSCache, modify the CoreDNS configuration so that UDP is preferentially used to communicate with the upstream DNS server, preventing resolution exceptions.

   Perform the following operations. In the forward add-on, specify prefer_udp as the protocol used by requests. After the modification, CoreDNS preferentially uses UDP to communicate with the upstream system.

   1. Log in to the CCE console and click the cluster name to access the cluster console.
   2. In the navigation pane, choose Add-ons. Then, click Edit under CoreDNS.
   3. Edit the advanced configuration under Parameters and modify the following content in the plugins field:

      {
          "configBlock": "prefer_udp",
          "name": "forward",
          "parameters": ". /etc/resolv.conf"
      }

2. Log in to the CCE console and click the cluster name to access the cluster console. Choose Add-ons in the navigation pane, locate NodeLocal DNSCache on the right, and click Install.
3. On the Install Add-on page, select the add-on specifications and set related parameters.
   • DNSConfig Automatic Injection: After this function is enabled, a DNSConfig admission controller will be created. The controller intercepts pod creation requests in the namespace labeled with node-localdns-injection=enabled based on admission webhooks and automatically configures DNSConfig for pods. If this function is disabled or the pod belongs to a non-target namespace, you must manually configure DNSConfig for the pod.
     After automatic injection is enabled, you can customize the following configuration items for DNSConfig (supported when the add-on version is 1.6.7 or later):

     

     If DNSConfig has been configured in the pod when automatic injection is enabled, DNSConfig in the pod will be used first.

     • IP Address of DNS Server (optional): IP address list of the DNS server obtained when the container resolves the domain name. NodeLocal DNSCache and CoreDNS IP addresses are added by default. You have the option to add an additional IP address, but duplicates will be removed.
     • Search Domain (optional): a search list for host-name lookup. When a domain name cannot be resolved, DNS queries will be attempted combining the domain name with each domain in the search list in turn until a match is found or all domains in the search list are tried. You can add up to three extra search domains, but any duplicates will be removed.
     • ndots: specifies that if a domain name has fewer periods (.) than the specified value of ndots, it will be combined with the search domain list for DNS query. If the domain name still cannot be resolved, it will be used for DNS query. The system will perform a DNS query on a domain name if the number of periods (.) in it is greater than or equal to the value of ndots. If the domain name cannot be resolved correctly, the system will sequentially combine it with the search domain list and then perform a DNS query.
   • Target Namespace: This parameter is available after DNSConfig Automatic Injection is enabled. Only NodeLocal DNSCache of v1.3.0 or later supports this function.
     • All Enabled: CCE adds the node-local-dns-injection=enabled label to all created namespaces excluding built-in ones (such as kube-system), identifies namespace creation requests, and automatically adds the label to newly created namespaces.
     • Manual configuration: You must manually add the node-local-dns-injection=enabled label to the namespaces requiring the injection of DNSConfig. For details, see Managing Namespace Labels.

4. Click Install.

Using NodeLocal DNSCache

By default, application requests are sent through the CoreDNS proxy. To use node-local-dns as the DNS cache proxy, use any of the following methods:

• Auto injection: Automatically configure the dnsConfig field of the pod when creating the pod. (This function is not available for pods in system namespaces such as kube-system.)
• Manual configuration: Manually configure the dnsConfig field of the pod.

Auto injection

The following conditions must be met:

• Automatic DNSConfig injection has been enabled during the add-on installation.
• The node-local-dns-injection=enabled label has been added to the namespace. For example, run the following command to add the label to the default namespace:

  kubectl label namespace default node-local-dns-injection=enabled

• The new pod does not run in system namespaces such as kube-system and kube-public namespace.
• The node-local-dns-injection=disabled label for disabling DNS injection is not added to the new pod.
• The new pod's DNSPolicy is ClusterFirstWithHostNet. Alternatively, the pod does not use the host network and DNSPolicy is ClusterFirst.

After auto injection is enabled, the following dnsConfig settings are automatically added to the created pod. In addition to the NodeLocal DNSCache address 169.254.20.10, the CoreDNS address 10.247.3.10 is added to nameservers, ensuring high availability of the service DNS server.

...
  dnsConfig:
    nameservers:
      - 169.254.20.10
      - 10.247.3.10
    searches:
      - default.svc.cluster.local
      - svc.cluster.local
      - cluster.local
    options:
      - name: timeout
        value: ''
      - name: ndots
        value: '5'
      - name: single-request-reopen
...

Manual configuration

Manually add the dnsConfig settings to the pod.

Create a pod and add the NodeLocal DNSCache IP address 169.254.20.10 to the DNSConfig nameservers configuration.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx:alpine
    name: container-0
  dnsConfig:
    nameservers:
    - 169.254.20.10
    - 10.247.3.10
    searches:
    - default.svc.cluster.local
    - svc.cluster.local
    - cluster.local
    options:
    - name: ndots
      value: '2'
  imagePullSecrets:
  - name: default-secret

Common Issues

• How Do I Avoid an Automatic DNSConfig Injection?

  Solution:

  To prevent automatic DNSConfig injection for a workload, add node-local-dns-injection: disabled to the labels field in the pod template. Example:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: test
    namespace: default
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: test
    template:
      metadata:
        labels:
          app: test
          node-local-dns-injection: disabled   # Prevent automatic DNSConfig injection.
      spec:
        containers:
          - name: container-1
            image: nginx:latest
            imagePullPolicy: IfNotPresent
        imagePullSecrets:
          - name: default-secret