Using NodeLocal DNSCache to Improve DNS Performance

Challenges

When the number of DNS requests in a cluster increases, the load of CoreDNS increases and the following issues may occur:

• Increased delay: CoreDNS needs to process more requests, which may slow down the DNS query and affect service performance.
• Increased resource usage: To ensure DNS performance, CoreDNS requires higher specifications.

Solution

To minimize the impact of DNS delay, deploy NodeLocal DNSCache in the cluster to improve the networking stability and performance. NodeLocal DNSCache runs a DNS cache proxy on cluster nodes. All pods with DNS configurations use the DNS cache proxy running on nodes instead of the CoreDNS service for domain name resolution. This reduces CoreDNS's load and improves the cluster DNS performance.

After NodeLocal DNSCache is enabled, a DNS query goes through the path as shown below.

Figure 1 NodeLocal DNSCache query path

Notes and Constraints

• Only clusters of version 1.19 or later support the NodeLocal DNSCache add-on.
• CCI does not support NodeLocal DNSCache. When the workload is dynamically scaled to CCI, pods on CCI cannot access NodeLocal DNSCache. As a result, domain name resolution times out. Therefore, when creating a workload that is scaled to CCI, add the node-local-dns-injection: disabled label to the pod to prevent automatic injection of DNSConfig. For details, see Common Issues.
• The node-local-dns-injection label is the system label used by NodeLocal DNSCache. Use this label only to prevent an automatic DNSConfig injection.

Installing the Add-on

CCE provides add-on NodeLocal DNSCache that simplifies the installation of NodeLocal DNSCache.

NodeLocal DNSCache serves as a transparent caching proxy for CoreDNS and does not provide plug-ins such as hosts or rewrite. If you want to enable these plug-ins, modify the CoreDNS configurations.

1. (Optional) Modify the CoreDNS configuration so that the CoreDNS preferentially uses UDP to communicate with the upstream DNS server.

   The NodeLocal DNSCache uses TCP to communicate with the CoreDNS. The CoreDNS communicates with the upstream DNS server based on the protocol used by the request source. However, the cloud server does not support TCP. To use NodeLocal DNSCache, modify the CoreDNS configuration so that UDP is preferentially used to communicate with the upstream DNS server, preventing resolution exceptions.

   Perform the following operations. In the forward add-on, specify prefer_udp as the protocol used by requests. After the modification, CoreDNS preferentially uses UDP to communicate with the upstream system.

   1. Log in to the CCE console and click the cluster name to access the cluster console.
   2. In the navigation pane, choose Add-ons. Then, click Edit under CoreDNS.
   3. Edit the advanced configuration under Parameters and modify the following content in the plugins field:

      {
          "configBlock": "prefer_udp",
          "name": "forward",
          "parameters": ". /etc/resolv.conf"
      }

      Corefile:

      Corefile: |-
        .:5353 {
            bind {$POD_IP}
            cache 30 {
                servfail 5s
            }
            errors
            health {$POD_IP}:8080
            kubernetes cluster.local in-addr.arpa ip6.arpa {
                pods insecure
                fallthrough in-addr.arpa ip6.arpa
            }
            loadbalance round_robin
            prometheus {$POD_IP}:9153
            forward . /etc/resolv.conf {
                prefer_udp
            }
            reload
            ready {$POD_IP}:8081
        }

2. Log in to the CCE console and click the cluster name to access the cluster console. Choose Add-ons in the navigation pane, locate NodeLocal DNSCache on the right, and click Install.
3. On the Install Add-on page, select the add-on specifications and set related parameters.
   • DNSConfig Automatic Injection: After this function is enabled, a DNSConfig admission controller will be created. The controller intercepts pod creation requests in the namespace labeled with node-local-dns-injection=enabled based on admission webhooks and automatically configures DNSConfig for pods. If this function is disabled or the pod belongs to a non-target namespace, you must manually configure DNSConfig for the pod.
     After automatic injection is enabled, you can customize the following configuration items for DNSConfig (supported when the add-on version is 1.6.7 or later):

     

     If DNSConfig has been configured in the pod when automatic injection is enabled, DNSConfig in the pod will be used first.

     • IP Address of DNS Server (optional): IP address list of the DNS server obtained when the container resolves the domain name. NodeLocal DNSCache and CoreDNS IP addresses are added by default. You have the option to add an additional IP address, but duplicates will be removed.
     • Search Domain (optional): a search list for host-name lookup. When a domain name cannot be resolved, DNS queries will be attempted combining the domain name with each domain in the search list in turn until a match is found or all domains in the search list are tried. You can add up to three extra search domains, but any duplicates will be removed.
     • ndots: specifies that if a domain name has fewer periods (.) than the specified value of ndots, it will be combined with the search domain list for DNS query. If the domain name still cannot be resolved, it will be used for DNS query. The system will perform a DNS query on a domain name if the number of periods (.) in it is greater than or equal to the value of ndots. If the domain name cannot be resolved correctly, the system will sequentially combine it with the search domain list and then perform a DNS query.
   • Target Namespace: This parameter is available after DNSConfig Automatic Injection is enabled. Only NodeLocal DNSCache of v1.3.0 or later supports this function.
     • All Enabled: CCE adds the node-local-dns-injection=enabled label to all created namespaces excluding built-in ones (such as kube-system), identifies namespace creation requests, and automatically adds the label to newly created namespaces.
     • Manual Configuration: You must manually add the node-local-dns-injection=enabled label to the namespaces requiring the injection of DNSConfig. For details, see Managing Namespace Labels.

4. Click Install.

Using NodeLocal DNSCache

By default, application requests are sent through the CoreDNS proxy. To use node-local-dns as the DNS cache proxy, use any of the following methods:

• Auto injection: Automatically configure the dnsConfig field of the pod when creating the pod. (This function is not available for pods in system namespaces such as kube-system.)
• Manual configuration: Manually configure the dnsConfig field of the pod.

Common Issues

• How Do I Prevent Auto DNSConfig Injection on a Specific Pod?

  Solution:

  Add node-local-dns-injection: disabled to the labels field in the pod template. Example:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: test
    namespace: default
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: test
    template:
      metadata:
        labels:
          app: test
          node-local-dns-injection: disabled   # Prevent auto DNSConfig injection.
      spec:
        containers:
          - name: container-1
            image: nginx:latest
            imagePullPolicy: IfNotPresent
        imagePullSecrets:
          - name: default-secret