Recommended Configurations for HA CCE Clusters

Deploying Master Nodes in Different AZs

Multiple regions are provided for you to deploy your services, and there are different availability zones (AZs) in each region. An AZ is a collection of one or more physical data centers with independent cooling, fire extinguishing, moisture-proof, and electricity facilities. AZs within a region are connected using high-speed optical fibers. This allows you to build cross-AZ HA systems.

When creating a cluster, enable the HA mode of the cluster and configure the distribution mode of the master nodes. The master nodes are deployed in different AZs as randomly as possible. This ensures a higher disaster recovery (DR) capability of the cluster.

Figure 1 Configuring an HA cluster

Selecting a Network

• Types of networks: CCE supports VPC, Cloud Native 2.0, and container tunnel networks. Different types of networks have different performance and functions. For details, see Networks.
• VPC: To enable your applications to access other cloud services like RDS, create related services in the same VPC as your cluster. This is because services using different VPCs are isolated from each other. If you have VPCs, use VPC peering connections to enable communication between VPCs.
• Pod CIDR block: Do not configure a small pod CIDR block. Otherwise, the number of supported nodes will be limited.
  • For a cluster using a VPC network, if the subnet mask of the pod CIDR block is /16, there are 256 × 256 IP addresses available. If the maximum number of pods reserved on each node is 128, the maximum number of nodes supported is 512.
  • For a cluster using a container tunnel network, if the subnet mask of the pod CIDR block is /16, there are 256 × 256 IP addresses available. The pod CIDR block allocates 16 IP addresses to nodes at a time by default. The maximum number of nodes supported by your cluster is 4,096 (65536/16=4096).
  • For a cluster using a Cloud Native 2.0 network, the pod CIDR block is a VPC subnet, and the number of containers that can be created depends on the size of the selected subnet.
• Service CIDR block: The Service CIDR block determines the maximum number of Services in a cluster. Evaluate your actual needs before adjusting this CIDR block. A Service CIDR block cannot be modified after its creation. Do not configure an excessively small one.

For details, see Planning CIDR Blocks for a Cluster.

Configuring Service Forwarding

kube-proxy is a key component in a Kubernetes cluster. It is responsible for load balancing and forwarding between a Service and its backend pods. When using clusters, consider the potential performance issues of the forwarding mode.

CCE supports iptables and IPVS.

• IPVS allows higher throughput and faster forwarding. It applies to scenarios involving large clusters or a large number of Services.
• iptables is the traditional kube-proxy mode. It applies to scenario involving a small number of Services or a large number of short concurrent connections on clients. If there are more than 1,000 Services in a cluster, network delay may occur.

Configuring Quotas and Limits

CCE allows you to configure quotas and limits for your cloud service resources and resources in your clusters. This prevents excessive use of resources. When deploying applications in CCE clusters, consider these limits and periodically review them. This will avoid scaling failures caused by insufficient quotas during application running.

• Configuring resource quotas for cloud services: Cloud services like ECS, EVS, VPC, ELB, and SWR are also used in CCE clusters. If the existing resource quotas cannot meet your requirements, submit a service ticket and increase the quotas.
• Configuring resource quotas for a cluster: You are allowed to configure the namespace-level resource quotas to limit the number of objects of a certain type in a namespace and the total computing resources like CPUs and memory consumed by the objects. For details, see Configuring Resource Quotas.

Monitoring Master Node Metrics

Monitoring master node metrics allows you to check the master nodes' performance and efficiently identify problems occurred on them. The master nodes that are not running properly may lower application reliability.

CCE can monitor kube-apiserver, kube-controller, kube-scheduler, and etcd-server on master nodes. You need to install the Cloud Native Cluster Monitoring add-on. With the Grafana add-on, you can use the Kubernetes monitoring overview dashboard to monitor metrics of Kubernetes API server requests and latency and etcd latency.

If an on-premises Prometheus instance is used, you can manually add monitoring metrics. For details, see Monitoring Master Node Components Using Prometheus.

Partitioning Data Disks Attached to a Node

A node's default data disk stores the container runtime and kubelet components. For details, see Space Allocation of a Data Disk.

The default space of this data disk is 100 GiB. You can adjust the space as required. Images, system logs, and application logs are stored on data disks. Therefore, you need to evaluate the number of pods to be deployed on each node, the size of logs, images, and temporary data of each pod, as well as some reserved space for the system. For details, see Selecting a Data Disk for a Node.

Running CCE Node Problem Detector

A failure in a worker node may affect the availability of applications. CCE Node Problem Detector is used to monitor node exceptions. It helps you detect and handle latent exceptions in a timely manner. You can also customize the check items, including target nodes, check period, and triggering threshold. For details, see Configuring Node Fault Detection Policies.

Configuring DNS Caches

When the number of DNS requests in a cluster increases, the load of CoreDNS increases and the following issues may occur:

• Increased delay: CoreDNS needs to process more requests, which may slow down the DNS query and affect service performance.
• Increased resource usage: To ensure DNS performance, CoreDNS requires higher specifications.

To minimize the impact of DNS delay, use NodeLocal DNSCache to improve the networking stability and performance. NodeLocal DNSCache runs a DNS cache proxy on cluster nodes. All pods with DNS configurations use the DNS cache proxy running on nodes instead of the CoreDNS service for domain name resolution. This reduces CoreDNS' load and improves the cluster DNS performance.

You can install the NodeLocal DNSCache add-on. For details, see Using NodeLocal DNSCache to Improve DNS Performance.

Properly Deploying CoreDNS

Deploy the CoreDNS instances across different AZs and nodes to reduce the impact of a single-node or single-AZ failure.

Ensure that the CPUs and memory of the node where CoreDNS is running are not fully used. Otherwise, the QPS and response of domain name resolution will be affected.

For details about how to properly configure CoreDNS, see Properly Configuring CoreDNS.

Configuring Multiple Pods for a Single Application

If your application runs in one pod, the application will be unavailable if the pod is abnormal. Use Deployments or other types of ReplicaSets to deploy your applications. Each time a pod fails or is terminated, the controller automatically restarts a new pod that has the same specifications as the original one. This ensures that a specified number of pods are always running in the cluster.

When creating a workload, set the number of pods to a value greater than 2. If one of the pods is faulty, the remaining pods still run until Kubernetes automatically creates another pod to compensate for the loss. You can also use HPAs and CAs (Using an HPA and a CA for Auto Scaling of Workloads and Nodes) to automatically scale in or out the workloads as required.

Using Containers to Isolate Processes

Containers provide process-level isolation. Each container has its own file system, network, and resources. This prevents interference between different processes and avoids attacks and data leakage from malicious processes. Using containers to isolate processes can improve the reliability, security, and portability of applications.

If several processes work together, you can create multiple containers in a pod so that they can share the same network, PV, and other resources. Taking an init container as an example. An init container runs before the main container is started to complete some initialization tasks like configuring environment variables, loading databases or data, and pulling Git repositories.

Multiple containers in a pod share the lifecycle of this pod. So, if one of the containers is abnormal, the entire pod will be restarted.

Configuring Resource Quotas for a Workload

Configure and adjust resource requests and limits for all workloads.

If too many pods are scheduled to one node, the node will be overloaded and unable to provide services.

To avoid this problem, when deploying a pod, specify the resource request and limit required by the pod. Kubernetes then selects a node with enough idle resources for this pod. For example, the resource request for an Nginx pod is 1 CPU core and 1,024-MiB memory, and the resource limit for it is 2 CPU cores and 4,096-MiB memory.

Kubernetes statically schedules resources. The remaining resources on each node are calculated as follows: Remaining resources on a node =Total resources on the node – Allocated resources (resources not in use). If you manually run a resource-consuming process, Kubernetes cannot detect it.

Additionally, you must configure resources for all pods. For a pod that does not have resources configured, after it is scheduled to a node, Kubernetes does not deduct the resources used by this pod from the node on which it runs. Other pods may still be scheduled to this node.

Deploying an Application in Multiple AZs

You can run pods on nodes in multiple AZs to prevent an application from being affected by faults of a single AZ.

When creating a node, manually specify an AZ for the node.

Figure 2 Specifying an AZ of a node

During application deployment, configure anti-affinity rules for pods so that the scheduler can schedule the pods across multiple AZs. For details, see Implementing High Availability for Containers. The following is an example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
  labels:
    app: web-server
spec:
  replicas: 4
  selector:
    matchLabels:
      app: web-server
  template:
    metadata:
      labels:
        app: web-server
    spec:
      containers:
      - name: web-app
        image: nginx
      imagePullSecrets: 
      - name: default-secret
      affinity:
        podAntiAffinity:  # Workload anti-affinity
          preferredDuringSchedulingIgnoredDuringExecution:  # The rules should be followed whenever possible; otherwise, pods may fail to schedule when the number of pods exceeds the number of AZs.
          - podAffinityTerm:
              labelSelector:  # Pod label matching rule. Configure anti-affinity rules between pods and their labels.
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - web-server
              topologyKey: topology.kubernetes.io/zone  # Topology domain of the AZ where the node is located
            weight: 100

You can also use Pod Topology Spread Constraints to deploy pods in multiple AZs.

Deploying a System Add-on in Multiple AZs

The Deployment pods of CCE system add-ons like CoreDNS and CCE Container Storage (Everest) can be deployed in multiple AZs, the same way as deploying an application. This function can satisfy different requirements.

Configuring a Container Health Check

Kubernetes automatically restarts pods that are not running properly. This prevents service interruption caused by pod exceptions. In some cases, however, even if a pod is running, it does not mean that it can provide services properly. For example, a deadlock may occur in a process in a running pod, but Kubernetes does not automatically restart the pod because it is still running. To solve this problem, configure a liveness probe to check whether the pod is healthy. If the liveness probe detects a problem, Kubernetes will restart the pod.

You can also configure a readiness probe to check whether a pod can provide services properly. During application startup, the application cannot provide services to external systems because it may take some time for it to initialize. After you configure a readiness probe for the application pod, Kubernetes allows the Service to forward requests to this pod only after the readiness probe detects that the pod is ready. When the pod is faulty, the readiness probe can prevent new traffic from being forwarded to it.

A startup probe checks whether a container has been started. It ensures that the container can start successfully before the liveness and readiness probes do their tasks, and the startup is not affected by the liveness and readiness probes. Configuring startup probes ensures that the startup of slow-start containers can be detected thoroughly to prevent Kubernetes from terminating them before they are started.

You can configure these probes for an application during its deployment. The following is an example:

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: nginx:alpine
    args:
    - /server
    livenessProbe:
      httpGet:
        path: /healthz
        port: 80
        httpHeaders:
        - name: Custom-Header
          value: Awesome
      initialDelaySeconds: 3
      periodSeconds: 3
    readinessProbe:
      exec:
        command:
          - cat
          - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5
    startupProbe:
      httpGet:
        path: /healthz
        port: 80
      failureThreshold: 30
      periodSeconds: 10

For details, see Configuring Container Health Check.

Configuring Auto Scaling

Auto scaling can automatically adjust the number of application pods and nodes as required. It ensures that pods and nodes can be quickly scaled to save resources and costs.

During peak hours, the following auto scaling may occur:

• Workload scaling: When containerizing applications, you can configure resource requests and limits for the application pods to prevent them from infinitely using resources during peak hours. However, after the resource limit is reached, an error may occur. To resolve this problem, you can scale the number of pods for each workload.
• Node scaling: After the number of pods is increased, the node resource usage may increase to a certain extent. As a result, the pods that are scaled out cannot be scheduled to that node. To resolve this problem, scale in or out nodes based on the resource usage.

For details, see Using an HPA and a CA for Auto Scaling of Workloads and Nodes.

Configuring Logs, Monitoring Metrics, and Alarm Rules

• Logs
  • Control plane logs are reported from the master nodes. CCE supports kube-controller-manager, kube-apiserver, kube-scheduler, and audit logs. For details, see Collecting Control Plane Component Logs.
  • Application logs are generated from pods. These logs include logs generated by pods that run services and Kubernetes system components like CoreDNS. CCE allows you to configure log policies to collect, manage, and analyze logs periodically to prevent logs from being over-sized. For details, see Logging Overview.
• Monitoring
  • Master node metrics: Monitoring these metrics enables you to identify problems occurred on the master nodes. For details, see Monitoring Master Node Metrics.
  • Application metrics: CCE comprehensively monitors applications in clusters. In addition to standard metrics, you can configure custom metrics that comply with the specifications for your applications to improve the observability. For details, see Monitoring Center Overview.
• Alarms

  You can add alarm rules for metrics to detect cluster faults and generate warnings in a timely manner with the monitoring function. This helps you maintain service stability. For details, see Configuring Custom Alarms on AOM.