Implementing Node Pool-level HA

Node Layer: Physical and AZ-level Isolation

Configuring Physical Host Anti-Affinity Using an ECS Group

An ECS group is a Huawei Cloud ECS feature that controls the locations where ECSs are placed.

To create an anti-affinity ECS group, select the Anti-affinity policy during the ECS group creation on the ECS console. ECSs in this group will be scheduled onto different physical hosts. For details, see Managing ECS Groups.

Associating an ECS Group with a Node Pool

When creating or editing a node pool on the CCE console, expand Advanced Settings and select the created anti-affinity ECS group from the drop-down list.

In this case, all new nodes added to the node pool will automatically join the ECS group and be distributed across different physical hosts.

Creating a Multi-AZ Node Pool for AZ Anti-Affinity

This is the basis for building a multi-AZ HA architecture.

When creating a node pool, select node flavors from different AZs.

Nodes will be created in the specified AZs according to the configuration. Even if one AZ becomes unavailable, nodes in other AZs continue to provide services.

Pod Layer: Fine-Grained Distribution Using Scheduling Policies

After the node resources are ready, you need to use native Kubernetes scheduling rules to distribute pods evenly and properly across nodes.

Configuring Pod Affinity and Anti-Affinity

Pod anti-affinity is typically used to distribute pods of the same application across different nodes, preventing all pods from being scheduled onto a single node.

Example: Ensuring only one pod per node

The following YAML uses requiredDuringSchedulingIgnoredDuringExecution (a hard constraint) to ensure that pods with the app: nginx-ha label are not scheduled onto the same node:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ha
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-ha
  template:
    metadata:
      labels:
        app: nginx-ha
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nginx-ha
            topologyKey: kubernetes.io/hostname  # Use the host name as the topology domain to ensure that the pods are scheduled to different nodes.
      containers:
      - name: nginx
        image: nginx:latest
      imagePullSecrets:
        - name: default-secret

Using Pod Topology Spread Constraints

Topology spread constraints provide a more flexible, stronger way to distribute pods across topology domains, such as nodes or AZs, than anti-affinity. They allow you to define the maximum allowed imbalance between domains.

Core concepts

• topologyKey: the node label defining the topology domain, such as kubernetes.io/hostname (node) and topology.kubernetes.io/zone (AZ)
• maxSkew: the maximum allowed difference in pod count. The value must be greater than 0.
• whenUnsatisfiable: behavior when constraints cannot be met. The options include DoNotSchedule and ScheduleAnyway.

Example: Evenly distributing pods across AZs and nodes

The following YAML example uses two constraints to ensure that pods are evenly distributed across nodes and AZs.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-topo
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx-topo
  template:
    metadata:
      labels:
        app: nginx-topo
    spec:
      topologySpreadConstraints:
      - maxSkew: 1  # The maximum allowed difference in pod counts for different selectors in the same topology domain is 1.
        topologyKey: kubernetes.io/hostname  # The node is used as the topology domain.
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: nginx-topo
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone  # The AZ is used as the topology domain.
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: nginx-topo
      containers:
      - name: nginx
        image: nginx:latest
      imagePullSecrets:
        - name: default-secret