Help Center/ Cloud Container Engine/ User Guide/ Settings/ Scheduling
Updated on 2024-11-11 GMT+08:00
View PDF

Scheduling

Basic kube-scheduler configurations and Volcano-backed advanced scheduling are available. You can enable advanced scheduling functions such as bin packing, priority-based scheduling and preemption, AI task performance enhancement, and heterogeneous resource management for improved cluster resource utilization at low costs.

Default Cluster Scheduler Configuration

Default cluster scheduler (default-scheduler)

The Kubernetes scheduler discovers newly created pods that have not been accepted by nodes and assigns them to proper nodes. You are allowed to use multiple schedulers in the same cluster. kube-scheduler is the default cluster scheduler provided by the Kubernetes community. CCE also supports the enhanced Volcano scheduler, which offers general computing capabilities like high-performance job scheduling engine, heterogeneous chip management, and task management.

You can use kube-scheduler together with the Volcano scheduler, or use kube-scheduler or Volcano scheduler separately.

Table 1 Default cluster scheduler

Scheduler Name

Description

Configuration

kube-scheduler

kube-scheduler provides the community native, standard scheduling capabilities.

If kube-scheduler is set as the default scheduler in a cluster and the Volcano Scheduler add-on (Volcano Scheduler) is also installed in the cluster, the enhanced Volcano capabilities are enabled by default. This ensures that you have access to advanced scheduling capabilities, including resource utilization optimization, AI job performance enhancement, and heterogeneous resource management. They help reduce costs while improving resource utilization. In this case, kube-scheduler schedules common workloads, while the Volcano scheduler schedules some specified workloads. For details, see Scheduling Workloads.

kube-scheduler configurations:

Enhanced configurations after the Volcano scheduler is installed:

Volcano scheduler (available in clusters v1.27 or later)

If the Volcano scheduler is set as the default scheduler in a cluster, kube-scheduler will no longer work. The Volcano scheduler schedules all workload tasks in the cluster.

Volcano provides enhanced scheduling capabilities in addition to the capabilities provided by kube-scheduler. Before using this scheduler, you need to install the Volcano Scheduler add-on first. For details, see Volcano Scheduler.

Enhanced configurations of the Volcano scheduler:

kube-scheduler

kube-scheduler provides the community native, standard scheduling capabilities.

Before enabling volcano enhanced capabilities, install Volcano Scheduler. Enabling this function will provide advanced scheduling capabilities, including optimizing resource utilization, enhancing AI job performance, and managing heterogeneous resources. This will ultimately improve cluster resource utilization and reduce costs.

Enhanced configurations of the Volcano scheduler:

Scheduler Performance Configuration

Only kube-scheduler supports this configuration.

Table 2 Parameters

Item

Parameter

Description

Value

QPS for communicating with kube-apiserver

kube-api-qps

QPS for communication with kube-apiserver
  • If the number of nodes in a cluster is less than 1,000, the default value is 100.
  • If the number of nodes in a cluster is 1,000 or more, the default value is 200.

Burst for communicating with kube-apiserver

kube-api-burst

Burst for communication with kube-apiserver
  • If the number of nodes in a cluster is less than 1,000, the default value is 100.
  • If the number of nodes in a cluster is 1,000 or more, the default value is 200.

Priority-based Scheduling

Scheduling based on priority

This is a basic scheduling capability and cannot be disabled. The scheduler preferentially guarantees the running of high-priority pods, and will not evict low-priority pods that are running. For details, see Priority-based Scheduling.

Whether to enable preemption (supported by the Volcano scheduler)

After this function is enabled, when cluster resources are insufficient, the scheduler will proactively evict low-priority pods to make it possible to schedule pending high-priority pods. For details, see Priority-based Scheduling.

  • This configuration is supported when Volcano is selected as the default scheduler.
  • Enabling both priority-based preemption scheduling and delayed pod creation simultaneously is not possible.

Resource Utilization Optimization Scheduling (Supported by the Volcano Scheduler)

Bin packing

With this option enabled, the cluster scheduler schedules pods to nodes that have the most requested resources. This reduces resource fragments on each node and improves the resource utilization of the cluster. For details, see Bin Packing.

For details about the bin packing weight and weights for each resource to score nodes, see Table 3.
Table 3 Bin packing weight

Item

Description

Default Value

Binpack Scheduling Strategy Weight

A larger value indicates a higher weight of the bin packing policy in overall scheduling.

10

CPU Weight

A larger value indicates a higher cluster CPU usage.

1

Memory Weight

A larger value indicates a higher cluster memory usage.

1

Custom Resource Type

Other custom resource types requested by pods, for example, nvidia.com/gpu. A larger value indicates a higher usage of the specified cluster resource.

None

Load-aware scheduling (usage)

This function uses the Cloud Native Cluster Monitoring (kube-prometheus-stack) add-on to obtain the actual CPU and memory load of each node, calculates the average load of each node based on the specified period, and preferentially schedules jobs to the node with the lightest load to balance load. For details, see Load-aware Scheduling.

AI Job Performance Enhancement Scheduling (Supported by the Volcano Scheduler)

Fair Scheduling Policy (DRF)

Dominant Resource Fairness (DRF) is a scheduling algorithm based on the dominant resource of a container group. It supports fair allocation of multiple types of resources and is suitable for batch AI training and big data jobs. DRF is suitable for batch process small scale services like single AI model training and single big data computing and query, because it preferentially considers the throughput of services in clusters.

DRF helps you enhance the service throughput of clusters and improve service performance. For details, see DRF.

Workload Group Scheduling Policy (Gang)

Gang scheduling meets the scheduling requirements of "All or nothing" in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of pods. It is mainly used in scenarios that require multi-process collaboration, such as AI and big data scenarios.

Gang scheduling effectively resolves pain points such as resource waiting or deadlocks in distributed training jobs, thereby significantly improving the utilization of cluster resources. For details, see Gang.

Heterogeneous Resource Scheduling (Supported by the Volcano Scheduler)

Support GPU resource scheduling

To use this capability, the CCE AI Suite (NVIDIA GPU) add-on (CCE AI Suite (NVIDIA GPU)) must be installed. With this option enabled, GPUs can be used for AI training jobs, and the scheduler provides full GPU dispatch and GPU sharing to improve resource utilization.

Support NPU resource scheduling

To use this capability, the CCE AI Suite (Ascend NPU) add-on (CCE AI Suite (Ascend NPU)) must be installed. With this option enabled, Ascend NPUs can be used for AI training jobs, and the scheduler provides NPU topology-aware scheduling to improve training job efficiency.

Parent Topic: Settings