Managing Costs for a Cluster

CCE provides many cluster types. You can select a proper one based on your service characteristics. The following table lists the differences between these clusters.

For details, see Comparison Between Cluster Types.

ECSs come in various flavors, each offering different computing and storage capabilities. Typically, higher specifications (such as CPU and memory) and specialized features (like GPUs and NPUs) result in higher costs per node. It is important to configure stable, cost-effective ECSs that align with the specific needs of your services.

Different services have varying resource usage periods and stability requirements. To achieve cost-effectiveness, you can choose the appropriate billing mode based on the service characteristics.

For details, see Billing Items.

It is a good option to identify and clear idle cloud services or resources in clusters in a timely manner, such as unused ECSs, EVS disks, OBS buckets, ELB load balancers, and EIPs.

To ensure that your workloads have enough resources and to avoid wasting resources due to excessive requests, it is important to configure appropriate requests and limits.

Quota management sets limits on the total number of resources that teams and users can use when they share cluster resources. These resources include the number of objects of a specific type created in a namespace, as well as the total number of compute resources like CPUs and memory used by these objects.

This approach help minimize unnecessary resource overhead.

For details, see Configuring Resource Quotas.

CCE provides auto scaling for applications. This feature enables applications that experience traffic surges or periodic peak and off-peak hours to automatically adjust compute resources.

Application scaling on demand (HPA)

Application auto scaling helps dynamically adjust compute resources based on service requirements and policies. It enables quick scale-out during peak hours and scale-in during off-peak hours, optimizing resource utilization and reducing costs.

The following table lists the auto scaling approaches supported by CCE.

Burst scaling (interconnected with CCI)

In short-term high-load scenarios, besides HPA, pods in a CCE cluster can be scaled to CCI. There is no need to create new nodes, reducing resource consumption caused by scale-out. The CCE Cloud Bursting Engine for CCI add-on must be installed in the cluster. For details, see Elastic Scaling of CCE Pods to CCI.

Application auto scaling helps dynamically adjust the number of pods based on workload metrics. If there are not enough cluster resources and new pods cannot run properly, you can add more nodes to the cluster.

For details, see Node Scaling Rules.

The following table lists auto scaling policies that you can select.

If your service meets the following requirements, it is recommended that you use this capability:

• Nodes are deployed in different clusters. They cannot share compute resources with each other, resulting in an increase in resource fragments.
• The node flavors are not ideal for applications that undergo frequent changes. At first, the node flavors match the application requirements, resulting in a high resource allocation rate. However, as the applications evolve, their resource demands change, causing a significant difference in the ratio of requested resources to node flavors. This leads to a decrease in the allocation rate of node resources and an increase in compute resource fragments.
• There are a large number of reserved resources. Online services experience daily peaks and troughs. To ensure service performance and stability, users apply for resources based on peak usage, which may result in many idle resources in the cluster during certain times.
• Online and offline services are deployed in separate Kubernetes clusters, and resources cannot be shared between them at different time. This means that during off-peak hours for online services, the resources cannot be used by offline services.

The following table lists cloud native hybrid deployment features that can help you improve resource utilization, reduce costs, improve efficiency in the scenarios mentioned earlier.

The Volcano Scheduler is used to improve cluster resource usage. It provides bin packing, descheduling, node pool affinity, and load-aware scheduling policies.

A pod priority indicates the importance of a pod relative to other pods. Volcano supports pod PriorityClasses in Kubernetes. After PriorityClasses are configured, the scheduler preferentially schedules high-priority pods. When cluster resources are insufficient, the scheduler will proactively evict low-priority pods to make it possible to schedule pending high-priority pods. For details, see Priority-based Scheduling and Preemption.

The following table lists the types of priority-based scheduling and preemption supported by CCE.

GPU virtualization allows for the separation of compute and GPU memory, optimizing the utilization of GPUs. CCE GPU virtualization leverages the proprietary xGPU virtualization technology to dynamically separate GPU memory and compute. This virtualization solution offers greater flexibility compared to static allocation. While ensuring maximum service stability, you have the freedom to define the number of GPUs to be used, thereby enhancing GPU utilization.

For details, see GPU Virtualization.

In AI and big data collaborative scheduling scenarios, Volcano Dominant Resource Fairness (DRF) and group can be used to improve training performance and resource utilization.

DRF

DRF is a scheduling algorithm based on the dominant resource of a container group. DRF can be used to enhance the service throughput of a cluster, shorten the overall service execution time, and improve training performance. It is suitable for batch AI training and big data jobs. For details, see DRF.

In actual services, limited cluster resources are often allocated to multiple users. Each user has the same rights to obtain resources, but the number of resources they need may be different. It is crucial to fairly allocate resources to each user. A common scheduling algorithm is the max-min fairness share, which allocates resources to meet users' minimum requirements as far as possible and then fairly allocates the remaining resources. The rules are as follows:

• Resources are allocated in order of increasing demand.
• No source gets a resource share larger than its demand.
• Sources with unsatisfied demands get an equal share of the resource.

Gang

Gang scheduling meets the scheduling requirements of "All or nothing" in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of pods. It is mainly used in scenarios that require multi-process collaboration, such as AI and big data scenarios. Gang scheduling effectively resolves pain points such as resource waiting or deadlocks in distributed training jobs, thereby significantly improving the utilization of cluster resources. For details, see Gang.

The Gang scheduler algorithm checks whether the number of scheduled pods in a job meets the minimum requirements for running the job. If yes, all pods in the job will be scheduled. If no, the pods will not be scheduled.

It is applicable to the scenarios where multi-process collaboration is required.

• AI scenarios typically involve complex processes. Data ingestion, data analysts, data splitting, trainers, serving, and logging which require a group of containers to work together are suitable for container-based Gang scheduling.
• Multi-thread parallel computing communication scenarios under MPI computing framework are also suitable for Gang scheduling because master and slave processes need to work together.
• Containers in a pod group are highly correlated, and there may be resource contention. The overall scheduling allocation can effectively resolve deadlocks.

  If cluster resources are insufficient, Gang scheduling can significantly improve the utilization of cluster resources.

When working with high-performance computing (HPC), real-time applications, or memory-intensive workloads that require frequent communication between CPUs, accessing nodes across non-uniform memory access (NUMA) in a cloud native environment can lead to decreased system performance due to increased latency and overhead. Volcano's NUMA affinity scheduling policy resolves the issue by scheduling pods to the worker node that requires the least number of cross-NUMA nodes. This reduces data transmission overheads, optimizes resource utilization, and enhances overall system performance.

Volcano targets to lift the limitation to make scheduler NUMA topology aware so that:

• Pods are not scheduled to the nodes that NUMA topology does not match.
• Pods are scheduled to the most suitable node for NUMA topology.

For details, see NUMA Affinity Scheduling.

With application scaling priority policies, you have precise control over the scaling priorities of pods on different types of nodes, allowing for optimized resource management. The application scaling priority policies include the following aspects:

• Scale-out: Volcano schedules new pods in a cluster based on preset node priority for scale-out.
• Scale-in: When a workload is specified, Volcano scores the workload based on preset node priority to determine pod deletion sequence during scale-in.

If the default scaling priority policy is applied, pods will be scheduled first to yearly/monthly nodes during scale-out, followed by pay-per-use nodes and virtual-kubelet nodes (scaling pods to CCI). During scale-in, pods are deleted sequentially from virtual-kubelet nodes (scaling pods to CCI), pay-per-use nodes, and yearly/monthly nodes. You can adjust the scaling priority policies based on your service scenarios. For details, see Application Scaling Priority Policies.