Managing Costs for a Cluster

CCE provides many cluster types. You can select a proper one based on your service characteristics. The table below lists the differences between different types of clusters.

For details, see Comparison Between Cluster Types.

ECSs come in various flavors, each offering different computing and storage capabilities. Typically, higher flavors, such as CPUs and memory, and specialized features like GPUs and NPUs result in higher costs per node. It is important to configure stable, cost-effective ECSs that align with the specific needs of your services.

Different services have varying resource usage periods and stability requirements. To achieve cost-effectiveness, you can choose the appropriate billing mode based on the service characteristics.

For details, see Billing Items.

It is a good option to identify and clear idle cloud services or cluster resources in a timely manner, such as unused ECSs, EVS disks, OBS buckets, ELB load balancers, and EIPs.

To ensure that your workloads have enough resources and to avoid wasting resources due to excessive requests, configure appropriate requests and limits.

Quota management sets limits on the total number of resources that teams and users can use when they share cluster resources. These resources include the number of objects of a specific type created in a namespace, as well as the total number of compute resources like CPUs and memory used by these objects.

This approach helps minimize unnecessary resource overhead.

For details, see Configuring Resource Quotas.

CCE provides auto scaling for applications. This feature enables applications that experience traffic surges or periodic peak and off-peak hours to automatically adjust their compute resources.

Application scaling on demand (HPA)

Application auto scaling helps dynamically adjust compute resources based on service requirements and policies. It enables quick scale-outs during peak hours and scale-ins during off-peak hours, optimizing resource utilization and reducing costs.

The table below lists the auto scaling approaches supported by CCE.

Application auto scaling helps dynamically adjust the number of pods based on workload metrics. If there are not enough cluster resources and new pods cannot run properly, you can add more nodes to the cluster.

For details, see Node Scaling Rules.

The table below lists auto scaling policies that you can select.

Use cloud native hybrid deployment if your services are deployed in the scenarios below.

• Nodes are deployed in different clusters. They cannot share compute resources with each other, resulting in an increase in resource fragments.
• The node flavors are not ideal for applications that undergo frequent changes. At first, the node flavors match the application requirements, resulting in a high resource allocation rate. However, as the applications evolve, their resource demands change, causing a significant difference in the ratio of requested resources to node flavors. This leads to a decrease in the allocation rate of node resources and an increase in compute resource fragments.
• There are a large number of reserved resources. Online services experience daily peaks and troughs. To ensure service performance and stability, users apply for resources based on peak usage, which may result in many idle resources in the cluster during certain times.
• Online and offline services are deployed in separate Kubernetes clusters, and resources cannot be shared between them across different times. This means that during off-peak hours for online services, the resources cannot be used by offline services.

The table below lists cloud native hybrid deployment features that can help you improve resource utilization, reduce costs, improve efficiency in the scenarios mentioned earlier.

Volcano Scheduler can improve cluster resource utilization. It provides bin packing, descheduling, node pool affinity, and load-aware scheduling policies.

A pod priority indicates the importance of a pod relative to other pods. Volcano supports pod PriorityClasses in Kubernetes. After PriorityClasses are configured, the scheduler preferentially schedules high-priority pods. When cluster resources are insufficient, the scheduler will proactively evict low-priority pods to make it possible to schedule pending high-priority pods. For details, see Priority-based Scheduling and Preemption.

The table below lists the types of priority-based scheduling and preemption supported by CCE.

GPU virtualization allows for the separation of compute and GPU memory, optimizing the utilization of GPUs. CCE GPU virtualization leverages the proprietary xGPU virtualization technology to dynamically separate GPU memory and compute. This virtualization solution offers greater flexibility compared to static allocation. While ensuring maximum service stability, you have the freedom to define the number of GPUs to be used, thereby enhancing GPU utilization.

For details, see GPU Virtualization.

In AI and big data collaborative scheduling scenarios, Volcano Dominant Resource Fairness (DRF) and Gang scheduling can be used to improve training performance and resource utilization.

DRF

DRF is a scheduling algorithm based on dominant-resource fairness. It is designed for scenarios involving large-scale submissions of AI training and big data jobs, helping improve overall cluster throughput, shorten job execution time, and enhance training performance. For details, see DRF.

In actual services, limited cluster resources are often allocated to multiple users. Each user has the same rights to obtain resources, but the number of resources they need may be different. It is crucial to fairly allocate resources to each user. A common scheduling algorithm is the max-min fairness share, which allocates resources to meet users' minimum requirements as far as possible and then fairly allocates the remaining resources. The rules are as follows:

• Resources are allocated in order of increasing demand.
• No user gets a resource share larger than their demands.
• Users with unsatisfied demands get an equal share of the resource.

Gang

Gang scheduling meets the scheduling requirements of "All or nothing" in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of pods. It is mainly used in scenarios that require multi-process collaboration, such as AI and big data scenarios. Gang scheduling effectively resolves pain points such as resource waiting or deadlocks in distributed training jobs, thereby significantly improving the utilization of cluster resources. For details, see Gang.

The Gang scheduler algorithm checks whether the number of scheduled pods in a job meets the minimum requirements for running the job. If yes, all pods in the job will be scheduled. If no, the pods will not be scheduled.

It is applicable to the scenarios where multi-process collaboration is required.

• AI scenarios typically involve complex workflows, such as data ingestion, data analysis, data splitting, training, serving, and logging. These workflows require multiple containers to run collaboratively and are well-suited for Gang scheduling.
• Multi-thread parallel computing communication scenarios under MPI computing framework are suitable for Gang scheduling because primary and secondary processes need to work together.
• Containers in a pod are highly correlated, and there may be resource contention. The overall scheduling allocation can effectively resolve deadlocks.

  If cluster resources are insufficient, Gang scheduling can significantly improve the utilization of cluster resources.

When working with high-performance computing, real-time applications, or memory-intensive workloads that require frequent communication between CPUs, accessing nodes across non-uniform memory access (NUMA) in a cloud native environment can lead to decreased system performance due to increased latency and overhead. Volcano's NUMA affinity scheduling policy resolves the issue by scheduling pods to the worker node that requires the least number of cross-NUMA nodes. This reduces data transmission overheads, optimizes resource utilization, and enhances overall system performance.

Volcano aims to resolve the NUMA topology-aware scheduling restrictions of the scheduler so that:

• Pods are not scheduled to nodes that do not match the NUMA topology.
• Pods are scheduled to the most suitable node for NUMA topology.

For details, see NUMA Affinity Scheduling.

With application scaling priority policies, you have precise control over the scaling priorities of pods on different types of nodes, allowing for optimized resource management. The application scaling priority policies include:

• Scale-outs: Volcano schedules new pods in a cluster based on preset node priority for scale-outs.
• Scale-ins: When a workload is specified, Volcano scores the workload based on preset node priority to determine pod deletion sequence during scale-ins.

If the default scaling priority policy is applied, pods will be scheduled first to yearly/monthly nodes during a scale-out, followed by pay-per-use nodes and virtual-kubelet nodes (scaled to CCI). During a scale-in, pods are deleted sequentially from virtual-kubelet nodes (scaled to CCI), pay-per-use nodes, and yearly/monthly nodes. You can adjust the scaling priority policies based on your service scenarios. For details, see Application Scaling Priority Policies.