CPU Rate Limiting for Low-Priority Workloads

In colocation environments, high-priority (online) and low-priority (offline) workloads share nodes to improve resource utilization. However, offline workloads can preempt CPU resources, degrading online workload performance.

CPU rate limiting protects online workloads by dynamically capping the CPU quota for best-effort workloads. It calculates quotas based on allocatable CPU capacity minus real-time CPU usage. Key features of CPU rate limiting:

Quota budgeting: derives the best-effort CPU budget from remaining allocatable capacity after accounting for online workload consumption.
Jitter control: applies a threshold percentage to prevent excessive quota adjustments from minor usage fluctuations.
QoS awareness: restricts only best-effort cgroup. Higher-priority workloads are protected via request-based CPU accounting.
Runtime configurability: supports dynamic parameter updates without restarting workloads or reloading the daemon.

Prerequisites

Cluster: CCE Turbo cluster v1.29 or later
OS: Huawei Cloud EulerOS 2.0 or later
Volcano: v1.22.1 or later; hybrid deployment: enabled

Notes and Constraints

CPU rate limiting is designed for resource oversubscription scenarios. In this model, offline pods do not request standard CPU or memory resources. Instead, they consume extended resources: kubernetes.io/batch-cpu and kubernetes.io/batch-memory.

Example

Log in to the CCE console and click the cluster name to access the cluster console.
In the navigation pane, choose Nodes. On the Node Pools tab, locate the target node pool and choose More > Mixed configuration.

Ensure that node pool hybrid deployment and resource oversubscription are enabled. For details, see Procedure.
Enable CPU rate limiting.
```
kubectl edit cm -n kube-system  volcano-agent-configuration
```
Add the following cpuThrottlingConfig settings to nodesConfig:
```
"cpuThrottlingConfig": {
  "enable": true,
  "cpuThrottlingThreshold": 80,    # The best-effort quota is capped at 80% of allocatable CPU resources.
  "cpuJitterLimitPercent": 1,      # Updates are suppressed unless the quota changes by at least 1%.
  "cpuRecoverLimitPercent": 10    # Each quota increase is capped at 10%.
}
```
- cpuThrottlingThreshold: the maximum percentage of allocatable CPU that the best-effort quota can occupy.
- cpuJitterLimitPercent: the minimum percentage point change that triggers a quota update. For example, if set to 1, updates occur only when the quota change is ≥ 1%.
- cpuRecoverLimitPercent: the maximum percentage increase allowed per single quota update. For example, 10 caps each quota increase at 10%.
Check the best-effort quota, which is computed as (Node allocatable CPU x cpuThrottlingThreshold% − cpuUsage). This value is written to the cpu.max file in the cgroup besteffort directory.
```
cat "/sys/fs/cgroup/kubepods/besteffort/cpu.max"
```
The cpu.max file contains two values in the following format:
```
<quota> <period>
```
- quota: the CPU time in microseconds available per period.
- period: the length of the period in microseconds. The default value is typically 100000 microseconds (100 ms).
The CPU limit is calculated as (Quota/Period). In this example, 154300/100000 yields a limit of 1.543 cores.

Verification

Deploy high-priority jobs to use standard CPU and memory resources, and low-priority jobs to use extended resources kubernetes.io/batch-cpu and kubernetes.io/batch-memory.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: press
  namespace: default
  labels:
    app: press
spec:
  replicas: 1
  selector:
    matchLabels:
      app: press
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: press
    spec:
      containers:
      - name: test
        image: your-image  # High-priority jobs
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: "250m"
            memory: "512Mi"
          limits:
            cpu: "2"
            memory: "512Mi"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cputhrottle
  namespace: default
  labels:
    app: cputhrottle
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cputhrottle
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: cputhrottle
    spec:
      containers:
      - name: test
        image: your-image  # Low-priority jobs
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            kubernetes.io/batch-cpu: "1"
            kubernetes.io/batch-memory: "512Mi"
          limits:
            kubernetes.io/batch-cpu: "4"
            kubernetes.io/batch-memory: "512Mi"

Increase CPU load on the low-priority pod, and verify its CPU usage (requires Kubernetes Metrics Server):
```
kubectl top pod
```
Observe that cpu.max is updated accordingly.
```
cat "/sys/fs/cgroup/kubepods/besteffort/cpu.max"
```
Increase CPU load on the high-priority pod. After a brief stabilization period, check CPU usage again. The low-priority pod's CPU usage is continuously throttled.

Computation of cpu.max in the besteffort directory