Volcano Queues

Resource Planning

Before running this example, ensure your cluster meets the following requirements:

• At least 6 CPU cores, 12 GiB of memory, and 3 GPUs are allocatable.
• Additional resources are reserved for system components.
• After the two initial jobs are running, at least one GPU node can provide 1 CPU core, 2 GiB of memory, and 1 GPU, with no other services consuming these resources. This ensures verification results are not affected by node-level resource exhaustion.
• No other queues have reserved resources in the CPU, memory, or GPU dimensions, which would reduce the effective upper limit available for this example.

Step 1: Check and Configure the Scheduler

1. Run the following command to view the current scheduler configuration:

   kubectl get configmap volcano-scheduler-configmap -n volcano-system \
     -o jsonpath='{.data.volcano-scheduler\.conf}'

   The configuration must satisfy the following requirements:

   • actions: must include enqueue and allocate.
   • Add-ons: must include capacity and must not include proportion. (The two add-ons are mutually exclusive.)

2. (Optional) If the current configuration does not meet these requirements, back up the existing configuration before modifying it and run the following command:

   kubectl edit configmap volcano-scheduler-configmap -n volcano-system

   Retain any existing add-ons required by other services in the cluster. Do not overwrite the entire ConfigMap. Example configuration:

   data:
     volcano-scheduler.conf: |
       actions: "enqueue, allocate, backfill"
       tiers:
       - plugins:
         - name: priority
         - name: gang
           enablePreemptable: false
         - name: conformance
       - plugins:
         - name: drf
           enablePreemptable: false
         - name: predicates
         - name: capacity
         - name: nodeorder
         - name: binpack

   Key parameters are described as follows.

   Parameter	Type	Mandatory	Description
   enqueue	Scheduling action	Yes	Checks whether the job's minimum resource requirements can be satisfied by the queue's remaining quota. Upon passing, the PodGroup status transitions from Pending to Inqueue.
   allocate	Scheduling action	Yes	Selects suitable nodes for the job within the queue's resource upper limit and allocates resources.
   backfill	Scheduling action	No	Attempts to utilize remaining node resources within the current scheduling cycle.
   capacity	Add-on	Yes	Validates admission and allocation against the queue's resource configuration.
   proportion	Add-on	No	Calculates resource shares based on the queue weight. This parameter cannot be used together with capacity.
   gang	Add-on	No	Checks whether the job satisfies its minAvailable requirement for gang scheduling.

3. After updating the configuration, restart the Volcano Scheduler for changes to apply.

   kubectl rollout restart deployment volcano-scheduler -n volcano-system
   kubectl rollout status deployment volcano-scheduler -n volcano-system

Step 2: Create Two Queues

1. Create a file named queue-isolation-demo.yaml (filename may be customized). The content is as follows:

   apiVersion: scheduling.volcano.sh/v1beta1
   kind: Queue
   metadata:
     name: queue-team-a
   spec:
     deserved:
       cpu: "2"
       memory: 4Gi
       nvidia.com/gpu: "1"
     capability:
       cpu: "2"
       memory: 4Gi
       nvidia.com/gpu: "1"
   ---
   apiVersion: scheduling.volcano.sh/v1beta1
   kind: Queue
   metadata:
     name: queue-team-b
   spec:
     deserved:
       cpu: "4"
       memory: 8Gi
       nvidia.com/gpu: "2"
     capability:
       cpu: "4"
       memory: 8Gi
       nvidia.com/gpu: "2"

   Key parameters are described as follows.

   Parameter	Type	Description
   metadata.name	String	Queue name, which must be unique in a cluster. The name can contain a maximum of 253 characters and must conform to the DNS subdomain name format.
   spec.deserved	ResourceList	Resources the queue is entitled to. In this example, it is set to the same value as capability so that the example focuses only on fixed quota boundaries.
   spec.capability	ResourceList	Hard upper limit on resources the queue can consume. The total resource requests of allocated and queued jobs cannot exceed this value.
   spec.guarantee.resource	ResourceList	(Optional) Resources exclusively reserved for this queue. It must be less than or equal to deserved. This field is not configured in this minimal example.
   spec.parent	String	(Optional) Name of the parent queue. If omitted, the queue is associated with root. Training jobs can be submitted only to leaf queues.
   nvidia.com/gpu	Integer	NVIDIA GPU extended resource name. Its value must be a non-negative integer. It is used in this example to set GPU quotas. If your cluster registers GPU resources under different names, update accordingly in both queue and job configurations.

2. Apply the queue configuration.

   kubectl apply -f queue-isolation-demo.yaml

   If information similar to the following is displayed, the queues are successfully created:

   queue.scheduling.volcano.sh/queue-team-a created 
   queue.scheduling.volcano.sh/queue-team-b created

3. Check queue statuses.

   kubectl get queue queue-team-a queue-team-b \
     -o custom-columns='NAME:.metadata.name,STATE:.status.state,PARENT:.spec.parent'

   Ensure that STATE is Open for both queues.

Step 3: Submit a Job Within the Quota

1. Create a namespace.

   kubectl create namespace queue-demo

2. Create a file named jobs-within-quota.yaml with the following content:

   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: team-a-within-quota
     namespace: queue-demo
   spec:
     queue: queue-team-a
     schedulerName: volcano
     minAvailable: 1
     tasks:
       - name: worker
         replicas: 1
         template:
           spec:
             restartPolicy: Never
             containers:
               - name: worker
                 image: busybox:1.36.1
                 command: ["sh", "-c", "sleep 3600"]
                 resources:
                   requests:
                     cpu: "1"
                     memory: 2Gi
                     nvidia.com/gpu: "1"
                   limits:
                     cpu: "1"
                     memory: 2Gi
                     nvidia.com/gpu: "1"
   ---
   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: team-b-within-quota
     namespace: queue-demo
   spec:
     queue: queue-team-b
     schedulerName: volcano
     minAvailable: 1
     tasks:
       - name: worker
         replicas: 1
         template:
           spec:
             restartPolicy: Never
             containers:
               - name: worker
                 image: busybox:1.36.1
                 command: ["sh", "-c", "sleep 3600"]
                 resources:
                   requests:
                     cpu: "2"
                     memory: 4Gi
                     nvidia.com/gpu: "1"
                   limits:
                     cpu: "2"
                     memory: 4Gi
                     nvidia.com/gpu: "1"

   Extended GPU resources cannot be overcommitted. In this example, requests and limits for GPUs are set to the same integer value. The busybox container does not perform actual GPU computation, but the pod still holds the requested GPU resources. If your cluster cannot pull busybox:1.36.1 directly, mirror the image to SWR and update the image field accordingly.

3. Submit the job.

   kubectl apply -f jobs-within-quota.yaml

4. Check the job status.

   kubectl get vcjob -n queue-demo -o wide
   kubectl get podgroup -n queue-demo -o wide
   kubectl get pod -n queue-demo -o wide

   When cluster resources satisfy the requirements, the expected results are:

   • Both team-a-within-quota and team-b-within-quota transition to Running.
   • Their corresponding PodGroups transition to Running.
   • Both pods are in the Running state.
   • queue-team-a has allocated 1 CPU core, 2 GiB of memory, and 1 GPU, with 1 CPU core, 2 GiB of memory, and 0 GPUs remaining.
   • queue-team-b has allocated 2 CPU cores, 4 GiB of memory, and 1 GPU, with 2 CPU cores, 4 GiB of memory, and 1 GPU remaining.

5. View allocated resources.

   kubectl get queue queue-team-a \
     -o jsonpath='{.status.state}{"\n"}{.status.allocated}{"\n"}'
   kubectl get queue queue-team-b \
     -o jsonpath='{.status.state}{"\n"}{.status.allocated}{"\n"}'

   Queue status synchronization may be delayed briefly. If values are not updated, wait one scheduling cycle and check again.

Step 4: Submit a Job That Exceeds the Remaining Quota

1. Create a file named job-over-quota.yaml with the following content:

   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: team-a-over-quota
     namespace: queue-demo
   spec:
     queue: queue-team-a
     schedulerName: volcano
     minAvailable: 1
     tasks:
       - name: worker
         replicas: 1
         template:
           spec:
             restartPolicy: Never
             containers:
               - name: worker
                 image: busybox:1.36.1
                 command: ["sh", "-c", "sleep 3600"]
                 resources:
                   requests:
                     cpu: "1"
                     memory: 2Gi
                     nvidia.com/gpu: "1"
                   limits:
                     cpu: "1"
                     memory: 2Gi
                     nvidia.com/gpu: "1"

   This job requests 1 CPU core, 2 GiB of memory, and 1 GPU. While the CPU and memory requests fit within queue-team-a's remaining quota (1 CPU core, 2 GiB), the GPU request exceeds the remaining GPU quota of 0.

2. Submit the job.

   kubectl apply -f job-over-quota.yaml

3. Check the job and PodGroup statuses.

   kubectl get vcjob team-a-over-quota -n queue-demo -o wide
   kubectl get podgroup team-a-over-quota -n queue-demo -o wide
   kubectl describe podgroup team-a-over-quota -n queue-demo

   The expected results are:

   • team-a-over-quota remains in Pending.
   • The corresponding PodGroup remains in Pending and does not transition to Inqueue or Running.
   • The PodGroup events indicate insufficient queue resource quota and insufficient nvidia.com/gpu.
   • With delayed pod creation enabled (default in most configurations), pods are not created until the queue quota check passes. In some versions or configurations, pending pods may be created but cannot obtain resources exceeding the queue's upper limit.
   • The remaining 1 GPU in queue-team-b is not allocated to team-a-over-quota.
   • Running jobs team-a-within-quota and team-b-within-quota are unaffected.

   This verifies that the two queues independently manage CPU, memory, and GPU consumption, and this capability prevents jobs from exceeding their queue's GPU upper limit.

Step 6: Clean Up Example Resources

Before deleting a queue, ensure no PodGroups remain in Running or Inqueue state, and confirm no other service jobs need to be retained.

1. Delete the example jobs.

   kubectl delete vcjob team-a-over-quota team-a-within-quota team-b-within-quota -n queue-demo

2. Confirm that related PodGroups and pods have been deleted.

   kubectl get podgroup,pod -n queue-demo

3. Delete the example queues and namespace.

   kubectl delete queue queue-team-a queue-team-b
   kubectl delete namespace queue-demo

What Do I Do If Error "unable to find job queue" Is Reported When a Training Job Is Created

• Possible cause: The target queue does not exist, or the Volcano Admission controller's queue cache has not yet synchronized.
• Troubleshooting
  1. Verify the queue exists.

     kubectl get queue

  2. Check that the value of spec.queue in your job matches a queue name from the output above.
  3. If the queue was recently created, wait a few seconds for the cache to synchronize, and retry.

What Do I Do If Error "can only submit job to queue with state Open" Is Reported When a Training Job Is Created?

• Possible cause: The target queue is not in the Open state.
• Troubleshooting
  1. Check the queue status and events.

     kubectl get queue <queue-name> -o yaml
     kubectl get events --field-selector involvedObject.kind=Queue,involvedObject.name=<queue-name>

  2. Submit the job only after the queue status transitions to Open.
     

     Do not manually modify status.state to bypass queue lifecycle management.

Training Job Stuck in Pending State

• Possible cause: The issue is typically caused by insufficient resources, scheduling policy conflicts, or node exceptions.
• Troubleshooting
  1. Check the PodGroup status to identify the blocking cause.

     kubectl describe podgroup -n <namespace> <podgroup-name>

  2. Review the queue configuration and allocated resources.

     kubectl get queue <queue-name> -o yaml

     Check whether the job's requested resources exceed the queue's capability or whether the queue has already reached its allocation upper limit.

  3. Check the job's minimum resource requirements against the queue's remaining quota. If the sum of currently allocated resources, resources already enqueued, and the job's minimum resource requirements exceeds the queue's effective upper limit, the job remains Pending.
  4. Verify that enqueue, allocate, and capacity are enabled in the scheduler configuration, and that proportion is not enabled simultaneously.
  5. If the queue quota is sufficient, check node-level constraints, including allocatable resources, taints, labels, node affinity, and device add-on status.
  6. Review scheduler logs for anomalous scheduling events.

     kubectl logs -n volcano-system <volcano-scheduler-pod>

Why Is the Resource Size Relationship Invalid When Configuring a Queue?

• Possible cause: The capacity add-on's resource constraint formula is violated.
• Troubleshooting: Verify each resource dimension satisfies the following:

  guarantee.resource <= deserved <= capability

  Additionally, confirm that resource names and units are correct. For example, use 4 or 4000m for CPU, 16 Gi for memory, and nvidia.com/gpu: "1" for GPU.

Why Does a Job Still Fail to Run When the Queue Quota Is Sufficient?

Queue quota validation is only one phase of the scheduling process. The job must also satisfy the following requirements:

• The cluster has sufficient CPU, memory, GPU/NPU, or other extended resources.
• Node selectors, node affinity, taints, and tolerations allow pod placement on available nodes.
• The PodGroup's minAvailable requirement can be satisfied.
• The node can pull the container image.

To identify the cause, run the following command:

kubectl describe podgroup -n <namespace> <podgroup-name>
kubectl describe pod -n <namespace> <pod-name>

Why Does a New Job Show Insufficient Queue Quota Even When Queue Quota Remains?

status.allocated reflects only resources that have been allocated. During the enqueue phase, the scheduler also accounts for resources consumed by jobs that have entered the queue but are not yet allocated. Consequently, the apparent remaining quota calculated from status.allocated may exceed the actual remaining quota the scheduler can accept.

Check for PodGroups in Inqueue or Pending state within the same queue.

kubectl get podgroup -A -o wide

Why Is a Job Blocked by Insufficient Queue Quota When GPUs/NPUs Are Still Available Cluster-Wide?

This is expected behavior when the queue's capability limit is enforced. capability caps the maximum resources a queue can consume, regardless of cluster-wide idle capacity.

To grant the queue additional resources, a cluster administrator should evaluate other queues' configurations and overall cluster capacity, and adjust the queue's capability accordingly. Do not modify the job to bypass the target queue.