Volcano Jobs

Basic Concepts

Prerequisites

• A CCE standard or Turbo cluster of v1.27 or later is available.
• The Volcano Scheduler add-on of 1.19.11 or later has been installed in the cluster. If it is not installed, you can go to the CCE console and search for and install it on the Add-ons page.
• The cluster nodes have enough resources like CPUs and memory to run batch processing tasks.
• (Optional) To use queues, create Volcano queues in advance using kubectl.

If Volcano is used as the scheduler, you are advised to use Volcano to schedule all workloads in the cluster to avoid scheduling conflicts.

Notes and Constraints

• vcjobs rely on the Volcano Scheduler add-on. If this add-on is not installed, vcjobs could not work.
• vcjobs cannot be created using the console. They must be deployed using kubectl or YAML.
• When minAvailable is specified, the cluster must provide enough resources. Otherwise, jobs may remain pending for extended periods.
• vcjobs cannot be directly integrated with Kubernetes CronJobs. They are triggered by an external controller.

Example: Using a vcjob to Run a PyTorch Distributed Training Task

This YAML example shows how to use a vcjob to start a PyTorch distributed training task with one master and two workers. It applies policies and the pytorch plugin to ensure high reliability and automatic configuration.

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: resnet-dist-train
spec:
  # The job is successful only when at least three pods (one master and two workers) are running successfully.
  minAvailable: 3
  # Specify Volcano and enable advanced scheduling capabilities such as Gang scheduling.
  schedulerName: volcano
  # Policies for a job: When a pod is evicted (PodEvicted) or fails (PodFailed),
# restart the job to rebuild all pods of the job.
  # This is critical for distributed training, since it prevents training from being unsynchronized due to the failure of some nodes.
  policies:
    - events: [PodEvicted, PodFailed]
      action: RestartJob
  # Enable the pytorch plugin to automatically inject the environment variables and startup parameters required for the distributed training task.
# Based on the master and worker names and number of replicas in tasks,
# the plugin automatically configures MASTER_ADDR, RANK, WORLD_SIZE, MASTER_PORT, and more.
  plugins:
    pytorch: ["--master=master", "--worker=worker", "--port=23456"]
    status-mounter: ["-statusJson=true","-mountPoint=/status","-values=retryCount,pending,running"]
  tasks:
    # Master: coordinates the training process.
    - replicas: 1
      name: master
      # Policies for a task: When the master task is complete, the entire job is automatically marked as complete.
      policies:
        - event: TaskCompleted
          action: CompleteJob
      template:
        spec:
          containers:
            - image: gcr.io/kubeflow-ci/pytorch-dist-sendrecv-test:1.0
              imagePullPolicy: IfNotPresent
              name: master
          restartPolicy: OnFailure
    # Worker: executes model training.
    - replicas: 2
      name: worker
      template:
        spec:
          containers:
            - image: gcr.io/kubeflow-ci/pytorch-dist-sendrecv-test:1.0
              imagePullPolicy: IfNotPresent
              name: worker
              workingDir: /home
          restartPolicy: OnFailure

Key Features

Policy

• Policies for a job (spec.policies): apply to the entire job. For example, if any pod fails, RestartJob terminates all pods and recreates them to ensure that distributed training restarts from a consistent state.
• Policies for a task (tasks[].policies): apply to a task. For example, the TaskCompleted event triggers CompleteJob, which automatically terminates the entire job once the master task has completed.

pytorch plugin

• The built-in pytorch plugin can automatically inject the following environment variables to pods:
  • MASTER_ADDR: IP address of the master pod
  • MASTER_PORT: specified by --port (23456 in this example)
  • WORLD_SIZE: the total number of processes, one master and two workers (three in total)
  • RANK: the unique sequence number assigned to each pod. The master pod is always 0, while worker pods are numbered sequentially starting from 1.
• Manual service discovery and IP address configuration are not required. This simplifies distributed training deployment.
  

  This plugin requires the use of a torch.distributed startup (such as torch.distributed.run or torch.distributed.launch) within a pod, and it must read the previously defined environment variables.

status-mounter plugin

This built-in plugin automatically injects the status of a vcjob into pods through environment variables or file mounting. It supports:

• mountPoint: specifies the file path where status information is written. If it is not set, no files are mounted. An example is mountPoint="/status".
• addEnv: controls whether status information is injected as environment variables into a pod. It is enabled by default. An example is -addEnv=true.
• values: defines the vcjob status field to be injected into a pod. By default, only retryCount is included. An example is -values=retryCount,pending,running, which injects the retryCount, pending, and running status values into the pod.
• statusJson: determines whether the status object of an entire vcjob is serialized into the JSON format and mounted. By default, it is disabled. An example is -statusJson=true, which enables the function. After it is enabled, a field named jsonStatus is generated, which contains the complete status data in the JSON format.
• prefix: specifies the prefix of an environment variable to be injected. The default value is Job_. An example is -prefix=Job_, which will result in environment variable names such as Job_retryCount and Job_pending.
• containers: specifies a container for state injection. The default value is *, which means the configuration applies to all containers. An example is -containers=*.

Deployment and Verification

1. Apply the YAML.

   kubectl apply -f resnet-dist-train.yaml

2. Check the job status.

   # Check the job status.
   kubectl get vcjob resnet-dist-train
   # Check the pods.
   kubectl get pods -l volcano.sh/job-name=resnet-dist-train
   # View logs, for example, worker-0.
   kubectl logs resnet-dist-train-worker-0

   If the job status is Completed, the training has been completed.

   During training, you can monitor the running status of distributed tasks in real time by reading the environment variables injected by the plugin or mounted files. This approach helps determine the scheduling and execution status of the current tasks, enabling corresponding adjustments or decision-making. For example, the status-mounter plugin writes task status to a file at the specified path. You can run the following command to view the status values:

   kubectl exec resnet-dist-train-worker-0 -- cat /status/pending
   1
   kubectl exec resnet-dist-train-worker-0 -- cat /status/running
   2

   This command output indicates that one pod is pending startup while two pods are already running. Based on this information, it can be determined that the tasks have not fully started, as one worker is still waiting for scheduling. This approach is not only convenient for debugging but also useful for dynamically reading job statuses within the training scripts. It enables more intelligent task control and resource management.

Components

The table below lists the core components required by vcjobs. All these components are in the kube-system namespace.

Common Issues

• What are the differences between vcjobs and Kubernetes native jobs?

  vcjobs support advanced features such as multiple tasks, minAvailable, queues, and Gang scheduling, making them well-suited for AI and HPC batch processing scenarios. In contrast, Kubernetes native jobs support only a single pod template and a simple retry policy.

• How do I view the scheduling logs of a vcjob?

  You can use the following command:

  kubectl logs -n kube-system -l app=volcano-scheduler