Repacking Volcano Heterogeneous Resources

Measuring Repacking Benefits by Released Complete Nodes

For workloads with fixed requirements for the number of processing units per pod or for complete nodes, whether idle processing units can be used for scheduling depends on whether the remaining capacity of a single node meets the resource request. Repacking keeps the total amount of existing resource requests unchanged and measures fragmentation improvement by reducing the number of nodes on which the target resource is allocated to running workloads (by emptying complete nodes for the target resource).

How to Calculate Fragmentation Rate

For target resource R:

                Number of currently occupied nodes – Theoretically minimum number of occupied nodes
Fragmentation rate (R) = ------------------------------------------------------- × 100%
                      Number of nodes providing resource R

• Number of nodes providing a specific resource: the number of nodes for which Allocatable[R] is greater than 0.
• Number of currently occupied nodes: the number of nodes on which resource R is allocated to running workloads.
• Theoretically minimum number of occupied nodes: the minimum number of nodes required for compact packing while keeping current resource requests unchanged.

For example, 15 out of 20 NPU nodes are in use, and only 12 nodes are theoretically required at minimum. In this plan, two nodes are to be released:

• Before repacking: (15 – 12)/20 × 100% = 15%
• After repacking: (13 – 12)/20 × 100% = 5%
• Improvement: 10 percentage points

minFragImprovementPercent: 10 indicates that the fragmentation rate must decrease by at least 10 percentage points. The fragmentation rate is calculated based on the entire cluster. scope.nodes only limits the nodes that can be emptied and does not change the statistical scope. This metric measures the compactness of resource distribution and does not guarantee that a future workload can be scheduled. Each planned migration still needs to pass the scheduler's feasibility verification.

Repacking uses goals to specify target resources and minimum improvement. The solution must release at least one target resource node and reach the configured improvement threshold.

Configuration Example

goals:
  - resource: huawei.com/ascend-1980
    # The repacking is worth executing only if the fragmentation rate decreases by at least 10 percentage points.
    minFragImprovementPercent: 10

If the resource distribution is already compact enough, the run ends with NoFragmentation. If there are fragments but the benefit threshold cannot be reached, the run ends with InsufficientImprovement. Both are normal evaluation results.

Using Scope to Limit Eviction

Repacking evicts running workloads. The scope limits the operation from two dimensions:

• scope.podGroups: defines which workload pods can be evicted.
• scope.nodes: defines target nodes for emptying.

For example, when training and inference workloads are deployed in the same cluster, low-priority training and offline inference can be allowed for repacking, while critical online inference is excluded. On the node side, you can select only a specific node pool as the scope for emptying.

Configuration Example

scope:
  podGroups:
    include:
      selector:
        matchLabels:
          repack.volcano.sh/eligible: "true"
    exclude:
      selector:
        matchLabels:
          repack.volcano.sh/protected: "true"
  nodes:
    include:
      selector:
        matchLabels:
          accelerator-pool: ascend-npu
    exclude:
      names: [ascend-maintenance-01]

The VCJob controller copies the job label to its PodGroup, and Kthena copies the ModelServing label to the PodGroup corresponding to each ServingGroup. The general pg-controller also inherits stable labels from pod templates of workloads such as Deployments and StatefulSets. You only need to maintain labels on workloads and configure the eviction scope using label selectors. You do not need to identify or maintain automatically generated PodGroup names.

Within the same include or exclude, names and selectors are evaluated with the OR logic (union), and exclude has a higher priority. PodGroup names are in the format namespace/name, and node names are used directly.

scope.nodes specifies which nodes can be emptied, but does not specify which node receives the pod. If a workload cannot leave a specified resource pool, this should be expressed through node affinity, taints and tolerations, or other native scheduling constraints. Repacking inherits these constraints during planning.

Verifying Migration Feasibility Based on Complete Scheduling Semantics

If the number of processing units requested by a pod is less than the number of idle ones on a node, this only indicates that the plan is feasible in terms of total resource quantity, but does not prove that the pod can actually be scheduled. When generating a plan, repacking reuses the scheduling policy of Volcano Scheduler to filter nodes, simulates rescheduling of each pod to be migrated, and checks whether original scheduling constraints and remaining node resources allow the migration.

Whether a node can be used as a candidate for emptying is also determined through unified migration feasibility assessment:

• DaemonSets, CNIs, kube-proxy, and regular pods that do not request target resources are not involved in migration and do not prevent target resources from being emptied.
• Pods that request target resources must be scheduled by Volcano, associated with an identifiable PodGroup, and belong to a workload that scope.podGroups allows to be evicted.
• If any target resource pod is not migratable or any pod to be migrated does not have a feasible receiving node, the candidate is eliminated.

The simulation process tracks the cumulative total of planned migrations to prevent receiving node capacity from being exceeded when multiple locally feasible solutions are combined. It checks whether a complete migration path exists in the current snapshot, but does not guarantee future cluster status. The receiving node selected through simulation does not trigger resource reservation. The Execute phase replans and provides candidate node suggestions through nominatedNodeName. The final binding result is still determined by the scheduler based on real-time status.

Evaluating the Impact of Workload Interruption Using Gang Semantics

In distributed training and model inference, the repacking solution should not distribute migration across a large number of workloads. Even if the total number of pods to be migrated is the same, migrating multiple pods from one workload usually has a smaller impact scope than migrating one pod from each of multiple workloads. Therefore, repacking uses PodGroup as the unit for measuring interruption impact and calculates multi-dimensional cost for each feasible emptying solution.

Whether the Number Is Less Than the Gang MinAvailable

For a PodGroup, repacking reads the number of running members and MinAvailable from the scheduler snapshot and calculates the maximum number of pods that can be migrated without affecting the minimum number of available gang members:

Number of pods that can be safely migrated = max(Running − MinAvailable, 0)

• If the number of planned pods to migrate does not exceed this value, available replicas in the PodGroup still meet the MinAvailable requirement, and the affected target resource quantity is calculated based on that number.
• If the number of planned pods to migrate exceeds this value, the count of PodGroups whose available members fall below MinAvailable increases, and the number of affected target resources is calculated based on the target resource usage of the entire PodGroup.

For example, a training job has eight running workers, and minAvailable is 6.

• After one or two workers are migrated, minAvailable is still met.
• After the third worker is migrated, the number of available members falls below minAvailable. Repacking counts the NPUs used by the entire PodGroup as the affected target resources.

Multi-Dimensional Interruption Impact Scoring

Each feasible candidate is calculated across five dimensions based on the complete plan, instead of evaluating only partial pods on the current node.

• Number of affected workloads: The default weight is 1.0. Minimize the number of training or inference workloads affected by migration.
• Number of PodGroups whose available members fall below MinAvailable: The default weight is 0.8.
• Affected target resources: The default weight is 0.6. If the PodGroup still meets the MinAvailable requirement, the number of actually migrated processing units is calculated. Otherwise, the number of processing units in the entire PodGroup is calculated.
• Target resources to be migrated: The default weight is 0.3. Reduce the number of NPUs or GPUs to be migrated when workload impact is similar.
• Number of pods to be migrated: The default weight is 0.1. This metric further reduces the number of evicted and rebuilt objects.

Each dimension is normalized using Min-Max across the current round of candidates, then multiplied by the default weight and summed. The lower the total score, the higher the priority of the solution. The weights reflect comprehensive preferences, not a strict hierarchical order. The scores are used only for relative selection within the same planning round and cannot be compared across clusters or runs.

For example, all three solutions can release one node and migrate three processing units. Solution A migrates three pods from the same workload, and the MinAvailable requirement is still met after migration. Solution B migrates one pod from each of three workloads. Solution C affects only one workload, but the number of pods in the PodGroup falls below MinAvailable after migration. Generally, solution A is preferred based on the comprehensive score. Solution B has a larger workload impact scope, and solution C has higher costs in terms of the number of PodGroups below MinAvailable and the amount of affected target resources.

The scope and maxPerRun are hard limits, and the interruption impact score is only used to rank candidate solutions that meet the limits. The scoring tends to reduce affected workloads and avoid making PodGroups fall below MinAvailable, but it is not used as a prohibition condition. Workloads that cannot be interrupted should be explicitly excluded using scope.podGroups.exclude.

Dynamically Selecting Targets to Empty and Suppressing Secondary Fragmentation

Repack planning does not process nodes one by one after determining the node sequence in advance. The planner works in incremental mode: Each time an emptying solution is selected, the corresponding migration is accounted for in the simulation state, and the remaining candidates are re-evaluated. This allows handling situations where receiving capacity is consumed by previous migrations, the PodGroup impact scope grows, or the cost of emptying subsequent nodes changes.

Selecting Candidates for Emptying

In each round, candidates that do not meet migration feasibility, execution budget, receiving capacity, or scheduler feasibility are first excluded, and then candidates are selected in ascending order of total interruption impact score. The total score also considers the number of affected workloads, the number of PodGroups below MinAvailable, the amount of affected target resources, the amount of resources to be migrated, and the number of pods to be migrated. If scores are the same, emptying benefits and candidate names are further compared to obtain stable results.

Repacking does not select the node with the most currently idle processing units, but rather the node that can be emptied, has a complete migration path, and has relatively low workload interruption impact.

Filling Receiving Nodes

Removing pods from the source node does not necessarily mean that fragmentation is improved. If the receiving node is not properly selected, the migration may only move fragments from one location to another. Repacking organizes receiving nodes based on the following rules:

1. Exclude nodes that are being emptied and avoid reoccupying nodes whose target resources are completely idle.
2. Preferentially schedule pods to nodes that are determined to remain occupied due to workload eviction restrictions, scope limitations, or existing migrations.
3. Among nodes that may still be defragmented later, preferentially use nodes that will incur higher costs in the future, and reserve nodes that are easier to release.
4. Among candidate nodes of the same type, preferentially select the node with smaller remaining target resource capacity after migration.

This policy aims to empty the current node while retaining a compact resource distribution for subsequent jobs and the next round of repacking. Node sorting only determines the simulation sequence. Final candidates still need to be filtered by the scheduler.

Limiting the Impact Scope of a Single Run Through Budgeting

maxPerRun limits the change scope of a single run across two dimensions: the number of workloads and the number of target resources.

• podGroups: the maximum number of workloads that can be affected. The system counts workloads by PodGroup.
• resources.<resource-name>: the maximum number of target resources that can be migrated.

podGroups is an API field name. You do not need to maintain PodGroups based on this field. A VCJob usually corresponds to a PodGroup. Each ServingGroup of ModelServing corresponds to a PodGroup. Therefore, you need to set the budget based on the actual number of ServingGroups that may be affected.

Configuration Example

maxPerRun:
  # A maximum of two training or inference workloads can be affected.
  podGroups: 2
  resources:
    # Even if the workload scale is larger, a maximum of eight processing units can be migrated in this run.
    huawei.com/ascend-1980: 8

Any candidate exceeding the upper limit will be eliminated in the planning phase. For the first rollout, it is recommended that you start with one workload and a small number of processing units, and gradually increase the budget based on the DryRun, historical recovery duration, and operations window.

Execute runs serially within the cluster and enters a cooldown period after each execution to prevent multiple rounds from affecting each other. Pod eviction is performed through the Kubernetes Eviction API. The API server checks the PDB based on real-time status. DryRun does not guarantee that the PDB still allows eviction during execution.

Providing Candidate Node Suggestions Through nominatedNodeName

The receiving nodes provided in the planning phase are based on the current cluster snapshot. After entering the Execute phase, resource usage and scheduling queues may still change. Therefore, repacking does not consider planned nodes as mandatory binding results.

After the original pod is evicted, the workload controller creates a new pod. The built-in repack controller of Volcano Controller Manager identifies the rebuilt pod based on the persistent relocation record. After confirming that the candidate node is still available, the controller writes the node name into pod.status.nominatedNodeName. Volcano Scheduler preferentially evaluates the node and continues to apply complete scheduling rules.

• If the recommended node is still available, the rebuilt pod is preferentially scheduled to this node, improving plan completion probability.
• If the recommended node is unavailable, the scheduler can select another feasible node to prevent expired plans from blocking workload recovery.
• Repacking does not restrict node usage through cordoning, taints, or resource reservations. Other workloads in the scheduling queue can still be scheduled.
• status.relocations[].placement records the planned node, the node selected during execution, and the node actually bound, which are used to identify alternative scheduling and timeouts.

nominatedNodeName is a scheduling suggestion, not node binding or resource reservation.

Closed-Loop from Solution Evaluation to Result Verification

DryRun and Execute use the same planning logic but have different responsibilities. DryRun is used to confirm the benefit threshold, affected workloads, and migration path. Related information is stored in status.plan.summary, status.plan.moves, and status.plan.freedNodes.

After review is passed, a new Execute run should be created and replanned based on real-time status. Execute writes the pod-by-pod eviction and scheduling process to status.relocations and writes the actual node eviction and final fragmentation improvement to status.result.

A difference between planned and actual values does not necessarily indicate a fault. The differences may be caused by PDB status changes, pods being scheduled to other nodes after being rebuilt, or new resource contention during execution. You can view the plan, execution process, and actual result to determine whether the repacking meets the capacity target.

Workload Eviction and Rebuilding Mechanism

VCJob Eviction and Recovery

VCJob allows you to define workload-level policies. For example, if a VCJob is configured with lifecycle policies such as event: PodEvicted and action: RestartJob or action: RestartTask, a repack eviction may trigger restart of the entire workload or all pods under the same job role. As a result, the actual interruption scope becomes much larger than the planned number of pods to be migrated. You are not advised to configure such policies for VCJobs to be involved in repacking. If overall restart semantics are required, evaluate the recovery cost based on all affected pods and use a stricter scope and execution budget.

Eviction and Recovery of ModelServing

spec.replicas of ModelServing indicates that multiple independent ServingGroups are created, instead of creating only the corresponding number of pods. Each ServingGroup contains an entry pod and a worker pod, which share a single PodGroup. The maxPerRun.podGroups of repacking is counted based on the PodGroups corresponding to these ServingGroups.

The example uses recoveryPolicy: ServingGroupRecreate. In this mode, after a pod is evicted by repacking, Kthena not only rebuilds the pod, but also deletes all pods and the old PodGroup in the same ServingGroup, and then creates a new PodGroup and all pods. Therefore, the final effect may be that "Repacking evicts only one pod, but the controller rebuilds the entire ServingGroup."

• restartGracePeriodSeconds controls the waiting time before a ServingGroup is restored due to an abnormal pod. It is different from spec.eviction.gracePeriodSeconds, which controls the graceful termination time of the pod requested by repack eviction.
• Repacking retains the original PodGroup as the plan audit identity and identifies the new generation of PodGroups through the same ModelServing owner. When the name changes, the new name is recorded in relocations[].replacementPodGroupName. This field can be left empty if the original name is reused.
• After reconstructed pods in the new PodGroup are coordinated by the placement lease and SchedulerGate, suggestions on candidate nodes are offered via nominatedNodeName. Final binding is determined by Volcano Scheduler.

This recovery semantics expands the scope of workload interruption. maxPerRun limits the number of PodGroups and target resources to be migrated in the repack plan and cannot replace the availability design of ModelServing. Production inference services must retain at least one other active ServingGroup to ensure successful traffic draining and model warm-up, treating a complete ServingGroup as the atomic unit for assessing disruption impact. ModelServings that cannot accept ServingGroup rebuilding should be marked with repack.volcano.sh/protected=true and explicitly excluded by scope.

Step 1: Perform DryRun Evaluation

For the first use, you only need to specify the mode and target resources. The following CR evaluates fragmentation and migration feasibility on nodes that provide huawei.com/ascend-1980 and does not evict any pod:

apiVersion: repack.volcano.sh/v1alpha1
kind: RepackRun
metadata:
  name: ascend-npu-dryrun
spec:
  mode: DryRun
  goals:
    - resource: huawei.com/ascend-1980

Run the apply command and observe the result.

kubectl apply -f ascend-npu-dryrun.yaml
kubectl get repackrun ascend-npu-dryrun -w
kubectl get repackrun ascend-npu-dryrun -o yaml

Pay attention to status.conditions and status.plan. In DryRun mode, status.phase=Succeeded only indicates that evaluation is complete, but does not mean that migration has been performed. Meaning of conditions:

• RepackRecommended: There are feasible plans with positive benefits. You can continue to add security controls and review them.
• NoFragmentation: The current target resource does not need repacking.
• InsufficientImprovement: There are fragments, but no feasible plan meets the requirements.

If scope is not specified, workloads and evicted nodes are not limited. If maxPerRun is not specified, the impact scope of a single operation is not limited. This minimum configuration is suitable for the first evaluation. Before entering the Execute phase, supplement scope and maxPerRun as needed.

When extending from this baseline CR, you only need to add corresponding fields as needed:

• Specifying workloads or node pools that can be evicted: Configure the scope field.
• Filtering low-benefit plans: Configure the goals[].minFragImprovementPercent field.
• Limiting the number of workloads and processing units affected per operation: Configure the maxPerRun field.
• Performing migration: Create a RepackRun and change the mode to Execute.
• Adjusting graceful termination time of pods or CR retention time: Configure the eviction or ttlSecondsAfterFinished field.

Step 2: (Optional) Prepare Workload and Node Labels

If you have configured stable labels for VCJob and ModelServing that can be used for RepackRun scope filtering, skip this step and go to Step 3: Add a Scope as Needed.

This section describes how to prepare labels and sample workloads for the example environment.

1. Add labels to nodes.

   kubectl label node ascend-node-01 accelerator-pool=ascend-npu
   kubectl label node ascend-node-02 accelerator-pool=ascend-npu
   kubectl label node ascend-node-03 accelerator-pool=ascend-npu

   Ensure the nodes have correctly reported the NPU capacity.

   kubectl describe node ascend-node-01

2. Create workloads.

   The following example creates a VCJob training job and a Kthena ModelServing inference service. Both types of workloads are explicitly marked as eligible for repacking. Online services that cannot be migrated in the production environment should be labeled with repack.volcano.sh/protected=true (this is only an example; you can customize the label) and excluded from subsequent scopes.

   apiVersion: v1
   kind: Namespace
   metadata:
     name: repack-demo
   ---
   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: ascend-batch-training
     namespace: repack-demo
     labels:
       workload-type: training
       repack.volcano.sh/eligible: "true"
       repack.volcano.sh/protected: "false"
   spec:
     schedulerName: volcano
     queue: default
     minAvailable: 2
     tasks:
       - name: worker
         replicas: 2
         template:
           spec:
             nodeSelector:
               accelerator-pool: ascend-npu
             terminationGracePeriodSeconds: 30
             containers:
               - name: trainer
                 image: alpine:3.20
                 command: ["sh", "-c", "tail -f /dev/null"]
                 resources:
                   requests:
                     huawei.com/ascend-1980: 1
                   limits:
                     huawei.com/ascend-1980: 1
             restartPolicy: Never
   ---
   apiVersion: workload.serving.volcano.sh/v1alpha1
   kind: ModelServing
   metadata:
     name: ascend-online-serving
     namespace: repack-demo
     labels:
       workload-type: inference
       repack.volcano.sh/eligible: "true"
       repack.volcano.sh/protected: "false"
   spec:
     schedulerName: volcano
     replicas: 2
     recoveryPolicy: ServingGroupRecreate
     template:
       restartGracePeriodSeconds: 30
       gangPolicy:
         minRoleReplicas:
           inference: 1
       roles:
         - name: inference
           replicas: 1
           workerReplicas: 1
           entryTemplate:
             spec:
               nodeSelector:
                 accelerator-pool: ascend-npu
               containers:
                 - name: leader
                   image: alpine:3.20
                   command: ["sh", "-c", "tail -f /dev/null"]
                   resources:
                     requests:
                       huawei.com/ascend-1980: 1
                     limits:
                       huawei.com/ascend-1980: 1
               restartPolicy: Always
           workerTemplate:
             spec:
               nodeSelector:
                 accelerator-pool: ascend-npu
               containers:
                 - name: worker
                   image: alpine:3.20
                   command: ["sh", "-c", "tail -f /dev/null"]
                   resources:
                     requests:
                       huawei.com/ascend-1980: 1
                     limits:
                       huawei.com/ascend-1980: 1
               restartPolicy: Always

3. Run the following command:

   kubectl apply -f ascend-workloads.yaml
   kubectl get vcjob -n repack-demo
   kubectl get modelserving -n repack-demo
   kubectl get pod -n repack-demo -o wide
   kubectl get podgroup -n repack-demo --show-labels

   The VCJob controller creates a PodGroup for a job and replicates the VCJob metadata.labels to the PodGroup. Kthena creates a PodGroup for each ServingGroup and replicates the ModelServing metadata.labels to these PodGroups. You only need to maintain labels on the workload CR. Later, you can use scope.podGroups.selector to select both training and inference workloads.

   

   Evicting a pod may trigger restart of the entire VCJob or rebuilding of the entire ServingGroup of ModelServing. The actual interruption scope may be greater than the planned migration volume. Read Workload Eviction and Rebuilding Mechanism carefully before using Execute.

Step 3: Add a Scope as Needed

The minimum DryRun is evaluated globally. If only workloads with the repack.volcano.sh/eligible=true label are allowed for repacking and only target nodes can be selected from the accelerator-pool=ascend-npu node pool, copy the minimum CR, use a new name, and add the following to spec:

metadata:
  name: ascend-scoped-dryrun
spec:
  # Retain the mode and goals in the baseline and add scope at the same level.
  scope:
    podGroups:
      include:
        selector:
          matchLabels:
            repack.volcano.sh/eligible: "true"
    nodes:
      include:
        selector:
          matchLabels:
            accelerator-pool: ascend-npu

If you also need to exclude workloads that cannot be interrupted, such as key online inference workloads, add exclude under the existing scope.podGroups. To exclude nodes that are being maintained, add exclude under scope.nodes.

spec:
  scope:
    podGroups:
      # Retain the existing include.
      exclude:
        selector:
          matchLabels:
            repack.volcano.sh/protected: "true"
    nodes:
      # Retain the existing include.
      exclude:
        selector:
          matchLabels:
            maintenance-state: draining

The preceding code block is a field fragment that needs to be merged into the baseline CR and is not two independent RepackRuns.

kubectl apply -f ascend-scoped-dryrun.yaml
kubectl get repackrun ascend-scoped-dryrun -w

# Check the number of PodGroups and nodes selected by the selector.
kubectl get repackrun ascend-scoped-dryrun \
  -o jsonpath='podGroups={.status.plan.summary.resolvedScope.podGroupCount}, nodes={.status.plan.summary.resolvedScope.nodeCount}{"\n"}'

# View affected workload IDs, pods to be migrated, and source and target nodes.
kubectl get repackrun ascend-scoped-dryrun \
  -o jsonpath='{range .status.plan.moves[*]}{.owner.kind}{"/"}{.owner.name}{"\t"}{.namespace}{"/"}{.podGroupName}{"\t"}{.cards}{" cards\n"}{range .pods[*]}{"  "}{.name}{": "}{.fromNode}{" -> "}{.toNode}{"\n"}{end}{end}'
kubectl get repackrun ascend-scoped-dryrun \
  -o jsonpath='{.status.plan.freedNodes}{"\n"}'

Result Confirmation

• The number of resolvedScopes meets the expectation.
• plan.moves[].owner contains only VCJobs or ModelServings allowed by the scope and not protected.
• plan.freedNodes contains only nodes with accelerator-pool=ascend-npu.

Step 4: Add the Benefit Threshold and Execution Budget

Before executing in production, you are advised to add the following fields as needed:

• To avoid executing plans with low benefits, add minimum fragment improvement to existing goals.
• To limit impact scope of a single execution, add maxPerRun. Both upper limits apply concurrently.
• To automatically clear completed DryRuns, add TTL.

All preceding fields are optional extensions. After target resources, scope, benefit threshold, and execution budget are combined, the recommended production DryRun is as follows:

apiVersion: repack.volcano.sh/v1alpha1
kind: RepackRun
metadata:
  name: ascend-guarded-dryrun
spec:
  mode: DryRun
  scope:
    podGroups:
      include:
        selector:
          matchLabels:
            repack.volcano.sh/eligible: "true"
      exclude:
        selector:
          matchLabels:
            repack.volcano.sh/protected: "true"
    nodes:
      include:
        selector:
          matchLabels:
            accelerator-pool: ascend-npu
  goals:
    - resource: huawei.com/ascend-1980
      minFragImprovementPercent: 10
  maxPerRun:
    podGroups: 1
    resources:
      huawei.com/ascend-1980: 4
  ttlSecondsAfterFinished: 86400

Run the following command:

kubectl apply -f ascend-guarded-dryrun.yaml
kubectl get repackrun ascend-guarded-dryrun -w

# Check expected benefits, number of emptied nodes, and number of migrated processing units at a time.
kubectl get repackrun ascend-guarded-dryrun \
  -o jsonpath='fragmentation={.status.plan.summary.fragBeforePercent}% -> {.status.plan.summary.fragAfterPercent}%, freedNodes={.status.plan.summary.freedNodeCount}, movedCards={.status.plan.summary.movedCardCount}{"\n"}'

Assume that DryRun output is as follows, indicating that two processing units in the batch training PodGroup are migrated, ascend-node-01 is expected to be released, and the fragmentation rate decreases from 33% to 0%.

status:
  phase: Succeeded
  plan:
    summary:
      fragBeforePercent: 33
      fragAfterPercent: 0
      freedNodeCount: 1
      movedCardCount: 2
      resolvedScope:
        podGroupCount: 1
        nodeCount: 3
    moves:
      - namespace: repack-demo
        podGroupName: ascend-batch-training-2f6c840d-8b14-4e45-a4cf-b43c249acbfd
        owner:
          apiVersion: batch.volcano.sh/v1alpha1
          kind: Job
          name: ascend-batch-training
        cards: 2
        pods:
          - name: ascend-batch-training-worker-0
            fromNode: ascend-node-01
            toNode: ascend-node-02
            cards: 1
          - name: ascend-batch-training-worker-1
            fromNode: ascend-node-01
            toNode: ascend-node-03
            cards: 1
    freedNodes:
      - ascend-node-01
  conditions:
    - type: Complete
      status: "True"
      reason: RepackRecommended

Step 5: Create an Independent Execute Run

DryRun and Execute are two independent execution modes. Execute does not run a plan saved from a prior DryRun. Instead, it generates a fresh plan based on real-time status. After you review the production DryRun, copy its complete YAML, retain the confirmed scope, goals, and execution budget, and modify only the following fields:

metadata:
  # You must use a new name to create a RepackRun.
  name: ascend-execute-001
  labels:
    change-ticket: change-20260730
spec:
  # Change DryRun to Execute.
  mode: Execute

  # (Optional) Override the pod's graceful termination period used by the Eviction API.
  eviction:
    gracePeriodSeconds: 30

  # (Optional) Retain the Execute result for seven days for auditing.
  ttlSecondsAfterFinished: 604800

This code block is a modification relative to the production DryRun. When creating the Execute file, retain the scope, goals, and maxPerRun from the original CR.

kubectl apply -f ascend-execute-001.yaml
kubectl get repackrun ascend-execute-001 -w

# Continuously observe the evicted pods, rebuilt pods, and node changes.
kubectl get pod -n repack-demo -o wide -w

Execute uses the Kubernetes Eviction API, and PDB constraints still apply. If another Execute is running or still in the execution cooldown period, the new run remains pending and can be identified by AnotherRunActive or ExecuteCooldownActive in Conditions.

Step 6: Check Whether Execute Applies

1. Check the Conditions and final benefits.

   kubectl get repackrun ascend-execute-001 -o wide
   kubectl get repackrun ascend-execute-001 \
     -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
   
   # Compare the planned value with the actual value.
   kubectl get repackrun ascend-execute-001 \
     -o jsonpath='plan: frag {.status.plan.summary.fragBeforePercent}% -> {.status.plan.summary.fragAfterPercent}%, freed {.status.plan.summary.freedNodeCount}, moved {.status.plan.summary.movedCardCount}{"\n"}result: frag {.status.result.fragAfterPercent}%, freed {.status.result.freedNodeCount}, moved {.status.result.movedCardCount}, verified {.status.result.metricsVerified}{"\n"}'

2. Check the rebuilt pod and the actual scheduled node corresponding to each evicted pod.

   kubectl get repackrun ascend-execute-001 \
     -o jsonpath='{range .status.relocations[*]}{.namespace}{"/"}{.victimPodName}{"\teviction="}{.eviction.phase}{"\tplacement="}{.placement.phase}{"\tplanned="}{.plannedNodeName}{"\tselected="}{.placement.selectedNodeName}{"\tactual="}{.placement.actualNodeName}{"\treplacement="}{.placement.replacementPodName}{"\n"}{end}'
   
   # Check whether the VCJob, ModelServing, and pods are restored and check the planned node to be emptied.
   kubectl get vcjob -n repack-demo
   kubectl get modelserving -n repack-demo
   kubectl get podgroup -n repack-demo
   kubectl get pod -n repack-demo -o wide
   kubectl get pod -A --field-selector spec.nodeName=ascend-node-01 -o wide
   kubectl describe node ascend-node-01

   The table below lists the involved fields.

   Field	Description
   plannedNodeName	Target node calculated in the Execute phase, recorded as an immutable plan record.
   selectedNodeName	Node to which repacking writes nominatedNodeName for the rebuilt pod based on the real-time scheduling snapshot.
   actualNodeName	Node finally bound by Volcano Scheduler. It is written by the repack controller after obtaining the spec.nodeName of the rebuilt pod.

   nominatedNodeName does not forcibly bind or reserve resources. Therefore, actualNodeName may differ from the first two fields. If the eviction target is still reached after the pod is scheduled to another node, the run can end successfully with ExecutionCompletedWithAlternativePlacement.

3. Verify ModelServing in special scenarios.

Step 7: Submit a Large-Scale Training Job on the Repacked Node

Repacking only consolidates node resources. It does not forcibly reserve capacity for subsequent jobs. Therefore, after the Execute operation is complete and the node is confirmed to be freed, you should submit a large-scale training job as soon as possible and schedule the job to the Ascend resource pool through the node constraints of the workload.

1. Job submission example: The Volcano job requires two workers to be admitted simultaneously. Each worker occupies four NPUs, totaling eight NPUs, which can be accommodated by a complete node freed after repacking.

   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: ascend-large-training
     namespace: repack-demo
     labels:
       workload-type: training
   spec:
     schedulerName: volcano
     queue: default
     minAvailable: 2
     tasks:
       - name: worker
         replicas: 2
         template:
           spec:
             nodeSelector:
               accelerator-pool: ascend-npu
             containers:
               - name: trainer
                 image: alpine:3.20
                 command: ["sh", "-c", "tail -f /dev/null"]
                 resources:
                   requests:
                     huawei.com/ascend-1980: 4
                   limits:
                     huawei.com/ascend-1980: 4
             restartPolicy: Never

2. Run the commands and verify the job.

   kubectl apply -f ascend-large-training.yaml
   kubectl get job.batch.volcano.sh ascend-large-training -n repack-demo
   kubectl get pod -n repack-demo -l volcano.sh/job-name=ascend-large-training -o wide

   Adjust the number of replicas, minAvailable, and the number of requested NPUs per pod based on the actual number of processing units on the node and the training parallelism policy. This verification focuses on whether the capacity of the completely freed node can accommodate the job, rather than the accumulated number of idle processing units.

Example YAML

apiVersion: repack.volcano.sh/v1alpha1
kind: RepackRun
metadata:
  # RepackRun is a cluster-level resource. You do not need to configure a namespace for it.
  name: ascend-complete-execute-001
  labels:
    # metadata.labels is used only for retrieval and audit. It does not determine the repacking scope.
    accelerator-pool: ascend-npu
    change-ticket: change-20260730
spec:
  # DryRun only generates a plan. Execute replans and evicts pods through the Eviction API.
  mode: Execute

  # Specify the workloads that can be evicted and the nodes that can be emptied.
  # If scope is left blank, neither dimension is restricted.
  scope:
    podGroups:
      # The workload eviction scope. The selector matches the workload labels inherited by the PodGroup.
      include:
        # Within the same include, names and selectors are evaluated with the OR logic (union).
        selector:
          matchLabels:
            repack.volcano.sh/eligible: "true"
        # The PodGroup name is in the format of namespace/name. You are advised to use selector for automatically generated PodGroups.
        names:
          - repack-demo/manually-managed-training-pg
      # The priority of exclude is higher than that of include. The matched workloads will not be evicted.
      exclude:
        selector:
          matchExpressions:
            - key: repack.volcano.sh/protected
              operator: In
              values:
                - "true"
        names:
          - repack-demo/protected-training-pg
    nodes:
      # Only the nodes that can be emptied are restricted. The nodes that receive the pods to be migrated are not restricted.
      include:
        # Names and selectors are evaluated with the OR logic (union).
        selector:
          matchLabels:
            accelerator-pool: ascend-npu
        names:
          - ascend-node-03
      # Exclude nodes that are under maintenance or cannot be repacked from the repacking target.
      exclude:
        selector:
          matchLabels:
            maintenance-state: draining
        names:
          - ascend-node-08

  # Specify the target extended resources to be repacked. Only one type of resource can be configured for each run.
  goals:
    - resource: huawei.com/ascend-1980
      # Repacking is recommended only if the fragmentation rate decreases by at least 10 percentage points.
      minFragImprovementPercent: 10

  # Limit the impact scope of a single execution. If any upper limit is exceeded, the candidate solutions will be filtered out.
  maxPerRun:
    # A maximum of two workloads can be affected. The system counts workloads by PodGroup.
    podGroups: 2
    resources:
      # A maximum of eight Ascend NPUs can be migrated.
      huawei.com/ascend-1980: 8

  eviction:
    # Override the graceful termination period specified in the eviction request. If omitted, the pod's own configuration applies.
    gracePeriodSeconds: 30

  # The run is automatically deleted seven days after it enters the final state. Deleting the CR does not roll back the completed migration.
  ttlSecondsAfterFinished: 604800

Within the same include or exclude, names and selectors are evaluated with the OR logic (union), and exclude has a higher priority. PodGroup names are in the format namespace/name, and node names are Kubernetes node names. The names of automatically generated PodGroups are not fixed. In production configurations, the labels inherited from the workload are preferred.