Help Center/ ModelArts/ Model Training/ Creating a Training Job/ Priority and Scheduling Policy
Updated on 2026-11-11 GMT+08:00

Priority and Scheduling Policy

When using a dedicated resource pool for training jobs, you can set the job priority when creating a training job or adjust the priority when a job is in the Pending state for a long time. By adjusting the job priority, you can reduce the job queuing duration.

Overview

Some training jobs, such as unimportant tests or experiments, are of low priority. Some low-priority tasks are for testing or simple, non-essential experiments. High-priority jobs enter the scheduling queue faster than low-priority ones.

You can adjust the job execution sequence by configuring the priority of training jobs to ensure normal operation of important services at peak hours. For details, see Changing the Priority.

For details about training job scheduling policies, see Training Job Scheduling Policy.

Constraints

  • You can set the priority of a training job only if it is created using a dedicated resource pool.
  • The job priority can be set to 1, 2, or 3. A larger number indicates a higher priority. The default priority is 1, and the highest priority is 3. To set the priority to 3, you must have the permission. For details about how to configure the permission, see Assigning the Permission to Set the Highest Job Scheduling Priority to an IAM User.
  • If you want to change the priority of a training job in the Pending state, it must be queued for resources in a dedicated resource pool.

Configuring the Priority

On the Create Training Job page, select Increase job scheduling priority. For details, see Configuring Resources. The priority can be set to 1, 2, or 3. A larger number indicates a higher priority. The default priority is 1, and the highest priority is 3.

Changing the Priority

  • Method 1: Change the priority of a training job on the ModelArts console.
    1. Log in to the ModelArts console.
    2. In the navigation pane, choose Model Training > Training Jobs (new console: Training).
    3. On the training job list page, find the target training job in the Pending status.
    4. Click the target training job to access its details page.
    5. Click next to Job Priority. In the displayed dialog box, change the priority and click OK.
  • Method 2: Change the priority of a job using an API.

    Call the API for updating a training job to change the job priority. For details, see Modifying the Description of a Training Job.

Assigning the Permission to Set the Highest Job Scheduling Priority to an IAM User

You can set the job priority to 1 or 2 by default. Once permission to set the highest priority is granted, you can set it to 1, 2, or 3. If you have been assigned with the modelarts:trainJob:setHighPriority permission, you can set this parameter to 3.

  1. Log in to the Huawei Cloud console as a tenant user, hover over your username in the upper right corner, and choose Identity and Access Management from the drop-down list to switch to the IAM management console.
  2. On the IAM console, choose Permissions > Policies/Roles from the navigation pane, click Create Custom Policy in the upper right corner, and configure the following parameters.
    • Policy Name: Enter a custom policy name, for example, Allowing Users to Set the Highest Job Priority.
    • Policy View: Select Visual editor.
    • Policy Content: Select Allow, ModelArts Service, modelarts:trainJob:setHighPriority, and default resources.

  3. In the navigation pane, choose User Groups. Then, click Authorize in the Operation column of the target user group. On the Authorize User Group page, select the custom policies created in 2, and click Next. Then, select the scope and click OK.

    After the configuration, all users in the user group have the permission to set the training job priority to 1 to 3.

    If no user group is available, create a user group, add users using the user group management function, and configure authorization. If the target user is not in a user group, you can add the user to a user group through the user group management function.

Training Job Scheduling Policy

ModelArts scheduler supports four job queuing and scheduling policies, suited for different resource management and service scenarios. The policy is configured at the resource pool level and affects the scheduling order and behavior of all jobs within that resource pool.

  1. First In, First Out (FIFO)

    Policy description: Jobs are sorted by priority and creation time, and scheduled sequentially from the front of the queue. When the resources requested by a job are locked by other jobs, the job is not scheduled. If a job fails to be scheduled due to insufficient resources, a lock is applied to the requested resources of the current job, and the scheduler continues to attempt the next job.

    Use cases

    • Scenarios with tight resources and strict priorities, ensuring that high-priority jobs obtain resources first, and low-priority jobs cannot cut the line to occupy resources that should belong to preceding jobs.
    • Dedicated resource pools: The resource pool capacity is limited, and jobs are expected to queue strictly in order.
    Figure 1 FIFO scheduling policy
  2. Traverse scheduling.

    Policy description: Jobs are sorted by priority and creation time and scheduled sequentially. When a job fails to be scheduled due to insufficient resources, the scheduler skips that job and continues attempting to schedule subsequent jobs until the entire queue has been traversed.

    Use cases

    • Fragmented resource scenarios: Jobs in the queue have significantly varying resource demands, and the failure of a large job ahead should not block smaller jobs behind it.
    • Resource-utilization-first: Desiring to schedule as many jobs as possible to maximize resource utilization.
    Figure 2 Traversal scheduling policy
  3. Tiered round-robin scheduling.

    Policy description: Introduces a timeout mechanism based on FIFO: After a job fails to be scheduled, a resource lock is applied and subsequent jobs are blocked within a timeout window (behaving like FIFO). After the timeout expires, the resource lock is released, subsequent jobs are automatically allowed to pass, and the schedulable scope is expanded to prevent long-term resource idling caused by front jobs occupying the queue indefinitely.

    Use cases

    • Balancing order and utilization: Needing to maintain a certain queuing order while avoiding resource waste caused by long waiting times for preceding jobs.
    • Elastic resource release scenarios: The resources awaited by preceding jobs may be released shortly (e.g., other jobs completing); waiting is maintained before the timeout, while subsequent jobs are allowed to use resources after the timeout.

  4. Hierarchical scheduling.

    Policy description: Divided into two tiers based on job priority (the larger the number, the higher the priority), with each tier scheduled independently. Jobs with priorities 1 and 2 use the traverse scheduling policy, while jobs with priority 3 use the tiered round-robin scheduling policy. Jobs with priorities 1 and 2 are scheduled only after all priority-3 jobs have been completely scheduled.

    Use cases

    • Multi-tenant/Multi-workload scenarios: Different workloads have different priorities, requiring high-priority workloads to acquire resources first.
    • Hybrid scheduling requirements: Different priority tiers require distinct scheduling behaviors (e.g., strict queuing for high priority, allowing queue bypass for low priority).