Priority and Scheduling Policy

Overview

Some training jobs, such as unimportant tests or experiments, are of low priority. Some low-priority tasks are for testing or simple, non-essential experiments. High-priority jobs enter the scheduling queue faster than low-priority ones.

You can adjust the job execution sequence by configuring the priority of training jobs to ensure normal operation of important services at peak hours. For details, see Changing the Priority.

For details about training job scheduling policies, see Training Job Scheduling Policy.

Constraints

• You can set the priority of a training job only if it is created using a dedicated resource pool.
• The job priority can be set to 1, 2, or 3. A larger number indicates a higher priority. The default priority is 1, and the highest priority is 3. To set the priority to 3, you must have the permission. For details about how to configure the permission, see Assigning the Permission to Set the Highest Job Scheduling Priority to an IAM User.
• If you want to change the priority of a training job in the Pending state, it must be queued for resources in a dedicated resource pool.

Configuring the Priority

On the Create Training Job page, select Increase job scheduling priority. For details, see Configuring Resources. The priority can be set to 1, 2, or 3. A larger number indicates a higher priority. The default priority is 1, and the highest priority is 3.

Changing the Priority

1. Log in to the ModelArts console.
2. In the navigation pane, choose Model Training > Training Jobs (new console: Training).
3. On the training job list page, find the target training job in the Pending status.
4. Click the target training job to access its details page.
5. Click next to Job Priority. In the displayed dialog box, change the priority and click OK.

Assigning the Permission to Set the Highest Job Scheduling Priority to an IAM User

You can set the job priority to 1 or 2 by default. Once permission to set the highest priority is granted, you can set it to 1, 2, or 3. If you have been assigned with the modelarts:trainJob:setHighPriority permission, you can set this parameter to 3.

1. Log in to the Huawei Cloud console as a tenant user, hover over your username in the upper right corner, and choose Identity and Access Management from the drop-down list to switch to the IAM management console.
2. On the IAM console, choose Permissions > Policies/Roles from the navigation pane, click Create Custom Policy in the upper right corner, and configure the following parameters.
   • Policy Name: Enter a custom policy name, for example, Allowing Users to Set the Highest Job Priority.
   • Policy View: Select Visual editor.
   • Policy Content: Select Allow, ModelArts Service, modelarts:trainJob:setHighPriority, and default resources.

3. In the navigation pane, choose User Groups. Then, click Authorize in the Operation column of the target user group. On the Authorize User Group page, select the custom policies created in 2, and click Next. Then, select the scope and click OK.

   After the configuration, all users in the user group have the permission to set the training job priority to 1 to 3.

   If no user group is available, create a user group, add users using the user group management function, and configure authorization. If the target user is not in a user group, you can add the user to a user group through the user group management function.

Training Job Scheduling Policy

Jobs are scheduled based on priority and First In, First Out (FIFO). At the start of each scheduling round, jobs are sorted by priority (highest first) and creation time (oldest first) to create a queue. The system then tries to process each job, starting from the front of the queue. If the top job lacks sufficient resources, the current round ends, and no further jobs are processed.

• Priority Definition

  The priority of a training job can be 1, 2, or 3. A larger value indicates a higher priority. The priority is specified when a job is created and can be dynamically adjusted during job running. For details about how to change the priority, see Changing the Priority.

• Scheduling Process
  1. Fetch: Retrieve all pending jobs in the current queue.
  2. Sort: Descending order by priority → Ascending order by creation time.
  3. Dequeue: Take a job from the head of the queue and attempt to schedule it.

  ├── Sufficient resources: Dequeue and run; current scheduling round ends.

  └── Insufficient resources: Current scheduling round ends; wait for the next round.

  4. The next scheduling round restarts from step 1.
• Typical Scenarios
  • Scenario 1: Priority Determines Scheduling Order

    The queue contains task A (priority 3, created at 9:00), task B (priority 2, created at 8:30), and task C (priority 3, created at 9:30).

    Sorted order: A → C → B Although task B was created earliest, its priority is lower than A and C, so it must wait until the higher-priority tasks finish scheduling.

  • Scenario 2: FIFO Does Not Skip the Head of the Queue

    The queue contains task A (requires 8 PUs, priority 3) and task B (requires 2 PUs, priority 2). Currently, 2 PUs are available.

    Task A is at the head of the queue, but available resources are insufficient. Although there are enough resources for task B, the FIFO policy does not allow skipping the head of the queue. As a result, both tasks remain queued until enough free resources become available for task A.

  • Scenario 3: Dynamic Priority Adjustment

    Task A (priority 1) is queued behind task B (priority 2). After an administrator increases Task A's priority to 3, the sorted order updates in real time, moving task A to the head of the queue for priority scheduling.