Help Center/ ModelArts/ ModelArts User Guide (Standard)/ Using ModelArts Standard to Train Models/ Managing Model Training Jobs/ Priority of a Training Job
Updated on 2025-08-18 GMT+08:00
View PDF
Share

Priority of a Training Job

When using a dedicated resource pool for training jobs, you can set the job priority when creating a training job or adjust the priority when a job is in the Pending state for a long time. By adjusting the job priority, you can reduce the job queuing duration.

Overview

Some training jobs, such as unimportant tests or experiments, are of low priority. In this case, you need to prioritize training tasks (jobs). Jobs with higher priority are scheduled before those with lower priority.

You can adjust the job execution sequence by configuring the priority of training jobs to ensure normal running of important services at peak hours.

Constraints

  • You can set the priority of a training job only if it is created using a dedicated resource pool.
  • The job priority can be set to 1, 2, or 3. A larger number indicates a higher priority. The default priority is 1, and the highest priority is 3. To set the priority to 3, you must have the permission. For details about how to configure the permission, see Assigning the Permission to Set the Highest Job Scheduling Priority to an IAM User.
  • If you want to change the priority of a training job in the Pending state, it must be queued for resources in a dedicated resource pool.

Configuring the Priority

On the Create Training Job page, select Increase job scheduling priority. The priority can be set to 1, 2, or 3. A larger number indicates a higher priority. The default priority is 1, and the highest priority is 3.

Changing the Priority

  1. On the training job list page, find the target training job in the Pending status.
  2. Click the target training job to access its details page.
  3. Click next to Job Priority. In the displayed dialog box, change the priority and click OK.

ModelArts Standard manages training job queues using a priority system.

  1. For jobs with equal priority, scheduling follows these rules:
    • If resources are enough, jobs run based on their submission order using the FIFO method.
    • If resources fall short, some jobs from separate logical pools might get prioritized instead, disrupting strict FIFO execution.
  2. For jobs with adjusted priority, scheduling follows this rule:

    After a training job is delivered to a resource pool, its priority cannot be changed.

Assigning the Permission to Set the Highest Job Scheduling Priority to an IAM User

You can set the job priority to 1 or 2 by default. Once permission to set the highest priority is granted, you can set it to 1, 2, or 3. If you have been assigned with the modelarts:trainJob:setHighPriority permission, you can set this parameter to 3.

  1. Log in to the Huawei Cloud console as a tenant user, hover over your username in the upper right corner, and choose Identity and Access Management from the drop-down list to switch to the IAM management console.
  2. On the IAM console, choose Permissions > Policies/Roles from the navigation pane, click Create Custom Policy in the upper right corner, and configure the following parameters.
    • Policy Name: Enter a custom policy name, for example, Allowing Users to Set the Highest Job Priority.
    • Policy View: Select Visual editor.
    • Policy Content: Select Allow, ModelArts Service, modelarts:trainJob:setHighPriority, and default resources.
  3. In the navigation pane, choose User Groups. Then, click Authorize in the Operation column of the target user group. On the Authorize User Group page, select the custom policies created in 2, and click Next. Then, select the scope and click OK.

    After the configuration, all users in the user group have the permission to set the training job priority to 1 to 3.

    If no user group is available, create a user group, add users using the user group management function, and configure authorization. If the target user is not in a user group, you can add the user to a user group through the user group management function.