Help Center/ ModelArts/ Troubleshooting/ Resource Pool/ Troubleshooting Model Warmup Failures
Updated on 2026-09-02 GMT+08:00
View PDF
Share

Troubleshooting Model Warmup Failures

ModelArts dedicated resource pools support model warmup. By configuring warmup tasks in the console, you can pre-cache weight files into the in-memory acceleration directory. This significantly reduces service initialization time and improves the overall user experience.

When creating or modifying a model warmup task, requests may fail if the configuration or the resource pool environment does not meet the necessary requirements. If you encounter a request error, you are advised to troubleshoot based on the error code provided in Creating or Modifying a Warmup Task Fails.

During the model warmup process, issues such as warmup failures or partial success may occur. These typically manifest as model loading failures, response timeouts, or incomplete initialization of certain functions. In the event of a warmup failure, first verify that the model files are complete and not corrupted. Once the files are confirmed to be intact, check whether system resources, such as memory and CPU, are sufficient. If the warmup is only partially successful, it is recommended to refer to the corresponding error messages and take appropriate corrective actions based on the reported status. For details, see Model Warmup Task Abnormal or Partially Successful.

Creating or Modifying a Warmup Task Fails

An error is reported when a model warmup task is created or modified.

Figure 1 Adding a model warmup task

Table 1 Model warmup error codes

Error Code

Description

Possible Cause

Solution

ModelArts.50015028

Request to create or update warmup task failed. Invalid format. Validation rules not met.

Request to create model warmup task failed. There is no node bound to subpool.

Before creating a model warmup task for a resource queue (old version: logical subpool), enable resource pool node binding. For details, see Enabling/Disabling Node Binding for a Dedicated Resource Pool.

Request to create model warmup task failed. There is a resource queue (old version: logical subpool).

When there is a resource queue (old version: logical subpool), you cannot create model warmup tasks in the physical resource pool. Instead, create them directly in the resource queue (old version: logical subpool).

Request to create model warmup task failed. There are more target pre-warmed nodes than available nodes in the resource pool.

Ensure the target node count for the model warmup task does not exceed the available nodes in the resource pool.

Request to create model warmup task failed as the maximum allowed number has been reached.

Remove unused model warmup tasks to release quotas for new ones.

Request to create model warmup task failed due to a duplicate weight warmup path in the resource pool.

Ensure that each model warmup task has a unique weight path in the resource pool.

Request to modify model warmup task failed. The number of target pre-warmed nodes cannot be 0.

Choose a number higher than 0 for the target warmup nodes in the model warmup task.

Request to create warmup task failed. Invalid name format.

Create a warmup task with a name that follows the validation rules.

Request to create a model warmup task failed because the remaining resources in the resource pool are insufficient.

Before creating or updating a warmup task, ensure that the remaining resources in the resource pool are sufficient.

ModelArts.50015029

Request to disable node binding failed. There are model warmup tasks in the subpool.

Request to disable node binding failed. There are model warmup tasks in the subpool.

Ensure that there are no model warmup tasks in the resource queue (old version: logical subpool) before disabling node binding.

ModelArts.50015030

Request to node rebinding failed. There are model warmup tasks in the subpool.

Request to node rebinding failed. There are model warmup tasks in the subpool.

Enable node draining when rebinding nodes in a resource queue (old version: logical subpool) with weight warmup tasks. For details, see Rebinding a Resource Queue (Old Version: Logical Subpool).

ModelArts.50020100

Task creation or update failed due to insufficient available resources in the resource pool.

The memory or CPU resources in the resource pool are insufficient.

Release unused jobs to meet the resource pool's weight warmup requirement.

ModelArts.50005000

Internal service error.

The error may be caused by the failure to obtain the agency information of the ModelArts IAM user.

Ensure that the IAM user who creates the warmup task has been authorized. For details about how to configure agency authorization, see Configuring Agency Authorization for ModelArts with One Click.

Model Warmup Task Abnormal or Partially Successful

The model warmup task is abnormal or partially successful.
Figure 2 Model warmup error

Table 2 Weight warmup subtask status description

Status

Description

Solution

Warmup succeeded

The nodes in the resource pool have completed weight directory warmup.

No action is required.

Deleting

The nodes in the resource pool are clearing the weight directory.

No action is required.

Warming up

The nodes in the resource pool are warming up the weight directory.

No action is required.

No schedulable node

There is no schedulable node in the resource pool for weight directory warmup. As a result, the subtasks of the weight warmup task cannot be completed on the specified number of nodes. The possible causes are as follows: resource pool scale-in, node tainted, node fault, node draining, node restart, or insufficient node memory.

Check whether the number of available nodes in the resource pool is greater than the target number of nodes to be warmed up.

The possible solutions to avoid no schedulable node are as follows:

  1. Add available nodes to the resource pool.
  2. Delete idle jobs or warmup tasks to release memory resources and drain resources on specified nodes.
  3. Decrease the number of target warmup nodes in the weight warmup task.

Warmup timeout

Weight warmup subtask timed out.

Check whether the size of the weight directory to be warmed up is too large or the download rate of the OBS image is too low. If the fault is not caused by the preceding reasons, contact O&M personnel for troubleshooting.

Insufficient memory

The actual size of the OBS directory selected when a weight warmup task is created is greater than the size of the space occupied by files set by you.

When creating a weight warmup task, set the space occupied by files to be greater than the actual size of the OBS directory.

Unknown error

The failure cause is not identified.

Possible causes:

You have not granted the OBS Administrator permission to the ModelArts agency. As a result, the model warmup fails. Authorize the OBS Administrator permission to the ModelArts agency.

If the fault is not caused by the preceding reasons, contact O&M personnel for troubleshooting.

Parent topic: Resource Pool