Help Center> ModelArts> Resource Management> Elastic Cluster> Viewing Details About a Resource Pool

Updated on 2024-06-12 GMT+08:00

View PDF

Viewing Details About a Resource Pool

Log in to the ModelArts management console. In the navigation pane, choose Dedicated Resource Pools > Elastic Cluster.
In the resource pool list, click a resource pool to go to its details page and view its information.
- If there are multiple resource pools, click in the upper left corner of the details page of one resource pool to switch between resource pools. Click More in the upper right corner to perform operations such as adjust capacity, delete the resource pool, or set the job type. The available operations vary depending on resource pools.
- In the Network area of Basic Information, you can click the number of resource pools associated to view associated resource pools.
- In the extended information area, you can view the monitoring information, jobs, nodes, specifications, events, and subpools. For details, see the following section.

Viewing Resource Pool Jobs

On the resource pool details page, click Jobs. You can view all jobs running in the resource pool. If a job is queuing, you can view its queuing position.

Only training jobs can be viewed.

Viewing Resource Pool Monitoring Information

On the resource pool details page, click Monitoring. The resource usage including used CPUs, memory usage, and available disk capacity of the resource pool is displayed. If AI accelerators are used in the resource pool, the GPU and NPU monitoring information is also displayed.

Figure 1 Viewing resource views

Figure 2 Jobs

Viewing Resource Pool Nodes

On the resource pool details page, click Nodes. You can view all nodes in the resource pool and the resource usage of each node.

Some resources are reserved for cluster components. Therefore, CPUs (Available/Total) does not indicate the number of physical resources on the node. It only displays the number of resources that can be used by services. CPU cores are metered in milicores, and 1000 milicores equal 1 physical core.

Deleting nodes:
On the Nodes tab, locate the node to be deleted. In the Operation column, click Delete.

To delete nodes in batches, select the check boxes next to the node names, and click Delete.

Delete abnormal nodes in a resource pool one by one or in batches and add new ones for substitution.
- Before deleting a node, ensure that there are no running jobs on this node. Otherwise, the jobs will be interrupted.
- If there is only one node, it cannot be deleted.
Replacing nodes:
On the Nodes tab, locate the node to be replaced. In the Operation column, click Replace. No fee is charged for this operation.

Check the node replacement records on the Records page. Running indicates that the node is being replaced. After the replacement, you can check the new node in the node list.

The replacement can last no longer than 24 hours. If no suitable resource is found after the replacement times out, the status changes to Failed. Click to check the failure cause.
- The number of replacements per day cannot exceed 20% of the total nodes in the resource pool. The number of nodes to be replaced cannot exceed 5% of the total nodes in the resource pool.
- Ensure that there are idle node resources. Otherwise, the replacement may fail.

Figure 3 Nodes

Figure 4 Operation records

Viewing Resource Pool Specifications

On the resource pool details page, click Specifications. You can view the specifications used by the resource pool and the number of each specification.

Figure 5 Specifications

Viewing Resource Pool Events

On the resource pool details page, click Events. You can view all events of the resource pool. The cause of an event is PoolStatusChange or PoolResourcesStatusChange.

On the trace list, click on the right of Event Type to filter events.

When a resource pool starts to be created or becomes abnormal, the resource pool status changes and the change will be recorded as an event.
When the number of nodes that are available or abnormal or in the process of being created or deleted changes, the resource pool node status changes and the change will be recorded as an event.

Figure 6 Events