Viewing Details About a Standard Dedicated Resource Pool

Resource Pool Details Page

1. Log in to the ModelArts console. In the navigation pane on the left, choose Standard Cluster under Resource Management.
2. You can search for the resource pools by name, ID, resource pool status, node status, resource pool type, and creation time.
3. In the resource pool list, click a resource pool to go to its details page and view its information.
   • If there are multiple ModelArts Standard resource pools, click in the upper left corner of the details page of one resource pool to switch between resource pools.
   • For a pay-per-use standard resource pool: On the details page, click More in the upper right corner to scale in/out, delete a resource pool, change the billing mode to yearly/monthly, and perform other operations. Operations that can be performed vary depending on resource pools.
   • For a yearly/monthly standard resource pool: On the details page, click More in the upper right corner to perform operations such as scale-out, unsubscription, renewal, auto-renewal enabling or modification, driver upgrade, and job type setting. Operations that can be performed vary depending on resource pools.
   • In the Network area of Basic Information, you can click the number of resource pools associated to view associated resource pools. You can view the number of available IP addresses on the network.
   • In the extended information area, you can view the monitoring, jobs, nodes, specifications, events, tags, logical subpools, and plug-ins. For details, see the following section.

Viewing Jobs in a Resource Pool

On the resource pool details page, click Jobs. You can view all jobs running in the resource pool. If a job is queuing, you can view its queuing position.

Currently, only training jobs and inference jobs can be viewed.

Viewing Resource Pool Events

On the resource pool details page, click Events. You can view all events of the resource pool. The cause of an event is PoolStatusChange or PoolResourcesStatusChange.

In the event list, click on the right of Event Type to filter events.

• When a resource pool starts to be created or becomes abnormal, the resource pool status changes and the change will be recorded as an event.
• When the number of nodes that are available or abnormal or in the process of being created or deleted changes, the resource pool node status changes and the change will be recorded as an event.

Figure 1 Events

Viewing Resource Pool Nodes

On the resource pool details page, click Nodes. You can view all nodes in the resource pool and the resource usage of each node. Hover the cursor over the node to obtain the node name and resource ID. You can use the resource ID to obtain the bills or query the billing information of yearly/monthly resources in the Billing Center.

Some resources are reserved for cluster components. Therefore, CPUs (Available/Total) does not indicate the number of physical resources on the node. It only displays the number of resources that can be used by services. CPU cores are metered in milicores, and 1,000 milicores equal 1 physical core.

You can delete, unsubscribe from, reset, or restart nodes in batches, or enable or disable HA redundancy for nodes in batches, as shown in the following figure. For details, see Rectifying a Faulty Node in a Standard Dedicated Resource Pool. You can add, edit, and delete resource tags for nodes in batches. You can renew yearly/monthly nodes and enable or modify auto-renewal for them in batches.

Figure 2 Performing operations on nodes in batches

To delete, replace, rectify, reset, restart, authorize a node, and enable or disable HA redundancy for a single node, locate the target node in the list and click the button in the Operation column. For details, see Rectifying a Faulty Node in a Standard Dedicated Resource Pool. You can view the running jobs of a single node. For details, see View the Running Jobs of a Resource Pool Node. You can view the plug-in resource usage of a single node. For details, see Querying Plug-in Component. You can also edit resource tags.

Figure 3 Operations on a single node

In the node search box, you can search for nodes by node name, node status, HA redundancy, batch, driver version, driver status, IP address, and resource tag.

You can export the node information of a standard resource pool to an Excel file. Select the node names, click Export > Export All Data to XLSX or Export > Export Part Data to XLSX above the node list, and click in the browser to view the exported Excel files.

On the node list page, click to customize the information displayed in the node list.

View the Running Jobs of a Resource Pool Node

In the Nodes tab of the resource pool details page, you can view the running jobs of a single node.

To do so, locate the node and choose More > Running Jobs in the Operation column. In the displayed dialog box, you can view the running jobs of the node by searching for the keyword.

Figure 4 View the running jobs of a resource pool node

Querying Plug-in Component

On the resource pool details page, choose Node Management from the navigation pane to view the plug-in usage of the current node.

To view the instance usage of the plug-in, locate the target node and choose More > Query Plug-in Component in the Operation column.

Figure 5 Component instances

Viewing Resource Pool Specifications

On the resource pool details page, click Specifications. You can view the resource specifications used by the resource pool and the number of resources corresponding to the specifications, and adjust the container engine space.

Figure 6 Viewing resource pool specifications

Viewing Resource Pool Monitoring Information

On the resource pool details page, click Monitoring. The resource usage including used CPUs, memory usage, and available disk capacity of the resource pool is displayed. If AI accelerators are used in the resource pool, the GPU and NPU monitoring information is also displayed.

You should set the statistical period to less than seven days. If the period is too long, the query may be delayed due to a large number of AOM metrics.

Figure 7 Viewing monitoring metrics

Managing Resource Pool Tags

You can add tags to a resource pool for quick search.

On the resource pool details page, click Tags. You can view, search for, add, modify, and delete tags of a resource pool.

You can add up to 20 tags.

Managing Logical Subpools

You can split logical subpools from a ModelArts Standard dedicated resource pool and set resource quotas for each subpool, including reserved quotas and quota rollout. After a logical subpool is split, tasks in the training, inference, and development environments must be delivered to the logical subpool and restricted by the quota. Only tasks within the quota can be delivered and run.

On the resource pool details page, switch to the Logical Subpool tab. You can create, resize, and delete a logical subpool, as well as setting the job type for the subpool. For details, see Managing Logical Subpools of a Standard Dedicated Resource Pool.

Binding to a Logical Subpool

1. After node binding is enabled for the standard dedicated resource pool, nodes in a resource pool can be bound to a specified logical subpool.
2. After a logical subpool is bound, the jobs on the associated nodes may be preempted. To add a node to the target logical subpool, you can move the redundant nodes from the target subpool to the available nodes. For details, see Migrating a Node.
3. Log in to the ModelArts console. In the navigation pane on the left, choose Standard Cluster under Resource Management.
4. Click the resource pool with node binding enabled. The resource pool details page is displayed.
5. Switch to the Nodes tab, select the target nodes, and click Bind Logical Subpool above the list.
6. In the displayed dialog box, select the target logical subpool and click OK.

   For logical subpools with node binding enabled, check whether node draining needs to be enabled during scaling. If node draining is enabled, the drainage will be performed on the nodes to be scaled only this time. You can check the drainage information in the Nodes tab. If the draining fails, hover the cursor over the node status to view the failure cause.

   Node draining is to clear node resources to release or remove nodes that are no longer needed.

   

   During the draining, nodes cannot be scheduled and job loads on the nodes will be evicted, which may cause job failures.

Viewing Resource Pool Plug-ins

1. ModelArts provides various plug-ins for you to selectively expand resource pool functions to meet service requirements.
2. On the resource pool details page, click the Plug-ins tab. You can view the information about all plug-ins of the resource pool.
3. Installing a plug-in: Locate the target plug-in, click Install, configure related information, and click OK. For details, see Managing Standard Dedicated Resource Pool Plug-ins.
   Figure 8 Installing a plug-in
   

1. View plug-in details: Click a plug-in name to view its details, including the introduction and component list.
   Figure 9 Plug-in details
   

Checking the Disk Specifications of a Resource Pool

1. On the resource pool details page, choose More > Scaling in the upper right corner. On the displayed page, you can view the disk type, size, quantity, write mode, container engine space, and mount path of the system disk, container disk, and data disk.