Upgrading the Lite Cluster Resource Pool Driver

Description

If GPUs or Ascend resources are used in a dedicated resource pool, you may need to customize GPU or Ascend drivers. ModelArts allows you to upgrade GPU or Ascend drivers of your dedicated resource pools.

There are two driver upgrade modes: secure upgrade and forcible upgrade.

Secure upgrade: Running services are not affected. After the upgrade starts, the nodes are isolated (new jobs cannot be delivered). After the existing jobs on the nodes are complete, the upgrade is performed. The secure upgrade may take a long time because the jobs must be completed first.
Forcible upgrade: The drivers are directly upgraded, regardless of whether there are running jobs.

Notes and Constraints

The target Lite Cluster resource pool must be running, and the resource pool contains GPU or Ascend resources.

Procedure

Log in to the ModelArts console. In the navigation pane, choose AI Dedicated Resource Pools > Elastic Clusters. Click the ModelArts Lite tab to view the resource pool list.
In the resource pool list, locate the target resource pool, and choose > Upgrade Driver in the Operation column.
In the displayed dialog box, you can view the driver type, number of instances, current version, target version, upgrade mode, upgrade scope, and rolling switch of the dedicated resource pool.
- Target Version: Select a target driver version from the drop-down list.
- Upgrade mode: You can select secure upgrade or forcible upgrade.
  - Secure upgrade: Perform the upgrade when no job is running on the node. The upgrade may take a long time.
  - Forcible upgrade: Ignore the running jobs and perform the upgrade directly. This may cause the running jobs to fail.
- Rolling Mode: Once enabled, you can upgrade the driver in rolling mode. Currently, By node percentage and By instance quantity are supported.
  - By node percentage: The number of instances to be upgraded is the percentage multiplied by the total number of instances in the resource pool.
  - By instance quantity: You can set the number of instances to be upgraded in each batch.
  Different upgrade instances have different policies for selecting nodes.
  - Secure upgrade chooses nodes that are not running any services.
  - Forcible upgrade chooses nodes randomly.
  - To check if a node has any service, go to the resource pool details page. In the Nodes tab, ensure all GPUs and Ascend cards are available. If they are, the node has no services.
  - Nodes with abnormal drivers will be upgraded during a rolling upgrade, just like other nodes.
Click OK to start the driver upgrade.