NPU Topology-aware Affinity Scheduling on a Single Node

Only Snt9A nodes support small-network group affinity scheduling. Configure this function as needed.

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Overview. In the navigation pane, choose Settings. Then click the Scheduling tab.
2. Set Default Cluster Scheduler to Volcano. After Volcano Scheduler is enabled, enable small-network group affinity scheduling.
3. (Optional) In Expert mode, click Try Now and set gpuaffinitytopologyaware.weight for the cce-gpu-topology-priority plugin to specify the priority weight of each small-network group. Before configuring the priority weights, ensure that the cce-gpu-topology-priority plugin has been configured.

   ...
   tiers:
       - plugins:
           - name: priority
           - enableJobStarving: false
             enablePreemptable: false
             name: gang
           - name: conformance
       - plugins:
           - enablePreemptable: false
             name: drf
           - name: predicates
           - name: nodeorder
       - plugins:
           - name: cce-gpu-topology-predicate
           - name: cce-gpu-topology-priority  # If this parameter exists, the cce-gpu-topology-priority plugin has been configured. If not, add this parameter.
             arguments:   # Configure the priority weights.
               gpuaffinitytopologyaware.weight: 10
           - name: xgpu
       - plugins:
           - name: nodelocalvolume
           - name: nodeemptydirvolume
           - name: nodeCSIscheduling
           - name: networkresource
   ...

   Table 2 Parameters

   Parameter	Example Value	Description
   arguments.gpuaffinitytopologyaware.weight	10	Priority weight. The value ranges from 0 to 2147483647. A higher weight indicates that a node with a higher NPU usage in the group is more likely to be selected to reduce resource fragments.

4. In the lower right corner of the tab, click Confirm Settings. In the displayed dialog box, confirm the modification and click Save.

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Overview. In the navigation pane, choose Settings. Then click the Scheduling tab.
2. Set Default Cluster Scheduler to Volcano. After Volcano Scheduler is enabled, enable basic affinity scheduling without other configurations.
3. In the lower right corner of the tab, click Confirm Settings. In the displayed dialog box, confirm the modification and click Save.

Only Snt9C and Snt3P IDUO2 nodes support die affinity scheduling. Configure this function as needed.

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Overview. In the navigation pane, choose Settings. Then click the Scheduling tab.
2. In the Volcano Scheduler configuration, die affinity topology scheduling is enabled by default. You can take the following steps to verify it.
   1. Set Default Cluster Scheduler to Volcano and click Expert mode > Try Now.
      Figure 2 Expert mode > Try Now
      

   2. In the YAML file, check the parameters. Die affinity topology scheduling depends on the cce-gpu-topology-predicate and cce-gpu-topology-priority plugins, which are enabled by default in the Volcano Scheduler configuration. If the following parameters do not exist, manually configure them.

      ...
      tiers:
          - plugins:
              - name: priority
              - enableJobStarving: false
                enablePreemptable: false
                name: gang
              - name: conformance
          - plugins:
              - enablePreemptable: false
                name: drf
              - name: predicates
              - name: nodeorder
          - plugins:
              - name: cce-gpu-topology-predicate
              - name: cce-gpu-topology-priority
              - name: xgpu
          - plugins:
              - name: nodelocalvolume
              - name: nodeemptydirvolume
              - name: nodeCSIscheduling
              - name: networkresource
      ...

   3. After the cce-gpu-topology-predicate plugin is configured, the hard affinity is used by default. The following parameters can be configured for the cce-gpu-topology-predicate and cce-gpu-topology-priority plugins.

      ...
      tiers:
          - plugins:
              - name: priority
              - enableJobStarving: false
                enablePreemptable: false
                name: gang
              - name: conformance
          - plugins:
              - enablePreemptable: false
                name: drf
              - name: predicates
              - name: nodeorder
          - plugins:
              - name: cce-gpu-topology-predicate
                arguments:   # No affinity is configured
                  npu-die-affinity: none    
              - name: cce-gpu-topology-priority
                arguments:   # Configure the priority weight value, which is only applied in hard affinity.
                  gpuaffinitytopologyaware.weight: 10
              - name: xgpu
          - plugins:
              - name: nodelocalvolume
              - name: nodeemptydirvolume
              - name: nodeCSIscheduling
              - name: networkresource
      ...

      Table 3 Parameters

      Parameter	Example Value	Description
      arguments.npu-die-affinity	none	Die affinity scheduling. The options are as follows: required: Hard affinity is used. none: No affinity is configured. If this parameter is not specified, hard affinity is used by default.
      arguments.gpuaffinitytopologyaware.weight	10	Priority weight, which is only applied in hard affinity. The value ranges from 0 to 2147483647. A higher weight indicates that a node with a higher NPU usage in the die is more likely to be selected to reduce resource fragments.

   4. Click Save in the lower right corner.

3. In the lower right corner of the tab, click Confirm Settings. In the displayed dialog box, confirm the modification and click Save.