Using FlexNPU to Virtualize and Isolate NPU Resources

Core Architecture

In traditional AI architectures, applications are directly bound to underlying drivers and exclusively occupy physical accelerator resources, resulting in rigid resource allocation and inefficient scheduling. FlexNPU breaks this paradigm through a client-server distributed architecture that completely decouples API calls from physical execution. It consists of the following core components:

• flexnpu-device-plugin and flexnpu-runtime: serve as the bridge between the Kubernetes control plane and the physical hardware layer.
  • Device Plugin: dynamically partitions physical NPU resources and reports them to Kubernetes, enabling accurate scheduling and fine-grained compute allocation.
  • Runtime: As the underlying OCI interception engine, it transparently and securely injects virtualized device nodes and communication configurations into the service sandbox during container startup, ensuring strict inter-service isolation.
• flexnpu-daemon: a resident daemon process deployed on the host. It receives FlexNPU registration requests from the upper layer (for example, the Kubernetes scheduling system), and dynamically starts and manages the underlying aclserver process group (the actual physical execution endpoint). It serves as the core control plane for single-node virtualized compute scheduling.
• flexnpu-client: deployed in the service container as a dynamic link library (DLL), it enables zero-intrusion integration for upper-layer AI applications (for example, PyTorch and vLLM). It transparently intercepts underlying hardware API calls initiated by applications at the frontend, and forwards compute requests to the aclserver on the host with minimal latency through high-speed shared memory or lightweight network protocols for actual physical execution.

FlexNPU Advantages

• Zero-intrusion API interception

  Users can run native AI frameworks directly without recompiling code or modifying environment variables. aclrt intercepts all core instructions such as aclInit and aclrtMalloc at the lowest layer of the stack, converting locally executed function calls into network or shared memory requests. This significantly reduces service access costs.

• High-throughput dual-link RPC communication

  To address high-frequency AI operator dispatch and large-volume data transmission, FlexNPU implements a dual-link communication architecture:

  • TAP (TCP/IP-based): transmits control flows and lightweight state instructions, ensuring connectivity and reliability across isolated environments.
  • Shared memory: For frequent tensor data interaction, the system dynamically establishes memory mapping via /dev/shm. The client writes data directly to shared memory, and the server reads it directly, achieving zero-copy data transfer and significantly reducing communication latency.
• Cloud native-oriented containerized orchestration

  FlexNPU is designed for cloud-native environments. In OCI-compliant container runtimes such as Docker or containerd, physical devices (for example, /dev/davinciX) are exclusively managed by ACLServer within the host's security sandbox. Only virtualized communication configurations (for example, client.conf) and shared memory directories are mounted into the service container. This design achieves complete resource isolation while eliminating the risk of unauthorized physical hardware access from containers. It integrates seamlessly with custom runtimes for resource injection.

Prerequisites

You have created a CCE standard or Turbo cluster v1.29 or later.

Notes and Constraints

Precautions

• Do not uninstall the NPU add-on when containers in the cluster are using virtual NPU resources. Otherwise, workloads will become unavailable.
• Do not forcibly delete containers that use virtual NPU resources. Otherwise, there may be residual resources, affecting subsequent resource scheduling and provisioning. If you have forcibly deleted the containers, take the following steps:
  1. Check all nodes where NPU virtualization is enabled. Run the flexnpu-smi info command on each node and compare all entity-id values with the UIDs of all workload pods that use virtual NPU resources on the current node.
  2. If an entity-id exists but the pod UID does not exist, the NPU virtualization device is a residual device. In this case, run the following command to clear the device:

     flexnpu-smi unregister -u <FlexNPU-ID-of-the-residual-device>

     

• After a node is restarted unexpectedly, a pod that uses virtual NPU resources may be in the Execution failed state due to the delay in reporting virtual NPU resources. This is normal and can be ignored.
• After installing the CCE AI Suite (Ascend NPU) add-on, you can upgrade the add-on version if needed.

  If NPU virtualization is enabled for CCE AI Suite (Ascend NPU), ensure that there are no workloads that are using virtual NPU resources on the nodes before performing a version upgrade. If workloads are using virtual NPU resources on NPU nodes, drain those nodes before upgrading the add-on. For details, see How Do I Drain NPU Nodes After Upgrading the CCE AI Suite (Ascend NPU) Add-on?

Step 1: Install CCE AI Suite (Ascend NPU) and Enable FlexNPU

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Add-ons. In the right pane, find the CCE AI Suite (Ascend NPU) add-on and click Install.
3. On the add-on installation page, select version 2.4.3.
4. Click Install Using YAML. In the spec.values.custom area, add enable_flexnpu and set its value to true. Then, click Submit.

   

5. On the Add-ons page, view details about the installed add-on. Ensure that the add-on pods are in the running state.

Step 2: Install and Configure the Volcano Scheduler

1. In the navigation pane, choose Add-ons. On the right, find the Volcano Scheduler add-on, select version 1.21.17 or later, and click Install. On the Add-ons page, check the status of Volcano Scheduler and ensure that it is running.
2. In the navigation pane, choose Settings. Switch to the Scheduling tab and set the default scheduler to Volcano.
3. Click Try Now on the right of Expert mode. On the Expert Mode (Volcano Scheduler) panel, add the following content to enable the FlexNPU capability, and click Save.

   

    - arguments:
        scheduleMode: binpack
      name: flexnpuaffinity

   Table 1 Parameters

   Parameter	Description
   scheduleMode	Scheduling mode, which determines how tasks are allocated to NPU nodes. Options: binpack: fully utilizes existing NPUs and avoids scheduling pods to idle NPUs. It scores nodes based on resource usage. Higher usage results in higher scores and priority. This kind of algorithm aims to fully utilize NPUs and schedule application loads on some NPUs. spread: distributes application loads across different NPUs or nodes to the maximum extent, avoiding scheduling multiple pods to the same NPU. The scheduler scores nodes based on resource usage. Higher usage results in higher scores and priority. This kind of algorithm maximizes the use of idle resources in the cluster and evenly distributes application loads across the cluster.
   name	Name of the scheduling add-on. The value is fixed at flexnpuaffinity.

4. On the tab for scheduling configuration, click Confirm for the configuration to take effect.

Step 3: Prepare an Image for Containers That Use Virtual NPU Resources

Step 4: Create a Workload That Uses Virtual NPU Resources

1. In the navigation pane, choose Workloads. On the displayed page, click Create Workload.

   In this example, a workload will be created. To modify an existing workload, click the workload name and switch to the Containers tab.

2. In the basic information for container configuration, select the image created in Step 3 and set NPU Quota to NPU virtualization. You can split the compute and memory of physical NPUs and allocate them to service containers.
   • Equal NPU distribution disabled

     Application scenario: single-card splitting, where the resources of a physical NPU are split and allocated to the container

     

     Parameter	Description
     NPU Memory	The memory allocated to the container. The value must be an integer multiple of 128 MiB.
     NPU Compute	The maximum compute that can be used by a container, expressed as a percentage (%) of the total compute of a single physical NPU.

   • Equal NPU distribution enabled

     Application scenario: multi-NPU splitting, where the resources of one or more physical NPUs are evenly allocated to multiple FlexNPU instances, which will be used by the container.

     

     Parameter	Description
     NPUs	The number of FlexNPU instances that the total resources (compute and memory) are evenly distributed to. For example, if this parameter is set to 2, the container will have two FlexNPU instances.
     Total NPU Memory	The memory allocated to the container. The value must be an integer multiple of 128 MiB.
     Total Compute	The maximum compute that can be used by a container, expressed as a percentage (%) of the total compute of a single physical NPU.

   For example, you request two virtual NPUs, 256 MiB of memory, and 20% of the compute. The system will create two virtual NPUs for you. The resources allocated to each virtual NPU are as follows:

   • Compute: 10% (20%/2)
   • Memory: 128 MiB (256 MiB/2)

   The container can use the resources of two virtual NPUs.

3. Click Create Workload and check the status of each pod until they are running normally.
4. Check the UID of the pod, and then run the command highlighted in the following figure on the node to view the information about the virtual NPUs allocated to the pod.

   

   flexnpu-smi info

   Expected result: The virtual NPU resource usage of each pod is the same as configured, and the pods are independent of each other without interference.