CPU Core Pinning for Inference

Overview

In LLM inference deployment scenarios, the default CPU scheduling policy of the operating system dynamically migrates inference processes across different CPU cores. This leads to a decreased CPU cache hit rate and disrupts the affinity between the NPU and the CPU, thereby introducing additional scheduling overhead and impacting inference performance. For scenarios deploying LLMs using Ascend NPUs, this CPU migration overhead is particularly pronounced and can cause significant performance degradation in high-throughput inference scenarios.

The ModelArts inference deployment platform provides a CPU core pinning capability. When deploying inference services, you can choose to enable this feature. Once enabled, the platform automatically executes a core pinning script during the inference pod startup phase. This script identifies the NPU topology and CPU affinity relationships, binding inference-related processes (including underlying communication processes, core computation threads, and resource recycling threads) to designated CPU cores. This eliminates CPU migration overhead caused by the OS scheduler, thereby enhancing inference performance.

Application scenarios:

• Deploying LLM inference services using Ascend NPUs, targeting ultimate inference performance.
• Core production workloads with high demands on inference throughput and latency.
• Deploying real-time inference services using dedicated resource pools, where the goal is to eliminate the impact of CPU scheduling jitter on inference performance.

Constraints

• Model limitations: Only Snt9b and Snt9b23 support core pinning.
• Resource pool limitations: Dedicated resource pools support core pinning; public resource pools do not support core pinning.
• Specification limitations: The core pinning option is visible and can only be enabled when preset or custom specifications using eight (or a multiple of eight, using the entire node) Ascend accelerators are selected. In other scenarios, this feature is hidden.
• Operator plugin version limitations: The core pinning feature relies on a specified version of the KubeInfer (ModelArts Infers Operator) plugin, requiring version 7.6.2 or higher. If the version is too low, the core pinning feature is unavailable. For instructions on checking the plugin version, see KubeInfer (ModelArts Infers Operator).
• API compatibility notes: The core pinning capability is implemented via a new optional field (cpu_affinity), with a default value of false, ensuring zero impact on existing services.
• Performance limitations: The platform provides a core pinning acceleration solution, but cannot guarantee the exact degree of service performance improvement for every model after pinning. Use this feature as needed.

Prerequisites

• A dedicated resource pool has been created, and the resource pool node flavor belongs to the models supporting core pinning (Snt9b or Snt9b23). For details about how to create a dedicated resource pool, see Creating a Dedicated Resource Pool.
• The basic information configuration for the inference service has been completed as instructed in Configuring Service Information.
• Your account has a positive balance and sufficient resources.

Console Configuration

1. Log in to the ModelArts console. In the navigation pane, choose Model Inference > Real-Time Services.
2. Complete basic information, resource configuration, model configuration, and other settings on the deployment page.
3. In the Unit Settings area, set Unit Instance Specifications to Ascend 8-PU (or a multiple of 8, using the entire node). Set the custom specification to 8 PUs.
   Figure 1 Configuring acceleration via core pinning
   

4. Select Acceleration via core pinning. By default, it is unchecked (core pinning disabled). If this function is enabled, the system automatically identifies the NPU topology and CPU affinity and binds inference-related processes to specified CPU cores. This eliminates the CPU migration overhead caused by the OS scheduler.
5. After completing the remaining configuration, click Next to go to the confirmation page. Verify that the configuration is correct, and click Confirm Deployment to start deploying the inference service. The platform backend will automatically configure CPU core pinning, with no manual intervention required.

API Configuration

When deploying an inference service via API, set the cpu_affinity field (Boolean type; default value: false) within the inference unit configuration (unit_configs) in the request body of the deployment creation or upgrade request to control the core pinning switch.

APIs involved:

• Creating a deployment: POST /v2/{project_id}/services/{service_id}/deployments
• Upgrading a deployment: PUT /v2/{project_id}/services/{service_id}/deployments/{deployment_id}

Request body example (key fields only):

{
     "unit_configs": [
         {
             "role": "COMMON",
             "image": {
                 "source": "SWR",
                 "swr_path": "swr_path"
             },
             "count": 1,
             "cpu_affinity": true,
             "name": "role-0",
             "custom_spec": {
                 "memory": 40,
                 "cpu": 0.05,
                 "ascend": 0
             }
         }
     ]
 }

Verifying the Result

After deployment is complete, you can verify whether core pinning has taken effect using the following methods:

Method 1: Verifying the Result via the Process Status

1. Log in to the node where the inference service resides or enter the inference container.
2. Check whether the core pinning script is running:

   ```bash
   ps -ef | grep cpu_bind
   ```

   Core pinning disabled: The core pinning script is not running, and no related process output appears.

   Core pinning enabled: The core pinning script runs successfully, and its process is visible.

3. Check the CPU affinity of inference-related processes to confirm they are bound to fixed CPU cores:

   ```bash
   top
   ```

   After entering the top command, press F, move to the P option, press the Spacebar to enable it, and press ESC to return. The effect is shown in the figure below.

   Figure 2 Effect
   

   Core pinning disabled: The CPU value for the target process spans a range of all CPU cores (e.g., 0~127), indicating the process is not bound to fixed CPU cores.

   Core pinning enabled and effective: The CPU value for the target process shows a fixed CPU core range (e.g., 0~15), indicating the process is bound to designated CPU cores.

Method 2: Verifying the Result via Service Events

On the ModelArts console, go to the inference service details page. Check the service event list in the Deploy tab to confirm that the core binding init container started normally and executed successfully.

FAQs

Question 1: The core pinning option is invisible or cannot be selected

Possible Causes: The currently selected flavor is not an 8-Ascend setup (or a multiple of eight, using the entire node), or a resource pool with non-Snt9b/Snt9b23 models is being used.

Solution:

1. Verify that a dedicated resource pool is being used rather than a public pool.
2. Confirm that the selected flavor supports core pinning (Snt9b or Snt9b23), and the number of Ascend accelerators is eight (or a multiple of eight, using the entire node).
3. If the options still do not appear despite meeting all specifications, submit a service ticket to contact O&M personnel and verify whether the Operator version meets the requirements.

Question 2: Inference service deployment fails after enabling core pinning

Possible cause: The core pinning image fails to be pulled. This is usually because the core pinning script image has not been pushed to the SWR repository at the same region, or the image address in the configuration center is incorrect.

Symptom: The pod's init container fails to start due to an image pull failure, entering the ImagePullBackOff state, and the inference business container will not start.

Solution:

1. Go to the service details page in the ModelArts console, check the service event list, and confirm if it is an image pull failure.
2. Submit a service ticket to contact O&M personnel to check whether the init.container.image.url and arm.init.container.image.url configuration items in the configuration center point correctly to the SWR image address.
3. Confirm that the core pinning script image has been pushed to the SWR repository.
4. After correcting the configuration, recreate the pod to recover.

Question 3: Inference performance does not meet expectations after core pinning

Possible cause: Unreasonable CPU allocation policy, incorrect affinity, or the computational characteristics of the model itself are unsuited for core pinning optimization.

Solution:

1. Confirm whether core pinning has actually taken effect by checking the CPU affinity of inference-related processes (see Verifying the Result – "Method 1").
2. If core pinning has taken effect but performance has not improved, the model's computational characteristics may not benefit from core pinning optimization. Core pinning primarily optimizes performance fluctuations caused by CPU scheduling jitter; for compute-bound models, performance gains may be limited.

Question 4: How do I disable an enabled core pinning capability

Solution: Go to the inference service details page in the ModelArts console, enter the deployment configuration page via Upgrade Deployment or Modify Configuration, enable CPU pinning, and redeploy the service. After the pod is recreated, core pinning operations will cease, restoring original behavior.

Question 5: Will the core pinning feature take effect automatically after upgrading the Operator version

Solution: After an Operator version upgrade, running inference services remain unaffected, and their core pinning state stays unchanged. If core pinning is turned on and all requirements are met, new inference services will automatically use core pinning. Running inference services must have their pods rebuilt (for example, by upgrading the deployment) to use the core pinning capabilities provided by the new-version Operator.