Deploying Inference Workloads

Introduction

Inference workloads (ServingGroups) are used to deploy large language model (LLM) inference services in CCE clusters. This capability is built on Kthena ModelServing. It uses the vLLM inference engine to support single-node or multi-Role (Entry + Worker) distributed inference. It also uses gang scheduling and topology-aware policies to implement flexible compute allocation and stable service scheduling.

Prerequisites

Before deployment, ensure the following requirements are met:

• A CCE standard or Turbo cluster v1.29 or later is available.
• Volcano v1.21.7 or later is installed in the cluster.

Procedure

1. Switch to the deployment page.
   1. Log in to the CCE console.
   2. In the navigation pane, choose AI Containers. Then, click the Inference Workloads tab.
   3. Click Deploy Application.

2. Configure basic information and click Next: Configure Roles.

   Table 1 Basic settings

   Parameter	Description
   Application Name	Name of the inference application to be deployed
   AI Application Templates	Predefined application templates
   Image	When using a third-party image, ensure the pod can access the Internet. For details, see Using Third-Party Images.
   Cluster Name	The target CCE cluster
   Namespace	Namespace to which the inference workload will belong
   Pods	Number of pods of the inference workload

3. Configure the Role.

   Inference workloads use Roles to organize containers for distributed inference. Each Role contains one entry node and multiple worker nodes.

   Table 2 Role settings

   Parameter	Description
   Role Name	Custom identifier, which cannot be modified after the Role is created.
   Replicas	Number of instances to create for the Role during deployment.
   Entry Container	The entry container is the entry node of a Role. Each Role contains exactly one entry instance, which receives inference requests and coordinates computing tasks for the Role. Container Name: defaults to container-entry, identifying the container's responsibilities. Resource Quota: If the default configuration does not meet your requirements, click to adjust CPU, memory, and NPU resources. Boot Command: defines the main process of the container and executed when the container starts. The main process status determines the container lifecycle. For details, see Startup Command. Port: port used by the container to provide services.
   Worker Container	The worker container is the worker node of a Role. Each Role can contain multiple worker instances, which respond to entry scheduling and execute inference tasks. Replicas: number of instances for this Role in the same inference service group. Adjust replicas per Role to optimize compute allocation. Container Name: defaults to container-worker. Container Configuration: Select Same as entry container to reuse all configurations except the container name from the entry container. Select Custom to configure resources, startup commands, and ports independently. Boot Command: defines the main process of the container and executed when the container starts. The main process status determines the container lifecycle. For details, see Startup Command. Port: port used by the container to provide services.

4. (Optional) Configure advanced settings.

   Table 3 Advanced settings

   Parameter	Description
   Custom Inference Metrics	When enabled, Cloud Native Cluster Monitoring automatically collects metrics from inference pods. Required annotations are automatically added to the pod to facilitate metric collection. For details, see Monitoring Custom Metrics Using Cloud Native Cluster Monitoring.
   Upgrade Policy	Upgrade mode for the workload: Not configured: The upgrade policy is not used. Configured: Partition: used for grayscale release to control which ServingGroup is upgraded. If set to 0, all ServingGroups are upgraded. If set to N (N > 0), only ServingGroups with ordinals greater than or equal to N are upgraded. Max unavailable pods: maximum number of pods that can be unavailable during the upgrade. It is used to control service availability.
   Scheduling Policies	Supported policies: Gang scheduling: ensures that a group of pods (for example, multiple Roles of a distributed job) are scheduled together or not at all. Not configured: Gang scheduling is not used. Min Role replicas: defines the minimum number of Role replicas. The scheduler schedules pods only when sufficient resources are available. Topology-aware Scheduling Not configured: Topology-aware scheduling is not used. Configured: Select inference pod affinity or Role affinity. Pod Affinity Tier: affinity tier for ServingGroup pod scheduling. For example, deploy a PD disaggregation group within the same network domain. Role Affinity Tier: affinity tier for Role pod scheduling. For example, deploy distributed inference pods (entry and workers) within the same network domain.
   Tolerance Policies	Using both taints and tolerations allows (but does not force) the pod to be scheduled to a node with matching taints, and controls pod eviction policies after the node is tainted. For details, see Configuring Tolerance Policies.
   Labels and Annotations	Add labels or annotations to workload pods as key-value pairs. For details, see Configuring Labels and Annotations.
   Description	Description of the inference workload.

5. Confirm the configurations and click Submit.

   On the Inference Workloads page, view the new inference workload in the Deploying state.

Helpful Links

• For more information about topology scheduling, see Network Topology-aware Scheduling.
• For more information about the underlying architecture and working rules of inference workloads, see Kthena ModelServing.