Kthena ModelServing

Architecture Design

To overcome the limitations of Kubernetes native resources (such as Deployments or StatefulSets) in multi-role collaboration scenarios, Kthena adopts a three-layer architecture: ModelServing → ServingGroup → Role. Each layer serves a distinct function, and they collaborate to deliver high-performance, scalable inference services.

Table 1 Architecture layers and functions

Layer	Managed Object	Function
ModelServing	A set of ServingGroups	As a control plane throughout the lifecycle, it maintains global configuration consistency, implements revision control, and provides the capability to query the status of inference clusters. It provides a unified interface to manage global replicas and revisions.
ServingGroup	A set of Roles	As an independent functional unit of the inference services, it is a logical group that connects the top-level control and bottom-level execution. It contains all roles required to complete an end-to-end inference, ensuring the integrity and independence of the inference logic.
Role	A set of pods	As the smallest functional execution unit, it handles specific computing tasks and consists of a group of pods with deterministic IDs. It executes specific computing or distribution tasks, such as the execution of prefill and decode roles.

At the Role layer, Kthena uses a dual-pod template design to support responsibility decoupling in distributed inference scenarios and to implement refined control and collaboration of pods through the definition of specific environment variables.

• Entry pod: acts as the traffic ingress and task coordinator. It receives external requests and distributes computing tasks.
• Worker pod: serves as a compute execution unit. It executes actual tensor computation and inference tasks.

  To help you identify the roles of pods in your services, the controller automatically injects the following key environment variables into entry and worker pods:

  Table 2 Key environment variables

  Environment Variable	Description
  GROUP_SIZE	The total number of pods in the current Role instance, used for coordinating collaboration logic across pods.
  ENTRY_ADDRESS	Internal address (in the format of a headless service) of the entry pod. This address is used for communication with worker pods.
  WORKER_INDEX	Unique index of each worker pod in the current role instance, which is used for identity identification and task assignment

Prerequisites

• A CCE standard or Turbo cluster v1.29 or later is available.
• The Volcano Scheduler add-on v1.21.5 or later has been installed in the cluster. If it is not installed, you can go to the CCE console and search for and install it on the Add-ons page.

  To use auto scaling, ensure Volcano v1.22.3 or later is installed.

If Volcano is used as the scheduler, you are advised to use Volcano to schedule all workloads in the cluster to avoid scheduling conflicts.

Notes and Constraints

• ModelServing depends on the Volcano Scheduler add-on. If this add-on is not installed in the cluster, related functions will not work.
• ModelServing workloads cannot be created using the console. They must be deployed using kubectl or YAML.
• When minRoleReplicas is specified, the cluster must provide enough resources. Otherwise, jobs may remain pending for extended periods.

Key Features

• Gang Scheduling

  Kthena integrates deeply with the Volcano scheduler to support atomic, all-or-nothing scheduling. This prevents scheduling deadlocks and resource waste that occur when only some pods in a distributed inference group become ready. The scheduler checks whether the number of pods scheduled for a job meets the minimum required for execution. If the threshold is met, all job pods are scheduled simultaneously. Otherwise, none are scheduled.

  The system uses minMember to determine the minimum number of pods required for each ModelServing instance, ensuring efficient and predictable scheduling. The calculation logic is as follows:

  • Default mode: minRoleReplicas is not configured. In this mode, the system calculates the minimum number of pods based on the total replica count across all roles.

    minMember = replicas × role.replicas × (1 + role.workerReplicas)

  • Refined mode: minRoleReplicas is configured. In this mode, the system independently calculates the minimum number of pods that must be ready for each role, using index logic (0 to minRoleReplicas[roleName] - 1). This supports more flexible startup policies.

    Configuration example: In this example, the prefill and decode roles each has four replicas. The system ensures that at least two replicas per role are successfully scheduled and ready, improving overall scheduling efficiency and resource utilization.

    apiVersion: workload.serving.volcano.sh/v1alpha1
    kind: ModelServing
    metadata:
      name: sample
      namespace: default
    spec:
      schedulerName: volcano
      replicas: 1  # servingGroup replicas
      template:
        restartGracePeriodSeconds: 60
        gangPolicy:
          minRoleReplicas:
            prefill: 2
            decode: 2
        roles:
          - name: prefill
            replicas: 4 
            # ... additional role pod configuration
          - name: decode
            replicas: 4
            # ... additional role pod configuration

• Rolling Update

  Kthena supports rolling updates controlled by the Partition parameter. The update proceeds from the replica with the highest sequence number (last pod) downward to ensure an orderly and stable process. The following example uses four replicas (R-0 to R-3) to illustrate the state transitions:

  1. Initialization: All replicas run the old revision. (R-0 to R-3 are in the ready state.)
  2. Initiation: R-3 with the highest index is recreated first. (R-3 is in the restarting state.)
  3. Switchover: After R-3 becomes ready, R-2 is updated. (R-3 is of the new revision, and R-2 is in the restarting state).
  4. Progression: R-1 and R-0 are updated in sequence.
  5. Completion: All replicas run the new revision. (R-0 to R-3 are in the ready state.)

  The Partition parameter defines the update boundary. Only replicas with an index greater than or equal to the Partition value are updated. The controller proceeds to the next replica only after the current replica changes to the Running state and passes the readiness check.

  

  During a rolling update, the next replica is processed only after the current one is running and ready. If cluster resources are insufficient, the first updated replica may fail to start or become ready, blocking the entire update. Ensure sufficient resources are reserved before initiating an update so that the first replica can start successfully and prevent the rolling update from stalling.

• Recovery Policy

  This policy defines the granularity and behavior for handling faulty pods in model serving.

  • ServingGroupRecreate: Fine-grained control is performed by ServingGroup. If any pod in the group fails, all pods in the ServingGroup will be recreated to ensure strong consistency across the entire group.
  • RoleRecreate: Fine-grained control is performed by role. If any pod under a role fails, all pods under that role will be recreated.

    Configuration example: The following YAML defines a ModelServing resource with recoveryPolicy set to RoleRecreate:

    apiVersion: workload.serving.volcano.sh/v1alpha1
    kind: ModelServing
    metadata:
      name: sample
      namespace: default
    spec:
      schedulerName: volcano
      replicas: 1  # servingGroup replicas
      recoveryPolicy: RoleRecreate
      template:
      # ...

Example: a ModelServing Workload (Nginx)

In this example, there are two roles: prefill and decode. The prefill role has two replicas, each containing two pods: one entry and one worker. The decode role has three replicas, each containing three pods: one entry and two workers. For details about other scheduling features, see Key Features.

apiVersion: workload.serving.volcano.sh/v1alpha1
kind: ModelServing
metadata:
  name: sample
  namespace: default
spec:
  schedulerName: volcano
  replicas: 1  # servingGroup replicas
  recoveryPolicy: RoleRecreate # Recovery policy
  template:
    restartGracePeriodSeconds: 60
    gangPolicy: # Gang scheduling policy
      minRoleReplicas:
        prefill: 2
        decode: 1
    roles:
      - name: prefill
        replicas: 2       # role replicas
        entryTemplate:
          spec:
            containers:
              - name: worker
                image: nginx:latest
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                  requests:
                    cpu: 250m
                    memory: 512Mi
            imagePullSecrets:
            - name: default-secret
        workerReplicas: 1
        workerTemplate:
          spec:
            containers:
              - name: worker
                image: nginx:latest
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                  requests:
                    cpu: 250m
                    memory: 512Mi
            imagePullSecrets:
            - name: default-secret
      - name: decode
        replicas: 3  # role replicas
        entryTemplate:
          spec:
            containers:
              - name: worker
                image: nginx:latest
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                  requests:
                    cpu: 250m
                    memory: 512Mi
            imagePullSecrets:
            - name: default-secret
        workerReplicas: 2
        workerTemplate:
          spec:
            containers:
              - name: worker
                image: nginx:latest
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                  requests:
                    cpu: 250m
                    memory: 512Mi
            imagePullSecrets:
            - name: default-secret

Example: Automatic Scaling (Using vLLM)

Kthena decouples policies from targets through two core custom resources (CRDs), enabling flexible autoscaling configurations:

• AutoscalingPolicy: defines scaling policies, including monitoring metrics, target values, stable/emergency mode thresholds, and observation windows.
• AutoscalingPolicyBinding: defines the scaling target, binds a policy to a specific ModelServing or Role, and configures constraints such as replica limits and the metrics scrape endpoint.

Pods must expose service metrics in a Prometheus-compatible format (for example, vLLM's built-in /metrics endpoint). The Autoscaler periodically scrapes these metrics to drive scaling decisions.