Kthena ModelServing

Architecture Design

To overcome the limitations of Kubernetes native resources (such as Deployments or StatefulSets) in multi-Role collaboration scenarios, Kthena adopts a three-layer architecture: ModelServing -> ServingGroup -> Role. Each layer serves a distinct function, and they collaborate to deliver high-performance, scalable inference services.

Table 1 Architecture layers and functions

Layer	Managed Object	Function
ModelServing	A set of ServingGroups	As a control plane throughout the lifecycle, it maintains global configuration consistency, implements revision control, and provides the capability to query the status of inference clusters. It provides a unified interface to manage global replicas and revisions.
ServingGroup	A set of Roles	As an independent functional unit of the inference services, it is a logical group that connects the top-level control and bottom-level execution. It contains all roles required to complete an end-to-end inference, ensuring the integrity and independence of the inference logic.
Role	A set of pods	As the smallest functional execution unit, it handles specific computing tasks and consists of a group of pods with deterministic IDs. It executes specific computing or distribution tasks, such as the execution of prefill and decode roles.

At the Role layer, Kthena uses a dual-pod template design to support responsibility decoupling in distributed inference scenarios and to implement refined control and collaboration of pods through the definition of specific environment variables.

• Entry pod: acts as the traffic ingress and task coordinator. It receives external requests and distributes computing tasks.
• Worker pod: serves as a compute execution unit. It executes actual tensor computation and inference tasks.

  To help you identify the roles of pods in your services, the controller automatically injects the following key environment variables into entry and worker pods:

  Table 2 Key environment variables

  Environment Variable	Description
  GROUP_SIZE	The total number of pods in the current Role instance, used for coordinating collaboration logic across pods.
  ENTRY_ADDRESS	Internal address (in the format of a headless service) of the entry pod. This address is used for communication with worker pods.
  WORKER_INDEX	Unique index of each worker pod in the current role instance, which is used for identity identification and task assignment

Prerequisites

• A CCE standard or Turbo cluster v1.29 or later is available.
• The Volcano Scheduler add-on v1.21.5 or later has been installed in the cluster. If it is not installed, you can go to the CCE console and search for and install it on the Add-ons page.

  To use auto scaling, ensure Volcano v1.22.3 or later is installed.

If Volcano is used as the scheduler, you are advised to use Volcano to schedule all workloads in the cluster to avoid scheduling conflicts.

Constraints

• ModelServing depends on the Volcano Scheduler add-on. If this add-on is not installed in the cluster, related functions will not work.
• ModelServing workloads cannot be created using the console. They must be deployed using kubectl or YAML.
• When minRoleReplicas is specified, the cluster must provide enough resources. Otherwise, jobs may remain pending for extended periods.

Key Features

Example: a ModelServing Workload (Nginx)

In this example, there are two roles: prefill and decode. The prefill role has two replicas, each containing two pods: one entry and one worker. The decode role has three replicas, each containing three pods: one entry and two workers. For details about other scheduling features, see Key Features.

apiVersion: workload.serving.volcano.sh/v1alpha1
kind: ModelServing
metadata:
  name: sample
  namespace: default
spec:
  schedulerName: volcano
  replicas: 1  # servingGroup replicas
  recoveryPolicy: RoleRecreate # Recovery policy
  template:
    restartGracePeriodSeconds: 60
    gangPolicy: # Gang scheduling policy
      minRoleReplicas:
        prefill: 2
        decode: 1
    roles:
      - name: prefill
        replicas: 2       # role replicas
        entryTemplate:
          spec:
            containers:
              - name: worker
                image: nginx:latest
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                  requests:
                    cpu: 250m
                    memory: 512Mi
            imagePullSecrets:
            - name: default-secret
        workerReplicas: 1
        workerTemplate:
          spec:
            containers:
              - name: worker
                image: nginx:latest
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                  requests:
                    cpu: 250m
                    memory: 512Mi
            imagePullSecrets:
            - name: default-secret
      - name: decode
        replicas: 3  # role replicas
        entryTemplate:
          spec:
            containers:
              - name: worker
                image: nginx:latest
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                  requests:
                    cpu: 250m
                    memory: 512Mi
            imagePullSecrets:
            - name: default-secret
        workerReplicas: 2
        workerTemplate:
          spec:
            containers:
              - name: worker
                image: nginx:latest
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                  requests:
                    cpu: 250m
                    memory: 512Mi
            imagePullSecrets:
            - name: default-secret

Example: Automatic Scaling (Using vLLM)

Kthena decouples policies from targets through two core custom resources (CRDs), enabling flexible autoscaling configurations:

• AutoscalingPolicy: defines scaling policies, including monitoring metrics, target values, stable/emergency mode thresholds, and observation windows.
• AutoscalingPolicyBinding: defines the scaling target, binds a policy to a specific ModelServing or Role, and configures constraints such as replica limits and the metrics scrape endpoint.

Pods must expose service metrics in a Prometheus-compatible format (for example, vLLM's built-in /metrics endpoint). The Autoscaler periodically scrapes these metrics to drive scaling decisions.