Elastic Degradation Based on Multi-Role Deployment (P-to-D Resource Reallocation)

What Is P-to-D Resource Reallocation?

P-to-D resource reallocation is an elastic degradation and scheduling capability provided by the ModelArts inference deployment platform specifically for prefill-decode (PD) disaggregated deployment scenarios.

In LLM PD disaggregated deployments, the P role handles request preprocessing and initial token computation, while the D role handles subsequent token generation and incremental decoding. In deployment patterns such as xP1D, the D role is critical. If the D role fails and cluster resources are insufficient to recover the full workload, service continuity is completely interrupted.

Core concept

When the D role fails and remaining cluster resources are insufficient to run all replicas, the platform reduces the number of P replicas to free up resources. These freed resources are then prioritized to satisfy the replica requirements of the D role, thereby maintaining service continuity in a degraded state. Similarly, when the P role fails, it supports self-degraded recovery.

This feature is implemented based on Volcano's multi-dimensional gang scheduling capabilities. By configuring the environment variable minSubGroups to specify the minimum allowed replica group count for each role, the scheduler can perform partial scheduling based on minimum specifications when resources are constrained, rather than strictly waiting for all resources to become ready before scheduling globally.

P-to-D resource reallocation vs. traditional gang scheduling

Concepts

• Prefill (P): The prefill role workload for inference requests, responsible for initial token computation and request preprocessing.
• Decode (D): The decode role workload for inference requests, responsible for sequential token generation and incremental decoding.
• PD disaggregated deployment: An xPyD deployment pattern where P and D roles are deployed separately (e.g., 4P1D, 2P1D).
• Degradation: The process of reducing replica counts for certain roles when resources are constrained to ensure overall service availability.
• P-to-D resource reallocation: A specialized degradation scenario where P replicas are reduced to prioritize satisfying D replica requirements, allowing service recovery.
• minSubGroups: A field in subGroupPolicy of Volcano's PodGroup. This field specifies the minimum number of subgroups required for a role, controlling the minimum replica floor during degradation.
• Gang scheduling: A Volcano scheduling policy requiring a group of workloads to meet all resource requirements simultaneously before unified scheduling occurs; otherwise, no scheduling takes place. For more information about gang scheduling, see Multi-dimensional Gang Scheduling.
• Elastic gang scheduling: An extension of gang scheduling that permits the configuration of minSubGroups, enabling partial scheduling when resources are insufficient.

Application Scenarios

Constraints

• Deployment pattern constraints: Applicable exclusively to multi-role disaggregated deployment scenarios, including but not limited to configurations containing a single D replica (e.g., 4P1D, 2P1D). If degradation environment variables are configured for non-disaggregated scenarios (such as single-role deployments), the inference platform management plane records an error log and returns a failure.
• Deployment scenario constraints: The P-to-D resource reallocation feature applies exclusively to fault recovery scenarios.
• Model image constraints: The model image must support xPyD disaggregated deployment capabilities. Recovery behavior depends on the chosen recovery policy during service deployment and the model image's awareness of the ranktable and network topology. If the image lacks proper ranktable and network topology awareness, service anomalies may occur due to pod rebuild or scale-in. If the image is insensitive to ranktable and network topology, degradation causes no service impact.
• Plugin version dependencies: P-to-D resource reallocation relies on specific Volcano and KubeInfer versions in the dedicated resource pool. If Volcano version is < 1.22.x or KubeInfer is < 7.6.2, the message "Volcano plugin version: {actual_version} does not meet degradation requirements. Required volcano>=1.22.0" or "Operator plugin version: {actual_version} does not meet degradation requirements. Required Operator>=7.6.2" is displayed. Navigate to Auxiliary Tools > Plugin Marketplace to upgrade the plugin versions.
• Resource pool constraints: P-to-D resource reallocation is only supported on resource pools running Kubernetes versions v1.29.x, v1.30.x, v1.31.x, v1.32.x, v1.33.x, v1.34.x, and v1.35.x. If the version is lower than v1.29.x, Volcano cannot be upgraded to version 1.22.x or later.
• Degradation behavior limitations:
  • Degradation only supports configuring minSubGroups; it does not modify subGroupSize.
  • When both P and D roles are configured with minSubGroups, Volcano's scheduling result can be unpredictable (e.g., 5P3D might degrade to 4P3D or 5P2D, depending on Volcano's scheduling decision). You are advised to configure minSubGroups only for the P role to ensure predictable resource reallocation behavior.
• Rebuild policy constraints: The degradation supports rebuild policies at all levels. However, P-to-D resource reallocation depends on deployment replica-level rebuild.
• Reserved environment variable key constraints: ENV_MODELARTS_DEGRADATION_ENABLE and ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS are system-reserved environment variable keys. You must not define custom environment variables using these keys. If duplicate key names exist, the inference task will fail.
• Performance impact after the degradation: During degraded operation, reduced P replicas decrease prefill parallel processing capacity, potentially increasing time to first token (TTFT) and lowering throughput. This performance decline is expected behavior; you should monitor degradation status and scale out resources promptly to restore full performance.

Feature Configuration

Configuration method: Configuration is supported exclusively via environment variables through the inference deployment platform console.

In the deployment creation or deployment upgrade page, add environment variables under unit settings, enable automatic rebuild under more settings, and select deployment replica rebuild. For details about P and D role configurations, see Configuration Examples.

Figure 1 Adding environment variables

Figure 2 Selecting the deployment replica rebuild policy

Feature configuration principles:

The P-to-D resource reallocation feature passes degradation configurations via environment variables:

1. Management plane: When creating or updating a deployment, the inference platform's management plane parses user-configured environment variables to retrieve the degradation switch and minSubGroups value, writing these configurations into the corresponding kubeinfer definition.
2. Service plane: Reads the degradation configuration from the kubeinfer CR annotations and sets the minSubGroups field for the corresponding role when creating the PodGroup.
3. Scheduling plane (Volcano): Performs elastic scheduling based on the minSubGroups value in subGroupPolicy of the PodGroup when resources are constrained, adhering to the minimum required replica group count.

Typical configuration:

To reduce P replicas and preferentially meet the requirements of D replicas, it is recommended that only the degradation switch for the P role be configured:

P nodes: Enable P degradation and retain a minimum floor of one P replica:

ENV_MODELARTS_DEGRADATION_ENABLE = true 
ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS = 1

Configuration Examples

Example 1: 4P1D P-to-D resource reallocation configuration

Scenario: 4P1D disaggregated deployment service configured with P-to-D resource reallocation capabilities, maintaining a minimum floor of one replica for the P role.

P-role environment variables:

ENV_MODELARTS_DEGRADATION_ENABLE = true 
ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS = 1

D-role environment variables: Leave unconfigured.

D-role rebuild: Enable automatic rebuild and select deployment replica rebuild.

Degradation effect:

• Normal operation: 4P1D; resources are sufficient, and full scheduling is applied.
• Degraded operation (D fault recovery): 2P1D; when resources are constrained, P degrades to two replicas while D recovers one replica.
• Minimum degraded operation: 1P1D; under severe resource constraints, P degrades to one replica while D recovers one replica.

Example 2: 2P1D P-to-D resource reallocation configuration

Scenario: 2P1D disaggregated deployment service maintaining a minimum floor of one replica for the P role.

P-role environment variables:

ENV_MODELARTS_DEGRADATION_ENABLE = true 
ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS = 1

D-role environment variables: Leave unconfigured.

D-role rebuild: Enable automatic rebuild and select deployment replica rebuild.

Degradation effect:

• Normal operation: 2P1D; resources are sufficient, and full scheduling is applied.
• Degraded operation: 1P1D; when resources are constrained, P degrades to one replica while D maintains one replica.

Example 3: Independent P and D degradation control

Scenario: 5P3D disaggregated deployment service maintaining a minimum floor of two replicas for the P role and one replica for the D role.

P-role environment variables:

ENV_MODELARTS_DEGRADATION_ENABLE = true 
ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS = 2

P-role rebuild: Enable automatic rebuild and select deployment replica rebuild.

D-role environment variables:

ENV_MODELARTS_DEGRADATION_ENABLE = true 
ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS = 1

D-role rebuild: Enable automatic rebuild and select deployment replica rebuild.

Degradation effect:

When minSubGroups is configured for both P and D roles simultaneously, the Volcano scheduler determines which role's replicas to reduce, leading to potentially unpredictable outcomes (e.g., degrading to 4P3D or 5P2D). To ensure predictable P-to-D resource reallocation (prioritizing P reduction), you are advised to configure the degradation switch for the P role only.

Example 4: Configuring environment variables using an API

Scenario: Configure P-to-D resource reallocation using environment variables when creating a deployment via the ModelArts inference deployment API. For details about the inference deployment API, see ModelArts API Reference.

API request example (simplified):

 POST /v2/{project_id}/services
    Content-Type: application/json
    {
      "service": {
        "name": "llm-pd-separated-service",
        "deployment": {
          "replicas": 1,
          "roles": [
            {
              "name": "Prefill",
              "replicas": 4,
              "envs": [
                {
                  "key": "ENV_MODELARTS_DEGRADATION_ENABLE",
                  "value": "true"
                },
                {
                  "key": "ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS",
                  "value": "1"
                }
              ]
            },
            {
              "name": "Decode",
              "replicas": 1,
              "envs": [],
              "recovery":"Instance"
            }
          ]
        }
      }
    }

Result Verification

After completing the configuration, perform the following steps to verify the environment variables:

1. Log in to the ModelArts console and choose Model Inference > Real-Time Inference.
2. Click the target service name. On the service details page that is displayed, switch to the Deploy tab. Select a deployment card and view environment variables in the unit settings area. Ensure that ENV_MODELARTS_DEGRADATION_ENABLE of the P role is set to true and ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS is set to the expected value.

   In the unit settings area, check the automatic rebuild setting and ensure that the deployment replica rebuild policy is enabled.

After the configuration is complete, perform the following steps to view the degradation details:

1. Log in to the ModelArts console and choose Model Inference > Real-Time Inference.
2. Click the target service name. On the service details page that is displayed, switch to the Deploy tab. Select a deployment card and click View Details. On the deployment replica list page that is displayed, you can view the degradation details of the instances by checking the pod count of the replicas, or click the pod list to view detailed information.

FAQs

1. How does P-to-D resource reallocation differ from standard gang scheduling?

   Standard gang scheduling requires all workloads to meet resource requirements simultaneously before scheduling, causing prolonged service outages when resources are scarce. P-to-D resource reallocation is an elastic variant that uses minSubGroups to permit partial scheduling under resource constraints, keeping services running in a degraded state to guarantee service continuity.

2. Does the service remain in a degraded state when resources are sufficient?

   No. Degradation is inactive when resources are plentiful; services run normally at full capacity. Degradation triggers only when resources fail to meet full global requirements, and automatically restores partial or full replicas once cluster resources recover.

3. Why are results unpredictable when both P and D configure minSubGroups?

   Volcano decides which role to scale down based on its scheduling logic when both roles specify minimum bounds. For example, a 5P3D configuration may be degraded to 4P3D or 5P2D. This feature does not restrict this choice. It is strongly recommended that you configure minSubGroups only on the P role to protect D replicas from reduction.

4. How do I restore full replicas after degradation?

   The service runs stably in a degraded state until available resources recover, at which point remaining replicas are restored. You can also expand cluster resources (add nodes, release other services) to satisfy full scheduling requirements.

5. What is the difference between setting minSubGroups to 0 versus setting it to the replica count?
   • minSubGroups = 0: Subgroup checks are bypassed, rendering degradation ineffective (not currently supported).
   • minSubGroups = replica count: Equivalent to full gang scheduling at the role level, leaving no degradation headroom.

   You are advised to set this to a value ≥ 1 but less than the total replica count (e.g., setting minSubGroups=1 or 2 for a 4P setup).

6. What happens if degradation environment variables are set in non-disaggregated scenarios?

   The inference platform service plane checks if the deployment is multi-role disaggregated when creating PodGroups. If degradation environment variables are configured for non-disaggregated scenarios (such as single-role deployments), the inference platform management plane records an error log and returns a failure.

7. What happens if environment variable values are invalid (e.g., negative numbers, non-numeric values)?

   The management plane strictly validates input values. Invalid values (negative numbers, zero, non-numeric values, exceeding replica count) trigger error logs and return failures, causing the deployment/update request to fail.

8. What is the performance impact of P-to-D resource reallocation?

   Performance is unaffected when resources are sufficient. Once degradation triggers, reduced P replicas lower prefill parallel processing capacity:

   • The TTFT can be increased.
   • The inference throughput may decrease.
   • However, decode capabilities remain unaffected, and the subsequent token generation speed is not affected.

   You should monitor degradation status via metrics and expand capacity promptly.

9. How do I disable P-to-D resource reallocation?

   Update the deployment to remove or modify the environment variables:

   1. Set ENV_MODELARTS_DEGRADATION_ENABLE to false or remove the variable.
   2. Remove the ENV_MODELARTS_DEGRADATION_MIN_SUB_GROUPS variable.
   3. Upon updating, the platform stops setting minSubGroups, reverting the PodGroup to default gang scheduling behavior.
10. What should I do if the Volcano version does not meet requirements?

    If Volcano version is < 1.22.x or KubeInfer is < 7.6.2, the message "Volcano plugin version: {actual_version} does not meet degradation requirements. Required volcano>=1.22.0 or "Operator plugin version: {actual_version} does not meet degradation requirements. Required Operator>=7.6.2" is displayed. Upgrade the Volcano plugin in Auxiliary Tools > Plugin Marketplace to enable P-to-D resource reallocation functionality.

11. How does P-to-D resource reallocation interact with fault recovery strategies?

    Degradation links with fault recovery strategies, leading to different expectations:

    • Deployment replica rebuild: If normal-running replicas post-exception still satisfy degradation requirements, new deployment replicas are not created; degradation continues until abnormal replicas drop below minSubGroups.
    • Unit rebuild: Recreates units following policy; rebuilds units only when pods are restarted; a degradation process will take place during the rebuild.
    • Unit replica rebuild: Recreates unit replicas with degradation.
    • Pod rebuild: Recreates pods with degradation.

    Degradation can occur under various recovery policies, but P-to-D resource reallocation specifically relies on the deployment replica-level recovery policy.

12. Does degradation support initial deployments?

    No. Initial deployments permit setting the minSubGroups parameter, but if resources required by the full inference units exceed remaining cluster resources at creation time, the platform logs an error and returns a failure.